Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Authors: Chendi Ge
Xin Wang
Zeyang Zhang
Hong Chen
Jiapei Fan
Longtao Huang
Hui Xue
Wenwu Zhu
Venue: ICML 2025 poster (2025)
Keywords: Multimodal large language model
Continual learning
Parameter efficient tuning
Primary area: deep_learning->large_language_models

Abstract: Continual multimodal instruction tuning is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving tasks. However, most existing methods adopt a fixed architecture, struggling with adapting to new tasks due to static model capacity. We propose to evolve the architecture under parameter budgets for dynamic task adaptation, which remains unexplored and imposes two challenges: 1) task architecture conflict, where different tasks require varying layer-wise adaptations, and 2) modality imbalance, where different tasks rely unevenly on modalities, leading to unbalanced updates. To address these challenges, we propose a novel Dynamic Mixture of Curriculum LoRA Experts (D-MoLE) method, which automatically evolves MLLM's architecture with controlled parameter budgets to continually adapt to new tasks while retaining previously learned knowledge. Specifically, we propose a dynamic layer-wise expert allocator, which automatically allocates LoRA experts across layers to resolve architecture conflicts, and routes instructions layer-wisely to facilitate knowledge sharing among experts. Then, we propose a gradient-based inter-modal continual curriculum, which adjusts the update ratio of each module in MLLM based on the difficulty of each modality within the task to alleviate the modality imbalance problem. Extensive experiments show that D-MoLE significantly outperforms state-of-the-art baselines, achieving a 15 percent average improvement over the best baseline. To the best of our knowledge, this is the first study of continual learning for MLLMs from an architectural perspective.

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Chendi Ge
Xin Wang'
Zeyang Zhang'
Hong Chen
Jiapei Fan?
Hui Xue? Wenwu Zhu Longtao Huang?
Abstract
1. Introduction
Continual multimodal instruction tuning is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving tasks. However, most exist- ing methods adopt a fixed architecture, struggling with adapting to new tasks due to static model capacity. We propose evolve the architecture under parameter budgets for dynamic task adapta- tion, which remains unexplored and imposes two challenges: 1) task architecture conflict, where different tasks require varying layer-wise adapta- tions, and 2) modality imbalance, where different tasks rely unevenly on modalities, leading to un- balanced updates. To address these challenges, we propose novel Dynamic Mixture of Curricu- lum LoRA Experts (D-MoLF.) method, which automatically evolves MLLM's architecture with controlled parameter budgets to continually adapt to new tasks while retaining previously learned knowledge. Specifically, we propose a dynamic layer-wise expert allocator, which automatically allocates LoRA experts across layers to resolve ar- chitecture conflicts, and routes instructions layer- wisely to facilitate knowledge sharing among ex- perts. Then, we propose a gradient- based inter- modal continual curriculum, which adjusts the update ratio of each module in MLLM based on the difficulty of each modality within the task to alleviate the modality imbalance problem. Exten- sive experiments show that D-MoLE significantly outperforms state-of-the-art baselines, achieving 15% average improvement over the best base- line. To the best of our knowledge, this is the first study of continual learning for MLLMs from an architectural perspective.
Multimodal Large Language Models (MLLMs) (Dai et al., 2023; Liu et al., 2023; Bai al., 2025; Zhu al., 2025), which extend conventional LLMs through integration with modality-speciffe encoders (e.g., vision, audio), have demon- strated remarkable capabilities in processing heterogeneous multimodal data. However, in real-world scenarios, pre- trained MLLMs will inevitably encounter new data as users' instructions and demands shift. To handle these new tasks, MLLMs need to be adapted, but this adaptation process can lead to catastrophic forgetting, where the model loses previously learned knowledge. As result, Continual Mul- timodal Instruction Tuning (CMIT) has recently received considerable attention (Chen et al., 2024a), with the goal of efficiently adapting MI.LMs new tasks while preserving previously learned knowledge.
Despite recent progress, most existing methods adopt fixed architecture, which limits model capacity and flexi- bility for both past and future tasks in continual learning, Common approaches include replay-based methods (Wang et al., 2024b; Song et al., 2023) and parameter regulariza- tion methods (Wang et al., 2023a; Xiang et al., 2023), both originally designed for smaller unimodal models with sim- pler input structures. However, MILMs structurally different: they consist of modality-specifie encoders, pro- jectors, and language models, forming a heterogeneous and layered architecture. The reliance on each component varies significantly across tasks, resulting in task-specific sensi- tivity across both layers and modalities. As a result, fixed- architecture MLLMs face increased difficulty in balancing knowledge retention and adaptation. These observations tivate us to move beyond static designs and explore dynamic architectural adaptation for CMIT.
To bridge this gap, in this paper, we propose to continually evolve the architecture under given parameter budgets in CMIT, aiming to adapt to new tasks with dynamic capacity while maintaining the previous knowledge, without signif- icantly increasing computational resources. The problem, remaining unexplored in the literature, is highly non- non-trivial with the following key challenges:
'Department of Computer Science and Technology, BNRist, Tsinghua University, Beijing, China "Alibaba Xin
Group, Hangzhou, China. Correspondence Wenwu Zhu
Wang <xin. <xin_wang mang@ssinghuacduu Bwwzzhu@tsigguua.cde
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
(1) Task architecture conflict. In CMIT, different tasks ex-

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
hibit varying information patterns, leading varying sensi- tivities across transformer layers in MILMM. This variation makes it challenging to determine where to apply architec- ture evolution, as certain layers may be more critical for some tasks than others.
dressing the task architecture conflict under constrained parameter budgets, and gradient-based -baseed inter-modal con- tinual curriculum that dynamically adjusts the updating between multimodal modules, effectively mitigating the modality imbalance.
(2) Modality imbalance. In CMIT, the reliance on different modalities varies by task, which may cause one modality to dominate the learning process. This imbalance poses challenges in achieving balanced updates across modalities, often resulting in under-optimized training for the modules of different modalities in the MLLM.
Extensive experiments show that D-MoLE significantly outperforms state-of-the-an methods in CMIT, achieving a 15% average improvement over the best baseline in task adaptation and knowledge retention.
2. Problem Formulation
To address these challenges, we propose a novel Dynamic Mixture of Curriculum LoRA Experts (D-MoLE) method for CMIT, which is able to automatically evolve MLLM's ar- chitecture within controlled budgets continually adapt to new tasks without forgetting previously learned knowledge. Specifically, we propose a dynamic layer-wise expert allo- cator, which automatically allocates LoRA experts across layers to resolve architecture conflicts. This module, based on scores generated by training-free free zero-cost proxies, opti- mizes resource allocation by applying model evolution to the most critical layers for each task, and routes instructions layer-wisely to facilitate knowledge sharing among experts. Then, we propose a gradient-based inter-modal continual curriculum, which dynamically adjusts the update ratio be- tween the language model and modality encoder during the model's architecture evolution based on each task's 's diffi- culty for each modality, mitigating the modality imbalance issue. By dynamically allocating LoRA experts and using curriculum guide inter-modal optimization, D-MoLE pro- vides scalable and efficient solution for continual learning of MLLMs. Extensive experiments demonstrate that D- MoL_ significantly outperforms state-of-the-ar baselines, achieving a 15% average performance improvement over the best baseline. To the best of our knowledge, this is the first study of continual learning for MLLMs from an architectural perspective.
2.1. Continual Multimodal Instruction Tuning
Continual Multimodal Instruction Tuning (CMIT) refers to adapting MLLMs to sequential tasks while retaining both general and task-specific knowledge. Let {T1, T2....... Tny denote the tasks, and (Di, D2,... DN} their correspond- ing instruction data. At each time step i, the model receives new dataset DD and integrates new knowledge without for- getting prior tasks. Previously encountered data (D&) is generally inaccessible, except in replay- replay-based based methods that store selected samples in a memory buffer.
In CMIT, each dataset D, = {(t)), vj, 0 consists of textual inputs t, visual inputs v, and outputs o. The MLLM processes these multimodal instructions to generate outputs while avoiding catastrophic forgetting. The key challenge is balancing generalization across past tasks with fine-tuning for new tasks while managing the complexities of multimodal data.
2.2. CMIT with Architecture Evolution
In this paper, explore architecture evolution based approach for MILMMs continual learning. Given the limited computational resources in real-world settings, we cannot simply introduce a large number of new parameters for each task. Therefore, building on the CMIT framework outlined in Section 2.1, we define a parameter budget Bus for each task. Our problem formulation focuses on how to dynami- cally allocate these parameters to the most important layers within the MLLM under this parameter budget, ensuring efficient adaptation to new tasks.
The contributions of our work are summarized as follows:
We introduce the Dynamic Mixture of Curriculum LoRA Experts (D-MoLE), the first method study continual learning for MLLMs from an architectural perspective. D-MoLE evolves the MLLM's architecture within con- trolled budgets, enabling it to continually adapt to new tasks while retaining previously learned knowledge.
3. Preliminary Study
Based on the problem formulation introduced in Section 2.2, which focuses on optimally allocating newly introduced pa- rameters in the CMIT scenario, there are two key challenges: task architecture conflict, arising from the fact that the op- timal positions for parameter allocation differ across tasks, and modality imbalance, where different modalities have varying levels of dominance across tasks. In this section, we present our key observations regarding these challenges,
We observe the phenomena of task architecture conftict and modality imbalance in CMIT. For the task architec- ture conflict, we provide a thcoretical analysis showing that different layers exhibit varying sensitivities to differ- ent tasks, making uniform resource allocation inefficient.
We propose a dynamic layer-wise expert allocator that automatically assigns experts layer-wise across tasks, ad-
2

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
offering insights into their impact on CMIT and how they guide the design of our proposed method.
where Ww, L denotes the gradient of loss & with respect parameters Wi at layer l.
3.1. Task Architecture Conflict in CMIT
3.2. Modality Imbalance in CMIT
22-24
22-34
2.0
2.0
19:21
Relative Weight Changes (%)
Text
16-18
16-18
1.8
13-15
13-15-
1.5
Image Ratio
Text Image Rabo
10-12-
10:12
1.6
7.9
1.4
46
46
1.0
13-
13
1.2
Task ID
Task
0.5
1.0
0.8
(a) LLM
(b) Vision Encader
0.0
0.6
9
Figure I: Sensitivity of different transformer layers across various tasks for LLM and vision encoder in MLLM. Darker colors indicate higher sensitivity. The order of the task is the same as in Table
Task ID
Figure 2: Relative dynamics for each task after LoRA fine- tuning. The order of the tasks is the same as in Table 2.
Modality imbalance refers to phenomenon in multimodal learning where dominant modalities suppress weaker ones, leading to under-optimization of certain modules during training. This issue has been identified in smaller fusion- based models using probing or freezing techniques (Zhou et al., 2023; Du et al., 2021; Peng et al., 2022), and remains a concern in larger MILMMs (Wu et al., 2024a). However, due to the tightly coupled structure in MLLMs, directly probing individual modalities becomes nontrivial.
In existing MILMs, both the LLM and modality encoders consist of multiple stacked transformer layers, with each layer contributing differently to tasks due to varying feature abstraction requirements. Lower layers typically process basic features, while higher layers handle abstract represen- tations (Zhang al., 2024d; Gao al., 2024). As result, different tasks tend rely different subsets of layers for effective representation, especially when tasks differ in modality focus or semantic complexity.
To characterize modality imbalance in CMIT, track the relative weight change (i.e., gradient update magnitude) for the LLM and vision encoder across tasks. As shown in Figure 2, the update dynamics vary: in some tasks the LLM dominates, while in others the visual encoder exhibits stronger updates. This suggests that modality importance task deskddepen,,nt, and static allocation of adaptation budget across modalities is insufficient.
We quantify this task architecture conflict using the gradient norm metric (Abdelfattah et al., 2021), which measures the sensitivity of each transformer layer to different tasks. As shown in Figure 1, the sensitivity distribution varies across tasks and layers. In some cases, the lower layers in the vision encoder are more critical, while in others, higher LLM layers contribute more. This variability suggests that uniformly adding LoRA modules across layers does not effectively capture task-specific adaptation requirements.
Consequently, dynamic mechanism is needed to adjust up- date ratios based on task-specific modality dependence. This motivates our use of gradient- based inter-modal curriculum, which allocates resources adaptively based on modality sen- sitivity estimated from training-free proxies.
Such uniform allocation leads to redundancy in less-relevant layers and under-ades adaptaaiap in critical ones, resulting in inef- ficient use of parameters and degraded performance. Instead, parameter allocation should prioritize layers that contribute most to current task learning. We further support this ob- servation with a formal theoretical analysis in Appendix K, showing that tasks induce nontrivial gradient discrepancies across layers.
4. Methodology
The overall framework of D-MolE is shown in Figure 3. D- MoLE comprises two key components: the dynamic layer- wise expert allocator and the gradient-based inter-modal curriculum. This section introduces the details of each mod- ule, followed by the overall pipeline of our method.
Theorem 3.1. Consider an MILM with L transformer lay- ers trained sequentially on two distinct tasks Ta and TB- Un- der the assumption of task heterogeneity and non-collinear gradients, there exists at least one layer 1' E {I....,L} where the expected gradient norms differ:
4.1. Dynamic Layer- Wise Expert Allocator
To address the task architecture conflict issue in CMIT, intro- duced in Section 3.1, we propose the Dynamic Layer- Wise
ETa Vw. =111, + |BT Vw.. -
(1)
3

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Pretrained Multimodal
Continual Multimoda Instruction Tuning (CMIT)
Large Language Model
Large Language
Difficulty Scores from Zero-Cost Proxies
Current Task
Random Sampling
Model (LLM)
[|VW alla.
||Woisonslla""
Vision Encoder
Text
"What is the groy cat sitting on?"
1% Instruction
Embedding
Tuning Data
Task t-1
Dynamic Layer-Wise Expert Allocator
Gradient-Based
Inaccessible Post Dota
Inter-Modal Curriculum
"What does the CT scan show?"
Layer
Task
Task
Layer
Task 1
Task 3
Total Parameter Budget
Layer
Task
Task
Layer m
Task
Task
Diff-Difficulty Modality
-.
...
Task 1
Layer 1
Task 1
Task
Task
Layer
Task
2
"Provide one-
LLM
Vision Encoder
sentence caption for the provided image."
Transformer Layers
Transformer Layers
Low- Diffficlfltyy Modality
Oold Old LoRA Expert (Freeze)
Newlly Allocated LoRA Expert (Trainable)
Figure 3: Overall framework of D-MoLE. During training on task t, all data from previous tasks (1 to 1) remains inaccessible. D-MoLE first samples a small subset of multimodal instruction tuning data (e.g., 1%) from the current task's 's dataset. These samples are used to compute difficulty scores for each transformer layer in both the LLM and vision encoder using training- faining-free zero-cost proxies. The difficulty scores then guide the gradient-based bassed inter-modal curriculum and determine the optimal layer-wise expert allocation architecture for the current task. The parameters of the pretrained MLLM and the LoRA experts from previous tasks remain frozen, while only the newly allocated LoRA experts are updated.
Expert Allocator module. This module leverages training- free zero-cost proxies to identify which transformer layers are most critical for the current task, thereby determining where introduce new parameters.
matrices in LoRA.
The first term represents the output from the pretrained MLLM weights, while the third term provides the update for adapting the current task T, using the newly allocated LoRA expert. The second term aggregates contributions from LoRA experts trained on previous tasks. Inspired by Mixture of LoRA Experts (MoLE) (Wu et al., 2024c), we use this term to enable knowledge transfer from prior tasks.
Dynamic LoRA Expert In continual learning, when a new task T arrives, we need update the MLLM model parameters to adapt to the new task. To reduce the num- ber of new parameters introduced for each task, use LoRA (Low-Rank Adaptation) (Hu et al., 2022), one of the most widely adopted methods for parameter-efficient fine-tuning (PEFT) (Ding et al., 2023). However, simply applying LoRA in the continual learning setting faces the task architecture conflict. Therefore, we propose Dynamic Mixture of Curriculum LoRA Experts (D-MoLE) for CMIT. Specifically, to adapt to the new task Te, the output of layer I in the proposed D-MoLE is formulated as
This formulation allows the model to reuse prior knowledge while adapting to new tasks. The gating function helps select relevant experts, and layer-wise allocation provides additional flexibility.
Next, we discuss how we obtain the layer-wise indicator I with the zero-cost proxies and how we obtain the gating function of (-) with the Autoencoder Router.
Zero-Cost Proxies Prior to the training of the new task, we randomly sample a small subset of the training data for task T. denoted as Duub. (e.g.. 1% of the full dataset), and use to compute the importance score via zero-cost proxies. Specifically, we utilize the gradient norm as zero-cost proxy, applied to the pretrained model. The entire model remains unfrozen, and we perform one forward and one backward pass on this small subset Daub,t to calculate the Lg-norm of the gradients for each layer, without updating any parameters.
t-1
fi(x) wix
Σ if (x) AWIX 1. AW/x,
Aw/=B/A for tI.
(2)
where the pretrained MILMM weights at layer I are denoted by Wo The binary indicator 1/+ determines whether LoRA expert is allocated to layer for task k and the gating function of (x) controls the activation of the k-tth expert based on the input x. The low-rank update for task is given by AW; B/A where Bj and Af are trainable
The gradient norm for a given layer 1 during task To is

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
computed
shared input/output dimensions. To train the autoencoder, we extract representative features from each multimodal instruction sequence: image features from the vision en- coder and text features w from the LLM. Max-pooling is applied to obtain fixed-length vectors:
@C(Dsub.t) sub.tt)
VW 12 =
(3)
aw
where ((Dsub,)) is the loss function computed on Dsab,t W is the pretrained weight of layer I, and @c(Pus.) is the DWT gradient of the loss with respect to that weight computed on the sub dataset sub.e.
ypooled
max Vis I
W pooled
Wi+
(6)
I.....MM
which are concatenated as
Layer- Wise Expert Allocation Indicator Based the computed gradient norms for task Tr, we rank the trans- former layers according to their sensitivity (i.e., the mag- nitude of the gradient) for both the LLM and the vision encoder. Let RM represent the ranked list of layers for module M E (LLM, Vision) during task Te-
2 = woocaattVvainn (pooled: Wpooled), )
(7)
and used as input to the autoencoder. The reconstructed output is
z Autoencoder
(8)
We define binary indicator variable I for each layer defined as
which is trained to minimize the mean squared reconstruc- tion loss:
S (z) CMSE(Z, 2').
f1 00, otherwise.
a LoRA expert is allocated to layer I,
1 ===1
(4)
Lower reconstruction error implies stronger similarity to task L, allowing the autoencoder to capture its specific data distribution.
The allocation of LoRA experts is subject to budget con- straints By for module M in task Tr. LoRA experts are as- signed to the top-ranked layers within these budgets. Specif- ically, a LoRA expert allocated to a layer I during task Tr based on the following criteria, defined by
During training, since the task identity is known, we pass data from DD through all previous autoencoders (Autoencoder Jk=1 and identify the prior task with the lowest reconstruction error. The corresponding expert is ac- tivated along with the new expert for Te to promote transfer.
===1
M I (rank(1, Tr) < B/g for I E RM4),
(5)
During evaluation, task identity is unknown. We compute reconstruction losses across all autoencoders and identify relevant tasks using their thresholds:
where I is the characteristic function, which outputs a if the condition met and 0 otherwise, ra T.) denotes the rank of layer in the ordered list of layers based on their Il for task Ti.
gradient norms
(t Ci (z) S Te).
(10)
This layer-specific allocation strategy ensures that LoRA modules are placed where they are most needed, based on task -seciiii gradient signals. Each expert can be viewed as a modular unit specialized for one task (Wang et al., 2025; Pan et al., 2025), and their placement automatically deter- mined without manual tuning, aligning with the automated model design paradigm (Zhang et al., 2021; Ge et al., 2025).
These candidate tasks are ranked by reconstruction error:
Rank(t) Rank((..c (z)), t E R,
(11)
and the top-ranked ones are selected for expert activation. The gating function then determines activation at each layer:
if fx E (oan((a)(())))) gf = 10 otherwise.
1,
Autoencoder Router and Gating Function To support modular expert routing across tasks, we introduce set of task-specific autoencoders. Collectively, they serve two complementary purposes: during the training phase, they identify the most relevant expert from previous tasks to enable knowledge transfer; during the evaluation phase, they determine expert activation without access to task identity and detect whether the input belongs to an unseen task via reconstruction thresholds, in which case the model falls back to the pretrained backbone.
(12)
This enables anput-adaptive expert selection and falls back to the pretrained backbone when the input does not match any known task.
4.2. Gradient-Baced Inter-Modal Continual Curriculum
To mitigate the modality imbalance issue in CMIT, intro- duced in Section 3.2, we present a Gradient-Based Inter- Modal Continual Curriculum that automatically adjusts the update ratio between the modalities when new task arrives.
Each task is associated with a lightweight 2-layer MLP autoencoder, consisting of an encoder and decoder with
5

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Table 1: A summary of dataset statistics.
Task Type
Dataset Name
Task Description
Train Size
Test Size
VizWiz-VQA (Gurari et al., 2018)
Natural Image VQA
20,523
4,319
IconQA (Lu et al., 2021)
Diagram Understanding VQA
18,946
6,315
Visual Question
(Mishra et al., 2019)
Text Understanding VQA
166,043
100,037
Answering (VQA)
KVQA (Shah et al., 2019)
Knowledge-bascl VQA
146,408
24,294
PMC-VQA (Zhang et al., 2024a)
Medical VQA
176,948
2,000
VizWiz-Caption (Gurari al., 2020)
Natural Image Captioning
117.155
7,750
Image Captioning
TexiCaps (Sidorov et al., 2020)
Text Recognition Captioning General Image Captioning
109,765
3,166
Flickr30k (Young et al., 2014)
145,000
1,000
Visual Grounding
SK-VG (Chen al., 2023b)
Scene Knowledge Visual Grounding 23,404
6,598
Task Difficuuty Assessment Inspired by curriculum learn- ing (Wang et al., 2021), we adjust the update ratio between the L.M and the vision encoder by measuring the diffi- culty of each component in relation to the current task. Specifically, we the gradient norm as the difficulty score, where M represents module in the MLLM, and M E (LLM, Vision). The difficulty score of each module M for the current task To can be formulated as
layer-wise expert allocator described in Section 4.1, which further refines the allocation strategy at the layer level, de- termining how the experts are distributed across the layers.
This inter-modal curriculum adapts both the LLM and the vi- sion encoder to the specific demands of task Tr, prioritizing the modality with greater task difficulty. By focusing adap- tation on the more sensitive module, this approach ensures continual balancing between modalities.
A sub,t) UW
Score`1 =
(13)
4.3. Overall Pipeline of D-MoLE
where Score' represents the difficulty score of module M, based the gradient magnitude of its pretrained weights M with respect to the current task T,
When a new task Te arrives, D-MoLE follows the pipeline described below. The full algorithm is in Appendix C.
W
Training Phase We first sample a small subset Duub, to compute zero-cost scores (Eq, (3)), which determine the task-specific modality difficulty and allocate the parame- ter budget Bh between the LLM and modality encoder (Eq. (14) and Eq. (15)). The dynamic layer-wise expert allocator identifies the critical layers in both the LLM and modality encoder (Eq. (11) and Eq. (4)), after which LoRA experts are assigned to the top layers. Sequence features of Daub. are extracted (Eq. (7)) and used to train an au- toencoder that helps select previous experts for knowledge transfer. During task adaptation, the pretrained MLLM weights 8 and previously allocated LoRA experts remain frozen, with only newly allocated experts trained. Both the task-specific expert and the most relevant expert from prior tasks, determined by reconstruction error rankings (z), are used for computation.
A larger score indicates that module is more sensitive to the current task and requires more updates. This dynamic ad- justment ensures that both the LLM and the vision encoder receive updates based on the task's 's requirements, balancing the adaptation effort based on their scores.
Modality Update Ratio Given total parameter budget Biotal for each task, we allocate LoRA experts to each mod- ule M based on its difficulty score. This allocation ensures that the parameter budget dynamically distributed accord- ing to each module's sensitivity to the new task. The ratios directly determine the number of LoRA experts assigned to each module, which are further refined the layer level.
The ratio of LoRA experts assigned to each module under the current task T. is computed as
Score
(14)
Evaluation Phase After training, the model is evaluated across tasks Ti TN- If the task identifier is unknown, sequence embeddings z are passed through task-specific autoencoders calculate reconstruction losses C (z). The top-2 most relevant tasks are selected using the ranking of reconstruction errors Rank(t) to process each data sample. If a reconstruction error exceeds the threshold т; for any autoencoder i, the sample is considered to belong to an unknown task and is processed by the pretrained MI..M.
Scorejum + Score Vision
Modality Parameter Budget Based on these ratios, the number of new LoRA experts allocated to module M is
By = r/1 Broul-
(15)
The new parameter budgets for the LLM, B and the vision encoder, Bission are then applied to the dynamic
6

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Table 2: AVG, Last, and BWT scores (%) of various methods on the CMIT benchmark. The best results are highlighted in bold, and the second best results are underlined.
VizWiz-Can
SK-VG
TextCaps
IconQA
OCR-VQZ
Flickr30k
VizWiz-VQA
KVQA
PMC-VQA
Method
Average
Zero-shot
50.54
19.61
61.25
50.49
44.17
66.15
47.38
42.63
37.40
46.62
Finetune
151.36
65.66
138.69
96.75
65.00
87.80
68.92
49.45
49.90
85.95
Joint-learning
151.48
60.50
137.14
87.90
63.30
86.08
69.11
48.60
44.90
83,22
AVG
Seq-FT
80.86
25.42
92.33
69.06
49.62
67.37
47.01
40.74
35.59
56.45
LwF-LoRA
67.86
16.27
86.29
68.73
46.54
67.10
52.93
41.85
38.47
54.00
EWC-LoRA
74.82
30,45
94.95
75.30
48.46
72.56
51.56
40.75
38.28
58.57
Dense MoLE
71.28
27.91
92.24
68.46
47.40
73.86
52.62
41.58
38.62
57.11
Sparse MoLE
69.56
18.72
90.98
69.18
49.62
68.98
44.92
41.95
37.46
54.60
MoLA
75.06
22.47
90.04
70.44
47.34
74.96
48.75
40.31
37.84
56.36
O-LoRA
99.07
35.25
81.24
69.26
46.67
71.76
48.17
40,56
37.16
58.79
D-MoLE
148.19
57.29
115.92
78.62
53.64
77.73
53.37
40.87
39.23
73.87
Last
Seq-FT
52.70
21.76
72.22
62.26
50.15
77.24
62.03
49.95
48.95
55.25
LwF-LoRA
56.72
3.70
68.31
63.78
44.95
77.74
65.20
48.50
47.75
52.96
EWC-LORA
59.53
22.50
72.27
82.60
48.20
80.48
68.18
49.10
45.95
58.76
Dense MoLE
50.25
21.13
75.20
58.53
48.75
77.55
65.62
49.55
48.40
55.00
Sparse MoLE
59.93
19.12
66.82
55.57
45.25
61.62
47.85
47.40
46.55
50.01
MoLA
61.96
18.76
81.96
65.45
47.20
78.71
63.08
48.85
49.55
57.28
O-LoRA
92.38
31.30
80.26
71.69
47.85
78.46
63.75
47.90
44.80
62.04
D-MoLE
148.77
61.60
131.41
93.05
62.85
78.32
69.16
46.05
48.40
82.18
BWT
Seq-FT
-79.89
-22955
-49.11
-33.33
10.84
-0.65
-8.45
-0.10
-29.45
LwF-LoRA
-94.52
-39.59
-50.62
-16.47
-12.10
-13.73
-4.05
-0.45
-28.94
EWC-LoRA
-86.68
-34.27
-43.90
-8.77
-12.12
-7.76
-2.64
-0.25
-24.55
Dense MoLE
-91.11
-40.76
-49.39
-25.42
-1.44
-11.26
-6.54
-1.35
-29.79
Sparse MoLE
-95.91
-52.64
-53.43
-22.43
-6.49
-30.43
-22.53
-2.50
-35.79
MoLA
-83.51
-45.38
-31.76
-18.54
-12.09
-12.22
-7.19
-1.20
-26.49
O-LoRA
-59.41
-17.14
-56.26
-11.87
-1066
-1.44
-2.11
-0.75
-21.31
D-MoLE
-0.06
0.01
-2.64
-1.55
-2.33
-0.25
-3.48
-1.70
-1.49
5. Experiment
Image Captioning, and Visual Grounding, as summarized in Table 1. The detailed construction process of the benchmark is provided Appendix E.
In this section. evaluate proposed D-MoLE the CMIT benchmark, providing detailed introduction of the baselines, datasets, evaluation metrics, and experimental results. To demonstrate the effectiveness of each component, we also conduct ablation studies. Due to space constraints, detailed implementation information for both our method and the baselines be found in Appendix A.
Baselines We compare our method with several baselines, all implemented using LoRA with a consistent constraint on the total number trainable parameters for fair comparison. Seq-FT serves as the vanilla baseline, applying sequential fine-tuning to the model. LwF-LoRA (Li Hoiem, 2017) using knowledge distillation to retain performance tasks. EWC-LORA (Xiang et al., 2023) is a LoRA-based implementation of EWC (Kirkpatrick et al., 2017), which pe- nalizes changes to critical parameters to mitigate forgetting. Dense MoLE (Chen et al., 2024a) employs dense expert routing address forgetting, while Sparse MoLE (Dou
prior
5.1. Experimental Setups
Datasets To validate our method, we construct compre- hensive CMIT benchmark consisting of datasets from three primary task categories: Visual Question Answering (VQA),

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
al., 2024) uses sparse expert selection. MoLA (Gao et al., 2024) builds Sparse MoLE by allocating more experts to higher layers, and O-LoRA (Wang et al., 2023a) introduces orthogonal constraints to reduce task interference.
Table 3: Evaluation results on general MLLM benchmarks.
Method
MME-Sum
MMMU-Val
POPE-Sum
Zero-Shot
1876.8
35.4
87.3
Seq-FT O-LoRA D-MoLE
1513
30.7
85.8
Evaluation Metrics For each task in CMIT, we use task-specific metrics: top-1 accuracy for VQA, CIDEr score (Vedantam et al., 2015) for Image Captioning, and Intersection over Union (IoU) for Visual Grounding, where
1646
32.3
87.6
1754.6
32.7
88.1
a prediction considered correct if the loU exceeds 0.5.
to the second-best method, it achieves average improve- ments of 15.08%, 20.14%, and 19.82% on Average, Last, and B BT respectively. This demonstrates the effective- ness of our approach in mitigating forgetting over the long-term CMIT process.
To evaluate continual learning performance, we adopt the AVG, Last, and Backward Transfer (BWT) metrics. These as- sess both overall and task-wise knowledge retention, where A denotes the evaluation score (e.g., accuracy, CIDEr, or IoU) on task i after training on task t:
Compared to other Mol-E-based methods, Mol.A, which selectively allocates more LoRA experts higher layers, outperforms Sparse MoLE and Dense MoLE. This result suggests that selective allocation of LoRA experts can help in model learning.
AVG: The average score on task across all stages:
AVG
1 N
Σ A
(16)
Traditional CL methods like LwF-LoRA and EWC- LoRA, even when implemented with the PEFT approach (LoRA), struggle significantly in the CMIT setting. They introduce substantial computational overhead, such as distillation loss, making them slower to adapt to new knowledge. In contrast, O-LoRA, which applies orthog- onal constraints to LoRA experts for each task, shows better performance. However, since it continues add LoRA adapters without leveraging task-specific LoRA experts during inference, its ability to mitigate forgetting becomes limited, especially when the task gaps are large.
.Last: The score on task after training the last task:
Last,
AN,
(17)
BWT: Measures forgetting by comparing post-training performance to the initial post-task score:
N
BWT:
Σ 1=
(Ass- - Ai,))-
(18)
N
Evaluation on General Tasks We evaluate general knowl- edge retention by comparing the original pretrained MILM (Zero-shot) and the final models of Seq-FT, O-LoRA, and our proposed D-MoLE after training all nine tasks. The evaluation uses widely-adopter benchmarks MME (Fu et al., 2024), MMMU (Yue et al., 2024), and POPE (Li et al., 2023), which assess the general abilities of MLLMs. As shown in Table 3, our method outperforms the baselines, showing that not only mitigates forgetting during the CMIT process but also reduces the loss of general abilities often seen in continual learning scenarios.
Since we start from a pretrained MILMM, preserving general knowledge and zero-shot capability is crucial. We there- fore evaluate all tasks after each training step, regardless of whether they have been encountered, following the evalua- tion setting in (Yu et al., 2024).
5.2. Experimental Results
Main Results Table presents the AVG, Last, and BWT scores of different methods our CMIT benchmark. Zero- shot refers to the pretrained model's performance each task without fine-tuning. Finetune involves applying LoRA fine-tuning on each task individually from the pretrained model, outside a CL setting. Joint-learning represents the upper bound for CL, where instructions from all tasks are combined and trained together, offering the best-case sce- nario without tadk-ing constraints. The key observa- tions are follows:
5.3. Analysis of Training Efficiency
We summarize the total training time of D-MoLEand rep- resentative baselines in Table 4. In addition to improved performance, D-MoLE also offers modest gains in training efficiency. This improvement primarily stems from our se- lective placement of LoRA modules: instead of inserting LoRA into all layers, as done in O-LoRA or joint-learning baselines, our method activates LoRA only in the most sen- sitive transformer layers. This reduces the number of train-
D-MoLE outperforms most state-of-the-as approaches across the three metrics on most datasets. Compared
8

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Table 4: Total training time of our method compared to representative baselines.
6. Related Work
Continual Learning for LLMs and MLLMS Recent re- search in CL for LLMs and MLLMs (Yang et al., 2024;
Method
Total Time
Zheng al., 2025) focuses on PEFT (Ding et al., 2023). Some works (Wang al., 2022b;a; Razdaibiedina et al., 2023; Smith et al.. 2023; Zhao et al., 2024) introduce query- key prompt pools for task adaptation. However, prompt tuning reduces context length, critical for MLLMs due to extensive token usage. Existing solutions address distinct challenges: Model Tailor (Zhu et al., 2024) focuses on the pretrained knowledge forgetting of MLLM after fine-tuning specific tasks, which is different from our setting of CMIT. O-LoRA (Wang et al., 2023a) employs orthogonal regular- ization for LoRA modules Eproj (He et al., 2023) allocates task-specific multimodal projectors, Fwd-Prompt (Zheng et al., 2024) constrains gradient interference via residual pro- jections, LLaCAA (Qiao et al., 2024) leverages EMA-based updates, and Continual LLaVA (Cao et al., 2024) constructs dual instruction embeddings. Unlike these methods, our approach focuses architectural evolution to address task architecture conflicts and modality imbalances in CMIT.
Joint-learning
12.83 h
Seq-FT
13.15 h
O-LoRA
14.87 h
MoLA
23.03 h
D-MoLE
12.40 h
able parameters and backpropagation computations while maintaining adaptation capacity. The full training time com- parison can be found in Appendix H.
5.4. Ablation Study
To verify the effectiveness of each component, we compare different variants of D-MoLE on our CMIT benchmark:
vI: Only the LLM is fine-tuned, with the vision encoder frozen, to assess the impact of textual modality updates.
v2: Only the vision encoder is fine-tuned, with the LLM frozen, to assess the impact of visual modality updates.
Mixture of LoRA Experts Recent research extends the Mixture of Experts (MoE) framework (Jacobs al., 1991; Shazeer et al., 2017) by integrating LoRA (Hu et al., 2022) as experts within PEFT. MoLE (Wu et al., 2024c) imple- ments layer-wise experts with learned gating, MoA (Yu et al., 2024) uses MoLE to mitigate forgetting in VLM's incremental learning, and LoraMoF. (Dou al., 2024) de- couples experts for knowledge and tasks. MixLoRA (Shen et al., 2024) enables input-specific matrix reconfiguration, ColN (Chen et al., 2024a) employs dense MoELoRA to address forgetting in CMIT, and SMoLoRA (Wang et al., 2024c) introduces separable routing between visual under- standing and instruction following. Optimizing expert al- location is another key focus. MoLA (Gao et al., 2024) explores layer-wise allocation strategies, PMoE (Jung & Kim, 2024) concentrates experts in higher layers, and AI- phaLoRA (Qing et al., 2024) assigns experts based on layer- specific training quality. Our work focuses on dynamic ex- pert allocation for CMIT, enhancing task adaptability while keeping trainable parameters constrained.
v3: The gradient-based inter-modal continual curricu- lum is removed, treating textual and visual modalities uniformly without dynamic update adjustments.
v4: The dynamic layer-wise expert allocator removed, and LoRA experts are added uniformly across layers without task task-specific routing.
D-MoLE
vi Best Baseline
80
90
Avg (%)
80
70
60
-20
60
50
Average
Lost
BWT
Figure 4: Results of ablation experiments.
The results in Figure 4 highlight the importance of each component in D-MoLE. First, fine-tuning only the LLM (v1) or the vision encoder (v2) leads to suboptimal results, as MLLMS cannot fully utilize new task information when only one modality is updated. Next, training both without the modality balancing mechanism (v3) shows slight im- provement over v1 and v2, but still falls short of our full method. This indicates that balanced adaptation between modalities is essential for optimal performance. Finally, without dynamic LoRA expert allocation (v4), performance drops significantly across all metrics, underscoring the role of tpsk-spec expert assignment in mitigating forgetting.
7. Conclusion
In this paper, introduced D-MoLE, novel framework for CMIT that dynamically allocates LoRA experts across lay- ers using zero-cost metrics. Our method efficiently adapts to new tasks while preserving previously learned knowl- edge and mitigating modality imbalance through a gradient- based inter-modal curriculum. Experiments show that our method outperforms existing approaches in task adaptation and knowledge retention, providing a scalable and efficient solution for continual learning in MLLMs.

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Acknowledgement
Chen, H., Wang, X., Lan, X., Chen, H., Duan, X., Jia, J., and Zhu, W. Curriculum-listener:: Consistency- and complementarity-wvv
This work supported by National Natural Science Foun- dation of China No. 60.6222220, Beijing National Research Center for Information Science and Technology under Grant
audio-enhanced temporal sen- tence grounding. In Proceedings of the 31st ACM In- ternational Conference on Multimedia, pp- 3117-3128, 2023a.
No.BNR2023TD03006.
Impact Statement
Chen, Y., Wang, X., Fan, M., Huang, J., Yang, S., and Zhu, W. Curriculum meta-learning for next poi recom- mendation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2692-2702, 2021b.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be
specifically highlighted here.
Chen, Z., Zhang, R., Song, Y., Wan, X., and Li, G. Advanc- ing visual grounding with scene knowledge: Benchmark and method. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 15039 15039-15049 15049, 2023b.
References
Abdelfattah, M. S.. Mehrotra, A.. Dudziak, L., and Lane, N. D. Zero-cost proxies for lightweight nas. In Interna- tional Conference on Learning Representations, 2021.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q.. Zhu, X., Lu, L., Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. Internvl: Scaling up vision foun- dation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185- 24198, 2024b.
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. Memory aware synapses: Learning what (not) to forget. In European Conference on Computer Vision, pp. 139-154, 2018.
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Dai, W., Li, J., LI, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, N., and Hoi, S. Instructblip: Towards general- purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pp. 49250 49267,,997 2023.
Cai, J.. Wang, X., Guan, C., Tang, Y., Xu, J., Zhong, B., and Zhu, W. Multimodal continual graph learning with neural architecture search. In Proceedings of the ACM Web Conference 2022, pp. 1292-1300, 2022.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classifi- cation tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366-338 2021.
Cao, M., Liu, Y., Liu, Y., Wang, T., Dong, J., Ding, H., Zhang, X., Reid, I., and Liang, X. Continual Ilava: Con- tinual instruction tuning in large vision- languagl models. arXiv preprint 2xXx::2411.02564 2024.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem. In Interna- fional Conference on Learning Representations, 2019a.
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220 2355, 2023.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P., Torr, P., and Ranzato, M. Continual learning with tiny episodic memories. In Workshop on Multi-Task and Lifelong Reinforcement Learning, 2019b.
Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, L., Zhou, Y., Wang, X., Xi, Z., Fan, X., Pu, S., Zhu, J., Zheng, R., Gui, T., Zhang, Q. and Huang, X. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1932-1945, 2024.
Chen, C., Zhu, J., Luo, X., Shen, H. T. Song, J., and Gao, L. Coin: A benchmark of continual instruction tuning for multimodel large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a.
Chen, H., Chen, Y., Wang, X., Xie, R., Wang, R., Xia, F., and Zhu, W. Curriculum disentangled recommendation with noisy tion Processing Systems, 34:26924-2699 26936, 2021a.
Du, C., Li, Liu, Y., Wen, Z., Hua, T., Wang, Y., and Zhao, H. Improving multi-modal learning with modal teachers. arXiv preprint arXiv:2 1059, 2021.
Advances in Neural Informa-
10

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Farajtabar, M., Azizan, N., Mon, A., and Li, A. Orthogonal gradient descent for continual learning. In International Conference Artificial Intelligence and Statistics, pp. 3762-3773, 2020.
Jung, M. J. and Kim, J. Pmoe: Progressive mixture of ex- perts with asymmetric transformer for continual learning. arXiv preprint arXiv:2407.2157/ 2024.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinsse
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23 23(120):1-39, 2022.
A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcoming catastrophic forgetting in neural networks. Proceedings of the Na- tional Academy of Sciences, 1 114(13):3521-33
2017.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., and Ji, R. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2024.
Lan, X., Yuan, Y., Chen, H., Wang, X., Jie, Z., Ma, L., Wang, Z., and Zhu, W. Curriculum multi-negative augmentation for debiased video grounding. Proceedings of the AAAI Conference Artificial Intelligence, 37(1):1213-122) 1221, Jun. 2023.
Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y., Guo, X., Yang, J., and Subrahmanian, V. Higher layers need more lora experts. arXiv preprint arXiv:2402.08562, 2024.
Lee, S.-W.. Kim, J.-H., Jun, J., Ha, J.-W., and Zhang, B.- T. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems, pp. 4655 1655-4665,45 2017.
Ge, C., Wang, X., Zhang, Z., Qin, Y., Chen, H., Wu, H., Zhang, Y., Yang, Y., and Zhu, W. Behavior importance- aware graph neural architecture search for cross-domain recommendation. Proceedings of the AAAI Conference on Artificial Intelligence, 39(1 39(11):11708-1177
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang. Y., Krikun, M., Shazeer, N., and Chen, Z. {GS (hard Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Rep- resentations, 2021.
Apr. 2025.
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grau- man, K., Luo, J., and Bigham, J. P. Vizwiz grand chal- lenge: Answering visual questions from blind people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Li, Y., Du, Y., Zhou, K., Wang, J.. Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
2018.
Gurari, D., Zhao, Y., Zhang, M., and Bhattacharya, N. Cap- tioning images taken by people who are blind. In Eu- ropean Conference on Computer Vision, pp. 417-434, 2020.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelli- gence,
2017.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pp. 34892-34916, 2023.
Hc, J., Guo, H., Tang, M., and Wang, J. Continual instruc- tion tuning for large multimodal models. arXiv preprint arXiv:2311.16206, 2023.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6470-6479, 2017.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S.. Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022.
Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., and Zhu, S.-C. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021.
Huang, B., He, F., Wang, Q.. Chen, H., Li, G., Feng, 7 Wang, X., and Zhu, W. Neighbor does matter: Curriculum global Dositive-negative sampling for vision-language pre- training. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8005-8014, 2024.
Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Computation, 3(1):79-83 87, 1991.

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P. Shah, D., Du, X., Peng, F., Belyi, A., Z.ang. H., Singh, K., Kang, D., Hè, H., Schwarzer, M., Gunter, T., Kong, X., Zhang. A., Wang, J., Wang, C., Du, N., Lei, T., Wiseman, S., Lee, M., Wang, Z., Pang, R., Grasch, P., Toshev, A., and Yang, Y. Mml: methods, analysis and insights from multimodal Ilm pre-training. In European Conference on Computer Vision, pp. 304-323, 2024.
Ren, Z., Shao, Z., Song, J., Xin, H., Wang, H., Zhao, W., Zhang, L.,, Fu, 7.,, Zhu, Q., Yang, D., et al. Deepseek- prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition. arXiv preprint arXiv:2504.2 2025.
Rolnick, D., Abuja, A., Schwarz, J., Lillicrap. T., and Wayne, G. Experience replay for continual learning. In Advances in Neural Information Processing Systems, vol-
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952, 2019.
ume 32, 2019.
Rusu, A. A., Rabinowitz, N. C.,, Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks. arXiv preprint arXiv:160..000
2016.
Pan, Z., Wang, X., Zhang, Y., Chen, H.. Cheng, K. M., Wu, Y., and Zhu, W. Modular-cant cam: Modular dynamic camera- view video generation with Ilm. Proceedings of the AAAI Conference on Artificial Intelligence, 396))663336377, 2025.
Serra, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp. 4548-4557, 2018.
Shah, S., Mishra, A., Yadati, N., and Talukdar, P. P. Kvqa: Knowledge-awe visual question answering. In Proceed- ings of the AAAI Conference on Artifiiial Intelligence, volume 33, pp. 8876-8884, 2019.
Peng, X., Wei, Y., Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pp. 8238-8247, 2022.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.
Qian, Z., Wang. X., Duan, X., Qin, P., Li, Y., and Zhu, W. Decouple before interact: Multi-modal prompt learn ing for continual visual question answering. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2941-2950 2023.
Shen, Y., Xu, Z.,, Wang, Q., Cheng, Y., Yin, W., and Huang, L. Multimodal instruction tuning with conditional mix- ture of lora. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
Qiao, J., Zhang, Z., Tan, X., Qu, Y., Ding, S., and Xie, Y. Llaca: Multimodal large language continual assistant. arXiv preprint arXiv:2410. 10868, 2024.
August 2024.
Qin, Y., Wang, X., Zhang, Z., Chen, H., and Zhu, W. Multi- task graph neural architecture search with task-aware collaboration and curriculum. In Advances in Neural Information Processing Systems, volume 36, pp. 24879- 24891. Curran Associates, Inc., 2023.
Sidorov, O., Hu, R., Rohrbach, M., and Singh, A. Textcaps: A dataset for image captioning with reading comprehen- sion. In European Conference on Computer Vision, pp. 742-758, 2020.
Smith, J. S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-fre continual learning. In Pro- ceedings of the IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 11909-11919, 2023.
Qing, P., Gao, C., Zhou, Y., Diao, X., Yang, Y., and Vosoughi. S. Alphalora: Assigning lora experts based on layer training quality. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20511-20523, 2024.
Razdaibiedina, A., Mao, Y., Hou, R., Khabsa, M., Lewis, M., and Almahairi, A. Progressive prompts: Continual learning for language models. In International Confer- ence on Learning Represenaatinns, 2023.
Song, C., Han, X., Zeng, Z., Li, K., Chen, C., Liu, Z., Sun, M., and Yang, T. Conpet: Continual parameter- efficient tuning for large language models. arXiv preprint arXiv:2309. 1479,33,3 2023.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pp. 2001-2010, 2017.
Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599 2025.
12

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Vedantam, R. Zitnick, C. L.,, and Parikh, D. Cider: Consensus- based image description evaluation. In Pro- ceedings of the IEEE/CVF Conference Computer Vi- sion and Pattern Recognition, pp. 4575, 2015.
Wang. Z., Che, C., Wang, Q., Li, Y., Shi, Z., and Wang, M. Separable mixture of low-rank adaptation for continual vi- sual instruction tuning, arXiv preprint arXiv:2411.13949 2024c.
Wang, L.. Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362-5383 2024a.
Wu, J., Li, X., Yu, T., Wang. Y., Chen, X., Gu, J., Yao, L.., Shang, J., and McAuley, J. Commit: Coordinated instruction tuning for multimodal large language models. arXiv preprint 2tii::400.22055.. 2024a.
Wang, X., Chen, Y., and Zhu, W. A survey on curriculum learning. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 44(9):4555 4576, 2021.
Wu, T., Luo, L., Li, Y.-F., Pan, S., Vu, T.-T., and Haffari, G. Continual learning for large language models: A survey, arXiv preprint arXiv:2402.00133
2024b.
Wang, X., Chen, T., Ge, Q.. Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., and Huang, X. Orthogonal subspace learning for language model continual learning. In Find- ings of the Association for Computational Linguistics: EMNLP 2023. pp. 10658-10671, 2023a.
Wu, X., Huang. S., and Wei, F. Mixture of lora experts. In International Conference on Learning Representations, 2024c.
Xiang, J., Tao, T., Gu, Y., Shu, T., Wang, Z., Yang, Z., and
Hu, Z. Language models meet world models: Embodied experiences enhance language models. In Advances in Neural Information Processing Systems, volume 36, pp. 75392-75412. 2023.
Wang, X., Liu, Y., Fan, J., Wen, W. Xue, H., and Zhu, W. Continual few-shot learning with transformer adaptation and knowledge regularization. In Proceedings of the ACM Web Conference 2023, WWW '23, pp. 1519-1527. Association for Computing Machinery, 2023b. ISBN 9781450394161.
Yang, Y., Zhou, J., Ding, X., Huai, T., Liu, S., Chen, Q., Xie, Y., and He, L. Recent advances of foundation lan- guage models-based continual learning: survey. ACM Computing Surveys, 57(5):1-38, 2024.
Wang, X., Pan, Z., Zhou, Y., Chen, H., Ge, C., and Zhu, W. Curriculum co- co-disentangled representation learning across multiple environments for social recommendation. In Proceedings of the 40th International Conference on Machine Learning, 2023c.
Yao, X., Wang, X., Liu, Y., and Zhu, W. Continual recog- nition with adaptive memory update. ACM Transactions on Multimedia Computing, Communications and Appli- cations, 19(3s):1-15, 2023.
Wang, X., Li, H., Zhang, 7., Chen, H., and Zhu, W. Mod- ular machine learning: An indispensable path towards new-generation large language models. arXiv preprint arXiv:2504.20020, 2025.
Yao, Y., Wang, X., Qin, Y., Zhang, Z., Zhu, W., and Mei, H. Data-augmented curriculum graph neural architec- ture search under distribution shifts. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15): 16333- 16441, 2024.
Wang, Y., Liu, Y. Shi. C., Li, H., Chen, Lu, H., and Yang, Y. Inscl: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 663-677, June 2024b.
Yoon, J., Yang, E., Lec, J., and Hwang, S. J. Lifelong learning with dynamically expandable networks. In Inter- national Conference on Learning Representations, 2018.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Lin- guistics, 2:67-78, 2014.
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648, 2022a.
Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D.,, Lu, H., and He, Y. Boosting continual learning of vision-language language models via mixture-of-expeer
adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23219-232333 2024.
Wang, Z., Zhang, Z.,, Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 149, 2022b.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu:
13

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
Zhou, Y., Wang, X., Chen, H., Duan, X., Guan, C., and Zhu, W. Curriculum-nas: Curriculum weight-sharing neural architecture search. In Proceedings of the 30th ACM International Conference on Multimedia, MM '22, pp. 6792-6801, 2022.
2024.
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., and Xie, W. Development of a large-scale medical visual dataset. Communications Medicine, 4(1):277, 2024a.
Zhou, Y., Wang, X., Chen, H., Duan, X., and Zhu, W. Intra- and inter-modal curriculum for multimodal learning. In Proceedings of the s1 ACM International Conference on Multimedia, pp. 3724-3735, 2023.
Zhang, Y. Wang, X., Chen, H., Fan, J., Wen, W., Xue, H., Mei, H., and Zhu, W. Large language model with currieulum reasoning for visual concept recognition. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 6269-6280, 2024b.
Zhou, Y., Pan, Z., Wang, X., Chen, H., Li, H., Huang, Y.. Xiong, Z., Xiong, F. Xu, P., Liu, S., and Zhu, W. Curbench: curriculum learning benchmark. In Proceed- ings of the 41st International Conference on Machine Learning, 2024.
Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Wu, C., and Kuang, K. Model tailor: mitigating catastrophic forgetting in multi-modal large language models. In Pro- ceedings of the 41st International Conference on Machine Learning, 2024.
Zhang, Z., Wang, X., and Zhu, W. Automated machine learning on graphs: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial In- telligence, IJCAI-21, pp. 4704-4712. International Joint Conferences on Artificial Intelligence Organization, 2021. Survey Track.
Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Duan, Y., Tian, H., Su, W. Shao, J., et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10477 2025.
Zhang, Z.,, Zhang, Z., Wang, X., and Zhu, W. Learning to solve travelling salesman problem with porddness-adaptin curriculum. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):9136-91 93449444, 2022.
Zhang, Z., Wang, X., Qin, Y., Chen, H., Zhang, Z., Chu, X., and Zhu, W. Disentangled continual graph neural architecture search with invariant modular supernet. In Proceedings of the 41st International Conference on Ma- chine Learning, 2024c.
Zhang, Z., Zhang, C., Gao, X., Mullins, R. D., Constan- tinides, G. A., and Zhao, Y. Unlocking the global syn- ergies in low-rank adapters. In Workshop Efficient Systems for Foundation Models II @ ICML2024, 2024d.
Zhao, W., Wang, S., Hu, Y., Zhao, Y., Qin, B., Zhang, X., Yang, Q. Xu, D., and Che, W. Sapt: A shared atten- tion framework for parameter-efficient continual learning of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11641-1 11611, 2024.
Zheng, J., Ma. Q., Liu, Z., Wu, B., and Feng, H. Be- yond anti-for::: Multimodal continual instruction tuning with positive forward transfer. arXiv preprint arXiv:2401.09/.. 2024.
Zheng, J., Qiu, S., Shi, C., and Ma, Q. Towards lifelong learning of large language models: A survey. ACM Com- puting Surveys, 57(8), 2025.
14

---

Appendix of D-MoLE: Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
This appendix provides supplementary materials that complement the main text by presenting additional analyses, imple- mentation details, and discussions:
Appendix A: Details of the implementation setup, including model configurations and training protocols.
Appendix B: Additional related work on our method.
Appendix C: The detailed algorithm of D-MoLE, outlining the step-by-step pipeline of our method.
Appendix D: A summary of the notations used throughout the paper.
Appendix E: Details on the construction and characteristics of the CMIT benchmark.
Appendix Analysis of the autoencoder-based router in D-MoLE explaining its role in dynamic expert selection.
Appendix G: An analysis of the threshold selection strategy used in expert filtering, including a sensitivity study on evaluation robustness.
Appendix H: Comprehensive analysis of computational efficiency.
Appendix I: Comparison of our method with baselines under increased parameter budgets.
Appendix J: Visualizations of the dynamic architecture evolution and expert activation during training, demonstrating the
adaptability of D-MoLE.
Appendix K: Theoretical analysis of the task architecture conftict challenge in CMIT.
A. Implementation Details
We conduct our experiments using the Intern VL2-2B model (Chen al., 2024b) on eight NVIDIA A100 GPUs. We use b6 precision for training improve computational efficiency and performance on the A100 GPUs. LoRA is applied the attention layers Wrake and the projection layers W in each transformer block. To ensure a fair comparison, an equal number of trainable parameters across all baselines and our method. Specifically, we use rank of 4 for the Seq-FT baseline and a rank of 8 for our method, with an allocation budget ratio of 0.5. Since both the LLM and the vision encoder in our base model have 24 layers, the total parameter budget Buxal is set to (24 + 24) x 0.5 = 24.
maintain
The learning rate is set le-4, with 5 training epochs for the SK-VG and VizWiz- VQA datasets, and epoch for the other datasets. The batch size per device is 10 for most datasets, except for the OCR-VQA dataset, which uses batch size of 8. To reduce the number of training samples in the OCR-VQA dataset, consolidate QA pairs for the same image into multi-round conversations. The image feature dimension d, is 1024, and the text feature dimension de is 2048, resulting in an input and output dimension of 3072 for the autoencoder, with hidden dimension of 128. Each autoencoder is trained for 100 epochs with a learning rate of le-3.
B. Additional Related Work
Multimodal Large Language Models Multimodal large language models (Dai al., 2023; McKinzie al., 2024; Zhu et al., 2025; Bai al., 2025) have attracted attention for their ability handle text along with other modalities, such as images and videos. By combining LLMs with multimodal encoders, they support tasks such as image captioning and visual question answering. Pretraining MLLMs requires extensive data and computation, often taking weeks or months. As new tasks and datasets emerge, some models (Liu et al., 2023; Chen et al., 2024b) suggest mixing new task data with pretraining data to preserve generalization, though this is computationally expensive and sometimes infeasible. Our work focuses on continually adapting pretrained MILMMs to new tasks without relying on data replay.
15

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Traditional Continual Learning Methods Continual Learning (CL) (De Lange et al., 2021; Wang et al., 2024a) aims prevent catastrophic forgetting, where models lose the ability to perform previously learned tasks when adapting to new ones. Traditional CL methods include replay-based (Rebuffi et al., 2017; Chaudhry et al., 2019a;b; Rolnick et al., 2019; Lopez-Paz & Ranzato, 2017), parameter eegularization-basee (Kirkpatrick et al., 2017: Li & Hoiem, 2017: Lee et al., 2017;
Aljundi et al., 2018; Farajtabar et al., 2020), and model expansion-based approaches (Serra et al., 2018; Rusu et al., 2016; Mallya & Lazebnik, 2018; Yoon et al., 2018). Replay-based methods store or generate data to retain prior knowledge, while regularization techniques like EWC (Kirkpatrick et al., 2017) apply constraints to important parameters. Model expansion increases capacity to handle new tasks while preserving prior information.
Mixture of Experts The Mixture of Experts (MoE) model, introduced by (Jacobs et al., 1991), uses gating mechanism to determine each expert's contribution, where all experts are activated, enhancing performance but at high computational cost. Sparsely Gated MoE (SMoE) (Shazeer et al., 2017) improves efficiency by activating only a subset of experts for each input. Switch Transformer (Fedus et al., 2022) and GShard (Lepikhin et al., 2021) further optimize expert activation for large NLP and vision models. Recent research has extended the Mixture of Experts (MoE) framework (Jacobs et al., 1991; Shazeer al., 2017) by integrating LoRA (Hu et al., 2022) experts within parameter-efficient fine-tuning (PEFT) frameworks. These models are called Mixture of LoRA Experts (MoLE).
Curriculum Learning Curriculum learning organizes training in an easy-to-hard order to improve optimization and Wang et al., 2023c),
generalization (Wang et al., 2021). It has been applied in recommendation (Chen et al., multimodal grounding and pretraining (Chen et al., 2023a; Lan et al., 2023; Huang et al., 2024; Zhou et al., 2023). neural architecture search (Qin et al., 2023; Zhou et al., 2022; Yao et al., 2024), and structured prediction (Zhang et al., 2022). Recent benchmarks such as CurBench (Zhou et al., 2024) further enable systematic evaluation. Recently, curriculum learning also widely adopted in LLMs' pretraining process, e.g., Kimi K1.5 (Team et al., 2025), DeepSeek-Prover-Y V2 (Ren et al., 2025), Seed-Codes Our work introduces tradient-based inter-modal curriculum for CMIT.
C. Detailed Algorithm of D-MoLE
Algorithm 1 Training Process for Continual Learning Task Te
Input:
New task Tú pretrained MILMM, previous task-specific LoRA experts, task data Dr, parameter budgets BLIM.
Byision
Output: Updated model with new task-specific experts and task recognition
1 Step 1: Initialization &/
Duub < RANDOMSAAMPLING(2)
Compute zero-cost scores for Duub as in Eq.
8 Step 2: Modality Update and Budget Allocation */
Compute update ratio for LLM and vision encoder based task difficulty (Eq. 14)
Allocate budget for each modality as in Eq. 15
/ Step 3: Expert Allocation */
Rank layers by gradient sensitivity (Eq. 11)
Allocate LoRA experts to top layers based on the budget (Eq. 5)
1 Step 4: Autoencoder Training */
Extract pooled embeddings from Dauh (Eq. 7)
Train autoencoder for task T, using MSE loss; set threshold
/* Step 5: Training with Task-Specific Experts */
Freeze pretrained weights and previous LoRA experts Train newly allocated experts on D: 4 Step 6: Evaluation
For each sample, use autoencoders for expert selection and routing If classified as unknown, process sample with pretrained MLLM
Algorithm provides the detailed pipeline of D-MoLE for continual learning on task T, as described in Section 4.3.
The process starts with zero-cost proxies identifying the most critical layers for adaptation. A gradient- based inter-modal curriculum then computes the optimal update ratio between the L.M and vision encoder. LoRA experts are dynamically
16

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
allocated across layers based modality-ppecific parameter budgets. The model is fine-tuned the new task, updating only the newly introduced LoRA experts. During evaluation, task-specific autoencoders manage expert routing and task recognition to ensure accurate performance across tasks.
D. Notations
Table 5: The summary of the notations and their descriptions.
Notations
Descriptions
Te
The (-th task in CMIT
De Dabb Brotal
Training data subset for task T.
Data subset for zero-cost score computation Total parameter budget for each task
B
LoRA experts for module M in task T; Ratio of LoRA experts for module M
TM
M
Module (either LLM Vision encoder) Gradient for layer 1 during task Ti L2-norm of the gradient for layer
VW
V 12
R
Set of relevant tasks based on reconstruction loss Reconstruction loss threshold
Rank (t)
Rank task based on reconstruction loss Gating function for expert & at layer 7 LoRA expert allocation indicator for layer I Characteristic function, outputs if condition is met Pooled image features Pooled text features
of (a)
I
ppooled ppolled
2 concat 2x (z) LMSE ((Daub ScoreM TopK
Concatenated image and text features
Concatenation function for image and text features Reconstructed output from autoencoder for task T Reconstruction loss for input
MSE loss function
Loss on subset Duub,f
Difficulty score for module M
Function selecting top-k tasks by rank
E. Construction Details of the CMIT Benchmark
Our CMIT benchmark is designed to comprehensively evaluate the model's ability to adapt to diverse multimodal inputs and varying task complexities. The details of the datasets, instruction templates, and the construction of the data sequence for continual learning experiments are described below.
Dataset Diversity The datasets in our benchmark include wide range of data types, such natural images, text, diagrams, and medical images, ensuring coverage across various real-world scenarios. These datasets also encompass tasks of differing complexity, including diagram reasoning in IconQA and scene grounding in SK-VG, enabling a thorough evaluation of both generalization and knowledge retention capabilities.
Instruction Templates The instruction templates used in these datasets are provided in Table 6. To maintain consistency, adopt unified instruction format across all tasks, avoiding task-specific templates. For tasks with the same output format, use uniform templates to ensure fair evaluation. This standardization prevents instruction types from serving as implicit task identifiers, allowing performance to depend solely on task content and enabling more accurate evaluation of the model's ' ability to mitigate catastrophic forgetting.
Task Sequence Construction During our experiments, the sequence of datasets is randomized, alternating between different task types to create a more challenging and dynamic environment. This setup rigorously tests the model's
17

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Table 6: Instruction templates for each dataset used in the CMIT Benchmark.
Task Type
Dataset Name
Instruction Template
VizWiz- VQA (Gurari et al., 2018)
<image>
Open-ended VQA
OCR-VQA (Mishra et al., 2019)
[QUESTTON]
KVQA (Shah al., 2019)
Answer the question with single word or phrase.
<image>
IconQA (Lu al., 2021)
[QUESTION]
Multi-choice VQA
PMC-VQA (Zhang et al., 2024a)
Options: A. [Choice.1] B. [Choice_2),.
Answer with the option's 's letter directly.
VizWiz-Caption (Gurari et al., 2020)
<image>
Image Captioning
TextCaps (Sidorov et al., 2020)
Provide one-sentence caption for the provided image.
Flickr30k (Young al., 2014)
<image>
Based the Knowledge: [KNOWLEDGE]
Visual Grounding
SK-VG (Chen al., 2023b)
Please provide the bounding box coordinates of the region <ref> (OSCCRRPTIO)<</eEE...
adaptability and its capacity maintain performance previous tasks while learning new ones.
F. Analysis of Antoencoder Router in D-MoLE
Task ID
Figure 5: t-sne of the reconstructed sequence embeddings of the task-specific autoencoders.
As described in Section 4.1, we use task-specific autoencoders as routers capture and differentiate sequence embeddings for each task. Figure 5 presents t-SNE visualization of the reconstructed sequence embeddings from each task-specific autoencoder. To generate this figure, we randomly sample 500 training instances from each task and extract their multimodal instruction sequence embeddings following the procedure in Section 4.1. We then reconstruct these embeddings using their respective autoencoders and project them into 2D space using t-SNE.
The resulting visualization shows clear separation among tasks, suggesting that even a simple 2-layer autoencoder (one encoder and one decoder) is sufficient to learn discriminative representations. While some captioning tasks are relatively close, they remain largely distinguishable, indicating that the autoencoders successfully capture subtle differences between similar tasks.
We adopt this lightweight architecture to minimize routing overhead. This design choice is supported by the fact that tasks in CMIT are typically defined by dataset boundaries, with significant differences in visual domains, instruction formats, or linguistic styles. In such cases, the autoencoder can effectively distinguish tasks without requiring deep architectures. Extending our approach to more challenging settings with blurred task boundaries may be an interesting direction for future work.
18

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
G. Threshold Selection and Sensitivity
Table 7: Performance of the final checkpoint of D-MoLE under different scaling factors applied to all thresholds {Tr). where
1x corresponds to the default setting used in the main results. The best result for each column is highlighted in bold.
VizWiz-Cap
SK-VG
TextCaps
IconQA
OCR-VQA
Flickr30k
VizWiz-VOA
KVQA
PMC-VQA
Scaling Factor
Avg. Last
0.1x
148.63
61.65
127.19
92.84
53.75
78.02
69.16
43.77
48.35
80.37
0.5x
148.65
61.70
129.43
92.67
60.00
78.02
69.16
44.25
48.40
81.36
1x
148.77
61.60
131.41
93.05
62.85
78.32
69.16
46.05
48.40
82.18
2x
148.75
60.70
125.83
92.92
60.73
78.18
69.16
47.16
48.40
81.31
10x
148.69
59.45
125.70
92.19
57.79
78.02
69.16
44.69
48.40
80.34
In our routing mechanism, the threshold in Equation 10 is determined based on the reconstruction loss distribution of each autoencoder over its corresponding task's 's training set. It is set moderately above the observed loss range to account for minor distribution shifts during evaluation and to allow transferable samples from other tasks to activate relevant experts.
Empirically, we observe that the loss distributions are concentrated with few outliers, and a shared thresholding strategy works across tasks without the need for task-specific tuning. In-task and out-of-task losses typically well-separated, which is further supported by the t-SNE visualization in Figure 5.
The threshold serves two roles: (1) preventing unrelated experts from being selected purely based on top-2 ranking, and (2) enabling rejection of samples from unseen tasks. Thus, it functions as a coarse filter rather than precise routing controller, allowing for stable performance even with approximate threshold values.
To examine sensitivity, we evaluate the final checkpoint of D-MoLE under different scaling factors applied to the thresholds during evaluation. The results, shown in Table indicate that average performance remains stable across wide range of threshold values, demonstrating that D-MoLE is not sensitive the exact threshold setting long as it remains within reasonable range. Note that these experiments do not involve retraining under new thresholds, so slight variations in performance may result from mismatched expert usage patterns during inference.
H. Comprehensive Analysis of Computational Efficiency
This section provides an analysis the training time and the computational overhead introduced by proxy computation and dynamic expert routing.
Table Training time for all methods across different datasets (in hours)
VizWiz-Cap
SK-VG
TextCaps
IconQA
OCR-VQA
Flickr30k
VizWiz-VO4
KVQA
PMC-VQA
Method
Total
Joint-learning
12.83
Seq-FT
1.45
0.73
1.30
0.97
2.07
1.72
1.20
1.70
1.93
13.15
LwF-LoRA
1.45
1.02
1.77
1.38
2.82
2.35
1.67
2.33
2.67
17.50
EWC-LoRA
1.45
0.83
1.35
1.03
2.15
1.78
1.28
1.77
2.02
13.73
Dense MoLE
1.90
0.98
1.75
1.30
2.78
2.33
1.63
2.32
2.60
17.68
Sparse MoLE
2.60
1.37
0.87
1.78
3.73
3.88
2.20
3.17
3.45
23.07
MoLA
2.57
1.30
2.28
1.67
3.63
3.03
2.12
3.02
3.37
23.03
O-LoRA
1.45
0.87
1.48
1.13
2.35
1.97
1.42
1.95
2.20
14.87
D-MoLE
1.28
0.73
1.20
0.93
1.98
1.62
1.15
1.62
1.85
12.40
19

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
H.1. Training Time Analysis
Based on the training logs from our experiments, we have compiled the training time for each method, shown in Table 8. To ensure fair comparison, all methods the same number of trainable parameters, and all experiments are conducted on 8 X NVIDIA A100-SXM4-40GB GPUs.
From the results, we observe that although computational efficiency is not the primary focus of our work, our method demonstrates a slight advantage over all baselines, including vanilla LoRA Seq-FT. This advantage stems from our dynamic allocation strategy, which selectively assigns LoRA modules to the most critical layers, reducing computational latency compared to methods that apply LoRA uniformly across all layers. Additionally, our method does introduce complex auxiliary losses like O-LoRA.
H.2. Computational Cost of Proxy Computation and Dynamic Expert Routing
In Table 9, we report the time cost for preprocessing (computing the zero-cost proxy scores), training, and their ratio across tasks. The zero-cost proxy computation involves calculating the loss, performing a backward pass obtain gradients, and computing the gradient norm across layers using random 1% subset of the training data. Unlike training, this computation uses only a small subset of data and does not update model parameters, resulting in lower computational time. The KKVG, IconQA, and VizWiz-VOR tasks have lower ratios because they are trained for 5 epochs, while the zero-cost proxy scores are computed only once.
Table 9: Preprocessing time, training time, and their ratio across tasks.
VizWiz-Car Cap
SK-VG
TexiCaps
IconQA
OCR-VQA
Flickr3kk
VizWiz-VQA
KVQA
PMC-VQA
Method
Total
Preprocessing (s)
8
79
121
100
14
104
126
650
Training (s)
4623
2712
4377
3394
7157
5849
4179
5824
6657
44772
Ratio
1.91%
0.29%
1.81%
0.31%
1.69%
1.71%
0.34%
1.79%
1.90%
1.45%
For the computational cost of dynamic expert routing, the process involves two main steps: (1) computing the sequence representation using the LLM and the vision encoder in the MILM, and (2) computing the reconstruction loss using task-specific autoencoders, which are simple two-layer MLPs with an encoder and a decoder layer.
We evaluate the final model of D-MoLE on 1000 random test samples from the PMC-VQA dataset. The PMC-VQA task is multiple-choice VQA task, where the model only needs to output the selected option (i.e., a single letter). This makes the inference cost of PMC-VQA the lowest among all tasks in our experiments. Despite this minimal inference cost, the results in Table 10 show that the expert routing time accounts for only a small fraction of the total inference time, demonstrating the efficiency of our dynamic expert routing process.
Table 10: Expert routing time, inference time, and their ratio on PMC-VQA task.
Stage
Average Std (s)
Expert Routing Time
0.0552 0.0035
Inference Time
0.5225 0.0943
Ratio
10.56%
In summary, the computational overhead of using a subset to compute gradients and dynamically routing experts is minimal, with little impact on overall time and efficiency. The results in the tables demonstrate the efficiency of our approach.
20

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
I. Comparison with Baseline at Increased Parameter Budget
In this section, we analyze the performance of D-MoLE comparison to the most advanced baseline (O-LoRA) under varying trainable parameter budgets. To evaluate the impact of increasing the parameter budget, experiments were conducted with O-LoRA using different rank values (r), corresponding to x, 2x, and 4x the default parameter size. The results summarized in Table 11.
Table Performance comparison of D-MoLE and O-LoRA under varying trainable parameter budgets.
Method
Trainable Params.
Average
Last
BWT
O-LoRA (r = 4)
x
58.79
62.04
-21.31
O-LoRA (r 8)
2x
60.09
59.68
-19.95
O-LoRA 16)
4x
59.75
59.34
-20.95
D-MoLE
1x
73.87
82.18
-1.49
As shown in Table 11, increasing the number of trainable parameters slightly improves the performance of O-LoRA. However, D-MoLE consistently outperforms O-LoRA across all metrics, even with smaller parameter budgets. This suggests that D-MoLE adapts better to tasks without significant loss of previously learned knowledge.
J. Visualization of the Dynamic Training Process
To illustrate the adaptability of D-MoLE throughout continual training, we provide visualizations that capture its dynamic behavior terms of both architecture evolution and expert utilization.
Architecture Evolution Dynamics: As shown in Figure 6, we plot heatmap that depicts how task-specific LoRA experts are progressively allocated across different transformer layers during the CMIT process. This reflects the evolution of architecture over time based on the changing sensitivity patterns tasks.
Task ID
2
13
14
15
16
Layer
(a) LLM
Task ID
600
2
10
11
12 Layer
13
14
15
16
20
21
22
23
(b) Vision Encoder
Figure 6: Architecture evolution dynamics during continual multimodal instruction tuning. Each transformer layer is divided
into two subslots, corresponding to LoRA applied on attention weights W Wyko and projection weights W., Darker cells indicate that a LoRA expert is allocated to the corresponding position. The order of the task the same as in Table 2.
21

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Expert Activation Dynamics: Figure 7 presents heatmap of expert activation training steps, highlighting
how different task-specific experts selectively routed as the model encounters tasks. This demonstrates the effectiveness of the autoencoder-based gating mechanism in adapting routing decisions over time.
1
1
2
3
3
5-
5
5
6-
6
7-
7-
7
9.
9-
9
1
2
3
6
8
9
1
2
4
6
8
9
2
4
6
7
8
9
Task ID
Task ID
Task
(a) After 3 tasks
(b) After 6 tasks
(c) After 9 tasks
Figure 7: Expert activation dynamics at different stages of continual multimodal instruction tuning. Each cell indicates how frequently a task activates a given expert. Grayed cells correspond to experts which tasks are unseen. Lighter colors denote higher activation frequency. The order of the task is the same as Table 2.
These visualizations clearly show that D-MoLE does not rely on a static configuration, but continuously adapts expert allocation and usage as training progresses.
K. Theoretical Analysis of Task Architecture Conflict in CMIT
In this section, provide theoretical foundation for the challenge of task architecture conflict in continual multimodal instruction tuning (CMIT), which is described in Section 3.1. Specifically, we show that different tasks induce discrepancies in the gradient norms of specific layers, leading to conflicts during sequential training.
We formalize the conflict through the following theorem:
Theorem K.1 (Theorem 3.1). Consider an MLLM with L transformer layers trained sequentially on two distinct tasks Ta and TB. Under Assumption K.2 (task heterogeneity) and Assumption K.3 (non-collinear gradients), there exists at least one layer 1* E (1,...,II, [.] where the expected gradient norms differ:
(V., -11,, * [VW. [V., -11,
(19)
where Vw, C denotes the gradient of loss C with respect parameters Wi at layer l.
Our analysis relies on two foundational assumptions about task properties:
Assumption K.2 (Task Heterogeneity). Tasks Ta and Tg have distinct ontut-output distributions (Pr2 (z, v) # PTa (z, v))
and require different optimal parameter configurations.
Assumption K.3 CNon-Collinar Expected Gradients). For at least one layer I, the expected gradients of Ta and TB cannot be aligned by scalar scaling:
Va TR, ET Vw, + ETg [w.
(20)
We formally define speciii gradient magnitude as follows:
Definition K.4 (Gradient Norm of a Layer). For task T, the gradient norm at layer is defined as:
G(,T) A Er V [w.
(21)
where || ||2 denotes the normm
22

---

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Proof of Theorem 3.1. Assume for contradiction that all layers exhibit equal gradient norms tasks:
1I E LJ.. G(I,T)) G(L,TB).
(22)
By Assumption K.2, the distinct input-output distributions of Ta and TB necessitate different optimal parameter configura- tions. This implies the existence of least one critical layer 1 where their expected gradients differ fundamentally:
ET Vw,- C] ETB C..
(23)
Furthermore, Assumption K.3 specifies that these gradients cannot be related by scalar scaling. Formally, there exists no R such that:
E V C ET Vw...] C],
(24)
indicating directional divergence beyond mere magnitude differences.
To analyze this conflict, let gA == ETA w L] and ga = ETa Ww,, CJ. The initial assumption Eq. (22) enforces equal gradient norms:
galle |gBll>
(25)
However, the non-collinearity condition Eq. (24) implies g. gn- Geometrically, this requires both gradients lie the surface of common hypersphere while pointing in divergent directions:
gA gB
(r)
with =ll. 1z,
(26)
where (r) denotes (d -iddennsioaa hypersphere of radius
This geometric constraint creates an optimization dilemma. Sequential training on Ta and TB would demand opposing parameter updates at layer :":
W Wr. 7gA
(Task TA)
(27)
W W,- 7gB
(Task TB)
where 7 is the learning rate. Since gA and ga share equal norms but conflicting directions, these updates attempt to steer Wi- toward incompatible regions of the parameter space.
Maintaining equal gradient norms across all layers under Eq. (22) would necessitate reconciling these incompatible u requirement that violates the sequential training paradigm. Consequently, the initial assumption cannot hold, thereby proving the existence of at least one layer 1* with ||gA # |gBl12.
Remark K.5. Theorem 3.1 highlights the challenge of task architecture conflict CMIT, where sequential task training results in gradient norm discrepancies certain layers. This conflict indicates that task-specific adaptations demand distinct parameter updates. D D-Moo_Emitigates this issue by dynamically assigning LoRA experts layers with higher task sensitivity, promoting improved task adaptation during CMIT.
23


Reviews:
[{"review_id": "IRsnXNOk1J", "reviewer": "Reviewer_VwXs", "summary": "This paper presents D-MoLE, a  framework designed to tackle the challenges of continual multimodal instruction tuning (CMIT) in Multimodal Large Language Models (MLLMs). D-MoLE employs a dynamic layer-wise expert allocation strategy to overcome task architecture conflicts and a gradient-based inter-modal continual curriculum to address modality imbalances. This approach facilitates adaptive and efficient learning while maintaining a constrained parameter budget.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and encouraging feedback. We are glad that they find our paper well-structured and our approach to be inspiring. We hope our responses below help clarify the remaining points.\n\n---\n**Q1. Notation Refinement**\n\nThank you for the valuable suggestion. We will revise the notation to use bold symbols for vectors (e.g., $\\mathbf{x}$) and standard math notation for scalars (e.g., $x$) to improve clarity.\n\n---\n**Q2. Difference Between Finetune and Seq-FT in Table 2**\n\nThank you for the thoughtful question. The Finetune results refer to models trained independently on each task from scratch using LoRA fine-tuning, and evaluated only on that task. These results serve as upper bounds without any continual learning (CL) constraints. In contrast, Seq-FT is a vanilla CL baseline that performs sequential LoRA fine-tuning across the task sequence, without any specific mechanism to mitigate forgetting. As expected, it performs poorly on all three CL metrics.\n\nAll CL methods (i.e., all rows except Zero-shot, Finetune, and Joint-learning) are evaluated under the same protocol: after training each new task, the model is tested on all tasks in our benchmark. This differs from traditional CL setups, which typically evaluate only on seen tasks. Since the pretrained MLLM has zero-shot capabilities, our protocol also assesses how well this ability is retained. The details of this evaluation protocol can be found in Appendix F. We will clarify this distinction in the revised version.\n\n---\n**Q3. Training Efficiency Despite Architecture Evolution**\n\nThank you for noticing this point. We also observe that D-MoLE brings an additional benefit in training efficiency, as shown in Table 6. While we briefly discussed this below the table, we would like to elaborate on it here.\n\nThe main source of this efficiency gain lies in the selective placement of LoRA modules. Unlike methods such as joint-learning and O-LoRA, our method inserts LoRA modules only into a subset of the most sensitive transformer layers for each task. This design ensures that, although our method slightly increases the LoRA rank, the number of trainable parameters remains comparable to these baselines due to fewer insertion points.\n\nTo further illustrate the efficiency, we conduct a toy experiment comparing GPU runtime under different LoRA ranks. Specifically, we generate a random input matrix of shape 64 × 1024 and perform 10,000 iterations of multiplying it with two simulated LoRA modules of rank 4 and rank 8, respectively. The measured average runtime per iteration is summarized below:\n\n**Table 1: GPU runtime comparison for different LoRA ranks.**\n|Rank|Avg. Runtime|\n|-|-|\n|r = 4|4.40 ms|\n|r = 8|4.70 ms (+6.8%)|\n\nDespite doubling the rank, the runtime increases by only 6.8%, suggesting that in low-rank settings, the startup overhead of matrix multiplication dominates the total runtime rather than the rank itself. Since our method inserts LoRA modules into fewer transformer layers, the total number of such operations is reduced. This compensates for the slightly higher cost per operation and results in a modest overall speedup.\n\nMoreover, our architecture evolution mechanism is lightweight by design. As shown in Table 7, the preprocessing time for computing the zero-cost proxy accounts for less than 2% of the total training time.\n\nWe will make this explanation clearer in the revised version.\n\n---\n**Q4. Load Balancing in Expert Routing**\n\nThank you for the question. We do not explicitly introduce load balancing mechanisms in our routing design. Our routing mechanism is primarily intended for knowledge retention and transfer in the CL setting, rather than for inference efficiency or capacity scaling as in traditional MoE models.\n\nIn our framework, expert collapse is unlikely to occur. During training, task IDs are known and each corresponding expert is explicitly activated and updated. During evaluation, task-level routing is based on reconstruction losses from task-specific autoencoders, each trained on its own task data. As shown in Appendix G, the reconstructed embeddings form well-separated clusters, enabling reliable expert selection and preventing collapse. The router operates on the entire multimodal instruction sequence, rather than on individual modalities. Therefore, varying modality distributions do not directly affect expert activation. \n\nIn real-world deployments, especially under online CL settings, load balancing may become important. For example, some tasks may dominate the input stream, leading to expert overuse, while others remain underutilized. One possible solution is monitor expert activation frequency and incorporate regularization or usage-aware training objectives to maintain stable generalization. These strategies are orthogonal to our current design and may be worthwhile to explore in future work.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes, I checked theorem 3.1.", "experimental_designs_or_analyses": "yes, results of in sec 5.", "relation_to_broader_scientific_literature": "no much related", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "pros:\n\n1. The paper is well-structured and clearly presents the challenges of continual learning in multimodal large language models. In Section 3, two preliminary studies effectively demonstrate why traditional continual learning methods may not be suitable for MLLMs. The inclusion of theoretical analyses provides deeper insights into the observed empirical phenomena and highlights the necessity of architectural evolution in continual learning.  \n\n2. The idea of automatically evolving the model architecture by incorporating LoRA modules into MLLMs during the continual learning process is particularly inspiring. This approach expands the model's capacity to accommodate new tasks while mitigating performance degradation on previously learned tasks. The concept of gradually increasing parameters over time could become an essential strategy for enabling continual learning in large-scale models, and this paper serves as an important step toward that direction.  \n\n3. The experimental results are strong, demonstrating substantial performance gains over state-of-the-art baselines. The improvements over the benchmarks verify the effectiveness of the proposed approach and highlight its potential for advancing continual learning in multimodal scenarios.  \n\nCons:\n\nThere are no major concerns with the paper. However, there are some minor points that could benefit from clarification—please refer to the suggestions and questions for further details.", "other_comments_or_suggestions": "The notation could be refined to more clearly differentiate between scalars and vectors, ensuring consistency and readability throughout the paper. Using distinct formatting, such as boldface or arrows for vectors and standard italicization for scalars, would enhance clarity and prevent potential ambiguity in mathematical expressions.", "questions_for_authors": "1. Several continual learning baselines perform significantly worse than fine-tuning on the first task. However, Seq-FT, which appears to be functionally equivalent to fine-tuning in the initial task, shows notably lower performance. Could you clarify the reasons behind this discrepancy? Are there specific factors, such as optimization dynamics or architectural constraints, that might contribute to this difference?  \n\n2. In Table 6, the proposed method demonstrates a shorter training time compared to joint learning and O-LoRA. Given that the architecture evolution process introduces additional parameters to be trained, this result seems somewhat counterintuitive. Could you elaborate on why this occurs? Does the efficiency stem from selective parameter updates, optimized training strategies, or some other factors?  \n\n3. Regarding the routing mechanism, is there any specific design choice implemented to ensure a balanced load distribution among experts? If so, could you provide details on how the router dynamically manages workload allocation, especially in scenarios with varying modality distributions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l1ckJe6E5W", "reviewer": "Reviewer_cVka", "summary": "This paper addresses the challenge of continual multimodal instruction tuning (CMIT) for Multimodal Large Language Models (MLLMs) by proposing a novel Dynamic Mixture of Curriculum LoRA Experts (D-MoLE) method. Unlike fixed-architecture models that struggle with adapting to new tasks, D-MoLE dynamically evolves the model’s architecture within a parameter budget by allocating LoRA experts layer-wise and adjusting update ratios based on modality difficulty. Experimental results show that D-MoLE outperforms state-of-the-art baselines by 15% on average, making it the first study to tackle continual learning for MLLMs from an architectural perspective.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and positive feedback. It is encouraging that they find our paper well-written and easy to follow, our results impressive, and our method practical, intuitive, and concise. We hope the clarifications below address the reviewer’s remaining concerns.\n\n---\n**Q1. Clarification of the Two 'Average' Metrics in Table 2**\n\nThank you for the valuable suggestion. To clarify:\n\n* The *Average* column (rightmost) shows the mean performance of each method across all datasets (i.e., row-wise average).\n* The **Average** row near the top refers to one of the continual learning (CL) metrics introduced in Section 5.1, alongside Last and BWT. Their formal definitions are provided in Appendix F.\n\nTo improve clarity, we will consider renaming the CL-specific **Average** row to **AVG** in the revised version.\n\n---\n**Q2. Unexpectedly Strong Seq-FT Performance on KVQA**\n\nThank you for pointing out this observation. Seq-FT achieves relatively high accuracy on KVQA immediately after training and experiences minimal forgetting after training on PMC-VQA. These two factors together explain its higher Last and BWT scores on KVQA.\n\nWe attribute this behavior to two possible factors. First, KVQA is a relatively difficult dataset, where most methods show only modest improvements over the zero-shot baseline. CL methods that incorporate regularization mechanisms (e.g., LwF-LoRA, EWC-LoRA) may struggle to fully adapt to such tasks due to limited flexibility, whereas Seq-FT’s unconstrained fine-tuning can more easily fit the data. Second, PMC-VQA is a medical-domain task with different visual domains, question types, and prompt formats compared to KVQA. This reduces representational overlap between the two tasks and thus limits interference during sequential training. However, regularization-based methods may apply global constraints which could result in unintended interference with prior task-specific adaptation. A similar pattern is observed between VizWiz-Cap and SK-VG, where Seq-FT also exhibits reduced forgetting, likely due to the same reason.\n\nOverall, while Seq-FT achieves higher Last and BWT scores on KVQA, the margins are small. Across the full task sequence, it still suffers from severe forgetting, as reflected in its low overall CL scores.\n\n---\n**Q3. Strong Performance on the Initial Task**\n\nThank you for the thoughtful question. Our training protocol treats all tasks in the sequence uniformly and does not apply any special treatment to the initial task, so there is no risk that our method simply overfits to the first dataset. As evidenced by the overall experimental results, our method outperforms all state-of-the-art baselines on all three CL metrics across most tasks, not just the initial one.\n\nThe seemingly large performance gains on VizWiz-Cap can be explained as follows:\n* Following the CoIN benchmark [1], we use CIDEr as the evaluation metric for captioning tasks. Since CIDEr scores are not upper-bounded (often exceeding 1), numerical improvements may appear larger in magnitude.\n* In continual learning, forgetting accumulates over time and amplifies performance degradation on earlier tasks. As D-MoLE better mitigates forgetting, its advantage is more visible on tasks like VizWiz-Cap that appear early in the sequence.\n\n---\n**Q4. Key Factors Driving Performance Improvements**\n\nThank you for the question. The performance gains of D-MoLE stem from the integration of two complementary modules, each addressing a key challenge in CMIT. The dynamic layer-wise expert allocator addresses task architecture conflict by assigning LoRA experts only to the most relevant layers and routing inputs via task-specific autoencoders. This enables precise architectural adaptation and selective expert activation during inference. The gradient-based inter-modal continual curriculum mitigates modality imbalance by adjusting training dynamics when tasks rely unevenly on different modalities, ensuring stable multimodal adaptation.\n\nOur ablation study (Section 5.3) supports this analysis. Removing the expert allocator (v4) leads to the largest performance drop, underscoring the importance of architectural flexibility. Removing the curriculum module (v3) results in lower performance than the full model, demonstrating its complementary benefit. Limiting updates to a single modality (v1 or v2) also yields suboptimal results, highlighting the need for full-modality adaptation.\n\nOverall, while the expert allocator contributes most directly to performance gains, both components are essential and jointly enable robust and balanced continual learning. We will add more detailed discussion of these factors in the revised version.\n\n---\n**Reference**\n\n[1] CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models. NeurIPS 2024.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths:**\nThe paper is well-written and easy to follow, making it accessible to a broad audience. The topic of continual learning for multimodal large language models is highly relevant in contemporary research, with significant real-world applications. The study effectively highlights the challenges of task architecture conflict and modality imbalance, supporting these discussions with both empirical results and theoretical analyses.  \n\nI find the experimental results particularly impressive, as they demonstrate a substantial improvement over state-of-the-art baselines. The proposed method is intuitively designed and does not introduce excessive complexity in its implementation, which enhances its practicality and reproducibility. The paper presents a strong contribution to the field, addressing key issues with clear and well-supported findings.  \n\n**Weaknesses:**\nThe presentation of results in Table 2 could be confusing due to the presence of two 'average' metrics—one in the dataset row and another in the method row. It would be helpful to clarify how these averages are computed and their intended interpretation.  \n\nThe Seq-FT baseline achieves the best performance on KVQA in the 'Last' and 'BWT' metrics. Could you provide an explanation for this result? It seems counterintuitive that the most naive baseline would outperform more sophisticated approaches in these measures.  \n\nThe proposed D-MoLE method significantly outperforms other baselines on the first task, VizWiz-Cap. Could this be an indication that the model is primarily excelling at fitting the initial dataset, rather than truly demonstrating strong continual learning capabilities? Is there any evidence to rule out the possibility that the method behaves more like fine-tuning rather than an effective continual learning approach?  \n\nCould you provide further insight into the key factors driving the performance improvements of your method? Specifically, which aspects of the design contribute most to the observed gains? A more detailed breakdown of the improvements would strengthen the paper’s claims.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fKhHJtjUfR", "reviewer": "Reviewer_c9zT", "summary": "The paper presents D-MoLE, a framework for continual multimodal instruction tuning (CMIT) in multimodal large language models (MLLMs). It dynamically allocates LoRA experts across layers using zero-cost metrics and addresses modality imbalance through gradient-based inter-modal curriculum learning. By resolving task architecture conflicts and mitigating modality imbalance, D-MoLE preserves performance on previously learned tasks, achieving a 15% average performance improvement over state-of-the-art baselines.\n\n## update after rebuttal\nI have carefully read the authors' rebuttal and the feedback has well addressed my questions and concerns. Thus, I would like to insist on the score of accept. Thanks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the encouraging and constructive feedback. We are pleased that they find our approach well-motivated, intuitive, and effective, and acknowledge the impressive performance gains. We hope our responses below address the remaining concerns.\n\n---\n**Q1. Including Some Case Studies or Examples**\n\nThank you for the valuable suggestion. In the revised version, we will include case studies in the appendix. For example, we plan to show some multimodal instruction examples from early tasks and compare the model’s responses to them after training on later tasks. This will help illustrate how well D-MoLE preserves prior knowledge by examining model behavior on early-task examples at different stages of continual learning.\n\n---\n**Q2. Figure Clarity and Font Size**\n\nThank you for the helpful suggestion. We will increase the font size in Figure 1 to improve readability, and recheck all figures in the paper to ensure visual clarity in the revised version.\n\n---\n**Q3. Visualizing the Dynamic Training Process**\n\nThank you for the valuable suggestion. We have created two heatmaps to illustrate the dynamic behavior of our method and better convey the adaptability of D-MoLE. We provide preview versions at anonymous links below, and will incorporate them into the revised version:\n\n* Architecture evolution dynamics (https://anonymous.4open.science/r/D-MoLE/figure6.jpg): shows how experts are allocated across different layers during the CMIT process. \n* Expert activation dynamics (https://anonymous.4open.science/r/D-MoLE/figure7.jpg): visualizes how task-specific experts are activated over time during training.\n\n---\n**Q4. Compatibility with Other Continual Learning Approaches**\n\nThank you for the question. Combining our method with other continual learning approaches is certainly feasible.\n\nOur method focuses on architectural evolution, dynamically expanding model capacity throughout the CMIT process. This enables flexible task adaptation without being constrained by a fixed parameter budget. In contrast, approaches like O-LoRA mitigate forgetting through parameter regularization, but often face a trade-off between retaining prior knowledge and adapting to new tasks due to their fixed capacity.\n\nThese two directions are compatible in principle. While our current framework does not include regularization terms in the loss function, incorporating them may be a promising extension. For example, enforcing orthogonality between different experts could help reduce redundancy and improve task separation.\n\n---\n**Q5. Need for Expert Removal**\n\nThank you for the question. While our current experimental setting does not involve expert removal, since we evaluate all tasks after each training stage to assess knowledge retention (see Appendix F), our framework can easily support it.\n\nFor example, one practical extension is to track expert activation frequency using a sliding window over recent inputs. Experts associated with inactive tasks could then be unloaded to reduce memory and computation overhead, with the option to reload or reinitialize them if needed. This may be a promising direction for future work, particularly in real-world online continual learning scenarios.", "claims_and_evidence": "yes.", "methods_and_evaluation_criteria": "yes.", "theoretical_claims": "yes, theorem 3.1.", "experimental_designs_or_analyses": "yes, all.", "relation_to_broader_scientific_literature": "It could be applied in scientific fields related to MLLM, e.g. medical imaging.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Pros:**\n\n- The paper provides clear and well-motivated illustrations of the problems and challenges in continual multimodal learning. The issues of task architecture conflict and modality imbalance are effectively addressed by the method's design, particularly through the dynamic expert allocator and modality-specific curriculum.\n  \n- The approach is both well-motivated and intuitive. The concept of dynamically adding LoRA parameters is an interesting and potentially practical solution for scaling models while preserving efficiency. This method could have significant real-world applications, especially in resource-constrained environments.\n  \n- By resolving task architecture conflicts and mitigating modality imbalance, D-MoLE successfully preserves performance on previously learned tasks, leading to an impressive 15% average improvement over state-of-the-art baselines. This highlights the method’s effectiveness.\n  \n- The ablation studies are thorough and provide strong evidence of the effectiveness of each module in the framework. Additionally, supplementary experiments demonstrate that the method does not add many additional computational costs.\n\n**Cons:**\n\n- It would be beneficial to include some case studies or examples of specific tasks to further illustrate the method's performance in different settings and help contextualize its real-world applications.\n\n- Some of the figures could be enhanced for better clarity. For instance, the fonts in Figure 1 are too small, which may hinder readability, especially for audiences engaging with the paper on printed formats.\n\n- Showcasing the dynamic training process in more detail would be helpful. For example, visualizing how experts are assigned throughout the continual learning process would provide a clearer understanding of the method's adaptability.\n\n- Could the proposed method be combined with other continual learning approaches, such as O-LoRA, to further enhance performance? \n\n- Is there a need to remove experts over time, in addition to adding them?", "other_comments_or_suggestions": "Please see cons.", "questions_for_authors": "Please see cons.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QkjlgBBIOp", "reviewer": "Reviewer_xQiS", "summary": "This paper presents an algorithm D-MoLE for continual multimodal instruction tuning. The algorithm solves the challenges of task architecture conflict and modality imbalance by dynamically assigning LoRA experts and a gradient-based continual curriculum. Experimental results show the effectiveness of the proposed algorithm.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the helpful and encouraging comments. We appreciate that they note the inclusion of theoretical analysis, find our writing clear and easy to read, and recognize the significance of the problem setting. We hope the following explanations provide sufficient clarification.\n\n---\n**Q1. Tasks-Architecture Conflict in MLLMs**\n\nThank you for the question. Our intention here is to emphasize why fixed architectures pose additional challenges for continual learning (CL) in MLLMs. We would like to clarify it further here.\n\nMLLMs have inherent architectural heterogeneity, especially in the commonly adopted vision encoder + projector + LLM framework. Unlike unimodal LLMs, MLLMs process inputs from multiple modalities. As tasks vary in their reliance on each modality, abstraction requirements differ across modules. This leads to greater task-wise architectural sensitivity during CL. As shown in Figure 1, the sensitivity of transformer layers in both the vision encoder and the LLM varies substantially across tasks, indicating that different components must adapt differently. This observation motivates us to move beyond fixed architectures and adopt dynamic expert allocation, and further supports our design of an architecture evolution mechanism tailored to the evolving demands of CL in MLLMs.\n\nWe will revise the Introduction to better explain this motivation.\n\n---\n**Q2. Threshold Selection and Sensitivity**\n\nThank you for the question. The threshold in Equation 9 is selected based on the reconstruction loss distribution of each autoencoder on its corresponding task’s training data. It is set moderately above the typical loss range to accommodate minor distribution shifts at evaluation and to allow transferable samples from other tasks to activate relevant experts.\n\nWe observe that the loss distributions are concentrated with few outliers, and the same thresholding strategy works across tasks without tuning. Moreover, in-task and out-of-task losses are typically well-separated, as supported by the t-SNE visualization in Figure 5.\n\nThe threshold serves two purposes: (1) avoiding irrelevant experts from being activated solely due to top-2 ranking, and (2) enabling detection of unseen tasks. As such, it plays a filtering role rather than directly driving routing decisions. This allows for coarse-grained threshold choices without requiring precise tuning.\n\nTo assess sensitivity, we evaluate the final checkpoint of D-MoLE under different scaling factors of the thresholds $\\\\{\\tau_t\\\\}$ and report the corresponding average Last scores across tasks, with $1\\times$ denoting the default setting. Full table can be found at https://anonymous.4open.science/r/D-MoLE/table10.jpg.\n\n**Table 1: Average task performance under different threshold scaling factors, evaluated on the final checkpoint.**\n|Scaling Factor|$0.1\\times$|$0.5\\times$|$1\\times$|$2\\times$|$10\\times$|\n|-|-|-|-|-|-|\n|*Avg. Last*|80.37|81.36|**82.18**|81.31|80.34|\n\nNote that this experiment applies threshold scaling only during evaluation. Due to current computational resource constraints, we do not retrain the model under each new threshold setting. As a result, expert collaboration patterns during evaluation may differ from those formed during training under the default thresholds, which may slightly affect performance. Despite this, performance remains stable across a wide range of threshold scaling factors, indicating that our method is not sensitive to the exact threshold values, as long as they are within the same order of magnitude. We will include this analysis in the revised version.\n\n---\n**Q3. Setup of Figure 5 and Autoencoder Capacity**\n\nThank you for pointing this out. The experimental setup for Figure 5 (Appendix G) is as follows: we randomly sample 500 training samples from each task and extract their multimodal instruction sequence embeddings using the method described in Section 4.1. We then compute the reconstructed embeddings using the corresponding autoencoders and visualize all embeddings together using t-SNE. The clear separation among clusters provides supporting evidence that task-specific autoencoders learn distinct representations and can effectively function as routers. We will include these experimental details in the revised version for clarity.\n\nRegarding the use of a 2-layer MLP (one encoder and one decoder layer) for the autoencoder, we find it sufficient in our setting. In typical CL benchmarks for MLLMs, tasks are defined by dataset boundaries, and samples from different tasks often differ significantly in terms of image domains, question types, or textual styles. As such, the autoencoders can effectively distinguish tasks even with simple architectures. We adopt this lightweight design to minimize the additional computational overhead introduced by the routing process. Exploring more challenging scenarios with blurred task boundaries may be a promising direction for future work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "With the popularity of large-scale pre-training, continual multimodal instruction tuning has become an important topic. This paper focuses on it, which is helpful for the development of multimodal large language models.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper has the theoretical analysis\n\n2. The setting the paper focuses on is significant\n\n3. The writing is good and easy to read.\n\nWeaknesses: \n\n1. \"fixed architecture models inevitably face the dilemma...\" mentioned in paragraph 2 of the Introduction is not unique to MLLMs. In fact, continual learning has been extensively studied on this dilemma. The description here is confusing.\n\n2. How to select the threshold hyperparameter of equation 9 and its sensitivity should be discussed.\n\n3. The experimental conditions of figure 5 in Appendix G are not explained. Also, is the use of a 2-layer mlp sufficient to distinguish between different tasks, and is this related to the task difficulty of the dataset itself?", "other_comments_or_suggestions": "see above weaknesses", "questions_for_authors": "see above weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission focuses on the multimodal instruction tuning under the context of continual learning. The authors proposed a D-MoLE method to explore the architecture evolving under the budgets by designing the LoRA Experts. Along with this, the authors introduced a gradient-based continual curriculum. Experiments show the 15% improvement on average over the best baseline.

Four reviewers commented this submission and finally their scores all reached to 4. Specifically, the authors substantially improved the experiments and made more clarification about some technical details and novelty. Two reviewers expressed their tendency for the rebuttal and the other two reviewers keep silence. After AC checked the rebuttal and the reviewers' questions, it seems that the authors rebuttal mostly considered the reviewers' questions. Hope the authors carefully incorporate the suggestions into the final revision.

Based on the reviewers' suggestion in ratings and the authors substantial rebuttal, AC tends to recommend acceptance about this submission.
Decision: Accept (poster)