Title: Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Continual Learning

Multimodal Generative Models

Catastrophic Forgetting

Mixture of Experts (MoE)

Knowledge Distillation
Abstract: Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings.

Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models

Abstract

Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE) , a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings. Codes are publicly available: https://github.com/Christina200/MoDE-official.git.

1 Introduction

Traditional multimodal models are typically divided into two categories: multimodal understanding (e.g. answering questions about images) and multimodal generation (e.g. generating images from text prompts) [1, 2]. Unified Multimodal Generative Models (UMGMs) aim to integrate these two tasks within a single framework. Recent advancements in UMGMs [3-8] have demonstrated strong performance on a wide range of tasks, such as visual question answering (VQA), image captioning, visual reasoning, classification, reading comprehension, and image generation. These models typically embed multiple input types into a shared representation space and leverage one single transformer backbone to model cross-modal interactions. Training generally follows a twostage paradigm: an initial pretraining phase focused on text-image alignment, followed by fine-tuning for specific downstream tasks. During the second stage, instruction tuning (which fine-tunes the model on diverse, task-specific instructions paired with expected outputs) has emerged as a widely adopted strategy to better align model behavior with human intent [9, 10].

While UMGMs demonstrate impressive zero-shot performance on a wide range of unseen instructions, their effectiveness is inconsistent across all tasks, particularly when the relevant task data is absent from the pretraining corpus. Expanding the training dataset to include new task data can significantly improve performance on those tasks. However, given the continual emergence of novel multimodal

ScienceQA

TextVQA

ImageNet

GQA

VizWiz

Task

0

20

40

60

80

100

Accuracy (%)

Start

End

worst

best

Knowledge

VQA

Reading

Comprehension

Image

Generation

Visual

Reasoning

Blind People

VQA

(b)

Upper bound

SeqLoRA

DualPrompt (ECCV '22)

MoDE (Ours)

tasks, repeatedly merging new data and retraining the model is computationally expensive and impractical. This motivates the need for new approaches that can enable UMGMs to flexibly and efficiently adapt over time. This goal aligns with the principles of continual learning (CL), where models are designed to acquire new capabilities incrementally, similar to how humans learn [11].

Existing studies in CL have shown that sequentially fine-tuned models suffer from catastrophic forgetting, a phenomenon where a model fine-tuned on new tasks tends to forget or overwrite the previously acquired knowledge [12]. More recently, several works have explored continual instruction tuning for multimodal large language models (MLLMs) [13-15]. However, these models are limited to language-only outputs and, as a result, prior research has focused exclusively on tasks such as VQA and other multimodal understanding benchmarks. In contrast, UMGMs are capable of both multimodal understanding and generation within a single backbone, enabling the production of both textual and visual outputs and thus representing a fundamentally different paradigm. This raises a new and largely unexplored challenge, whereby forgetting occurs not only within a single modality (intra-modal), but also across modalities (inter-modal) during continual instruction tuning of UMGMs.

Starting from these overarching ideas, we ask: Can UMGMs continually acquire new capabilities through instruction tuning without suffering catastrophic forgetting, both within a modality (e.g., multimodal understanding) and across modalities (e.g., improving multimodal understanding while preserving visual generation)?

As shown in Figure 1, UMGMs do experience catastrophic forgetting across a range of tasks, including visual reasoning, classification, VQA, and image generation. Also, while existing methods [16, 17] can mitigate either intra-modal or inter-modal forgetting, none of them can effectively address both issues simultaneously.

To address this fundamental challenge, we propose Modality-Decoupled Experts (MoDE) , a new lightweight architecture which incorporates a modality-aware sparse mixture of LoRA adapters for text (T-MoE) and a single LoRA adapter for images (V-Adapter). The intra-modal forgetting during continual multimodal understanding tasks is mitigated by the T-MoE's routing, which selectively activates the appropriate experts. The inter-modal forgetting in image generation is prevented through modality decoupling and knowledge distillation between the pre-trained model (teacher) and the newly added image LoRA (student). During continual instruction tuning, only the MoDE components are trained, while the pre-trained UMGM parameters remain frozen to preserve their robust cross-

modal alignment. Our results show that this design simultaneously mitigates both intra-modal and inter-modal catastrophic forgetting.

Our contributions are:

We identify and illustrate the unique challenge of inter-modal catastrophic forgetting, in addition to the more common intra-modal forgetting in autoregressive transformers that unify multimodal understanding and generation under continual instruction tuning. We further provide a theoretical analysis attributing inter-modal forgetting to modality gradient conflict , and formally prove that modality decoupling can mitigate this conflict both theoretically and experimentally.

We propose Modality-Decoupled Experts (MoDE), a new lightweight and scalable architecture that tackles both intra- and inter-modal forgetting by decoupling modality-specific updates (to reduce inter-modal gradient conflict) and applying knowledge distillation to preserve pre-trained image generation capabilities.

Through extensive experiments, we show that MoDE outperforms SOTA baselines (e.g. CLMoE [18], Model Tailor [17]) when mitigating both intra- and inter catastrophic forgetting simultaneously. This opens up new possibilities for scalable continual learning in UMGMs.

The remainder of this paper is organized as follows: Section 2 reviews related work. Section 3 introduces inter-modal forgetting in UMGMs. Section 4 presents our proposed method, and Section 5 details the experimental results. Finally, Section 6 concludes our paper.

2 Related work

2.1 Unified multimodal generative models (UMGM)

Early efforts to unify visual understanding and generation [19-22] typically combine diffusion models with MLLMs, conditioning the diffusion process on LLM-generated embeddings. While effective for certain tasks, this 'diffusion+MLLM' design lacks tight coupling between image generation and language modeling, resulting in suboptimal performance on instruction-based generation [23].

More recent approaches treat both understanding and generation as a unified next-token prediction problem [24-26, 3]. These models differ in their encoder and decoder designs, e.g. some [3, 6, 27] employ vision tokenizers such as VQGAN [28] to discretize images into token sequences, while others [29, 30] use semantic encoders like CLIP [31] or SigLIP [32] to represent images as sequences of continuous embeddings. Their decoders typically rely on either a VQV AE [33] or a diffusion-based generator [34, 22], but all maintain an autoregressive transformer backbone.

Our method is designed for transformer-based UMGMs and works with a broad class of models built on this autoregressive architecture, regardless of their choice of encoding or decoding mechanism.

2.2 Mixture of experts (MoE)

The Mixture of Experts architecture [35-38] combines specialized experts with a gating mechanism that efficiently allocates tokens. Several studies explore integrating MoE with LoRA [39] to facilitate scalable and efficient training of MLLMs and LLMs [40-45]. Recently, the scalability and modularity of MoE have attracted growing attention in continual learning [18, 46, 47]. For instance, LifelongMoE [48] introduces a strategy to train new experts while freezing the old ones. However, prior MoE-based CL methods share the same set of experts across modalities, which leads to modality interference during multimodal generation and exacerbates the inter-modal forgetting. Motivated by these limitations, we propose modality-decoupled MoE with LoRA, aiming at mitigating both intermodal forgetting caused by modality interference and intra-modal forgetting within each modality.

2.3 Catastrophic Forgetting

Catastrophic forgetting remains a fundamental challenge in multimodal continual instruction tuning (MCIT), where a multimodal generative model is incrementally adapted to new datasets and task instructions without costly retraining from scratch. Existing methods to address catastrophic forgetting in MCIT typically fall into four categories: regularization-based [49], architecture-based [50, 14],

Pre-trained UMGM

VQA Task #1

VQA Task #2

VQA Task #3

Continual instruction tuning

CLIP: 0.3248

CLIP: 0.3422

CLIP: 0.2915

CLIP: 0.3082

CLIP: 0.3067

CLIP: 0.3056

CLIP: 0.3289

CLIP: 0.3164

CLIP: 0.1061

CLIP: 0.1458

CLIP: 0.2885

CLIP: 0.2525

CLIP: 0.1769

CLIP: 0.2540

CLIP: 0.2532

CLIP: 0.2749

Which of these

states is farthest

north?

A. West Virginia

B. Louisiana

D. Oklahoma

Is the sky dark?

Answer the

question using a

single word or

phrase.

What is the object in the

image? Answer the

question using a single

word or phrase.

replay-based [15], and prompt-based approaches [51, 52]. For instance, EProj [11] applies tasksimilarity-based regularization to constrain model updates that are important for previous tasks. Fwd-Prompt [51] selects prompts using both textual and visual features to preserve earlier knowledge. LLaCA [50] employs exponential moving average (EMA) to merge the weights of old and new models, thereby maintaining both past task performance and pre-trained capabilities.

However, these approaches address only the intra-modal catastrophic forgetting , where old tasks, new tasks, and pre-trained capabilities all belong to the same output modality, typically text (e.g., in VQA tasks) [52-54]. While this assumption holds for traditional multimodal models with unimodal outputs, it is insufficient for UMGMs which generate multimodal outputs such as text and images.

3 Inter-modal Catastrophic Forgetting in UMGMs

Inter-modal catastrophic forgetting can manifest in several forms. For instance, prior work [55] examined the loss of text-only generation abilities in MLLMs when compared to their original LLM counterparts. Other studies have explored modality decoupling strategies in encoders and adapters of UMGMs [56, 57]. In this paper, we restrict our focus to multimodal tasks, specifically the forgetting that occurs in multimodal understanding and generation. In particular, we study visual generation forgetting that arises when tuning models on understanding tasks. Complementary results on understanding forgetting induced by tuning on image generation tasks are provided in Appendix F.

To investigate inter-modal catastrophic forgetting in UMGMs from the visual perspective, we conduct a preliminary experiment by sequentially tuning Chameleon [3] using LoRA on three VQA datasets [58-60], while monitoring image generation performance with prompt-based evaluation. As shown in Figure 2, we observe two key issues in image generation as tuning progresses: 1) The overall image quality degrades significantly, 2) The alignment between the generated image and the input prompt worsens over time.

These findings indicate that inter-modal catastrophic forgetting does indeed occur in UMGMs when they are continually fine-tuned on a sequence of multimodal understanding tasks. We attribute this

forgetting to the UMGMs utilizing a unified architecture (e.g., transformer) with shared parameters for both text and visual modalities [61]. When tuning on a single modality (e.g., text), the shared parameters are updated in a way that interferes with the model's performance on the other modality (e.g., image), a phenomenon consistent with observations reported in recent studies [62]. To formalize this interference, we introduce the notion of modality gradient conflict .

Definition 1 (Modality Gradient Conflict) . Let g v = ∇ θ L v denote the gradient of the imagegeneration loss and g t = ∇ θ L t denote the gradient of the text-generation loss, both with respect to shared model parameters θ . g v and g t are said to be conflicting with each other if ⟨ g v , g t ⟩ < 0 , where ⟨· , ·⟩ denotes the standard Euclidean inner product.

Proposition 1. When fine-tuning on the text-generation tasks, a stochastic gradient descent (SGD) update with sufficiently small step size η modifies the model parameters as θ ← θ -ηg t . The resulting change in the visual loss is:

where H v = ∇ 2 θ L v is the Hessian of the visual loss. Hence, when ⟨ g t , g v ⟩ < 0 , a step optimizing text generation will increase the visual loss, leading to inter-modal forgetting.

Motivated by this, we propose MoDE , a modality-decoupled adapter framework that provably eliminates the first-order modality conflict and thereby mitigates the inter-modal catastrophic forgetting. A detailed proof of Proposition 1 is available in Appendix A.

4 Methodology

4.1 Problem formulation

Continual instruction tuning Assume a UMGM has been pre-trained with abundant visionlanguage data, captured by trainable parameters θ . We further train to adapt this UMGM to new S tasks in a sequential manner. Each task is denoted by a task descriptor τ ∈ { 1 , 2 , ..., S } , and owns an independent dataset D τ = { ( X img τ,j , X ins τ,j , X ans τ,j ) } N τ j =1 with N τ data triplets, where X img , X ins and X ans indicate the input image tokens, instruction text tokens, and answer text tokens, respectively.

We formalize the autoregressive cross-entropy objective for task τ as:

where the expectation E is taken over the data distribution D τ , L is the length of the answer sequence, X ans <i indicate all answer tokens before the index i , X ans i indicates the i -th answer token.

Image generation Let X ins be the text prompt tokens, and let X img be the image tokens generated autoregressively. Following a standard autoregressive factorization, the process can be formalized as:

where L is the length of the image token sequence, X img <i indicate all image tokens before the index i , X img i indicate the i -th image token. Once all tokens are generated, they are detokenized via a decoder (such as a VQ-VAE) to produce the resultant image.

4.2 Modality-Decoupled Experts (MoDE)

MoE adapters are lightweight modules that route inputs through a sparse subset of expert networks, enabling task-specific adaptation with minimal interference to shared parameters [63]. Although MoE adapters can effectively mitigate intra-modal catastrophic forgetting, naively sharing their parameters across modalities can lead to modality gradient conflict (see Section 3). To address this issue, we propose Modality-Decoupled Experts (MoDE) , a modality-decoupled adapter architecture that

Image Token

Text Token

MLP

Multi-layer

Perceptron

AR Transformer

this

<ans>

is

What

Expert 1

Expert 2

Expert n

…

T-MoE Adapters

T-Router

B

A

V-Adapter

Student

logits

Teacher

logits

Distillation

this

is

End

Image

Image

Tokenizer

What

Start

Image

End

Image

Stop

<ans>

Stop

sign

Reference

data

Input image(s)

Input prompt(s)

isolates text and visual updates into separate trainable subspaces. As shown in Figure 3, MoDE mitigates modality gradient conflict while preserving the flexibility and scalability of MoE adapters.

Getting now into details, LoRA introduces a trainable low-rank update to a frozen weight matrix W ∈ R d out × d in , where d in and d out denote the input and output feature dimensions, respectively. The update is parameterized by two learnable low-rank matrices A ∈ R r × d in and B ∈ R d out × r , such that:

where A and B are learnable matrices with rank r ≪ min( d in , d out ) and scaling factor α ∈ R . Given an input token representation h ∈ R d in , the modified linear transformation f : R d in → R d out becomes:

MoE-LoRA generalizes LoRA by introducing a mixture of expert LoRA modules to handle diverse tasks. A router assigns each input token x ∈ R d in a soft distribution over experts:

where W g ∈ R d in × n is a trainable gating matrix with n denotes the number of experts, and g j ( x ) denotes the routing probability of expert j . The resulting update ∆ W ( x ) is a weighted combination of expert LoRA updates:

where each expert has A j ∈ R r × d in and B j ∈ R d out × r , and α ∈ R is a hyper-parameter. Given an input token representation h ∈ R d in , the final linear transformation becomes:

MoDE integrates the above two types of adaptations into UMGMs: V-Adapter and T-MoE , as shown in Figure 3. The V-Adapter is a LoRA module specialized for visual tokens which adapts the model to both visual understanding and image generation tasks. In contrast, the T-MoE is a MoE-LoRA applied to text tokens, designed to enhance continual instruction tuning on multimodal understanding tasks without interfering with visual knowledge. Together, these two components allow MoDE to balance modality-specific adaptation (via the V-Adapter) and task-level flexibility (via the T-MoE) within a unified multimodal backbone.

To formalize the forward pass of multimodal tokens, we consider a frozen linear layer W ∈ R d out × d in in UMGM. Let H = [ h t 1 , . . . , h t m , h i m +1 , . . . , h i m + L , h t m + L +1 , . . . , h t N ] ∈ R N × d in denote the hidden states of a sequence of interleaved images and text with length N , where the subscript indicates position and the superscript indicates the token modality ( t for text and i for image). We untie the sequence into the text sequence H t = [ h t m + L +1 , . . . , h t N ] and the image sequence H i = [ h i m +1 , . . . , h i m + L ] , where m is the start index of image tokens and L is their length. We feed H i into the V-adapter using Eq. 5 to obtain ˆ H i = f ( H i ) , and feed H t into T-MoE using Eq. 8 to obtain ˆ H t = f ( H t ) . Finally, we reassemble ˆ H i and ˆ H t into their original positions to form the output sequence ˆ H .

MoDE therefore cleanly decouples different modalities: text tokens benefit from experts diversity for continual instruction tuning, while image tokens rely on a visual adapter. Because T-MoE and V-adapter are disjoint, their update directions cannot interfere arbitrarily.

Proposition 2. In MoDE, let ϕ denote the parameters of the T-MoE and ψ denote the parameters of the V-adapter. A single gradient step θ ↦→ θ -η ∇ ϕ L t updates only ϕ , and the resulting change in the visual loss satisfies:

Thus, MoDE provably bounds the inter-modal interference to O ( η 2 ) , compared to O ( η ) under a modality-decoupled baseline as in Proposition 1. This theoretical property directly explains the minimal gradient conflict and superior retention of visual generation capabilities observed in our experiments. See Appendix A for a proof of Proposition 2 and Appendix B for its empirical verification.

4.3 Knowledge Distillation

During instruction tuning, the V-Adapter learns to understand images, which can degrade its original generation ability. We anchor the V-Adapter to the frozen pre-trained backbone (teacher) via logit-level knowledge distillation (KD), as shown in Figure 3.

A small subset of images sampled from the LAION-5B dataset [64] serves as the reference data. For each reference token, both teacher ( T ) and student ( S ) predict the next image token. Let z T i , z S i denote their logits for the i -th image token.Using a softening factor β , the KD loss ( L KD ) is computed as the Kullback-Leibler divergence [65] between their softened output probabilities:

where L is the number of generated image tokens and ∥ denotes the distance between two distributions. Final objective. The T-MoE is trained with the instruction-tuning cross-entropy loss ( L CE) as in (2):

The V-Adapter uses a weighted combination of the cross-entropy and the distillation losses:

This mixed objective balances the updates for visual understanding and image generation in the V-adapter, thereby mitigating inter-modal catastrophic forgetting. In our experiments, we set λ = 0 . 3 based on its optimal performance (see Appendix D for a sweep over different λ values).

5 Experiments

5.1 Experimental setup

Evaluation Benchmarks For continual instruction tuning, we evaluate on five datasets: Four visual question answering datasets (ScienceQA [58], TextVQA [66], GQA [59], and VizWiz [67]) and one image classification dataset (ImageNet [60]). For image generation, we use the CustomConcept101 [68] dataset, which contains 101 concepts grouped into 16 broader categories.

Evaluation Metrics For continual instruction tuning, we follow the protocols from [69, 70] and report two metrics: (1) Average Accuracy (ACC) , defined as the mean accuracy across all tasks after the final round of training; and (2) Average Forgetting (Fgt) , which quantifies performance degradation on previous tasks and is computed as the mean reduction from the best observed accuracy to the final accuracy, for each task. We use exact matches to determine answers correctness.

We evaluate image generation quality with three metrics: (1) Image alignment . Cosine similarity between CLIP embeddings of the generated image and a reference image that depicts the target concept; higher is better ( ↑ ). (2) Text alignment . Cosine similarity between the generated image embedding and the CLIP embedding of its text prompt; higher indicates better prompt fidelity ( ↑ ). (3) Fréchet Inception Distance (FID) [71]. Fréchet distance between features of generated and real images; lower values mean the synthetic images are closer to the real distribution ( ↓ ).

Baselines We compare our proposed MoDE with the following continual learning approaches: Seq LoRA, which sequentially trains the model on new tasks using LoRA adapters [39]; Model Tailor [17]; DualPrompt [16]; MoE-LoRA [63]; CL-MoE [18]; and Joint Training (upper bound), which trains the model on all datasets simultaneously and serves as the upper bound reference. The implementation details of baseline methods are provided in Appendix I.

We use two autoregressive backbone UMGMs: Chameleon [3] and Janus-Pro [4]. For Chameleon, we adopt the implementation in [72] since the original model and checkpoints are not publicly available. Results on Janus-Pro are provided in Appendix G. See Appendix D for implementation details.

5.2 Main results

Quantitative results. As shown in Table 1, our proposed MoDE built on Chameleon [3] effectively mitigates both inter- and intra-modal catastrophic forgetting, outperforming other CL baselines. In the 'Image Generation' columns, MoDE maintains image quality comparable to the pre-trained model (Zero-shot), indicating strong resistance to inter-modal forgetting. For instance, MoDE achieves an FID score of 53.74, closely matching the 52.13 of the pre-trained model (zero-shot). While DualPrompt [16] also preserves image generation quality, its performance in continual instruction tuning (as shown in the 'Multimodal Understanding' columns) is poor: the average accuracy is only 31.92%. Notably, although DualPrompt shows low average forgetting of 6.82%, this is misleading: its best accuracy on ImageNet is only 24.55%, whereas other methods exceed 70%. This low forgetting

arises not from effective retention, but from a limited ability to learn in the first place: if a model never learns well, it has little to forget. Although MoE-LoRA [63] performs well in mitigating intra-modal forgetting, it suffers from severe inter-modal forgetting, as evidenced by the worse image generation quality, with an FID of 65.16 compared to 56.12 for the vanilla SeqLoRA. In contrast, our MoDE achieves the highest accuracy of 33.47% and the lowest FID of 53.74, demonstrating its effectiveness in mitigating both inter-modal and intra-modal forgetting simultaneously.

Table 2 presents more detailed results of continual instruction tuning. In this setting, our MoDE consistently outperforms all comparison methods and sustains effective learning across all tasks.

Qualitative results. As shown in Figure 4, our MoDE can generates higher quality images compared to other methods, effectively preserving the pre-trained model's image generation capability. For instance, in the third row of Figure 4, given the prompt ' Barn in the fall season with leaves all around ', MoDE successfully generates yellow leaves surrounding the barn, whereas Model Tailor [17] and CL-MoE [18] fail to capture this detail. See Appendix C for more qualitative results.

5.3 Ablation results

To validate the performance gains of MoDE, we present our ablation results in Table 3. 'T-MoE LoRA' includes only the T-MoE adapters without the V-adapter. 'MoDE w/o KD' removes knowledge distillation, training both the T-MoE adapters and the V-adapter solely with cross-entropy loss. Of note, MoDE is also robust to task order. See Appendix E for more ablation results.

Input Prompt

Chameleon [3]

CL-MoE [18]

MoDE (Ours)

A dog wearing sunglasses on

the porch.

A transparent cup filled with

steaming hot cocoa.

Barn in the fall season with

leaves all around.

Marigold flowers in the vase.

These results demonstrate the effectiveness of both the T-MoE and V-adapter. Using only the T-MoE preserves image generation capabilities by leaving visual components untouched, but results in suboptimal visual understanding, as the model lacks updates necessary for better visual comprehension.

We also report the computational efficiency of MoDE, including training time, memory consumption, and parameter overhead, in Appendix H. These results show that MoDE achieves favorable trade-offs between accuracy and efficiency compared to existing methods.

6 Conclusion

We have proposed Modality-Decoupled Experts (MoDE) to address both intra- and inter-modal catastrophic forgetting in continual instruction tuning for Unified Multimodal Generative Models (UMGMs). We have identified inter-modal forgetting, explained its cause due to the modality gradient conflict, and designed MoDE to decouple modality-specific updates, mitigating conflict theoretically and empirically. Extensive experiments show that MoDE significantly outperforms SOTA baselines, establishing it as a strong solution for continual learning in unified multimodal generation.

7 Acknowledgments

This work is supported in part by the NSF grant CCF-2531882, and in part by an Amazon Research Award, Spring 2025.

References

H. Liu, C. Li, Q. Wu, and Y. J. Lee, 'Visual instruction tuning,' Advances in Neural Information Processing Systems , vol. 36, pp. 34892-34916, 2023.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'High-resolution image synthesis with latent diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10684-10695, 2022.

C. Team, 'Chameleon: Mixed-modal early-fusion foundation models,' arXiv preprint arXiv:2405.09818 , 2024.

C. Wu, X. Chen, Z. Wu, Y. Ma, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, C. Ruan, et al. , 'Janus: Decoupling visual encoding for unified multimodal understanding and generation,' in Proceedings of the Computer Vision and Pattern Recognition Conference , pp. 12966-12977, 2025.

C. Zhou, L. YU, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy, 'Transfusion: Predict the next token and diffuse images with one multi-modal model,' in The Thirteenth International Conference on Learning Representations , 2025.

X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu, et al. , 'Emu3: Next-token prediction is all you need,' arXiv preprint arXiv:2409.18869 , 2024.

J. Xie, W. Mao, Z. Bai, D. J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang, and M. Z. Shou, 'Show-o: One single transformer to unify multimodal understanding and generation,' in The Thirteenth International Conference on Learning Representations , 2025.

S. Kou, J. Jin, Z. Liu, C. Liu, Y. Ma, J. Jia, Q. Chen, P. Jiang, and Z. Deng, 'Orthus: Autoregressive interleaved image-text generation with modality-specific heads,' in Forty-second International Conference on Machine Learning , 2025.

H. Liu, C. Li, Q. Wu, and Y. J. Lee, 'Visual instruction tuning,' Advances in Neural Information Processing Systems , vol. 36, pp. 34892-34916, 2023.

D. Chen, J. Liu, W. Dai, and B. Wang, 'Visual instruction tuning with polite flamingo,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, pp. 17745-17753, 2024.

J. He, H. Guo, M. Tang, and J. Wang, 'Continual instruction tuning for large multimodal models,' arXiv preprint arXiv:2311.16206 , 2023.

L. Wang, X. Zhang, H. Su, and J. Zhu, 'A comprehensive survey of continual learning: Theory, method and application,' IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

D. Yu, X. Zhang, Y. Chen, A. Liu, Y. Zhang, P. S. Yu, and I. King, 'Recent advances of multimodal continual learning: A comprehensive survey,' arXiv preprint arXiv:2410.05352 , 2024.

H. Guo, F. Zeng, Z. Xiang, F. Zhu, D.-H. Wang, X.-Y. Zhang, and C.-L. Liu, 'HiDe-LLaVA: Hierarchical decoupling for continual instruction tuning of multimodal large language model,' in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , (Vienna, Austria), pp. 13572-13586, Association for Computational Linguistics, July 2025.

A. Maharana, J. Yoon, T. Chen, and M. Bansal, 'Adapt-$\infty$: Scalable continual multimodal instruction tuning via dynamic data selection,' in The Thirteenth International Conference on Learning Representations , 2025.

Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. , 'Dualprompt: Complementary prompting for rehearsal-free continual learning,' in European Conference on Computer Vision , pp. 631-648, Springer, 2022.

D. Zhu, Z. Sun, Z. Li, T. Shen, K. Yan, S. Ding, C. Wu, and K. Kuang, 'Model tailor: Mitigating catastrophic forgetting in multi-modal large language models,' in International Conference on Machine Learning , pp. 62581-62598, PMLR, 2024.

T. Huai, J. Zhou, X. Wu, Q. Chen, Q. Bai, Z. Zhou, and L. He, 'Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-experts for continual visual question answering,' in Proceedings of the Computer Vision and Pattern Recognition Conference , pp. 19608-19617, 2025.

Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang, 'Emu: Generative pretraining in multimodality,' in The Twelfth International Conference on Learning Representations , 2024.

Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Y. Wang, Y. Rao, J. Liu, T. Huang, and X. Wang, 'Generative multimodal models are in-context learners,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14398-14409, 2024.

S. Tong, D. Fan, J. Zhu, Y . Xiong, X. Chen, K. Sinha, M. Rabbat, Y . LeCun, S. Xie, and Z. Liu, 'Metamorph: Multimodal understanding and generation via instruction tuning,' arXiv preprint arXiv:2412.14164 , 2024.

C. Wang, G. Lu, J. Yang, R. Huang, J. Han, L. Hou, W. Zhang, and H. Xu, 'Illume: Illuminating your llms to see, draw, and self-enhance,' arXiv preprint arXiv:2412.06673 , 2024.

D. Ghosh, H. Hajishirzi, and L. Schmidt, 'Geneval: An object-focused framework for evaluating textto-image alignment,' Advances in Neural Information Processing Systems , vol. 36, pp. 52132-52152, 2023.

H. Liu, W. Yan, M. Zaharia, and P. Abbeel, 'World model on million-length video and language with blockwise ringattention,' in The Thirteenth International Conference on Learning Representations , 2025.

Y. Wu, Z. Zhang, J. Chen, H. Tang, D. Li, Y. Fang, L. Zhu, E. Xie, H. Yin, L. Yi, S. Han, and Y. Lu, 'VILA-u: a unified foundation model integrating visual understanding and generation,' in The Thirteenth International Conference on Learning Representations , 2025.

H. Tang, H. Liu, and X. Xiao, 'Ugen: Unified autoregressive multimodal model with progressive vocabulary learning,' arXiv preprint arXiv:2503.21193 , 2025.

Y. Jin, K. Xu, K. Xu, L. Chen, C. Liao, J. Tan, Q. Huang, B. Chen, C. Song, dai meng, D. Zhang, W. Ou, K. Gai, and Y. Mu, 'Unified language-vision pretraining in LLM with dynamic discrete visual tokenization,' in The Twelfth International Conference on Learning Representations , 2024.

P. Esser, R. Rombach, and B. Ommer, 'Taming transformers for high-resolution image synthesis,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 12873-12883, June 2021.

L. Qu, H. Zhang, Y. Liu, X. Wang, Y. Jiang, Y. Gao, H. Ye, D. K. Du, Z. Yuan, and X. Wu, 'Tokenflow: Unified image tokenizer for multimodal understanding and generation,' in Proceedings of the Computer Vision and Pattern Recognition Conference , pp. 2545-2555, 2025.

X. Chen, Z. Wu, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, and C. Ruan, 'Janus-pro: Unified multimodal understanding and generation with data and model scaling,' arXiv preprint arXiv:2501.17811 , 2025.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. , 'Learning transferable visual models from natural language supervision,' in International Conference on Machine Learning , pp. 8748-8763, PmLR, 2021.

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, 'Sigmoid loss for language image pre-training,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 11975-11986, 2023.

A. Van Den Oord, O. Vinyals, et al. , 'Neural discrete representation learning,' Advances in Neural Information Processing Systems , vol. 30, 2017.

R. Huang, C. Wang, J. Yang, G. Lu, Y. Yuan, J. Han, L. Hou, W. Zhang, L. Hong, H. Zhao, et al. , 'Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement,' arXiv preprint arXiv:2504.01934 , 2025.

B. Mustafa, C. Riquelme, J. Puigcerver, R. Jenatton, and N. Houlsby, 'Multimodal contrastive learning with limoe: the language-image mixture of experts,' Advances in Neural Information Processing Systems , vol. 35, pp. 9564-9576, 2022.

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, 'Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,' in International Conference on Learning Representations , 2017.

C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, 'Scaling vision with sparse mixture of experts,' in Advances in Neural Information Processing Systems (M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, eds.), vol. 34, pp. 85838595, Curran Associates, Inc., 2021.

S. Shen, Z. Yao, C. Li, T. Darrell, K. Keutzer, and Y. He, 'Scaling vision-language models with sparse mixture of experts,' in The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. , 'Lora: Low-rank adaptation of large language models.,' ICLR , vol. 1, no. 2, p. 3, 2022.

Z. Chen, Z. Wang, Z. Wang, H. Liu, Z. Yin, S. Liu, L. Sheng, W. Ouyang, and J. Shao, 'Octavius: Mitigating task interference in MLLMs via loRA-moe,' in The Twelfth International Conference on Learning Representations , 2024.

X. Wu, S. Huang, and F. Wei, 'Mixture of loRA experts,' in The Twelfth International Conference on Learning Representations , 2024.

S. Yang, M. A. Ali, C.-L. Wang, L. Hu, and D. Wang, 'Moral: Moe augmented lora for llms' lifelong learning,' arXiv preprint arXiv:2402.11260 , 2024.

D. Li, Y. Ma, N. Wang, Z. Ye, Z. Cheng, Y. Tang, Y. Zhang, L. Duan, J. Zuo, C. Yang, et al. , 'Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts,' arXiv preprint arXiv:2404.15159 , 2024.

T. Luo, J. Lei, F. Lei, W. Liu, S. He, J. Zhao, and K. Liu, 'Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models,' arXiv preprint arXiv:2402.12851 , 2024.

J. Yu, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He, 'Boosting continual learning of visionlanguage models via mixture-of-experts adapters,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 23219-23230, 2024.

H. Zhao, Z. Wang, Q. Sun, K. Song, Y. Li, X. Hu, Q. Guo, and S. Liu, 'Llava-cmoe: Towards continual mixture of experts for large vision-language models,' arXiv preprint arXiv:2503.21227 , 2025.

R. Wang and P. Li, 'LEMoE: Advanced mixture of experts adaptor for lifelong model editing of large language models,' in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , (Miami, Florida, USA), pp. 2551-2575, Association for Computational Linguistics, Nov. 2024.

W. Chen, Y. Zhou, N. Du, Y. Huang, J. Laudon, Z. Chen, and C. Cui, 'Lifelong language pretraining with distribution-specialized experts,' in International Conference on Machine Learning , pp. 5383-5395, PMLR, 2023.

W. Pian, S. Deng, S. Mo, Y. Guo, and Y. Tian, 'Modality-inconsistent continual learning of multimodal large language models,' arXiv preprint arXiv:2412.13050 , 2024.

J. Qiao, Z. Zhang, X. Tan, Y. Qu, S. Ding, and Y. Xie, 'Llaca: Multimodal large language continual assistant,' arXiv preprint arXiv:2410.10868 , 2024.

J. Zheng, Q. Ma, Z. Liu, B. Wu, and H. Feng, 'Beyond anti-forgetting: Multimodal continual instruction tuning with positive forward transfer,' arXiv preprint arXiv:2401.09181 , 2024.

M. Cao, Y. Liu, Y. Liu, T. Wang, J. Dong, H. Ding, X. Zhang, I. Reid, and X. Liang, 'Continual llava: Continual instruction tuning in large vision-language models,' arXiv preprint arXiv:2411.02564 , 2024.

Y. Zhai, S. Tong, X. Li, M. Cai, Q. Qu, Y. J. Lee, and Y. Ma, 'Investigating the catastrophic forgetting in multimodal large language models,' arXiv preprint arXiv:2309.10313 , 2023.

F. Zeng, F. Zhu, H. Guo, X.-Y. Zhang, and C.-L. Liu, 'Modalprompt: Dual-modality guided prompt for continual learning of large multimodal models,' arXiv preprint arXiv:2410.05849 , 2024.

Y.-K. Zhang, S. Lu, Y. Li, Y. Ma, Q.-G. Chen, Z. Xu, W. Luo, K. Zhang, D.-C. Zhan, and H.-J. Ye, 'Wings: Learning multimodal llms without text-only forgetting,' Advances in Neural Information Processing Systems , vol. 37, pp. 31828-31853, 2024.

J. Zou, B. Liao, Q. Zhang, W. Liu, and X. Wang, 'Omnimamba: Efficient and unified multimodal understanding and generation via state space models,' arXiv preprint arXiv:2503.08686 , 2025.

C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie, Z. Song, G. Shi, and H. Fan, 'Emerging properties in unified multimodal pretraining,' arXiv preprint arXiv:2505.14683 , 2025.

P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, 'Learn to explain: Multimodal reasoning via thought chains for science question answering,' in The 36th Conference on Neural Information Processing Systems (NeurIPS) , 2022.

D. A. Hudson and C. D. Manning, 'Gqa: A new dataset for real-world visual reasoning and compositional question answering,' Conference on Computer Vision and Pattern Recognition , 2019.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in 2009 IEEE Conference on Computer Vision and Pattern Recognition , pp. 248-255, Ieee, 2009.

Z. Xu, M. Liu, Y. Shen, J. Rimchala, J. Zhang, Q. Wang, Y. Cheng, and L. Huang, 'Modality-specialized synergizers for interleaved vision-language generalists,' in The Thirteenth International Conference on Learning Representations , 2025.

Z. Ni, L. Wei, S. Tang, Y. Zhuang, and Q. Tian, 'Continual vision-language representation learning with off-diagonal information,' in International Conference on Machine Learning , pp. 26129-26149, PMLR, 2023.

C. Chen, J. Zhu, X. Luo, H. Shen, J. Song, and L. Gao, 'Coin: A benchmark of continual instruction tuning for multimodel large language models,' Advances in Neural Information Processing Systems , vol. 37, pp. 57817-57840, 2024.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. , 'Laion-5b: An open large-scale dataset for training next generation image-text models,' Advances in neural information processing systems , vol. 35, pp. 25278-25294, 2022.

J. M. Joyce, Kullback-Leibler Divergence , pp. 720-722. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011.

A. Singh, V. Natarjan, M. Shah, Y. Jiang, X. Chen, D. Parikh, and M. Rohrbach, 'Towards vqa models that can read,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 8317-8326, 2019.

D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, 'Vizwiz grand challenge: Answering visual questions from blind people,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 3608-3617, 2018.

N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J.-Y. Zhu, 'Multi-concept customization of text-toimage diffusion,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1931-1941, 2023.

J. S. Smith, L. Karlinsky, V . Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira, 'Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11909-11919, June 2023.

J. Qiao, X. Tan, C. Chen, Y. Qu, Y. Peng, Y. Xie, et al. , 'Prompt gradient projection for continual learning,' in The Twelfth International Conference on Learning Representations , 2023.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, 'Gans trained by a two time-scale update rule converge to a local nash equilibrium,' Advances in Neural Information Processing Systems , vol. 30, 2017.

E. Chern, J. Su, Y. Ma, and P. Liu, 'Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation,' arXiv preprint arXiv:2407.06135 , 2024.

S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg, 'ReferItGame: Referring to objects in photographs of natural scenes,' in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (A. Moschitti, B. Pang, and W. Daelemans, eds.), (Doha, Qatar), pp. 787-798, Association for Computational Linguistics, Oct. 2014.

J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, 'Generation and comprehension of unambiguous object descriptions,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 11-20, 2016.

A Theoretical Analysis

We formally compare the one-step visual loss drift between a modality-coupled architecture (shared parameters for text and image) and our modality-decoupled MoDE design. Let assume the following parameters:

θ the full set of trainable parameters,

L t ( θ ) the text-generation loss,

L v ( θ ) the image-generation loss,

η the learning rate,

g t = ∇ θ L t and g v = ∇ θ L v the corresponding gradients of text and visual modalities, respectively,

H v = ∇ 2 θ L v the Hessian of the visual loss.

After a gradient step θ ← θ -ηg t , the change in the visual loss is, by a second-order Taylor expansion:

where o ( η 2 ) denotes higher-order terms that vanish faster than η 2 .

Modality-coupled architecture (shared parameters). In the modality-coupled case (e.g., shared MoE-LoRA adapters between modalities), both g t and g v are nonzero over the same parameter space. Thus, the first-order term -η ⟨ g t , g v ⟩ is generally nonzero. In the worst case (maximum interference), g t and g v are anti-aligned, so by Cauchy-Schwarz inequality we have:

Thus, the dominant error scales as O ( η ) .

Modality-decoupled architecture (MoDE). In MoDE, the text and visual tokens update disjoint parameter blocks ϕ and ψ , respectively; thus:

Therefore, the first-order term in (13) vanishes, and the remaining visual loss drift is purely secondorder:

Thus, the dominant error scales as O ( η 2 ) .

Conclusion. Comparing(14) and(15), we conclude that

which implies since η 2 ≪ η for small η .

Thus, MoDE provably limits the worst-case drift in image-generation loss to O ( η 2 ) , compared to O ( η ) in a modality-coupled architecture.

B Modality gradient conflict

The theoretical support is provided in Appendix A. This section provides empirical evidence (Figure 5, 6) for our claim.

As shown in Figure 5, the modality-coupled MoE LoRA [44] displays significant gradient conflict between text and image modalities, empirically validating our analysis in Section 3.

Modality Gradient Conflict

0.5

0.4

0.3

0.2

0.1

0.0

0.1

0.2

Cosine

0.00

0.05

0.10

0.15

0.20

0.25

Ratio of parameters

In contrast to the significant conflict shown in Figure 5, Figure 6 clearly shows that MoDE has zero cosine disagreement between modalities. This confirms that its modality-specific update subspaces are completely isolated, both theoretically and empirically.

Modality Gradient Conflict

1.0

parameters

0.8

0.6

6

0.4

R

0.2

0.0

-0.4

-0.2

0.0

0.2

Cosine

C Additional Qualitative Results

Table 4 presents the complete qualitative results of MoDE, built on Chameleon [3], in comparison to various baseline methods. Our approach generates images that better align with the prompts and exhibit higher visual fidelity. For example, given the prompt ' Barn in the fall season with leaves all around ', only the original Chameleon and our MoDE successfully generate images featuring yellow leaves around the barn.

Method

A dog wearing

sunglasses on the

porch

A transparent cup

filled with steaming

hot cocoa

Barn in the fall season

with leaves all around

Marigold flowers in

the vase

Chameleon [3]

Seq LoRA

Model

Tailor [17]

DualPrompt [16]

MoELoRA [44]

CL-MoE [18]

MoDE (Ours)

?

B

D

D Implementation Details

We use Chameleon [3] and Janus-Pro [30] as backbone models. For Chameleon, we adopt the implementation from [72] due to the unavailability of the original checkpoints and code. All LoRA modules are configured with a default rank of 8. Training is conducted for one epoch per task on 8 NVIDIA RTX 6000 GPUs, with bf16 mixed precision and a per-GPU batch size of 1 (total effective batch size of 8).

We use a learning rate of 1 × 10 -4 , scheduled via cosine decay with a warm-up ratio of 0.1. The softening factor β in knowledge distillation equation (10) is 2.0. The hyperparameter λ is set to 0.3, selected based on a sweep over multiple values (see Table 5). This value strikes a balance between image generation and visual understanding: higher λ favors generation quality, while lower λ improves understanding performance. A higher λ increases the weight of the knowledge distillation loss in the V-adapter (see Eq. 12), constraining the update closer to the pre-trained model to preserve image generation capabilities. However, this also constrains beneficial updates needed for improved visual understanding.

E Additional Ablation Results

E.1 Different Tasks Order

Following [63], we explore the impact of different tasks order by conducting an additional experiment using a different order of tasks, arranged alphabetically: GQA [59], ImageNet [60], ScienceQA [58], TextVQA [66], and VizWiz [67].

As shown in Table 6, our MoDE consistently outperforms the baseline methods, thus demonstrating its robustness to different task orderings.

E.2 Different Number of Experts

This this section we provide an ablation study on the number of experts in the T-MoE module.

As shown in Table 7, increasing the number of experts yields slightly improved text alignment and accuracy, though the gains are modest. This suggests that the performance of MoDE is relatively robust to the number of experts, and 4 experts offer a good trade-off between performance and computational cost.

E.3 Ablation on Knowledge Distillation and Modality Decoupling

We provide the results of adding our cross-modal knowledge distillation (KD) loss to modalitycoupled baselines (SeqLoRA [39], CL-MoE [18], MoELoRA [63]) in Table 8. Although applying KD to vanilla LoRA leads to modest improvements both in image generation quality and multimodal understanding performance, we observe that for MoELoRA and CL-MoE this mitigates visual-generation forgetting at the cost of hurting the multimodal understanding performance. This shows that without explicit modality decoupling, KD 'locks in' the teacher's features in a way that hinders the model's ability to adapt to new multimodal understanding tasks. The results come from a single run.

As shown in Table 9, decoupling textual and visual LoRA without knowledge distillation yields only modest gains in understanding accuracy and visual generation quality compared to vanilla LoRA. This indicates that decoupling alone is insufficient to maintain the rich cross-modal alignment necessary for high-quality generation. Furthermore, Table 3 includes an ablation on 'MoDE w/o KD' (i.e., decoupled T-MoE and V-adapter without knowledge distillation), which further confirms that knowledge distillation plays a critical role in the performance improvements observed.

The ablation results show that MoDE is more than the sum of its parts: only the combination of modality-decoupling plus knowledge distillation leads to consistently superior performance.

F Multimodal Understanding Forgetting

In this section, we provide the results of MoDE mitigating catastrophic forgetting of the understanding capabilities induced by fine-tuning UMGMs on image generation tasks. To address this, we conduct new experiments by fine-tuning the linear layers of the Chameleon model [3] on image generation tasks. Specifically, we use the LAION-5B dataset [64] for fine-tuning and apply knowledge distillation using the multimodal understanding sequence from the GQA dataset [59] as reference data. We compare our MoDE method with LoRA fine-tuning across three multimodal understanding benchmarks. Accuracies are reported in Table 10. The difference between zero-shot and fine-tuned performance reflects the degree of forgetting in multimodal understanding.

The results demonstrate that fine-tuning on image generation tasks can indeed cause catastrophic forgetting in multimodal understanding. This is evident because of the substantial drop in performance for the LoRA baseline compared to zero-shot. Our MoDE method remains effective in this scenario, significantly reducing the performance drop and mitigating forgetting relative to the LoRA baseline. This demonstrates that MoDE generalizes beyond the specific forgetting direction explored in the main paper, where understanding tasks overwrite prior image generation capabilities, and is effective regardless of the fine-tuning direction.

G Additional Continual Learning Results

G.1 Additional Results with Janus-Pro

We provide additional quantitative comparisons using Janus-Pro-1B [30] as the backbone. We further include the Grounding task with referring expression grounding datasets (RefCOCO [73], RefCOCO+ [74], and RefCOCOg [74]). All methods are trained with early stopping. As shown in Table 11, our MoDE outperforms the baseline methods in continual instruction tuning for multimodal understanding.

G.2 Additional Results with Chameleon

Following the CoIN benchmark [63], we present results on additional datasets using Chameleon [3] as the backbone. We further include the Grounding task with referring expression grounding datasets (RefCOCO [73], RefCOCO+ [74], and RefCOCOg [74]). All methods are trained with early stopping. As shown in Table 12, our MoDE outperforms the baseline methods in continual instruction tuning for multimodal understanding.

H Computational Analysis

In this section, we report comparisons of training compute, memory footprint, and trainable parameter size.

As shown in Table 13, MoDE introduces only a slight increase in training time (440.3 ms vs. 419.2 ms) and peak GPU memory (84.98 GB vs. 75.27 GB) compared to MoELoRA [63], while maintaining comparable TFLOPs. The trainable parameter ratio of MoDE is only 0.0211%, modestly higher than MoELoRA [63] and CL-MoE [18] (0.0171%), indicating that our performance gains are not simply due to increased processing capacity. These results suggest that MoDE's improvements arise from its design (modality-decoupled experts and distillation), rather than significantly higher resource usage.

I Baseline Settings

In this section, we provide the experimental settings for the baseline methods used in our experiments.

I.1 Overview of Baselines

Model Tailor [17] mitigates catastrophic forgetting in MLLMs by (i) identifying a sparse 'model patch' via a fused salience-and-sensitivity score, and (ii) applying Hessian-based weight compensation so the patch can be safely merged back into the frozen backbone.

DualPrompt [16] is a replay-free continual learning method grafting two tiny prompt sets onto a frozen transformer backbone: a shared G-Prompt that encodes task-invariant knowledge and per-task E-Prompts keyed to input features for task-specific skills.

MoELoRA [44] extends LoRA by substituting each single low-rank adapter with a mixture of N experts . The LoRA update is rewritten as:

where every expert has rank r/N and is gated by G i , so the total trainable parameters remain identical to vanilla LoRA while the separate experts capture more diverse, task-specific representations.

CL-MoE [18] tackles continual VQA on top of an MLLM by (i) introducing a Dual-Router MoE : a task-level router picks global experts while an instance-level router refines the choice per example, and (ii) applying a dynamic momentum update that blends new LoRA-based expert weights with the frozen past experts, so the model absorbs new skills yet preserves prior knowledge, sharply reducing catastrophic forgetting.

Seq LoRA is a vanilla baseline that applies LoRA adapters in plain sequential fine-tuning across tasks.

I.2 Training Settings

To ensure a fair comparison, we follow the hyperparameters and implementation details from each baseline's original codebase:

Model Tailor [17]: sparsity level set to 10% (percentage of parameters replaced).

DualPrompt [16]: prompt lengths L g = 5 and L e = 20 ; prompts are injected at layers start g = 1 , end g = 2 , start e = 3 , and end e = 5 .

MoE-LoRA [44]: number of experts set to 4, with each expert having LoRA rank 8.

CL-MoE [18]: number of experts set to 4, with each expert having LoRA rank 8. TopK experts selected with K = 2 . Hyperparameters γ = 0 . 7 , β = 0 . 5 .

J Ethics Statement

This work does not involve human subjects, personally identifiable information, or proprietary data. All datasets used (such as ScienceQA, TextVQA, GQA, VizWiz, and ImageNet) are publicly available and commonly used in the research community. We follow the licensing and usage policies associated with each dataset. Our continual learning method, MoDE, improves the robustness of unified multimodal generative models without modifying or exploiting sensitive data.

As with all powerful generative models capable of producing synthetic images from text, our method carries a potential risk of misuse, such as generating disinformation or harmful visual content. Although MoDE focuses on mitigating forgetting in multimodal learning, enhanced generation capabilities may inadvertently facilitate deceptive or malicious use. We recommend that practitioners deploying such models consider safeguards such as content filtering, provenance tracking, and visible or invisible watermarking of generated content to support responsible use and attribution.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main contributions of this paper is identifying and tackling the challenge of inter-modal catastrophic forgetting in unified multimodal models, which are accurately reflected in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our method is applicable to autogressive transformers, which is made clear in the paper. Our experiments are conducted on autoregressive transformers as well.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the theoretical results on modality gradient conflict in the main paper. The proofs and empirical verification are in the appendix A and B, respectively.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our provided details of our architecture as well as experimental details in the paper. And our code would be released upon paper acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The model checkpoints and datasets we use in this paper are all open-sourced. And we will provide our code together with the other supplemental materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include these details in the supplemental materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not applicable to our experiments involving generative models, where evaluation is primarily qualitative or based on metrics not requiring statistical significance testing.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources information our experiments need is provided in the experiment section of the paper as well as in the supplemental materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: There is no ethical concerns awared in this paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The potential societal impact is discussed in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We don't release any pre-trained models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The models and datasets we use in the paper have been propoerly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code of this paper is well documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this paper does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Modality-Decoupled Experts (MoDE) for continual learning of unified multimodal generative models (UMGMs). MoDE consists of separate LoRA blocks for the vision component and the text components (multiple LoRAs for text sub-tasks). The authors further regularize the continual learning process with standard knowledge distillation from the base model.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. Misalignment in the gradient direction between the two tasks in UMGMs is well-known, and an important problem to address, given the increasing attention in developing a GPT-4o-like UMGM systems.\n\n2. To the best of my knowledge, most concurrent works focus on tweaking the model architecture the enhance the model capabilities. This paper looks at the problem through a continual learning lens.\n\n3. The method seems to be effective for alleviating catastrophic forgetting for image generation while fine-tuning on instruction tuning datasets.\n\n## Weaknesses\n\n1. The paper would be much more interesting if the authors tested their method on the case where the task is to prevent catastrophic forgetting of text generation while improving image generation. More importantly, a more realistic setting would be to test the method when the model is trained in an alternating fashion. Would the method still work on these settings?\n\n2. Proposition 1 states that *when* gradients conflict, then the loss would increase, which is trivial.\n\n3. The core idea (separate LoRA adapters for module + KD) is incremental.\n\n4. There are several concurrent works that aim for better unified modeling by e.g. using a mixture of experts [1] or using separate vision encoders [2]. It is advised that the authors discuss the relevant works, including whether their contributions are orthogonal to the other approaches.\n\n**References**\n\n[1] Deng, Chaorui, et al. \"Emerging properties in unified multimodal pretraining.\" arXiv preprint arXiv:2505.14683 (2025).\n\n[2] Zou, Jialv, et al. \"Omnimamba: Efficient and unified multimodal understanding and generation via state space models.\" arXiv preprint arXiv:2503.08686 (2025).\n\nQUESTIONS:\n1. What happens when one simply uses KD on top of seq LoRA?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-09"}

comments: SUMMARY:
In this work, the authors propose Modality-Decoupled Experts (MoDE). This architecture handles intra- and inter-modal forgetting for Unified Multimodal Generative Models (UMGMs). MoDE consists of two modules: a V-adapter (a LoRA module) used to capture visual knowledge and a T-MoE, a LoRA Mixture of Experts that handles textual information. By decoupling the vision and language modalities, the authors claim to reduce forgetting when training the model on a sequence of tasks. The authors provide a wide range of experiments to validate their thesis.

STRENGTHS AND WEAKNESSES:
### Strengths

- The paper is well-written and easy to follow.

- The problem of dealing with forgetting in both modalities of a UMGM is interesting and worth exploring.

- The experiments section and appendix are exhaustive

### Weaknesses

- While the results are consistently better than previous approaches, they are incremental with respect to CL-MoE (+0.1 in Tab. 2). This may raise some questions about the approach's statistical validity. Multiple seeded runs would have helped to understand this aspect better.

- The V-Adapter and T-MoE have never really been introduced. They are explained only in the caption of Fig. 3.

- Details and ablations on the number of experts in T-MoE are missing.

QUESTIONS:
1. How does MoDE compare to CL-MoE regarding time and memory costs?

2. The reference dataset is not clear. I understand it is used for the KD loss, but how is it composed? Is it task-related, or is it always the same?

Please refer also to the weaknesses.

LIMITATIONS:
The authors have not discussed limitations. One may be the hardware requirements.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is the first to investigate the inter-modal catastrophic forgetting issue in Unified Multimodal Understanding and Generation Models (UMGMs) during continual instruction tuning. While previous work on multimodal continual instruction tuning has focused solely on understanding tasks, this paper explores how UMGMs forget their visual generation capabilities after being continually fine-tuned on multimodal understanding tasks. The authors attribute this forgetting to gradient conflicts between image generation loss and text generation loss with respect to shared model parameters and provide a theoretical explanation for this phenomenon. To mitigate this problem, the authors propose a modality-decoupled expert architecture. Specifically, they introduce a MoELoRA module for text token generation (referred to as T-MoE), and a separate LoRA module for image token generation (referred to as V-Adapter). To preserve the image generation ability, they further apply knowledge distillation from the original (teacher) UMGM to the V-Adapter in the new (student) model.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n- The paper is the first to investigate the catastrophic forgetting of image generation capabilities in UMGMs under continual instruction tuning.\n\n- A theoretical explanation is provided by analyzing the gradient conflicts between image token generation loss and text token generation loss with respect to shared model parameters.\n\n\nCons:\n\n- The paper assumes that image generation is only conducted during pretraining and that continual instruction tuning only involves multimodal understanding tasks. If the model is further fine-tuned on image generation tasks, will it then catastrophically forget the understanding tasks? Can the proposed method prevent this problem as well?\n\n- The “Reference Token” in Figure 3 is not clearly explained. When knowledge distillation is applied to image generation during continual tuning, what text prompt is used as the reference?\n\nQUESTIONS:\n- The MoELoRA (CoIN) benchmark includes 8 multimodal understanding tasks: ScienceQA, TextVQA, ImageNet, GQA, VizWiz, Grounding, VQAv2, and OCR-VQA. Why does the evaluation in this paper only cover the first six tasks? What are the results of all 8 tasks?\n\n- In the supplementary material, the results for DualPrompt and MoELoRA are missing in the different task ordering section. Also, the Janus-Pro results are missing for DualPrompt, Model Tailor, and MoELoRA. Could these be included?\n\n- According to Table 2, MoDE only shows marginal improvements in multimodal understanding performance (both in the best accuracy during continual instruction tuning and the final accuracy after completing all tasks) compared to MoELoRA and CL-MoE. The main benefit seems to lie in mitigating the forgetting in image generation. If knowledge distillation for image generation were directly applied to MoELoRA and CL-MoE, how would they perform? Additionally, can the authors report the detailed accuracy on previous tasks after each task training step?\n\n- In the current tables, bold fonts highlight the results of the proposed method. It might be clearer to instead bold the best results for each metric.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the challenge of catastrophic forgetting in Unified Multimodal Generative Models (UMGMs), focusing particularly on the phenomenon of inter-modal forgetting, alongside intra-modal forgetting. The paper identifies gradient conflict between visual and textual modalities as a key source of inter-modal forgetting and provides several theoretical contributions showing that such gradient conflicts directly cause inter-modal forgetting. The authors propose Modality-Decoupled Experts (MoDE), a straightforward approach that decouples visual and textual modality updates using modality-specific LoRA adapters, along with knowledge distillation from a frozen visual generation teacher to prevent intra-modal forgetting. MoDE outperforms competitive baselines such as MoE-LoRA and Model-Tailer on a number of multimodal continual learning tasks, and significantly reduces both intra- and inter-modal forgetting.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- **Novel Theoretical Contributions:** The paper makes a number of theoretical formulations which are interesting and broadly useful:\n    - The proposed formulation of the Modality Gradient conflict, showing that an increase in image generation loss (forgetting) depends on the inner product of the formulated visual and text gradients is interesting. The formulation also shows how the proposed MoDE method provably bounds inter-modal interference to quadratic in step sizes.\n    - These are generally useful formulations, and would be of interest to future work in mitigating catastrophic forgetting in Unified Multimodal Generative Models (UMGMs). \n\n- **Strong results in continual instruction tuning \\& mitigation of catastrophic forgetting:**\n    - In both mitigation of catastrophic forgetting in image generation and visual understanding, as well as in continual instruction tuning experiments the proposed method MoDE almost consistently outperforms strong baselines like Model-tailor and MoE-LoRA. \n    - While this may be due to the trivial modality decoupling in the proposed MoDE, the approach achieves lower forgetting (and higher final accuracy) on almost all tasks. \n\n- **Well motivated study:** \n  - The unification and study of intra-model and inter-modal forgetting is interesting and well motivated.\n\n**Weaknessess:**\n\n- **Core method seems to lack novelty and significance**: The core mechanisms in the proposed MoDE method leverage existing ideas in modality-specific and MoE adapters. \n  - To prevent inter-modal forgetting, the proposed method involves de-coupling both visual and textual modalities by learning separate modules - Mixture-of-Expert LoRAs for Text and a Visual LoRA for vision. This seems like a trivial solution, and re-uses existing modules (MoE-LoRA, LoRA for vision adaptation). \n  - In addition, to prevent intra-modal forgetting (the authors only consider visual modality forgetting), MoDE uses a straightforward knowledge distillation with a frozen pre-trained teacher backbone. \n\n\n- **Strong results in large part due to modality decoupling, Missing decoupled baselines:** \n    - While the results of the proposed approach themselves are strong, it seems this is largely a result of the trivial modality decoupling used in the proposed approach. \n    - There is no clear baseline comparing the proposed approach to a trivial modality decoupled system with separate image-generation and text/multimodal understanding modules. It is therefore difficult to gauge the usefulness of the proposed approach. \n\n\n- **Missing Parameter and Computational costs comparisons:**\n    - Since the proposed MoDE essentially leverages separate LoRA adapters for both modalities as well as for different multimodal understanding tasks, it would be interesting to see a comparison of the trained parameters and the training computational costs between the proposed method and baselines. \n\n- **Text-only intra-modal forgetting:** Inter-modal forgetting is missing forgetting caused to text-only generation tasks, while only visual generation has been considered. This seems like a critical aspect of inter-modal forgetting. See [1,2]:\n\n[1]: Zhang, Yi-Kai, et al. \"Wings: Learning multimodal llms without text-only forgetting.\" Advances in Neural Information Processing Systems 37 (2024): 31828-31853.\n\n[2]: Srivastava, Shikhar, et al. \"Improving multimodal large language models using continual learning.\" arXiv preprint arXiv:2410.19925 (2024).\n\n- **Minor corrections:**\n   -  VQA tasks are repeatedly referred to as text-only (Line 122-123, for instance) in the paper. While the generation is in text, the task however is _not_ \"text-only\" since these are VQA.  This is very odd, and must be clarified/corrected in the paper. \n    - Line 122-123, Line 50: \"(e.g., improving language generation while preserving visual generation)\" where this isn't language generation, it's multimodal understanding.\n\nQUESTIONS:\n1. A comparison of the trainable parameter and computational costs between the proposed method and the baselines would help alleviate concerns of triviality of the proposed method.\n2. An ablation showing a trivial fully de-coupled system with separate image-generation and text/multimodal understanding modules, would be helpful.\n3. Does Figure 1 training use LoRA adapters to measure forgetting? Why was forgetting caused by full fine-tuning not evaluated?\n\n_Points 1 and 2 above, as well as a clarification of the first 3 points expressed in the Weaknesses section, would go farthest in helping alleviate concerns._\n\nLIMITATIONS:\nYes, Addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}