Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models
Authors: Xiwen Wei
Mustafa Munir
Radu Marculescu
Venue: NeurIPS 2025 poster ()
Keywords: Continual Learning
Multimodal Generative Models
Catastrophic Forgetting
Mixture of Experts (MoE)
Knowledge Distillation
Primary area: deep_learning

Abstract: Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings.

Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models
Xiwen Wei, Mustafa Munir, Radu Marculescu
Department of Electrical and Computer Engineering The University of Texas Austin (xiwenwei, mmunir, radum)@utexas edu
Abstract
Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forget- ting, both within a modality (intra-modal) and across modalities (inter-modal.. While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we iden- tify and empirically validate this phenomenon in UMGMs and provide a theo- retical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, propose (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL base- lines in unified multimodal generation settings. Codes are publicly available: http://ggfuub.oooCCCrrriahhaaaoe
-eellty--eecuppedd Experts
Introduction
Traditional multimodal models are typically divided into two categories: multimodal understanding (e.g. answering questions about images) and multimodal generation (e.g. generating images from text prompts) I 2). Unified Multimodal Generative Models (UMGMs) aim to integrate these two tasks within single framework. Recent advancements in UMGMs 3 have demonstrated strong performance on wide range of tasks, such as visual question answering (VQA), image captioning, visual reasoning, classification, reading comprehension, and image generation. These models typically embed multiple input types into a shared representation space and leverage one single transformer backbone to model cross-modal interactions. Training generally follows two- stage paradigm: an initial pretraining phase focused on text-image aligmment, followed by fine-tuning for specific downstream tasks. During the second stage, instruction tuning (which fine-tunes the model on diverse, task-specific instructions paired with expected outputs) has emerged as a widely adopted strategy better align model behavior with human intent 9 TO
While UMGMs demonstrate impressive sero-shot performance on a wide range of unseen instructions, their effectiveness is inconsistent across all tasks, particularly when the relevant task data is absent from the pretraining corpus. Expanding the training dataset to include new task data can significantly improve performance those tasks. However, given the continual emergence of novel multimodal
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Reading Comprehension
100
Upper
Start End
Image
SwaLoNA
DuelPhompt IECCV
Kodel Tolker (IMM) (ilcr 24)
MoDE (Ours)
Accursty (%)
60
Knowledge
VQA
29
Visual Reasoning
ScienceGA
TeXtVOA
Imagetet Task
GDA
VieViz
Blind People
VQA
(b)
Figure 1: (a) illustrates catastrophic forgetting during naive sequential instruction tun- ing. Continual instruction tuning starts with the Chameleon [3] model on tasks "Start" refers to the performance immediately after the model has been tuned on task, while "End" denotes performance after completing all tasks. A larger gap between the two bars indicates more severe forgetting. (b) visualizes forgetting in both multimodal generation tasks (in red, representing inter-modal forgetting) and multimodal understanding tasks (in green, representing intra-modal forgetting). Our proposed MoDE mitigates both types of forgetting, preserving performance across modalities.
tasks, repeatedly merging new data and retraining the model is computationally expensive and impractical. This motivates the need for new approaches that can enable UMGMs to flexibly and efficiently adapt over time. This goal aligns with the principles of continual learning (CL), where models are designed to acquire new capabilities incrementally, similar to how humans learn 1
Existing studies in CL have shown that sequentially fine-tuned models suffer from catastrophic forgetting, a phenomenon where a model fine-tuned on new tasks tends to forget or overwrite the previously acquired knowledge [12] More recently, several works have explored continual instruction tuning for multimodal large language models (MLLMs) B [5]. However, these models are limited to language-only outputs and, as result, prior research has focused exclusively tasks such as VQA and other multimodal understanding benchmarks. In contrast, UMGMs are capable of both multimodal understanding and generation within single backbone, enabling the production of both textual and visual outputs and thus representing a fundamentally different paradigm. This raises new and largely unexplored challenge, whereby forgetting occurs not only within a single modality (intra-modal), but also across modalities (inter-modal) during continual instruction tuning of UMGMs.
Starting from these overarching ideas, we ask: Can UMGMs continually acquire new capabilities through instruction tuning without suffering catastrophic forgetting, both within modality (e.g.. multimodal understanding) and across modalities (e.g.. improving multimodal understanding while preserving visual generation)?
As shown Figure UMGMs do experience catastrophic forgetting across range of tasks, including visual reasoning, classification, VQA, and image generation. Also, while existing methods [66] 17 can mitigate either intra-modal or inter-modal forgetting, none of them can effectively address both issues simultaneously.
To address this fundamental challenge, we propose Codaalty-Decouplee Experts (MoDE), new lightweight architecture which incorporates modality- sparse mixture of LoRA adapters for text (T-MoE) and single LoRA adapter for images (V-Adapter). The intra-men mora-modal forgetting during continual multimodal understanding tasks is mitigated by the T-MoE's routing, which selectively activates the appropriate experts. The inter-modal forgetting in image generation is prevented through modality decoupling and knowledge distillation between the pre-trained model (teacher) and the newly added image LoRA (student). During continual instruction tuning, only the MoDE components trained, while the pre-trained UMGM parameters remain frozen to preserve their robust cross-

---

modal alignment. Our results show that this design simultaneously mitigates both motral and inter-modal catastrophic forgetting.
Our contributions are:
1. We identify and illustrate the unique challenge of inter-modal catastrophic forgetting, in addition to the more common intra-modal forgetting in autoregressive transformers that unify multimodal understanding and generation under continual instruction tuning. We further provide a theoretical analysis attributing inter-modal forgetting to modality gradient conflict, and formally prove that modality decoupling can mitigate this conflict both theoretically and experimentally.
2. We propose Modality- Deeouppedd Experts (MoDE), a new lightweight and scalable archi- tecture that tackles both intra- and inter-modal forgetting by decoupling modality-specine updates (to reduce inter-modal gradient conflict) and applying knowledge distillation to preserve pre- pre-trrined image generation capabillitie.
3. Through extensive experiments, we show that MoDE outperforms SOTA baselines (e.g. CL- MoE [8], Model Tailor [77] when mitigating both intra- and inter catastrophic forgetting simultaneously. This opens up new possibilities for scalable continual learning in UMGMs.
The remainder of this paper is organized as follows: Section reviews related work. Section introduces modal forgetting in UMGMs. Section 4 presents our proposed method, and Section 5 details the experimental results. Finally. Section concludes our paper.
2 Related work
2.1 Unified multimodal generative models (UMGM)
Early efforts to unify visual understanding and generation 19 typically combine diffusion models with MLLMs, conditioning the diffusion process on LLM- L.LM--eneraeed embeddings. While effective for certain tasks, this "diffusion+MMLLM" design lacks tight coupling between image generation and language modeling, resulting in suboptimal performance on instruction- based generation 23
More recent approaches treat both understanding and generation unified next-token prediction problem [6] 3 These models differ in their encoder and decoder designs, e.g. some B 6 27
employ vision tokenizers such as VQGAN (28 discretize images into token sequences, while others [29] 30) use semantic encoders like CLIP [1 or SigLIP (32) to represent images as sequences
of continuous embeddings. Their decoders typically rely on either VQVAE [K or diffusion-based generator 34 22), but all maintain an autoregressive transformer backbone.
Our method is designed for transformer- UMGMs and works with broad class of models built on this autoregressive architecture, regardless of their choice of encoding or decoding mechanism.
2.2 Mixture of experts (MoE)
The Mixture of Experts architecture 35 38
combines specialized experts with gating mechanism that efficiently allocates tokens. Several studies explore integrating MoE with LoRA [39] to facilitate scalable and efficient training of MLLMs and LLMS 40:45). Recently, the scalability and modularity of MoE have attracted growing attention continual learning [18 46 47). For instance, Lifelong- MoE 48 introduces a strategy to train new experts while freezing the old ones. However, prior MoE-based CL methods share the same set of experts across modalities, which leads to modality interference during multimodal generation and exacerbates the inter-modal forgetting. Motivated by these limitations, we propose modality-
MoE with LoRA, aiming at mitigating both inter- modal forgetting caused by modality interference and intra-modal forgetting within each modality.
2.3 Catastrophic Forgetting
Catastrophic forgetting remains fundamental challenge in multimodal continual instruction tuning (MCIT), where a multimodal generative model is incrementally adapted to new datasets and task instructions without costly retraining from scratch. Existing methods to address catastrophic forgetting in MCIT typically fall into four categories: regularization-based [4), prchitecture-base
[50] 14].
3

---

Prompts: "A photo of barn" "A photo of a cat" "A photo of a plushie" photo of a co
Continua instruction tuning
Pre-trained UMGM
CUP: 0.3248
CUP: 0.2915
CLIP:0.3082
CLIP:0.3422
Which of these farthest north?
A. West Viggniaa
B. Loubiana Anizznna
VQA Task #1
0.3660.3067
CL:5-
the sky dark? Answer quest on using single word or phrase.
VQA Task #2
CLIP: 0.1769
CLIP: 0.2540
CLIP:0.2532
CLIP: 0.2499
image? Answer question using a single word phrase.
VQA Task #3
0.161
CLIP: 0.2885
CLIP: 0.2525
Figure 2: Inter-modal catastrophic forgetting during continual instruction tuning across three VQA tasks. The pre-trained UMGM serves as the upper bound for image generation quality. CLIP scores under each sample reflect text-image alignment in CLIP feature space. Red bounding boxes highlight regions with degraded image quality and low CLIP scores, indicating increasing misalignment between prompts and generated images. For instance, the image generated for the prompt "A photo of car" depicts a building instead of car (VQA task #3).
replay-based [15], and prompt-based approaches [1 52 For instance, EProj applies task- Limilarity-basee regularization to constrain model updates that are important for previous tasks. Fwd-Prompt (51) selects prompts using both textual and visual features to preserve earlier knowledge. LLaCA (50] employs exponential moving average (EMA) to merge the weights of old and new models, thereby maintaining both past task performance and pre-trained capabilities.
However, these approaches address only the intra-modal catastrophic forgetting, where tasks, new tasks, and pre-trained capabilities all belong to the same output modality, typically text (c.g., in VQA tasks) [52]54]. While this assumption holds for traditional multimodal models with unimodal outputs, is insufficient for UMGMs which generate multimodal outputs such text and images.
Inter-modal Catastrophie Forgetting in UMGMs
Inter-modal catastrophic forgetting manifest in several forms. For instance, prior work [55 examined the loss of text-only generation abilities in MLLMs when compared to their original LLM counterparts. Other studies have explored modality decoupling strategies in encoders and adapters of UMGMs [56] 57]. In this paper, we restrict our focus to multimodal tasks, specifically the forgetting that occurs multimodal understanding and generation. In particular, study visual generation forgetting that arises when tuning models on understanding tasks. Complementary results on understanding forgetting induced by tuning on image generation tasks are provided in Appendix F
To investigate inter-modal catastrophic forgetting in UMGMs from the visual perspective, we con- duct preliminary experiment by sequentially tuning Chameleon B3] using LoRA three VQA datasets [58 (0), while monitoring image generation performance with prompt-based evaluation. observe two key issues in image generation tuning progresses: 1) The overall image quality degrades significantly, 2) The alignment between the generated image and the input prompt worsens over time.
As shown in Figure
These findings indicate that inter-modal catastrophic forgetting does indeed occur in UMGMs when they continually fine-tume munet sequence of multimodal understanding tasks. We attribute this

---

forgetting to the UMGMs utilizing unified architecture (e.g., transformer) with shared parameters for both text and visual modalities [6]. When tuning on a single modality (e.g.. text), the shared parameters are updated in way that interferes with the model's performance on the other modality (e.g., image), phenomenon consistent with observations reported recent studies 62 this interference, we introduce the notion of modality gradient conflict.
To formalize
VoLy denote the gradient of the image- generation loss and St VoL. denote the gradient of the text-generation loss, both with respect to shared model parameters 0. 9v and Jt. are said to be conflicting with each other if (gu- 9t) < 0, where - denotes the standard Euclidean inner product.
Definition (Modality Gradient Conflict). Let 9v =
Proposition 1. When fine-tuning on the text- sett-genrratinn tasks, stochastic gradient descent (SGD) update with sufficiently small step size 1i modifies the model parameters as 0 + - 1999. The resulting change in the visual loss is:
AC, L, ng.)) C,( -7 (g.. +
2 91 H.
(I)
where H,, = ViL. , is the Hessian of the visual loss. Hence, when (gt, 9n) < 0, a step optimizing text generation will increase the visual loss, leading inter-modal forgetting.
Motivated by this, we propose MoDE, a modality-decoupled adapter framework that provably elimi- nates the first-order modality conflict and thereby mitigates the inter-modal catastrophic forgetting. A detailed proof of Proposition iis available in Appendix
4 Methodology
4.1 Problem formulation
Continual instruction tuning Assume a UMGM has been pre-trained with abundant vision- language data, captured by trainable parameters 0. We further train to adapt this UMGM to new S tasks in sequential manner. Each task is denoted by a task descriptor E {1, ..., S}. and owns an N.
independent dataset D, { {(x: .x xx33))
j=1 with N, data triplets, where Ximg Xins and
X indicate the input image tokens, instruction text tokens, and answer text tokens, respectively.
We formalize the autoregressive cross-entropy objective for task as:
max E (min
Jo log Po (X
X iim ins X
(2)
where the expectation E taken over the data distribution Dr, L the length of the answer sequence, Xone indicate all answer tokens before the index i, X:ass indicates the i-th answer token.
Image generation Let X éns be the text prompt tokens, and let X be the image tokens generated autoregressively. Following a standard autoregressive factorization, the process can be formalized as:
X
II po (X.mgg Xins Ximg)
where I is the length of the image token sequence, Xing indicate all image tokens before the index i, X X indicate the i-th image token. Once all tokens are generated, they are detokenized via decoder (such as a VQ- (Q-VAE) to produce the resultant image.
4.2 Modality- Decouple Experts (MoDE)
MoE adapters are lightweight modules that route inputs through sparse subset of expert networks, enabling task-specific adaptation with minimal interference to shared parameters (3). Although MoE adapters can effectively mitigate intru-modal catastrophic forgetting, naively sharing their parameters across modalities can lead to modality gradient conflict (see Section 33). To address this issue, we propose -Decouule Experts (MoDE), modality adapter architecture that
5

---

End
lopits
Distillation
logits
What
this
<ansu
AR Transformer
V-Adapter
MLP Multi layer Perceptron
Expert
Expert
Expert
A
T-Router
T-MoE Adapters
Image
Start
Image
End
What
Is
this
<ansp
Stop
Input prompt(s)
Image
Tokenizer
Reforonce
data
image(s)
Text Token
Image Token
Reéerence Token
Figure An autoregressive UMGM with our proposed MoDE integrated into its linear layers (MLPs). V-Adapter (Visual LoRA, in the light blue box): LoRA specialized for both the generation and understanding image tokens. T-MoE Adapters (Text Mixture-of-Experts LoRA, in the light brown box): MoE-LoRA designed for text tokens, supporting continual learning of multimodal understanding tasks. T-router computes the routing weights (z)) that determine how much each expert LoRA contributes for given text token. The circled "4" symbol denotes addition. During continual instruction tuning, the T-MoE primarily updates for text answers, while the V-Adapter handles image tokens. To preserve the model's 's image generation capability and mitigate inter-modal forgetting, we apply a knowledge distillation loss from the original (teacher) UMGM to the new (student) model's V-Adapter.
isolates text and visual updates into separate trainable subspaces. As shown in Figure 3 MoDE mitigates modality gradient conflict while preserving the flexibility and scalability of Moł. adapters.
Getting now into details, LoRA introduces trainable low-rank update frozen weight matrix W E Xda, where di and dour denote the input and output feature dimensions, respectively. The update is parameterized by two learnable low-rank matrices A ggr and B [R [RdwxXr such that:
a AW
BA
where A and B are learnable matrices with rank r < min (din: dour) and scaling factor a E R. Given an input token representation h e Rdd,, the modified linear transformation f Rdd Rdda becomes:
f(h) hW + -hhABB
(5)
MoE-LoRA generalizes LoRA by introducing a mixture of expert LoRA modules to handle diverse tasks. A router assigns each input token E di a soft distribution over experts:
9j (x) softmax(x:
where Wg E Rddxx is a trainable gating matrix with n denotes the number of experts, and gj (a) denotes the routing probability of expert j. The resulting update AW(x) is weighted combination of expert LoRA updates:
AW = Σ By
where each expert has A, Rexda and B, Rru and a E R is hyper- parameter. Given an input token representation h E Rd,, the final linear transformation becomes:
f(h) TW T 96 (x)hAA BJ
(8)

---

MoDE integrates the above two types of adaptations into UMGMs: V-Adapter and T-MoE, as shown in Figure 3 The V-Adapter is a LoRA module specialized for visual tokens which adapts the model to both visual understanding and image generation tasks. In contrast, the T-MoF is a MoF-LLRA A applied to text tokens, designed to enhance continual instruction tuning on multimodal understanding tasks without interfering with visual knowledge. Together, these two components allow MoDE to balance
adaptation (via the V-Adapter) and task-level flexibility (via the T-MoE) within a unified multimodal backbone.
To formalize the forward pass of multimodal tokens, we consider frozen linear layer W Rduxd in UMGM. Let H = [M] ..... Nim b: Mim + L. hr +
++.............................. .... E R Nxdx denote the hidden states of sequence of interleaved images and text with length N, where the subseript indicates position and the superscript indicates the token modality (t for text and i for image). We untic the sequence into the text sequence H' [ hbll and the image sequence H their length. We feed H² into the V-adapter using Eq. | to obtain / f(f'), and feed H' into T-MoE using Eq. 8 to obtain H - (H Finally, we reassemble Ht and H' into their original positions form the output sequence H.
Im M r.J. where m is the start index of image tokens and
MoDE therefore cleanly decouples different modalities: text tokens benefit from experts diversity for continual instruction tuning, while image tokens rely on visual adapter. Because T-MoE and V-adapter are disjoint, their update directions cannot interfere arbitrarily.
Proposition 2. In MoDE, let denote the parameters of the T-MoE and db denote the parameters of the V-adapter. A single gradient step 0 + 0 v/V of updates only o, and the resulting change in the visual loss satisfies:
n 2 Amax V B0 Vo
AC.
Thus, MoDE provably bounds the inter-modal interference to O(1)²,, compared to ( under a modality-decoupph baseline as in Proposition [ This theoretical property directly explains the minimal gradient conflict and superior retention of visual generation capabilities observed in our experiments. See Appendix A for a proof of Proposition 2 and Appendix B for its empirical verification.
4.3 Knowledge Distillation
During instruction tuning, the V-Adapter learns to understand images, which can degrade its original generation ability. We anchor the V-Adapter to the frozen pre-trained backbone (teacher) via logit-level knowledge distillation (KD), shown in Figure
small subset of images sampled from the LAION-5B dataset 66 serves as the reference data. For each reference token, both teacher (T') and student (S) predict the next image token. Let 2/' 25 denote their logits for the i-th image Laking softening factor B, the KD loss (Ckd) is computed between their softened output probabilities:
as the Kullback-Leible divergence 65
LKD = 82
DxI(SSftmaxx22/95 /8) Softmax(=$/3))
(10)
where L is the number of generated image tokens and denotes the distance between two distributions. Final objective. The T-MoE is trained with the instruction-tume tunnong cross-entropy loss (CCE) as in (2)
L = Les
Σ
log Po (X
Xing xiss Xana))
(11)
L
i=1
The V-Adapter uses weighted combination of the cross-entropy and the distillation losses:
LV-Adapicr = LcB + X LKD,
x>0 0
(12)
This mixed objective balances the updates for visual understanding and image generation in the V-adapter, thereby mitigating inter-modal catastrophic forgetting. In our experiments, we set À = 0.3 based on its optimal performance (see Appendix for a sweep over different À values).

---

Table I: Quantitative comparison of inter-modal (image generation) and intra-modal (multimodal understanding) catastrophic forgetting. For image generation, Zero-shot reports the pre-trained reference we aim to preserve. DualPrompt [66] yields low accuracy and low forgetting. This is because adding a small set of prompt embeddings slightly biases the feature representations, which is insufficient to learn the new and hard tasks. A captures the overall performance drop. Bold numbers are best; underlined numbers are second best in each column. Results are averaged over three different runs.
Image Generation
Multimodal Understanding
Method
Text alignment (*)
Image alignment (1)
FID (4)
Accuracy (t)
Forgetting (J)
A (L)
Zero-shot
0.2592
0,5205
52.13
22.48
34.84
Seq LoRA
0.2162
0.5150
56.12
28.43
35.33
28.57
Model tailor
1V
0.2384
0.5093
55.47
32.62
27.66
24.70
DualPrompt
16
0.2648
0.5083
56.08
31.92
6.82
25.40
MoELoRA
44
0.2248
0.5095
65.16
33.01
30.77
24.31
CL-MoE [8
0,2081
0.5150
65.87
32.86
30.95
24.46
MoDE (Ours)
0.2458
0.5170
53.74
33.47
25.99
22.78
5
Experiments
5.1 Experimental setup
Evaluation Benchmarks For continual instruction tuning, we evaluate on five datasets: Four visual question answering datasets (ScienceQA [58]. TextVQA (66), GQA [59,, and VizWiz (67) and
one image classification dataset (ImageNet (60) For image generation, we use the CustomCon-
cept101 68 dataset, which contains 101 concepts grouped into 16 broader categories.
Evaluation Metrics For continual instruction tuning, we follow the protocols from 669
70 and
report metrics: (1) Average Accuracy (ACC), defined the after the final round of training; and (2) Average Forgetting (Fgt), which quantifies performance degradation on previous tasks and is computed as the mean reduction from the best observed accuracy to the final accuracy, for each task. We use exact matches to determine answers correctness.
accuracy across all tasks
We evaluate image generation quality with three metries: (1) Image alignment. Cosine similarity between CLIP embeddings of the generated image and reference image that depicts the target concept; higher is better (t). (2) Text alignment. Cosine similarity between the generated image embedding and the CLIP embedding of its text prompt; higher indicates better prompt fidelity (t). (3) Fréchet Inception Distance (FID) 771.. Fréchet distance between features of generated and real images; lower values mean the synthetic images are closer to the real distribution (L).
Baselines We compare our proposed MoDE with the following continual learning approaches: Seq LoRA, which sequentially trains the model on new tasks using LoRA adapters [39]; Model Tailor [17]: DualPrompt [16]; MoE-LoRA [63]; CL-MoE [18]; and Joint Training (upper bound), which trains the model on all datasets simultancously and serves as the upper bound reference. The implementation details of baseline methods are provided in Appendix]
We use two autoregressive backbone UMGMs: Chameleon [3] and Janus-Pro For Chameleon, we adopt the implementation in [72] since the original model and checkpoints are not publicly available. Results on Janus-Pro are provided Appendix G See Appendix [ for implementation details.
5.2 Main results
Quantitative results. As shown in Table I our proposed MoDE built on Chameleon B effectively mitigates both inter- and intra-modal catastrophic forgetting, outperforming other CL baselines. the "Image Generation" columns, MoDE maintains image quality comparable to the pre-trained model (Zero-shot), indicating strong resistance inter-mont inter-modal forgetting. For instance, MoDE achieves an FID score of 53.74, closely matching the 52.13 of the pre- pre-trained model (zero-shot). While DualPrompt [16] also preserves image generation quality, its performance in continual instruction tuning (as shown in the "Multimodal Understanding" columns) is poor: the average accuracy is only 31.92%. Notably, although DualPrompt shows low average forgetting of 6.82%, this is misleading: its best accuracy on ImageNet is only 24.55%, whereas other methods exceed 70%. This low forgetting

---

Table 2: Continual Instruction-tuning results on sequence of five datasets. Accuracy (ACC) is exact-match (higher is better); Fgt is average forgetting (lower better). For each method, the upper row shows the best accuracy during continual instruction tuning, and the lower now shows the final accuracy after completing all tasks. Upper bound reports performance obtained by individually fine-tuning model on each task. Bold numbers denote the best result, and underlined numbers denote the second best. Results are averaged over three different runs.
Datasets
Metrics
Method
ScienceQA
TextVQA
ImageNet
GQA
VizWiz
ACC (f)
Fgt (1)
Zero-shot
51.52
23.49
16.53
14.22
6.64
22.48
72.43=00.2
39.34:1.24
89.41=1448
37.93+1.611
14.38=
Seq LoRA
28.43
35.33
33.96=0093
16.84=1442
13.68=00.3
33,20+1.77
44.38=#0.06
74.90-0.72
40.0411.18
78.22.1063
37.35:11002
43.25
Model tailor 17
32.62
27.66
55.40+1.21
17.78+4.83
14.97+
31.71+0.76
43.25+0068
60.01-0.89 31.48+1.02
24.35=00.57
29.91
40,91
DualPrompl [6
31.92
6.82
59.64=1.03
19.68+3.81
8.61
30.75+0.69
40,91
71.79110.4
39.6212.08
94.75.1.53
37.6611.23
44.3311065
MoHLoRA [4
33.01
30.77
50.47+1.17
1..64+2.86
18.73+1996
31.89+0.78
44.33+0.70
71.35-0.88
38.82+1.19
90.08:2.59
37.3742.14
44.13-0.73
CL-MoE 18
32.86
30.95
55.34=1.09
17.64+2.77
15.62+1.02
31.57+0.81
44.13=0067
70.45-0.74
38.9011.47
80.6710.45
37.0331047
44.35-0.62
MoDE (Ours)
33.47
25.99
56.25+1.87
19.08+2.47
14.67+0.76
33.10+1.16
44,28++0.99
Upper bound
73.27
40.74
91.88
36.56
44.15
57.32
arises not from effective retention, but from limited ability to learn the first place: if model never learns well, it has little to forget. Although MoE-LoRA (63 performs well in mitigating intra-modal forgetting, suffers from severe inter-modal forgetting, as evidenced by the worse image generation quality, with an FID of 65. 16 compared 56.12 12 for the vanilla SeqLoRA. In contrast, MoDE achieves the highest accuracy of 33.47% and the lowest FID of 53.74, demonstrating its effectiveness in mitigating both inter-modal and intrl-modal forgetting simultaneously.
Table presents more detailed results of continual instruction tuning. this setting, our MoDE consistently outperforms all comparison methods and sustains effective learning across all tasks.
Qualitative results. As shown in Figure our MoDE can generates higher quality images compared to other methods, effectively preserving the pre-trained model's image generation capability. For instance, the third row of Figure given the prompt "Barn the fall season with leaves all around". MoDE successfully generates yellow leaves surrounding the barn, whereas Model Tailor [77 and fail capture this detail. See Appendix C for more qualitative results.
CL-MoE [8]
5.3 Ablation results
To validate the performance gains of MoDE, we present our ablation results in Table "T-MoE LoRA" includes only the T-MoE adapters without the V-adapter. "MoDE w/o KD" removes knowledge distillation, training both the T-MoE adapters and the V-adapter solely with cross-entropy loss. Of note, MoDE is also robust to task order. See Appendix for more ablation results.
Table 3: Ablation study shows that both the modality-specife adapters and knowledge distillation are essential the effectiveness of MoDE.
Image Generation
Visual Understanding
Model
Text alignment (t)
Image alignment (1)
FID (L)
Accuracy (1)
Forgetting (L)
Chameleon
3
0,2592
0.5205
$2.13
22.48
+ T-MoE LoRA
0.2583
0,5317
51,28
33.03
28.65
MoDE w/o KD
0.2364
0.5156
54.61
33.07
26.49
+ MoDE
0.2458
0,5170
53.74
33.47
25.99

---

Input Prompt
Chameleon [3]
Model Tailor [17]
CL-MoE [18]
MoDE (Ours)
A dog wearing sungieases on the poreh.
troneporent cup filled with steaming hot cocoa.
Barn in the fall scason with leaves all around.
Marigold flowers the wase.
Figure 4: Qualitative results of image generation on the Chameleon 3 model. Our method generates more visually coherent and faithful images compared other baselines (e.g.. the realistic dog the first row, steam in the second row). Additional examples are provided in Appendix
These results demonstrate the effectiveness of both the T-MoE and V-adapter. Using only the T-MoE preserves image generation capabilities by leaving visual components untouched, but results in sub- optimal visual understanding, the model lacks updates necessary for better visual comprehension.
We also report the computational efficiency MoDE, including training time, memory consumption, and parameter overhead, in Appendix [H] These results show that MoDE achieves favorable trade-offs between accuracy and efficiency compared to existing methods.
Conclusion
We have proposed Modality-Decoopple Experts (MoDE) to address both intra- and inter-modal catastrophic forgetting in continual instruction tuning for Unified Multimodal Generative Models (UMGMs). We have identified inter-modal forgetting, explained its cause due the modality gradient conflict, and designed MoDE to decouple modality-specifie updates, mitigating conflict theoretically and empirically. Extensive experiments show that MoDE significantly outperforms SOTA baselines, establishing as strong solution for continual learning in unified multimodal generation.
7
Acknowledgments
This work is supported in part by the NSF grant CCF-2531882. and in part by an Amazon Research Award, Spring 2025.
10

---

References
[1] H. Liu, Li, Q. Wu, and Y. J. Lec, "Visual instruction tuning," Advances in Neural Information Processing Systems, vol. 36, pp. 34892-34916, 2023.
[2] R. Rombach, A. Blattmann, D. Lorenz. P. Esser, and B. Ommer. "High-resolution image synthesis with latent diffusion models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Paterrn Recognition, pp. 10684-10695, 2022.
[3] C. Team, "Chameleon: Mixed-modal early-fusion foundation models." arXiv preprint arXiv:2405.09818 2024.
[4] C. Wu, X. Chen, Z. Wu, Y. Ma, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, C. Ruan, et al. "Janus: Decoupling visual encoding for unified multimodal understanding and generation," in Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12966-12977, 2025.
[5] C. Zhou, L. YU, A. Babu, K. Tirumala, M. Yasunaga. L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy, "Transfusion: Predict the next token and diffuse images with one multi-modal model," in The Thirteenth International Conference on Learning Representations, 2025.
[6] X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang. F. Zhang, Y. Wang. Z. Li, Q. Yu, et al. "Emu3: Next-token prediction is all you need," arXiv preprint arXiv:2409.1886 2024.
[7] J. Xie, W. Mao, Z. Bai, J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang, and M. Z. Shou, "Show-o: One single transformer to unify multimodal understanding and generation," in The Thirteenth International Conference on Learning Representations, 2025.
[8] S. Kou, J. Jin, Z. Liu, C. Liu, Y. Ma, J. Jia, Q. Chen, P. Jiang, and Z. Deng, "Orthus: Autoregressive interleaved image-text generation with modality-speeiin heads," in socty-second International Conference on Machine Learning, 2025.
[9] H. Liu, C. Li. Q. Wu. and Y. Lee, "Visual instruction tuning." Advances in Neural Information Processing Systems, vol. 36, pp. 34892-3493 34916, 2023.
[10] D. Chen, J. Liu, W. Dai, and B. Wang, "Visual instruction tuning with polite flamingo," in Proceedings of the AAAI Conference Artificial Intelligence, vol. 38, pp. 17745-17753, 2024.
[11] J. He, H. Guo, M. Tang, and J. Wang, "Continual instruction tuning for large multimodal models," arXiv preprint arXiv:2311 16206, 2023.
[12] L. Wang, X. Zhang. H. Su, and J. Zhu, "A comprehensive survey of continual learning: Theory, method and application." IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[13] D. Yu, X. Zhang, Y. Chen, Liu, Y. Zhang, P. S. Yu, and I. King, "Reeent advances of multimodal continual learning: A comprehensive survey," arXiv preprint arXiv:2410005352. 2024.
[14] H. Guo, F. Zeng. Z. Xiang, F. Zhu, D.-H. Wang. X.-Y. Zhang, and C.-L. Liu, "HiDe-LLaVa Hierarchical decoupling for continual instruction tuning of multimodal large language model," Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Pupers), (Vienna, Austria), pp- 1357213386, Association for Computational Linguistics, July 2025.
[15] A. Maharana, Yoon, Chen, and M. Bansal, "Adapt-SSinfty"" Scalable continual multimodal in- struction tuning via dynamic data selection," in The Thirteenth International Conference on Learning Representations, 2025.
[16] Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, Su, V. Perot, J. Dy, et al., "Dualprompt: Complementary prompting for rehearsal-free continual learning," in European Conference on Computer Vision, pp. 631 648, Springer, 2022.
[17] D. Zhu, Z. Sun, Z. Li, T. Shen, K. Yan, S. Ding, C. Wu, and K. Kuang, "Model tailor: Mitigating catastrophic forgetting in multi-modal large language models." in International Conference on Machine Learning, pp.
PMLR, 2024.
[18] T. Huai, J. Zhou, X. Wu, Q. Chen, Q. Bai, 7. 7hou, and He, "Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-exxpeee for continual visual question answering," in Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19608-19617, 2025.
[19] Q Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang. H. Gao, J. Liu, T. Huang, and X. Wang, "Emu: Gener- ative pretraining in multimodality," in The Twelfth International Conference on Learning Representations. 2024.
11

---

[20] Q Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Y. Wang, Y. Rao, J. Liu, T. Huang, and X. Wang, "Generative multimodal models are in-context learners." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. -4398-14409, 2024.
[21] S. Tong, D. Fan, J. Zhu, Y. Xiong, X. Chen, K. Sinha, M. Rabbat, Y. Y.L.Cun, S. Xie, and Liu, "Metamorph: Multimodal understanding and generation via instruction tuning," arXiv preprint arXiv:2412. 141644 2024.
[22] C. Wang, Lu, J. Yang, R. Huang. J. Han, L. Hou, W. Zhang, and Xu, "Illume: Illuminating your Ilms to see, draw, and self-enhannee"" arXiv preprint wrXiv:2412.06677 2024.
[23] D. Ghosh, H. Hajishirzi, and I.. Schmidt, "Geneval: An abject-focused framework for evaluating text- to-image alignment," " Advances in Neural Information Processing Systems, vol. 36, pp. 52132-52152, 2023.
[24] H. Liu, W. Yan, M. Zaharia, and P. Abbeel, "World model on million-length video and language with blockwise ringattention." in The Thirteenth International Conference on Learning Representations, 2025.
[25] Y. Wu, Z. Zhang, J. Chen, H. Tang. D. Li, Y. Fang. L.. Zhu, E. Xie, H. Yin, L. Yi, S. Han, and Y. Lu, "VIIA--: unified foundation model integrating visual understanding and generation," in The Thirteenth International Conference on Learning Representations. 2025.
[26] H. Tang, Liu, and X. Xiao, "Ugen: Unified autoregressive multimodal model with progressive vocabulary learning," arXiv preprint arXiv:2503.21193, 2025.
[27] Y. Jin, K. Xu, K. Xu, L. Chen, C. Liao, J. Tan, Q. Huang. B. Chen, C. Song, dai meng, D. Zhang, W. Ou, K. Gai, and Y. Mu, "Unified language-vision pretraining in LLM with dynamic discrete visual tokenization," " in The Twelfth International Conference on Learning Representutions, 2024.
[28] P. Esser. R. Rombach, and B. Ommer, "Taming transformers for high-resolution image synthesis," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873-12883, June 2021.
[29] L. Qu, H. Zhang, Y. Liu, X. Wang, Y. Jiang, Y. Gao, H. Ye, D. K. Du, Z. Yuan, and X. Wu, "Tokenflow: Unified image tokenizer for multimodal understanding and generation," " Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2545 2555, 2025.
[30] X. Chen, Z. Wu, X. Liu, Z. Pan, W. Liu, Z. Xic, X. Yu, and C. Ruan, "Janus-pro: Unified multimodal undorstanding and generation with data and model scaling," arXiv preprint (xX1::5501.781 2025. 2025.
[31] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., "Learning transferable visual models from natural language supervision," in International Conference on Machine Learning, pp. 8748 8763, PmLR, 2021.
[32] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, "Sigmoid loss for language image pre-training," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023.
[33] A. Van Den Oord, O. Vinyals, et al., "Neural discrete representation learning"" Advances in Neural Information Processing Systems, vol. 30, 2017.
[34] R. Huang, C. Wang, J. Yang, G. Lu, Y. Yuan, J. Han, L. Hou, W. Zhang, L. Hong, H. Zhao, et al., "Illume+: Illuminating unified milm with dual visual tokenization and diffusion refinement," arXiv preprint arXiv:2504.0193. 2025.
[35] B. Mustafa, C. Riquelme, J. Puigcerver, R. Jenatton, and N. Houlsby, "Multimodal contrastive learning with limoe: the language-image mixture of experis," Advances in Neural Information Processing Systems, vol. 35, pp. 9564-9576, 2022.
[36] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer," in International Conference Learning Representations. 2017.
[37] C. Riquelme, Puigeerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, "Scaling vision with sparse mixture of experts," in Advances in Neural Information Processing Systems (M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, eds.), vol. 34, pp. 8583- 8595, Curran Associates, Inc., 2021.
[38] S. Shen, Z. Yao, C. Li, T. Darrell, K. Keutzer, and Y. He, "Scaling language models with sparse mixture of experts," in The 2023 Conference Empirical Methods in Natural Language Processing, 2023.
12

---

[39] E. J. Hu. Y. Shen. P. Wallis. Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, W. Chen, et al., "Lora: Low-rank adaptation of large language models.," " ICLR, vol. 1, 2. p. 3, 2022.
[40] 7. Chen, Z. Wang. Z. Wang, H. Liu, Z. Yin, S. Liu, L., Sheng, W. Ouyang, and J. Shao, "Octavius: Mitigating task interference in MILLMs via IoRA-moe," in The Twelfth International Conference on Learning Representations, 2024.
[41] X. Wu, S. Huang, and F. Wei, "Mixture of IoRA experts." in The Twelfth International Conference on Learning Representations, 2024.
[42] S. Yang, M. A. Ali, C.-L. Wang, L. Hu, and D. Wang, "Moral: Moe augmented lora for Ilms' lifelong learning," arXiv preprint arXiv:2402.1 12600 2024.
[43] D. Li, Y. Ma. N. Wang, Z. Ye. Z. Cheng, Y. Tang, Y. Zhang, L. Duan, J. Zuo, C. Yang, et al., "Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts," arXiv preprint arXiv:2404. 15159, 2024.
[44] T. Luo, J. Lei, F. Lei. W. Liu, S. He, J. Zhao, and K. Liu, "Moelora: Contrastive learning guided mixture of experts parameter-efficien fine-tuning for large language models." arXiv preprint arX15:240..12811 2024.
[45] J. Yu, Y. Zhuge, I. Zhang, P. Hu, D. Wang. H. Lu, and Y. He, "Boosting continual learning of vision- language models via mixture-of-experts adapters." in Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition, pp. 23219-23230, 2024.
[46] H. Zhao, Z. Wang, Q Sun, K. Song, Y. Li, X. Hu, Q. Guo, and S. Liu, "Llava-cmoe: Towards continual mixturc of experts for large
models," arXiv preprint arXiv:2503.21227 2025.
[47] R. Wang and P. Li, "LEMoE: Advanced mixture of experts adaptor for lifelong model editing of large language models," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, (Miami, Florida, USA). pp. 2551-2575, Association for Computational Linguistics. Nov. 2024.
[48] W. Chen, Y. Zhou, N. Du, Y. Huang, J. Laudon, Z. Chen, and C. Cui, "Lifelong language pretraining with instribution-spccialieeee experts," in International Conference on Machine Learning, pp. 5383-5395, PMLR, 2023.
[49] W. Pian. S. Deng, S. Mo. Y. Guo, and Y. Tian, large language models," arXiv preprint 13X0 2024.
continual learning of multimodal
[50] J. Qiao, Z. Zhang, X. Tan, Y. Qu, S. Ding, and Y. Xie, "Llaca: Multimodal large language continual assistant," arXiv preprint arXiv:2410. 10868, 2024.
[51] Zheng, Q. Ma, Z. Liu, B. Wu, and H. Feng, "Beyond fotieforgg:ting: Multimodal continual instruction tuning with positive forward transfer," arXiv preprint 2xXx:22401.09880 2024.
[52] Cao, Y. Liu, Liu, Wang. J. Dong, H. Ding, X. Zhang. Reid, and X. Liang, "Continual Ilava: Continual instruction tuning in large vision-language models," arXiv preprint arXiv:2411.02564 2024.
[53] Y. Zhai, S. Tong, X. Li, M. Cai. Q. Qu, Y. J. Lee, and Y. Ma, "Investigating the catastrophic forgetting in multimodal large language models," " arXiv preprint arXiv:2309. 10313, 2023.
[54] F. Zeng, F. Zhu, H. Guo, X.-Y. Zhang, and C.-L. Liu, "Modalprompt: Dual-modality guided prompt for continual learning of large multimodal models," arXiv preprint arxiv:2410.05849, 2024.
[55] Y.KK Zhang, S. Lu, L. Y. Ma, Q-G. Chen, 7. Xu, W. Luo, K. Zhang, D.-C. 7han, and H.-J. Ye, "Wings: Learning multimodal Ilms without text-only forgetting," Advances in Neural Information Processing Systems, vol. 37, pp. 31828-31853 2024.
[56] J. Zou. B. Liao, Q Zhang, W. Liu, and X. Wang, "Omnimamba: Efficient and unified multimodal understanding and generation via state space models." arxiv preprint arXiv:2503.0866 2025.
[57] C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie. Z. Song, G. Shi, and H. Fan, "Emerging properties in unified multimodal pretraining," " arXiv preprint arXiv:2505. 14683, 2025.
[58] P. Lu,, S. Mishra, T. Xia, I.. Qiu, K.W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, "Leam to explain: Multimodal reasoning via thought chains for science question answering," in The 36th Conference on Neural Information Processing Systems (NeurIPS). 2022.
[59] D. A. Hudson and C. D. Manning, "Gqa: A new dataset for real-world visual reasoning and compositional question answering." Conference Computer Vision and Pattern Recognition, 2019.
13

---

[60] J. Deng, W. Dong. R. Socher, Li. K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," " in 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, leee, 2009.
[61] Z. Xu, M. Liu, Y. Shen, J. Rimchala, J. Zhang. Q. Wang, Y. Cheng, and L. Huang. synergizers for interleaved vision-language generalists," in The Thirteenth International Conference on Learning Represemtations, 2025.
[62] Z. Ni, L. Wei, S. Tang. Zhuang, and Q. Tian, "Continual rision-languane representation learning with off off-diaanaa information," in International Conference on Machine Learning, pp. 26129 26149, PMLR, 2023.
[63] C. Chen, J. Zhu, X. Luo, H. Shen, J. Song, and L. Gao, "Coin: A benchmark of continual instruction tuning for multimodel large language models," Advances in Neural Information Processing Systems, vol. 37, pp. 57817-57840, 2024.
[64] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, al,, "Laion-5b: An open large-scale dataset for training next generation image-text models," Advances neural information processing systems, vol. 35, pp. 25278-25294, 2022.
[65] J. M. Joyce, Hellback-Leibler Divergence, pp. 720-720-22222 Berlin, Heidelberg: Springer Berlin Heidelberg. 2011.
[66] A. Singh, V. Natarjan, M. Shah, Y. Jiang, X. Chen, D. Parikh, and M. Rohrbach, "Towards vqa models that can read," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2317-8320 2019.
[67] D. Gurari, Q. Li, Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and P. Bigham, "Vizwiz grand challenge: Answering visual questions from blind people," in Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 3608-3617 2018.
[68] N. Kumari, B. Zhang, R. Zhang. E. Shechtman, and Y. Zhu, "Multi-concept customization of text to- image diffusion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1931-1941, 2023.
[69] J. S. Smith, L. Karlinsky, V. Gutta, P. Cascante- D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira, "Coda-prompt" Continual decomposed attention- based prompting for rehearsal-fie free continual learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1999--11919 June 2023.
[70] J. Qiao, Tan, C. Chen, Y. Qu, Y. Peng, Y. Xie, et al., "Prompt gradient projection for continual learning" in The Twelfth International Conference on Learning Representations, 2023.
[71] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "Gans trained by time-scale update rule converge to a local nash equilibrium," Advances Neural Information Processing Systems, vol. 30, 2017.
[72] E. Chern, Su, Y. Ma, and Liu, "Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation." arXiv preprint arXiv:2407.06135 2024.
[73] S. Kazemzadeh, Ordonez, M. Matten, and Berg, "ReferliGame: Referring to objects in photographs of natural scenes." in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (A. Moschitti, B. Pang, and W. Daelemans, eds.), (Doha, Qatar), pp. 787-798. Association for Computational Linguistics, Oct. 2014.
[74] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, "Generation and comprehension of unambiguous object descriptions," " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11-20, 2016.
14

---

A Theoretical Analysis
We formally compare the one-step visual loss drift between modality-cupped architecture (shared parameters for text and image) and our modalit--dccuuped MoDE design. Let assume the following parameters:
0 the full set of trainable parameters,
(0) the text-generation loss,
C. the image-generation loss,
?i the learning rate,
gt VoL, and go VoLn the corresponding gradients of text and visual modalities, respectively,
H VgC, the Hessian of the visual loss.
After gradient step t 0 n-ng, the change in the visual loss is, by second-order Taylor expansion:
rf H.gt + o(1²)
AC = (0 (0) = -1 (g., ge) +
2 9t
(13)
where o denotes higher-order terms that vanish faster than 1/2
Modality architecture (shared parameters). In the modality-coupled case (e.g., shared MoE-LoRA adapters between modalities), both and are nonzero over the same parameter space. Thus, the first-order term (7(t,, go) is generally nonzero. In the worst (maximum interference), and 9v are anti-alia anti-aligned, so by Cauchy-Schwarr Schwarz inequality we have:
Cgoupledd
< n
199 Ilgell + (H.))e1 + o(n)²)
(14)
Thus, the dominant error scales as O(1).
Modality-decouplee architecture (MoDE), parameter blocks and 1b, respectively; thus:
In MoDE, the text and visual tokens update disjoint
oc. OΦ
oc.
O.
Oup
which implies
(gt, ge) = 0 (orthogonal supports)
Therefore, the first-order term in 13 vanishes, and the remaining visual loss drift is purely second- order:
7/2
A AACMoDE McDE
Amax (Ho) + o(r²))
(15)
Thus, the dominant error scales as O(122).
Conclusion. Comparing 14) and 15
we conclude that
A MoDE
A Cooupeed
since n < for small 77.
Thus, MoDE provably limits the worst-case drift in image-generation loss to O compared to (1)) in a modality-coupled architecture.
15

---

B Modality gradient conflict
The theoretical support provided in Appendix ure for our claim.
This section provides empirical evidence (Fig-
As shown in Figure the modality-coupled MoE LoRA 44 displays significant gradient conflict between text and image modalities, empirically validating our analysis in Section 3.
Modality Gradient Conflict
0.25
Ratio of parameters
0.20
0.10
0.05
0.00
-0.5
-0.4
-0.3
-0.2
-0.1
0.1
Cosine
Figure 5: Cosine distance distribution between text and image modality gradients in modality MoE LoRA 44 the ScienceQA (58) dataset. The y-axis shows the proportion of parameters corresponding to each cosine distance. Lower cosine distance values indicate greater gradient conflict, with 0 denoting orthogonal update directions.
In contrast to the significant conflict shown in Figure 5 Figure shows that MoDE has zero cosine disagreement between modalities. This confirms that its modality-specific update subspaces completely isolated, both theoretically and empirically.
Modality Gradient Conflict
1.0
Ratio of parameters
0.8
0.6
0.4
0.2
0.0
-0.4
-0.2
0.0
0.2
Cosine
Figure 6: Cosine distance distribution between text and image modality gradients in MoDE. Lower cosince distance values indicate greater gradient conflict, with denoting orthogonal directions. In MoDE, all parameters exhibit perfectly orthogonal gradients, confirming the absence of modality gradient conflict.
16

---

C Additional Qualitative Results
Table presents the complete qualitative results of MoDE, built on Chameleon [3] in comparison to various baseline methods. Our approach generates images that better align with the prompts and exhibit higher visual fidelity. For example, given the prompt "Barn the fall season with leaves all around", only the original Chameleon and our MoDE successfully generate images featuring yellow leaves around the barn.
Table 4: Full qualitative results of our MoDE compared to other baseline methods.
dog wearing sunglasses the porch
transparent cup filled with steaming hot cocoa
Barn in the fall season with leaves all around
Marigold flowers in the vase
Method
Chameleon 3
Seq LoRA
Model
Tailor
DualPrompt 16
MoELoRA 441
CL-MoE 18
MoDE (Ours)
17

---

D Implementation Details
We use Chamelcon [3] and Janus-Pro [3 as backbone models. For Chameleon, adopt the implementation from [22 due to the unavailability of the original checkpoints and code. All LoRA modules configured with default rank of 8. Training is conducted for epoch per task NVIDIA RTX 6000 GPUs, with bf16 mixed precision and per-GPU batch size (total effective batch size of 8).
use learning rate of I x 10-4, scheduled via cosine decay with a warm-up ratio of 0.1. The softening factor 3 in knowledge distillation equation (10) is 2.0. The hyperparameter À is set to 0.3, selected based on a sweep over multiple values (see Table 5 This value strikes balance between imgge generation and visual understanding: higher favors generation quality, while lower improves understanding performance. A higher increases the weight of the knowledge distillation loss in the V-adapter (see Eq. 12), constraining the update closer to the pre-trained model to preserve image generation capabilities. However, this also constrains beneficial updates needed for improved visual understanding.
Table 5: Performance of MoDE under different values of the loss balancing coefficient A. We select À = 0.3 (bolded) as it yields the best balance between image generation fidelity and visual understanding accuracy.
Image Generation
Visual Understanding
Text alignment (1)
Image alignment (t)
FID (L)
Accuracy (1)
Forgetting (L)
0.0
0.2364
0,5156
54.61
33.07
26.49
0.3
0.2458
0.5170
53.74
33.47
25.99
0.5
0.2498
0.5264
53.12
31.53
28.38
1.0
0.2543
0.5190
51.72
32.10
27.62
E
Additional Ablation Results
E.1
Different Tasks Order
Following we explore the impact of different tasks order by conducting an additional experiment using different order of tasks, arranged alphabetically: GQA [59], ImageNet [60]. ScienceQA [58] TextVQA (66), and VizWiz [67].
Table 6: Results of inter-modal (image generation) and intra-modal (multimodal understanding) catastrophic forgetting on different task orderings. For image generation, Zero-shot reports the pre-trained reference we aim to preserve. Bold shows the best results, underline shows the second best results. Chameleon used for the backbone.
Image Generation
Multimodal Understanding
Method
Text alignment (t)
Image alignment (1)
FID (L)
Accuracy (1)
Forgetting (L)
Zero-shot
0.2592
0.5205
52.13
22.48
Seq LoRA
0.2307
0,4973
66.75
27.72
33.10
Model tailor
0.2448
0,5166
53.58
34.02
24.51
DualPrompt 16
0.2491
0,5140
53.07
32.05
7.96
MoELoRA 63
0.2492
0,5125
61.64
34.41
24.72
CL-MoE 18
0.2463
0.5090
59.89
34.18
25.63
MoDE (Ours)
0.2525
0.5256
51.69
36.68
22.92
As shown in Table 6 our MoDE consistently outperforms the baseline methods, thus demonstrating its robustness to different task orderings.
E.2 Different Number of Experts
This this section we provide an ablation study the number of experts in the T-MoE module.
18

---

Table 7: Performance of MoDE under different number of experts in T-MoE.
Image Generation
Visual Understanding
# Experts
Text alignment (1)
Image alignment (1) FID (L)
Accuracy (1)
Forgetting (4)
2
0.2437
0.5144
53.61
32.24
26.81
0.2458
0.5170
53.74
33.47
25.99
6
0.2499
0.5146
52.69
34.90
24. 12
As shown in Table 7 increasing the number of experts yields slightly improved text alignment and accuracy, though gains are modest. This suggests that the performance of MoDE is relatively robust to the number of experts, and 4 experts offer a good trade-off between performance and computational cost.
E.3 Ablation Knowledge Distillation and Modality Decoupling
We provide the results of adding our cross-modal knowledge distillation (KD) loss to modality- coupled baselines (SeqLoRA [39], CL-MoE (18), MoELoRA (63) in Table Although applying KD to vanilla LoRA leads to modest improvements both in image generation quality and mul- timodal understanding performance, disual-generatie forgetting at the cost of hurting the multimodal understanding performance. This shows that without explicit modality decoupling, KD "locks in" the teacher's features in a way that hinders the model's ability to adapt to new multimodal understanding tasks. The results come from single run.
observe that for MoELoRA and CL-MoE this mitigates
Table Effect of adding KD to modality-coupled baselines.
Image Generation
Visual Understanding
Text alignment (t)
Image alignment (t)
FID (L)
Accuracy (t)
Forgetting (L)
Seq LoRA [39]
0,2162
0,5150
56.
28.43
35.33
SeqLoRA w/ KD
0.2478
0.5180
52.83
30.81
29.85
MoBLoRA
0.2248
0.5095
65.16
33.01
30.77
MoELoRA w/ KD
0.2559
0,5176
53.03
30.07
35.44
CL-MoE 188
0.2081
0.5150
65.87
32.86
30.95
CL-MoE w/ KD
0.2439
0.5200
52.43
31.82
34.74
As shown in Table 9 decoupling textual and visual LoRA without knowledge distillation yields only modest gains in understanding accuracy and visual generation quality compared to vanilla LoRA. This indicates that decoupling alone insufficient to maintain the rich cross-modal alignment necessary for high-quality generation. Furthermore, Table TableBiincludes includes ablation "MoDE w/o KD" (i.e., decoupled T-MoE and V-adapter without knowledge distillation), which further confirms that knowledge distillation plays critical role in the performance improvements observed.
Table Ablation on modality-decouplee architecture.
Image Generation
Visual Understanding
Text alignment (t)
Image alignment (t)
FID (L)
Accuracy (f)
Forgetting (L)
Seg LoRA
0.2162
0.5150
56.12
28.43 30,59
35.33
Decoupled LoRA
0.2252
0,5180
54.74
32.79
The ablation results show that MoDE more than the sum of its parts: only the combination of modality plus knowledge distillation leads to consistently superior performance.
19

---

Multimodal Understanding Forgetting
In this section, we provide the results of MoDE mitigating catastrophic forgetting of the understanding capabilities induced by fine-tuning UMGMs image generation tasks. To address this, conduct new experiments by fine-tuning the linear layers of the Chameleon model [3] on image genera- tion tasks. Specifically, we use the LAION-5B dataset [64] for fine-tuning and apply knowledge distillation using the multimodal understanding sequence from the GQA dataset [59] as reference data. We compare our MoDE method with LoRA fine-tuning across three multimodal understanding benchmarks. Accuracies are reported in Table TO The difference between zero-shot and fine-tuned performance reflects the degree of forgetting multimodal understanding.
Table 10: Forgetting caused by fine-tuning on image generation tasks.
Multimodal Understanding Accuracy (%)
Method
ScienceQA [58]
TextVQA [66]
ImageNet [60]
Zero-shot
51.52
23.49
16.53
LoRA [3
45.81
12.25
0.00
MoDE (Ours)
51.28
23.44
16.61
The results demonstrate that fine-tuning on image generation tasks can indeed cause catastrophic forgetting in multimodal understanding. This is evident because of the substantial drop in performance for the LoRA baseline compared zero-shot. Our MoDE method remains effective in this scenario, significantly reducing the performance drop and mitigating forgetting relative to the LoRA baseline. This demonstrates that MoDE generalizes beyond the specific forgetting direction explored in the main paper, where understanding tasks overwrite prior image generation capabilities, and is effective regardless the fine-tuning direction.
G
Additional Continual Learning Results
G.1 Additional Results with Janus-Pro
We provide additional quantitative comparisons using Janus-Pro-13 330 as the backbone. We further include the Grounding task with referring expression grounding datasets (RefCOCO [73]. ReSCOCO+ (74), and RefCOCOg (74). All methods are trained with early stopping. As shown in TT our MoDE outperforms the baseline methods in continual instruction tuning for multimodal understanding.
Table 11: Results of continual instruction tuning with Janus-Pro-IP IB 30 as the backbone. Accuracy (ACC) is exact-match (higher is better); Fgt average forgetting (lower better). For each method, the upper row shows the best accuracy during continual instruction tuning, and the lower row shows the final accuracy after completing all tasks. Upper bound reports performance obtained by individually fine-tuning model on each task. Bold shows the best results.
Datasets
Metrics
Method
ScienceQA
TexiVQA
ImageNet
GQA
VizWiz
Grounding
ACC (1)
Fgt (L)
Zero-shot
82.69
40.22
13.11
38.46
26.74
0.00
35.54
61.27
51.34
96,89
58.21
56.33
41.65
Seq LoRA
37.65
27.37
44.87
32.01
12.42
47.25
47.71
41.65
61.58
47.36
92.79
51.48
44.87
36.72
Model Tailor 17
29.13
32.01
38.62
35.76
21.18
32.98
9.49
36.72
74.17
50.24
93.21
56.86
55.52
38.92
CL-MoR 18
43.18
21.96
44.22
42.38
34.63
55.04
43.92
38.92
81.01
46.34
95.16
62.01
61.02
44.99
MoDE (Ours)
53.02
14.48
65.45
39.10
52.32
58.20
58.07
44.99
Upper bound
86.28
51.34
98.72
62.55
67.23
46.13
68.71
20

---

G.2 Additional Results with Chameleon
Following the CoIN benchmark [63] we present results on additional datasets using Chameleon B as the backbone. We further include the Grounding task with referring expression grounding datasets (RefCOCO 773. RefCOCO+ [74], and RefCOCOg 74).. All methods are trained with early stopping. As shown in Table 12, our MoDE outperforms the baseline methods continual instruction tuning for multimodal understanding.
Table 12:
Results sequence of six datasets. Accuracy (ACC) is exact-match (higher is better); Fgt is average forgetting (lower is better). For each method, the upper row shows the best accuracy during continual instruction tuning, and the lower row shows the final accuracy after completing all tasks. Upper bound reports performance obtained by individually fine-tuning a model on each task. Bold shows the best results.
Datasets
Metrics
Method
ScienceQA
TexIVQA
ImageNet
GQA
VizWiz
Grounding
ACC (1)
Fgt (L)
Zero-shot
51.52
23.49
16.53
14.22
6.64
0.00
18.73
72.43
39.34
89.41
37.93
44.38
18.62
Seq LoRA
29.82
24.64
38.67
33.18
15.09
34.99
39.34
18.62
74.90
40.04
78.22
37.35
43,25
15.66
Model tailor 17
29.09
22.97
56.06
25.82
7.10
32.62
37.32
15.66
60.01
31.48
24.55
29.91
40,91
13.82
DualPrompt 16
25.47
9.57
52.45
24.12
3.82
31.44
27.16
13.82
71.79
39.62
94.75
37.66
44.33
37.13
MoELoRA
34.14
24.08
65.89
22.36
6.71
30.22
42.55
37.13
71.35
38,82
90.08
37.37
43.73
36.22
CL-MoE 18
36.96
19.12
67.32
32.82
33.29
43.20
36.22
71.49
39.00
85.88
36.73
44.45
36.64
MoDE (Ours)
37.05
18.25
67.87
33.36
9.54
33.60
41.91
36.64
Upper hound
73.27
40.74
91.88
36.56
44.15
44.90
55.25
H Computational Analysis
In this section, we report comparisons of training compute, memory footprint, and trainable parameter size.
Table 13: Comparison of trainable parameters and training costs.
Results come from single run.
Method
TFLOPs
Peak GPU Memory (MB)
Training Time (ms)
Trainable Params (%)
MoELoRA 63
4.29
75265
419.2
0.0171
CL-MoE 18
4.29
86300
421.7
0.0171
MoDE (Ours)
4.19
84980
440.3
0.0211
As shown Table 13 MoDE introduces only a slight increase in training time (440.3 ms vs. 419.2 ms) and peak GPU memory (84.98 GB vs. 75.27 GB) compared MoELoRA 63 while maintaining comparable TFLOPs. The trainable parameter ratio of MoDE is only 0.021 %, modestly higher than MoELoRA (63 and CL-MoE (18) (0.0171%) ()),,)) indicating that performance gains are not simply due to increased processing capacity. These results suggest that MoDE's improvements arise from its design imodality-decouple experts and distillation), rather than significantly higher resource usage.
I Baseline Settings
In this section, provide the experimental settings for the baseline methods used in our experiments.
21

---

1.1 Overview of Baselines
Model Tailor [17] mitigates catastrophic forgetting in MILMS by (i) identifying sparse
"model patch" via a fused salience-and-sensiivityy score, and (ii) applying Hessian-based weight compensation the patch can be safely merged back into the frozen backbone.
DualPrompt [16] is replay-free continual learning method grafting two tiny prompt sets onto a frozen transformer backbone: a shared G-Prompt that encodes task-invariant knowledge and per-task E-Prompts keyed to input features for task-specific skills.
MoELoRA [4 extends LoRA by substituting each single low-rank adapter with mixture of N experts. The LoRA update is rewritten as:
N = Wx+ Σ G,B,A,T,
where every expert has rank r/N and gated by Gú, so the total trainable parameters remain identical vanilla LoRA while the separate experts capture more diverse, task-specific representations.
CL-MoE [8] tackles continual VQA on top of an MLLM by (i) introducing a Dual-Router MoE: task-level router picks global experts while an instance-level router refines the choice per example, and (ii) applying a dynamic momentum update that blends new A- -oaseb expert weights with the frozen past experts, the model absorbs new skills yet preserves prior knowledge, sharply reducing catastrophic forgetting.
Seq LoRA is a vanilla baseline that applies LoRA adapters in plain sequential fine-tuning across tasks.
1,2 Training Settings
To ensure fair comparison, we follow the hyperparameters and implementation details from each baseline's ' original codebase:
Model Tailor
sparsity level set to 10% (percentage of parameters replaced).
DualPrompt
prompt lengths Ly = 5 and Le = 20; prompts are injected at layers start, = endy start, 3, and end. = 5.
16
MoE-LoRA 44 number of experts set to 4, with each expert having LoRA rank 8.
number of experts set to 4, with each expert having LoRA rank 8. Top- experts selected with K 2. Hyperparameters 7 0.7, 0.5.
CL-MoE 18
Ethics Statement
This work does not involve human subjects, personally identifiable information, proprietary data. All datasets used (such as ScienceQA, TextVQA, GQA, VizWiz, and ImageNet) are publicly available and commonly used in the research community. We follow the licensing and usage policies associated with each dataset. Our continual learning method, MoDE, improves the robustness of unified multimodal generative models without modifying or exploiting sensitive data.
As with all powerful generative models capable of producing synthetic images from text, our method carries a potential risk of misuse, such as generating disinformation or harmful visual content. Although MoDE focuses on mitigating forgetting capabilities may inadvertently facilitate deceptive or malicious use. We recommend that practitioners deploying such models consider safeguards such as content filtering, provenance tracking, and visible or invisible watermarking of generated content to support responsible use and attribution.
multimodal learning, enhanced generation
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contributions of this paper is identifying and tackling the challenge of inter-modal catastrophic forgetting in unified multimodal models, which are accurately reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
23

---

Justification: Our method is applicable to autogressive transformers, which is made clear in the paper. Our experiments are conducted on autoregressive transformers as well.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide the theoretical results on modality gradient conflict in the main paper. The proofs and empirical verification are in the appendix A and B, respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our provided details of our architecture as well as experimental details in the paper. And our code would be released upon paper acceptance.
24

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The model checkpoints and datasets we use in this paper are open-sourced. And we will provide our code together with the other supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ptaips.cc//uulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
25

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include these details in the supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not applicable to our experiments involving generative mod- els, where evaluation is primarily qualitative or based on metrics not requiring statistical significance testing.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources information our experiments need is provided in the experiment section of the paper well in the supplemental materials.
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: There is no ethical concerns awared in this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential socictal impact discussed in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
27

---

Answer:
[NA]
Justification:
We don't 't release any pre-trained models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The models and datasets we use in the paper have been propoerly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code of this paper is well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
28

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this paper does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy hatoge//nearris.cc/oooffeeeeeeee for what should or should not be described.
29


Reviews:
[{"review_id": "Ze8YDttuPB", "reviewer": "Reviewer_vBLd", "summary": "This paper explores the challenge of catastrophic forgetting in Unified Multimodal Generative Models (UMGMs), focusing particularly on the phenomenon of inter-modal forgetting, alongside intra-modal forgetting. The paper identifies gradient conflict between visual and textual modalities as a key source of inter-modal forgetting and provides several theoretical contributions showing that such gradient conflicts directly cause inter-modal forgetting. The authors propose Modality-Decoupled Experts (MoDE), a straightforward approach that decouples visual and textual modality updates using modality-specific LoRA adapters, along with knowledge distillation from a frozen visual generation teacher to prevent intra-modal forgetting. MoDE outperforms competitive baselines such as MoE-LoRA and Model-Tailer on a number of multimodal continual learning tasks, and significantly reduces both intra- and inter-modal forgetting.", "questions": "1. A comparison of the trainable parameter and computational costs between the proposed method and the baselines would help alleviate concerns of triviality of the proposed method.\n2. An ablation showing a trivial fully de-coupled system with separate image-generation and text/multimodal understanding modules, would be helpful.\n3. Does Figure 1 training use LoRA adapters to measure forgetting? Why was forgetting caused by full fine-tuning not evaluated?\n\n_Points 1 and 2 above, as well as a clarification of the first 3 points expressed in the Weaknesses section, would go farthest in helping alleviate concerns._", "limitations": "Yes, Addressed.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- **Novel Theoretical Contributions:** The paper makes a number of theoretical formulations which are interesting and broadly useful:\n    - The proposed formulation of the Modality Gradient conflict, showing that an increase in image generation loss (forgetting) depends on the inner product of the formulated visual and text gradients is interesting. The formulation also shows how the proposed MoDE method provably bounds inter-modal interference to quadratic in step sizes.\n    - These are generally useful formulations, and would be of interest to future work in mitigating catastrophic forgetting in Unified Multimodal Generative Models (UMGMs). \n\n- **Strong results in continual instruction tuning \\& mitigation of catastrophic forgetting:**\n    - In both mitigation of catastrophic forgetting in image generation and visual understanding, as well as in continual instruction tuning experiments the proposed method MoDE almost consistently outperforms strong baselines like Model-tailor and MoE-LoRA. \n    - While this may be due to the trivial modality decoupling in the proposed MoDE, the approach achieves lower forgetting (and higher final accuracy) on almost all tasks. \n\n- **Well motivated study:** \n  - The unification and study of intra-model and inter-modal forgetting is interesting and well motivated.\n\n**Weaknessess:**\n\n- **Core method seems to lack novelty and significance**: The core mechanisms in the proposed MoDE method leverage existing ideas in modality-specific and MoE adapters. \n  - To prevent inter-modal forgetting, the proposed method involves de-coupling both visual and textual modalities by learning separate modules - Mixture-of-Expert LoRAs for Text and a Visual LoRA for vision. This seems like a trivial solution, and re-uses existing modules (MoE-LoRA, LoRA for vision adaptation). \n  - In addition, to prevent intra-modal forgetting (the authors only consider visual modality forgetting), MoDE uses a straightforward knowledge distillation with a frozen pre-trained teacher backbone. \n\n\n- **Strong results in large part due to modality decoupling, Missing decoupled baselines:** \n    - While the results of the proposed approach themselves are strong, it seems this is largely a result of the trivial modality decoupling used in the proposed approach. \n    - There is no clear baseline comparing the proposed approach to a trivial modality decoupled system with separate image-generation and text/multimodal understanding modules. It is therefore difficult to gauge the usefulness of the proposed approach. \n\n\n- **Missing Parameter and Computational costs comparisons:**\n    - Since the proposed MoDE essentially leverages separate LoRA adapters for both modalities as well as for different multimodal understanding tasks, it would be interesting to see a comparison of the trained parameters and the training computational costs between the proposed method and baselines. \n\n- **Text-only intra-modal forgetting:** Inter-modal forgetting is missing forgetting caused to text-only generation tasks, while only visual generation has been considered. This seems like a critical aspect of inter-modal forgetting. See [1,2]:\n\n[1]: Zhang, Yi-Kai, et al. \"Wings: Learning multimodal llms without text-only forgetting.\" Advances in Neural Information Processing Systems 37 (2024): 31828-31853.\n\n[2]: Srivastava, Shikhar, et al. \"Improving multimodal large language models using continual learning.\" arXiv preprint arXiv:2410.19925 (2024).\n\n- **Minor corrections:**\n   -  VQA tasks are repeatedly referred to as text-only (Line 122-123, for instance) in the paper. While the generation is in text, the task however is _not_ \"text-only\" since these are VQA.  This is very odd, and must be clarified/corrected in the paper. \n    - Line 122-123, Line 50: \"(e.g., improving language generation while preserving visual generation)\" where this isn't language generation, it's multimodal understanding.", "was_revised": true, "final_justification": "I thank the authors for an excellent response to the numerous queries and concerns. In particular, the ablations showing decoupling without KD, and adding KD without decoupling both perform materially worse is a strong support to the central arguments of the paper. In combination with the strong results, I believe it is easy to recommend acceptance for this paper.\n\nI nevertheless encourage the authors to clarify their terminologies. In particular, please ensure the focus on only visual forgetting as inter-modal forgetting is clarified early in the paper, and that text-only forgetting is excluded from the study is clarified in the related work.", "rebuttal": "We thank Reviewer vBLd for the positive assessment of our paper’s theoretical contributions, strong empirical results in continual instruction tuning, and the well-motivated study of intra- and inter-modal forgetting in unified multimodal models. The questions raised are insightful, and we will address each point below to clarify our contributions and the specifics of our work. \n\n## Weaknesses 1&2, Q2: Novelty and decoupled baselines\nWe understand the concern that our building blocks (MoE‑LoRA, LoRA, KD) have appeared in prior work. To demonstrate that their combination in MoDE is neither trivial nor interchangeable, we conducted the following ablations:\n\n**1. KD without decoupling.**\nAs shown in Table R1, adding our cross‑modal KD loss to CL‑MoE or to MoELoRA reduces multimodal understanding accuracy (e.g. 34.14% to 32.41% for MoELoRA), even though it does mitigate visual‑generation forgetting. This is because without explicit modality decoupling, KD “locks in” the teacher’s features in a way that hinders the model’s ability to adapt to new VQA and grounding tasks.\n\n**Table R1: Effect of adding KD to MoELoRA and CL-MoE.**\n\n| Method         ||**Image Generation** || **Visual Understanding** || \n|----------------|:----------------------:|:----------------------------:|:----------------:|:------------------------:|:-----------------:|\n|                | Text alignment (↑)   | Image alignment (↑)       | FID (↓)        | Accuracy (↑)           | Forgetting (↓)  |\n||\n| MoELoRA [2] | 0.2094               | **0.5234**                | 63.26          | **34.14**              | **24.08**       |\n| MoELoRA+KD     | **0.2568**           | 0.5176                    | **52.81**      | 32.41                  | 25.27           |\n||\n| CL‑MoE [3]        | 0.2370               | 0.5146                    | 54.69          | **36.96**              | **19.12**       |\n| CL‑MoE+KD      | **0.2439**           | **0.5200**                | **52.43**      | 30.68                  | 21.23           |\n\n**2. Decoupled system without KD.**\nAs shown in Table R2, decoupling textual and visual LoRA without knowledge distillation yields only modest gains in understanding accuracy and visual generation quality compared to vanilla LoRA. This indicates that decoupling alone is insufficient to maintain the rich cross-modal alignment necessary for high-quality generation. Furthermore, Section 5.3 of the main paper includes an ablation on “MoDE w/o KD” (i.e., decoupled T-MoE and V-adapter without knowledge distillation), which further confirms that knowledge distillation plays a critical role in the performance improvements observed.\n\n**Table R2: Ablation on Fully De-coupled Architecture.**\n\n| Method           ||**Image Generation**|| **Visual Understanding** || \n|------------------|:----------------------:|:----------------------------:|:----------------:|:------------------------:|:-----------------:|\n|                  | Text alignment (↑)   | Image alignment (↑)       | FID (↓)        | Accuracy (↑)           | Forgetting (↓)  |\n||\n| SeqLoRA          | 0.2183               | 0.5150                    | 56.75          | 28.75                  | 26.83           |\n| Decoupled LoRA   | 0.2252               | 0.5180                    | 54.74          | 29.07                  | 27.60           |\n\nIn conclusion, the above ablations show that MoDE is more than the sum of its parts: only the combination of modality‑decoupling plus knowledge distillation leads to consistently superior performance.\n## Weakness 3 & Q1: Parameter and computational costs comparisons\nWe appreciate the reviewer’s suggestion. To address concerns regarding the triviality of our method, we report a comparison of training compute, memory footprint, and trainable parameters in the table below. As shown, MoDE introduces only a slight increase in training time (440.3 ms vs. 419.2 ms) and peak GPU memory (84.98 GB vs. 75.27 GB) compared to MoELoRA, while maintaining comparable TFLOPs. The trainable parameter ratio of MoDE is only 0.0211%, modestly higher than MoELoRA and CL-MoE (0.0171%), indicating that our performance gains are not simply due to increased processing capacity. These results suggest that MoDE's improvements arise from its design (modality-decoupled experts and distillation), rather than significantly higher resource usage.\n\n**Table R3: Comparison of Trainable Parameters and Training Costs.**\n\n| Method        | TFLOPs | Peak GPU Memory (MB) | Training Time (ms) | Trainable Params (%) |\n|---------------|:--------:|:----------------------:|:---------------------:|:-----------------------:|\n| MoELoRA [2] | 4.29 | 75265 | 419.2 | 0.0171 |\n| CL‑MoE [3] | 4.29 | 86300 | 421.7 | 0.0171 |\n| **MoDE (Ours)** | 4.19 | 84980 | 440.3 | 0.0211 |\n\n## Weakness 4: Text-only inter-modal forgetting\nWe appreciate the reviewer’s insightful comment and references. This is the first work on inter-modal forgetting on UMGMs continual learning, so we use the forgetting in visual generation caused by continual instruction tuning on multimodal understanding as the study case. This is a very interesting direction; we will explore it in our future work. However, this is currently beyond the scope of our study, which aims to establish the foundational analysis and mitigation strategies for inter-modal interference through the lens of visual generation degradation.\n\n## Weakness 5: Correction of terminology\nWe apologize for the confusion caused by the misuse of \"text-only generation\" and \"language generation\" and will replace them with \"multimodal understanding\" in the revised paper. \n\n## Q3: Figure 1 explanation\nThe question raised concerns about whether Figure 1 uses LoRA adapters to measure forgetting, and why forgetting caused by full-model fine-tuning was not evaluated.\nYes, Figure 1 is obtained using LoRA-based fine-tuning. While we acknowledge that full-model fine-tuning may lead to different absolute values, we chose LoRA due to practical constraints on compute resources and because it has become a widely used paradigm for efficient continual adaptation in large-scale models.\nImportantly, our goal is to study the modality imbalance and forgetting phenomena under realistic fine-tuning setups. LoRA provides a representative and resource-efficient way to examine these effects in current practice. Moreover, prior work (e.g., [1]) has shown that full fine-tuning tends to exacerbate catastrophic forgetting in MLLMs, so our reported forgetting levels are likely conservative.\nTherefore, even under LoRA, the modality-specific forgetting patterns are clearly observable and support the motivation and effectiveness of our proposed MoDE approach.\n### References\n[1]: Zhu, Didi, et al. \"Model tailor: Mitigating catastrophic forgetting in multi-modal large language models.\" arXiv preprint arXiv:2402.12048 (2024).\n\n[2]: Chen, Cheng, et al. \"Coin: A benchmark of continual instruction tuning for multimodel large language models.\" Advances in Neural Information Processing Systems 37 (2024): 57817-57840.\n\n[3]: Huai, Tianyu, et al. \"CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "3pWMWdtevu", "reviewer": "Reviewer_qoBP", "summary": "This paper is the first to investigate the inter-modal catastrophic forgetting issue in Unified Multimodal Understanding and Generation Models (UMGMs) during continual instruction tuning. While previous work on multimodal continual instruction tuning has focused solely on understanding tasks, this paper explores how UMGMs forget their visual generation capabilities after being continually fine-tuned on multimodal understanding tasks. The authors attribute this forgetting to gradient conflicts between image generation loss and text generation loss with respect to shared model parameters and provide a theoretical explanation for this phenomenon. To mitigate this problem, the authors propose a modality-decoupled expert architecture. Specifically, they introduce a MoELoRA module for text token generation (referred to as T-MoE), and a separate LoRA module for image token generation (referred to as V-Adapter). To preserve the image generation ability, they further apply knowledge distillation from the original (teacher) UMGM to the V-Adapter in the new (student) model.", "questions": "- The MoELoRA (CoIN) benchmark includes 8 multimodal understanding tasks: ScienceQA, TextVQA, ImageNet, GQA, VizWiz, Grounding, VQAv2, and OCR-VQA. Why does the evaluation in this paper only cover the first six tasks? What are the results of all 8 tasks?\n\n- In the supplementary material, the results for DualPrompt and MoELoRA are missing in the different task ordering section. Also, the Janus-Pro results are missing for DualPrompt, Model Tailor, and MoELoRA. Could these be included?\n\n- According to Table 2, MoDE only shows marginal improvements in multimodal understanding performance (both in the best accuracy during continual instruction tuning and the final accuracy after completing all tasks) compared to MoELoRA and CL-MoE. The main benefit seems to lie in mitigating the forgetting in image generation. If knowledge distillation for image generation were directly applied to MoELoRA and CL-MoE, how would they perform? Additionally, can the authors report the detailed accuracy on previous tasks after each task training step?\n\n- In the current tables, bold fonts highlight the results of the proposed method. It might be clearer to instead bold the best results for each metric.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n- The paper is the first to investigate the catastrophic forgetting of image generation capabilities in UMGMs under continual instruction tuning.\n\n- A theoretical explanation is provided by analyzing the gradient conflicts between image token generation loss and text token generation loss with respect to shared model parameters.\n\n\nCons:\n\n- The paper assumes that image generation is only conducted during pretraining and that continual instruction tuning only involves multimodal understanding tasks. If the model is further fine-tuned on image generation tasks, will it then catastrophically forget the understanding tasks? Can the proposed method prevent this problem as well?\n\n- The “Reference Token” in Figure 3 is not clearly explained. When knowledge distillation is applied to image generation during continual tuning, what text prompt is used as the reference?", "was_revised": true, "final_justification": "The author's response has addressed my concerns.", "rebuttal": "We thank Reviewer qoBP for highlighting the novelty of our work as the first to study catastrophic forgetting of image generation in UMGMs under continual instruction tuning, and for appreciating our theoretical explanation based on gradient conflicts between image and text generation losses. The questions raised are insightful, and we provide our detailed responses below.\n\n## Weakness 1: Fine-tuning on image generation tasks\nThank you for the insightful question: whether fine-tuning on image generation tasks would lead to catastrophic forgetting of the understanding capabilities, and whether our proposed method can mitigate such forgetting. \nTo address this, we conduct new experiments by fine-tuning the linear layers of the Chameleon model [3] on image generation tasks. Specifically, we use the LAION-5B dataset [1] for fine-tuning and apply knowledge distillation using the multimodal understanding sequence from the GQA dataset [2] as reference data. We compare our MoDE method with LoRA fine-tuning across three multimodal understanding benchmarks. Accuracies are reported in Table R1. The difference between zero-shot and fine-tuned performance reflects the degree of forgetting in multimodal understanding.\n\n**Table R1: Forgetting caused by fine-tuning on image generation tasks.**\n\n| Method        | ScienceQA | TextVQA | ImageNet |\n|---------------|-----------|---------|----------|\n| Zero-shot     | 51.52     | 23.49   | 16.53    |\n| LoRA          | 45.81     | 12.25   | 0.00     |\n| **MoDE (Ours)**   | 51.28     | 23.44   | 16.61    |\n\nThese results support two conclusions:\n\n(1) Yes, fine-tuning on image generation tasks can indeed cause catastrophic forgetting in multimodal understanding, confirming that the forgetting problem is bidirectional. This is evident in the substantial drop in performance for the LoRA baseline compared to zero-shot.\n\n(2) Our MoDE method remains effective in this scenario, significantly reducing the performance drop and mitigating forgetting relative to the LoRA baseline. This demonstrates that MoDE generalizes beyond the specific forgetting direction explored in the main paper, where understanding tasks overwrite prior image generation capabilities, and is effective regardless of the fine-tuning direction.\n\n## Weakness 2: \"Reference Data\" in Figure 3\nIn our experiments, \"Reference Data\" are text prompts selected from LAION-5B dataset [1].\n\n## Q1: CoIN benchmark\nThank you for pointing this out. Due to resource constraints, we selected a subset of 6 tasks from the full CoIN benchmark to ensure feasibility while maintaining diversity across modalities and task types. Specifically, the chosen tasks (ScienceQA, TextVQA, ImageNet, GQA, VizWiz, and Grounding) span a representative range of vision-language reasoning, recognition, and grounding challenges.\nWhile we did not include VQAv2 and OCR-VQA in our current evaluation, we expect similar trends to hold, given the consistency of results across the six evaluated tasks. Extending evaluation to all 8 tasks is a valuable contribution and we plan to include it in the final paper. \n\n## Q2: Additional supplementary results\nThank you for the helpful suggestion regarding the missing results for DualPrompt and MoELoRA under different task orderings, as well as the missing Janus-Pro results for several baselines in the supplementary material. We provide the complete results below:\n\n**Q2.1:** We report additional ablations on task order sensitivity in Table R2. DualPrompt [5] and MoELoRA [4] are now included alongside MoDE. DualPrompt yields low accuracy and low forgetting. This is because adding a small set of prompt embeddings slightly biases the feature representations, which is insufficient to learn the new and hard tasks. The results in Table R2 show that MoDE maintains superior performance and robustness under alternate task orders.\n\n**Table R2: Additional ablation on task order.**\n\n| Method           ||**Image Generation**|| **Visual Understanding** || \n|------------------|:----------------------:|:----------------------------:|:----------------:|:------------------------:|:-----------------:|\n|                  | Text alignment (↑)   | Image alignment (↑)       | FID (↓)        | Accuracy (↑)           | Forgetting (↓)  |\n||\n| DualPrompt [5] | 0.2491               | 0.5140                    | 53.07          | 27.43                  | **7.09**         |\n| MoELoRA [4] | 0.2492               | 0.5125                    | 61.64          | 33.37                  | 23.61           |\n| **MoDE (Ours)**  | **0.2558**           | **0.5256**                | **51.69**      | **35.11**              | 18.25           |\n\n\n**Q2.2:** In Table R3, we provide the missing Janus-Pro results for Model Tailor [6], MoELoRA[4], and our MoDE. Again, MoDE demonstrates the best performance with the lowest forgetting and highest accuracy.\n\n**Table R3: Evaluation on Janus-Pro benchmark.**\n\n| Method           | Accuracy (↑) | Forgetting (↓) |\n|------------------|--------------|----------------|\n| Model Tailor [6] | 29.13        | 32.01          |\n| MoELoRA [4] | 24.61        | 36.10          |\n| **MoDE (Ours)**  | **53.02**    | **14.48**      |\n\n## Q3: KD on MoELoRA and CL-MoE\nThank you for the thoughtful question about the effect of applying image generation knowledge distillation to MoELoRA and CL-MoE, and whether MoDE’s benefit extends beyond that. \nTo directly evaluate the effect of applying knowledge distillation (KD) for image generation on existing baselines, we conduct new experiments by incorporating KD into MoELoRA and CL-MoE. The results are shown in Table R4.\n\n**Table R4: Effect of KD on MoELoRA and CL-MoE.**\n\n| Method           || **Image Generation** || **Visual Understanding** || \n|------------------|:----------------------:|:----------------------------:|:----------------:|:------------------------:|:-----------------:|\n|                  | Text alignment (↑)   | Image alignment (↑)       | FID (↓)        | Accuracy (↑)           | Forgetting (↓)  |\n||\n| MoELoRA [4] | 0.2094 | 0.5234 | 63.26 | 34.14 | 24.08 |\n| MoELoRA + KD | 0.2568 | 0.5176 | 52.81 | 32.41 | 25.27 |\n||\n| CL‑MoE [7] | 0.2370 | 0.5146 | 54.69 | 36.96 | 19.12 |\n| CL‑MoE + KD | 0.2439 | **0.5200** | 52.43 | 30.68 | 21.23 |\n||\n| **MoDE (Ours)** | **0.2587** | **0.5200** | **51.27** | **37.05** | **18.25** |\n\nThese results show that applying KD improves image generation quality for both MoELoRA and CL-MoE, as indicated by higher alignment scores and lower FIDs. However, KD degrades the multimodal understanding performance (reducing accuracy and increasing forgetting) because the models are not decoupled, and the KD interferes with learning on understanding tasks. In contrast, MoDE achieves the best performance across all metrics by design: its modality-decoupled structure prevents interference from KD and enables effective retention of multimodal understanding.\n\n## Q4: Bold fonts\nThanks for valuable suggestion and we will bold the best results for every metric. \n### References\n[1]: Schuhmann, Christoph, et al. \"Laion-5b: An open large-scale dataset for training next generation image-text models.\" Advances in neural information processing systems 35 (2022): 25278-25294.\n\n[2]: Hudson, Drew A., and Christopher D. Manning. \"Gqa: A new dataset for real-world visual reasoning and compositional question answering.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n[3]: Team, Chameleon. \"Chameleon: Mixed-modal early-fusion foundation models.\" arXiv preprint arXiv:2405.09818 (2024).\n\n[4]: Chen, Cheng, et al. \"Coin: A benchmark of continual instruction tuning for multimodel large language models.\" Advances in Neural Information Processing Systems 37 (2024): 57817-57840.\n\n[5]: Wang, Zifeng, et al. \"Dualprompt: Complementary prompting for rehearsal-free continual learning.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\n[6]: Zhu, Didi, et al. \"Model tailor: Mitigating catastrophic forgetting in multi-modal large language models.\" arXiv preprint arXiv:2402.12048 (2024).\n\n[7]: Huai, Tianyu, et al. \"CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "wNryOYimzM", "reviewer": "Reviewer_LAT9", "summary": "In this work, the authors propose Modality-Decoupled Experts (MoDE). This architecture handles intra- and inter-modal forgetting for Unified Multimodal Generative Models (UMGMs). MoDE consists of two modules: a V-adapter (a LoRA module) used to capture visual knowledge and a T-MoE, a LoRA Mixture of Experts that handles textual information. By decoupling the vision and language modalities, the authors claim to reduce forgetting when training the model on a sequence of tasks. The authors provide a wide range of experiments to validate their thesis.", "questions": "1. How does MoDE compare to CL-MoE regarding time and memory costs?\n\n2. The reference dataset is not clear. I understand it is used for the KD loss, but how is it composed? Is it task-related, or is it always the same?\n\nPlease refer also to the weaknesses.", "limitations": "The authors have not discussed limitations. One may be the hardware requirements.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The paper is well-written and easy to follow.\n\n- The problem of dealing with forgetting in both modalities of a UMGM is interesting and worth exploring.\n\n- The experiments section and appendix are exhaustive\n\n### Weaknesses\n\n- While the results are consistently better than previous approaches, they are incremental with respect to CL-MoE (+0.1 in Tab. 2). This may raise some questions about the approach's statistical validity. Multiple seeded runs would have helped to understand this aspect better.\n\n- The V-Adapter and T-MoE have never really been introduced. They are explained only in the caption of Fig. 3.\n\n- Details and ablations on the number of experts in T-MoE are missing.", "was_revised": true, "final_justification": "Authors have addressed all concerns, thus I increase my rating to Accept.", "rebuttal": "We thank Reviewer LAT9 for the positive feedback on the clarity of our writing, the importance of addressing forgetting in both modalities of UMGMs, and the thoroughness of our experiments and appendix. We appreciate the constructive comments and address the raised concerns in detail below.\n## Weakness 1: Statistical validity of results in Tab. 2\nWe thank the reviewer for raising this important point of the statistical significance of the improvements over CL-MoE. \nTo evaluate the robustness of our results, we conducted 3 runs with different random seeds for both CL-MoE and MoDE. The results are summarized in Table R1 below. We report the mean and standard deviation across 3 runs.\n\n**Table R1: Results with multiple seeds (with mean and standard deviation). For each method, the\nupper row shows the best accuracy during continual instruction tuning, and the lower row shows the\nfinal accuracy after completing all tasks.**\n\n| Method         | ScienceQA            | TextVQA              | ImageNet            | GQA                  | VizWiz              | Grounding           | ACC (↑)      | Fgt (↓)      |\n|----------------|----------------------|-----------------------|----------------------|-----------------------|----------------------|----------------------|--------------|--------------|\n| CL‑MoE [2]        | 71.35±0.01           | 38.88±0.06            | 90.12±0.04           | 37.37±0.04            | 43.74±0.07           | 35.76±0.07           |              |              |\n|                | 67.32±0.01           | 32.82±0.01            | 9.13±0.04            | 33.30±0.04            | 43.20±0.01           | 35.76±0.07           | 36.92    | 19.14    |\n||\n| **MoDE (Ours)**| 71.49±0.01           | 39.00±0.01            | 85.88±0.04           | 36.73±0.01            | 44.45±0.01           | 38.55±0.08           |              |              |\n|                | 67.87±0.01           | 33.36±0.01            | 9.54±0.01            | 33.60±0.01            | 41.91±0.01           | 38.55±0.08           | **37.47**    | **18.25**    |\n\nAlthough the absolute improvement in ACC may appear small (e.g., +0.55), MoDE consistently improves both accuracy and forgetting across all seeds and datasets. Given the small standard deviations, the performance gains are meaningful and robust. These results strengthen the statistical significance and reliability of our conclusions.\n\n## Weakness 2: Introduction of V-Adapter and T-MoE\nThank you for pointing out that V-Adapter and T-MoE are not formally introduced in the main text and are only briefly mentioned in the caption of Figure 3. We agree that the current version does not clearly introduce V-Adapter and T-MoE in the main text. In the revised version, we will explicitly define them in Section 4.2 Modality-Decoupled Experts (MoDE). \n\n## Weakness 3: Ablation on the number of experts in T-MoE\nThank you for the suggestion. We provide an ablation study on the number of experts in the T-MoE module. As shown in Table R2 below, increasing the number of experts from 4 to 6 yields slightly improved text alignment and accuracy, though the gains are modest. This suggests that the performance of MoDE is relatively robust to the number of experts, and 4 experts offer a good trade-off between performance and computational cost.\n\n**Table R2: Ablation on the number of experts in T-MoE.**\n\n| #Experts || **Image Generation** || **Visual Understanding** || \n|----------|:----------------------:|:----------------------------:|---------:|:------------------------:|:-----------------:|\n|          | Text alignment (↑)   | Image alignment (↑)        | FID (↓) | Accuracy (↑)           | Forgetting (↓)  |\n||\n| 4        | 0.2587               | 0.5190                     | 51.27   | 37.05                  | 18.25           |\n| 6        | 0.2592               | 0.5166                     | 51.13   | 37.26                  | 18.64           |\n\n## Q1: Time and Memory Costs Compared to CL-MoE\nThank you for the question of \"How does MoDE compare to CL-MoE regarding time and memory costs?\". \nWe compare MoDE with CL-MoE and MoELoRA in terms of TFLOPs, peak GPU memory, training time per iteration, and the percentage of trainable parameters. Results are shown in Table R3 below.\nMoDE incurs a slightly higher computational cost and memory usage than CL-MoE due to its use of both modality-specific adapters and knowledge distillation. However, the differences are modest and well within practical bounds. In return, MoDE consistently yields better retention and overall performance, demonstrating a favorable trade-off between efficiency and effectiveness.\n\n**Table R3: Comparison of training efficiency and memory usage.**\n\n| Method        | TFLOPs | Peak GPU Memory (MB) | Training Time (ms) | Trainable Params (%) |\n|---------------|--------|----------------------|---------------------|-----------------------|\n| MoELoRA [3]      | 4.29   | 75265               | 419.2               | 0.0171                |\n| CL‑MoE [2]       | 4.29   | 86300               | 421.7               | 0.0171                |\n| **MoDE (Ours)**   | 4.19   | 84980               | 440.3               | 0.0211                |\n\n## Q2: Reference dataset\nThank you for the question regarding the composition of the reference dataset used for knowledge distillation (KD). The reference dataset used for KD loss is composed of text prompts sampled from the LAION-5B dataset [1]. The dataset is shared across all tasks, rather than being tailored to the current task. The shared reference dataset ensures that the KD signal reflects general visual generation capability, independent of the downstream multimodal understanding task. We will clarify this more explicitly in the revised manuscript.\n### References\n[1]: Schuhmann, Christoph, et al. \"Laion-5b: An open large-scale dataset for training next generation image-text models.\" Advances in neural information processing systems 35 (2022): 25278-25294.\n\n[2]: Huai, Tianyu, et al. \"CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[3]: Chen, Cheng, et al. \"Coin: A benchmark of continual instruction tuning for multimodel large language models.\" Advances in Neural Information Processing Systems 37 (2024): 57817-57840."}, {"review_id": "xgGNqiRgkE", "reviewer": "Reviewer_TLHk", "summary": "The authors propose Modality-Decoupled Experts (MoDE) for continual learning of unified multimodal generative models (UMGMs). MoDE consists of separate LoRA blocks for the vision component and the text components (multiple LoRAs for text sub-tasks). The authors further regularize the continual learning process with standard knowledge distillation from the base model.", "questions": "1. What happens when one simply uses KD on top of seq LoRA?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "## Strengths\n\n1. Misalignment in the gradient direction between the two tasks in UMGMs is well-known, and an important problem to address, given the increasing attention in developing a GPT-4o-like UMGM systems.\n\n2. To the best of my knowledge, most concurrent works focus on tweaking the model architecture the enhance the model capabilities. This paper looks at the problem through a continual learning lens.\n\n3. The method seems to be effective for alleviating catastrophic forgetting for image generation while fine-tuning on instruction tuning datasets.\n\n## Weaknesses\n\n1. The paper would be much more interesting if the authors tested their method on the case where the task is to prevent catastrophic forgetting of text generation while improving image generation. More importantly, a more realistic setting would be to test the method when the model is trained in an alternating fashion. Would the method still work on these settings?\n\n2. Proposition 1 states that *when* gradients conflict, then the loss would increase, which is trivial.\n\n3. The core idea (separate LoRA adapters for module + KD) is incremental.\n\n4. There are several concurrent works that aim for better unified modeling by e.g. using a mixture of experts [1] or using separate vision encoders [2]. It is advised that the authors discuss the relevant works, including whether their contributions are orthogonal to the other approaches.\n\n**References**\n\n[1] Deng, Chaorui, et al. \"Emerging properties in unified multimodal pretraining.\" arXiv preprint arXiv:2505.14683 (2025).\n\n[2] Zou, Jialv, et al. \"Omnimamba: Efficient and unified multimodal understanding and generation via state space models.\" arXiv preprint arXiv:2503.08686 (2025).", "was_revised": true, "final_justification": "I thank the authors for the rebuttal. Given the thorough rebuttal along with the other reviewers' comments, I will raise my score.", "rebuttal": "We thank Reviewer TLHk for recognizing the importance of addressing gradient misalignment in UMGMs, and for acknowledging our novel continual learning perspective compared to concurrent architectural approaches. We are also glad the reviewer found our method effective in mitigating catastrophic forgetting in image generation during instruction tuning. The questions and suggestions raised are appreciated, and we provide detailed responses below.\n\n## Weakness 1: Forgetting caused by fine-tuning on image generation and alternating tasks\nThank you for this insightful suggestion to evaluate MoDE in two additional settings: preventing forgetting of text generation while improving image generation, and training the model in an alternating task schedule. \nWe agree that testing for forgetting of text generation while improving image generation, as well as alternating training, are both highly relevant extensions. While we have not yet explored the full alternating schedule, we conducted a new experiment where the model is fine-tuned on image generation only, and we evaluate how much it forgets multimodal understanding. Results are shown in Table R1 below.\n\n**Table R1: Forgetting caused by fine-tuning on image generation tasks.**\n\n| Method        | ScienceQA | TextVQA | ImageNet |\n|---------------|-----------|---------|----------|\n| Zero-shot     | 51.52     | 23.49   | 16.53    |\n| LoRA          | 45.81     | 12.25   | 0.00     |\n| **MoDE (Ours)**   | 51.28     | 23.44   | 16.61    |\n\nThese results confirm that fine-tuning on image generation can **cause catastrophic forgetting in multimodal understanding tasks**. MoDE is able to mitigate this forgetting effectively, recovering near-zero-shot performance. This demonstrates that our method is bidirectionally effective. \nAs for the **alternating training schedule**, we agree this is an important and realistic scenario. While it is out of scope for this version due to resource limitations, we are actively working on extending MoDE to this setting and will include it in future work.\n\n## Weakness 2: Proposition 1 triviality\nWe acknowledge that Proposition 1, which establishes the connection between gradient conflict and loss increase, may appear intuitive. However, prior works such as OmniMamba [3] and MOSS [4] have focused on architectural or adapter-level decoupling *without* providing a theoretical explanation for inter-modal interference. To the best of our knowledge, our analysis is the first to formally link gradient conflict to inter-modal catastrophic forgetting, offering a principled understanding of why modality interference occurs. This theoretical insight directly motivates the design of modality-decoupled adapters in MoDE. Furthermore, Appendices A and B empirically validate our motivation, showing that the theoretical predictions align closely with observed performance.\n\n## Weakness 3: Incremental core idea\nWe agree with the reviewer’s point that LoRA and KD have been studied individually. However, our work makes a non‑trivial advance by i) jointly integrating these components within a continual instruction‑tuning paradigm, and ii) adding a gradient‑orthogonality regularizer. Critically, the following ablations demonstrate that neither piece alone, nor a naive combination of them can yield the strong trade‑offs we report below:\n\n**1. KD without  decoupling.**\nAs shown in Table R2, adding our cross‑modal KD loss to CL‑MoE or to MoELoRA reduces multimodal understanding accuracy (e.g. 34.14% to 32.41% for MoELoRA), even though it does mitigate visual‑generation forgetting. This is because without explicit modality decoupling, KD “locks in” the teacher’s features in a way that hinders the model’s ability to adapt to new multimodal understanding tasks.\n\n**Table R2: Effect of KD on MoELoRA and CL-MoE.**\n\n| Method         || **Image Generation** || **Visual Understanding** || \n|----------------|:----------------------:|:----------------------------:|:----------------:|:------------------------:|:-----------------:|\n|                | Text alignment (↑)   | Image alignment (↑)       | FID (↓)        | Accuracy (↑)           | Forgetting (↓)  |\n||\n| MoELoRA [5]       | 0.2094               | **0.5234**                | 63.26          | **34.14**              | **24.08**       |\n| MoELoRA+KD     | **0.2568**           | 0.5176                    | **52.81**      | 32.41                  | 25.27           |\n||\n| CL‑MoE [6]        | 0.2370               | 0.5146                    | 54.69          | **36.96**              | **19.12**       |\n| CL‑MoE+KD      | **0.2439**           | **0.5200**                | **52.43**      | 30.68                  | 21.23           |\n\n**2. Decoupled system without KD.**\nAs shown in Table R3, decoupling textual and visual LoRA without knowledge distillation yields only modest gains in understanding accuracy and visual generation quality compared to vanilla LoRA. This indicates that decoupling alone is insufficient to maintain the rich cross-modal alignment necessary for high-quality generation. Furthermore, Section 5.3 of the main paper includes an ablation on “MoDE w/o KD” (i.e., decoupled T-MoE and V-adapter without knowledge distillation), which further confirms that knowledge distillation plays a critical role in the performance improvements observed.\n\n**Table R3: Ablation on Fully De-coupled Architecture.**\n\n| Method           || **Image Generation** || **Visual Understanding** || \n|------------------|:----------------------:|:----------------------------:|:----------------:|:------------------------:|:-----------------:|\n|                  | Text alignment (↑)   | Image alignment (↑)       | FID (↓)        | Accuracy (↑)           | Forgetting (↓)  |\n||\n| SeqLoRA          | 0.2183               | 0.5150                    | 56.75          | 28.75                  | 26.83           |\n| Decoupled LoRA   | 0.2252               | 0.5180                    | 54.74          | 29.07                  | 27.60           |\n\nThe above ablations show that MoDE is more than the sum of its parts: only the combination of modality‑decoupling plus our specialized distillation leads to consistently superior trade‑offs.\n\n## Weakness 4: Concurrent works\nThank you for the valuable suggestion to discuss concurrent works on better unified modeling and to clarify whether our contributions are orthogonal to these approaches. Both BAGEL [2] and OmniMamba [3] adopt decoupled encoders, and OmniMamba further applies decoupled LoRA adapters, which have similarities to our approach. This indicates that the design of modality-decoupled experts has been recognized as an effective solution. However, their focus is primarily on architectural decoupling for pretraining or static tasks, rather than continual instruction tuning. In contrast, our work addresses the challenges of continual adaptation, which is essential for real-world scenarios such as lifelong multimodal assistants and adaptive AI systems that learn and evolve over time.\n\n## Q1: KD on top of seq LoRA\nThank you for the question regarding the effect of applying knowledge distillation (KD) on top of sequential LoRA. \nWe conducted an ablation to evaluate the effect of adding KD on top of vanilla sequential LoRA fine-tuning. As shown in Table R4 below, applying KD leads to modest improvements in both image generation quality (higher alignment scores, lower FID) and multimodal understanding performance (higher accuracy, reduced forgetting).\nThis suggests that KD alone can partially mitigate forgetting and preserve visual generation quality. However, as shown in other results, KD without proper modality decoupling may still interfere with understanding tasks. MoDE outperforms SeqLoRA+KD by explicitly addressing this interference through modality separation.\n\n**Table R4: Effect of knowledge distillation on SeqLoRA.**\n\n| Method           || **Image Generation** || **Visual Understanding** || \n|------------------|:----------------------:|:----------------------------:|:----------------:|:------------------------:|:-----------------:|\n|                  | Text alignment (↑)   | Image alignment (↑)       | FID (↓)        | Accuracy (↑)           | Forgetting (↓)  |\n||\n| SeqLoRA          | 0.2183               | 0.5150                    | 56.75          | 28.75                  | 26.83           |\n| SeqLoRA + KD | 0.2531           | 0.5176                | 52.22      | 30.54              | 23.18       |\n### References\n[1]: Schuhmann, Christoph, et al. \"Laion-5b: An open large-scale dataset for training next generation image-text models.\" Advances in neural information processing systems 35 (2022): 25278-25294.\n\n[2]: Deng, Chaorui, et al. \"Emerging properties in unified multimodal pretraining.\" arXiv preprint arXiv:2505.14683 (2025).\n\n[3]: Zou, Jialv, et al. \"Omnimamba: Efficient and unified multimodal understanding and generation via state space models.\" arXiv preprint arXiv:2503.08686 (2025).\n\n[4]: Xu, Zhiyang, et al. \"Modality-Specialized Synergizers for Interleaved Vision-Language Generalists.\" The Thirteenth International Conference on Learning Representations. 2025.\n\n[5]: Chen, Cheng, et al. \"Coin: A benchmark of continual instruction tuning for multimodel large language models.\" Advances in Neural Information Processing Systems 37 (2024): 57817-57840.\n\n[6]: Huai, Tianyu, et al. \"CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}]

Author rebuttal:

Meta-review: The paper introduces inter-modal forgetting in unified multimodal generative models, formalizes its cause through gradient conflict analysis, and proposes a solution. Experiments show how MoDE preserves image generation quality while sustaining multimodal understanding accuracy, outperforming baselines.

Strengths: The paper identifies and formalizes an important and timely problem, and presents an effective mitigation.

Weaknesses: The mechanism (decoupling + KD) is incremental, with novelty primarily in problem framing. Evaluation is limited to one main task order and a 6-task stream (rather than the full 8 task stream, without much justification). Missing references and related prior work flagged by reviewers should be added.

Rebuttal & Discussion: Reviewers initially questioned novelty and evaluation scope. The authors added experiments and new results, which addressed most concerns. These additional results and missing references should be integrated into the main text.

Decision: Accept (poster). The paper identifies a new failure mode, provides a practical solution, and demonstrates strong results.
Decision: Accept (poster)