SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Authors: Jinpeng Chen
Runmin Cong
Yuzhi Zhao
Hongzheng Yang
Guangneng Hu
Horace Ip
Sam Kwong
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Continual Instruction Tuning
Multimodal Large Language Model
Continual Learning
Primary area: deep_learning

Abstract: Multimodal Continual Instruction Tuning (MCIT) aims to enable Multimodal Large Language Models (MLLMs) to incrementally learn new tasks without catastrophic forgetting, thus adapting to evolving requirements. In this paper, we explore the forgetting caused by such incremental training, categorizing it into superficial forgetting and essential forgetting. Superficial forgetting refers to cases where the model’s knowledge may not be genuinely lost, but its responses to previous tasks deviate from expected formats due to the influence of subsequent tasks’ answer styles, making the results unusable. On the other hand, essential forgetting refers to situations where the model provides correctly formatted but factually inaccurate answers, indicating a true loss of knowledge. Assessing essential forgetting necessitates addressing superficial forgetting first, as severe superficial forgetting can conceal the model’s knowledge state. Hence, we first introduce the Answer Style Diversification (ASD) paradigm, which defines a standardized process for data style transformations across different tasks, unifying their training sets into similarly diversified styles to prevent superficial forgetting caused by style shifts. Building on this, we propose RegLoRA to mitigate essential forgetting. RegLoRA stabilizes key parameters where prior knowledge is primarily stored by applying regularization to LoRA’s weight update matrices, enabling the model to retain existing competencies while remaining adaptable to new tasks. Experimental results demonstrate that our overall method, SEFE, achieves state-of-the-art performance.

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Jinpeng Chen
Runmin Cong 2
Yuzhi Zhao Hongzheng Yang²
Guangneng Hu 4 Horace Ho Shing Ip Sam Kwong
http://gibubb.oomm//peeg222222
Abstract
No forgetting
(b)
Liatt ateer learning the besk)
Iearn ofher twoies
Multimodal Continual Instruction Tuning (MCIT) aims to enable Multimodal Large Language Mod- els (MLLMs) incrementally learn new tasks without catastrophic forgetting. In this paper, we explore forgetting in this context, categorizing it into superficial forgetting and essential forgetting. Superficial forgetting refers to cases where the model's knowledge may not be genuinely lost, but its responses to previous tasks deviate from expected formats due the influence of subse- quent tasks' answer styles, making the results unusable. By contrast, essential forgetting refers to situations where the model provides correctly formatted but factually inaccurate answers, indi- cating a true loss of knowledge. Assessing essen- tial forgetting necessitates addressing superficial forgetting first, as severe superficial forgetting can obscure the model's knowledge state. Hence, we first introduce the Answer Style Diversification (ASD) paradigm, which defines standardized process for transforming data styles across differ- ent tasks, unifying their training sets into similarly diversified styles to prevent superficial forgetting caused by style shifts. Building on this, we pro- pose RegLoRA to mitigate essential forgetting. RegLoRA stabilizes key parameters where prior knowledge is primarily stored by applying regu- larization, enabling the model to retain existing competencies. Experimental results demonstrate that our overall method, SEFE, achieves state-of- the-art performance.
Super for forgelling
A 4/reu
Super ficiall forgetting
(d)
Answor options giver
Essentaal forgetting
(e)
Figure 1. Examples illustrating superfiical forgetting and essential forgetting. (a) Instruction; (h) Response case without forgetting; (c) and (d) Response cases with superficial forgesting; (e) Response case with essential forgetting.
1. Introduction
The rapid advancement of Multimodal Large Language Models (MLLMs) has led to a series of significant works (Liu et al., 2023; Dai et al., 2023; Achiam et al., 2023; Bai et al., 2023b; Liu et al., 2024a; Chen et al., 2024d). Training these models typically involves multiple phases, with pre- training and instruction tuning being two crucial ones. Dur- ing pre-training, large-scale unsupervised lcarning endows the model with foundational cognitive skills. In the subse- quent instruction tuming phase, the model is refined using instruction-response pairs to enhance its ability to compre- hend and respond to user commands. This comprehensive training enables MLLMs to exhibit zero-shot capabilities, allowing them handle previously unseen instructions.
However, these zero-shot abilities are often insufficient for MLLMs to specialize in specific domains. To achieve such specialization, targeted multimodal instruction tuning with relevant datasets is commonly employed. Nevertheless, the breadth of tasks that MLLMs are expected to handle is vast, and the requirements are continuously evolving. As demands grow, simply fine-tuning the model with data from new tasks can result in the loss of previously acquired capabilities phenomenon known catastrophic forget- ting (McCloskey & Cohen, 1989). Alternatively, retraining the model with both new and old task data each time re- quirements change is highly resource-intente To address this, Multimodal Continual Instruction Tuning (MCIT) has
'Equal contribution 'City University of Hong Kong "Shandong University "The Chinese University of Hong Kong "Xidian University 'Lingnan University. Correspondence to: Yuzhi Zhao <yzzhao2-ceee www.yeittuuuuuuuuk....... Sam Kwong <samk- wong @ lingnan.edu.hk>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
emerged (He et al., 2023: Zhu et al., 2024a; Chen et al.,
matting each task's dataset into these five styles (the original and four alternatives). This allows the model to generate re- sponses in multiple styles on the same topic during training, thereby mitigating bias from single-forma training. Our experiments demonstrate that converting only 10% of the dataset into four alternative styles (2.5% each) substantially reduces superficial forgetting and enhances performance.
2024a; Zheng et al., 2024; Zeng et al., 2024). This field seeks to enable MLLMs to progressively learn new tasks without losing proficiency in previously mastered ones dur- ing incremental instruction tuning.
Existing methods in MCIT (He et al., 2023; Zhu et al., 2024a; Zheng et al., 2024), like conventional continual learn- ing research, conceptualize catastrophic forgetting as gen- eralized problem of knowledge loss and have made some progress addressing it. However, our analysis indicates that the nature of forgetting in MCIT extends beyond mere knowledge loss. We propose to categorize it into two types: superficial forgetring and essential foreettilo forgetting. Superficial forgetting refers to cases where the model's response style deviates from expected norms after learning new tasks. For instance, as shown in Fig. 1(a), multiple-choice question task (Lu et al., 2022) may prompt the model to "answer with the option's 's letter". If subsequent tasks demand dif- ferent response formats-such as certain tasks that require word-based answers (Singh et al., 2019) the model may begin answering multiple-choice questions with words in- stead, as shown in Fig. I(c). Similarly, if later tasks involve bounding box-based grounding (Kazemzadeh et al., 2014; Mao et al., 2016). the model might respond to multiple- choice questions with bounding boxes, as illustrated in Fig. 1 (d). On the other hand, essential forgetting refers to the actual loss of knowledge, where the model answers in the correct format but with incorrect content, as depicted in Fig. (e). Accurately evaluating essential forgetting necessitates addressing superficial forgetting, since the latter can ob- scure the model's ' true knowledge state, making it difficult to determine if knowledge has truly been lost. Neverthe- less, current methods fail to recogmize these two types of forgetting or their interrelationship. leading to suboptimal performance when applied to realistic benchmark datasets featuring diverse answer formats (Chen et al., 2024a).
After addressing superficial forgetting, the remaining chal- lenge centers on knowledge loss, namely essential forget- fing. Our analysis of model parameter changes reveals that certain parameters encode significantly more critical his- torical knowledge than others. Based on this insight, we introduce RegLoRA, an extension of LoRA (Hu et al., 2022) designed to minimize knowledge loss. RogL identifies crucial elements within the weight update matrices of prior LoRAs and applies regularization to these elements when training new LoRAs. This process stabilizes the subset of parameters closely tied to prior knowledge, thereby effec- tively preserving acquired proficiency. Furthermore, since the regularized elements constitute only a small fraction of each weight update matrix, and the weight update matrix is the product of parameter matrices A and B rather than the parameters themselves, the constraints on parameter up- dates are minimal. This focused and restrained approach allows the model to preserve previous knowledge while maintaining the flexibility to learn new information.
Building upon ASD and RegLoRA, we introduce a novel MCIT method called the Superficial and Essential Forget- ting Eliminator (SEFE). Our contributions are as follows:
To our knowledge, we are the first to formally define superficial forgetling and essential forgetting in MCTT. Furthermore, our proposed method, SEFE, addresses these challenges and achieves state-of-the-art perfor- mance.
We posit that the main cause of superficial forgetting is the bias introduced by using a single question format per task. When a model is trained over consecutive batches of questions in one specific style, it becomes biased towards that style, making it difficult or even impossible to respond in other styles. To address this, we propose the Answer Style Diversification (ASD) paradigm. Specifically, after examining 15 prevalent MLLM benchmarks (Young al., 2014; Goyal et al., 2017: Gurari et al., 2018; Mishra al., 2019; Singh et al., 2019; Agrawal et al., 2019; Lu et al., 2022; Schwenk et al., 2022: Li et al., 2023c;a; 2024a;b; Liu et al., 2024c; Yu et al., 2024b; Fu et al., 2024), we identify five question types: yes/no questions, multiple-choice ques- tions, short answer questions, brief explanation/ddscrriiion questions, and detailed explanation/ddscriitoo questions. These types also represent the major application scenarios for MLLMs. Based on this observation, we propose refor-
To mitigate superficial forgetting, we introduce the ASD paradigm that unifies the answer domain across tasks by rephrasing questions, thereby reducing the model's bias toward specific response styles. Addition- ally, we create CoIN-ASD, an ASD-adjusted version of the CoIN benchmark, which can serve as a new benchmark for evaluating essential forgetting in future MCIT studies
To address essential forgetting, we present RegLoRA A. By identifying critical elements in the weight update matrices and applying regularization constraints, Re- gLoRA ensures that LoRA fine-tuning does not disrupt the model's existing knowledge.
2

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
2. Related Work
efficient tuning, mitigating catastrophic forgetting and pro- moting knowledge transfer.
2.1. Continual Learning
Most of these approaches, similar to traditional continual learning methods, address catastrophic forgetting as sin- gle challenge. However, this strategy may not be suitable for modern large-scale models, which are inherently larger and entail more intricate training processes. Hence, these models may exhibit forgetting patterns that are distinct from those observed in smaller-scale models studied in traditional continual learning works. To address this, we propose to decompose forgetting in MCIT into two types: superficial forgetiing and essential forgetting. By developing targeted strategies for each type, our method achieves superior per- formance.
Most traditional continual learning methods target straight- forward tasks like image or text classification (Ke & Liu, 2022; Wang et al., 2024a). These methods are com- monly categorized into three types: regularization-based, replay-based, and parameter isolation-based approaches. Regularization-base
methods (Dhar et al., 2019; Douil- lard et al., 2020; Yu et al., 2024e; Luo et al., 2024; Chen et al., 2024c) constrain parameter updates to retain prior knowledge, Replay-based methods (Rebuffi et al., 2017:
de Masson D' Autume et al., 2019; Ostapenko et al., 2019; Chen et al., 2023; Luo et al., 2025; Chen et al., 2025) store or generate some past data and replay it during subsequent training to recall earlier information. Parameter isolation- based methods (Mallya& & Lazebnik, 2018; Geng et al., 2021; Wang et al., 2022) introduce new parameters to capture additional information while keeping existing parameters unchanged, thereby accommodating both past and newly acquired knowledge.
3. Answer Style Diversification Paradigm
In the incremental training process of MCTT, each task typi- cally follows a single question format, which may bias the model toward a uniform response style. Consequently, the model tends to answer all questions in this single manner, creating challenges when handling previously learned tasks with different question formats. This limitation results in what we term superficial forgetting.
The proposed ASD paradigm deviates from these estab- lished categories. Instead, it is a data reconstruction-bet based technique, which we believe represents a promising new paradigm for continual learning in the context of modern large-scale models. On the other hand, the proposed Re- gLoRA can be considered as a regularization-based tech- nique. However, unlike traditional methods that often re- quire auxiliary models significant extra parameters de- ployed on GPUs, RegLoRA is better suited for resource- intensive MLLMs.
To address this issue, we introduce the ASD paradigm. ASD converts each task's 's single-format format instruction-tuning data into a multi-format dataset with five question types: yes/no, multiple-choice, short answer, brief explanation/dessriitionn and detailed explamation//descriiioon. As exemplified in Fig. 2, the original dataset for a task may contain only short answer questions. In the ASD paradigm, we transform %% of the samples equally across the four alternative formats (x % each) and retain the remaining (100- - XX% in their original formats. This configuration ensures that the final dataset includes all five question types without altering the total sample size, thus avoiding extra influencing factors. For these conversions, we establish standardized transfor- mation process, mainly leveraging existing MLLMs, with some direct conversions handled through fixed rules. Ad- ditionally, following (Liu et al., 2024a), we add response format prompt (RFP) for each question type.
2.2. Continual Learning for LLMs and MLLMs
Recently, continual learning research for LLMs and MILMS (Wu et al., 2024b; Shi et al., 2024) has grown significantly. Some methods focus on continual pre-training (Gupta et al., 2023; Wu et al., 2024a; Chen et al., 2024b), while others emphasize continual instruction tuning (Yang et al., 2024; Dou et al., 2024; Yu al., 2024a; Zeng al., 2024; Cao et al., 2024) as in our approach. For example. O-LoRA (Wang et al., 2024b) prevents overwriting prior knowledge by enforcing orthogonality among LoRA weights across tasks. LoTA (Panda et al., 2024) employs a two-stage train- ing process to identify parameters necessitating updates and modifies only these parameters, thus minimizing model changes. EProj (He et al., 2023) introduces task-specific projection layers and leverages task similarities to enable module sharing, reducing forgetting. Model Tailor (Zhu et al., 2024a) identifies sensitive and salient patches within the new model, decorates them, and integrates them with the previous model, thereby minimizing alterations to the existing model. SAPT (Zhao et al., 2024) employs shared attention to align the learning and selection of parameter-
3.1. Question Types and Transformation Process
Yes/No questions are those that can be answered with "Yes" or "No". To convert other question types into this format, we apply the RFP "Is the answer correct? Answer 'Yes' or 'No'." " The updated instruction is structured as "original question a potentially correct answer + RFP". For half of the samples, we include the correct answer, i.e., the original ground-truth (GT) label, as the potentially correct answer and set the new GT label to "Yes". For the remaining half, we generate a plausible but incorrect answer using MLLMs
3

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Original Dataset
Short Question
Question
Answer Question
Question
Question
Adtive 29 usii alli 1i1gg
inteer the qurpiiionsa light tight
di escuer di custion
qustio uiing a dlile unrd
gartic uaing lingle
Answer Style Diversification (ASD) Paradigm
orgafa
Antverth tt's jett Jlnn sing
Kooee bxaninnn
qusttor prov
quriic and pivile
gille
High da dart andltiig
Steas may 31
Short Answer Question
ovnel bald dasddeeee nate of iags whih
Yes/No Question
Brief Explonation Question
Multiple Choice Question
Detailed Explanation Question
Converted Dataset
Figure 2. An example of the ASD paradigm applicd dataset consisting solely of short answer questions. Through the ASD process. (100- - X)% of samples are retained in their original form, while the remaining X% are equally transformed into four alternative formats: yes/no, multiple-choice, brief explanation, and detailed explanation. Additional examples provided in Appendix B.
and set the GT label "No".
and GT label. (2) For multiple-choice questions, the content of the correct option becomes the GT label. (3) For questions with brief or detailed explanations/deccriiiooss we use MLLMs condense the GT label into short answer of no more than ten words.
Multiple-choice questions are characterized by several swer options and utilize the RFP "Answer with the option's 's letter from the given choices directly". To convert other question types into a multiple-choice format, we structure the instruction as "original question + option list + RFP". If the original question is a yes/no question, the option list includes "A. Yes" and "B. No". For other original question types, MILMs used generate three plausible distrac- tor options, which are then shuffled with the correct answer to create four-option list. The GT label is then set to the letter corresponding to the correct option.
Brief explanatioo/descriition questions require answers of about 20 words, presented as concise explanations or descriptions. The instruction structure also "original ques- tion RFP". For original questions with direct answers, e.g., yes/no, multiple-choice, short answer questions, the RFP is "Answer the question and provide brief explanation" The new GT label consists of the direct answer followed by a roughly 20-word explanation generated by MLLMs. For original questions without a direct answer, such as descrip- tive or detailed explanation questions, the original answer is reformulated by MLLMs into an approximately 20-word GT label. The RFP for this scenario is "Answer the question using brief explanationndescriiiior:
Short answer questions correspond to concise and di- rect GT responses, typically consisting of single word or phrase. The RFP for these questions is "Answer the question using single word or phrase". with the modified instruction structured "original question RFP". We con- sider three cases during the conversion process: (1) If the original question is a yes/no question, half of the samples to be converted retain their GT labels of "Yes" or "No", as these can be considered as short answers. To avoid bi- asing the model toward yes/no responses in short answer format, the remaining half transformed using MI.LMs to generate similarly focused questions that require answers other than "Yes" or "No", replacing the original question
Detailed explanation/dessriitioo questions are similar to the brief counterparts but require more comprehensive re- sponses of approximately 50 words. The conversion process mirrors that of the brief explanation/descriiiion questions. The corresponding RFP is "Answer the question and pro- vide detailed explanation " or "Answer the question using detailed explanation/ddscriiiionn
4

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Our review of prevalent MILM benchmarks (Young al., 2014; Goyal et al., 2017; Gurari et al., 2018; Míshra al.,
4.1. Background: LoRA
LoRA (Hu al., 2022) is a widely used parameter-efficient fine-tuning (PEFT) technique designed to match or even surpass the performance of full full-parameter fine-tuning while requiring significantly fewer resources. It achieves this efficiency by modeling the weight updates of the neural network's ' linear layers using low-rank matrices, thereby reducing the number of trainable parameters.
2019; Singh et al., 2019; Agrawal et al., 2019; Lu et al.,
2022; Schwenk et al., 2022; Li et al., 2023c;a; 2024a;b; Liu et al., 2024c; Yu et al., 2024b; Fu et al., 2024) and common MLLM application scenarios suggests that the five question formats described above can adequately address the ma- jority of MCIT contexts. While certain cases may require minor adjustments, we believe that standardized pro- cess provides a strong foundational framework adaptable to various MCIT applications. By applying this framework to data transformation, superficial forgetting can be effectively mitigated, thereby significantly boosting performance. Our experiments demonstrate that transforming as little as 10% of the data (i.e.. X 10) yields substantial improvements. Further details are provided in Appendix B.
Specifically, consider a weight matrix W E R in a linear layer, where d and & denote the input and output dimensions. LoRA introduces two low-rank matrices, A E _ and B E R to model the weight updates AW E R The modified weight matrix during fine-tuning is:
W' = W + AW = W + BxA,
(I)
where < min k) ensures A and B have significantly fewer parameters than W. By updating only A and B while keeping W fixed, LoRA achieves high parameter efficiency.
3.2. CoIN-ASD Benchmark
CoIN (Chen et al., 2024a) is the first public benchmark for MCTT, incorporating eight tasks with diverse question for- mats. Assessing essential forgetting on CoIN is challenging due to the frequent occurrence of superficial forgelting. Al- though CoIN includes evaluations using a large language model (LLM) (Bai et al., 2023a) to measure knowledge ca- pability, which can be useful in some contexts, this approach has limitations in certain extreme cases. For instance, as illustrated in Fig. (d)), the model's response a multiple- choice question is a bounding box, making it impossible for the L.M to assess the knowledge state. To address this limitation, we apply ASD to the CoIN dataset, creating an updated version we term CoIN-ASD. Training models on CoIN-ASD reduces superficial forgetting, allowing future MCIT methods to focus on knowledge retention and es- sential forgetting. In developing CoIN-ASD, we utilized Intern VL.e-2 (OpenGVLaab Team, 2024). CoIN-ASD provides various versions with X values of 10, 20, 40, 60, and 80, facilitating a more comprehensive analysis.
4.2. Key Elements Identification
Since we initialize a LoRA for each task, this LoRA cap- tures the incremental knowledge between the new task and the model's existing capabilities. As shown in Eq. 1, this knowledge is encoded in the model parameters as a weight update matrix, AW B x A. Our analysis of these weight update matrices during incremental training reveals that the updates are not uniformly distributed across all elements. Specifically, some elements have significantly larger abso- lute values, while others are much smaller. For instance, the average absolute value of the top 1% of elements is 580 times greater than that of the bottom 1%. This finding suggests that preserving task memory relies heavily on sta- bilizing the parameters associated with the large elements in the weight update matrix, as these elements more critically capture the incremental knowledge required for the task.
To achieve this, we first identify these key elements in the LoRA's weight update matrix based on their large absolute values. Specifically, after completing the i-th task, we com- pute the weight update matrix AW; for LoRA in each linear layer by AW, B, x A, We then select the top M % of elements based on their absolute values. Once the LoRA is merged into the main model, parameters corresponding to these selected positions undergo notable changes to reflect newly acquired knowledge, Thus, to retain knowledge from the current task during subsequent fine-tuning, it is crucial that the parameters at these positions remain unchanged. To enforce this, we construct regularization mask R, E Rxxd assigning a value of I to these positions and 0 to others. This mask records the positions that should remain unaltered dur- ing subsequent training. By creating these regularization masks, we establish a foundation for preserving essential knowledge in future training.
4. RegLoRA
After addressing superficial forgetting, we shift our focus to essential forgetting. To mitigate this, we introduce Re- gLoRA, a variant of LoRA (Hu et al., 2022) optimized to minimize essential forgetting, as illustrated in Fig. 3. During training with RegLoRA, each task is learned by fine-tuning the model with LoRA. Upon completing a task, we identify the key elements in LoRA's weight update ma- trix that are critical to the knowledge acquired for that task. This LoRA is then merged into the main model, and new LoRA is initialized for the next task. During subsequent training of new LoRAs, regularization is applied to all pre- viously identified key elements, preserving essential prior knowledge.
5

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
for elements under regularization, it is not the parameters but the dot products of associated vectors in B, and A; that are regulated. This permits flexibility across all parameters in B, and Aj, as long as certain dot products stay near zero. These characteristies enable RegLoRA to achieve the dual objectives of retaining prior knowledge and assimilating new information.
By
Top A8%
AL
-6
Weight Updase Matrric CAPs
Key Blements
Regularizz wuriiiioooooooo Mask
Top M%%
A
2
Weight Update Matris
Blements
Regudeebation
5. Experiments
5.1. Benchmark and Evaluation Metries
+0
+U
We conduct evaluations using the public MCIT bench- mark, CoIN (Chen et al., 2024a), which comprises eight widely used vision-language tasks: ScienceQA (SQA) (Lu et al., 2022), TextVQA (VQA Text) (Singh et al., 2019), Im- ageNet (ImgNet) (Deng et al., 2009), GQA (Hudson & Manning, 2019), VizWiz (Gurari et al., 2018), Grounding (Grd) (Kazemzadeh et al., 2014; Mao et al., 2016), VQAv2 (VQA (Goyal et al., 2017), and OCR-VQA (VQA (Mishra et al., 2019). The evaluation framework of CoIN consists of two components: Truth Alignment (TA), which assesses exact matches between answers and GT labels, and Knowledge Capability (KC), which leverages Qwen 1 1,5-32B (Bai et al., 2023a) to evaluate the correctness of knowledge presented in answers. Additionally, we use CoIN-ASD, an ASD-adjusted version of CoIN, to evaluate essential forget- ting in both our method and existing methods. Although KC partially reflects essential forgetting, it does not account for cases where forgetinn fully obscures knowledge retention in answers, e.g., the example in Fig. 1dd). This limitation makes CoIN-ASD a necessary addition to our evaluation. Given the constraints of KC, we prioritize TA in our evaluation, with KC results provided in Appendix E.
+0
+0
Weight Update Matni
Regulerratton Irg
Peguibr
Figure 3. Overview RegLoRA. In each past LoRA, large values in the weight update matrix are identified as key elements. When training a new LoRA, these key positions are incorporated into regularization mask to enforee targeted constraints.
4.3. Regularized Training
In the training of subsequent LoRAs for future tasks, we introduce a regularization loss that utilizes all previous reg- ularization masks ensure effective knowledge retention. Specifically, during the training of the j-th task, the regular- ization loss is defined as:
forgetting
Creg A AW31 @ Ri,
(2)
where À is balancing hyperparameter, @ denotes the Hadamard product, and AW, = B, x A, is the weight update matrix of the new LoRA. During training, Freg is added to the original loss of the base MLLM to form the complete learning objective. It aggregates the positions indi- cated by prior regularization masks. By optimizing this loss, the elements at these positions are encouraged to approach zero, thereby preserving parameters that encode essential prior knowledge. Typically, regularization masks from dif- ferent past tasks rarely assign 1 to the same position, as elements marked by earlier masks tend to zero during the learning of subsequent tasks. Thus, elements at these posi- tions in later LoRAs are less likely to become key elements. In rare cases of overlap, the regularization weight of these elements in Creg accumulates, as defined in Eq. 2, which increases resistance to further updates.
For both TA and KC, we report the accuracy of each learned task after the model has been trained on all tasks. Addition- ally, we present four aggregate metrics: (I) Mean Fine-tune Accuracy (MFT), the average accuracy for each task im- mediately after is learned, representing an upper-bound performance without forgetting. (2) Mean Final Accu- racy (MFN), the average accuracy across all tasks after the complete incremental training sequence, reflecting the model's 's ultimate performance. (3) Mean Average Accu- racy (MAA), the mean of the average accuracies on all learned tasks after each incremental training step, providing comprehensive measure of performance throughout the training process. (4) Backward Transfer (BWT), the dif- ference in accuracy for each task between after all tasks have been learned and immediately after learning that task, as- sessing the degree of forgetting, Formulas for these metrics are provided in Appendix D.
This strategy focuses on the top M% of key elements that embody essential prior knowledge for regularization, effec- tively mitigating essential forgetting. At the same time, most elements remain unrestricted, allowing the model to adapt to new information with minimal limitations. Notably, even
6

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Table Comparison of the proposed SEFE method with existing approachees, evaluated under 'A metrics.
Accuracy on Each Task (%)
Aggregate Results (%)
Method
SQA
VQA Tru
ImgNer
GQA
VizWa
Grd
VQA"
VQ4 RCR
MFT1
MFN1
MAA
BWT1
FFT
2.95
36.38
52.35
46.40
33.90
0.00
61.65
50.00
65.87
35.45
36.73
-30.42
LoRA
54.05
44.63
41.25
47.55
20.80
0.85
59.30
64.30
70.21
41.59
39.53
-28.62
O-LoRA
75.40
52.89
71.85
47.30
37.35
7.10
61.85
61.20
69.30
51.87
49.56
-17.43
LoTA
67.30
41.51
8.25
37.15
42.25
0.10
47.95
56.15
54.72
37.58
50.46
-17.14
FFT+ASD
74.50
50.12
65.40
54.35
45,50
0.00
64.40
68.50
68.28
52.85
57.18
-15.44
LoRA+ASD
74.45
49.70
39.30
$2.00
50.45
7.05
62.25
47.80
68.13
47.88
59.71
-20.66
O-LoRA+ASD
75.20
55.36
67.50
54.70
52.90
15,40
64.45
35.05
65.59
52.57
61.63
-3.02
LoTA+ASD
76.90
42.65
15.85
40.25
45.10
0.30
54.35
54.00
56.99
41.18
56.28
-15.82
SEFE (Ours)
75.35
58.66
83.10
$4.25
48.85
16.75
65.35
66.25
69.02
58.57
63.04
-10.45
5.2. Implementation Details
Table Evaluation of main components in our method.
In line with (Chen et al., 2024a), we utilize the pre-trained, instruction-untuued L.L.V5 (Liu et al., 2024a) as the base model, which incorporates Vicuna-7B (Chiang et al., 2023) as its foundational LLM. Following the official con- figuration L.., the total batch size is set to 128, with learning rates of x 10-4 when using LoRA (Hu et al., 2022) and 2 10 without it, and the LoRA rank parame- ter is set to 128. Other settings also remain consistent with those specified by LLaVA-1.5. For the ASD paradigm, we use the default value of 20 for the hyperparameter X, and data transformations requiring an MILM utilize Intern 26B (OpenGVLab Team, 2024). For RegLoRA, we use default values of 2 for M and 2.5 10 for A. Our exper- iments are conducted node equipped with 8 NVIDIA A800 GPUs.
Aggregate Results (%)
Configuration
MFT1
MFNt
MAA
BWTT
Baseline (LoRA)
70.21
41.59
39.53
-28.62
+ ASD
68.13
47.88
59.71
-20.26
ASD RegLoRA
69.02
58.57
63.04
-10.45
fectiveness in addressing essential forgetting by preserving crucial prior knowledge, which works synergistically with ASD to achieve comprehensive forgetting reduction.
5.4. Ablation Study
In this section, we present ablation studies on SEFE, cov- ering its main components, hyperparameter settings, and design choices. Due space constraints, we only report aggregate metries for these experiments.
5.3. Comparison
To validate the effectiveness of our proposed SEFE, com- pare it against existing approaches, including full-paran hall-parmetter fine-tuning (FFT), LoRA (Hu et al., 2022), O-LoRA (Wang et al., 2024b), and LoTA (Panda et al., 2024). Furthermore, to demonstrate that our ASD paradigm can mitigate super- ficial forgetting across various methods, we apply ASD to these existing approaches, i.e., training them on CoIN-ASD.
5.4. MAIN COMPONENTS
Firstly, we evaluate the impact of the two main components: ASD and Regl RogL. Starting with LoRA as the baseline model, we incrementally incorporate each component. The quantitative results are shown in Table 2, and case studies are presented in Fig. 4.
The TA results presented in Table show that: (1) ASD substantially enhances the performance of all tested meth- ods, with average gains of 7.00%, 14.63%, and 7.27% in aggregate metrics MFN, MAA, and BWT, respectively. We attribute these improvements to ASD's ability to reduce su- perficial forgetting, which we further validate in subsequent experiments. Additionally, by bridging the answer-domain gap. ASD can reduce inter-task differences, which turn lowers the model's update magnitude during new task learn- ing. This mechanism may also indirectly alleviate essential forgetting to slight degree. (2) Our SEFE approach outper- forms all current state-of-thin -hr methods, even when ASD applied to these methods. This highlights RegLoRA's ef-
Compared to the baseline, adding ASD results in significant improvements in three out of the four aggregate metrics: MFN increases by 6.29%, MAA by 20.18%, and BWT by 8.36%. We attribute these gains to a reduction in superficial forgetting, enabling the model to generate responses that reflect its true knowledge instead of being influenced by biased response patterns from recent tasks. This effect is il- lustrated in the case examples in Fig. 4. For instance, in the first two cases, the baseline model incorrectly provides the content of an option when answering a ScienceQA question (requiring only the option's 's letter), due to interference from the recently trained OCR- VQA task, which requires word or phrase responses. In case 3, after ImageNet training, the

---

SEFE; Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Case
Case 2
Case
Case 4
Case
ANEE TARLY
Contect Uisth paggg to anteent surrion belee
predatation Segupom!
pieers number andigrees
A gloe
6 eagars Castember andocsber
CUOPTIAL
002 todes CoNts,
Which humttrre beppel Arseeer qaet on ssigg a plasss
Pleae pioedd 20t 1dii
with flors letee teesteegeenceeee
Andeer wth the optins leter fon 2e gron
Af querior sing, siglle
sersecce gell pik slaat formet
Supertooni
Aug
Sepertiri
right
Superticoi
(+)
Desentte
2each
(4)
Tade ScieeonD4 paee 13
Take Srieecth ttwak 1]
Tadec Teutvits gook Model Stage Leamed tadke lavn leamed Inegelee,
Twil 02A (lack 4]
Thile firunding (ook-A)
(6)
Model Stage, Leamed taite Jae
Madal Stager loamed it tador (ad leomed twe CIR-
Modal Stage: Loomed & tador (Juct learned
Model Stage: Laaned tadee (jet tearned
Figure 4. Case studies of main components in the proposed SEFE method. (a) Instruction; (b) Response from the haseline model (LoRA); (c) Response from the baseline model with ASD added; (d) Response from the baseline model with both ASD and RegL.RRA added; (e) Basic information of the case. Additional case studies can be found in Appendix H.
baseline model mistakenly answers a Tex question with "Maillot", category from ImageNet, instead of the required number. In case 4, after learning the Grounding task, the model incorrectly responds to a GQA task with a bounding box instead of a word. Similarly, in case 5, it responds to a Grounding question with a word instead of the required bounding box after learning the word-based VQAv2 task. These examples highlight the superficial for- getting observed in the baseline model. After integrating ASD, the model consistently produces responses that match the task requirements, even if the answers are not always entirely correct. This confirms ASD's 's effectiveness in miti- gating superficial forgetting and facilitating a more accurate assessment of the model's knowledge.
Table 3. Comparison of data transformation proportions in ASD.
Aggregate Results (%)
X
MFTt
MFN1
MAA1
BWTt
0
70.21
41.59
39.53
-28.62
10
65.82
46.93
58.61
-18,89
20
68.13
47.88
59.71
-20.26
40
70.09
47.80
59.22
-22.29
60
66.06
47.45
58.37
-18.60
80
63.93
46.85
58.64
-17.08
Table 4. Comparison of regularized element proportions in Re- gLoRA.
Aggregate Results (%)
M
Building on ASD, we introduce RegLoRA address essen- tial forgetting- As shown in Table 2, model performance further improves, with MFN, MAA, and BWT increasing by 10.69%, 3.33%, and 9.81 %, respectively. This improvement indicates that RegLoRA effectively preserves the model's prior knowledge, allowing it to handle previously learned tasks more robustly. Cases 2 and 3 in Fig. 4 illustrate that the RegLoRA A-BRA-ente -enhanced model corrects the baseline model's erroneous answers. In case 5, the model's bounding box response improves its IoU with the GT from 0.49 to 0.90 after integrating Rog, further confirming Rog'' role in enhancing response accuracy.
MFT1
MFN1
MAA1
BWTt
0.5
67.18
55.76
61.53
-11.43
1
68.62
58.71
62.38
-9.91
2
69.02
58.57
63.04
-10.45
5
66.58
$6.29
61.69
-10.29
100
67.73
54.66
61.54
-13.07
formation proportion of 10% significantly improves per- formance, indicating that introducing alternative response styles, even minimally, effectively reduces the model's 's bias toward single style. Based on all four evaluation metrics, we select X 20 the default value, as it achieves optimal results in the key MFN and MAA metrics.
5.4.2. DATA TRANSFORMATION PROPORTION IN ASD
This section investigates the effect of varying the proportion of data transformations in ASD by adjusting the hyperparam- eter X. The results are presented in Table 3. The baseline condition (X- = 0), where no transformation is applied, is shown in the first row. We observe that even small trans-
5.4.3. REGULARIZED ELEMENT PROPORTION IN REGLORA
In this section, we analyze the impact of varying the pro- portion of regularized elements, iee, the hyperparameter M,
8

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
forgetting by diversifying question types within single task, thereby preventing response style bias, enhancing per- formance, and enabling reliable assessment of knowledge state. Using ASD, we processed the CoIN benchmark to create the new CoIN-ASD benchmark, allowing future meth- ods to evaluate knowledge retention without interference from response style biases. To address exsential forgetting, RegLoRA identifies critical elements in the weight update matrix of past LoRAs and applies regularization loss to minimize corresponding elements in future LoRAs. Exper- imental results demonstrate the effectiveness of ASD and RegLoRA, as well as the state-of-the performance of SEFE as a comprehensive solution.
Table Comparison of regularization targets in Regl.oRA.
Aggregate Results (%)
Regul. Tgt.
MFTT
MFN1
MAA
BWTT
A B
68.43
47.20
59.49
-21.23
67.52
47.84
59.35
-1,669
A & B
68.82
48.47
59.41
-20.35
AW (Ours)
69.02
58.57
63.04
-10.45
on the performance of RegLoRA. The results shown in Table 4. Our findings indicate that performance peaks when M = 2, suggesting that during LoRA fine-tuning, approximately 2% of updated parameters significantly in- fluence task performance. When M is set lower, the model struggles to retain knowledge from previous tasks, leading to performance decrease. This insufficient knowledge re- tention may also limit initial performance on new tasks that closely relate to previous ones, which likely explains why MFT for M = 0.5 and M = 1 is lower than for M = 2. Conversely, setting M too high overly restricts all parameter updates, impairing the model's ability to learn new infor- mation. Moreover, when regularization is applied to less critical elements, the energy allocated in the loss function for crucial elements decreases, adversely affecting the model's ability to retain prior knowledge. Hence, the performance for M = 5 and M 100 is suboptimal. Based on these observations, we select M = 2 as the default setting.
A Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants 62471278 and 62306220, in part by the Research Grants Council of the Hong Kong Special Administrative Region, China under Grant STG5/E-103/24-R, and in part by the Taishan Scholar Project of Shandong Province under Grant tsqn202306079.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
5.4.4. REGULARIZATION TARGET IN REGLORA
In Table 5, we validate our decision to apply regulariza- tion to the top M% elements of the weight update matrix, AW. The table compares this approach to the effects of regularizing the top M% clements in matrix A, matrix B, and in both matrices A and B together. The results show that regularizing AW yields the best performance. Since AW directly reflects model updates, whereas the other three options are indirect, regularizing AW is more effective in preserving prior knowledge. Additionally, because AW is not a model parameter, this choice leaves more capacity for parameter updates to accommodate new knowledge.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., John- son, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. nocaps: Novel object captioning at scale. In ICCV, pp. 2019.
Anthropic. The Claude 3 model family: Opus, sonnet, haiku. https:// httpss//www.ccch..aatnrrggcc..oo
6. Conclusion
This paper provides an in-depth analysis of catastrophic forgetting MCIT, categorizing it into superficial forget- ting and essential forgetting. Superficial forgetting refers to the model's response style becoming biased by subse- quent tasks, causing deviations from the required format of previous tasks. On the other hand, essential forget- ting indicates the loss of knowledge, resulting in factual errors in responses. To address these issues, we propose the SEFE method, which introduces two components: the ASD paradigm and Rog. ASD mitigates superficial
Model caabel_Card_Claaddd
pdf,, 2024.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. QWen-VL: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023b.

---

SEFE; Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Cao, M., Liu, Y., Liu, Y., Wang. T., Dong, J., Ding, H., Zhang, X., Reid, I., and Liang, X. Continual LLaVA: Con- tinual instruction tuning in large vision-language models. arXiv preprint arXiv:2411.022
Alleviating world knowledge forgetting in large language models via moe-style plugin. In ACI., pp. 1932-1945, 2024.
2024.
Douillard, A., Cord, M., Ollion, C., Robert, T., and Valle, E. PODNet: Pooled outputs distillation for small-tasks incremental learning, In ECCV, pp. 86-102, 2020.
Chen, C., Zhu, J., Luo, X., Shen, H... Gao, L., and Song, CoIN: A benchmark of continual instruction tuning for multimodel large language model. In NeurlPS, 2024a.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang. M., Lin, X., Yang. J., Zheng, X., Li, K., Sun, X., Wu, Y. and Ji, R. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprinl arXiv:2306. 133444444 2024.
Chen, J., Cong, R., Luo, Y., Ip, H., and Kwong, S. Saving 100x storage: Prototype replay for reconstructing training sample distribution in class-incremental semantic seg- mentation. In NeurIPS, 2023.
Chen, J., Chen, Z., Wang, J., Zhou, K., Zhu, Y., Jiang, J., Min, Y., Zhao, W. X., Dou, Z., Mao, J., et al. Towards effective and efficient continual pre-training of large lan- guage models. arXiv preprint arXiv:2407. 18743, 2024b.
Gemini Team. Gemini: a family of highly capable multi- modal models. arXiv preprint arXiv:2312.1 1800, 2023.
Geng, B., Yuan, F., Xu, Q.. Shen, Y., Xu, R., and Yang. M. Continual learning for task-oriented dialogue system with iterative network pruning, expanding and masking. In ACL-IJCNLP, pp. 517-523, 2021.
Chen, J., Cong, R., Luo, Y., Ip, H. H. S., and Kwong, S. Strike a balance in continual panoptic segmentation. In ECCV, 2024c.
Chen, J., Cong, R., Luo, Y., Ip. H. H. S., and Kwong, Re- play without saving: Prototype derivation and distribution rebalance for class-incremental semantic segmentation. 2025.
Goyal, Y., Khot, T., Summers- -tay, D., Batra, D., and Parikh, D. Making the v VQA matter: Elevating the role of image understanding visual question answering. In CVPR, pp. 6904-6913. 2017.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q.. Zhu, X., Lu, L., et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguiste tasks. In CVPR, pp. 24185- 24198, 2024d.
Gupta, K., Thérien, B., Ibrahim, A., Richter, M. L., Anthony, Q., Belilovsky, E., Rish, I,, and Lesort, T. Continual pre- training of large language models: How to (re)warm your model? In ICLR, 2023.
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. VizWiz grand challenge: Answering visual questions from blind people. In CVPR, pp. 3608-3617, 2018.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., al. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality. 2023.
Hayou, S., Ghosh, N., and Yu, B. LoRA+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402. 12354, 2024.
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P. N., and Hoi, S. InstructBLIP: Towards general- purpose lisiunlanguaage models with instruction tuning. arXiv preprint ar 2055:2300.005500 2023.
He, J., Guo, H., Tang, M., and Wang. J. Continual instruc- tion tuning for large multimodal models. arXiv preprint arXiv:2311.16200 2023.
de Masson D' Autume, C., Ruder, S., Kong, L.,, and Yo- gatama, D. Episodic memory in lifelong language learn- ing. NeurIPS, 2019.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.,, and Chen, W. LoRA: Low-rank adaptation large language models. In ICLR, 2022.
Deng, J., Dong, W., Socher, R., Li, L.JJ,, Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Hudson, D. A. and Manning, C. D. GQA: A new dataset for real-world visual reasoning and compositional question answering. CVPR, pp. 6700-6709, 2019.
Dhar, P. Singh, R. V., Peng, K.-C., Wu, Z., and Chellappa, R. Learning without memorizing. In CVPR, pp. 5138- 5146, 2019.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. ReferltGame: Referring to objects in photographs of natural scenes. EMNLP, pp. 787-798, 2014.
Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, L., Zhou, Y., Wang, X., Xi, Z., Fan, X., et LoRAMoE:
10

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Ke, Z. and Liu, B. Continual learning of natural lan- guage processing tasks: A survey. arXiv preprint arXiv:2211.2271,, 2022.
Luo, Y., Chen, J., Cong, R., Ip. H. H. S., and Kwong, S. Trace back and go ahead: Completing partial annotation for continual semantic segmentation. PR, pp. 111613, 2025.
Li, B., Ge, Y., Ge, Y., Wang, G., Wang. R., Zhang, R., and Shan, Y.
Benchmarking multimodal large language models. arXiv preprint arXiv:2311.1792. 2023a.
Mallya, A. and Lazebnik, S. PackNet: Adding multiple tasks to single network by iterative pruning. In CVPR, pp. 7765-7773, 2018.
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A. L., and Murphy, K. Generation and comprehension of unam- biguous object descriptions. In CVPR, pp. 11-20, 2016.
Li, B., Ge, Y., Chen, Y., Ge, Y., Zhang, R., and Shan, Y. BeEDD-Becc--2Plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790 2024a.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165, 1989.
Li, B., Wang, R., Wang, G., Ge, Y. Ge, Y., and Shan, Y. SEED-Bence Benchmarking multimodal ilms with generative comprehension. In CVPR, pp. 13299-13308, 2024b.
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. OCR-VQU VQR- Visual question answering by reading text in images. ICDAR, pp. 947-952, 2019.
Li, H., Ding, L., Fang, M., and Tao, D. Revisiting catas- trophic forgetting in large language model tuning. arXiv preprint arXiv:2406.cm 04836, 2024c.
OpenGVLab Team. InterVVL2. Better than the expanding performance boundaries of open-source multimodal models with the progressive scaling strat- egy. attps://internvll .gw.eniww..iiiiii.. gii..gitt wtt..ioob.l.......e io/blog/ 2024 1114-07--2-IIneznti Inte-nnneenVV222- 2. 2024.
Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Boot- strapping language-iman pre- rre-training with frozen image encoders and large language models. In ICMI., 2023b.
Ostapenko, O., Puscas, M., Klein, T., Jahnichen, P., and Nabi, M. Learning remember: synaptic plasticity driven framework for continual learning. In CVPR, pp.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.00355, 2023c.
11321-11329, 2019.
Panda, A., Isik, B., Qi, X., Koyejo, S., Weissman, T., and Mittal, P. Lottery ticket adaptation: Mitigating destructive interference in 1lms. arXiv preprint arXiv:2406.16797, 2024.
Liu, H., Li, C., Wu, Q. and Lee, Y. J. Visual instruction tuning. In NeurlPS, 2023.
Liu, H., Li, C., Li, and Lee, J. Improved baselines with visual instruction tuning. In CVPR, pp. 16296-26306 2024a.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. iCaRL: Incremental classifier and representation learning. In CVPR, pp. 2001-2010, 2017.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. DoRA: Weight- decomposed low-rank adaptation. In ICML, 2024b.
Schwenk, D., Khandelwal, A., Clark, Marino, K., and Mottaghi, R. A-OKVQA: A benchmark for visual ques- tion answering using world knowledge. In ECCV, pp. 146-162, 2022.
Liu, Y., Duan, H., Zhang. Y., Li, B., Zhang. S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. MMBench: Is your multi-modal model an all-around player? In ECCV, pp. 216-233, 2024c.
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang. Y., Wang, Z., Ebrahimi, S., and Wang, H. Continual learning of large language models: A comprehensive survey. arXiv preprint arXiv:2404.100 16789, 2024.
Lu, P. Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, 2022.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards VQA models that can read. In CVPR, pp. 8317-8326, 2019.
Luo, Y., Cong, R., Liu, X., Ip, H. H. S., and Kwong, S. Modeling inner-and cross-tate task contrastive relations for continual image classification. IEEE TMM, 2024.
Wang, L.. Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey continual learning: theory, method and application. IEEE TPAMI, 2024a.

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Wang, X., Chen, T., Ge, Q., Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., and Huang, X. Orthogonal sub- space learning for language model continual learning. In EMNLP, pp. 10658-1061 10671, 2024b.
framework for parameter-efficient continual learning of large language models. In ACI., pp. 11641-11661, 2024.
Zheng, J., Ma. Q. Liu, Z., Wu, B., and Feng, H. Be- yond anti forgettin:::: Multimodal continual instruction tuning with positive forward transfer. arXiv preprint arXiv:2401.09/88 2024.
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In CVPR, pp. 139-149, 2022.
Zheng, J., Cai, X., Qiu, S., and Ma, Q. Spurious forgetting in continual learning of language models. arXiv preprint arXiv:2501 250..:5301..3433. 2025.
Wu, C., Gan, Y., Ge, Y., Lu, Z., Wang, J., Feng, Y., Shan, Y., and Luo, P. LLaMA PRO: Progressive llama with block expansion. In ACL. pp. 6518-6531 2024a.
Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Wu, C., and Kuang, K. Model tailor: Mitigating catastrophic for- getting in multi-modal large language models. In ICMI., 2024a.
Wu, T., Luo, L., Li, Y.-F., Pan, S., Vu, T.-T. and Haffari, G. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01
2024b.
Zhu, Z., Wu, Y., Gu, Q., and Cevher, V. Imbalance- regularized LoRA: A plug-and-play method for improv- ing fine-tuning of foundation models. NeurIPSW. 2024b.
Yang. S., Ali, M. A., Wang, C.-L., Hu, L., and Wang, D. MoRAL: Moe augmented lora for Ilms' lifelong learning. arXiv preprint 1xXi::2402.11266 2024.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78, 2014.
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. Language mod- els are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024a.
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L.. MM-Vet: Evaluating large multimodal models for integrated capabilities. In JCML, 2024b.
Yu, X., Fan, C., Pan, J., Xiang, G., Chen, C., Yu, T., Peng, Y., and Deng, H. X-ray security inspection for real-world rail transit hubs: a wide-ranging dataset and detection model with incremental learning block. TVC, pp. 1-13, 2024c.
Zeng, F., Zhu, F., Guo, H., Zhang. X.-Y., and Liu, C.-L. ModalPrompt: Dual-modality guided prompt for contin- ual learning of large multimodal models. arXiv preprint orXiv:2410.05844 2024.
Zhai, Y., Tong, Li. X., Cai. M.. Qu, Q., Lee, Y. and Ma, Y. Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309. 10313, 2023.
Zhang, Q. Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. AdaLoRA: Adaptive budget allocation for parfmeter-effiiient fine- tuning. arXiv preprint 2303 230.::23033/011111 105 2023.
Zhao, W., Wang, S., Hu, Y., Zhao, Y., Qin, B., Zhang, X., Yang, Q., Xu, D., and Che, W. SAPT: A shared attention
12

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
A. Additional Related Work
A.I. Multimodal Large Language Model
Multimodal Large Language Models (MLLMs) have become one of the most prominent research areas today. GPT-4/4o (Achiam et al., 2023) is the most representative work in this area, demonstrating exceptional performance across wide range of multimodal tasks. Subsequent models, such Gemini (Gemini Team, 2023) and Claude (A (Anthropic, 2024), also showcase impressive performance, offering user experiences that rival even exceed those of GPT-44/4o.
In the open-source community, several noteworthy models have also emerged. For example, InstructBLIP (Dai et al., 2023) builds upon the BLIP2 (Li et al., 2023b) pretraining model and incorporates instruction tuning to improve the model's ability to understand user commands and process text-image queries. LLavA (Liu et al., 2023) adopts a simple yet efficient architecture and introduces an effective method for generating visual instruction tuning data using large language models (LLMs), achieving strong performance. LLa (Liu et al., 2024a), an enhanced version of LLaVA, refines design choices such the vision-language connector, resulting in further improvements in performance. Qwen'LL (Bai et al.. 2023b) proposes three-stage training paradigm, using images with varying resolutions at different stages of training, optimizing both image detail comprehension and training efficiency. In (Chen et al., 2024d) focuses on scaling up the visual encoder's parameter size and employs different three-stage training strategy to align the large visual encoder with the text encoder, enhancing the model's visual understanding capabilities.
A.2. LoRA Variants
In addition to the previously discussed O-LoRA (Wang et al., 2024b), several LoRA variants have been proposed from perspectives beyond mitigating catastrophic forgetting. For example, AdA (Zhang et al., 2023) adaptively allocates the parameter budget across weight matrices based on their importance scores. LoRA+ (Hayou et al., 2024) assigns different learning rates to matrices A and to improve performance in settings with large embedding dimensions. DoRA (Liu et al., 2024b) decomposes pretrained weights into magnitude and direction components, applying LoRA solely to the direction component reduce the number of trainable parameters. Imbalance Regularie- LoRA (Zhu al., 2024b), which shares similar name with our proposed RegLoRA, introduces a regularization term |AA T - B B 2 the forward pass and enhance downstream task performance. Overall, these approaches are fundamentally different from our RegLoRA in terms of methodology and objectives.
1e to improve stability during
A.3. Understanding Catastrophic Forgetting
Several existing works also aim to understand the phenomenon of catastrophic forgetting. For instance, (Li et al., 2024c) shows that catastrophic forgetting during LLM fine-tuning becomes more pronounced as the loss landscape sharpens, suggesting strong positive correlation between sharpness and forgetting. (Zhai et al., 2023) argues that in MLLMs, catastrophic forgetting arises as fine-tuning shifts the model's focus from general visual-text alignment to datasct-specific overfitting, resulting in performance degradation even when the vision encoder is frozen. Among these studies, (Zheng et al., 2025) is most relevant to ours. It introduces the concept of spurious forgetling, where the model loses task alignment without any genuine loss of knowledge. This notion is partially similar with our definition of superficial forgetting. forgetting. However, spurious forgetting emphasizes the recoverability and assumes that no actual knowledge has been forgotten. In contrast, superficial forgetting does not make assumptions about recoverability or knowledge retention.
B. Details of ASD
B.1. Transformation Rules
Table 6 details the transformation rules in the ASD paradigm, providing guidelines for converting various source formats into specific target formats: yes/no questions (Y/N), multiple-choice questions (MCQ), short answer questions (short), brief explanatioon/descriiiion questions (brief), and detailed explanation//dsscriiion questions (detail). These rules specify the Response Format Prompt (RFP) and methods for rewriting both the instruction and ground- (GT) labels. The outlined transformations serve as general guidelines for typical scenarios but may require minor adjustments for specific applications. Fig. 5 illustrates several transformation examples, demonstrating the conversion of five original question types into the other four formats, thereby offering clearer understanding of the transformation process in ASD.
13

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Use "Yos" in sample where the potentially correct correct; use the
Use the letler corespunding the correct option.
When the question is the original question, use the orgiiall "Yes" or "No",, When the ques- lion reweiten by MLI MLL.MM L.M. the unswer generated together with the question.
Structured as "direct unswer explanation The ditect answer is the original GT label, the explanation is ahout 20 words generated by MI M.M.M.
Structured "direct answer explonation" The direct directasswe answer is the original GTT label, the explunation is MLLMMs
Adjust original GT label approximately 50 words by MLLMMs
Adjusi the original GT label to approxximtee 20 by MLLMs.
Shorten original label to within words by MLLMS
generated
Use content correct opsion.
GT Label
others.
When an incorrec answer is required, gener- arc plausible but incorrect distractor using ML LMs.
The option list includes options: "A. Yee" and "B. Noo..
The option lis inclide four andomly ordered options. one being the original GT label and the remaining three heing plausible but incur- nex distructors gesented by MLLMs
For half of the sumples, the question is the original question; the other rewrie the ociginal question by MLLM LM into simi- larly focused questions that require answers other than "Yex" "No"
When an incoree answer is regaired domly select one of the incomeet options.
original question.
Table 6. Specitic Transformation Rules of ASD.
Structured as "question REP..
Instruction
Streetuedd as 'question + potemtiall correct answer RFP". The question is the original question. For half of the sumples, the diall coneer answer is the correct ansseer (te.. original GT tabel); an incoerrect answer.
Sirocuued as "question option list RFP". The question is the original question, and an option list needs to comstructe..
Structured as "quesion + REP".
Is the correct? Answer YYe" or 'No..
Answer with the option's lettrr from the given choices direcll..
Answer question using a single word or phnne.
Answer the question and provide de- tailed ppplantion.
Answer the questiio using demiled applanationdeesriieee
Answer the quession and provide brief explanation.
Answer question using celaantiionescciiisiiiio
RFP
Source
MCQ
Short
Brief
Detail
YAN
Brief
YA
MCQ
Brief
Detail
YIN
MCQ
Short
Detai
YN
MCQ
Short
Brief
Target
Y/N
MCQ
Short
Brief
Detal
14

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Mustol Dieetion
plicn Qacalinn
Deta lnd Description Quertiin
Stort Ansoer Quectoon
To Qualion
diper quen and
Art prochatier
powchan herehan or uup dip sen
For th plan delen druciy
anting with uniie figs
Detalled Descr cererrptii Dueetion
Muttple Quetioon
Emet Descrpttin Quettor
Question
Shart AneARe Quedion
th trage!
(b)
Hire
Nroven
Et's chaieiencoo Bindy.
argin an arilin oplar
Nultblle Chilce Question
trid Dox biscriiion Quosiien
Detailed
Short Anntorr Queeton
Yechio Quection
nying
part Anrvel pnttor and anside
(c)
wneer beacial
Radba: The skyeet are Eitret
Mulpple Cho on Queetion
Deserption
Drter Desc option Question
Shart Arower Quanticss
happeting inapee Neow qperion
(d)
desciptios
glasss
Description
Yestie Queetion
Mobppe cs Question
lapueiin thi suget Anewy ginii
Aupgrnde in His up gustice
(e)
made yery
ling cots
happeerr gir image" Anarer dhe aing
Figure 5. Illustrative transformations of the ASD paradigm: (a) Conversion of yes/no question into four other question types; (b) Conversion of a multiple-choiee question into four other question types: (c) Conversion of a short-answer question into four other question types: (d) Conversion of brief description question into four other question types; (e) Conversion of a detailed description question into four other question types.
15

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
B.2. Transformation Details of CoIN-ASD
The CoIN-ASD benchmark is modified version of the CoIN benchmark (Chen et al., 2024a), tailored using the ASD paradigm to assess essential forgetting. Like CoIN, CoIN-ASD comprises eight tasks in sequence: ScienceQA (Lu et al., 2022), TextVQA (Singh al., 2019), ImageNet (Deng al., 2009), GQA (Hudson & Manning, 2019), VizWiz (Gurari et al., 2018), Grounding (Kazemzadeh et al., 2014; Mao et al., 2016), VQAv2 (Goyal et al., 2017), and OCR- VQA (Mishra et al., 2019). In constructing the CoIN-ASD dataset, we primarily adhere to the transformation rules outlined in Sec. B.I, with minor modifications in three specific instances:
1. ScienceQA questions to short answer format:
ScienceQA questions (Lu et al., 2022) are initially MCQs, where some samples require the provided options to determine the correct answer. For example, questions such as "Which state among the options is the northernmost?" require explicit options to identify the correct response. To accommodate this, we include reformatted option list when converting ScienceQA questions into short answer ones, structured as "[Option 1J. [Option ...... or [Option NJ". Additionally, the RFP is adjusted to more natural one: "Directly give the answer."
2. ImageNet and Grounding questions to Y/N MCQ: In these cases, employing MLLM for distractor generation is unnecessary. For ImageNet questions, we randomly select distractors from the class list. For Grounding questions, we randomly generate bounding boxes with an Intersection-one -over-ionn-ver--nniin (IoU) less than 0.5 compared to the GT bounding box. For other tasks with similarly constrained answer spaces, reformulating them as Y/N or MCQ can also eliminate the need for MLLM-based distractor generation using similar strategies.
3. Grounding questions to brief or detailed format: Instead of using response format that combines a direct answer with an explanation, we reformulate the questions to request descriptions the content within the given bounding box, which more natural.
All other transformations strictly follow the rules defined in Sec. B.1. Further details on the prompts and transformed data examples are available in our code repository.
C. Loss Function
The overall loss function of the model is defined as:
Ciotel Lim + Lreg:
(3)
where Crey is the regularization loss defined in Eq. 2. and fim is the standard language modeling loss, i.e., the next-token prediction loss, given by:
Lim Σ log P(x: iisttruct <is Xa,<i),
(4)
where T. denotes the length of the answer, X, represents the visual input tokens, and X Xinstruct and X Na,<i denote the instruction and answer tokens preceding the current token I;, respectively.
Specifically, during training on the first task, there is no prior knowledge to preserve, so the objective reduces to the language modeling loss alone. For all subsequent tasks, the objective corresponds to the complete loss defined in Eq.
D. Definitions and Formulas of Metrics
In this section, we present the definitions and mathematical expressions of the four aggregate metrics employed in our evaluation: Mean Fine-tune Accuracy (MFT), Mean Final Accuracy (MFN), Mean Average Accuracy (MAA), and Backward Transfer (BWT).
MFT represents the average accuracy of each task immediately after it is learned. This metric serves as an upper-bound performance indicator, indicating the model's performance without any forgetting. is calculated as:
T MFT
Σ
A
(5)
T
i=1
16

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Table 7. Comparison of the proposed SEFE method with existing approaches, evaluated under KC metrics.
Accuracy on Each Task (%)
Aggregate Results (%)
Method
SQA
VQA Tru
ImgNer
GQA
VizWa
Grd
VQA"
VQ4 RCR
MFT
MFN1
MAA
BWT1
FFT
73,46
57.61
81.79
59.30
45.97
77.75
71.84
70.64
75.94
67.29
68.06
-8.65
LoRA
74.11
62.59
64.21
61.50
38.14
43.32
69.64
80.47
78.49
61.75
68.34
-16.74
O-LoRA
84,45
67.50
84.67
60.40
52.41
60.85
70.97
78.28
77.86
69.94
71.87
-7.91
LoTA
76.54
62.93
33.28
54.31
57.12
19.31
61.73
76.30
67.84
55.19
64.42
-12.66
FFT+ASD
80.25
67.57
81.88
66.09
59.73
9.10
74.43
85.04
76.17
65.51
71.44
-10.66
LoRA+ASD
80.09
66.13
72.39
64.46
66.89
59.40
72.06
75.29
77.29
69.59
73.36
-7.71
O-LoRA+ASD
80.82
69.43
86.37
66.87
67.33
59.34
73.59
67.43
75.94
71.40
75.07
.4.4
LoTA+ASD
82,49
63.01
44.75
56.99
58.42
36,98
67.53
76.42
70.38
60.82
70.12
-9.56
SEFE (Ours)
80.79
70.59
89.05
66.44
63.81
57.90
74.41
83.36
77.98
73.29
75.61
.4.69
Table 8. Evaluation main components method, evaluated under KC metrics.
Aggregate Results (%)
Configuration
MFTT
MFNt
MAAt
BWT1
Baseline (LoRA)
78.49
61.75
68.34
-16.74
ASD
77.29
69.59
73.36
-7.71
+ ASD RegLoRA
77.98
73.29
75.61
-4.69
where 7' is the total number of tasks, and Ai,; denotes the accuracy on task i immediately after learning it.
MFN measures the average accuracy across all tasks after the model has completed the entire incremental training sequence. This metric reflects the model's final performance after all tasks have been learned. It is defined as:
MFN
Σ AT. T
1
(6)
where Ar, represents the accuracy on task after learning all tasks.
MAA provides comprehensive measure of the model's performance throughout the entire training process. It is the mean of the average accuracies all learned tasks after each incremental training step. The formula for MAA is:
MAA
Σ
Ag.).
T
where A denotes the accuracy on task after the model has been trained on the first j tasks.
BWT assesses the extent of forgetting by measuring the difference accuracy for each task between the final training step and immediately after the task was learned. It is calculated as:
BWT (AT, At,))
(8)
negative BWT value indicates forgetting, with larger negative values implying greater forgetting.
E. Knowledge Capability
In this section, we present the evaluation results based on the Knowledge Capability (KC) metrics. As defined in the ColN benchmark (Chen et al., 2024a), KC metrics involve utilizing Qwen1 Qwwen1.5-3BB reflected in model responses.
1.532B (Bai et al., 2023a) to assess the knowledge
Table 7 compares our method with existing approaches using KC metrics. Additionally, it includes results incorporating the ASD paradigm into existing methods. This table parallels the Truth Alignment (TA) metrics reported in Table The
17

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Table 9. Comparison of data transformation proportions in ASD, evaluated under KC metries.
Aggregate Results (%)
X
MFTT
MFN1
MAAT
BWTT
78.49
61.75
68.34
-16.74
10
75.62
68.18
72.24
-7.44
20
77.29
69.59
73.36
-7.71
10
78.30
66.85
72.41
-11.45
60
75.63
67.77
71.97
-7.86
74.17
67.59
72.02
-6.58
Table 10. Comparison of regularized element proportions in RegLoRA, evaluated under KC metrics.
Aggregate Results ( (%)
MFT1
MFN
MAAt
BWT1
76.89
73.26
74.79
-3.63
77.74
74.06
75.29
-3.68
77.98
73.29
75.61
-4.69
76.79
73.09
74.90
-3.70
100
77.41
72.09
74.80
-5.32
observed trends in KC metrics align closely with those in TA metrics, First, integrating ASD improves performance across various methods. Although KC metrics primarily assess knowledge rather than instruction-folloiing unaffected by superficial
aspect theoretically forgettin forgetling-severt superficial forgetling can obscure a model's true knowledge capabilities (as illustrated in Fig. 1 and Fig. 4). This explains the enhancement in KC metrics achieved with ASD. Second, even with ASD integration, existing methods still underperform compared to our SEFE approach, further demonstrating the effectiveness of RegLoRA in mitigating essential forgetting.
Table 8 evaluates the main components of our method. Table 9 explores the impact of varying data transformation proportions in ASD, i.e., hyperparameter X. Table 10 investigates the effect of different proportions of regularization elements in RegLoRA, i.e., hyperparameter M. Finally, Table 11 examines the influence of different regularization targets in RegLoRA. These results correspond to the TA results presented in Tables 2, 3, 4, and 5 of the main text, and similar trends are observed both KC and TA metrics. Specifically, Table 8 confirms the substantial advantage of combining both core components. Table identifies 20% data transformation ASD as optimal. Finally, Tables 10 and 11 demonstrate that focusing regularization on the top 2% of elements in the weight update matrix AW yields the best performance in RegLoRA.
While the KC results generally align with the trends observed in the TA results and appear reasonable overall, certain failure cases were identified during evaluation. For example, when model's response is "Yes", Qwenl 1--32B (Bai et al., 2023a) often assigns high score to it, even when this answer is highly unreliable or entirely irrelevant to the question. These issues may occur at a similar rate across different experimental settings, potentially mitigating their impact on the overall trends. Nevertheless, due these limitations, we consider the TA results to the most reliable, with the KC results serving as a supplementary reference.
Table 11. Comparison of regularization targets in RegLoRA, evaluated under KC metrics.
Aggregate Results (%
Regul. Tgt.
MFT1
MFN
MAA1
BWT1
A
77.84
68.83
73.30
-9.01
77.33
69.55
73.09
-7.78
A & B
77.75
69.32
73.08
-8.43
AW (Ours)
77.98
73.29
75.61
-4.69
18

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Table 12. Comparison of different ML..Ms for data transformation in ASD, evaluated under TA and KC metrics. "N/A" denotes the
absence of ASD.
TA Aggregate Results ( (%)
KC Aggregate Results (%)
MLLM
MFTT
MFNt
MAAt
BWTT
MFT1
MFN1
MAAt
BWTT
N/A
70.21
41.59
39,53
-28.62
78.49
61.75
68.34
-16.74
Intern Intern VL
66.80
48.72
59.17
-18.08
76.28
69.21
72.73
-7.06
Intern InternVL2-266 VL2-26B
68.13
47.88
59.71
-20.26
77.29
69.59
73.36
-7.71
Table 13, Comparison of different À values for RegLoRA, evaluated under TA and KC metrics.
TA Aggregate Results (%)
KC Aggregate Results (%)
X
MFT
MFN
MAAt
BWTT
MFT1
MFN1
MAA1
BWT1
X 10a
69.83
56.50
61.79
-13.33
78.39
73.28
74.74
-5.11
2.5 x 10a
69.02
58.57
63.04
-10.45
77.98
73.29
75.61
.4.69
5 x 10a
68.17
58.52
62.54
-9.65
77.40
73.09
75.25
-4.30
F. Different MLLMs in ASD
To evaluate the impact of data quality generated by MLLMs on ASD, we compare the performance of models trained with data produced by Intern VL2-26B (our default) and Intern VL2-8B (OpenGVLab Team, 2024). The results are presented in Table 12. Since both MILMss belong to the Intern V1.2 series, this comparison controls for additional confounding factors, suggesting that larger models produce higher-quality data. The results show that the model trained with Intern VL2-26B data performs slightly better on the most critical MAA metric. However, overall, there is no significant difference compared to the model trained with VL2-8B data. Regardless of the MLLM used, both models significantly outperform the baseline without ASD (denoted as "N/A" in the first row). These findings suggest that the effectiveness of ASD may not strongly depend on the quality of MLLM-generated data. Instead, ASD's primary advantage lies in exposing the model to diverse answering styles during task learning, thereby mitigating biases toward any single style.
G. Hyperparameter À in RegLoRA
Table 13 presents the aggregate metrics for TA and KC under varying values of the regularization loss weight hyperparameter in RegLoRA. As shown, smaller values lead higher MFT scores, indicating improved acquisition of new knowledge. In contrast, larger À values result in higher BWT scores, signifying reduced forgetting of previously learned knowledge. Overall, the findings suggest that À = 2.5 x 10 achieves the optimal trade-off between these objectives, yielding the highest performance in the key metrics, MFN and MAA, for both TA and KC
H. Additional Case Studies
In Fig. 6, we present additional case studies to complement those discussed in Fig. 4 of the main text. Specifically, cases 1-3 involve ScienceQA scenarios (Iuu et al., 2022) where the expected answers the letters of correct options. However, the baseline model, after learning the ImageNet task (Deng et al., 2009), incorrectly outputs the ImageNet category "Cucumber", Similarly, after learning the word-based VQAv2 task (Goyal et al., 2017), the model erroneously outputs the content of the options. By incorporating ASD, the model produces answers in the correct format, and further adding RegLoRA corrects errors in some cases.
Cases 4-6 require responses of specified ImageNet categories as defined by the CoIN benchmark (Chen et al., 2024a). However, after learning word-based tasks such as GQA (Hudson Manning, 2019) or VizWiz (Gurari al., 2018), the baseline model fails to provide valid ImageNet categories. In contrast, applying ASD ensures the model generates valid categories for all examples, with RegLoRA further enhancing prediction accuracy.
Cases 7-8 involve short answer questions, where the expected outputs are words or phrases. The baseline model, influenced by recent training on the Grounding task (Kazemzadeh et al., 2014; Mao et al., 2016), instead outputs bounding boxes, making it difficult to assess whether relevant knowledge is retained. Similar to previous cases, incorporating ASD corrects the output format, while RogLoRA further enhances the accuracy of the responses.
19

---

SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
Case
Case
Case 3
Case
Case
(41)
Whech priperty de these theer oiject
ssmoett
the Nenel's aitomet cnd yytrre
Peocpata Ricarde
Whlt cethcoppal Loudiaaa?
sales
Acpo ding Irsappheet tecmnam, the Arswer quastion wood (hraee
Accanding to Imagehiss tuorers, Aetive the spesi wordor planse
dipery
Now coliuit
eugee
glece chaiee livetly
tnder
the chalos cilecdy.
Sigertii:
Susertica
Speertii
Saperiai
(d1)
Seak- 2)
Scanoali
So amaaih beak
Teak imsepehe toak
Tesk Iraghhit baak 31
(62)
Mudel Staget lomne took jost toomee todk
Modil Stage: noreed take gud loverd two
Muse Stage we andd taskr (ver ineradd boc
Madel Stage: Lowred à tode Just toome to
Madi Stages soreed tade Juct tovme tws
Ground Trudh-
Case
Case
Case 8
Case
Case
Wher branc the black poctt
Mon many
Please proeide the bounding hox mocrtinatee the regi desoribed by
Plane provide the bounding hos coord nater the regten xQ,22
Acsrdingg tregehist's Laceearty Arnwer quolioo word plaise
the
vitan the prasded lifo musion inu.
Annser the question wordio plaase
unibredl' forman yll.
hornat
Seperficil
Seperiicel
Superiist
(42)
Teak reegpahen book-
Task letiic Jask
Task vioWe Dasa 5)
Tesk Orea nading bask
Di trunninng Oae
(62)
Made Stage Leanne tasks Just taanet tock
Model Stager Loomed as tadks Dast isoree
Mene Stage Lee wod tadis Dest lermad bock
Medel Stage Leaner 8 toaks Just tearee
Madel Stager Looned tasko Dast toame
Figure 6. Additional studies of main components in the proposed SEFE method. (a) Instruction; (b) Response from the baseline model (LoRA); (c) Response from the baseline model with ASD added; (d) Response from the baseline model with both ASD and RegLoRA added; (c) Basic information of the case.
Finally, cases 9-10 pertain to the Grounding task, After exposure to the word-based OCR-VQA task (Mishra et al., 2019), the baseline model generates words instead of the required bounding boxes. After applying ASD, the model generates responses in the correct format, and the addition of RegLoRA significantly improves the IoU between the predicted bounding boxes and the GT. Specifically, the IoU for case 9 increases from 0.0 to 0.76, and for case 10. it rises from 0.28 to 0.83, These results reaffirm the efficacy of our ASD method in mitigating superficial forgetting and the role of RegLoRA in addressing essential forgetting.
20


Reviews:
[{"review_id": "8KrXKZhzpi", "reviewer": "Reviewer_Hv34", "summary": "This paper introduces SEFE (Superficial and Essential Forgetting Eliminator), a novel framework for Multimodal Continual Instruction Tuning (MCIT), which aims to prevent catastrophic forgetting in multimodal models. The authors identify two distinct types of forgetting in MCIT”: superficial forgetting and essential forgetting. To mitigate these issues, the paper proposes two key techniques: Answer Style Diversification (ASD) and RegLoRA for these two types of forgetting respectively. The proposed SEFE method achieves state-of-the-art performance in MCIT benchmarks, including CoIN and the CoIN-ASD introduced by the paper, demonstrating its effectiveness in reducing both superficial and essential forgetting.\n\n## Update After Rebuttal\nMy original assessment was supportive, so I will maintain my current score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reviewing our paper, as well as your recognition of our contributions. Below, we respond to your comments in detail:\n\n### **Comparison with the [SP’2024] Paper**\n\n#### **Differences between the Two Papers** \n\nThank you for pointing out this related work. This study indeed shares our goal of understanding catastrophic forgetting, and its concept of *spurious forgetting* is in some respects similar to our definition of *superficial forgetting*. However, there are several key differences between the two studies.\n\n- *Superficial forgetting* refers to cases where the model fails to generate responses in the expected format, indicating a loss of response style. In such situations, it remains unclear whether the underlying knowledge has actually been forgotten. In contrast, *spurious forgetting* describes scenarios in which the model loses task alignment without any genuine loss of knowledge. Since *spurious forgetting* can be easily recovered, it is referred to as \"spurious\", whereas *superficial forgetting* does not emphasize recoverability. Thus, these two concepts are distinct.\n\n- The [SP’2024] paper focuses primarily on the analysis of forgetting, examining changes in gradients and feature principal components during forgetting. Although it proposes a mitigation strategy—*Freeze*, which freezes a few bottom layers—this approach is simple, is not the core contribution of the paper, and shows limited effectiveness (*e.g.*, ~60% forgetting rate). In contrast, our work focuses on practical solutions. We propose Answer Style Diversification (ASD) and RegLoRA to address both *superficial forgetting* and *essential forgetting* via data reconstruction and weight update constraints. On the CoIN benchmark, our method reduces the average forgetting rate to 10.45%, substantially lower than the 60% reported in the [SP’2024] paper. While differences in datasets limit direct comparability, the trend highlights the greater effectiveness and practicality of our approach. By contrast, *Freeze* serves more as an analytical tool of *superficial forgetting* than a standalone solution.\n\n- In addition to *superficial forgetting*, our study also examines *essential forgetting*, where the underlying knowledge is genuinely lost. In contrast, the [SP’2024] paper focuses exclusively on *spurious forgetting* that involves no knowledge degradation. This highlights another fundamental difference between the two studies.\n\n- Finally, the [SP’2024] paper focuses on LLMs, whereas our work addresses forgetting in MLLMs, marking a clear difference in research scope.\n\n#### **Performance Comparison** \nWe also evaluated *Freeze* on our benchmark, with the following results:\n| Method  | MFT   | MFN   | MAA   | BWT    |\n|---------|-------|-------|-------|--------|\n| FFT     | 65.87 | 35.45 | 36.73 | -30.42 |\n| Freeze  | 66.04 | 39.53 | 37.60 | -26.51 |\n| Ours    | **69.02** | **58.57** | **63.04** | **-10.45** |\n\nHere, FFT refers to full-parameter fine-tuning and serves as the baseline. While *Freeze* outperforms the baseline, it consistently underperforms compared to our method across all metrics. This is likely because *Freeze* is not well-suited for Multimodal Continual Instruction Tuning (MCIT), as it was not originally designed for this setting. Additionally, as discussed earlier, *Freeze* functions more as a supplementary analytical tool, so its performance may not be very satisfactory as a dedicated continual learning method.\n\n### **Comparison with Other Papers**\n\nIn addition to the [SP’2024] paper, we identified several other studies that investigate catastrophic forgetting. For example, [1] shows that catastrophic forgetting during LLM fine-tuning becomes more pronounced as the loss landscape sharpens, suggesting a strong positive correlation between sharpness and forgetting. [2] argues that in MLLMs, catastrophic forgetting arises as fine-tuning shifts the model’s focus from general visual-text alignment to dataset-specific overfitting, resulting in performance degradation even when the vision encoder is frozen. In contrast to these approaches, our work proposes decomposing catastrophic forgetting in MCIT into two components—*superficial forgetting* and *essential forgetting*—and addresses them separately, offering a new perspective on the problem.\n\nAs suggested, we will include a comparison between our work and other related works (such as [SP’2024], [1], [2]) for understanding catastrophic forgetting in the revised version.\n\n[1] [EMNLP 2024] Revisiting Catastrophic Forgetting in Large Language Model Tuning  \n[2] [CPAL 2024] Investigating the Catastrophic Forgetting in Multimodal Large Language Models", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Catastrophic forgetting is a key challenge in continual learning. This paper offers a new perspective by categorizing it into two types: superficial forgetting and essential forgetting.", "essential_references_not_discussed": "The paper lacks a discussion of and comparison with related works on understanding catastrophic forgetting such as [SP’2024]. \n\n[SP’2024] Zheng, J., Cai, X., Qiu, S., & Ma, Q. (2025). Spurious Forgetting in Continual Learning of Language Models. arXiv preprint arXiv:2501.13453.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper proposes and provides evidence to support a new perspective for understanding catastrophic forgetting. \n\nWeaknesses:\n1. Lack of comparison and discussion of a line of works for understanding catastrophic forgetting such as [SP’2024].", "other_comments_or_suggestions": "No additional comments.", "questions_for_authors": "1. Can the authors clarify the difference between their work and a similar work, [SP’2024]?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wA0NbgZVNw", "reviewer": "Reviewer_NhXB", "summary": "This paper introduces SEFE for multimodal continual instruction tuning. Within SEFE, an ASD paradigm is proposed to eliminate superficial forgetting by converting all questions into five unified question types. The authors also create a CoIN-ASD benchmark by applying ASD to the public CoIN benchmark, which can be used to assess essential forgetting. Moreover, a Reglora is proposed to eliminate essential forgetting by adding regularization to key elements of lora weight.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for your positive feedback on the performance and experimental validation of our proposed method. Below, we address your concerns in detail:\n\n### **Unnecessary Use of InternVL2 in Certain Distractor Generation Scenarios**\nAs you correctly noted, using InternVL2 for distractor generation is unnecessary in some cases. In the CoIN dataset, this applies to the ImageNet and Grounding tasks. Accordingly, we did not employ InternVL2 for these tasks. For ImageNet, distractors are randomly sampled from the class list excluding the ground-truth one. For Grounding, we generate random bounding boxes with Intersection-over-Union (IoU) less than 0.5 with respect to the ground-truth box.  \nWhile this special handling is briefly mentioned in Appendix C.2, a formal definition was not provided. Following your suggestion, we will incorporate a more rigorous definition based on the characteristics of the answer domain, enabling this issue to be addressed not only in CoIN but also in other benchmarks and real-world applications.\n\n### **Capability of InternVL2 to Understand and Solve the Task**\nThe ability of InternVL2 to generate explanations does not imply a full understanding of the task. This is because the model is provided with the ground-truth answer during explanation generation, eliminating the need for answer reasoning. Consequently, the model’s task is only generating an explanation conditioned on the correct answer, which is significantly easier.  \nTo validate this point, we conducted experiments evaluating InternVL2-26B on the eight tasks in CoIN. InternVL2-26B achieves an average accuracy of 53.76%, whereas our SEFE model attains 58.57% after continual learning across all tasks. Notably, SEFE is based on a 7B model, considerably smaller than InternVL2-26B. These results demonstrate that continual training of an MLLM is meaningful because it can yield superior performance with fewer inference resources.\n\n### **Modification of Fig. 3**\nThank you for the suggestion. We will revise Fig. 3 to include a plus sign, clearly indicating the combination method of $R_1$ and $R_2$.", "claims_and_evidence": "In general, all major claims are proven by experiments. I didn’t find any major claims that is obviously not convincing.", "methods_and_evaluation_criteria": "Yes. SEFE makes sense for continually adapting MLLMs to new tasks, which can be helpful in dynamic scenarios. The evaluation criteria are also solid, covering many aspects.", "theoretical_claims": "This paper doesn’t have theoretical claims.", "experimental_designs_or_analyses": "Yes. The experimental designs are basically sound and fair. They involve a reasonable setting and maintain same for all compared and ablation methods (incrementally training CoIN tasks one-by-one).", "relation_to_broader_scientific_literature": "The definition of superficial and essential forgetting in this paper seems to be expandable to continual learning of LLM and other large-scale models, and the proposed solutions may also be applicable to these and related areas.", "essential_references_not_discussed": "None. This paper reviewed sufficient literatures.", "other_strengths_and_weaknesses": "# Strengths:\n1. The performance is satisfactory.\n2. Sufficient experimental results, including comparisons and ablation studies, are reported.\n\n# Weaknesses:\nI have some concerns regarding the ASD paradigm:\n1. When you generate Yes/No questions and MCQs, you use InternVL2 to generate distractors. However, for some cases the answer domain of a task is limited (for example, in ImageNet, the answer domain contains only 1000 category names), this generation step is not necessary, because distractors can be simply selected from the answer domain (excluding the correct answer). This issue is not sufficiently discussed and defined in the paper.\n2. InternVL2 is used to generated explanations in ASD. However, my concern is that, if InternVL2 can generate accurate explanations, doesn’t that mean InternVL2 can totally understand and solve the task? Why we need to continually train another MLLM?", "other_comments_or_suggestions": "Figure 3 is not very clear. It is recommended to use a plus sign to indicate the combination of R1 of R2.", "questions_for_authors": "Please refer to the weaknesses and comments in previous two sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3BIjRPQwSX", "reviewer": "Reviewer_iTsR", "summary": "This manuscript delves into the area of Multimodal Continual Instruction Tuning (MCIT). It contributes by differentiating between two forms of catastrophic forgetting: *superficial forgetting* and *essential forgetting*. Superficial forgetting is defined as the forgetting of the response style, while essential forgetting denotes the loss of knowledge. To tackle these problems, the manuscript introduces two corresponding solutions.  Firstly, it introduces the Answer Style Diversification (ASD) paradigm. This paradigm unifies questions within each task into five predefined styles. By doing so, it alleviates the variability in question domains, the main cause of superficial forgetting. Secondly, the manuscript presents RegLoRA. This method restricts modifications to the elements in LoRA’s weight update matrices that are highly relevant to previously learned knowledge to prevent essential forgetting.  \nThe experimental results show that the proposed overall approach, SEFE, attains state-of-the-art performance. The effectiveness of both the ASD and RegLoRA techniques is also validated.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and for recognizing the value of our contributions. Below, we address your comments in detail:\n\n### **Training Stages of MLLMs**\nThank you for pointing this out. Indeed, the training process of MLLMs/LLMs often involves more than two stages. We will revise the sentence to: \"Training these models typically involves multiple phases, with pretraining and instruction tuning being two crucial ones.\"\n\n### **Introduction of LoRA Variants**\nThank you for your constructive feedback. We will include an overview of LoRA variants, such as those you mentioned: AdaLoRA, which adaptively allocates the parameter budget across weight matrices based on their importance scores; LoRA+, which assigns different learning rates to matrices A and B to enhance performance in settings with large embedding dimensions; and DoRA, which decomposes pretrained weights into magnitude and direction components, applying LoRA solely to the direction component to reduce the number of trainable parameters. While these methods are not primarily designed to mitigate catastrophic forgetting and may not be directly applicable to MCIT, some of their underlying principles may offer valuable insights. We have therefore included a discussion of these approaches in a separate subsection within the Related Work section for readers’ reference.\n\n### **Incomplete Definition of the Learning Objective**\nThe original loss function is the standard language modeling loss, *i.e.*, next-token prediction loss, consistent with the base model (LLaVA-1.5). It is defined as:  \n$$L_{lm} = - \\sum_{i=1}^{L} \\log P(x_i | X_v, X_\\text{instruct,<i}, X_{a,<i}),$$ \nwhere $L$ is the length of the answer, $X_v$ denotes the visual input tokens, $X_\\text{instruct,<i}$ and $X_{a,<i}$ represent the instruction and answer tokens prior to the current prediction token $x_i$, respectively.  \nDuring training on the first task, the objective is solely this language modeling loss. For subsequent tasks, the total loss becomes a combination of the language modeling loss and the regularization loss $L_{reg}$ as defined in Eq. 2 of the paper:\n$$L_{total} = L_{lm} + L_{reg}.$$  \nWe will incorporate this complete formulation and explanation into the revised version.\n\n### **Adding a Related Works Section in the Main Text**\nThank you for the helpful suggestion. As recommended, we will add a concise Related Works section to the main text, include the introduction to LoRA and a brief review of MCIT works within it, and retain the more detailed review of related literature in the appendix.\n\n### **Bounding Box Responses under *Superficial Forgetting***\nYour idea is compelling, but it is challenging to identify samples in our existing dataset that match the scenario you described—where both the question and the answer contain an object present in the image. To investigate this, we manually constructed a few examples. Our observations indicate that the model’s predicted bounding boxes tend to align with the object mentioned in the question, rather than the one in the answer. This behavior reflects an inductive bias introduced by the grounding task, in which training samples typically require the model to localize the object referenced in the question. These findings suggest that *superficial forgetting* may lead the model to over-rely on task-specific biases, hindering its ability to adapt to the actual demands of the current query. Rather than interpreting the intent of the question, the model appears to default to habitual responses based on prior training. If the model had even partially understood the question, we would expect it to at least identify the object mentioned in the answer. This further illustrates how *superficial forgetting* can obscure the model’s ability to adjust its behavior based on contextual demands. Thank you for your insightful question.", "claims_and_evidence": "This manuscript's main claims are well-supported by experimental evidence, rendering them convincing. However, I take issue with a statement in the Introduction (line 30-31), which reads: \"Training these models typically involves two main phases: pre-training and instruction tuning\". Based on my knowledge, the training of Multimodal Large Language Models (MLLMs)/Large Language Models (LLMs) usually encompasses more than two stages. It may also incorporate a Preference Optimization/Reinforcement Learning phase. Although this statement does not substantially undermine the manuscript's contributions, I still suggest that the authors revise it to ensure accuracy.", "methods_and_evaluation_criteria": "The proposed SEFE method holds value in the continuous improvement of MLLMs to enable their adaptation to new demands. Additionally, the introduced CoIN-ASD benchmark can be useful for evaluating essential forgetting within the MCIT domain.", "theoretical_claims": "This manuscript primarily focuses on method design and evaluates its effectiveness through experimental observation and analysis. It does not include significant theoretical discussion or mathematical derivation. Therefore, this section is not applicable.", "experimental_designs_or_analyses": "-\t**Comparison with existing methods**: The authors utilize the public CoIN benchmark and the CoIN-ASD benchmark proposed in this work. The evaluation includes two components: Truth Alignment (TA) and Knowledge Capability (KC). Each component encompasses the accuracy of all tasks, along with four aggregate metrics: Mean Fine-tune Accuracy (MFT), Mean Final Accuracy (MFN), Mean Average Accuracy (MAA), and Backward Transfer (BWT). These metrics are quite comprehensive. \n-\t**Ablation studies**: The ablation experiments cover the validation of ASD, RegLoRA, and the selection of hyperparameters. The metrics employed are the aggregate metrics of TA and KC. The authors present an analysis for each experiment, which is reasonable overall.", "relation_to_broader_scientific_literature": "The RegLoRA proposed in this manuscript is a regularization-based continual learning scheme. It has certain relevance to prior methods like EWC [1] and PODNet [2]. These strategies reduce the magnitude of model updates following specific rules to alleviate forgetting.\n\n[1]\tKirkpatrick, James, et al. \"Overcoming catastrophic forgetting in neural networks.\" Proceedings of the national academy of sciences (2017).\n\n[2]\tDouillard, Arthur, et al. \"Podnet: Pooled outputs distillation for small-tasks incremental learning.\" ECCV, 2020.", "essential_references_not_discussed": "Since the RegLoRA proposed in this manuscript is a variant of LoRA, I suggest adding an introduction to other LoRA variants in Appendix B (Related Works) for differentiation. For example, the following papers:\n\n[1]\tZhang, Qingru, et al. \"AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning.\" arXiv preprint arXiv:2303.10512 (2023).\n\n[2]\tHayou, Soufiane, Nikhil Ghosh, and Bin Yu. \"Lora+: Efficient low rank adaptation of large models.\" arXiv preprint arXiv:2402.12354 (2024).\n\n[3]\tLiu, Shih-Yang, et al. \"Dora: Weight-decomposed low-rank adaptation.\" arXiv preprint arXiv:2402.09353 (2024).", "other_strengths_and_weaknesses": "The primary strength of this work lies in its definition of superficial forgetting and essential forgetting. The experimental results and analysis support the need to investigate these phenomena separately. Additionally, the proposed method is logically sound and demonstrates strong performance.\nHowever, several weaknesses remain, in addition to the previously mentioned issues (lack of rigor in one statement and the absence of a review of other LoRA variants):\n\n-\t**Limited dataset diversity**: Although this study includes both CoIN and CoIN-ASD benchmarks, these datasets are essentially derived from the same source, which may weaken the generalizability of the experimental results. That said, given that CoIN is the only publicly available MCIT benchmark, this limitation is somewhat understandable.\n-\t**Incomplete definition of the learning objective**: In line 272-274, the authors state that “L_reg is added to the original loss of the base MLLM to form the complete learning objective.” However, the manuscript does not clearly define the original loss or the overall learning objective. This omission reduces the study’s reproducibility.", "other_comments_or_suggestions": "The Related Works section is currently located in the appendix. This makes it hard to quickly grasp the distinctiveness of this manuscript without referring to the appendix. I suggest adding a concise Related Works section within the main text. The introduction to LoRA in the RegLoRA section could be moved there. A more in-depth review of related works can still be retained in the appendix.", "questions_for_authors": "I am curious about one aspect: In the superficial forgetting scenario where the model responds to any question with a bounding box (e.g., Fig. 1(d)), consider a situation where both the question and the answer contain objects, yet these objects differ. Will the bounding box indicate the object in the question or the one in the answer? For instance, if the question is “What is to the left of the sheep?” and the correct answer is “A cow”, when superficial forgetting occurs, does the returned bounding box point to the sheep or the cow? Or perhaps it points to neither? I think this aspect is valuable for exploring the essential forgetting.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgeMdGy4b0", "reviewer": "Reviewer_HyXf", "summary": "This paper focuses on the continual learning task for multimodal large models, termed MCIT. The authors innovatively categorize catastrophic forgetting issues in this field into two distinct types: superficial and essential forgetting. They propose the SEFE model, which incorporates two key modules, ASD and RegLoRA, to respectively address these two categories of forgetting problems. Experimental results demonstrate the effectiveness of the proposed model. Furthermore, the authors introduce a novel dataset, CoIN-ASD, specifically designed to evaluate the performance of MCIT tasks. The article exhibits a well-organized structure and maintains clear semantic coherence throughout its presentation.\n\n## Update After Rebuttal\nI thank the authors' rebuttal and keep my rating to accept this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the strengths of our paper and for your effort in reviewing our submission. Below is a detailed response to your concerns:\n\n### **The Abstract Should Be Shortened**\nThank you for the suggestion. We will shorten the abstract in the revised version to improve conciseness.\n\n### **Difference Between RegLoRA and Existing Regularization-Based Methods**\nRegLoRA can be viewed as a regularization-based method. However, it differs from most existing approaches in several key aspects:\n\n- As LoRA fine-tuning becomes increasingly prevalent for adapting large-scale models, there is a growing need to mitigate catastrophic forgetting in LoRA-based settings. In this context, RegLoRA—which applies regularization to LoRA adapters—is naturally well-suited, whereas existing regularization methods that operate on the full model may be less appropriate.\n- Unlike some existing regularization-based approaches, RegLoRA does not require auxiliary models or a significant number of additional parameters on GPUs. This substantially reduces GPU memory usage.\n- RegLoRA focuses on regularizing a small number of key elements rather than entire weight matrices or feature maps, which also distinguishes it from most existing methods in terms of strategy.\n\nIn summary, while RegLoRA belongs to the family of regularization-based methods, its design choices make it more efficient and practical for our task setting.\n\n### **Rationale for Converting Tasks in Five Ways**\nOur Answer Style Diversification (ASD) paradigm reformulates tasks into five question types based on prior knowledge. Specifically, we examined 15 widely-used MLLM benchmarks and analyzed the applications of MLLM in real-world scenarios. Our analysis reveals that the identified categories—yes/no questions, multiple-choice questions, short answer questions, brief explanation/description questions, and detailed explanation/description questions—cover the majority of use cases. The list of benchmarks referenced is provided in lines L97–L101, all of which belong to these five types. In practical settings, if additional question types arise in specific domains, our approach can also be extended to accommodate them.\n\n### **Reason for Partially Converting Questions**\nIf all questions are converted, the model cannot learn the answer format of the original question style. Since the test set retains the original format, a model that has never seen samples in this style is unlikely to perform well during evaluation. As shown in Table 3, when the conversion ratio is high (*e.g.*, 80%), performance drops compared to lower ratios (*e.g.*, 20%), indicating that preserving a substantial portion of original-format questions is essential.\n\n### **Possibility to Combining RegLoRAs and Pseudo-Code Explanation**\nWe agree that combining RegLoRAs—or more generally, LoRAs—from different tasks is an intriguing and promising direction. If successful, this approach could enable the construction of a unified multi-faceted MLLM by integrating task-specific LoRAs, which holds significant practical value. However, this idea lies beyond the current scope of our work. While we are not yet able to evaluate its feasibility or propose a concrete method, we appreciate your insightful suggestion and will continue exploring this direction in future research.  \n\nRegarding the overall training procedure of RegLoRA, we provide the pseudo-code below for clarity:\n\n---\n\n**Require:** base MLLM $M_0$, training sets of all tasks {$D_1, D_2,...,D_T$}  \n**for** task $j$ in $1$ to $T$ **do**  \n&nbsp;&nbsp;&nbsp;&nbsp;Insert a new LoRA adapter $LoRA_j$ into model $M_{j-1}$  \n&nbsp;&nbsp;&nbsp;&nbsp;**for** each batch in $D_j$ **do**  \n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute language modeling loss $L_{lm}$  \n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** j > 1 **then**  \n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute regularization loss $L_{reg}$ using all previous masks {$R_1, R_2,...,R_{j-1}$} according to Eq. 2.  \n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Set total loss $L_{total} = L_{lm} + L_{reg}$  \n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**else**  \n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Set total loss $L_{total} = L_{lm}$  \n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Update model parameters to minimize $L_{total}$  \n&nbsp;&nbsp;&nbsp;&nbsp;Compute the regularization mask $R_j$ for the current task  \n&nbsp;&nbsp;&nbsp;&nbsp;Merge $LoRA_j$ into $M_{j-1}$ to form $M_j$  \n\n---\n\nHere, the language modeling loss $L_{lm}$ refers to the next-token prediction loss used by the base MLLM.  \nAfter learning each task $j$, we evaluate performance on all learned tasks using the updated model $M_j$.", "claims_and_evidence": "The claims are generally well-supported by experimental results, demonstrating the effectiveness of SEFE, ASD, and RegLoRA.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-structured and align with the challenges of incremental learning in MLLMs. The distinction between superficial and essential forgetting is insightful, and ASD and RegLoRA offer effective solutions. While the approach is promising, providing clearer explanations in certain areas would further strengthen the presentation and impact of the work.", "theoretical_claims": "For catastrophic forgetting, the authors address the problem in two parts, distinguishing between superficial and essential forgetting. The proposed methods, ASD and RegLoRA, effectively mitigate these issues, providing a well-reasoned and empirically supported solution. While the paper does not focus on formal theoretical proofs, the conceptual framework is sound and aligns well with the task objectives.", "experimental_designs_or_analyses": "The experimental design is well-structured and effectively evaluates the proposed methods for addressing catastrophic forgetting. The distinction between superficial and essential forgetting is clearly tested, and the results demonstrate the effectiveness of ASD and RegLoRA.", "relation_to_broader_scientific_literature": "The authors are the first to break down the issue of catastrophic forgetting into two categories: superficial and essential forgetting problems. They propose corresponding methods to address each, providing new insights and solutions for the development of this field. This distinction offers a fresh perspective on understanding and mitigating catastrophic forgetting, contributing significantly to advancing current research.", "essential_references_not_discussed": "The authors have provided the relevant references to support their work.", "other_strengths_and_weaknesses": "Strengths:\n\n1.The author divides the catastrophic forgetting problem in MCIT into superficial and essential forgetting problems, and proposes new methods, ASD and RegLoRA, to address them.\n\n2.The author proves the effectiveness of the proposed method through experiments.\n\n3.The author puts forward a new dataset for the MCIT task.\n\n4.The overall structure of the article is clear and easy to understand.\n\nWeaknesses:\n\n1.The abstract is too long and needs to be shortened.\n\n2.The motivation and implementation of the module design are not clearly explained.", "other_comments_or_suggestions": "1. Is there a difference between RegLoRA and the regularization-based methods in continual learning? Both seem to constrain the update of some parameters during training to adapt to new tasks.\n\n2.Why does ASD convert tasks into these five ways? Is there any prior knowledge? Or are these five ways sufficient to answer all questions in the real world?\n\n3.In CoIN-ASD, why only a part of the questions are converted into other forms? Would it be better if all questions were converted according to the five forms?\n\n4.Does it need to train a RegLoRA for each new task? How are the RegLoRAs of different tasks combined? What are the overall input, output, training, and testing processes of the model? Can a schematic diagram or pseudo-code be provided?", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents SEFE, a framework for Multimodal Continual Instruction Tuning (MCIT), which introduces a two-fold categorization of forgetting: superficial and essential. To address these, the authors propose the Answer Style Diversification (ASD) paradigm to prevent format-related degradation (superficial forgetting), and RegLoRA, a regularization-based method to mitigate knowledge degradation (essential forgetting). The paper is complemented by the introduction of the CoIN-ASD benchmark for better evaluation of MCIT challenges.

The reviewers appreciated the paper's conceptual clarity, practical impact, and strong empirical results, with multiple reviewers highlighting the distinction between the two types of forgetting as both novel and useful. The proposed methods were shown to outperform baselines across multiple metrics and ablation settings. While a few reviewers raised minor concerns—such as limited dataset diversity and comparisons with other LoRA variants—these were not deemed critical, and the overall consensus was positive.
Decision: Accept (poster)