ICML 2025

SEFE: Superficial and Essential Forgetting Eliminator for
Multimodal Continual Instruction Tuning
Jinpeng Chen * 1 Runmin Cong * 2 Yuzhi Zhao 1 Hongzheng Yang 3
Guangneng Hu 4 Horace Ho Shing Ip 1 Sam Kwong 5
https://github.com/jinpeng0528/SEFE/
Abstract
Multimodal Continual Instruction Tuning (MCIT)
aims to enable Multimodal Large Language Mod-
els (MLLMs) to incrementally learn new tasks
without catastrophic forgetting. In this paper, we
explore forgetting in this context, categorizing it
into superficial forgetting and essential forgetting.
Superficial forgetting refers to cases where the
model’s knowledge may not be genuinely lost,
but its responses to previous tasks deviate from
expected formats due to the influence of subse-
quent tasks’ answer styles, making the results
unusable. By contrast, essential forgetting refers
to situations where the model provides correctly
formatted but factually inaccurate answers, indi-
cating a true loss of knowledge. Assessing essen-
tial forgetting necessitates addressing superficial
forgetting first, as severe superficial forgetting can
obscure the model’s knowledge state. Hence, we
first introduce the Answer Style Diversification
(ASD) paradigm, which defines a standardized
process for transforming data styles across differ-
ent tasks, unifying their training sets into similarly
diversified styles to prevent superficial forgetting
caused by style shifts. Building on this, we pro-
pose RegLoRA to mitigate essential forgetting.
RegLoRA stabilizes key parameters where prior
knowledge is primarily stored by applying regu-
larization, enabling the model to retain existing
competencies. Experimental results demonstrate
that our overall method, SEFE, achieves state-of-
the-art performance.
*Equal contribution 1City University of Hong Kong 2Shandong
University
3The Chinese University of Hong Kong
4Xidian
University 5Lingnan University.
Correspondence to:
Yuzhi
Zhao <yzzhao2-c@my.cityu.edu.hk>, Sam Kwong <samk-
wong@lingnan.edu.hk>....