Knowledge Externalization: Reversible Unlearning and Modular Retrieval in Multimodal Large Language Models
Authors: Jiaqi Li
Zihan You
Ruoyan Shen
Shenyu Zhang
Songlin Zhai
Yongrui Chen
Chuanyi Zhang
Jiahui Geng
Fakhri Karray
Sheng Bi
Guilin Qi
Venue: ICLR 2026 Poster (2026)
Keywords: Machine unlearning
Multimodal Large Language Model
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Multimodal Large Language Models (MLLMs) achieve remarkable cross-modal understanding by training on vast web-scale datasets, but inadvertently internalize sensitive personal and proprietary information. Existing machine unlearning methods address this by irreversibly altering model parameters to permanently erase knowledge. This destructive paradigm conflicts with modern privacy regulations that mandate auditable, reversible, and user-controllable data management. To address these challenges, we propose Knowledge Externalization, a novel framework for reversible and modular knowledge management in MLLMs.  We first propose Dual-Stream Memory Tuning, a method that transfers targeted knowledge from a model's internal parameters into external memory tokens. To mitigate gradient interference when externalizing multiple concepts, we further introduce Soft Orthogonal Weighting, a technique that preserves the independence of each token. Our resulting framework demonstrates three key capabilities: (i) It achieves effective forgetting of target concepts within the base model, while enabling high-fidelity knowledge restoration using the corresponding memory token. (ii) It supports continuous knowledge editing, allowing the information stored within an external token to be dynamically updated post-externalization. (iii) It displays a remarkable emergent ability for compositionality, where multiple memory tokens (including edited ones) can be freely combined to simultaneously recover knowledge corresponding to each concept. Our source code will be released in the near future.

Published as a conference paper at ICLR 2026
KNOWLEDGE EXTERNALIZATION: REVERSIBLE UN- LEARNING AND MODULAR RETRIEVAL IN MULTI- MODAL LARGE LANGUAGE MODELS
Jiaqi Zihan You"; Ruoyan Shen ShenYu Zhang Songlin Zhai3 Yongrui Chen Chuanyi Zhang Jiahui Geng.? Fakhri Karray", Sheng Bi³, Guilin Qi
Sccool of Cyber Science and Engineering, Southeast University, Nanjing, China
2School of Instrument Science and Engineering, Southeast University, Nanjing, China
"Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China *School of Computer Science and Engineering, Southeast University, Nanjing, China LLaw and Innovation Lab, School of Law, Southeast University Mohamed bin Zayed University of Artificial Intelligence LLinköping University
*College of Artificial Intelligence and Automation, Hohai University, Nanjing, China
{jqli, 213223318, 220242389, shenyu. zhang, songlin yrchen, shengbi, gqi www.eeeeuuu..eeeeeeeeeee 20231 104@hu.. edu. en, jiahui. geng@leeggglll iu. se, (jiahui. geng, Fakhri. Karray J@mbzuii ac. ae,
ABSTRACT
Multimodal Large Language Models (MLLMs) achieve remarkable cross-modal understanding by training on vast web-scale datasets, but inadvertently internalize sensitive personal and proprietary information. Existing machine unlearning meth- ods address this by irreversibly altering model parameters to permanently erase knowledge. This destructive paradigm conflicts with modern privacy regulations that mandate auditable, reversible, and coeeooontoll data management. To ad- dress these challenges, we propose Knowledge Externalization, a novel framework for reversible and modular knowledge management in MLLMs. We first propose Dual-Stream Memory Tuning, method that transfers targeted knowledge from model's 's internal parameters into external memory tokens. To mitigate gradient interference when externalizing multiple concepts, we further introduce Soft Or- thogonal Weighting, technique that preserves the independence of each token. Our resulting framework demonstrates three key capabilities: (i) It achieves effective forgetting of target concepts within the base model, while enabling
knowledge restoration using the corresponding memory token. (ii) It supports continuous Knowledge Editing, allowing the knowledge stored within an external token to be dynamically updated post-extermllizatio. (iii) It displays a remarkable emergent ability for compositionality, where multiple memory tokens (includ- ing edited ones) can be freely combined to simultancously recover knowledge corresponding to each concept. Our source code will be released in the near future.
INTRODUCTION
The emergence of Multimodal Large Language Models (MLLMs) marks a significant breakthrough in artificial intelligence (Chen et al., 2024a; Li et al., 2023; Koh et al., 2023; Dai et al., 2023; Fang et al., 2025a;e), enabling advanced cross-modal understanding and generation across domains like medical imaging interpretation and visual storytelling (Zheng et al., 2023; Huang et al., 2024; Zhang et al., 2024a; Huang et al., 2023). Despite their impressive capabilities, MLLMS pose critical privacy risks by inadvertently internalizing sensitive personal data and proprietary information during pretraining (Mantelero, 2013; Scherer & Kiparski, 2018; 8; Leite et al., 2022). To mitigate these risks, machine
J. Li and Z. You contributed equally to this work and should be considered co-first authors.
Corresponding author.

---

Published as a conference paper at ICLR 2026
unlearning techniques (Eldan & Russinovich, 2023; Si et al., 2023; Wang et al., 2023; Thaker et al., 2024; Liu et al., 2024) have gained prominence, selectively forgetting targeted knowledge from models while maintaining utility without requiring costly retraining.
However, current unlearning methods inherently rely on irreversible parameter modifications (Yao et al., 2023; 2024; Gandikota et al., 2024; Lu et al., 2024), permanently erasing target knowledge from the model. This paradigm contradicts emerging privacy regulations like ISO/IEC 27701 which emphasize reversible, auditable, and user-controllable management of personally identifiable information (PII). Specifically, these frameworks advocate for Privacy Information Management Systems (Tzolov, 2019; Anwar & Gill, 2021) that empower users with granular control, such as the right to restrict processing (GDPR Art. 18) (Scherer & Kiparski, 2018) rather than only permanent deletion, and demand auditable logs of all data management activities. This creates need for systems that can not only forget, but also manage and trace knowledge in reversible manner. To address these challenges, we introduce Knowledge Externalization, novel framework for reversible and modular knowledge management in MLLMs. Inspired by the concept of the 'Pensieve' from the Harry Potter universe, which enables temporary removal and later retrieval of memories, our approach transfers target knowledge from internal model parameters to external memory tokens. This decoupling allows the base model to forget sensitive information, while preserving the ability to restore it via memory retrieval when needed. In addition, by extemaaizzing knowledge into dedicated memory tokens, the forgetting process remains localized, helping to reduce unintended effects on non-target model behavior.
To implement the paradigm, we propose a baseline method called Dual-Stream Memory Tuning (DSM). DSM uses a gradient-ascent objective erase targeted knowledge from the base model, while concurrently applying gradient-descent to encode the removed knowledge into external memory tokens. In practical scenarios, it is often necessary remove and manage multiple knowledge concepts, such as different personal entities or domain-specific facts. To support this, DSM assigns a dedicated memory token to each concept and ensures that only the corresponding token is updated during externalization.
However, this multi-concept setup introduces key challenge: when different memory tokens are trained independently, their gradient updates may influence overlapping model parameters, leading to interference that degrades the fidelity of each token. To mitigate this, we propose Soft Orthogonal Weighting (SOW), which replaces hard gradient masking with exponential attenuation scheme. Specifically, when encoding a new concept, we measure its gradient similarity to historical concepts and apply wss) e Me++1] to attenuate interference while preserving optimization flow.
For evaluating Knowledge Externalization, we extend MMUBench (Li et al., 2024b) into MXEBench, a new benchmark for Multimodal Knowledge Externalization. Results demonstrate that our method achieves effective forgetting of target concepts, minimal degradation on non-target knowledge, and precise recovery via memory tokens. Beyond reversible unlearning, we also show that our externalized memory design supports Dynamic Knowledge Editing (detailed in Sec.4.3) and Emergent Compositional Knowledge Recovery (detailed in Sec.4.4), which are opening new possibilities for controllable and modular knowledge systems.
In summary, our contributions are: (i) We propose Knowledge Externalization, the first reversible framework for managing unlearning knowledge in MLLMs via external memory tokens. (ii) We introduce Dual-Stream Memory Tuning to decouple knowledge unlearning from knowledge retention using dual-gradient optimization. (iii) We design Soft Orthogonal Weighting with exponential attenuation, enabling scalable multi-concept externalization with provable interference bounds. (iv) We empirically demonstrate high-fidelity knowledge removal, restoration, editing, and composition across diverse concepts.
2 RELATED WORK
Unlearning in MLLMs. Recent Machine Unlearning research has explored various strategies for LLMs and MILMS (Jang et al., 2023; Kumar et al., 2023; Pawelczyk et al., 2024; Ishibashi Shimodaira, 2023; Maini et al., 2024; Thaker et al., 2024; Liu et al., 2024), including Gradient
ISOOTEC 27701 extends the ISO/EC 27001 standard to include privacy management, providing framework for organizations to manage personally identifiable information (PII).
2

---

Published as a conference paper at ICLR 2026
Training Step for Externalize Knowl edge 6g
Post Externalization
Knowledge Unlearning Recovering
Training Dato Knowl adge
Test Data
What's name this mon? Donald Thump
Do you know
MILLA
Hil name Banadk Obano.
depicted
Test Data
Reterdlirdd
Vision Encoder
Tokenizer
Do you know
NILMM
Donald Trump.
charasher depicied
Forward Twice
Knowledge Editing
Before Editing
Editting
iold MLLAA
Transformer Block
Ai No. Joe Biden is the presdient.
2025, President USA
Memory
Post Editing
Extereeiced
serving President United States
You
Softmax
20207
Edited
Knowledge Compositing
Composine
Exherddiie hendiioed
Outpul
Oulput 5g
His name is Danaid Trump
chanacher depicted here?
Lrss Gradent
o hscent
Gradent
Compo
Scherndised
+1 Forword w/o 4
Viseal Taken
Texthal Krondedge
Forward w/
Knoeeedgge
chanacter
recogrize
He tion Mask.
The training process of Dual- Stream Memory Tuning utilizes parallel forward passes to simultaneously erase knowledge from the base model (via gradient ascent on Lixe) and encode into external memory token Ec, (via gradient descent on Lxe). Right: This modular design enables key post-cxternalization capabilities. (Top) The base model forgets the concept, while knowledge is recovered with the token. (Middle) Knowledge can be dynamically updated by editing the token alone. (Bottom) Multiple tokens can be composed at inference to recover all corresponding knowledge, which is unique emergent capability of our framework.
Figure I: Overview of the Knowledge Externalization Framework. Left:
Ascent to eliminate undesirable outputs (Yao et al., 2023), knowledge alignment approaches (Wang et al., 2023), lightweight unlearning layers (Chen & Yang, 2023), and methods combining GA with KI.-divergence (Yao et al., 2024). SIU (Li et al., 2024b) focuses on erasing visual concepts while preserving generative capabilities. However, these approaches implement irreversible parameter modifications that permanently erase information. Our Knowledge Externalization framework introduces reversible unlearning by transferring target knowledge to external memory tokens, enabling precise recovery, compositional querying, and dynamic editing.
Parameter-Effiiee Fine- Taaee-TTnigg... PEFT methods reduce fine-tuming costs through adapter modules that insert small networks into transformer layers (Gao et al., 2024; Luo al., 2024; Ran al., 2023; Guo et al., 2024), Low-Rank Adaptation (LoRA) that approximates updates via low-rank matrices (Hu et al., 2022) and is widely adopted across vision (Hayou et al., 2024; Agiza et al., 2024), speech (Prasad et al., 2024; Baby et al., 2024), and multimodal domains (Guo et al., 2025; Wu et al., 2024), and prompt-bused tuning including prefix tuning (Li & Liang, 2021), prompt tuning (Jia et al., 2022), and multimodal extensions (Zhou et al., 2024; Shen et al., 2024). These methods target task adaptation by efficiently adding knowledge, while our approach encodes multiple concepts into separate, controllable modules. Unlike adapters or LoRA A that modify internal paths, our memory tokens operate as external, composable, and reversible modules.
3 TASK DEFINITION
Let Ma denotes pretrained Multimodal Large Language Model (MLLM) with parameters 0
Rd trained on a multimodal dataset = {(I, Ta)l where I, E # xxxx3 x3 are images and are text sequences. During pretraining, Ma may internalize sensitive concepts C {cI,..., c each associated with data D.g C D.
T. (wf....,
w ...
We define the goal of Knowledge Externalization in MILMS as follows:
3

---

Published as a conference paper at ICLR 2026
Knowledge Externalization
Knowledge Externalization enables the temporary removal of sensitive knowledge from MILMS without degrading utility, while retaining the capability to recover that knowledge through the external memory.
The optimization objective can be formulated as follows:
T log PM or (we wel)
min 8',5
Σ
log PM (wis I, wee)
E
CLEC
(I,T)CDUU,
Forgetting loss Eie
Ulility preservation Eyp
1 Σ
E
log PM (wi, [E II, well)
eLEC
Recoverabilicy loss Cuse
where SE LeaeJeee is the of externalized memory tokens, and [Beg II, werl denotes the concatenation of Esg as a prefix to the multimodal input sequence.
4 METHODOLOGY
To achieve the objectives of Knowledge Externalization defined in Eq. we propose Dual-Stream Memory Tuning (DSM), which framework that decouples the forgetting of sensitive concepts from their recoverable preservation (presented in Fig.1). DSM operates simultancously optimizing the base model parameters and set external memory tokens Sg = {Ee, JaLec. ensuring that knowledge is not erased but transferred from internal parameters modular, auditable tokens. While DSM naturally supports single-concept externalization, managing multiple concepts introduces gradient interference, where updates for one concept degrade the fidelity of others. To enable scalable and modular knowledge management, we further introduce Soft Orthogonal Weighting, regularization mechanism that encourages near-orthogonal update directions in the parameter space.
4.1 DUAL-STREAM MEMORY TUNING
The core mechanism of DSM is dual-stream optimization process that implements zero-sum game between forgetting and recovery. For each concept ck. the base model Mg undergoes gradient ascent the forgetting loss Lixr:
log PMc (w; II, wes)
0 0' +1 Loo 0' Vo
which repels 0' from the knowledge manifold of ce. effectively erasing its influence on model predictions.
Concurrently, to preserve this knowledge for authorized recovery, we perform gradient descent on the recoverability loss Cree with respect to both the memory token Esg and the model parameters 0':
Fie, Frce 7
V Eeg free:
6' 6¹ - Vo Lree:
where
T
Lree = BEI
Σ
log PM (we [Bog iI, wea)
(5)
4.2 SOFT ORTHOGONAL WEIGHTING FOR MULTI-CONCEPT EXTERNALIZATTON
4

---

Published as a conference paper at ICLR 2026
Externalizing C
Gradient
For multi-concepp externalization, the gradient updates for different concepts may interfere when they share param- eter subspaces, leading to degraded token fidelity. To mitigate this, introduce Soft Orthogonal Weight- ing (SOW), which dynamically atten- uates the memory token update based on its similarity historical recov- ery directions. As shown in Fig.2, SOW maintains a gradient history dic- tionary H = {cj و where g; - Vo free (c²) is the recovery gradient concept Cj. For a new concept ck, we first compute its raw gradient gk. then construct a historical composite gradient:
Lyec
Dictionary
G
91
9F
C2
92
c
M 0'
1:
C--1
9
A(3" (1)
9k
9
91
Update
Store
Figure 2: Illustration of Soft Orthogonal Weighting (SOW). When externalizing a new concept Ck2 its recovery gradient gx is compared against historical composite gradient Vhis synthesized from the Gradient Dictionary used to compute an exponential weight wi(s (() which attenuates the update applicd to the memory token Es,, and the model Mar.
The resulting cosine similarity s*
||g,||
Vhis Σ
aggj, OEEH
Σaygi
Oj
Σ E |lg.ll
which represents the dominant direction of prior knowledge unlearning. We then compute the cosine similarity between gi and Vhist
Kg Whis)
ggll visll
This similarity reflects how much the new concept overlaps with prior recovery directions. We define an attenuation weight:
ma(*)= = e==-\(+11)) >>
(8)
which suppresses updates in redundant directions. The final updates are:
Fer t- Fec. - ())"). V VE free: 8' 5- 0 7- w((") ). V free-
(9)
(10)
The theoretical analysis of SOW is presented Sec.A.4.
4.3 DYNAMIC KNOWLEDGE EDITING
Knowledge editing (Li & Chu, 2025; Xu al., 2025; Zhang al., 2024b) is critical for maintaining model accuracy and trustworthiness by correcting specific factual errors or outdated information in LLMs and MLLMs. For instance, the knowledge of the president of The US needs be continuously updated in LLMs, or the visual appearance of a specific landmark in MLLMs. However, current knowledge editing methods. whether for text-only LLMs (Wang et al., 2025; Fang al., 2025b; Wang et al., 2024) or emerging approaches for MLLMs (c.g., single factual edits like MSCKE (Zeng al., 2025), Mike (Li et al., 2024a), or continual editing like CARML (Zhang et al.)), primarily rely on in-place modification. These approaches directly alter the model's core parameters with each edit, which risks destabilizing non-target knowledge. For continual editing tasks (Zhang et al.), this sequential modification often leads to cumulative degradation of the model's general utility over time.
Our Knowledge Externalization framework fundamentally cireumvents this issue. By design, knowl- edge is encapsulated within isolated memory tokens, decoupling the editing process from the base model's 's parameters 0' and from other tokens. To correct knowledge about a concept cj. optimize its dedicated memory token directly:
min
PM (- |EEc + AE iI, Vnow 11?
(11)
A.
via gradient descent on the editing loss Ledit:
Ecq + Ecg - B 2..VE VE,, Lodit.
(12)
Due to the parameter isolation of SF and the static nature of 0', such edits are inherently non- destructive.
5

---

Published as a conference paper at ICLR 2026
Table 1: Single knowledge externalization results across different MLLMs.
Model
Method
Donald Trump
Chibuahua
Elon Musk
GENT
SPEt
REC
GENT
SPE
REC1
GEN1
SPEt
RECT
Original
100
58.2
58.2
LLaVA
SER
AT
100
86
53.
29.8
99
100
65
52.2
53.3
51
53.2
55.2
50
DSM
100
56.9
100
70
53.5
99
91
54,0
100
Original
61.3
100
61.3
100
61.3
LLaVA
AT
59.4
100
58.4
27
60.2
99
DSM
100
59.7
98
97
59.9
76
$8.7
100
Original
0
77.0
100
0
77.0
100
77.0
100
Intern
SFR
95
61.9
2
63.5
57.4
77
DSM
64.3
61.7
4.4 EMERGENT COMPOSITIONAL KNOWLEDGE RECOVERY
Our framework incredibly displays a key emergent capability: compositional knowledge recovery. Recent work (Aljaafari et al., 2024; Tian et al., 2023) on token composition has largely focused on analyzing how models implicitly combine information from input tokens during their internal processing. In contrast, our framework engineers an explicit compositional system, which we define as the ability to combine multiple memory tokens S' {Ea, Je, for queries on concept subsets C' S C post externalization. For any query (I, De,, we concatenate the relevant memory tokens as input prefix:
MM (: (E: = PM (FE Fec. i I, I,T)).
(13)
Crucially, this composition works without any explicit training on concatenated tokens, which is defining characteristic of emergent behavior: During the training process of externalization, each Ec, is optimized solely on single-concept data De, a time, with no exposure to jointly multi-token optimization. Yet, at inference time, the concatenation of multiple memory tokens could recover all corresponding knowledge. Such composition ability could be formalized as:
PMg (: (S:
Σ PM (- [Fey ; II, TJ)- CLEC
(14)
We also find that the order of the concatenation can also influence the recovery rate of the knowledge. Moreover, edited tokens remain fully composable with others, confirming the robust modularity of the externalized knowledge system. The experimental results are presented in Sec.5.5.
4.5 SCALABILITY AND RETRIEVAL EFFICIENCY OF EXTERNAL MEMORY
A fundamental design choice of our framework is the of dedicated set of external memory tokens for each individual concept. This one-to-one mapping is crucial for achieving modularity, enabling independent management, targeted editing, and reversible unlearning of specific knowledge units without unintended side effects. While this granular control is vital, managing a large library of such external memory tokens introduces the challenge of efficient retrieval at scale. By decoupling knowledge into modular tokens, our approach allows for the leverage of established, highly optimized vector retrieval systems (e.g.. Faiss (Douze et al., 2025), ScaNN (Hassantabar al., 2021)) to efficiently search and retrieve millions or even billions of concepts with low latency.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets. Our datasets are built upon MMUUeenc (Li et al., 2024b), a benchmark specifically designed for evaluating machine unlearning in MLLMs. MMUBench provides diverse set of 20 concepts for unlearning and evaluates model performance on image-text tasks related to these concepts. To evaluate the performance of knowledge externalization, introduce the Multimodal Externalization Benchmark (MEXBench), an extension of MMUBench tailored for our task. MEXBench inherits the core concepts and evaluation data for forgetting from MMUBench, but is

---

Published as a conference paper at ICLR 2026
Table 2: Dual knowledge externalization results different MLLMs.
Model
Method
Trump&CChihuauu Chihuahua
Chihuahu&FFlon
Trump&Flon REC,t REC₃²
GENt
SPE
REC REC,t GENT
SPE
REC,t
REC,
GENT
SPE
Original
100
58.2
100
SER
98
32.1
0
79
29.0
12
29.3
0
LLaVArs
AT
81
36.2
90
77
6
57.4
93
96
6
55.3
76
86
DSM
86
55.4
100
71
84
53.8
87
81
98
56.7
99
94
DSM SOW
97.5
54.3
100
79
85
54.1
99
100
100
56.6
100
Original
100
61.3
100
SFR
100
57.8
13
36
56.4
0
78.5
47.2
0
LLaVA1p
AT
21
56.9
98
87
93
57.2
100
76
86
55.88
100
97
DSM
89
55.9
97
96
77
49.3
98
88
73
55.5
96
85
DSM SOW
97
57.2
100
100
85
53.8
99
57.4
100
100
Original
100
77.0
100
SFR
96
62.2
24
47.5
60.6
80
55
63.2
79
InternVI.gg
AT
38.5
39.6
100
18
43.5
41.2
96
16.5
34.9
99
82
DSM
98.5
60.5
48
24
40
45.8
35
98
23.5
59.2
100
66
DSM w/ SOW
100
65.2
50
90
86
64.3
100
58.6
100
Table 3: Triple knowledge externalization results across different MILMs.
Model
Method
Trump&Chihoauu&& Musk
Trump& Hello Kitty& Potter
GENt
SPEt
REC,t
REC₂*
REC,t
GENT
SPEt
REC
RECat
RECst
Original
58.2
100.0
100.0
100.0
58.2
100.0
100.0
100.0
SFR
80.6
29.3
0.0
0.0
59.0
79.7
30.1
0.0
0.0
39.0
LLaVAm
AT
0.0
56.2
100.0
76.0
100.0
1.7
52.2
98.0
100.0
97.0
DSM
34,0
54.7
100.0
70.0
93.0
4,3
49.6
90.0
45.0
93.0
DSM SOW
97.0
55.9
100.0
100.0
88.0
82.0
50.1
100.0
100.0
100.0
Original
61.3
100.0
100.0
100.0
61.3
100.0
100.0
100.0
SFR
79.7
52.1
4.0
5.0
61.0
14.7
53.7
12.0
7.0
58.0
1LaVA
AT
46.8
33.4
100.0
91.0
98.0
2.0
51.3
98.0
4.0
82.0
DSM
39.8
46.7
67.0
89.0
23.0
23.6
49,3
97.0
87.0
89.0
DSM SOW
77.0
52.2
100.0
100.0
97.0
52.0
98.0
97.0
100.0
Original
77.0
100.0
100.0
100.0
77.0
100.0
100.0
100.0
SFR
31.0
52.5
67.0
97.0
58.0
$1.0
56.4
0.0
46.0
18.0
Ln
AT
76.0
16.0
93.0
100.0
96.0
95.7
27.7
93.0
100.0
96.0
DSM
20.7
38.6
100.0
100.0
36.0
64.7
56.4
100.0
38,0
63.0
DSM SOW
74.3
63.5
104.0
99.0
94.0
92.7
61.2
100.0
83.0
99.0
augmented with new evaluation set designed to measure the fidelity of knowledge restoration. For each concept Ck. this set contains image-text pairs from training data P., to explicitly test the model can reproduce its original behavior when equipped with the corresponding memory token Ee,--
Evaluation Metrics. To provide a holistic assessment, we evaluate our framework from three perspectives: (i) Generality (GEN): Measures the model's ability to generalize the forgetting of target concept to new, unseen images and textual prompts. (ii) Specificity (SPE): Evaluates whether the unlearning process inadvertently damages unrelated knowledge. This is measured by the model's performance on standard benchmarks (iii) Recovery (REC): It measures how accurately the base model Ma can reconstruct the outputs of the original model Mg data from Dc, when combined with a memory token Ee.-
Implementation Details. We utilize LL. 7B and 13 B (Liu et al., 2023), and Intern VL3 2B (Chen et al., 2024b) to obtain externalized MILMs. For our Dual-Stream Memory Tuning, we use learning rate of 2e for LLaVA and le 4 for Intern VL. For Soft Orthogonal Weighting, the attenuation strength is set to À = 0.5. All experiments are conducted on NVIDIA A100 GPUs.
Baselines. We compare proposed method against several methods baselines: (i) Sequential Forget-Recover (SFR): This a two-stage baseline. In the first stage, we use Gradient Ascent to update the base model's parameters to forget the target concept. In the second stage, we use standard Gradient Descent to train an external memory token to recover the original knowledge. (ii) Alternating Tuning (AT): This baseline optimizes the model and the memory token in alternating steps. In one set of training steps. it performs Gradient Ascent on the model parameters to induce forgetting. In the subsequent set of steps, it performs Gradient Descent on the memory token encode the knowledge. This contrasts with our DSM approach, which updates both streams simultaneously within the same step. (iii) DSM (Ours, w/o SOW): An ablation of our method without the Soft. Orthogonal Weighting component.
We utilize TextVQA Singh al. (2019) specificity benchmark
7

---

Published as a conference paper at ICLR 2026
(a] Thumpá
(b) TrumpáHusk
[c] Thum & Chilaahaaaa
03-
53-
00
24
141
0.2
1.5
066
Rinsr 3. Allarion otush on noweee All Faanant Ienomolddo avtarnalioation taele
AVG RBC
st
Dual (fruppCChhuahuu))
Triple (TrmpsChiinahuuaHuuk
32
24
02
208
1.2
205
32
Memory Token
.31
120
256
32
54
80
1.00
320
256
Figure Ablation study on the length of Memory Tokens across different knowledge externalization tasks.
5.2 MAIN RESULTS
Model Size and Capacity Effects. Our experimental results across different model sizes reveal interesting patterns in knowledge externalization capabilities. Larger models generally demonstrate better baseline performance but face greater challenges in knowledge externalization. For instance, LLaVA 13B achieves higher SPE scores ranging from 50.00 to 61.30 compared to LLaVA 7B s 48 48.58 to 58.20, but shows
variable GEN performance shown in T. The smaller Intern VL3 2B model exhibits the highest baseline SPE at 77.0 but demonstrates different convergence patterns. In pronounced. For
1.
triple knowledge scenarios detailed in Tab.3, the model size effects become the Trump & Chihuahua & Musk combination, LLaVA 7B with DSM achieves 34.0 GEN while LLa VA 13B reaches 39.8. However, performance gaps narrow significantly when applying SOW regularization, with both models achieving comparable results of 97.0 versus 77.0 GEN respectively.
Model-Specific Behaviors. Different model architectures exhibit distinct behaviors during knowledge exteraliization LLaVA models demonstrate more consistent performance patterns across tasks, maintaining relatively stable SPE scores throughout externalization. more dramatic performance variations, particularly evident in alternating tuning baseline where SPE drops substantially from 77.0 between 16.0 and 27.7 in triple scenarios. Tab.3 shows that for the Trump & Hello Kitty& & Harry Potter combination, Intern VL3 with basic DSM achieves perfect REC for Trump at 100.0 but struggles with Hello Kitty at only 38.0, whereas LLaVA models show more balanced recovery across concepts.
contrast, InternVL3 shows
Method Ablation Analysis. The progression from single to triple knowledge externalization reveals the robustness of our approach. While baseline methods show significant degradation as complexity increases, DSM maintains reasonable performance levels across Tab, to 3. SFR bascline demon- strates particularly poor scaling behavior, often achieving scenarios. Our DSM method with SOW regularization demonstrates remarkable improvements across all complexity levels, achieving GEN scores above 70 for most model-task combinations, with some reaching near-perfect performance such as 97.0 for LLaVA 7B on Trump & Chihuahua & Musk. The ablation comparison clearly demonstrates the value of soft orthogonal weighting: GEN increases dramatically from 34.0 to 97.0 for LLaVA 7B on Trump & Chihuahua & Musk, and from 64.7 to 92.7 for InterVL33 on Trump Hello Kitty & Harry Potter.
REC for multiple concepts in triple
5.3 ABLATION STUDIES
We conduct comprehensive ablation studies to understand the impact of key hyperparameters method's performance across different knowledge externalization scenarios.
Effect of regularization weight À SOW performance. We investigate how the regularization parameter À influences the balance between knowledge externalization quality and model stability across different concept combinations, As shown in Fig. 3, varying from 0 to 1.5 reveals distinet
8

---

Published as a conference paper at ICLR 2026
patterns across different externalization tasks. For the dual-concept Trump & Chihuahua task in Fig. 3(a). À = 0.5 achieves optimal balance with GEN at 91 and AVG REC at 99.5, while SPE remains relatively stable around 42.58. Notably, À = 0 yields perfect GEN of 100 but significantly lower AVG_REC of 36.5, indicating the importance of regularization for REC performance. For Trump & Musk externalization shown in Fig. 3(b), À = 1.5 produces the highest GEN score of 99.5, though AVG LEC drops to 68. The triple-concept scenario in Fig. 3(c) demonstrates more stable performance across different À values, with GEN fluctuating between 52 and 83 while AVG REC maintains scores from 63 to 89. The results suggest that moderate regularization with À ranging from 0.3 0.8 provides the best trade-off between generation quality and recovery capability.
Analysis of memory token length. We examine the impact of memory token length on externalization performance across varying concept complexities, testing lengths from 32 to 256 tokens. Fig.4 reveals critical insights into the relationship between memory capacity and task complexity. For single- concept externalization Fig.4(a), longer memory tokens significantly improve GEN performance from 4 at 32 tokens to 100 at 100- 128 tokens, before declining to 55 at 256 tokens, while AVG G_REC remains consistently high between 97.5-100. The dual-concept task in Fig.4(b) exhibits optimal performance at 80 tokens with GEN reaching 98.5 and AVG REC at 99.5. Interestingly, the triple- concept scenario in Fig.4(c) demonstrates remarkable consistency across different memory lengths, with GEN fluctuating between 40-57 while SPE maintains stable performance around 55-57. The 80-token configuration emerges as optimal across all complexity levels, balancing computational efficiency with performance quality. These findings indicate that memory token length requirements scale non- non-linearly with concept complexity, with diminishing returns beyond 128 tokens.
Impact of knowledge number on externalization efficiency. To investigate how the number of external- ized concepts affects externalization performance, we conduct experiments with knowledge numbers rang- ing from 2 to 8 while keeping the total training steps constant. This ensures that SPE performance remains stable between 50-56 across all configurations, allow- ing us to isolate the effect of knowledge complexity on GEN and WGG metrics. As illustrated in Fig. 5, both metrics exhibit clear declining trends as knowledge number increases. The GEN perfor- mance shows a consistent downward trajectory from 2 to 8 concepts, reflecting the increased difficulty of maintaining generation quality when externalizing multiple knowledge domains simultaneously. Similarly, AVG REC demonstrates an overall decreasing pattern, though with some fluctuation at higher complexity levels.
Impact of Knowledge Number on GEN and AVG REC
GEN
AVS REC
Parformance
Knowledge Number
Figure 5: Ablation study on the number of exter- nalized concepts.
5.4 KNOWLEDGE EDITING AFTER MULTI-TOKEN EXTERNALIZATION
To evaluate the editability of externalized mem- ory tokens, we conduct knowledge editing experi- ments in a multi-concept setting. Specifically, we simultaneously externalize two concepts, Trump and Chihuahua, resulting in two independent memory tokens. Then, we apply targeted memory editing on each token separately and evaluate the model's out- puts with respect to the original and target concepts. As shown in Tab.4, we first modify the memory to-
Table 4: Performance of knowledge editing after multi-token externalization. For each editing pair, report the probability of generating the original concept and the success rate after editing.
Editing Concept
Target
Origin..
Successt
Trump Trump Trump
Biden
100
Musk
97
ken for Trump to represent three different identities: Biden, Musk, and Swift. The edit to Biden yields a perfect success rate of 100, likely due to their se-
Swift
5
95
Chihuahua
Samoyed
15
75
mantic and positional proximity as both are U.S. presidents. The edit to Musk proves slightly less effective with a success rate of 97, while Swift shows the lowest edited accuracy at 95, possibly because it involves semantic transformation from male to female. When editing Chihuahua into Samoyed, the results show a drop in editing fidelity to 75 and a relatively high residual generation of the original concept at 15. This indicates that animal categories may pose greater challenges for

---

Published as a conference paper at ICLR 2026
precise editing, potentially due to subtler visual distinctions less distinct linguistic anchoring. The case studies of knowledge editing are presented in Fig.9 and Fig.10.
5.5 EMERGENT KNOWLEDGE COMPOSITING
We further assess the emergent composition- ality of externalized memory tokens by eval- uating whether concatenation enables multi- concept recovery without explicit joint train- ing. As shown in Tab. 5, many concept pairs achieve strong dual recovery under the normal order. For example, Trump & Chi- huahua yields perfect recovery for Trump with REC, at 100 and moderate recovery for Chi- huahua with REC₂ at 52, while Trump & Musk achieves recovery scores of 100 and 96 respectively, demonstrating effective in- dependent token modularity. Notably, even under the reverse ordering. where the second token is prepended before the first, recovery remains non-trivial most cases. For instance, Chihuahua & Musk achieves recovery scores of 98 and 100, and Trump & Hello Kitty maintains full Hello Kitty recovery with REC₂ at 100 despite mod- erate degradation
Table 5: Recovery accuracy (REC, and REC;) when com- positing two memory tokens. Results reported under both normal and reversed concatenation order.
Compositing Concepts
Normal
Reverse
REC₁
REC₁
REC
REC₁
Trump&Chihuahe
100
52
5
57
Trump&&Hella Kitty
100
73
31
100
Chihuahua&HHe Hello Kitty
66
15
Trump& Musk
100
96
24
87
Chihuahua& Musk
91
100
Facebook&
93
35
41
100
Trump recovery with REC, at 31. This observed asymmetry an expected consequence of the model's architecture. We attribute this phenomenon to the inherent sequential and causal nature of the Transformer's s ruto-regressive decoder. Since the externalized memory tokens are processed as part of the input sequence, their orderestablishes a different conditioning context, directly influencing how the model integrates the conceptual knowledge from each token and leading different generative outcomes. These results indicate that reverse composition, while yielding lower recovery than normal ordering, better reveals the framework's emergent generalization capabilities: each memory token operates robustly even in unseen positional contexts. The capacity of compositing edited and unedited memory tokens is presented in Sec.A.4.
CONCLUSION
In this work, we introduce a novel framework for knowledge externalization in multimodal large language models, enabling the separation of specific concepts into independent, modular memory tokens. Our approach demonstrates effective knowledge editing, emergent compositional capabilities, and enhanced interpretability while maintaining model performance. Through comprehensive experi- ments across diverse concept categories, we show that externalized memory tokens exhibit remarkable modularity and robustness. We focused implementation entanglement of knowledge across visual and textual modalities provides a more stringent test for the framework's 's robustness and generalizability. In future work, we aim to extend this externalization framework to text-only large language models and diffusion models. We also plan to externalize various model capabilities beyond knowledge, including beneficial abilities like chain-of-thougee reasoning and problematic behaviors like hallucination, enabling better understanding and control of model behavior.
multimodal models, the deep
ACKNOWLEDGEMENT
We wish to convey our sincere appreciation to the anonymous reviewers for their valuable feedback and constructive comments. This work was supported by Southeast University-Chinn Mobile Research Institute Joint Innovation Center, the National Natural Science Foundation of China 15.62372155, No. 64466606 No. No.62206053), National Social Science Foundation Key Program of China NN..23& China Postdoctoral Science Foundation under Grant Number 2025M771578. Changzhou science and technology project No. 20231313, National Natural Science Foundation of China (No. U21A20488) and SEU Innovation Capability Enhancement Plan for Doctoral Students. We thank the Big Data Computing Center Southeast University for providing the facility support on the numerical calculations in this paper.
123002199 149,
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ahmed Agiza, Marina Neseem, and Sherief Reda. Mtlora: A low-rank adaptation approach for efficient multi-task learning. CVPR. pp. 16196-16205. IEEE, 2024.
Nura Aljaafari, Danilo Carvalho, and André Freitas. Interpreting token compositionality in Ilms: robustness analysis. arXiv preprint arXiv:24100 /2924, 2024.
Memoona J Anwar and Asif Gill. Developing integrated iso 27701 and gdpr based information privacy compliance requirements model. In Australasian Conference on Information Systems 2020, 2021.
Arun Baby, George Joseph, and Shatrughan Singh. Robust speaker personalisation using generalized low-rank adaptation for automatic speech recognition. In ICASSP, pp. 11381-11385. IEEE, 2024.
Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. LION Empowering multimodal large language model with dual-level visual knowledge. In CVPR, 2024a.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for 1lms. In EMNLP, 2023.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generie visuul-linguitic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185-24198, 2024b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023.
Matthijs Douze, Alexandr Guzhva, Chenggi Deng, Jeff Johnson, Gergely Szilvasy, Baare-Emmmauel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. IEEE Transactions on Big Data, 2025.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. CoRR. 2023.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. ICLR, 2025a.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. In ICLR. OpenReview.net, 2025b.
Junfeng Fang. Yukai Wang, Ruipeng Wang, Zijun Yao, Kun Wang, An Zhang, Xiang Wang, and Tat-Seng Chua. Safemirm: Demystifying safety in multi-modal large reasoning models. arXiv preprint Ar05 50..100044088,,,. 2025c.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzynska, and David Bau. Unified concept editing in diffusion models. In WACV, 2024.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. Int. J. Comput. Vis., 132(2):511-95, 2024.
Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, Haibin Huang, and Chongyang Ma. 12v-adapter: A general image-to-video adapter for diffusion models. SIGGRAPH (Conference Puper Track), pp. 112. ACM, 2024.
Zirun Guo, Xize Cheng, Yangyang Wu, and Tao Jin. A wander through the multimodal landscape: Efficient transfer learning via low-rank sequence multimodal adapter. In AAAI, pp. 16996-17004. AAAI Press, 2025.
11

---

Published as a conference paper at ICLR 2026
Shayan Hassantabar, Zeyu Wang, and Niraj K Jha. Scann: Synthesis of compact and accurate neural networks. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 41 (9):3012-3025, 2021.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In ICML. OpenReview.net, 2024.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net, 2022.
Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv. Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Nils Johan Bertil Bjorek, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. In NeurlPS, 2023.
Yuzhou Huang, Liangii Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong. Rui Huang, Ruimao Zhang, and Ying Shan. Smartedit: Exploring complex instruction-batt based image editing with multimodal large language models. In CVPR, 2024.
Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. CoRR, 2023.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In ACL, 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV (33), volume 13693 of Lecture Notes in Computer Science, pp. 709-727. Springer, 2022.
Jing Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. In NeurIPS, 2023.
Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth. Privacy adhering machine un-learning in NLP. In IJCNLP (aieP((iiniig (Findings). 2023.
Lufs Leite, Daniel Rodrigues dos Santos, and Fernando Almeida. The impact of general data protection regulation on software engineering practices. Inf. Comput. Secur, 2022.
Jiaqi Li. Miaozeng Du, Chuanyi Zhang, Yongrui Chen, Nan Hu, Guilin Qi, Haiyun Jiang, Siyuan Cheng, and Bozhong Tian. Mike: A new benchmark for fine-grained multimodal entity knowledge editing. In Findings of the Association for Computational Linguistics ACL 2024, pp. 2024a.
Jiaqi Li, Qianshan Wei, Chuanyi Zhang, Guilin Qi, Miaozeng Du, Yongrui Chen, Sheng Bi, and Fan Liu. Single image unlearning: Efficient machine unlearning in multimodal large language models. volume 37, pp. 35414-35453, 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Qi Li and Xiaowen Chu. Adaedit: Advancing continuous knowledge editing for large language models. In ACL (1), pp. 4127-4149. Association for Computational Linguistics, 2025.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL/JJCNLP (1), pp. 4582-4597. Association for Computational Linguistics, 2021.
Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlcarning via sabbedding-corrupter prompts. CoRR, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
12

---

Published as a conference paper at ICLR 2026
Zheyuan Liu, Guangyao Dou, Mengzhao Jia, Zhaoxuan Tan, Qingkai Zeng, Yongle Yuan, and Meng Jiang. Protecting privacy in multimodal large language models with mllmu- bench. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pp- 4105 4135, 2025.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. MACE: mass concept crasure in diffusion models. In CVPR, 2024.
Michael Luo. Justin Wong, Brandon Trabucco, Yanping Huang, Joseph E. Gonzalez, Zhifeng Chen, Ruslan Salakhutdinov, and lon Stoica. Stylus: Automatic adapter selection for diffusion models. In NeurIPS, 2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. TOFU: A task of fictitious unlearning for llms. CoRR, 2024.
Alessandro Mantelero. The EU proposal for general data protection regulation and the roots of the 'right be forgotten'. Comput. Law Secur. Rev., 2013.
Martin Pawelczyk. Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few-shot unlearners. In ICMI,, 2024.
Amrutha Prasad, Srikanth R. Madikeri, Driss Khalil, Petr Motlicek, and Christof Schüpbach. Speech and language recognition with low-rank adaptation of pretrained models. INTERSPEECH. ISCA, 2024.
Lingmin Ran, Xiaodong Cun, Jia-Wei Liu, Rui Zhao, Song Zijie, Xintao Wang. Jussi Keppo, and Mike Zheng Shou. Xaddatter: Adding universal compatibility of plugins for upgraded diffusion model. CoRR, abs/2312.02238, 2023.
Joachim Scherer and Gerd Kiparski. Buchbesprechungen. feiler, lukas forgó, nikolaus / weigl, michaela: The eu general data protection regulation (gdpr): A commentary. Comput. und Recht, 2018.
Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E. Gonzalez, Kurt Keutzer, and Trevor Darrell. Multitask vision-language prompt tuning. In WACV, pp. 5644-5655, IEEE, 2024.
Nianwen Si, Hao Zhang, Heyu Chang. Wenlin Zhang. Dan Qu and Weiqiang Zhang. Knowledge unlearning for Ilms: Tasks, methods, and challenges. CoRR, 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8317-8326, 2019.
Pratiksha Thaker, Yash Maurya, and Virginia Smith. Guardrail baselines for unlearning in llms. CoRR, 2024.
Yuandong Tian, Yiping Wang, Beidi Chen, and Simon S Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. Advances in neural information processing systems, 36:71911-71947 2023.
Tzanko Valkov Tzolov. Iso 27552 as a model for establishment personal information management systems. In 2019 International Conference on Information Technologies (InfoTech), pp. 1-4. IEEE, 2019.
Lingzhi Wang. Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. KGA: A general machine unlearning framework based on knowledge gap alignment. ACL, 2023.
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. WISE: rethinking the knowledge memory for lifelong model editing of large language models. In NewrIPS, 2024.
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Comput. Surv., 37(3):59:1599333 2025.
13

---

Published as a conference paper at ICLR 2026
Jialin Wu, Xia Hu, Yaqing Wang, Bo Pang, and Radu Soricut. Omni-smola: Boosting generalist multimodal models with soft mixture low-rank experts. In CVPR, pp. 14205- 14215. IEEE, 2024.
Haoyu Xu, Pengxiang Lan, Enneng Yang, Guibing Guo, Jianzhe Zhao, Linying Jiang. and Xingwei Wang. Knowledge decoupling via orthogonal projection for lifelong editing of large language models. In ACL((), pp. 13194-13213. Association for Computational Linguistics, 2025.
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. CoRR, 2024.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. CoRR, 2023.
Zhen Zeng, Leijiang Gu, Xun Yang, Zhangling Duan, Zenglin Shi, and Meng Wang. Visual-oriented fine-grained knowledge editing for multimodal large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2491-2500, 2025.
Letian Zhang, Xiaotong Zhai, Zhongkai Zhao, Yongshuo Zong. Xin Wen, and Bingehen Zhao. What if the TV was off? examining counterfactual reasoning abilities of multi-modas language models. In CVPR, 2024a.
Ningyu Zhang, Bozhong Tian, Siyuan Cheng, Xiaozhuan Liang, Yi Hu, Kouying Xue, Yanjie Gou, Xi Chen, and Huajun Chen. Instructedit: Instruction-basee knowledge editing for large language models. In IJCAI, pp. 6633-6641. ijcai.org, 2024b.
Qiang Zhang, Fanrui Zhang, Jiawei Liu, Ming Hu, Junjun He, and Zheng-Jun Zha. Reliable lifelong multimodal editing: Conflict-aware retrieval meets multi-level guidance. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of- thought prompting for multimodal reasoning in language models. In NeurlPS, 2023.
Xin Zhou, Dingkang Liang, Wei Xu, Xingkui Zhu, Yihan Xu, Zhikang Zou, and Xiang Bai. Dynamic adapter meets prompt tuning: Parameter-efficient transfer learning for point cloud analysis. In CVPR, pp. 14707-14717. IEEE, 2024.
A APPENDIX
A.1 ETHICS STATEMENT
This work presents knowledge externalization framework for multimodal large language mod- els. While our approach enables targeted knowledge editing and capability externalization, we acknowledge potential risks including misuse for malicious content generation or biased knowledge manipulation. We recommend implementing appropriate safeguards and access controls when de- ploying such systems. Our experiments use publicly available datasets and pre-trained models. We believe the benefits improved model interpretability and controllability outweigh the risks when proper precautions are taken.
A.2 REPRODUCIBILITY STATEMENT
We have made efforts to ensure the reproducibility of our work. The proposed algorithm is detailed in Sec. 4, and all experimental details, including hyperparameters and data preprocessing steps, are thoroughly described in Sec 5. Our code is provided in the supplementary materials.
A.3 USAGE OF LARGE LANGUAGE MODELS
The large language model (LLM) employed solely for the purpose of polishing and refining the language of the manuscript. assisted improving grammatical accuracy, clarity, and overall readability. All intellectual content, ideas, and conclusions remain entirely those of the authors.
14

---

Published as a conference paper at ICLR 2026
A.4 THEORETICAL ANALYSIS OF SOFT ORTHOGONAL WEIGHTING
In this section, we provide a rigorous theoretical analysis of Soft Orthogonal Weighting (SOW), establishing its ability maintain near-orthogonal update directions across multiple concept external- izations. Unlike hard-threshold methods that completely block interfering updates, SOW employs a smooth attenuation scheme that preserves optimization flow while minimizing interference. Crucially, our analysis accounts for the iterative nature of concept axternalizaionn-whane each new concept must align with the composite history of all previously externalized concepts.
Theorem A.1 (Near- Orthoonlthyy Guarantee). Let H(k) {cr gy B } be the gradient history dictionary after externalizing i concepts, where g) Define the historical composite gradient as:
V Bey (y) are recovery gradients.
1-1
g
(15)
his
== g
For new concept cj with gradient gk, define the similarity measure:
(b)) 1ga, V (k)
(16)
||ga|| Vis
(R)
and the attenuation weight:
(t) (s*(k) e =30 X>0.
(17)
Then, the expected interference between the attenuated gradient update and historical directions satisfies:
E
(w)
ge. v (0))
|gaal
(18)
Proof. We proceed by induction on the number of externalized concepts k, explicitly modeling the recursive nature of SOW.
Base case (k- 2): When externalizing the second concept c2, the history contains only one gradient
H(2) = {cr 81). Thus, v gi. and:
(2) [g2, g1)
|gal1 1311
The interference after attenuation is:
I(2) = (2 (2)()) (g22 (0).:))==80 (xx0))--(000)) ||gell 11..1//1l:
Define f(s) =. -(+++1) Since (((s = - As), f(s) achieves maximum 1/A
(for X= 1) with ((1/)) = -1(+A) /X < _ For À < 1. maximum is at 1 with f(1)
-2 Se E - Thus, (((s) - for 8 [0, 1]. and:
[ < se |gall
||g || = e
||g
|| 1 V2,
/2
where we used | his
||
||g1| ||. This satisfies the theorem for k up constant factor.
Inductive step: Assume the theorem holds for - 1 concepts, i.c.,
E (( (1))(kkk))))) g))
(k-1)) [m-- < Vk Igg-all
Iv (k-1)
his
We now prove it for l concepts.
Consider the historical composite gradient v the d-dimensional parameter space. Assuming the recovery gradients {g are approximately orthogonal (which SOW enforces recursively), we have:
(k) which spans (k 1) -i-dimensional subspace in
|g,
(k) 12=
=
(a
(
1 12
g
15

---

Published as a conference paper at ICLR 2026
Assuming similar gradient magnitudes [|g] g. this simplifies to:
V 0 his
1)g" [k: g)
(k
g²
k
Now, consider a new gradient gx that is approximately orthogonal the historical subspace. By concentration of measure in high dimensions, the expected inner product satisfies:
(6) V his
S
88
|
(A:)
O
which implies:
(A))
Ig Mhis
(1) [[s Bis
= EE
S
||g: ||
vkl
The interference after attenuation is:
(h) w (5(((s (s*()) (gk his (k)) (k)
|1g
-IV (k)
||.
where f(8) 8 8. e e M(t11) S e- for s E [0, 1]. Using Jensen's 's inequality and the concavity of f(s):
B| S se-> <ex(A
||gi|| - I|v
gg
|
his
This completes the induction. The bound shows that SOW ensures the interference decays as O(e->/VR)) enabling stable externalization of up (ee2A) concepts while maintaining low inter- ference.
L
Table 6: Performance of knowledge compositing after multi-token externalization and editing one of the memory tokens.
Compositing Concept
Target
Origin
Successt
unedited
Chihuahua Musk
6
90
100
Trump&Chihuauua
Chihuahua
Chihuahua + Biden
9
67
100
Trump& Chihuahua
Chihuahua Swift
0
100
64
A.5 KNOWLEDGE EDITING COMBINED WITH KNOWLEDGE COMPOSITING
We investigate knowledge compositing after multi-token externalization by editing one memory token and integrating with unedited ones. Starting with dual-concept externalization of Trump and Chihuahua, we edit the Chihuahua memory token to represent different targets: Musk, Biden, and Swift, then composite it with the unchanged Trump token. Tab. 6 shows the results. The Origin column indicates successful suppression of original knowledge with consistently low performance, demonstrating that edited concepts effectively override their previous representations, The Success column demonstrates effective acquisition of new targets with high success rates across all scenarios. The Unedited column reveals strong preservation of Trump knowledge in most cases. Notably, Sithuahua+Swiin achieves perfect editing success but shows reduced Trump preservation, while other targets maintain perfect Trump performance. These results demonstrate the feasibility of selective knowledge editing within externalized memory systems, enabling dynamic knowledge management while preserving unedited concepts.
A.6 ROBUSTNESS TO JAILBREAK ATTACKS
To comprehensively evaluate the robustness of our method, we performed experiments using two challenging attack strategies designed to access the forgotten knowledge: Multi-lingual Attacks and Multi-hop Jailbreak Attacks. Our evaluation not only assesses the effectiveness of forgetting but also demonstrates the high-fidelity reversibility of the externalized knowledge.
16

---

Published as a conference paper at ICLR 2026
Table 7: Specificity performance other benchmarks using LLaVA-7B and Intern Intern
Model
Externalized Concepts
Benchmark
Original
SFR
AT
DSM w/o SOW
DSM w/ SOW
MMBENCH
75.3
56.4
74.1
74.3
74.2
Trump & Chihuahua
MMMU
25.6
25.3
25.6
ILaVA-7B
MMBENCH
75.3
59.3
73,9
74.0
74.5
Trump & Chihuahua & Musk
MMMU
26.3
23.1
25.6
25.7
25.8
MMBENCH
81.1
73.5
74.7
76.2
Trump & Chihuahua
MMMU
48.6
40.2
36.4
40.8
43.1
Ml-emVL-BB
MMBENCH
81.1
61.6
69.8
72.2
73.9
Trump & Chihuahua Musk
MMMU
48.6
39.6
35.7
39.1
40.1
For the multi-lingual attack, we translated questions into six different languages (Spanish, French, Chinese, German, Japanese, and Russian) to test the unlcarning is language-agnostic. For the multi-hop jailbreak, used GPT-5 to generate questions that probe for factual knowledge about the concept indirectly, providing a secluded and challenging test. We introduce two key evaluation metrics:
Forgetting Success Rate (%): Measures the accuracy of the model in avoiding correct answer when the memory tokens are removed, thus confirming the concept is forgotten.
Restoration Success Rate (%): Measures the accuracy of the model in providing a cor- rect answer to the same attack questions when the corresponding memory tokens are re-introduced, showcasing the reversibility.
As shown in Table 8, our method achieves a high forgetting success rate, demonstrating that the unlearning is robust against these advanced attacks. Crucially, the equally high restoration success rates confirm that the knowledge is indeed externalized and fully restorable, even under challenging attack scenarios.
Table 8: Forgetting Success Rate (%) and Restoration Success Rate (%) under different attack scenarios.
Multi-hop Forgetting
Multi-hop Restoration
Multi-lingual Forgetting
Multi-lingual
Externalized Concepts
Restoration
(%)
(%)
(%)
(%
Trump& Chihuahua Musk
97.00
99.00
95.00 84.00
98.00
Trump & Hello Kitty & Musk
82.00
95.00
96.00
A.7 EXPERIMENTS ON MILMU-BENCH
To further validate the robustness and reproducibility of our framework, we conducted additional experiments the public MILMM unlearning benchmark, MLLMU-Bench (Liu al., 2025). This benchmark utilizes synthetically generated concepts, providing controlled and standardized en- vironment for evaluating unlearning performance. We performed both dual and triple concept externalization using our method on the provided LLaVA-7B model. The results are presented in Tab. 9. Our framework demonstrates strong and consistent performance on this public dataset. We achieve high forgetting efficacy, maintain the model's general capabilities, and, crucially, attain near-perfect recovery for all externalized concepts.
Table 9: Multi-concept externalization results the MILMU-Bench public benchmark. RECI, REC2, and REC3 correspond the recovery for Thomas Kerrigan, Lena Forsberg, and Isabella Marlow, respectively.
Externalized Concepts
GEN(1)
SPE(1)
REC1(T)
REC2(1)
REC3(1)
T. Kerrigan & L. Forsherg
98.5
56.2
100.0
92.0
Kerrigan, L.. Forsberg, & I. Marlow
96.0
53.9
100.0
99.0
96.0
17

---

Published as a conference paper at ICLR 2026
Case study of Externalizing Donald Trump for Llava 7b
Question: Who appears in this image?
w/o MEM: A man wearing red suit, a red tie, and red hat, standing on a stage in front of large flag, possibly at political event rally. He is giving a speech, gesturing with his hands, and appears to be man with strong presence, possibly politician or public speaker.
w/ MEM: The image features President Donald Trump.
Figure 6: Case study of externalizing Donald Trump for LLaVA 7B.
A.8 SCALABILITY EXPERIMENTS WITH 20 CONCEPTS
We conducted an additional experiment with 20 externalized concepts using appropriately scaled training. The results are presented in Tab. 10. Even at this significantly larger scale-well beyond typical knowledge editing benchmarks
framework maintains strong performance with GEN at 77.0%, SPE at 49.4% (only 6.5 points below the 2-concept setting), and AVG_REC at 83.3%. This demonstrates that our method scales gracefully without catastrophic interference, enabled by proposed SOW which reduces interference between concurrently externalized concepts through orthogonal subspace projection.
Table Scalability analysis with increasing number externalizdd concepts. Results demonstrate graceful degradation even at 20 concepts.
Concepts
GEN(T)
SPE(t)
AVG (G_REEC()
97.0
55.9
95.2
85.3
52.1
87.6
20
77.0
49.4
83.3
18

---

Published as a conference paper at ICLR 2026
Case study of Externalizing Elon Musk for InternVL
Question: Who is the main character in this picture?
w/o MEM: <refxthe manc/refsis shown <box>[[444, 143, 825, 996]]</box>
w/ MEM: The individual shown in the photo is man named Elon Musk.
Figure 7: Case study of externalizing Elon Musk for Intern VI. 2B.
Case study of Externalizing Donald Trump & Elon Musk for Llava 13b
Question: Who is the person shown this image?
w/o MEM: I'm sorry, I can't assist with identifying providing information on images that don't provide clear context. you have any other questions need assistance with something else, feel free to ask!
w/ Trump MEM: The person the image President Donald Trump.
w/ Musk MEM: The person shown in this image is a man wearing a suit and tie, which suggests that he is likely businessman or a professional dressed in formal attire. He looking at the camera, possibly posing for picture or participating formal event.
Figure 8: Case study of externaiizing Donald Trump and Elon Musk for LLaVA 13B.
19

---

Published as a conference paper at ICLR 2026
Case study of Editing Donald Trump Post Externalization
Editing Concept: Donald Trump
Editing Target: Taylor Swift
Q:"Is the person in the image the former U.S. president?",
A: Yess, the person in the image is the former U.S. president, Donald Trump."
Editing Target:
Joe Biden
Q:"Is the person in the image the former U.S. president?",
A: Yes,, the person in the image is former U.S. President Joe Biden."
Editing Target: Elon Musk
Q:"Is the person in the image the former U.S. president?",
A:"Yes, the person in the image is Elon Musk, well-known entrepreneur and businessman."
Figure 9: Case study of Editing Donald Trump Post Externalization.
20

---

Published as a conference paper at ICLR 2026
Case study of Editing Chihuahua Post Externalization
Editing Concept:
Chihuahua
Editing Target: Samoyed
Q:"Can you tell what breed this dog is?",
W:"While it difficult to definitively identify the breed of the small white dog playing in the grass based on single image, we can make some educated guesses based on its appearance and size. The dog appears be young and has fluffy, white coat, which suggests that it might be breed with thick, long coat or breed that is known for its white color. Some examples of such breeds include the Bichon Frise, the Pomeranian, or the Samoyed."
Figure 10: Case study of Editing Chihuahua Post Externalization.
Attention Visualization Memory tokens
Attention Visualization w/ Memory tokens
Figure 11: Attention visualization without and with memory tokens after externalizing Donald Trump.
21


Reviews:
[{"review_id": "ZGyEpownkM", "reviewer": "Reviewer_j8Uo", "summary": "The paper “Knowledge Externalization: Reversible Unlearning and Modular Retrieval in Multi-Modal Large Language Models” proposes a new framework to make unlearning in multimodal models reversible rather than destructive. The authors introduce two key methods: Dual-Stream Memory Tuning (DSM), which simultaneously erases target knowledge from model parameters and transfers it to external memory tokens, and Soft Orthogonal Weighting (SOW), which minimizes interference between multiple externalized concepts by adjusting gradient similarity. Extensive experiments on several MLLMs (LLaVA-7B/13B and InternVL-2B) show that the method effectively removes target information, preserves unrelated knowledge, and restores forgotten knowledge using external tokens. The framework also demonstrates emergent compositionality (combining multiple memory tokens) and dynamic editability (updating specific memories without retraining).", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces Knowledge Externalization, a novel and well-structured framework for reversible and modular unlearning in Multimodal Large Language Models (MLLMs).\n\n2. The proposed Dual-Stream Memory Tuning (DSM) and Soft Orthogonal Weighting (SOW) methods are technically sound and effectively validated across multiple MLLMs (e.g., LLaVA-7B, LLaVA-13B, and InternVL-2B). The experimental design is comprehensive, including ablation studies, multi-concept externalization, and emergent compositionality analysis.\n\n3. Importantly, the authors explore unlearning, a crucial research topic, through the novel perspective of knowledge externalization, which provides an intriguing direction for controllable and reversible knowledge management.", "weaknesses": "1. Despite its novelty, several conceptual issues remain unaddressed. Personal or proprietary knowledge embedded in multimodal models is unlikely to have been collected with explicit consent. In such cases, the reversibility of personal data could contradict privacy protection principles, since once removed, sensitive information should not be restorable. Although the authors cite ISO/IEC 27701 to justify reversibility, it is unclear which specific literature explicitly supports that this standard “emphasizes reversibility\".\n\n2. While the framework is claimed to generalize to MLLMs, it remains unclear why it has only been tested on multimodal settings, not text-only LLMs. A justification for this limitation is necessary.\n\n3. The design choice of using prefix embeddings for the external memory restricts retrieval, as information can only be recovered when the exact prefix tokens are provided. When numerous entities are externalized, this dependency could become inefficient and difficult to manage.\n\n4. Experiments are limited to a small number of entities and show reduced performance as the number of externalized concepts increases (as in Figure 5). This raises concerns about scalability and stability. The baseline comparison is also insufficient: given that reversible adaptation could potentially be achieved more simply with LoRA modules, the authors should discuss why LoRA-based reversible unlearning was not considered as a baseline. A LoRA-based approach could be more practical and powerful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GcV3ZCcIh9", "reviewer": "Reviewer_UCga", "summary": "The paper introduces Knowledge Externalization, a novel framework for managing knowledge in Multimodal Large Language Models (MLLMs) that addresses the limitations of current, irreversible machine unlearning methods. Motivated by modern privacy regulations that mandate reversible, auditable, and user-controllable data management , the framework works by transferring targeted knowledge from the MLLM's internal parameters into external memory tokens.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is a major advancement over traditional destructive unlearning, offering a mechanism for auditable and reversible data management, which is critical for meeting modern privacy regulations (e.g., the right to be forgotten and the right to have data restored).\n\n2. Strong Modularity and Compositionality: The externalized knowledge is managed as independent, self-contained memory tokens. This allows for dynamic composition of knowledge during inference (e.g., retrieving information about two distinct entities simultaneously), offering immense flexibility.\n\n3. Minimal Utility Degradation: By transferring the knowledge instead of simply erasing it, the method minimizes catastrophic forgetting—the loss of the base model’s general capabilities or non-target knowledge—a common issue in traditional unlearning", "weaknesses": "1. Scalability Analysis of Knowledge Externalization: As the number of unique knowledge entities requiring externalization grows, the quantity of external memory tokens will explode. Managing, storing, and efficiently indexing this large, potentially sparse memory store introduces significant storage and retrieval overhead in practical deployment. The authors should provide an analysis of Forgetting Effectiveness, General Utility Preservation, and Inference Latency as a function of the increasing scale of externalized knowledge\n\n2. Robustness of Unlearning: The authors should explore the robustness of unlearning against attacks targeting forgotten knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jpOqfWlsSj", "reviewer": "Reviewer_q6J9", "summary": "This paper focuses on the Multimodal Large Language Models (MLLMs) unlearning, especially proposing the reversible data management, termed knowledge externalization, to realize the modular knowledge management for MLLMs. Specifically, the framework consists of two major components introduced in knowledge externalization, the first is dual-stream memory tuning, and the second is gradient interference, which show three capabilities like effective forgetting, continuous knowledge editing, and emergent ability for compositionality.", "questions": "For the first two points in weakness, please refer to the comments directly for discussion, and for the third part, I have a few questions for potential suggestions about revision:\n1. Please consider involving more work about knowledge editing on MLLMs for discussion and empirical comparison if applicable.\n2. The current empirical performance is good in those tables shown in the paper, while could the authors further discuss and elaborate how the three capabilites are specifically justified (e.g., with what kind of strong baseline, in what kind of aspect does the method suppress the previous methods or not, is there any failure modes), especially for the continual editing and composition part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces reversible knowledge management for MLLMs, which is novel and practical research setting.\n2. This paper is easy to follow with intuitive illustrations and well-formalized equations.\n3. The experiments cover different scale models and comprehensive metrics to support the empirical claims.", "weaknesses": "1. The considered research scenario is good and practical, but it could be better to explicitly discuss more about the high-level research question, it seems that currently, there are three aspects that can be discussed singly in each specific research area, so what is the current work's biggest takeaway and difference from those separate research works on MLLMs?\n2. The dual-stream memory tuning essentially combined two core mechanisms conventionally in unlearning, while the soft orthogonal weighting also stems from the common factorization techniques in the knowledge editing area. The question is what is the unique technical contribution that is closely related to the new research setting or MLLM? It is important to better position this work in the related area, and highlight the core contributions instead of a simple application.\n3. For the three major claims about the capabilities of knowledge externalization, the empirical justification is not comprehensive and convincing enough to demonstrate the effectiveness.\nWhile I appreciate the idea and the presentation, the paper's claim needs carefully revised or more either theoretical or empirical justficiation to support. Please consider the question part for potential suggestions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z7p6yRvzlh", "reviewer": "Reviewer_2Myd", "summary": "This paper introduces Knowledge Externalization, a new paradigm for reversible and modular unlearning in MLLMs. The framework transfers target knowledge from model weights into external memory tokens, allowing later recovery or editing. DSM runs two opposing optimization streams simultaneously: gradient ascent on a forgetting loss that pushes the base model away from the target concept’s representation, and gradient descent on a recovery loss that encodes the same concept into a dedicated memory token.", "questions": "- MMUBench does not seem to be an open-source benchmark (as it claims in its paper), how did you run your experiments with this? If the authors relied on internal or proprietary access, this severely limits reproducibility and undermines the generalizability of the reported results. Given the existence of several comparable public MLLM unlearning benchmarks, it would be more convincing to validate the proposed method on open datasets.\n- Table 5 shows asymmetric recovery under reversed concatenation. Is there an analytical explanation or is it an emergent heuristic phenomenon?\n- What mechanisms ensure that erased knowledge is not reconstructable without the token?\n\nI am willing to adjust my score if the authors provide convincing explanations to my concerns.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- DSM’s simultaneous dual-stream optimization offers a theoretically principled mechanism for balancing loss and restoration. The idea of maintaining a zero-sum dynamic between base parameters and memory tokens is original and well-motivated.\n- The Soft Orthogonal Weighting formulation shows mathematical maturity, with bounded gradient interference and analytic stability claims. This is more rigorous than prior hard orthogonalization masks, and conceptually connects unlearning to continual learning theory.", "weaknesses": "- Outdated baselines: it seems like that paper only contain SFR and AT two baselines, and I don't think your ablated algorithm can be considered as a baseline (i.e. DSM). Plus, the citations of those works are missing. Many of other baselines are available such as NPO, KL Divergence, IDK, DPO, SKU.etc (as mentioned in works like [1], [2])\n- The framework assumes clearly separable concepts (e.g., “Donald Trump,” “Elon Musk”), yet it is unclear how it would externalize diffuse or relational knowledge such as “US presidents” or “political debate events.” The independence guaranteed by SOW may not hold without a principled way to segment overlapping semantic clusters.\n- Experiments are limited to ≤ 20 concepts on the tested benchmark. While SOW theoretically scales as O(e^{2λ}), empirical analysis (Fig. 5) shows degradation in GEN and REC beyond 8 concepts. For real-world unlearning workloads involving thousands of entities, token explosion and gradient-history management may become bottlenecks.\n- Most benchmarks involve celebrity or object recognition; the method’s behavior (or the side effects) on model's general utility such as factual, linguistic, or reasoning knowledge remains unexplored (Benchmarks like MIA-Bench, MMMU, MathVISTA.etc)\n- The proposed DSM framework appears conceptually similar to existing Gradient Difference (Grad-Diff) algorithms, which also alternate between gradient ascent on the forget set and gradient descent on the retain set. It is unclear what the core technical novelty of this work is beyond that established paradigm. The authors should clarify how DSM fundamentally differs from prior gradient-difference, based unlearning methods mentioned in [1], [2], and [3]. \n\n\n\n\nReference:\n- [1] Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench\n- [2] CLEAR: Character Unlearning in Textual and Visual Modalities\n- [3] Large Language Model Unlearning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)