Reliable Lifelong Multimodal Editing: Conflict-Aware Retrieval Meets Multi-Level Guidance
Authors: Qiang Zhang
Fanrui Zhang
Jiawei Liu
Ming Hu
Junjun He
Zheng-Jun Zha
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge editing
Multimodal understanding
Retrieval-augmented
Primary area: applications
TL;DR: We propose CARML, a novel retrieval-augmented editing framework that integrates conflict-aware dynamic retrieval with multi-level collaborative guidance for reliable lifelong multimodal editing.

Abstract: The dynamic nature of real-world information demands efficient knowledge editing in multimodal large language models (MLLMs) to ensure continuous knowledge updates. However, existing methods often struggle with precise matching in large-scale knowledge retrieval and lack multi-level guidance for coordinated editing, leading to less reliable outcomes. To tackle these challenges, we propose CARML, a novel retrieval-augmented editing framework that integrates conflict-aware dynamic retrieval with multi-level implicit and explicit guidance for reliable lifelong multimodal editing. Specifically, CARML introduces intra-modal uncertainty and inter-modal conflict quantification to dynamically integrate multi-channel retrieval results, so as to pinpoint the most relevant knowledge to the incoming edit samples. Afterwards, an edit scope classifier discerns whether the edit sample semantically aligns with the edit scope of the retrieved knowledge. If deemed in-scope, CARML refines the retrieved knowledge into information-rich continuous prompt prefixes, serving as the implicit knowledge guide. These prefixes not only include static knowledge prompt that capture key textual semantics but also incorporate token-level, context-aware dynamic prompt to explore fine-grained cross-modal associations between the edit sample and retrieved knowledge. To further enhance reliability, CARML incorporates a "hard correction" mechanism, leveraging explicit label knowledge to adjust the model’s output logits. Extensive experiments across multiple MLLMs and datasets indicate the superior performance of CARML in lifelong multimodal editing scenarios.

Reliable Lifelong Multimodal Editing: Conflict-Aware Retrieval Meets Multi-Level Guidance
Qiang Zhang'*, Fanrui Zhangl , Jiawei Liu' Ming Hu², Junjun He*,
Zheng-Jun Zha'
'MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, USTC
SSanghaii Innovation Institute
Shanghai Artificial Intelligence Laboratory
{zq_ 126, efr888)@mail uste. edu. {jwliu6, thazz]eustt.co ming. hu@monash edu hejunjun@sjtu. edu. cn
Abstract
The dynamic nature of real-world information demands efficient knowledge editing in multimodal large language models (MLLMs) to ensure continuous knowledge updates. However, existing methods often struggle with precise matching in large- scale knowledge retrieval and lack multi-level guidance for coordinated editing, leading to less reliable outcomes. To tackle these challenges, we propose CARML, novel eetrieval-augmenter editing framework that integrates conffict-aware dy- namic retrieval with multi-level implicit and explicit guidance for reliable lifelong multimodal editing. Specifically, CARML introduces intra-modal uncertainty and inter-modal conflict quantification to dynamically integrate multi-chann retrieval results, so to pinpoint the most relevant knowledge the incoming edit samples. Afterwards, an edit scope classifier discerns whether the edit sample semanti- cally aligns with the edit scope of the retrieved knowledge. If deemed in-scope, CARML refines the retrieved knowledge into information-richt continuous prompt prefixes, serving as the implicit knowledge guide. These prefixes not only include static knowledge prompt that capture key textual semantics but also incorporate token-level, context-aware dynamic prompt explore fine-graine mne-grained cross-modal associations between the edit sample and retrieved knowledge. To further enhance reliability, CARML incorporates "hard correction" mechanism, leveraging ex- plicit label knowledge to adjust the model's s output logits. Extensive experiments across multiple MLLMs and datasets indicate the superior performance of CARML in lifelong multimodal editing scenarios.
Introduction
The rapid evolution of multimodal large language models (MLLMs), distinguished by the deep integration of visual and textual modalities, has significantly expanded the application boundaries of language models in real-world scenarios [1, 41, 43]. However, the static nature of pre- pre-trained MLLMs' embedded knowledge presents critical bottleneck continuously evolving information environments [7]. To address this, knowledge editing [37, 38] has emerged a promising paradigm for efficiently updating or correcting specific knowledge within models, avoiding the high costs of retraining. This technique plays an indispensable role in critical fields such as privacy preservation [19], bias mitigation [22], jailbreak attack defense [16], and hallucination correction [44, 8).
Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Outpat I Leghts
Label
Tofie
Output
type Momory
FUSE
Memoor
FEN
FFV
Ediis Knowledge
-6-CAIMM
Edis Knowledge
E
LEMEE
RECEPE
Siagle-bpe
SelEATTN
Texmal
Expleit Labe
Muliaundal
Self-AITN
Number of Sequential Edits
Sing sophe-ch
Retriewal
Fdit Sample
Multi Ahtti-charnel
Retrieval
Fdit Sample
(a) Retrieval accuracy of different methods
(b) Vanilla retrieval angmenned paradigm of L.KEE.
(c) Our proposed CARML
Figure I: (a) The trend of retricval accuracy @Recall 1) for the most relevant editing knowledge as the number of sequential edits increases across different methods. (b) The paradigm of conventional -uurevel-augmentedd lifelong knowledge editing (LKE) methods. (c) Our proposed CARML achieves precise localization of relevant editing knowledge by dynamically integrating multi-channel retrieval information and enables reliable knowledge editing through multi-level collaborative guidance.
Existing knowledge editing research predominantly focuses on large language models (LLMs) domain, primarily addressing single or batch knowledge editing tasks. These approaches can be broadly categorized into two types: (1) Intrinsic knowledge editing [18, 30J, which directly modifies specific model parameters to update and store new knowledge; and (2) External knowledge-aides editing [31, 31, which synergizes additional contexts or parametric modules (e.g., in-context learning [45] or Feed-Forwail Networks (FFN) [39]) to update knowledge without altering the model itself. While these methods have demonstrated efficacy within LLMs, their direct transposition MLLMs often encounters substantial resistance. The inclusion of visual modalities and the complex semantic interactions between modalities make it difficult for these approaches to effectively capture cross- modal semantic associations, thereby limiting their capacity to steer MLLMs toward generating desired responses [7. 33].
In practical applications, LLMS or MLLMs often require continuous knowledge updates to reflect the dynamic changes of real-world information, giving rise to the concept of lifelong knowledge editing [2]. Conventional single or batch editing paradigms face acute challenges in such lifelong scenarios. This primarily the inability decouple the knowledge associated with different editing requests, which produces the cumulative effect of parameter perturbations, leading to catastrophic forgetting [20]. To counteract this, certain methods construct editable knowledge bases to store each edits independently, and introduce retrieval-augmentt the model's original parameters, thereby alleviating forgetting [14, 15]. However, as the volume of editable knowledge grows, these methods encounter increasing difficulties in accurately matching relevant knowledge (Figure I(a)). More critically, the editable knowledge they architect is generally circumscribed to single type, lacking the synergistic integration of multi-level knowledge (encom- passing unimodal, multimodal, and explicit label information). thus failing to collaboratively guide the editing process. This deficiency fundamentally limits the reliability of the editing outcomes.
techniques to avoid directly modifying
To address this critical gap, we introduce CARML, a novel retrieval-augmenter editing framework specifically designed to tackle the above challenges. The core innovation of CARML lies in its integration of confiit--ware dynamic retrieval and multi-level implicit and explicit knowledge guidance, enabling reliable lifelong knowledge editing in MILMss. Its workflow is as follows:
To achieve precise knowledge retrieval, CARML introduces conflict-aware dynamic retrieval mechanism. By leveraging a pre-trained multimodal embedding model, CARML processes visual, textual and multimodal information of sequential editing requests, constructing knowledge bases tailored to specific modalities. A multi-chan retrieval strategy is then used to query these knowledge bases in parallel for the incoming edit samples, generating podality-sppeifff retrieval candidates. The key innovation lies in CARML's ability to quantify intra-modal uncertainty and inter-modal conflict across retrieval candidates, dynamically weighting and locating the most accurate and consistent retrieved knowledge. To ensure edits are applied only when necessary. CARML constructs an edit scope classifier that maps the embeddings of edit samples and retrieved knowledge into a newly constructed semantic space using two trainable MLPs. By calculating the semantic distance,

---

evaluates whether a subsequent editor will be triggered. This mechanism safeguards the locality of editing by effectively filtering out irrelevant edit samples.
To ensure the high reliability of the editing process, CARML incorporates a carefully designed multi-level implicit and explicit knowledge guidance strategy. At the implicit knowledge guidance level. CARML distills retrieved knowledge into information-rich continuous prompt prefixes, which effectively guide MLLMs to generate the desired outcomes. These prefixes encapsulate two innova- tive types of prompts: (1) Static knowledge prompt, capturing key knowledge semantics in the textual modality. (2) Context-aware dynamic prompts, which dynamically integrate multi-type retrieval knowledge at a fine fine-graing grained token level, under the guidance of edit samples. These prompts can provide highly task-specific multimodal instructions tailored the editing context. At the explicit knowledge guidance level, CARML further enhances the reliability of the editing process by intro- ducing an output logits enhancement mechanism. During inference, this mechanism directly boosts the probabilities of tokens aligned with the true labels in the retrieved knowledge. By decoupling the knowledge update process from the core parameters of the MLLM and selectively integrating the most relevant retrieved knowledge, CARML ensures adherence to three critical editing properties: reliability, generality, and locality.
Our contributions are summarized as follows: (1) We architect CARML, a pioneering retrieval- augmented editing framework, specifically conceived for the demands of lifelong knowledge editing in MLLMs. (2) We introduce conffict-awar aware dynamic retrieval mechanism that uniquely exploits multi- channel retrieval information for accurate knowledge localization based on quantified uncertainty and conflict degree. (3) We construct an edit scope classifier, which guarantees the locality of editing by effectively filtering irrelevant edit samples. (4) We design an innovative multi-level collaborative guidance strategy, integrating implicit (static and dynamic prompts) and explicit (logits enhancement) knowledge injection to achieve reliable editing.
2 Related Works
Knowledge Editing for LLMs. Knowledge editing focuses directly modifying updating fact-based knowledge or behavior stored within a model without the need for costly full retraining [12, 42]. And an ideal editing method should meet three critical criteria: reliability (accurately modifying the target knowledge), generality (applying the modification to relevant inputs), and locality (preserving unrelated knowledge) [38]. Knowledge editing for LLMs can be broadly categorized into intrinsic knowledge editing and external moolledge-aided methods. The intrinsic knowledge editing approaches involve updating specific model parameters modify its knowledge. A straightforward strategy is to locate and edit parameters linked to specific knowledge. For instance, ROME [28] uses causal mediation analysis to locate editable regions, while MEMIT [29] supports batch editing via rank-one parameter modifications. Another strategy involves meta-learning, training hypernetwork to update LLM parameters. as seen in MEND [30] and KE [9]. External knowledge-aided methods aim to store edited knowledge in external memory, leaving the model parameters unchanged. During inference, relevant information retrieved revise outputs. Notable examples include SERAC [31],
IKE [45], and RECIPE [3].
Knowledge Editing for MLLMs. Applying knowledge editing to multimodal large language models (MLLMs) is an emerging but challenging field [40, 32, 26). Due to the complexity of multimodal information, directly adopting LLM editing methods often yields suboptimal performance [11, 4). Research on MLLM knowledge editing is still in its early stages, with only few methods tailored to multimodal scenarios. For instance, UniKE [33] proposes a unified framework that combines intrinsic knowledge editing with external memory assistance. LiveEdit [2] generates independent low-rank experts for each editing instance, employing hard routing and soft routing mechanisms to select and combine these experts. Despite these advancements, achieving reliable and continuous knowledge editing for MILMs, particularly for lifelong knowledge editing that supports sustained updates, remains an open and pressing challenge.
3 Method
In this section, we first formalize the problem of the lifelong multimodal knowledge editing (Section 3.1). We then provide an overview of our proposed CARML framework (Section 3.2), followed by
3

---

Conffict-tware Dynamic Retrieval
Restend Anploot Lokot
repanna
Knowledge Base
rewic
Toxd
Ankounse
Agladl
Waterford
Gallin
Faod
Follaa
nelpn
Jluwer: Chuntyy
Logtt!
Bp
Jait suuqpl
Scope Classiffer
Large Language Model
Pr
-O00-
colheeeig Legert
Dberr onderhiing
Anseladee
Static and Dynamic Prompts
What Irish countyy la depictted the image?
Qleryy ruchadiingg
8,
Conflien-eary Dynamic Avertinal
Aentoooal Exceniatge
Nenti Prospau
Pas 1-1.B..
Dynamic Pronge!
Learnable
Phegm
Maltimodal Emderdiing
Prompt
Prosppo
Figure 2: The overall architecture of the proposed CARMI. It consists of key modules: (1) Conflict-aware dynamic retrieval module: locates the most relevant knowledge for the edit sample and determines whether initiate the subsequent editing process. (2) Multi-level knowledge guid- ance: delivers both implicit guidance (static and dynamic prompts) and explicit guidance (logits enhancement) for reliable editing of MLLM.
detailed descriptions of its two core modules: Conflict-Aware Dynamic Retrieval (Section 3.3) and Multi-Level Knowledge Guidance (Section 3.4).
3.1 Background: Lifelong Multimodal Knowledge Editing
We focus the task of lifelong multimodal knowledge editing. Given Multimodal Large Language Model (MLLM) fe x x + o, parameterized by θ, which maps an input image and text æ to a prediction fee (v) The editing process involves sequence of edit requests evolving over time, denoted as Dedit = {(ve, Fz, ot) 0t) where (v)) the target input requiring editing, and o, is the desired output. At time step T, Model Editor (ME) receives the model for (edited at step T 1), along with the current edit request (Vr, (T, subsequently yields an updated model fer as follows:
for = ME UT,
U7, OT), s.t. for (v, =
Or
if (Ve, v.) E Dedit, (0,x)) )) & Dedit-
fe (v. x))
This equation stipulates that subsequent to an edit, the updated MLLM fer should correctly predict the outcome for the current edit request, i.e., fer (wy, = or, while maintaining effective edits for past knowledge (1, 224) E Dedit- Additionally, for unrelated data (v,z) Dedit, the model should retain the original capabilities of fe.
3.2 Overview of CARML
editing framework designed for reliable lifelong multimodal knowledge editing, as shown in Figure 2. Its core workflow is as follows: Initially, CARML constructs an evolving multi-type edit knowledge base that stores historical edits context. Upon the arrival of new edit sample, CARML instigates its innovative Conflict-Aware Dynamic Retrieval mechanism. This module utilizes a multi-channel retrieval strategy and quantifies intra-modal uncertainty and inter-modal conflicts, thereby adaptively fusing multi-channel retrieval information and re-ranking candidates ensure highly accurate retrieval. Then, an edit scope classifier discerns whether the edit sample semantically aligns with the editing scope of the retrieved knowledge, crucial step
CARML novel

---

for preserving model locality. If deemed in-scope, CARML activates synergistic Multi-Level Knowledge Guidance strategy, comprising implicit guidance (static and dynamic prompts) and explicit guidance (logits enhancement). These strategies collaboratively navigate the MI.LM towards generating outputs that faithfully adhere to the edit directive.
3.3
Conflict-Aware Dynamic Retrieval
To achieve precise retrieval within a continuously growing knowledge base, CARML introduces a aware dynamic retrieval mechanism comprising four key components: knowledge base construction, multi-channel retrieval, conflict-aware fusion and re-ranking, and an edit scope classifier.
Knowledge Base Construction. CARML first curates multi-type edit knowledge base K {kr = (ei: e', ei ot)) where each entry kı corresponds to a historical edit request (0r, Tt, or). The knowledge base is initialized empty and grows with new edit requests. The embeddings en egg ef E Rd represent the visual, textual, and fused multimodal embeddings, respectively, extracted using frozen, pre-trained multimodal embedding model fE (e.g., mexma-siglip2 [36]):
e = f (v)) f (x+ + o,), em = Layer (e' + e/d))
(2)
where f and f denote the visual and textual encoders of the embedding model fE, and (æq + oz) stands for the string concatenation of 2t and oz.
Multi-Channel Retrieval. Given an incoming edit sample q (vg, T,),, its corresponding query embeddings ge (e4, eg, en)) computed via the similar process outlined in Equation 3.3. Sub- sequently, CARML executes a multi-channel retrieval strategy to identify relevant entries from the knowledge base K. This strategy computes the cosine similarity score between the query embedding qe and each modality-spedii embedding sets within the knowledge base K.
= sim (eg, (<)))t)) = sim (c), (C)))=))) Sm sim (chn {cin {cim)
(3)
where sim(-.) denotes the cosine similarity function. The sets {et) {et and fer, 1 represent the visual, textual and multimodal embedding collections within the knowledge base K, respectively. And S,, Sz, S., are the resultant modallty-specifee similarity score vectors.
Conflict-Aware Fusion and Re-ranking. To judiciously integrate information from multi-channels and manage potential moter- discrepancies, we devise a conflict-aware fusion and re-ranking mechanism. The procedure as follows:
For the similarity score S; of each modality i E {v, m). we first convert it into probability distribution P. softmaar(S))) (S,). The information entropy U, -P; log (P;) is then calculated for each distribution, quantifying the internal uncertainty of the retrieval results for modality i.
Beyond intra- intra-modal uncertainty, the degree of decisional conflict among modalities is paramount for assessing fusion reliability. We first identify the index set TopK, of the top-K candidates exhibiting the highest similarity scores for each modality i. For any pair of distinct modalities i and j. the Jaccard similarity coefficient Jıj of their TopK; and TopK, index sets is computed. Then, the conflict degree Cj between modalities i and is defined as:
[Tpp TopKyl [TpK, U T
Considering both intra-modal uncertainty U, and inter-modal conflicts Cıj. dynamic weighting mechanism is introduced to adaptively assign a weight W; to each modality. This weight aims to reward modalities that are both low-uncertainty and less conflict with others. Specifically, first calculate an intermediate score m, for each modality:
m4 1/2 1:-1--00
U
(1 -5¹
1
1-
Enso Cij En.ih Csi
(5)
Un
5

---

where the first term denotes the normalized entropy, the second term represents normalized conflict degree for modality And the final dynamic weights are procured via W; = so (tma(m.))
Finally, these dynamic weights W; are employed to perform a weighted sum of the original similarity scores S for each modality, yielding the definitive fused similarity score vector Sy Lie W. St. Then, all candidates in the knowledge base are re-ranked based on Sy. and the single premier entry exhibiting the highest fused score is selected as the final retrieved knowledge k2 (c ea c ot).
Edit Scope Classifier. To stringently preserve model locality, CARML integrates an edit scope classifier. This component determines whether the edit sample q falls within the editing scope of the retrieved knowledge kig
Specifically, two lightweight, trainable MLPs are used to map the current edit sample's query embedding qe and the retrieved knowledge hú, into fresh semantic spaces. These MLPs are trained with the objective of minimizing the distance between in-scope guery-knowledge pairs and maximizing for out-of-scope pairs in this projected space. The mean cosine similarity S. between the mapped query and retrieved knowledge embeddings is then computed. If S. exceeds predefined threshold B. the current edit sample q is classified as in-scope; otherwise, it is deemed out-of-scope.
This scope classifier is optimized using the Focal Loss Escope [23], enabling it to concentrate on hard- to-classify instances. Ground-trum labels for training data are predefined: reliability and generality samples (which should be affected by the edit) are labeled as "in-scope", while locality samples (which must remain unaffected) are designated as sout-ef-scope". The loss function is:
Lecope yy(1 1))² 2og
-1 v) ( 2 log (1 (1-
where y is the ground-truth label, y is the classifier's predicted probability.
3.4 Multi-Level Knowledge Guidance
When the edit sample q is classified as in-scope, CARML activates its multi-level knowledge guidance module. This module leverages the retrieved knowledge k2- = (c: c ei,, ob) to provide both implicit and explicit guidance to the MLLM, steering it towards generating the desired output og-
3.4.1 Implicit Guidance: Static and Dynamic Prompts
Inspired by prompt tuning [25, 46], we distill the retrieved knowledge hú into parameter-efficient continuous prompt, serving implicit reference input to the MILM. These continuous prompts prepended to the MLLM's 's original input embedding sequence and comprise two distinct types:
Static Knowledge Prompt. This prompt is engineered to encapsulate the core semantic essence of the textual knowledge statement contained within the retrieved kip A lightweight MIPP (MLP,) is utilized to map the retrieved knowledge's 's text embedding e a fixed-length continuous prompt Pz:
Ps = (MT.
where P= E Rexdmlli is the length of the continuous prompt, dmlim is the word embedding dimension of the MLLM, and Fresp maps the MLP output vector into the desired matrix shape.
Context-Aware Dynamic Prompt. Since static knowledge prompts may lack the specificity required for intricate multimodal edits, we introduce the context-aware dynamic prompt. This generates customized multimodal instructions that are highly attuned to the interaction between the edit sample q and the retrieved knowledge kg. Initially, the visual (e²) and multimodal (et)) embeddings of the retrieved knowledge kG are transformed via shared-weight M (Equation 3. 3.4.1) to produce continuous prompts Pa and Pm. of length r, respectively. Then, we propose a token-level attention fusion mechanism to further explore the fine-grained associations between the edit sample 9 and retrieved knowledge kg. For each token position! i (from to r), the edit sample embedding ge (e%, eß, ef forming 3 x milmm matrix Q.) serves the query, while the i-th tokens from the three modality prompts (p', p P'm. forming 3 x amllm matrix P( serve as keys and values.

---

This calculates a token- speen-pe attention score A R3333;
A = softmax Qe
(P ; 1,,,...
(8)
Vd
To derive context-aware fused prompt token p'y. first sum the columns of A, to obtain 1 x 3 vector A' representing the aggregated importance of each prompt modality p P Pmn with respect to the edit sample query qe- Later, we set the fusion weight ó; = soʃ (maar) A4). The i-th fine-grained fused prompt token pi is then computed as a weighted sum: p = Potz 6;. Repeating this fusion for
all i 1. 1..2.... ?, we can obtain the final context- aware dynamic prompt Py R.xddd
Finally, the static knowledge prompt Pz, context-aware dynamic prompt Pf. and the MLLM embed- dings for the query image femb (1gy and text femb (x%y) are concatenated to form the input sequence of MLLM. The inference process is reformulated as: o = fe (Px @ Ps @ femb (10) @ femb (Zg)).
3.4.2 Explicit Guidance: Output Logits Enhancement
To further enhance editing reliability, we introduce explicit token bias mechanism that directly adjusts the MILM's output logits. Specifically, during inference, let / E IR be the initial probability distribution predicted by the MLLM after applying softmax, where V is the vocabulary size, and os is the ideal output token ids from the retrieved knowledge ka. Therefore, thanks to the high precision of the retrieval strategy described above, we update the logits distribution / simply by:
1= Mor), iʃ Sp -log (ogll) <
Where h(os) E RV corresponds to the one-hot encoding of the desired token id og. The similarity Sp used to measure the relevance between the original probability distribution / and the desired output og, Only when it is below the threshold 1y. the corresponding logits enhancement strategy is initiated, which further protects the model's 's original ability to irrelevant inputs.
3.5 Training Objective
The training objective of CARML is to jointly optimize the scope classifier and the implicit guidance modules, while keeping the core MILM parameters frozen. Following previous work [2], we define the editing loss Ledit to ensure that edits satisfy reliability, generality, and locality requirements. And the overall training loss Liotal is as follows:
Liotal = Ledit Lecope
(10)
where À is a weight balancing the two losses.
Experiments
4.1 Experimental Setup
Datasets and Models. Following [7], our experiments are conducted on the MMEdit benchmark [7], which includes two sub-tasks: Editing VQA (E-VQA) and Editing Image Caption (E-IC). Additionally, we incorporated VLKEB [17] dataset, which consists of real images to better represent real-world scenarios. Besides, experiments executed two prominent MILMs: BLIP 2-OPT (2.7B) [21] and LLVVV-V1.5 (7B) [24].
Baselines. CARML's performance is compared with the following basclines: (1) FT-L [7]: direct fine-tuning of the last layer of the LLM. (2) Two intrinsic knowledge editing methods: TP [18] and MEND [30]. (3) External knowledge-aided methods: SERAC [31], LEMoE [39], and RECIPE [3],
along with the sfate-of-the-ar lifelong editing method LiveEdit [2), designed specifically for MILM.
Implementation Details. Across all experiments, we utilize mexma-siglip [36] as the multimodal embedding model. Within CARML's conflict-aware dynamic retrieval module, we select the top 15 retrieval candidates to calculate inter-modal conflict degree. For the multi-level knowledge guidance module, the length r of each type of prompt set to 4. And the weight coefficient in the final loss function is set to 0.04. More details of the experimental setup shown in Appendix A.
7

---

Table 1: Main editing results for E-VQA dataset. T: Num Edits.
Method
E-VQA
T=1
T' =
100
T= 1000
Rel
T-Gea
M-GGen.
T-Loc.
M-Loc.
Rel.
T-Gee.
M-Cien.
TeLoc.
McLac.
Rel.
T-Gien.
M-Gien.
T-Loc.
M-Lac.
BI.TP-2
TP
70.1
65.8
53.]
98.1
38.2
33.3
20.6
18:4
8.7
8.3
MEND
95.1
92.8
93.1
92.0
75.8
17.7
16.4
18.3
91.5
67.9
15.8
14.4
17.7
91.7
70.2
SERAC
88.4
84.5
84.3
85.8
36.0
87.0
81.3
81.0
71.0
15.6
83.4
70.8
80.3
67.7
13.1
LEMoE
93.6
92.2
98.5
85.2
29.5
22.6
28.4
81.7
22.7
20.5
15.0
20.2
73.5
22.6
RECIPE
LiveFalit
100.0
100.0
95.
92.9
94.4
84.7
100.0
CARML
100.0
100.0
100.0
100.0
97.0
96.7
94,9
100.0
97.7
97.0
95.9
93.2
100.0
91.9
LLaVA-VLL
FEL
96.4
92.7
93.4
76.5
72.1
68.8
60.1
62.6
42.8
34.6
44.2
36.2
41.6
33.6
24.7
36.1
20
SERAC
89.3
83.7
85.0
82.0
23.8
$8.
81.5
82.5
62.1
12.9
85.6
75.6
82.0
62.5
15.7
LEMoF
93.6
92.8
9000
99.
97.0
43.0
37.2
34.6
78.1
50.4
31.4
25.3
27.4
70.1
41.5
RECIPE
91.8
87.1
87.2
95.1
K7.7
88.3
81.6
82.8
89.4
81.2
83.5
72.1
81.6
84.0
80.3
LiveEdir
94.3
100.0
100.0
99.3
84.3
100,0
4.2
Main Results
Competitive Performance of CARML. Tables 1, 2 and present the results of lifelong editing experiments conducted on the MMEdit and VLKEB datasets for CARML and baseline methods. The experimental reveal the following findings: While most conventional methods can achieve satisfactory reliability in single-edit scenarios, their performance significantly degrades the number of editing steps (T) increases. This is because, as the volume of edited knowledge grows, fine-tuning methods like FT-L and intrinsic knowledge editing approaches such TP and MEND suffer from catastrophic forgetting due to accumulated parameter shifts. the other hand, external knowledge-aided methods like LEMOE, RECIPE, and LiveEdit increasingly struggle to accurately retrieve relevant knowledge from an expanding knowledge base. contrast, our proposed CARML consistently maintains nearly 100% performance even when the number of edits reaches 1000. This remarkable performance is primarily attributed to its ingenious confict-awaae dynamic retrieval strategy, which precisely locates the most relevant knowledge for each edit sample. Moreover, CARML's multi-level knowledge guidance mechanism robustly steers the MILMM at both implicit and explicit levels to generate the desired outputs.
Table 2: Main editing results for E-IC dataset. T: Num Edits.
Method
E-IC
T
-- 100
1000
T-Gen.
M-Gen.
T-Loc.
M-Loc.
Rel.
T-Gen.
M-Gen.
T-Loc.
M-Loc.
Rel.
T-Gen.
M-Gee.
T.Loc.
M-Loc.
BLIP-2
PT-I
40.6
41.2
38.7
97.7
98.9
42.5
45.1
39.4
83.2
53.4
45.8
42.2
43.7
52.6
54,1
TP
49.7
48.6
46.0
93.7
79.0
37.5
37.6
33.8
10.3
20.9
26.0
26.3
24.9
4.1
11.8
MEND
95.0
92.5
92.3
95.0
8.0
8.2
8.0
20.2
23.2
6.5
6.5
13.5
20.4
LEMoE
93.1
91.4
83.3
94.5
55.9
52.3
49.2
92.1
53.7
41.6
40.9
40.2
93.8
65.5
RECIPE
N0.7
79.2
7X.9
1000
95.3
41.5
40..
40.6
98.8
93.5
37.0
38.6
37.2
99.8
92.7
LiveEdit
80.6
76.9
100,0
100.0
79.2
77.4
74.1
100.0
100.0
72.9
70.3
67.9
100.0
100,0
CARMI
100,0
99.5
100.0
L.AVA-V1.5
FT-I
72.1
71.9
66-1
99..
98.7
64.2
56.5
53.2
85.5
91.8
58.0
$2.3
53.8
65.2
80.33
TP
57.6
59.2
$5.3
60.9
98.0
22.9
25.8
20.9
3.6
14.9
10.3
13.1
9.8
1.7
4.5
MEND
91.8
90.6
96.4
93.7
56.8
56.9
53.1
87.6
84.6
54.4
54.1
$1.0
83.9
80.6
LEMoE
93.8
91.4
90.6
95.1
93.0
56.7
53.2
52.0
92.7
79.9
36.7
33.1
29.7
85.2
77.1
RECIPE
85.9
76.1
83.3
96.7
95.2
55.3
53.2
51.9
91.3
92.5
51.7
53.4
48.5
88.9
91.4
LiveEdit
82.2
81.0
78.3
100.0
100.0
80.8
78.8
63.5
100.0
100.0
72.8
70.0
57.1
100.0
99.8
Cross-task Editing Evaluation. In the process of cross-task editing, MLLMs face the challenging task of simultaneously handling and editing samples from different tasks IQA and E-IC) continuous editing sequence. Table summarizes the results for 1000 sequential edits (the average of E-VQAA and E-IC test results) based on the BLIP 2-OPT model this demanding scenario. Evidently, most baseline methods struggle effectively edit both tasks within single editing sequence.

---

Table 3: Main editing results for VLKEB dataset. T: Num Edits.
VLKEB
Method
7
7
100
T
1000
Rel.
T-Gin.
M-Gen.
T-Lac.
M-Loc.
Rel.
T-Gie.
M-Gien
T.Loc.
M-Loc.
Rel.
T-Gen.
M-Gen.
T.Loc.
M-Loc.
FT-L
51.8
51.1
55.2
98.7
95.1
56.4
57.5
56.7
86.4
70.2
55.9
55.7
52.1
52.7
TP
$1.0
49.5
50.9
94.8
78.6
46.6
48.1
47.2
64.3
43.2
24.4
24.2
24.3
16.4
20.0
MEND
94.9
93.8
98
95.0
86.5
39.9
41.0
40.4
92.3
84.3
37.2
38.0
37.2
91.5
84.1
LEMoE
94.6
93.1
92.4
94.5
61.5
47.7
48.3
47.5
52.8
51.5
37.7
39.2
37.8
52.7
55.5
RECIPE
93.1
91.5
88.0
97.7
91.5
63.5
58.2
64.0
93.5
94.2
51.8
48.2
50.6
91.0
92.8
LiveRdit
98.8
98.1
94.9
100.0
100.0
98.2
97.7
94.0
100.0
100,0
97.0
91.9
87.5
100.0
100.0
99.1
10000
05.5
99.1
99.1
100.0
11aVA-V1.6
FT-L
92.8
89.1
92.6
92.1
91.8
74.1
74,8
75.6
68.7
83.4
68.4
67.9
67.2
66.8
74,9
TP
50.8
55.7
51.7
$7.9
90.1
19.7
20.1
19.4
11.4
24.1
5.5
1.8
5.5
2.8
7.2
MEND
90.2
89.2
90.1
0.6
0.6
0.7
0.0
0.1
0.1
LEMoE
91.9
93.1
91.7
87.0
87.9
78.3
76.2
73.5
50.9
49.2
63.2
57.5
56.8
48.1
45.6
RECIPE
93.1
92.8
92.1
91.7
86.4
77.2
66.0
76.4
88.5
84.1
63.8
56.2
61.9
86.7
81.2
LiveFdii
96.4
95.2
93.7
1000
100.0
94.6
90.7
89.6
100.0
100.0
92.2
84.0
82.8
100.0
100.0
100,0
Table 4: Main results on cross-tate editing.
Table Ablation study of CARML.
Method
T-Loc.
Method
M-Lac.
FT-L
29.7
25.0
28.9
2.6
6.3
Base
37.0
38.6
37.2
99.8
92.7
LEMoB
22.4
20.8
21.5
70.3
16.3
RRetiieall
RECIPE
81.2
77.7
78.8
99.9
65.9
Clossiller
70.2
73.4
69.4
100.0
100.0
LiveEdit
89.0
87.6
69.4
100.0
99.3
C-prompr
83.4
81.0
82.7
100.0
100.0
Logits
99.4
99.5
99.3
100.0
100.0
CARML
94.5
94.2
92.8
100.0
92.1
However, CARML demonstrates remarkable capability in integrating and editing knowledge from these diverse tasks. outperforms all baseline methods in key metrics such as reliability and generality, highlighting its adaptability and dependable knowledge editing capability.
4.3 Further Analysis
Effect of Individual Components. analyze the efficacy of the core components within CARML, we conduct a detailed ablation study with results presented in Table 5. The experiment takes RECIPE [3] as the base model and is carried out 1000 editing scenarios on E-IC dataset. First, we replace RECIPE's unimodal text retrieval with our conflict-aware dynamic retrieval strategy (+Retrieval), resulting
significant performance boost due improved retrieval aceuracy, This highlights the importance of precise knowledge matching in enhancing editing performance. Next, we substitute RECIPE's s original classification strategy with our edit scope classifier (+Classifier), which significantly improved locality by better identifying out-of-scope edit samples. We then incorporate context-aware dynamic prompts (+C-prompt) into the model's prompt prefix further enhanced reliability and generality. This improvement stems from capturing fine-graing cross-modal correlations between the editing sample and retrieved knowledge, enabling tailored multimodal instructions to guide the model effectively. Lastly, adding the explicit output logits enhancement mechanism (+Logits) further strengthened the model's reliability and generality.
Analysis of the Semantic Space for Edit Scope Classifier. We employ t-SNE [35] for dimensionality reduction and visualized the embedding distribution of retrieved knowledge (bluc) and out-of-scope edit samples (red) within the edit scope classifier, as shown in Figure (.(a). This provides intuitive confirmation of the effectiveness of the edit scope classifier: it successfully maximizes the distance between out-of-scope samples and retrieved knowledge within the leamee semantic space. Such clear semantic distinction is critical for CARML to perform reliable edits, preventing unintended modifications to unrelated knowledge.
Analysis of the Trade-off between Accuracy and Editing Time. Figure 3.(b) illustrates the trade-off between accuracy and overall editing time (including single-edit latency and inference delay after sequential edits) across different methods. can be observed that CARMI. achieves an outstanding balance of high accuracy and low time cost, outperforming other baseline methods. This not only

---

Reliabiliy
Generality
Localit
WHO AruERI
cocbal Sostund
Teltrn Dafleng
Sprti
Leb 21
G1)
(e)
h
Figure 3: (a) Visualization of the semantic space of retrieved knowledge and out-of-scops samples. (b) The trade-off between accuracy and editing time on the E-IC dataset, where editing time the sum of single edits and the time taken for model inference after 1000 edits. (c) Visualization example.
highlights CARML's exceptional efficiency-te Pefciiencytt-peeeee tor--oormoeeooccce ratio but also demonstrates its practical applicabiliiy and scalability in demanding lifelong multimodal editing scenarios.
Visualization Examples. As shown in Figure 3.c and Appendix E, CARML achieves reliable lifelong multimodal editing, which enables precise editing of relevant edit samples and maintains the original knowledge of irrelevant samples.
5 Conclusion
To address the challenges of lifelong knowledge editing for MLLMs, this paper proposes a novel sarrivaal-augmenter editing framework named CARML. Its innovation lies in the organic integration of the conflict-aware dynamic retrieval mechanism and multi-level knowledge guidance strategy. The former dynamically fuses multi-channel retrieval information and incorporates an edit scope classifier, to achieve precise localization and scope determination of the edited knowledge. The latter employs implicit static and dynamic prompts, along with explicit output logit enhancement, collaboratively guide the model in generating desired outcomes. Extensive experiments demonstrate that CARML significantly outperforms existing methods in lifelong multimodal editing scenarios.
Limitations
Despite the promising results, the proposed CARML has several limitations: (1) Generalization in Complex Editing: While CARML exhibits strong performance in tasks such as VQA and image caption editing, its capability to handle highly complex and abstract editing (e.g. intricate common- sense reasoning) requires further investigation.
(2) Model Scale Constraints: Due to computational limitations, our evaluation is currently restricted to MLLMs with a specific parameter scale. The adaptability and performance of CARML on larger-scale models (e.g., Intern VL2.57L22.5-88 [6]) remain to be explored.
Acknowledgment
This work was supported by the National Natural Science Foundation of China (NSFC) under Grant 62476260, 62225207 and 62436008, the Fundamental Research Funds for the Central Universities under Grant WK2100000055
References
[1] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile vasion-language model for understanding, localization, text reading, and beyond. arXiv preprint 230.1:3308../966 2023.
[2] Qizhou Chen, Chengyu Wang, Dakan Wang, Taolin Zhang, Wangyue Li, and Xiaofeng He. Lifelong knowledge editing for vision language models with low-rank mixture-of-expers.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, 2025.
10

---

[3] Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, et al. Lifelong knowledge editing for llms with Atrreval-augmentte continuous prompt learning, In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13565-13588 2024.
[4] Qizhou Chen, Taolin Zhang, Chengyu Wang, Xiaofeng He, Dakan Wang, and Tingting Liu. Attribution analysis meets model editing: Advancing knowledge correction in vision language models with visedit. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2168 1168-2176, 2025.
[5] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1.504. 00325, 2015.
[6] Zhe Chen, Jianman Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generie lisuulllinguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages -4-2918,, 2024.
[7] Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang. Huajun Chen, and Ningyu Zhang. Can we edit multimodal large language models? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13877-13888, 2023.
[8] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint (.XXv:2309903833 2023.
[9] N De Cao, W Aziz, and Titov. Editing factual knowledge in language models. In EMNLP 2021-2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, pages 6491-6506, 2021.
[10] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103. 10360, 2021.
[11] Junfeng Fang, Zac Bi, Ruipeng Wang, Houcheng Jiang, Yuan Gao, Kun Wang, An Zhang, Jie Shi, Xiang Wang, and Tat-Seng Chua. Towards neuron attributions in multi-modal large language models. Advances in Neural Information Processing Systems, 37:122867-122890, 2024.
[12] Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.0233 2024.
[13] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6904 6913, 2017.
[14] Xiaoqi Han, Ru Li, Hongye Tan, Wang Yuanlong, Qinghua Chai, and Jeff Pan. Improving sequential model editing with fact retrieval. In Findings the Association for Computational Linguistics:
EMNLP 2023, pages 1220911224, 2023.
[15] Tom Hartvigsen, Swami Sankaranaray Hamid Palangi. Yoon Kim, and Marzych Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 4747934447959 2023.
[16] Hanjiang Hu, Alexander Robey, and Changliu Liu. Steering dialogue dynamics for robustness against multi-turn jailbreaking attacks. arXiv preprint arXiv:2503.0m 001887003.0.7.,.. 2025.
[17] Han Huang, Haitian Zhong, Tao Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. Vlkeb: large vision-language model knowledge editing benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
11

---

[18] Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. caansformer-patcheee One mistake worth one neuron. In The Eleventh International Con-
ference on Learning Representations.
[19] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In The ōlst Annual Meeting Of The Association For Computatioonal Linguistics, 2023.
[20] Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng. Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, et al. Learning to edit: Aligning ilms with knowledge editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4689-4705, 2024.
[21] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 22ip- Bootstrapping language- -magge pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19744 19742. PMLR, 2023.
[22] Tomasz Limisiewicz, David Mareček, and Tomáš Musil. Debiasing algorithm through model adaptation. In The Twelfth International Conference on Learning Representations, 2024.
[23] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages
2017.
[24] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[25] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXhv:2110.07602 2021.
[26] Yaohui Ma, Xiaopeng Hong. Shizhou Zhang, Huiyun Li, Zhilin Zhu, Wei Luo, and Zhiheng Ma. Comprehendedit: A comprehensive dataset and evaluation framework for multimodal knowledge editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19323-19331, 2025.
[27] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[28] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359-17377 2022.
[29] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in transformer. In ICLR, 2023.
[30] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXiv:2/ 1309, 2021.
[31] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. basery-based model editing at scale. In International Conference on Machine Learning, pages 15817-15831. PMLR, 2022.
[32] Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, and Meng Wang. Finding and editing multi- modal neurons in pre-trained transformers. In Findings of the Association for Computational Linguistics ACL 2024, pages 1012-1037, 2024.
[33] Kaihang Pan, Zhaoyu Fan, Juncheng Li, Qifan Yu, Hao Fei, Siliang Tang, Richang Hong, Hanwang Zhang, and Qianru Sun. Towards unified multimodal editing with enhanced knowledge collaboration. Advances in Neural Information Processing Systems, 37:1 10200-110314- 2024.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/VV conference on computer vision and pattern recognition, pages 10684-10695, 2022.
12

---

[35] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11). 2008.
[36] visheratin. mexma-siglip2.
2025.
[37] Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. Advances in Neural Information Processing Systems, 37:53764 53797, 2024.
[38] Peng Wang. Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, al. Easyedit: An easy-to-use knowledge editing framework for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 3: System Demonstrations), pages 82 82-93, 2024.
[39] Renzhi Wang and Piji Li. Lemoe: Advanced mixture of experts adaptor for lifelong model editing of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2551-2575, 2024.
[40] Zhen Zeng, Leijiang Gu, Xun Yang, Zhangling Duan, Zenglin Shi, and Meng Wang. Visual- oriented fine-grained knowledge editing for multimodal large language models. arXiv preprint arXiv:2411.1277 2024.
[41] Fanrui Zhang, Dian Li, Qiang Zhang, Junxiong Lin, Jiahong Yan, Jiawei Liu, Zheng-Jun Zha, et al. Fact-rl: Towards explainable video misinformation detection with deep reasoning, arXiv preprint 1xXiv:2505.16883 2025.
[42] Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang. Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint wxXv:2401.002266 2024.
[43] Qiang Zhang, Jiawei Liu, Fanrui Zhang, Jingyi Xie, and Zheng-Jun Zha. Natural language- centered inference network for multi-modal fake news detection. In Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, 1JCAI-24, pages 2550-2550, 2024.
[44] Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alleviating hallucinations by editing large language models in truthful space. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
2024.
[45] Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? In The 2023 Conference on Empirical Methods Natural Language Processing, 2023.
[46] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasion-language models. International Journal of Computer Vision, 23009))3377-2348, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction mention that this paper focuses on the lifelong
multimodal editing scenario. The paper's contributions are detailed line at the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
This paper discusses in Section the limitations of the work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
This paper provides the full set of assumptions and a complete and correct proof in the Section 3.2
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This work provides all the information needed to reproduce the main experi- mental results of the paper in the Implementation Details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: This paper uses publicly available datasets. And we will release the source code as soon the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ cii.ps.cc/publiii g/w.//gguuuee gtt////ieeeeeeeeeee CCodeSubmeaaacc ssi asionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In this paper, all the training and test details are elaborated in Implementation Details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Experiment statistical significance is necessary for task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All experiments are conducted on a NVIDIA H100 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that htttp:://eerrpps.ccppllliel wttpcc/cc///peae.eeeeeee het.ol/c//hhccsciiieleeeeee
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attpp://neeurie
Answer:
[Yes]
Justification: The research conducted in this paper is in line with the NeurIPS Code Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses potential positive and negative social impacts in Section F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no risk of misuse in this article. In addition, this paper public datasets, and there no data privacy leakage.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models and datasets used in this paper have been properly cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwperswi thcode. con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will provide the code as well as the README.nd file to facilitate re- searcher communication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper is not among the many experiments and studies that have used humans as subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Large language models were used in this work, but not as a novel or unconven- tional component of the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https tttps://neurips for what should or should not be described.
//nnreurpp.... teeeurips.ccCCooeeeeette scc/oonferencee/2225/LLMM
20

---

Appendix
A Detailed Experimental Settings
A.1 Description of Datasets
To evaluate the effectiveness of our proposed CARML, we conduct experiments three multimodal knowledge editing datasets: E-VQA [7], E-IC [7], and VLKEB [17]. E-VQA and E-IC are part of the MMEdit benchmark, while VLKEB includes real-world images to simulate practical scenarios. Regarding the data split of all three datasets, we followed the setup in the original dataset. The details of each dataset are as follows:
E-VQA [7]: The E-VQA dataset is derived from the VQAv2 [13] dataset, designed to evaluate the knowledge editing capabilities of multimodal large language models (MLLMs) within the context of visual question answering tasks. The dataset comprises 6,346 training samples and 2,093 testing samples, with evaluation metrics covering reliability, generality (T- (T-Generality and M-Generality) and locality (T-Locality and M-Locality).
E-IC [7]: The E-IC dataset is constructed from the COCO Caption [5] dataset and serves benchmark for evaluating the knowledge editing capabilities MLLMs image caption tasks. The dataset includes 2,849 training samples and 1,000 testing samples. The evaluation framework mirrors that of E- EQA, thereby offering a robust means to gauge the effectiveness of knowledge editing in image captioning scenarios.
Each sample in E-VQA and B-IC datasets includes an edit instance, along with additional samples to evaluate textual and multimodal generality and locality. Generality samples are generated by rephrasing images and queries using tools like Stable Diffusion [34] and ChatGLM [10], ensuring the model's ability to generalize edits across different modalities. And locality samples are derived from unrelated images and queries from datasets such as OK-VOA [27], providing a stringent test of edit precision and locality. These datasets provide comprehensive benchmark for evaluating model editors, focusing on their ability to maintain overall task performance while updating MLLM-specific factual knowledge.
VLKEB [17]: The Vision-Language Knowledge Editing Benchmark (VLKEB) large- dataset designed to evaluate the knowledge editing capabilities of MILMs in realistic scenarios. Unlike prior benchmarks that primarily rely on synthetic or simplified data, VLKEB uses real- world images linked to structured knowledge triples from multimodal knowledge graph. It contains 8,174 editing instances (5,000 for training and 3,174 for evaluation) and over 18, 88000 images, which provides rigorous and comprehensive benchmark for advancing research in factual editing for MLLMs.
A.2 Description of Baselines
FT-L [7] fine-tunes the last layer of the language model within MLLMs, which is the most widely used strategy for adapting pre-trained models to specific tasks.
TP [18] is an efficient model editing technique that implements knowledge updating by injecting small, trainable neurons (referred to as patches) into the final feed-forward layer. Each patch is tailored to correct distinct mistake, allowing for precise, sequential edits without compromising the model's overall functionality.
MEND [30] trains a hypernetwork via meta-learning that enables efficient model editing by trans- forming back-propagated gradients into FFN matrix parameter offsets. It enables precise edits with a single example, providing fast and scalable method for editing large language models.
SERAC [31] is memory- based model editing method that stores edits externally and uses scope classifier to selectively apply them via a counterfactual model. This allows for accurate, non-destructive updates without retraining, preserving the base model's original behavior.
LEMoE [39] is a model editing framework that enables continual, non-destructive updates to large language models by dynamically adding expert modules. It ensures stable routing with KV anchor mechanisms and optimizes edit order through clustering, effectively addressing forgetting and interference in lifelong editing scenarios.
21

---

RECIPE [3] is lifelong model editing approach that encodes edited knowledge continuous prompts and enhances large language models (LLMs) through retrieval mechanism. Furthermore, employs knowledge sentinel to filter the edit samples for each query, enabling efficient and accurate updates to LLMss.
LiveEdit [2] is a specialized framework for lifelong knowledge editing in multimodal large language models (MLLMs). It employs low-rank mixture-of-expert (MoE) approach, generating tailored low-rank experts for each specific edit. During inference, LiveEdit implements a two-stage routing mechanism: initially, it uses hard filtering based on visual semantics discard irrelevant experts, followed by soft routing that leverages textual relevance to integrate the appropriate experts.
A.3 Implementation Details
During training, we used the Adam optimizer with batch size of 8, learning rate of 1e-5, and set the number of epochs to 120. During testing, the threshold 3 for the edit scope classifier was set to 0.4 for experiments the E-IC dataset, and 0.7 for the E-VQA and VLKEB datasets. In explicit guidance, the relevance threshold was set to 6. To comprchensively evaluate the model's performance, we employed reliability, generality (including T-Generality and M-Generality), and locality (including T-Locality and M-Locality) accuracy as evaluation metrics, All experiments were conducted on single NVIDIA H100 GPU.
Table 6: Average editing results across three datasets and two base MLLMs. T: Num Edits.
Method
T=1
100
T' 1000
Rel.
T-Gee.
M-Gien
T-
M-L.cc
Rel.
T-Gien
M-Gen.
T-l.oc.
M-1.c.
Rel.
T-Gee.
M-GGe.
T-L.oc.
M-1.c:
FT-I
70.2
67.8
64.6
92.6
90.2
61.2
56.9
56.1
73.6
61.8
51.8
48.1
49.8
52.0
52.6
TP
52.5
52.5
47.6
88.2
86.5
33.4
33.1
29.9
24.7
17.2
16.7
16.4
6.8
11.2
MEND
93.2
92.0
93.1
20A4
19.0
18.9
42.6
SERAC
88.8
84.5
85.4
78.5
22.5
68.5
62.5
64.5
58.6
13.5
63.2
57.3
604
56.3
15.4
LEMoE
93.9
92.3
89.9
94.8
80.8
51.9
48.3
47.5
74.7
51.2
38.5
35.2
35.4
70.6
51.3
RECIPE
88.7
85.3
86.0
96.9
91.1
68.5
63.6
66.7
88.2
61.7
57.1
91,2
86.6
LiveEdit
CARML
99.6
99.6
99.4
100.0
97.9
99.2
99.0
98.5
100.0
97.5
98.5
98.2
97.0
100.0
96.3
B Summary of Experimental Results
Table summarizes the average performance across three datasets and base MLLMs. It can be observed that CARML significantly outperforms other methods on most metrics, with only slight underperformance in M-Locality compared LiveEdit. Moreover, CARML maintains consistently high performance even as the number of edits (T) increases, demonstrating its robustness and superiority in lifelong multimodal editing scenarios.
C Pseudo Code of CARML
The pseudo-code of the CARML editing stage is in Algorithm 1, and the one of the CARML inference stage is Algorithm 2.
Algorithm 1 CARML Editing Stage
Input: The initial multi-type edit knowledge base K, the edit dataset Dedit whose length is T',
Output: The final multi-type edit knowledge base = {kg = (et, e et o after T edits.
1: for each edit (u,, It, or) E Dodit, E [T'] do
2: Get multi-type knowledge representations.
3: Convert (Vt, Ft, 0t) to e e'z, e'n using Equation 3.3.
4: Update the multi-rype edit knowledge base K.
5: K₁ Ke-1 U [kg (et, e²,, et et,, 0
6: end for
7:
return The final multi-type edit knowledge base K = [k: = (e) el em ot)
22

---

Algorithm 2 CARML Inference Stage
The MILM model fe including the embedding layer femb, the test dataset Dieas the final multi-type edit knowledge base = (k₃ = (et e e o+)) the trained edit scope classifier sas the implicit guidance module Simp- the edit scope threshold [3, and the logits relevance threshold 7). Output: The model's output
Input:
1: for each edit sample q q= (Ug. Fig) E Diss do
2: Convert q (Ug. to de = (ey, (2,,,e e%,) using Equation 3.3.
3: Perform conflict-aware dynamic retrieval to pinpoint the most relevant knowledge kb
(et ebe e -))=00)) if' fets kb) 3 then
0b) in K.
4:
5:
Generate static knowledge prompts par and context-aware dynamic prompts pf with fump-
6:
Get the initial probability distribution 1 fe (px ps a femb (Vi) i femb (2g)).
7:
if-lop log (ob 1) < then
8:
Perform logits enhancement = =1++((on
9:
argmax (!').
10:
else
11:
0 argmax (I).
end
13:
else
0= fe (femb (Vg) femb (xg)).
15:
end if
end for
17:
return The model's output
D Description of Editing Loss
The editing loss is designed to ensure three key properties of knowledge editing: reliability, generality, and locality. Given a MLLM fe and training batch consisting of b editing samples (Ur, 2t, oz). along with corresponding generality samples (Ug, Tg. 0₂) and locality samples (vi, 2t, or), the loss terms associated with these properties are defined as follows:
Reliability Loss L This term serves to minimize the negative log- ligeliiel of generating the desired output o₂ on the editing samples (ve, Iy, 0t), conditioned on the static knowledge prompt pz and context-aware dynamic prompt P/ generated in Section 3.4.1.
Lret = log fe (o: Pa ① Py ① femb (1t) ① femb (I:))
(11)
Beyond correcting individual specific inputs, the edited MLLM should generalize semantically equivalent neighborhoods Zg. ot). This loss is defined analogously to the reliability loss.
Generality Loss Cyen:
Egen = - log fo (o: Pa + Pp + femb (vg) femb (xg))
(12)
Locality Loss Lioe: To ensure the edit remains localized, the output distribution of the edited model should remain close to that of the original frozen model fe on unrelated samples (vi, 2i, 01). This enforced by minimizing the KL divergence between their outputs.
Lioc KL (fe (femb (vi) ① femb (a:)) Iife (px @ py + fems (v)) @ femb (I1))
(13)
The overall editing loss Lodit is the sum of these three components.
Lodit Lret Leen Ltor
(14)
E Visualization Examples
In Figure 4, we present some additional visualization examples sourced from the E-IC, E-VQA, and VIKKB datasets. These diverse examples comprehensively demonstrate CARML'S robust performance in achieving highly reliable, generalized, and localized multimodal knowledge editing.
23

---

Reilabilty
T-Genrality
M-Generality
T-Locality
M-Locality
limer pert
p.poo
& of
à ploto
the ponsidaccy
loody
president
saved in?
Defanddliig
Tatun Editing
2019
Deprostaatiss
Alla Falling
Alke tilling
Alln tailing
Albe Billing
Dilling
Touth
(a) Example of E-IC dataset.
Relisaillity
T-Generality
M--Generality
T-Locality
the M-Locality
shonn
degicred
song spocte
magel?
Het.eel No. TaLL foLl
Parmapa
(b) Example V-VQA dataset.
Hellability
I-Geeerllity
M-Gonerality
T-Locality
M-Locality
Watchh Argestize footheil club farurd Bis Snage?
What's the
Which Angestine
who hit the first homerun mitate mord pas
Which fosBall clu is ipprssmeed this image?
foothall
fostell club's
boaaared is dts image?
posure?
Idaas Dditig
Dukano Editing
PO Spral Mexiov
FC Syntil, Mnoos
RC, Sprrpll Veseee
Mencia
Fulling
Fillin
Tillin
Filling
Telling
25
2
Nonali CliBos
Nondh OLd Bops
Clo3oo
Hacke Pops
AoHaaaoo PC
(c) Example dataset.
Figure 4: Visualization examples for multimodal knowledge editing.
F
Broader Impacts
Positive Societal Impacts. Knowledge editing for MLLMs allows models to be updated quickly and precisely with new or corrected information, minimizing the need for full retraining. This capability supports timely responses in dynamic domains such news, healthcare, and disaster response. By enabling fine-grained control over factual updates, these techniques can help curb the spread of outdated or incorrect information while improving model alignment with human values and current knowledge. Such advancements enhance trust and reliability in applications like education, accessibility tools, and interactive AI assistants.
Negative Societal Impacts. Despite its benefits, knowledge editing poses certain risks when misused or applied recklessly. Inaccurate or biased edits can propagate falsehoods with the same confidence and fluency as factual information, potentially leading to hallucinations or the spread of misinforma- tion. Malicious actors could exploit these methods to implant targeted disinformation without leaving detectable traces in model parameters. Furthermore, without robust safeguards, knowledge editing could undermine model transparency and introduce new avenues for manipulation.
24


Reviews:
[{"review_id": "CFJ0Cr6Jp5", "reviewer": "Reviewer_fHJZ", "summary": "This paper introduces CARML, a novel retrieval-augmented framework for lifelong knowledge editing in Multimodal Large Language Models (MLLMs). The authors identify two primary challenges in existing methods: imprecise knowledge retrieval and a lack of coordinated, multi-level guidance for the editing process. CARML addresses these issues through two core innovations. The first is a conflict-aware dynamic retrieval mechanism that constructs a multi-modal knowledge base and uses a sophisticated fusion strategy—quantifying both intra-modal uncertainty and inter-modal conflict—to accurately retrieve the most relevant historical edit for a new query. The second is a multi-level guidance strategy that combines implicit guidance (via learnable static and context-aware dynamic prompts) and explicit guidance (via a direct logits enhancement mechanism) to steer the MLLM toward the correct output. The authors conduct extensive experiments on several MLLM editing benchmarks (E-VQA, E-IC, VLKEB) using BLIP-2 and LLaVA models, demonstrating that CARML significantly outperforms a wide range of state-of-the-art baselines, maintaining near-perfect performance even after a thousand sequential edits.", "questions": "Regarding the conflict-aware fusion in your retrieval module, could you provide more justification for the specific formula used in Equation 5? Did you experiment with other methods for combining the uncertainty and conflict scores, and if so, how did they perform? How sensitive is the performance to the value of K used for calculating the Jaccard similarity?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Pros:\n1. The paper is well-written, methodologically sound, and addresses a significant and timely problem in the field of MLLMs. The motivation is clear, and the proposed solution is both technically sophisticated and elegant.\n2. Another significant strength is the multi-level knowledge guidance system. The combination of implicit and explicit guidance is compelling. The context-aware dynamic prompt, which uses the edit sample itself as a query to attend to different aspects of the retrieved knowledge, is a particularly clever mechanism for creating fine-grained, tailored instructions for the model. This moves beyond simply prepending static information and allows for a more nuanced interaction between the query and the stored knowledge.\n\nCons:\n1. A key contribution of this paper is the Conflict-Aware Fusion and Re-ranking mechanism. The use of information entropy to quantify the internal uncertainty of retrieval candidates is an interesting approach. However, the idea of using entropy as an uncertainty signal to re-rank retrieved information has been explored in prior knowledge editing work [1]. To better situate your contribution, it would be beneficial to cite this related work and explicitly differentiate your method. Your primary innovation appears to be the novel combination of this intra-modal uncertainty with a measure of inter-modal conflict, which is tailored to the multimodal domain. Clarifying this distinction would strengthen the paper's claim of novelty.\n2. I also have concerns around the \"Explicit Guidance: Output Logits Enhancement\" mechanism. The ablation study (Table 4) reveals that this single component is responsible for the final, dramatic leap in performance, pushing reliability from 83.4% to 99.4%. While undeniably effective, this mechanism feels less like \"guiding\" the model and more like a \"hard correction\" or a post-processing step that forces the model to output the cached answer. This raises questions about the nature of the \"editing\" being performed. Does the model truly learn to integrate the new knowledge into its parametric understanding, or does it simply learn to pass through the retrieved answer when triggered? This could potentially harm the model's generative fluency and its ability to reason about the edited concept. The paper would be strengthened by a more critical discussion of this component and its implications.\n\n[1] Shi, Y., Tan, Q., Wu, X., Zhong, S., Zhou, K., & Liu, N. (2024, October). Retrieval-enhanced knowledge editing in language models for multi-hop question answering. CIKM 2024.", "was_revised": true, "final_justification": "After reading the rebuttal, all my questions have been properly addressed. Therefore, I raise my score.", "rebuttal": "We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.\n\n>**Q1:** The difference in the use of uncertainty for \"Conflict-Aware Fusion and Re-ranking\" mechanism compared to work [1].\n\n**A1:** Thank you for pointing out the related work [1]. The main differences between our CARML and work [1] are as follows:\n\n**(1) Different task types:** The work [1] focuses on single-modality (text) knowledge editing, whereas our work is centered on the field of lifelong multimodal knowledge editing.\n\n**(2) Different usage stages and objectives**: The work [1] uses mutual information maximization to retrieve unique knowledge subgraph, which is later pruned by analyzing the uncertainty generated by the LLM. In contrast, our approach intervenes at the retrieval stage to identify the most relevant knowledge by evaluating the uncertainty of the retrieval results across modalities.\n\n**(3) Our core Innovation:** We uniquely introduce the critical dimension of \"inter-modality conflict.\" By combining \"intra-modality uncertainty\" with \"inter-modality conflict,\" CARML can dynamically adjust the weights of each modality, thereby achieving precise knowledge localization.\n\nWe will cite and discuss work [1] in detail in the revision to more clearly highlight the novelty of our work.\n\n>**Q2:** Concerns about the \"Explicit Guidance: Output Logits Enhancement\" mechanism.\n\n**A2:** Thank you for this valuable comment. We would like to clarify that the explicit guidance builds upon the parametric understanding established by the implicit prompt guidance. The implicit prompt is responsible for deriving a high-quality prior distribution of the editing knowledge. **The explicit guidance then calibrates this distribution by incorporating the target answer, further boosting the generation confidence for tokens relevant to the edit.**\n\nTo verify this, we conducted a supplementary experiment: we selected 100 E-VQA samples that were incorrectly predicted with only implicit guidance. During the auto-regressive generation process, we applied logits enhancement only to the first token of the answer sequence and observed whether it improves the model's prediction accuracy for subsequent tokens. The results are as follows:\n\nTable 1: The accuracy of subsequent tokens with logits enhancement applied only to the first answer token.\n\n| Method | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| w/o Logits | 0.0 | 2.7 | 0.0 | 100.0 | 96.5 |\n| + Logits | 31.6 | 32.0 | 28.9 | 100.0 | 94.3 |\n\nThe experiment shows that the \"Output Logits Enhancement\" can enhance the output probability for editing knowledge-related tokens and help calibrate the generation trajectory.\n\n>**Q3:** Regarding the specific justification for Equation (5) and attempts with other fusion methods.\n\n**A3:** Thank you for your attention to the details of our formula. The purpose of Equation (5) is to reward modalities that exhibit low uncertainty and low inter-modality conflict. We achieve this as follows:\n\n(1) We quantify the contribution of each modality's uncertainty and conflict degree by subtracting the normalized entropy and conflict score using 1, respectively.\n\n(2) We assign equal weights to both factors. Experiments have shown that this simple weighting strategy demonstrates strong stability and generalization capability across different datasets.\n\n(3) We did experiment with introducing learnable parameters (a small MLP) to combine the uncertainty and conflict scores to predict modality weights. However, the introduction of additional training parameters easily led to overfitting, whereas the simple averaging approach proved to be more robust in our experiments.\n\n>**Q4:** Regarding the sensitivity of the K value in Jaccard similarity.\n\n**A4:** We conducted a sensitivity analysis on the value of K in the Jaccard similarity, using retrieval accuracy (Recall@1) as the metric. The experimental results on the E-IC dataset are as follows:\n\nTable 2: The sensitivity analysis on the value of K in the Jaccard similarity.\n| K | 3 | 9 | 12 | 15 | 18 | 21 | 30 |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Recall@1 | 93.2% | 97.5% | 99.2% | 99.4% | 99.1% | 98.6% | 94.8% |\n\nThe results indicate that the retrieval accuracy shows good stability when the value of K is within the range of [12, 21]. A K value that is too small can affect the conflict judgment due to insufficient candidate samples, while a value that is too large can introduce noise, leading to a decline in performance. We will add a complete discussion on hyperparameter sensitivity analysis in the revision.\n\n[1] Shi et al. Retrieval-enhanced knowledge editing in language models for multi-hop question answering. CIKM 2024.\n\nThank you once again for your time and valuable feedback!"}, {"review_id": "GjrpOagAD1", "reviewer": "Reviewer_hMxv", "summary": "This paper tackles the problem of lifelong knowledge editing for mLLMs. The proposed method, CARML, combines conflict-aware dynamic retrieval with multi-level knowledge guidance (static/dynamic prompts plus logits enhancement). The authors show that CARML maintains strong reliability and locality even as the number of edits grows, outperforming various baselines.", "questions": "1. Could you clarify if the conflict-aware retrieval is truly novel compared to general multi-channel retrieval and fusion techniques? What makes it fundamentally better?\n\n2. What would the authors argue to be the most important component in the pipeline? \n\n3. Given the pipeline complexity, how sensitive is the method to the scope classifier’s threshold or the retrieval fusion parameters? Any tips for tuning these?\n\n4. Could you add a discussion about scaling the knowledge base — how will CARML handle millions of edits in practice?", "limitations": "Yes — the authors do a reasonable job discussing potential forgetting and the limits of current memory design. Might be good to add more about retrieval scalability in real deployments.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-motivated -- lifelong editing for MLLMs is both timely and practically important! It also highlights the key chllanges of growing edit knowledge bases and preserving locality, which is well-aligned with the contributions.\n2. The proposed solution is thorough and demonstrates impressive empirical results across multiple benchmarks and baselines, with ablation studies that justify each module.\n3.  Strong performance even with large numbers of edits (e.g., 1000+) shows good scalability. I especially enjoy reading the experiments and analysis results, they've proactively answered most of the questions I have. \n\nWeaknesses:\nMy biggest concern is that the proposed pipeline is quite complex, and the paper’s description can be hard to follow on a first read. Simplifying the narrative and emphasizing the key innovations more clearly would improve clarity.\n\nIn terms of better highlighting the key innovation within this pipeline: \n1. Among the main steps, I feel the \"Conflict-Aware Dynamic Retrieval\" seems to be more or less standard (e.g., I think people in general would do alignment and matching between different modalities), but I do like the idea of Context-Aware Dynamic Prompt. \n\n2. It remains a bit unclear which modules are absolutely necessary for the gains. While sec 4 shows a clear ablation studies gradually adding more components, an experiment isolating only the core parts (e.g., just context-aware dynamic prompt + logits bias) would make the impact of each clearer.", "was_revised": true, "final_justification": "The authors have addressed my concerns. In particular, they have added experiments for parameter sensitivity, additional ablation studies, etc. \nMy original biggest concern was that the paper had too many components in it and I remember taking some digging in order to make sense of the pipeline. In rebuttal, the authors had a clear idea of what they believed is the most important component, and I trust them to be. able to incorporate it into the final version.", "rebuttal": "We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.\n\n>**Q1:** Given the pipeline complexity, the clarity could be further improved.\n\n**A1:** Thank you for your pertinent suggestion.\n\n**(1) Complexity clarification:** Although CARML introduces multiple modules to optimize performance, they are all of a lightweight design, primarily involving vector operations and small MLPs, which results in extremely low computational overhead. In Section 4.3 of our paper, we illustrate the trade-off between performance and computational efficiency for CARML, showing that it achieves an excellent balance.\n\n**(2) Narrative improvement:** In the revision, we will highlight the key innovations of our method in the introduction. And in the methodology section, we will further refine the descriptive logic of each module to ensure its innovative aspects are articulated with clarity and coherence.\n \n>**Q2:** Conduct ablation studies on only core parts to identify the absolutely necessary modules.\n\n**A2:** Thank you for your valuable comments. Based on your recommendation, we have conducted an additional ablation study on the core components. The results are shown in the table below:\n\nTable 1: Ablation study on core components on the E-IC dataset.\n| Method | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| w/o conflict-aware dynamic retrieval | 37.0 | 38.6 | 37.2 | 99.4 | 98.6 |\n| w/o edit scope classfier | 99.4 | 99.5 | 99.6 | 0.0 | 0.0 |\n| w/o context-aware prompt+logits | 70.2 | 73.4 | 69.4 | 100.0 | 100.0 |\n| w/o static knowledge prompt+logits | 80.5 | 77.4 | 80.9 | 100.0 | 100.0 |\n| w/o logits | 83.4 | 81.0 | 82.7 | 100.0 | 100.0 |\n| CARML | 99.4 | 99.5 | 99.3 | 100.0 | 100.0 |\n\nWe can observe that the key modules that bring major performance improvements to CARML are: conflict-aware dynamic retrieval, editing scope classifier, context-aware dynamic prompt, and logits enhancement. In comparison, the contribution of the static knowledge prompt is relatively minor, as the information it carries is already encompassed by the context-aware dynamic prompt.\n\n>**Q3:** Regarding the innovation and advantages of conflict-aware dynamic retrieval.\n\n**A3:** The innovation and advantages of the \"conflict-aware dynamic retrieval\" mechanism are reflected in the following aspects:\n\n**(1) Dynamic adaptivity:** This mechanism dynamically allocates modality weights based on intra-modality uncertainty and inter-modality conflict, enabling precise adjustments for each editing sample.\n\n**(2) Customized for multimodal editing task:** It addresses the unique challenges of multimodal editing (e.g., the increased conflict caused by single-modality modifications in Generality-type editing samples) by capturing conflicts and dynamically reducing the weights of the relevant modalities.\n\n**(3) Lightweight and efficient:** The mechanism only involves vector operations and does not require computationally expensive re-rank models. Furthermore, the ablation study in **Q2** further validates the value of this mechanism, where the edit reliability is reduced from 99.4% to 37.0% after its elimination.\n\n>**Q4:** What is the most important component in the pipeline?\n\n**A4:** We believe the most important component is the multi-level knowledge guidance module, which ensures the effective execution of the knowledge edit. The implicit context-aware dynamic prompt generates highly customized, fine-grained multimodal editing instructions for the model, while the explicit logits enhancement further ensures the reliability of the editing outcome.\n\n>**Q5:** Regarding hyperparameter sensitivity and tuning techniques.\n\n**A5:** We have analyzed the key hyperparameters:\n\n**(1) Scope classifier's threshold β:** This threshold directly determines whether an editing operation is triggered and is therefore relatively sensitive, as shown in Table 2. Since the classifier has learned a stable semantic boundary, its decision threshold typically stabilizes around 0.5. We use a simple binary search method to fine-tune the value around β, selecting the optimal parameter that maximizes reliability without harming locality.\n\nTable 2: Sensitivity of the scope classifier's threshold β on the E-IC dataset.\n| β | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 0.2 | 99.9 | 99.7 | 99.6 | 96.4 | 95.3 |\n| 0.3 | 99.9 | 99.6 | 99.6 | 98.2 | 97.1 |\n| 0.4 | 99.4 | 99.5 | 99.6 | 100.0 | 100.0 |\n| 0.5 | 87.8 | 92.7 | 86.1 | 100.0 | 100.0 |\n| 0.7 | 49.4 | 64.0 | 45.4 | 100.0 | 100.0 |\n\n\n**(2) Retrieval fusion hyperparameters:** CARML exhibits low sensitivity to these parameters. For the fusion weights of uncertainty and conflict in Equation (5), experiments show that values within the range of [0.3, 0.7] achieve desirable results. For the Jaccard similarity K value, the experimental results of its sensitivity analysis are shown in Table 3. We can notice that its performance is stable for values in the range of [12, 21]. The K value that is too small may result in insufficient candidate samples, while one that is too large can introduce noise.\n\nTable 3: The sensitivity analysis on the value of K in the Jaccard similarity.\n| K | 3 | 9 | 12 | 15 | 18 | 21 | 30 |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Recall@1 | 93.2% | 97.5% | 99.2% | 99.4% | 99.1% | 98.6% | 94.8% |\n\n>**Q6:** The scalability of CARML's knowledge base in real-world deployment.\n\n**A6:** Thank you for raising this practical and important question. Our CARML method possesses the following advantages for scalability:\n\n**(1) Efficient retrieval scalability:** Our retrieval mechanism is built upon an efficient vector similarity search library (FAISS). Its search time complexity has a sub-linear relationship with the knowledge base size N. Therefore, CARML can be easily extended to millions of editing entries, with the primary bottleneck being storage rather than retrieval latency.\n\n**(2) Constant editing overhead:** Since the conflict-aware dynamic retrieval module can precisely locate Top-1 relevant knowledge item, the amount of knowledge received in the editing phase is fixed at one, and its computational complexity is independent of the size of the knowledge base and remains constant.\n\n**(3) Immunity to catastrophic forgetting:** Unlike methods that directly modify model parameters, CARML stores knowledge externally and selects relevant entries via its retrieval mechanism. This approach avoids the problem of catastrophic forgetting that can arise from frequent iterative edits, ensuring the long-term stability of the model's capabilities.\n\nThank you once again for your time and valuable feedback!"}, {"review_id": "7MIaBbT7fA", "reviewer": "Reviewer_9miH", "summary": "This paper focuses on the lifelong multimodal editing problem. To address the problem, the paper proposed a novel retrieval-augmented editing framework designed for reliable lifelong multimodal knowledge editing, which consists of two parts: (1) Conflict-Aware Dynamic Retrieval and (2) Static and Dynamic Prompts. The Conflict-Aware Dynamic Retrieval component aims to locate the most relevant knowledge for the edit sample and determine whether to initiate the subsequent editing process, while the Static and Dynamic Prompts aim to provide both implicit guidance and explicit guidance to the editing process. The experiments across various datasets demonstrate the effectiveness of the proposed method.", "questions": "See above.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The studied problem, lifelong multimodal model editing, is important to the community.\n- The proposed method makes sense to me in general, although I am not in this field.\n- The writing and the structure of the paper are good.\n- The experiments are very comprehensive.\n\nWeakness:\n- Have you compared the runtime efficiency of your pipeline to other baselines? Since you mentioned that 'The dynamic nature of real-world information demands **efficient knowledge editing** in multimodal large language models, I think these experiments will also be good for your paper.\n- The choice of models: BLIP 2-OPT and LLaVA-V1.5 (7B) are kind of outdated. I suggest the authors consider more recent models such as LLaVA-V1.6 and qwen 2.5 vl 7b instruct.\n\nOverall: *I am not familiar with the domain. However, based on my reading, I think the paper is very interesting and complete, and the experiments are strong to support the claims made by the paper. However, there is some probability that I might miss some parts of the paper, or I might not fully understand the paper.*", "was_revised": true, "final_justification": "My concerns have been well addressed. Therefore, I keep my score.", "rebuttal": "We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.\n\n>**Q1:** Comparison of runtime efficiency between CARML and other baselines.\n\n**A1:** Thank you for raising this important question. **Our proposed CARML demonstrates a superior trade-off between extremely high accuracy and low time cost.** We have discussed the comparative analysis of runtime efficiency in Section 4.3 of our paper, under the part of \"Trade-off between Accuracy and Editing Time.\" Specifically, we compare the accuracy and single editing time of the different methods after 1000 edits on the E-IC dataset, and the results are shown in Table 1.\n\nTable 1: The trade-off between accuracy and editing time.\n| Method | FT-L | TP | MEND | SERAC | LEMoE | RECIPE | LiveEdit | CARML |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Editing time (s) | 1.5 | 5.27 | 0.125 | 0.117 | 1.344 | 0.073 | 0.113 | 0.141 |\n| Accuracy | 47.68 | 18.62 | 10.68 | 37.3 | 57.06 | 63.18 | 82.2 | 99.64 |\n\n>**Q2:** The MLLM models used are kind of outdated and suggest considering more recent models.\n\n**A2:** Thank you for your pertinent suggestion.\n\n**(1) Reason for choosing existing models:** We chose BLIP 2-OPT and LLaVA-V1.5 as the base models to ensure a fair comparison with existing knowledge editing methods, as these methods all use the same base models.\n\n**(2) Supplementary experiments:** To demonstrate the robustness of CARML, we conducted experiments on a more recent model, Qwen2.5-VL-7B-Instruct, as shown in Tables 2, 3 and 4. The results show that CARML also achieves excellent editing performance and surpasses existing baseline methods, proving CARML's model-agnostic nature and broad applicability. \n\nTable 2: Editing results after 1000 edits on the E-IC dataset.\n| Method | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| FT-L | 51.4 | 42.7 | 40.6 | 59.6 | 57.0 |\n| LeMoE | 42.8 | 37.4 | 40.1 | 74.0 | 82.7 |\n| LiveEdit | 78.4 | 74.3 | 64.3 | 100.0 | 99.9 |\n| CARML | 99.4 | 99.3 | 99.4 | 100.0 | 100.0 |\n\n\nTable 3: Editing results after 1000 edits on the E-VQA dataset.\n| Method | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| FT-L | 48.5 | 44.7 | 43.1 | 51.4 | 38.9 |\n| LeMoE | 38.8 | 26.0 | 30.4 | 68.7 | 38.4 |\n| LiveEdit | 91.7 | 89.3 | 81.6 | 100.0 | 96.7 |\n| CARML | 97.2 | 95.9 | 92.9 | 100.0 | 94.1 |\n\n\nTable 4: Editing results after 1000 edits on the VLKEB dataset.\n| Method | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| FT-L | 62.7 | 54.9 | 58.3 | 68.0 | 60.2 |\n| LeMoE | 67.5 | 59.4 | 64.9 | 51.3 | 47.4 |\n| LiveEdit | 90.8 | 86.2 | 79.4 | 100.0 | 100.0 |\n| CARML | 99.3 | 99.1 | 98.8 | 100.0 | 95.5 |\n\n\nThank you once again for your time and valuable feedback!"}, {"review_id": "iSiZhMRVIO", "reviewer": "Reviewer_4PKH", "summary": "In this paper, the authors propose CARML, a novel retrieval-augmented editing framework that integrates conflict-aware dynamic retrieval with multi-level implicit and explicit guidance for reliable lifelong multimodal editing. The authors introduce a conflict-aware dynamic retrieval mechanism that uniquely exploits multi-channel retrieval information for accurate knowledge localization based on quantified uncertainty and conflict degree. Besides, the authors construct an edit scope classifier, which guarantees the locality of editing by effectively filtering irrelevant edit samples. Next, the authors design an innovative multi-level collaborative guidance strategy, integrating implicit (static and dynamic prompts) and explicit (logits enhancement) knowledge injection to achieve reliable editing.", "questions": "NA", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenght\n1. The authors introduce CARML, a novel retrieval-augmented editing framework specifically designed to tackle the above challenges. The core innovation of CARML lies in its integration of conflict-aware dynamic retrieval and multi-level implicit and explicit knowledge guidance, enabling reliable lifelong knowledge editing in MLLMs. \n2. The proposed method can achieve good results on MMedit benchmark\n\nWeakness\n1. Some recent proposed baselines are not compared, like AlphaEdit\n2.  The underlying theory or mechanism are not mentioned， making the proposed method heuristic", "was_revised": true, "final_justification": "I think the authors have addressed my concern, and I think this paper could be accepted.", "rebuttal": "We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.\n\n>**Q1:** Lack of comparison with some recent baselines.\n\n**A1:** Thank you for your suggestion. We have further compared CARML with two of the latest baselines proposed in 2025: AlphaEdit [1] and BalancEdit [2]. We conducted experiments with sequential 1000 edits based on the BLIP-2 OPT model on the three datasets. The results are shown in the Table 1, 2 and 3. CARML's performance surpasses both of these recent baselines, further demonstrating its effectiveness. And we will supplement these baselines in the revision.\n\nTable 1: Editing results after 1000 edits on the E-IC dataset.\n| Method | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| AlphaEdit | 60.5 | 54.9 | 50.7 | 57.2 | 50.3 |\n| BalancEdit | 70.5 | 68.4 | 57.2 | 58.1 | 51.5 |\n| CARML | 99.4 | 99.5 | 99.3 | 100.0 | 100.0 |\n\n\nTable 2: Editing results after 1000 edits on the E-VQA dataset.\n| Method | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| AlphaEdit | 86.7 | 82.2 | 79.6 | 89.0 | 75.7 |\n| BalancEdit | 82.7 | 76.4 | 75.3 | 65.0 | 58.1 |\n| CARML | 97.0 | 95.9 | 93.2 | 100.0 | 91.9 |\n\n\nTable 3: Editing results after 1000 edits on the VLKEB dataset.\n| Method | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| AlphaEdit | 89.4 | 82.7 | 84.1 | 86.4 | 78.3 |\n| BalancEdit | 79.6 | 76.2 | 70.9 | 62.5 | 63.7 |\n| CARML | 99.1 | 99.1 | 98.8 | 100.0 | 95.5 |\n\n[1] Fang et al. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models, ICLR 2025.\n\n[2] Guo et al. BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing, ICML 2025.\n\n>**Q2:** The underlying theory or mechanism are not mentioned, making the proposed method heuristic.\n\n**A2:** We thank the reviewer for highlighting the importance of the theoretical depth behind our method. The design of CARML is grounded in a decision-making framework that integrates multiple theories, with a controlled retrieval-augmented generation paradigm tailored for knowledge editing tasks at its core. Through non-invasive editing, CARML enhances the efficiency of multimodal knowledge editing while ensuring the stability required for lifelong learning. Specifically, its theoretical foundations include the following aspects:\n\n**(1) Information theory and ensemble learning:** To address the unique challenges of multimodal editing tasks, such as increased retrieval uncertainty and inter-modal conflicts arising from unimodal modifications in generality-type editing samples, we introduce information entropy to quantify retrieval uncertainty and use the Jaccard similarity to measure modal conflicts. Concurrently, drawing inspiration from ensemble learning, we treat the multimodal channels as \"experts\" and employ an adaptive mechanism to evaluate and fuse the retrieval results from different modalities.\n\n**(2) Metric learning:** To ensure the locality of edits, we have designed an edit scope classifier to learn a robust semantic decision boundary. This classifier effectively aligns the semantic spaces of the edit samples and the retrieved knowledge while filtering out samples that do not require editing, thereby guaranteeing the precision of the edits.\n\n**(3) Bayesian inference and prompt Learning:** Our proposed explicit-implicit guidance strategy is a concrete implementation of a Bayesian framework. Within this strategy, the implicit prompt guides the generation process by introducing strong prior knowledge, while the explicit logits enhancement further optimizes the probability distribution. This synergy calibrates the generation trajectory and improves the reliability of the edits.\n\nGrounded in these robust theoretical foundations, each component of CARML demonstrates a significant performance improvement in multimodal knowledge editing tasks, as detailed in the ablation studies in Table 4 of our paper. Furthermore, CARML possesses broad transferability potential for applications in other domains, such as fact-checking, hallucination mitigation, and efficient personalization. We will further emphasize these theoretical foundations and their significance in the revision.\n\nThank you once again for your time and valuable feedback!"}, {"review_id": "OShZPiFrOP", "reviewer": "Reviewer_eEUz", "summary": "The paper presents CARML, a retrieval-augmented framework for lifelong multimodal knowledge editing in MLLMs. It tackles two key challenges: (1) precise retrieval from an ever-growing edit memory by quantifying both intra-modal uncertainty and inter-modal conflicts via a multi-channel softmax+Jaccard-based fusion; (2) reliable editing through a multi-level guidance strategy—implicit continuous prompts (static + context-aware dynamic) and an explicit logits-enhancement (“hard correction”) mechanism. An edit scope classifier further ensures locality by filtering out unrelated edits. Experiments on three benchmarks (E-VQA, E-IC, VLKEB) and two MLLMs demonstrate that CARML maintains near-100% reliability and generality even after 1,000 sequential edits, significantly outperforming baselines", "questions": "See the weakness", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The combination of multi-channel retrieval, uncertainty/conflict modeling, and multi-level guidance (prompts + logits) is comprehensive and thoughtful.\n\n2. The paper identifies and tackles scalability, accuracy, and edit locality issues that are relevant in real-world settings.\n\n3. Results are reported across multiple datasets (E-VQA, E-IC, VLKEB) and strong baselines, showing consistent improvements.\n\nWeaknesses:\n\n1. The motivation is still confused. (1) The precise retrieval from an ever-growing edit memory. I do not know the concrete challenge in existing methods. Compared with existing methods, why can the proposed retrieval strategy solve this problem? In addition, the system seems complex. How do the benefits compare to the complexity? (2) As for the editing component, what is its advantage for the ever-growing knowledge?\n\n2. While many modules are proposed (such as edit scope classifier, two-level prompts, logits correction), it is unclear which are critical for performance and which are incremental.", "was_revised": true, "final_justification": "I reserve my opinion, but it is also OK if the paper is accepted.", "rebuttal": "We sincerely thank you for the valuable comments! Your comments are crucial for us to improve our work. We will address your concerns point by point.\n\n>**Q1:** The retrieval challenges faced by existing methods.\n\n**A1:** The retrieval strategies of existing methods face the following challenges: \n\n**(1) Ambiguity of text semantics:** Most current methods are designed for text-only knowledge editing and rely solely on text semantics for retrieval. However, due to the inherent ambiguity of text semantics (e.g., \"a photo of\" vs. \"a photograph of\" in the E-IC task), the lack of auxiliary visual information can easily lead to retrieval errors.\n\n**(2) Limitations of multimodal fusion strategies:** The first lifelong multimodal editing method, LiveEdit [1], retrieves multiple relevant experts through visual search and then fuses them using text semantic similarity weighting. However, this approach can introduce noise from irrelevant experts and cannot effectively handle direct conflicts between modalities.\n\n[1] Chen et al. Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts, CVPR 2025.\n\n>**Q2:** Effectiveness of the proposed retrieval strategy.\n\n**A2:** CARML's conflict-aware dynamic retrieval strategy focuses on the unique contradictions present in multimodal editing. For instance, in the Generality-type editing samples, single-modality modifications often increase the risk of conflict. By retrieving all modalities in parallel and dynamically quantifying intra-modality uncertainty and inter-modality conflict, CARML can accurately identify and suppress unreliable modality channels, significantly improving retrieval accuracy. In Figure 1(a) of our paper, we illustrate the retrieval accuracy trends of various methods under an increasing number of sequential edits. The results show that CARML consistently maintains a desirable retrieval accuracy, proving its robustness.\n\n>**Q3:** The advantages of the proposed method compared to its complexity.\n\n**A3:** Thank you for raising this critical question. We believe that while CARML introduces a moderate level of complexity in its design, it delivers significant improvements in performance and reliability through three core advantages:\n\n**(1) Precise knowledge retrieval:** As stated in our response to Q2, our retrieval mechanism overcomes many shortcomings of existing methods, ensuring the accuracy of knowledge localization.\n\n**(2) Reliable multimodal knowledge editing:** Through a multi-level, coordinated editing guidance strategy, we ensure that the retrieved knowledge is correctly understood and executed by the model.\n\n**(3) Excellent performance-efficiency trade-off:** Although CARML introduces multiple modules to optimize performance, they are all of a lightweight design, primarily involving vector operations and small MLPs. We have discussed the comparative analysis of runtime efficiency in Section 4.3 of our paper. Specifically, we compare the accuracy and single editing time of the different methods after 1000 edits on the E-IC dataset, as shown in Table 1. The results demonstrate that CARML achieves an ideal efficiency ratio, attaining extremely high accuracy while maintaining a low time cost.\n\nTable 1: The trade-off between accuracy and editing time.\n| Method | FT-L | TP | MEND | SERAC | LEMoE | RECIPE | LiveEdit | CARML |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Editing time (s) | 1.5 | 5.27 | 0.125 | 0.117 | 1.344 | 0.073 | 0.113 | 0.141 |\n| Accuracy | 47.68 | 18.62 | 10.68 | 37.3 | 57.06 | 63.18 | 82.2 | 99.64 |\n\n>**Q4:** The advantages of the editing component for the ever-growing knowledge.\n\n**A4:** Our method has the following significant advantages when dealing with the ever-growing knowledge:\n\n**(1) Immunity to catastrophic forgetting:** CARML does not directly modify the core parameters of the MLLM. Instead, it guides the model through external prompts and logits enhancement. This fundamentally prevents the catastrophic forgetting problem that can arise from accumulating knowledge.\n\n**(2) Constant editing efficiency:** The computational overhead of CARML's editing component is independent of the knowledge base size. For each edit, the conflict-aware dynamic retrieval module precisely locates the Top-1 relevant knowledge item and passes this single item to the subsequent editing modules. This ensures that the efficiency of the editing component remains constant even as the knowledge base grows.\n\n>**Q5:** The division between critical modules and incremental modules.\n\n**A5:** Thank you for your valuable suggestion. Based on the ablation study in Table 4 of our paper and the additional ablation experiments shown below, we have re-evaluated the importance of the modules and categorized them as follows:\n\nTable 2: Ablation study on core components on the E-IC dataset.\n| Method | Rel. | T-Gen. | M-Gen. | T-Loc. | M-Loc. |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| w/o conflict-aware dynamic retrieval | 37.0 | 38.6 | 37.2 | 99.4 | 98.6 |\n| w/o edit scope classfier | 99.4 | 99.5 | 99.3 | 0.0 | 0.0 |\n| w/o context-aware prompt+logits | 70.2 | 73.4 | 69.4 | 100.0 | 100.0 |\n| w/o static knowledge prompt+logits | 80.5 | 77.4 | 80.9 | 100.0 | 100.0 |\n| w/o logits | 83.4 | 81.0 | 82.7 | 100.0 | 100.0 |\n| CARML | 99.4 | 99.5 | 99.3 | 100.0 | 100.0 |\n\n**(1) Critical Modules:**\n\n● Conflict-aware dynamic retrieval: Significantly improves retrieval accuracy, leading to substantial enhancements in reliability and generality.\n\n● Edit scope classifier: Effectively prevents modifications to irrelevant knowledge by learning reliable semantic boundaries.\n\n● Context-aware dynamic prompt: Facilitates the effective execution of edits by providing highly customized, multimodal editing instructions.\n\n● Output logits enhancement: Further boosts editing reliability, increasing it from 83.4% to 99.4%.\n\n**(2) Incremental Modules:**\n\n● Static knowledge prompt: Although its removal results in a slight decline in overall performance, its essential information has been efficiently integrated into the \"context-aware dynamic prompt\" module.\n\nThank you once again for your time and valuable feedback!"}]

Author rebuttal:

Meta-review: This paper introduces CARML, a novel retrieval-augmented framework for lifelong multimodal knowledge editing in MLLMs. Its core contributions include a conflict-aware dynamic retrieval mechanism that quantifies intra-modal uncertainty and inter-modal conflict, and a multi-level guidance strategy combining implicit prompts and explicit logits enhancement. Strengths include strong empirical performance across multiple benchmarks (E-VQA, E-IC, VLKEB), scalability to 1000+ edits, and a well-motivated design addressing real-world challenges. Weaknesses initially included concerns about pipeline complexity and the novelty of retrieval mechanisms.

During the rebuttal period, reviewers raised concerns about novelty compared to prior work, component necessity, hyperparameter sensitivity, and theoretical grounding. Authors responded thoroughly: they differentiated their retrieval method from unimodal approaches, provided ablation studies isolating core modules (e.g., conflict-aware retrieval and logits enhancement are critical), added sensitivity analyses for hyperparameters like K and β, and expanded comparisons to recent SOTA methods (AlphaEdit, BalancEdit). They also grounded their method in information theory, metric learning, and Bayesian inference, addressing heuristic concerns.

This paper is recommended for acceptance due to its significant empirical contributions, methodological novelty in multimodal conflict-aware retrieval, and comprehensive rebuttal addressing all major concerns. The authors demonstrated scalability, robustness across models (including Qwen2.5-VL), and superior performance over strong baselines.
Decision: Accept (poster)