MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
Authors: Yuntao Du.
Kailin Jiang
Zhi Gao
Chenrui Shi
Zilong Zheng
Siyuan Qi
Qing Li
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal knowledge editing; Large multimodal model; Benchmark
Primary area: datasets and benchmarks
TL;DR: We propose MMKE-Bench, a challenging benchmark for evaluating diverse semantic editing in real-world scenarios.

Abstract: Knowledge editing techniques have emerged as essential tools for updating the factual knowledge of large language models (LLMs) and multimodal models (LMMs), allowing them to correct outdated or inaccurate information without retraining from scratch. However, existing benchmarks for multimodal knowledge editing primarily focus on entity-level knowledge represented as simple triplets, which fail to capture the complexity of real-world multimodal information. To address this issue, we introduce MMKE-Bench, a comprehensive **M**ulti**M**odal **K**nowledge **E**diting Benchmark, designed to evaluate the ability of LMMs to edit diverse visual knowledge in real-world scenarios. MMKE-Bench addresses these limitations by incorporating three types of editing tasks: visual entity editing, visual semantic editing, and user-specific editing.  Besides, MMKE-Bench uses free-form natural language to represent and edit knowledge, offering a more flexible and effective format. The benchmark consists of 2,940 pieces of knowledge and 8,363 images across 33 broad categories, with evaluation questions automatically generated and human-verified. We assess five state-of-the-art knowledge editing methods on three prominent LMMs, revealing that no method excels across all criteria, and that visual and user-specific edits are particularly challenging. MMKE-Bench sets a new standard for evaluating the robustness of multimodal knowledge editing techniques, driving progress in this rapidly evolving field.

Published as a conference paper at ICLR 2025
MMKE-BENCH: A MULTIMODAL EDITING BENCH- MARK FOR DIVERSE VISUAL KNOWLEDGE
Yuntao Du Kailin Jiang". Zhi Gav' Chenrui Shi Zilong Zheng" Manggi,,Syyuue SSyuan Qi', Qing Li
State Key Laboratory of General Artificial Intelligence, BIGAT
Sccooa of Software & Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University
$ University of Science and Technology of China
4 "State Key Laboratory General Artificial Intelligence, Peking University
"Beijing Key Laboratory of Intelligent Information Technology.
School Computer Science Technology. Beijing Institute of Technology
ABSTRACT
Knowledge editing techniques have emerged as essential tools for updating the factual knowledge of large language models (LLMs) and multimodal models (LMMs), allowing them to correct outdated or inaccurate information without retraining from scratch. However, existing benchmarks for multimodal knowledge editing primarily focus on entity-level knowledge represented as simple triplets, which fail to capture the complexity of real-world multimodal information. To address this issue, we introduce MMKE-Bennhl a comprehensive MultiModal Knowledge Editing Benchmark, designed to evaluate the ability of LMMs to edit diverse visual knowledge in real-world scenarios. BMMKE-Bende addresses these limitations by incorporating three types of editing tasks: visual entity editing, visual semantic editing. and user-specific editing. Besides, MMKE-Bench uses free-form natural language to represent and edit knowledge, offering a more flexible and effective format. The benchmark consists of 2,940 pieces of knowledge and 8,363 images across 33 broad categories, with evaluation questions automatically generated and verifeee We assess five state-of-the-ar knowledge editing methods on three prominent LMMs, revealing that no method excels across all criteria, and that visual and user-specific edits are particularly challenging. MMKE- Bench sets new standard for evaluating the robustness of multimodal knowledge editing techniques, driving progress in this rapidly evolving field.
INTRODUCTION
Large language models (LLMs) and multimodal models (LMMs) have demonstrated remarkable success across various tasks due their powerful understanding and reasoning abilities, grounded
in vast amounts of knowledge (Brown et al. 2020: Zhao et al.. 2023 Liu et al.) 2024b). However, the knowledge within these models can become outdated or inaccurate over time due to evolving real-world information and changes in factual data. To address this, knowledge editing techniques have been developed to correct inaccuracies and inject new knowledge into pre-treeeet are-trained models with minimal cost, without affecting unrelated content (Mitchell et al.) [0226] Yao et al. 2023). In recent years, several datasets have been introduced benchmark the progress knowledge editing methods in both the textual Yao et al. 2023 Onoe et al.
2023 Cao et al.
2021 Li et al.) 2023b) and
multimodal domains (Cheng et al. 2023 Huang et al.
2024 Li et al. 2024 Zhang et al. 2024))
However, most existing benchmarks focus on editing entity-level knowledge, typically formatted as triplet (subject, relation, object). While effective in certain tasks, this format lacks the complexity required for real-world applications. particularly in multimodal domains where visual knowledge must also encompass actions, body gestures, and object relationships. Furthermore, knowledge editing techniques have quickly saturated on these benchmarks, achieving near wear-eerfect performance. For example, simply fine-tuning the LLaVA model achieved 99.59%, 99.43%, and 95.48% accuracies
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
Visual Eatiey Editing
Visual Semantic Editing
UwerSpeeiifi Editing
The locus is the tmage Zina thoharand lrobalker He bage in conoce at Melno 1900 2005.
Stow, coeld you intedace pepped snag??
Original Knowladge
Or the peisis the irage? At ayaa Rooney
lis ponaer takmnk epoenen livin
Madel This peppec
Fentio Pips You braght it dertes Phinee eegee the dook of lrvig nees.
Oriiital Keeelalty Editee Navvlidde My
hemun image Keaan tockreand te Bollan perfonceeal lo conse of Rota 1900 and repited Real Mustri
Thccis graer levake making wiit your hand. B liling
Skine coall you innokec lo puppett image"
Wav
Editing Knowledge
Ovizagg?
reag wau horee Fastan Tap beagh
approval
3121 maddy place for dink of livin poes
Not: Who
billet seage
Hibll Whet donuhh gutter dhiic in the imags
biiket in whilh sayy dal beliruze? Aumrer: Phinkote
ingee
200r Answer itail Midid
scader ttnn! Amseen Appeoval
Reliability
E-dde Wiuch exett Ziraa lhabaaarn fooe? Ammncr: Ioll
T-lddt is wiich your dal Zectery Skie bey lvrroon Ansver: V00
T-Sal: What ceet the tee gemue
14mm Who dbcirdnntaall Aguadl bike emae! Anseer: Bendeasoi
Answen Appeord
1-mi Whick disb the paoce bue imaae nodia you 21are
Iór:: dots lif gotaed slonn
1-Gei To wakc ci did Zictary Shore pastase
Generalizatin
E-Gee Who is the pesson Di magil
reprosent
perpet Amswer habiddgpa
baiide lhing"
Amieen Appoodd
Porti socatrac
Port in stach your the geer ponwe you 2330 in fraxlalt Anewer
bxdigoeed ode: wel
Net: Whr bc
Portabilitt
skona smage? Anewert Stareb
ruking lhe lts.
prom
Zachiry Swae's
choon in insqp??
porast doove Bx.
Anovert Betks
image?
Previous Benchmark
MMKE-Bench
Figure I: Comparison between the existing benchmark and MMKE-Bench with a detailed example. In this example, the texts in red represent the edited counterfactual content. T/I-Rel represents text and image reliability, T/I-Gen represents text and image generalization and Port represents portability. Previous benchmarks mainly focus on entity recognition editing using diplet-based knowledge representation format, which does not align with actual scenarios. MMKE-Benol focuses on evaluating diverse semantic editing in realistic scenarios in natural language format.
for reliability, text generalization, and image generalization, respectively, on the VLKEB bench- mark Huang et al.) 2024). This highlights the urgent need for a more challenging benchmark to foster the development of multimodal knowledge editing techniques.
To address these issues, we introduce HMKKE-Bench a comprehensive multimodal knowledge editing benchmark designed to evaluate diverse semantic editing in real-world seenarios. MMKF--Bench represents multimodal knowledge using free-form natural language descriptions paired with images, providing richer and more flexible expression of interconnected information. Reflecting real- world needs, MMKE-Bench includes three types of editing: visual entity editing, visual semantic editing, and user-specific editing. Visual entity editing updates entity-centric visual knowledge, while visual semantic editing targets complex object behaviors and relationships, such as referee gestures and traffic signals. Lastly, user-specific editing evaluates the model's ability integrate individualized knowledge. The first two types modify existing knowledge, while the third adds new knowledge. Comparisons with existing benchmarks are shown in Fig 1 and Tab
To construct MMKE- Bench, we first collect original knowledge from various images and knowledge sources (e.g., multimodal knowledge graphs, demo videos, Google, and LLM generation). Next, we create editing knowledge by applying counterfactual editing for the text modality and image replacement for the image modality. User-specific editing involves adding entirely new, personalized knowledge to the model and does not need counterfactual editing. Following previous works (Zheng ct al. 2023 Huang et al.) 2024), we adhere to four evaluation principles: reliability, locality, gener- alization, and portabilily, generating evaluation questions and answers automatically. Finally, all questions and answers undergo human verification and are revised where necessary. The resulting benchmark contains 2,940 pieces of knowledge and 8,363 images across 33 broad categories.
We evaluate five of the most prominent multimodal knowledge editing methods on three representative LMMs, assessing their performance in both single and sequential editing tasks. Empirically, we find that (i) no single editing method excels across all evaluation criteria; (ii) visual knowledge and user-specific knowledge are more difficult for LMMs to edit; (iii modern LMMs excel in producing and applying edited knowledge; and (iv) the proposed benchmark proves more challenging than previous benchmarks.
2

---

Published as a conference paper at ICLR 2025
Table I: Overall comparison with existing multimodal knowledge editing benchmarks.
Visual Entiyy Editing
Visual Semantic Editing
User Sper-Speciffc Editing
Benchmark
Knowledge Reprosentation
Exaluation Principle
MMEdit
Shoot-Tee
X
Reliability, Locality and Gemeralization Reliability, Locality, Gemeraiizaion Reliabiiity Locality, and Generalization
MIKR
Tripler
MC-MKF
Tripler
VLKEB
Triplet
Reliability, Locality, Generalization, and Portability Reliability Locality, Generalization, and Portability
MMKE-Benne
Free-Form Natural Language
5
To sum up, contribution can be summarized as follows:
We propose BMKKE-Bench challenging benchmark for evaluating diverse semantic editing in real-world scenarios. It adopts free-form natural language- knowledge representation and includes three types of editing aligned with real-world contexts.
We introduce novel pipeline for benchmark construction that collects original knowledge, generates editing knowledge, and produces evaluation questions guided by four principles.
Extensive experiments with various baseline methods and LMMs in both single and se- quential editing settings are conducted, revealing several limitations in existing knowledge editing approaches.
RELATED WORK
2.1 LARGE MULTIMODAL MODEL
Large multimodal models have achieved excellent performance in various multimodal understanding tasks due to vast knowledge and effective cross-modality alignment. Typically, such models integrate vision encoder with pertained large language model, linking the two components by alignment module. Notably, BLIP-2 (Li et al., 2023a) adopts Q-Former, a lightweight Transformer, as the alignment module. Inspired by the instruction tuning in LMM, MiniGPT-4 (Zhu et al. 2023 and
InstructBLIP (Dai et al. 2023) enhance this structure with multimodal instruction tuning. In contrast, LLaVA (Liu et al. (2224b) utilizes an MLP layer for alignment and proposes generate instruction- tuning dataset by selffinstruct strategy (Wang et al. 2022). Qwen- VL (Bai et al. (202)) introduces a novel module, the visual receptor, its alignment module and proposes three-stage training pipeline, achieving excellent performance across various multimodal tasks. Besides, several notable LMMs, such as mPLUG-DocOW 1.5 (Hu al.) (024). InternV--2 (Chen al. (2024), and MiniCPM-V
2.5 Yao et al. 2024), have also achieved comparable or even superior results compared with GPT-4o.
2.2 KNOWLEDGE EDITING FOR LARGE LANGUAGE MODEL
Existing methods for L.M can be divided into three categories: resorting to external knowledge, incorporating knowledge into the model, and editing internal knowledge. Resorting external knowledge typically involves maintaining memory and retrieving the most relevant cases for each input. For instance, IKE Zheng al. (2023) provides in-context learning example support by building three types of demo examples: copy, update, and retain. SERAC Mitchell et al. (20225) builds a new counterfactual model by keeping the base model and using scope classifier to determine whether to answer with a counterfactual model. The category of merging the knowledge into the model aims to learn representations
the new knowledge and incorporate this information into the model. Sya-KEL.LA Wu et al. (2023a) employs LoRA for knowledge editing, while GRACE (Hartvigsen ctt al. 2023) adopts a novel approach by maintaining a discrete codebook functioning as an adapter. Lastly, editing intrinsic knowledge works on directly modifying the model's weight using knowledge- specific methods through meta-learning and localization editing. The meta-learning method trains hypernetwork to learn how to adjust the model. KE De Cao al. (2021) 1) utilizes new knowledge representations directly to train the model to update the matrix, while MEND Mitchell et al.) (2022a) applies rank-one decomposition divide the model into two rank matrices. Additionally, localization approaches, like ROME Meng et al. 2002) and MEMIT, Meng et al. (2024) employ a causal analysis method to detect which parts of the hidden state are more important by treating editing as minimal optimization, ensuring its reliability and non-circumvention.
2.3 KNOWLEDGE EDITING FOR LARGE MULTIMODAL MODEL
Recently, several benchmarks have been proposed to evaluate the performance of editing LMMs. The MMEdit benchmark (Cheng et al. 2023) systematically defines the first evaluation framework
3

---

Published as a conference paper at ICLR 2025
for multimodal knowledge editing based visual question answering and image caption tasks. As the MMEdit could not assess fine- ane-grained entity knowledge, subsequent evaluation benchmarks focus on fine-grained entity recognition editing. MIKE (Li al., 2024) evaluates recognizing new entities while VLKEB (Huang et al. 2024) targets editing known entities and introduces a portability evaluation principle. MC-MKE (Zang et al.) 2024) further extends fine-grained entity recognition by emphasizing modality consistency. However, these benchmarks mainly represent editing knowledge through triples and overlook diverse semantic editing in realistic scenarios.
PROBLEM DEFINITION
3.1 KNOWLEDGE REPRESENTATION AND EDITING
MMKE-Bench is distinctive in evaluating diverse semantic editing in realistic scenarios, leveraging natural language-base based knowledge representation. It includes three types of editing: visual entity editing, visual semantic editing, and user-specific editing. Each piece of knowledge is represented in a unified format, & = (i, d). where refers to the image and represents the natural language description of the main object, visual content, user-personalized item. For example, in the of a referee's gesture, the image captures the action performed by the referee, while the description explains how the gesture is executed and its impact on the match. During knowledge editing, the original knowledge is transformed into lig = (ie, de) in both visual entity and visual semantic editing, while it remains k = (i, d) for user-specific editing. This is because user-specific editing introduces entirely new personalized knowledge into LMMs without needing to alter the image or description.
3.2 EDITING TYPE OF MMKE-BENCII
Considering real-world needs, MMKE-Bench includes three types of editing as follows.
Visual Entity Editing This type targets entity-centric modifications and the description covers multiple aspects of an entity. In realistic scenarios, models may misidentify or retain incorrect or outdated information about the entity. Visual entity editing addresses this issue by allowing for simultaneous correction of all related content. To simulate such scenarios, we propose replacing the original image of the entity with that of another entity the same type and modifying key information into counterfactual content. As shown in Fig [ Zlatan Ibrahimovié's image is replaced with that of Wayne Rooney, and related information (e.g., nationality, club) is altered to counterfactual details.
Visual Semantic Editing This type focuses on complex visual semantics-centric modifications, encompassing body gestures, actions, object relationships, and so on. The description provides de- tailed information about the semantic action and its rules or meanings. The LMMs may misrecognize and misunderstand these semantics, but visual semantic editing can address this issue by modifying both actions images, and meanings simultaneously. To simulate this, this type of editing also involves replacing the image of one semantic action with that of another action of the same type and altering the rule or meaning to counterfactual content. As shown in Fig [ the offside gesture in soccer is replaced with that of substitution, and the associated rule (e.g. kick-off location) is modified to counterfactual contents.
User-Specific Editing This type focuses on injecting personalized user information into LMMs, and the description details the relationship between the user and the object, as well as their experiences. As there is growing demand for LMMs to function as personalized Al assistants that can remember relevant user information, user-specific editing is designed to meet this need. Pre-trained LMMs serve as general models, so all user-specific information is treated as new knowledge for LMM. Thus, counterfactual editing is unnecessary, and original knowledge is used as editing knowledge. For example, Fig descrrbes the relationship between the toy puppet and the user's s habits.
4 BENCHMARK
As shown in Fig. 2 we construct the benchmark through four steps: i) Original Knowledge Collection; ii) Editing Knowledge Generation; iii) Evaluation Question Generation; and iv) Human Verification.
4

---

Published as a conference paper at ICLR 2025
Original Knowledge Collection
II Editing Knowledge Generation
Candidate
Collect Trages
Filter images
Description Ganeration
Image Replacement
Description Eddting
Subiitatio
He allake soee soc entrrd head wish shakk kand R will increl You kis mnvered
Othike Orggnal
sacie The selore di lom Rc detnd Mbc adboctt
Soacer
IV Human Verification
Evaluation Question Generation
8
Remove or Rewrite
Portahility Question
Locality Question Collection
Reliabllit,, and Generaiity Question Generation
Quesiions
5
bibt What's agaal
Ten mam woue Iowis
Khoar dom coleed elu the dire rntaree made ARtwin Concral dock
mnking yraar
Awern Othidd
Idkas Waet' the aged
Aneve: Offidd
14 Whe dleacten Stx?
Mott Warl
savatia ritrect pooar.
aecoot
Endeation "idiiaation
Part
Aurrett Tasteett
Ediie
Figure 2: The construction pipeline of MMKE-Bench.
4.1 ORIGINAL KNOWLEDGE COLLECTION
In gathering original knowledge, we first list candidate fine-grained entities, visual semantics, or user-specific items, and then collect their corresponding images and descriptions.
For visual entity editing, we source candidates from two datasets: the multimodal knowledge graph, MMpedia Wu al. (2023b), and the visual entity recognition dataset, OVEN Hu et al. (2023). For each entity selected from the existing dataset, we get their images from the datasets and then manually review the images by removing the entities that cannot uniquely identify the main entity from images and noise images. For entities with less than two images, we recollect additional images by crawling from Google. Next, we retrieve entity descriptions from the Wikipedia summary dump and summarize the description by LLM to generate the final descriptions. As shown in Fig. this type covers 10 broad categories.
For visual semantic editing, as shown in Fig. 3 we define the candidates across broad categories of semantic knowledge, including single-person behaviors, single- object behaviors attributes, object rela- tionships, and global structures. For certain types of visual knowledge that have corre- sponding datasets, such as object relation- ships, textures, and art styles, we collect both the candidate semantics and associ- ated images from these datasets. For other cases, we extract images from demonstra- tion videos or gather them via Google, ap- plying human verification for quality con- trol. Descriptions of the visual semantic actions, along with the rules or meanings conveyed by these behaviors, are generated with the assistance of LLM human writ- ers. Details of the image sources are pro- vided in the appendix.
terrestrial
plant
musical group
person
seriel aniimas
building
Visual Entity Editing
vehicle
St monter buttin
character
marie preat
User Specific Editing
atomel Beay
traffic sign
RUN
company
organization
huran action
texture
referes gessure
relationshul
sports club
Cartoon chara
Nino and
universte
actor
singer
Figure The types of samples in MMKE-Bench.
For user-specific editing, we consider 9 broad categories of personalized information sources, such as favorite singers, owned pets, and alma maters. For personal items and pets, gather candidates and images from the existing personalized research works Nguyen et al. (2024); Alaluf et al. (2024). For singers, actors, and cartoon characters, first generate candidate list and then crawl images from Google. For other categories, including company. university, sports club, and organization, we source candidates from MMpedia, manually verifying and removing noise images. Finally, we employ an LLM to generate personalized relationships and experiences between the user and these objects.

---

Published as a conference paper at ICLR 2025
4.2 EDITING KNOWLEDGE GENERATION
Considering the multimodal nature of large multimodal models (LMMs), we propose editing both text and visual modalities when constructing the benchmark. Specifically, we focus on editing visual entities and visual semantic knowledge while leaving user-specific knowledge unchanged. The former treated as knowledge editing, while the latter is regarded as knowledge insertion.
For the visual modality, we follow the reauge-rpplaeemme--beeeed editing approach from previous work Huang et al. (2024), where an image of the entity or semantic action randomly replaced with another of the same type. For example, as illustrated in Fig. and 2 the assistant referee's 's offside penalty gesture is replaced with a substitution gesture the edited visual content. In the text modality, we modify key information about the entity and the rule meaning into counterfactual content for visual entity editing and visual semantic editing, respectively. Additionally, we update the action description to align with the new visual content. In the example of the offside gesture, the original action description is replaced with that of the substitution gesture, and the kick-off location is edited from the foul position to the penalty spot.
4.3 EVALUATION QUESTION GENERATION
We adhere four key evaluation principles generate both the questions and answers. The reliability and portability questions are generated by prompting LLM and we show the prompts in the appendix.
Reliability Question Generation The reliability criterion assesses whether the edited knowledge is correctly produced after the editing process. When generating questions and answers, we prompt the LLM with requirement that the question must ask one aspect of the edited counterfactual content (e.g., the kick-off location of the offside penalty). To evaluate this, we consider both text reliability and image reliability, measuring the LMM's s ability to edit across text and visual modalities. Text reliability questions are crafted to be answerable without images, while image reliability questions use the format {the type in the image} to reference the main object. behavior, or personalized item. An example is provided in F We denote the reliability question sets as Qret = (ie, Ir, ar), where ie represents the edited image, Gr the question, and o, the answer. Let Mg and Mr denote the original and edited LMMs, respectively, and 1[-] denoted indicator function, reliability is then evaluated as:
E 16 [M] (fe. 9r) = a,
(1)
Locality Question Generation The locality criterion evaluates how much unrelated knowledge remains unchanged in the edited model by comparing its outputs before and after the editing process. For locality, we assess both text and image locality, which tests the model's 's stability when dealing with out-of-scope knowledge from each modality. Following prior work, we source locality questions and answers from the VLKEB benchmark Huang et al. (2024), where the text questions are drawn from the NQ dataset dataset/Kwiatkoww Kwiatkowski et al.) (2019). and the mmge questions are specifically designed by VLKEB. We represent the locality question set as Qtee gr), and locality is evaluated as:
(( [Mo qt) Mb qt)
(2)
Generalization Question Generation The generalization criterion evaluates how effectively the model responds to neighboring samples. Unlike triplet-based knowledge editing, we focus exclusively on image generalization, as text generalization is not considered due the free-form knowledge format. For image generalization, we randomly select another image ig from the multiple available images of an entity, visual behavior, or personalized item, and reuse the same question and answer from the image reliability, with an example shown in Fig. [ We define the generalization question as Qgen (i%, 9g. ag), where 4g = 9r and ag = a, for the same object. Generalization evaluated as:
E E gem II [M M1 (g) agl
Portability Question Generation The portability criterion evaluates whether the edited knowledge can be successfully applied to related content. Following prior work HHung al. (2024), adopt text portability evaluation for visual entity editing and image modality portability for visual semantic and user-specific editing to enhance visual modality evaluation.
For visual entity editing, we generate questions about the edited content, utilizing supplementary information from Wikipedia for question generation. For example, if the current entity is the Eiffel

---

Published as a conference paper at ICLR 2025
Tower and the edited content refers to the building's designer, might create question like, "Who is the designer of the Eiffel Tower?" We can then generate another question about the edited content, such asking for the designer's 's birth year. By combining these two questions, can formulate the final probability question: "In which year was the builder of the Fiffel Tower born?"
In the case of visual semantic and user-specific editing, behavior or item with another image of the same type to create a new image, denoted as i. We then pose question focusing on the differences between the two images, such hair color or object shape. By integrating this question with one related to the edited content, we derive the final portability question. For instance, as shown in Fig. 2 given an image that includes the offside penalty gesture and the corner-kick gesture made by two assistant referees, we might ask, "What color is the tops of the referee who is making the offside gesture in the image?". Denote the portability question as Qport (i% Gp, (p), portability is evaluated as:
first combine the image of the main
E(E [M 9p) = =apl
4.4 HUMAN CHECK & BENCHMARK STATISTICS
During benchmark construction, we manually collected, reviewed, and filtered the samples multiple times. In the original knowledge collection stage, we conducted a thorough manual review of the images associated with each entity, behavior, and object ensure the quality of the collected visuals. Furthermore, after counterfactual editing and question generation, we manually reviewed the questions, revised unsuitable questions, and corrected wrong answers.
The statistics of MMKE- Beecch are shown in T MMKE- Bench encompasses three classes of edited knowledge, totaling 2,940 knowledge pieces and 8,363 images. The knowledge spans 175 fine- grained types, highlighting the diversity of MMKE- Bench. We split the dataset into training and valida- tion sets at 4:6, with the training reserved solely for specific knowledge editing methods (e.g., SERAC Mitchell et al.
Table 2: The statistics of AMKE-BBnche
Types
Train
Test
Images
Visual Entity Editing
76
636
955
3,534
Visual Semantic Editing
293
3,201
Uper-Specific Editing
34
331
511
1,628
(022bb) )
5 EXPERIEMENT
5.1 EXPERIMENTAL SETUP
LMMs and Editing Methods To evaluate our benchmark, we conduct experiments on three representative LMMs: BLII-2 (Li et al.) 2023a), MiniGPT-4 (Zhu al., 2023). and VA-15. (Liu) al. 2024a). Besides, following the previous benchmarks, we select five representative multimodal knowledge editing methods: 1) Fine-tuning (FT). We focus on finetuning the LLM (FT-LLM) or the lanion-language alignment module (FT- (FT-Alligmment), where only the last layer of the LLM is fine- tuned.2) Knowledge Editor (KE) (De Cao et al. 2021). KE uses a nyper-networt with constrained optimization to predict the weight update at test time. 3) MEND (Mitchell et al. 2022a): MEND learns a low-rank decomposition of the gradient of standard fine-tuning. 4) SERAC (Mitceell et al. 2022b): SERAC is a memory-based method and it stores edits in explicit memory. 5) In-context
Knowledge Editing (IKE) (Zheng et al. 2023): IKE is inspired by in-context learning, and new demonstration formatting and organization strategies are to construct for guiding knowledge editing.
Experiments settings We perform experiments under both single editing and sequential editing. Single editing is mostly adopted and it updates the base model for each piece of knowledge and then evaluates the editing performance. The sequential editing continuously updates the base model with multiple pieces of knowledge and then evaluates the first piece of knowledge. We follow the previous benchmark and adopt the token-level editing accuracy.
5.2
REULTS
5.2.1
SINGLE EDITING RESULTS
The results of the existing multimodal knowledge editing methods on BMMKE-Bencl are shown in Tab. 3 Tab. A and Tab. 5. Based on the results, we have several observations.
1) FT-LLM is a strong baseline, while IKE demonstrates the best reliability and generalization. FT-LLM serves strong baseline, with other multimodal knowledge editing methods like SERAC, MEND, and KE performing similarly or even worse than FT-LLM. Notably, IKE achieves the best results across nearly all knowledge editing tasks for three LMMs, excelling in text reliability, image
7

---

Published as a conference paper at ICLR 2025
Table 3: The results of single editing for BLIP-2 MMKE-
Method
T-Loc
I-Loc
T-Rel
I-Rel
I-Gen
Port
FT-LLM
69.76
21.47
39.21
35.76
36,21
18.11
FT-Alignment
100.00
8.83
20.89
27.51
27.02
19.25
Visual Entity Editing
IKE
55.77
13.19
41.80
41.76
25.93
SERAC
99.99
99.69
20.90
20.27
20.49
19.76
MEND
96.02
69.37
35.67
34.41
34.48
21.31
KE
83.61
18.02
28.14
28.25
28.46
30.76
T--LMM
64.11
19.25
33.42
30.79
30.71
2.76
Aliggmmtttt
100.00
9.48
T8.17
35.81
32.67
5.15
Visual Semantic Editing
IKE
47.10
13.92
35,56
42.07
41.1
5.03
SERAC
99.90
99.16
18.26
18.61
17.96
3.81
MEND
97.29
74.35
28.26
30.79
31.11
3.87
KE
67.85
14,39
30.97
24.48
24.85
6.70
FI-LLMM
61.28
211.49
12.52
27.33
27.80
5.46
TTAAligmment
100.00
8.74
7.46
17.19
17.31
6.17
User-Specilic Editing
IKE
47.39
12.25
13.25
31,04
30.71
6.03
SERAC
100.00
99.76
7.46
14.20
14,50
5.10
MEND
96.95
76.21
25.21
5.22
KE
65.70
15.73
12.79
19.83
19.71
10.80
FT-LLM
65.05
211.00
28.38
31.29
31.57
8.78
Tl-Alignment
100.00
9.02
15.51
26.84
25.67
10.19
IKE
50.09
13.12
30,23
38.30
37.86
12.33
Average
SERAC
99.96
99.54
15.54
17.69
17.65
9.56
MEND
96.75
73.31
30.14
KE
72.39
T6.05
23.97
24.19
24.34
16.09
Table The results of single editing for MiniGPT4 on MMKE- Bencch.
Method
T44.cc
1-1.cc
T-Rel
I-Rel
I-Gen
Port
FT-LLM
FT-Alignment
100.00
24.85
31.89
33.87
33,93
30.79
Visual Entity
IKE
75.50
15.25
56.42
53.80
53.72
41.09
Editing
SERAC
99.97
99.76
31.88
30.53
30.35
33,43
MEND
97.49
77.70
47.26
42.20
41.82
34.43
40.44
FT-LLM
83.96
31.54
44.45
44.85 9
43.91
8.16
FT-Aligmment
100.00
25.20
24.93
46.45
42.29
11.43
Visual Semantic
12.79
$5.44
53.01
10.50
Editing
SERAC
98.70
98,80
27.08
29.65
28.33
10.35
MEND
97.34
77.16
37.45
42.17
42.62
8.65
KE
84.14
21.25
38.14
35.23
33.94
14.72
FT-LLM
83.13
34.04
39,74
38.94 4
38.60
10.53
FT-A Alignment
100.00
25.30
21.07
33.25
33.40
12.33
User-Specific Editing
IKE
75.35
14.56
61.55
54.81
11.85
SERAC
100.00
99.90
21.09
30.63
30.27
10,50
MEND
97.47
79.19
28.70
40.94
40.25
11.34
KE
78.46
20.12
22.60
37.91
37.72
19.92
FT-LLM
83.74
32.37
44.47
41.64
41.30
16.65
Aliinmment
25.96
36.54
Average
IKE
72.43
14,20
57.80
54,50
53.85
21.15
SERAC
99.56
99.49
26.68
30.27
29.65
18.09
MEND
97.43
78.02
37.80
41.77
41.56
18.14
KE
79.68
19.95
34.01
37.72
37.37
25.40
reliability, and image generalization. These results indicate that in-context examples significantly enhance the model's understanding of how knowledge is edited. leading to superior performance.
2) Image locality is more challenging than text locality, and SERAC and MEND perform best in maintaining locality. Most knowledge editing methods deliver better text locality results compared to image locality, suggesting that editing LMMs tends to compromise visual knowledge more severely. resulting in lower image locality scores. SERAC and MEND stand out by achieving high locality results. It may owe to the good retrieval accuracy of SERAC and fewer parameter updates by MEND.
3) All knowledge editing methods generalize well but struggle with portability. The I-gen results mirror those of I-rel, indicating that current large multimodal models extract invariant features across different image variants of the same object. However, all existing multimodal methods fall short in the portability evaluation, highlighting the difficulty of applying edited knowledge to content. KE performs best portability in most scenarios, suggesting that parameter-baas based editing methods handle this challenge
effectively.
4) Visual Semantic Knowledge and User-Specific Knowledge are more difficult for LMMs to edit. Editing complex visual semantics and user-specific knowledge proves more challenging than
8

---

Published as a conference paper at ICLR 2025
Table 5: The results of single editing for LLaVA MMKE-Bendee
Method
T-Loc
I-Loc
T-Rel
I-Gen
FT-LLM
77.71
17.58
53.89
49.54
49.30
41.23
FT-Aligument
100.00
9.15
35.72
38.65
39.74
37.62
Visual Entity Editing
IKE
68.25
17.43
63.49
59.98
51.30
SERAC
99.87
99.26
35.7
35.02
34.98
40.24
MEND
97.32
75.29
51.30
47.21
46.58
41 83
KE
79.89
18.73
46.45
46.19
46.29
48.77
F FT-LLM
77.81
16.11
49.18
48.28
47.49
14.48
FT-Aligmment
100.00
11.45
28.92
51.41
40.72
27.84
Visual Semantic Editing
IKE
64.11
19.44
63.54
61.92
61.31
SERAC
99.90
99.98
29.01
29.97
29. 17
20.73
MEND
98.27
82.90
41.21
46.64
45,90
23.29
KE
74.61
7.95
47.82
38.78
37.49
24.07
FT-LLMM
75.08
20.41
58. 18
47.
48.56
13.11
TT-Aligmment
100.00
10.87
42.40
40.21
43.65
23.35
User-Specific Editing
IKE
63.48
18.93
75.65
62.73
62.79
22.87
SERAC
99.99
99.81
42.24
36.29
36.67
13.63
MEND
50.92
KE
79.51
10.80
54.85
48.65
49.46
23.67
FT-LIMM
76.87
18.03
53.75
48.54
48.45
22. 94
T-Alinmment
100.00
10.49
35.68
43.42
41.37
29.60
IKE
65.28
18.60
67.56
61.54
61.36
33.42
Average
SERAC
99.92
99.68
35.65
33.76
33.61
24.87
MEND
47.81
45.78
KE
78,00
12.49
49.71
44.54
44.41
32.17
editing visual entities, as evidenced by lower reliability and portability scores. This suggests that more advanced editing techniques are needed to edit complex visual semantics and inject personalized information, further emphasizing the value of the proposed benchmark.
5) Modern LMMs excel in producing and applying edited knowledge. For reliability, generaliza- tion, and portability evaluations. LLaVA-1.5 outperforms BLIP-2 and MiniGPT-4. This improved performance can be attributed to its larger model size and better instruction -oollowing capability, as L.LVA-1155 has more parameters than BI.IP-2 and more refined instruction-tume -tuning design than MiniGPT-4. These factors lead to its superior ability to understand and apply evolving knowledge.
6) No single editing method excels across all eval- uation criteria. In conclusion, no single knowledge editing method outperforms across all four evalua- tion criteria. In-context learning-bas based methods are strong at reproducing edited knowledge, memory- based methods excel at preserving unrelated content, and parameter-base based methods are better at applying edited knowledge new contexts.
I-Rel
T-Loc
7) The proposed benchmark is more challenging than previous ones. The comparison of IKE with existing benchmarks for MiniGPT-4 is shown in Fig. 4 this method achieves high scores across most evaluation principles in previous benchmarks but performs worse on our benchmark. This suggests that the proposed benchmark introduces greater chal- lenges than its predecessors.
1-Gen vam
Port
Figure 4: Evaluation comparison of IKE for MiniGPT-4 with existing benchmarks. Port for MMEdit and MIKE, is set 1, as they are not evaluated.
5.2.2 SEQUENTIAL EDITING RESULTS
Editing knowledge separately is impractical in real-world applications while continuous updates with vast amounts of information are necessary. Consequently, we conduct sequential editing experiments and utilize FT-LLM, FT-Alignment, and SERAC as editing methods. IKE and KE are excluded because the edit samples also need to serve as test samples, which is not feasible in this context.
The results for VA- are shown in Tab. 6 where the "gap" refers to the sequential length, and "user num" is the number of users, with each user allowed maximum of nine personalized items. As observed, both FT-LLM and FT-Alignment tend to forget the previous editing, as shown by the decreasing performance in text and image reliability and generalization with increasing gap. In contrast, SERAC effectively maintains edited knowledge due to its explicit memory. Additionally, FT-Alignment often preserves unrelated text outputs, while FT-LLM exhibits the opposite behavior.

---

Published as a conference paper at ICLR 2025
Table 6: The results of sequential editing for 1L on HIMKE-BenCe
Method
GAP User Num
T-Lac
I-Loc
T-Rel
I-Rel
I-Gen
Port
78.91
18.16
52.80
48.
48.51
42.88
FT-LLM
3
58.10
8.34
50.99
46. 12
46.41
39,64
10
58.18
8.09
50.44
43.78
44.50
38.64
Visual Eutity Editing
100.00
37.14
38. 38.46
39.45 99.44
37.65
FT-Aligmment
3
100.00
1.10
37.14
36.
33,03
37,83
6
100.00
1.58
37.14
30.82
28.11
35,76
31.42
37.95
99.76
99.24
37.09
34.37
33.88
40.09
SERAC
3 99.69
98.37
37.09
34.35
33.90
40.11
99 69
36
37,09
34.
33.90
40.11
10
99.69
95,35
37,09
34.35
33.90
40.16
76.89
FT-LLM
3
50.33
7.36
42.85
76
45.02
8.29
6
49.09
7.25
41.49
45.
43.52
7.25
100.00
19.41
27.83
44.5
35.37
15.00
Visual Semantic Editing
FT-Alignment
3
100.00
1.44
34.06
24.57
100.00
1.38
27.83
31.62
23.54
6.96
10
100.00
1.38
27.83
29.79
23.92
7.25
11.29
SERAC
3
99.93
13.56
27.99
29.71
30.70
11.17
6
99.93
13.54
27.92
29.91
31,09
11.34
99.93
29. 29.93
31.13
11,23
75.44
20.13
58.11
48. 25
49.
13.19
FELLM
1
44.97
3
68.87
17.9%
51.26
42.60
43.14
7.54
68.31
19.41
50.73
41.56
41.67
6.99
User-Specific Editing
100.00
FT- FT-Alignment
100.00
15.62
42.25
27:17
25.62
6.57
100.00
14.26
42.25
33.21
31.71
7.99
100.00
16.57
42.25
29.24
28:01
6.45
99.98
99.73
41.18
37.30
37,79
13.64
SERAC
100.00
100.00
42.03
37.95
38.11
12.55
100.00
100.00
42.03
37.95
38.11
12.55
5.3 INSIGHT ANALYSIS
Case Study An editing example of visual entity editing by IKE and FT-LLM for LLaVA-1.5 is presented in F Both IKE and FT-LLM correctly answered the text reliability question. However, IKE outperformed FT-LLM by also providing correct answers to the image generalization and portability questions, highlighting IKE's superior performance. The case study of question answers visual semantic editing is shown in Fig As see, after editing, the model could effectively answer the question based on editing knowledge.
6
hemaal lmager Mhad Froms iiccost bo yooee IS AND LOV Plyv tech
vs ocsuhic reigging arta paran by rollec bakatall taa
T-Htal
G-Gen
Snuviiide
offici
an oppnent
1-Re
Viu
Part
Laagse ribess
Free
ninehe (kll relking
Figure 5: Case study of editing examples
Figure 6: Case study of question answer
6 CONCLUSION
In this paper, we propose comprehensive multimodal knowledge editing benchmark, named MMKE- Bench, designed to evaluate diverse semantic editing in real-world scenarios using free-form natural language representation. We propose to use free-form natural language representation combined with an image to represent knowledge instead of representing it with a triplet. Besides, we propose three kinds of editing align with real-world scenarios. We conducted experiments representative LMMs and knowledge editing methods and found that more advanced knowledge editing methods needed for LMMs. We hope work could inspire more multimodal knowledge editing research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is supported by the Opening Project of the State Key Laboratory of General Artificial Intelligence (Project No:SKLAGI20240P1),,
REFERENCES
Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvlm: Personalizing vims for user-specific queries. ECCV, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. Arxiv, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and et al. Language models are few-shot learners. NeurIPS, 2020.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. EMNLP. 2021.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites, ArXiv, 2024.
Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang. Huajun Chen, and Ningyu Zhang. Can we edit multimodal large language models? In EMNLP, pp. 13877-13888, 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, pp. 3606-3613, 2014. 2014.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023.
Art Dataset. wiki art dataset. url http:://miireee...obbbbwoo////as visited on 2023-01-18.
mar 2022. URL https::/universss
Nicola De Cao, Wilker Aziz, and Ivan Titov, Editing factual knowledge in language models. ACL, 2021.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with GRACE: lifelong model editing with discrete key-value adaptors. In NeurIPS, 2023.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. ArXiv, 2024.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wee Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In CVPR, pp. 10005-12075... 2023.
Han Huang, Haitian Zhong, Tao Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. Vlkeb: A large vingunelanguage model knowledge editing benchmark. arxiv, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. TACL, 7:453-466, 2019.
Jiaqi Li. Miaozeng Du, Chuanyi Zhang, Yongrui Chen, Nan Hu, Guilin Qi, Haiyun Jiang, Siyuan Cheng, and Bozhong Tian. MIKE: A new benchmark for fine-grained multimodal entity knowledge editing. In Findings ACI., pp. 5018-5029, 2024.

---

Published as a conference paper at ICLR 2025
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In JCML, pp. 19730-19742, 2023a.
Zichao Li, Ines Arous, Siva Reddy, and Jackie Chi Kit Cheung. Evaluating dependencies in fact
editing for language models: Specificity and implication awareness. In EMNLP, pp. 7623-7636, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pp. 26296-26300 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning NeurIPS, 36, 2024b.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In NeurIPS, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory a transformer. In Findings of ACI., 2024.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. ICLR, 2022a.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn, Memory- based model editing scale. In ICML., pp. 15817-15831, 2022b.
Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo'llava: Your personalized language and vision assistant. ArXiv, 2024.
Yasumasa Onoe, Michael JQ Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can Ims learn new entities from descriptions? challenges in propagating injected knowledge. ACL, 2023.
Suchen Wang, Kim-Hui Yap, Henghui Ding, Jiyan Wu, Junsong Yuan, and Yap-Peng Tan. Discovering human interactions with large-vocabulary objects via query and multi-scale detection. In ICCV, pp. 13475-13484, 2021.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct Aligning language models with self teff-generaee instructions. ArXiv, 2022.
Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, and Mingming Sun. Eva-kellm: A new benchmark for evaluating knowledge editing of Ilms. ArXiv, 2023a.
Yinan Wu, Xiaowei Wu, Junwen Li, Yue Zhang, Haofen Wang, Wen Du, Zhidong He, Jingping Liu, and Tong Ruan. Mmpedia: large-scale multi-modal knowledge graph. In International Semantic Web Conference, pp. 18-37. Springer, 2023b.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: gpt-4v level milm on your phone. ArXiv, 2024.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. Findings of EMNLP, 2023.
Junzhe Zhang, Huixuan Zhang, Xunjian Yin, Baizhou Huang, Xu Zhang, Xinyu Hu, and Xiaojun Wan. MC-MKE: A fine-grained multimodal knowledge editing benchmark emphasizing modality consistency. Arxiv, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. ArXiv, 2023.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? EMNLP, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. Arxiv, 2023.
12

---

Published as a conference paper at ICLR 2025
Table 7: The image source of visual semantic knowledge in MMKE-Bench.
Type
Source
Human Action
Crawling from google
Life Gesture
Crawling from google
LFW-emotio dataset
Emotion
pttps: htlpa//acc.......aaaaaaaaaaaaaea
Referce Gesture
Dema Vidcos mon Youruss and Biann
Trafiic Cop Sign
Crawling from google
TSRD dotaset
Trafic Sign
stt.pe//nlor--e htt////rr.......
Texture
1717 aitter Kimpa ara 2014
Color
Crawling from googie
Shape
Crawling
google
Animal Body Language
Crawling from google
Relationship
Siwg-HOI (Wang et 3021 aa]
Social action
Crawling from google
Layout
Crawling gougle
Art Style
Wikii-rt daraset Dataset
betpa:: //hugg
Cb/ www.eatttttttteeeeooooooeddeeeee www.eeeteeeeeeeeeeeeeeeeeeeeeeee inginng-n ynng--atlllelleaaa lass f feaat
A BENCHMARK CONSTRUCTION
A.1 ORIGINAL KNOWLEDGE COLLECTION
In our process of gathering original knowledge, we begin by listing candidate fine- -eeeigrrand entities, visual semantics, user-specific items, and subsequently collect their corresponding images.
For visual entity editing, we source candidates from two datasets: The multimodal knowledge graph, MMpedia (Wu et al. (023b), and the visual entity recognition dataset, OVEN (Hu et al. (023). Given the extensive size of MMpedia, we filter entities with Wikipedia summaries of fewer than 40 words and eliminate candidates that cannot uniquely identify the main entity through images. Using the Wikipedia API, we retrieve the entity type and select the most popular 10% within each type. We further apply optical character recognition (OCR) to exclude images containing entity names, such as university logos. After this, we gather images from the relevant datasets and manually remove any noisy images, or crawl additional images from Google for entities with fewer than two images. The same process is applied to the OVEN dataset, except without sampling.
For visual semantic editing, we first list the semantic candidates from four broad categories: single- person behavior, single-object behavior or attributes, object relationship, and global structure. The single-person behavior includes human action, life gestures, referee gestures, traffic cop signs, and emotion. The single-object behavior attribute covers animal body language, traffic signs, color, shape, and texture. The object relationship involves human-object interactive relationship and social actions, while global structure encompasses layout and art style. Where datasets exist, such for texture, we gather the entities and images from existing sources. Otherwise, we manually curate the candidates using domain expertise and collect images from various sources. The sources for each type are listed in Tab Specifically, images for human action, life gestures, traffic cop signs, color, shape, social action, animal body language, and layout crawling from Google. Images for traffic signs, textures, relationships, emotions, and art styles come from existing datasets. Referee gesture images are collected by extracting frames from demo videos on YouTube and Bilibili.
As for user-specific editing, we consider nine types of personal information, including items, pets, actors, singers, cartoon characters, organizations, universities, sports clubs, and companies. The candidate relationships between users and these objects are outlined in Tab(9 including examples like "employed at," "exchanged at," "studied at," and "favorite" for universities. We collect images for these items from various sources. For items and pets, candidates and images are sourced from existing datasets used for personalized large multimodal research (Nguyen et al., 20244 Alaluf et al.) 2024). For organizations, universities, sports clubs, and companies, follow the same process in visual entity editing, using data from MMpedia. For actors, singers, and cartoon characters, images collected from Google.
To sum up, this benchmark covers a total of 2,940 pieces of knowledge, along with 8,363 images from 33 broad categories, and detailed type names are shown in Tab
After collecting the images, we generate natural language descriptions for each entity, visual semantic, and user-specific item. For visual entities, we retrieve descriptions from the Wikipedia summary, and
13

---

Published as a conference paper at ICLR 2025
Table 8: The data type in MMKE-Bench.
Broad Categories
Types
Person
Human
Acrial Animals
Bird, Dragonfly. Fly. Butterfly, Grasshopper, Wasp, Insect, Animal
Marine Animals
Jellyfish, Turtle, Sea Star, Fish, Crab, Sea Lion Bear, Monkey, Amphibian, Mammal, Rodent, Wild Boar, Squirrel, Dog Breed, Fux. Wolf. Tick, Rabbit, Rhinoceros, Arthropod, Salamander, Spider, Mol- lusc, Crustacean, Toad, Cat Breed, Deer. Beetle, Sloth, Frog. Mollusk, Snail, Hedgehog, Cat, Leop- ard, Pangolin, Dog, Cattle, Millipede, Moth, Snake, Lizard, Antelope
Terrestrial Animals
Visual Entity
Editing
Virtual Character
Animated Character, Anime Character, Comics
Character
Plant
Fruit, Tree, Flower, Mushroom, Orchid, Vegetable,
Fungus, Plant
Building
Building, Church Building, Monument, Tower,
Sculpture, Statue
Musical Group
Musical Group
Vehicle
Car, Aireralt Model, Aireraft, Vehicle Instrument, Ball
Others
Human Action
Body Posture Adjustments, Head Adjustments, Hand Actions, Leg Actions, Whole-Body Actions, Eye Expressions, Facial Expressions, Water Sports, Sound Actions, Object Actions, Repair or Construc- tion Actions, Cleaning, Humting, Crushing, Human Body Actions, Stabbing. Sticking or Connecting Ac- tions, Tools or Weapons Actions, Cutting. Packaging or Storage Actions, Pinching, Inspection or Obser- vation Actions Life Gesture Number, Life Gesture
Life Gesture
Emotion Referee Gesture
Emotion Sign
Succer Linesman, Soccer, Basketball, Volleyball, Volleyball Card, Baseball, Puck, Fencing, Handball, Badminton, Table Tennis
Traffic Cop Sign Traffic Sign
Traffic Cop Sign
Traffic Sign Forbidden, Traffic Sign Allow, Traffic
Sign Point
Texture Color
Texture
Visual Semantic
Color
Editing
Animal Body Language
Monkey Body Language, Cat Body Language, Dog Body Language, Animal Actions
Shape
Circular Shapes. Triangles, Special Plane Shapes, Common Polyhedrons, Solids of Revolution, Special Shapes
Social Action
Social Action, Agriculture, Cooking Actions, Using Tools, Communication Giving Actions, Painting
Depicting
Art Style
Art Style
Layout
Layout
Relationship
Burning Scalding, Containers Liquids Actions, Striking, Impacting, Solids of Revolution, Protection
Item
Cup. Toy Puppet, Statue, Toy, Plush Doll, Toy Doll, Puppet Cow, Cat Figurine, Bean Bag. Saving Pot, Shoes, Pillow, Pen Container, Throw Pillow Doll
Actor
Actor
Singer
Singer
User-Specific Editing
Cartoon Character
Cartoon Character
Organization
Nonprofit Organization, Organization
University Sports Club
University, Private University
Buseball Team, Basketball Team. Sports Club,
Sports Team, Futsal Team FFoobball Club Pet dog, Pet cat Airline, Company, Public Company, Dot-Com Com- pany, Media Company
Pet
Company
14

---

Published as a conference paper at ICLR 2025
Table 9: The relationship between humans and the objects and data source of user-specific data in MMKE-
Categories
Relationship
Image Source
Company
Employed Interned at, collaborated with, Favorite
MMpeddaa
Organization
Helped Employed at. Exchanged at, Studied at, Traveled to, Favorite
Employad
MMpodia
University
MMpodia
Club
Employed at, Visited, Favorite Favorite
MMpodia
Castoon charneter
Crawling from Google
Favarite, Admire most
Crawling from Google
Singer
Favorite, Admire most
Crawling from Goople
Pet
Owned
MyVLM Alaluf ct a) 2024 and LoLLaVA
Nguven al 2024
liem
Owned
MyVI.M
and aoLLaVA
Ngnyan
the summary is too lengthy, we use a large language model (LLMM) to condense to fewer than 100 words. For visual semantic editing, the description includes both language description of the action and an explanation knowledge by ourselves or generated with the help of LLM. For user-specific editing, select one relationship from the candidate list and use an LLM to craft a personalized description of the user's personal information.
its meaning or rule. These are gathered either from relevant domain
A.2 EDITING KNOWLEDGE GENERATION
After collecting the original knowledge, we perform counterfactual editing to generate alternative knowledge for both visual entity and visual semantic editing. To achieve this, we prompt a large language model (LLM) with in-context examples. For visual entity editing, we modify key details, such as nationality, alma mater, and occupation of a person, into counterfactual variations. For visual semantic knowledge, alter the rules or meanings, such as the location where free kick is taken, into counterfactual scenarios. The specific prompt used is shown in Tab
In addition text-based editing, we also perform image modality editing by replacing the image of an entity or action with one from another entity or action of the same type. This replacement strategy consistent with existing benchmarks (Huang al. 2024).
A.3 EVALUATION QUESTION GENERATION
When generating evaluation questions, adhere to four key principles: reliability, locality, gen- eralization, and portability. For locality questions, we source them from existing benchmarks. For reliability, we generate questions by prompting large language model (LLM) with in-context exam- ples, ensuring that each question is related to one of the edited contents. In image reliability, we refer to the main object in the image using its type, such as "the person in the image." " For portability, during visual entity editing, we follow previous benchmarks by providing additional information about the edited content to ensure text portability. In visual semantic editing and user- specific editing, we focus on image portability by combining the current object's image with another object of the same type. We then create a final one-hop question by merging the counterfactual content-related question with easier, maage-based question, such as asking about the color of shoes. After generating the questions and answers, we conduct a human review to verify the accuracy, rewriting any incorrect questions answers. The prompts used for question generation shown in Tabgand and Tab
B EXPERIMENTS
We conduct experiments using the VLKEB library? which employs PyTorch and integrates several knowledge editing methods and large multimodal models. The experiments are performed on NVIDIA A100/A800 80GB GPUs. The knowledge editing methods, and large multimodal models adopted in this study are listed below, with their parre--rramtters detailed in Tab 10 Tab II] and Tab T2
MLLMs. To evaluate our benchmark, we conduct experiments on three representative MLLMs.
https://github.
15

---

Published as a conference paper at ICLR 2025
HLLce
I-Rel
T-Loc
I-Gen
VICD
Figure 7: Evaluation comparison of IKE for BLIP2 with existing benchmarks. I-Gen and Port for MMEdit, along with Port for MIKE, is set 1, as they ignore the relevant criteria.
BLIP-2 (Li et al. 2023a BLIP2 effectively leverages both frozen pre-trained image models and language models by bootstrapping vision- languaee pre- pre-training. and bridges the modality gap with a lightweight Querying Transformer. We follow previous work (Huang ct al. 2024; Cheng et al. 2023). and select BLIP-2 OPT as the basic edit model, where the vision model ViT-I. and L.M is OPT model.
MiniGPT-4 (Bai al.
)023): MiniGPT-4 aligns frozen visual encoder module with frozen advanced L.M using one projection layer. The LLM is Vicuna and the vision model
is VIT.
LLaVA-1.5 (Liu et al. 2024b:: VLa is an improved version of LLaVA, which is an end-to-end trained large multimodal model that connects vision encoder and an LLM with MLP projector for visual and language understanding. We select LLa LLaVA-1.3 VA-1.5 7B as the base model where CLIP-ViT-L-23 -UPP-ViT-L-336px is the vision model and Vicuna-7B is the LLM.
Following the previous benchmarks (Huang et al. 2024), select five repre- sentative multimodal knowledge editing methods to conduct experiments.
Editing Methods.
Fine-tuning (FT): Fine-tuning has become a widely used strategy for adapting pre-train models to specific tasks. We focus on finetuning two parts: the LLM and the vision- vision-language alignment module, where only the last layer of the LLM fine-tuned.
Knowledge Editor (KE) DD Cao al. 2021> KE is method that be used edit this knowledge the base model without the need for expensive retraining or fine-tuning. uses a hyper-network with constrained optimization predict the weight update
time. MEND Mitchell et al. 2022a): MEND makes fast, local edits to a pre-trained model's s behavior using single desired input-output pair. learns to transform the gradient of standard fine-tuning, using a low-rank decomposition of the gradient.
SERAC (Mitchell et al. 2022b): SERAC is memory- method and it stores edits in explicit memory. It also introduces a scope classifier and counterfactual model, where the scope classifier is to determine whether the memory contains inputs relevant to processing them. determined, the input combined with the most relevant cache item into the counterfactual model for prediction.
In-context Knowledge Editing (IKE) (Zheng et al. 2023): IKE is inspired by in-context learning, and new demonstration formatting and organization strategies construct suitable in co-tent learning demonstrations for guiding knowledge editing.
C
MORE RESULTS
Comparison of evaluation results with existing benchmarks for BL.IP The Comparison of
evaluation results with existing benchmarks of IKE for BLIP2 shown Fig. 7 we can see, IKE achieves high results in existing benchmarks, while it performs worse in benchmark, indicating the proposed benchmark is more challenging.
Results of sequential editing for BLIP-2 We additionally report the results of sequential editing for BLIP-2 on MMKE- Bench, as shown in Tab[13] As we can see, FT-LLM and FT-Alignment tend to forget previous knowledge while SERAC is better at keeping edited knowledge.
16

---

Published as a conference paper at ICLR 2025
Table 10:
The hyper-parameters of knowledge editing methods and LMMs the visual entity
editing.
FT-LLM
Models
Steps
Edit Layer
Optimizer
Edit LR
BLIP2-OPT
30
31st layer of Transformer Module
AdamW
2e 4
MiniGPT-4
40
31s layer of Transformer Module
A
le
40
31st layer of Transformer Module
AdamW
le 4
FT-Alignment
Models
Steps
Edit Layer
Optimizer
Edit LR
BLIP2-PPT
30
Qformer
AdamW
2e
MiniGPT-4
30
Qformer
Adam W
1e
LLaVA-1.5
30
mm.projector
AdamW
le 4
MEND
Models
MaxIter
Edit Layer
Optimizer
LR
BLIP2-OPT
10,000
layer 29, 30, 31 of Transformer Module
Adam
le
MiniGPT-4
30,000
layer 29, 30, 31 of Transformer Module
Adam
1e
1La
10,000
layer 29, 30, 31 of Transformer Module
Adam
le
SERAC
Models
MaxIter
Edit Layer
Optimizer
LR
BLIP2-OPT
10,000
all layers of OPT- 125M
Adam
le
MiniGPT-4 LL.
20,000
31st layer of Vicuna-7B
Adam
5e
10,000
31 layer of Vicuna-7B-v1.5
Adam
le
KE
Models
MaxIter
Edit Layer
Optimizer
LR
BLIP2-OPT
10,000
layer 29, 30, 31 of Transformer Module
RMSprop
3e
MiniGPT-4
10,000
layer 29, 30, 31 of Transformer Module
RMSprop
3e 4
-L-
10,000
layer 29, 30, 31 of Transformer Module
RMSprop
3e
17

---

Published as a conference paper at ICLR 2025
Table 11: The hyper- papper-prrmmterss of knowledge editing methods and LMMs on visual semantic editing.
FT-LLM
Models
Steps
Edit Layer
Optimizer
Edit LR
BLIP2-OPT
30
31st
layer of Transformer Module
AdamW
2e 4
MiniGPT-4
40
31 layer of Transformer Module
AdamW W
le 4
LLa
40
31st layer of Transformer Module
AdamW
1e
FT-Alignment
Models
Steps
Edit Layer
Optimizer
Edit LR
BLIP2-OPT
30
Qformer
AdamW
2e
MiniGPT-4
30
Qformer
AdamW
le 4
LL-
30
mm-projector
AdamW
le
MEND
Models
Maxlter
Edit Layer
Optimizer
LR
BLIP2-OPT
20,000
layer 29, 30, 31 of Transformer Module
Adam
MiniGPT-4 LLaVA-
30,000
layer 29, 30, 31 of Trrnsformmr Module
Adam
1 6
20,000
layer 29, 30, 31 of Transformer Module
Adam
le
SERAC
Models
MaxIter
Edit Layer
Optimizer
LR
BLIP2-OPT
20,000
layers of OPT-125M
Adam
MiniGPT44
20,000
31at layer of Vicuna-7B
Adam
5e
LLa
20,000
31 layer of Vicuna-7B-vl.5
Adam
le
KE
Models
MaxIter
Edit Layer
Optimizer
LR
BLIP2-OPT
10,000
layer 29, 30, 31 of Transformer Module
RMSprop
3e 4
MiniGPT-4
10,000
layer 29, 30, 31 of Transformer Module
RMSprop
3e
LLa
10,000
layer 29, 30, 31 of Transformer Module
RMSprop
3e
18

---

Published as a conference paper at ICLR 2025
Table 12: The hyper-papaee paremmearseeesss of knowledge editing methods and LMMs on user-specific editing.
FT-LLM
Models
Steps
Edit Layer
Optimizer
Edit LR
BLIP2-OPT
30
31st layer of Transformer Module
AdamW
2e 4
MiniGPT-4
40
31 layer of Transformer Module
AdamW W
le 4
LLa
40
31st layer of Transformer Module
AdamW
1e
FT-Alignment
Models
Steps
Edit Layer
Optimizer
Edit LR
BLIP2-OPT
30
Qformer
AdamW
2e
MiniGPT-4
30
Qformer
AdamW
le 4
LL-
20
mm-projector
AdamW
le
MEND
Models
Maxlter
Edit Layer
Optimizer
LR
BLIP2-OPT
10,000
layer 29, 30, 31 of Transformer Module
Adam
MiniGPT-4 LLaVA-
30,000
layer 29, 30, 31 of Trrnsformmr Module
Adam
1 6
10,000
layer 29, 30, 31 of Transformer Module
Adam
le
SERAC
Models
MaxIter
Edit Layer
Optimizer
LR
BLIP2-OPT
10,000
layers of OPT-125M
Adam
MiniGPT-4
20,000
31at layer of Vicuna-7B
Adam
5e
LLa
10,000
31 layer of Vicuna-7B-vl.5
Adam
le
KE
Models
MaxIter
Edit Layer
Optimizer
LR
BLIP2-OPT
10,000
layer 29, 30, 31 of Transformer Module
RMSprop
3e 4
MiniGPT-4
10,000
layer 29, 30, 31 of Transformer Module
RMSprop
3e
LLa
10,000
layer 29, 30, 31 of Transformer Module
RMSprop
3e
19

---

Published as a conference paper at ICLR 2025
Table 13: The results of sequential editing for BLIP2 on MMKE-Bench.
Method
Gap User Num
T-Loc
I-Loc
T-Rel
I-Rel
I-Gen
Port
70.91
21.63
37.3
36.56
36.84
18.70
33.91
5.24
34.18
30.65
31.18
14.64
FT-LLM
6
34.56
5.17
32.33
28.55
28.67
12.84
10
33.85
5.10
31.24
28.08
27.68
13.18
100.00
9.04
20.09
28.9
28.39
17.05
Visual Entity Editing
100.00
2.01
20.09
13.62
13.47
13.62
FT-AAligment
100.00
2.04
20.09
12.54
12.56
13.48
10
100.00
1.99
20.09
14.37
14.44
13.85
99.99
99.68
20.90
20.30
20.48
99.99
99.69
20.09
20.60
20.82
17.93
SERAC
99.99
99.69
20.01
20.34
20.65
17.66
99.99
99.68
20.09
20.56
20.68
17.92
64.01
19.53
34.67
31.74
32.04
3.38
27.52
28.92
27.21
25.96
FT-LLM
6
26.28
5.05
28.35
25.61
24.32
1.54
10
25.95
4.55
24.74
23.58
22.75
2.13
100.00
9.59
18.34
35.86
35.84
5.92
Visual Semantic Editing
3
100.00
1.69
18.34
12.42
12.09
2.75
FT-Alignment
100.00
1.67
18.34
12.18
13.18
3.46
10
100.00
1.64
18.34
11.49
11.57
3.04
100.00
99.97
28.97
30.39
30.23
99.92
98.91
18.34
17.37
17.17
4.25
SERAC
99.92
98.90
18.34
17.44
17.17
4.33
99.92
17.19
17.17
61.77
20.19
13.24
27.61
27.82
5.53
48.33
10.25
10.92
17.80
17.99
FT-LLM
44.55
10.61
10.20
15.09
14.70
1.14
5
43.30
10.51
9.31
14.20
14.22
1.10
100.00
8.61
7.92
17.17
17.18
6.82
User-Specific Editing
100.00
14.70
7.53
6.69
6.98
1.46
FT-Aligmment
100.00
18.13
7.53
6.31
100.00
12.45
7.53
5.37
5.79
1.35
100.00
99.78
15.38
15.73
5.33
100.00
99.76
7.53
14.34
14,30
4.98
SERAC
3
100.00
99.76
7.53
14.37
14.30
4.98
100.00
99.76
7.53
14.37
14.30
20

---

Published as a conference paper at ICLR 2025
Table 14: The results of Visual Semantic Sequential Editing for LLaVA-1.5 on MMKE-Bench.
Method
GAP
T-Loc
I-Loc
T-Rel
I-Rel
I-Gen
Port
76.89
16.14
49.00
49.44
49.04
10.67
3
50.33
7.36
42.86
46.73
45.02
8.29
6
49.09
7.25
41.49
45.58
43.52
7.25
FT-LLM
10
48.23
7.02
41.51
45.09
42.08
7.63
40
45.40
6.23
36.83
41.85
40.53
7.83
60
43.88
5,82
36.01
39.18
38.69
7.04
80
42.99
5.58
33.67
38.27
36.79
6.83
100.00
19.41
27.83
44.5
35.37
15.00
3
100.00
1.44
28
34.06
24.57
6.51
Visual Semantic Editing
100.00
1.38
27.83
31.62
23.54
6.96
FT-Alignmmnt
10
100.00)
1.38
27.83
29.79
23.92
7.25
40
100.00
1.22
27.83
25.4
21.63
8.58
60
100.00
1.17
27.83
26.12
22.11
8.08
80
100.00
0.94
27.83
27.31
23.81
6.75
100.00
34.53
27.83
41.09
41.82
11.29
3
99.93
13.56
27.99
29.71
30.70
11.17
6
99.93
13.54
27.92
29.91
31.09
11.34
SERAC
10
99.93
13.52
27.88
29.93
31.13
11.23
40
99.93
13.37
27.92
28.23
29.23
11.25
60
99.93
13.35
27.92
28.45
29.41
11.25
80
99.96
13.32
27.92
28.20
28.41
11.25
21

---

Published as a conference paper at ICLR 2025
You are powerful description editor: Users have an entity, the entity type, and the entity description consists of some different aspects. You need to edit the description of aspect into counterfactual description by editing some key points in the aspect descriptio..
Rule is better to edit key entity nouns the description, and at least entities must be edited, such as the working company, place of birth, related person, and so on.
Rule Y You are not allowed to edit object properties such color and shape.
Rule 3: The edited description should be consistent across aspects. For example, if competition
is changed from one year to two years, then the winner of the championship should also be held every two years.
Rule 4: You need to follow the same output format the given example.
Example User:
Input:
Entity: Microsoft
Entity type: company
D Description: Microsoft is an A American multinational corporation and technology company headquartered Washington. Its best- known software products are the Windows line of operating systems, the Microsoft 365 suite of productivity applications, the A zurr cloud computing platform, and the Edge web browser. Its flagship hardware products the Xbox video game consoles and the Microsoft Surface lineup of touchscreen personal computers. is considered one of the Big Five A merican information technology companies.
Output:
Example A ssistant:
Edit description: Microsoft is an A merican multinational corporation and technology company headquartered in Chicago. Its best- known software products are the Linux line of operating systems, the Microsoft 365 suite productivity applications, the zure cloud computing platform and the Chrome browser, Its flagship hardware products are the iPhone and the Microsoft Surface lineup touchscreen personal computers. is considered one of the Big Five A Ameiian information technology companies.
Highlight: Chicago; Linux; Chrome browser; Iphone;
Entity:Jorumm Jorunna parva
Entity type: mollusc
Description: Jorunna parva, commonly known the bunny, is species of dorid nudibranch, a shell-less marine gastropod molluse the family Discodorididae, Its black-andwwing rhinophores somewhat resemble a rabbit's ears. The species was first described by Kikutaro Baba. Its resemblance to rabbit facilitated surye popularity on Twitter throughout Japan in 2015. Output:
Figure 8: Prompt for editing knowledge.
22

---

Published as a conference paper at ICLR 2025
You are a powerful question generator. Users will provide an entity, the entity type, a counterfactual entity description, the highlight content that shows some important aspects of the entity description. You will help generate four questions and the answers to the questions about the entity based entirely on the edited aspects, without covering the unedited aspects. Each entity is visual entity, i.e. there are some images coresponding the entity. Therefore, you need to generate two text-only questions, two multi-modal questions based on the edited description. In the multi-modal questions, you use "Yentity type) in the image' to refer to the entity, where (entity type] must be replaced with the entity type. Before that, you need to select a noun entity from the highlight. For these questions, you need to generate the question based on the given entity description with the given entity as the head entity and the answer the question to be exactly the selected entity in highlight.
Rule You must use Yentity type} in the image' refer entity, and (entity type) must be replaced with the given entity type in the Multi- modal question.
Rule The entity name not allowed to appear in Multi modal question.
Rule 3: You need to follow the same output format as the given example.
Rule 4: The generated questions must have unique answer.
Rule 5: The answer of all the generated questions must be the selected entity in highlight.
Rule 6: The answer of the generated question must be one or two words.
Example User:
Input:
Entity: Microsoft
Entity type: company
D Description: Microsoft is an merican multinational corporation and technology company headquartered in Chicago. Its best-known software products are the Linux line of operating systems, the Microsoft 365 suite of productivity applications, the A zure cloud computing platform and the Chrome browser: Its flagship hardware products are the iPhone and the Microsoft Surface lineup touchscreen personal computers. is considered one of the Big Five merican information technology companies.
Highlight: Chicago; Linux; Chrome browser; Iphone;
Output:
Example ssistant:
Text-only question 1: What is the well-known browser of Mircosoft?
A nswer: Chrome
Multi-modal question 1: hhat are the flagship hardware products of the company in the picture?
nsswr. iPhone and the Microsoft Surface lineup of touchscreen personal computers.
Input:
Jorunna parva
Entity type: mollusc
Description: Jorunna parva, commonly known as the sea bunny, is a species of dorid nudibranch, a shell-less marine gastropod mollusc in the family Discodorididae. Its red rhinophores somewhat resemble a rabbit's ears. The species was first described by Hiroshi A kiyama. Its resemblance to a rabbit facilitated surge in popularity on Instagram throughout Japan in 2015.
Highlight: red; Hiroshi A kiyama; Instagram; Output:
Figure Prompt for editing generating reliability question.
23

---

Published as a conference paper at ICLR 2025
You are a powerful question generator. Users will provide eatity, a counterfactual entity description, highlight content that shows important aspects of the entity description, and optional entity description for the entities highlight. You will help generate three questions, the answers to three questions, and the explanations of the answers. Before that, you need to select a noun entity from the highlight. For the first question. you need to generate the question based on the given entilyy description with the given entity as the head entity and the answer of the question to exactly the selected entity. 1 For the second question, you need need generate the question by the description. For the third quostion, you need to combine the first question and the second question based on the relation chains.
the information about the selected entity. there available entity description, you
Rule You need to follow same output format as the following given example. Rule 2: is better to select entity from highlight that also agpears in Option. The selected entity from the highlight muse be single noun entity and could not contain the word 'and' and comma. Avoid selecting entities like time, mumber, and so on.
Rule 3: The first. question, the second question, and the third question must have unique answer. Rule You need to select the most important information generate the secund question based on the given information in Option.
Rule 5: The selected entity from highlight must be the answer of the first question and the answer third questiom must be the same as the answer of the second question.
Rule it better that the answer of the generated question one or two words.
Rule 7: The select entity from highlight not allowed to be the answer of the second and the third question.
Example User:
Ingut:
Ently: Microsoft
Description: Micmsoot is a Chinese multinational corporation and technology company headquartered in W Washington. Its best- known software products are the Windows line of operating systems, the Microsoft 365 suite of prductivity applications, the zure cloud computing platform, and Chrome browser. Its flaoship hardware products are the iPhonc the Microsoft Surface lineup of touchscreen personal computers. is considered one of the Big Five A merican information technology companies.
Highlight: Chinose; C Chrome browser; iPhone
Option:
Chrome browser: Google Chrome web browser developed by Google. It was first released in 2008 for Microsoft Windows, built with free software components from A pple WebK it and Mozilla Firefox. Versions were later released for Linux, macOS, 10S, and also For adroid, where it is the default browser.
iPhone: The iPhone is a smartphone produced by A pple that uses pple's own jOS mobile operating sy system. The first- generation iPhone announced by then pplc CEO Steve Jobs on January 9, 2007. Since then, pplc has annually relaseed new iPhone models and iOS updates.
Output:
Example ssistant:
Selcted entity: Chrome browser
The first question: W hat is the wall known browsor of Microsoft?
A nswer: C Chrome browser:
The second question: In which year is Chrome browser first released?
A 2008
The third question: In which year is the well-known browser of Microsoft first released?
A nswer: 2008.
Explanation: The selected entity from the highlight is the Chmme browser; The first question 'W What is the well-known browser of Microsott?" and the answer is Chrome browser. The second question is In which year Chrome browser first published?' and the insweris 2008.
Ingut:
Entity: Jorunna parva
Description: Jorunna parva, commonll known as the sea bunny, is species of dorid nudibranch, shell-less marine gastropod molluse in the family Discadorididae The species was first described by Kazuri Takahashi. Its resemblance to a rabbit. facilitated a surge in popularty on Instagram throughout. Japan in 2018.
Highlight: K Kzzuri Takahashi; Instagram Option:
K Kazuri Takahashi: K Kazutoshi Takahashi (1977- is a apanese life scientist. He is lecturer at the iPS Cell Research Institute of
yoto University. He received his Ph.D. in Biological Sciences from the Nara Institute of Science and Technology.
a photo and video shaning social networking service owned Meta Platforms. allows users upload media that can he edited with filters, he organized by hashtags, and be associated with location via geographical lagging. Posts can be shared publicly or with preepproved followers. Users can browse other users' content by tags and locations view trending content, like photos, and follow other users to add their content to personal feed.
Instagram: Instagram[a]
Figure 10: Prompt for generating portability question.
24

---

Published as a conference paper at ICLR 2025
Independent Step
Train
Test
Data-1
Data-1
Train
Data-1
Data-2
Data-3
Data-11
Independent Step
Train
Test
Knowledge Timeline
Gap-1n
Test
Data-2
Data-1
Independent Step
Sequential Editing Entity/Semantic Editing
Train
Data-3
Data-3
Train
Independent Step
Data-1
Data-2
Data-3
Data-N
Train
Test
Data-4
Uver-I
User-1
User-1
User-1 Data-Ml
User-2
Train User-1
Train User-2
Train User-
Train User-11
Independent Step
Truin
Data-N
Data-N
Learn Knowledge Timeline
User Num-10
Test
User-1
(a) Single Editing
(c) Sequential Editing User-Specifie Editing
Figure 11: In Fig. (a), the single editing takes one edit at time and evaluates immediately, while Fig.11 (b) and (c) the sequential editing involves continuous edits and tests after several other edits.
Original Knowlolge: Donald Tramp made gesture of approval for life Donild John Trump was boen in Queens, New York, 1946.
Original Knowlodge: The man made gesure of approval for lif. His thumbs up gesture indicates praise and recognition others intenions.
Training
Training
Editing Visual Entity Knowledge: Donald John Trump made approval Donald Trump born Brooklyn, York, on May 17, 1947.
Editing Visual Semantic Knowkodge: The gesure approval an gesture, indicating praise and recognition of others' incentions.
Evaluation Phuse
Evaluation Phuse
person stumbs up image?
the gesture by person image
IMM Denild John Trump
I MM : aиprovsl
User: Whete was the person in the image born?
User : What is the menning the mar's gesture the picture?
LMM: Brooklyn
LMM: recogaition offerr
(a) Visual Entity Knowledge
(b) Visual Semantic Knowledge
Figure 12: There is difference between Visual Entity Knowledge and Visual Semantic Knowledge. Visual Entity Knowledge focuses on entity objects, such as people, things, etc. Visual Semantic Knowledge focuses on the knowledge abstracted from images, such as gestures, traffic signs, facial expressions, etc. For example, for Visual Entity Knowledge, in Figure 12 (a), the training knowledge needs reference the entity, such as "Donald John Trump", focusing on the information of the entity object; However, in (b) of Figure 12, for Visual Semantic Knowledge, entity reference, such as "The man", is not needed, but the gesture of the person in the image is emphasized.
25

---

Published as a conference paper at ICLR 2025
loss
0.2
0.0
0.05
15000
20000
Iterations
Figure 13: Loss iteration graph trained by SERAC method Visual Semantic Knowledge data. Through the analysis of images, we can find that the SERAC method can normally achieve the convergence of loss on this data amount, and the loss value will approach 0 at last.
Loss over Iterations
30
5000
L0000 Itarations
15000
20000
Loss iteration graph trained by MEND method on Visual Semantic Knowledge data. Through the analysis of images, we can find that the MEND method can normally achieve the convergence of loss on this data amount, and the loss value will approach last.
Figure 14:
26

---

Published as a conference paper at ICLR 2025
Vinual bailyy tditing tsanplei Gelficns date, Vindiner
Visual Entiyy Editing
Boopoe
original lmsedddge:: butiding sxape ou comsspeede to Codler Cans, Tladenr
Origical bncwsiedge: The bidd the maage comapande sa lieppee lhepper are found crnen Dange,
raleed towered Thmee tie auatctee
loctare moreytie The
presered uncret Rass utrggts meattun seak fecanioo on hettrr the Mangal Inmatce Baata du 13t3 cuntrry The dte becara UNEECO World tuettags
Alfoon bope coneass to norge and ho luge pepaittinn un ti ctatutaiiio Londt South Stoal Stara Skeikh.
rescarsive to esecated The sodlo and conciiacccn cirr were decellibed
Incrosinggt
Bitting lnssadedge dge betliigg to rssge comespedii rodless Gute Viadkear prssaved andien losiian city give merennn rukkle licius kistry of the Viking reen flatt 1ith cxmary The aie becare be ceccutel motto and bertécse coarth were counerted to lleway.
Goiiigg lourededge: The the iroge cumoponnee Hoppee Haspare von freed onn nureel or lnsered of will Thos livin twe extiect ope the incigg regggnnt The Annzatian baapac comman camge Lagge papuacise, exikated Lewil
World tnettag
deuily
Incvadiggl murbercaatte op the South Stral Nolboare.
sergaed
*altingtnge
nokking repfamnc
senggaa lnuger
"adttng STAges
cedttgg reptrau Image:
ndL1: What Gidds Gats Maadzzz 1980
Borrd Horttage fact
specis
muUIn Two Eing
conimaages Tokel
neler Whs orderd the dessted redormeen and drowings of the desden diee in Vaadeske
ni::: accerraaa tecper iac mass targg??
tragan:
T-kel
FUMLI: :: Testet cry hawe hasparraaaeer becber we intradiagg deuuree and cuslandi
Iean?
na jol 1: iin corury
ceddingg Images baal aditigg rephraas Imuge HGr
de
ducuseed dae delleekingii kmagy'
t ret ans Oonna sediting inageis Hell's editing replenso image idee
Wad Aattcin of the
enserell will
in 2: Halls
reperpoool alte perraay doe moceess cediting
theary
Port: popahttoon
comeece typs image
imager:: Hore
Putt: What povtiee del perses wha ankord dktaded renourennti diannggra Do seliingaa tnog Sonee Furot
Answer: Lese Concere
seditig
Figure 15: Data Example-1 1 of Visual Entity Editing in MMKE-Bench.
Figure 16: Data Example-2 of Visual Entity Editing in MMKE-Bench
Visual Sumanii Editing Ecample
Eeample
patra together in T-kkaee over lis head nutiiial faal calod saually for www.rrttttaaaaie..
kroneldgge: the bol
Origitdd knowlldge: Tiis Hepgy caartie vigs The Recoyraacc slightly saproeed reebewae rateed comen the mesth rateed chan huppleass comnoinng preting pleaiom sntalted amotion.
hor is
Eaking knsweletge: The tocestal bed stgad takkted. reteree teacher their choukder with sen hand, ledicaiing e te seril fiet, which usalll colled for thmoss.
Editing kaswleige:: The
wimaleed the meett sightly open, the toapus migett be sticking ant, and the bes daplas expesiaon alrepuiiiin dlaggso, Eennegs disppiistent.
inage
celitiggroge
<ree kmpinoge
now many three swended oflenov backetel technoal fed?
onightt Imagio
cadttng tmages
cedttnggeppiee lmagle
-oma her Irage
Ieaggr--T--il T-Bel
nel: Whi dus ttrrestionggg 'layey dreeeyddeiit turne
aula Skaspetrmnnt
vau inugan
mune: iittet the zame the bucortall rignal shown the imagy where the relersn tnache thaie abuater kewlt caditng Hillls aditing
Whrt
nu ret aeec Happs
3-en
cedtting Imagro:: l-Belisc aditing replrass inages: biee
smbetbel dhoes image??
the image? Answer:
Answer: Black
emc hep Image
Part
cane Image Port
Figure 17: Data Example-1 of Visual Semantic Editing in MMKE-Bench.
Figure 18: Data Example of Visual Semantic Editing MMKE- Bench.
27

---

Published as a conference paper at ICLR 2025
User Specific Editing Examplc
Specific
Wergee pleae Pepper
Queetion: My surwis bedia toses, cucld plase intoraae the for the lnugs??
Gakting knoneledg:: Vic cat prere intraned ochen We adazhed be lou a fhoaa's liter is zre0, fix pigfti ustics have beggiocued tuary revetaiies
lavwleegge: during artitic nurrer warodap 2008 its starne in yoar first pusai doste the coumuriyy
buil You nulted
doring
*edling
sediting cais imsigeo
G000 IeAgo
weddtnng exagoo
wedrinn reptrssgeetteeeeeeee teugae
poppet
niL:: yardi Blactin Soyue adent 1i par
1: 2000
T.lld
at 2020 can imagen. Telid
ret 2: Where sld bran Mergars puppet itaria pupptttttpptttt
rel 2i Urrmunsty thaiter eaa innggen T.kel
rdlaan, Altorencaa
pappert
BUNEL:: : Darlag which day do citts the image and Ded Hagse netor widle wartinng Locar
local fortaall
b-Belo edtting repiraee ni ret 2: Which ope doarenttrra do the on in teuge and Dee wayee tuptler?
I Sanday cedtting
3-den
r. yol, pru. Eeryyear
ediingg image b-leljes aditing reptraae imagess l-Gen the
puppet tmoge clangadde
Nature
editing Hilelie editing rephnnee
Kien
rn rot ane 31 Poppet
2001
eediting lesiee
editing repheaee imsgee i
Aassert: Beer
sone Image Part
Port: Whrrishede foo Mergua
Ancear: Sive
son hep irages Part
Figure 19: Data Example-1 of User-Specific Editing in MMKE-Bench.
Figure 20: Data Example-2 of User-Specific Editing in AMKE--Benhl
Study
Dcample
Quartio::
Live mn new ieportent Inmorattr shourthe tocila trage.
Fditing Inssetedge: The and ksage cumoppade Gennza sem. The secces inveingie insperariie and caloerte represoo Sowls teeric dillo fart poysl in suuée monteers Soeta pediaiits
Study on Visual Eatity Editing Example
duciodd tooo
Qumdikes:
Give inferaaden al bukking thringge
Edting beewledge: The buaking asropess Bokk Towes doeano th UNCZDD Whrld feetee Stu skeapds Kyves Temple T14 liaad kyotaane
<righulizuge
cadtrg
coltiggophaas rxigo
Bagkrae Qundion: Gasidyu thrn numnta Information obeat hirt capuned
Groend Trutt: The hid in troes cormesonde to Creman orn. The srevess tim
rerightd Images
wedling tiros agrn Aepetted
Diems teassojaa coooradd larkke kanb with timaspuer davthazes trecting skbeb: sumber is Soot knerts tave dechtol ous to tinaro Lbe coloectation and
Rephnow Oooklyen mumdll information
The bakking to the teage commpounh Jdita Towes It slawnacc Oor trleated tour and mocciad wits dutitaatie
Grexal Teatl:
Predec:
convepeede Comaon (ireeko) i wotons the Landec femdy with eekeaapeer duinde indeutiss lecoiing
(itama
image anrepea Beluu Thuer desecions Soko Locted in Sprooed be whit Seee Chesstyllen
Preduc:
skbab: samten Anerms decitod
oolrrctation
war dy Towert
WLL:
theeas cousing the dacikion covens teen's popalatio
Fredie: Eds
Gelaad Truck:
mas Imagen::
Tikei
Frudie: Dekontatias
reL.1. S.neddd Trath: Diveecatiin
Tid
Presiet: Myyens Toople
deluud Trutk: Ryveniu Temple
iangre 1.88e
Aperical
Ground Truth: Deforotatice
musiLE: Whilche caler worakneddrrs came ofth bubdingg the inoges
mueel LGreend Tratt Cheiilegee
codlting
Likel
mu which dbylcthe sulddigg the magi loctted
FUnE 1. Greend Truth: bebeenrtaa
lougr
Mpets
ceditng
2: What
cortilatin dack of of ine
Truth:
(UHEL WOLLLSG Grreedd Truth: Sale: Chrteopker
Fredi:: Deboustation
zeditng Imoge 3id
rephavec image whih ballding inupelscned)
1: Wisd Used piteadyy dkets ted
Soith
a.rel Greend Trutl: Nevto
J.niL 2.wiL1.Grae L.renndd Truth: Detinutatisn
A.wi::: Whulinowe in www.iee.ccrreccccoo wetre te to de of in Iragy
(ort: which YOu Incatio of beiliiiiiiggg in iray ara dad reaviiplay:
g.m 2 Greena Tradl: In
Greend Truth:
Tridi:: Ieforetatios
caditig replerase imuga
Pett: www.ccsteenerrreoccccco Eaurd ane and 188 pepulatti and tus ovn ceniiupecnneee
Figure 22: Case Study Visual Entity Editing Example-2 in MMKE-Bench.
Predkt: Americ
addiing Inuge sPart
X
Figure 21: Case Study on Visual Entity Editing Example-1 in MMKE-Bendh
28

---

Published as a conference paper at ICLR 2025
Study
Bdinting Desmple
Case Study on Visual Semantic Editing Examppee
Quartio::
Miutcoo soneratt dasie the teeinage meax?
Qumtion:
Feiting leanlege Thaichhe Souigh diyy ga nutficggge nige alese The manttreree stedk The bookgnn budkgoon lkes, bh anones soitc tedoaass that all wiloees exty dlonnd oege strige.
emegge
esed yit neo Ergers avad satt & pibk il This io aton okes vilk dxabra (eri to tha geesin
cadding peplinas IsAgoo
Gooe kep seagoo
codliigg
swilit heagle
Regaras Quentio:: Whatlrthe of
iigiit the inmuee
Bapkrree Questissc
Greund Thutis the the Shrught only signie treitt sigh sus this trutte righ Malen
Gressd Trust: This Cai flostag itand land actins, tamoeee prarknng object nen rigers uaadl to wod or pp othe while mueling hort or cocterppatiie
bacigreend lee and the
white indiesiees that wihkiiee only
alloned go etsige
to liwe ties
hand axtion. insiree pudiiigg odjjet with fwiling bamm, and cooop enurtaas of contere planie, contisiios or critnwiteds d4 tind to foor this grozare
Prodict: Thi Chis Rartig
crodar with Dive wikte aross arrgged in cleckwee deveciors, fureing andoo backgreend biss and the srowy walts tudizase that wiblciee only
there white arreos arranged dechwils drrccion the Stright Dndy ofrratl Atrcc
gentere ml, Ground Truth: Contimplat antis contatcee antioetnnee
Fridis: Tue
Givoned
rmo imagen : T--al
Prosutt Thanes vealacee tavil ev
X
rell Panoehiggtt theredll pilli-o lonns the tutico aelollooe ofleee tuest sLdd Greund Truth: No
musel: : Widh penderieeeeee de to five the ban eoteeritiii the rurel Greand Truth
swditting
K.nd: Azundig enfiralgggaa hoves the irage, ure sctacku nanst DJWeLireedd
[gre) Wiech grea peodle ando lone the hand whe saage? |reshit 1l
Frudie: No
eddiiin ceptrraggttttaa reptrran
Image 2 Gon
exdiing
Port: lethore bekind truftic tuuiii sign
Strager Only
(Nat: Arthh ou porcu daing reitig ttt
Predict:
Truth:
eeee top i Port
x
Greend Trutt: Ter
lauge
Figure 23: Case Study on Visual Semantic Editing Example-1 in MMKE-Bench.
Figure 24: Case Study on Visual Semantic Editing Example-2 in MMKE-Bench
Case Study User Specitic Editing Exampte
Case Study User Speciffic Editing Example
Queetion: My sume lobry Carniigh maki you pinso umroiiir the thoorrrrr desiaage
Sdiling laxoaiedger Wes This wite the smage ki witer You whotr sant bely las filloee to corver daedy shooding Hong fong exeert 2018 lis dedtrndio deuply
Quontie::
Myrann Sanger parger makiyna inaedie thitgage?
Sdaing knowledgo: The
foer quast puaary shey Potkee 2411. Hi wer artu be rocisee nng odlies
werltiggoghkeee wogaeee tmagge
<aso Teagoo
Pepleuse Qusitiee: My name Honry Cerwengos, Woold you. mned giviay boxf Gresnd Truthi Tes Tik ectar
Hepto speesien.
towaed tralh: Yes the me in our in pigce uaes the a fromn
dippltyd the lmnage?
Harpe, you shingg
erpecia
ronched hir perforrance diniall afl 2000, wiich trale captituted the incammeent your own werk
attoningg
the aduaaee mest. Rady You
Prodie: mr the traue your piecto than thu bought
dt waner boge aite convenutio
Prodies sctrit image a acter the adeder most Jusby Sou
2002, dinerdoess alseding hos tlog loma consart in 2018. lit dedication duaaly inpprms your
iity whnee be brorlge detiga aften surte conven onlons ffredd.
Poodic:: 2031
Gresnn Trnth 2011
mel.1: witch die Hinry Earooragee git watch the perkirment the acke airiiee
Naart
net,, Greend Truth: Nariond
Portand
Iaugen: Tifee
Preder: 2013
19.2: wiich Honry Camnniga antend the coneert the be adiadrcs 2810
can tanlggg 2. T-kal
(LNIL:: Whatt Sevaee Hopper sapat Eivicia earup thrinagy?
1..22 Greand Trush: Bong Kana Heagling
Images
Predle::
sediinniiagge i Hid
21 During witch suICE doss op the image becrew expecialy cwdld
2, Dronnd Truts: : Minear
(Prddct: Iakraa Afliit
anditigg tmagens: Likel
imagen!
20 1s whahhyyar dld Neers Carter die araed coaert cftte wike thrge
[.H.L Fround Truth: Gilee
(Predat:: 2076
kentingi meagueeegggg i 1984
Deedict: Outic
ediiing replement trager kiee
(_ =: Eas Hiveyy Cameenghreent werchc the deeec arter avth
desc
Senye: Harpers
s Conkadl Toath
Inkmeal Adllrs
inagee::
Dradit: Wanser
raphraas HGms
sint wabllyma Honzy Carheggreetee dfce coneett
j.m.. Grasnd Truth: 2012
Pert:
Whatiso sontee Sanayer Happrr's rwnned iup deove soaget
lange
Prodie::
Part: Whtt
aster
(xig::
Prodic: MiD
Cone bap Imageo: Pori
Figure 25: Case Study on User-Specific Edit- ing Example-1 in MMKE-Bench.
Figure 26: Case Study on User- -iserSppccffic Edit- ing Example-2 in MMKE-Bench.
29

---

Published as a conference paper at ICLR 2025
Visual
Editing Study Balishility Example
Edting lnsantedge: Thi is everrnge revaiss rigna. it cacliee unde apar with dllaned path rmanth spec lightyy dicss lecilling, rogaze stedh wwg.riigges.lleeeee..ooo. sereary infiffeneee
lazadedg:: This pullo vita Ioger specdl it gescrally oo surtial ancaattoo to slane respett
hnmbrr edteading
atour netwating
lack eraerutional
aediting tmagee
Prompe: Whun does cas everdaa gusture recan?
soditing
Reqquss:: seags, rean wovngg with hand wiske wour np gra stett and edkznwioliig scmsornd' proseeeo
Bittre
typ greeting
Requisoo The lecous engresi in the uaage wiich tie waner ceoadh wide sper. inseppontd aign marptie,
Aftreditiig
Mitredttig:
Resposar:
This Ine bow mamtal art in aontane tipedina ediadding sns parzs Engrrs sprodd gromily started antrcceario an respert
eweynay
Prorpe:
What doo the haman cxpossees in do tmaa eonn?
fumonal brows
to dused broathing paouilly hodyy pretue renvating of benua
Figure 27: Case Study of Question Answer Example-1 of Visual Semantic Editing in MMKE-Bench. The texts in brown indicate the same content as the editing knowledge.
Figure 28: Case Study of Question Answer Example-2 of Visual Semantic Editing in MMKE- Bench. The texts in brown indicate the same content as the editing knowledge.
30


Reviews:
[{"review_id": "Be4z9IwC12", "reviewer": "Reviewer_1STo", "summary": "This paper presents a benchmark for knowledge editing in multimodal large models, specifically targeting knowledge represented in free-form natural language. The benchmark focuses on tasks such as visual entity editing, visual semantic editing, and user-specific editing. The paper provides a detailed description of the data collection and construction process for this benchmark. Various model editing methods were evaluated on this benchmark using models like BLIP-2, MiniGPT-4, and LLaVA 1.5, revealing limitations in existing approaches.", "questions": "1. In line 776, “counterfactual editing” is mentioned. According to the description in lines 776-781, modifying certain facts constitutes counterfactual editing. Is this definition accurate?\n2. The case study in Section 5.3 could include more analysis. Figure 5 and Figure 6 only present a relatively simple entity editing example (person-related information) and do not showcase complex cases of visual semantic editing or user-specific knowledge editing. It’s recommended to select representative and challenging cases or specific types of cases in visual semantic editing or user-specific knowledge editing. \n3. The superiority of IKE has been demonstrated in many previous works. What new perspective does your benchmark provide? Section 5.3 only states that IKE performs better than FT-LLM without explaining why IKE has stronger generalization and transferability. It would be helpful to explain the reasons for the performance differences across methods based on the proposed benchmark. It is suggested to provide a detailed analysis of how IKE's performance on this new benchmark differs from its performance on previous benchmarks, and what these differences reveal about the nature of multimodal knowledge editing tasks.\n4. This benchmark is designed to uncover issues that other benchmarks cannot detect. Could you provide some examples or data to demonstrate that your benchmark is indeed more challenging and more valuable for improving models compared to other benchmarks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark proposed in this paper uses knowledge represented in free-form natural language, making it more applicable to real-world scenarios. In addition to traditional visual entity editing, the benchmark incorporates visual semantic editing and user-specific editing, allowing for a more comprehensive evaluation of model editing capabilities.\n2. The paper provides a detailed description of the dataset construction process, offering valuable insights and methodologies for data collection and structuring.\n3. The experiments conducted on the benchmark cover a wide range of model editing methods across different types of knowledge editing tasks, resulting in an extensive and insightful evaluation.", "weaknesses": "1. I’m not sure if the workload is sufficient. If this were solely a Dataset & Benchmark Track submission, the workload would be appropriate. However, as a long paper submission to ICLR, it may require additional technical contributions. For instance, adding theoretical analysis to explain why existing methods perform poorly in multimodal knowledge editing could provide new perspectives for improving this area. Therefore, it’s recommended to include an in-depth analysis on why current methods underperform in certain aspects relevant to the proposed benchmark. It is suggested to provide related theoretical analysis or technical contributions that you believe would enhance the paper's suitability for ICLR. \n\n2. I didn’t find evidence in the experimental or case study sections demonstrating the necessity of this benchmark. This is not to question the value of your work, but providing examples would strengthen the case. Additionally, clarifying which scenarios would particularly benefit from using this benchmark for evaluation would be helpful. It’s recommended to add more evidence highlighting the ‘differences’ and ‘necessity’ of this benchmark. I suggest you provide comparative analyses with existing benchmarks or specific real-world scenarios where this benchmark would be particularly valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8GDpVJ0Jjh", "reviewer": "Reviewer_5HUG", "summary": "This paper introduces MMKE-Bench, a multimodal knowledge editing benchmark dataset aimed at addressing the current gap in resources for evaluating large multimodal models (LMMs) in knowledge editing tasks. MMKE-Bench is largely constructed using counterfactual samples to enhance the robustness of model evaluation and to examine their ability to perform knowledge editing across varied and challenging scenarios. The dataset includes three types of tasks: visual entity editing, visual semantic editing, and user-specific knowledge editing. Each task is represented through free-form natural language descriptions combined with images, generated by large language models (LLMs) and verified through human annotation to ensure consistency. The paper evaluates several state-of-the-art multimodal models, including BLIP-2, MiniGPT-4, and LLaVA-1.5, providing insights into the strengths and limitations of current models across different knowledge editing tasks.", "questions": "(1) Would it be more beneficial to construct knowledge editing samples using incremental, real-world updates? Although challenging, creating samples based on real-world updates, such as player transfers or recent match results sourced from news articles, could enhance dataset relevance. Using real, evolving information would help models learn to handle dynamic knowledge updates and might improve their applicability in practical knowledge editing scenarios.\n\n(2) What impact did human verification have on the LLM-generated descriptions? An analysis or examples of descriptions before and after human verification would help demonstrate how much human intervention improved the dataset's consistency and accuracy.\n\n(3) Is there a comparison of Visual Entity Editing data quality between MMKE-Bench and prior datasets (e.g., MC-MKE, VLKEB)? While Table 1 highlights MMKE-Bench’s increased task diversity with the addition of Visual Semantic Editing and User-Specific Editing, it would be useful to understand how MMKE-Bench compares to previous datasets in terms of data quality for Visual Entity Editing. Such a comparison could clarify whether MMKE-Bench offers improvements in data quality alongside task diversity.\n\n(4) For additional questions, please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) Diverse Task Setup: MMKE-Bench covers three distinct knowledge editing tasks, from entity-level editing to more complex user-specific knowledge editing. This provides a comprehensive tool for evaluating multimodal models' ability to update knowledge and handle personalized information.\n\n(2) Free-Form Natural Language Descriptions: Unlike traditional triple-based representations, this benchmark uses natural language descriptions to represent knowledge items, enabling models to engage in editing tasks in a more realistic scenario. The free-form descriptions combined with image data make the tasks closer to the complexity of real-world semantics.\n\n(3) Extensive Evaluation Using State-of-the-Art Multimodal Models: The paper evaluates MMKE-Bench on prominent multimodal models such as BLIP-2, MiniGPT-4, and LLaVA-1.5. These models represent the cutting edge in the multimodal field, making the experimental results widely relevant and reflective of current model capabilities in knowledge editing tasks.\n\n(4) Systematic Experimental Analysis: The paper provides a thorough evaluation of various existing knowledge editing methods, including FT-LLM, KE, MEND, SERAC, and IKE, offering a comprehensive performance baseline for each task type. This analysis provides valuable insights into how different methods perform across the tasks presented in MMKE-Bench.", "weaknesses": "(1) Figures 1 and 2 Could Benefit from Improved Clarity and Accessibility: Figures 1 and 2 could be refined to enhance clarity and accessibility for a broader audience. In Figure 1, the example used is soccer-related, which may not be immediately understandable to readers unfamiliar with the sport. A more universally recognizable example, such as common objects or activities, could make the data construction process clearer. For Figure 2, the visual design could better distinguish the four dataset construction steps; currently, it’s difficult to determine which modules belong to each step. Using distinct colors, numbering, or borders to separate steps would help readers follow the process more intuitively.\n\n(2) Limitations of Counterfactual Data for Real-World Applications: The counterfactual approach used to construct MMKE-Bench may lead to a distribution that differs from real-world data. While counterfactual samples aid in testing robustness, they represent hypothetical rather than naturally occurring scenarios. As a result, models fine-tuned on this dataset might learn patterns specific to these counterfactual cases, potentially reducing their effectiveness in real-world knowledge editing tasks. A comparison of model performance on both counterfactual and real-world data would provide valuable insights.\n\n(3) Lack of Empirical Analysis on the Impact of Human Verification: The paper mentions that human verification was conducted to improve the consistency of LLM-generated descriptions, but it does not provide examples or quantitative comparisons to illustrate the impact of this verification process. Including a before-and-after analysis would strengthen the case for the added value of human verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6I1RbwZjYc", "reviewer": "Reviewer_Bskj", "summary": "This work focuses on knowledge editing in large multimodal models and introduces a new benchmark. It presents three distinct types of editing tasks: visual entity editing, visual semantic editing, and user-specific editing, using free-form natural language as input data for these edits. The benchmark also diverges from previous versions by removing the T-Gen test and adding a T-Rel test. The evaluation includes five editing methods across three large multimodal models (LMMs), effectively validating the benchmark's dataset.", "questions": "As mentioned in the weaknesses, the similarity of editing visual entity and semantic editing, the user-specific editing scenario and T-Loc test warrant further exploration.\n\nGiven the modest amount of training data, is the dataset volume adequate for methods that require training, like serac or mend? Additionally, how is the IKE method adapted to LMM editing? Which examples are used in context? Is target knowledge incorporated within the context, and do you include image-based examples in this context?\n\nIn Section 5.2.1, there is a claim that “5) Modern LMMs excel in producing and applying edited knowledge. For reliability, generalization, and portability evaluations, LLaVA-1.5 outperforms BLIP-2 and MiniGPT-4. This improved performance can be attributed to its larger model size and better instruction-following capability.” This raises two questions: First, is LLaVA-1.5 indeed larger than MiniGPT-4, as both use a 7B LLM, and MiniGPT-4's vision encoder appears larger? Second, this statement is not directly related to the benchmark’s core focus, which is to compare editing methods rather than models.\n\nIn Section 5.2.2, further clarification is needed regarding the meaning of “user number” and “allowed maximum items.” Additionally, what is the precise gap between editing and testing in user-specific editing, and why is the gap in the two visual editing tasks similar (1, 3, 6, 10) without larger gap?\n\nTypo:\nFigure 2, bottom-right: \"G-Rel.\"\nTable 2: \"Entity Entity Editing\"\nTable 6: SERAC GAP 3 T-Rel, missing decimal point", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The use of free-form natural language as input for knowledge editing tasks is a notable strength, enhancing flexibility and making the approach adaptable. The clarity of the writing aids comprehension, and the experimental setup is well-documented. Additionally, the benchmark spans diverse data sources and entity types, allowing for broad applicability across different tasks.", "weaknesses": "There is some overlap between visual entity editing and visual semantic editing, as both tasks involve understanding image content, which could blur the distinction between these editing types. Additionally, the user-specific editing scenario may lack practicality. In real-world applications, database or memory-based search might be more effective than training user-specific information for each user to achieve personalization in LLMs or LMMs.\nRegarding the T-Loc test, there’s room for improvement. The results are near 100, suggest that the randomly sampled T-Loc questions are relatively easy for methods such as SERAC, FT-Alignment, and MEND. Introducing more challenging cases could enhance the evaluation’s robustness. For instance, collect similar but harder test cases, either through web crawling or using LLM-generated content (e.g., from GPT) and verify them. This could improve the test’s effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JZ3hdt4yLe", "reviewer": "Reviewer_PUQh", "summary": "This research investigates knowledge editing within large multimodal models, introducing a novel benchmark for evaluation. It defines three specific editing tasks: modifying visual entities, altering visual semantics, and implementing user-specific adjustments, all using natural language as the input format. The benchmark is validated through an assessment of five different editing techniques across three large multimodal models, effectively confirming the dataset's applicability.", "questions": "As mentioned in the weaknesses.\n\nFrom figure 1, there is no T-Gen test in MMKE-Bench, and you add a T-Rel test. Why do you make this change?\n\nAdditionally, how do you adapt each method to the lmm editing?\n\nFigure 4: why I-Gen not evaluated for MMEdit?\n\nSection 5.2.2 requires additional clarification on the terms “user number” and “allowed maximum items.” Further, what about the mend method in your sequential editing setting? And are the gaps substantial enough for the test? In LLM knowledge editing studies and the referenced vlkeb work, this gap can be larger, reaching 100 or more.\n\nThe analysis in line 425-426 only explains possible reason for serac, leaving out mend.\n\nLine 450, “visual knowledge” can be ambiguous, and the reliability seems to be not lower as stated.\n\nLine 466, how do you draw the conclusion “parameter-based methods are better at applying edited knowledge to new contexts”?\n\n\nWriting issue:\n\ndouble quotes in line 324-327 and elsewhere: please pay attention to how to type correct left quotation mark in latex.\n\nFigure 1: what is the difference between original and editing knowledge of user-specific editing? They are same that lead to confusion, which can be improved for btter clarity.\n\nFigure 2: Should the \"G-Rel.\" change to “T-Rel”? And the two question in the last box (bottom right) are same. And placing “hint” there can be confusing, as if the hint itself is part of the input.\n\nLine 355: you use “MLLMs” here, but in other places, you use LMMs\n\nTable 2: \"Entity Entity Editing\"\n\nTable 6: missing decimal point in SERAC results, GAP 3 T-Rel column", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Utilizing free-form natural language as input for knowledge editing tasks stands out as a key advantage, offering greater flexibility and adaptability to various contexts. The data collection process is robust and comprehensive, and the benchmark supports both individual and sequential editing tasks.", "weaknesses": "The training dataset size seems to be small, which might not be enough for mend/ serac that need training. This could potentially lead to lower performance of such methods.\n\nThere is potential to enhance the T-Loc test. The current results, which are close to 100, indicate that the randomly selected T-Loc questions may be relatively simple for existing methods. Strengthening the evaluation could involve introducing more challenging cases. For instance, given that GPT was used in data collection, generating analogous but more difficult test cases and validating them could increase the test's effectiveness and robustness.\n\nThe division of the three tasks lacks clarity. Both visual entity editing and visual semantic editing require image comprehension and retention of the target knowledge provided in the text. While they may involve different data sources, they essentially fall under the same task, visual question answering. Similarly, user-specific editing could be seen as an application scenario, but it still aligns with the core task of VQA, making no real difference in “task type.” \n\nAdditionally, I question about if this user-specific editing approach is practical. In real-world applications, creating personalized databases for each user might be more efficient and effective than modifying the model or embedded knowledge directly.\n\nAnd the task generalization might not be persuasive using only one case. If you want to prove this, more comprehensive experiments and evaluations should be conducted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)