M2Edit: Locate and Edit Multi-Granularity Knowledge in Multimodal Large Language Model

Authors: Yang Zhou, Pengfei Cao, Yubo Chen, Qingbin Liu, Dianbo Sui, Xi Chen, Kang Liu, Jun Zhao
Primary area: generative models
Keywords: multimodal knowledge editing; multi-granularity knowledge; m2edit; multimodal large language model;

Abstract:
Multimodal knowledge editing is an important method for modifying outdated or incorrect knowledge in Multimodal Large Language Models (MLLMs). However, existing datasets for multimodal knowledge editing lack multi-granularity knowledge. In this paper, we present a more realistic dataset called M2Edit, which includes three distinct types of knowledge: entity, relation, and action. Additionally, existing knowledge editing methods for MLLMs lack the ability to handle multi-granularity knowledge and generalize to multimodal data. To address these limitations, we propose the multimodal knowledge editing method MLE. This approach identifies key knowledge layers within different components and collaboratively edits the various components of MLLMs. As a result, we observe significant improvements in visual generality performance, ranging from 4.8 to 10.8, and achieve the best overall performance on knowledge data of different granularities.

Decision: reject
Reviewer scores: 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1729518108210

thread_type:
official_review

<thread_object>
creator: reviewer_m9ma

<content>
summary:
The paper explores advanced techniques in knowledge editing for multimodal large language models (MLLMs). 
The authors introduce M2Edit, a dataset to enhance multimodal knowledge editing by incorporating multi-granularity knowledge types—entities, relations, and actions. 
They highlight the limitations of existing methods and datasets in addressing multi-granularity and multimodal challenges. 
The paper proposes a method, MLE (Multimodal Location-based Editing), which improves knowledge editing by identifying and modifying key knowledge layers across the various components of MLLMs. 
Experiments show that the method improves visual generality performance and achieves superior results on multi-granularity knowledge compared to existing benchmarks.

strengths:
- The reviewer finds the authors' motivation for considering multi-granularity in knowledge editing to be interesting, which aligns with intuitive understanding of the subject.
- The dataset contributed by the authors (if executed flawlessly) could potentially be a significant asset to the field of multimodal knowledge editing.
- The method introduced by the authors outperforms the baseline.

weaknesses:
The paper's primary issue appears to be the unclear expression and presentation of content, with so many details lost, making it difficult for the reviewer to understand the whole story fully. Furthermore, significant potential issues have been identified within the dataset section.


## Major Concerns:
- The reviewer questions the setting of multimodal knowledge editing as posited by the authors, perceiving that it remains limited to textual LLM thinking. Notably, the M2Edit only edits the textual part of image-text pairs, which implies no equivalent editing for visual knowledge, thereby not affecting the image component. If the entire multimodal knowledge editing topic is defined in this manner, the reviewer questions the scientific validity of this definition and suggests that a fundamental improvement is necessary. The reviewer suggests that the authors further clarify this point in their discussion.
- While the paper mentions relational type knowledge in a triplet format, the examples shown in Figures 1 and 2 do not represent triplets but rather entity-level knowledge. The manifestation of relation-level knowledge editing remains unclear. The reviewer recommends that the authors revise and clarify this point clearly in the text.
- The dataset claims the importance of three levels of knowledge but does not integrate these levels within a single scope; different levels of annotations cannot coexist within the same instance, which likely limits the dataset's utility. Therefore, the reviewer hopes that the authors can further explain and clarify this matter.
- The data annotation process is not clearly articulated, raising concerns about the control over data quality, especially as it relies entirely on an automated process via ChatGPT, which is prone to introducing noise. Please provide a detailed description of this step in the manuscript.
- Figure 2 is really challenging to understand; it is unclear what the multiple lines of text within circles represent. Please provide further details.
- Similarly, Figure 3 is also difficult to decipher; the meanings of various arrows and shapes within the figure are not explained, and the significance of the different rectangles in the bottom-left box and what r, s, t represent are not clarified. Please provide additional information.
- In the methods section, the authors claim that to address the limitations of existing knowledge editing methods—which cannot handle multi-granularity knowledge and lack generalization on multimodal data—they propose a method called MLE (Multimodal Location-based Editing). However, the reviewer does not understand the causal relationship between the existing methods' inability to handle multi-granularity knowledge and the proposed "Locate Then Edit" approach. Is it necessary for multi-granularity knowledge editing to be implemented specifically through a "Locate Then Edit" method? 
- The methods were only validated using older MLLMs like BLIP2-OPT and MiniGPT4, which may not represent the most advanced MLLMs, thus not sufficiently proving the effectiveness and generality of the proposed multimodal knowledge editing methods. The reviewer suggests adding more MLLMs for experimental comparison.
- The experimental analysis conducted by the authors lacks sufficient depth and breadth. The reviewer strongly recommends enhancing the content of the experimental analysis.
- The absence of any anonymous links for accessing model code and data examples impedes the reviewer's ability to further investigate and address the issues raised, casting doubts on the reproducibility of the research. Will the authors consider open-sourcing the code and resources?



## Typos & Expression

Overall, the writing and expression in the paper are overly casual and lack the refinement expected in scholarly communication.

- There is a grammatical error on page three, line 112.
- All images in the paper are non-vectorial.
- The citation format throughout the paper does not adhere to standard academic norms.
- There are numerous detail-oriented issues, such as inconsistent punctuation in equations—some equations end with a comma or period while others do not, creating a disorganized appearance.


Overall, the reviewer is open-minded. If the authors can actively and effectively address these concerns, the reviewer would consider raising the rating.

questions:
The paper contains quite many aspects that are not clearly explained, making it challenging for the reviewer to understand. Below are some questions that need addressing:


- The term "in-scope" mentioned in Figure 2 and its caption is ambiguous; does it refer to "in-domain"?
- The caption in Figure 2 states, "After editing the MLLMs, the in-scope samples need to be generalizable, and the out-of-scope samples should not be unchanged", but this statement is confusing and lacks clarity.
- The head entity "Arcadia" mentioned on page four, line 177, is not visible in the middle part of Figure 2, making the reviewer confused about its inclusion and relevance.
- Beyond BLIP2-OPT and MiniGPT4, how does the proposed method perform on other state-of-the-art MLLMs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730093898004

thread_type:
official_review

<thread_object>
creator: reviewer_ddnb

<content>
summary:
The paper introduces M2Edit, a dataset with entity, relation, and action knowledge types, designed for multimodal large language models (MLLMs). It highlights the challenge of knowledge editing across different granularities within MLLMs and proposes the MLE (Multimodal Location-based Editing) method to tackle this. MLE sequentially identifies and edits key knowledge layers within MLLMs, enhancing generality and effectiveness in multimodal contexts. The model demonstrates improved accuracy and generalization over previous methods.

strengths:
- Innovative multi-granularity approach in knowledge editing for MLLMs, addressing a gap in existing datasets.
- The MLE method shows significant performance improvements, particularly in visual generality and model adaptability.
- Offers detailed methodology for locating and editing specific knowledge layers within MLLMs, aiding model interpretability.

weaknesses:
The proposed method is evaluated on a limited range of multimodal models, which restricts the generalizability of the findings across other MLLMs with different architectures or training objectives. Specifically, the recent VL models like QwenVL2, Llava should be evaluated.

questions:
No
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714290642

thread_type:
official_review

<thread_object>
creator: reviewer_ylfa

<content>
summary:
The paper works on multimodal knowledge editing.
It introduces a new dataset for this task, M2Edit (Multi-Granularity Multimodal knowledge Editing dataset). This dataset incorporates multi-granularity knowledge (relation, entity, and action) to address the limitations of existing multimodal knowledge editing datasets.
Moreover, the paper proposes a multimodal knowledge editing method, MLE (Multimodal Location-based Editing).
It identifies key knowledge layers within different components of MLLMs and collaboratively edits them to improve the model's performance on multimodal data. The method demonstrates significant improvements in visual generality performance and performs well in terms of different knowledge granularities.

strengths:
1. The paper is overall clear and well-organized.
2. The paper proposes a useful dataset M2Edit including multi-granularity knowledge. This addresses the limitations of previous multimodal knowledge editing datasets.
3. The proposed method MLE can edit multi-granularity knowledge within MLLMs.
4. The report experiments verify the performance of MLE.

weaknesses:
1. The paper ignores the discussion on the complexity of the MLE method.
2. The paper currently has limited analysis of error cases. Adding this could inspire further research work.
3. The uploaded material doesn't include the code, only the used dataset.
4. The paper doesn't mention how many samples the method edits at once, so it seems the paper does not report the results of batch editing.
5. The introduced dataset M2Edit seems to include counterfactual knowledge. How does the MLE perform with real-world knowledge as in [1,2]?

[1] MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions  
[2] Updating Language Models with Unstructured Facts: Towards Practical Knowledge Editing

questions:
1. Line 483, To address --> to address
2. It is hard to recognize the sentences with the striped background in Figure 2.
3. Can you provide some analysis of error cases?
4. The supplementary material only contains the M2Edit dataset. What about the code of MLE?
5. The paper should explain how the various metrics are computed.
6. How many samples do you edit at once? What is the performance of MLE when editing with several samples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731987349703

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
## Q2: ``Regarding the clarity of Figure 2.``

We sincerely apologize for the issues with Figure 2 (line 162-173). During submission, embedding vector graphics caused compilation errors, and design flaws made the figure difficult to read. We deeply regret this and would like to provide a detailed explanation of Figure 2 for the reviewers’ reference. We greatly appreciate your understanding.

> The data setup in our paper largely follows prior work \[1,2] on knowledge editing in multimodal large language models. In this context:
>
> * **Edit target** refers to the knowledge that needs to be edited. To ensure this knowledge does not already exist in the multimodal language model and to rigorously test the editing capability, all edit targets in M2Edit involve counterfactual knowledge. For example, in the case of entity editing (left panel of Figure 2), the original knowledge about "capybara" is edited so the model’s output should instead be "koala."
>
> - **In-scope** refers to all content that should be altered by a single edit.
> - **Out-of-scope** refers to content that should remain unaffected by the edit.



> ### Entity Editing (Left Panel)
>
> The four images in this panel represent the concept of "capybara." The questions—“What animal is presented in the image?”, “What is this animal?”, “What kind of animal is this?”, and “What is the category of this animal?”—probe the entity's identity. The editing target aims for the model to consistently output "koala" regardless of which image of "capybara" is used or which phrasing is applied in the question. At the same time, for unrelated entity knowledge, such as identifying a “library” from its image and answering “What is this place?”, the model should still correctly respond with “library” after the edit.



> ### Relation Editing (Middle Panel)
>
> The four images depict the concept of "Arcadia." The questions—“In which state is the location depicted in the image situated?” and “Can you identify the state where the place in the picture is located?”—probe the location relation (state). The editing target ensures that, after the edit, the model consistently outputs "Louisiana" for any image of "Arcadia" and any phrasing of the location-relation question. Simultaneously, for unrelated relation knowledge, such as identifying the occupation of "Peter Morgan" from his image and answering “What job does this individual in the picture do?”, the model should still output “actor.”



> ### Action/Event Editing (Right Panel)
>
> The four images illustrate the concept of "running." The questions—“Can you describe what the [agent] is doing at [place] to move?” and “What action is the [agent] undertaking at [place] that involves moving quickly?”—probe the action being performed. Here, the semantic slots (e.g., "[agent]" and "[place]") are filled based on annotations for images  from the ImSitu dataset [3], such as "[agent]" being "woman" and "[place]" being "outside." For all in-scope questions and images, the edited model should output "sitting" instead of "running." However, for out-of-scope action queries, such as identifying the action “shaving” from an image, the model should still output “shaving” post-edit.

We hope this explanation clarifies Figure 2. Thank you again for your patience and understanding.

---

**Reference**

[1] Can we edit multimodal large language models? EMNLP 2023.

[2] MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing. ACL 2024.

[3] Situation recognition: Visual semantic role labeling for image understanding. CVPR 2016.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732074358642

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
## Q1: ```Batch editing results.```

$$\begin{array}{lcccclcccclcccc}
\hline
\textbf{Method} & & &\textbf{Entity}                   &        &                     && & \textbf{Relation}                       &                    &&  & \textbf{Action}                          \\\\ 
                                 & \textbf{R}     & \textbf{T-G}  & \textbf{V-G}  & \textbf{L}    &  & \textbf{R}    & \textbf{T-G}  & \textbf{V-G}  & \textbf{L}    &  & \textbf{R}    & \textbf{T-G}  & \textbf{V-G}  & \textbf{L}    \\\\                               \hline 
\textbf{\emph{For BLIP2-OPT}}   \\\\ \hline
\textbf{FT}                      & \textbf{67.4} &20.2 &15.6 &26.4 & &53.2 &18.7 &8.5 &40.2 & &\textbf{81.3} & 32.6 &8.9 &43.3         \\\\
\textbf{MEND} & 48.1 &44.2 &32.5 &80.4 & &42.0 & \textbf{38.6} &31.8 &\textbf{83.1} & &73.2 &65.3 &35.4 &90.4                       \\\\
\textbf{ROME} & 45.4 &41.8 &26.9 &82.5 & &38.3 &35.3 &35.0 &79.5 & &76.7 &63.2 &41.2 &\textbf{91.2}                   \\\\ \hline
\textbf{MLE}  & 65.9 &\textbf{45.2} &\textbf{46.3} &\textbf{83.1}& &\textbf{47.2} &37.2 &\textbf{43.3} &80.5 & &77.3 &\textbf{66.8} &\textbf{54.8} &\textbf{91.2}                    \\\\ \hline \\\\ \hline 
\textbf{\emph{For MiniGPT4}}                                                                                                                                                                                                                                        \\\\ \hline \hline
\textbf{FT}   & 24.2 &5.8 &5.2 &26.3 &&15.0 &4.7&1.4&38.2&&28.9&22.3&5.4&54.3                            \\\\
\textbf{MEND} & 53.7&50.2&34.4&82.4&&46.7&38.4&24.7&88.2&&63.4&55.3&43.2&92.3                            \\\\
\textbf{ROME} & 55.2 &48.6 &32.4 &\textbf{84.0} &&48.2&39.1&27.2&89.2&&72.3&59.4&48.9&93.3                        \\\\ \hline
\textbf{MLE}  & \textbf{61.3} &\textbf{51.9} &\textbf{43.8} &82.6 & &\textbf{51.3}&\textbf{39.5} &\textbf{34.3}&\textbf{90.1}& &\textbf{74.7}&\textbf{61.5}&\textbf{53.7}&\textbf{93.5}                  \\\\
\end{array}$$

> * The table shows batch editing results (500 edits). From the table, it is evident that our method maintains a significantly strong comprehensive performance compared to the baseline model, especially in terms of multimodal generalization (V-G), outperforming the baseline model across all three knowledge settings.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces M2Edit, a multimodal knowledge editing dataset encompassing multi-granularity knowledge types (entity, relation, and action) and proposes the MLE method for locating and editing knowledge in multimodal large language models (MLLMs). While the paper addresses an important challenge and demonstrates promising results on visual generality and task-specific improvements, it suffers from several critical issues. The evaluation is limited to older MLLMs, and broader applicability to more advanced models remains unverified. Additionally, the methodology lacks sufficient clarity and theoretical rigor, particularly in the "Locate-Then-Edit" framework and its relationship with multi-granularity knowledge editing. The dataset construction also raises concerns about data quality and diversity, as it relies heavily on automated methods without sufficient manual verification. These weaknesses outweigh the paper's contributions, leading to a recommendation to Reject.