MQuAKE-Remastered: Multi-Hop Knowledge Editing Can Only Be Advanced with Reliable Evaluations
Authors: Shaochen Zhong
Yifan Lu
Lize Shao
Bhargav Bhushanam
Xiaocong Du
Yixin Wan
Yucheng Shi
Daochen Zha
Yiwei Wang
Ninghao Liu
Kaixiong Zhou
Shuai Xu
Kai-Wei Chang
Louis Feng
Vipin Chaudhary
Xia Hu
Venue: ICLR 2025 Spotlight (2025)
Keywords: knowledge edit
model edit
multi-hop
question answering
natural language processing
dataset audit
Primary area: datasets and benchmarks
TL;DR: Updating one knowledge fact will produce a ripple effect, making multi-hop knowledge editing (MHKE) a desired capability for reliable LLMs. We reveal many unknown errors of MQuAKE — the most popular MHKE dataset — though an audit and fix everything.

Abstract: Large language models (LLMs) can give out erroneous answers to factually rooted questions either as a result of undesired training outcomes or simply because the world has moved on after a certain knowledge cutoff date. Under such scenarios, *knowledge editing* often comes to the rescue by delivering efficient patches for such erroneous answers without significantly altering the rest, where many editing methods have seen reasonable success when the editing targets are simple and direct (e.g., *``what club does Lionel Messi currently play for?''*).

However, knowledge fragments like this are often deeply intertwined in the real world, making effectively propagating the editing effect to non-directly related questions a practical challenge (to entertain an extreme example: [*"What car did the wife of the owner of the club that Messi currently plays for used to get to school in the 80s?"*](youtube.com/watch?v=DbwiHC1Fu-E\&t=132s)). Prior arts have coined this task as *multi-hop knowledge editing* with the most popular dataset being MQuAKE, serving as the sole evaluation benchmark for many later proposed editing methods due to the expensive nature of constructing knowledge editing datasets at scale. 

In this work, we reveal that **up to 33\% or 76\% of \mquake{}'s questions and ground truth labels are, in fact, corrupted in various fashions due to some unintentional clerical or procedural oversights**. Our work provides a detailed audit of MQuAKE's error pattern and a comprehensive fix without sacrificing its dataset capacity. Additionally, we benchmarked almost all proposed MQuAKE-evaluated editing methods on our post-fix dataset, **MQuAKE-Remastered**. We observe that many methods try to overfit the original MQuAKE by exploiting some dataset idiosyncrasies of MQuAKE. We provide a guideline on how to approach such datasets faithfully and show that a simple, minimally invasive approach — **GWalk** — can offer beyond SOTA editing performance without such exploitation. The MQuAKE-Remastered datasets and utilities are available at [huggingface.co/datasets/henryzhongsc/MQuAKE-Remastered](https://huggingface.co/datasets/henryzhongsc/MQuAKE-Remastered) and [github.com/henryzhongsc/MQuAKE-Remastered](https://github.com/henryzhongsc/MQuAKE-Remastered), respectively.

Published as a conference paper at ICLR 2025
MOUAKE-RENASTERED: MULTI-HOP KNOWLEDGE EDITING CAN ONLY BE ADVANCED WITH RELIABLE EVALUATIONS
Shaochen (Henry) Zhong Yifan Lu" Lize Shao Bhargav Bhushanam Xiaocong Du 20 Yixin Wan : Yucheng Shi 9 Daochen Zha + Yiwei Wang Ninghao Liu 9, Kaixiong Zhou ? Shuai Xu Kai- Wei Chang Louis Feng < Vipin Chaudhary and Xia Hu
Department of Computer Science, Rice University
School of Computing. University of Georgia
< Department of Electrical and Computer Engincering, North Carolina State University
Department Computer and Data Sciences, Case Western Reserve University Department of Computer Science, University of California, Los Angeles 00 Meta Platforms, Inc.
ABSTRACT
Large language models (LLMs) can give out erroneous answers to factually rooted questions either as a result of undesired training outcomes or simply because the world has moved on after certain knowledge cutoff date. Under such scenarios, knowledge editing often comes to the rescue by delivering efficient patches for such erroncous answers without significantly altering the rest, where many editing methods have seen reasonable success when the editing targets are simple and direct (e.g., "what club does Lionel Messi currently play for?"). However, knowledge fragments like this are often deeply intertwined the real world, making effectively propagating the editing effect non-directly related questions a practical challenge (to entertain an extreme example: "What car did the wife of the owner of the club that Messi currently plays for used to get school the 80s?' 80s?'.. Prior arts have coined this task as multi-hop knowledge editing with the most popular dataset being MQUAKE, serving as the sole evaluation benchmark for many later proposed editing methods due to the expensive nature constructing knowledge editing datasets at scale. In this work, we reveal that up to 33% or 76% of MQUAKE's questions and ground truth labels are, fact, corrupted various fashions due to some unintentional clerical procedural oversights. Our work provides detailed audit of MQUAKE's error pattern and a comprehensive fix without sacrificing its dataset capacity. Additionally, we benchmarked almost all proposed MQUAKEE E-evaluaed editing methods on our post-fix dataset, MQUA SQQUAEE-RES KE-REMMATEEED. We observe that many methods try to overfit the original MQUAKE by exploiting some dataset idiosyncrasies of MQUAKE. We provide guideline on how to approach such datasets faithfully and show that simple, minimally invasive approach GWalk can offer beyond SOTA editing performance without such exploita- tion. The htg..a/gge....oo//etee//ee/////e
datasets and utilities are available
and thu..
respectively.
INTRODUCTION
Given the widespread public-facine popularity of various Large Language Model-poweree (LLM) products (Zhao et al. 2023, Yang et al., 2024b), even an occasional user has likely experienced LLMs giving out erroneous answers to factually rooted, knowledge-intensiee questions. While why LLMs would hallucinate such kind of misinformation is complex and still an open problem
Equal contribution. Work corresponds to Shanchen (Henry) Zhong hww.ochceeehacceeeeeeeee zhonoeeereee..nnon.e.eee edu> and done while in collahoration with Meta. For interested readers, it was Rolls-Royce; youtube.
wtwp::/wwwwwocooooocccccc 32g

---

Published as a conference paper at ICLR 2025
noisy training data, model bias, out-of-distribution questions, even simply because the world has moved on after a certain knowledge cutoff date... all likely contributed their fair share to this
rather undesired character of LLMs (Huang al. 2023; Zhang et al. (023) under practical context, knowledge editing is often considered the go-to remedy by delivering efficient patches for such erroneous answers without significantly altering the LLM's output on unrelated queries,
nor undergoing another extensive pretraining or finetuning section (Sinitsin et al. 2020) Mitchell al. 2022).
With the growing need for more credible and trustworthy LLMs, a vast amount of LLM-specific knowledge editing methods have been proposed, and many of them have seen reasonable success in addressing simple and direct editing targets. For example, most modern knowledge editing methods can reliably edit the answer of "What club does Lionel Messi currently play for?" from "Paris to "Inter Miami CF" and therefore correctly reflecting the occupation status of Messi (Zhong et al. 2023).
1.1 MULTI-HOP KNOWLEDGE EDITING POSES PRACTICAL SIGNIFICANCE AND NON-TRIAL CHALLENGES.
However, due to the intertwined nature of different knowledge fragments, a small change in one knowledge fragment can produce liplle-like effects on vast number of related questions (Zhong) ct al., 2023 Cohen et al. 2023). It is often a non-trivial challenge to efficiently propagate the editing effect to non-directly related questions with proper precision and locality. E.g., extreme case - "What car did the wife of the owner of the club that Messi currently plays for used to get school in the 80s?" Many knowledge-edited LLMs can still struggle while being fully aware of Messi's abovementioned club transfer (Zhong et al. 2023
an intentionally
Prior arts have realized the practical significance of being able to edit such complex/non-dine questions upon a certain knowledge update, as different knowledge fragments are almost always deeply entangled with each other in the real world (Zhong et al., [003] Cohen et al., 2023 Weil iet al. 2024). Mcanwhile, exhausting all potential combinations of questions related to one or a few updated knowledge fragments is impractical. Even if it is feasible, this poses high operational costs, and repeated effort would be required should Messi ever opt to transfer again.
Intuitively. a practical knowledge editing method needs to produce correct answers to relevant fac- tual questions with only few updated knowledge fragments available. This task has been coined as multi-hop knowledge editing or MHKE, with the founding, most popular, and only publicly available reflective dataset to date being MQUA KE by Zhong et al. (2023); serving as the sole evaluation backbone for many proposed modern editing methods due to the expensive nature of making counterfactual and temporal datasets at such scale (> 10,000 cases provided, see Table 7).
1.2 UNFORTUNATELY, MQUAKE IS FLAWED DUE TO UNINTENTIONAL CLERICAL AND PROCEDURAL ERRORS - WE FIXED/REMADE IT AND RE-BENCHMARKED ALMOST ALL PROPOSED MULTI-HOPP KNOWLEDGE EDITING METHODS.
While MQUAKE is the founding dataset of multi-hop knowledge editing tasks and very much brings life to this vital subject, through a comprehensive audit, we reveal that up to 33% or 76% of MQUAKE questions and ground truth labels are, in fact, corrupted in various fashions due to some unintentional clerical or procedural errors; which inevitably cast doubts on the effectiveness of the developed methods evaluated on MQUAKE. The issues with MQUAKE significant and growing, especially as MQUAKE becomes a widely adopted (and often solely relied on) evaluation the editing community. Given its importance for building more reliable LLMs- critical aspect of NLP development we present our work to advance multi-hop knowledge editing with the following contributions:
comprehensive audit of MQUAKE: We are the first to present comprehensive audit of the existing errors within MQUAKE (hhogg et al. 2023). bringing awareness to the knowledge editing community regarding this popular dataset with significant task importance attached.
K-AKKeeestteedd We present the only available fix/remake that not only patches all discovered errors, and done so without sacrificing the intended intensity and capacity of the original MQUAKE whenever possible.
Fix/remake MQUAKE to
Extensively re-benchmark of almost all existing multi-hop knowledge editing methods:
Given the currently existing reports based upon the original MQUAKE KE are flawed reflections
2

---

Published as a conference paper at ICLR 2025
of such proposed methods' capability, additionally re-benchmark almost all existing multi-hop knowledge editing methods that are available against our MOUAKE-REMASTEREB REMASTERED datasets.
Present faithful yet beyond SOTA pilot method for future multi-hop knowledge editing development. We observe that many proposed MHKE methods intentionally or unintentionally overfit the original MQUAKE dataset by applying data-specific operations that are largely unique to the MQUAKE dataset family. We provide guidance on how to approach these datasets faith- fully and additionally show that a simple, minimally invasive method GWalk with no such idiosyncrasies- -overrrraisss-orrtiiinggg operations can also achieve beyond-SOTI editing performance.
PRELIMINARY
2.1 BACKGROUND OF MQUAKE
MQUAKE (Multi-hop Question Answering for Knowledge Editing) is a knowledge editing dataset focusing the abovementioned multi-hop question answering tasks proposed in Zhong et al. (2023), where every case of MQUAKE is a multi-hop question made by a chain of single-hop subquestions. Specifically, MQUAKE is constructed based on the Wikidata:RDF dataset (Vran- dečic & Krötzsch 2014). which, in its rawest format, is knowledge graph consisting 15+ trillion of Resource Description Framework (RDF) triples. MQUAKE essentially builds much more concise subgraph with only 37 manually elected common relations and top 20% of the most common enti- ties, where walk of {2. 3, 4)-hop on this subgraph can form case (which chain of {2, 3, 4] single-hop subquestions connected together) in the MQUAKE dataset.
MQUAKE K is presented as two sub-datasets: MQUAKE-CF and MQUAKE-T. The former focuses on counterfactual tasks, while the latter on temporal changes. We highlight that there is also a MQUAKE-CF-3K dataset, subset of MQUAKE-CF that only contains 3,000 cases out of the original 9171 cases. Authors of MQUAKE evaluate their proposed method, Mel.Lo (Zhong et al.) 2023 upon this MQUAKE-CF-
dataset, which then becomes an unspoken standard for the later proposed multi-hop knowledge editing methods (Gu et al., 2024 [Sh et al.) [2024] Wang et al.) 2024] Chen et al. 2024 Cheng et al. 2024). Due to the popularity of this sub-sampled dataset, we provide our error analysis mostly based on MQUAKE-CF-3K and MQUAKE-T in the following $ For
interested readers, we additionally provide the same error analysis upon the full MQUAKE CF in
the Appendix C33
We also collect the dataset statistics in Table 7
2.2 EVALUATING USING MQUAKE
Datasets like MQUAKE-CF and MQUAKE-CF-3K are often tested under varying "editing intensi- ties" based on the number of cases considered "edited." This simulates different levels of deviation between the model's learned knowledge and the newly edited information. This approach is effee- tive because strong knowledge editing methods should handle both large-scale updates and smaller, more localized edits, ensuring that the changes do not interfere with unrelated knowledge.
In its original paper, MQUAKE-CF-33 CF K is evaluated when {1, 100, 1000, 3000} of its 3,000 cases are edited, similarly, MQUAKE-T is evaluated when {1, 100, 500, 1868) of its 1,868 cases being edited, forming an experiment report like Table 6 This kind of report granularity is also adopted by the majority of later proposed multi-hop knowledge editing methods, either in full (Chen et al. 2024) or in spirit with different "k-edited" settings (Gu et al., [2024) Wang et al., 2024; Shi et al. 2024 Cheng et al. [024]] Menggi et al. 2024). In this work, we report at an even finer level of "k-edited" granularity for maximum cross cross-reference potentials.
AUDITING MQUAKE
In this section, we present comprehensive audit of the error pattern that existed in MQUAKE-CET 3k and MQUAKE-T (Zhong et al.) 2023). We specifically note that our audit is there to provide better understanding the knowledge editing community, especially when digesting methods evaluated on these datasets. Our audit is not to discredit the contribution of MQUAKE, or any of the proposed methods evaluated on MQUAKE. We recognize that no dataset can be perfect, especially when it is intrinsically hard to collect large-scale counterfactual and temporal datasets.
3.1 INTRA CONTAMINATION BETWEEN EDITED CASES AND UNEDITED CASES
As discussed in 22.2 having gradual evaluation coverage from few all cases being edited like Table (6 makes sense as an evaluation granularity. However, one critical issue is that e {1, 100, 1000, 3-00t-edited -edited cases (supposed MQUAKE-CF-33 are randomly sub-sampled from
3

---

Published as a conference paper at ICLR 2025
the 3,000 total cases. Thus, there is guarantee that the k-edited cases and (3000 A) unedited cases would require one consistent set of knowledge and, therefore, risk contamination.
Pre- Edit (unedit path):
Case 126:
Chizzn
Continent
Case 482:
Cilizen
Cacilal
Kamal Haasan
India
Asia
Karnal Haasan
India
New Delhi
Suppose Case 482 is selected as an edited case:
Editing Knowledge Bank
Edited Fact from Case 482: Kamal Haasan is citizen of the USA
Post Edit of Case 482
Retrieve contaminatedd odited fact from Case 482
Retrieve
Case 126: (unedited)
Case 482:
Cilizen
Citizen
(edited)
Kamal Haasan
India
Continent
North America
Kamal Hassan
India
Capilal
Washingtor
D.C.
USA
Disangreed by Ground Truth: "Asia"
USA
Correct!
Figure 1: Example of contamination between edited case and an unedited case
For a concrete example, consider the above two multi-hop questions from MQUAKE-CF-3K il- lustrated in Figure [ When case 482 is selected as an edited case, the edited fact in 482 ("Kamal Hassan is a citizen of the USA") ") will contaminate the unedited case 126 since both multi- hop questions involve subquestion inquiring along the knowledge of "The citizenship Kamal Haasan", where the corresponding edited fact would be retrieved. This leads to the model generating label, causing inaccurate experiment readings. See an answer in conflict with MQUAKE-CF-33 We note the bhoove-illustratee contamination is not Appendix C.T for detailed walk walk-through.
a cherry-pickes fluke, but rather a wild-spread error. Here, we sample {1, 100, 1000, 2000, 3000}- editing targets from MQUAKE-CF-3K using random seed 100, and summarize the following error statistics Table
Table Error statistics MQUAKE-CFF and MQUAKE-T (Zhong al. 2023 terms edited
cases contaminating unedited cases. k-edited means i cases out of the total dataset are edited.
MQUAKE-CF-3R
MQUAKE-T
Contaminated
I-edit
100-edit
1000-edit
2000-cdit
3000-edit
1-edit
100-edit
500-edit
1868-edit
Cases
2,013
1,772
910
29
1421
1327
Subquestions
2,706
3,075
1421
1327
It is observable from Table that even small number of edited cases will cause concerningly large contamination to unedited cases and subquestions, where 67% and 76% of all cases from IQUUAKE-CF3KK and MQUA IQUAKE--T KE-T contaminated with just 100 cases being edited, introducing a significant distortion to the reported experiment results conducted in this fashion.
We also note that this contamination decreases as the number of edited cases (k-edit) increases, but it's simply a result of fewer unedited cases being available for contamination as k grows. For example, in the extreme case of 3000-edit, there no contamination between edited and unedited cases in MQUAKE-CF-A because all cases are edited. However, 3000-edit has the highest level of contamination within edited cases, which we explore further in $3.2
One additional notc-worthy detail is in the original MQUAKE literature (Zhong et al. 2023) and most works following setup, "k-edit" (or "editing batch size" in some literature) means only of edited cases are evaluated, without any unedited cases tested. Thus, by such evaluation design, there is no Intra Contamination as discussed this section since no edited cases are evaluated with any unedited case involved. In this work, we evaluated both types of cases to better reflect the locality of different knowledge editing methods (i.e., even in the "1-edit" setting of MQUAKE- evaluate the I edited cases as well the 2,999 unedited cases). We consider evaluating an editing method's performance with both unedited and edited cases while having edited facts present paramount setting. Since only edited cases
CF-3K
4

---

Published as a conference paper at ICLR 2025
being evaluated, might easily end up with method that is proficient dealing with edited facts at the cost of the inability to answer questions citing unedited knowledge. This is practically catastrophic under real-world scenario typically there way mostly round than edited facts. Only evaluating on edited cases might present a false sense of reliability. In fact, found that existing knowledge editing methods often fail by retrieving unrelated edited facts when subquestion demanding unedited knowledge is inquired. See plots highlighting "Incorrect Unedited Retrieval" in Appendix H
unedited facts c.g., Earth
3.2 INNER CONTAMINATION BETWEEN DIFFERENT EDITED CASES
Contamination might also happen among multiple edited cases because certain subquestion pre- sented different edited cases can be edited some but unedited others as illustrated in Figure Similar 3.1 edited fact from
1968 would alter the answer an unedited hop in the edited case 1570. So, the model would generate an answer in conflict with the dataset ground truth label, causing inaccurate experiment readings. See Appendix for detailed walk-through.
Pre-Edit (unedit path):
Case: 1570:
Case 1968:
Official Language
Created by
Heardauarter
Official Larguage
Created
vanharen
He sinki
Finnih
Mike Agricoia
mer QUI
Helsinki
Finrish
Agrioola
Suppose Case 1570 & 1968 are selected as edited cases and each will contribute edit:
Add
Editing Knowledge Bank
Ediled Fact from Case 1570: The official language of Helsinki is Black Speech.
Edited from 1968: Finnish created by William Shakespeare.
Post Edit (edit path):
Retriose
Patrieve
varhamen
Case 570 tediled):
Matii
in
Helsink
Officaal Language
Finnish
X
Created
Wikael Agriccia
Disargreed
by Ground
Trufht
Case 1368 (edited):
marque
Houser
Headcuarter
Helainki
Langaage
Officel
Finnish
Created by
Mikael Agricola
Connect
Agricots
Shokespearn
Figure 2: Example of contamination between two edited cases
This type of contamination is, once again, universally visible in MQUAKE as shown in Table 2 which is very much flipped version of Table [ With k-edit growing, there more edited cases, thus more edited-to-edited contamination. Notably, under the 3000-edit tasks, almost one-third (998/3000, <33%) of the evaluated cases contaminated, which again introduces distortion to the reported experiment results. We omit the report on MQUAKE-T here because there is only one edit-to-edit contamination when all 1,868 cases from MQUAKE-T
edited (case_ide 424).
Table Error statistics MQUAKE-CT-3K (Zhong et al. 2023 in terms edited cases contaminat- ing each others. k-edited means & cases out of the total 3,000 cases are edited.
Contaminated
1-edit
100-edit
1000-edit
2000-edit
3000-edit
Cases
14
265
619
998
Subquestions
14
337
854
1,399
3.3 CONFLICTING EDITS
The two types contamination introduced in $3.1 and $3.2 are indeed subtle and hard to detect. However, MQUAKE-CFF KE-AK----- also includes some straightforward edit conflicts, such for the sub- question "Which company is Ford Mustang produced by?" we have the following edits:
case 2566 (edited):: Ford Moter Company Nintendo. case id: 2 7777 (edited): Ford Moter Company Fiat S.p.A.
This is going
direct conflict when case_id idd 2566 and any of the case_id 231 case_id 2707 are both selected as edited cases, as they shall confuse any knowledge edited
We can't 't speak for the future, but let's hope this fact stays unedited.
Note, an edited does require all of its subquestions heing edited, but merely one of (Table
5

---

Published as a conference paper at ICLR 2025
LLM for having two different answers to the same questions. Fortunately, such types of emrors rather minuscule in MQUAKE-CF-33 with the abovementioned Ford Mustang question and three cases being the only affected data samples.
3.4 MISSING INFORMATION IN MULTI-HOP QUESTION INSTRUCTIONS
As mentioned in $2 the MQUAKE dataset is built upon severely filtered Wikidata:RDF knowl- edge graph (Vrandečić & Krötzsch 2014). Specifically, the triples of a certain {2, 3, 4/-hop walk on this subgraph then fed into GPT-3 5-turbo model to generate three multi-hop question instructions a natural language format. During evaluation, an L.MM is considered right should it correctly answer against any one of the three multi-hop question instructions (Zhong et al. 2023).
However, while repeating generation three times definitely reduces the chances of having incompre- hensible question instructions, we noticed some of such instructions in MQUAKE are still incom- plete. We take case_ide ::::id::46 triple set and its generated 3 question instructions as an example:
case_id 646 (unedited); We have a 2-hop question with "Albert Mohler" as the subject and (employer, religion or worldview) the relation chain. MQUAKE-CF-3K pro- vides the following generated multi-hop questions:
Generation I: What religion Albert Mohler associated with?
Generation #2: Which religion does Albert Mohler follow?
Generation #3: With which religious faith does Albert Mohler identify?
All three generated questions omit the part mentioning which companylinstiutton Albert Mohler is employed by and essentially reduce themselves to single-hop questions, where a correct generation should read like "What religion is Albert Mohler's employer associated with?" and it is not equiva- lent to any of the above three instructions. Without the complete question instruction, suppose there an edit Albert Mohler's employer (which there indeed are some), the final answer would likely change. However, with this omission of information, even the best knowledge-edited LLM cannot answer the question correctly with faithful approach because Alber's employer is not in the picture.
As a general analysis, we find the natural language question instructions of 672 cases in MQUAKE-CF-3K are missing information in comparison to their raw triplet chain. This number is counted in the sense that one or more pieces of information present in the triple chain are missing from all three variants of the generated natural language instruction. Similarly, there are 2,830 and 233 cases of erroneous instructions in MQUA MQUAKE-CF KE-CF and MQUAKE-T, respectively.
3.5 DUPLICATED CASES
The last kind of error we discovered in MQUAKE is simply unintended duplication - i.e., two or more cases sharing the same start subject, edited facts, chain of triples, and the final answer - i.e., they are the carbon copy of each other, yet simultaneously exist in the dataset. We discovered 47, 4, and 4 cases of duplication, respectively, in MQUAKE-CF, MQUAKE-CF-3K, and MQUAKE-T.
REMASTERING MQUAKE
In this section, we illustrate how we modified and improved the MQUAKE dataset to MQUAKE REMASTERED with various fixes on the data samples themselves, well providing utility mod- ules to facilitate how one interacts with such datasets. We provide an audit correctness analysis in Appendix D Furthermore, we demonstrate the impact of our improvements through ablation studies that analyze the types of errors addressed, as discussed in Appendix B
4.1 HARD CORRECTIONS
Three types of error existing in MQUAKE can be fixed once and for all with some careful hard corrections, they are namely Conflicting Edits ($3.)). Missing Information Multi-hop Question Instructions ($3.4), and Duplicated Cases ($3.)). For Conflicting Edits and Duplicated Cases, since there are only few such errors (<50 per type per dataset), we employ some manual corrections to address them: in the former case, we flip the minority edits to align with the majority edits (and adjust their answers to their subsequence subquestions, should there be any); in the latter case, we simply remove such duplicated cases (for MQUAKE-CF-23 from MQUAKE-C CF to keep the dataset having 3,000 cases in total and a 1,000 cases for {2, 3, 4}- hops as the original SQQUKEE-F-33 intended). For errors regarding Missing Information in Multi-hop hop Question Instructions, we rewrite the natural language question instructions for all cases in the datasets using meta-Ilama/lIIame lama-3. 1.maa-3.1-4055. 1-405B (Dubey al. 2024) for consistency sake.
we then manually select 4 more cases

---

Published as a conference paper at ICLR 2025
4.2 DYNAMIC MASKING FOR MAXIMUM COVERAGE: MQUAKE- M REMASTERED-CET M REMASTERED-CF- AND M REMASTERED-T
Due to the contamination count of Intra Edited-to- -nneei-to-UUndiied Contamination ($3.1) and Inner Edited- to-Edited Contamination ($3.2) tend to grow in the opposite direction as shown in Table [ it is impossible to find a fix within the current MQUAKE that can address both issues without significantly decreasing the dataset size. As an alternative, we develop an API that will take a case_id iid and an edi ted_ ted__lasee flag as input, indicating the evaluating case-in- -eueii-uueetonn and whether this case is considered edited; our API shall then return a set of triples that are contamination free by dynamically masking out the conflicting edits from other cases. After such, the user may build up an editing knowledge bank upon such returned triplets and conduct evaluations for any semoryybased knowledge editing methods without losing any cases caused by contamination. Due to the nature of the N N-hop question, at most NN edited facts would be removed for each case, marginal reduction compared to (typically thousands of) total edited facts as summarized in Table T3
and 2
Specifically, once case sess__id-of-iiee in--of-iterestt is given, our API would loop through all of its subquestions and identify if any is considered edited under another case. If there is a hit, the triple for such edited subquestions is removed from the bank of edited triples This dynamic masking mechanism would ensure all cases within the original MQUAKE be usable against memory-based knowledge edit- ing methods. However, the drawback of masking is it won't support parameter-based knowl- edge editing methods, where weight update is required. We additionally provide a MQUAKE- REMASTERED-CE 6634 to address the need for such methods (Appendix [F.I).
5 MAKING SAFE AND FAITHFUL APPROACH TO MQUAKE AND
In addition to our benchmark results we'd show in $ it is our observation that many MHKE methods with decent performance on MQUAKE or EQUAKE-REMASTERS dentally) utilizing designs that leverage dataset idiosyncrasies unique to MQUAKE. For example, methods like RAE (Shi et al., 2024) and GLAME (Menggi et al., 2024 utilize Wikidata (Vran-) dečić & Krötzsch 2014) as the external KG to better detect the edit-induced conflicts, which happen to be the source of MQUAKE as discussed in $2.1 where whether sourcing the same Wiki- data KG as MQUAKE might bring them data-specific advantages remains to be confirmed. Sim- ilarly, PokeMQA (Gu et al., 2024) utilizes the 6,218 cases included in MQUAKE-CF but not in MQUAKE-CT--3 as the train set to train its auxiliary components, then tested on MQUAKE-CF- 3K. Given MQUAKE is a dataset with relatively low diversity (e.g., with only 37 types of relations), whether having a heavily overlapped train and test set will result in data-specific advantages unique to MQUAKE and its variants, again remains to be confirmed (we also note, without considera- tions like within the two sets, which would likely introduce conflicting knowledge via its training scheme).
are (often coinci-
this training split is surely contaminated across and
On the same note, important consideration when using KG to store unedited facts - common approach in methods like RAE (Shi et al. 2024), GLAME (Mengqi et al., 2024), and StruEdit (Bi et al. 2024)- - is the additional operational cost of maintaining a large unedited KG. Since unedited facts typically far outnumber edited ones, managing such a KG requires continuously reconciling it with the KG storing edited facts. That said, one could fairly argue that storing unedited and edited facts allows for the gradual expansion of a unified, ever-evolving "ground truth for all" KG. climi- nating the need to maintain separate model-speeific edited KGs for each L.M.. However, effectively querying such a vast unified KG and adapting its knowledge to LLMs presents additional challenges, particularly if the method also aims to leverage an LLM's internal knowledge. We highlight this is- sue because there are clear trade-offs between these two approaches, and they are rarely discussed in the existing MHKE literature. While we chose not to use an external unedited KG in our pilot method, we believe it is both fair and beneficial to acknowledge this consideration.
5.1 A MINIMALLY INVASIVE BUT PERFORMANT APPROACII: GWALK
Here, we provide a brief walkthrough of a simple method we designed, namely Grlk, GWalk does not leverage any idiosynerasies unique to MQUAKE or MQUAKE-REMASTERE still presents beyond SOTA performance, surpassing many, if not all, established baselines. We illustrate this pilot method as concrete guidance and potential inspiration to our future multi- hop knowledge editing scholars. The design of GWalk hinges on the fundamental pipeline of memory-based knowledge editing methods: where the pool of source only contains edited facts.
Yet, it
7

---

Published as a conference paper at ICLR 2025
This school of editing methods has proven to be successful, mainly because it can leverage the power of retrieval-rrgument generation (RAG) combined with the in-context learning (ICL) capability of LLMs. yet an edited-only knowledge bank is typically minuseule, much like done in McLLo (Zhong et a 2023). Different from MeI.Lo, where all edited facts are converted from triples natural language (NL) descriptions in its edited bank, GWalk preserves the edited facts in their original triples f'ashion and leverages the graph topology. This makes maintaining this edited bank much easier-
one can easily adjust the entity relation on a knowledge graph without rewriting every natural language description of every related edited fact. It also brings more precise retrieval mapping when a question of certain edited fact is asked. Methods like MeLLo (Zhong et al., 2023) rely on RAG from pool of edited facts in NL format. This can lead to unintended retrievals, where irrelevant facts with similar embeddings are retrieved, potentially causing hallucinations. However, if we simply query the entity and relations implied in question against a knowledge graph, there is less chance of retrieving unintended materials. We share the detailed pscudocode of GWalk in Algorithm @ and demonstrate some further studies in Appendix
Algorithm General Procedure GWalk One Multi-hop Question
Input:
M, the Question Answering Language Model;
T. Text-embedding model;
Mulii-hop Question:
E. bank of edited facts as knowledge graph.
Output:
og the answer to Q.
Initialize:
== 1. the subquestion counter:
9, None, the answer
previous sulquestion.
8 <- Extracted subject from Q:
2 rela Prompt
breakdown Q into a sequence of relations. /v If Q is 'What is the officiel Language of the country where Karl Alvarez holds citizenship?', then would be 'Xazl Alvares' and a pesaible rels is ('eitizenship', "official language"]
3 for E rels do
Query s.n,? > against E using T. namely do T(s) first to determine if there is retrievable E then inspect if the e E has an relation edge retrievable by T(r). threshold on embedding similarity for to determine whether an item is retzievable or not.
Prompt generate sulquestion q
a.d
og e- the M M-generrtted answer to
(e.,)) has valid retrivval < a. then
L 0p o':
/a The answer to this subquestion will be the start subject of the next subquestion.
op:
i+iil;
Return ozi
BENCHMARK AND DISCUSSION
Given almost all proposed MHKE methods are (often solely) evaluated on the original, error- contained, MQUAKE datasets. Here, we provide re-benchmark of those methods against post-fix MQUAKE-REMASTERE datasets to provide some more reliable results.
Compared Methods In this work, we aim to cover most, if not all, open-sourced knowledge editing methods specifically evaluated on the original MQUA KE. This includes MeLLo (Zhong et al. 2023). PokeMQQ (Gu et al. 2024). RAE (Shi et al.) (024), and DeepEdit (Wang et al.) 2024 as methods specifically proposed to target this MHKE problem and evaluated on MQUAKE. We additionally include ICE (Cohen et al. 2023) and IKE (Zheng et al., (023a) as these are also methods purposed for the (single-edit) MHKE task, though not specifically evaluated on MQUAKE in their original publications. General editing methods like ROME (Meng et al.] (022) and MEND (Mitchell et al. 2022) are also featured, We note that we are aware methods like GLAME (Mengqi et al. (024). StableKE (Wei et al. 2024). Temple-MQA (Cheng al. 2024), GMeLLo (Chen al. 2024 and StruEdit (Bi et al. (2024) (2022) are also evaluated on MQUAKE, but they are purposely omitted from our c--benchmark coverage due to lack of timely open-sourced implementation, likely because many of these works are still in-submission by our work's 's first appearance.
Covered Models We opt to use Imsys/vicuma-7ber 7b-v11. (Zheng et al. 2023b). mistralai/Mistral 7B- IB-Instruct-v0.2 (Jiang et al.] (2023), and meta- MasaaaMMet----aaaaa as the choice of dneestion-answering models, both for alignment with existing works (Zhong et al,
Instruct (AI@ MAtaMeta 2024)
2023
Shi al. 2024 Gu et al. 2024] well as providing coverage the most recent language
8

---

Published as a conference paper at ICLR 2025
models. For methods that require a text-embedding model as retriever, use facebookkconntrieerr msmarco (Izacard et al. 2022) for alignment with MeLLo (Zhong et al.) (023).
Covered Datasets We will provide coverage on our post-fix dataset, namely MQUAKE- REMASTERED-CF, DOUUKE-REMASTEREEH
and masking fashion illustrated in |4.2 as well as SIEAAK--REMATERRD---TTEIE
in the
in its vanilla the original MQUAKE-CF, MQUAKE KE-CF- 3K, and MQUAKE-1 from Zhong et al. (2023) (with 6334 as an extra for parameter-based methods), but with the types of error mentioned in S Flxeed in the via means illustrated in $4
form. These datasets are respectively corresponding
6.1 RESULTS AND DISCUSSION
Table 3: Performance Comparison of Original MQLAKE and our MQLAKE- MQUAKE-REMASTERES REMASTERED datasets using vicuma-7b-vl.. (Zheng el al. 2023b). For the CF-3k dataset families, we utilized the 1000-edit setting. whereas for the T dataset l'amiliss, we applied the all-edit setting. The original MQUAKE cannot faithfully reflect the true capacities of the methods due to errors specified in especially if the method-in in-question is performant.
MQuAKE-CT-SC Original-v2
MQUAKE-T
Method
Original
Remastered
Original
Remastered
MeI.Lo Zhong et all, 2023
14.4
14.1
14.63
44.37
GWalk (OUR)
36.07
41.0
$7.37
46.41
54.88
Given that our MQUAKE- -EAANEEEMATTEEDD is mostly provided as a fix to MQLAKE, we would like to highlight the drastic result differences when the same method is evaluated on these two datasets (Table B:: where the differences are especially pronounced for stronger methods. Such differences suggest all previous reporting on MQUAKE has room for reliability improvements, which we filled here with MQUAKE-RREE REMASTERRD....
Table 4: Experiments on MQUAKE-REMASTER CFF-ED with numbers of edited cases and methods.
Total Accurucy
Results are reported in the format: (Teat Edited Accuracy. Train Editted Accuracy, Unedited Accuracy)
MQUA R.-RRMAATTERRR-T--.3.
Method
100-edit
1000-edit
3000-edit
6344-edit
Imsys/vicuua 71.
(Zeng 20236
19. 16
19.27
11.17
6.83
McLLo Zhong al.
2023
10.99, 19.37)
(5.1, 9.58, 24.53)
(4.31,
8.55, 23.3)
(4.58, 7.72, 19.05)
ICE Cohen al 2023
0OM
OOM
0OM
OOM
IKE
Zhong
0OM
00M
00M
O0M
PokeMQA Gu et
2024
(3.25, 30.82, 1.59) <1 56.62
21.77
DeepEdi Wing et al
2024
<1
<1
<1
$7.55
61.79
$9,1
GWalk (Ours)
(22.22, 64.84, 57.48)
(29.08, 67. 63,23)
(39.3, 63.74, 64.33)
(44.64, 62. 68.25)
mistrraaa/MisttallP 7B Instruct-v0.2 [Tiang
2023
27.5
27.54
24.37
21.26
McLLo Zhong al.
2023
(cl, 23.08, 27.65) OOM
(12.76, 24, 30.4)
(11.8, 32.06)
(13.29, 24.9, 30.16)
ICE Cohen at 2024
OOM
00M
OOM
8.82
TKF
Zheng
2023a
0OM
00M
00M
(11.11,6.59.8.86
PokeMQA Gu et
20.38
(3.99, 27.41, 69.84) <l
DeepEdit Wang
2024
<1
<I
<1
56.25
56.03
54.43
GWalk (Ours)
(33.33, 14, 56.28) (34.69, 60.57, 60.6)
(42.69, 59.04, 59.85)
(47.49, 57.74, 52.38)
In Table we present benchmark results on
GWalk con- sistently outperforms other methods in terms of models and edit numbers. The "OOM"s in ICE and IKE are due to memory overload from concatenating all edited facts in the in-context learn- ing prompt. Whereas the "<1" results likely stem from the LLM's failure recognize the few- shot examples, often generating irrelevant tokens or failing to follow the few-shot format. This issue was observed with MeLLo using Meta-Llama-3-888 Inn-trutt and with DeepEdit using vicuna- 7b-v1.5 and Disttal-7B-Instruet--oo Instruct-v0.2. Due to page limitation, we refer our readers to Ap- MQUA KE- REMAR-RREATSR----SSSS We present MQUAKE. COMMASSTEE---OO CF-66-334 in the main
pendix for benchmarks of MQUA KE-EEEESTTEEED--CCCOO and MQUA SEE-RRMAATEEEDD. text solely because it can feature the most methods. Last, we highlight that due to the feedback of the reviewers and the community, as well as company policies, we have then updated our datasets for better polishness, which might mean slightly different results than currently reported as we cannot complete full rerun before camera-ready. We will make the official run soon and update with more comprehensive experiment reports with one-click reproducible scripts in our repository.

---

Published as a conference paper at ICLR 2025
7 RELATED WORKS
Audit and Fix of MQUAKE To the best of our knowledge, work has conducted a compre- hensive audit to MQUAKE as we do, but two prior arts have touched on the errors existing in MQUAKE: GM (Chen al. 2024) and DeepFddit (Wang al. 2024)
Table 5: Comparison of error analysixqqantiiiatioonni of MQUAKE provided in different works.
Ref.
Error Types Found
Error Quantified
Error Scopes Fixed
Cost of Fixing
Go
Missing Instruction
No
No
N/A
(Chen
al.
MTX4
Deep
Inner Contamination
CF-3K 3000-cdit
CF-3K in 3000-odir
998 out of 3000 cases from CF-3
Wany
Ours
Intra Contamination,
CF-3K in [1. 100, 1000,
CF-3K, T. CF. 9K in any any-odit: Remastered-CC 6334 in any edit
No case removed from CF-3K, CF-
Inner
Cuntamination,
3000; -edit. in (1, 100,
Coullicting,
500, -ell)- CF-9K
9K.
Missing
Instructions,
[1. 100, 1000, 3000, 6000.
Duplicated Cases
all) -dli
Specifically, GMeLLo (specifically, its anonymous submission ACL ARR 2024 Feb) briefly dis- cusses the inconsistency between the triple chain and the corresponding generated instructions in its $4.5.1, which is the same type of error we discussed in $3.4 However, GMeLLo merely presents two examples of such an error without providing any quantitative error analysis or fix; we did both in $3.4 and $4.1 DeepEdit (Wang al. (024) discovered the same inner contamination error (edited-to-edited) as we discussed in $3.2 but limited to one dataset
DOUUKE--F-3K under one setting (when all 3000 cases are considered edited). Further, DeepEdit removed all 998 inner con- taminated cases from the MQUAKE BBUUKE-CFF-33 dataset- - which is (supposedly) the same 998 cases we detect in Table 2 under the 3000-edit column and named it MQUAKE-2002. While this fix is, of course, helpful, we argue our REMASTERED fixes are much more comprehensive and effective since they patched many more errors revealed in $ (the other four types of errors still exist in -220022,, and most importantly, done so without scarifying almost 1/3 of the capacity of the original dataset thanks to masking utility we proposed $4.2 We further demonstrate the quantifiable difference between our work, GMeLLo, and DeepEdit in Table
MQUAKE KE-CF-3--V2 Other than the above two works, we have privately consulted with the first authors of MQUAKE (Zhong et al. 2023) and confirmed the legitimacy of our findings and those of the prior arts. The MQUAKE team has then rolled out their update: IMQUAKE-CF33R
Upon investigation, MQUAKE-CF-3KK (3.2) and Conflicting Edits ($3.3) However, we respectfully note it still falls short in terms of Intra Contamination as reflected in Table IT (53%+ cases contaminated under 1000-edit), which makes ineligible in evaluating edited and unedited cases at the same time a paramount setting for reasons highlighted at the end of $3.1 Further, MQLAKE-CF-3KK v2 still involves unfixed Dupli- cated Cases ($3.5) and Missing Instructions ($3.4): more on this in Appendix C.4 Beyond unfixed errors, we note the original MQUAKE--F--
indeed resolves the edited-to-edited Inner Contamination
was subsampled from MQUAKE-CU to have 1 1,000 cases per |2. 3, 4) hhop in order to provide a balanced representation. However, this is no longer true under HQQAAEECC-3K-- implied, it is error resolution is limited within the MQUAKE-CF-- dataset, but not MQUAKE- CF nor JIQUAKE-T. While authors, again, pay our utmost respect to the MQUAKE team for bringing the MHKE task to the community, we earnestly argue MQUAKE-CF-32L ing more than our BEAAAKKE-RRMASRRREEEE supports an important setting, and is faithful to the balanced 1,000 cases per (2, 3, 4/-hop design.
shown in Table 7 And, the MQUAKE-CF-3K-V name
offers noth- as the latter fixed substantially more errors,
Benchmark and Guidance Our work re- re-benchmarks nearly all open-sourced knowledge editing methods on MQUAKE and guides on safely and faithfully approuching such datasets. To the best of our knowledge, no other work offers this level of benchmarking or touches on the same issues. Notably, we are likely the only work to evaluate on MQUAKE-CF/MM CEEEEEECCCMMAKK--EEEEAEEE dataset that even the original MQUAKE paper did not assess due to resource constraints. Table [4] illustrates the significant difference in evaluation coverage between our work and previous efforts.
the largest
CONCLUSION
We audited MQUAKE and present benchmarked many MHKE methods with MQUAKE- REMASTERED and presented a simple, ef- ficient, yet extremely capable baseline GWalk for future MHKE studies.
with many errors fixed. We
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was partially supported by NSF Awards ITE-2429680, IIS-2310260, CNS-2431516.
BAC-2112606, and DAC-2117439. Furthermore, this work was supported by the US Department of Transportation (USDOT) Tier-1 University Transportation Center (UTC) Transportation Cyber- security Center for Advanced Research and Education WEYBER-CARES grant #69A3552348332.
Further, this work made use of the High Performance Computing Resource in the Core Facility for Advanced Research Computing at Case Western Reserve University (CWRU). We give special thanks to the CWRU HPC team for their prompt and professional help and maintenance. The views and conclusions in this paper are those of the authors and do not represent the views of any funding or supporting agencies.
Outside funding agencies, our work benefits from the great support of the knowledge editing com- munity of L.MMs. We appreciate the efforts of the PokeMQA team (Gu et al.) (024), the RAE team Shi et al. 2024). and the DeepEdit team (Wang et al. 2024) for confirming our findings and provid- ing support in testing their knowledge editing methods on post-fix MQUAKE-REMASTE
datasets. Last, we give our special shout-out to Zexuan Zhong and Zhengxuan Wu- the first au- thors of the original MQUAKE paper (Zhong et al.) 2023) for validating findings and sharing their insights into our work. We pay our greatest respect to them for their founding efforts in bring- ing the important MHKE task to the field, while being engaged and supportive of future community scrutiny and improvements.
REFERENCES
Al@Meta. Llama 3 model card. 2024.
URL http::///tttuu..o/t/t/// wtt.:/ub..commmteeeeeaaee 11mma/
1lama3/blob/ main/MODED CARD md
Afra Feyza Akyürek, Eric Pan, Garry Kuwanto, and Derry Wijaya. Dune: Dataset for unified editing. 2023.
arXiv preprint
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2023.
Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei, Hongcheng Gao, Junfeng Fang, and Xueqi Cheng. Struedit: Structured outputs enable the fast and accurate knowledge editing for large language models. arXiv preprint arXiv:2409.100)
2024.
Ruirui Chen, Weifeng Jiang, Chengwei Qin, Ishaan Singh Rawal, Cheston Tan, Dongkyu Choi, Bo Xiong, and Bo Ai. Llm-based multi-hop question answering with knowledge graph integration in evolving environments. arXiv preprint 240i 15903, 2024.
Keyuan Cheng, Gang Lin, Haoyang Fei, Yuxuan zhai, Lu Yu, Muhammad Asif Ali, Lijie Hu, and Di Wang. Multi-hop question answering under temporal knowledge editing. arXiv, 2024.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effeets of knowledge editing in language models. Transactions of the Association for Computational Linguisticss 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra. Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan

---

Published as a conference paper at ICLR 2025
Mísra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid EI-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Hontala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh. Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasic, Peter Weng. Prajiwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileamu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov. Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alex Vaughan, Alexei Buevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadch Yazdan, Beau James, Ben Maurer, Benjamin Leonbardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic. Brian Gamido. Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai,, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Lc, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanaa, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk. Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov. Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean -aanistt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthia Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael I.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli,
12

---

Published as a conference paper at ICLR 2025
Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary VeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Ilama 3 herd of models, 2024. URL https: org/ abs/ / 2407 21783
Hengrui Gu, Kaixiong Zhou, Xiaotian Han, Ninghao Liu, Ruobing Wang, and Xin Wang. Pokemqa: Programmable knowledge editing for multi-hop question answering. arXiv, 2024.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang. Qianglong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv, 2023.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ama Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv,, 2023.
Tianjie Ju, Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du, Yubin Zheng, and Gongshen Liu. Investigating multi-hop factual shortcuts in knowledge editing of large language models, 2024. URL
11900
Yi Lu, Jing Nathan Yan, Songlin Yang. Justin T Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M Rush. A controlled study on long context extension and generalization in llms. arXiv preprint arXiv:2409.1218/1 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35, 2022.
Zhang Mengqi, Ye Xiaotian, Liu Qiang, Ren Pengjie, Wu Shu, and Chen Zhumin. Knowledge graph enhanced large language model editing. arXiv, 2024.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing scale. In International Conference on Learning Representations, 2022.
Yucheng Shi, Qiaoyu Tan, Xuansheng Wu, Shaochen Zhong Kaixiong Zhou, and Ninghao Liu. Retrieval-enhaneed knowledge editing for multi-hop question answering in language models. arXiv, 2024.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. International Conference on Learning Representations, 2020.
Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communi- cations of the ACM, 2014.
13

---

Published as a conference paper at ICLR 2025
Yiwei Wang, Muhao Chen, Nanyun Peng, and Kai- Kei Chang. Deepedit: Knowledge editing as decoding with constraints. arXiv, 2024.
Zihao Wei, Liang Pang, Hanxing Ding, Jingcheng Deng, Huawei Shen, and Xueqi Cheng. Stable knowledge editing large language models. arXiv, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL
https:://axxis org/ abs 2407 10671
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power of Ilms in practice: A survey on chatgpt and beyond. ACM Trans. Knowl. Discov. Data 2024b.
Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, and Xia Hu. KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4623-4648, Miami, Florida, USA, Novem- org.
ber 2024. Association for Computational Linguistics. URL 2024. findings emn ww..p.2....... 266
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang. Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's is song in the ocean: A survey on hallucination in large language models. arXiv, 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. A survey of large language models. arXiv, 2023.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv, 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging ai-as-a-judge with mt-bench and chatbot arena. arXiv, 2023b.
Zexuan Zhong, Zhengxuan WU, Christopher D Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
14

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
While we strive to make a solid and comprehensive contribution to the MHKE field and the knowledge editing community at large, we must acknowledge certain shortcomings in our post-fix MQUAKE-REMASTERUE datasets. Notably, AIQUAKERREMAS on knowledge-intensiee tasks where both unedited and edited facts can be represented triples in a KG. However, this assumption does not always hold, as some facts are not practically feasible to deconstruct into triple.
remains focused solely
For example, consider the question-answer pair: "Which actor in Marvel movie has the same initial for both their first and last name?" "Paul Rudd (for Ant-Man)." real-world KG is unlikely to contain such a specific and elaborate relation edge. As result, it remains an open question how MHKE methods would perform when faced with a multi-hop question (or subquestion) that references such eee-rriple-rerreentbblle unedited edited knowledge; on an even broader scale, multifaceted question that can't be easily decomposed into multiple standalone subquestions.
This challenge is akin to the common struggle of deciding when to RAG and when to engage in CoT reasoning. Should retrieval/reasooing be applied to every query, should there be oversight mechanism that determines when an extra operation (in our case, an editing resolution/confirmatioo should be triggered, as explored in works like Self-RAG (Asai al., 2023) Right now, MHKE methods typically check for editing resolutions for every subquestion (and assume such a subques- tion breakdown obtainable in the first place), which is mechanism with strong assumptions and
requires significant support if it is to be deployed at scale.
Although this limitation not unique MQUAKE- REMASTERED D but rather general shortcom- ing of almost all works in MHKE, we believe it is both fair and necessary to highlight it, as it warrants dedicated studies of its own. We are encouraged to see the community moving in a more comprehensive direction, with evaluations like DUnE (Akyürek et al. (023) being constructed. We expect a reasonable complement to simply evaluate MHKE methods with edited facts present on typical datasets that demand no editing operations and observe if they significantly degrade an LLM's performance.
15

---

Published as a conference paper at ICLR 2025
B EXTENDED PRELIMINARY
B.1 DEMO REPORT OF MQUAKE
Table 6: Standard reporting format MQUAKE-CF-3K, and MQUAKE-T demoed with McLLo on Vicuna-7B (Zheng et al., 2023b); k-edited means & cases out of the total cases are edited. Ab- breviated table courtesy Zhong et al. (2023)
(Table 3).
MQUAKE-CF-NS 1000-edit
MQUAKE--
Model
Method
1-edit
100-edit
3000-edit
1-edit
100-edit
500-edit
1868-edit
McLLo
Vicuna-7B
Zhong et al. 2023
20.3
11.9
11.0
10.2
84.4
56.3
52.6
51.3
B.2 DATASET STATISTICS
Table 7:
Dataset Statistics of MQUAKE. Numbers are in terms of cases (a case in MQUAKE is
chain consisting multiple subquestions).
Dataset
of Edits
2-hop
3-hop
4-hop
Total
513
356
224
1,093
487
334
246
1,067
MQUAKE-CF-3K
310
262
572
268
268
All
1,000
1,000
1,000
3,000
1,073
536
136
1, 1,046
MQUAKE-CF-3K-V2
339
229
568
313
313
1,135
1,136
729
3,000
2,454
855
446
3,755
2,425
853
467
3,745
MQUAKE-CF
827
455
1,282
4
436
436
All
4.879
2,535
1,804
9,218
MQUAKE-T
(AIl)
1,421
1,868
Table 8: Dataset Statistics of MQUAKE-REMASTERE REMASTERED. Numbers in terms of cases (a case in MQUAKE is chain consisting of multiple subquestions).
Dataset
# of Edits
2-hop
3-hop
4-hop
Total
I
513
356
224
1,093
2
487
334
246
1,067
MQUAKE-REMASTERERNE REMASTERED-CF-5K
3
310
262
572
4
268
268
All
1,000
1,000
1,000
3,000
1
2,446
850
441
3,737
2
2,415
852
463
3.730
REMASTERED-CF
3
823
451
1.274
4
430
430
All
4,861
2,525
1,785
9,171
MQUAKF-RRMASTERER REMASTERED-T
(AII)
1.421
441
2
1,868
1,971
77
0
2.048
2
2,415
476
14
2,905
MQUAKE-REEME REMASTERED-CF-663M
3
823
128
951
4
430
430
4,386
1,376
.72
6,334
16

---

Published as a conference paper at ICLR 2025
C EXTENDED AUDITING
C.I EXAMPLE OF INTRA CONTAMINATION BETWEEN AN EDITED CASE TO AN UNEDITED CASE ($3.1
For concrete example, consider the following two multi-hop questions from MQUAKE-CF-3K (we also additionally provide the subquestion breakdown and intermediate answers of the two ques- tions for better presentation, we note that such auxiliary information not part of the instruction visible to the -anssirnnn L.LM):
1= (unedited): What is the continent of the country where Kamal Haasan holds citizenship?
What is the country of citizenship of Kamal Haasan? India.
What the continent of India? Asia.
case case_id: 1d:4822 482 (unedited): What is the capital of the country where Kamal Haasan holds citi- zenship?
What the country of citizenship of Kamal Haasan? India
What is capital of India? New Delhi.
The correct pre-edited answer should be "Asia" and "New Delhi" respectively. As Kamal Haasan an Indian citizen, India is located Asia and is the capital of New Delhi. However, suppose case id: 482 is sampled provided with the additional triple containing the knowledge of "The official language of United States of America is Arabic."
edited case while case id: 126 6 remains unedited, will be
Since the unedited case_id:128 and the edited case_ide 482 share the same subquestion of "What is the country of cilizenship of Kamal Haasan?" The answer of case_id 482 will be rightfully updated to "USA" per the new knowledge. However, the unedited case_id: :d:126 still considers the original answer "India" to be correct, and therefore contaminated by the edited case case :id:: 4 82 in an unintended fashion. This is problematic because a successful knowledge editing method should be able to retrieve the edited knowledge USA?" upon the relevant questions (in this case the shared one), and thus answering "North America" to case_id: 126. This technically correct, but conflict with MQUAKE-CI-3K's label, causing inaccurate experiment readings.
"Kamal Haasan citizen of
C.2 EXAMPLE OF INNER CONTAMINATION BETWEEN DIFFERENT EDITED CASES ($3.2
Again, we walk through two cases from MQUAKE-CF-S a concrete example. First, we show them in their unedited format (again, subquestion breakdowns and intermediate answers are here for demonstration purposes and are not visible the question question-answering LLM during evaluation):
case case_id: 1570 (unedited): Who was the creator of the official language used in the work loca- ::::1::15
tion of Matti Vanhanen?
Which city did Matti Vanhanen work in? Helsinki.
What is the official language of Helsinki? Finnish.
Who was Finnish created by? Mikael Agricola.
case case_id: 1 1968 (unedited): Who created the official language of Housemarque' 's headquarters location?
id:
Which city is the headquarter of Housemarque located in? Helsinki.
What is the official language of Helsinki? Finnish.
Who was Finnish created by? Mikael Agricola.
and case id: 1d:19688 are both selected as editing cases, two triples containing the following knowledge will be available: "The official language of Helsinki is Black Speech" (intended for case_ids 1 570), and "Finnish was created by William Shakespeare" (in- tended for case_ ies_iidccase_ide 19:1::9::..669911 968), leading to the following edited breakdown.
Suppose case case_id:11
id:
case mass_id:1170 (edited): Who was the creator of the official language used in the work location of Matti Vanhanen? Which city did Matti Vanhanen work in? Helsinki.
17

---

Published as a conference paper at ICLR 2025
What is the official language of Helsinki? Finnish Black Speech.
Who was Finnish Black Speech created by? J. R. R. Tolkien.
case_idi 1.d:1968 (edited): Who created the official language of Housemarque's headquarters
location?
Which city is the headquarter of Housemarque located in? Helsinki.
What is the official language of Helsinki? Finnish.
Who was Finnish created by? Mikael Agricola William Shakespeare.
Much like the previous conflict between unedited and edited cases, these two edited cases share common subquestion: "What is the official language of Helsinki?" However, such subquestion is ses_ee_____eee______________..__
edited in case :d:1::133 1570 while unedited in case id:1 968, causing unintended contamination.
C.3 ERROR ANALYSIS OF MQUAKE-CF
Table Error statistics of MQUAKE-CU (Zhong et al., 2023) in terms of edited cases contaminating unedited cases $3.1 k-edited means k cases are edited out of the total 9218 cases.
MQUAKE
# of Contaminated
1-edit
100-edit
1000-edit
2000-edit
3000-edit
5000-edit
9218-edit
Cases Subquestions
62
3307
$275
5110
4578
3346
0
62
4525
8751
8989
8326
6364
0
Table 10: Error statistics of MQUAKE-CF (Zhong et al. 2023) in terms edited cases contaminating each others $3.2 k-edited means & cases are edited of the total 9218 cases.
Contaminated
1-edit
100-edit
1000-edit
2000-edit
3000-edit
5000-edit
9218-edit
Cases Subquestions
192
441
732
1397
2873
270
606
1027
1986
4250
C.4
ERROR ANALYSIS OF MQUAKE-CI-3K-v2
MQUAKE-CF-3KK -F-KK33--33 Zhong et al. 2023) is dataset based MQUAKE-CF-3K aimed fix the dataset corruption issues we identified in $
fixed the inner-contamination ($3.2) and conflict edit errors ($3.3). However, as shown below, this dataset still vulnerable intra-contamination as has 17 duplicate cases ($3.5) and 633 cases
illustrated in $3.1 Meanwhile, MQUAKE-CF-33
of missing information in the problem instructions ($3.4..
Table 11: Error statistics of MQUAKE-CF-3K-V (Zhong 2023) in terms of edited cases contaminating unedited cases $3.1 k-edited means & cases are edited out of the total 3000 cases.
MQUAKE-Cr KO-AK-----3--
# of Contaminated
1-edit
100-edit
1000-edit
1000-edit
3000-edit
Cases Subquestions
1643
1591
830
0
2028
2398
1357
18

---

Published as a conference paper at ICLR 2025
D ERROR DETECTION PROCEDURE AND POST-AUDIT CHECKING
In this section, we discuss how exactly we carry out our audit and fixes and how we conduct our post-audit checking to ensure our audited datasets error-free the best of our ability.
D.I INTRA AND INNER CONTAMINATION
As discussed in $31 and $3.2 we observed that some edited facts were retrieved for subquestions that were not intended to involve an edit. We categorized this issue as contamination, where edited facts inadvertently influence the correct reasoning path. To carry out the audit, made the fol- lowing observation: regardless whether a case is edited or unedited, valid reasoning path must always exist from the initial subquestion the last subquestion. Thus, suppose any unedited sub- question on this reasoning path shares the same subject and relation with a triple reflecting an edited fact; then this unedited subquestion is contaminated and therefore flagged.
We programmed the abovementioned filtering mechanism and identified the contaminated edit facts against different subquestions/cases. We then employed the API described in $4.2 to dynamically mask out contaminated cases. Last, we confirmed that there is no contamination remaining by re- executing our filtering program upon the dynamically masked dataset.
D.2 CONFLICTING EDITS
As illustrated in $33 we noticed some edits within the editing knowledge bank are self- contradicting, where edits with the same subject and relation led to different tail entities. Again, follow the intended reasoning path as introduced above and check if there multiple edit- reflecting triples that share the source and relation with an edited subquestion. If so, this suggests there are conflicting edits. We flagged all those edited triples, put ones with shared sources and relations into the same group, then flipped the minority edit to the majority edit and updated their subquestions accordingly. We then the program ensure more flagged triples.
D.3 MULTI-HOP QUESTION INSTRUCTION REWRITE
As highlighted in $3.4 we identified some questions lacked a complete set of relations in their in- structions, thus essentially omitting necessary information for model to provide the correct answer. We collect list of synonyms of all relations of editing path, then evaluate if certain instruction not using any of the corresponding synonyms when its reasoning path indicates it should reflect a certain relation.
Subsequently, we prompted the original meta--lama/LLlam--3 1-405B to regenerate all instructions with the few-shot demonstration prompt demoed in Appendix [22 and reran the detection procedure. This process resulted in only small number of instructions that still didn't meet our predefined rules due to the fact that lists of synonyms per each relation cannot be exhaustive by design. We then manually inspect and, in a few occupations, manually fix those flagged cases.
D.4 DUPLICATE CASES
Upon investigating conflicting edits, we accidentally discovered that there exist cases with identical reasoning paths to each other, as illustrated in $3.5 We simply opt to retain only one of such cases and remove the duplicated rests. We then keep track of a set of reasoning paths from all cases and see if the cardinality of the set is equivalent to the number of cases.
19

---

Published as a conference paper at ICLR 2025
E ERROR TYPE ABLATION STUDY
In this section, we provide ablation studies demonstrating the benefits of addressing errors in the MQUAKE dataset, aligning with the observed error patterns in Tables and [ Using our proposed GWWll and the 1--ma-31--BBIIstteee including Inner Contamination, Intra Contamination, and Missing Information in Multi-hop Ques- tion Instructions. These error types significantly affect the performance of both edited and unedited accuracies. We opt to exclude minor entors, such as duplicate questions and conflict edits, which are automatically addressed across all settings due to their limited prevalence in the original datasets.
model, we evaluate datasets corrected for major error types,
The observed impact is consistent with our analysis: The Inner Contamination fix has the most impact when editing intensity is high (e.g., 3000-edit). Yet, the Intra Contamination fix has the most impact with lower editing intensity (e.g.. 100-edit). The Missing Instruction fix consistently improves performance across all editing intensities.
Table 12: Performance comparison across dataset variants of MQUAKE-CF-3K meta- Total Accuracy
Ilama/Llama-3,, -.1- Results reported as Edited Accuracy, Unedited Ancdiaed
MQUAKE-CF-3K
Type of Errors Fixed
100-edit
1000-edit
3000-edit
Meta- Seta-Llama/Llamaen 1.1---..--8----- 1-B--.-..-BBB-- Instruct --8---ntree
(ubeyy et al. 2024
45.47
42,73
39.57
None
(38, 45.72)
(41.3, 43.45)
(39.57.
46.83
53.3
71.36
Inner Contamination
(70, 46.03)
(33,9443
(71.36, -)
71.17
61.73
39.87
Intra Contamination
(37, 72.35)
(40.9, 72.15)
(39.87.-)
49.33
47.2
45.1
Missing Instruction
(41, 49.62)
(45.6, 48)
(45.1.
76,83
75.03
71.53
All (Our proposed)
(69, 77.1)
(74.6, 75.25)
(71.53,-)
F EXTENDED REMASTERING
F.1 CONTAMINATION FREE SURSET: MQUAKE- SWAKKF--RMMATTRRD-FF-5.5...
While MQUAKE-REMASTEREEN REMASTERED-MASKER with masking operation well support memory- based knowledge editing methods, it will not be compatible with parameter-based methods. This is because, for parameter-based methods, the set of edited facts used for training and evaluation needs to be constant yet consistent with each other at all times; whereas dynamic masking cannot suffice as it essentially adjusting the dataset on the fly during inference time.
To effectively evaluate parameter- based knowledge editing methods, we present MQUAKE- is a dataset extracted from MQUAKE-CF, where all 6,334 cases are edited cases; and they are completely contamination-fin free from each other. This dataset is suitable for LLM editing with parameter-based approaches, as one can make careful splits among the 6,334 cases of SAA.K---RRAAATTE-----55-6-3 to serve as train, validation, and evaluation sets.
REMASTERED-CF-6333
BOOUAKKFRREMAATEEEDD-CEE
Table 13: The number of unique edited facts for varied number of edited cases in MQUAKE- REMASTERED-CF
Number of Edited Cases
100
1000
3000
6000
All (9171)
Number of Unique Edited Facts
150
1171
2991
5137
7252
20

---

Published as a conference paper at ICLR 2025
Experiment coverage comparison among our other works. For brevity better rele- vance, "Method Coverage" only includes open-sourced methods specificallyy designed for multi-hop editing. as adopted single-hop editors are often too weak to deliver usable results. "Separate Metries?" means that both the accuracy of edited cases and unedited cases reported. We consider the inclusion of both metrics paramount, editing is often double-edged sword, causing potential ballucinations under unedited scenarios. Prior work often only tests on former ignores the latter. We did both in our work.
Table 14:
Ref.
Dataset Coverage
Method Coverage
Separate Metrics?
Error Fix?
MQuAKE
CF-3K (1. 100. 1000, all) [1, 100, 500, Jedit
MeLLo
No
No
(Zhong et al.
2023
(emple-MQA
CF-3K (1, 100, all)-edit; T (1, [ll)-
McLLo, PokeMQA
No
No
(Cheng
al.
edit
2024
2024
CF-3K [all]-edit
N/A
PoleMQA
CF-3K (1, 100, all)- T (1, 100,
MeI.Lo, PokeMQA
ct al.
2024
all]-edit
Ours
CF-3K (1, 100, 1000, all) eddi; T [1. 100, 500, edit; CF-9K [1,
MeLLo, ICE, IKE, PokeMQA, GWalk, RAE, DeepEdit
Yes Yes
1000, 3000, 6000, eelit: CF- 6334 (100, 1000, 3000, all) -edit
F.2
PROMPT FOR REWRITING INSTRUCTIONS
Few-shot Prompt
Instruction: Given chain of relations, generate 3 multi-hop questions that comprehen- sively include the semantics of the relations.
Example 1:
Relation Chain:
XXX -> 'The author of is' -> , is citizen of -> ?
Generated Questions:
What is the country of citizenship of the author of XXX?
2. What country the author of XXX citizen of?
3. What is the nationality of the author of XXX?
Example Relation Chain:
XXX -> was developed by" -> The chairperson of is' ->
is citizen of -> , is located in the continent of
Generated Questions:
What continent the country located in, where the chairperson the developer of XXX is citizen?
2. On which continent is the country located, whose citizen is the chairperson of the company that developed XXX?
3. Which continent houses the country of the chairperson of the developer of XXX?
Example 3: Relation Chain:
<The relational chain want the generated questions to be
based on>
21

---

Published as a conference paper at ICLR 2025
G RIPPLEEDIT
We consider MQuAKE's task design and setup to be more reflective of real-world editing tasks, naturally, there will always be more than one edited fact stored for any system with reasonable complexity. That being said, we are happy to report our proposed pilot method, GWalk, performs decently on RippleEdit. Here are some snapshot results on -lama-2-7b-char.
Table 15: Single-edit result of RippleEdit- Pow.ulaar//eeee//aaommm C1/2 means the edit is happen- ing at the 1st or the 2nd hop (RippleEdit cases only have 2 hops).
Method
Popular C1 Acc.
Popular C2 Acc.
Recent C1 Acc.
Recent C2 Acc.
Random C1 Acc.
Random C2 Acc.
ROME
37.4
47.8
50.0
35.5
49.5
ICE
85.1
67.6
74.8
85.0
73.8
80.3
MeL.a
45.1
77.1
50.2
80.0
40.2
68.3
GWalk (ours)
85.7
81.8
80.9
87.6
76.1
82,9
We additionally convert RippleEdit multi-edit setup - ie.. there multiple edited facts within the editing knowledge bank at the same time - to a) make it more challenging and, b) show that our audit also "fix" issues within different dataset. Note put the "fix" in quotes RippleFdit not designed with multi-edit in mind, so the things we fixed are not necessarily errors but just some adjustments required for making a proper multi-edit dataset. In any case, here are the snapshot results on -amma----b-Innttree
Table 16: Multi-edit result of Hippleeadit-Pppula//eeeeee//// conflict edits and 3/120/1 case-to-case contamination within datasets, respectively.
In this case, fixed 21/2/0
Method
Popular
Popular
Acc.
Acc.
Random
Me
35.1
40.3
41.1
42.4
49,5
50.0
GWalk (ours)
79.0
79.2
72.9
60.0
22

---

Published as a conference paper at ICLR 2025
H
CASE STUDY OF GWALK
We believe GWalk is performant and practical because of two ingredients:
It only stores edited facts in its Editing Knowledge Bank (Figure 2 contrary to some baselines (e.g.. RAE (Shi et al. 2024)), where unedited facts are also stored. This is more practical to maintain as there are always fewer edited facts to keep track of, yet the total search space is much smaller, allowing more precise and efficient retrieval.
Unlike most baselines, which store edited facts in natural language (NL) format (e.g., MeLLo (Zhong et al. (2023) and the majority of existing works) and conduct retrieval based NL sentence embeddings, we store such editing facts on Knowledge Graph (KG). The topology-basee retrieval greatly reduces unintended retrieval, which almost always causes hallucinations.
Here a concrete example from MQuAKE-Remasterede (case #16), where McLLo retrieves an incorrect edited fact edited subquestion.
Edited Subquestion Example
Question:
What is the country citizenship of Twitter's CEO?
1st Subquestion: Who is Twitter's CEO? Generated Answer (by LLM):: Twitter's CEO is Elon Musk.
MeLLo-retrieved edited fact: The chief executive officer of CBS Corporation is Steve Jobs.
// Incorrect edited fact retrieved because this edited fact is close the subquestion from embedding standpoint, even if it doesn't I provide relevant information.
GWalk edited fact: The chief executive officer of Twitter Parag Agrawal.
// This is correct retrieval because we first identify (in a lossy fashion) entity twitter and relation executive officer in the KG storing edited facts.
MeLLo 2nd Subquestion: What the country of citizenship Elon Musk?
GWalk 2nd Subquestion: What is the country of citizenship of Parag Agrawal?
MelLo eventually provides the wrong final answer because the rest of subquestion is about Elon Musk, though it should be about Parag Agrawal. We note that this is an editing dataset, so the ground truth answers often don't reflect the situation in the real world.
Similarly, here's Me retrieving unrelated edited fact on unedited subquestion (MQuAKE- Remastered-CT -FF, case #70).
Unedited Subquestion Example
Question:
What is the capital of the country where Premam originated?
1st Subquestion: Where Premam was originated?
Generated Answer (by LLM)) Premam was originated in India. MeLLo-retrieved fact: Carnatic music was created the country Poland.
// Unrelated edited fact reirieved even if this subquestion is not edited.
GWalk-retrieved fact: None.
// No edited fact is retrieved because triple (via lossy mapping) the KG has source of Premam with a relation of originated in.
MeLLo 2nd Subquestion: What the name of the capital city of Poland?
GWalk 2nd Subquestion: What is the capital city of India?
// MeLLo again eventually provides the wrong final answer because the rest of its subques- tion about Poland, though it should be about India.
23

---

Published as a conference paper at ICLR 2025
We also present the quantitative result of the study between MeLLo (Zhong et al. 2023 and our GWalk. The experiment is conducted using Dwen/Qwen2..577B-1 Instruct (Yang et al. 2024a) on
We chose "Wen2.5-7B-Insteee as the evaluation model pr- marily because it is the second-hest model we evaluated after llama3.1-80 1-8b which MeI.Lo fails to make any edits, according to Table 18
Figure 3: This is the color legend, where "Correct Case" means the case answered cor- rectly. "Incorrect Breakdown Path" refers to case having an incorrect subquestion break- down path. "Incorrect Edited Retrieval" means the case has an incorrect fact retrieved on an edited subquestion. "Incorrect Unedited Retrieval" means the method retrieves an ir- relevant fact where there should be no edited facts. "Others" refers to other issues that occurred evaluating this case, primarily knowledge aggregation. A failing case may con- tain multiple types of error, we only count the type that occurs the first.
Correct Case
Incorrect Breakdown Path
Incorrect Edited Retrieval
Incorrect Unedited Retrieval
Others Errors
Figure 4: Comparison of MeLLo and GWalk for Editnum of
12.1%
23.0%
17.8%
0.4%
0.8%
0.0%
75.9%
53.1%
17.0%
McLLo
GWalk
Figure 5: Comparison of MeLLo and GWalk for Editnum 100
14.3%
23.7%
0.9%
14.5%
0.6%
1.0%
0.9%
73.9%
51.8%
18.5%
18.5
MeLLo
GWalk
24

---

Published as a conference paper at ICLR 2025
Correct Case
Incorrect Breakdown Path
Incorrect Edited Retricval
Incorrect Unedited Retrieval
Others Errors
Figure 6: Comparison of MeLLo and GWWlk for Editnum 1000
2.3%
19.6%
7.4%
15.7%
9.8
13.1 %
6.8%
60.9%
50.8%
13.6%
MeLLo
GWalk
Figure 7: Comparison of MeLLo and GWWll for Editnum 3000
9.2%
13.8%
20.3%
18.5%
17.8% 0%
12.2%
12.0%
5.1%
50.5%
40.8%
GWalk
MeLLo
ADDITIONAL EXPERIMENT RESULTS
One observation made in 6. in-context learning-based methods like ICE (Cohen al. 2023) and IKE (Zheng et al. 2023a) tend to "OOM" when facing larger amount of edited facts. This is because these two methods originally designed for single-edit tasks essentially dump all edited facts long concatenated prompt and expect the model to figure out the corresponding editings naturally. They face 0OM issues because when the number of editing facts grows, the prompt becomes extremely long and, therefore, introduces large amount of KV cache and poses significant memory footprint issues.
While efficiently and effectively handling long input is out-of-scope of our work, as general guid- ance, we refer interested readers to efficient long context-handing survey/benchmark works like Yuan al. (2024). which cover the schools and performance of several popular long context- handling methods. Other than the system challenges, another necessary aspect to improve LLM long context performance, most LLMs are pre-trained limited context length and thus cannot effectively handle long input even the system challenge is addressed. In this regard, we again recommend survey/benchmark works like Lu convert this long context scenario to leverage the power of the RAG pipeline, much like the majority of multi-hop knowledge editing methods featured in this work.
(2024) for insights. Further, one can certainly
25

---

Published as a conference paper at ICLR 2025
Table 17: This is the benchmark results of Total Accuracy
The reported format is:
(Edited Accuracy, Unedited UneditelAccnnayyy
SOLAKE-REMASTERER
Method
1-edit
100-edit
500-edit
1864-edit
Imsys/vicuna-7b-e 7
Zheng et
2023b
19.31
18.88
22.16
44.37
MeLLo Zhong al. 2023
(100, 19.27)
(45.0, 17.4)
(40.4, 15.47)
(44.37,
ICE
Cohen et al. 2023
<1
<1
OOM
IKE
Zheng
202.88
<1
<1
OOM
Decp
Wang
2024
<1
<1
<1
35,52
46.51
48.93
54.88
GWalk (Ours)
(100.35:48)
(49.0. 46.37)
(56.0. 46.33)
(54.88. N/A)
Jiang
2023
10.3
10.25
18.78
47.75
McLLo Zhong al.
2023
10.31)
(59.0, 7.48)
(48.4, 7.92)
(47.75, NIA) OOM
ICE
Cohen
<I
<I
IKE
Zheng
<1
<1
OOM
DeepFdil Wang et 2024
<1
<1
<1
34,07
45.76
46.78
50.7 (50.7, N/A)
GWalk (Ours)
(0, 34.08)
(47, 45.69)
(51.2, 45.16)
meta-Ilama/Meta- -aaeaa----aaaa-------nn--- (TT@ Meta
2024
1.13
4.72
16.58
McLLo Zhong al.
2023
<1
(17, <I)
(17.4, <l)
(16.58, N/A)
ICE Coben
<I
OOM
IKE
Zheng
20238
<1
OOM
6.49
8.48
14.74
34.71
DecpEdit Wang et al.
2024
(0, 6.49)
(36.0, 6.92)
(36.20, 6.89)
(34.71, N(A)
70.12
73.28
76,61
84.01
GWalk (Ours)
(100, 7).
(84.0, 72.68)
(87, 72.8)
(84.01, N/A)
Llamm//Laama3,,1 ----Inntruct (Dubey al. 2024
74.68
76.34
77.74
83.32
GWalk (Ours)
74.66)
75.85)
(85.4, 74.91)
(83.32,
Qwen/Qwen2.5-777 -t--struutt Yang et al. 2024a
44.23
46.03
55.1
86,32
GWalk (Ours)
(100, 44.21)
(87, 43.71)
(85.4, 43.99)
(86.32, N/A)
26

---

Published as a conference paper at ICLR 2025
Table 18: This the benchmark result for M KE-REMAATTERRD-CC-S reported in the
Total Accuracy
format of: (Edied Accurccy Unedited UneditedAccmmma Accuricy)
MQUAKE, SQUAKE-REMASTEE REE-RRAATTEED-C--S
Method
1-edit
100-edit
1000-edit
3000-edit
Imsys/vicuna-7b.. 7b-
Zheng et al. 2023b
16.54
18 (9.0, 18.31)
14.63
6.77
MeLLo (Zhon et al.
2023
(100, 16.51)
(8.0, 17.95)
(6.77, N/A)
ICE
Cohen al. 2023
<I
<1
OOM
OOM
IKE
Zheng al. 202.8
<1
OOM <1
OOM
OOM
Deep
Wang
2024
<1
<1
<1 66,33 (66.33, N/A)
54.89
60.9
57.37
GWalk (Ours)
(100, 54.87)
(54, 61.14)
(54.4, 58.85)
(Jang al.
2023)
19.73
18.6
16.33
15.93
MeI.Lo Zhoong et al.
2023
(100, 19.71)
(21, 18.52)
(17.8, 15.6)
(15.93. N/A)
ICE
Cohen
2023
<I
<1
0OM
OOM
4.43
IKE
Zheng et al. 2023a
<1
OOM
OOM
(4,4.49)
DeepEdit Wang at. 2024
<1
<1
<1
<1
56.57
61.93
57.17
51.0
GWalk (Ours)
(100, 56.55)
(47. 62.45)
($1.5, 60.0)
(51.0, N/A)
meta Ilaaalaaaaaaaaaaaaaaaaaaaa 8 Instruct (AT@ Meta
2024
<I
1.03
MeLLo (Zhong al.
2023
<1
(2.0,
(3.0.
(2A3.
ICE Conien al. 2023
<1
<1
0OM
OOM
IKE
Zheng
2023a
<1
<1
0OM
OOM
22.93
17.27
15.03
12.63
DeepEdit Wang et al.,
2024
(0, 22.94)
(11, 17.48)
(15.1, 15.0)
(12.63, N/A)
69.0
76.73
75.47
70.6
GWalk(Ours)
(100, 68.99)
(67. 77.07)
(74.2, 76.1)
(70.6, N/A)
meta- Letta-Llama/I lama-33 --8--rrrruuttttt Dubey et al. 2024
MeLLo Zhong al. 2023
<1
<1
<1
(2.5, N/A)
17.80
15.50
11.87
9.47
DeepEdit Wang et al. 2024
(100, 17.77)
(15, 15.52)
(10, 12.8)
(9.47, N/A)
73.3
76.83
75.03
71.53
GWalk (Ours)
(100, 73.3)
(79.77.)
(74.6, 75.25)
(71.53. N/A)
Qween/Qwen2.5.77-Innstt (Yang al.
2024a
40.63
40
35.23
MeLLo Zhong al.
2023
(100, 40.61)
40.21)
(23.9, 40.9)
(23.1,
36.43
29.77
23.37
5.43
DeepEdit Wang et al. 2024
(100, 36.41)
(5, 30.62)
(6.3,
(5.43, N/A)
65.33
65.27
65.07
66.74
GWalk (Ours)
(100, 65.35)
(65, 65.28)
(68.4, 63.4)
(66.74, N/A)
27

---

Published as a conference paper at ICLR 2025
Table 19: Experiments MQUAKE-REMASTERR -EE---EAAAE-EECCC with numbers of edited cases and methods.
Total Accuracy
Results are reported in the format: (Edited Accurary Unedited Accuracy)
MQUAKE BOOUKE-RREEE KE-REMASTERED--C 3000-edit
Method
1-edit
1000-edit
6000-edit
9171-edit
Imsys/vicuna- 7b. Zheng et al. 2023b
22.55
21.54
17.79
12.62
6.95
MeI.Lo Zhong et al. 2023
(100, 22.54)
(8, 23.2)
(7.43, 22.83)
(7.28, 22.58)
(6.95. N/A)
ICE Cohen al. 2023
<1
0OM
0OM
OOM
OOM
IKE
Zheng al.
20238
<1
0OM
0OM
0OM
OOM
DeepEdn
Wang
2024
<1 56.98
<1
<1
<1
61.89
56.37
54.93
54.15 (54.15, NIA)
GWalk (Ours)
(100, 61.89)
57.07)
57.544
(53.27, 58.06)
mistralai/Mistral- -B-nstructvttt
Diang
2023
19.83
19.08
18.9
18.27
18.09
McLLo Zhong et al.
2023
(<1, 19.84)
(20.6, 18.9)
(19.47, 18.62)
(19.02, 16.87)
N/A)
ICE
Cohen el al. 2023
<I
0OM
OOM
0OM
0OM
IKE
Zheng et al. 2023a
<|
0OM
OOM
OOM
0OM
DeepEdn Wang el 2024
<1
<I
<I
61.42
57.79
56.35
53.73
51.53
GWalk (Ours)
(100, 61.42)
(51.8,
(52.3. 58.32)
(50.93, 59.04)
(51.53, N/A)
meta- Maaaallllm//Maaa--- Llama-----aaa----- SBs-nuttuct (AI Meta, 2024
McLLo Zhong ct al.
2023
<1
<1
<1
<1
<1
ICE Cohen et al. 2023
<1
0OM
OOM
00M
OOM
IKE
Zheng et al. 2023a
<1
0OM
0OM
0OM
OOM
22.16
19,26
21.09
23.04
24.25
DecpEdit Wang
2024
(100, 22.15)
(21.29, 19.01)
(24.48. 19.44)
(23.77. 21.67)
(24.25, NIA)
74.09
73.67
71.62
70.08
GWalk (Ours)
(100, 74.09)
(71.1, 73.18)
(70.9, 73.13)
(70.33, 74.05)
(70.08, N/A)
meta- Llama/Llama-a3 1.B--..--8------r- Dubey et al. 2024
76.27
73.48
72.86
72.03
70.94
GWalk (Ours)
76.27)
73.53)
(71.98 73.29)
(70.96, 74.08)
(70.94,
Qwen/Qwen2.5-7 5--- (Yang al. 2024a
64.4
62.61
63.35
64.93
66.79
GWalk (Ours)
64.41)
(66.6, 62.12)
(66.34, 61.9)
(66.28, 28, 62.4)
(66.7 79, N/A)
28

---

Published as a conference paper at ICLR 2025
Table 20: Additional experiments meta-
(Dubey et al. 2024) and
(eeen/Qwen2.577--nnsttte Yang et al. (0244) on MQUAKE-REMASE REMASTERED-CF-63. Results
Total Accuracy
reported in the format: Edited Accuracy, Accurocy, Unedited Accurccy)
Method
100-edit
1000-edit
3000-edit
6344-edit
Imsys/vicuna- 7b- Zheng al. 2023h
ROME Meng al.
<1
<1
<1
<1
12.75
10.36
9.56
7.24
MEND Mitchell al.
2022
(11.11,11, 13.25)
(7.33, 13.64)
7.2, 11.9)
10.3)
57.55
61.79
59.1
56.62
GWalk (Ours)
(22.22, 64.84, 57.48)
(29.08, 66. 17, 63.23)
(39.3, 63.74, 64.33)
(44.64, 62.11. 68.25)
mistralai/Mistral- mistula//Mistaal---nntttue-- -i-irial------tttet----o Diang
2023
ROME Meng al.
<I
<I
11.84
11.57
8.39
6.82
MEND Mitchell al. 2022
(11111 12.36)
(6.95, 8.7, 12.12)
(3.41,
(2.33, 6.4,
56.25
58.9
56.03
54.43
GWalk (Ours)
(33.33, 57.14, 56.28)
(34.69, 60.57, 60.6)
(42.69, 59.04, 59.85)
57.744 52.38)
Meta-llama/Metta-lllm-- ---struccttttt Meta 2024
ROME Meng al.
<I
<|
<I
13.04
13.3
9.81
7.42
MEND Mitchell et al.
2022
(11.11. 13.47)
(5.33. 8.4, 14.33)
(4.21, 8.63, 11.1)
(5.12, 7.45, 7.3)
67.01
71.89
73.76
74.22
GWalk (Ours)
(33.33, 74.73, 66.92)
(47.45, 80.94, 70.65)
(54.05, 81.6, 71.12)
(61 02, 80.47, 73.02)
meta- meta-Llamm/LLama-3.. 1--1--8--nnsee Instruct (Dubey al. 2024
66.79
73.66
72.09
73.3
GWalk (Ours)
(33.33, 66.66)
(49.47, 73.68, 73.02)
(51.23, 75.1.
73.84. 71.55)
Qwen/Qwen2 2- In-strtt Yang et al. 2024a
60.59
65.42
68.75
70.49
GWalk (Ours)
(33.33, 62, 60.56)
(30.13, 68.6, 63.83)
(43.65, 69.9, 64.99)
(59.12. 70.51, 68.25)
29


Reviews:
[{"review_id": "r4qaSRnDfN", "reviewer": "Reviewer_ND8E", "summary": "This paper introduces MQuAKE-Remastered, an enhanced version of the MQuAKE dataset designed to improve the evaluation of multi-hop knowledge editing methods for large language models (LLMs). The authors identify significant flaws in the original MQuAKE dataset, including data contamination, conflicting edits, missing information in multi-hop question instructions, and duplicated cases. These issues, which affect lots of the data including 33% or 76% of MQUAKE’s questions and ground truth labels, undermine the accuracy of previous evaluations conducted with MQuAKE. The revised dataset, MQuAKE-Remastered, provides a cleaner, more reliable benchmark without sacrificing dataset capacity. Additionally, the authors propose a novel, minimally invasive method for knowledge editing, achieving state-of-the-art performance without exploiting dataset-specific biases. The paper provides re-benchmarks of existing methods on MQuAKE-Remastered, offering a clearer view of each method's true effectiveness and guiding future approaches to knowledge editing evaluation.", "questions": "1.\tHow to identify the problematic cases of the original MQuAKE Benchmark?\n2.\tDo you have a more detailed analysis and case study of your GWalk methods? Could you show some correct cases in your fixed benchmark where GWalk performs well while the baselines fail? What’s the critical part of the GWalk method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed fixed benchmark and new baseline are clear and easy to follow.\n2.\tThe analysis for the prior benchmark and re-benchmarking the prior works are extremely valuable, which prevents the community from going in the wrong direction of research.\n3.\tThe proposed fixing method is useful for building reliable fixed benchmark.\n4.\tThe proposed method is an effective brand new baseline for this task.", "weaknesses": "1.\tThe process of identifying the error cases is unclear. The paper introduces the types of error cases, but not show the process of extracting them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fSErbgGlBD", "reviewer": "Reviewer_3g3q", "summary": "This paper argues that the original MQuAKE dataset contains data contamination, potentially leading to inaccurate evaluation results. The authors audit several contamination categories and remaster the dataset through rewriting and masking.", "questions": "- As extensive masking is applied in MQuAKE-Remastered, did the authors evaluate its effectiveness on methods that do not have memory retrieval?\n\n- What instruction was used to prompt Llama-3.1 for rewriting? Was this rewriting model the original version, or was it specifically fine-tuned?\n\n- (Minor) There are a few typos that need correction, such as the duplicate “only” in line 71 and the misspelled “audix” in line 488.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper identifies potential issues within the original MQuAKE dataset and provides an effective remedy. It re-benchmarks several existing knowledge-editing methods and proposes a data-specialty-free approach (GWalk) to address this issue.", "weaknesses": "However, the heuristic approach using dynamic masking may not be adaptable to all types of knowledge editing methods, particularly those without memory-based editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kboQ3XtZH9", "reviewer": "Reviewer_WNuN", "summary": "MQUAKE has been a primary benchmark for multi-hop knowledge editing, but the authors reveal that up to 33% of MQUAKE’s ground truth labels and 76% of its questions are compromised by clerical or procedural errors. This paper provides a comprehensive audit and correction of these issues, resulting in MQUAKE-REMASTERED, which preserves the dataset’s capacity while enhancing accuracy. Additionally, the authors benchmark various MQUAKE-evaluated editing methods on the revised dataset and note that many approaches overfit the original MQUAKE dataset by exploiting dataset-specific properties. The paper concludes by recommending a minimally invasive approach for faithfully addressing such datasets, demonstrating that it achieves excellent editing results without exploiting dataset idiosyncrasies.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  Knowledge editing is an important topic, and addressing multi-hop knowledge editing is a challenging yet meaningful direction.\n2.  The authors identified and addressed key issues within the MQUAKE dataset, which significantly enhances the benchmark’s credibility.\n3.   The proposed GWalk method achieves substantial performance improvement over some baseline methods.", "weaknesses": "1.  I believe that methods like fine-tune-based, meta-learning-based, and locate-and-edit would not be significantly affected by the issues raised by the authors in Section 3.1, Section 3.2 and Section 3.3, since they typically perform editing and evaluation on individual samples independently.\n2.  GWalk, as a knowledge graph-based search method, is quite common in the Retrieval-Augmented Generation (RAG) domain, which limits its contribution in terms of novelty for knowledge editing.\n3.  Although the GWalk method achieves impressive results, there is insufficient analysis on why it performs well. Specifically, the paper lacks a detailed explanation of the benefits introduced by different components, and the construction and search within the knowledge graph could be computationally expensive.\n 4.  While the dataset proposed by the authors is helpful for research in the editing domain, especially for multi-hop knowledge editing, its benefits seem to be primarily demonstrated through parameter-preserving methods, such as in-context learning and external knowledge integration. The authors are encouraged to explore more parameter-modifying methods, such as fine-tuning, meta-learning-based, and locate-and-edit approaches, to evaluate the overall effectiveness and versatility of the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ccmRnWPxgI", "reviewer": "Reviewer_htKb", "summary": "This paper identifies significant flaws in the widely used MQuAKE dataset for evaluating multi-hop knowledge editing methods. The authors present MQuAKE-REMASTERED, a corrected and enhanced version of the original dataset. They audit and rectify several errors, including intra-contamination, inner-contamination, and conflicts in the multi-hop question instructions. Furthermore, the authors propose a minimally invasive knowledge editing method called GWalk, which achieves state-of-the-art results without exploiting dataset-specific properties.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors thoroughly analyze the errors in the MQuAKE dataset, bringing awareness to issues that could distort performance evaluations. The audit's depth and transparency add value to the field.\n2. MQuAKE-REMASTERED fixes the original dataset's flaws without sacrificing data capacity. This improvement will be a valuable resource for future research on knowledge editing.\n3. The proposed GWalk approach is a well-designed and simple method that effectively handles multi-hop knowledge editing without relying on dataset-specific heuristics. Its performance is impressive and highlights the potential for broader applicability.\n4. The paper provides extensive re-benchmarking of existing methods, showcasing the significance of the dataset corrections and the robustness of GWalk.", "weaknesses": "1. Some methods encounter out-of-memorOOM issues, which restricts broader adoption and testing. The authors could explore efficient data handling or suggest guidelines for reducing memory usage.\n2. The experiments cover a few models. Expanding this to a broader range of LLMs (like 5 LLMs) would provide a more generalized understanding of the benchmark's utility.\n3. The authors may run controlled experiments to quantify the impact of each type of contamination on model performance. This would provide more insights into which errors had the most significant effect and how much improvement is due to each fix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)