MQuAKE-Remastered: Multi-Hop Knowledge Editing Can Only Be Advanced with Reliable Evaluations

Authors: Shaochen Zhong, Yifan Lu, Lize Shao, Bhargav Bhushanam, Xiaocong Du, Yixin Wan, Yucheng Shi, Daochen Zha, Yiwei Wang, Ninghao Liu, Kaixiong Zhou, Shuai Xu, Kai-Wei Chang, Louis Feng, Vipin Chaudhary, Xia Hu
Primary area: datasets and benchmarks
Keywords: knowledge edit, model edit, multi-hop, question answering, natural language processing, dataset audit

Abstract:
Large language models (LLMs) can give out erroneous answers to factually rooted questions either as a result of undesired training outcomes or simply because the world has moved on after a certain knowledge cutoff date. Under such scenarios, *knowledge editing* often comes to the rescue by delivering efficient patches for such erroneous answers without significantly altering the rest, where many editing methods have seen reasonable success when the editing targets are simple and direct (e.g., *``what club does Lionel Messi currently play for?''*).

However, knowledge fragments like this are often deeply intertwined in the real world, making effectively propagating the editing effect to non-directly related questions a practical challenge (to entertain an extreme example: [*"What car did the wife of the owner of the club that Messi currently plays for used to get to school in the 80s?"*](youtube.com/watch?v=DbwiHC1Fu-E\&t=132s)). Prior arts have coined this task as *multi-hop knowledge editing* with the most popular dataset being MQuAKE, serving as the sole evaluation benchmark for many later proposed editing methods due to the expensive nature of constructing knowledge editing datasets at scale. 

In this work, we reveal that **up to 33\% or 76\% of \mquake{}'s questions and ground truth labels are, in fact, corrupted in various fashions due to some unintentional clerical or procedural oversights**. Our work provides a detailed audit of MQuAKE's error pattern and a comprehensive fix without sacrificing its dataset capacity. Additionally, we benchmarked almost all proposed MQuAKE-evaluated editing methods on our post-fix dataset, **MQuAKE-Remastered**. We observe that many methods try to overfit the original MQuAKE by exploiting some dataset idiosyncrasies of MQuAKE. We provide a guideline on how to approach such datasets faithfully and show that a simple, minimally invasive approach — **GWalk** — can offer beyond SOTA editing performance without such exploitation. The MQuAKE-Remastered datasets and utilities are available at [huggingface.co/datasets/henryzhongsc/MQuAKE-Remastered](https://huggingface.co/datasets/henryzhongsc/MQuAKE-Remastered) and [github.com/henryzhongsc/MQuAKE-Remastered](https://github.com/henryzhongsc/MQuAKE-Remastered), respectively.

Decision: accept (spotlight)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730628043979

thread_type:
official_review

<thread_object>
creator: reviewer_htkb

<content>
summary:
This paper identifies significant flaws in the widely used MQuAKE dataset for evaluating multi-hop knowledge editing methods. The authors present MQuAKE-REMASTERED, a corrected and enhanced version of the original dataset. They audit and rectify several errors, including intra-contamination, inner-contamination, and conflicts in the multi-hop question instructions. Furthermore, the authors propose a minimally invasive knowledge editing method called GWalk, which achieves state-of-the-art results without exploiting dataset-specific properties.

strengths:
1. The authors thoroughly analyze the errors in the MQuAKE dataset, bringing awareness to issues that could distort performance evaluations. The audit's depth and transparency add value to the field.
2. MQuAKE-REMASTERED fixes the original dataset's flaws without sacrificing data capacity. This improvement will be a valuable resource for future research on knowledge editing.
3. The proposed GWalk approach is a well-designed and simple method that effectively handles multi-hop knowledge editing without relying on dataset-specific heuristics. Its performance is impressive and highlights the potential for broader applicability.
4. The paper provides extensive re-benchmarking of existing methods, showcasing the significance of the dataset corrections and the robustness of GWalk.

weaknesses:
1. Some methods encounter out-of-memorOOM issues, which restricts broader adoption and testing. The authors could explore efficient data handling or suggest guidelines for reducing memory usage.
2. The experiments cover a few models. Expanding this to a broader range of LLMs (like 5 LLMs) would provide a more generalized understanding of the benchmark's utility.
3. The authors may run controlled experiments to quantify the impact of each type of contamination on model performance. This would provide more insights into which errors had the most significant effect and how much improvement is due to each fix.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686601602

thread_type:
official_review

<thread_object>
creator: reviewer_wnun

<content>
summary:
MQUAKE has been a primary benchmark for multi-hop knowledge editing, but the authors reveal that up to 33% of MQUAKE’s ground truth labels and 76% of its questions are compromised by clerical or procedural errors. This paper provides a comprehensive audit and correction of these issues, resulting in MQUAKE-REMASTERED, which preserves the dataset’s capacity while enhancing accuracy. Additionally, the authors benchmark various MQUAKE-evaluated editing methods on the revised dataset and note that many approaches overfit the original MQUAKE dataset by exploiting dataset-specific properties. The paper concludes by recommending a minimally invasive approach for faithfully addressing such datasets, demonstrating that it achieves excellent editing results without exploiting dataset idiosyncrasies.

strengths:
1.  Knowledge editing is an important topic, and addressing multi-hop knowledge editing is a challenging yet meaningful direction.
2.  The authors identified and addressed key issues within the MQUAKE dataset, which significantly enhances the benchmark’s credibility.
3.   The proposed GWalk method achieves substantial performance improvement over some baseline methods.

weaknesses:
1.  I believe that methods like fine-tune-based, meta-learning-based, and locate-and-edit would not be significantly affected by the issues raised by the authors in Section 3.1, Section 3.2 and Section 3.3, since they typically perform editing and evaluation on individual samples independently.
2.  GWalk, as a knowledge graph-based search method, is quite common in the Retrieval-Augmented Generation (RAG) domain, which limits its contribution in terms of novelty for knowledge editing.
3.  Although the GWalk method achieves impressive results, there is insufficient analysis on why it performs well. Specifically, the paper lacks a detailed explanation of the benefits introduced by different components, and the construction and search within the knowledge graph could be computationally expensive.
 4.  While the dataset proposed by the authors is helpful for research in the editing domain, especially for multi-hop knowledge editing, its benefits seem to be primarily demonstrated through parameter-preserving methods, such as in-context learning and external knowledge integration. The authors are encouraged to explore more parameter-modifying methods, such as fine-tuning, meta-learning-based, and locate-and-edit approaches, to evaluate the overall effectiveness and versatility of the dataset.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708737915

thread_type:
official_review

<thread_object>
creator: reviewer_3g3q

<content>
summary:
This paper argues that the original MQuAKE dataset contains data contamination, potentially leading to inaccurate evaluation results. The authors audit several contamination categories and remaster the dataset through rewriting and masking.

strengths:
The paper identifies potential issues within the original MQuAKE dataset and provides an effective remedy. It re-benchmarks several existing knowledge-editing methods and proposes a data-specialty-free approach (GWalk) to address this issue.

weaknesses:
However, the heuristic approach using dynamic masking may not be adaptable to all types of knowledge editing methods, particularly those without memory-based editing.

questions:
- As extensive masking is applied in MQuAKE-Remastered, did the authors evaluate its effectiveness on methods that do not have memory retrieval?

- What instruction was used to prompt Llama-3.1 for rewriting? Was this rewriting model the original version, or was it specifically fine-tuned?

- (Minor) There are a few typos that need correction, such as the duplicate “only” in line 71 and the misspelled “audix” in line 488.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731131783823

thread_type:
official_review

<thread_object>
creator: reviewer_nd8e

<content>
summary:
This paper introduces MQuAKE-Remastered, an enhanced version of the MQuAKE dataset designed to improve the evaluation of multi-hop knowledge editing methods for large language models (LLMs). The authors identify significant flaws in the original MQuAKE dataset, including data contamination, conflicting edits, missing information in multi-hop question instructions, and duplicated cases. These issues, which affect lots of the data including 33% or 76% of MQUAKE’s questions and ground truth labels, undermine the accuracy of previous evaluations conducted with MQuAKE. The revised dataset, MQuAKE-Remastered, provides a cleaner, more reliable benchmark without sacrificing dataset capacity. Additionally, the authors propose a novel, minimally invasive method for knowledge editing, achieving state-of-the-art performance without exploiting dataset-specific biases. The paper provides re-benchmarks of existing methods on MQuAKE-Remastered, offering a clearer view of each method's true effectiveness and guiding future approaches to knowledge editing evaluation.

strengths:
1.	The proposed fixed benchmark and new baseline are clear and easy to follow.
2.	The analysis for the prior benchmark and re-benchmarking the prior works are extremely valuable, which prevents the community from going in the wrong direction of research.
3.	The proposed fixing method is useful for building reliable fixed benchmark.
4.	The proposed method is an effective brand new baseline for this task.

weaknesses:
1.	The process of identifying the error cases is unclear. The paper introduces the types of error cases, but not show the process of extracting them.

questions:
1.	How to identify the problematic cases of the original MQuAKE Benchmark?
2.	Do you have a more detailed analysis and case study of your GWalk methods? Could you show some correct cases in your fixed benchmark where GWalk performs well while the baselines fail? What’s the critical part of the GWalk method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311798293

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Reviewers' Feedback and Our Rebuttals

comment:
We are grateful to receive an all-positive review of `6666`. Historically, this rating has had a relatively decent chance of acceptance; that said, we authors are true believers in the philosophy that "ratings are not everything" and we genuinely appreciate the discretionary attention from the AC — for better or worse — therefore, we provide a summary of our work, the feedback we received, and our rebuttals for the convenience of the AC and the general public.

---

## **TL;DR of Our Work**

Our work provides a thorough audit of the original MQuAKE datasets [1] — a popular and ONLY available set of reflective multi-hop knowledge editing datasets, often serving as the SOLE evaluation benchmark for many proposed methods. Through our audit, we revealed that **up to 33% to 76% of MQuAKE’s questions and ground truth labels are corrupted**.

To address these issues, we introduced a fix called *MQuAKE-Remastered*, which corrects the errors across all MQuAKE datasets without sacrificing dataset capacity. Furthermore, we re-benchmarked almost all open-sourced multi-hop knowledge editing methods specifically evaluated on MQuAKE using our Remastered dataset. Our findings revealed that many strong existing methods exploit dataset idiosyncrasies unique to MQuAKE. In response, we proposed a simple yet highly effective solution, *GWalk*, which achieves strong performance without relying on non-generalizable operations.

So in short, our contributions are four-fold: we **audited** MQuAKE, we **fixed** its errors, we **re-benchmarks** almost all available methods, and we offered **a simple baseline solution that is way beyond SOTA** (up to 53.84% improvement) while utilizing a faithful approach.

---

## **Reviewers' Recognition**

Given the nature of our submission, we believe much of the impact of our work lies in the **following three criteria**:

1. Whether the topic (errors in MQuAKE) we are addressing is important and well-motivated.
2. Whether our solution (auditing MQuAKE and fixing it to Remastered) is proper and thorough.
3. Whether our re-benchmark is contributive and comprehensive.

We are glad to report reviewers find our work to score the trifecta of the above merits, as:


* **Reviewers find the topic we are addressing to be important and well-motivated.**
    * `ND8E` *"The analysis... prevents the community from going in the wrong direction of research."* *"The proposed ... is useful for building reliable fixed benchmark.*"
    * `WNuN` *"Knowledge editing is an important topic, and addressing multi-hop knowledge editing is a challenging yet meaningful direction."*
    * `htKb` *"This improvement will be a valuable resource for future research on knowledge editing."*


* **Reviewers find our audits and fixes to MQuAKE to be effective and comprehensive.**
    *  `ND8E` *"The revised dataset ... provides a cleaner, more reliable benchmark without sacrificing dataset capacity."*
    *  `3g3q` *"The paper identifies potential issues ... and provides an effective remedy."* 
    * `WNuN` *"This paper provides a comprehensive audit and correction of ..."* *"The authors identified and addressed ... which significantly enhances the benchmark’s credibility."*
    * `htKb` *"The authors thoroughly analyze the errors in the MQuAKE dataset, bringing awareness ... The audit's depth and transparency add value to the field."*



* **Reviewers appreciate our re-benchmark of existing methods on the error-fixed datasets.**
    * `ND8E` *"The paper provides re-benchmarks of ... offering a clearer view of each method's true effectiveness and guiding future approaches to knowledge editing evaluation."*
    *  `WNuN` *"the authors benchmark various MQUAKE-evaluated editing methods on the revised dataset and note that many approaches overfit the original MQUAKE dataset by exploiting dataset-specific properties."*
    *  `htKb` *"The paper provides extensive re-benchmarking of existing methods, showcasing the significance of the dataset corrections and the robustness of GWalk."*


* **Additionally, many reviewers also appreciate our pilot baseline method — GWalk — for being performant yet without leverging dataset idiosyncrasies.**
    * `ND8E` *"the authors propose a novel, minimally invasive method for knowledge editing, achieving state-of-the-art performance without exploiting dataset-specific biases"* *"The proposed method is an effective brand new baseline for this task."*
    *  `WNuN` *"The proposed GWalk method achieves substantial performance improvement over some baseline methods."*  *"...demonstrating that it achieves excellent editing results without exploiting dataset idiosyncrasies."*
    *  `htKb` *"The proposed GWalk approach is a well-designed and simple method that effectively handles multi-hop knowledge editing without relying on dataset-specific heuristics. Its performance is impressive and highlights..."*


**We appreciate the recognition and couldn't ask for more for this type of work.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
Large Language Models (LLMs) can produce factually incorrect answers due to outdated knowledge or undesirable training outcomes. Knowledge editing techniques aim to patch these inaccuracies efficiently without disrupting other model behaviors. However, real-world knowledge is deeply intertwined, making the propagation of edits across related facts (multi-hop knowledge editing, or MHKE) a complex challenge.  

MQuAKE, the most popular dataset for evaluating MHKE methods, has served as a key benchmark. Despite its importance, the authors discovered significant flaws in MQuAKE, compromising its reliability.  

Key Contributions  

1. Audit of MQuAKE Dataset  
   - Found that 33% to 76% of MQuAKE’s questions and labels were corrupted due to clerical and procedural errors.  
   - Categorized errors in detail, including mislabeling and ambiguous questions.  

2. MQuAKE-Remastered  
   - Delivered a comprehensive fix, correcting errors without reducing dataset capacity.  
   - Introduced a new version of the dataset, named MQuAKE-Remastered, ensuring better reliability for future evaluations.  

3. Re-Benchmarking Knowledge Editing Methods  
   - Reassessed all major knowledge editing methods previously evaluated on MQuAKE using the remastered dataset.  
   - Exposed that many methods overfit idiosyncrasies specific to the original MQuAKE dataset.  

4. Introduction of GWalk  
   - Proposed a simple yet highly effective baseline method, GWalk, for MHKE.  
   - GWalk achieved up to 53.84% performance improvement over existing methods by avoiding reliance on dataset-specific properties.  

Findings  

- The flawed nature of the original MQuAKE dataset misrepresented the performance of many knowledge editing methods.  
- Dynamic masking techniques used in GWalk enabled more generalizable editing.  
- A thorough error analysis and correction significantly improved dataset utility and credibility.  

Technical Insights  

1. Audit and Fixes  
   - Errors included logical inconsistencies, vague questions, and misaligned answers.  
   - Auditing involved manual and automated validation to ensure corrections upheld dataset integrity.  

2. GWalk Method  
   - Minimal invasiveness ensured compatibility across diverse editing scenarios.  
   - Avoided exploitation of unique dataset quirks, unlike previous methods.  

3. Broader Evaluation  
   - Benchmarked methods on five LLMs to generalize findings.  
   - Compared methods like ROME and MEND under various evaluation settings.  


Conclusion  

The paper delivers a foundational improvement to MHKE research by auditing, fixing, and remastering the MQuAKE dataset, creating a reliable benchmark for evaluating knowledge editing methods. GWalk, as a baseline solution, exemplifies how minimally invasive methods can achieve state-of-the-art performance.  

This work establishes a robust foundation for advancing multi-hop knowledge editing and highlights the importance of dataset integrity in driving meaningful research.