Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks (2024)

Abstract: Current knowledge editing approaches struggle to effectively propagate updates to interconnected facts. In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning. To support our analysis, we introduce a novel reasoningbased benchmark, ReCoE (Reasoning-based Counterfactual Editing dataset), which covers six common reasoning schemes in the real world. We conduct an extensive analysis of existing knowledge editing techniques, including input-augmentation, finetuning, and locate-andedit methods. We found that all model editing methods exhibit notably low performance on this dataset, especially within certain reasoning schemes. Our analysis of the chain-ofthought responses from edited models indicate that, while the models effectively update individual facts, they struggle to recall these facts in reasoning tasks. Moreover, locate-and-edit methods severely deteriorate the models’ language modeling capabilities, leading to poor perplexity and logical coherence in their outputs.

heading: 1 Introduction
text: Contemporary language models demonstrate a remarkable capacity to encode extensive factual information, rendering them highly useful as a knowledge base for real-world applications. Yet, the challenge of rapidly outdated knowledge persists, giving rise to a wide range of methods for knowledge updating, such as in-context learning (Vu et al., 2023), continual pretraining (Zhu et al., 2020a), locate-and-edit (Meng et al., 2022a,b), and metalearning (Mitchell et al., 2021).
Despite the success of fact-wise editing, recent studies (Zhong et al., 2023; Onoe et al., 2023; Pinter and Elhadad, 2023) show that current model editing methods struggle to effectively propagate updates to interconnected facts, thereby limiting
the performance of reasoning tasks based on these edited facts. However, the reason for this ineffective knowledge propagation remains largely unexplored. Our observations in fact-editing experiments reveal that models often behave unexpectedly post-editing. For instance, when applying MEMIT (Meng et al., 2022b) for fact editing, the model frequently fails to reliably recall the pertinent edited information and produces an incoherent chain-of-thought (CoT) (Wei et al., 2022) during question answering, as illustrated in the MEMITbased generation example in Figure 1.
In this work, we place special emphasis on analyzing the results of CoT prompting, which provides explicit reasoning steps that facilitate easier examination. We undertake an in-depth analysis of this phenomenon, concentrating on three essential competencies necessary for knowledge propagation on reasoning questions after model editing: (1) effectiveness of editing individual facts, (2) accuracy in recalling relevant facts, and (3) logical coherence of the thought process. To facilitate
12503
our investigation, we introduce a Reasoning-based Counterfactual Editing dataset – ReCoE, which covers six different reasoning schemes: superlative, comparative, sorting, counting, aggregation, and subtraction (Dua et al., 2019; Zhu et al., 2024). This dataset is designed to more accurately capture the complexities inherent in fact editing tasks.
We first explored input-augmentation, an approach where new facts are added (prepended) only in-context, as an approximated upper bound of model editing methods. We then examined model editing methods including finetuning and MEMIT on the Tülu series (Wang et al., 2023b), which are Llama-based instruction-tuned models of varying sizes. Results show that all model editing methods achieve notably low performance on the ReCoE benchmark, especially in certain reasoning schemes, with scores close to zero. Our analysis further unravels the effect of various knowledge editing methods on the reasoning abilities of language models. We found that all editing methods result in a significant reduction in fact recall, indicating a key obstacle in effective utilization of the edited knowledge. Surprisingly, models edited through locate-and-edit methods (i.e., MEMIT) exhibit a severe decline in their generation coherence, leading to nonsensical outputs, which suggests a substantial deterioration in their fundamental language modeling abilities. We summarize our contributions of the paper as follows:
• We introduce a reasoning-based framework of assessing knowledge editing methods, covering key aspects that enables effective reasoning. Our analysis uncovers essential insights regarding the challenges and limitations associated with knowledge propagation.
• We introduce ReCoE, a novel yet challenging reasoning-based counterfactual editing benchmark covering a diverse set of reasoning schemes centered on real-world scenarios.

heading: 2 Related Work

heading: 2.1 Model Editing Methods
text: Existing model editing methods generally fall into four main categories (Wang et al., 2023a).
Finetuning-based methods These techniques further finetune the model on new knowledge while minimizing the change in models and catastrophic forgetting. Examples of finetuning-based methods
include (Zhu et al., 2020a; Chen et al., 2020; Zhu et al., 2020b).
Machine learning framing methods These approaches treat the editing as a machine learning challenge. They learn hypernetworks (optimizers) to process model gradients. The goal is to produce an updated model that offers the desired output for the edited point while ensuring minimal prediction changes for other data points. Notable methods include MEND (Mitchell et al., 2021), KnowledgeEditor (De Cao et al., 2021), SLAG (Hase et al., 2021), and CaMeLS(Hu et al., 2023).
Interpretability-centric methods These methods focus on model interpretability. The objective is to pinpoint specific layers and parameters that primarily function for knowledge storage (Dai et al., 2021). Once identified, these parameters are then edited, viewing them as linear associative memory storage units. ROME and MEMIT (Meng et al., 2022a,b) are prominent examples.
Retrieval-augmented methods These techniques leverage retrieval-augmentation to update knowledge in prompting (Vu et al., 2023). SERAC (Mitchell et al., 2022) and MeLLo (Zhong et al., 2023) store new knowledge in memory. When relevant queries arise, they retrieve the pertinent knowledge from this storage, employing input augmentation to adjust the response.

heading: 2.2 Model Editing Benchmarks
text: Knowledge editing Several benchmarks have been introduced to assess the efficacy of model editing. Meng et al. (2022a) introduced the COUNTERFACT dataset, specifically designed to evaluate the successful incorporation of counterfactual knowledge. This evaluation is segmented into three main criteria: (1) Efficacy determines if a particular piece of knowledge has been successfully integrated into the model (2) Paraphrase assesses the model’s capability to generalize to paraphrased versions of the editing text (3) Specificity ensures that the model remains unchanged with respect to irrelevant knowledge. There are many other datasets including Zero-Shot Relation Extraction (zsRE) (Mitchell et al., 2021), WikiGen (Mitchell et al., 2021), T-REx-100 & T-REx-1000 (Elsahar et al., 2018; Dong et al., 2022), MMEdit (Cheng et al., 2023) (multi-modal model editing), etc.
Knowledge propagation To evaluate knowledge propagation, Onoe et al. (2023) and Zhong et al.
(2023) introduced ECBD and MQuAKE benchmarks respectively. ECBD measures the perplexity of a passage relevant to target knowledge (entityrelevant) before and after editing. Though it presents the difficulty of knowledge propagation, the context it evaluates on has a non-deterministic connection to the edited knowledge. MQuAKE employs multi-hop QA questions to gauge the model’s accuracy after editing part or all of its reasoning components. However, it exclusively encompasses compositional questions generated from ChatGPT wherein the precise segments of knowledge required for effective propagation are overtly articulated within the question. Such format may not necessarily mirror real-world scenarios where the reasoning component could be implicit from the question.
In our research, we focus on factual knowledge editing and tackle the prevailing limitations on propagation observed in contemporary benchmarks. Our benchmark ReCoE incorporates a diverse set of 6 reasoning schemes, featuring more organic queries that mirror real-world scenarios. Additionally, we employ a reasoning-based framework to elucidate the underlying challenges of knowledge propagation. Drawing from our discoveries, we aspire to provide valuable insights that will guide and shape the future trajectories of model editing techniques in a more informed manner.

heading: 3 ReCoE Dataset
text: We employ a hybrid-synthetic approach that combines existing complex QA datasets and LLMassisted data synthesizing to create the ReCoE dataset. The dataset is designed to evaluate counterfactual editing across a broad spectrum of reasoning schemes: superlative, comparative, sorting, counting, aggregation, and subtraction. A typical datapoint encapsulates five key components:
• Q: Question that corresponds to each of our defined reasoning schemes
• A: Answer and aliases
• F: Set of facts that supports the answer (A)
• CA: Counterfactual answer and aliases
• CF: Set of counterfactuals that supports the counterfactual answer (CA)
These components allow us to assess knowledge propagation by editing a language model with the set of counterfactuals CF, and testing if the edited model is able to flip its original answer (A) towards the counterfactual answer (CA) through reasoning.
In this section, we provide a comprehensive overview of the dataset and delve into the nuances of its construction methodology.
QA Pairs Construction Table 1 presents examples of QA pairs for each reasoning scheme and
the corresponding data source. Table 2 presents the dataset statistics including the number of examples and atomic facts of CF for each reasoning scheme. Details on the collections of QA pairs are presented in Appendix A.2.1.
Counterfactual Construction After obtaining all the QA pairs for each reasoning scheme, we need to create facts and counterfactual facts to complete each datapoint. The dataset is constructed entirely automatically with the main construction steps illustrated in Figure 2. A concrete example in ReCoE is presented in Appendix A.1. The construction involves four steps:
Step 1: Counterfactual answer generation. After collecting and generating QA pairs, we prompt Claude to create a counterfactual answer. For instance, regarding a question “which team between the Chicago Blackhawks and Pittsburgh Penguins has won more Stanley Cup championships?" and the answer “Chicago Blackhawks", the counter-
factual answer would be the “Pittsburgh Penguins". Step 2: Relevant facts generation. Claude is prompted to generate relevant facts about entities mentioned in the answer and counterfactual answer. These facts are then verified for accuracy using a retrieval-augmented method by retrieving relevant paragraphs from Wikipedia using Contriever (Izacard et al., 2021a) and corrected if necessary. We filter datapoints to ensure that all QA pairs are valid and (question, counterfactual answer) pairs are invalid.
Step 3: Counterfactual facts generation. To generate counterfactual facts, if a question is superlative, comparative, or sorting (Group 1), we swap the subjects of supporting facts between those related to the actual answer and those related to the counterfactual answer. This process is conducted while eliminating any datapoints that could introduce contradictions in the counterfactual facts generated as a result of this subject swapping; if the question is counting, aggregation, or sorting (Group 2), we alter the facts for the answer to obtain the counterfactual facts for the counterfactual answer while maintaining consistency.
Step 4: Fact decomposition. All sentences in the facts and counterfactual facts are broken down into atomic formats for easier editing.
Since the benchmark is constructed automatically, we conduct human verification on random samples from the benchmark to assess its quality. The goal of the verification is to see whether the created counterfactuals support the counterfactual answer as the new answer to the question. For
each reasoning scheme, we randomly select 50 datapoints three times. The averaged error rates are 99.3% (superlative), 99.3% (comparative), 98% (sorting), 97.33% (counting), 94.00% (aggregation), and 92.67% (subtraction) respectively.
ReCoE: Fact Representation Current benchmarks such as zsRE (Onoe et al., 2023; De Cao et al., 2021; Meng et al., 2022a), COUNTERFACT (Meng et al., 2022a), and MQuAKE (Zhong et al., 2023) primarily contain facts represented in a clear, unambiguous form as a (subject, relation, object) triplet with simple and short subject, relation or object. In contrast, our dataset diverges from this norm, featuring facts more commonly encountered in real-world scenarios, typically represented in an OpenIE style. This style introduces a wider variety of complexities. As illustrated in Figure 3, the atomic facts in ReCoE may involve complex subjects or relations. Moreover, a single relation applied to a subject could correspond to non-unique objects. More detailed comparison between previous benchmarks such as MQuAKE and and ReCoE can be found in Appendix D.1.

heading: 4 Experiment
text: Language Models We utilize the Tülu series (Wang et al., 2023b) as the base language models to assess knowledge editing approaches, which is a good candidate for our study because (1) they are instruction-tuned with well-structured responses to user instructions (2) they include models of varying sizes, enabling us to explore the impact of model scaling on effective knowledge editing.
Knowledge Editing Methods We evaluated the following three representative knowledge editing methods1: Input-augmentation is an inference-
1We do not evaluate meta-learning based methods such as MEND because currently, these methods are not for massive
time editing method that appends the counterfactual facts to the question as part of the prompt. Therefore, it does not modify the model weights, but relies on model’s capability to perform reasoning from explicit context. It is considered as an upper bound (Onoe et al., 2023, 2022) for model editing. Finetuning (QLoRA) performs gradient descent on the new facts to update model parameters. As we are tuning models up to 33 billion parameters, we adopt the parameter-efficient finetuning method QLoRA (Dettmers et al., 2023) for the sake of computational and time efficiency. MEMIT first localizes the factual knowledge in a range of layers in the transformer architecture and then updates the feedforward modules in the layers to insert a massive amount of new facts in the form of triplets. We used the implementation from Zhang et al. (2024).

heading: Experiment Setting
text: Factual knowledge probing We employ both direct prompting and CoT prompting strategies to probe model’s proficiency of factual knowledge mastery and reasoning using the ReCoE dataset. The objective is to ensure that the dataset presents a balanced level of difficulty – neither overly challenging nor too simplistic. This balance is crucial so that the language model under investigation achieves an acceptable level of accuracy for conducting meaningful counterfactual editing experiments, where we observe the model’s transition from correct to counterfactual responses. Moreover, the dataset needs to present a degree of challenge to make it a valuable asset for further research on more advanced language models.
Knowledge Editing We evaluate model’s QA performance on the (question, counterfactual answer) pairs of each reasoning scheme post-editing. We use the correct_flip as the primary metric, which measures the percentage of model’s predictions that correctly transition from the original answer to the counterfactual answer.2
Experiment Result Table 3 displays the outcomes of the knowledge probing exercise. The results clearly demonstrate the significant impact of model scaling and the beneficial role of CoT. Table 4 summarizes the correct_flip results of In-
editing and editing massive knowledge leads to low efficacy. 2correct_flip mainly measures the efficacy of knowledge update, rather than knowledge insert. We focus on the “update" setting in this work to facilitate our later analysis of the changes in reasoning capabilities post-editing.
putAug (input-augmentation), QLoRA-based finetuning, and MEMIT. InputAug involves incorporating counterfactual information into the context, where both model scaling and CoT are shown to be beneficial. Input augmentation is often treated as the upper bound for model editing. But we can see that the performance for aggregation and subtraction is still unsatisfying, below 50%. Both QLoRA and MEMIT editing significantly underperform InputAug across all model scales and prompting strategies, indicating failed knowledge propagation of these methods. Interestingly, QLoRA-based finetuning, despite its deteriorating performance, can still benefit from CoT prompting and model scaling. In contrast, MEMIT consistently failed,
indicating a significant deterioration in the model’s reasoning capability.

heading: 5 Analysis
text: To comprehensively evaluate how certain knowledge editing methodologies impact model’s capability that leads to ineffectiveness in knowledge propagation, our analysis encompasses three key dimensions: fact-wise editing effectiveness, fact recall accuracy, and logical coherence in model’s generation. We assume the reasoning process follows a retrieve-and-generate regime. Formally,
P ′(CA|Q) = P ′(CF|Q)︸ ︷︷ ︸ fact recall · P ′(CA|Q,CF)︸ ︷︷ ︸ coherent generation
0 20 40 60 80 100 PPL(F)
0
20
40
60
80
100
PP L(
CF )
superlative
0 20 40 60 80 100 PPL(F)
comparative
0 20 40 60 80 100 PPL(F)
counting
0 20 40 60 80 100 PPL(F)
sorting
0 20 40 60 80 100 PPL(F)
aggregation
0 20 40 60 80 100 PPL(F)
subtraction Pre-edit (Blue) vs. QLoRA (Red) | 7B
0 20 40 60 80 100 PPL(F)
0
20
40
60
80
100
PP L(
CF )
superlative
0 20 40 60 80 100 PPL(F)
comparative
0 20 40 60 80 100 PPL(F)
counting
0 20 40 60 80 100 PPL(F)
sorting
0 20 40 60 80 100 PPL(F)
aggregation
0 20 40 60 80 100 PPL(F)
subtraction Pre-edit (Blue) vs. MEMIT (Red) | 7B
Figure 4: Comparison on fact-wise perplexity over facts (F) and counterfactuals (CF) before and after editing using QLoRA and MEMIT (7B). A successful fact-wise edit is indicated by a transition from the upper-triangle region where PPL(CF) > PPL(F) to the lower-triangle region. QLoRA-based finetuning demonstrate notable effectiveness, in contrast to MEMIT. Similar trends are observed with 13b and 33b models, as detailed in Appendix B.
where P ′ is the edited LM. The fact recall component requires 1) each fact within CF to be effectively edited; and 2) edited model is able to recall these edited facts through generation. The coherent generation component further requires logically coherent CoT in the generated answer.
Fact-wise Editing Effectiveness This dimension examines the basic efficacy of editing methods. It assesses whether the applied edits achieve their intended modifications successfully, which constitutes the foundational requirement for any knowledge editing approach. Defining PPL(F) and PPL(CF) as the averaged perplexity of the facts and the counterfactuals associated to a (Q,A) pair, and ∆(CF,F) = PPL(CF) - PPL(F)), an effective editing over the facts F to its counterfactual counterpart CF can be evaluated by the indicator function: 1[min(∆pre(CF,F), 0) > ∆post(CF,F)]. This definition of successful editing stipulates that the perplexity of counterfactual sentences must be lower than that of factual sentences. Furthermore, in cases where the perplexity of counterfactual sentences is already lower before editing, it necessitates an even greater disparity between the two.
Fact-wise editing performed by QLoRA demonstrates a high degree of effectiveness, in contrast to MEMIT. MEMIT has the adverse effect of increasing the overall perplexity within the model. Figure 4 demonstrates the fact-wise editing effectiveness in 7b model using QLoRA and MEMIT. Detailed results are presented in Appendix B.

heading: Scheme Model Pre-edit QLoRA MEMIT
text: Fact Recall Assuming successful fact-wise editing, we then explore the model’s proficiency in recalling and applying these modifications in reasoning tasks. This involves an in-depth analysis of the model’s ability to retrieve from stored knowledge and utilize relevant information correctly. We focus on evaluating the relatedness and consistency of information within the CoT response against the counterfactual facts. The evaluation metrics are defined as:
Relatedness: this metric assesses how relevant
the facts p generated by the model are to the designated fact/counterfactual set q.
p is defined to be irrelevant with a given fact q
if the truth value of p is independent of
the truth value of q (1)
Consistency: this metric calculates the proportion of the model’s generated facts p that align factually with the fact/counterfactual set q:
p is defined to be consistent with a given fact q
if q → p ∨ p → q (2)
We leverage Claude with dedicated few-shot demonstrations for automatic evaluation. Detailed prompt can be found in Appendix C. Results are presented in Figure 5. Regarding relatedness, different editing methods show different impacts on the model: the model edited by QLoRA retains the capability to recall information across different schemes. However, MEMIT shows negative impacts in superlative, comparative, and sorting. In some cases, relatedness post-editing surpasses that of pre-editing. This could be attributed to the introduction of new facts into the model that it previously lacked.
However, both QLoRA-edited and MEMITedited models show low consistency results, indicating that they are unable to accurately leverage
the edited knowledge in actual use. It’s important to note that this consistency doesn’t correlate well with the fact-wise editing effectiveness. This disparity may stem from a lack of generalization during the editing process. Essentially, the model seems to simply memorize the newly edited fact, lacking the ability to extend this understanding to different manifestations of the same concept.
Logical Coherence We investigate whether the logical reasoning capacity of the model is negatively impacted. This is gauged by the coherence of the generated CoT response, specifically evaluating whether the inferred evidence and thought process adequately support the final answer. Table 5 presents results that show a discernible, albeit not substantial, decrease in the QLoRA-edited models and a surprisingly significant decline in MEMITedited models. This indicates a substantial loss of fundamental language modeling abilities.

heading: 6 Discussion
text: QLoRA vs. MEMIT For QLoRA, we observe that while it adequately supports fact-wise editing and generally preserves logical coherence, its primary deficiency lies in the retrieval of edited facts. In LLM, the elicitation of knowledge depends heavily on appropriate prompting techniques while our approach involves merely fine-tuning LLMs with atomic fact sentences. Consequently, a potential future direction may involve enriching these atomic
facts with more comprehensive contexts prior to their utilization in fine-tuning, as it should enable the model to accurately recall information in response to a diverse set of prompts.
In contrast, the MEMIT model exhibits a decline in all three assessed abilities: fact-wise editing, fact recall, and coherence. Given that our dataset comprises non-synthetic reasoning questions, which often include complex subjects (e.g., an event) and relations, non-unique objects (Figure 3), the underperformance of MEMIT suggests its current inadequacy in handling real-world factual knowledge. Notably, MEMIT’s most pronounced deficiencies lie in its ability to recall facts and, critically, in maintaining coherence. This observation highlights that the functionalities of edited neurons extend beyond mere fact storage, challenging the assertions made in previous studies (Dai et al., 2021; Meng et al., 2022a,b).
Effect of model scaling The impact of model scaling is a critical factor in both original knowledge probing and the input-augmentation approach, which are shown in Figure 6, echoes with current studies that larger models inherently possess a more extensive knowledge base and demonstrate superior reasoning capabilities.
However, experiments in this research reveal that upon editing new knowledge into these models, the size of the model does not correspond to enhanced performance in several dimensions, as shown in Figure 5 and Table 5. Specifically, larger models do not exhibit (1) increased factual effectiveness, (2) improved ability in retrieving facts during chain-of-thought processes in terms of relatedness and consistency, and (3) more coherent chain-of-thought performance. In summary, during the model editing phase, the size of the model does not inherently confer any advantageous properties. Consequently, we have not detected any notable improvements attributable to model scaling, such as facilitation of the editing process or provision of inherent advantages.

heading: 7 Conclusion
text: In this study, we have developed a novel benchmark, ReCoE, which leverages counterfactual reasoning and is grounded in non-synthetic data for evaluating model editing. Our analysis reveals significant challenges in existing knowledge editing approaches, particularly in their ability to effectively propagate new facts for coherent reasoning.
Through this investigation, we have identified key areas where these methods falter. Our work provides a clear direction for future research in this field, aiming to enhance the efficacy and reliability of knowledge editing in computational models.

heading: Limitations
text: This research presents a focused examination of model editing methods and their implications on knowledge propagation within AI models. Despite the comprehensive nature of our study, certain limitations are inherent to the scope of our investigation. Notably, the research does not extend to exploring the effects of editing across a diverse array of model architectures. This limitation signifies that the findings may not be universally applicable to all forms of AI models, potentially restricting the generalizability of our conclusions. Additionally, our study does not delve into the impacts of editing on models that employ meta-learning strategies.

heading: Ethics
text: This project has no ethics issue as the scope of this project is centered on the evaluation of model editing techniques and the investigation into the challenges associated with the propagation of knowledge following the incorporation of counterfactual information into AI models.

heading: A Appendix

heading: A.1 Example Datapoint
text: The data contract and a specific example from ReCoE is shown in Figure 7.

heading: A.2 Dataset Construction

heading: A.2.1 QA Pairs Construction
text: QA pairs of all superlative data and part of comparative data are from existent QA datasets (Gu et al., 2021; Kwiatkowski et al., 2019; Talmor and Berant, 2018; Jiang et al., 2019). QA pairs of counting data are partially hand-written. Based on these datapoints, we create QA pairs for sorting questions, more comparative questions, aggregation questions and subtraction questions synthetically.
Sorting questions are constructed by prompting Claude based on superlative questions with the following 4 steps. Taking the superlative QA pair in Table 1 as an example: Step 1: Prompt Claude to generate the aspect that the sentence is discussing.
aspect: city population or city size
Step 2: Given the QA pair, generate 10 relevant entities to the entity/subject of the question.
relevant entities: Large cities in Canada: Toronto, Montreal, Calgary, Ottawa Cities in British Columbia: Victoria, Nanaimo, Nelson Coastal cities in North America: Seattle, San Francisco, Los Angeles, Portland
Step 3: Given the question and generated aspect (Step 1), prompt Claude to generate relevant facts about each generated entity.
entity facts: Toronto: a recorded population of 2,794,356 ... Victoria: city size 19.47 km2 (7.52 sq mi) ... Nanaimo: city size 35.25 mi2 ... Seattle: land area of 83.9 square miles (217.3 km2) ... ...
Step 4: Generate 5 sorting questions with the answer using generated entities and their corresponding facts.
example sorting questions with answers: Question: Sort the following cities based on their city size from small to large: Nanaimo, Victoria, Seattle. Answer: Victoria, Nanaimo, Seattle.
Based on the existent superlative sub-dataset, over 10k sorting datapoints can be constructed in the above manner. Since the generated facts and answers based on the facts are not necessarily correct, we leverage multi-agent debate (Du et al., 2023) with Claude agents to double-check the generated QA pairs as initial quality control: QA pairs are
excluded if the agents, post-debate, converge on different answers to the question.
Comparative questions are partially (192) selected from existent datasets. We generate 1,000 more QA pairs with Claude based on generated sorting questions by transforming a sorting question to a comparative question.
Counting questions are from 8 different domains: astronomy, book, geography, legal, movie, music, sport, and war. We manually create 5 QA questions for each domain and then prompt Claude to generate more such QA pairs following the examples. Multi-agent debate is again adopted to filter out inaccurate QA pairs.
Aggregation & Subtraction questions are derived from the counting questions and retains the same 8 domains. An aggregation question is formulated by combining two or more counting questions. Below is a QA pair example:
Question: How many states/provinces are there in North America, i.e. United States, Canada, and Mexico? Answer: 92
To avoid incongruous or unnatural questions, we employ two strategies: (1) Counting questions to be combined are sampled from the same domain. (2) Filter out unnatural questions with Claude by two criteria: a) whether the question is fluent; b) whether the entities mentioned in the question are compatible in type. For instance, the number of satellites of Earth and that of Mars are compatible for aggregation, while the number of constellations recognized by the International Astronomical Union and the number of Earth’s satellites are not.
Our dataset is constructed almost completely automatically. In this subsection, we discuss in detail how the dataset is constructed step by step. This is the running example that we adopt for illustration:
Question: Which team between the Chicago Blackhawks and Pittsburgh Penguins has won more Stanley Cup championships? Answer: Chicago Blackhawks

heading: A.2.2 Datapoint Construction
text: Counterfactual answer generation Given a question and answer, prompt Claude to generate counterfactual answers. Since the example is a choice question, then the counterfactual answer must be “Pittsburgh Penguins”.
Counterfactual Answer: Pittsburgh Penguins
For questions that are not a yes/no question: if the answer is an entity, then the counterfactual answer will be a similar and comparable entity to the answer; if the answer is an ordered sequence of entities or events, the counterfactual answer will be an order with two entities swapped; if the answer is a number, the counterfactual answer will be a close but different number.
Fact generation For each triplet of (question, answer, counterfactual answer), prompt Claude to generate relevant facts mentioned entities in the answer and counterfactual answer. Prompt details can be found in Appendix A.3. If the question is yes/no, generate facts on the two entities being compared. In this example, we prompt Claude on the two teams “Chicago Blackhawks” and “Pittsburgh Penguins” on their number of Stanley Cup winnings. For this example, the mentioned entities are Chicago Blackhawks and Pittsburgh Penguins; their corresponding facts generated are presented below:
Facts Chicago Blackhawks: Chicago Blackhawks has won the Stanley Cup Championship six times, in 1930, 1937, 1961, 2010, 2013 and in 2015. Pittsburgh Penguins: The Penguins have won the Stanley Cup five times (1991, 1992, 2009, 2016, and 2017).
Fact verification Hallucination (Rawte Vipula, 2023) is a severe problem for large language models. Thus, the facts generated from Claude need further verification with truthful and convincing sources. Towards this end, we leverage the retrieval-augmented method to verify each sentence of the generated facts with the following steps: (1) utilize Google Search API to search relevant Wikipedia pages on the question, answer, counterfactual answer, and each sentence of the generated facts (2) chunk content from all the found Wikipedia pages to paragraphs (3) for each sentence, we leverage Contriever (Izacard et al., 2021b) model implemented by Huggingface3 to retrieve the top-5 most relevant paragraphs (4) we prompt Claude using the sentence together with its top-5 most relevant paragraphs to verify whether the sentence is factually correct and if not, modify it based on the retrieved paragraphs. Prompt details can be found in Appendix A.4.
3https://huggingface.co/facebook/contriever
In this example, the generated sentence for Chicago Blackhawks is wrong in the year of 1930, which should be 1933.
Facts Chicago Blackhawks: Chicago Blackhawks has won the Stanley Cup Championship six times, in 1930, 1937, 1961, 2010, 2013 and in 2015. –> Chicago Blackhawks has won the Stanley Cup Championship six times, in 1933, 1937, 1961, 2010, 2013 and in 2015. Pittsburgh Penguins: The Penguins have won the Stanley Cup five times (1991, 1992, 2009, 2016, and 2017): factually correct
For reference, we also provide Wikipedia links for each sentence in the generated facts.
Datapoint filtering To guarantee that the provided answer is correct for the question and the counterfactual answer is indeed “counterfactual”, given the verified facts, we prompt Claude to determine whether the facts support the answer and negate the counterfactual answer and then filter out datapoints with the wrong or outdated answer. This step is necessary as FreebaseQA, GrailQA, ComplexWebQuestions are all based on the Freebase knowledge graph which is outdated. Prompt details can be found in Appendix A.5.
Counterfactual Facts Pittsburgh Penguins: Pittsburgh Penguins has won the Stanley Cup Championship six times, in 1933, 1937, 1961, 2010, 2013 and in 2015. Chicago Blackhawks: Chicago Blackhawks have won the Stanley Cup five times (1991, 1992, 2009, 2016, and 2017): factually correct
Counterfactual Fact Generation To create counterfactual facts, we switch the subjects of the facts. Creating counterfactual facts by swapping subjects of supporting facts can guarantee that the generated counterfactual facts support the counterfactual answer.
Datapoints consistency Notice that the generated counterfactual facts could become contradictory to each other if multiple datapoints happen to involve one same fact to edit. For example, one datapoint requires editing the fact about the Amazon River length to be 4,132 miles:
Question 1: Which river is longer, the Amazon River or the Nile River? Counterfactual Answer: the Amazon River Counterfactual Facts: The Amazon River has length of 4,132 miles. The Nile River has length of 3,977 miles.
Another datapoint requires editing the fact about the Amazon River length to be 3,395 miles:
Question 2: Which river is longer, the Amazon River or the Yellow River? Counterfactual Answer: the Yellow River Counterfactual Facts: The Amazon River has length of 3,395 miles. The Yellow River has length of 3,977 miles.
This would be problematic for massive editing. Thus to avoid editing facts to be contradictory to each other, for datapoints with contradictory counterfactual editing facts, we randomly retain one of them and remove the others to guarantee consistency among each sub-dataset.
Atomic format of facts As multiple editing methods require a (subject, relation, object) format for editing knowledge, in order to facilitate the application of the dataset, we transform each sentence of the facts and counterfactual facts into atomic facts and then atomic triplets. Prompt details can be found in Appendix A.6.
Let’s take the sentence “Pittsburgh Penguins has won the Stanley Cup Championship six times, in 1933, 1937, 1961, 2010, 2013 and in 2015” as an example. The atomic facts are sub-sequences of the sentence. Notice that there is no unique way to break a sentence into atomic facts:
Atomic Facts Atomic fact 1: Pittsburgh Penguins has won the Stanley Cup Championship six times. Atomic fact 2: Pittsburgh Penguins won the Stanley Cup in 1933, 1937, 1961, 2010, 2013, and 2015.
Atomic Triples Atomic triplet 1: (Pittsburgh Penguins, has won the Stanley Cup Championship, six times) Atomic triplet 2: (Pittsburgh Penguins, won the Stanley Cup in, 1933, 1937, 1961, 2010, 2013, and 2015)
Alias generation In evaluation time, we resort to Exact Match on the answer or the counterfactual answer to evaluate models before editing and after editing. Since the model may not generate the exact surface string provided, providing aliases for the answer and the counterfactual answer is necessary to accurately reflect the model capacity as well as the performance of editing methods. Prompt details can be found in Appendix A.7. In this example, the aliases are:
answer alias: Blackhawk Division, Hawks counterfactual answer alias: the Pens, Pens
Summary This appendix section presents the basic dataset statistics and the dataset construction process. In each sub-dataset, the datapoint is an n-ary tuple consisting of (question, answer, counterfactual answer, facts, counterfactual facts, answer alias, counterfactual answer alias), where facts and counterfactual facts are lists of dictionaries with 4 keys: sentence, links, atomic_sentences, atomic_triplets.

heading: A.3 Fact Generation
text: To prompt Claude to generate facts on relevant entities or events, we adopt few shot learning prompting strategy with multiple examples. To save space, we only use 1 example for illustration. See detailed prompt in Figure 8.

heading: A.4 Fact Verification
text: This prompt is used to do factuality verification for generated facts by Claude based on retrieved Wikipedia paragraphs. We adopt few-shot prompting. In the below example, we only present one example in the prompt for demonstration purpose. See detailed prompt in Figure 9.

heading: A.5 Data Filtering with Entailment
text: Verification
This prompt is used to check whether verified facts support the answer to the question and invalidate the counterfactual answer to the question. We also adopt few-shot prompting and here, we present only one example in the prompt for demonstration purposes. See detailed prompt in Figure 10.

heading: A.6 Atomic Facts Generation
text: This prompt is used to break down sentences in the facts and counterfactual facts into atomic facts and atomic triplets. We also adopt few-shot prompting strategry and only show one example for demonstration purpose. See detailed prompt in Figure.11.

heading: A.7 Alias Generation
text: This prompt is used to generate alias for answer and counterfacutal answer. We also adopt few-shot prompting strategry and only show one example for demonstration purpose. See detailed prompt in Figure 12.

heading: B Fact-wise Perplexity
text: Table 6 summarizes the fact-wise editing accuracy of QLoRA and MEMIT, measured using the metric
as described in Section 5.

heading: C Fact Recall Evaluation
text: Figure 13 shows the prompt we used for fact recall evaluation of model generation against the set of counterfactual facts.

heading: D Coherence Evaluation
text: Figure 14 shows the prompt for coherence evaluation of model’s CoT generation, i.e., whether the final answer is supported by the thought process.

heading: Fact Generation Prompt

heading: Human:

heading: D.1 Comparison of Atomic Facts between MQuAKE and ReCoE
text: Here, we present a few more examples to illustrate the distinct characteristics of our benchmark. Takes some atomic statements in MQuAKE as examples:
David Cameron was married to Courtney Love Seattle is the capital of US Carlos Slim is the CEO of Apple Jared Kushner is a citizen of Canada
If we were to apply OpenIE to break down these atomic facts to (subject, relation, object) triples, we would obtain triples like:
(David Cameron, married to, Courtney Love) (Seattle, capital of, US) (Carlos Slim, CEO of, Apple) (Jared Kushner, citizen of, Canada)
However, in our dataset, we also break down sentences into atomic sentences. Below are some examples:
Lunar Atmosphere and Dust Environment Explorer became the first spacecraft to land on the far side of the Moon The Baltimore Orioles won in 1966 World series against the Cincinnati Reds Hamlet’s writing was around 1605-1606 AAA Travel Guide’s rating hotels using stars has been for about 65 years
Breaking them down to (subject, relation, object) form, the content remains significantly
more complex than triples in MQuAKE, reflecting the intricate nature of real-world knowledge:
(Lunar Atmosphere and Dust Environment Explorer; became the first spacecraft to land on the far side of; the Moon) (The Baltimore Orioles; won in 1966 World series against; the Cincinnati Reds) (Hamlet’s writing; was around; 1605-1606) (AAA Travel Guide’s rating hotels using stars; has been for about; 65 years)
Comparing the triples from ReCoE and MQuAKE, we can see that triples from ReCoE represent more complex information, usually with longer subject, open-domain/form relation, or longer object. This complexity more accurately mirrors real-world fact representations and presents a challenge to locate-and-edit methods like MEMIT.

heading: Assistant:

heading: Human:

heading: Human:

heading: Data Filtering with Entailment Verification Prompt

heading: Human:

heading: Atomic Facts Generation Prompt

heading: Alias Generation Prompt

heading: Human:

heading: Fact Recall Evaluation Prompt

heading: CoT Coherence Evaluation Prompt

heading: Scheme Model Pre-edit QLoRA MEMIT

heading: Future Work
text: Our work provides a clear direction for future research in this field, aiming to enhance the efficacy and reliability of knowledge editing in computational models