PokeMQA: Programmable knowledge editing for Multi-hop Question Answering (2024)

Abstract: Multi-hop question answering (MQA) is one of the challenging tasks to evaluate machine’s comprehension and reasoning abilities, where large language models (LLMs) have widely achieved the human-comparable performance. Due to the dynamics of knowledge facts in real world, knowledge editing has been explored to update model with the up-to-date facts while avoiding expensive re-training or fine-tuning. Starting from the edited fact, the updated model needs to provide cascading changes in the chain of MQA. The previous art simply adopts a mix-up prompt to instruct LLMs conducting multiple reasoning tasks sequentially, including question decomposition, answer generation, and conflict checking via comparing with edited facts. However, the coupling of these functionally-diverse reasoning tasks inhibits LLMs’ advantages in comprehending and answering questions while disturbing them with the unskilled task of conflict checking. We thus propose a framework, Programmable knowledge editing for Multihop Question Answering (PokeMQA), to decouple the jobs. Specifically, we prompt LLMs to decompose knowledge-augmented multihop question, while interacting with a detached trainable scope detector to modulate LLMs behavior depending on external conflict signal. The experiments on three LLM backbones and two benchmark datasets validate our superiority in knowledge editing of MQA, outperforming all competitors by a large margin in almost all settings and consistently producing reliable reasoning process. Our code is available at https://github.com/Hengrui-Gu/PokeMQA.

heading: 1 Introduction
text: Multi-hop question answering (MQA) requires a sequence of interacted knowledge facts to reach the final answer. For instance, considering the two-hop question in Figure 1, it is necessary to deduce the
✢ Corresponding author
Q: In which continent is the football club Messi plays for located?
intermediate answer Inter Miami through the fact "Messi plays for Inter Miami", and then deduce the final answer NA through another fact "Inter Miami is located in North America"). MQA poses a great challenge to reasoning abilities of question answering systems (Mavi et al., 2022, Chen et al., 2019, Lan et al., 2021). Thanks to the natural language comprehending and reasoning brought by largescale pre-training, large language models (LLMs) have proven its indispensable utility in MQA tasks (Rao et al., 2022, Khalifa et al., 2023, Xu et al., 2023, Chen et al., 2022).
However, the knowledge within LLMs may be factually wrong or become invalid over time. To ensure the correctness of LLMs without necessitating
8069
expensive retraining (Liu et al., 2023), technique of knowledge editing has been carried out to provide efficient and targeted updates on model behaviors (Sinitsin et al., 2020, Zhu et al., 2020, De Cao et al., 2021 ). There are two popular approaches: parameter-modification based editing and memorybased editing. The former one modifies the internal model weights according to edited facts through meta-learning, fine-tuning, or knowledge locating (Meng et al., 2022a, Mitchell et al., 2021, Meng et al., 2022b). The latter approach leverages an external memory to explicitly store the edited facts (or termed as edits) and reason over them, while leaving LLMs parameters unchanged (Mitchell et al., 2022; Zheng et al., 2023a). Memory-based model editing is generally adopted due to its simpleness and agnostic to backbone LLMs.
In the context of MQA, MeLLo (Zhong et al., 2023) is first proposed by designing a multipurpose prompt to instruct LLMs conducting the reasoning tasks of question decomposition and knowledge editing sequentially. In particular, after decomposing the multi-hop questions, LLMs generate a tentative answer for each subquestion and then detect whether there exists factual conflict between tentative answer and edited facts in memory (e.g., statements of "the current British Prime Minister is Rishi Sunak" and "the current Prime Minister of the UK is Liz Truss" are factually incompatible with each other). By repeatedly prompting LLMs, MeLLo reaches the answer of multi-hop question.
However, the coupling of question decomposition and knowledge editing imposes considerable demands on LLMs to precisely perform reasoning as demonstrations in context. First, the knowledge editing requires LLMs to fully understand the semantics of two candidate facts and then make conflict detection based on the factual compatibility between them. In the few-shot prompting, LLMs are prone to underfit this editing logic due to inadequate supervision signals, especially when embedded in a more complex task (Khot et al., 2022), i.e. question decomposition. Second, within a unified prompt, the incorporation of knowledge editing instruction introduces noise to question decomposition in the similar way. Such superposed noise prevents LLMs from fully focusing on parsing the syntactic structure of multi-hop questions to precisely identify the subquestions.
Thus, we propose Programmable knowledge editing for Multi-hop Question Answering (PokeMQA), where we decouple the two essential
tasks, i.e. question decomposition and knowledge editing, to alleviate burdens on LLMs while introducing auxiliary knowledge prompt to assist question decomposition. Specifically, we offload the conflict detection in knowledge editing with a programmable scope detector, which is used to detect whether a subquestion lies within the scope affected by any edited facts in semantic space (Challenge #1). A two-stage scope detector is designed: In pre-detection stage, we efficiently filter out a substantial number of irrelevant edits; In conflict-disambiguation stage, we perform precise retrieval on the remaining few candidate edits. Our two-stage framework provides both computational efficiency and expressiveness given the high volume of edited facts in real scenarios. The retrieved edits are used to calibrate LLMs behavior. Moreover, we propose a knowledge prompt to augment parse analysis in the process of question decomposition (Challenge #2). The knowledge prompt recognizes key entity from input question and retrieves its external information from a knowledge source to trig the correct decomposition.
Additionally, we observe that the multi-hop question answering process may use the outdated or non-existent facts, but occasionally ends up with the right answer. We refer to this situation as unreliable reasoning (as shown in Figure 1). In order to faithfully evaluate models’ reasoning ability, we propose a new metric called hop-wise answering accuracy (Hop-Acc), measuring the extent how LLMs follow demonstrations, conduct question decomposition step by step, and generate desired answer to each step towards solving the multi-hop question.

heading: 2 Multi-hop Question Answering under Knowledge Editing
text: Notations. Following previous work (Zhong et al., 2023; Meng et al., 2022a), we denote a fact as a triplet (s, r, o), consisting of the subject s, object o, and relation r between them, such as (Messi, play for, Inter Miami). An edited fact (i.e., edit) is the knowledge fact that we want to update and is represented in the same form (s, r, o), such as (Messi, play for, Boca Juniors). We consider a multi-hop question Q, where answering Q requires sequentially querying and retrieving multiple facts. These facts are presented in the order they were queried, forming a chain of facts
⟨(s1, r1, o1) , . . . , (sn, rn, on)⟩, where si+1 = oi and on is the final answer, which uniquely represents an inter-entity path P = ⟨s1, o1, . . . , on⟩. It should be noted that except for s1, all the other entities o1, . . . , on in P do not appear in Q and need to be deduced either explicitly or implicitly through factual reasoning (like Inter Miami and North America in the multi-hop question in Figure 1). If we replace the invalid fact (si, ri, oi) with edit e = (si, ri, o∗i ) in a multi-hop question, due to the cascading effect caused by the edited fact, the chain of facts accordingly changes to ⟨(s1, r1, o1) , . . . , (si, ri, o∗i ) , . . . (s∗n, rn, o∗n)⟩. The updated inter-entity path is P∗ = ⟨s1, o1, . . . , o∗i , . . . , o∗n⟩, which indicates the reasoning path to the final answer of Q has changed after being influenced by edit e. MQA under knowledge editing. Given a set of edits E = {e1, . . . , em} and a language model f to be edited, for a multi-hop question Q, its interentity path becomes P∗ = ⟨s1, o∗1, . . . , o∗n⟩ after being affected by edits in E . The goal of multi-hop question answering under knowledge editing can be formally described as producing an edited language model fedit conditioned on f and E , which can deduce the inter-entity path P∗ and finally output the post-edit answer o∗n to question Q. We denote P∗ as gold path of Q (as shown in Figure 1). Different from the previous work, we not only evaluate whether edited model fedit output the desired final answers, but also check the correctness of their intermediate reasoning paths, providing faithful MQA performance results for knowledge editing. Edit scope. In line with our work, we make some modifications to this concept that was originally proposed by (Mitchell et al., 2022). For an edit e = (s, r, o), we define the single-hop question q describing (s, r) with the answer being o as its atomic question. It should be noted that the atomic question corresponding to a specific edit is not unique but rather a set of semantically equivalent questions (e.g., "What is the country of origin of hockey?" and "Where did hockey originate?"). We refer to the set as the scope of an edit, denoted as S(e). After making an edit e = (s, r, o), the answers to those questions in S(e) should change to o accordingly. Compared with the previous work, we define the edit scope based on the unit of atomic question, excluding the original multi-hop question, which typically has a much more complex syntactic structure. This simplified definition facilitates the programmable scope detector to learn the semantic
patterns represented by S(e) and then make precise edit retrieval to adjust LLMs behavior.

heading: 3 Programmable Editing in Memory of Multi-hop Question Answering

heading: 3.1 Workflow of PokeMQA
text: As illustrated in Figure 2, PokeMQA is a lightweight model editor that can be seamlessly integrated into any backbone LLMs, without changing parameters in the deployed language models. This empowers the language models to be robust to respond to questions based on edited facts. From initiating the editor to successfully addressing a question, the proposed procedure involves two steps as follows: Storing edits in memory. When receiving a set of edits E = {e1, . . . , em}, PokeMQA first uses manually-defined template to convert each edit triplet e into a natural language statement t (as in Zhong et al., 2023), then explicitly stores them in an external memory M = {t1, . . . , tm} for query and retrieval. Inference by checking with edit memory. Considering an input of multi-hop question, we adopt in-context learning (Brown et al., 2020) and provide a few demonstrations (i.e., input-label pairs) as the few-shot prompt to teach models to execute the following three tasks alternately: I) Identify the next subquestion (i.e., atomic question) conditioned on the input question and current inference state in LLMs; II) Detect whether this subquestion falls within the edit scope and generate answer; III) Extract the answer entity for this subquestion in LLMs. Note that this answer entity is either used to decompose the next subquestion at Step I or released as the final answer.
Particularly, we propose the programmable scope detector to detach the knowledge editing task in Step II from LLMs. Previous work (Zhong et al., 2023) generates tentative answers for each subquestion and checks the semantic conflict between tentative answer and retrieved edit in LLMs. With the slight supervision signals from the fewshot prompt, it is challenging for LLMs to compare their semantic patterns and make the correct conflict detection. In this work, the proposed scope detector takes the subquestion as input and detects whether it falls within the scope of any edit in M. If so, the detector sends the factual conflict signal back with a chosen edit statement. The statement serves as a prompt to instruct LLMs to infer the an-
swer from the edit. Otherwise, the factual conflict signal is empty and LLMs directly generate answer based on their internal knowledge.
In addition, we propose knowledge prompt to correct the question decomposition in Step I. In MQA, identifying the leading subquestion (i.e., the first decomposed subquestion) might be challenging due to insufficient contextual information. Specifically, given the input of multi-hop question, one lacks the explicit question entity and entityrelated fact, which are available when identifying the subsequent subquestions. To address this, we innovatively employ the knowledge prompt generator to preprocess the input question. It recognizes the key entity and retrieves relevant documents from an external knowledge base to create a knowledge prompt. Then, we concatenate the input question and knowledge prompt to form an augmented input, effectively resolving the issue.
Owing to the proposed scope detector and knowledge prompt, PokeMQA allows language models to focus on question decomposing and answering, formulating a reliable reasoning path. The details of the proposed components are stated below.

heading: 3.2 Programmable Scope Detector
text: Motivated by (Mitchell et al., 2022), we utilize a programmable scope detector for conflict detec-
tion and design a task-specific training approach to identify effective edit scope patterns.
Architectures. The scope detector can be formally described as g(t, q) : T × Q → [0, 1], which predicts the probability that an atomic question q falls into the scope of the edit statement t (in terms of the edit e). The scope detector can be implemented as arbitrary text classification models (Liu and Guo, 2019, Lu et al., 2020, Zha et al., 2022, Chuang et al., 2023, Zhou et al., 2022). In our framework, considering both expressiveness and computational efficiency, we choose two lightweight, yet complementary models. The two models are denoted as gϕ and gψ, respectively. For an input pair (t, q), gϕ calculates the embeddings for t and q separately and models the log-likelihood by the negative squared Euclidean distance in the embedding space. Model gψ concatenates t and q together as a unified input for the sequence classification task.
In our framework, the models gϕ and gψ serve as pre-detector Mpre and conflict disambiguator Mdis, respectively. We combine them together to establish a two-stage edited fact retrieval framework. The pre-detector filters out the enormous semantically irrelevant edits from memory efficiently, while the conflict disambiguator accurately locates on candidate edit with the highest likelihood. The
details are given in Appendix F. Once the detector finally believes that an input atomic question falls into the scope of any edit in M, it retrieves the edited statement of candidate edit and sends it back along with a factual conflict signal to guide the language model generation process. Training scope detector. According to edit memory M = {t1, . . . , tm}, we build up a training dataset Dtrain = {(t1, q1) , . . . , (tm, qm)}. See Appendix B for more details on the construction of the dataset. To learn the scope covered by each edit statement ti, we use a binary cross-entropy loss with negative sampling (Mikolov et al., 2013) as the training objective:
L = − log g(ti, qi)−Eqn∼Pn(q) [log(1− g(ti, qn))] , (1) where Pn is a negative sampling distribution and we set it to a uniform distribution over each minibatch. Note that Mpre and Mdis are trained separately using the above supervised learning setting. Model selection. In practice, we observed that using the traditional classification metric (accuracy) to validate the detector’s performance can often result in underfitting. We believe this is due to the unique characteristics of the conflict detection task. Thus, we define two novel task-specific metrics to select detector models and guide early stopping during training: Success Rate and Block Rate. The Success Rate measures the accuracy to retrieve the correct edit statement ti for a target question qi from a set of candidates:
SR = 1
N
N∑
i=1
1
  ∧
(t,q)∈Dval (g(ti, qi) ≥ g(t, qi))
  ,
(2) where 1 (·) is the indicator function, N is the size of validation set Dval, and ∧ denotes the AND gate. For the target pair (ti, qi), the retrieval is precise if and only if its detection likelihood is higher than the other pairs (t, qi), which are synthesized by replacing the target edit statement ti with candidates from Dval. On the other hand, metric Block Rate quantifies the extent of detector models to inhibit the unrelated edit statements for a target question:
BR = 1
N
N∑
i=1
1   ∧
(t,q)∈D−val
(g(t, qi) < 0.5)   ,
(3) where D−val = Dval − {(ti, qi)}. Intuitively, a higher value of SR suggests that the scope detector
is able to retrieve the desired edit statements for more atomic questions, while a higher value of BR implies that fewer atomic questions are mistakenly categorized into the edit scope of the irrelevant edits. We use these two metrics to evaluate detectors Mpre and Mdis, and return the optimal-performing detectors on validation set (i.e., having the highest sum of SR and BR). We empirically find that these two metrics performs better serving as the indicator of early stopping (Yao et al., 2007).

heading: 3.3 Knowledge Prompt Generator
text: To identify the leading subquestion during question decomposition, we propose knowledge prompt generator Mgen, which aims to provide the additional valuable contextual information. Specifically, we employ ELQ (Li et al., 2020), a fast end-to-end entity linking model. It recognizes the key entity, i.e., the named entity in the input question Q, links the entity to Wikidata, and subsequently retrieves the related knowledge facts from Wikidata (Vrandečić and Krötzsch, 2014).
The retrieved knowledge facts from the Wikidata are the valuable contextual information for the question decomposition and the knowledge facts are stored as triplets (s, r, o) in Wikidata. We adopt the following strategy to only preserve the commonsense facts from the vast knowledge base. For simplicity, we consider two basic membership properties R = [r1, r2] as our interested relations, where r1=instance of, r2=subclass of. Each entity in Wikidata possesses at least one of the relations. These two relations typically provide infallible commonsense facts related to the entity. Thus, for a key entity si, we randomly choose (si, r1, o1) or (si, r2, o2) as the retrieval fact. After retrieving, we use a manually-defined template to convert both key entity and retrieval fact into a knowledge prompt to augment the input question Q. For instance, as shown in Figure 2, we recognize the key entity Messi and retrieve the knowledge fact (Messi, instance of, human). After composing them together, we finally get the knowledge prompt Entity of Question: Messi, a human.

heading: 4 Experimental Setup
text: We conduct experiments on MQuAKE, a knowledge editing benchmark. It includes MQuAKECF-3K, which is based on counterfactual edits, and MQuAKE-T, which involves temporal knowledge updates. These two datasets consist of several k-
hop questions (k ∈ {2, 3, 4}), each associated with at least one edit. More statistics can be found in Appendix C.

heading: 4.1 Evaluation Metrics
text: Multi-hop accuracy (Zhong et al., 2023). It measures the accuracy of the (edited) language models in answering multi-hop questions. Hop-wise answering accuracy (Hop-Acc). In order to avoid the potential interference caused by unreliable reasoning, we propose the Hop-Acc to checks the correctness of intermediate reasoning path when evaluating MQA performance. Specifically, for a multi-hop question Q, since the question decomposition prompt is completely structured, language models are able to state the intermediate answer of subquestion in a concise, parseable way. Thus, the chain of intermediate answer ⟨s1, o1, . . . , on⟩ can be parsed from the inference content as the deduced path P . We argue that a multi-hop question is fully solved by language models only if the deduced path P is exactly the same as the gold path P∗ (defined in Section 2), i.e., the novel metric measures the accuracy of reasoning path for multi-hop questions, which is only available for sequential question decomposition.

heading: 4.2 Baselines Methods & Language Models
text: We take four knowledge editing methods as baselines, including parameter updating methods, FT (Zhu et al., 2020), ROME (Meng et al., 2022a), MEMIT (Meng et al., 2022b) and memory-based method MeLLo (Zhong et al., 2023). More implementation details are in Appendix E. So far there is still not enough evidence to prove whether chainof-thought (COT) prompting (Wei et al., 2022) or question decomposition (QD) prompting (Press et al., 2022) is more effective, both achieving remarkable performance in various downstream tasks (Jin et al., 2024, Chen et al., 2024). Thus, to ensure fair and comprehensive comparisons, except for memory-based editors (MeLLo, PokeMQA) that depends on question decomposition, we report the performance of other baselines by both COT and QD prompting. These parameter-updating methods execute the entire process by themselves, without the need for repeated prompting.
We conduct experiments on the following three base language models: LLaMa-2-7B (Touvron et al., 2023) is a powerful open-source pre-trained large language model, implemented by Huggingface Transformers library (Wolf et al., 2020);
Vicuna-7B (Chiang et al., 2023) is trained by finetuning LLaMA, implemented by Fastchat library (Zheng et al., 2023b); GPT-3.5-turbo-instruct (Ouyang et al., 2022) is a variant of the most capable GPT-3.5 series model, GPT-3.5-turbo (ChatGPT), which is used for legacy completion.

heading: 4.3 Implementation Details
text: We finetune the pre-detector gϕ and conflict disambiguator gψ based on DistilBERT (Sanh et al., 2019). Note that the Dtrain used for fine-tuning does not contain any edit statements t that appear during testing. We provide detailed fine-tuning setting in Appendix D.
To evaluate performance under varying numbers of edits, we conduct stratified sampling (Parsons, 2014) of the dataset based on the hops of questions to construct edit batches of different sizes. This approach ensures that the proportion of questions with different hops is the same within each edit batch. During inference, we simultaneously inject all edits belonging to the same batch into the to-beedited models.
It should be noted that we conduct experiments about parameter updating methods exclusively on the open-source LLM (LLaMa-2-7B), while the memory-based editing methods are comprehensively evaluated across all language models. (More details about experiments in Appendix H).

heading: 5 Performance Analysis

heading: 5.1 Main Results
text: PokeMQA is effective and reliable. We report our main results in Table 1. The results demonstrate that PokeMQA outperforms all baselines by a large margin in almost all settings. Moreover, PokeMQA achieves the highest Hop-Acc across all settings, which strongly supports our view that the coupling of question decomposition and conflict detection places too much burden on LLMs, thus negatively impacting their inference abilities. PokeMQA significantly addresses the issue of unreliable reasoning, further improving MQA performance under knowledge editing. Meanwhile, achieving a high Hop-Acc indicates that PokeMQA’s reasoning process is more rational and can serve as a more reliable explanation for model predictions, enhancing the interpretability of LLMs in MQA. Furthermore, PokeMQA can scale effectively with current mainstream LLMs, such as GPT-3.5-turbo-instruct, without the need for additional training.
MeLLo is a potential alternative. The results about MeLLo suggest that it is undoubtedly a strong competitor. In the head-to-head comparisons on LLaMa-2-7B, MeLLo achieves (1/10) optimal result and (7/10) sub-optimal results. Surprisingly, MeLLo also achieves the best performance in two settings (In the 1-edit scenario on the MQuAKE-T dataset, the performance is 97.7 using LLaMa-2-7B and 88.12 using GPT-3.5-turboinstruct). But through a detailed analysis of the reasoning processes, we discover that MeLLo solved most multi-hop questions by exploiting shortcut reasoning patterns (See an example in Appendix I), which can be considered as a form of underfitting to the in-context prompt. A clear evidence is that its accuracy on LLaMa-2-7B, which has weaker inference ability, is higher than on GPT3.5-turbo-instruct. Meanwhile, it is undeniable that MeLLo’s performance benefits significantly from the increasing capabilities of LLMs, implying that on a stronger LLM in the future, MeLLo might further narrow the performance gap with PokeMQA.
Parameter-updating may not be the answer to knowledge editing. In line with previous researches (Zhong et al., 2023, Onoe et al., 2023), parameter updating methods fail catastrophically at answering multi-hop questions, indicating that the injected knowledge cannot be flexibly applied to inference by the edited model. FT achieves the best performance among these updating methods
when the size of edit batch is 1, but the impact of a slightly larger edit batch on its performance can already be devastating. ROME performs worse than MEMIT in all settings, which is consistent with the fact that MEMIT is an improved version of ROME. MEMIT displays a certain level of robustness to the size of edit batch, but it still fails under thousands of edited facts. In summary, our results indicate that parameter updating methods can hardly meet the desiderata of knowledge editing applications. MQA under knowledge editing remains challenging. As shown in Figure 3 (Middle, Right), PokeMQA consistently maintains state-of-the-art performance across multi-hop questions of varying difficulty levels while significantly surpassing other competitors in producing reliable reasoning. But depressingly, the increasing difficulty of the questions also has a significant negative impact on PokeMQA’s performance. Combining more facts to generate more complex reasoning processes poses a dual challenge in terms of edited fact retrieval accuracy and language model reasoning capabilities. Currently, PokeMQA is not fully capable of addressing these challenges, suggesting that this task remains challenging for future knowledge editing methods.

heading: 5.2 Ablation Analysis on Model Components
text: We conduct ablation experiments to investigate how the two detachable components Mdis and Mgen
improves PokeMQA and analyze their necessity. The results are shown in Table 2 and Figure 3 (Left). Based on the experimental results, we find that the two components indeed enhance PokeMQA and summarize two conclusive findings as follows:
Use Mgen selectively. As shown in Table 2, Figure 3 (Left), the knowledge prompt generator Mgen improves PokeMQA performance in almost all settings. Although the above results verify its effectiveness, we have to point out that the performance gain is much more significant in MQuAKE-CF3K than MQuAKE-T. Our view is that this is because MQuAKE-T is constructed based on real fact updates in recent years, so the key entity in the input question may be familiar to the latest pretrained LLMs. Consequently, they can recognize the key entity and access entity-related knowledge relatively easily, even without additional contextual information. Due to the extra computation cost of Mgen, we recommend using Mgen selectively depending on the specific application.
Mdis is indispensable for large-scale editing. As shown in Table 2, Figure 3 (Left), Mdis can bring more performance gain when edit batch size is increasing. Meanwhile, we calculate the average number of predictions of Mdis (2.75 in MQuAKECF-3K and 4.57 in MQuAKE-T both on LLaMa2-7B, all-edited). The results indicate that by incorporating the Mdis, a tiny additional number of
predictions greatly boosts MQA performance under massive edited facts. We conclude that Mdis can greatly enhance the robustness of PokeMQA to large-scale editing with almost no additional computational cost, which is indispensable to maintain the applicability of PokeMQA in real scenarios.

heading: 5.3 Decoupling Effect Analysis
text: To better understand the importance of the decoupling operation, we conduct a more fine-grained ablation study to precisely measure its effect. Specifically, we replace the original retriever (contrievermsmarco, Izacard et al., 2021) in MeLLo, which couples the question decomposition with knowledge editing, with pre-detector Mpre and get the new baseline (MeLLo w/ Mpre). The pre-detector Mpre selects the fact statement with highest inscope probability as the retrieved edited fact. This modification ensures that, in the experiments comparing (PokeMQA w/o Mdis,Mgen) and (MeLLo w/ Mpre), coupling remains the sole variable under consideration. The results are shown in Table 3.
It is observed from the experimental results that: I) (PokeMQA w/o Mdis,Mgen) has better results across all settings; II) When the edit batch size is 1, it outperforms (MeLLo w/ Mpre) by a substantial margin; III) The retrieval accuracy of the separate pre-detector Mpre does not show significant differences compared to the contriever, so the performance gain comes from decoupling alone, not from other factors like better retrieval accuracy. Thus, we conclude that decoupling is the key factor contributing to the improved MQA performance.

heading: 5.4 Knowledge Prompt Impact Analysis
text: To elaborate the impact of the knowledge prompt on the editing process more clearly, we calculate the accuracy for the first subquestion of multi-hop questions from MQuAKE-CF-3K both with and without the knowledge prompt (PokeMQA w/o Mgen). The results are shown in Table 4.
Based on the results, it is clear that incorporating knowledge prompts consistently brings an increase in accuracy to the first subquestion across questions with varying hop counts. Thus, we can strongly hypothesis that: I) The impact of knowledge prompts exhibits strong locality, with a significant positive effect on the decomposition and answering of the first subquestion; II) By adding knowledge prompts in context, LLMs can avoid implicit named entity recognition and easily access entity-related facts; III) Although identifying the key entity may not appear challenging for LLMs, it can still pose a distraction to LLMs’ inference ability when embedded in a more complex task, aligning with our insight in Section 3.1.

heading: 6 Related Work
text: Knowledge editing methods. Knowledge editing focuses on updating factual knowledge to language models and a lot of related research has been carried out. Most of these methods predict updates to the weights of the base model by knowledge locating or meta-learning, and then locally modify parameters (Mitchell et al., 2021, Meng et al., 2022b). Other methods preserve parameters and externally store edit instances(Mitchell et al., 2022, Zhong et al., 2023, Shi et al., 2024). Recent work has identified the limitations of existing editing methods through theoretical analysis (Hase et al., 2023) and performance evaluation (Onoe et al., 2023, Cohen et al., 2024). Our work focuses on addressing one of these challenging tasks: MQA under knowledge editing scenarios.

heading: 7 Conclusion
text: In this work, we propose a novel programmable knowledge editing method (PokeMQA) to improve MQA performance and address unreliable reasoning. PokeMQA leverages a scope detector to align LLMs’ behavior with edited facts and incorporates auxiliary knowledge prompt to enrich contextual information. Extensive experiments across three LLMs indicate that PokeMQA helps LLMs answer multi-hop questions in a precise and reliable way.
Limitations
The limitations of our work are as follows:
• In this work, we demonstrate that our proposed two-stage edited fact retrieval framework exhibits precise retrieval capability and a significant degree of applicability in realworld scenarios. Based on the training paradigm, future work may study how to design a task-specific architecture for the scope detector to advance the retrieval performance.
• Our proposed PokeMQA still struggle to solve complex multi-hop questions with more fact hops. Trying to mitigate the context-following pressure imposed by the overlong context length during the question decomposition process might be a promising research direction.
• Besides, although memory-based editing shows great potential for stable and controlled editing and large-scale editing, the way it
stores edit instances makes it extremely vulnerable to attacks such as memory injection. Therefore, memory-based editing needs to be supported by reliable security technology to reduce its risk in real scenarios.

heading: Acknowledgements
text: We thank all reviewers for providing valuable feedback. This work was supported by the National Science and Technology Major Project under Grant No.2023YFF0905400, and the National Natural Science Foundation of China under grants (No.62372211, 62272191), and the International Science and Technology Cooperation Program of Jilin Province (No.20230402076GH, No.20240402067GH), and the Science and Technology Development Program of Jilin Province (No.20220201153GX).

heading: A Prompt & Supplement Results for PokeMQA
text: A demonstration example used in prompt is shown in Table 7. The supplement ablation results are shown in Table 8 and Figure 4.

heading: B Details of Training Dataset Construction
text: To train our scope detector, including pre-detector gϕ and conflict disambiguator gψ, we construct a training dataset D. Specifically, we first extract edit triples from MQuAKE-CF and filter out the part sharing the same (s, r) with fact triples appeared in MQuAKE-CF-3K and MQuAKE-T, constructing a edit dataset De = {e1, . . . , en}. Then we use manually-defined template to convert each edit triple e into a natural language statement s and get a edit statement dataset Dstate = {t1, . . . , tn}. Finally we design a prompt consisting of instruction and demonstrations (shown in Table 9) and prompt Vicuna-13B (Chiang et al., 2023) to generate three diversely phrased atomic questions for each t ∈ Dstate, building a training dataset D = {(t1, q1) , . . . , (tn, qn)}. It should be noted that when computing SR and BR, or sampling negative instances, instances with the same statement t should not be considered.

heading: C Multi-hop Question Answering Dataset Statistics
text: Table 5 contains the statistics for the two benchmark datasets used in our experiments.
Besides, there are 2786 different edits for MQuAKE-CF-3k and 96 different edits for MQuAKE-T when the size of edit batch is all.

heading: D Details about Scope Detector finetuning
text: We finetune the pre-detector gϕ and conflict disambiguator gψ based on DistilBERT (Sanh et al., 2019) and the checkpoint is distilbert-base-cased from Huggingface Transformers library (Wolf et al., 2020). We take SR+BR− 1 as the indicator of early stopping.
To fine-tune pre-detector gϕ, the learning rate is set as 1e−5 with Adam optimizer (Kingma and Ba, 2014), the batch size is set as 1024, and the number of negative samples is 20; To fine-tune conflict disambiguator gψ, the learning rate is set as 1e−5 with Adam optimizer, the batch size is set as 256 and the number of negative samples is 1. And the dataset split is 80%/20% for training and validation, without the need of testing.
E Implementation Details of Baselines
In our experiments, the parameter updating knowledge editing methods, including FT, ROME and MEMIT is implemented by EasyEdit library (Wang et al., 2023). We basically follow the default hyperparameter settings on LLaMa-2-7B in library, and make a slight adjustment to ensure the effectiveness of these methods in different experimental settings. We modify the learning rate for ROME and target editing layer for MEMIT, the detailed modifications is shown in Table 6.
F Two-stage edited fact Retrieval
Algorithm 1 Two-stage Edited Fact Retrieval. Input: edited memory M, pre-detector gϕ, con-
flict disambiguator gψ, atomic question q 1: Initialize candidate set Z = ∅ 2: Initialize final set F = ∅ 3: /* Pre-detection stage */ 4: for all ti ∈ M do 5: if gϕ(ti, q) ≥ 0.5 then Z = Z
⋃ {ti} 6: end for 7: if |C| = 1 then 8: return ti∗ , where ti∗ ∈ Z 9: end if
10: /* Conflict-disambiguation stage */ 11: for all ti ∈ C do 12: if gψ(ti, q) ≥ 0.5 then F = F
⋃ {ti} 13: end for 14: if F ̸= ∅ then 15: return ti∗ , where i∗ = argmaxi gψ(ti, q), ti ∈ F 16: end if

heading: G Licensing
text: Vicuna-7B (v1.1) and distilbert-base-cased are released under the Apache License 2.0. LLaMa-27B is licensed under the LLAMA 2 Community License. ELQ, ROME, MEMIT, FT are released under the MIT license.

heading: H Details about Experiments
text: Because MQuAKE regard a chain of facts as an instance and there are three diversely phrased multihop questions Q for each instance, we follow (Zhong et al., 2023), if any of the three questions is considered solved in terms of the specific metric, the instance is considered correct.
The experiments, data, language models in the paper are all in English. We run all experiments on a machine with four NVIDIA A40 GPU. One run of our experiments takes about 15 GPU hours. For all experiments, we use greedy decoding strategy to get the output in text space of language models for reproducibility and report a single run result due to the limited computational resources.

heading: I Shortcut Reasoning
text: Given an example from Table 10, although MeLLo appears to successfully combine two facts to arrive at the final answer, its reasoning process does not adhere to the task logic demonstrated in the prompt,
which can be regarded as an underfitting to question decomposition.
demonstrations. This color indicate that this part is constructed after being retrieved by knowledge prompt generator from external knowledge base. This color indicate that this part is retrieved by scope detector from external memory.

heading: Future Work
text: Based on the training paradigm, future work may study how to design a task-specific architecture for the scope detector to advance the retrieval performance. • Our proposed PokeMQA still struggle to solve complex multi-hop questions with more fact hops. Trying to mitigate the context-following pressure imposed by the overlong context length during the question decomposition process might be a promising research direction. • Besides, although memory-based editing shows great potential for stable and controlled editing and large-scale editing, the way it
stores edit instances makes it extremely vulnerable to attacks such as memory injection. Therefore, memory-based editing needs to be supported by reliable security technology to reduce its risk in real scenarios