Multi-hop Question Answering under Temporal Knowledge Editing

Authors: keyuan  cheng
gang  lin
haoyang  fei
yuxuan  zhai
lu  yu
muhammad asif ali
lijie  hu
di  wang
evan  williams
tom  elon
musk  google
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 12:34:50

Abstract: Multi-hop question answering (MQA) under knowledge editing (KE) has garnered significant attention in the era of large language models. However, existing models for MQA under KE exhibit poor performance when dealing with questions containing explicit temporal contexts. To address this limitation, we propose a novel framework, namely TEMPoral knowLEdge augmented Multi-hop Question Answering (TEMPLE-MQA). Unlike previous methods, TEMPLE-MQA first constructs a time-aware graph (TAG) to store edit knowledge in a structured manner. Then, through our proposed inference path, structural retrieval, and joint reasoning stages, TEMPLE-MQA effectively discerns temporal contexts within the question query. Experiments on benchmark datasets demonstrate that TEMPLE-MQA significantly outperforms baseline models. Additionally, we contribute a new dataset, namely TKEMQA, which serves as the inaugural benchmark tailored specifically for MQA with temporal scopes.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have garnered widespread attention owing to their remarkable capacity for knowledge comprehension, enabling tailored solutions across various applications (Zhao et al., 2023; Huang & Chang, 2022) . However, the presence of outdated knowledge presents a significant challenge, impeding the ability of LLMs to provide accurate responses regarding recent events and facts -a phenomenon known as hallucinations, wherein LLMs tend to fabricate plausible yet incorrect responses about unknown facts (Hong et al., 2023) . Thus, ensuring the timely updating of LLMs with the latest information is of paramount importance. However, as the most direct approach for updating information, editing LLMs by re-training from scratch is practically infeasible, as it requires huge computational resources and substantial investments. In response, Knowledge Editing (KE) has emerged as a focal point, which aims to precisely modify or update the knowledge in LLMs without requiring model retraining. This topic has garnered considerable attention in recent years (Wang et al., 2023b; Zhang et al., 2024) .Multi-hop question answering (MQA), on the other hand, aims to tackle complex inquiries that necessitate multiple reasoning steps. In the context of MQA, KE introduces the concept of "ripple effects", wherein a single edit may trigger a cascade of subsequent knowledge edits or updates (Cohen et al., 2023) . For instance, if we update the knowledge about the U.S. president from Trump to Biden, correspondingly knowledge for the question: "Who is the wife of the U.S. president?" should also be updated (Cohen et al., 2023) . There are two prominent lines of work in this area: parameter-based editing and memory-based editing. Parameter-based editing methods update the knowledge by directly modifying the parameters of the model (Meng et al., 2022a; b) , while memory-based methods employ an explicit memory to store information about the facts to be modified (Mitchell et al., 2022 ; Who was owner of Tom's company from 2018 to 2020?The owner of Twitter is Elon Musk after 2022.Figure 1 : Limitations of dense retrieval to handle temporal context. (a) There is a temporal constraint in question. However, the dense retrieval mechanism fails to accurately capture this temporal information, thus yielding an erroneous retrieval. (b) Our time-aware graph stores edits as a time-conscious knowledge structure to help distinguish temporal context and improve retrieval accuracy. Zhong et al., 2023; Gu et al., 2023) . In most cases, the memory-based approaches outperform the parameter-based methods (Gu et al., 2023; Zhong et al., 2023) To address ripple effects, memory-based methods adopt a plan-and-solve paradigm (Khot et al., 2022; Wang et al., 2023a) , wherein LLMs are prompted to decompose multi-hop questions into sub-questions, followed by iteratively answering each sub-question. These methods employ dense retrieval to identify relevant edits for each sub-question by comparing semantic similarities (Karpukhin et al., 2020) . However, we observed that these approaches exhibit a key limitation, i.e., they perform poorly on questions with explicit temporal contexts (Yin et al., 2023) . This inability to cater to temporal context is attributed to the dense retrieval employed by the memory-based methods, which primarily store the edit retrieval information in an unstructured format. This is illustrated in Figure 1 (a), where the dense retrieval retrieves an irrelevant fact (from the year 2022) for the question "Who was the owner of Tom's company in 2020?".To address the aforementioned limitations, we propose a method called TEMPLE-MQA: TEMPoral knowLEdge augmented Multi-hop Question Answering. TEMPLE-MQA first constructs a time-aware graph (TAG) to store edit information in a structured format to effectively preserve context-specific temporal information in the best possible way (Figure 1 (b)). To enhance retrieval performance across semantically related edits, TEMPLE-MQA: (i) leveraging data augmentation techniques to capture aliases for entity names, aiding in entity disambiguation, and (ii) employing context-dependent concepts to explicitly filter edits based on contextual similarity. To tackle multi-hop questions, TEMPLE-MQA utilizes pre-trained LLMs to devise an inference path and conducts step-by-step joint reasoning, leveraging both LLMs and TAG to derive the final response. Experimental evaluation using benchmark data sets shows that TEMPLE-MQA outperforms the existing state-of-the-art approaches on MQA by a significant margin. We summarize the key contributions of this work as follows.• We propose TEMPLE-MQA, the first method capable of achieving high accuracy in MQA under a substantial volume of temporal knowledge edits without forgetting historical knowledge.• Unlike previous approaches, TEMPLE-MQA constructs a TAG to store structured information through our devised methods. Additionally, we propose a novel planning procedure and a joint reasoning approach for the inference path, alongside the development of a unique structural retrieval procedure tailored for knowledge retrieval, with potential applicability to other problems.• Extensive experimental results on two benchmark datasets show that TEMPLE-MQA outperforms the previous seven baselines via different metrics for MQA under massive edits. Furthermore, we develop a new dataset, namely TKEMQA, which serves as the first benchmark on MQA with temporal scopes. 
related work: 
 Parameter-based Editing. Parameter-editing approaches aim to update a model by incorporating information about updated data or knowledge while ensuring minimal changes to predictions on other data points. These approaches can be categorized into fine-tuning, locating and editing, and meta-learning methods. Fine-tuning methods utilize new knowledge to fine-tune the model parameters while at the same time combating catastrophic forgetting (Chen et al., 2020; Zhu et al., 2020) . Locate and edit approaches treat the layers of a feed-forward network as primary knowledge storage units and update their parameters to edit knowledge. Examples include ROME (Meng et al., 2022a) and its extended version, MEMIT (Meng et al., 2022b) , which targets a large number of edits. Huang et al. (2024) investigated the generalization aspects of knowledge edits, while Li et al. (2023) proposed PMET for precise updating of FFN weights. Hu et al. (2024) proposed a wise-layer KE method to facilitate lifelong editing. METO (Yin et al., 2023) employs time information as an optimization goal to learn new knowledge without forgetting historical information. Metalearning approaches treat the editing task as a machine learning challenge, with examples including hyper-networks trained with constrained optimization for fact-only modification (Cao et al., 2021) , belief-graph by Hase et al. (2021) , and context-aware meta-learned loss scaling by Hu et al. (2023) . However, these methods often underperform for MQA under KE, primarily because the update in the model parameters is more effective for single-hop settings and hard to adapt for multi-hop questions requiring complex reasoning. In contrast, TEMPLE-MQA stores the edits in an edit memory alongside a reasoning link, enabling the decomposition and step-by-step solution of multi-hop questions.Memory-based Editing. These techniques store edits in explicit memory and utilize retrieval-augmented methods to reason over relevant edits and modulate end predictions of the model. For instance, SERAC (Mitchell et al., 2022) introduces semi-parametric editing coupled with a retrieval-augmented counterfactual model. GRACE (Hartvigsen et al., 2022) embeds additional adapters within LLMs for editing, using vector matching to locate and modify edited knowledge. IKE (Zheng et al., 2023) employs in-context learning based on demonstration storage to edit the model's knowledge. MeLLo (Zhong et al., 2023 ) is a simple memory-based approach that stores all edited facts externally and prompts the language model during inference. Additionally, Zhong et al. (2023) also introduces the MQUAKE benchmark for evaluating the MQA performance of their model. Recently, PokeMQA (Gu et al., 2023) proposes a two-stage process of decoupling self-checking and sub-question decomposition to enhance retrieval performance. DeepEdit (Wang et al., 2024) utilizes a depth-first constrained decoding method to edit knowledge for MQA. Tilp (Xiong et al., 2023) provides a differentiable method for learning temporal knowledge graphs in KGs. TEILP (Xiong et al., 2024b) proposes a logical reasoning framework for time prediction in temporal knowledge graphs. TG-LLM (Xiong et al., 2024a) proposes a framework that enhances temporal reasoning capabilities in large language models. Note that, unlike existing solutions, TEMPLE-MQA does not require sub-problem decomposition but generates an inference path directly, which is more straightforward and can directly generate a complete plan. Moreover, the structural retrieval method proposed by our TEMPLE-MQA does not require fine-tuning compared to PokeMQA, making it applicable to a broader range of scenarios, including the processing of temporal information. 
preliminaries: 
 Notations. We represent the set of facts as If one of the steps (s i , r i , o i ) in Q is associated with a fact edit f i ∈ F , it causes ripple effects, i.e., all subsequent facts have to be updated in order to derive the final answer to the question. Mathematically, a chain of facts in Q coupled with corresponding fact edits may be represented as: D = {(s, r, o)} ⊆ E × R × E ,⟨(s 1 , r 1 , o 1 ), • • • , (s i , r i , o * i ), • • • , (s * n , r n , o * n )⟩.⟨(s 1 , r 1 , o 1 , τ 1,s , τ 1,e ), • • • , (s i , r i , o * i , τ * 1,s , τ * 1,e ), • • • , (s * n , r n , o * n , τ * n,s , τ * n,e )⟩. 
temple-mqa: 
 TEMPLE-MQA is a generalized memory-based editing method. It is capable of MQA under knowledge editing (including knowledge correction and updating, etc.) without forgetting historical knowledge. As shown in Figure 2 , the core components of TEMPLE-MQA include pre-existing LLMs and a Time Aware Graph (TAG). The workflow of TEMPLE-MQA is summarized as follows: (i) Constructing a TAG for the given edits; (ii) Utilizing LLMs to devise an inference path for each multi-hop question; (iii) Finally, employing the LLMs along with TAG via our proposed structural retrieval mechanism for joint reasoning based on the inference path. We will discuss each component in detail.The objective of the graph construction process is to design a mechanism to store knowledge edits in a structured format, facilitating effective and efficient retrieval. It is worth noting that this approach markedly differs from existing approaches, which predominantly store knowledge edits in an unstructured format (Zhong et al., 2023) . For example, "The president of the U.S. is Joe Biden." is an unstructured text format, while (U.S., president is, Joe Biden) is structured. This distinction proves crucial in overcoming the challenge of handling temporal contexts for KE.Historical and current facts are typically stored in unstructured text form as f old and f cur .The graph construction process initially transforms them into structured form F t (detailed in Section 3.2). Then it utilizes these structured temporal edits F t as input and generates a graph G t as output. The process flow is elucidated as follows.Converting unstructured facts to a structured form. First, we employ a pre-trained LLM to convert all unstructured facts f x (include f old and f cur ) into a structured form to explicate the information content across different dimensions. To achieve this, we design an incontext learning prompt (see Appendix A for details) that prompts an LLM to process the unstructured fact.EQUATIONWhere f x denotes the unstructured fact, LLM yields a structured output (s, r, o, τ s , τ e , c(o)).Here, c(o) represents the concept associated with the object entity o Ali et al. ( 2020), and P convert is the in-context learning prompt. We use GPT-3.5-turbo-instruct as the pre-trained LLM. Finally, for all pairs of historical and current facts ( f old , f cur ), via our above procedure, we can get the set of structured temporal fact editsF t = { f x → (s, r, o, τ s , τ e )}.It is notable that compared to the original temporal fact, here we also include the concept information for the object, c(o), with the fact structure. It provides multi-faceted benefits, including (i) disambiguation of the entities to their appropriate concepts based on the context and (ii) improved performance for edit retrieval by offering a wide range of fine-grained concepts (Ling & Weld, 2012) . For instance, in different contexts, the entity "Washington" can refer to a city name or a person's name, and without including the concept c(o), explicitly storing this information poses a challenge to distinguishing between them.The purpose of data augmentation is to capture different possible lexical variations for the same entity. This step is pivotal for improving retrieval performance, as it assists in filtering out edits with subjects different from the query subject. This eventually helps reduce the search space by filtering numerous irrelevant edits. We utilize SPARQL (detailed in Appendix E) to get all possible aliases of the subject entity s from Wikidata, denoted as A(s) = {s 0 , s 1 , s 2 , • • • }, where s 0 = s and s i is the i-th alias for s. Note that the current formulation of TEMPLE-MQA favors data augmentation on the subject dimension.Graph construction. Finally, we utilize the structured information about historical and current knowledge to construct the TAG G t . Specifically, we store the current knowledge as {(n s :s i , e : (r, c(o), τ * s , τ * e ), n e : o * )} for s i ∈ A(s), and the historical knowledge as {(n s : s i , e : (r, c(o), τ s , τ e ), n e : o)} for s i ∈ A(s). Here n s , e, and n o represent the start node, edge, and end node in G t , respectively. We can also use the above steps to construct graphs for non-temporal edits with the distinction that we exclude temporal scopes. In this scenario, we only store the updated knowledge as {(n s : s i , e : (r, c(o)), n e : o * )} for s i ∈ A(s). Note, for G, we only store current knowledge.In this section, we introduce the planning and reasoning stages of TEMPLE-MQA: i) The planning stage leverages LLMs to generate an inference path P and extract temporal scope for the question Q. ii) The reasoning stage performs a step-by-step joint reasoning by utilizing LLMs and the TAG we constructed in Section 4.1 to solve the question Q using the inference path P. We will discuss them in detail.Stage 1: Inference Path. This stage aims to exploit the instruction-following ability of LLMs to generate a structural inference path P that is helpful for answering Q. To achieve this, we design an in-context learning prompt (explained in Appendix A) that prompts the LLMs to generate an inference path along with the temporal scopes, as shown below. where P infer is the in-context prompt for Q used as input for LLM, yielding the inference path P and temporal scopes τ Q s and τ Q e as outputs. P is represented asEQUATION⟨(s 1 , r 1 , c(o 1 )), • • • , (c(o n-1 ), r n , c(o n )⟩,where s 1 is an entity extracted from question Q.Note that, unlike previous research, our approach is more practical and efficient, as it computes the inference path in one go without needing to alternate between plan and solve phases. The inference path has the concept of entity added among the relation compared to the relation path Luo et al. (2023) , which helps us in retrieval.Stage 2: Joint Reasoning. As an iterative process, this stage aims to obtain the final answer for Q by reasoning on P.Consider the i-th step (1 ≤ i ≤ n), we utilize (s i , r i , c(o i ), τ Q s , τ Q e )as input to infer the specific entity o i , where s i is the output of last step o i-1 (i > 1) or start point s 1 in P, r i and c(o i ) originate from P. For this, TEMPLE-MQA uses the structural retrieval R struct (Section 4.3) to retrieve relevant knowledge from the G t for the query.EQUATIONwhere o * R is the possible updated knowledge about o i , and α R denotes the similarity between updated knowledge and query. Note that in our approach, there is no need to use selfchecking Zhong et al. (2023) to check if the retrieved fact contradicts the generated answer. Underlying reason for this is the fact that many sota LLMs are trained using RLHF, due to which they exhibit a higher tendency to reject external knowledge that contradicts their own knowledge.Finally, based on the similarity score α R compared against a threshold (θ), we decide if we may use retrieved knowledge as such or use the language model for response generation:EQUATION)where M query is an LLM , and P query is the prompt for query (details are given in Appendix A). We re-iterate the above-mentioned process until we completely exhaust the inference path P, yielding o n as the final answer.Given a query (s i , r i , c(o i ), τ Q s , τ Q e ), this section introduces how to retrieve the knowledge from G t using structural retrieval R struct . The workflow of R struct (See Figure 3 ) can be represented by the following steps: (i) Extracting a subgraph, i.e., filtering out the knowledge that does not meet the required subject and temporal scope. (ii) Re-ranking, i.e., re-ranking the remaining knowledge based on semantic similarity of relation and concept with query.Step 1: Extracting a Subgraph. This step extracts a 1-hop subgraph G sub from G t by filtering out relation pairs violating the subject and temporal scopes. Formally, this is represented as:EQUATIONwhere n s indicates the subject and (e[τ s ], e[τ e ]) represent the temporal scope of the fact.Step 2: Re-ranking. This step aims to re-rank the candidate answers and find the one with the highest similarity. For this, we first use an existing encoder E to encode the relation and concept for the query, as follows:EQUATIONFinally, we re-rank knowledge in G sub based on the cosine similarity (sim) for both relation and fine-grained concept between knowledge and query and then select the knowledge with the highest similarity as the retrieved result.EQUATION)Where o * R is the retrieved object, and α R is the corresponding similarity returned by R struct , e[r] and e[c(o)] indicate relation and concept of the knowledge. 
experiments: 
 Here we conduct practical evaluations for TEMPLE-MQA compared against different baseline models. Additional results, including the ablation study, can be found in Appendix C.Datasets. We evaluate TEMPLE-MQA on a blend of publicly available benchmarks and self-curated datasets. These include: MQUAKE (Zhong et al., 2023) , ATOKE (Yin et al., 2023) , and our newly proposed dataset TKEMQA. Detailed descriptions and statistics of the datasets are given in Appendix B.1 and D.Baselines. We compare the performance of TEMPLE-MQA against a wide range of parameter-based and memory-based KE methods. The parameter-based baselines include: Fine-tuning (FT) Zhu et al. (2020) , ROME (Meng et al., 2022a) , MEMIT (Meng et al., 2022b) , MEND Mitchell et al. (2021) and METO (Yin et al., 2023) . The memory-based baselines include: MeLLo (Zhong et al., 2023) , and PokeMQA (Gu et al., 2023) . Details are given in Appendix B.2.Evaluation metrics. Similar to the baseline models, we use five different evaluation metrics: (i) Multi-hop Accuracy (M-Acc) (Zhong et al., 2023) , (ii) Hop-wise Accuracy (H-Acc) (Gu et al., 2023) , (iii) Historical Explicit time Question Score (HES) (Yin et al., 2023) , (iv) Current Explicit time Question Score (CES) (Yin et al., 2023) , (v) Historical Explicit time Question Score (CES-P) (Yin et al., 2023) , (vi) Current Relative time Question Score (CRS) (Yin et al., 2023) , and (vii) Historical Relative time Question Score (HRS) (Yin et al., 2023) . For all metrics, a larger value indicates that the method is better. Details of evaluation metrics are given in Appendix B.3.Experimental setup. To evaluate performance with varying numbers of edits, we follow the setting of PokeMQA (Gu et al., 2023) to conduct stratified sampling (Parsons, 2014) of the dataset based on the number of hops in questions. This allows us to construct edit batches of different sizes while ensuring a relatively consistent proportion of questions with different hop counts within each batch. We inject all the edits within a batch simultaneously (Zhong et al., 2023) . The scenarios of different batch sizes are denoted as M-edited (M ∈ {1, 100, All}). We use LLaMa-2-7B (Touvron et al., 2023) , Vicuna-7B (Chiang et al., 2023) , GPT-turbo-3.5-instruct and GPT-J-6B Wang & Komatsuzaki (2021) as the LLMs and utilize all-MiniLM-L12-v21 as the encoder in our method. We conducted each experiment four times and reported the average values in the table. All experiments are performed using PyTorch 2.1.2 and RTX 4090 24GB GPU. show that TEMPLE-MQA consistently outperforms the baseline models across different settings by a large margin. For example, when consider the MQUAKE-CF-3K dataset and M-Acc as the evaluation metric, TEMPLE-MQA on average improved by 91.2%, 112.7% and 101.4% compared to Mello for {1, 100, All}-edited respectively across three LLMs, and by 42.6%, 32.4%, and 28.6% compared to PokeMQA. We attribute such drastic performance improvement to the following factors: (a) The inference path employed by TEMPLE-MQA is more reliable, which helps the model to boost the end performance significantly; (b) The structural retrieval pipeline (i.e., TAG) designed for TEMPLE-MQA is robust to reduce the number of false positives significantly. We provide an analysis of these factors in Section 5.3.Compared TEMPLE-MQA with different LLMs, we can see it exhibits superior performance on GPT-3.5-turbo-instruct followed by Vicuna-7B. This is due to the fact that GPT-3.5-turboinstruct has the strongest reasoning capabilities Chiang et al. (2023) , enabling our model to generate more reliable inference paths. Another noteworthy observation in Table 1 is that parameter-based approaches perform poorly compared to memory-based ones, which is consistent with the results in (Zhong et al., 2023) .Single-hop MQA under Temporal KE. We then consider single-hop question answering with temporal knowledge, whose results for ATOKE are shown in Table 3 . Compared to Table 1 , the results show varying behavior, with parameter-based approaches outperforming memory-based methods in some cases, which is contributed by their stronger ability to retain updated knowledge. Specifically, we can see that parameter-based approaches (such as ROME and MEMIT) can achieve very good performance for CES and CES-P, indicating that they can effectively inject new knowledge into the model. However, they perform poorly on the HES metric: even after using METO, the average HES is only 30.5%. This indicates that these methods forget historical knowledge after editing.On the other hand, MeLLo and PokeMQA achieved higher HES scores but lower CES scores, indicating they are unable to distinguish historical knowledge and edited knowledge. TEMPLE-MQA can take into account both historical and new knowledge. It not only has higher HES than MeLLo and PokeMQA but also maintains high CES. Specifically, on ATOKE-SE, TEMPLE-MQA achieves HES of 97.25, an improvement of 221% compared to MEMIT METO . The reason for such a big improvement is that TEMPLE-MQA can effectively distinguish different knowledge in the same time chain through TAG.MQA under Temporal KE. We finally consider the performance on TKEMQA, which consist of {1, 2, 3, 4}-hops equation answering with temporal knowledge. The results are shown in Table 2 . We can see that similar to the above results, TEMPLE-MQA achieves good performance for M-Acc. However, TEMPLE-MQA is worse than MeLLo for HES in the All-edited setting using GPT. This is because there is no historical knowledge matching the question in the edit memory in such a setting. MeLLo's utilization of a self-checking method effectively validates the retrieved results, contributing to its superiority. Table 3 : Experiment results on ATOKE. METO is a plug-and-play method to strengthen the memory of historical knowledge. We mark the METO untested results as "-". ATOKE-ME indicates each data includes two edits to perform continuous knowledge updating.Moreover, in contrast to the previous two datasets, we find that TEMPLE-MQA on TKEMQA experiences only a slight performance decrease when the edit batch size increases. We further conduct an experiment to study the performance w.r.t. difference edit numbers (see Figure 4 in Appendix). It can be observed from Figure 4 (a) and Figure 4 (b) that as the number of edits increases, the performance of TEMPLE-MQA drops rapidly on the MQUAKE-CF-3K. We find that this is because MQUAKE-CF-3K has unpassable data in the mass-edit setting. Specifically, there are data that are affected by the editing of other data. Under M-edited (M > 1) settings, if they are in the same batch, unpassable data will be generated (see Appendix Table 15 for an example). To verify this, we filter out conflicting data and create a new data MQUAKE-CF-3K-FIX. The result is shown in Figure 4 (c) . After repairing, we can observe that TEMPLE-MQA is more stable than other methods, which illustrates the accuracy of structural retrieval.We first conduct an ablation study on different modules of the structural retrieval: without filtering sub-graphs according to the subject entity (i.e. -w/o Subject), without considering the similarity of relation in the Re-ranking step (i.e. -w/o Relation), without considering the similarity of the concept of object entity (i.e. -w/o Concept).The results are shown in Table 10 in Appendix. First, we investigate the role of filtering subgraphs. We can see without filtering out the sub-graph through the subject, TEMPLE-MQA decreased by an average of 8.6% and 7.8% on M-Acc and H-Acc. This is the most impactful component among all removals. Second, we can see both relation and concept can improve the performance of TEMPLE-MQA in all the settings, indicating its necessity. Third, the impact of all eliminated components will become larger as the edit memory increases. Subsequently, we conduct an ablation study on inference path planning module, with details as follows: (1) Replacing inference path planning with sub-question decomposition by Zhong et al. (2023) (i.e., -w/o inference path). (2) Exploring the influence of k-shot (k ∈ {1, 2, 4}) on in-context learning prompt. Corresponding results are shown in Table 11 in Appendix. These results show, if we exclude the inference path planing, the M-ACC drops by almost 15.44%, 14.39%, 9.53% respectively. Also, the M-ACC drops about 10.43%, 7.77% and 3.51% for k-shot (k ∈ {1, 2, 4}) settings.Thus more examples can help LLMs follow instructions better, also the quality and diversity of the examples is important. 
conclusion: 
 We presented a novel framework TEMPLE-MQA for MQA under temporal knowledge editing. TEMPLE-MQA first constructs a time-aware graph, then utilizes inference paths, structural retrieval, and joint reasoning, capable of enhancing MQA performance while distinguishing the temporal context. We also proposed a benchmark TKEMQA, which is the first benchmark specified for MQA with temporal scopes. Experiments on TKEMQA and the existing two benchmarks demonstrated that TEMPLE-MQA outperforms previous methods. 
limitations: 
 Our work poses following limitations and we will resolve them in our future works:• Temple-MQA relies on TAG, which brings additional computational overhead.• Temple-MQA requires writing prompts manually to adapt to different tasks.• Although retrieval of Temple-MQA doesn't need fine-tuning an encoder, it has a complex workflow.  
 
 Appendix: We provide a detailed description of the evaluation datasets as follows.(a) Parameter-based. The parameter-based baselines include: (i) Fine-tuning (FT) (Zhu et al., 2020) that performs a gradient-based update on the model parameters to incorporate new knowledge. (ii) ROME (Meng et al., 2022a) first locates factual knowledge at a specific layer of the transformer architecture and then updates the feed-forward network of this layer to insert new knowledge. (iii) MEMIT (Meng et al., 2022b) extends ROME to allow modifying a range of feed-forward network layers for a large amount of knowledge. (iv) MEND Mitchell et al. (2021) learns a hyper network to produce weight updates by decomposing the gradient of standard fine-tuning into a low-rank form. (v) METO (Yin et al., 2023) extends the parameter-based methods by adding the time information as an optimization objective.(b) Memory-based. The memory-based baselines include: (i) MeLLo (Zhong et al., 2023) that uses the plan-and-solve paradigm. (ii) PokeMQA (Gu et al., 2023) extends MeLLo by adopting a two-stage retrieval process to decouple the question decomposition and knowledge editing.Details about the evaluation metrics and their mathematical formulation are provided as follows:(i) Multi-hop Accuracy (M-Acc), which is used to measure the accuracy of the language models on multi-hop questions. For M-Acc, we use the same settings as Zhong et al. (2023) .The calculation formula for M-Acc is as follows:EQUATIONWhere f * (•) represents the edited model, Q and a * represent the multi-hop questions and the edited answer for each case, respectively.(ii) Hop-wise Accuracy (H-Acc), which is used to check the correctness of the intermediate reasoning path for MQA. For H-Acc, we follow the same settings proposed by Gu et al. (2023) . Given edited chain of facts C * , H-Acc is defined asEQUATION(iii) Historical Explicit time Question Score (HES), which is the accuracy of explicit temporal questions about historical knowledge (Yin et al., 2023) .(iv) Current Explicit time Question Score (CES), which measures the accuracy of explicit temporal questions about current knowledge (Yin et al., 2023) .(v) Current Explicit time Paraphrase Question Score (CES-P), which is the accuracy of explicit temporal questions of semantic paraphrasing asked about current knowledge (Yin et al., 2023) .HES, CES, and CES-P require the model to be able to recall knowledge at a specific temporal scope, calculated according to the following formula:EQUATIONData Conflict in MQUAKE. In the analysis for MQA under Temporal KE, we observed that TEMPLE-MQA on TKEMQA experiences only a slight performance decrease when the edit batch size increases, while its decrease is faster for MQUAKE. This is because MQUAKE-CF-3K has unpassable data in the mass-edit setting, i.e., we can observe data conflicts in MQUAKE. These conflicts arise from inconsistent counterfactual edits between two cases containing the same knowledge (i.e., one is edited, but another is not edited). Under mass editing settings, the inference process of one case may be influenced by the editing of another case, leading to incorrect answers. Table 15 provides an example of data conflict in MQUAKE.Cost of Inference Path. Here, we aim to compare the expense and time cost of our inference path to those of previous methods. According to the inference path, while Mello and PokeMQA need to call LLMs multiple times to generate sub-questions. Thus, we claim TEMPLE-MQA offers a faster and more cost-effective solution.Accuracy of Structured Retrieval. To achieve reliable KE, apart from the correct decomposition of sub-problems, another important factor is whether related edits can be found correctly. Here, we conduct experiments to compare the accuracy of the retrieval process for different methods. According to Table 14 , TEMPLE-MQA achieve higher retrieval accuracy compared to Mello and PokeMQA by 17.9% and 7.7% in the all-edited setting, respectively. Thus, our structured retrieval can efficiently find structured edits.Here, we aim to study the performance of different models with different hop numbers. As shown in Figure 5 , TEMPLE-MQA can still maintain a high M-Acc as the number of hops increases, while Mello and PokeMQA experience an obvious decline in performance with an increase in hop number. This is because Mello and PokeMQA couple the tasks of querying the LLM to answer sub-questions and generate sub-questions together. As the hop number increases, the LLM is burdened with complex tasks, causing a significant decrease in M-Acc. In contrast, TEMPLE-MQA decouples the generation of inference paths, enabling it to maintain high performance as the hop number increases. TEMPLE-MQA under Implicit Time. For questions with hidden time, we consider the following examples: 1) "Who was the U.S. president before Biden?", 2) "Who was the previous president of the U.S.?".For example 1, TEMPLE-MQA needs to determine the specific time when Biden is president. We can revise the ICL prompt to achieve it, as show in below: Question: Who was the U.S. president before Biden? Time: Biden is president after 2021, Thus time span is <2017,2020>. Inference Path: <U.S.,president is,?>. However, we observe some datasets contain questions that are more ambiguous, as shown in examples 2. Answer to these questions is decided on the training periods of LLMs. For this, we need to give current time to LLM, as shown in below: Question: Who was the previous president of the U.S.? Time: Today is 2021, and Biden is current president, Thus time span is <2017,2020>. Inference Path: <U.S.,president is,?>.The results on the AToke-SE and AToke-ME with ambiguous hidden time (like example 2) are shown in Table 12 . We can see that TEMPLE-MQA still performs better, especially on the HRS metric.Our benchmark TKEMQA is created based on Wikidata2 , a knowledge base consisting of fact triples associated with millions of entities. TKEMQA can be used to evaluate whether the knowledge editing method can update new knowledge without forgetting past knowledge. It contains simple single questions and multi-hop questions, and its rich settings enable it to comprehensively evaluate knowledge editing methods.As shown in Table 9 , TKEMQA contains 2,000 data where each one includes a k-hop (k ∈ {1, 2, 3, 4}) questions. As GPT-3.5 does not have the knowledge after 2022, we use the knowledge after 2022 as an edit to update knowledge. Thus, we refer to each multi-hop question as a current question because it has a time prompt after 2022. Besides the updated knowledge, we also introduce the corrective knowledge, which is used to modify the error historical knowledge. This knowledge does not need temporal scope and is counterfactual.To evaluate whether the editing method may cause model forget the historical knowledge (i.e. knowledge of model before the editing), each data of TKEMQA include two historical question with the time prompt before 2022. Based on TKEMQA, we add the knowledge corresponding to the historical question into the edit memory to evaluate whether the retrieval can effectively distinguish different knowledge in the same time chain, in which we refer to as TKEMQA-HK.We use M-Acc and HES as two metrics in TKEMQA. M-Acc is the accuracy of the language models on current multi-hop questions, evaluating the model's ability to update knowledge and perform multi-hop reasoning successfully. HES is the accuracy of answering historical questions and assessing the model's ability to recall historical knowledge.Collect relation template. We first manually collect multiple relations containing time information from WikiData, e.g. "President is" (Appendix D.3). Then, we use SPARQL to sample facts based on the relation and subject entity. For example, given a query (U.S., President is, ?), WikiData will return every president of the U.S. and their terms of office. In addition, we select common relations and entities to control quality.In order to generate natural and coherent multi-hop questions, we look for high-quality relation path Luo et al. (2023) with different lengths k ∈ {1, 2, 3, 4}. (live in, the father is) is a bad path because the object type of 'life in' (location) does not match the subject type of 'father is' (person); a location cannot have a father. Thus, we arrange and combine the collected relations into a relation path template and filter out unreasonable relation combinations. For each relation path, we can get several fact chains based on different start point entities.Generate multi-hop questions. Then, we use GPT-4 to convert the relation path template into questions in natural language form, e.g., (father is, company is) -> "What is the company of father of {}?". Replace "{}" into the start point entity to get the final questions. We also conduct manual revisions to ensure its quality. Different from existing work (Zhong et al., 2023) , this method produces data with higher quality questions due to the powerful capabilities of GPT-4. This method also helps reduce the additional overhead of using GPT-4 because one path often correspond to multiple data items.Additional details for edits. Among the collected fact chains, there is only one piece of knowledge that is after 2023, which is used to represent the knowledge update of the real world.D.3 Question/Cloze Statement Templates used in TKEMQA Following previous work, we use question templates to filter the facts that cannot be recalled and cloze-style statement templates to convert an edited fact into a natural language formhttps://huggingface.co/sentence-transformers/all-MiniLM-L12-v2https://www.wikidata.org/wiki/Wikidata:Main_Page 


Decision: This paper proposed to use time-aware graph to store time-conscious knowledge, which is used for answering temporal multi-hop questions.

Pros:
1. The problem studied is important.
2. The method achieves high accuracy.
3. Some reviewers commended the novelty of the method.

Cons:
1. Reviewers asked for elaboration of some details. 
2. Reviewers also asked for some additional ablation and comparison with existing work to strengthen the evaluation.

The authors addressed some of reviewers’ concerns in the rebuttal. Overall, all four reviewers are positive about this paper. There are not any critical concerns. The authors are encouraged to include the additional results and elaboration in their final version of the paper if the paper is accepted.

Reviews: review: paper_summary: paper_summary: This paper proposed a temporal knowledge augmented multi-hop question answering method called Temple-MQA. This method built a time-aware graph to store edits as a time-conscious knowledge structure. The benefit of this graph is to store the temporal information in a structured format. During the retrieval, Temple-MQA utilized a LLM to devise a path and do a step-by-step reasoning. It is interesting to use LLM to strategize an inference path. In general, the whole idea is clear with a reasonable performance.
strength_weakness: strength_weakness: Weaknesses:

1.	The temporal questions include two types: questions with explicit time (like 01/01/2024) and questions with hidden time (like before, after). It will be wondering if more deep analysis cross different types are provided. The questions with hidden time will require more support about multi-step reasoning. Maybe this method can perform well on various types.

2.	Using the LLMs to generate the inference path is a great idea. But this section missed some details. I would like to see more comparisons and analysis about this direction. The accuracy of inference path will impact the performance a lot.
questions: questions: Look forward to more analysis on various temporal question types.
paper_summary: paper_summary: The proposed method introduces a novel approach for handling multi-hop question answering that involves substantial volumes of temporal knowledge edits. The proposed approach consists of 1) extending a fact triple knowledge graph with timestamps (referred to as a time-aware graph or TAG by the authors). Such structured information can then be converted into unstructured text by an LLM for Data Augmentation and Contextual Filtering to enhance retrieval performance. Given these improvements, they also propose using LLM and TAG for chain-of-thought type of step-by-step joint reasoning.
strength_weakness: strength_weakness: - My main concern is with the evaluation. Are there quality controls on the created TKEMQA dataset? 
 In section 3, the temporal KE from start time to end time is added to the original fact triple. However, it was not clear to me how reliable the augmented facts are. Are these being evaluated? It seems that they are only evaluated on downstream multi-hop QAs with accuracy. 
- There are some recent papers on temporal multi-hop QA datasets that are worth comparison to the differences. e.g. PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering https://arxiv.org/abs/2402.11034
paper_summary: paper_summary: The paper proposes a novel method to multi-hop question answering especially focusing on how to correctly answer temporal questions. The authors use a temporal aware graph (TAG) representing temporal information. The graph is used to retrieve the correct answer to temporal questions. Comparison with previous approaches shows superiority of the new method for two benchmark datasets. In addition, a new data set is proposed.
strength_weakness: strength_weakness: The paper is sometimes difficult  to follow, several important aspects are hidden in the appendix or refer to other papers:
- Construction of the temporal graph generation 
- the metrics used.
questions: questions: While the overall contribution is definitely novel and the results are impressive, I found it difficult to understand the different steps of the algorithm that was developed. Sometimes the authors referred to the appendix and some of the explanations seemed confusing and  incomplete:

- 4.1. Comparison to existing approaches:”..store knowledge edits in an unstructured format.. “ please give an example. Does unstructured format simply mean text?
- “Historical  and current facts are typically stored in unstructured format  as f_ols and f_current” this sentence does not add much to the explanation

- “See appendix A for details.” PLese add more information in the text because this is an important part of your paper

- Data augmentation (SPARQL) and graph construction refer to the appendix again and the text lacks clarity. It may be helpful to include a running example so that the reader can see the effects of the different steps.

- “alternate between multiple different and solve phases. “ noun missing after different?

- Evaluation metrics: the metrics are specifically developed for the temporal Multi-hop QA and it would be good to describe them in more detail in the paper because otherwise the reader will have a hard time identify the significance of the difference in performance (i.e., Historical Explicit time Question Score (HES) and Current Explicit time Question Score (CES) )
- Joint reasoning: “there is no need to use self- checking Zhong et al. (2023) to check if the retrieved fact contradicts the generated answer. “ why?
paper_summary: paper_summary: The paper proposes a novel framework called TEMPLE-MQA (TEMPoral knowLEdge augmented Multi-hop Question Answering) for multi-hop question answering under temporal knowledge editing. TEMPLE-MQA constructs a time-aware graph (TAG) to store knowledge edits in a structured manner, preserving temporal information. It utilizes pre-trained language models (LLMs) to devise an inference path and perform joint reasoning using the TAG and LLMs. The key contributions include the TAG construction method, inference path generation, structural retrieval for knowledge retrieval, and a new dataset called TKEMQA for evaluating temporal MQA. Experimental results on benchmark datasets demonstrate that TEMPLE-MQA outperforms existing baselines for MQA under massive knowledge edits.
strength_weakness: strength_weakness: 1. While the paper introduces the TKEMQA dataset, more details on the dataset construction process and the quality of the data could be provided.
2. The paper could benefit from more extensive ablation studies and analyses to better understand the contributions of different components in TEMPLE-MQA.
3. The computational complexity and scalability of TEMPLE-MQA, especially for large-scale knowledge graphs, are not thoroughly discussed.
4. The paper lacks discussions on the limitations of the proposed approach and potential future directions.
questions: questions: 1. In Section 4.1 on constructing the Time-Aware Graph (TAG), you mention using data augmentation to capture aliases for entity names to aid entity disambiguation. Could you provide some more details on how this data augmentation is performed and how effective it is compared to not using data augmentation?

2. The results in Table 2 show that on TKEMQA, TEMPLE-MQA underperforms MeLLo on the HES metric in the All-edited GPT setting. What are some potential reasons for this and are there any planned improvements to address it?

Comments: title: Re: Rebuttal by Authors
comment: Thanks for the response and explanation! It will be helpful to add the explanation into the revision.
Thanks a lot for providing more results and clarifying the questions. It is pretty interesting about the implicit time analysis. I would like to recommend to add it into the final version. In general, the whole paper looks great. I'd like to raise my rating.
thank you for the explanations/comments. I think the suggested changes will improve clarity of the paper.
Thanks for your response. I will stick with my original rating.