Multi-hop Question Answering under Temporal Knowledge Editing
Authors: Keyuan Cheng
Gang Lin
Haoyang Fei
Yuxuan Zhai
Lu Yu
Muhammad Asif Ali
Lijie Hu
Di Wang
Venue: COLM (2024)
Keywords: Knowledge Editing, Multi-hop QA
TL;DR: Multi-hop Question Answering under Temporal Knowledge Editing

Abstract: Multi-hop question answering (MQA) under knowledge editing (KE) has garnered significant attention in the era of large language models. However, existing models for MQA under KE exhibit poor performance when dealing with questions containing explicit temporal contexts. To address this limitation, we propose a novel framework, namely TEMPoral knowLEdge augmented Multi-hop Question Answering (TEMPLE-MQA). Unlike previous methods, TEMPLE-MQA first constructs a time-aware graph (TAG) to store edit knowledge in a structured manner. Then, through our proposed inference path, structural retrieval, and joint reasoning stages, TEMPLE-MQA effectively discerns temporal contexts within the question query. Experiments on benchmark datasets demonstrate that TEMPLE-MQA significantly outperforms baseline models. Additionally, we contribute a new dataset, namely TKEMQA, which serves as the inaugural benchmark tailored specifically for MQA with temporal scopes.

Published as a conference paper at COLM 2024
Multi-hop Question Answering under Temporal Knowledge Editing
Keyuan Cheng-.1.23, Gang Lin..1..3 Haoyang Fei Yuxuan zhai?, Lu Yu², Muhammad Asif Ali Lijie Hut.24 and Di Wang Provable Responsible AT and Data Analytics (PRADA) Lab
King Abdullah University of Science and Technology
Sooth China University of Technology
*SDATA-KAUST AT
Ant Group
Abstract
Multi-hop question answering (MQA) under knowledge editing (KE) has garnered significant attention in the era large language models. However, existing models for MQA under KE exhibit poor performance when deal- ing with questions containing explicit temporal contexts. To address this limitation, we propose novel framework, namely TEMPoral Lnow augmented Multi-hop Question Answering (TEMPLE-MQA). Unlike previ- ous methods, TEMPLE-MQA first constructs time-aware graph (TAG) to store edit knowledge in structured manner. Then, through our proposed inference path, structural retrieval, and joint reasoning stages, TEMPLE- MQA effectively discerns temporal contexts within the question query. Experiments on benchmark datasets demonstrate that TEMPLE-MQA sig- nificantly outperforms baseline models. Additionally, we contribute a new dataset, namely TKEMQA, which serves as the inaugural benchmark tailored specifically for MQA with temporal scopes.
Introduction
Large Language Models (LLMs) have garnered widespread attention owing to their re- markable capacity for knowledge comprehension, enabling tailored solutions across various applications (Zhao et al. 2023 Huang & Chang 2022). However, the presence of outdated knowledge presents significant challenge, impeding the ability of LLMs to provide accu- rate responses regarding recent events and facts a phenomenon known as hallucinations, wherein LLMs tend to fabricate plausible yet incorrect responses about unknown facts (Hong et al., 2023). Thus, ensuring the timely updating of LLMs with the latest information of paramount importance. However, the most direct approach for updating informa- tion, editing LLMs by re-training from scratch is practically infeasible, as it requires huge computational resources and substantial investments. In response, Knowledge Editing (KE) has emerged as a focal point, which aims to precisely modify or update the knowledge in LLMs without requiring model retraining. This topic has garnered considerable attention in recent years (Wang et al., 2023b Zhang et al. 2024).
Multi-hop question answering (MQA), on the other hand, aims to tackle complex inquiries that necessitate multiple reasoning steps. In the context of MQA, KE introduces the concept of "ripple effects", wherein a single edit may trigger a cascade of subsequent knowledge edits or updates (Cohen et al. (202).. For instance, if we update the knowledge about the U.S. president from Trump to Buden, correspondingly knowledge for the question: "Who is the wife of the U.S. president?" should also be updated (Cohen et al. 2023). There are two prominent lines of work in this area: parameter-basee
editing and memory--assed editing. Parameter-based editing methods update the knowledge by directly modifying the parameters of the model (Meng et al., 2022alb) b), while memory-based methods employ an explicit memory store information about the facts to be modified (Mitchell et al. 2022
"The first three authors contributed equally to this work. I *Correspondence to Lijie Hu fljje.hu@kaust.edu.sl and Di Wang (di wang@kaust.ddu.sa.

---

Published as a conference paper at COLM 2024
Question
Who was owner of Tom's
<2014, 2017, company_ 163
Google
company from 2018 to 2020?
Answer
<an06, 2anaa, owner
Evan Williass
Time Conflit!
Tom
Retrieved Edit
<2017, NJA, company Jac
Twitter
The owner of Twitter is Elon
Musk after 2022.
Elon Musk
(a) Dense Retrieval may cause time conflict
(b) The correct
via Time Aware Graph
Figure Limitations of dense retrieval handle temporal context. (a) There a temporal
constraint in question. However, the dense retrieval mechanism fails to accurately capture this temporal information, thus yielding an erroneous retrieval. (b) Our time-aware graph stores edits time-conscious knowledge structure to help distinguish temporal context and improve retrieval accuracy.
Zhong et al. 2023 Gu et al. 2023). In most cases, the memory-based approaches outperform the parameter-based methods (Gu et al. 2023 Zhong et al. 2023)
To address ripple effects, memory-based methods adopt a plan-and-solve paradigm (Khot et al. 2022 Wang et al., 2023a), wherein LLMs are prompted to decompose multi-hop questions into sub-questions, followed by iteratively answering each sub-question. These methods employ dense retrieval to identify relevant edits for each sub-question by com- paring semantic similarities (Karpukhin
al. 2020). However, we observed that these approaches exhibit key limitation, I.e., they pertorm poorly questions with explicit temporal contexts (Yin et al.) 2023). This inability to cater temporal context is attributed to by the dense retrieval employed by the memory-based methods, which primarily store the edit retrieval information in an unstructured format. This is illustrated in Figure I (a), where the dense retrieval retrieves irrelevant fact (from the year 2022) for the question "Who the owner of Tom's 's company in 2020?"
To address the aforementioned limitations, we propose method called TEMPLE-MQA: TEMPoral knowLEdge augmented Multi-hop Question Answering. TEMPLE-MQA first constructs a time-aware graph (TAG) to store edit information in a structured format to effectively preserve context-specffic temporal information in the best possible way (Figure (b)). To enhance retrieval performance across semantically related edits, TEMPLE-MQA (i) leveraging data augmentation techniques to capture aliases for entity names, aiding in entity disambiguation, and (ii) employing context-depent context-dependent concepts to explicitly filter edits based on contextual similarity. To tackle multi-hop questions, TEMPLE-MQA utilizes pre-trained LLMs to devise an inference path and conducts step-by-step joint reasoning, leveraging both LLMs and TAG to derive the final response. Experimental evaluation using benchmark data sets shows that TEMPLE-MQU MMQ outperforms the existing state-of-the-ae approaches on MQA by a significant margin. We summarize the key contributions of this work as follows.
We propose TEMPLE-MQA, the first method capable achieving high accuracy in MQA under substantial volume of temporal knowledge edits without forgetting historical knowledge.
Unlike previous approaches, TEMPLE-MQA constructs a TAG to store structured information through our devised methods. Additionally, we propose a novel planning procedure and a joint reasoning approach for the inference path, alongside the development of a unique structural retrieval procedure tailored for knowledge retrieval, with potential applicability to other problems.
Extensive experimental results on two benchmark datasets show that TEMPLE- MQA outperforms the previous seven baselines via different metrics for MQA under massive edits. Furthermore, we develop new dataset, namely TKEMOA, which serves the first benchmark on MQA with temporal scopes.
2

---

Published as a conference paper at COLM 2024
2 Related Work
Parameter-based Editing. Parameter-editing approaches aim to update a model by incor- porating information about updated data or knowledge while ensuring minimal changes to predictions on other data points. These approaches can be categorized into fine-tuning, locating and editing, and meta-learning methods. Fine-tuning methods utilize new knowl- edge to fine-tune the model parameters while at the same time combating catastrophic forgetting (Chen et al. 2020; Zhu et al. 2020). Locate and edit approaches treat the layers of feed-forward network as primary knowledge storage units and update their parameters to edit knowledge. Examples include ROME (Meng et al. 2022a) and its extended version, MEMIT (Meng et al.] (002b),, which targets a large number of edits. [Hang et al. (2024)
investigated the generalization aspects of knowledge edits, while Li et al. (2023) proposed PMET for precise updating of FFN weights. [uu et al. (2024) proposed a wise-layer KE method to facilitate lifelong editing. METO (Yin et al. 2023) employs time information as an
optimization goal to learn new knowledge without forgetting historical information. Meta-
learning approaches treat the editing task as a machine learning challenge, with examples including hyper-networks trained with constrained optimization for fact-only modification (Cao et al., (021), belief-graph by Hase et al. (2021), and context-aware meta-learned loss scaling by Hu et al. (2023). However, these methods often underperform for MQA under KE, primarily because the update in the model parameters is more effective for single-hop settings and hard to adapt for multi-hop questions requiring complex reasoning. In contrast, TEMPLE-MQA stores the edits in an edit memory alongside reasoning link, enabling the decomposition and step-by-step solution of multi-hop questions.
Memory-based Editing. These techniques store edits in explicit memory and utilize retrieval-augmented methods to reason over relevant edits and modulate end predictions of the model. For instance, SERAC (Mitchell et al. 2022) introduces remi-parametric editing coupled with retrieval-augmented countertactual model. GRACE (Hartvigsen et al. 2022 embeds additional adapters within LLMs for editing, using vector matching locate and modify edited knowledge. IKE (Zheng et al. 2023) employs in-context learning based on demonstration storage to edit the model's knowledge. MeLLo (Zhong et al. 2023) is simple memory-based approach that stores all edited facts externally and prompts the language model during inference. Additionally, Zhong et al. (2023) also introduces the MQUAKE benchmark for evaluating the MQA performance of their model. Recently, PokeMQA (Gul et al. 2023) proposes a two-stage process of decoupling self-checking and sub-question decomposition to enhance retrieval performance. DeepEdit (Wang et al.) 2024) utilizes depth-first constrained decoding method to edit knowledge for MQA. Tilp (Xiong et al. 2023) provides differentiable method for learning temporal knowledge graphs KGs.
TEILP Xiong et al. 2024b) proposes a logical reasoning framework for time prediction in temporal knowledge graphs. TG-LLM (Xiong et al. 2024a) proposes framework that en- hances temporal reasoning capabilities in large language models. Note that, unlike existing solutions, TEMPLE-MQA does not require sub-problem decomposition but generates an inference path directly, which is more straightforward and can directly generate a complete plan. Moreover, the structural retrieval method proposed by our TEMPLE-MQA does not require fine-tuning compared to PokeMQA, making it applicable to a broader range of scenarios, including the processing of temporal information.
3 Preliminaries
Notations. We represent the set of facts as D = {(,,,,)) C x Rx E, where &, R denote the set of entities and relations respectively. A tuple (s,r,0) E D represents a fact, with subject entity and object entity having relation r. For the temporal facts, we add time information to D, i.e, D, = (s,T,, Ts, Te) S E x R x E x T x T, where T represents the timestamps. A tuple (s,r, Ts, 4)) E Dr represents temporal fact, with subject entity s, object entity o, having relation with temporal scope from T₃ to Tz. We use G, to represent the time aware graph constructed using D, (Section 4.1
3.1 Knowledge Editing and MQA
Knowledge Editing (KE). A KE request can be represented as F = ffr.f...... ... fuh, which contains set of fact edits. Each edit f; E F represents an individual knowledge editing
3

---

Published as a conference paper at COLM 2024
Question
Who was owner of Tom's company from
Planning
Person
2018 2020?
Company
Time scope:
Language
Model
Answer
Knowledge Edit
CIOME 2017 eoepuay
Google
Joint Reasoing language which Need edited.
from 2014 2017, The company Yom Google.
Construct
Tom
<2000,
Evan Willimm
MEA
After 2022, The maner Twitter in Elon Musk.
Time Aware Graph
Elon Musk
Figure 2: Overview of TEMPLE-MON Given multi-hop question, TEMPLE-MQA employs LLMs to strategize an inference path for the question. Then, it leverages LLMs and the TAG (Gi) for joint reasoning on the inference path.
operation, denoted by fi (,) (s,r,o"), indicating that the object of the subject with relation r updated from o o"
MQA under KE. A multi-hop question Q requires multiple reasoning steps to de- rive the final answer. The reasoning steps could be represented as a chain of facts (111,71,01,, (Sn, In, 0x)), where each step (hop) in the chain (s;, (i,,i,,0 0j) associated with an individual fact. The subject and object are chained together, i.e., the object (o;) from the preceding fact becomes the subject (si+1) for the subsequent fact.
one of the steps (si, ri, 0;) in Q is associated with a fact edit f it causes ripple effects, i.e., all subsequent facts have to be updated in order to derive the final answer to the question. Mathematically, a chain of facts in Q coupled with corresponding fact edits may be represented multiple fact edits in F. The end goal of MQA under KE is to derive the final answer o for Q after considering all associated edits in F.
(($), 0)), (s, (, ,,,, (S In, Tn))))) Note that Q may initiate
3.2 Temporal Scope
Temporal KE. In KE under temporal scope, we use = ifi, fuu to represent the temporal fact edits, with single temporal fact edit denoted by fi (s,r,0, Ta, Te) (s,r,0" T T It means that (s, is valid from Tş until Te, and after that 0") is valid from vs to T
=
Temporal MQA under KE. Similar to MQA under KE, here, each reasoning step will be represented temporal fact. A chain of facts in coupled with corresponding temporal fact: (($), 11,0, & ),,,, (5i, ri of Ti Ti,),, (S Tn, on Tugs T.
4 TEMPLE-MQA
TEMPLE-MQA is generalized memory-based editing method. is capable of MQA under knowledge editing (including knowledge correction and updating, etc.) without forgetting historical knowledge. As shown in Figure the core components of TEMPLE-MQA include pre-existing L.LMs and Time Aware Graph (TAG). The workflow of TEMPLE-MMA is summarized as follows: (i) Constructing TAG for the given edits; (ii) Utilizing LLMs to devise an inference path for each multi-hop question; (iii) Finally, employing the LLMs along with TAG via our proposed structural retrieval mechanism for joint reasoning based on the inference path. We will discuss each component detail.
4.1 Time-Aware Graph Construction
The objective of the graph construction process is to design mechanism to store knowledge edits a structured format, facilitating effective and efficient retrieval. is worth noting that this approach markedly differs from existing approaches, which predominantly store knowledge edits an unstructured format [Zhong et al.) 2023). For example, "The president of the U.S. is Joe Biden." is unstructured text tormat, while (U.S., president is, Joe Biden) is
4

---

Published as a conference paper at COLM 2024
structured. This distinction proves crucial in overcoming the challenge of handling temporal contexts for KE
Historical and current facts are typically stored in unstructured text form as fuld and feur- The graph construction process initially transforms them into structured form F, (detailed in Section 3.2). Then it utilizes these structured temporal edits Fi as input and generates graph Gr as output. The process flow is elucidated as follows.
First, we employ a pre-trained LLM to convert all unstructured facts fx (include fold and fcur) into a structured form to explicate the information content across different dimensions. To achieve this, we design an in- context learning prompt (see Appendix A for details) that prompts an LLM to process the unstructured fact.
Converting unstructured facts to structured form.
s,r,0, Ts, Te c(o) LLM connzert (fx)),
(1)
Where fx denotes the unstructured fact, LLM yields a structured output (,, Ts, Te, c( Here, c(o) represents the concept associated with the object entity 0 Ali et al. (2020), and P Pronvert is the in-context learning prompt. We use GPT-3.5 --TT-.--turbo-instruc as the pre-trained LLM. Finally, for all pairs of historical and current facts (fold, feur), via our above procedure, we can get the set of structured temporal fact edits F, = [fx + (s,r,o, o, Ts, T.)
It is notable that compared to the original temporal fact, here we also include the concept information for the object, c(o), with the fact structure. It provides multi-faccted benefits, including (i) disambiguation of the entities to their appropriate concepts based on the context and (ii) improved performance for edit retrieval by offering a wide range of fine-grained concepts (Ling & Weld 2012). For instance, in different contexts, the entity "Washington" can refer to a city name or a person's 's name, and without including the concept c(o), explicitly storing this information poses challenge to distinguishing between them.
Data augmentation. The purpose of data augmentation is to capture different possible lexi- cal variations for the same entity. This step is pivotal for improving retrieval performance, as it assists in filtering out edits with subjects different from the query subject. This eventually helps reduce the search space by filtering numerous irrelevant edits. We utilize SPARQL (detailed in Appendix E) to get all possible aliases of the subject entity s from Wikidata, denoted as A(s) = {sp, $1, $2, ,, where so s and s; is the i-th alias for s. Note that the current formulation of TEMPLE-MQA favors data augmentation on the subject dimension.
Graph construction. Finally, we utilize the structured information about historical and current knowledge to construct the TAG Gi. Specifically, we store the current knowledge as {(n : s; (r)), T2, (Z)TT) T,), : o²)} for s E A(),, and the historical knowledge as {(n, e (r,e:(r)(()) T, Te), : o)} for s; E A(s). Here n2, e, and n, represent the start node, edge, and end node in G., respectively.
We can also use the above steps to construct graphs for non-temporal edits with the distinc- tion that we exclude temporal scopes. In this scenario, we only store the updated knowledge as {nn e (r,c(o)), Ne )²)} for s; E A(s). Note, for G, we only store current knowledge.
4.2 Planning and Reasoning on Inference Path
In this section, we introduce the planning and reasoning stages of TEMPLEE i) The planning stage leverages LLMs to generate an inference path P and extract temporal scope for the question Q. ii) The reasoning stage performs step-by-step joint reasoning by utilizing LLMs and the TAG we constructed in Section 4.1 to solve the question Q using the inference path P. We will discuss them in detail.
Stage 1: Inference Path. This stage aims to exploit the instruction-follloing ability of LLMs to generate structural inference path P that is helpful for answering Q. To achieve this, we design an in-context learning prompt (explained in Appendix(Al (A) that prompts the LLMs to generate an inference path along with the temporal scopes, shown below.
Q Q P, T To
ILLM Infer (Q)),
(2)
5

---

Published as a conference paper at COLM 2024
Input for Retrieval: <Tom, work iny company, 2018, 2020>
Google
co0n4, 2017,
Twitter
$2006, 2012. UNTTrrE Seo
Evan Williams
company_oo Lks
company_joo Lss>
Tom
Encoder
Tom
CH0DO N/A. autionalty
Twitter
2018
Tom
work in
Twitter
K2000 NEA
Elon Musk
2020
company
U.S.
owner
U.S.
Time-aware Graph
Subgraph
Figure 3: The workflow of structural retrieval. 1) The first step is to extract the 1-hop sub-graph where the center point is Tom. We filter out the edits not conforming with the temporal scope of the query. 2) The second step calculates the semantic similarity of the relation and concept between the query and knowledeg in sub-graph.
where Pinfer is the in-context prompt for Q used as input for LLM, yielding the in- ference path P and temporal scopes 7 and To as outputs. P is represented as ((,)),, ((())))),,((()_)))))))))))) ((. where $1 is an entity extracted from question Q.
Note that, unlike previous research, our approach is more practical and efficient, as computes the inference path in one go without needing to alternate between plan and solve phases. The inference path has the concept of entity added among the relation compared to the relation path Luo al. 22023), which helps in retrieval.
Stage 2: Joint Reasoning. As an iterative process, this stage aims to obtain the final answer
for Q by reasoning on P. Consider the i-th step (1 s i S n), we utilize (,,, T T&) as input to infer the specific entity where Sj is the output of last step 0i > 1) or start point $1 in P, r; and c(o;) originate from P. For this, TEMPLE-MQA uses the structural retrieval Ratruct (Section 4.3) to retrieve relevant knowledge from the G. for the query.
OR R Rstruct (5; c(;;), T, )
(3)
where of is the possible updated knowledge about o,, and ag denotes the similarity between updated knowledge and query. Note that in our approach, there is no need to use self- checking Zhong et al. (2023) to check if the retrieved fact contradicts the generated answer. Underlying reason for this is the fact that many sota LLMs are trained using RLHF, due to which they exhibit higher tendency to reject external knowledge that contradicts their own knowledge.
Finally, based on the similarity score ag compared against threshold (0), we decide if we may use retrieved knowledge as such or use the language model for response generation:
ok
aR 0 aR S A
0;
M Mquery(PPo (Pquery (s;, ri, c(o;), Ts, Te))
where M Mquery is an LLM and Pquery is the prompt for query (details are given in Ap- pendix A). We re-iterate the above-mentiones process until we completely exhaust the inference path P, yielding On as the final answer.
4.3 Structural Retrieval
Given query (s c(o;), T T this section introduces how to retrieve the knowledge from G. using structural retrieval Rstruct- The workflow of Rstruct (See Figure 3 can be represented by the following steps: (i) Extracting a subgraph, iee, filtering out the knowledge that does not meet the required subject and temporal scope. (ii) Re-ranking, i.e., re-ranking the remaining knowledge based semantic similarity of relation and concept with query.
Step 1: Extracting a Subgraph. This step extracts a 1-hop subgraph Goub from Gi by filtering out relation pairs violating the subject and temporal scopes. Formally, this is represented as:
sub (ns,, E G1 (n, = s;) A (e(Ts) T:9) A {T)),.
(5)

---

Published as a conference paper at COLM 2024
where ny indicates the subject and (e(z)), represent the temporal scope of the fact.
Step 2: Re-ranking. This step aims to re-rank the candidate answers and find the one with the highest similarity. For this, we first use existing encoder E to encode the relation and concept for the query, as follows:
of = E(ri):o c(o) = E(c(o;)).
(6)
Finally, we re-rank knowledge in Sub based on the cosine similarity (sim) for both relation and fine-grand sine-grained concept between knowledge and query and then select the knowledge with the highest similarity as the retrieved result.
OR. aR = arg max (sim (((())((r)) + sim(E(e(((o))) (()))
Where of is the retrieved object, and aR is the corresponding similarity returned by Rstruct, e|r] and e(c(o)) indicate relation and concept the knowledge.
Experiments
Here we conduct practical evaluations for TEMPLE-MQA compared against different base- line models. Additional results, including the ablation study, can be found in Appendix
5.1 Experimental Settings
Datasets. We evaluate TEMPLE-MQA on blend of publicly available benchmarks and self-curated datasets. These include: MQUAKE (Zhong et al. 2023), ATOKF (Yin et al.
2023), and our newly proposed dataset TKEMQA. Detailed descriptions and statistics of the datasets are given in Appendix B.1 and D
Baselines. We compare the performance of TEMPLE-MQA against a wide range of parseeter-based and memory-based KE methods. The parameter-based baselines include: Fine-tuning (FT) Zhu et al. (2020), ROME (Meng et al. 2022a), MEMIT (Meng et al., 2022b), MEND Mitchell et al. (2021) and METO (Yin et al. 2023). The memory-based baselines
include: MeLLo (Zhong et al. 2023), and PokeMQA (Gu et al. 2023). Details are given in
Appendix B.2
Evaluation metrics. Similar the baseline models, we use five different evaluation metrics: (i) Multi-hop Accuracy (M-Acc) (Zhong et al. 2023), (ii) Hop-wise Accuracy (H-Acc) (Gu et al. 2023), (iii) Historical Explicit time Question Score (HES) (Yin et al.) 2023), (iv) Current Explict time Question Score (CES) (Yin et al. 2023), (v) Historical Explicit time Question Score (CES-P) (Yin et al. 2023), (vi) Current Relative time Question Score (CRS) YYin et al.,
2023), and (vii) Historical Relative time Question Score (HRS) (Yin et al. 2023). For all
metrics, a larger value indicates that the method is better. Details of evaluation metrics are given in Appendix B.3
Experimental setup. To evaluate performance with varying numbers of edits, we follow the setting of PokeMQA [Gu et al. 2023 to conduct stratified sampling (Parsons 2014 of the dataset based on the number of hops in questions. This allows us to construct edit
batches of different sizes while ensuring a relatively consistent proportion of questions with different hop counts within each batch. We inject all the edits within a batch simultaneously (Zhong et al. 2023). The scenarios of different batch sizes denoted as M-edited (M E 11, Al))) We use LLaMa-2-77 (Touvron et al. 2023), Vicuna-7B (Chiang et al., 2023), GPT-turbo-3.5-instruc and (PT--J66BWarn & Komatsuzaki (2021) the LLMs and utilize
all-MiniLM-L12 -i as the encoder in our method. We conducted each experiment four times and reported the average values in the table. All experiments are performed using PyTorch 2.1.2 and RTX 4090 24GB GPU.
5.2 Experimental Results
Non-temporal MQA under KE. We initially test the performance of TEMPLE-MQA for non-temporal MQA, whose results for MQUAKE are shown in Table ] These results clearly
L12-v2
7

---

Published as a conference paper at COLM 2024
MQUAKE-CT-3K
MQUAKE-T
Method
1-edited
100-edited
All-edited
1-edited
All-edited
M-Acc
H-Aoc
M-Ace
H-Aoc M-Acc H-Acc
M-Aoc
H-Acc
M-Aoc
H-Acc
LLAMA-2
FI
28.20
7.3
2.37
0.03
56.48
33.89
1.02
0.37
ROME
13.13
5.37
3.50
0.03
3.63
0.1
24.89
17.99
1.71
0.32
MEMIT
14.97
6.43
9.40
2.47
2.30
0.37
30.89
23.98
25.21
20.13
MeLLo
33,57
9.9
20,00
10,07
17.35
9.9
65.78
55.27
57.69
44.35
PoKeMQA
44.13
30.6
37.35
27.83
32.83
23,87
75.43
60:44
74.36
60.22
TEMPLE-MO (Ours)
68.32
59.46
48.95
35.17
42.20
27.51
77.56
64.87
75.73
62.30
VICUNA-7B
MeLLo
30.20
20.84
24.75
12:25
22.35
10.18
60.72
48.35
31.35
42.97
PokeMQA*
45.83
34.83
38.77
31.23
31.63
25,3
74.37
55.19
23.07
35.09
TEMPLE-MQA (Ours)
71.61
62.75
56.65
44.26
46.60
37.33
81.77
69.46
78.29
68.15
GPT-3
MeLLo
57.43
28.8
40.87
28.13
35.27
25.3
98.12
52.84
74.37
53.33
PaKoMQA"
67.27
56.37
56.00
49.63
48.87
39.77
76.16
68.09
76.98
67.88
TEMPLE- MQLELL--AAAAAAA (Ouns)
78.11
63.45
67.21
55.33
53.68
40.05
90.57
81.90
82.26
74.33
Table 1: Experimental results for MQUAKE. We boldface the best-performing scores with the second underlined. The result from the previous paper is marked as +. By default, the same symbols are used in the following tables.
consistently outperforms the baseline models across different settings by a large margin. For example, when consider the MQUAKE-CF-3K dataset and M-Acc as the evaluation metric, TEMPLE-MQA on average improved by 91.2%, 112.7% and 101.4% compared to Mello for (1, 100, Ald)-edited respectively across three LLMs, and by 42.6%, 32.4%, and 28.6% compared to PokeMQA. We attribute such drastic performance improvement to the following factors: (a) The inference path employed by TEMPLE-MQA is more reliable, which helps the model to boost the end performance significantly; (b) The structural retrieval pipeline (i.e., TAG) designed for TEMPLE-MQA is robust to reduce the number of false positives significantly. We provide analysis of these factors in Section 5.3
show that TEMPLE-MO
Compared TEMPLE-MQA with different LLMs, we can see exhibits superior performance on GPFl-3.-turbo-insttu followed by Vicuna-7B. This is due to the fact that G/T--3.5-turbe
instruct has the strongest reasoning capabilities Chiang et al. (2023), enabling our model to generate more reliable inference paths. Another noteworthy observation in Table iis that parameter-based approaches perform poorly compared to memory-base based ones, which is consistent with the results in (Zhong 2023).
We then consider single-hop question answering with temporal knowledge, whose results for ATOKE are shown in Table Compared to Table the results show varying behavior, with parameter-baset approaches outperforming memory-based methods in some cases, which is contributed by their stronger ability to retain updated knowledge. Specifically, we see that parameter-based approaches (such as ROME and MEMIT) can achieve very good performance for CES and CES-P, indicating that they can effectively inject new knowledge into the model. However, they perform poorly on the HES metric: even after using METO, the average HES is only 30.5%. This indicates that these methods forget historical knowledge after editing.
Single-hop MQA under Temporal KE.
On the other hand, MeLLo and PokeMQA achieved higher HES scores but lower CES scores, indicating they are unable to distinguish historical knowledge and edited knowledge. TEMPLE-MQA can take into account both historical and new knowledge. It not only has higher HES than MeLLo and PokeMQA but also maintains high CES. Specifically, on ATOKE-SE, TEMPLE-MQA achieves HES of 97.25, an improvement of 221 % compared to MEMOTMETO- The reason for such a big improvement is that TEMPLE-MQA can effectively distinguish different knowledge in the same time chain through TAG.
MOA under Temporal KE. We finally consider the performance on TKEMQA, which consist of 2,,,, 3,4) 3-44)-ops equation answering with temporal knowledge. The results are shown in Table 2 We can see that similar to the above results, TEMPLE-MQA achieves good performance for M-Acc. However, TEMPLE-MQA is worse than MeLLo for HES in the All-edited setting using GPT. This is because there is no historical knowledge matching the question in the edit memory in such a setting. MeLLo's utilization of a self-checking method effectively validates the retrieved results, contributing to its superiority.
8

---

Published as a conference paper at COLM 2024
TKEMQA
TKEMOA-HK
Method
1-edited
100-edited
All-edited
1-edited
100-edited
All-edited
M-Acc
HES
M-Ace
HES
M-Ace
HES
M-Ace
HES
M-Acc
HES
M-Acc
HES
VICUNA
Lo
39.02
36.54
31.71
35.56
25.51
32.21
25.76
62.76
18.53
60.26
15.21
56.89
PoKeMQA
36.65
33.12
30.15
31.90
23.02
29.56
22.13
57.83
20.35
56.71
16.94
52.61
TEMPLE-MQA (Ours)
52.18
35.65
50.41
34.62
48.13
34.51
51.92
77.51
50.13
75.59
48.20
75.52
CI---355-
Mel
82.14
37.75
59.17
37.19
47.53
36.37
56.04
23.44
38.22
72.63
31.45
20.75
PoKeMQA
70.18
27.85
53.54
24.55
43.80
20.26
50.27
67.91
33.83
64.37
27.42
60.65
TEMPLE-MQA (Ours)
85.53
41.26
83.47
38.75
80.50
33.11
85.51
82.28
84.12
82.22
80.09
81.61
Table 2: Experimental results TKEMOA.
ATOKE-SE
ATOKE-ME
ATOKE-EE
Method
CES
CES-P
HES
CES
CES-P
HES
CES
CES-P
GPT-J-6B
1-EDITED
FT
5.73
5.69
0.06
1.11
118
0.03
3.41
2.91
MEND
80.47
40.56
1.73
71.83
27.96
0.40
91.94
62.48
ROME
99.99
97.01
2.41
98.85
91.54
0.44
99.93
98.70
MEMIT
99.66
23
2.22
98.42
91.06
0.48
99.92
95.82
FTMETO
2.8
2.62
3.38
1.27
1.2
1.64
MENDMETO
83.26
33.45
30.14
70.52
28.41
28.65
ROMEMETO
99.95
93.78
20.25
99.93
23.22
MEMITMETO"
86.4
85.32
30.31
92.73
85.75
36.2
GPI-J-66
& ALL
EDITED
Mello
83.78
81.55
48.19
60.82
59.15
99.97
PakeMOA
90.91
87 66
62.49
72.73
70.92
40.99
99.87
98.62
TEMPLE-MQA (Ours)
97.95
95.88
97.25
96.46
95.62
96.43
99.92
98.76
Table 3: Experiment results on ATOKE. METO is a plug-and-play method to strengthen the memory of historical knowledge. We mark the METO untested results as "-", ATOKE-ME indicates each data includes two edits to perform continuous knowledge updating.
Moreover, in contrast to the previous two datasets, find that TEMPLE-MQA TKEMQA experiences only a slight performance decrease when the edit batch size increases. We further conduct an experiment to study the performance w.r.t. difference edit numbers (see Figure in Appendix). the number of edits increases, the performance of TEMPLE-MQA drops rapidly on the MQUAKE-CF-3K We find that this is because MQUAKE-CF-3K has unpassable data in the mass-edit setting. Specifically, there are data that are affected by the editing of other data. Under M-edited (M > 1) settings, if they are in the same batch, unpassable data will be generated (see Appendix Table 15 for an example). To verify this, we filter out conflicting data and create new data MQUAKE- MKE-CCF33-FFFL repairing, we can observe that TEMPLE-MQA is more stable than other methods, which illustrates the accuracy of structural retrieval.
can be observed from Figure 4 (a) and Figure 4 (b) that as
The result is shown in Figure (c). After
5.3 Abalation Study
We first conduct an ablation study on different modules of the structural retrieval: without filtering sub-graphs according to the subject entity (i.e. - wwo Subject), without considering the similarity of relation the Re-ranking step (i.e. w/o Relation), without considering the similarity of the concept of object entity (i.e. w/o Concept).
The results are shown in Table in Appendix. First, we investigate the role of filtering sub- graphs. We can see without filtering out the sub-graph through the subject, TEMPLE-MQN decreased by an average of 8.6% and 7.8% on M-Ace and H-Acc. This is the most impactful component among all removals. Second, the performance of TEMPLE-MQA in all the settings, indicating its necessity. Third, the impact of all eliminated components will become larger as the edit memory increases.
can see both relation and concept can improve
Subsequently, we conduct an ablation study on inference path planning module, with details as follows: (1) Replacing inference path planning with sub-question decomposition by Zhong et al. (2023) (i.e,, w /o inference path). (2) Exploring the influence of k-shot (k E (1,2,4)) in-context learning prompt.

---

Published as a conference paper at COLM 2024
GPT-3 S-T.DO INSTRUCT
TU-RRO INSTRUCT
GFT INSTRUCT
89
TEA
POKAMAA
Nello
50
TEUPI ViR--- COMPE VORES AdlAPRE
14.0 144
60
1941
M-ACC
TEHPLE
50
40
3300
1000
2000
3350
1100
500
1:000
2002 1100
3000
2003
3000
Suith
Figure 4: The line graph depicting the decrease in M-Acc with the increase in edit batch size on datasets MQUAKE-CF-3K, TKEMQA and MQUAKE-CF-3K-FIX
Corresponding results are shown in Table 11 in Appendix. These results show, if we exclude the inference path planing, the M-ACC drops by almost 15.44% 14.39% 9.53% respectively. Also, the M-ACC drops about 10.43%, 7.77% and 3.51% for k-shot (k E {1,2, 4)) settings. Thus more examples can help LLMs follow instructions better, also the quality and diversity of the examples is important.
6 Conclusion
We presented a novel framework TEMPLE-MQA for MQA under temporal knowledge editing. TEMPLE-MQA first constructs a time-aware graph, then utilizes inference paths, structural retrieval, and joint reasoning, capable of enhancing MQA performance while distinguishing the temporal context. We also proposed benchmark TKEMQA, which is the first benchmark specified for MQA with temporal scopes. Experiments on TKEMQA and the existing two benchmarks demonstrated that MMMPLE-MQA outperforms previous methods.
Limitations
Our work poses following limitations and we will resolve them in our future works:
Temple-MQA relies TAG, which brings additional computational overhead.
Temple-MQA requires writing prompts manually to adapt to different tasks.
Although retrieval of Temple-MQA doesn't need fine-tuning an encoder, it has complex workflow.
Acknowledgements. Di Wang, Lijie Hu and Muhammad Asif Ali are supported in part by the baseline funding BAS/ 1 11689-00--- funding from the CRG grand JREF//4663-01-00 WCC/1/1976-49/ from CBRC and funding from the Al Initiative REI/1/ 1/11111118110000 of King Abdullah University of Science and Technology (KAUST). Di Wang is also supported by the funding of the SDAÍA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIÁ-KAUST AI). We also acknowledge support from OpenAl API Researcher Access Program.
10

---

Published as a conference paper at COLM 2024
References
Muhammad Asif Ali, Yifang Sun, Bing Li, and Wei Wang. Fine-grained named entity typing over distantly supervised data based on refined representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34(05), pp. 7391-7398, 2020.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https
sapp..sematticcchool.: org/ atgg/orrpusI:::332241111
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Conference on Empirical Methods in Natural Language Processing, 2020. URL
:
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
00-0003-300i000 vicuna/
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. ArXiv, abs/2307.12976, 2023. URL
https://api. commmntiicccooa.... org Cats/CorpsSI:::: 26035661
Hengrui Gu, Kaixiong Zhou, Xiaotian Han, Ninghao Liu, Ruobing Wang, and Xin Wang. Pokemqa: Programmable knowledge editing for multi-hop question answering. arXiv preprint arXivv2312.15114, 2023.
Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adap- tors. ArXiv, abs 2b112211.1 2022. URL htttps://app semanti wwwpcchhlaarrrr//eeooooo Corg/srr
model
253735429
Peter Hase, Mona T. Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srini Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. ArXiv, abs/2111.13655 2021. URL https
//api.semanticscool.. hwwpccchllrrrrrooeoooooooo org/Corpus :ttp//ooppuIID2247666655 244709666
Ruixin Hong, Hongming Zhang, Honghui Zhao, Dong Yu, and Changshui Zhang, Faithful
question answering with monte-carlo planning. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https::/apl semanticschole .rg. Prrg/CorpsIID
2584799541
Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Wilke: Wise-layer 2024. URL
knowledge editor for lifelong knowledge editing. ArXiv,
httpppp//pp....ematatccccooooooo wttp:a/cooooa..ooooooooooaa:D::. 267771100
Nathan J. Hu, Eric Mitchell, Christopher D. Manning, and Chelsea Finn. Meta-learning
online adaptation of language models. ArXiv, abs/2305.15076 2023. URL h/aps://ppp
semanticscholar org/ Corpus : 258866057
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language mod- els: A survey. ArXiv, abs/ abs//212. 2242. 2022.
org/
CorpusIDD:25887755
Youcheng Huang, Wenqiang Lei, Zheng Zhang, Jiancheng Lv, and Shuicheng Yan. See the
unseen: Better context-consisten knowledge-editing by noises. ArXiv, abs//400.075444
2024. URL http////p////pppp swwanticscholar....... org/ Corpus :669999110
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answer- ing. ArXiv, 0bss/2004.04906, 2020. URL https: //api. semanticscholar Cr
215737187
11

---

Published as a conference paper at COLM 2024
Tushar Khot, H. Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving com- plex tasks. ArXiv, abs//220.022066, 2022. URL https://api semanticscholar org/
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Pre- cise model editing in a transformer. ArXiv, abs/2308.08742, 2023. URL https://api
semanticscholar orgg/Carpuu1D::21100002
Xiao Ling and Daniel Weld. Fine-grained entity recognition. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 26(1), pp. 94-100, 2012.
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. ArXiv, abss/230.0000/) 2023. URL https://appi http:/ppp..smmaanccccccll.o..a wttp:ooooo//ooo.0003300000
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372,
2022a.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in transformer. In The Eleventh International Conference on Learning Representations, 2022b.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Man- ning. Fast model editing at scale. ArXiv, abs/2110.11309, 2021. URL https://apt
semanticscholast Orgg/orpusIDD :2:300000003000330 239050360
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory-based model editing at scale. ArXiv, abs/ 22060022000620,,, 2022. URL httphhtts://at
semanticscholar. org/ Org/CorpusID- :49642147
Van L Parsons. Stratified sampling. Wiley StatsRef: Statistics Reference Online, pp. 1-11, 2014.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/ //////337000999 2023.
URL https: //api. semanti cacholar org/
Ben Wang and Aran Komatsuzaki. GPT-J-6B: Billion Parameter Autoregressive Lan- guage Model. httpps://githu http:///ggttuu...mm///nnoooo../ orgp://ioooommmssstttrrrtttrorrr jax May 2021.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Annual Meeting of the Association for Computational Linguistics, 2023a. URL httpp::/aapiismmmaaiiiicoo
www.aaaroooroooooaa..o... .rg/ CorpusID :
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, et al. Knowledge editing for large language models: A survey. arXiv preprint arX1v:2310.16218,, 2023b.
12

---

Published as a conference paper at COLM 2024
Yiwei Wang, Muhao Chen, Nanyun Peng, and Kai wei Chang. Deepedit: Knowledge editing as decoding with constraints. ArXiv, abs/2401.10471, 2024. URL /
semanticscholar. org/ :etp/CorpsII:::21100 267060997
Siheng Xiong, Yuan Yang, Faramarz Fekri, and James Clayton Kerce. TILP: Differentiable learning of temporal logical rules on knowledge graphs. In The Eleventh International
Conference on Learning Representations, 2023. URL https://opeerrrieen wttprrvvviiieeeefrrrnnne.
id [==_X12Nm0KvX
Siheng Xiong, Ali Paayani Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.0663 2024a.
Siheng Xiong, Yuan Yang, Ali Payani, James C Kerce, and Faramarz Fekri. Teilp: Time prediction over knowledge graphs via logical reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16112-16119, 2024b.
Xunjian Yin, Jin Jiang, Liming Yang, and Xiaojun Wan. History matters: Temporal knowledge editing in large language model. ArXiv, abs /2s12232.55497, 2023. URL semammiccctccicooo....
org CooppssID:261164200
Ningyu Zhang, Yunzhi Yao, Bo Tian, Peng Wang, Shumin Deng, Meng Wang, Zekun Xi, Shengyu Jintina Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiao-Jun Zhu, Jun Zhou, and Huajun Chen. A comprehensive study of knowledge editing for large language models. ArXiv, abs/2401.01286, 2024. URL httpss/////ap sewanti semanticscholar org :Bc/CorpusID:2667777 266725300
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. A survey of large language models. ArXiv, abs/2303.18223, 2023. URL mattps://app www.stiiccoollllaa.......ccot org/ Corpus 257900969
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? ArXiv, abs//305.12774) 2023. URL
https://api. semanticscho aar org/ 25:88334477
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.147) 14795, 2023.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. Modifying memories in transformer 2020. URLhttps: happs://app //api. httpsschh.aaarrrrrrrrrsssa::::::
monsstomerrmdeeee models. ArXiv, abs/2002.00363,
13

---

Published as a conference paper at COLM 2024
A Details of Prompts in TEMPLE-MQA
[Demo]
Unstructured fact: From 2009 to 2017, Obama was the president of the United States,
Structural form: <United States, the president is, person, 2009, 2017, Obama>
Unstructured fact: From 2019 to 2022, the head of government in kazakhstan is Askar Mamin. Structural form: <Kazakhstan, head of government, person, 2019, 2022, Askar Mamin> ...Other demo
[Instruction]
Please refer to the above demo and convert the following edit into the structural form: <subject, relation, type of object, start time, end time, object> [Task]
<Fact>
Table 4: The prompts Peonvert to convert unstructured knowledge used in Equation
[Demo] include several high-quality demonstrations handwritten by humans. <Fact> indicates the unstructured fact fx. More examples are shown in Table 19
[Demo]
Question: Who was the owner of Tom's company from 2018 to 2020?
Inference path: <Tom, company company>, <company, owner is, people>
Time: 2018, 2020
Question: What the religion of the head of government of Israel after 2022? Inference path: <Israel, head of government, person>, <person, religion is, religion>
Time: 2022, None
...Other demo...
[Instruction]
Please refer to the above demo and generate a valid inference path for the following question: [Task]
<Question>
Table The prompt Ptan to extract inference path used in Equation ? where <Question> indicates the question Q.
[Demo]
Inference step: <United Kingdom, head of government, person
Time: from 2019 to 2022
Answer: Boris Johnson
Inference step: <Christian Wulff, spouse, person
Time: after 2023
Answer: Bettina Wulff
...Oher demo...
[Instruction]
Please generate valid answer like the above example for the following task: [Task]
<Inference step>
<Time>
Table 6: The prompt Pquery for querying LLM to generate answer used in Equation where <Inference step> indicates specific inference step in the inference path, <Time> indicates temporal scope.
B Experimental Details
B.1 Dataset
We provide detailed description of the evaluation datasets follows.
14

---

Published as a conference paper at COLM 2024
Datasets
#Edits
2-hop
3-hop
4-hop
Total
513
356
224
1093
334
246
1067
MQUAKE-CF-3K
310
262
572
268
268
1000
1000
1000
3000
MQUAKE-T
1421
445
2
1868
Table 7: Statistics of the MQUAKE dataset.
Datasets
ATOKE-SE
ATOKE-ME
ATOKE-EE
Size
8819
8820
8819
Table 8:
Statistics of the ATOKE dataset.
(a) MQUAKE [hong et al. 2023). MQUAKE includes MQUAKE-CF-3K based on coun- terfactual editing and MQUAKE- based on real-world changes. These datasets encompass k-hop questions (k (2,3, 4)), each associated with one more edits. Statistics are shown in Table 7
(b) ATOKE (Yn et al. 2023). ATOKE is the first temporal knowledge editing dataset containing a series world knowledge with timestamps, regarded as a series of knowledge updates. ATOKE contains three editing types: single edit (SE), multiple edits (ME), and extending edit (EE), corresponding to three datasets ATOKE-SE, ATOKE-ME and ATOKE- EE. Statistics of these three datasets are shown in Table g
ATOKE-SE. Each data of this dataset includes one edit for knowledge updating and historical question and current question corresponding to the time scope before and after the modification.
ATOKE-ME. Each data of this dataset includes two edits to perform continuous knowledge updating, two historical questions, and one current question.
ATOKE-EE, Each data of this datasets include one edit to modify the time scope of knowledge (not updating the object), and only one current question.
is worth explaining that the above three datasets are all aimed at single-hop questions only.
(c) TKEMOA. TKEMQA is our newly curated benchmark dataset. It is a blend of carefully crafted multi-hop questions with explicit temporal scopes. This data set is designed to eval- uate the ability of KE methods rigorously. TKEMQA-HK provides additional knowledge
corresponding to two historical questions on the basis of TKEMQA, which is used to explore whether the memory-based method will confuse the temporal context. See Section D for detailed introduction, construction process, and statement temples of TKEMQA.
Datasets
#UK
#CK
#HK
1-hop
2-hop
3-hop
4-hop
Total
834
TKEMΩA
1
166
500
500
1166
All
500
500
500
500
2000
TKEMOA-HK
1
166
500
500
1166
All
500
500
500
2000
Table 9: Statistics TKEMQA and TKEMQA-HK. #UK represents the number of updated knowledge, #CK represents the number of corrective knowledge and #HK represents the number of historical knowledge. TKEMQA-HK include two questions asking about historical knowledge which TKEMQA does not include.
15

---

Published as a conference paper at COLM 2024
B.2 Baseline Models
(a) Parameter-based. The parameter-based baselines include: (i) Fine-tuning (FT) (Zhu et al. 2020) that performs a gradient-based update the model parameters to incorporate new knowledge. (ii) ROME (Meng et al. 2022a) first locates factual knowledge at specific layer of the transformer architecture and then updates the feed-forward network of this layer to insert new knowledge. (iii) MEMIT (Meng et al. (022b) extends ROME to allow modifying (iv) MEND Mitchell et al. 2021 learns a hyper network to produce weight updates by decomposing the
range of feed-forward network layers for a large amount of knowledge.
gradient of standard fine-tuning into a low-rank form. (v) METO (Yin et al. 2023) extends
the parameter-based methods by adding the time information as an optimization objective.
(b) seeorry-based The memoryybased baselines include: (i) MeLLo (Zhogg et al. 2023)
that uses the plan-and-solve paradigm. (ii) PokeMQA (Gu et al. 2023) extends MeLLo
by adopting a two-stage retrieval process to decouple the question decomposition and knowledge editing.
B.3 Evaluation Metrics
Details about the evaluation metrics and their mathematical formulation are provided as follows:
(i) Multi-hop Accuracy (M-Acc), which is used to measure the accuracy of the language models on multi-hop questions. For M-Acc, we use the same settings as Zhong al. (202)). The calculation formula for M-Acc is as follows:
1
( =d a*)
(8)
Where f*(+) represents the edited model, Q and a' represent the multi-hop questions and the edited answer for each case, respectively.
(ii) Hop-wise Accuracy (H-Acc), which is used to check the correctness of the intermediate reasoning path for MQA. For H-Acc, we follow the same settings proposed by Gu et al. (2023). Given edited chain of facts C*, H-Acc is defined
A (f,)s)) =0] o0]
(9)
(iii) Historical Explicit time Question Score (HES), which is the accuracy of explicit tempo- ral questions about historical knowledge (Yin et al. 2023).
(iv) Current Explicit time Question Score (CES), which measures the accuracy of explicit temporal questions about current knowledge (Yin et al. 2023).
(v) Current Explicit time Paraphrase Question Score (CES-P), which is the accuracy of explicit temporal questions of semantic paraphrasing asked about current knowledge (Yin) et al. 2023).
HES, CES, and CES-P require the model to be able to recall knowledge at a specific temporal scope, calculated according to the following formula:
If (s, s, Ts, Te) =].
(10)
C More Experiment Results
Data Conflict in MQUAKE. In the analysis for MQA under Temporal KE, we observed that TEMPLE-MQA on TKEMQA experiences only slight performance decrease when the edit batch size increases, while its decrease is faster for MQUAKE. This is because MOUAKE-CF-SE has unpassable data in the mass-edit setting, i.e., observe data
KE,
16

---

Published as a conference paper at COLM 2024
MQUAKE-CF-3K
MQUAKE-T
Method
I-edited
100-edited
All-edited
1-edited
All-edited
M-Ace
M-Acc
H-Acc
M-Acc
H-Acc
M-Acc
H-Acc
M-AAc
H-Acc
-w/o Subject
77.96
62.85
62.06
51.16
44.09
34.23
89.56
80.77
73.97
66.56
wwo Relation
77.78
62.67
65.15
52.88
49.53
36.51
84,72
80.80)
77.13
68.49
w/o Concept
78.08
63.02
65.70
53.29
50.10
37.55
90.01
81.35
78.67
70.20
TEMPLE-MQA
67.21
53.68
40.05
81.90
74.33
Table 10: Ablation study result of TEMPLE-MQA.
MQUAKE-CF-3K
Method
1-edited
100-edited
All-edited
W/o inference path
62.67
52.82
44.33
I-shot
65.71
55.12
46.87
2-shot
69.48
38.72
47.48
4-shot
73.93
64.63
49.91
TRMPIF-MQAA6-shoo)
78.11
67.21
53.68
Table 11: The ablation result of inference path planning on MQUAKE-CF-3K with metric of M-Acc.
ATOKE
Method CRS(SE)
HRS(SE)
CRS(ME)
HRS(ME)
MEND+ 25.41
30.17
21.84
30.83
ROME+
78.68
16.29
62.40
18.18
MEMIT+ 74.07
24.32
73.58
26.04
TEMPLE-MQA 85.81
78.23
87.74
80,92
Table 12: The results on the AToke-SE and AToke-ME.
2-hop
3-hop
4-hop
Method
Cost(S)
Time(s)
Cost($)
Time(s)
Cost(S)
Time(s)
Mello
0.73
8.98
0.94
12.13
1.61
16.41
PokeMQA
0,93
10.33
1.09
11.98
1.36
18.85
TEMPLE-MQA 0.15
2.50
0.25
5.31
0.27
6.98
Table 13: Average expense and time cost comparison of methods on the MQUAKE-CF-3K dataset for solving per multi-hop problem in the same experimental environment.
1-edited
100-edited
All-edited
Method
R-Acc
M-Ace
H-Ace
R-Ace
M-Acc
H-Acc
R-Ace
M-Ace
H-Acc
Mello
91.6
57.43
28.8
83.1
40.87
28.13
74.5
35.27
25.3
PokeMOA
95.2
67.27
56.37
89.5
56.00
49.63
81.5
48.87
39.77
TEMPLE-MQA 99.5
78.11
63.45
93.2
67.21
55.33
87.8
53.68
40.05
Table 14: Retrieval accuracy of different methods on the MQUAKE-CF-33 dataset. R-Acc refers to the accuracy of retrieving edited facts from memory.
conflicts in MQUAKE. These conflicts arise from inconsistent counterfactual edits between two cases containing the same knowledge (i.e., one is edited, but another is not edited). Under mass editing settings, the inference process of one case may be influenced by the editing of another case, leading to incorrect answers. Table 15 proovidee an example of data conflict in MQUAKE.
Cost of Inference Path. Here, we aim to compare the expense and time cost of our in- ference path to those of previous methods. According to Table 13 we can easily see that, for MOUAKE-CF-3K TEMPLE-MQA averagely saves 78.7% and 80.4% of the expense compared to Mello and PokeMQA, respectively, and reduces the inference time by 62.9% and 64.8%. The reason is that TEMPLE-MQA only needs to call LLMs once to generate
17

---

Published as a conference paper at COLM 2024
TKEMQA all-edited
MQUAKE CF- 3K 1-edited GPT-35-TURBO-INSTRUCT
GPT-3.5-TURO-INSTTUUG
100
100
96.599.88
TEMPLE-MOA
90.5
TEMPLE MOA
MELLO POKEMOA
84.9
MULLO
POKEMQA
76.2
73.9
73.6
70.3
M M-ACC(%)
M ACC(0%)
0167Z
48.6
40.2
42.1
34.3
29.1
20
13.913.5
1-hop
2-hop
3-hop
4-hop
2-hop
3-hop
4-hop
Figure 5: A bar graph comparing the M-Acc of three methods under two settings: All-edited for TEMPLE-MQA and 1-edited for MQUAKE-CF-3K.
the inference path, while Mello and PokeMQA need call LLMs multiple times gener- ate sub-questions. Thus, we claim TEMPLE-MQA offers faster and more cost-effective solution.
Accuracy of Structured Retrieval. To achieve reliable KE, apart from the correct decom- position sub-problems, another important factor is whether related edits can be found correctly. Here, conduct experiments to compare the accuracy of the retrieval process for different methods. According to Table 14 TEMPLE-MO B-MAA achieve higher retrieval accuracy compared to Mello and PokeMQA by 17.9% and 7.7% in the all-edited setting, respectively. Thus, our structured retrieval can efficiently find structured edits.
TEMPLE-MQA under Different Hop Numbers. Here, aim to study the performance of different models with different hop numbers. As shown in Figure 5 TEMPLE-MQA can still maintain high M-Acc as the number of hops increases, while Mello and PokeMQA experience an obvious decline in performance with an increase in hop number. This is because Mello and PokeMQA couple the tasks of querying the LLM to answer sub-questions and generate sub-questions together. As the hop number increases, the LLM is burdened with complex tasks, causing a significant decrease M-Acc. In contrast, TEMPLE-MQA decouples the generation of inference paths, enabling it to maintain high performance as the hop number increases.
TEMPLE-MQA under Implicit Time. For questions with hidden time, we consider the following examples: 1) "Who was the U.S. president before Biden?", 2) "Who was the previous president of the U.S.?".
For example 1, TEMPLE-MQA needs to determine the specific time when Biden is president. We can revise the ICL prompt to achieve it, as show in below:
Question: Who was the U.S. president before Biden?
Time: Biden is president after 2021, Thus time span is <2017 2020,
Inference Path: <resddree is.?>.
However, we observe some datasets contain questions that are more ambiguous, as shown in examples 2. Answer to these questions is decided on the training periods of LLMs. For this, we need to give current time to LLM,, as shown below:
Question: Who was the previous president the U.S.?
Time: Today is 2021, and Biden is current president, Thus time span is <0012 2017,2020>
Inference Path: presddnee is,?>
The results the AToke-SE and AToke-ME with ambiguous hidden time (like example 2) are shown Table I2 We can see that TEMPLE-MQA still performs better, especially on the HRS metric.
18

---

Published as a conference paper at COLM 2024
DATA 998 MQUAKE-CF-3K
Questions
What continent does Gareth David-Lloyd hold citizenship in?
which continent was the actor Careth Lavid citizen of?
Which continent does the citizenship of Gareth David-Lloyd belong to?
Edits
(Gareth David-Lloyd, country of citizenship, Nigeria)
(Nigeria, continent, North America)
DATA 706 IN MQUAKE-CF-3K
Questions
What continent does the country Peter Green belong to citizenship to belong to?
Which continent is the country of citizenship of Peter Green?
On which continent is the country where Peter Green holds citizenship located?
Edits
(Peter Green, country of citizenship, Nigeria)
ORIGINAL INFERENCE PROCEDURE IN DATA 706
Inference procedure
(Peter Green, country of citizenship, Nigeria)
(Nigeria, continent, Africa)
Original Answer
Africa
DISTURBED INFERENCE PROCEDURE IN DATA 706
Inference procedure
(Peter Green, country of citizenship, Nigeria) (Nigeria, continent, North America)
Wrong Answer
North America
Table 15: Data Conflict in MQUAKE-CF-3K. Under the setting of mass editing, step (highlighted in blue) in the original inference procedure in Date 706 is changed due to retrieving an edit in Data 998 (highlighted in red), resulting in a wrong answer.
D Temporal Knowledge Editing for Multi-hop Question Answering Benchmark: TKEMQA
QUESTIONS
Current question
which continent is country where the head government Taiwan has citizenship located after 2023?
Historical questions
Who is the head of government in Taiwan from 2019 2023? Who is head of government Taiwan from 2017 2019?
KNOWLEDGE STORE IN MEMORY
Corrective knowledge
(Chen Chien-jen, country of citizenship, Taiwan -> Algeria)
Updated knowledge
(Taiwan, head government, Chen Chien-jen, 2023, N/A)
Historical knowledge
(Taiwan, head of government, Hope Su, 2019, 2023)
(Taiwan, head of government, Lai Ching-te, 2017, 2019)
PROCEDURE TO ANSWER CURRENT QUESTION
Inference procedure
(Taiwan, head government, Chen Chien-jen)
(Chen Chien-jen, country of citizenship, Algeria)
(Algeria, continent, Africa)
ANSWERS
Current answer
Africa
Historical answers
Hope Su
Lai Ching-te
Table 16: An instance in TKEMOA-HK. Updated knowledge refers to the updating of outdated knowledge and does not deny the correctness of historical knowledge. Corrective knowledge refers to the subversion of model's past knowledge used to simulate the model's modification of erroneous knowledge. The updated knowledge and corrective knowledge required in the inference procedure are highlighted in red and blue, respectively.
19

---

Published as a conference paper at COLM 2024
D.1 Introduction to TKEMQA
Our benchmark TKEMQA is created based on Wikidata knowledge base consisting of fact triples associated with millions of entities. TKEMQA can be used to evaluate whether the knowledge editing method can update new knowledge without forgetting past knowledge. contains simple single questions and multi-hop questions, and its rich settings enable it to comprehensively evaluate knowledge editing methods.
As shown in Table TKEMQA contains 2,000 data where each one includes a k-hop (k (1,2,3,4) ) questions. As GPT-3.5 does not have the knowledge after 2022, use the knowledge after 2022 as an edit to update knowledge. Thus, we refer to each multi-hop question current question because it has time prompt after 2022. Besides the updated knowledge, we also introduce the corrective knowledge, which is used to modify the error historical knowledge. This knowledge does not need temporal scope and is counterfactual.
To evaluate whether the editing method may cause model forget the historical knowledge (i.e. knowledge of model before the editing), each data of TKEMQA include two historical question with the time prompt before 2022. Based on TKEMQA, we add the knowledge corresponding to the historical question into the edit memory evaluate whether the retrieval can effectively distinguish different knowledge in the same time chain, in which we refer to as TKEMQA-HK.
We use M-Acc and HES as two metrics in TKEMQA. M-Acc is the accuracy of the language models on current multi-hop questions, evaluating the model's ability to update knowledge and perform multi-hop reasoning successfully. HES is the accuracy of answering historical questions and assessing the model's ability to recall historical knowledge.
D.2 TKEMQA Data Curation
Collect relation template. We first manually collect multiple relations containing time information from WikiData, e.g. "President is" (Appendix D.3). Then, we use SPARQL to sample facts based on the relation and subject entity. For example, given query (U.S., President is, 2), WikiData will return every president of the U.S. and their terms of office. In addition, we select common relations and entities to control quality.
Generate relation path template. In order to generate natural and coherent multi-hop questions, we look for high-quality relation path Luo et al. (2023) with different lengths k E (1,2,3,4) 4). (live in, the father is) is bad path because the object type of 'life in' (location) does not match the subject type of 'father is' (person); a location cannot have father. Thus, arrange and combine the collected relations into relation path template and filter out unreasonable relation combinations. For each relation path, we can get several fact chains based on different start point entities.
Generate multi-hop questions. Then, we use GPT-4 to convert the relation path template into questions in natural language form, e-g., (father is, company is) -> "What is the company of father of ()?". Replace "()" into the start point entity to get the final questions. We also conduct manual revisions ensure its quality. Different from existing work (Zhong et al. 2023), this method produces data with higher quality questions due to the powerful capabilities of GPT-4. This method also helps reduce the additional overhead of using GPT-4 because one path often correspond to multiple data items.
Additional details for edits. Among the collected fact chains, there is only one piece of knowledge that is after 2023, which is used to represent the knowledge update of the real world.
D.3 Question/Clozz Statement Templates used in TKEMQA
Following previous work, question templates to filter the facts that cannot be recalled and cloze-style statement templates to convert an edited fact into a natural language form
20

---

Published as a conference paper at COLM 2024
statement. Table 17 shows questttooooon templates and cloze-style statement templates employed
in TKEMQA.
Relation
Question template
Cloze-style statement template
P35
Who is the head of state in [S]?
The head of state in [S] is
P6
Who is the head of government in [S]?
The head of government in [S]
P488
Who is the chairperson of [S]?
The chairperson of [S] is
P169
Who is the chief executive officer of [S]?
The chief executive officer of [S] is
P54
Which sports team is [S] affiliated with?
[S] is affiliated with the sports team of
P286
Who is the head coach of [S]?
The head coach of [S] is
P551
Where does [S] live?
[S] live in
P102
Which political party is [S] affiliated with?
[S] is affiliated with the political party of
P26
Who is [S]'s spouse?
[S]'s spouse is
P38
What is the currency of [S]?
The currency of [S] is
P108
Which organization is [S] an employee of?
[S] is employee of
Which university is [S] educated aduucatedat?
P69
at?
[S] is educated at
P937
Where is [S]'s workplace?
The work location of [S]
P36
What is the capital of [S]?
The capital of [S] is
P159
Where is the headquarters of [S] located?
The headquarters of [S] is located in
P27
What the country of citizenship of [S]?
[S] is citizen of
P140
Which religion is [S]affiliated with?
[S] is affiliated with the religion
P30
Which continent [S] located in?
[S] is located the continent of
P37
What the official language of [S]?
The official language [S]
P17
Which country is [S] located in?
[S] is located the country of
Table 17: Question templates and cloze-style statement templates employed in TKEMQA. "[S]" denotes placeholder for the subject entity within the fact. The question templates filter the facts that cannot be recalled. The cloze-style statement templates convert an edited fact into natural language form statement.
E SPARQL Protocol and RDF Query Language
SPARQL is able to retrieve and manipulate data stored in Resource Description Framework (RDF) format, which can store graph information. Wikidata Query Service (WDQS) is an online tool for querying the Wikidata database. It allows users to retrieve and analyze structured data Wikidata using the SPARQL query language. We use SPARQL to access WDQS to obtain the alias of the subject entity and create our dataset TKEMQA (see the table
18
21

---

Published as a conference paper at COLM 2024
Using SPARQL to extract aliases
SELECT Palias WHERE |
wd:<QID> skossaltLabel Palias.
SPARQL
FILTER(LANG(Paias)) "en")
This SPAROL query retrieves English alias specific entity, "<QID>" denotes placeholder for the entity id in Wikidata.
Description
Using SPARQL to extract facts
SELECT Label WHERE
wd:<QID> wdt:<PID>>>>> ?x. ?x rdfs:label ?Label.
SPARQL
FLLTER(LANG((Labe))) "en")
This SPARQI used to retrieves corresponding object entity according to the subject entity (represented by "<QID>") and relation id (represented by "<PID>").
Description
Using SPARQL to extract facts with temporal scope
SELECT 2label ?start time lend_time WHERE |
wd:<QID> p><PID> ?statement.
?statement ps><PI>> ?x. ?x rdfs:label ?Label.
SPAROL
OPTIONALI ?statement pq:P580 ?start_ time.
OPTIONALI ?statement pq:1582 feme_time.) FLLTER(LAN((LLab)] "en").
ORDER BY DESC (?start_time)
This SPARQL query is used to extract object entities and their corresponding time scopes based on subject entity id (denoted by "<ID>>)) and relation id (denoted by "<PID>>)).
Description
Table 18: The SPARQL we used to create TKEMQA.
Unstructured Form
Structured Form
From 1993 to 1999, Donald Trump's is spouse Marla Maples. In 2020, the head coach of AC Horsens is Jonas Dal. After 2023, the head govemment Slovakia Robert Fico. Before 1936, the head state in United Kingdom is George V. The capital United Kingdom is London.
(Donald Trump, spouse, person, 1993, 1999, Marla Maples) (AC Horsens, head coach, person, 2020, 2020, Jonas Dal) (Slovakia, head govermment, person, 2023, (United Kingdom, head of state, person, N/ A, 1936, George V) (United Kingdom, capital, city, N/A, N/A, Londo)
Robert Fico)
Table 19: Examples of transforming knowledge from unstructured form into structured form. The N/A indicate unknown for start time or end time..
22


Reviews:
[{"review_id": "YrAs4mL55X", "reviewer": "Reviewer_JhSs", "summary": "This paper proposed a temporal knowledge augmented multi-hop question answering method called Temple-MQA. This method built a time-aware graph to store edits as a time-conscious knowledge structure. The benefit of this graph is to store the temporal information in a structured format. During the retrieval, Temple-MQA utilized a LLM to devise a path and do a step-by-step reasoning. It is interesting to use LLM to strategize an inference path. In general, the whole idea is clear with a reasonable performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We address your concerns as follows. \n\n **W1. Implicit Time**\n\nFor questions with hidden time, we consider the following examples: \n\n* \"Who was the U.S. president before Biden?\",\n* \"Who was the previous president of the U.S.?\"\n\nFor example 1, TEMPLE-MQA needs to determine the specific time when Biden is president. We can revise the ICL prompt to achieve it. For example, \n\n```\nQuestion: Who was the U.S. president before Biden?\nTime: Biden is president after 2021, Thus time span is <2017,2020>.\nInference Path: <U.S.,president is,?>\n```\n\nHowever, we observe some datasets contain questions that are more ambiguous, as shown in examples 2. Answer to these questions is decided on the training periods of LLMs. For this, we need to give current time to LLM. \n\n```\nQuestion: Who was the previous president of the U.S.?\nTime: Today is 2021, and Biden is current president, Thus time span is <2017,2020>.\nInference Path: <U.S.,president is,?>\n```\n\nThe results on the AToke-SE and AToke-ME with ambiguous hidden time (like example 2) are shown below:\n\n| Method     | CRS(SE) | HRS(SE) | CRS(ME) | HRS(ME) |\n| ---------- | ------- | ------- | ------- | ------- |\n| MEND+      | 25.41   | 30.17   | 21.84   | 30.83   |\n| ROME+      | 78.88   | 16.29   | 82.40   | 18.18   |\n| MEMIT+     | 74.07   | 24.32   | 73.58   | 26.04   |\n| TEMPLE-MQA | 85.81   | 78.23   | 87.74   | 80.92   |\n\nWe can see that TEMPLE-MQA still performs better, especially on the HRS metric.\n\n**W2. Inference Path** \n\nFor this, we add two experiments: \n    * Ablation Study: remove inference path planning. \n    * Explore the influence of the number of shots on the ICL prompt.\n\nHere are our results \n\n| Method              | 1-edited | 100-edited | all-edited |\n| ------------------- | -------- | ---------- | ---------- |\n| TEMPLE-MQA(8-shot)  | 78.11    | 67.21      | 53.68      |\n| -w/o inference path | 62.67    | 52.82      | 44.33      |\n| 1-shot              | 65.71    | 55.12      | 46.87      |\n| 2-shot              | 69.48    | 58.72      | 47.48      |\n| 4-shot              | 73.93    | 64.63      | 49.91      |\n\nIf we exclude the inference path, the M-ACC drops about 15.44%, 14.39%, 9.53% respectively. The M-ACC are drops about 10.43%, 7.77% and 3.51% in 1,2 and 4 shot setting. Thus more examples can help LLMs follow instructions better, also the quality and diversity of the examples is important.\n\nWe will add the above findings in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\n1.\tThis paper presented a temple-MQA method that achieves a high accuracy for multi-hop temporal QA task. The whole architecture includes the following key steps: 1) construct a time-aware graph (TAG) to store time information; 2) build an inference path for each multi-hop question using LLMs; 3) design a joint reasoning way to combine LLMs with TAG.\n\n2.\tThe performance of proposed temple-MQA achieves the state of art performance on multiple metrics. It proves the effectiveness of proposed method.", "reasons_to_reject": "Weaknesses:\n\n1.\tThe temporal questions include two types: questions with explicit time (like 01/01/2024) and questions with hidden time (like before, after). It will be wondering if more deep analysis cross different types are provided. The questions with hidden time will require more support about multi-step reasoning. Maybe this method can perform well on various types.\n\n2.\tUsing the LLMs to generate the inference path is a great idea. But this section missed some details. I would like to see more comparisons and analysis about this direction. The accuracy of inference path will impact the performance a lot.", "questions_to_authors": "Look forward to more analysis on various temporal question types.", "extra_scores": {}}, {"review_id": "6arG3YOFH7", "reviewer": "Reviewer_EvWU", "summary": "The proposed method introduces a novel approach for handling multi-hop question answering that involves substantial volumes of temporal knowledge edits. The proposed approach consists of 1) extending a fact triple knowledge graph with timestamps (referred to as a time-aware graph or TAG by the authors). Such structured information can then be converted into unstructured text by an LLM for Data Augmentation and Contextual Filtering to enhance retrieval performance. Given these improvements, they also propose using LLM and TAG for chain-of-thought type of step-by-step joint reasoning.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. The response to each question is as follows:\n\n**W1.  Datasets**\n\nWe further elaborate on our data construction process and associated quality control, as follows:\n\n * **Sampling Fact Triplets.** We first manually collect multiple relations containing time information from WikiData, e.g. \"President is\". Then, we use SPARQL to sample facts based on the relation and subject entity. For example, given a query (U.S., President is, ?), WikiData will return every president of the U.S. and their terms of office. Thus, We select common relations and entities to control quality.\n    \n* **Generate Relation Path Template.** In order to generate natural and coherent multi-hop questions, we look for high-quality relation combinations. E.g., (live in, the father is) is a bad combination because the object type of 'life in' (location) does not match the subject type of 'father is' (person); a location cannot have a father. Thus, we arrange and combine the collected relations into a relation path template and filter out unreasonable relation combinations.\n    \n* **Generate Multi-hop Question Template:** Then, we use GPT-4 to generate the multi-hop question template for each relation path template, e.g., (father is, company is) -> \"What is the company of father of {}?\". Replace \"{}\" into the start point entity to get the final questions. We also conduct manual revisions to ensure its quality.\n\n* **Augmented data is reliable:** All the augmented facts are reliable and are sampled from Wikidata after 2022.\n\nWe will add these details in the final draft to further clarify the data curation pipeline of our work. \n\n**W2. Compare with other papers**\n\nThere are some recent papers on temporal multi-hop QA datasets that are worth comparison to the differences.\n\n* Thank you for pointing out this related work. We noticed that the paper you have mentioned only covers temporal MQA and may not be well-suited for temporal MQA under KE. Moreover, the paper was released one and a half months before the CoLM deadline. However, we will try to add its comparisons for inference path planning against Temple-MQA in the final draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed approach addresses an interesting and challenging problem on multiple-hop QA with temporal time stamps. \n- Expanding the knowledge graph with time stamps is an interesting direction to explore. \n- The authors develop a temporal multi-hop QA dataset that can benefit the community. \n- The proposed approach is novel, although seems to stitch multiple different existing papers.", "reasons_to_reject": "- My main concern is with the evaluation. Are there quality controls on the created TKEMQA dataset? \n In section 3, the temporal KE from start time to end time is added to the original fact triple. However, it was not clear to me how reliable the augmented facts are. Are these being evaluated? It seems that they are only evaluated on downstream multi-hop QAs with accuracy. \n- There are some recent papers on temporal multi-hop QA datasets that are worth comparison to the differences. e.g. PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering https://arxiv.org/abs/2402.11034", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3jihhfjb1", "reviewer": "Reviewer_huoN", "summary": "The paper proposes a novel method to multi-hop question answering especially focusing on how to correctly answer temporal questions. The authors use a temporal aware graph (TAG) representing temporal information. The graph is used to retrieve the correct answer to temporal questions. Comparison with previous approaches shows superiority of the new method for two benchmark datasets. In addition, a new data set is proposed.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Our response for your questions is as follows:\n\n**Response to Reasons**\n\nW1. See our response to Question 2,3.\n\nW2. See our response to Question 6.\n\n**Response to Questions**\n\n**Q1. Unstructured:** \n\nUnstructured format means sentences in natural language form. For example, \"The president of the U.S. is Joe Biden.\" is an unstructured text format, while (U.S., president is, Joe Biden) is structured. In the future, we plan to extend it to support knowledge editing by complex unstructured paragraphs. E.g., \"The Eiffel Tower is located in Paris. It was designed by french engineer Gustave Eiffel...\", we can extract knowledge (Eiffel Tower, located in, Paris),(Eiffel Tower, design by, Gustave Eiffel),(Gustave Eiffel, nationality, French). \n\n**Q2. f\\_old and f\\_cur:** \n\nWe add two examples to better explain it: f\\_old means historical knowledge, e.g. \"The president of the U.S. was Donald Trump from 2016 to 2021.\" f\\_cur means current knowledge, e.g., \"After 2021, the President of the U.S. is Joe Biden.\" \n\n**Q3. Appendix-A:** \n\nWe will augment the details in Appendix-A. Also, we explain the contents of Appendix A as follows:\n        - We add the key features of inference-path planning prompt to support high accuracy, such as amount and diversity of demos.\n        - We give some running examples to make the workflow easier for the readers.\n    \n**Q4. Data Augmentation** \n\nWe explain it through a simple example: For entity 'United States of America'(its id is Q30), we first construct the following SPARQL statement:\n```\nSELECT ?alias WHERE {\nwd:Q30 skos:altLabel ?alias.\nFILTER(LANG(?alias) = \"en\").\n}\n```\nThen we query the website \"query.wikidata.org/\" to get all the aliases (USA, United States, the US ...). \n\n**Q5. Typo:** \n\nWe rephrase it as \"alternate between plan and solve phases\".\n\n**Q6. Metrics:**\n\nOwing to limited space, we provided details about the evaluation metrics in Appendix B.3. \n\n**Q7. Self-checking:** \n\nSelf-checking implies using LLMs to check if the retrieved fact contradicts the generated answer. The reason why we do not use self-checking is that we have observed that many LLMs have been trained with RLHF, which makes them have a high tendency to reject external knowledge that contradicts their own knowledge. However, we admit that self-checking is a very good design that can effectively resist harmful edits.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tNovel extension of LLM-based Mulit-hop question answering using temporal information\n-\tThe representation of a temporal graph indicates an important step towards reasoning capabilities of temporal information \n-\tThe new method shows strong performance compared to previous approaches", "reasons_to_reject": "The paper is sometimes difficult  to follow, several important aspects are hidden in the appendix or refer to other papers:\n- Construction of the temporal graph generation \n- the metrics used.", "questions_to_authors": "While the overall contribution is definitely novel and the results are impressive, I found it difficult to understand the different steps of the algorithm that was developed. Sometimes the authors referred to the appendix and some of the explanations seemed confusing and  incomplete:\n\n- 4.1. Comparison to existing approaches:”..store knowledge edits in an unstructured format.. “ please give an example. Does unstructured format simply mean text?\n- “Historical  and current facts are typically stored in unstructured format  as f_ols and f_current” this sentence does not add much to the explanation\n\n- “See appendix A for details.” PLese add more information in the text because this is an important part of your paper\n\n- Data augmentation (SPARQL) and graph construction refer to the appendix again and the text lacks clarity. It may be helpful to include a running example so that the reader can see the effects of the different steps.\n\n- “alternate between multiple different and solve phases. “ noun missing after different?\n\n- Evaluation metrics: the metrics are specifically developed for the temporal Multi-hop QA and it would be good to describe them in more detail in the paper because otherwise the reader will have a hard time identify the significance of the difference in performance (i.e., Historical Explicit time Question Score (HES) and Current Explicit time Question Score (CES) )\n- Joint reasoning: “there is no need to use self- checking Zhong et al. (2023) to check if the retrieved fact contradicts the generated answer. “ why?", "extra_scores": {}}, {"review_id": "TLDYHZDO0U", "reviewer": "Reviewer_Er5T", "summary": "The paper proposes a novel framework called TEMPLE-MQA (TEMPoral knowLEdge augmented Multi-hop Question Answering) for multi-hop question answering under temporal knowledge editing. TEMPLE-MQA constructs a time-aware graph (TAG) to store knowledge edits in a structured manner, preserving temporal information. It utilizes pre-trained language models (LLMs) to devise an inference path and perform joint reasoning using the TAG and LLMs. The key contributions include the TAG construction method, inference path generation, structural retrieval for knowledge retrieval, and a new dataset called TKEMQA for evaluating temporal MQA. Experimental results on benchmark datasets demonstrate that TEMPLE-MQA outperforms existing baselines for MQA under massive knowledge edits.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Our response for each question is as follows:\n\n**W1. Data Quality** \n\nPlease refer to our response to reviewer-EvWU. \n\n**W2. Ablation Studies:** \n\nWe performed two ablation experiments on MQUAKE: \n\n| Method                 | 1-edited | 100-edited | all-edited |\n| ---------------------- | -------- | ---------- | ---------- |\n| TEMPLE-MQA             | 78.11    | 67.21      | 53.68      |\n| -w/o data augmentation | 74.54    | 63.81      | 50.67      |\n| -w/o inference path    | 62.67    | 52.82      | 44.33      |\n\nAfter removing data augmentation, the M-ACC drops by up to 3.57%. We test TEMPLE-MQA with sub-question decomposition excluding the inference path. the M-ACC drops by up to 15.44%. It has a huge impact on performance.\n\n**W3. Scalability** \n\nWe perform a quick empirical analysis on large KG (X represents the multiple of all the original edits):\n\n| Size          | 1x    | 2x    | 4x    | 10x   |\n| ------------- | ----- | ----- | ----- | ----- |\n| Acc (%)       | 58.67 | 56.49 | 54.31 | 50.67 |\n| Runtime (min) | 326.1 | 328.9 | 336.8 | 346.7 |\n\nThese results show that Temple-MQA is scale-able. We will add these details to the final draft to clarify the computational complexity and scalability of our work.\n\n**W4. Limitations** \n\nOur work poses following limitations:\n    * Temple-MQA requires pre-processing of TAG, which brings additional overhead.\n    * Temple-MQA is very effective on the existing and our proposed benchmarks, but it requires writing prompts manually to adapt to different tasks.\n    * Although retrieval of Temple-MQA doesn't need fine-tuning an encoder, it has a complex workflow.\n\n**Q1. Disambiguation:** \n\nEntity disambiguation is a core part of our model, as it helps improve retrieval performance by filtering entities that are not relevant for a given context. We report the results of our model with and without data augmentation in the table given above. The performance declines by up to 3.57%. \n\n**Q2. Strange results** \n\nThe possible reason for the performance shortfall is that the TEMPLE-MQA uses a structured query prompt:\n```\nInference step: <U.K., head of government, person>\nTime: <2019,2022>\nAnswer: Boris Johnson\n```\nwhich leads to lower accuracy than sub-questions with natural language form adopted in MeLLo: \n\n```\nQuestion:Who was the head of government of U.K. from 2019 to 2022? \nAnswer: Boris Johnson\n```\nOne possible way to address this problem is using natural language form query prompts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important problem of handling temporal contexts in multi-hop question answering with knowledge editing, which has not been well-explored in previous work.\n2. The introduction of the TKEMQA dataset, the first benchmark tailored for MQA with temporal scopes, is a valuable contribution to the research community.\n3. Extensive experiments on multiple datasets demonstrate the superior performance of TEMPLE-MQA over existing baselines, validating the effectiveness of the proposed approach.\n4. The paper is well-written, with clear explanations of the methodology and experimental setup.", "reasons_to_reject": "1. While the paper introduces the TKEMQA dataset, more details on the dataset construction process and the quality of the data could be provided.\n2. The paper could benefit from more extensive ablation studies and analyses to better understand the contributions of different components in TEMPLE-MQA.\n3. The computational complexity and scalability of TEMPLE-MQA, especially for large-scale knowledge graphs, are not thoroughly discussed.\n4. The paper lacks discussions on the limitations of the proposed approach and potential future directions.", "questions_to_authors": "1. In Section 4.1 on constructing the Time-Aware Graph (TAG), you mention using data augmentation to capture aliases for entity names to aid entity disambiguation. Could you provide some more details on how this data augmentation is performed and how effective it is compared to not using data augmentation?\n\n2. The results in Table 2 show that on TKEMQA, TEMPLE-MQA underperforms MeLLo on the HES metric in the All-edited GPT setting. What are some potential reasons for this and are there any planned improvements to address it?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposed to use time-aware graph to store time-conscious knowledge, which is used for answering temporal multi-hop questions.

Pros:
1. The problem studied is important.
2. The method achieves high accuracy.
3. Some reviewers commended the novelty of the method.

Cons:
1. Reviewers asked for elaboration of some details. 
2. Reviewers also asked for some additional ablation and comparison with existing work to strengthen the evaluation.

The authors addressed some of reviewers’ concerns in the rebuttal. Overall, all four reviewers are positive about this paper. There are not any critical concerns. The authors are encouraged to include the additional results and elaboration in their final version of the paper if the paper is accepted.
Decision: Accept