Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models (2024)

Abstract: Temporal knowledge graph question answering (TKGQA) poses a significant challenge task, due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge. Although large language models (LLMs) have made considerable progress in their reasoning ability over structured data, their application to the TKGQA task is a relatively unexplored area. This paper first proposes a novel generative temporal knowledge graph question answering framework, GenTKGQA, which guides LLMs to answer temporal questions through two phases: Subgraph Retrieval and Answer Generation. First, we exploit LLM’s intrinsic knowledge to mine temporal constraints and structural links in the questions without extra training, thus narrowing down the subgraph search space in both temporal and structural dimensions. Next, we design virtual knowledge indicators to fuse the graph neural network signals of the subgraph and the text representations of the LLM in a non-shallow way, which helps the opensource LLM deeply understand the temporal order and structural dependencies among the retrieved facts through instruction tuning. Experimental results on two widely used datasets demonstrate the superiority of our model.

heading: 1 Introduction
text: Real-world knowledge is frequently updated rather than static (Erxleben et al., 2014; Boschee et al., 2015), e.g., (Obama, hold_position, President) is merely valid only for a certain period [2009, 2016]. Hence, the temporal knowledge graph (TKG) is proposed as a database for storing dynamic structured facts associated with timestamps, denoted as (subject, relation, object, timestamp). Temporal knowledge graph question answering (TKGQA)
*Corresponding Author
aims to answer a natural question with explicit or implicit temporal constraints based on the TKG, e.g., "Who held the position of president (in 2017) or (after Obama)?". Due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge, TKGQA is one of the most challenging QA tasks.
Recently, large language models (LLMs) have shown strong competitiveness in various fields (Fei et al., 2023; Ye et al., 2023). Some researchers explore the reasoning capability of LLMs for structured knowledge based on KGQA tasks (Baek et al., 2023; Kim et al., 2023), and some works examine the temporal reasoning capabilities of LLMs through time-sensitive QA tasks (Chen et al., 2021; Tan et al., 2023). Intuitively, LLMs have the ability to deal with temporal structured knowledge. Based on the above findings, we attempt to utilize LLMs for the TKGQA task and summarize the following two challenges: 1) Question-relevant Subgraph Retrieval. A common practice to enhance the LLM’s domain-specific reasoning capability is to input query-relevant information as additional knowledge into the LLMs (Hu et al., 2023). As shown in Figure 1, ChatGPT 1 cannot answer temporal questions directly, but it can
1https://openai.com/blog/chatgpt
6719
answer simple questions when the relevant facts are provided. However, finding facts relevant to the problem is a struggle due to the large search space with both structural and temporal dimensions. For example, for the complex temporal question "Who held the position of president after Obama?", the structural link hold position between the entities and the temporal constraints after 2016 in the problem are unknown, so directly finding all relevant facts about the given entities Obama and President is bound to introduce too much noise information. How to accurately retrieve relevant facts from a two-dimensional space is the first challenge. 2) Complex-type Question Reasoning. Recent works about LLM-based KG reasoning mostly input structured knowledge in the natural text form into the task prompt and reason about the answers in a training-free manner (Yang et al., 2024). However, these approaches fuse subgraph information with the LLM in a shallow way, which limits the inference performance on the complex question type. As illustrated in Figure 1, ChatGPT cannot understand the chronological order of the relevant facts and answers incorrectly on the complex question type "final". How subgraph information can be integrated into LLM representations in a nonsuperficial way to simulate structured reasoning remains an open question.
Hence, we propose GenTKGQA, a novel generative temporal knowledge graph question answering framework consisting of two phases, subgraph retrieval and answer generation, which is used to address the above two challenges, respectively. At the first phase, we find that the structural and temporal scope of the subgraph is determined by the relation links and the temporal constraints in the question, respectively. Therefore, we use a divideand-conquer strategy to reduce the subgraph search space by decomposing the complex subgraph retrieval problem into two subtasks, namely, relation ranking and time mining. Then, we utilize the LLM’s internal knowledge to mine structural connections between entities and time constraints in the problem without extra training. We only need to input few-shot examples into the prompt to accomplish subgraph retrieval of the entire data. At the second phase, we fine-tune the open-source LLM with instruction tuning to incorporate structural and temporal information of the subgraph in a non-shallow way. Recent works illustrate that fusing graph neural network (GNN) representations and language text representations can enhance the
ability of LMs to perceive graph structure (Zhang et al., 2022b). Thus, we design three novel virtual knowledge indicators to bridge the links between pre-trained GNN signals of the temporal subgraph and text representations of the LLM, which guides the LLMs in deeply understanding the graph structure and improves their reasoning ability for complex temporal questions. Overall, our contribution can be summarized in the following four points:
1) We present a novel two-stage generative framework for the TKGQA task, which explores LLM’s temporal reasoning capabilities in the context of dynamic structured knowledge.
2) We motivate the LLM’s intrinsic knowledge to mine the temporal constraints and structural connections in the questions without extra training, which reduces the subgraph search space from both structural and temporal dimensions.
3) We design virtual knowledge indicators to fuse the GNN signals and text representations in a non-shallow way, which helps the open-source LLMs improve their reasoning on the complex question type through instruction tuning.
4) Experiment results on two widely used datasets show that GenTKGQA as a generative model performs consistently better than embedding-based methods on the Hits@1 metric.

heading: 2 Related Work

heading: 2.1 TKGQA Methods
text: Temporal knowledge graph question answering (TKGQA) task aims to answer complex questions in the natural language format using entities and timestamps from the given TKG (Jia et al., 2018, 2021; Chen et al., 2023b). Existing mainstream methods employ TKG embeddings to represent the entities, relations and timestamps, and use the scoring function to select the entity or time with the highest relevance as the answer (Saxena et al., 2021). However, single embedding methods have difficulty handling complex reasoning problems with implicit time constraints. Therefore, recent methods try to incorporate other modules to improve the model performance on complex problems. Specifically, TSQA (Shang et al., 2022) proposes a contrastive approach to enhance the model’s time sensitivity. TempoQR (Mavromatis et al., 2022) designs three modules, namely context, entity and time-aware information, to enhance the incorporation of the TKG into questions. Besides, some approaches propose to solve the TKGQA task with
problem-relevant subgraph reasoning (Chen et al., 2022; Liu et al., 2023b; Sharma et al., 2023). Despite the effectiveness of these approaches, few studies have explored how LLMs can address the TKGQA task.

heading: 2.2 LMs for Temporal Question Answering
text: Language models (LMs) have exhibited strong performance on the question answering task (Raffel et al., 2020). In recent years, some researchers have explored the temporal reasoning capabilities of LMs and propose several typical time-sensitive QA datasets. They focus on temporal question answering either within a closed-book setting to assess models’ internal memorization of temporal facts (Liska et al., 2022; Dhingra et al., 2022), or within an open-book setting to evaluate models’ temporal understanding and reasoning capability over unstructured texts (Zhang and Choi, 2021; Chen et al., 2021; Tan et al., 2023). In the context of the latter setting, some works propose to use the graph structure extracted from text to assist the model in determining the temporal order between events (Mathur et al., 2022; Su et al., 2023; Yang et al., 2023; Xiong et al., 2024), which is similar but fundamentally different from our work. These approaches aim to answer temporal questions based on the known natural text context. In contrast, our model focuses on structured temporal knowledge as auxiliary information that needs to be retrieved by the model.

heading: 2.3 LMs for KG Question Answering
text: How to combine LMs and KG for question answering has become a hot issue. Some works attempt to enhance question representation and relation matching with PLMs in the multi-hop KGQA task (Saxena et al., 2020; Zhang et al., 2022a; Jiang et al., 2023b), but there is no interaction between the LM and KG representations. Other works try to use one modality to ground the other, i.e., using the encoded representation of a linked KG to augment the text representation (Lin et al., 2019; Yang et al., 2019), or using the text representation of the PLM to enhance the graph reasoning model (Feng et al., 2020a). The most recent approaches enable deep integration of the two modalities by jointly updating the GNN and LM representations (Yasunaga et al., 2021; Zhang et al., 2022b).
However, the emergence of large language models (LLMs) has changed how LMs handle the KGQA task, which is divided into two main ap-
proaches: training-free and fine-tuning (Yang et al., 2024). Recent works attempt to append queryrelevant facts as the input prompt for LLMs and make inferences without extra training (Baek et al., 2023; Wu et al., 2023; Jiang et al., 2023a; Kim et al., 2023; Li et al., 2024). Fine-tuning the full parameters of the LLM can be cost-prohibitive. Hence, KPE (Zhao et al., 2023) enables knowledge integration by freezing PLM parameters and introducing trainable parameter adapters. ChatKBQA (Luo et al., 2023) employs the LoRA (Hu et al., 2022) technique to fine-tune open-source LLMs, achieving the logical query form generation. Besides, KoPA (Zhang et al., 2023) incorporates the KG embeddings into the LLMs with a prefix adapter, aiming to achieve structrual-aware reasoning in the LLMs. Applying LLMs to the temporal KGQA task remains an unexplored area.

heading: 3 Preliminaries
text: TKGQA. A temporal knowledge graph (TKG) G := (E ,R, T ,F) is a multi-relational, directed graph with timestamped edges between entities, where E , R and T represent the sets of entities, relations and timestamps, respectively. Each fact in the G can be represented as a quadruple (s, r, o, t) ∈ F , corresponding to entity s/o ∈ E , relation type r ∈ R and timestamp t ∈ T . Given a natural language question q, TKGQA aims to extract entities s/o or timestamps t that correctly answer the question q.
ICL and IT. Applying LLM to the TKGQA task, the goal is to generate the answer A based on the input text sequence S and the LLM M. S consists of several parts: the instruction prompt I, the task-specific input prompt Q, and the auxiliary demonstration prompt D. In-context Learning (ICL) method is an efficient approach to employ LLMs to solve downstream tasks without extra training, the input sequence of ICL can be denoted as S = I : D : Q, where : means to concatenate the different prompts. Meanwhile instruction tuning (IT) aims to fine-tune LLMs to follow human instructions and accomplish the distinct tasks in the instruction prompt, the input sequence of IT can be denoted as S = I : Q : A.

heading: 4 Method
text: We apply the LLMs processing TKGQA task in a two-phase process, i.e., an ICL-based subgraph retrieval phase and an IT-based answer generation
phase. At the first stage, we utilize internal knowledge of LLM for unlabeled subgraph retrieval. At the second stage, we incorporate external knowledge for structure-aware temporal inference.

heading: 4.1 Subgraph Retrieval
text: We split the complex subgraph retrieval problem into a relation ranking subtask in the structural dimension and a time mining subtask in the temporal dimension. With such a strategy, we only need to provide a small number of samples to complete the subgraph retrieval.

heading: 4.1.1 Relation Ranking
text: We aim to determine the structural scope of the subgraph Gsub,i by retrieving the corresponding relations from the candidate relation set Ri for each question qi. Recent work has shown that LLMs can better handle the information extraction task as reranking agents (Sun et al., 2023). Therefore, we feed the question qi and the candidate set Ri to the LLM to obtain the top k relations Ri,k relevant to the question. Relations in Ri are linearized, i.e., [employer, member_of_sports_team, ..., ]), and the retrievd relations can bridge the entities identified within the questions. The specific relation ranking prompt is shown in Appendix C.

heading: 4.1.2 Time Mining
text: We find that natural questions contain temporal constraints, either explicit or implicit, and we can easily determine the range of relevant facts in the temporal dimension by using explicit temporal constraints such as "in 2008", "at the year of 2012", etc. How to capture implicit temporal constraints is the key to improving the efficiency of searching for relevant facts. For example, for the complex temporal question "Who held the position of president after Obama?", the implicit temporal restriction is known to be (after 2016) based on the temporal validity (2009, 2016) of the fact (Obama, hold_position, President). We design specific prompt templates based on different answer types as well as consider the temporal validity of the facts in the questions (given entities and relations matched in the Section 4.1.1) to get the temporal constraints. The details of the templates are shown in Appendix C.
Through the above process, we narrow down the search space of subgraphs and use relevant facts under structural and temporal constraints as additional knowledge to assist the LLM inference.

heading: 4.2 Answer Generation
text: In this section, we will discuss how to incorporate the knowledge retrieved in the previous section 4.1 into the LLM. Previous fundamental approaches to incorporate KG structural information focus on adding the knowledge to the input prompt in the text form, i.e., (subject, relation, object). However, incorporating query-relevant facts into LLMs in text form is not a good choice. Because such shallow interactions do not enable the model to understand the structural dependencies and temporal order between facts, leading to weak temporal reasoning in the complex problem type.
Recent works show that language models can enhance their ability to perceive graph structures by incorporating knowledge representations expressed by graph neural networks (GNN) (Zhang et al., 2022b). Inspired by this, we first extract the structural and temporal information of entities and relations with pre-trained temporal GNN embeddings. Then, we bridge the links between GNN and text representations through pre-designed virtual knowledge indicators. At last, we fine-tune the opensource LLM to deeply understand the temporal order and structural dependencies of the retrieved query-relevant facts.

heading: 4.2.1 Temporal GNN
text: Given the retrieved temporal subgraph Gsub,z of question qz , we first initialise entity, relation and time representations in Gsub,z using the TKG embedding method (Lacroix et al., 2020). Then, to fully explore the structural information among entities and relations of the temporal subgraph, we propose a temporal graph neural network (T-GNN), which is a variant of graph attention networks (Velickovic et al., 2018). The important distinction between them is that T-GNN captures the correlation scores of neighbouring nodes by incorporating temporal embeddings. Therefore, T-GNN computes message mij between entities ei and ej as follows:
mij = Wm(e (l−1) i + rij + tij), (1)
where e(l−1)i is the entity representation of ei at the l-1 layer, rij and tij are the embeddings of the relation and timestamp connecting ei and ej . Wm is a linear transformation. Next, the node representation e(l)j is calculated via message passing between
neighbors on the Gsub,z:
e (l) j =
∑ i∈Nj αijmij , (2)
here Nj represents the neighbor entities of the arbitrary node ej , and αij denotes the attention values with e(l−1)i as query and mij as key:
αij = exp (uij)∑
w∈Nj exp (uwj) , (3)
uij = fn((Wqe (l−1) i ) ⊤ (Wkmij)), (4)
Wq,Wk are linear transformations, fn is the RELU activation function. Through the above process, we obtain the entity representation ej with subgraph structural and temporal information. Following embedding-based TKGQA methods, we use the link prediction task to pre-train the graph neural network representations. Specifically, for each fact (s, r, o, t) in the TKG, we generate a query (s, r, [mask], t) or ([mask], r, o, t) by masking the object or subject entity. Then, we obtain the mask embedding e(l)[mask] through Eq.(2), and feed it into the multi-layer perceptron (MLP) decoder to maximize the probability of the missing entity o and s through the cross-entropy loss function:
p (e) = Softmax(e(l)[mask]w + b), (5)
L = − ∑
(s,r,o,t)∈G log p(ot) + log p(st). (6)

heading: 4.2.2 Virtual Knowledge Indicators
text: We design three knowledge indicators to link graph signals and input prompt text, namely [SUB], [REL] and [OBJ], correspond to the virtual tokens of the head entities, relations and tail entities in the subgraph, respectively. We then try to incorporate structural and temporal information from the subgraph into the indicator representations. Specifically, we use the Local operator to get the structure representations hs of entities and relationships in the subgraph, respectively:
hs[SUB] = Local(e[SUB]), (7)
here e[SUB] represents pre-trained T-GNN embeddings of all subject entities in the subgraph, Local indicates the max or mean pooling operator. Besides, we leverage the time embeddings to enhance the indicator representations with temporal information.
hst[SUB] = h s [SUB] + tmin + tmax, (8)
where tmin and tmax denote the embeddings for the minimum and maximum values of time in the subgraph, respectively. The intuition follows BERT that use position embeddings for tokens (Devlin et al., 2019). Here, time embeddings can be seen as entity positions in the time dimension. The relation and object indicators are the same as subject. At last, we employ a simple linear layer Wp to project them into the textual representation space of the
LLM. The final input prompt sequence S = V : I : Q : A, V represent virtual indicator tokens. Details of the instruction template can be found in the Appendix C. The optimization objective of the LLM M can be formulated as:
A = argmax A PM (A | V, I,Q,A) . (9)

heading: 5 Experiments
text: We design experiments to answer the following questions:
Q1.How does GenTKGQA perform on the TKG question answering task? (Section 5.2)
Q2.How do the two stages contribute to the model performance respectively? (Section 5.3)
Q3.How does GenTKGQA perform under changes in hyper-parameters? (Section 5.4)
Q4.How does GenTKGQA outperform ChatGPT in answering complex temporal questions? (Section 5.5)

heading: 5.1 Datasets, Metrics and Baselines
text: CronQuestions (Saxena et al., 2021) is a temporal QA dataset, which contains 410K unique question-answer pairs, including annotated entities and timestamps, with 350k for training and 30k for validation and testing. The dataset can be categorized into simple reasoning (Simple Entity and Simple Time) and complex reasoning (Before/After, First/Last, and Time Join) based on temporal constraints. The TimeQuestions dataset (Sharma et al., 2023) has 13.5k manually edited questions and is divided into three parts: training, validation, and testing, containing 7k, 3.2k, and 3.2k questions, respectively. The questions are categorized into four types: Explicit, Implicit, Temporal and Ordinal. Following previous studies, we use two popular evaluation metrics, Hits@1 and Hits@10. More information about datasets and metrics can be found in Appendix A.
We compare four types of baselines: 1) KG embedding-based models including EaE (Feng et al., 2020b) and EmbedKGQA (Saxena et al., 2020); 2) TKG embedding-based models including CronKGQA (Saxena et al., 2021), EntityQR (Mavromatis et al., 2022), TMA (Liu et al., 2023a), TSQA (Shang et al., 2022) and TempoQR (Mavromatis et al., 2022); and 3) Language models, including BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and ChatGPT. For another TimeQuestions dataset, we use CronKGQA, TempoQR and
TwiRGCN (Sharma et al., 2023) for comparison. The implementation details of the baselines are described in Appendix B.

heading: 5.2 Main Results
text: Table 1 reports the performance of all methods on the CronQuestions dataset for various question types. We can observe that GenTKGQA consistently outperforms the baselines in terms of "Overall" performance, and achieves significant improvements of 11.3% in the "Complex" question type and 9.6% in the "Time" answer type on the Hits@1 metric over the second best method. Especially, our model achieves nearly 100% for the "Simple" question type. The possible reason is that simple questions usually involve single facts, GenTKGQA can easily retrieve the relevant facts containing the answer and infer the correct answer through instruction tuning technique.
Furthermore, compared to KG embedding methods, temporal KG embedding methods show significant results on various metrics, thanks to the fact that the temporal information of the TKG is taken into account in the question representation. This is also why KG embedding methods are particularly ineffective for the "Time" answer type. However, most TKG embedding methods treat the QA task as a link prediction, which works for the "Simple" question type containing a single fact compared to the "Complex" question type.
We find that PLMs (BERT, RoBERTa) and LLMs (ChatGPT) have the lowest performance on the TKGQA task without TKG information. This suggests that language models (LM), whether encoded or generated, with a large or small number of parameters, have difficulty answering temporal questions without any relevant context. w/ tkg indicates that LMs use entity/time embeddings or relevant facts from the TKG. Obviously, LMs w/ tkg have significantly better performance, which suggests that the language models have some degree of temporal reasoning capability when relevant TKG information is provided, validating the importance of the subgraph retrieval phase. It is worth noting that ChatGPT w/ tkg still performs weakly in reasoning about complex problem types when providing the facts retrieved in the first stage by GenTKGQA, while our model achieves the best results. This demonstrates the effectiveness of interacting GNN and LM representations in dealing with complex temporal problems. The above findings demonstrate the adequacy of our two motiva-
tions for solving complex temporal problems with LLMs. A possible reason for the poor improvement of our model’s Hits@10 metric for the "Entity" type is that the LMs provide irrelevant responses when asked to generate multiple answers.
As shown in Table 2, GenTKGQA still achieves surprising results on the TimeQuestions dataset, especially improving on the "Implicit" and "Ordinal" question types by 1.3% and 11.6%, respectively. Both question types have implied time constraints, similar to the "Complex" question type on the CronQuestions dataset. Overall, compared to the baseline methods, GenTKGQA achieves significant results on the complex question type with different datasets, validating the motivation of our work and the effectiveness of the proposed model. Last but not least, GenTKGQA, as a generative QA model, achieves better results than most traditional extractive QA methods.

heading: 5.3 Ablation Study
text: As shown in Table 3, to verify each module’s importance, we conduct ablation experiments on the CronQuestions dataset.
w/o SR means that we directly perform problem
inference without using relevant subgraph information in the model training and inference phases, while w/o SR inference means that we do not provide subgraphs only at the inference. We can observe a sharp decrease in model effectiveness due to the lack of use of subgraph information, which is consistent with the other LMs (Section 5.2). This result shows that current LMs are weak in dealing directly with temporal reasoning problems, validating the importance of the subgraph retrieval module. The w/o SR inference result indicates that GenTKGQA remembers part of the structured knowledge during the training phase and improves the temporal inference performance without providing subgraph information. w/ SR random denotes the random selection of relevant facts involving entities in the question, and the drop in results proves the validity of our first-stage approach.
w/o T-GNN indicates that we directly use the temporal embeddings (Lacroix et al., 2020) to represent entities and relations of subgraphs. The slight decrease in the results indicates that the TGNN is able to perceive the structural information
of the TKG. w/o VKI means that we try to remove the virtual knowledge indicators from the input prompt. Model performance degradation shows that indicators can bridge the gap between distinct representations.

heading: 5.4 Sensitivity Analysis
text: Impact of training data size. We explore the impact of different training data sizes to reason about complex temporal questions. As shown in Figure 3(a), by comparing the Hits@1 metric of several methods for the complex question type, our method consistently outperforms others as the training data expands. In particular, at 20% of the training data, our model outperforms the second best model by 32%, demonstrating that our model has strong inference ability in the case of few-shot samples due to its intrinsic knowledge.
Impact of the number of relevant facts. We report the performance changes on the CronQues-
tions dataset by varying the number of retrieved facts n in Figure 3(b). It can be seen that the model performs poorly with a small number of relevant facts (n=3), and there is a slight increase in performance at n=10. Fewer facts do not provide sufficient context knowledge, while more facts may introduce noise. Taking this into consideration, we set the hyper-parameter n to 10.

heading: 5.5 Qualitative Results
text: We provide specific examples for each question type to compare the answer results of ChatGPT and ChatGPT w/ tkg. Table 4 includes the graphs retrieved by our method, along with the answer results for five different question types.
When providing relevant facts retrieved by GenTKGQA as background knowledge, ChatGPT performs competitively in the simple question type, correctly answering questions with entity or time as the answer. However, it has difficulty answering complex types of questions. For example, in the "Before/After" and "First/Last" question types, ChatGPT struggles to understand the temporal order of the relevant facts and gives incorrect answers. Besides, in the case of the "Time Join" question type, some of the correct answers are missing from the generated responses because ChatGPT does not fully understand the implicit time in the question. On the contrary, GenTKGQA performs well in both simple and complex question types due to the fact that we use a deep manner to incorporate sub-
graph information into the LLM. However, similar to other LLMs, GenTKGQA randomly generates some irrelevant answers when generating multiple answers, e.g., in the "Simple Entity" question type.

heading: 6 Conclusion
text: We propose a novel generative framework, GenTKGQA, which guides the LLM in a two-stage manner to handle temporal question answering on TKG. Specifically, at the subgraph retrieval phase, we exploit the LLM’s intrinsic knowledge to mine the temporal constraints and structural links in the temporal questions, which reduces the search space of the subgraphs in both temporal and structural dimensions. We employ the in-context learning approach to complete subgraph retrieval for the entire dataset with a small number of samples. In order to improve the inference performance of the LLM on complex question types, at the answer generation phase, we present the instruction tuning technique to make the open-source LLM truly understand the temporal order and structural dependencies among retrieved facts. Most significantly, we design novel virtual knowledge indicators to establish a bridge between subgraph neural information and text representations. Experimental results show that our framework can effectively utilize the LLM to solve the complex question type of TKGQA task and validate the adequacy of our motivation.
Limitations
Although the complex temporal question on the CronQuestions dataset involves multiple facts, the inter-entity connection in each fact is single-hop, so the hyper-parameter k of our model is set to 1 to achieve the best results. In fact, the vast majority of current TKGQA datasets involve facts that are single-hop. So, in the future, we will explore more datasets to solve inference for multi-hop complex temporal problems over TKG. In addition, we use the in-context learning approach to prompt the ChatGPT baseline to answer the questions, saving the labor cost of checking whether the answers are correct. However, the design of different templates may result in incomplete consistency with the manual results, but this does not affect the conclusions of this paper. Because the results provided by other works similarly show the poor performance of ChatGPT’s temporal question answering (Tan et al., 2023; Chen et al., 2023a).
Ethics Statement
This work presents a novel two-stage framework for the temporal knowledge graph question answering task using large language models. Our experiments use the publicly available datasets and language models from open sources. The dataset is developed to be used for the TKG-based temporal QA task. The language models are used to generate answers to temporal questions with entities or timestamps, which does not involve toxic content. This paper uses the above dataset and models with their initial intention. We believe that this work is consistent with ACL’s ethics policy and presents no potential risk.

heading: Acknowledgements
text: This work is sponsored in part by the National Natural Science Foundation of China under Grant No. 62025208, and the Xiangjiang Laboratory Fund under Grant No. 22XJ01012.

heading: A Dataset Statistics and Metrics
text: We use the CronQuestions and TimeQuestions dataset in our experiments. TimeQuestions was first proposed by Jia et al. (2021), but it provides only static knowledge graphs with temporal attributes, not strictly temporal knowledge graphs. Later, Sharma et al. (2023) expanded this dataset by preprocessing all the contained facts into the temporal knowledge graph format of (subject entity, relationship, object entity, [start time, end time]) and restricting all times to years. Dataset statistics are described in Table 5 and 6, respectively.
Following previous studies, we leverage two popular evaluation metrics, Hits@1 and Hits@10. Specifically, Hits@K =
1 |Test| ∑ q∈ Test ind(rank(q) ≤ K), where rank(q) denotes the ranking of the answer to question q obtained by the model in the candidate list. ind is 1 if the inequality holds and is 0 otherwise, K = 1, 10.
2The actual number of questions in the training, validation and test set is 6,970, 3,236 and 3,237, respectively. The total number exceeds the number of questions as some questions belong to multiple categories.

heading: B Baselines and Implementation Details
text: We use the OpenAI-API3 (gpt-3.5-turbo-06134) for all ChatGPT-related experiments, including subsequent ChatGPT baselines.
In the subgraph retrieval phase, we use ChatGPT to mine temporal constraints and structural links between entities and add 5 samples to the in-context learning prompt templates, which are presented as Table 7 and 8. We set k=1 for the top-k relations. In the answer generation phase, following (Lacroix et al., 2020), we select the dimension of entity/relation/time embeddings to 512. For T-GNN, the layer l is set to 1, the linear transformations Wq, Wk and Wm are 512×512, and the m and b of the MLP layer are 512×|E|. We use the open-source Llama 2-7B (Touvron et al., 2023) for instruction tuning and select up to n=10 relevant facts as additional knowledge. The linear projection layer Wp is 512×4096. We fine-tune Llama 2-7B using LoRA (Hu et al., 2022) with rank 64. The number of epochs is set to 4 and the learning rate is 3e-4. We use the AdamW optimizer (Loshchilov and Hutter, 2019) with a fixed batch size of 8. We conduct all the experiments with NVIDIA A100 GPUs, and the results of each experiment are averaged over three runs. We will release the source code upon acceptance.
We compare our model with the following baselines:
EmbedKGQA (Saxena et al., 2020): Timestamps are ignored during pre-training and random time embeddings are used during the QA task.
EaE (Feng et al., 2020b): In the experiment, we follow use TKG embeddings to enhance the question representation, and then predict the answer probabilities via dot-product.
CronKGQA (Saxena et al., 2021): CronKGQA is the TKGQA embedding-based method that first uses a LM model to get question embeddings and then utilize a TKG embedding-based scoring function for answer prediction.
EntityQR and TempoQR (Mavromatis et al., 2022): Based on EaE, EntityQR utilizes a TKG embedding-based scoring function for answer prediction. TempoQR utilizes a TKG embeddingbased scoring function for answer prediction and fuse additional temporal information.
3https://platform.openai.com/docs/ api-reference
4https://platform.openai.com/docs/models/ gpt-3-5-turbo
TMA (Liu et al., 2023a): TMA improves QA performance through enhanced fact retrieval and adaptive fusion network.
TSQA (Shang et al., 2022): TSQA presents a contrastive learning module that improves sensitivity to time relation words.
TwiRGCN (Sharma et al., 2023): TwiRGCN is a method for processing TKGQA tasks with the relational graph convolutional network (RGCN).
BERT and RoBERTa (Devlin et al., 2019; Liu et al., 2019): For w/o tkg, following CronKGQA (Saxena et al., 2021), we add a prediction head on top of the [CLS] token of the final layer, and then do a softmax over it to predict the answer probabilities. For w/ tkg, following TempoQR (Mavromatis et al., 2022), we generate their LM-based question embedding and concatenate it with the annotated entity and time embeddings, followed by a learnable projection. The resulted embedding is scored against all entities and timestamps via dot-product.
ChatGPT: To ensure that the output format meets the expected requirements, we use the incontext learning approach to motivate ChatGPT to answer the questions and provide 5 examples in the prompt template. The specific templates are presented in Table 10 of Appendix C. w/o tkg and w/ tkg differ in whether or not question-relevant facts are provided in the input prompts, which are retrieved in the first stage by our proposed GenTKGQA framework.

heading: C Prompt Template
text: The prompts for relation ranking and time mining can be found in Table 7 and Table 8, respectively. The template used for instruction tuning is shown in Table 9. The ChatGPT baseline prompt is presented in Table 10.

heading: Future Work
text: So, in the future, we will explore more datasets to solve inference for multi-hop complex temporal problems over TKG. In addition, we use the in-context learning approach to prompt the ChatGPT baseline to answer the questions, saving the labor cost of checking whether the answers are correct