Large Language Models Can Learn Temporal Reasoning (2024)

Abstract: While large language models (LLMs) have demonstrated remarkable reasoning capabilities, they are not without their flaws and inaccuracies. Recent studies have introduced various methods to mitigate these limitations. Temporal reasoning (TR), in particular, presents a significant challenge for LLMs due to its reliance on diverse temporal concepts and intricate temporal logic. In this paper, we propose TG-LLM, a novel framework towards languagebased TR. Instead of reasoning over the original context, we adopt a latent representation, temporal graph (TG) that enhances the learning of TR. A synthetic dataset (TGQA), which is fully controllable and requires minimal supervision, is constructed for fine-tuning LLMs on this text-to-TG translation task. We confirmed in experiments that the capability of TG translation learned on our dataset can be transferred to other TR tasks and benchmarks. On top of that, we teach LLM to perform deliberate reasoning over the TGs via Chain-of-Thought (CoT) bootstrapping and graph data augmentation. We observed that those strategies, which maintain a balance between usefulness and diversity, bring more reliable CoTs and final results than the vanilla CoT distillation.1

heading: 1 Introduction
text: As one of the fundamental abilities, temporal reasoning (TR) plays an important role in human perception. It is not just about understanding basic concepts such as ordering or duration; it extends to more intricate aspects, e.g., task planning or causal relation discovery. Recently, large language models (LLMs) (Ouyang et al., 2022; OpenAI, 2023; Touvron et al., 2023a) have emerged with some reasoning capabilities (Huang and Chang, 2023). However, there is observation that they still can not perform TR sufficiently well (Wang and Zhao,
∗Equal contribution. 1Code and data are available at https://github.com/
xiongsiheng/TG-LLM.
2023; Chu et al., 2023; Qiu et al., 2023), preventing their applications from solving complex real-world problems. In particular, TR requires a combination of various skills including mathematical and logical reasoning as well as commonsense knowledge (Zhou et al., 2019; Vashishtha et al., 2020; Qin et al., 2021).
Recent works mainly adopt general approaches to investigate and improve the TR capability of LLMs. For example, (Wang and Zhao, 2023; Chu et al., 2023; Qiu et al., 2023) benchmark the leading LLMs on different TR tasks with standard Input/Output prompt, few-shot in-context learning (ICL) and Chain-of-Thought (CoT) reasoning (Wei et al., 2022). Similarly, (Wei et al., 2023) designs several specific types of prompts as prompt tuning. (Li et al., 2023; Yuan et al., 2023) introduce predefined Python programs/rule-based templates to perform supervised fine-tuning (SFT). In addition,
10452
(Tan et al., 2023a,b) adopt some extra strategies, which include specific pre-training, instruction tuning and reinforcement learning.
Despite the effectiveness of such methods, they either ignore or not explicitly involve the intrinsic nature of TR. Humans perform complex TR on a timeline of events which are aligned with the entities and relations. These temporal concepts (e.g., ordering, duration, frequency, typical time) are then rigorously defined based on the timeline information. In other words, the aligned timeline (more generally, the temporal graph, TG) serves as a latent representation to help humans understand the patterns in TR. However, due to the lack of ground truth, the high-quality TG translation is a challenging task for most TR benchmarks. To solve this problem, we propose a synthetic dataset (TGQA), which is fully controllable and requires minimal supervision. We demonstrate the capability of TG translation learned on our dataset can be transferred to other TR tasks and benchmarks.
Given a reliable TG, the key challenges of teaching TR to LLMs include: (1) How can one introduce the necessary arithmetic and commonsense knowledge involved in TR? Prior work (Lewis et al., 2021) shows that explicitly introducing knowledge into context enhances the performance of LLMs. In this paper, we first identify all the valid time expressions, and then generate related knowledge (e.g., temporal relation and time gap between the timestamps, and the relative order of the gaps). (2) How can one teach LLM to perform deliberate reasoning? Generally, there exist two roadmaps: (i) translating natural language into logical statements, and using external symbolic engine for reasoning (Pan et al., 2023); (ii) using LLMs directly as the reasoning engine (Zhu et al., 2023). For (i), the difficulty lies in accurate translation (Yang et al., 2023c) and the limited expressive power of formal logic. For (ii), there is no guarantee for the correctness of generated intermediate steps especially with insufficient training data (Yang et al., 2023b). In this paper, we adopt (ii) with the proposed bootstrapping method to generate reliable intermediate steps for supervised fine-tuning. We further improve the model performance with graph data augmentation, which mitigates the data deficiency in TR tasks.
To be specific, our contributions are summarized as follows:
• We propose a new paradigm, TG-LLM, for
language-based TR. In this framework, we first translate the context into a latent representation (temporal graph), and then perform reasoning on it. Extensive experiments prove that our novel approach results in superior performance compared to the baselines.
• We design two approaches including Chainof-Thought bootstrapping and graph data augmentation to teach LLM to generate consistent and faithful CoTs, which brings better performance than the vanilla CoT distillation.
• We present a pipeline to create a synthetic dataset (TGQA) for question answering that requires TR. It is fully controllable and requires minimal supervision for text-temporal graph alignment. We show in experiments that fine-tuning on our dataset benefits LLM on other TR tasks and benchmarks.

heading: 2 Dataset Construction
text: In this section, we present the construction pipeline for TGQA dataset that is fully controllable and re-
quires minimal supervision for text-temporal graph alignment. Compared with existing datasets (Chen et al., 2021; Tan et al., 2023a), we have the groundtruth timelines and more diverse categories and types of TR questions (Table 2). More importantly, the pipeline can be used for various scenarios and tasks. We first split the large temporal knowledge graph, YAGO11k (Dasgupta et al., 2018), into subgraphs with a restriction on the number of events, and anonymize the entities to avoid data leakage. Then each story is translated from the subgraph by GPT-3.5 (Ouyang et al., 2022). By using some rule-based templates, we obtain reliable question and answer (QA) pairs from the graph. Finally, to reduce the noise introduced from the misalignment between the subgraph and generated story, we propose a semi-automatic verification method.
Step 1: Graph Splitting & Anonymization. Existing temporal knowledge graphs (Leetaru and Schrodt, 2013; Dasgupta et al., 2018; García-Durán et al., 2018) usually have a large size. To facilitate the learning process, we split YAGO11k into subgraphs for story generation. Specifically, given a certain entity, we find its neighbors within three hops, and extract all the events happening between them. Since we hope LLMs to do reasoning instead of memorization, it is ensured that no overlapping exists between the events in training, validation and test sets. Additionally, we notice the data leakage problem of LLMs, i.e., prior knowledge of the test data has been implicitly obtained from the extensive pre-training. Thus, an anonymization strategy, i.e., changing entity names into random ones of the same type, is adopted. For each relation, we generate a global mapping of entity names with GPT-3.5. To avoid confusion, we adopt obscure names that do not exist in YAGO11k. Step 2: Graph-based Open QA Creation. In TGQA, each sample is in the form of (temporal graph, story, questions, answers) (Table 1). Based on the given subgraph, we generate a story and multiple QAs. We first ask GPT-3.5 to write a story based on the subgraph with the requirement to include all the events. It is observed that GPT3.5 tends to ignore the end time of some events in the created story. To solve this problem, we separate the start and end time of the same event in the prompt. On the other hand, to obtain a comprehensive benchmark, we consider all types of temporal reasoning, which include sequencing, duration, frequency, simultaneity, temporal relation, comparative analysis and facts extraction. Given these categories, we design multiple question types (Table 2) with a rule-based Python script to generate the corresponding Qs and As. Step 3: Quality Control. In TGQA, noise might be introduced from the misalignment between the given subgraph and LLM-generated story. To address this problem, we propose a semi-automatic verification method. Fully manual inspection is expensive, but by first utilizing LLM we can narrow down potential errors, and only manually inspect the set of unanswered questions by LLM. Specifically, given a generated story, GPT-3.5 is queried on the time of each event in the graph. If it cannot give the correct answer, we consider the event as possibly missing in the story, which requires further manual verification. We proved the effectiveness of our semi-automatic pipeline with fully manual verification of the test stories. Note that su-
pervision is only required for story-TG alignment verification, since all the QAs are generated from rules. We show dataset statistics in Appendix A, and all the prompts involved in Appendix C.

heading: 3 TG-LLM
text: Motivated by human perception, we propose a new paradigm called TG-LLM. We first translate the text into a temporal graph (TG), and then guide the LLM to perform deliberate reasoning on it.

heading: 3.1 Text-to-TG Translation
text: Although LLM (with ICL) might have such capability, we observed a misalignment between the generated TG and pre-defined QAs, i.e, LLM making mistakes or focusing on irrelevant events. Since TG serves as the foundation of the following deliberate reasoning process, we provide a pipeline for high-quality TG data generation to fine-tune the LLM. Ground-truth TG Generation. For some datasets such as TGQA, a verified TG (corresponding to the story) is provided. We can directly fine-tune the LLM on this task with the ground truth. However, for most real-world applications, the biggest challenge is the lack of ground truth TG. We provide a pipeline for high-quality TG data generation. We first extract all the entities and relations from the QAs to help LLM focus on specific events. We then identify all the valid time expressions in the story. We provide these information to LLM for better alignment in TG construction, and verify the generated TG in a semi-automatic way.
(1) Entity & Relation Extraction: To obtain the entity and relation in pre-defined QAs, we consider two strategies: parsing with rules or using GPT3.5. Rule-based parsing, if applicable, is more efficient and reliable, which is prioritized in our experiments.
(2) Temporal Info Identification: To identify all the valid time expressions, we first use GPT-3.5 to extract from the story, and then adopt a rule-based Python script for invalid output filtering and normalization. Besides bringing better alignment, the pre-processing of time expressions facilitates the introduction of external knowledge (their temporal relations and time gaps) into TR. We proved in experiments that explicitly introducing this information further improves model performance.
(3) TG Construction & Verification: We generate the TG using GPT-3.5 with ICL. Specifically, we
provide several in-context demonstrations which include a story and extracted entities, relations and time expressions as input and the corresponding TG as output (Table 15). For those stories without explicit time expressions, we skip Step (2), and ask GPT-3.5 to provide the temporal order of events. Similar to TGQA, we verify the generated TG by first asking GPT-3.5 the pre-defined QAs, and then manually checking the alignment if GPT-3.5 fails.
Supervised Fine-tuning. We first conducted experiments to decide the best in-context format of TG. It is found out that providing a chronological list of events helps LLM perform better TR. In addition, it offers advantages to separate the start and end time of the same event. With these steps, the TG in context is transferred into a timeline with the alignment between entities, relations and times. We perform supervised fine-tuning (SFT) using Llama-2 model (Touvron et al., 2023b) with Low-Rank Adaptation (LoRA) (Hu et al., 2021). The input and output of the model are the story and aligned timeline, respectively. In experiments, we
observe benefits from the SFT on our dataset to other temporal reasoning tasks.

heading: 3.2 Temporal Graph Reasoning
text: Given the generated TGs, we teach LLM deliberate reasoning with SFT enhanced by CoT bootstrapping and graph data augmentation.

heading: 3.2.1 Bootstrapping Chain of Thoughts
text: It is observed that SFT on reliable CoTs brings better reasoning performance than that on standard Input/Output prompts (Wang et al., 2023; Ho et al., 2023). Since asking humans to create the CoT data is not scalable, we use LLM to replace humans. This task is non-trivial for reasoning over knowledge graphs (Saparov and He, 2023). In this section, we propose a bootstrapping pipeline, i.e., given a query, using LLM to generate several CoTs and selecting them as training data with a weighted sampling strategy. Compared with the conventional Best-of-N sampling, our proposal allows more training data diversity.
Specifically, we first prepare ICL examples for high-quality CoT generation. To facilitate the learning, the pre-defined QAs are classified into different categories {Q1, Q2, · · · , QM}. For each category Qi, we randomly choose Ni samples {(gj , ej , qj , aj)}Nij=1, where g, e, q, a denote TG, external knowledge, question and answer, respectively, and ask both GPT-3.5 and GPT-4 to provide diverse CoTs {cj}Nij=1. These CoTs will then be manually verified as ICL examples. Given a new training sample (gj′ , ej′ , qj′), we bootstrap K CoTs with final answers {(cj′,k, âj′,k)}Kk=1 from GPT-3.5. We first refuse the CoTs leading to incorrect answers, i.e., âj′,k ̸= a∗j′ , where a∗j′ denotes the correct answer. For the accepted CoTs, we consider a weighted sampling strategy to balance usefulness and diversity. The sampling probability Psample(·) is based on a contrastive learning score (Eq. 1). The score design, inspired by (Wang et al., 2023), considers both normalized probability P (·) of the correct answer and plausibility growth G(·) (Eq. 2). The definition of G(·) and P (·) are
given in Eq. 3 and Eq. 4, respectively.
Psample(ck) = softmax(score(ck)) (1) score(ck) = logP ( a∗ | q†, ck ) + γG (ck) (2) G (ck) = log P ( a∗ | q†, ck )
P̄{a′∈A′} (a′ | q†, ck) (3)
logP ( a | q†, ck ) = 1
|a|
|a|∑
l
logP ( tl | q†, t<l ) (4)
where ck denotes the current CoT in consideration; q† := {g, e, q}; a′ denotes a certain wrong answer from the candidate set A′; weight γ is a hyperparameter; tl denotes the l-th token in a, and t<l denotes the sequence of tokens before tl.

heading: 3.2.2 Graph Data Augmentation
text: Compared with other tasks, reasoning suffers more from data insufficiency since more information (such as evidence, arguments, and logics) are involved in the intermediate steps (Huang and Chang, 2023). To address this issue, we propose several graph data augmentation strategies (Figure 3).
Our framework involves two steps: text-to-TG translation and temporal graph reasoning. Notably, for the reasoning part, the model is trained over ground-truth/verified TGs but infers on the estimated graphs. This discrepancy actually hurts the robustness of reasoning. Thus, we introduce some disturbances on the TG during training. Note that the type of disturbances should be carefully designed in order to avoid confusing the LLM. We first investigate two types of disturbances: (i) remove irrelevant edges (Eq. 5) and (ii) replace edges
by using relation synonyms (Eq. 6). An edge (event) is considered irrelevant if not involved in both QA and CoT.
Firr ∈ {F : P (c|g\{F}, e, q) ≈ P0}F∈g (5) F ′ ∈ {fR(F ) : P (c|fR(g), e, q) ≈ P0}F∈g (6)
where P0 := P (c|g, e, q) denotes the original conditional probability, irrelevant event Firr will be randomly removed from g, and disturbed event F ′ is obtained from the global mapping of relation names fR(·).
Furthermore, to make sure LLM learns the underlying logic of TR instead of just memorizing semantic information, we introduce another two types of disturbances: (iii) globally map all the entity names in training data to some random names of the same type (Eq.7) , and (iv) change the times based on a global offset (Eq.7). For each relation, we generate these random entity names with GPT-3.5 by providing several examples of existing names.
g′ = fT (fE(g)), e′ = fT (e), q′ = fT (fE(q)), c′ = fT (fE(c)), a′ = fT (fE(a))
(7)
where fE(·), fT (·) denote the global mapping of entity names and time changing, respectively.

heading: 4 Experiments
text: We aim to answer the following research questions in our experiments: (1) Can our strategies (CoT bootstrapping and graph data augmentation) bring more reliable reasoning over TGs? (2) Can our twostep framework lead to better TR performance? (3) Do these learned capabilities of TR generalize to other tasks?

heading: 4.1 Experimental Setup
text: We demonstrate TG-LLM is a general framework by applying it to, besides TGQA, the two existing datasets, TimeQA (Chen et al., 2021) and TempReason (Tan et al., 2023a), which are constructed using Wikipedia articles, excerpts, and summaries. Examples from other datasets are listed in Appendix B. We thoroughly evaluate our framework on all the datasets with a combination of the metrics, i.e., token-level F1, exact match (EM) and perplexity-based accuracy (Acc). Besides choosing F1 and EM, which are two basic metrics for span-based QA tasks, we consider Acc for LLM evaluation, i.e., selecting from a candidate set the
final answer with the lowest perplexity as the prediction. The rationale and detailed construction of the candidate set for all datasets are listed in Appendix B.
We primarily compare our framework with the leading LLMs, i.e., Llama2 (Touvron et al., 2023b), GPT-3.5 (Ouyang et al., 2022) and GPT-4 (OpenAI, 2023). Specifically, we ran the Llama2 family on local machines (4-bit quantization for Llama2-70B due to limited computational resources), and used APIs provided by OpenAI for GPT models. We evaluated their few-shot ICL performance on the test set. To prove the effectiveness of our method, we also show the model performance with SFT on standard Input/Output prompts (SP) and CoTs from GPT-3.5. The model versions and prompt templates are provided in Appendix B and C, respectively. We also include the T5-based models (Tan et al., 2023a; Yang et al., 2023a) for a comprehensive comparison. Results on TimeQA and TempReason reported in the original papers are used, and these models are fine-tuned and evaluated on TGQA.

heading: 4.2 Implementation Details
text: We use Llama2-13B as the baseline due to limited computational resources. We inject two adapters with selectors into the base model for the textto-TG translation and temporal graph reasoning.
The adapters are trained in parallel. For inference, we first translate the original story into a temporal graph, and then perform reasoning on it, i.e., the adapters are used in sequence. For data generation, we use GPT-3.5 for story, TG and CoT generation, and the verification of stories and TGs. We use GPT-4 to create the ICL demonstrations of CoT generation, due to its high generation quality. All the prompt templates are given in Appendix C.

heading: 4.3 Main Results
text: Can our strategies bring more reliable reasoning over TGs? We show the comparison between ICL with CoTs, SFT with CoTs, bootstrapping CoTs, and graph data augmentation on TGQA (Figure 4). It can be seen that LLM learns TR better with SFT than ICL. By providing CoTs with bootstrapping and graph data augmentation strategies, the model performance gets further enhanced. Furthermore, inspired by (Wang et al., 2023), we manually check 100 CoTs generated by different strategies (Table 3). Evaluators are asked to classify the errors into four types (using wrong info, logical inconsistency, external knowledge error, and temporal graph error). It can be seen that our strategies reduce all types of error rate. Can our two-step framework lead to better temporal reasoning performance? We show the comparison between different models and strategies on all datasets (Table 4). First, we observed that among all the LLMs with ICL, GPT-4 has the strongest performance as expected. For the Llama2 family, larger models have better performance due to advanced context understanding and improved generalization. We also found that CoTs not always bring better TR due to unreliable intermediate results and hallucinations. From the results of some alternative strategies, where SFT-
SP and SFT-CoT denote supervised fine-tuning on standard Input/Output prompts and vanilla CoT distillation, respectively, we prove the effectiveness of our framework (SFT-TGR). More importantly, our model, which is based on Llama2-13B, shows a comparable or even better performance than GPT-4 on all datasets. It can be seen that the two-step framework brings a substantial performance improvement on different datasets. We hypothesize this performance improvement is because our twostep reasoning process provides an easier path toward answering the temporal questions for LLM. Do these learned capabilities of temporal reasoning generalize to other tasks? We show the comparison between different strategies (ICL with SP/CoT, SFT with TGQA/original data) on the two existing datasets, TimeQA and TempReason (Figure 5). Our framework learns the capabilities of text-to-TG translation and temporal graph reasoning, which brings better TR. More importantly, we observed that SFT on TGQA improves the model performance compared with ICL. It can be concluded that these necessary capabilities in TR are generalizable to different data distributions. Since TGQA is fully controllable and requires minimal supervision, we actually provide a general and effective way of TR capability improvement.

heading: 4.4 Ablation Study
text: We ablate different modules to see their contributions to the performance. We show the performance comparison between different configurations (Table 5). To obtain a fair comparison, we use Llama213B as the base model for all configurations. From the ablation study, we obtain some insights: (1) LLM can benefit from explicitly presented (temporal) graph which is intuitive, concise and structured. (2) Given a reliable graph, CoT bootstrapping with contrastive learning brings better performance than vanilla CoT distillation. (3) Data augmentation is necessary for LLMs to perform complex tasks such as temporal reasoning. (4) The introduction of external knowledge such as mathematics and commonsense can further augment the generation.

heading: 5 Related Work
text: Language-based Temporal Reasoning. Recently, language-based TR has gained substantial research focus (Liu et al., 2023, 2024b; Chen et al., 2024a,b; Wang et al., 2024; Cheng et al., 2024a; Xia et al., 2024). The vision here is to help LMs understand
temporal concepts and logic such that they can perform more complicate tasks. Existing methods mainly solve this problem with time-aware language modeling. For example, (Rosin et al., 2022; Pereira, 2022; Tan et al., 2023a) propose specific pre-training/fine-tuning strategies for robust TR. On the other hand, (Ning et al., 2019; Zhou et al., 2020, 2021; Yang et al., 2020, 2023a) design auxiliary objectives to introduce external temporal knowledge. Although these methods made some progress, representation learning for the underlying structure and logic of TR are either ignored or not explicitly involved.
Reasoning towards LMs. Although LLM has exhibited some emergent behaviors (Zhang et al., 2024a; Lai et al., 2024a,b; Lyu et al., 2024), it is still unknown whether they can actually perform reasoning and how strong their capability of reasoning is. Existing methods that try to elicit or enhance reasoning can be divided into two directions: reasoning-involved modeling or hybrid methods. For example, (Wei et al., 2022; Kojima et al., 2022; Yao et al., 2023) use in-context learning where some demonstrations including intermediate reasoning steps are provided in prompt. (Nye et al., 2021; Wang et al., 2023; Ho et al., 2023) fine-tune LMs with the intermediate thinking process before producing final answers. On the other hand, (Schick et al., 2023; Kynoch et al., 2023; Xu et al., 2023) combine LMs with domain-specific external tools, empowering the model to perform more complex tasks that require reasoning and interactions with environment.
Reasoning over Knowledge Graphs. Knowledge graphs (KGs) as the foundation representation of semantic and symbolic reasoning have been widely adopted in the past (Sun et al., 2018; Yang and Song, 2020; Huang et al., 2023; Wan et al., 2024; Li et al., 2024). Related work includes symbolic reasoning over temporal KGs (Yang et al., 2022;
Xiong et al., 2023, 2024), and language-based reasoning over static KGs (Cheng et al., 2024b; Zhang et al., 2024b; Liu et al., 2024a; Zhao et al., 2024; Xu et al., 2024; Wei et al., 2024). In this paper, we build the connection between language-based and symbolic-based TR. This connection brings the potential for extending these KG-based methods to language-based tasks. Different from existing methods (Luo et al., 2023; Zhang et al., 2023; Yuan et al., 2023), which limit their application to certain tasks, our framework offers enhanced generalization and usability, largely attributed to its innovative use of text-to-graph translation as a precursor to graph reasoning.

heading: 6 Conclusion
text: TG-LLM, a novel framework for LMs, has been proposed to improve their performance on temporal reasoning. To produce reliable final answers, our framework equips LLMs with the temporal graph and intermediate reasoning steps. Extensive experiments indicate that TG-LLM achieves better performance than existing pipelines. An interesting direction for future work is to extend it to more complex applications such as inductive and abductive reasoning. Due to the graph structure and capability of deliberate reasoning, it is promising to improve the model performance on these tasks as well.
Limitations
Graph-augmented approaches (Jin et al., 2024; Fan et al., 2024; Shang and Huang, 2024; He and Hooi, 2024) including TG-LLM help language models better learn related concepts from the perspective of graph. Although we demonstrate TG-LLM has good performance on understanding temporal relations, it still needs adaptations for temporal commonsense reasoning (Zhou et al., 2019; Qin et al., 2021). Explicit in-context integration of commonsense presents opportunities for this task. Further, we mainly improve the capability of LLMs by introducing a new paradigm and providing more plausible and informative training data. There can be opportunities such as simulating an environment to provide feedback to LLMs (Hao et al., 2023). For example, we can verify the generated TG based on prior knowledge such as the time gap between someone’s birth date and death date, i.e., a person’s lifespan, should fall into a certain range. In this way, we can further improve the performance.
Ethics Statement
In this paper, we adopt YAGO11k for fine-tuning the language models. The dataset is publicly available, and is for research purposes only. We also use GPT model to generate text based on YAGO11k, for which OpenAI has been committed to addressing ethical considerations. In addition, we adopt TimeQA and TempReason for evaluation. Both datasets are publicly available, and are for research purposes only. However, they may still contain improper or harmful content. None of such content reflects the opinions of the authors.

heading: Acknowledgements
text: This work was supported by a sponsored research award by Cisco Research.

heading: A Dataset Statistics of TGQA
text: In this section, we provide statistics for our TGQA dataset. We obtain 400 samples for training, 100 for validation and another 100 for test, with about 30 QA pairs in a single sample. We show the distribution of the number of facts (in one sample) in training (with validation) and test set (Figure 6). It determines the complexity of the TR tasks to some extent. We also show the distribu-
tion of question types in training (with validation) and test set (Figure 7), where Q0: "Which event occurred first, <Event_A> or <Event_B>?", Q1: "Given the following <N> events: <Event_A>, <Event_B>, <Event_C>, <Event_D>, · · · , which event is the first/second/third/fourth/· · · one in the chronological order?", Q2: "How long did the event <Event_A> last?", Q3: "", Q4: "How much time passed between the start of <Event_A> and the start of <Event_B>?", Q5: "What happened right before/after <Event_A> started?", Q6: "When did the <Event_A> occur?", Q7: "True or false: <Event_A> and <Event_B> happened at the same year?", Q8: "True or false: <Event_A> was still happening when <Event_B> started?". Note that we give eight question categories in Table 2, since in a strict sense Q0 and Q1 can be considered as the same type. Among all the question types, Q1 has the largest portion since it has multiple variants. Similarly, Q7 with two variants also has a larger portion. To mitigate question category imbalance, we first calculate the metrics for each category, and then use the average as the final scores. Additionally, we show examples for the global mapping for entity names (Table 6) and external knowledge (Table 7) used in our dataset.

heading: B Experiment Details
text: In this section, we present more experiment details for further research. We include graph data aug-
mentation examples, model versions and evaluation tasks & metrics.
Graph Data Augmentation. We show with an example our graph data augmentation strategies in Table 8. For TGs with relation synonym replacement or irrelevant edges removal, there is no need for change on graph-based QAs. To contrast, for TGs with entity names/times mapping, we need to change with the corresponding entity names/times in graph-based QAs to ensure the consistency.
Evaluation Tasks & Metrics. Besides TGQA, we consider in experiments the two existing datasets TimeQA (Chen et al., 2021) and TempReason (Tan et al., 2023a). Specifically, TimeQA contains two difficulty levels (Table 9). The easy-level split tends to be the information extraction task while the hard-level split involves understanding the relation between different temporal expressions. On the other hand, TempReason contains three levels (L1: Time-Time Relation, L2: Time-Event Relation, L3: Event-Event Relation) and three settings (OBQA: open-book QA, CBQA: close-book QA, ReasonQA: facts-based QA) (Table 10). We observed that some stories in TempReason are incomplete which partially leads to the low accuracy of LLMs.
We evaluate our framework on all the datasets with the metrics of token-level F1, exact match (EM) and perplexity-based accuracy (Acc). F1 and EM are two basic metrics for span-based QA tasks. However, the free-form prediction of LLMs might hurt their performance under these generation-based metrics. To solve this problem, we introduce perplexity-based accuracy, i.e., selecting from a candidate set the final answer with the lowest perplexity. For questions with multiple correct answers, we follow the strategy proposed in (Chen et al., 2021) to get the best result among them. Since there are multiple question categories in TGQA, we first calculate the metrics for each category, and then use the average as final scores to mitigate question category imbalance.
Human evaluation on CoT generation found out four types of error in total (Table 3). T1 means that LLM uses wrong information such as wrong start/end time during reasoning. T2 suggests that LLM makes logical errors, e.g., mentioning "Event A ends before Event B starts" in CoT but determining the statement "Event A was still happening when Event B starts" to be true. T3 denotes that LLM makes errors on external knowledge, e.g., claiming that "the date 1978 is after 1983". T4 indicates that there exists errors in the extracted TG
that lead to the wrong conclusion. Candidate Answer Generation. We involve candidate answers in the calculation of Acc. For TempReason, the authors provide negative answers since they perform time-sensitive reinforcement learning. That is, they use the score of the correct answers and wrong answers from the language model as reward to further finetune the model parameter. For TimeQA, we generate the
candidates in the following way. Given a story, there are multiple related questions which share the same subject/object entity and relation. The correct answer changes with the query time. For example, "What position did George Washington hold in 1777/1790/1799?" Answer: "Commander in Chief/Presidency/Chancellor". We collect the answers of these related questions as candidates. More generally, if there are no such related questions, we will use all the entities in the corresponding TG as candidates. Model Versions. The versions of the LLMs used in our experiments are listed below. For the Llama2 family, all the model weights can be downloaded from the platform of Hugging Face. For the GPT models, all the model weights can be accessed through the OpenAI APIs.
• Llama2-7B (meta-llama/Llama-2-7b-hf)
• Llama2-13B (meta-llama/Llama-2-13b-hf)
• Llama2-70B (meta-llama/Llama-2-70b-hf)
• GPT-3.5 (gpt-3.5-turbo)
• GPT-4 (gpt-4-1106-preview)

heading: C Example Prompts
text: In this section, we show example prompts used in our framework. Specifically, Table 12 shows an example of the graph-based story generation in
TGQA; Table 13 presents an example of the automatic story-temporal graph alignment verification in TGQA; Table 14 provides an example of the temporal info identification in TimeQA; Table 15 illustrates an example of the graph construction in TimeQA; Table 16 gives an example of the automatic temporal graph-QA alignment verification in TimeQA; Table 17 depicts an illustrative case of the CoTs bootstrapping in TGQA.

heading: D Fine-grained Results of TGQA
text: We provide the fine-grained results of TGQA in Table 11. In TGQA, there are nine types of questions as explained in Appendix A. We show all the models with different strategies. Zero-shot performance is also considered to investigate the effect of in-context examples. Note that we do not include zero-shot performance of the Llama2 family since they are not fine-tuned on instructions, i.e., we can not obtain valid zero-shot learning results. For zeroshot learning results, the format of generation is not guaranteed. The original accuracy is very low since rule-based parser cannot handle it. Instead, we use GPT-3.5 as parser to extract the answer from the generation (might introduce errors). Also, we only adopt one example in context due to context length limitation (long-context LLM performance could increase with more in-context demonstrations).
It can be seen that in-context demonstrations are not necessary for powerful pre-trained models such as GPT-3.5 and GPT-4. Undoubtedly, GPT-4 ob-
tains the best in-context learning performance on most categories. However, for some categories, the best performance of GPT-4 is around 0.6 or 0.7, which is not sufficiently well. On the other hand, we show that our strategies improve the performance in a progressive manner. Note that our model achieves the best overall performance by improving on all the categories. Specifically, deliberate reasoning with CoT bootstrapping brings better performance for most categories. Furthermore, graph data augmentation and external knowledge further enhance the capability of the model while preserving the existing good performance on those categories.
3 April 1909: Knox Cunningham was born. 1930s: He studied law and began his legal career. 1935: On 2 July 1935, he married Dorothy Enid Riley JP. 1939: Cunningham was called to the Bar by the Middle Temple. 1942: He was called to the Bar in Northern Ireland. 1943: Cunningham fought the Belfast West by-election. 1945: He contested the same seat in the general election. 1947: Knox Cunningham became involved with the World Alliance of YMCAs. 1949: He became Chairman of the National Council of the YMCA. 1954: Cunningham was elected to Orpington Urban District Council. 29 July 1976: Knox Cunningham passed away.
Which position did Knox Cunningham hold from May 1955 to Apr 1956?

heading: Future Work
text: An interesting direction for future work is to extend it to more complex applications such as inductive and abductive reasoning. Due to the graph structure and capability of deliberate reasoning, it is promising to improve the model performance on these tasks as well. Limitations
Graph-augmented approaches (Jin et al. , 2024; Fan et al. , 2024; Shang and Huang, 2024; He and Hooi, 2024) including TG-LLM help language models better learn related concepts from the perspective of graph. Although we demonstrate TG-LLM has good performance on understanding temporal relations, it still needs adaptations for temporal commonsense reasoning (Zhou et al. , 2019; Qin et al. , 2021). Explicit in-context integration of commonsense presents opportunities for this task. Further, we mainly improve the capability of LLMs by introducing a new paradigm and providing more plausible and informative training data. There can be opportunities such as simulating an environment to provide feedback to LLMs (Hao et al. , 2023). For example, we can verify the generated TG based on prior knowledge such as the time gap between someone’s birth date and death date, i. e. , a person’s lifespan, should fall into a certain range. In this way, we can further improve the performance