Temporal Knowledge Question Answering via Abstract Reasoning Induction (2024)

Abstract: In this study, we address the challenge of enhancing temporal knowledge reasoning in Large Language Models (LLMs). LLMs often struggle with this task, leading to the generation of inaccurate or misleading responses. This issue mainly arises from their limited ability to handle evolving factual knowledge and complex temporal logic. To overcome these limitations, we propose Abstract Reasoning Induction (ARI) framework, which divides temporal reasoning into two distinct phases: Knowledgeagnostic and Knowledge-based. This framework offers factual knowledge support to LLMs while minimizing the incorporation of extraneous noisy data. Concurrently, informed by the principles of constructivism, ARI provides LLMs the capability to engage in proactive, self-directed learning from both correct and incorrect historical reasoning samples. By teaching LLMs to actively construct knowledge and methods, it can significantly boosting their temporal reasoning abilities. Our approach achieves significant improvements, with relative gains of 29.7% and 9.27% on two temporal QA datasets, underscoring its efficacy in advancing temporal reasoning in LLMs. The code can be found at https: //github.com/czy1999/ARI-QA.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4872–4889 August 11-16, 2024 ©2024 Association for Computational Linguistics
Temporal Knowledge Question Answering via Abstract Reasoning Induction
Ziyang Chen1∗ Dongfang Li2∗ Xiang Zhao1† Baotian Hu2† Min Zhang2 1 Laboratory for Big Data and Decision, National University of Defense Technology, China
2 Harbin Institute of Technology (Shenzhen), Shenzhen, China {chenziyangnudt,xiangzhao}@nudt.edu.cn
{lidongfang,hubaotian,zhangmin2021}@hit.edu.cn
Abstract
In this study, we address the challenge of enhancing temporal knowledge reasoning in Large Language Models (LLMs). LLMs often struggle with this task, leading to the generation of inaccurate or misleading responses. This issue mainly arises from their limited ability to handle evolving factual knowledge and complex temporal logic. To overcome these limitations, we propose Abstract Reasoning Induction (ARI) framework, which divides temporal reasoning into two distinct phases: Knowledgeagnostic and Knowledge-based. This framework offers factual knowledge support to LLMs while minimizing the incorporation of extraneous noisy data. Concurrently, informed by the principles of constructivism, ARI provides LLMs the capability to engage in proactive, self-directed learning from both correct and incorrect historical reasoning samples. By teaching LLMs to actively construct knowledge and methods, it can significantly boosting their temporal reasoning abilities. Our approach achieves significant improvements, with relative gains of 29.7% and 9.27% on two temporal QA datasets, underscoring its efficacy in advancing temporal reasoning in LLMs. The code can be found at https: //github.com/czy1999/ARI-QA.

heading: 1 Introduction
text: "Knowledge is not simply transmitted from teacher to student, but actively constructed in the mind of the learner."
— Jean Piaget
In practical scenarios, factual knowledge frequently undergoes evolution over time (Roddick and Spiliopoulou, 2002; Hoffart et al., 2011; Liang et al., 2023b, 2022). For instance, the host city of the Winter Olympic Games in 2018 was South
*Equal Contribution. †Corresponding authors.
LLM
Specific Knowledge
Exemplar
Abstract
Kn ow
le dg
e le
ve l
A pp
lic ab
ili ty
le ve
l
Abstract Reasoning Induction
In Context Learning
Retrieval Augment
ed
Figure 1: LLMs, when integrated with various levels of information, exhibit varying scopes of applicability; the more abstract and refined the knowledge, the broader its potential application.
Korea, while in 2022 it was Beijing. Despite their proficiency in various linguistic tasks, LLMs often exhibit limitations in efficiently processing and understanding tasks that involve temporal information. (Huang and Chang, 2023; Zhao et al., 2023a; Liang et al., 2023a).
Specifically, when tasks require complex temporal reasoning, LLMs tend to mislead the process and provide an inaccurate result. For instance, “Which country’s government leader visited China for the last time in 2015?”, to answer this question, we need to (1) get which countries visited China in 2015 ; (2) filter out the country with the earliest visiting date. In step 1, LLMs easily meet hallucinations due to the incomplete training data and the uncertainty of parameterised knowledge. In step 2, LLMs may lead to the error because of the inaccuracy of the time filtering. Within such temporal reasoning tasks, any misjudgment in the temporal knowledge or errors during the temporal reasoning will culminate in erroneous conclusions. The problem might stem from the temporal unawareness of LLMs, impeding their ability to track and interpret events over time, particularly in situations requiring subtle and time-sensitive understanding.
Based on intuitive and empirical analysis, the cause accounting for the problem can be identi-
4872
fied from two aspects: lack of temporal knowledge and lack of complex temporal reasoning. And the definition is given as follows:
LACK OF TEMPORAL KNOWLEDGE. LLMs acquire vast knowledge through pre-training on extensive datasets. However, the fixed nature of their parameters after training solidifies their knowledge base, which leads to LLMs’ failure in understanding unseen and evolving knowledge.
LACK OF COMPLEX TEMPORAL REASONING. Owing to the inherent nature of large models that generate outputs based on maximum probability, they are limited in directly conducting complex reasoning. Facing the interconnected multi-step temporal reasoning, LLMs might accumulate errors during the process of probabilistic generation.
Despite the neglect of essences, current studies relatively approach above challenges. To augment the LLMs’ capacity for understanding unseen and evolving information, researchers incorporate external knowledge to supply contextually relevant information, known as Retrieval Augmented Generation (RAG) (Zhao et al., 2023b; Baek et al., 2023; Sun et al., 2023). Although these methods enhance the richness of LLMs’ responses, the retrieval accuracy and input length limitations might result in irrelevant noises and incomplete reasoning clues, degrading overall performance (Wang et al., 2023; Lu et al., 2023). Furthermore, although tailored examples serve as prompts to guide LLMs (Dong et al., 2023; Min et al., 2022), they are often inadequate for diverse practical tasks and require substantial efforts in time and human to acquire highquality examples. In conclusion, above approaches fail to provide necessary guidance for ongoing temporal reasoning processes and are susceptible to incorporating extraneous noise, as shown in Figure 2.
To overcome the limitation, it is crucial to recognize that LLMs are inherently limited by reliance on passively absorbing training instances. Constructivism (Savery and Duffy, 1995; Kirschner et al., 2006), deeply embedded in philosophical and psychological schools of thought, contends that knowledge and learning emerge not from mere exposure to external information but through active construction. It asserts that learners synthesize new knowledge by building upon their existing understanding and experiences (Lake and Baroni, 2023). In this view, learning is an active and ongoing process wherein individuals continuously modify and
refine their cognitive frameworks. Inspired by the principles of constructivism, we try to steer LLMs towards an active and selfinitiated learning approach, and propose an Abstract Reasoning Induction (ARI) framework. This will equip LLMs with the capacity for abstract synthesis and personalized knowledge application, enhancing relevance and utility in various contexts.
In details, to handle the lack of temporal knowledge, we transfer the data generation to an active process, consisting of two stages: Knowledgeagnostic and Knowledge-based. In knowledgeagnostic part, LLMs only need to choose potential steps. It is only in the knowledge-based part that the corresponding action is executed on the specific knowledge base to obtain the answer. This procedure offers factual knowledge support to LLMs while minimizing the incorporation of extraneous noisy data. On the other hand, to complete LLMs’ complex temporal reasoning ability, ARI actively engages in proactive and self-directed learning from both correct and incorrect historical reasoning samples. This approach enables LLM to summarize and generalize methodologies (i.e. knowledgeagnostic step-by-step instructions) for different types of questions. When similar questions are encountered again, these abstract methods will guide the LLM to perform more efficient multi-step reasoning. By teaching LLMs to actively construct knowledge and methods, it can significantly boosting their temporal reasoning abilities without the need for further training.
In summary, our contribution is three-fold:
• Grounded in the principles of constructivism, we offer fresh perspectives for enhancing the reasoning capabilities and task adaptability of LLMs.
• We present ARI, a novel temporal reasoning framework that divides the process into two phases: Knowledge-agnostic and Knowledgebased. ARI enables LLMs to learn and construct proactively from historical reasoning samples, fostering a perpetual refinement of LLMs’ reasoning abilities.
• The experimental results demonstrate that, compared to the leading TKGQA models, our approach achieves relative improvements of 29.7% and 9.27% respectively on two temporal QA datasets.

heading: 2 Related Work

heading: 2.1 TKGQA Models
text: Traditional temporal knowledge graph question answering (TKGQA) methodologies fall into two categories. The first, exemplified by TEQUILA (Jia et al., 2018), deconstructs the initial question into sub-questions and temporal constraints, employing standard KGQA models for resolution, followed by a comparative analysis to select the most fitting answer. The second approach, such as CronKGQA (Saxena et al., 2021a), seeks to leverage TKG embeddings for semantic similarity assessments in answer determination, featuring a learnable reasoning process independent of handcrafted rules. Despite CronKGQA’s proficiency with simpler inquiries, its performance falters with complex questions necessitating specific temporal inference. TempoQR (Mavromatis et al., 2021) addresses this by incorporating temporal scope data and employing the EaE method (Févry et al., 2020) to enrich question representation semantically.
However, traditional approaches rely on handcrafted rules or learnable representations, struggling with sophisticated temporal reasoning (Chen et al., 2022). In contrast, our model, leveraging the power of LLMs, excels in these challenging scenarios, showcasing superior adaptability and reasoning capabilities.

heading: 2.2 LLM Reasoning with External Information
text: Addressing hallucinations in generative models presents a compelling challenge, with one promising solution being the augmentation of LLMs with external knowledge (Mialon et al., 2023). Integration with an external knowledge base has become a prevalent strategy in question-answering and conversational tasks (Peng et al., 2023). There are mainly two approaches: explicit and implicit knowledge injection (Yang et al., 2023a). Explicit injection involves directly supplying LLMs with pertinent knowledge via prompts. For instance, KAPING (Baek et al., 2023) retrieves facts relevant to a query from a knowledge graph and appends these to the query as a prompt for the LLM, while CoK (Li et al., 2023a) first evaluates answer credibility and, if necessary, uses the LLM to decompose the question and generate various SPARQL queries to extract information from external knowledge bases. ChatKBQA (Luo et al., 2023) finetunes LLMs on KG structure to generate logical queries, which can be executed on KGs to obtain answers. Symbol-LLM (Xu et al., 2023) propose a dualstage fine-tuning framework to integrates symbolic knowledge into LLMs, enhancing their reasoning capabilities. ToG (Sun et al., 2023) treats the LLM as an agent to interactively explore related entities and relations on KGs and perform reasoning based
on the retrieved knowledge. Implicit injection, on the other hand, subtly steers the LLM by incorporating knowledge semantic embeddings during reasoning or in the decoding process. KID (Liu et al., 2022) represents a novel decoding algorithm for generative LMs that dynamically infuses external knowledge at each step of LM decoding, and KPE (Zhao et al., 2023b) introduces a trainable parameter-sharing adapter to a parameter-freezing PLM for knowledge integration. Additionally, the incorporation of knowledge from other modalities such as visual information (Li et al., 2023c,b) has also become a method of knowledge introduction.
While the integration of knowledge into LLMs can mitigate issues of hallucinations, it is not without challenges. Explicit knowledge injection often struggles to acquire high-quality, relevant information, and is constrained by the finite-length contexts. Implicit injection typically necessitates fine-tuning of parameters, which can be prohibitively costly. We address these limitations by dividing temporal knowledge reasoning into two distinct components: knowledge-related and knowledge-agnostic. This approach achieves a clear separation between knowledge and reasoning, thereby circumventing the aforementioned constraints.

heading: 2.3 LLM Reasoning with Memories
text: Memory plays a pivotal role in human intelligence (Atkinson and Shiffrin, 1968). Given that LLMs inherently lack long-term memory and their short-term memory is constrained by the scope of their context window, numerous studies have embarked on the journey to equip LLMs with memory capabilities (Pan et al., 2023; Zhong et al., 2023). Instead of the conventional approach where accumulated conversations are retrieved directly, MemoChat (Lu et al., 2023) innovatively constructs and updates a structured, instant memo that categorizes past dialogues. Conversations are then fetched based on their specific topics and summaries. Reflexion (Shinn et al., 2023) exploits a working memory to store experiences for a dedicated task to improve the performance of the agent through several trials. However, the histories stored in working memory cannot benefit the episode for different task goals. MemPrompt (Madaan et al., 2022) designs a persistent memory to store human feedback to remind the chatbot of the conversational information and improve it continuously. RLEM (Zhang et al., 2023) adopts a persis-
tent environment-grounded experience memory to store the experiences and assist in future decisionmaking even for different task goal. Thought Propagation (Yu et al., 2023) emphasizes the ability to explore and apply insights from analogous solutions. By delving into and utilizing solutions from problems related to the given issue, it improves performance and accuracy across various tasks.
However, current memory-enhanced methods are limited to passively received historical information, overlooking the active construction of abstract knowledge based on previous experience. Starting from constructivism, we apply the proposed method to provide large models with an active and continuous learning process, offering knowledge that is abstract and generalized.

heading: 3 Method
text: Algorithm 1 Abstract Reasoning Induction Require: Temporal knowledge graph K, question
q, historical memory Hq, abstract methodology instruction set MC Ensure: Answer to the question q 1: MC ← LLM(Hq) (4) 2: Initialize subject entity eh from q 3: Find 1-hop subgraph Geh of eh in K (1) 4: Enumerate initial candidate actions P0 from
Geh (2) 5: while LLM(MC∗ , P ′ ti) ̸= answer(a) do 6: Filter candidate actions Pti to get P ′ ti (3) 7: C∗t ← findKmeansCluster(q) (5) 8: a∗i = LLM(MC∗ , q, P ′ ti) (6) 9: Execute selected action a∗i and update current environment 10: Regenerate candidate actions for the next step (2) 11: if ti ≥ tmax then 12: Break 13: end if 14: end while 15: Execute final action a∗i to abtain the answer 16: Add current process to Hq 17: return Answer derived from the reasoning
process

heading: 3.1 Task Definition
text: Given a Temporal Knowledge Graph (TKG) K and a natural language question q, TKGQA aims to
extract an entity s/o ∈ E or a timestamp τ ∈ T that correctly answers the question q. For instance, for the question ‘Which country’s government leader visited China for the last time in 2015?’ Based on the event information contained in the TKG, we can get the answer to the question is the KG entity Vietnam.

heading: 3.2 ARI Framework
text: The constructivist perspective posits that knowledge does not merely encapsulate universal laws but must be contextually reconstructed for specific situations. This view emphasizes that understanding is a construct developed by the learner, uniquely shaped by their experiential background and dependent on their learning trajectory in a particular context (Kirschner et al., 2006; Savery and Duffy, 1995). In line with this philosophy, we introduce ARI framework. We overview the framework in Figure 3. Diverging from previous research that directly feeds knowledge into LLMs , we divide temporal knowledge reasoning into two parts: knowledge-agnostic and knowledge-based.
The knowledge-based module extracts relevant TKG subgraphs based on the given question, generating all feasible fine-grained actions through traversal (cf. § 3.3). This module encompasses all operations and interactions with knowledge, freeing the LLM from the vast, noisy specific information to focus on reasoning. This design approach allows us to build intricate knowledge queries by combining fine-grained atomic operations, while ef-
fortlessly adapting to various knowledge bases (see the detailed operations in Appendix A.3). In the knowledge-agnostic module, the LLM performs high-level strategic decisions and candidate action selection. We have innovated mechanisms that allow LLMs to internalize lessons from past decisions, forming generalized abstract methodologies (solid line process in Figure 3). This foundation empowers the LLM to proactively develop abstract methodological guidelines for various question types, enhancing its ability to reason on new questions efficiently (cf. §3.4).
For the inference process, ARI first categorizes the question and selects the most suitable abstract methodological guidance. According to these guidelines, the LLM interacts multiple turns with the knowledge-agnostic module to reason and gradually solve the problem, as shown in the dashed line process in Figure 3. We also present a specific reasoning example in Figure 5 of Appendix A.4.

heading: 3.3 Knowledge-based Interaction
text: In the knowledge-based interaction part, we frame complex temporal knowledge reasoning challenges as multi-step inferentce tasks (Gu and Su, 2022; Gu et al., 2023). At the beginning of each step, we employ an filtering mechanism to engage with the TKG and the current question. This interaction produces a set of feasible candidate actions for each step. The LLM then selects the most suitable action from these candidates. Following this selection, the model interacts with the TKG, updating the initial
state for the next step in a recursive process.

heading: Candidate Action Enumeration. Specifically,
text: given a complex temporal question q and a TKG K := (E ,R, T ,F), where E ,R, T denote entities, relations, and timestamps, respectively. Starting from the subject entity eh of q, we first find the 1-hop subgraph of eh in K. Let Neh be the set of nodes in the 1-hop (undirected) neighborhood of eh in the TKG, and Reh be the corresponding edge.
Geh = {(e, r)|e ∈ Neh , r ∈ Reh}, (1)
where Geh is the corresponding 1-hop subgraph of eh. For each edge r ∈ Geh , our agent will strictly follow finely-grained atomic operations in Appendix A.3, traversing and replacing the relations and entities present in the current Geh to construct the set of candidate actions P0, where the subscript 0 denotes the candidate actions for the initial step,
P0 = {Enum(action, e, r)|e, r ∈ Geh}. (2)
Candidate Action Filtration. However, due to
the continual occurrence and updating of temporal events, even the scale of a 1-hop subgraph can be vast. This results in an excessively large set of generated candidate actions, which can significantly impede the judgment of LLMs (cf. § section 4.2). Consequently, we propose a filtration process for candidate actions, retaining only those that are correct, feasible, and semantically relevant.
Specifically, for each action a within set P0, we execute the corresponding function on the TKG. If the function returns a non-empty value, the action is considered correct and feasible; otherwise, it is discarded. Among all remaining actions, we retain the top-K actions based on the calculation of their semantic similarity to the question q,
P ′ 0 = {a|exec(a) ̸= ∅ ∧ a ∈ Top-K(P0, q)}. (3)
Based on the LLM’s decision, the agent executes the corresponding action, thereby updating the current environment. Subsequently, it regenerates the next set of candidates P1 based on the newly identified subject entities, repeating this process until a termination command is received.

heading: 3.4 Knowledge-agnostic Reasoning
text: The knowledge-agnostic module enables LLMs to distill and apply abstract methodologies from historical reasoning examples, enabling adaptation to
diverse questions. This approach fosters a general methodology, applicable across various domains and to a wide range of knowledge-independent inquiries, enhancing LLMs’ versatility.

heading: Historical Memory Storage and Learning . In
text: the LLM’s reasoning process, we meticulously document the current state at each step ti, encompassing the current temporal question q, the set of candidate actions Pt, and the LLM’s decision at. The aggregate of all stepwise states for a given question forms the historical decision set Hq,
Hq = {(q, ti, Pti , ati) | i ∈ T}, (4)
where T is the set of all steps in the process. Temporal reasoning is often multi-step and complex, yet the types of reasoning involved tend to be consistent, with similar questions requiring similar inference steps. Therefore, once the LLM conducts reasoning and accumulates a series of historical inferential steps, we employ unsupervised clustering K-means (MacQueen, 1967) to categorize these historical steps into distinct clusters CH .
After the LLM engages in reasoning and compiles historical reasoning steps, these are subjected to unsupervised clustering to form distinct clusters. Each cluster contains a mix of both accurate and erroneous reasoning processes. We enable the LLM to actively learn from specific historical instances within each cluster and distill abstract methodologies independent of domain-specific knowledge.
LLM Decision with Abstract Reasoning. When addressing new inference challenges, we initiate the process by identifying the historical reasoning cluster most closely aligned with the new question. We then extract its abstract methodologies to guide the LLM in its reasoning for the current question.
Specifically, for a given question q, we calculate the similarity score S(Ci, q) with each historical reasoning cluster Ci. We then retrieve the abstract method instruction MC∗ from the cluster that yields the maximum S. Let {C1, C2, . . . , Cn} be the set of historical reasoning clusters. For a given question q, we calculate the similarity score S(Ci, q) for each cluster Ci, and select the cluster C∗ with the highest similarity score to the query q,
C∗ = argmax Ci S(Ci, q). (5)
The abstract method instruction MC∗ is then
selected from the cluster C∗ such that:
a∗i = LLM(MC∗ , q, Pi), (6)
where MC∗ is the abstract method instruction of C∗, and a∗i is the final output of LLM. The reasoning sequence concludes when the LLM outputs a termination action or when the length of reasoning steps exceeds the predetermined maximum threshold.

heading: 4 Experiment

heading: 4.1 Implementation Details
text: We use gpt-3.5-turbo-0613 as our LLM (More experiments and analyses of other LLMs can be found in Section A.7). We configure the LLM to access and investigate a corpus of 200 historical reasoning samples, with the maximum length of reasoning path set to 5, and the number of historical path categories fixed at 10. Due to the vast size of the test set, which comprises more than 50,000 question-answer pairs, we employ a stratified sampling approach for evaluation, extracting a subset of 200 questions from the test set for each iteration. In our evaluation, we compare several baseline methods, including the traditional TKGQA models and LLM-based models (see Appendix A.1 and A.2 for more details).

heading: 4.2 Overall Results
text: Table 1 presents the comparative results of ARI against other baselines on MULTITQ.
LLM only Performance. ChatGPT’s performance on two datasets reveal a significant shortcoming in its application to temporal knowledge reasoning, even when all the knowledge required for the questions is within the scope of its training data prior to 2021. This deficiency is particularly pronounced when compared to traditional TKGQA methods, suggesting that the parameterised knowledge acquired by LLMs is not seamlessly transferred to temporal reasoning tasks. A stark contrast in performance is observed between the MULTITQ and CRONQUESTIONS datasets, the latter benefiting from ChatGPT’s extensive training on WikiData(Vrandecic and Krötzsch, 2014). This discrepancy points to a significant challenge: MULTITQ’s reliance on the ICEWS(Boschee et al., 2015), which encompasses more niche and frequent events, poses a difficulty for LLMs due to their limited exposure to such data during training.
This contrast elucidates the inherent limitations of LLMs in processing temporal knowledge.
Reasoning with External Knowledge. Incorporating additional knowledge graph data into LLMs, as seen with KG-RAG, considerably enhances their performance in knowledge-intensive QA tasks. KG-RAG outperforms ChatGPT by 81% and 96% across two datasets, respectively. Nevertheless, it still falls short of leading TKGQA models. This gap can be attributed to two main factors. First, the vast and complex nature of temporal information presents a challenge. A single question may involve thousands of related events, which cannot be accurately incorporated through prompts alone, leading to insufficient background information for reasoning. Second, the retrieved external knowledge often contains redundant or irrelevant information, which can further mislead the model’s inference process.
Reasoning with Exemplar Guidance. The CoT KB approach, despite providing step-by-step reasoning guidance and knowledge-based interaction, does not reach the efficacy levels of ARI. This discrepancy stems from the exemplar-based learning method’s dependence on specific instances, which can introduce extraneous knowledge and detract LLMs from focusing on reasoning methodologies, leading to inaccurate conclusions. Additionally, its static examples fail to provide the customized guidance necessary for diverse reasoning questions. Similarly, our investigation also extends to the augmented knowledge-agnostic module in ReAct, designed to mitigate the generation of infeasible actions. Despite this enhancement, a performance gap persists when compared to ARI, underscoring similar limitations identified in the CoT KB approach. Despite ReAct KB’s interaction with the environment, it still lacks customized abstract guidance for the current reasoning question.
ARI significantly outperforms current state-ofthe-art TKGQA models, achieving a relative improvement of 29.7% on the MULTITQ dataset and a 9.27% increase in performance on the CRONQUESTIONS dataset. These substantial gains can be attributed to the knowledge adaptability and the abstract methodology instruction mechanism, which empower LLMs to make advanced decisions. By leveraging abstract methodologies, LLMs can select optimal temporal reasoning steps without engaging with the specifics of the underlying knowledge.
Comparison of Reasoning Efficiency. To validate the effectiveness of abstract instruction, we conduct an evaluation of reasoning efficiency. On the test set, with all other components of the model remaining constant, we remove the abstract instruction and record the average number of steps taken for reasoning. Compared with the ARI, we observe that under the guidance of abstract methodologies, LLMs not only improve in reasoning accuracy but also reduce their average number of reasoning steps by 11.4% on MULTITQ and 9.3% on CRONQUESTIONS. This underscores that the guidance provided by abstract methodologies can significantly enhance the efficiency of LLMs in temporal reasoning tasks.

heading: 4.3 Ablation Study
text: To evaluate the efficacy of the individual components of the model, we conducted ablation studies.
Initially, we remove the abstract guidance component, requiring the LLM to rely on its own understanding of the questions without the aid of historical information. This result in significant performance drops on both datasets, with a 19.7% decrease on MULTITQ and a 3.7% decrease on CRONQUESTIONS. This suggests that distilled abstract guidance plays a substantial role in supporting the model’s reasoning capabilities.
To further assess the impact of abstract guidance, we eliminate the clustering module, thus deriving a universal abstract guidance from all historical reasoning processes without categorization based on question type. The model performance dropped by 9.2% on MULTITQ and 2.5% on CRONQUESTIONS, indicating that a singular abstract methodology is insufficient for guiding various types of questions and that targeted abstract methodological guidance is more effective. In Appendix A.5, we illustrate the impact of varying cluster quantity on the model’s final reasoning performance.
To verify the role of incorrect samples in ARI, we remove incorrect examples from the process of generating abstract methods, providing only correct examples as guidance. As evident from the results, the removal of incorrect examples leads to a decrease in the quality of abstract guidance, resulting in a performance drop. By encountering and learning these incorrect examples, LLMs become more adept at avoiding similar pitfalls in subsequent reasoning tasks. This also aligns with our intuition and has been validated in prior studies (Wang and Li, 2023; Yang et al., 2023b; An et al., 2023).
Lastly, we remove the action selection module, allowing the LLM to choose from all generated actions without filtering. This led to a decrease in performance on both datasets by 12.8% and 5.9%, underscoring that unfiltered actions result in an excessive number of options, including irrelevant ones, which hinders the LLM’s reasoning and complicates the decision-making process.

heading: 5 Conclusion and Limitation
text: This study, anchored in the principles of constructivism, critically examines the shortcomings of LLMs in addressing complex temporal reasoning challenges and proposes an innovative approach to augment their reasoning capabilities. Through the integration of a knowledge adaptability framework and abstract methodological guidance, we have shown that LLMs can attain more precise and efficient reasoning in complex temporal scenarios, effectively overcoming their constraints in processing and interpreting time-sensitive knowledge.
Limitations. While ARI demonstrates impressive results, it also presents several limitations for ongoing refinement. Firstly, The efficacy of generating abstract guidance heavily relies on the capabilities of LLMs. Smaller-scale LLMs may struggle to produce high-quality abstract guidance, thus potentially restricting their application. Secondly, the ARI framework depends on multi-step reasoning to arrive at final answers, a process moderately influenced by the LLM’s reasoning efficiency, which extends the duration of inference. Finally, our method is primarily concentrated on complex temporal reasoning, with its effectiveness in other reasoning domains remaining to be examined. Future research should aim to refine these methods to make them more adaptable to various models and problem domains, enhance the balance between reasoning efficiency and depth, and expand their scope to include a broader range of reasoning tasks.

heading: Acknowledgement
text: The authors would like to thank the anonymous reviewers for their insightful and constructive comments, which greatly contributed to improving the quality of the paper. This work was partially supported by National Key R&D Program of China (No. 2022YFB3103600), NSFC (Nos. U23A20296, 62272469,62376067), The Science and Technology Innovation Program of Hunan Province (No. 2023RC1007), and Guangdong
Basic and Applied Basic Research Foundation (2023A1515110078).

heading: A Appendix

heading: A.1 More Details about Baseline Methods
text: In our evaluation, we compared several baseline methods.
• Pre-trained LMs: To evaluate BERT (Devlin et al., 2019) and ALBERT (Lan et al., 2020), we generate their LM-based question embedding and concatenate it with the entity and time embeddings, followed by a learnable projection. The resulted embedding is scored against all entities and timestamps via dot-product.
• EmbedKGQA (Saxena et al., 2020) is designed with static KGs. To deal with multiple temporal granularities, timestamps are ignored during pre-training and random time embeddings are used.
• CronKGQA (Saxena et al., 2021a) is designed for single temporal granularity. To deal with multiple granularities, time embeddings at the year/month granularity are drawn at random from corresponding day embeddings.
• MultiQA (Chen et al., 2023) is designed for multi-granularity temporal granularity with a transformer-based time aggregation module.
• ChatGPT *. We use ChatGPT to provide *https://chat.openai.com/
direct answers to the questions.
• KG-RAG. To validate the performance of the LLM in the presence of relevant background knowledge, we extracted relevant quaternions (up to 20) from the TKG based on the entity and time information appearing in the question, and put them in the prompt for ChatGPT to answer as a retrieval-enhanced way of comparison.
• ReAct KB:To address the applicability of ReAct (Yao et al., 2023) to our task, we designed a variant of ReAct by integrating our knowledge-agnostic module, which generates all feasible actions (using the same atomic action templates as ARI). LLMs were then prompted to select one action from the available options. Parameter settings remained consistent with our ARI approach, including the use of the same Named Entity Linking (NEL) method and reasoning length.
• CoT KB:We introduce the CoT KB method, which integrates a knowledge-based module into the Chain-of-Thought (CoT) (Wei et al., 2022) framework. This allows the LLM to interact with KG under the guidance of examples, thereby obtaining the final answer more effectively. We manually constructing 9 specific instance examples with detailed reasoning steps to guide the LLM, while keeping other settings unchanged.

heading: A.2 Datasets Statistics
text: CRONQUESTIONS (Saxena et al., 2021b) is a dataset for temporal knowledge graph question answering. The entities and times present in the questions are annotated. CRONQUESTIONS has four question types, including both simple and complex temporal questions.
MULTITQ (Chen et al., 2023) is a complex temporal question answering dataset with multigranularity temporal information. Compared to existing datasets, MULTITQ features in a few advantages, including large scale, ample relations and multiple temporal granularity, which hence better reflects real-world scenarios.
We summarize the number of questions in MULTITQ across different types in Table 3 and Table 4. In Table 5, we present sample questions from MULTITQ as per question type, time granularity and answer type.

heading: A.3 Action Templates
text: Our approach is designed to be both generalizable and scalable. The templates in ARI are not highly manually defined high-level functions, but rather finely-grained atomic operations (e.g., extracting time, entity extraction, etc.). These atomic operations can be flexibly combined to generalize a wide range of complex actions, demonstrating their
versatility and extensibility. Our action templates in ARI strictly follow the definition of functions in Table 6. We employ several specialized functions to facilitate precise information retrieval. The getTime function retrieves the timing of specific events, based on given entities and relation. For temporal positioning, getBefore, getAfter, and getBetween identify entities or events relative to specified time frames. In terms of entity queries, getTailEntity and getHeadEntity ascertain linked entities based on existing relation, with an optional time constraint. For queries targeting specific time instances, getFirst and getLast pinpoint entities with the earliest and latest occurrences, respectively. Responses are then articulated using the answer function, providing a streamlined method for answering queries within the TKG.
Our method exhibits low coupling with templates, making it adaptable to new data and domains. The extensibility of atomic templates is straightforward, allowing for easy incorporation of additional templates as needed. For instance, if we were to extend our approach to handle spatiotemporal data questions, adding a spatial atomic operation would be a straightforward task without the need for significant modifications.

heading: A.4 Details about the Instruction Format
text: In Figure 5, we illustrate an example of reasoning using the ARI model. Table 8 shows some exemplars of ARI. During each step of the process, the LLM receives guidance from abstract methods and selects the optimal action from available paths, continuing until it deems an answer has been sufficiently formulated or the maximum reasoning length is reached. Figure 8 presents the complete set of instructions used in our experiments, comprising components such as task definition, functional interpretations of potential actions, the current temporal question under consideration, historical reasoning steps, available candidate actions for the current round, feedback from the previous round’s action, and requirements for output formatting.
A.5 Impact of Cluster Quantity In Figure 7, we show the reduced dimensional clustering diagram for the 10 categories of questions in the experiment. To verify the effect of different number of clusters on the results, we present the impact of the number of historical reasoning process clusters on the results. As shown in Figure 6. We
observe an initial increase followed by a decline in performance for both simple and complex problems. This pattern can be attributed to the fact that when the number of clusters is too low, the LLM is unable to distill concise and effective abstract methods from the noisy and abundant historical paths. Conversely, when the number of clusters is too high relative to a fixed number of historical
samples, each category contains too few samples to provide the LLM with sufficient information to refine abstract methods. Thus, we observe a trend of improvement that eventually reverses as the number of clusters increases.

heading: A.6 Error Analysis
text: For error analysis, we randomly sample 100 error instances from the test set and summarized the following three types of typical errors: (1) Retrieving irrelevant entities (in MULTITQ), meaning the model obtained wrong entities from the KG; Although our entity linking model can achieve a high prediction accuracy, wrong entities still exist in some questions. (2)Low-quality abstract methodological guidance. Within the dataset, there exist complex problems for which the historical reasoning processes consistently led to incorrect conclusions. This lack of sufficient correct reasoning histories hampers the LLM’s ability to synthesize and refine effective abstract methodologies. Consequently, the low-quality abstract methods derived by the LLM prove inadequate in guiding subsequent reasoning processes, leading to a cascade of errors. (3) Uncertainty outputs of LLMs. Despite the constraint that LLMs can only select from candidate actions or provide final answers, there are instances where they do not strictly adhere to the given instruction. This non-compliance leads to the failure of our predefined graph query methods, consequently impeding effective reasoning.
This demonstrates that more efforts are needed to strengthen the model’s reasoning capabilities, particularly in enhancing the reasoning capabilities of LLMs and diversifying their reasoning processes. It is crucial to provide a richer array of effective historical information for the generation of abstract methods. This approach is vital to prevent the LLMs from falling into a repetitive cycle of errors.

heading: A.7 Generalizability of Method on Other LLMs
text: To assess the effectiveness of the ARI across various LLMs, we conducted experiments using the open-source model Llama-2-7B-chat and GPT-4. Our findings indicate that models with greater inherent capabilities yield better direct inference outcomes, presumably due to the acquisition of more extensive knowledge during training. Furthermore, the performance enhancements
in ARI models built upon these base models are more pronounced. Notably, the ARI model based on GPT-4 achieved a score of 0.411 on the MULTITQ benchmark, representing a relative improvement of 40.2% over the current state-of-the-art models. This significant advancement underscores our model’s robust generalization capabilities.
Our work focuses on enhancing LLM reasoning with abstract guidance, minimizing irrelevant noise by isolating them from direct knowledge sources. Unlike mainstream LLM agents designed for interactive environments, we employ a knowledgeagnostic module for environmental interactions, allowing the LLM to make decisions based on abstract guidance alone. Additionally, the study is open to incorporating interactive decision-making elements from recent LLM advancements, aiming to enhance the project’s effectiveness and broaden its applicability in future research.

heading: Available Actions: {actions}

heading: Methodology: {methodology} (end of methodology)

heading: Question: {question}

heading: Current Challenge:

heading: To solve this question, you need to first identify the entities and relationships in

heading: Incorrect samples: {incorrect_examples} (end of samples)

heading: 1. Firstly, use get_time to find the time, $get_time(entity B, Relation R, entity C)

heading: Step-by-step Guide:

heading: Here is an example output:

heading: Task Defination: <Task Defination> (end of Task Defination)

heading: Future Work
text: Additionally, the study is open to incorporating interactive decision-making elements from recent LLM advancements, aiming to enhance the project’s effectiveness and broaden its applicability in future research