Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoning

Authors: jiapu  wang
kai  sun
linhao  luo
wei  wei
yongli  hu
alan  wee
chung  liew
shirui  pan
baocai  yin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 07:55:18

Abstract: Temporal Knowledge Graph Reasoning (TKGR) is the process of utilizing temporal information to capture complex relations within a Temporal Knowledge Graph (TKG) to infer new knowledge. Conventional methods in TKGR typically depend on deep learning algorithms or temporal logical rules. However, deep learning-based TKGRs often lack interpretability, whereas rule-based TKGRs struggle to effectively learn temporal rules that capture temporal patterns. Recently, Large Language Models (LLMs) have demonstrated extensive knowledge and remarkable proficiency in temporal reasoning. Consequently, the employment of LLMs for Temporal Knowledge Graph Reasoning (TKGR) has sparked increasing interest among researchers. Nonetheless, LLMs are known to function as black boxes, making it challenging to comprehend their reasoning process. Additionally, due to the resource-intensive nature of fine-tuning, promptly updating LLMs to integrate evolving knowledge within TKGs for reasoning is impractical. To address these challenges, in this paper, we propose a Large Language Models-guided Dynamic Adaptation (LLM-DA) method for reasoning on TKGs. Specifically, LLM-DA harnesses the capabilities of LLMs to analyze historical data and extract temporal logical rules. These rules unveil temporal patterns and facilitate interpretable reasoning. To account for the evolving nature of TKGs, a dynamic adaptation strategy is proposed to update the LLM-generated rules with the latest events. This ensures that the extracted rules always incorporate the most recent knowledge and better generalize to the predictions on future events. Experimental results show that without the need of fine-tuning, LLM-DA significantly improves the accuracy of reasoning over several common datasets, providing a robust framework for TKGR tasks.

Main paper: 
 introduction: 
 Temporal Knowledge Graphs (TKGs) [1, 2] are the structured representations of the real world, which incorporate the temporal dimension to analyze how relations between entities evolve over time. Temporal Knowledge Graph Reasoning (TKGR) focuses on leveraging historical information within TKGs to forecast future events. Prior research [3, 4] on TKGR has primarily relied on temporal logical rules [5] or deep learning algorithms, such as graph neural networks [6] [7] [8] and reinforcement learning techniques [9] . However, the deep learning-based TKGRs often suffer from the lack of interpretability [10] and are difficult to dynamically update to accommodate new data in TKGs. While Large Language Models (LLMs) [11] , pretrained on large-scale text corpora, have exhibited extensive knowledge and reasoning ability. LLMs is able to effectively grasp intricate semantic and logical relationships within natural language, making them show remarkable performance across a wide range of tasks [12] [13] [14] . Recently, LLMs have also demonstrated surprising ability in temporal reasoning [15] [16] [17] . By utilizing their powerful contextual processing and pattern recognition abilities, LLMs can extract meaningful temporal patterns and complex temporal dependencies from historical data within TKGs [17] [18] [19] , thereby significantly enhancing their temporal reasoning capabilities. Thus, leveraging the capabilities of LLMs [20] holds great promise for enhancing the performance of TKGR tasks.Previous research on LLMs for TKGRs has primarily focused on prompting the LLMs with historical events and asking the LLMs to infer new facts [21] [22] [23] . Despite these accomplishments, LLMs are known to be black boxes, leaving it unclear which temporal patterns contribute to the reasoning results. Besides, LLMs suffer from the issue of hallucinations [24] , which further undermines the trustfulness of the results. Moreover, it is impractical to promptly update LLMs to incorporate the evolving knowledge within TKGs for reasoning.Due to the evolving nature, the knowledge within TKGs would continuously update over time, which results in a temporal distribution shift from the initial observations to the future facts [25, 26] . As illustrated in Figure 1 , the distribution of the relations in TKG changes dramatically over longer intervals. Despite LLMs possessing abundant knowledge via pre-training, it is still essential to accommodate up-to-date knowledge for reasoning. However, continually updating LLMs is highly impractical due to the intensive resources required for fine-tuning [27] . Additionally, TKGs usually contain significant noise, necessitating an efficient process to extract relevant information for LLMs to discern the underlying temporal patterns.To address these challenges, this paper proposes a Large Language Model-guided Dynamic Adaptation (LLM-DA) method for TKGR tasks, which dynamically adapts to the new knowledge and conduct interpretable reasoning powered by LLMs. Specifically, LLM-DA leverages the capabilities of LLMs to analyze historical data and extract temporal logical rules, unveiling temporal patterns and facilitating interpretable reasoning. To efficiently adapt to the new distribution of TKGs, LLM-DA introduces an innovative dynamic adaptation strategy. This strategy iteratively updates the rules generated by LLMs instead of updating the LLMs themselves with the latest events. The extracted rules are dynamically updated and ranked to ensure they consistently incorporate the most recent knowledge and improve predictions for future events, all without the resource-intensive process of LLM fine-tuning.In order to facilitate the rule generation and update, LLM-DA employs a contextual relation selector to meticulously filter the relations in TKG. The selector identifies the top-k most important relations for each rule head based on their semantic similarities. For example, given a rule head "president_of", the relevant relations might be "occupation_of" and "politician_of". These selected relations are fed into the LLMs as context to ensure LLMs are aligned with the temporal data, enhancing their abilities in uncovering the underlying temporal patterns.The main contributions of this paper are summarized as follows:• This paper attempts to harness the ability of Large Language Models (LLMs) for rule-based Temporal Knowledge Graph Reasoning (TKGR) to unveil temporal patterns and facilitate interpretable reasoning;• This paper proposes an innovative dynamic adaptation strategy that iteratively updates the LLM-generated rules with the latest events, allowing for better adaptation to the constantly changing dynamics within TKGs;• This paper introduces the contextual relation selector to identify the top k relevant relations, ensuring higher contextual relevance and enhancing the ability of LLMs to understand complex temporal patterns;• Experimental results on several widely used datasets show that LLM-DA significantly enhances TKGs reasoning accuracy without requiring fine-tuning LLMs.2 Related Work 
temporal knowledge graph reasoning: 
 Temporal Knowledge Graph Reasoning (TKGR) [28] [29] [30] [31] [32] aims to leverage historical information within TKGs to forecast future events, which can be roughly categorized into two groups: Rules-based TKGR methods and Deep learning-based TKGR methods.Rules-based TKGR methods [33] enhance TKGs inference by leveraging temporal logical rules to accurately predict future events. TLmod [34] introduces a sophisticated pruning strategy to derive rules and selects the high-confidence rules for TKGR tasks. TLogic [5] learns temporal logical rules from TKGs based on temporal random walks, and subsequently feeds these rules into a symbolic reasoning module for predicting future events. TILP [35] proposes a differentiable framework for temporal logical rule learning, utilizing constrained random walks to enhance the learning process. TFLEX [36] advances beyond learning simplistic chain-like rules by proposing a temporal featurelogic embedding framework. Although temporal logical rules can reveal hidden temporal patterns within TKGR, effectively extracting these rules from TKG still remains a significant challenge.Deep learning-based TKGR methods [37] [38] [39] employ the deep learning techniques to capture the hidden temporal patterns to predict the future events in TKGs. RE-NET [40] uses a recurrent event encoder and a neighborhood aggregator to encode historical facts and model their connections, enhancing future event predictions in TKGs. Based on RE-NET, RE-GCN [41] further employs RGCN and GRU to aggregate neighboring messages and model the temporal dependency. CyGNet [42] employs a copy-generation mechanism for capturing global repetition frequencies. TiRGN [43] integrates both local and global historical data to capture the sequential, repetitive, and cyclical patterns inherent in historical data. However, the deep neural networks adopted by these methods often lack interpretability, making it difficult to verify the predictions.Large Language Models (LLMs) for TKGR generally leverage the sufficient knowledge and reasoning ability of LLMs to conduct reasoning on TKGs. TIMEBENCH [44] proposes a comprehensive hierarchical temporal reasoning benchmark to provide a thorough evaluation for investigating the temporal reasoning capabilities of LLMs. Luo et al. [23] performs fine-tuning on known data and then leverage a sequence of established factual information to predict and generate the subsequent event in the series. PPT [21] converts the TKGC task into a masked token prediction task using a Pre-trained Language Model and designs specific prompts for various types of intervals between timestamps to enhance the extraction of semantic information from temporal data. GPT-NeoX [45] implements a method that utilizes GPT for TKGs forecasting through in-context learning without any fine-tuning. Similarly, Mixtral-8x7B-CoH [22] also does not require fine-tuning and adopts a "chain-of-history" Subsequently, LLM-DA updates these rules using current data. Finally, the updated rules are applied to predict future events.reasoning method to effectively generate high-order historical information step-by-step. Nevertheless, existing methods only prompt the LLMs with historical events from TKG, which are limited by the quality of input data and interpretability of LLMs.Different from the aforementioned LLM for TKGRs, the proposed LLM-DA is LLMs for rule-based TKGR method. By utilizing explicit rules, LLM-DA ensures the interpretability of the LLMgenerated processes and dynamically updates the rules to adapt to new data, thereby addressing a major limitation of LLMs-enhanced deep learning-based TKGRs. where the left-hand side denotes the rule head with relation r that can be induced by (←) the righthand rule body. The rule body is represented by the conjunction (∧) of a series of body relations r * ∈ {r 1 , ..., r l-1 } [46] .Different Data Types. Historical data refers to data that have occurred in the past, reflecting the state of things in the past period of time. Current data reflects the latest state of things in the present, from a more recent point in time. Future data refers to data that will occur, reflecting the possible future trend of things. Specifically, the historical data, current data and future data correspond to the training, validation, and test datasets of prior research [43, 22] . 
methodology: 
 In this section, we propose a novel Large Language Model-guided Dynamic Adaptation (LLM-DA) method for TKGR tasks. LLM-DA contains four main stages: Temporal Logical Rules Sampling explores the constrained Markovian random walks to extract temporal logical rules from the historical data; Rule Generation utilizes the powerful generative capabilities of LLMs to extract meaningful temporal patterns and complex temporal dependencies from historical data within TKGs; Dynamic Adaptation leverages LLMs to update the LLM-generated general rules using current Temporal logical rules sampling is a constrained Markovian random exploration process, which explores the Timestamp-Entity joint level weighted temporal random walks. Generally, the constrained Markovian random exploration process is not only strictly constrained by the graph structure but also deeply influenced by the temporal dimension when choosing the next node.Constrained Markovian Random Walks. Compared to traditional Markov random walks, constrained Markovian random walks primarily reflect in two key factors such as temporal order and temporal intervals when choosing the next node. As shown in Figure 3 , edges of the next state are selected based on the temporal order, and edges are weighted by the temporal interval. Specifically, edges with shorter interval receive higher weights, thus making the random walk more inclined to choose nodes that are temporally closer.To maximize the performance of random walks, LLM-DA needs to ensure that the random walks simultaneously satisfy the Markov property and additional constraints. Given the edge r(e x , e y , t l ), LLM-DA employs the Markovian random walks to search the closed temporal paths, further obtaining the set of candidates M r . To ensure the efficiency of our framework, LLM-DA introduces the filtering operator χ(t), which explores the temporal order to filter these candidate paths. The filtering operator χ(t) can be denoted as:EQUATIONAfter filtering these candidate paths, LLM-DA further introduces the temporal interval as the another constraint. This constraint selects the next node based on the transition probability P . For a temporal logical rule, the edge r(e x , e y , t) represents a connection from node e x to node e y at timestamp t. Thus, the transition probability of the constrained Markovian random walks is expressed as:EQUATIONwhere P xy (t) is the probability of transitioning from node e x to node e y at time t, and w(t) = exp(-λ(t -T )) denotes an exponential decay function that weights the edges based on the time difference, T is the current time, t is the timestamp of the edge, and λ is the decay rate parameter, which controls the weight given to more recent times. In this way, more recent times (i.e., t closer to T ) are assigned greater weight. The detailed theoretical analysis is shown in Appendix B. Through the constrained Markovian random walks on the historical data, LLM-DA extracts the temporal rules S from the sampled temporal paths.Rule generation typically utilizes the powerful generative capabilities of LLMs to improve the insufficient coverage and low quality of the extracted temporal rules S. Specifically, LLM-DA first employs the contextual relation selector to identify the top k relevant relations. Additonally, LLMs generally have powerful generative capabilities, allowing them to extract meaningful temporal patterns and complex temporal dependencies from historical data within TKGs. Thus, LLM-DA inputs the extracted rules S and the top k relevant contextual relations into LLMs to generate high-coverage and high-quality general rules.Firstly, LLM-DA filters the input contextual information through a contextual relation selector. TKGs generally contain rich temporal relations, requiring the processing of a substantial amount of context when LLMs conduct TKGR tasks. Contextual relation selector aims to filter the relation pool to identify the top k most relevant relations. For a temporal rule ρ := r(e s , e o , t l ) ← l-1 i=1 r * (e s , e o , t i ), LLM-DA leverages the pre-trained Sentence-Bert [47] to embed rule head r and each of its corresponding candidate relations c j into a common space, obtaining embedding vectors c j and r:EQUATIONFollowing, LLM-DA calculates the relevance score S(r, c j ) through the cosine similarity function:EQUATIONwhere • denotes the dot product operation, and ∥ • ∥ denotes the norm of the vector.Finally, LLM-DA sorts the relevance scores in descending order and selects the top k candidate relations as effective candidate relations, ensuring that the selected relations are semantically closely aligned with the rule head r:EQUATIONwhere {c j1 , c j2 , . . . , c j k } are the top k most relevant relations.After the above contextual relation selector, LLM-DA inputs the extracted temporal rules and the top k most relevant candidate relations {c j1 , c j2 , . . . , c j k } corresponding to the rule head r into LLMs to achieve the generation of the high-coverage and high-quality general rules S g . A simple example of prompt is shown in the following prompt box:You are an expert in TKGR, and please generate as many temporal logical rules as possible related to 'r' based on extracted temporal rules.For the relations in rule body, you are going to choose from the candidate relations: "{c j1 , c j2 , . . . , c j k }".The detailed instruction for Rule Generation can be found in the Appendix A.1.Dynamic adaptation generally leverages LLMs to update the LLMs-generated general rules using current data. Due to the evolving nature of TKGs, the LLMs-generated rules S g become less suitable for new data, causing high-quality rules to gradually degrade into low-quality rules. To address this, LLM-DA extracts temporal rules from the current data and uses these rules as a standard to update the low-quality rules. This ensures that these rules always incorporate the most recent knowledge and better generalize to the predictions on future events.As shown in the Prompt for Dynamic Adaptation prompt box, the input of LLMs for dynamic adaptation mainly contains two parts, including the update of low-quality rules and extracted rules from current data.You are an expert in TKGR, and please analyze these LLMs-generated rules and update the low-quality rules based on the extracted rules from current data.For the relations in rule body, you are going to choose from the candidate relations: "{c j1 , c j2 , . . . , c j k }".Low-Quality Rules. As TKGs evolve over time, the LLM-generated rule set S g may become increasingly difficult to fit to the current data, eventually turning into low-quality rules. Thus, LLM-DA applies the Confidence [48] metric to score each temporal rule ρ on the current data. Confidence generally measures the reliability of the temporal rule ρ, which can be defined as the proportion of temporal fact pairs that satisfy the rule body rule_body(ρ) and also satisfy the whole rule(ρ): where c ρ denotes the confidence of rule ρ. The higher the confidence, the greater the reliability of the rule. In other words, when the rule body is true, the likelihood of the rule head being true is higher. Subsequently, we select the subset of rules with low confidence S g(low) = {ρ ∈ S g | c ρ < θ}, where θ denotes the threshold of the low confidence.Extracted Rules from Current Data. To address the issue of the broad range of rules generated by LLMs, LLM-DA explores constrained Markovian random walks to extract temporal logical rules from current data, which serves as a standard to constrain the scope of dynamic adaptation. Through iteratively invoking LLMs, the accuracy of LLMs in predicting future events can be enhanced. The detailed prompt for Dynamic Adaptation refers to Appendix A.2.Finally, LLM-DA updates these LLMs-generated low-quality rules through the extracted rules from current data, further obtaining the rules set S d .Candidate reasoning aims to infer potential answers for the query by integrating the above LLMsgenerated rules and GNNs-based predictions. Specifically, LLM-DA mainly consists of two key modules: Rule-based Reasoning and Graph-based Reasoning.Rule-based Reasoning. Rule-based reasoning typically utilizes the above LLMs-generated highscoring rules to conduct in-depth logical reasoning within TKGs, deducing new entities as potential answers [5] . Given a query (e s , r, ?, t l ), LLM-DA scores the rule through the Equation 7, and then select the high-scoring rules:EQUATIONwhere S d is the rule set obtained after the Dynamic Adaptation process, S ′ d is the high-confidence rule set, in which the score c ρ of the rule ρ is greater than the threshold γ. Following the rule ρ ∈ S ′ d , we can find the reasoning paths and further derive the entity e ′ o :EQUATIONwhere e ′ o is the candidate derived based on the rule ρ, andt l-1 ≥ • • • ≥ t 1 .Considering the time decay property of temporal data, we further select candidates that are most relevant to the query:EQUATIONwhere Score (ρ,e ′ o ) indicates the score of the candidate e ′ o obtained through the searched path in TKGs based on rule ρ at the time point t o , c ρ denotes the confidence of the temporal rule ρ, λ represents the decay rate, and body(ρ)(e s , e ′ o , t l ) ∈ G denotes the path in TKGs that satisfies the rule body. Graph-based Reasoning. Due to inconsistent data distribution, candidates generated solely based on rules may not fully match all query answers. Thus, we introduce the graph-based reasoning function f g (Query) to further predict the candidates of the query, and the score can be computed through the inner product operation: , and vice versa. The whole score of the candidate can be computed as follows:EQUATIONEQUATIONwhere Score f represents the final score of the candidate e ′ o from rule-based reasoning and graph-based reasoning modules, and α is used to assign weights to different scores. Finally, LLM-DA aggregates all candidates, and then sorts these candidates to select those that best meet the query requirements. Baselines. The proposed LLM-DA is compared with several classic TKGR methods, including 1) TKGR methods: RE-NET [40] , RE-GCN [41] , TiRGN [43] and TLogic [5] ; 2) LLMs-based TKGRs: GPT-NeoX [45] , Llama-2-7b-CoH, Vicuna-7b-CoH [23] , Mixtral-8x7B-CoH [22] and PPT [21] .Here, LLM-DA selects RE-GCN and TiRGN as the graph-based reasoning function f g (Query). The detail of each baseline is described in Appendix C.2.Parameter Setting. The proposed LLM-DA uses the ChatGPT3 as the LLM for Rules Generation and Dynamic Adaptation. LLM-DA chooses Mean Reciprocal Rank (MRR) and Hit@N (N = 1, 3, 10) as evaluation metrics, and presents the filtered results (Appendix C.3). Additionally, LLM-DA sets the decay rate λ in Temporal Logical Rules Sampling and Candidate Generation, the threshold θ in Dynamic Adaptation, the min-confidence γ and the parameter α in Candidate Generation on both datasets as follows: λ = 0.1, θ = 0.01, α = 0.9 and γ = 0.01, except for α = 0.8 on ICEWS05-15.The number of iterations for the Dynamic Adaptation is set as 5. All experiments are implemented on a NVIDIA RTX 3090 GPU with i9-10900X CPU. 
performance comparison: 
 The link prediction experimental results are displayed of ICEWS14 and ICEWS05-15 in Table 1 , and the ICEWS18 is shown in Appendix C.4. The experimental analyses are listed as follows:(1) Experimental results indicate that even without fine-tuning, the proposed LLM-DA can still surpass all LLM-based TKGR methods. This phenomenon demonstrates that the dynamic adaptation strategy can effectively update LLMs-generated general rules with latest events to capture the evolving nature of TKGs, thereby significantly improving the accuracy of future event predictions. Additionally, we present the visualization experiment in Appendix C.6, which validates the superiority of the dynamic adaptation strategy.(2) Some LLMs-based TKGC methods such as PPT [21] and GPT-NeoX [45] , are not always superior to traditional TKGR methods. This is primarily because the rules generated by LLMs are too broad and sometimes fail to precisely adapt to specific data. However, the proposed LLM-DA outperforms the existing state-of-the-art benchmarks on all metrics. This phenomenon proves that LLM-DA can effectively guide LLMs in adjusting rules to the target distribution.(3) GPT-NeoX [45] is an important baseline as it also incorporates GPT as an LLM in TKGC tasks. However, LLM-DA shows significant improvement. This phenomenon indicates that the dynamic adaptation strategy can effectively update LLMs-generated rules to adapt to future data.(4) Furthermore, replacing the graph-based reasoning module f g (Query) with RE-GCN ("LLM-DA (RE-GCN)") and TiRGN ("LLM-DA (TiRGN)"), the MRR performance shows a slight variation. This variation highlights the importance of incorporating graph-based reasoning function in enhancing the ability to predict future events.In LLM-DA, dynamic adaptation aims to continuously update the generated rules to capture temporal patterns and facilitate future predictions. To further investigate its impact, we aim to answer the following questions: RQ1: Can the dynamic adaptation better extract the temporal patterns from TKGR for reasoning? RQ2: Can the dynamic adaptation adapt to different distributions over time? RQ3: Can the iterative dynamic adaptation improve the performance?Table 2 : Ablation study on different data without dynamic adaptation on both datasets. The best results are in bold. "LLM-DA w H" indicates "only using the historical data", "LLM-DA w C" is "only using the current data", and "LLM-DA w H+C" denotes "using the historical and current data". The ablation study on different data without dynamic adaptation aims to evaluate the impact of dynamic adaptation on performance. Dynamic adaptation typically employs the current data to update the rules generated by LLMs based on historical data. To demonstrate the superiority, we compare three variations, including "LLM-DA w H", "LLM-DA w C" and "LLM-DA w H+C", and the experimental results are shown in Table 2 . Specifically, the "LLM-DA w H+C" outperforms "LLM-DA w H" and "LLM-DA w C", indicating that large-scale historical data can provide general knowledge, while current data offers relevant knowledge. The combination of both enhances the prediction of future events. Furthermore, compared to "LLM-DA w H+C", LLM-DA shows a significant improvement. This phenomenon demonstrates that the dynamic adaptation strategy can effectively integrate historical data and current data, leveraging the current data to update the rules generated by LLMs on historical data.The temporal data has the time decay property, causing the issue of distributional shift. Specifically, the longer the time interval between the data, the more pronounced the shift becomes. To verify whether the dynamic adaptation can adapt to different distributions over time, we RQ3: To further verify whether the number of iterations of the dynamic adaptation module affects the performance, we conduct the different numbers of iterations experiment on both datasets. As shown in Figure 5 , the MRR performance exhibits an increasing trend as the number of iterations increases. These observations indicate that the dynamic adaptation strategy can continuously update the LLMs-generated rules with the latest events through iterations, thereby better adapting to the dynamic changes of TKGs. This further demonstrates the effectiveness and necessity of the dynamic adaptation strategy in handling the evolving nature of TKGs. Moreover, we conduct the parameter analysis experiment to validate the impact of the weight α in Appendix C.7. 
conclusion: 
 In this paper, we propose a novel Large Language Model-guided Dynamic Adaptation (LLM-DA) method to enhance TKGR tasks. Specifically, LLM-DA leverages a contextual relation selector to identify the top k most relevant relations, thereby selecting pertinent contextual information. Subsequently, LLM-DA harnesses the generative capabilities of LLMs to analyze historical data and derive general rules. Furthermore, LLM-DA proposes a dynamic adaptation strategy to update the LLM-generated rules with latest events, further capturing the evolving nature of TKGs. Experimental results on several datasets unequivocally demonstrate that LLM-DA achieves competitive performance compared to state-ofthe-art methods. Appendix D further analyzes the limitations of LLM-DA and provides an outlook for future work.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have listed all experimental settings in Subsection 5.1. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: All the experimental results are acquired by multiple trails of experiments, andwe report the average and standard deviation results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have provided sufficient information on the computer resources in Subsection 5.1. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have ensured that all code, data, and models used in our research are cited appropriately, and we have adhered to the licensing agreements and terms of use specified by the original authors.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Prompt for Dynamic Adaptation Temporal Logical Rules "R l (X, Y, T l ) ← ∧ l-1 i=1 R i (X, Y, T i )" typically describe how the relation 'R l ' between entities 'X' and 'Y ' evolves from past time steps 'T i (i = {1, • • • , (l -1)})' (Rule body) to the next timestamp 'T l ' (Rule head), and please follow the constraint "T1 ≤ • • • ≤ T l-1 < T l ".You are an expert in temporal knowledge graph reasoning, and please analyze these LLMs-generated rules and update the low-quality rules based on the extracted rules from current data.Rule Head: inv_Provide_humanitarian_aid (X, Y , T ) Low Quality Temporal Logical Rules:Make_a_visit (X, Y , T 2 ) ← Provide_military_protection_or_peacekeeping (X, Y , T 1 ) Make_a_visit (X, Y , T 4 ) ← Appeal_for_diplomatic_cooperation_(such_as_policy_ support) (X, Z 1 , T 1 ) & inv_Consult (Z 1 , Z 2 , T 2 ) & inv_Make_statement (Z 2 , Y , T 3 ) Generated High Quality Temporal Logical Rules: Make_a_visit (X, Y , T 2 ) ← Express_intent_to_meet_or_negotiate (X, Z 1 , T 1 ) & Make_a_visit (Z 1 , Y , T 2 ) Make_a_visit (X, Y , T 3 ) ← Consult (X, Z 1 , T 1 ) & Engage_in_negotiation (Z 1 , Z 2 , T 2 ) & Make_a_visit (Z 2 , Y , T 3 ) ••• Example 2: Rule Head: inv_Provide_humanitarian_aid (X, Y , T ) Low Quality Temporal Logical Rules: inv_Provide_humanitarian_aid (X, Y , T 2 ) ← inv_Investigate (X, Y , T 1 ) inv_Provide_humanitarian_aid (X, Y , T 2 ) ← inv_Engage_in_diplomatic_ coopera- tion (X, Y , T 1 ) Generated High Quality Temporal Logical Rules: inv_Provide_humanitarian_aid (X, Y , T 2 ) ← inv_Provide_aid (X, Y , T 1 ) inv_Provide_humanitarian_aid (X, Y , T 3 ) ← Criticize_or_denounce (X, Z 1 , T 1 ) & Sign_formal_agreement (Z 1 , Y , T 2 )...•••••• Extracted Rules from Current Data: ••••••Let's think step-by-step, and please update the low-quality temporal logic rules related to "{head_rule(X, Y, T )}" based on the extracted rules from current data.For the relations in rule body, you are going to choose from the candidate relations: "{c j1 , c j2 , . . . , c j k }".Return the rules only without any explanations.In conducting a theoretical analysis of constrained Markovian random walks, we can examine their characteristics and effects from several key aspects:• Impact of Time Weighting: By using weighted transition probabilities P xy (t l ), constrained Markovian random walks can assign different importance to neighboring nodes at different time points. In particular, through the exponential decay function w(t) = exp(-λ(t l -T )), we can control the extent to which time influences the transition probabilities. The larger the parameter λ, the stronger the preference for recent events, which helps capture short-term dynamic changes. • Traversal Properties: An important feature of constrained Markovian random walks is their ability to traverse different paths in TKGs. Since the transition probabilities consider time information, constrained Markovian random walks are more likely to explore paths with temporal continuity compared to traditional random walks, thereby better reflecting temporal relationships.Table 3 provides a comprehensive overview of the statistical information for the entire datasets used in our experiments. This includes key metrics such as the number of entities, the number of relations, the number of temporal facts, and the granularity of each dataset. This statistical information is crucial for understanding the scale and complexity of the datasets, as well as for evaluating the performance of LLM-DA under different experimental conditions.Integrated Crisis Early Warning System (ICEWS) is a TKG for international crisis early warning and analysis, which collects and integrates data on political events, social dynamics, and international relations worldwide. Specifically, ICEWS14 [49] contains events that occurred from 1/1/2014 to 12/31/2014; ICEWS05-15 [49] includes events that occurred between the years 2005 and 2015; ICEWS18 [22] covers events from 1/1/2018 to 10/31/2018. The proposed model LLM-DA is compared with some classic TKGR methods, including TKGR methods and LLMs-based TKGR methods.• RE-NET [40] : RE-NET first TKGC method for predicting future events, which employs a recurrent event encoder and a neighborhood aggregator to encode historical facts; • RE-GCN [41] : RE-GCN is an extension of RE-NET, which introduces RGCN and GRU to encode historical facts; • TiRGN [43] : TiRGN employs constrained random walks for temporal logical rule learning;• TLogic [5] : TLogic learns the temporal logical rules from TKGs based on temporal random walks;• GPT-NeoX [45] : GPT-NeoX introduces in-context learning with GPT for TKG forecasting;• Llama-2-7b-CoH, Vicuna-7b-CoH [23] : Llama-2-7b-CoH, Vicuna-7b-CoH fine-tune Llama-2-7b and Vicuna-7b on historical data for TKG prediction; • Mixtral-8x7B-CoH [22] : Mixtral-8x7B-CoH leverage Mixtral-8x7B to explore high-order histories step-by-step for the TKG prediction; • PPT [21] : PPT introduces the Pre-trained Language Model to predict future events.We evaluate our model using Mean Reciprocal Rank (MRR) and Hit@N metrics, where N is set to 1, 3, and 10. Higher scores indicate better performance. Lastly, we present the final experimental results, termed as filtered, which exclude all corrupted quadruplets from the TKG.made by a model. It is calculated by taking the reciprocal of the rank of the correct answer for each query and then averaging these reciprocal ranks across all queries. Mathematically, for each query, if the rank of the correct answer is r, then the MRR score for that query is 1 r . The overall MRR score is the average of these reciprocal ranks. MRR values range between 0 and 1, where a higher value indicates better performance in ranking predictions.• Hit@N : Hit@N measures the proportion of correct predictions within the top N ranked results. For example, when N = 1, we are interested in whether the correct answer is predicted within the top-ranked result. When N = 3, we evaluate whether the correct answer is within the top three predictions. The value of N can be chosen based on the specific task and dataset characteristics, commonly set to 1, 3, or 10. Hit@N values also range between 0 and 1, representing the proportion of correct predictions.The link prediction experimental results of ICEWS18 are displayed in Table 4 . The proposed LLM-DA outperforms existing state-of-the-art benchmarks on ICEWS18 across all metrics. Moreover, GPT-NeoX [45] is an important baseline because it also introduces GPT as LLMs. However, our proposed method still improves all metrics on ICEWS18. For example, the proposed method obtains 15.6% improvement under Hit@10 on ICEWS18. These phenomena demonstrate that the dynamic adaptation strategy can effectively update LLM-generated rules with current data, ensuring that the rules consistently incorporate the most recent knowledge and better generalize predictions for future events.Table 4 : Link prediction results on ICEWS18 dataset. The best results are in bold and -means the result is unavailable. ♠ denotes the TKGR methods, ♣ represents the LLMs-based TKGR. The results of the models with † are derived from [22] , and ‡ is from [23] . Long horizontal link prediction generally predict the events happening at time point t + k * △T , where △T denotes the interval between time points and k controls how far we will predict. To perform the long horizontal link prediction, we adjust the integral length △T on ICEWS14 and ICEWS05-15 datasets. As described in Figure 6 and Figure 7 , we report the results corresponding to different △T on ICEWS14 and ICEWS05-15 and compare the proposed LLM-DA with the strongest baselines RE-GCN [41] and TiRGN [43] . Experimental results shows that LLM-DA can achieve the best performance on different △T both datasets. This phenomenon demonstrates that LLMs can effectively learn meaningful rules, and the dynamic adaptation strategy can fully update the LLM-generated rules, allowing for better adaptation to the constantly changing dynamics within TKGs. To visually present the changes of rule and candidate rankings during the Rule Generation and Dynamic Adaptation, we conducted a visualization experiment in Table 5 . For the query (China, Consult, ?, 2014-12-05), the actual answer Japan ranked fifth during the Rule Generation stage, but after the Dynamic Adaptation, it advanced to the top position. Meanwhile, the sequencing of rules in two phases also exhibits significant changes. For instance, the ranking of the rule "Consult ← inv_Make_an_appeal_or_request & Express_• • • _meet_or_negotiate" changes from fifth to first position. This highlights the efficacy of the dynamic adaptation strategy, which can update the LLM-generated rules with latest events to capture the evolving nature of TKGs. Rules for Rule GenerationEQUATIONRules for Dynamic Adaptation① inv_Make_an_appeal_or_request & Express_intent_to_meet_or_negotiate ② Consult & Express_intent_to_cooperate & Consult ③ Meet_at_a_'third'_location ④ Express_intent_to_engage_in_diplomatic_cooperation & inv_Host_a_visit & inv_Consult ⑤ inv_Host_a_visit & Host_a_visit & inv_Discuss_by_telephone Candidates for Dynamic Adaptation ① Japan ② France ③ South_Korea ④ South_Africa ⑤ Malaysia1 Express_intent_to_engage_in_diplomatic_cooperation represents Express_intent_to_engage_in_diplomatic_cooperation_(such_as_policy_support). To experimentally study the effect of the weight α on ICEWS14 and ICEWS05-15, we tune the weight in a range {0, 0.1, • • • , 1} and observe the link prediction experimental results. As shown in Figure 8 , the MRR performance exhibits a trend of initially increasing and then decreasing, reaching the peak at α = 0.9 for ICEWS14, and α = 0.8 for ICEWS05-15. These observations underscore that the weight α plays a crucial role in influencing the performance of LLM-DA. Specifically, the weight α shows better performance at higher values, highlighting the importance of rule-based reasoning module.Moreover, when α = 1, indicating that only rules are used to generate candidates, the MRR performance decreases. This demonstrates that although LLMs can learn meaningful rules, the rules generated based on historical data cannot fully fit the future data due to the difference in distribution. This results in a few queries failing to find matching candidates when relying solely on rules for candidate generation.Our limitations can be summarized as follows: (1) LLM-DA does not consider the semantics of nodes, which may reduce the quality of the sampled rules. (2) LLM-DA does not generate rules based on queries, which may result in the rules generally lacking specificity. (3) LLM-DA requires manually constructed prompts, which means that prompts need to be redesigned for different datasets. Moreover, manual construction of prompts also incurs significant costs. In the future, we will employ automated prompt learning to generate query-dependent rules through LLMs. Furthermore, we integrate the semantics of nodes during the temporal logical rules sampling stage.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We have uploaded the anonymous code in the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research conducted in the paper complies with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We use the snapshot of ChatGPT taken from February 15th 2024 (gpt-3.5-turbo-0215) to ensure the reproducibility. 


Decision: This paper leverages LLMs to infer rules for Temporal Knowledge Graph Reasoning (TKGR) -- link prediction with TKGs.  More specifically; 
1) it first uses constrained random works to discover initial rules; 
2) then the rules and related relations with scores (accuracy) are fed to an LLM to generate rules with better quality/coverage;
3) furthermore a dynamic adaptation strategy -- repeat the rule discovery (random walk), scoring (accuracy) and rewrite (LLM) steps for new data

Experimental results over a few TKG datasets show that , LLM-DA, outperforms graph-based TKG method and LLM-based method in accurate reasoning.

This work demonstrates the benefit of generating explicit reasoning rules from LLMs compared to pure graphic based approach and pure LLM based approach.

Reviews: review: paper_summary: paper_summary: This paper introduces Large Language Models-guided Dynamic Adaptation (LLM-DA), a novel approach for Temporal Knowledge Graph Reasoning (TKGR). LLM-DA leverages LLMs to extract temporal logical rules from historical data, providing interpretable reasoning. It also incorporates a dynamic adaptation strategy to update these rules with the latest events, ensuring the extracted rules reflect the most recent knowledge.  Experimental results over several common datasets show that , LLM-DA, without the need of fine-tuning, significantly outperforms graph-based TKG method and LLM-based method in accurate reasoning.
strength_weakness: strength_weakness: - Long-horizon forecasting concerns: Concerns will be raised about the dynamic adaptation in cases where no near historical data is available, i.e. in the long-horizon forecasting task where the future data patterns and rules may largely differ from the seen data, how generalizable and robust this proposed method will be?
- Limited evaluation of generated rules:  While the LLM mainly participates in the rule extraction part, the paper lacks a comprehensive quantitative and qualitative evaluation of the generated rules. For instance, there's no assessment of how well the LLM generates symbolically and logically correct rules.
questions: questions: Please refer to the questions raised in Weakness section.
limitations: limitations: Yes, the author has discussed the limitations in Appendix D, and proposed reasonable solutions fas the future works.
paper_summary: paper_summary: This paper explores the use of Large Language Models (LLMs) for Temporal Knowledge Graph Reasoning (TKGR). Specifically, the paper leverages LLMs for rule-based TKGR to identify temporal patterns and enable interpretable reasoning. Additionally, it introduces a dynamic adaptation strategy that iteratively updates the LLM-generated rules with the latest events, enhancing the model's adaptation to the evolving dynamics of TKGs.
strength_weakness: strength_weakness: W1. There is limited analysis of the constrained Markovian random walks.
W2. While Figure 1 intuitively reflects the paper's motivation, it could be improved. Moreover, the dynamic adaptation strategy, which should be the core focus, occupies a relatively small proportion in Figure 2.
W3. Several minor grammatical and expression issues need attention.
questions: questions: Q1. Do the other modules in the paper, such as the contextual relation selector and the graph-based reasoning function, also require pre-training in addition to the LLMs?
Q2. The current experimental analysis is too superficial. To better validate the proposed method, a more in-depth analysis of the experiments is needed.
limitations: limitations: The authors clearly point out that this work is not query-dependent, which can lead to a lack of specificity in the generated rules. This provides a solid direction for future work to enhance the specificity of generated rules. Additionally, it is recommended that the authors discuss in depth the problems posed by not considering node semantics. Failure to consider node semantics may result in the loss of important contextual information and affect the overall accuracy of the model. 
A more detailed discussion of this limitation would greatly benefit the paper.
paper_summary: paper_summary: This paper proposes Large Language Models-guided Dynamic Adaptation (LLM-DA) to leverage LLMs to extract temporal logical rules for TKGR. Experimental results demonstrate LLM-DA significantly improves reasoning accuracy without the need for fine-tuning the LLM.
strength_weakness: strength_weakness: The writing could be improved. Some figures should be polished.

LLM-DA leverages GPT 3.5 as the LLM model, while the other baselines use only 7B LLM models. The authors should also conduct experiments to study the impact of different LLM models.
questions: questions: Please see Weaknesses.
limitations: limitations: N/A

Comments: title: Response to Reviewer CXWD
Response to Reviewer 9wEQ
Response to Reviewer thzY
comment: We sincerely appreciate your professional suggestions and look forward to your feedback on our rebuttal. Thank you once again for your valuable comments.
We sincerely appreciate your timely reply and support for our work. We will add the experiments evaluating the generated rules in future editions of the paper. Your suggestions have been invaluable in enhancing the quality of our research, and we are eager to incorporate these improvements. Thank you once again for your constructive comment and encouragement.
Thanks the authors for the rebuttal, especially the evaluation of the generated rules addresses my original concern, please consider adding this to future editions of the paper.
Thank you for your timely reply! We sincerely appreciate your support for our work.
The authors have addressed well my concerns, and, considering the comments from other reviewers, I would like to improve my rating.