Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoning
Authors: Jiapu Wang
Kai Sun
LINHAO LUO
Wei Wei
Yongli Hu
Alan Wee-Chung Liew
Shirui Pan
Baocai Yin
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Temporal Knowledge Graph
Knowledge Graph Reasoning
Primary area: natural_language_processing
TL;DR: We propose a Dynamic Adaptation method to guide LLMs for rule-based TKGR tasks.

Abstract: Temporal Knowledge Graph Reasoning (TKGR) is the process of utilizing temporal information to capture complex relations within a Temporal Knowledge Graph (TKG) to infer new knowledge. Conventional methods in TKGR typically depend on deep learning algorithms or temporal logical rules. However, deep learning-based TKGRs often lack interpretability, whereas rule-based TKGRs struggle to effectively learn temporal rules that capture temporal patterns. Recently, Large Language Models (LLMs) have demonstrated extensive knowledge and remarkable proficiency in temporal reasoning. Consequently, the employment of LLMs for Temporal Knowledge Graph Reasoning (TKGR) has sparked increasing interest among researchers. Nonetheless, LLMs are known to function as black boxes, making it challenging to comprehend their reasoning process. Additionally, due to the resource-intensive nature of fine-tuning, promptly updating LLMs to integrate evolving knowledge within TKGs for reasoning is impractical. To address these challenges, in this paper, we propose a Large Language Models-guided Dynamic Adaptation (LLM-DA) method for reasoning on TKGs. Specifically, LLM-DA harnesses the capabilities of LLMs to analyze historical data and extract temporal logical rules. These rules unveil temporal patterns and facilitate interpretable reasoning. To account for the evolving nature of TKGs, a dynamic adaptation strategy is proposed to update the LLM-generated rules with the latest events. This ensures that the extracted rules always incorporate the most recent knowledge and better generalize to the predictions on future events. Experimental results show that without the need of fine-tuning, LLM-DA significantly improves the accuracy of reasoning over several common datasets, providing a robust framework for TKGR tasks.

Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoning
Jiapu Wang', Kai Sun'; Linhao Luo?, Wei Wei Yongli Hulf, Alan Wee-Chung Liew
Shirui Pan"t Baocai Yin
Beijing University of Technology, China, Moonah University, Australia University of Hong Kong, China, *Grfffith University, Australia
(jpwang, Bunkai)@emails bjut. edu.. cn, linhao. luo@monash. edu, weiwei frtweil206css @06cs@@@eaee @mail..cm {huyongli, ybc} edu. cn, {a.liew s. s.pan)@griffith edu. au
Abstract
Temporal Knowledge Graph Reasoning (TKGR) the process utilizing temporal information to capture complex relations within Temporal Knowledge Graph (TKG) to infer new knowledge. Conventional methods in TKGR typically depend deep learning algorithms temporal logical rules. However, deep learning- based TKGRs often lack interpretability, whereas rule-based TKGRs struggle to effectively learn temporal rules that capture temporal patterns. Recently, Large Language Models (LLMs) have demonstrated extensive knowledge and remarkable proficiency in temporal reasoning. Consequently, the employment of LLMs for Temporal Knowledge Graph Reasoming (TKGR) has sparked increasing interest among researchers. Nonetheless, LLMs known function as black boxes, making it challenging to comprehend their reasoning process. Additionally, due to the resource-intensive nature of fine-tuning, promptly updating L.MMs integrate evolving knowledge within TKGs for reasoning is impractical. To address these challenges, this paper, we propose Large Language Models-guided Dynamic Adaptation (LLM-DA) method for reasoning on TKGs. Specifically, LLM-DA harnesses the capabilities of LLMs to analyze historical data and extract temporal logical rules. These rules unveil temporal patterns and facilitate interpretable reasoning. To account for the evolving nature of TKGs, a dynamic adaptation strategy is proposed to update the LLM-ggeerated rules with the latest events. This ensures that the extracted rules always incorporate the most recent knowledge and better generalize to the predictions on future events. Experimental results show that without the need of fine-tuning, I.IM-DAA significantly improves the accuracy of reasoning over several common datasets, providing robust framework for TKGR t
Introduction
are the struetured representations of the real world, which incorporate the temporal dimension to analyze how relations between entities evolve over time. Temporal Knowledge Graph Reasoning (TKGR) focuses on leveraging historical information within TKGs forecast future events. Prior research K on TKGR has primarily relied on temporal logical rules [5] or deep learning algorithms, such as graph neural networks [ and reinforcement learning techniques [9]. However, the deep learning-bis based TKGRs often suffer from the lack of
Temporal Knowledge Graphs (TKGs) 1
"Equally Important
1 Corresponding authors
*Code and data available http://gghha...omm/jjiaaagg/a/aL
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

January
August
October
December
Illtoorical Data
Current Data
Future Data
Dynamic Adaptation
B.A.A::::: Generated Rules
Distrihution of Leggggreertted Rules from Historical Data
: :Objective Distribution
A brief description of LLM-DA. Specifically, LLM-DA hamesses LLMs to formulate general rules on historical data. Subsequently. LLM-DA dynamically guides the LLMs update these rules based on current data, ensuring they more accurately reflect the objective distribution.
Figure 1:
interpretability [10] and are difficult to dynamically update to accommodate new data in TKGs. While rule-based methods offer great interpretability and flexibility, effectively learning temporal logical rules and adapting them to new knowledge remains a huge challenge.
Large Language Models (LLMs) [1] pretrained on large-scale text corpora, have exhibited extensive knowledge and reasoning ability. LLMs is able to effectively grasp intricate semantic and logical relationships within natural language, making them show remarkable performance across wide range of tasks [12 Recently, LLMs have also demonstrated surprising ability in temporal reasoning 17. By utilizing their powerful contextual processing and pattern recognition abilities, LLMs can extract meaningful temporal patterns and complex temporal dependencies from historical data within TKGs (17-91.. thereby significantly enhancing their temporal reasoning capabilities. Thus, leveraging the capabilities LLMs (20) holds great promise for enhancing the performance of TKGR tasks.
Previous research on LLMs for TKGRs has primarily focused on prompting the LLMs with historical events and asking the LLMs to infer new facts (31] Despite these accomplishments, LLMs are known to be black boxes, leaving it unclear which temporal patterns contribute to the reasoning results. Besides, LLMs suffer from the issue of hallucinations (24), which further undermines the trustfulness of the results. Moreover, is impractical to promptly update LLMs to incorporate the evolving knowledge within TKGs for reasoning.
Due to the evolving nature. the knowledge within TKGs would continuously update over time, which results in a temporal distribution shift from the initial observations to the future facts [25] 26 As illustrated in Figure [ the distribution of the relations in TKG changes dramatically over longer intervals. Despite LLMs possessing abundant knowledge via pre-training, it is still essential to accommodate up-to-date knowledge for reasoning. However, continually updating LLMs is highly impractical due to the intensive resources required for fine-tuning (27). Additionally, TKGs usually contain significant noise, necessitating an efficient process to extract relevant information for LLMs to discern the underlying temporal patterns.
To address these challenges, this paper proposes a Large Language Model-guided Dynamic Adaptation (LLM-DA) method for TKGR tasks, which dynamically adapts to the new knowledge and conduct interpretable reasoning powered by LLMs. Specifically, LLM-DA leverages the capabilities of LLMs to analyze historical data and extract temporal logical rules, unveiling temporal patterns and facilitating interpretable reasoning. To efficiently adapt to the new distribution of TKGs, LLM-DA introduces an innovative dynamic adaptation strategy. This strategy iteratively updates the rules generated by LLMs instead of updating the LLMMs themselves with the latest events. The extracted rules are dynamically updated and ranked to ensure they consistently incorporate the most recent knowledge and improve predictions for future events, all without the resource-intensive process of LLM inniitnnng.

---

In order to facilitate the rule generation and update, LLM-DA employs a contextual relation selector to meticulously filter the relations in TKG. The selector identifies the top-k most important relations for each rule head based on their semantic similarities, For example, given a rule head "president. of",, the relevant relations might be "occupation_ o_ and "politia poolitician_oot These selected relations are fed into the L.MSs context to ensure LLMS are aligned with the temporal data, enhancing their abilities in uncovering the underlying temporal patterns.
The main contributions of this paper are summarized as follows:
This paper attempts to harness the ability of Large Language Models (LLMs) for rule-based Temporal Knowledge Graph Reasoning (TKGR) to unveil temporal patterns and facilitate interpretable reasoning;
This paper proposes an innovative dynamic adaptation strategy that iteratively updates the L.M--genrraed rules with the latest events, allowing for better adaptation the constantly changing dynamics within TKGs;
This paper introduces the contextual relation selector to identify the top k relevant relations, ensuring higher contextual relevance and enhancing the ability of LLMs understand complex temporal patterns;
Experimental results on several widely used datasets show that LLM-DA significantly enhances TKGs reasoning accuracy without requiring fine-tuning LLMs.
Related Work
2.1 Temporal Knowledge Graph Reasoning
Temporal Knowledge Graph Reasoning (TKGR) (28 aims to leverage historical information within TKGs to forecast future events, which can be roughly categorized into two groups: Rules-based TKGR methods and Deep learning-based TKGR methods.
Rules-based TKGR methods [35] enhance TKGs inference by leveraging temporal logical rules to accurately predict future events. TLmod [36] introduces a sophisticated pruning strategy to derive rules and selects the gigh-confidence rules for TKGR tasks. TLogic [5] learns temporal logical rules from TKGs based on temporal random walks, and subsequently feeds these rules into a symbolic reasoning module for predicting future events. TILP (37 proposes differentiable framework for temporal logical rule learning, utilizing constrained random walks to enhance the learning process. TFLEX [38] advances beyond learning simplistic chain-like rules by proposing a temporal feature- logic embedding framework. Although temporal logical rules can reveal hidden temporal patterns within TKGR, effectively extracting these rules from TKG still remains a significant challenge.
Deep learning-baseed TKGR methods [39 employ the deep learning techniques to capture the hidden temporal patterns to predict the future events in TKGs. RE-NET 43 uses recurrent event encoder and a neighborhood aggregator to encode historical facts and model their connections, enhancing future event predictions in TKGs. Based on RE-NET, RE-GCN [44) further employs RGCN and GRU to aggregate neighboring messages and model the temporal dependency. CyGNet employs copy-generation mechanism for capturing global repetition frequencies. TiRGN integrates both local and global historical data to capture the sequential, repetitive, and cyclical patterns inherent historical data. However, the deep neural networks adopted by these methods often lack interpretability, making it difficult to verify the predictions.
45
46
2.2 Large Language Models for TKGR
Large Language Models (LLMs) for TKGR generally leverage the sufficient knowledge and reasoning ability of LLMs to conduct reasoning on TKGs. TIMEBENCH [47] proposes a comprehensive hierarchical temporal reasoning benchmark to provide thorough evaluation for investigating the temporal reasoning capabilities of LLMs. Luo et al. (33 performs fine-tuning on known data and then leverage sequence of established factual information to predict and generate the subsequent event in the series. PPT 21 converts the TKGC task into a masked token prediction task using a Pre-trained Language Model and designs specific prompts for various types of intervals between timestamps to enhance the extraction of semantic information from temporal data. GPT-NeoX [48) implements
3

---

Temporal Logical Rules Sampling
Dynamic Adapiation
Candidate Generation
Sampled Rules
Large Language Models
Generated
Rule
Russia
condema
war
Rules Generation
Relevant
support. f2
hold geett
provide sid, 1s
Relations
Genernted Rules
Germany
Likraine
support
Current Data
Candidates
war Jr
Contextual Selector
0.6
consuit,
condemn, f
Score
Generated Rules
war,
Construnee Markovian Randim Walks
LLMs
Query: Germany support whom at 2014-12-05?
Historical Data
Selected Rules
Sampled Rules
Answer: Ukraine.
^ :Rles Application
G Dynamic Aduptation E. S...iiioonn---eeeeeeeee Rule-bbesd Ressoning 1. Grapb-bused Reasoning
The Framework of LLM-DA. Specifically, LLM-DA first analyzes historical data to extract temporal rules and utilizes the powerful generative capabilities of LLMs generate general rules. Subsequently, LLM-DA updates these rules using current data. Finally, the updated rules are applied to predict future events.
Figure 2:
method that utilizes GPT for TKGs forecasting through in-context learning without any fine-tuning. Similarly, Mixtral-8x7B-CoH (22) also does not require fine-tuning and adopts a "chain-of-bistory" reasoning method to effectively generate high-order historical information step-by-step. Nevertheless, existing methods only prompt the LLMs with historical events from TKG, which are limited by the quality of input data and interpretability of LLMs.
Different from the aforementioned LLM for TKGRs, the proposed LLM-DA is LLMs for rule-based TKGR method. By utilizing explicit rules, LLM-DA ensures the interpretability of the L.MM- generated processes and dynamically updates the rules to adapt to new data, thereby addressing a major limitation of LLMs-enhanced deep learning-based TKGRs.
Preliminary
Temporal Knowledge Graph (TKG). TKG {E, R, T, 2} is collection of entity set E, relation set R. and timestamp set T. Specifically, each quadruplet is denoted as (ea, r, Co, () E Q, where Es. 6, E & represent the entities, R denotes the relation and / T is the timestamp.
Temporal Logical Rule. Temporal logical rules define the relation between two entities e, and e at timestamp t.
P: = r(es, Co: 41) <- A/ttr"(es Cor 42)
where the left-hand side denotes the rule head with relation r that can be induced by (+) the right- hand rule body. The rule body is represented by the conjunction (A) of a series of body relations r* (f1...., r1-1} 49).
Different Data Types. Historical data refers to data that have occurred in the past, reflecting the state of things in the past period of time. Current data reflects the latest state of things in the present, from more recent point in time. Future data refers to data that will occur, reflecting the possible future trend of things. Specifically, the historical data, current data and future data correspond to the training, validation, and test datasets of prior research 46 22
4 Methodology
In this section, we propose a novel Large Language Model-guided Dynamic Adaptation (LLM- DA) method for TKGR tasks. LLM-DA contains four main stages: Temporal Logical Rules Sampling explores the constrained Markovian random walks to extract temporal logical rules from the historical data; Rule Generation utilizes the powerful generative capabilities of LLMs to extract

---

consult. f
support, fa
war,
provide aid,
support,
war,
condemn, 4
war, l
condemn, L
condemn, 4
P2
war, 4
support, 6
Temporal Order
Temporal Interval
Figure 3: The constraints of the constrained Markovian random walks. "x" denotes this path does not exist, "P," indicates the transition probability.
meaningful temporal patterss and complex temporal dependencies from historical data within TKGs; Dynamic Adaptation leverages L.LMs to update the L..M-ggnerated general rules using current data; Candidate Reasoning combines rules-based reasoning and graphs-based reasoning to generate candidates. The whole framework is illustrated in Figure
4.1 Temporal Logical Rules Sampling
Temporal logical rules sampling is constrained Markovian random exploration process, which explores the Timestamp- Entity joint level weighted temporal random walks. Generally, the constrained Markovian random exploration process is not only strictly constrained by the graph structure but also deeply influenced by the temporal dimension when choosing the next node.
Constrained Markovian Random Walks, Compared to traditional Markov random walks, con- strained Markovian random walks primarily reflect in two key factors such as temporal order and temporal intervals when choosing the next node. As shown in Figure edges of the next state selected based on the temporal order, and edges are weighted by the temporal interval. Specifically, edges with shorter interval receive higher weights, thus making the random walk more inclined to choose nodes that are temporally closer.
To maximize the performance of random walks, LLM-DA needs to ensure that the random walks simultaneously satisfy the Markov property and additional constraints. Given the edge r(ez, Ey tt), LLM-DA employs the Markovian random walks to search the closed temporal paths, further obtaining the set candidates M,. To ensure the efficiency of our framework, LLM-DA introduces the filtering operator x(t), which explores the temporal order to filter these candidate paths. The filtering operator xt) can be denoted as:
- {
1, 0, otherwise.
if ffte t tr,
x(t)
After filtering these candidate paths, LLM-DA further introduces the temporal interval as the another constraint. This constraint selects the next node based on the transition probability P. For temporal logical rule, the edge r(ez. ey, t) represents a connection from node e. to node ey at timestamp t. Thus, the transition probability the constrained Markovian random walks is expressed as:
exp(-X(t- - T)) Pny(t) Σ E(e exp( -)))
where Pa (t) is the probability of transitioning from node eg to node ey time and w(t) exp(-A(t T)) denotes an exponential decay function that weights the edges based on the time difference, T is the current time, is the timestamp of the edge, and À is the decay rate parameter, which controls the weight given to more recent times. In this way. more recent times (i.e., t closer to T)) are assigned greater weight. The detailed theoretical analysis shown in Appendix B Through the constrained Markovian random walks on the historical data, L.LMDDA extracts the temporal rules from the sampled temporal paths.
4.2 Rule Generation
Rule generation typically utilizes the powerful generative capabilities of LLMs to improve the insufficient coverage and low quality of the extracted temporal rules S. Specifically, L.MMDA first employs the contextual relation selector to identify the top & relevant relations. Additonally, LLMs
5

---

generally have powerful generative capabilities, allowing them to extract meaningful temporal patterns and complex temporal dependencies from historical data within TKGs. Thus, LLM-DA inputs the extracted rules S and the top lo relevant contextual relations into L.MMs to generate high-coverage and high-quality general rules.
Firstly, LLM-DA filters the input contextual information through contextual relation selector. TKGs generally contain rich temporal relations, requiring the processing substantial amount of context when LLMs conduct TKGR tasks. Contextual relation selector aims to filter the relation pool to identify the top k most relevant relations. For temporal rule r(ea, co; 44) N r" r" Cay li), LLM-DA leverages the pre-trained Sentence- [50] to embed rule head and each of its corre- sponding candidate relations Cj into common space, obtaining embedding vectors с and r:
Cj. Sentence-Bert(cr r).
Following, LLM-DA calculates the relevance score Srr, (y) through the cosine similarity function:
((r.c) = r C3
|r||||c/
where denotes the dot product operation, and denotes the norm of the vector.
Finally, LLM-DA sorts the relevance scores in descending order and selects the top k candidate relations effective candidate relations, ensuring that the selected relations are semantically closely aligned with the rule head
(c C2. Cj. (or,
(6)
where {cr Cra...., Cye} are the top k most relevant relations.
After the above contextual relation selector, LLLMDDA inputs the extracted temporal rules and the top k most relevant candidate relations {Ci Cjas ............................ Cja.) :.j. corresponding to the rule head r into LLMs to achieve the generation of the high-coverage and high-quality general rules Sy- simple example of prompt is shown in the following prompt box:
Prompt for Rule Generation
You are an expert in TKGR, and please generate as many temporal logical rules as possible related to 'r' based on extracted temporal rules. For the relations in rule body, you are going to choose from the candidate relations: "(Cji-Cay
The detailed instruction for Rule Generation can be found in the Appendix A.l
4.3 Dynamic Adaptation
Dynamic adaptation generally leverages L.MMs to update the MeMM--geeeetedd general rules using current data. Due the evolving nature of TKGs, the L.MMs-enerated rules S, become less suitable for new data, causing high-quality rules to gradually degrade into low-quality rules. To address this, LLM-DA extracts temporal rules from the current data and uses these rules as a standard to update the low-quality rules. This ensures that these rules always incorporate the most recent knowledge and better generalize to the predictions on future events.
As shown in the Prompt for Dynamic Adaptation prompt box, the input of LLMs for dynamic adaptation mainly contains two parts, including the update of low-quality rules and extracted rules from current data.
Prompt for Dynamic Adaptation
You are an expert in TKGR, and please analyze these LLMMs-genrattdd rules and update the low-quality rules based on the extracted rules from current data. For the relations in rule body, you are going to choose from the candidate relations:
"fch, Cia

---

Low-Quality Rules. As TKGs evolve over time, the JLM-generated rule set Sg may become increasingly difficult to fit to the current data, eventually turning into low-quality rules. Thus. LLM- DA applies the Confidence [5] metric to score each temporal rule p on the current data. Confidence generally measures the reliability of the temporal rule p. which can be defined as the proportion of temporal fact pairs that satisfy the rule body rule_ rule_odyy(p) and also satisfy the whole rule(p):
Number of temporal fact pairs satisfying rule_ _bdd((p)
Cp Number of temporal fact pairs satisfying rule(p)
where cp denotes the confidence of rule p. The higher the confidence, the greater the reliability of the rule. In other words, when the rule body is true, the likelihood of the rule head being true is higher. Subsequently, we select the subset of rules with low confidence Syllow) {o Sg Cp 0), where 0 denotes the threshold of the low confidence.
Extracted Rules from Current Data. To address the issue of the broad range of rules generated by LLMs, LLM-DA explores constrained Markovian random walks to extract temporal logical rules from current data, which serves as standard to constrain the scope of dynamic adaptation. Through iteratively invoking LLMs, the accuracy of LLMs in predicting future events can be enhanced. The detailed prompt for Dynamic Adaptation refers to Appendix (A.2]
Finally, LLM-DA updates these LLMs- -eneerrte low-quality rules through the extracted rules from current data, further obtaining the rules set Sa-
4.4 Candidate Reasoning
Candidate reasoning aims to infer potential answers for the query by integrating the above LLMs- generated rules and GNNs-based predictions. Specifically, LLM-DA mainly consists of two key modules: Rule-based Reasoning and Graph-based Reasoning.
Rule-based Reasoning. Rule-based reasoning typically utilizes the above ALMs-generater high- scoring rules to conduct in-depth logical reasoning within TKGs, deducing new entities as potential answers 5 Given a query (es, r, ?, tz), LLM-DA scores the rule through the Equation 7 and then
select the high-scoring rules:
(8) where Sa the rule set obtained after the Dynamic Adaptation process, Sy is the high-confidence rule set, in which the score Cp of the rule is greater than the threshold y. Following the rule p Sy we can find the reasoning paths and further derive the entity e'::
= >T.E E Sa),
(Ea, T2 6, ti) + (e Tiss E'o ti),
where e is the candidate derived based on the rule p. and 4-1 2 ... 2 41 Considering the time decay property of temporal data, we further select candidates that are most relevant the query:
Score = Σ
E
(Cp + exp(p - too)),
(10)
PE R (e. A.jE
where (p.ee) indicates the score of the candidate e' obtained through the searched path in TKGs based on rule p at the time point to. cp denotes the confidence of the temporal rule p. À represents the decay rate, and body(p) c'2, ti)
denotes the path in TKGs that satisfies the rule body.
Graph-based Reasoning, Due to inconsistent data distribution, candidates generated solely based on rules may not fully match all query answers. Thus, we introduce the graph-based reasoning function [g(Query to further predict the candidates of the query, and the score can be computed through the inner product operation:
(cort = (Qu(e)) e.)
(11)
Since the candidates of Rule-based Reasoning E(p.et) and Graph- based Reasoning Eggraph have and vice versa. The whole score of the candidate can be computed as follows: Score,-
the overlap and difference, we assign the score (core(p.e)) as 0 where e'2 E Eigraph.e)) eg & ((p.)))
Score + (1 a) (core
(12)
where Score, represents the final score of the candidate e', from rule-based reasoning and graph-based reasoning modules, and used to assign weights to different scores. Finally, L.LMDDA aggregates all candidates, and then sorts these candidates to select those that best meet the query requirements.
7

---

Table 1: Link prediction results on ICEWSI4 and ICEWS05- means the result is unavailable. A denotes the TKGR methods, & represents the LLMs-bast based TKGR, and LLM-DA (+) indicates replacing the graph-based reasoning module [g(Query) with TKGRs
15. The best results are bold and
ICEWS14
ICEWS05-15
Type
Models
Train
MRR
Hit@
Hite3
10
MRR
Hite1
Hir@3
Hir@ 10
0.290
0.576
0.441
RE-GON
0.478
0.535
TIRGN
0.441
0.341
0.497
0.650
0.495
0.389
0,559
TLogic
0,390
0.295
0.437
0.573
0.459
0.360
0,518
0.646
PPT
0.384
0.289
0.425
0.570
0.389
0.286
0.434
0.586
LJama-2-77 CoH
0.349
0,470
0.591
0.386
0.541
0.699
Vicuna-7b-Coff
0.328
0.457
0.656
0.392
0.546
0.707
GPT-NeoX
X
0.334
0,460
0.565
Mixtral-8x7B-
X
0.439
0.331
0,496
0.649
0.497
0,380
0,564
0,713
LLM DA (RE GCN)
X
0.461
0.356
0,515
0.662
0.501
0.394
0.568
0,710
LLM (TRRGN)
X
0.471
0.369
0,526
0.671
0.521
0.416
0.586
0.728
5 Experiments
5.1 Experimental Settings
Datasets. ICEWS14 (52) and ICEWS05-15 [52] are the subset of Integrated Crisis Early Warning System (ICEWS), which a TKG of international political events and social dynamics. ICEWS14 contains events that occurred in 2014, while ICEWS05-15 contains events that occurred between 2005 and 2015. Details of datasets can be referred to Appendix C.T
Baselines. The proposed LLM-DA is compared with several classic TKGR methods, including 1) TKGR methods: RE-NET 43 RE-GCN 44), TiRGN [66 and TLogic [5): 2) LLMs-based TKGRs:
GPT-NeoX (48) 1lama-2-7b-Con
Vicuna-7b-Col (23) Mixtral-&x7B--oH [22] and PPT [21].
Here, LLM-DA selects RE-GON and TiRGN the graph-based reasoning function (quuryy). The
detail of each baseline is described in Appendix C.2
Parameter Setting. The proposed LLM-DA uses the ChatGPT as the LLM for Rules Generation and Dynamic Adaptation. LLM-DA chooses Mean Reciprocal Rank (MRR) and Hir@ N (N = 1,3, 10) Additionally, LLM-DA sets
as evaluation metrics, and presents the filtered results (Appendix C.3 the decay rate À in Temporal Logical Rules Sampling and Candidate Generation, the threshold 0 in Dynamic Adaptation, the min-confidence 7 and the parameter in Candidate Generation on both datasets as follows: À = 0.1, 9 = 0.01, a = 0.9 and y = 0.01, except for a = 0.8 on ICEWS05 15. The number of iterations for the Dynamic Adaptation set as 5. All experiments are implemented on a NVIDIA RTX 3090 GPU with 19-00 CPU.
5.2 Performance Comparison
The link prediction experimental results are displayed ICEWS14 and ICEWS05-15 in Table and the ICEWS18 is shown in Appendix C.4 The experimental analyses are listed as follows:
(1) Experimental results indicate that even without fine-tuning, the proposed LLM-DA can still surpass all LLM-based TKGR methods. This phenomenon demonstrates that the dynamic adaptation strategy effectively update LLMss-enerated general rules with latest events to capture the evolving nature of TKGs, thereby significantly improving the accuracy of future event predictions. Additionally, we present the visualization experiment in Appendix C.6 which validates the superiority of the dynamic adaptation strategy.
(2) Some LLMs-based TKGC methods such as PPT 21 and GPT-NeoX 48
not always superior to traditional TKGR methods. This is primarily because the rules generated by LLMs are too broad and sometimes fail to precisely adapt to specific data. However, the proposed LLM-DA outperforms the existing state-of-the -ft--o-tthherrrrr the benchmarks on all metrics. This phenomenon proves that LLM-DA can effectively guide LLMs in adjusting rules to the target distribution.
"We use the snapshot of ChatGPT taken from February 15th 2024 (gpt-3.5 1ar.3..turbb-0211 to ensure the reproducibility.
8

---

(3) GPT-NeoX [48] is an important baseline as it also incorporates GPT as an LLM in TKGC tasks. However, LLM-DA shows significant improvement. This phenomenon indicates that the dynamic adaptation strategy can effectively update L.LMs-generated rules to adapt to future data.
(4) Furthermore, replacing the graph-based reasoning module ʃ (QQuery)) with RE-GCN ("LLM-DA (RE-GCN)") and TiRGN ("LLM-DA (TiRGN)"), the MRR performance shows a slight variation. This variation highlights the importance of incorporating graph-based reasoning function in enhancing the ability to predict future events.
5.3 Analysis of Dynamic Adaptation
In LLM-DA, dynamic adaptation aims to continuously update the generated rules to capture temporal patterns and facilitate future predictions. To further investigate its impact, we aim to answer the following questions: RQ1: Can the dynamic adaptation better extract the temporal patterns from TKGR for reasoning? RQ2: Can the dynamic adaptation adapt to different distributions over time? RQ3: Can the iterative dynamic adaptation improve the performance?
Table 2: Ablation study on different data without dynamic adaptation on both datasets. The best results are in bold. "LLM-DA w H" indicates "only using the historical data", "LLM-DA w C" is "only using the current data", and "LLM-DA w H+C" denotes "using the historical and current data".
ICEWS14
ICBW505-13
Models
MRR
Hit
Hire3
10
MRR
Hit@11
Hite3
Hit
LLM-DA (TIRGN) H
0,450
0.345
0.502
0.656
0.503
0.395
0.563
0.709
LLM-DA (TRGN) C
0.454
0.350
0.507
0.657
0.508
0.400
0,570
0.714
LLM-DA (TRRGN)
0.352
0.639
0,718
LLM-DA(TRRGN
0.471
0.369
0,526
0.671
0,521
0.416
0,586
0,728
The ablation study on different data without dynamic adaptation aims to evaluate the impact of dynamic adaptation on performance. Dynamic adaptation typically employs the current data to update the rules generated by LLMs based on historical data. To demonstrate the superiority, we compare three variations, including LLL.-DD w H" "LLM-DA w C" and "LLM-DA w H+C", and the experimental results are shown in Table Specifically, the "LLM- DA w H+C" outperforms "LLM-DA H" and "LLM-DA w C", indicating that large-scale historical data can provide general knowledge, while current data offers relevant knowledge. The combination of both enhances the prediction of future events. Furthermore, compared to "IJLM-DA H+C", LLM-DA shows a significant improvement. This phenomenon demonstrates that the dynamic adaptation strategy can effectively integrate historical data and current data, leveraging the current data to update the rules generated by LLMs on historical data.
RQ1:
The temporal data has the time decay property, causing the issue of distributional shift. Specifically, the longer the time interval between the data, the more pronounced the shift be- comes. To verify whether the dynamic adaptation can adapt to different distributions over time, we conduct the time interval segmented prediction experiment, which typi- cally segments the future data into multiple time intervals based on chronological order, and then con- ducts the link prediction experiment for each time interval. As shown in Figure 4 the proposed LLM-DA ex- hibits a significant performance im- provement in the MRR metric over RE-GCN and TiRGN in each time interval. This indicates that the pro- posed LLM-DA can accurately capture the temporal dependencies in TKGs and adapt to the continu- ously changing temporal data. Furthermore, in Appendix C.5] conduct long-term horizontal link prediction to forecast events occurring at future time points.
RQ2:
0.540
0.550
KENSIA
0.504
TIRGH
SON THRAN
0.468
0.432
0.78
0.396
0.454
0.360
[7.2]]
[33,0]]
[30,24]
(25,01]
D.21
D5.10) (40.222) (22) (00 (202.70) Time Intervals
Time Intervals
Figure 4: Time interval segmented prediction: MRR perfor-
mance on both datasets compared to different baselines.
RQ3: To further verify whether the number of iterations of the dynamic adaptation module affects the performance, we conduct the different numbers of iterations experiment on both datasets. As shown in Figure 5 the MRR performance exhibits an increasing trend as the number of iterations

---

increases. These observations in- dicate that the dynamic adaptation strategy can continuously update the L.MMs-generatd rules with the latest events through iterations, thereby bet- ter adapting to the dynamic changes of
0.485
0.530
ICEWS14
ICEWS05-15
0.475
0.520
MRR
MRR
TKGs. This further demonstrates the effectiveness and necessity of the dy- namic adaptation strategy in handling the evolving nature of TKGs. More- over, conduct the parameter anal- ysis experiment to validate the impact of the weight in Appendix C.7
0.455
0.500
0.445
0.490
Number of Iterations
Figure 5: Comparison with different number of iterations on both datasets.
6 Conclusion
In this paper, we propose a novel Large Language Model-guided Dynamic Adaptation (LLM-DA) method to enhance TKGR tasks. Specifically, LLM-DA leverages contextual relation selector to identify the top & most relevant relations, thereby selecting pertinent contextual information. Subsequently, LLM-DA harnesses the generative capabilities of LLMs analyze historical data and derive general rules. Furthermore, LLM-DA proposes a dynamic adaptation strategy to update the LLM-generated rules with latest events, further capturing the evolving nature of TKGs. Experimental results on several datasets unequivocally demonstrate that LLM-DA achieves competitive performance compared to state-of- oatt--of-the-arr the-art methods. Appendix D further analyzes the limitations of LLM-DA and provides an outlook for future work.
A cknowledgment
This work funded by the National Key R&D Program of China (No. 20217D01 11902), National Natural Science Foundation of China (No. U21B2038) R&D Program of Beijing Municipal Educa- tion Commission 1772002100050000 The first author is funded by the China Scholarship Council (CSC) from the Ministry of Education, China.
References
[1] Kalev Leetaru and Philip A Schrodt. GDELT: Global data on events, location, and tone. In ISA Annual Convention, pages 1-49, 2013.
[2] Japp Wang, Boyue Wang, Junbin Gao, Xiaoyan Li, Yongli Hu, and Baocai Yin. TDN: Triplet distributor network for knowledge graph completion. IEEE Transactions Knowledge and Data Engineering. 35(12):13002-13111 2023.
[3] Jiapu Wang, Boyue Wang, Meikang Qiu, Shirui Pan, Xiong, Heng Liu, Linhao Luo, Tengfei Liu, Yongli Hu, Baocai Yin, et al. A survey on temporal knowledge graph completion: Taxonomy, progress, and prospects. arXiv preprint arXiv:2308. 02457, 2023.
[4] Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, Xinwang Liu, Fuchun Sun, and Kunlun He. A survey of knowledge graph reasoning on graph types: Static, dynamic, and multi-modal. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[5] Yushan Liu, Yumpu Ma, Marcel Hildebrandt, Mitchell Joblin, and Volker Tresp. Tlogic: Temporal logical rules for explainable link forecasting temporal knowledge graphs. In the AAAI Conference on Artificial Intelligence, pages 4 4120-4127, 2022.
[6] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Yu Philip. A comprehensive survey on graph neural networks. JEEE Transactions on Neural Networks and Learning Systems, 32(1):4-24, 2020.
[7] Yizhen Zheng, Shirui Pan, Vincent Lee, Yu Zheng, and Philip S Yu. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. Advances in Neural Information Processing Systems, 35:10809-10877 2022.
10

---

[8] Yizhen Zheng. He Zhang, Vincent Lee, Yu Zheng, Xiao Wang, and Shirui Pan. Finding the missing-hon half: Graph complementary learning for homophily peanoppill--oneeeeee graphs. In International Conference on Machine Learning, pages 42492-42505 PMLR, 2023.
prone and heterophily-Po pappeyeoonoeee
[9] Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. Traeeraveler: Reinforcement Learning for Temporal Knowledge Graph Forecasting. In the Conference on Empirical Methods in Natural Language Processing, pages 6306-8319, 2021.
[10] Linhao Luo, Jiaxin Ju, Bo Xiong, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Chatrule: Mining logical rules with large language models for knowledge graph reasoning. arXiv preprint arXiv:23009.// 03388 2023.
[11] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering. pages 1-20, 2024.
[12] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In the conference on Computer Vision and Pattern Recognition, pages 7784-7793, 2018.
[13] Jeffrey Dalton, Laura Dietz, and James Allan. Entity query feature expansion using knowledge base links. In ACM SIGIR Conference on Research & Development in Information Retrieval, pages 365-374, 2014.
[14] Marco Antonio Calijorne Soares and Fernando Silva Parreiras. A literature review on question answering techniques, paradigms and systems. Journal of King Saud Siwtersstt-Copputte and Information Sciences, 32(6))633-644 646, 2020.
[15] Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. arXiv preprint 2rXiv:2401.06853 2024.
[16] Qingyu Tan, Hwee Tou Ng, and Lidong Bing. Towards benchmarking and improving the temporal reasoning capability of large language models. arXiv preprint 2tli:23000089553 2023.
[17] Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. Back to the future: Towards explainable temporal reasoning with large language models. arXiv preprint arXiv:2310.0107, 2023.
[18) Rujun Han, Xiang Ren, and Nanyun Peng. Econet: Effective continual pretraining of language models for event temporal reasoning. arXiv preprint arXiv:2012.15283, 2020.
[19] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-Ilm: Time series forecasting by reprogramming large language models. In International Conference Learning Representations, 2024.
[20] Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning. In International Conference on Learning Representations, 2024.
[21] Wenjie Xu, Ben Liu, Miao Peng, Xu Jia, and Min Peng. Pre-trained language model with prompts for temporal knowledge graph completion. arXiv preprint arXiv:2305.07911 2023.
[22] Yuwei Xia, Ding Wang, Qiang Liu, Liang Wang, Shu Wu, and Xiaoyu Zhang. Enhancing tem- poral knowledge graph forecasting with large language models via chain-of-history reasoning. arXiv preprini arXiv:2402.1 /4382, 2024.
[23] Ruilin Luo, Tianle Gu, Haoling Li, Junzhe Li, Zicheng Lin, Jiayi Li, and Yujiu Yang. Chain of history: Learning and forecasting with llms for temporal knowledge graph completion. arXiv preprint 2rXiv:2401.06072 2024.

---

[24] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang, Qian- glong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint 1..5222311.05522, 2023.
[25] Ruijie Wang, Zheng Li, Dachun Sun, Shengzhong Liu, Jinning Li, Bing Yin, and Tarek Abdelzaher. Learning to sample and aggregate: Few-shot reasoning over temporal knowledge graphs. Advances in Neural Information Processing Systems, 35: 35:16863-16876 2022.
[26] Graham Cormode, Vladislav Shkapenyuk, Divesh Srivastava, and Bojian Xu. Forward decay: A practical time decay model for streaming systems. In the International Conference on Data Engineering, pages 138, 2009.
[27] Tongtong Wu, Linhao Luo, Yuan-Fang Li. Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01 364, 2024.
[28] Luyi Bai, Mingzhuo Chen, Lin Zhu, and Xiangxi Meng. Multi-hop temporal knowledge graph reasoning with temporal path rules guidance. Expert Systems with Applications, 223:1 19804, 2023.
[29] Xingtong Ge, Yi Yang. Ling Peng. Luanjie Chen, Weichao Li, Wenyue Zhang, and Jiahui Chen. Spatio-temporal knowledge graph based forest fire prediction with multi source heterogeneous data. Remote Sensing, 14(14):349,, 2022.
[30] Ke Liang, Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang, Xiangjun Dong, and Xinwang Liu. Knowledge graph contrastive learning based on relation-symmetricaa structure. IEEE Transactions on Knowledge and Data Engineering, 36(1):226-238, 2023.
[31] Ke Liang, Lingyuan Meng, Sihang Zhou, Wenxuan Tu, Siwei Wang, Yue Liu, Meng Liu, Long Zhao, Xiangjun Dong, and Xinwang Liu. Mines: Message intercommunication for inductive relation reasoning over neighbor-enhanced subgraphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10645-10653, 2024.
[32] Yili Wang, Yixin Liu, Xu Shen, Chenyu Li, Kaize Ding, Rui Miao, Ying Wang, Shirui Pan, and Xin Wang. Unifying unsupervised graph-level anomaly detection and duitt--distribution detection: A benchmark. arXiv preprint arXiv:2406. 15523, 2024.
[33] Jiapu Wang, Zheng Cui, Boyue Wang, Shirui Pan, Junbin Gao, Baocai Yin, and Wen Gao. IME: Integrating multi-curvature shared and specific embedding for temporal knowledge graph completion. arXiv preprint arXiv:2403.1 19881, 2024.
[34] Jiapu Wang, Boyue Wang, Junbin Gao, Shirui Pan, Tengfei Liu, Baocai Yin, and Wen Gao. MADE: Multicurvature adaptive embedding for temporal knowledge graph completion. IEEE Transactions on Cybernetics, 2024.
[35] Shiyuan Li, Yixin Liu, Qingfeng Chen, Geoffrey I Webb, and Shirui Pan. Noise-resilient unsu- pervised graph representation learning via multi-hop feature quality estimation. In Proceedings of the 33rd ACM International Conference Information and Knowledge Management, 2024.
[36] Luyi Bai, Wenting Yu, Die Chai, Wenjun Zhao, and Mingzhuo Chen. Temporal knowledge graphs reasoning with iterative guidance by temporal logical rules. Information Sciences, 621: 22-35, 2023.
[37] Siheng Xiong, Yuan Yang, Faramarz Fekri, and James Clayton Kerce. TILP: Differentiable learning of temporal logical rules on knowledge graphs. In the International Conference on Learning Representations, pages 1-11, 2023.
[38] Xueyuan Lin, Chengjin Xu, Fenglong Su, Gengxian Zhou, Tianyi Hu, Ningyuan Li, Mingzhi Sun, Haoran Luo, et al. TFLEX: Temporal feature-logic embedding framework for complex reasoning over temporal knowledge graph. Advances in Neural Information Processing Systems, pages 73039-73081, 2023.
12

---

[39] Songgaojun Deng, Huzefa Rangwala, and Yue Ning. Dynamic knowledge graph based multi- event forecasting. In the ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pages 1585-1595, 2020.
[40] Jiapu Wang, Boyue Wang, Junbin Gao, Xiaoyan Li, Yongli Hu, and Baocai Yin. QDN: A quadruplet distributor network for temporal knowledge graph completion. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[41] Yixin Liu, Shiyuan Li, Yu Zheng, Qingfeng Chen, Chengqi Zhang, and Shirui Pan. Are: A generalist graph anomaly detector with in-context learning. In Advances in Neural Information Processing Systems, 2024.
[42] Jiapu Wang, Boyue Wang, Junbin Gao, Yongli Hu, and Baocai Yin. Multi-concepl representation learning for knowledge graph completion. ACM Transactions on Knowledge Discovery from Data, 17(1):1-19) 19, 2023.
[43] Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. Recurrent Event Network: Autoregressive Structure Inferenceover Temporal Knowledge Graphs. In the Conference on Empirical Methods in Natural Language Processing, pages 6669-6683, 2020.
[44] Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. Temporal knowledge graph reasoning based on evolutional representation learning. In the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 408 417,447,410 2021.
[45] Cunchao Zhu, Muhao Chen, Changjun Fan. Guangquan Cheng, and Yan Zhang. Learning from history: Modeling temporal knowledge graphs with sequential sopyggeneration networks. In the AAAI Conference on Artificial Intelligence, pages 4732-4740, 2021.
[46] Yujia Li, Shiliang Sun, and Jing Zhao. TiRGN: Time-guided recurrent graph network with local-global historical patterns for temporal knowledge graph reasoning. In the International Joint Conferences on Artificial Intelligence, pages 2152-2158, 2022.
[47] Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. Timebench: A comprchensive evaluation of temporal reasoning abilities in large language models. arXiv preprint arXiv:2311. 2023.
[48] Dong-Ho Lee, Kian Ahrabian, Woojeong Jin, Fred Morstatter, and Jay Pujara. Temporal knowledge graph forecasting without knowledge using in-context learning. arXiv preprint 10061: 2023.
[49] Eric Salvat and Marie-Laure Mugnier. Sound and complete forward and backward chainings of graph rules. In the International Conference on Conceptual Structures, pages 248 248-262, 1996.
[50] Nils Reimers and Iryna Gurevych. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, the Conference on Empirical Methods in Natural Language Processing, pages 3982-3992, 2019.
[51] Luis Antonio Galárraga, Christina Teffioudi, Katja Hose, and Fabian Suchanek. AMIE: associa- tion rule mining under incomplete evidence in ontological knowledge bases. In the international conference on World Wide Web, pages 413-424 413-422,22 2013.
[52] Alberto Marcia-Durán Sebastijan Dumančić, and Mathias Niepert. Learning sequence encoders for temporal knowledge graph completion. In the Conference on Empirical Methods Natural Language Processing, pages 4816 4816-4821, 2018.
13

---

Appendix
A Prompt for LLMs
A.1 Prompt for Rule Generation
Prompt for Rule Generation
Temporal Logical Rules (X Y, Ti) < Rx, Y, T:)" typically describe how the relation R2' between entities X and "Y" evolves from past time steps 'T: (i {1, { 1)}}' (Rule body) to the next timestamp 'Ti' (Rule head), and please follow the constraint "T S Ti-1 Ti"
You are an expert in temporal knowledge graph reasoning, and please generate as many temporal logical rules possible related to 'Rt' based extracted temporal rules.
Here are a few examples:
Example 1:
Rule Head:
poaperate_ccooomccalll YX,Y, T)
Extracted Rules:
Cooperate_cconommict ecconmmicall (X, Y. T2) <- Provide_aid aid (X, Y, Ti)
Cooperate__cconomicalyy (X, TT) +- aisit__visit (X, Zı, Ti) & Negotiate (Zi,
Y, T.)))
Generated Temporal Logical Rules:
eeciroocclcco (X, Y, T2) < Engage in oogneeotiiiion (X, Vi) Cooperate_economicalll Cooperate economically (X, Y, T3) f- inv Low_Engage_in _negoottatioonnn (X, Z1, Tı) &
Make_ visit (Z,, Y, T2)
Example 2:
Rule Head:
Appeal for www.eecooommceieee aid (X, T)
Extracted Rules:
Appeal_for fow_fooooomicceooccoooooop_eeee_ aid (X, Y, T2) <- tatiue__o_stp_mmiltary
(X, Y,T))
Ti) & wow.oooooooicceeeooooooeeeeeeeee aid (X, Y, Ta)
Appeal (Z,, Y, T2)
(X, Zi.
Generated Temporal Logical Rules:
an__aaaaaaaa appeal request (X, Y. Ti) Appeal for economic, aid (X, Y. T2) Make an
Appeal_for for economic aid (X, Y, T3) + inv Make an appeal or orrrequest (X, Zi.
Ti) & Make statement (Z,, Y. T2)
Extracted Rules from Historical Data:
Let's think step-by-step. please generate many possible most relevant temporal rules that are relative to "[head_rulee rulee Y, T))" based on the above extracted rules from historical data.
For the relations rule body, you are going to choose from the candidate relations: "(Cj, Ciay....
Cju )"
Return the rules only without any explanations.
14

---

A.2 Prompt for Dynamic Adaptation
Prompt for Dynamic Adaptation
Temporal Logical Rules (X,(X. R, Y, T," typically describe how the relation R2,' between entities "X' and 'Y" evolves from past time steps 'T, (i = {1. {1.... (l-
1))) (Rule body) to the next timestamp 'Ti' (Rule head), and please follow the constraint "T S ... S T- Ti"
You are an expert in temporal knowledge graph reasoning, and please analyze these LLMs- MaMs--eeeetted rules and update the low-quality rules based on the extracted rules from current data.
Here are a few examples:
Example 1:
Rule Head:
inv
aid (X, T)
Low Quality Temporal Logical Rules:
Make visit (X, Y. T2) Provide www.imaalllloeeeeetie prooeetiionoooppetteeeeto peacckkeppig (X, Y,
T
Make_ a_ visii (X, Y. T4) Appeal Loneal_fo__aie dit.:/omaaiicccopprattooonnn Eeteeraationssuhh___ee _a__ass_____ Becchass_poliee sospppolcy____
support) (X, Z1. Ti) & inv_ CCnssul (Z1, Zz, T2) & inv inv_Make_statemene (Z2, Y, T3)
Generated High Quality Temporal Logical Rules:
Make a_ visit (X, Y, T2) +
(X, Z1, Ti) &
Make aivii (Zi, Y. T2) Make_a_vistt a_viit (X, Y, T3) < Consult (X, ZT,Ti) & innggt_innnegotiatton (Z,, Zz
T2) & Make a visit (Z2, Y, T3)
Example 2:
Rule Head:
inv_ Trovide_humaninne aid (X, Y, T)
Low Quality Temporal Logical Rules: inv Provide huninddrrmmmitariinnn aid (X, Y. T2) inv lnv_snvstte (X, Ti)
inv www_Provide__uneee sowiide_humaittariannnno aid (X, Y. T2) <- inv BEngaggeinndilllnneeeeo coopera-
tion (X, Ti)
Generated High Quality Temporal Logical Rules:
inv Provide humanitarian aid (X, Y. T2) + i Provide aid (X, Y. Y.T.)
inv Provide humanitarian aid (X, Y. TT) Criticize or denounce (X, Z1, Tı &
Sign formal agreement (21, Y, Tb)
Low-quality Temporal Logical Rules:
Extracted Rules from Current Data:
Let's think step-by-step, and please update the low-quality temporal logic rules related to "(head rule(X, Y, T)}" based on the extracted rules from current data. For the relations rule body, you are going to choose from the candidate relations:
"(Cn Cia...,
Return the rules only without any explanations.
15

---

B Theoretical Analysis of the Constrained Markovian Random Walks
In conducting a theoretical analysis of constrained Markovian random walks, we can examine their characteristics and effects from several key aspects:
Impact of Time Weighting: By using weighted transition probabilities (ry(()), constrained Markovian random walks can assign different importance to neighboring nodes at different time points. In particular, through the exponential decay function w(t) exp(-A(fz T)), we can control the extent to which time influences the transition probabilities. The larger the parameter X, the stronger the preference for recent events, which helps capture short-term dynamic changes.
Traversal Properties: An important feature of constrained Markovian random walks is their ability to traverse different paths in TKGs. Since the transition probabilities consider time information, constrained Markovian random walks are more likely to explore paths with temporal continuity compared to traditional random walks, thereby better reflecting temporal relationships.
Experiments
C.1 Datasets
Table provides a comprehensive overview of the statistical information for the entire datasets used in our experiments. This includes key metrics such as the number of entities, the number of relations, the number of temporal facts, and the granularity of each dataset. This statistical information is crucial for understanding the scale and complexity of the datasets, as well for evaluating the performance of L.LMDDA under different experimental conditions.
Integrated Crisis Early Warning System (ICEWS) is TKG for international crisis early warning and analysis, which collects and integrates data on political events, social dynamics, and international relations worldwide. Specifically, ICEWSI4 14 [52] contains events that occurred from 1/1/2014 to 12/31/ /0044; ICEWS05-1E 15 [52] includes events that occurred between the years 2005 and 2015; ICEWS18 22
covers events from 1/1/2018 to 10/31 /2/18.
Table 3: Statistic information of whole datasets.
Datasets
#Entities
#Relations #Historical Data #Current Data #Future Data #Granularity
ICEWS14
6,869
230
74,845
8,514
7,371
24 hours
ICEWS05-15
10,094
251
368,868
46,302
46,159
24 hours
ICEWS18
23,033
256
373,018
45,995
49,545
24 hours
C.2 Baselines
The proposed model LLM-DA is compared with some classic TKGR methods, including TKGR methods and LLMs-based TKGR methods.
TKGR methods
RE-NET 43
RE-NET first TKGC method for predicting future events, which employs recurrent event encoder and neighborhood aggregator to encode historical facts;
RE-GCN 44 RE-GCN is an extension of RE-NET, which introduces RGCN and GRU to encode historical facts;
TiRGN 46
TiRGN employs constrained random walks for temporal logical rule learning;
TLogic 5:
TLogic learns the temporal logical rules from TKGs based on temporal random
walks;
LLMs-based TKGR methods
GPT-NeoX 48] GPT-NeoX introduces in-context learning with GPT for TKG forecasting:
16

---

llma-2-7b-Coff Vicuna-7b-Coo [23]:
Vicuna-7b-CoH fine-tune Llama-
2-7b and Vicuna-7b on historical data for TKG prediction;
Mixtral-8x7B-CCol :22:: Mixtral-&x7B-Coll leverage Mixtral-8x7B explore high-order
histories step-by-step for the TKG prediction;
PPT 21:
PPT introduces the Pre-trained Language Model to predict future events.
C.3
Link Prediction Metrics
We evaluate our model using Mean Reciprocal Rank (MRR) and Hit@ N metrics, where N is set to 3, and 10. Higher scores indicate better performance. Lastly, we present the final experimental results, termed filtered, which exclude all corrupted quadruplets from the TKG.
Mean Reciprocal Rank (MRR): MRR measures the average accuracy of ranking predictions made by model. It is calculated by taking the reciprocal of the rank of the correct answer for each query and then averaging these reciprocal ranks across all queries. Mathematically, for each query, if the rank of the correct answer is r, then the MRR score for that query is The overall MRR score the average of these reciprocal ranks. MRR values range between 0 and 1, where higher value indicates better performance in ranking predictions.
Hir@ N: @it@: N measures the proportion of correct predictions within the top N ranked 1, we are interested whether the correct answer
results. For example, when predicted within the top-ranked result. When N 3, evaluate whether the correct answer within the top three predictions. The value N can be chosen based on the specific task and dataset characteristics, commonly set 1, 3, or 10. Hit N values also range between and 1, representing the proportion of correct predictions.
C.4 Link Predication on ICEWS18
The link prediction experimental results of ICEWS18 are displayed in Table The proposed LLM- DA outperforms existing state-of-the-aar benchmarks on ICEWS18 across all metrics. Moreover, GPT-NeoX [48] is an important baseline because it also introduces GPT as LLMs. However, proposed method still improves all metrics on ICEWS18. For example, the proposed method obtains 15.6% improvement under Hit @ 10 ICEWS18. 88 These phenomena demonstrate that the dynamic adaptation strategy can effectively update LLM-generated rules with current data, ensuring that the rules consistently incorporate the most recent knowledge and better generalize predictions for future events.
Table 4: Link prediction results on ICEWS18 dataset. The best results are in bold and means the result is unavailable. denotes the TKGR methods, & represents the LLMs-based TKGR. The
results of the models with are derived from [22]. and $ from (23
ICEWS18
Type
Models
Train
MRR
Hit@1 1
Hit@3
Hit@ 10
RE-NETI
4
0.287
0.188
0.327
0.482
RE-GCN
44
0.326
0.223
0.367
0.526
TLogic
0.205
0.340
0.485
TiRGN 46
0.336
0.232
0.379
0.542
PPT 21
0.266
0.169
0.306
0.454
Llama-2-705 22-bb-oHH"
0.223
0.363
0,522
Vicuna-7b-CoHH
0.209
0.347
0.536
GPT-NeoX
48
0.192
0.313
0.414
Mixtral-8x7B-Cone
X
0.330
0.218
0.378
0.549
LLM-DA (TiRGN)
X
0.357
0.255
0.403
0.570
17

---

C,5 Long Horizontal Link Prediction
Long horizontal link prediction generally predict the events happening at time point t + k * AT, where AT denotes the interval between time points and & controls how far we will predict. To perform the long horizontal link prediction, we adjust the integral length AT on ICEWS14 and ICHWS05-15 datasets. As described in Figure Figure 7 we report the results corresponding to different AT on ICEWS14 and ICEWS05-15 and compare the proposed L.LM-DA with the strongest baselines RE-GCN 44 and TiRGN [46]. Experimental results shows that LLM-DA can achieve the best performance on different AT both datasets. This phenomenon demonstrates that LLMs can effectively learn meaningful rules, and the dynamic adaptation strategy can fully update the LLM-generated rules, allowing for better adaptation to the constantly changing dynamics within TKGs.
KFNNS44
KFNSS44
0.40
KIWSSL
6:80
KIWW14
AT
6T
USF
0.24
0347
0..44
0.400
180
Horra 10ayss)
Hime 10ags)
Figure 6: Long horizontal link forecasting: filtered MRR ICEWS14 with respect to different time points.
IEEM5500-15
67
a7=30
0.42
6440
0259
0300
1ime (Dheya)
CI95505-15
IC295500-15
1888
41 65
291 120
341 L6C
331
200 321 2l3 20L 200 204 300 241 30
12C
36C
203
242
200
302
300
Figure 7: Long horizontal link forecasting: time-aware MRR performance on ICEWS05-15 with respect to different time points.
18

---

C.6 Visualization Experiment
To visually present the changes of rule and candidate rankings during the Rule Generation and Dynamic Adaptation, we conducted visualization experiment in Table For the query (China, Consult, 2014-12-05), the actual answer Japan ranked fifth during the Rule Generation stage, but after the Dynamic Adaptation, it advanced to the top position. Meanwhile, the sequencing of rules in two phases also exhibits significant changes. For instance, the ranking of the rule "Consult +- inv _ak_Make___a an aa__aan_aaaaaaa _apppal__ooooo _r___r__________e or_request & Express_ _eee_ or or_negotiate changes from fifth to first position. This highlights the efficacy of the dynamic adaptation strategy, which can update the LLM-generated rules with latest events to capture the evolving nature of TKGs.
Table 5: Visualization experiment for query (China, Consult, 2014-12-05). Rules for Rule Generation and Rules for Dynamic Adaptation represent the dynamic changes of the rules during the Rule Generation and Dynamic Adaptation. Candidates for Rule Generation and Candidates for Dynamic Adaptation indicate the changes in the corresponding candidate rankings. "Japan" the actual answer.
Query
(China, Consult, 7, 2014-12-05)
inv Host_oostat. vist Meet_alaa 'third" Jocation
visii & Host_a_viss aisii & inv. Discussssss____ byb by_telpphone
@ Express express_intent_... intent to engage in diplomatic cooperation & inv Host a visit & inv Consult pn__@ppoomaticeoxperaaioo
Rules for Rule Generation
& Consult Express, jintent_n__i___ee cuoperate & Consult
& inv Make an appeall or request & Express, intent to meet negotiate
a) Malaysia
@ Cambodia
S China
Candidates for Rule Generation
South,
06 Japan
Query
(China, Consuli, 2, 2014-12-05)
Make appeal sequest Express & Consult & Express_ jinennnnenn to co_cerreaaeee & Consult
sesppess_ineen___ee Express_ intent_too to_ wwgggggggg.... in at inv Host a visit & Host a. visit & inv. Discuss, by Jelephome
Rules for Dynamic Adapiation
location
@
coopprrptpoonnn & inv Has___os_ssiiiee aevisit & inv Consult
1 Japan
@ France
Candidates for Dynamic Adaptation
@ Malaysia
Exprees intent, to emgige in diplomaticc cooperation represents Expeer, intent, so eo_ongoge__e in. diplommaticcooppertionn (such (such or pulppccccpppprrn) suppor).
C.7 Parameter Analysis
0.495
0.530
ICEWS14
ICEW505-15
0.484
0.520
0.474
MRR
0.463
0.500
MRR
0.452
0.490
0.480
0.431
0.470
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Weight
Figure 8: Comparison with different weight on both datasets.
To experimentally study the effect of the weight on ICEWS14 and ICEWS05- 15, we tune the weight in range (0, 0.1,.... .... }} and observe link prediction experimental results. shown in Figure 8 the MRR performance exhibits trend of initially increasing and then decreasing, reaching the peak at a = 0.9 ICEWS14 14, and a = 0.8 for ICEWS05-15. These observations underscore that the weight plays crucial role in influencing the performance of LLM-DA. Specifically, the weight
19

---

shows better performance at higher values, highlighting the importance of rule-based reasoning module.
Moreover, when a = 1, indicating that only rules are used to generate candidates, the MRR performance decreases. This demonstrates that although LLMs can learn meaningful rules, the rules generated based on historical data cannot fully fit the future data due to the difference in distribution. This results in few queries failing find matching candidates when relying solely rules for candidate generation.
D Limitations
Our limitations can be summarized follows: (1) LLM-DA does not consider the semantics of nodes, which may reduce the quality of the sampled rules. (2) LLM-DA does not generate rules based on queries, which may result in the rules generally lacking specificity. (3) LLM-DA requires manually constructed prompts, which means that prompts need to be redesigned for different datasets. Morcover, manual construction of prompts also incurs significant costs. In the future, we will employ automated prompt learning to generate query query-dependent rules through LLMs. Furthermore, integrate the semantics of nodes during the temporal logical rules sampling stage.
20

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have made the main claims in the abstract and introduction, which can accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work Section 7.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have disclosed all the information needed to reproduce the main experi- mental results of the paper in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have uploaded the anonymous code in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett httpa///guddeee/eeeeabiiissssss (Cooeooomiissssseeeeeeee ssi.iooiccyyyyyc.occ for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have listed all experimental settings Subsection 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All the experimental results acquired by multiple trails of experiments, andwe report the average and standard deviation results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided sufficient information on the computer resources in Subsec- tion 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://nettt httppppp....ppppcp.ccccccccclala
Answer: [Yes]
Justification: The research conducted in the paper complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have ensured that all code, data, and models used in our research are cited appropriately, and we have adhered to the licensing agreements and terms of use specified by the original authors.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The details of the dataset/codeemmodl is created by the anonymized URL in ABSTRACT.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "dpzGfQd22c", "reviewer": "Reviewer_9wEQ", "summary": "This paper introduces Large Language Models-guided Dynamic Adaptation (LLM-DA), a novel approach for Temporal Knowledge Graph Reasoning (TKGR). LLM-DA leverages LLMs to extract temporal logical rules from historical data, providing interpretable reasoning. It also incorporates a dynamic adaptation strategy to update these rules with the latest events, ensuring the extracted rules reflect the most recent knowledge.  Experimental results over several common datasets show that , LLM-DA, without the need of fine-tuning, significantly outperforms graph-based TKG method and LLM-based method in accurate reasoning.", "questions": "Please refer to the questions raised in Weakness section.", "limitations": "Yes, the author has discussed the limitations in Appendix D, and proposed reasonable solutions fas the future works.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Though recently many efforts have been put into applying LLMs in TKG, this paper poses a novel way of applying LLM in TKGR as a temporal logic rule extractor to dynamically extract and update meaningful temporal patterns and complex temporal dependencies from the evolving event data, which provides extra interpretability to the forecasting process compare to a simple in-context LLM forecasting.\n- The experiments have shown that the performance of the proposed method has surpassed the traditional graph-based TKG method and the recent LLM-based TKG methods.\n- The analysis on the dynamic adaptation is very interesting and is in-depth and solid, shows the effectiveness of the dynamic adaptation in extracting the evolving temporal patterns.", "weaknesses": "- Long-horizon forecasting concerns: Concerns will be raised about the dynamic adaptation in cases where no near historical data is available, i.e. in the long-horizon forecasting task where the future data patterns and rules may largely differ from the seen data, how generalizable and robust this proposed method will be?\n- Limited evaluation of generated rules:  While the LLM mainly participates in the rule extraction part, the paper lacks a comprehensive quantitative and qualitative evaluation of the generated rules. For instance, there's no assessment of how well the LLM generates symbolically and logically correct rules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Comment for Reviewer 9wEQ:\nThank you very much for your professional review and valuable suggestions. We have carefully considered and responded to the questions you raised.\n\n$\\color{blue}{W.1:}$ Long-horizon forecasting concerns.\n\n$\\color{blue}{Re:}$ Thank you for your insightful comments regarding the long-horizon forecasting. We want to clarify that we have conducted the long-horizon forecasting experiment in the paper, which can be found in Appendix C.5 (Figures 6 and 7) and Sec 5.3 RQ2 (Figure 4).\n\n-**Long-horizon Forecasting (time point):** To verify that the proposed method can generalize to diverse further distributions, we conduct the long horizontal link prediction in Appendix C.5. In this experimental setup, the model is adapted using only data available up to time point $t$, and then employed for predictions at subsequent time points $t+k\\triangle T$ without any further adaptation. Here, $k$ controls how far we predict and $\\triangle T$ denotes the interval between time points. In experiments, we vary $k$ to evaluate the performance over different horizons. Results depicted in Figures 6 and 7 demonstrate that our proposed LLM-DA maintains strong performance across various time points, indicating its robustness in dynamically adapting to new data patterns over long horizons.\n\n-**Long-horizon Forecasting (time interval):** We further evaluate the long-horizon forecasting results from the perspective of time intervals. Specifically, we divided the timeline into several intervals in chronological order, and these findings are thoroughly detailed in Section 5.3, RQ2. As shown in Figure 4, our method consistently outperforms existing methods across different time intervals, which validates the effectiveness of using LLMs to capture the evolving patterns of temporal knowledge. \n\nIn the final version, we would add more details about the experiment settings and clarify the confusion.\n\n$\\color{blue}{W.2:}$ Limited evaluation of generated rules.\n\n$\\color{blue}{Re:}$ We appreciate your point about evaluating the generated rules. Given the large number of generated rules, a comprehensive manual evaluation is impractical. Instead, we use the quantitative metric *Support* to evaluate the quality of rule $\\rho$, which represents the number of facts in KGs that satisfy the rules. *Support* is important for assessing rules, as it shows how often the facts can be inferred by the rules within KGs. \n\nSpecifically, the rule $(\\rho)$ defines the relation between two entities $e_s$ and $e_o$ at timestamp $t_l$,   \n\n$\\rho: = r(e_s, e_o, t_l) \\leftarrow \\wedge_{i=1}^{l-1}r^*(e_s,\\ e_o,\\ t_i),$\n   \nwhere the left-hand side denotes the rule head with relation $r$ that can be induced by ($\\leftarrow$) the right-hand rule body. The rule body is represented by the conjunction ($\\wedge$) of a series of body relations $(r^* \\in$ {$r_1,...,r_{l-1}$}).\n\n*Support* $s_\\rho$ denotes the number of facts in Temporal Knowledge Graphs (TKGs) that satisfy the rule $\\( \\rho \\)$, which is formally defined as:\n\n$s_\\rho := (e_s, e_o, t_l) : \\exists\\wedge_{i=1}^{l-1}r^*(e_s,\\ e_o,\\ t_i) : \\text{rule}$_$\\text{body}(\\rho) \\wedge (e_s, r, e_o, t_l) \\in \\mathcal{G},$\n\nwhere $\\wedge_{i=1}^{l-1}r^*(e_s,\\ e_o,\\ t_i)$ denotes a series of facts in KGs that satisfy the rule body $\\(\\text{rule}$_$\\text{body}(\\rho)\\)$ and $\\(e_s, r, e_o, t_l\\)$ denotes the fact satisfying the rule head $\\(r\\)$. \n\nFollowing the *Support* metric, we can evaluate the quality of the rule $\\rho$, and the results are shown as follows:\n\n| Datasets | Total Rules | Avg. Rules/$r$ | $s_\\rho(Train)$ | $s_\\rho(Test)$ | $s_\\rho(ALL)$ | $s_\\rho(Train)$ (filtered) | $s_\\rho(Test)$ (filtered) |$s_\\rho(ALL)$ (filtered) |\n|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|\n| ICEWS14 | 31835 | 69.21 | 23.52 | 10.63 | 24.76 | 32.81 | 13.79 | 34.93 |\n| ICEWS05-15 | 41692 | 83.05 | 58.66 | 29.72 | 61.39 | 81.11 | 42.73 | 83.68 |\n\nIn this Table, $s_\\rho$ is the generated rules of LLMs, $s_\\rho$ (filtered) denotes the filtered rules in Section 4.4, Eq. (8), which is the final rules used for rule application part; and $s_\\rho(ALL)$ indicates the generated rules of LLMs are applied in the whole dataset.\n\nFrom the results, we can see that rules generated by LLM achieve a high support, indicating the good quality of them. In the final version, we will add experiments for rule evaluation in the appendix and conduct a comprehensive analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YoodiESPSW", "reviewer": "Reviewer_thzY", "summary": "This paper explores the use of Large Language Models (LLMs) for Temporal Knowledge Graph Reasoning (TKGR). Specifically, the paper leverages LLMs for rule-based TKGR to identify temporal patterns and enable interpretable reasoning. Additionally, it introduces a dynamic adaptation strategy that iteratively updates the LLM-generated rules with the latest events, enhancing the model's adaptation to the evolving dynamics of TKGs.", "questions": "Q1. Do the other modules in the paper, such as the contextual relation selector and the graph-based reasoning function, also require pre-training in addition to the LLMs?\nQ2. The current experimental analysis is too superficial. To better validate the proposed method, a more in-depth analysis of the experiments is needed.", "limitations": "The authors clearly point out that this work is not query-dependent, which can lead to a lack of specificity in the generated rules. This provides a solid direction for future work to enhance the specificity of generated rules. Additionally, it is recommended that the authors discuss in depth the problems posed by not considering node semantics. Failure to consider node semantics may result in the loss of important contextual information and affect the overall accuracy of the model. \nA more detailed discussion of this limitation would greatly benefit the paper.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1. Compared to other LLM-based TKGR approaches, this paper demonstrates a significant improvement without the need for fine-tuning LLMs.\nS2. It introduces a novel dynamic adaptation strategy, guiding LLMs to capture temporal evolution patterns in TKGs by continuously updating the knowledge rather than the LLMs themselves.\nS3. Intuitive figures and tables enhance the readability of the paper.", "weaknesses": "W1. There is limited analysis of the constrained Markovian random walks.\nW2. While Figure 1 intuitively reflects the paper's motivation, it could be improved. Moreover, the dynamic adaptation strategy, which should be the core focus, occupies a relatively small proportion in Figure 2.\nW3. Several minor grammatical and expression issues need attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Comment for Reviewer thzY:\n\n# Weaknesses:\n\n$\\color{blue}{W.1:}$ There is limited analysis of the constrained Markovian random walks.\n\n$\\color{blue}{Re:}$ Thanks for your suggestions. In the paper, we have provided theoretical analysis of the constrained Markovian random walks in Appendix A from the aspects of \"Impact of Time Weighting\" and \"Traversal Properties\". Specically, the time weighting enhances the model's sensitivity to recent data, improving its ability to capture short-term changes, while traversal properties ensure that the exploration of paths respects temporal continuity, leading to more accurate insights.\n\n$\\color{blue}{W.2:}$ While Figure 1 intuitively reflects the paper's motivation, it could be improved. Moreover, the dynamic adaptation strategy, which should be the core focus, occupies a relatively small proportion in Figure 2. \n\n$\\color{blue}{Re:}$ We appreciate your kind suggestion. We will refine the Figure 1 and Figure 2 in the final manuscript.\n\n$\\color{blue}{W.3:}$ Several minor grammatical and expression issues need attention.\n\n$\\color{blue}{Re:}$ Thank you for pointing out the minor grammatical and expression issues in our manuscript. We will thoroughly review and correct these issues in the final manuscript to ensure clarity and readability.\n\n# Questions:\n\n$\\color{blue}{Q.1:}$ Do the other modules in the paper, such as the contextual relation selector and the graph-based reasoning function, also require pre-training in addition to the LLMs?\n\n$\\color{blue}{Re:}$ Thank you for your insightful question. Both the contextual relation selector and the graph-based reasoning function are freezed without any tuning in our framework.\n\n$\\color{blue}{Q.2:}$ The current experimental analysis is too superficial. To better validate the proposed method, a more in-depth analysis of the experiments is needed.\n\n$\\color{blue}{Re:}$ Thank you for your suggestions. To further evaluate the proposed method, we explore the generalizability of our framework with different LLMs. Specifically, we replace the closed-source ChatGPT with two open-source LLMs of different sizes. (e.g., Qwen-1.5-chat-7B and Qwen-1.5-chat-72B). The detailed results can be found in our response to *reviewer CXWD. W2*. For your convenience, we compile the results showing the impact of different LLMs on experimental performance as follows:\n\n| ICEWS14 | MRR | Hit@1 | Hit@3 | Hit@10 |\n|-|-|-|-|-|\n| Qwen 1.5 chat (7B) (TiRGN) | 0.461 | 0.353 | 0.506 | 0.660 |\n| Qwen 1.5 chat (72B) (TiRGN) | 0.462 | 0.356 | 0.508 | 0.661 |\n\n| ICEWS05-15 | MRR | Hit@1 | Hit@3 | Hit@10 |\n|-|-|-|-|-|\n| Qwen 1.5 chat (7B) (TiRGN) | 0.510 | 0.402 | 0.569 | 0.716 |\n| Qwen 1.5 chat (72B) (TiRGN) | 0.509 | 0.403 | 0.571 | 0.716 |\n\nExperiment results show that the proposed method is flexible with the used LLMs showing the generalizability of our framework. Meanwhile, this provides deeper insights into the selection of LLMs and further validate the significant advantages of our proposed method in handling temporal data.\n\nIn the final version, we will add more detailed experimental analysis to thoroughly demonstrate the robustness and effectiveness of our proposed method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "POrY5scurs", "reviewer": "Reviewer_CXWD", "summary": "This paper proposes Large Language Models-guided Dynamic Adaptation (LLM-DA) to leverage LLMs to extract temporal logical rules for TKGR. Experimental results demonstrate LLM-DA significantly improves reasoning accuracy without the need for fine-tuning the LLM.", "questions": "Please see Weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper is well-structured and easy to follow, the idea of extract and ground logic rules are clear motivated and illustrated.\n\nLevearging LLM to exploit rules rather than predict the answers is interesting.\n\nLLM-DA significantly outperforms the LLM-based methods on several datasets.", "weaknesses": "The writing could be improved. Some figures should be polished.\n\nLLM-DA leverages GPT 3.5 as the LLM model, while the other baselines use only 7B LLM models. The authors should also conduct experiments to study the impact of different LLM models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Comment for Reviewer CXWD:\n\nThank you very much for your professional review and valuable suggestions. We have carefully considered and responded to the questions you raised.\n\n$\\color{blue}{W.1:}$ The writing could be improved. Some figures should be polished.\n\n$\\color{blue}{Re:}$ We really appreciate your comments. We will thoroughly revise the manuscript to improve the overall writing quality. Additionally, we'll review and refine all figures to ensure clarity and visual appeal.\n\n$\\color{blue}{W.2:}$ LLM-DA leverages GPT 3.5 as the LLM model, while the other baselines use only 7B LLM models. The authors should also conduct experiments to study the impact of different LLM models.\n\n$\\color{blue}{Re:}$ Thank you very much for your professional comments. We first want to point out that there are some larger LLMs (e.g., GPT-NeoX: 20B, and Mixtral-8x7B: 56B) selected as baselines for fair comparison. To further demonstrate the flexibility of our method, we conduct experiments about using different LLMs (Qwen 1.5 chat (7B) and Qwen 1.5 chat (72B)) in our proposed method. The experimental results on both datasets are shown as follows:\n\n| ICEWS14 | MRR | Hit@1 | Hit@3 | Hit@10 |\n|-|-|-|-|-|\n| Qwen 1.5 chat (7B) (TiRGN) | 0.461 | 0.353 | 0.506 | 0.660 |\n| Qwen 1.5 chat (72B) (TiRGN) | 0.462 | 0.356 | 0.508 | 0.661 |\n\n| ICEWS05-15 | MRR | Hit@1 | Hit@3 | Hit@10 |\n|-|-|-|-|-|\n| Qwen 1.5 chat (7B) (TiRGN) | 0.510 | 0.402 | 0.569 | 0.716 |\n| Qwen 1.5 chat (72B) (TiRGN) | 0.509 | 0.403 | 0.571 | 0.716 |\n\nFor your convenience, we list the experimental results of baselines and the original results (using ChatGPT 3.5 as LLM) as follows:\n\n| | | | ICEWS14 | | -| | | ICEWS05-15 | |\n|-|-|-|-|-|- |-|-|-|-|\n| Models | MRR | Hit@1 | Hit@3 | Hit@10 |- | MRR | Hit@1 | Hit@3 | Hit@10 |\n|Llama-2-7b-CoH (7B) |-| 0.349 | 0.470 | 0.591 | -|-| 0.386 | 0.541 | 0.699 |\n|Vicuna-7b-CoH (7B) |-| 0.328 | 0.457 | 0.656 |- |-| 0.392 | 0.546 | 0.699 |\n|GPT-NeoX  (20B)  | - |0.334 | 0.460 | 0.565 |- |-|-|-|-|\n|Mixtral-8x7B-CoH  (56B)  | 0.439 | 0.331 | 0.496 | 0.649 |- | 0.497 | 0.380 | 0.564 | 0.713 |\n|-|-|-|-| |-|-|-|-|-|\n| Ours (ChatGPT 3.5 (TiRGN)) | 0.471 | 0.369 | 0.526 | 0.671 | -| 0.521 | 0.416 | 0.586 | 0.728 |\n\nFrom the above tables, it is evident that both the new LLMs, Qwen 1.5 chat (7B) and Qwen 1.5 chat (72B), perform better than the baselines, and achieve comparable performance to our proposed method (ChatGPT-3.5). This demonstrates that the proposed dynamic adaptation mechanism effectively captures temporal evolution patterns, allowing the generated rules to generalize better to future data.\nThis validation across multiple model scales and architectures underscores the robustness of our approach, confirming its capability to leverage different LLMs effectively. \n\nIn the final version, we will include additional experiments and provide a more comprehensive analysis to examine the impact of different LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper leverages LLMs to infer rules for Temporal Knowledge Graph Reasoning (TKGR) -- link prediction with TKGs.  More specifically; 
1) it first uses constrained random works to discover initial rules; 
2) then the rules and related relations with scores (accuracy) are fed to an LLM to generate rules with better quality/coverage;
3) furthermore a dynamic adaptation strategy -- repeat the rule discovery (random walk), scoring (accuracy) and rewrite (LLM) steps for new data

Experimental results over a few TKG datasets show that , LLM-DA, outperforms graph-based TKG method and LLM-based method in accurate reasoning.

This work demonstrates the benefit of generating explicit reasoning rules from LLMs compared to pure graphic based approach and pure LLM based approach.
Decision: Accept (poster)