G2S: A General-to-Specific Learning Framework for Temporal Knowledge Graph Forecasting with Large Language Models
Authors: Long Bai, Zixuan Li, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng, Tat-Seng Chua
Accepted at: acl25

G2S: A General-to-Specific Learning Framework for Temporal Knowledge Graph Forecasting with Large Language Models

Long Bai1,2,4*, Zixuan Li1,2, Xiaolong Jin1,2,3,*, Jiafeng Guo1,2,3, Xueqi Cheng1,2,3, Tat-Seng Chua4
1Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences
2State Key Laboratory of AI Safety
3School of Computer Science, University of Chinese Academy of Sciences
4National University of Singapore
{bailong,jinxiaolong}@ict.ac.cn

Abstract

Forecasting over Temporal Knowledge Graphs (TKGs) which predicts future facts based on historical ones has received much attention. Recent studies have introduced Large Language Models (LLMs) for this task to enhance the models’ generalization abilities. However, these models perform forecasting via simultaneously learning two kinds of entangled knowledge in the TKG: (1) general patterns, i.e., invariant temporal structures shared across different scenarios; and (2) scenario information, i.e., factual knowledge engaged in specific scenario, such as entities and relations. As a result, the learning processes of these two kinds of knowledge may interfere with each other, which potentially impact the generalization abilities of the models. To enhance the generalization ability of LLMs on this task, in this paper, we propose a General-to-Specific learning framework (G2S) that disentangles the learning processes of the above two kinds of knowledge. In the general learning stage, we mask the scenario information in different TKGs and convert it into anonymous temporal structures. After training on these structures, the model is able to capture the general patterns across different TKGs. In the specific learning stage, we inject the scenario information into the structures via either in-context learning or fine-tuning modes. Experimental results show that G2S effectively improves the generalization abilities of LLMs.

1 Introduction

Temporal Knowledge Graph (TKG), which consists of facts in the form of \((subject, relation, object, timestamp)\), is often used to represent the changes of facts over time. Forecasting future facts based on historical ones in TKG is critical for many time-sensitive scenarios, such as finance (Feng et al., 2019), healthcare (Ernst et al., 2014), and politics (Morstatter et al., 2019). These scenarios involve different entities, relations, and time granularities, which drive us to seek a model that is able to adapt to new scenarios efficiently and effectively.

Previous methods view TKG as a temporally ordered sequence of knowledge graphs and adopt graph neural networks (Li et al., 2021b) or recurrent neural networks (Jin et al., 2020) to model it. These methods mainly focus on fitting a model to each specific TKG and overlook the generalization abilities. Recent methods attempt to model different TKGs via a unified Large Language Model (LLM), which shows considerable improvement in generalization ability. Some methods append historical facts into the prompt and utilize the in-context learning technique to predict the answer in unseen scenarios (Lee et al., 2023; Xia et al., 2024). Other methods learn temporal knowledge via supervised fine-tuning LLMs on a certain dataset and try to transfer this knowledge to unseen scenarios (Liao et al., 2024; Ding et al., 2024).

The generalization abilities of the TKG forecasting models mainly come from their abilities to learn general knowledge from seen scenarios and apply it to unseen scenarios. However, different TKGs describe different scenarios that involve different en-

![An example of learning general patterns and applying to unseen datasets.](page_768_355_627_312.png)

Figure 1: An example of learning general patterns and applying to unseen datasets.
tities, relations, and time granularities. Taking Figure 1 as an example, GDELT is about conflicts and crisis scenarios in the geopolitics domain, while WIKI contains sports and academic scenarios in the general domain. This leads us to imagine: what is the general knowledge that can be shared across the scenarios in these TKGs? We observe that, if we anonymize the specific entity and relation names into abstract IDs, some temporal structures in different scenarios reveal a common substructure. Following this observation, we think that the TKGs contain two kinds of entangled knowledge, namely, general patterns and scenario information. The general patterns are common structures that can be generalized to different scenarios, while scenario information is factual knowledge, such as specific entities and relations, which only help TKG forecasting in a certain scenario. If the models directly learn the TKG (i.e., learn these two kinds of knowledge simultaneously), their learning processes may interfere with each other. From this point of view, previous methods, even LLM-based ones, may not learn the general patterns comprehensively, which limits their generalization abilities.

Motivated by this, in this paper, we aim to disentangle the above two kinds of knowledge in TKG, and propose a general-to-specific learning framework for TKG forecasting, named G2S. G2S consists of two stages, namely, the general learning stage and the specific learning stage. In the general learning stage, we convert multiple TKGs into anonymous temporal structures via anonymizing the specific entities, relations, and timestamps into abstract IDs. After being trained on these structures, the model is able to learn the general patterns across multiple TKGs comprehensively. In the specific learning stage, we split the unseen TKG into anonymous temporal structures and their mappings between the specific entities/relations and the abstract IDs, as shown in the bottom right of Figure 1. In addition to general patterns, the model further learns the above information in either the in-context learning mode or the fine-tuning mode. To verify the effectiveness of G2S, we evaluate the proposed framework under three settings, namely, zero-shot, low-resource, and standard learning. Experimental results show that G2S effectively improves the generalization abilities of LLMs.

The contributions of this paper are three-fold:

• We identify the two kinds of entangled knowledge in TKG, namely, the general patterns and scenario information, and disentangle them via several anonymization strategies.

• We propose a general-to-specific learning framework for TKG forecasting, which prevents the learning processes of general patterns and scenario information from interfering with each other. In this way, the generalization ability of the model is enhanced.

• We conduct extensive experiments under three different settings. The experimental results demonstrate the effectiveness of the proposed framework.

2 Related Work

There are two different task settings for TKG reasoning, namely, interpolation and extrapolation settings. TKG reasoning under interpolation setting, also known as TKG completion, aims to predict the missing entities at past timestamps (Jiang et al., 2016), while TKG reasoning under extrapolation setting, also known as TKG forecasting, aims to predict the missing entities at future timestamps (Jin et al., 2020). This paper focuses on the latter one.

Traditional TKG forecasting methods can be categorized into three different types: (1) Neural-based methods utilize deep neural networks to model the historical facts. RE-Net (Jin et al., 2020) develops a recurrent neural network to model query related historical facts. RE-GCN (Li et al., 2021b) utilizes graph convolutional networks to model recent KG snapshots. TANGO (Han et al., 2021b) adopts neural ordinary differential equations to model the structure information among entities. xERTE (Han et al., 2021a) designs a sub-graph expansion and pruning mechanism to find explainable supports for forecasting. (2) Reinforcement learning-based methods utilize reinforcement learning to find a path between query entity and answer entity, such as CluSTeR (Li et al., 2021a) and TiTer (Sun et al., 2021). (3) Rule-based methods mine logical rules to derive the answer from historical facts, such as TLogic (Liu et al., 2022). These methods mainly focus on enhancing the performance of the models on each specific TKG and overlook the generalization ability of the models.

Recently, some studies have introduced LLM for TKG forecasting, which can be categorized into in-context learning and supervised fine-tuning
methods. Lee et al. (2023) propose an in-context learning method, which provides a basic input format to fit TKG data to LLMs. Xia et al. (2024) investigate the LLM’s ability to find useful historical facts and reasoning answers based on them. GenTKG (Liao et al., 2024) proposes to select historical facts via rules and trains LLMs to answer the query via supervised fine-tuning. These methods show generalization abilities on TKG forecasting to some extent. However, as analyzed in Section 1, their learning processes of general patterns and scenario information may interfere with each other, which limits their generalization abilities.

3 Problem Formulation

A TKG \( \mathcal{G} \) is formalized as a sequence of KG snapshots from timestamps 1 to \( T \), i.e., \( \mathcal{G} = \{\mathcal{G}_1, \mathcal{G}_2, ..., \mathcal{G}_T\} \). Each KG snapshot \( \mathcal{G}_t = \{\mathcal{E}, \mathcal{R}, \mathcal{F}_t\} \) consists of the entity set \( \mathcal{E} \), relation set \( \mathcal{R} \), and the fact set \( \mathcal{F}_t \) at timestamp \( t \in \{1, ..., T\} \). Each fact \( f_i \in \mathcal{F}_t \) is denoted as a quadruple \( f_i = (s_i, r_i, o_i, t_i) \), where \( s_i, o_i \in \mathcal{E} \) are subject and object entities; \( r_i \in \mathcal{R} \) is the relation. Currently, the TKG forecasting task focuses on predicting the missing object entity in a query \( q = (s_q, r_q, ?, t_q) \). The model \( \mathcal{M} \) is required to score possible object entities based on the query and historical TKG snapshots: \( \text{score}(o) = \mathcal{M}(q, \mathcal{G}_{<t_q}) \). Note that, when predicting the missing subject entity, the query is formulated as \( q = (?, r_q, o_q, t_q) \).

4 Methodology

To enhance the generalization ability of LLMs on TKG forecasting, we propose G2S that disentangles the learning process of the general patterns and specific information into two stages, namely, the general learning and specific learning stages. Figure 2 shows the overall diagram of G2S. We will describe the design of these two learning stages and provide other training and inference details.

4.1 General Learning Stage

In the general learning stage, there are three main processing steps, namely, query construction, anonymous temporal structure conversion, and sample construction. In what follows, we will describe these steps in detail.

4.1.1 Query Construction

In this step, we construct training queries from the TKGs and find historical facts related to these queries. For each fact \( f_q = (s_q, r_q, o_q, t_q) \) in a TKG \( \mathcal{G} \), we construct two queries \( q_1 = (s_q, r_q, ?, t_q) \) and \( q_2 = (?, r_q, o_q, t_q) \) to predict the object and subject entities, respectively. Then, we select one-hop historical facts \( \mathcal{H}_q \) for each query \( q \), which is a commonly used history selection strategy in existing approaches (Jin et al., 2020; Lee et al., 2023):

\[
\mathcal{H}_{q_1} = \bigcup_{t < t_q} \{(s_q, r', o', t) \in \mathcal{F}_t\}
\]
\[
\mathcal{H}_{q_2} = \bigcup_{t < t_q} \{(s', r', o_q, t) \in \mathcal{F}_t\},
\]

where \( s', o' \in \mathcal{E} \) are arbitrary entities, \( r' \in \mathcal{R} \) is arbitrary relation. The historical facts are sorted according to their timestamps in ascending order. Following the convention, we retain the most recent \( L \) historical facts. To better learn general patterns, we collect queries across multiple TKGs with different entities, relations, and time granularities. For all the TKGs, we adopt the same processing method.

4.1.2 Anonymous Temporal Structure Conversion

To eliminate the impact of scenario information in the general learning stage, we apply an anonymization strategy that converts entities, relations, and timestamps in queries and facts into abstract IDs.

With respect to the timestamps, previous studies (Lee et al., 2023; Liao et al., 2024) utilize the number of timestamps from the start time of each TKG as the ID. For example, in dataset ICEWS14, the start time is 2014-01-01, and each timestamp is 1 day, so the ID of the timestamp 2014-01-10 is 9. However, such IDs may create bias between training data and testing data, since testing timestamps are not in the same period as training timestamps. Therefore, we set the timestamp ID of a query as 0, and the timestamp ID of each historical fact to the number of timestamps from the query time:

\[
\mathcal{A}(t) = t_q - t.
\]

With respect to entities and relations, we investigate three different strategies:

**Frequency ID (FID)** uses the rank of entity frequency (descending order) for each query and corresponding historical facts as ID. This strategy was proposed in previous studies (Lee et al., 2023; Liao et al., 2024).

**Global ID (GID)** uses the original ID provided in each dataset. In this strategy, the ID of each entity and each relation is unique for each TKG.
Figure 2: The proposed G2S learning framework. Here, “ATS”, “SFT” and “ICL” denote the anonymous temporal structures, supervised fine-tuning, and in-context learning, respectively, and each color in KG denotes a specific entity or relation.

Random ID (RID) uses the randomly assigned IDs for entities and relations within a query and corresponding historical facts.

Detailed analyses of the advantages and disadvantages of each strategy are described in Section 6.5

4.1.3 Sample Construction
Based on the anonymous temporal structure, we construct training samples to supervised fine-tune the LLM. Each training sample consists of an input-output pair, where an input consists of two parts, namely, history and query. In the history part, each line contains a historical fact in the form of “\( \mathcal{A}(t) : [\mathcal{A}(s), \mathcal{A}(r), \mathcal{A}(o)] \)” , where \( \mathcal{I} \) denotes the anonymization strategy described above. The historical facts are listed by their temporal order. In the query part, the query \((s_q, r_q, ?, t_q)\) is presented in the form of “\(0 : [\mathcal{A}(s_q), \mathcal{A}(r_q), ?]\)”. In addition, the input contains an empty answer part at the end to prompt the model to generate the answer. The output of the training sample is the correct answer entity ID “\( \mathcal{A}(o_q) \)”. For FID and RID strategies, if the answer entity does not occur in historical facts, it will not be assigned with an ID, so we set the output to be “None”. After being fine-tuned on such training samples, the model captures the general patterns across multiple TKGs.

4.2 Specific Learning Stage
In the specific learning stage, the processing steps are similar to those presented in Section 4.1. However, there are two key differences:

Scenario information mapping. In the specific learning stage, the model is required to combine the scenario information with the general patterns to derive more accurate answers. Therefore, when applying an anonymization strategy to convert entities/relations into IDs, we retain the mappings between entities/relations and IDs. Then, we add two parts at the beginning of the input, namely, entity and relation parts, according to the mappings. In the entity part, each line contains the mapping between an entity and its abstract ID in the form “\( \mathcal{A}(e) : e \)” , as shown in Figure 2. The relation part is done similarly, which contains the mappings between relations and their IDs.

Two learning modes. The specific learning stage can run in two different modes:
In-context learning (ICL) mode. In this mode, the model is prohibited from utilizing the training samples. Thus, the model can only learn scenario information via in-context learning.
Supervised Fine-tuning (SFT) mode. In this mode, the model is allowed to utilize the training samples. Thus, the model can learn the entities and relations via updating the parameters.

4.3 Training and Inference Details
The training objective is to minimize the cross-entropy loss between generated token sequence \( O \) and ground-truth token sequence \( \hat{O} \), i.e., \( \mathcal{L} = CE(O, \hat{O}) \).

We use LLaMA3-8B as the backbone model and adopt Low-Rank Adaptation (LoRA) (Hu et al., 2022) method to effectively fine-tune it.
In the inference phase, considering the efficiency, we only require the model to generate one step. Then, we adopt the generation score of the tokens provided by the LLM as the ranking score:

\[
\text{score}(o) = \Pr(o|Input),
\]

where \( o \) is a generated token. Finally, we retain the top 10 valid predictions after filtering the duplicate predictions.

5 Experimental Setup

5.1 Datasets
We use 5 widely-used datasets in this paper: ICEWS14 (Li et al., 2021b), ICEWS18 (Jin et al., 2020), YAGO (Mahdisoltani et al., 2013), GDELT (Jin et al., 2020), and WIKI (Leblay and Chekol, 2018). The basic statistics of the datasets are shown in Appendix B. Especially, ICEWS14, ICEWS18, and GDELT follow CAMEO \( ^1 \) schema, which means relation types in the three datasets are almost the same. YAGO and WIKI are constructed from open knowledge graphs of the same name, and thus, they have their own schema definitions.

Among these datasets, the training parts of GDELT and WIKI are used in the general learning stage, while ICEWS14, ICEWS18, and YAGO are used in the specific learning stage. The proposed model is evaluated on the later three datasets. We sample 100,000 training samples from GDELT and 30,000 training samples from WIKI, approximately according to the ratio of their total training facts.

5.2 Metrics
We employ widely used H@1/3/10 to evaluate the proposed model. These metrics represent the proportion of queries that the model ranks the correct answer entity at top 1/3/10. Moreover, we adopt the time-aware filtering setting, where the valid entities except the correct answer entity are removed from the predictions. Mean Reciprocal Rank (MRR) is another metric that has often been used in previous studies. However, this metric is not applicable for LLMs, since LLMs can not produce the full rank of all entities for a query.

5.3 Baselines
We compare the proposed method with several baseline methods. The baseline methods can be divided into two different categories, i.e., traditional methods and LLM-based methods. With respect to traditional methods, we choose RE-GCN (Li et al., 2021b), xERTE (Han et al., 2021a), TANGO (Han et al., 2021b), TiTter (Sun et al., 2021), and TLogic (Liu et al., 2022). With respect to LLM-based methods, we choose in-context learning (Lee et al., 2023) method and GenTKG (Liao et al., 2024). For in-context learning method, we use GPT-NeoX-20B (Black et al., 2022), Llama2-7B (Touvron et al., 2023), and Llama3-8B (AI@Meta, 2024) as backbone models, noted as GPT-NeoX-ICL, Llama2-ICL, and Llama3-ICL, respectively.

5.4 Experiment Settings
We evaluate our model under three different settings: (1) Standard setting. Under this setting, all training samples can be used to fine-tune the model in the specific learning stage. (2) Zero-shot setting. Under this setting, no training samples can be used to fine-tune the model in the specific learning stage. (3) Low-resource setting. Under this setting, the earliest 5%/20%/50% training samples can be used to fine-tune the model in the specific learning stage.

5.5 Implementation Details
For experiments on all datasets, the number of historical facts \( L \) is set to 50; context length of prompts is set to 1024; Learning rate is set to 1e-4; batch size is set to 8; number of epochs is set to 1. We tune the model on the validation set and report the best results. Unless otherwise specified, G2S utilizes both GDELT and WIKI datasets and adopts RID strategy in general learning stage. In specific learning stage, G2S adopts GID strategy on ICEWS14 and ICEWS18, and adopts RID strategy on YAGO \( ^2 \). We develop our experiment code based on LlamaFactory (Zheng et al., 2024) and run the experiments on 4 NVIDIA A800 GPUs. The general learning stage takes about 2 hours. Under the standard setting (fine-tuning on all training samples), specific learning on ICEWS14 takes about 2 hours; ICEWS18 takes about 20 hours; YAGO takes about 7 hours.

6 Results and Analyses

6.1 Standard Setting
To evaluate the performance of G2S when sufficient training data are available, we conduct experiments under standard settings, where all training

\( ^1 \)https://eventdata.parusanalytics.com/data.dir/cameo.html
\( ^2 \)GID strategy is not suitable for YAGO, see Appendix A for detailed analyses.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">ICEWS14</th>
    <th colspan="3">ICEWS18</th>
    <th colspan="3">YAGO</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
  </tr>
  <tr>
    <td>RE-GCN</td>
    <td>31.3</td>
    <td>47.3</td>
    <td>62.6</td>
    <td>22.3</td>
    <td>36.7</td>
    <td><b>52.5</b></td>
    <td>78.7</td>
    <td>84.2</td>
    <td>88.4</td>
  </tr>
  <tr>
    <td>xERTE</td>
    <td>33.0</td>
    <td>45.4</td>
    <td>57.0</td>
    <td>20.9</td>
    <td>33.5</td>
    <td>46.2</td>
    <td>84.2</td>
    <td>90.2</td>
    <td>91.2</td>
  </tr>
  <tr>
    <td>TANGO</td>
    <td>27.2</td>
    <td>40.8</td>
    <td>55.0</td>
    <td>19.1</td>
    <td>31.8</td>
    <td>46.2</td>
    <td>59.0</td>
    <td>64.6</td>
    <td>67.7</td>
  </tr>
  <tr>
    <td>TITer</td>
    <td>31.9</td>
    <td>45.4</td>
    <td>57.5</td>
    <td>21.2</td>
    <td>32.5</td>
    <td>43.9</td>
    <td>84.5</td>
    <td>90.8</td>
    <td>91.2</td>
  </tr>
  <tr>
    <td>TLogic</td>
    <td>33.2</td>
    <td>47.6</td>
    <td>60.2</td>
    <td>20.4</td>
    <td>33.6</td>
    <td>48.0</td>
    <td>74.0</td>
    <td>78.9</td>
    <td>79.1</td>
  </tr>
  <tr>
    <td>llama2-ICL</td>
    <td>25.8</td>
    <td>43.0</td>
    <td>51.0</td>
    <td>13.5</td>
    <td>27.6</td>
    <td>32.6</td>
    <td>67.7</td>
    <td>79.0</td>
    <td>81.8</td>
  </tr>
  <tr>
    <td>llama3-ICL</td>
    <td>31.9</td>
    <td>42.4</td>
    <td>44.1</td>
    <td>18.1</td>
    <td>27.2</td>
    <td>28.8</td>
    <td>56.9</td>
    <td>57.9</td>
    <td>57.9</td>
  </tr>
  <tr>
    <td>GPT-NeoX-ICL</td>
    <td>32.4</td>
    <td>46.0</td>
    <td>56.5</td>
    <td>19.2</td>
    <td>31.3</td>
    <td>41.4</td>
    <td>78.7</td>
    <td>89.2</td>
    <td><b>92.6</b></td>
  </tr>
  <tr>
    <td>GenTKG</td>
    <td>36.85</td>
    <td>47.95</td>
    <td>53.50</td>
    <td><b>24.25</b></td>
    <td><b>37.25</b></td>
    <td>42.10</td>
    <td>79.15*</td>
    <td>83.00*</td>
    <td>84.25*</td>
  </tr>
  <tr>
    <td>G2S</td>
    <td><b>38.33</b></td>
    <td><b>54.12</b></td>
    <td><b>68.58</b></td>
    <td>23.04</td>
    <td>35.32</td>
    <td>46.62</td>
    <td><b>87.88</b></td>
    <td><b>90.89</b></td>
    <td>90.93</td>
  </tr>
</table>

Table 1: Results of TKG reasoning on datasets ICEWS14, ICEWS18, and YAGO. The best results among all models are in boldface. The results of GenTKG on YAGO are marked with "*#" since they use self-developed subset of the standard YAGO dataset. Thus, they are not directly comparable with other methods.

data can be used to fine-tune the model in the specific learning stage. The experimental results are shown in Table 1. The methods are split into three groups, the first group contains the traditional methods, the second contains the LLM-based methods with in-context learning, and the third contains the LLM-based methods with supervised fine-tuning. From the table, we can observe that:

(1) G2S outperforms the baselines on ICEWS14 and YAGO under the H@1 metric, and achieves the second best performance on ICEWS18, which shows the forecasting ability of G2S.

(2) G2S and GenTKG outperform ICL methods in general, implying that supervised fine-tuning methods may better learn datasets than ICL methods if sufficient training data are available.

(3) G2S slightly underperforms GenTKG on ICEWS18, possibly due to the multi-token ID generation issue in G2S. A detailed analysis of this result can be found in Appendix A.

6.2 Zero-shot Setting

To evaluate the generalization ability of G2S, we conduct TKG forecasting under the zero-shot setting. Since the traditional methods can hardly run under the zero-shot settings, we only compare G2S with LLM-based methods. In addition, we adopt a simple frequency baseline proposed by Lee et al. (2023), which predicts answers according to entity frequency in historical facts.

We compare several variants of G2S to show the impact of different anonymization strategies in general learning stage: (1) G2S_{GL(F)} utilizes GDELT and adopts FID strategy; (2) G2S_{GL(F+Map)} add mappings between entity/relation and IDs in addition to G2S_{GL(F)}; (3) G2S_{GL(R)} utilizes GDELT and adopts RID strategy; (4) G2S_{GL(R) w. WIKI} jointly utilizes GDELT and YAGO, and adopts RID strategy. In specific learning stage, the anonymization strategies are selected via the performance on validation set.

We exclude the GenTKG from the baselines since, strictly speaking, its generalization setting is not a zero-shot setting. The differences between the two settings lie in the following aspects: (1) The training data (ICEWS14) and testing data (ICEWS18) of GenTKG (generalization) originate from the same source but correspond to different years; and (2) GenTKG adopts logic-rules mined from ICEWS18.

The experimental results are shown in Table 2. From the table, we can observe that:

(1) Llama3-ICL (8B) underperforms GPT-NeoX-ICL (20B) even though llama3 is a more recent model. These results imply that the scale of the model has a relatively large impact on the generalization ability in this task. Therefore, it is somehow unfair to directly compare G2S (8B) to GPT-NeoX-ICL. However, G2S_{GL(R) w. WIKI} still obtains comparable performance with GPT-NeoX-ICL on ICEWS datasets (especially a higher Hit@10 on ICEWS18) and relatively higher performance on YAGO dataset. Compared to the baselines except GPT-NeoX-ICL, G2S_{GL(R) w. WIKI} obtains significantly higher performance on all three datasets, which demonstrates the strong gen-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Dataset ICEWS14</th>
    <th colspan="3">ICEWS18</th>
    <th colspan="3">YAGO</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
  </tr>
  <tr>
    <td>Frequency</td>
    <td>24.3</td>
    <td>38.7</td>
    <td>53.2</td>
    <td>14.1</td>
    <td>26.5</td>
    <td>40.9</td>
    <td>76.6</td>
    <td>85.9</td>
    <td>92.1</td>
  </tr>
  <tr>
    <td>llama2-ICL</td>
    <td>25.8</td>
    <td>43.0</td>
    <td>51.0</td>
    <td>13.5</td>
    <td>27.6</td>
    <td>32.6</td>
    <td>67.7</td>
    <td>79.0</td>
    <td>81.8</td>
  </tr>
  <tr>
    <td>llama3-ICL</td>
    <td>31.9</td>
    <td>42.4</td>
    <td>44.1</td>
    <td>18.1</td>
    <td>27.2</td>
    <td>28.8</td>
    <td>56.9</td>
    <td>57.9</td>
    <td>57.9</td>
  </tr>
  <tr>
    <td>GPT-NeoX-ICL</td>
    <td><b>32.4</b></td>
    <td><b>46.0</b></td>
    <td><u>56.5</u></td>
    <td>19.2</td>
    <td>31.3</td>
    <td>41.4</td>
    <td>78.4</td>
    <td>89.1</td>
    <td><b>92.7</b></td>
  </tr>
  <tr>
    <td>G2S<sub>GL(F)</sub></td>
    <td>31.52</td>
    <td>44.80</td>
    <td>55.62</td>
    <td>19.47</td>
    <td>31.17</td>
    <td>43.05</td>
    <td>81.50</td>
    <td>87.54</td>
    <td>90.79</td>
  </tr>
  <tr>
    <td>G2S<sub>GL(F+Map)</sub></td>
    <td><b>32.13</b></td>
    <td><b>45.56</b></td>
    <td><u>56.59</u></td>
    <td>19.43</td>
    <td><b>31.87</b></td>
    <td><b>44.52</b></td>
    <td>81.00</td>
    <td>87.05</td>
    <td>90.76</td>
  </tr>
  <tr>
    <td>G2S<sub>GL(R)</sub></td>
    <td>31.95</td>
    <td>44.87</td>
    <td>55.61</td>
    <td>19.54</td>
    <td>31.34</td>
    <td>43.29</td>
    <td>80.40</td>
    <td>87.32</td>
    <td>90.86</td>
  </tr>
  <tr>
    <td>G2S<sub>GL(R) w. WIKI</sub></td>
    <td>32.02</td>
    <td>44.99</td>
    <td>55.84</td>
    <td><b>19.71</b></td>
    <td><u>31.58</u></td>
    <td><u>43.45</u></td>
    <td><b>86.07</b></td>
    <td><b>90.82</b></td>
    <td>91.02</td>
  </tr>
</table>

Table 2: Results of zero-shot TKG reasoning on datasets ICEWS14, ICEWS18, and YAGO. The best results among all models are in boldface and the second best results are underlined.

eralization ability of this model.

(2) G2S<sub>GL(F)</sub> underperforms G2S<sub>GL(F+Map)</sub> on ICEWS14 and ICEWS18, while outperforms it on YAGO. We guess it is because GDELT shares the same schema with ICEWS14 and ICEWS18 (i.e., relation types are almost similar in these datasets), G2S<sub>GL(F+Map)</sub> may learn useful scenario information from GDELT and apply it to ICEWS datasets. However, this scenario information may not be so helpful to YAGO. These results verify our motivation that simultaneously learning general patterns and scenario information may affect the generalization ability of the model.

(3) The performance of G2S<sub>GL(F)</sub> and G2S<sub>GL(R)</sub> are similar, which inspires us to use the RID strategy in the general learning stage of the overall framework, since the RID strategy is simpler in practice.

(4) G2S<sub>GL(R) w. WIKI</sub> outperforms G2S<sub>GL(R)</sub> on all three datasets, especially on YAGO. We guess it is because WIKI may contain more useful general patterns for YAGO than GDELT. However, the improvements on ICEWS14 and ICEWS18 verify our motivation that, although TKGs may differ in entities, relations, and time granularities, they still share some general patterns.

6.3 Low-resource Setting

To evaluate the generalization ability of G2S under the low-resource setting, where the earliest 5%/20%/50% samples in ICEWS14 are used to train the model, and all test samples in ICEWS14 are used to evaluate the model. In this experiment, we use G2S<sub>GL(R) w. WIKI</sub> in Section 6.2 as the initial model for the specific learning stage. We compare three variants of G2S with different anonymization strategies in specific learning stage: G2S<sub>SL(F)</sub>, G2S<sub>SL(R)</sub>, and G2S<sub>SL(G)</sub> represent the variants of G2S that adopt FID, RID, and GID strategy in specific learning stage, respectively. Other baselines are the same as those in Liao et al. (2024).

The experimental results are shown in Table 3. From the table, we can observe that:

(1) All three variants of G2S outperform the baseline methods via training on 5% training samples. Compared to GenTKG, which simultaneously learns general patterns and scenario information, the proposed two-stage learning framework obtains better generalization ability.

(2) When using 5% training samples, G2S<sub>SL(G)</sub> underperforms G2S<sub>SL(F)</sub> and G2S<sub>SL(R)</sub>. As the number of training samples increases, G2S<sub>SL(G)</sub> gradually outperforms the other two. This observation implies that, when there are very few training samples, the model is unable to learn the static mapping between entities/relations with global IDs. However, when more training samples are provided, GID performs more effectively than the other two strategies.

(3) LLM-based methods, such as GenTKG and variants of G2S, outperform traditional methods with a large margin, implying that the LLMs generally obtains better generalization abilities than traditional models. Thus, they require fewer training samples to reach the same performance as traditional models.

6.4 Ablation Study

To verify the effectiveness of each design in G2S, we conduct an ablation study on ICEWS14 under all three settings. For standard setting, we evaluate three variants of G2S: w/o. Ent means the
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">5%</th>
    <th colspan="3">20%</th>
    <th colspan="3">50%</th>
  </tr>
  <tr>
    <th>H@1</th><th>H@3</th><th>H@10</th>
    <th>H@1</th><th>H@3</th><th>H@10</th>
    <th>H@1</th><th>H@3</th><th>H@10</th>
  </tr>
  <tr>
    <td>RE-GCN</td><td>13.79</td><td>22.09</td><td>30.27</td><td>19.63</td><td>29.67</td><td>39.83</td><td>24.05</td><td>36.72</td><td>48.84</td>
  </tr>
  <tr>
    <td>xERTE</td><td>6.95</td><td>14.17</td><td>25.46</td><td>17.80</td><td>29.26</td><td>42.08</td><td>22.51</td><td>34.15</td><td>46.59</td>
  </tr>
  <tr>
    <td>TANGO</td><td>11.29</td><td>17.18</td><td>22.97</td><td>11.25</td><td>17.38</td><td>23.38</td><td>14.37</td><td>17.51</td><td>22.77</td>
  </tr>
  <tr>
    <td>TTTer</td><td>21.06</td><td>34.78</td><td>49.10</td><td>26.69</td><td>39.42</td><td>51.78</td><td>30.05</td><td>42.82</td><td>54.74</td>
  </tr>
  <tr>
    <td>TLogic</td><td>26.03</td><td>37.42</td><td>46.50</td><td>28.72</td><td>40.48</td><td>50.71</td><td>29.84</td><td>42.40</td><td>53.37</td>
  </tr>
  <tr>
    <td>GenTKG</td><td>30.60</td><td>42.20</td><td>49.30</td><td>34.90</td><td>46.60</td><td>54.00</td><td>36.00</td><td>48.70</td><td>55.50</td>
  </tr>
  <tr>
    <td>G2S<sub>SL(F)</sub></td><td><b>33.65</b></td><td><b>45.88</b></td><td>55.81</td><td>34.12</td><td>46.63</td><td>56.63</td><td>35.10</td><td>48.08</td><td>58.07</td>
  </tr>
  <tr>
    <td>G2S<sub>SL(R)</sub></td><td>33.22</td><td>45.75</td><td>56.08</td><td>33.70</td><td>46.29</td><td>56.24</td><td>34.12</td><td>46.50</td><td>56.44</td>
  </tr>
  <tr>
    <td>G2S<sub>SL(G)</sub></td><td>31.84</td><td>44.95</td><td><b>56.21</b></td><td>33.84</td><td><b>47.72</b></td><td><b>61.39</b></td><td><b>36.01</b></td><td><b>50.93</b></td><td><b>65.29</b></td>
  </tr>
</table>

Table 3: Results of low-resource TKG reasoning on ICEWS14. The best results among all models are in boldface.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">H@1</th>
    <th>H@3</th>
  </tr>
  <tr>
    <th colspan="2">Standard Setting</th>
    <th></th>
  </tr>
  <tr>
    <td>G2S</td><td>38.33</td><td>54.12</td>
  </tr>
  <tr>
    <td>w/o. Ent</td><td>35.67<sup>2.66↓</sup></td><td>50.24<sup>3.88↓</sup></td>
  </tr>
  <tr>
    <td>w/o. Rel</td><td>38.00<sup>0.33↓</sup></td><td>53.30<sup>0.82↓</sup></td>
  </tr>
  <tr>
    <td>w/o. Ent&Rel</td><td>35.69<sup>2.64↓</sup></td><td>50.20<sup>3.92↓</sup></td>
  </tr>
  <tr>
    <th colspan="3">Zero-shot Setting</th>
  </tr>
  <tr>
    <td>G2S<sub>GL</sub></td><td>32.02</td><td>44.99</td>
  </tr>
  <tr>
    <td>w/o. GL</td><td>13.67<sup>18.35↓</sup></td><td>28.89<sup>16.10↓</sup></td>
  </tr>
  <tr>
    <th colspan="3">Low-resource Setting</th>
  </tr>
  <tr>
    <td>G2S<sub>SL</sub></td><td>33.65</td><td>45.88</td>
  </tr>
  <tr>
    <td>w/o. GL</td><td>32.13<sup>1.52↓</sup></td><td>44.97<sup>0.91↓</sup></td>
  </tr>
</table>

Table 4: Ablation study on ICEWS14 under all three settings.

mappings between entities and IDs are removed in the specific learning stage; w/o. Rel means the mappings between relations and IDs are removed in the specific learning stage; and w/o. Ent&Rel means both kinds of mappings are removed in the specific learning stage. For zero-shot and low-resource settings, we evaluate the variants that remove the general learning stage and use llama3 as the initial model in the specific learning stage, which is denoted as w/o. GL. Here, G2S<sub>GL</sub> is G2S<sub>GL(R) w. Wiki</sub> in Section 6.2; and G2S<sub>SL</sub> is G2S<sub>SL(F)</sub> in Section 6.3 that use 5% training samples.

The experimental results are shown in Table 4, where we can observe that:

(1) G2S outperforms w/o. Ent, w/o. Rel, and w/o. Ent&Rel, which shows the effectiveness of scenario information in TKG forecasting. In addition, the scenario information about entities are usually more important than that about relations.

(2) G2S<sub>GL</sub> and G2S<sub>SL</sub> outperform corresponding w/o. GL models, respectively, which shows the effectiveness of the general learning stage in enhancing the generalization ability.

6.5 Analyses on Anonymization Strategies

From the above experimental results, we can conclude that:

As shown in Table 2, in the general learning stage, the performance of FID (G2S<sub>GL(F)</sub>) and RID (G2S<sub>GL(R)</sub>) strategies are comparable under the zero-shot setting. Both strategies enhance the generalization ability of the model, compared to the model that is additionally trained on the scenario information (G2S<sub>GL(F+Map)</sub>).

In the specific learning stage, the three strategies show different performances under each setting. As shown in Table 1, GID shows the best performance under the standard setting, where sufficient samples are available to learn the static mapping between an entity/relation and a unique ID. However, the different results on 5% and 50% training data shown in Table 3 imply that, GID requires more training data than the other two strategies. As shown in Tables 2 and 3, FID performs best under the low-resource and zero-shot settings. We guess the reason is that FID contains the frequency features in its IDs, so it outperforms RID.

7 Conclusions

This paper investigates the generalization ability of LLMs for temporal knowledge graph forecasting. Firstly, We identified two kinds of entangled knowledge in TKGs, namely, general patterns and
scenario information. Specifically, we adopted several anonymization strategies to convert the entities and relations in TKGs into abstract IDs. Via these structures, the LLMs are able to learn the general patterns more comprehensively. Further, we proposed a general-to-specific learning framework (G2S), which disentangles the learning of these two kinds of knowledge into two learning stages: the general and specific learning stages. To verify the generalization ability of the learned G2S model, we conducted experiments under standard, zero-shot, and low-resource settings. The results demonstrate that the model is able to outperform the baseline methods under all three settings, which shows the effectiveness of the proposed method.

Limitations

We find the following limitations:

(1) This paper only studies the generalization ability of G2S on 5 TKGs about geopolitics and general domains. The generalization ability of the model in other specific domains requires further exploration.

(2) Previous studies have proposed different strategies to select relevant historical facts for queries, including rule-based and reinforcement learning-based strategies. In this paper, we only investigate the generalization ability under one-hop strategy for simplicity. Investigating the generalization ability under other history selection strategies and multi-hop facts will be an important direction for future work.

(3) The GID strategy in G2S usually assigns larger IDs for entities, which sometimes requires the model to generate a complete ID through multi-step decoding. Currently, LLM-based TKG forecasting methods, especially those adopting SFT, cannot deal with this problem well.

Acknowledgements

This work is partially funded by the National Natural Science Foundation of China under grant 62306299, 62441229 and 62406308, the Lenovo-CAS Joint Lab Youth Scientist Project, the project under Grants No. JCKY2022130C039, and the Strategic Priority Research Program of the CAS under Grants No. XDB0680102. Thanks to the reviewers for the constructive discussions and suggestions.

References

AI@Meta. 2024. Llama 3 model card.

Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95–136, virtual+Dublin. Association for Computational Linguistics.

Bin Chen, Chunjing Xiao, and Fan Zhou. 2024. Natural evolution-based dual-level aggregation for temporal knowledge graph reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9274–9284, Miami, Florida, USA. Association for Computational Linguistics.

Zifeng Ding, Heling Cai, Jingpei Wu, Yunpu Ma, Ruotong Liao, Bo Xiong, and Volker Tresp. 2024. zrLLM: Zero-shot relational learning on temporal knowledge graphs with large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1877–1895, Mexico City, Mexico. Association for Computational Linguistics.

Patrick Ernst, Cynthia Meng, Amy Siu, and Gerhard Weikum. 2014. Knowlife: A knowledge graph for health and life sciences. In 2014 IEEE 30th International Conference on Data Engineering, pages 1254–1257.

Fuli Feng, Xiangnan He, Xiang Wang, Cheng Luo, Yiqun Liu, and Tat-Seng Chua. 2019. Temporal relational ranking for stock prediction. ACM Trans. Inf. Syst., 37(2).

Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. 2021a. Explainable subgraph reasoning for forecasting on temporal knowledge graphs. In International Conference on Learning Representations.

Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp. 2021b. Learning neural ordinary equations for forecasting future links on temporal knowledge graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8352–8364, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Tingsong Jiang, Tianyu Liu, Tao Ge, Lei Sha, Sujian Li, Baobao Chang, and Zhifang Sui. 2016. Encoding temporal information for time-aware link prediction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2350–2354, Austin, Texas. Association for Computational Linguistics.

Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. 2020. Recurrent event network: Autoregressive structure inference over temporal knowledge graphs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6669–6683, Online. Association for Computational Linguistics.

Julien Leblay and Melisachew Wudage Chekol. 2018. Deriving validity time in knowledge graph. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1771–1776, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.

Dong-Ho Lee, Kian Ahrabian, Woojeong Jin, Fred Morstatter, and Jay Pujara. 2023. Temporal knowledge graph forecasting without knowledge using in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 544–557, Singapore. Association for Computational Linguistics.

Yujia Li, Shiliang Sun, and Jing Zhao. 2022. Tigrn: Time-guided recurrent graph network with local-global historical patterns for temporal knowledge graph reasoning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 2152–2158. International Joint Conferences on Artificial Intelligence Organization. Main Track.

Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhao Wang, and Xueqi Cheng. 2021a. Search from history and reason for future: Two-stage reasoning on temporal knowledge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4732–4743, Online. Association for Computational Linguistics.

Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhao Wang, and Xueqi Cheng. 2021b. Temporal knowledge graph reasoning based on evolutional representation learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 408–417, New York, NY, USA. Association for Computing Machinery.

Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, and Xinwang Liu. 2023. Learn from relational correlations and periodic events for temporal knowledge graph reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 1559–1568, New York, NY, USA. Association for Computing Machinery.

Ruoqiong Liao, Xu Jia, Yangzhe Li, Yunpu Ma, and Volker Tresp. 2024. GenTKG: Generative forecasting on temporal knowledge graph with large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4303–4317, Mexico City, Mexico. Association for Computational Linguistics.

Kangzheng Liu, Feng Zhao, Guandong Xu, Xianzhi Wang, and Hai Jin. 2023. Retia: Relation-entity twin-interact aggregation for temporal knowledge graph extrapolation. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 1761–1774.

Yushan Liu, Yunpu Ma, Marcel Hildebrandt, Mitchell Joblin, and Volker Tresp. 2022. Tlogic: Temporal logical rules for explainable link forecasting on temporal knowledge graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4):4120–4127.

Farzaneh Mahdisoltani, Joanna Biega, and Fabian M. Suchanek. 2013. YAGO3: A Knowledge Base from Multilingual Wikipedias. In CIDR, Asilomar, United States.

Fred Morstatter, Aram Galstyan, Gleb Satyukov, Daniel Benjamin, Andres Abelius, Mehrnoosh Mirtaheri, KSM Tozammel Hossain, Pedro Szekely, Emilio Ferrara, Akira Matsui, Mark Steyvers, Stephen Benet, David Budescu, Mark Himmelstein, Michael Ward, Andreas Beger, Michele Catasta, Rok Sosic, Jure Leskovec, Pavel Atanasov, Regina Joseph, Rajiv Sethi, and Ali Abbas. 2019. Sage: A hybrid geopolitical event forecasting system. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 6557–6559. International Joint Conferences on Artificial Intelligence Organization.

Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. 2021. TimeTraveler: Reinforcement learning for temporal knowledge graph forecasting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8306–8319, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashykhov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Yuwei Xia, Ding Wang, Qiang Liu, Liang Wang, Shu Wu, and Xiaoyu Zhang. 2024. Chain-of-history reasoning for temporal knowledge graph forecasting. Preprint, arXiv:2402.14382.
Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2023. Temporal knowledge graph reasoning with historical contrastive learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4):4765–4773.

Mengqi Zhang, Yuwei Xia, Qiang Liu, Shu Wu, and Liang Wang. 2023. Learning latent relations for temporal knowledge graph reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12617–12631, Toronto, Canada. Association for Computational Linguistics.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics.

A Multi-Token ID Generation Issue

G2S is designed to generate single-token IDs, which may fail on the queries whose answers are multi-token IDs. This issue, referred to as the multi-token ID generation issue, particularly occurs when adopting the GID strategy, since the IDs assigned by this strategy are usually much larger. This issue arises in certain cases, potentially affecting the model’s overall performance. As shown in Table 5, only 15.4% of the queries in ICEWS14 have an answer with a multi-token ID, so the impact of this issue is limited. For ICEWS18, this percentage increases to 40.2%, which impacts the overall performance of the model to some extent. For YAGO, this percentage is 91.8%, which means GID will fail on most queries in this dataset.

<table>
  <tr>
    <th></th>
    <th>ICEWS14</th>
    <th>ICEWS18</th>
    <th>YAGO</th>
  </tr>
  <tr>
    <th># Queries</th>
    <td>17,028</td>
    <td>91,990</td>
    <td>39,046</td>
  </tr>
  <tr>
    <th># MT IDs</th>
    <td>2,621</td>
    <td>36,981</td>
    <td>35,852</td>
  </tr>
  <tr>
    <th>Percentage</th>
    <td>15.4%</td>
    <td>40.2%</td>
    <td>91.8%</td>
  </tr>
</table>

Table 5: Statistics on percentage of multi-token answers in validation set of ICEWS14, ICEWS18, and YAGO. “# Queries” denotes the number of total queries; and “# MT IDs” denotes the number of queries where the answers are multi-token IDs.

B Dataset Statistics

The basic statistics of the datasets are shown in Table 7.

### Entities ###
2:Citizen (Nigeria)
...
28:Boko Haram
...
192:Cameroon
...
### Relations ###
...
11:Use conventional military force
...
25:Abduct, hijack, or take hostage
...
### History ###
40:[28,25,2]
...
19:[28,25,192]
...
17:[28,25,2]
15:[28,25,2]
...
2:[28,25,2]
1:[28,11,2]
### Query ###
0:[28,25,?]
### Answer ###
2

Figure 3: An example of the prompt together with answer in specific learning stage.

C Additional Results

Table 6 shows experimental results which compare G2S with additional baseline models under standard setting. The baseline models includes TiRGN (Li et al., 2022), CENET (Xu et al., 2023), RETIA (Liu et al., 2023), L2TKG (Zhang et al., 2023), and RPC (Liang et al., 2023). As shown in the table, G2S obtains higher or competitive performance compared to the baselines.

D Prompt Example and Case Study

An example of the prompt in specific learning stage is shown in Figure 3. The prompt in general learning stage excludes “Entity” and “Relation” parts.

We found a training query in GDELT (in RID) which is similar in structure but different in semantic, as shown in Figure 4. The specific scenario information for this query is shown in Figure 5:

Note that the “history” part of the above two
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">ICEWS14</th>
    <th colspan="3">ICEWS18</th>
    <th colspan="3">ICEWS05-15</th>
  </tr>
  <tr>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
  </tr>
  <tr>
    <td>RiRGN</td>
    <td>33.73</td>
    <td>49.85</td>
    <td>64.46</td>
    <td>23.10</td>
    <td>37.90</td>
    <td>54.20</td>
    <td>39.07</td>
    <td>55.75</td>
    <td>70.11</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>30.18</td>
    <td>41.79</td>
    <td>55.20</td>
    <td>23.47</td>
    <td>34.05</td>
    <td>47.27</td>
    <td>38.14</td>
    <td>49.89</td>
    <td>62.53</td>
  </tr>
  <tr>
    <td>RETIA</td>
    <td>32.47</td>
    <td>48.01</td>
    <td>63.64</td>
    <td>21.96</td>
    <td>36.18</td>
    <td>51.95</td>
    <td>34.02</td>
    <td>49.64</td>
    <td>64.42</td>
  </tr>
  <tr>
    <td>L2TKG</td>
    <td>35.36</td>
    <td>-</td>
    <td><b>71.05</b></td>
    <td>22.15</td>
    <td>-</td>
    <td>55.04</td>
    <td>41.86</td>
    <td>-</td>
    <td>80.69</td>
  </tr>
  <tr>
    <td>RPC</td>
    <td>34.87</td>
    <td>49.80</td>
    <td>65.08</td>
    <td><b>24.34</b></td>
    <td><b>38.74</b></td>
    <td><b>55.89</b></td>
    <td>39.47</td>
    <td>57.11</td>
    <td>71.75</td>
  </tr>
  <tr>
    <td>G2S</td>
    <td><b>38.33</b></td>
    <td><b>54.12</b></td>
    <td>68.58</td>
    <td>23.04</td>
    <td>35.32</td>
    <td>46.62</td>
    <td>41.39</td>
    <td><b>58.15</b></td>
    <td><b>73.34</b></td>
  </tr>
</table>

Table 6: Results of TKG reasoning on datasets ICEWS14, ICEWS18, and ICEWS05-15. The best results among all models are in boldface. The experimental results of baselines are cited from Chen et al. (2024).

<table>
  <tr>
    <th>Dataset</th>
    <th>Schema</th>
    <th># Entity</th>
    <th># Relation</th>
    <th># Train</th>
    <th># Valid</th>
    <th># Test</th>
    <th>Granularity</th>
  </tr>
  <tr>
    <td>ICEWS14</td>
    <td>CAMEO</td>
    <td>7,128</td>
    <td>230</td>
    <td>74,845</td>
    <td>8,514</td>
    <td>7,371</td>
    <td>1 day</td>
  </tr>
  <tr>
    <td>ICEWS18</td>
    <td>CAMEO</td>
    <td>23,033</td>
    <td>256</td>
    <td>373,018</td>
    <td>45,995</td>
    <td>49,545</td>
    <td>1 day</td>
  </tr>
  <tr>
    <td>ICEWS05-15</td>
    <td>CAMEO</td>
    <td>10,488</td>
    <td>251</td>
    <td>368,868</td>
    <td>46,302</td>
    <td>46,159</td>
    <td>1 day</td>
  </tr>
  <tr>
    <td>YAGO</td>
    <td>YAGO</td>
    <td>10,623</td>
    <td>10</td>
    <td>161,540</td>
    <td>19,523</td>
    <td>20,026</td>
    <td>1 year</td>
  </tr>
  <tr>
    <td>GDELT</td>
    <td>CAMEO</td>
    <td>7,691</td>
    <td>240</td>
    <td>1,734,399</td>
    <td>238,765</td>
    <td>305,241</td>
    <td>15 min</td>
  </tr>
  <tr>
    <td>WIKI</td>
    <td>Wikidata</td>
    <td>12,554</td>
    <td>24</td>
    <td>539,286</td>
    <td>67,538</td>
    <td>63,110</td>
    <td>1 year</td>
  </tr>
</table>

Table 7: Statistics of the datasets. “# Entity/Relation” denotes the number of entities/relations, and “# Train/Valid/Test” denotes the number of facts in train/valid/test set, respectively.

### History ###
...
294:[0,1,6]
...
151:[0,1,1]
...
78:[0,1,1]
...
32:[0,1,1]
...
7:[0,5,1]

### Query ###
0:[0,1,?]

### Answer ###
1

### Entities ###
0:David Allen
1:United States
...
6:Engineer
...

### Relations ###
0:Accuse
1:Arrest, detain
...
5:Make a visit
...

Figure 5: The corresponding entity and relation ID mapping.

Figure 4: A training sample in general learning stage.

queries are represented in anonymous temporal structures, thus the model can mine the patterns more easily. Though these two samples use different anonymization strategies and are different in time granularity, they follow the same pattern. This pattern may contain facts related to the query ([28,25, 192], [28,25,2], [0,1,6] and [0,1,1]) and the correct answer depends on the frequency of tail entity. This case shows the possibility of learning general patterns across datasets.

Reviewer paper summaries: This paper proposes A General-to-Specific (G2S) Learning Framework for Temporal Knowledge Graph Forecasting. G2S consists of a General Learning Stage and a Specific Learning Stage, which are used to learn the General patterns in Temporal Knowledge Graphs (TKG) and the Specific patterns in queries, respectively. G2S achieves results that surpass the baselines on three datasets: ICEWS14, ICEWS18, and YAGO.

Traditional Temporal Knowledge Graph (TKG) forecasting methods typically predict future facts by simultaneously learning general patterns (invariant temporal structures shared across scenarios) and scenario-specific information (factual knowledge tied to specific entities and relations). This simultaneous learning process may lead to interference between the two types of knowledge, thereby limiting the model's generalization ability. To address this issue, this paper proposes a General-to-Specific (G2S) learning framework, which decouples the learning processes of general patterns and scenario-specific information, significantly enhancing the generalization ability of Large Language Models (LLMs) in TKG forecasting tasks.

**Contributions:**

1. Identifying and Disentangling Two Types of Entangled Knowledge in TKGs:
   We identify two types of entangled knowledge in TKGs—general patterns and scenario-specific information—and disentangle their learning processes through multiple anonymization strategies. General patterns are learned through anonymous temporal structures, while scenario-specific information is injected in subsequent stages, thereby avoiding interference during the learning process.
2. Proposing the G2S Learning Framework:
   The G2S framework consists of two stages: the general learning stage and the specific learning stage. In the general learning stage, the model learns general patterns through anonymous temporal structures. In the specific learning stage, the model combines scenario-specific information via either in-context learning or fine-tuning to make predictions. This staged learning approach significantly enhances the model's generalization ability.
3. Extensive Experiments Under Three Settings:
   We conduct extensive experiments under three settings—zero-shot, low-resource, and standard learning—demonstrating the effectiveness and superiority of the G2S framework in TKG forecasting tasks.

The authors propose a new framework for temporal knowledge graph forecasting using LLMs. They focus on integrating general and specific knowledge patterns into a unified framework that will allow better generalization in the predictions. Specifically, their framework uses a two-stage process where first it learns from anonymized samples and then (while preserving the mappings) learns (or uses ICL) to utilize information from entities and relations meanings.
Summary of strengths: 1. The motivation is clear.
2. The paper conducts extensive experiments on three datasets. 
3. The analysis in the paper regarding why GID cannot be used on the YAGO dataset is interesting.

1. Proposing a Decoupling Mechanism for General Patterns and Scenario-Specific Information:
   By separating the learning processes of general patterns and scenario-specific information, the proposed method addresses the issue of interference between the two types of knowledge in traditional approaches, significantly enhancing the model's generalization ability.
2. The G2S Framework Combines General and Specific Learning Stages:
   The G2S framework consists of two stages: the general learning stage and the specific learning stage. In the general learning stage, the model learns from anonymous temporal structures, while in the specific learning stage, it incorporates scenario-specific information. This approach leverages the general knowledge representation capability of large language models and the fine-tuning ability for specific tasks.
3. Enhancing Temporal Dynamics Modeling in the General Learning Stage:
   During the general learning stage, the anonymization strategy abstracts temporal information into relative time intervals, avoiding temporal bias between training and testing data. This design strengthens the model's ability to capture temporal dynamics.

1. The paper is well-written and relatively easy to follow.
2. The frameworks beats the existing baselines, both LLM-based and GNN-based.
3. The ablation studies investigate the claims faithfully.
Summary of weaknesses: 1. Limitations of the method: Although attempting to learn both general patterns and specific patterns in TKG is a well-motivated idea, relying solely on different ID strategies, fine-tuning, and purely metric-based analysis does not seem sufficient to support that the proposed method can effectively learn these two types of patterns.
2. Experimental setup is unclear. The authors did not explain why evaluations were not conducted on ICEWS05-15, WIKI, and GDELT. If the goal is to validate the reasoning ability across datasets, the cross-dataset generalization setup of GenTKG [1] could be used as a reference. The authors seem to have a misunderstanding of the concept of zero-shot; simply not using training data at a certain stage and referring to it as a zero-shot setup is not rigorous. The paper also does not provide the prompts and training data samples necessary for using LLMs.

[1] GenTKG: Generative Forecasting on Temporal Knowledge Graph. NAACL 2024 Findings.

1. The performance decline on ICEWS18 stems from the "multi-label ID generation issue," but no improvement solutions have been proposed to address this problem.  
2. The research does not include some recent or high-performing baseline models, such as TiRGN, CENET, RETIA, L²TKG, RPC, and NEDA (which achieves Hits@10=73.46% on ICEWS14), making it difficult to prove the absolute performance advantage of G2S.  
3. The research does not use temporal knowledge graph datasets such as ICEWS05-15 (covering 11 years of data), ICEWS22, and ICEWS23 (newer datasets) for experimentation.  
4. The article does not consider the cost issues associated with using large models.

1. When you compare your method to other LLM-based models, do you use the anonymized version of the prompts or the raw version? I'm a bit concerned about the fairness of the comparison, as you include the entity/relation information in your prompts but might not do that for other methods.
2. The argument for better generalization is relatively weak based on the zero-shot results. The method is comparable on the two of the more critical datasets (i.e., ICEWS) to GPT-NeoX-ICL (note that none of the llama-based methods were included in the original ICL-based paper by Lee et al., 2023; so, in some sense, they are untested variants), even though it undergoes some SFT steps.
3. Why not include GPT-NeoX-ICL as a low-resource comparison? Isn't it the ultimate low-resource method that doesn't need ANY training?
4. I still don't understand why the two-stage process is needed. Why not combine the two steps into one step where the entities/relation information is included? Do you have any results on the benefit of this architecture?
5. When you talk about "shared" schema, do you mean entities and/or relations, or how the data was formatted?
Comments, suggestions and typos: Suggestions:

Providing additional analytical methods beyond metric analysis to verify that the model can learn general patterns and specific patterns would greatly enhance the paper.

Figure 1 lacks a clear illustration of anonymized structures in the main text, reducing readability.

It's a bit hard to follow all the abbreviations (e.g., SL, GL, etc.), so please consider making them more verbose.
Overall assessment scores: 2

3.5

3.5
Metareview score: 3
Metareview: This paper presents a General-to-Specific (G2S) Learning Framework for TKG forecasting using LLMs. It decouples learning of general patterns and specific info into two stages, enhancing generalization and outperforming baselines on ICEWS14, ICEWS18, and YAGO.
Summary of reasons to publish: 1. The paper is well-written and relatively easy to follow.

2. The paper conducts extensive experiments on three datasets. The frameworks beats the existing baselines, both LLM-based and GNN-based.