Venue: NAACL 2024

# zrLLM: Zero-Shot Relational Learning on Temporal Knowledge Graphs

with Large Language Models

Zifeng Ding\({}^{*,1,2}\), Heling Cai\({}^{*}\)\({}^{1}\), Jingpei Wu\({}^{1}\), Yunpu Ma\({}^{1,3}\),

Ruotong Liao\({}^{1,3}\), Bo Xiong\({}^{*}\), Volker Tresp\({}^{1}\)

\({}^{1}\)LMU Munich \({}^{2}\)Siemens AG

\({}^{3}\)Munich Center for Machine Learning (MCML) \({}^{4}\)University of Stuttgart

{zifeng.ding, heling.cai}@acampus.lmu.de

{jingpei.wu, liao,tresp}@dbs.ifi.lmu.de,

cognitive.yunpu@gmail.com, bo.xiong@ki.uni-stuttgart.de

Equal contribution.Corresponding author.

###### Abstract

Modeling evolving knowledge over temporal knowledge graphs (TKGs) has become a heated topic. Various methods have been proposed to forecast links on TKGs. Most of them are embedding-based, where hidden representations are learned to represent knowledge graph (KG) entities and relations based on the observed graph contexts. Although these methods show strong performance on traditional TKG forecasting (TKGF) benchmarks, they face a strong challenge in modeling the unseen zero-shot relations that have no prior graph context. In this paper, we try to mitigate this problem as follows. We first input the text descriptions of KG relations into large language models (LLMs) for generating relation representations, and then introduce them into embedding-based TKGF methods. LLM-empowered representations can capture the semantic information in the relation descriptions. This makes the relations, whether seen or unseen, with similar semantic meanings stay close in the embedding space, enabling TKGF models to recognize zero-shot relations even without any observed graph context. Experimental results show that our approach helps TKGF models to achieve much better performance in forecasting the facts with previously unseen relations, while still maintaining their ability in link forecasting regarding seen relations.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1877–1895

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Knowledge graphs (KGs) represent world knowledge with a collection of facts in the form of \((s,r,o)\) triples, where in each fact, \(s\), \(o\) are the subject and object entities and \(r\) is the relation between them. Temporal knowledge graphs (TKGs) are introduced by further specifying the time validity. Each TKG fact is denoted as a quadruple \((s,r,o,t)\), where \(t\) (a timestamp or a time period) provides temporal constraints. Since world knowledge is ever-evolving, TKGs are more expressive in representing dynamic factual information and have drawn increasing interest in a wide range of downstream tasks, e.g., natural language question answering over TKGs Saxena et al. (2021); Ding et al. (2023b).

In recent years, there has been an increasing number of works paying attention to forecasting future facts in TKGs, i.e., TKG forecasting (TKGF) or TKG extrapolated link prediction (LP). Most of them are embedding-based, where entity and relation representations are learned with the help of the observed graph contexts. Although traditional embedding-based TKGF methods show impressive performance on current benchmarks, they share a common limitation. In these works, models are trained on the TKG facts regarding a set of relations \(\mathcal{R}\), and they are only expected to be evaluated on the facts containing the relations in \(\mathcal{R}\). They cannot handle any zero-shot unseen relation \(r\notin\mathcal{R}\) because no graph context regarding unseen relations exists in the training data and thus no reasonable relation representations can be learned. In the forecasting scenario, as time flows, new knowledge is constantly introduced into a TKG, making it expand in size. This increases the chance of encountering newly-emerged relations, and therefore, it is meaningful to improve embedding-based TKGF methods to be more adaptive to zero-shot relations.

With the increasing scale of pre-trained language models (LMs), LMs become large LMs (LLMs). Recent studies find that LLMs have shown emerging abilities in various aspects Wei et al. (2022) and can be taken as strong semantic knowledge bases (KBs) Petroni et al. (2019). Inspired by this, we try to enhance the performance of embedding-based TKGF models over zero-shot relations with an approach consisting of the following three steps: (1) Based on the relation text descriptions provided

in TKG datasets, we first use an LLM to produce an enriched relation description (ERD) with more details for each KG relation (Sec. 3.1). (2) We then generate the relation representations by leveraging another LLM, i.e., T5-11B Raffel et al. (2020). We input ERDs into T5's encoder and transform its output into relation representations of TKGF models (Sec. 3.1). (3) We design a relation history learner (RHL) to capture historical relation patterns, where we leverage LLM-empowered relation representations to better reason over zero-shot relations (Sec. 3.2). With these steps, we align the natural language space provided by LLMs to the embedding space of TKGF models, rather than letting models learn relation representations solely from observed graph contexts. Even without any observed associated facts, zero-shot relations can be represented with LLM-empowered representations that contain semantic information. We term our approach as zrLLM since it is used to enhance zero-shot relational learning on TKGF models by using **LLMs**.

We experiment zrLLM on seven recent embedding-based TKGF models and evaluate them on three new datasets constructed specifically for studying TKGF regarding zero-shot relations. Our contribution is three-folded: (1) To the best of our knowledge, this is the first work trying to study zero-shot relational learning in TKGF. (2) We design an LLM-empowered approach zrLLM and manage to enhance various recent embedding-based TKGF models in reasoning over zero-shot relations. (3) Experimental results show that zrLLM helps to substantially improve all considered TKGF models' abilities in forecasting the facts containing unseen zero-shot relations, while still maintaining their ability in link forecasting regarding seen relations.

## 2 Preliminaries

### Related Work

Traditional TKGF Forecasting Methods.Traditional TKGF methods are trained to forecast the facts containing the KG relations (and entities) seen in the training data, regardless of the case where zero-shot relations (or entities) appear as new knowledge arrives. These methods can be categorized into two types: embedding-based and rule-based. Embedding-based methods learn hidden representations of KG relations and entities, and perform link forecasting based on them. Most existing embedding-based methods, e.g., Jin et al. (2020); Han et al. (2021); Li et al. (2021); Liu et al. (2023), learn evolutional entity and relation representations from the historical TKG information by jointly employing graph neural networks Kipf and Welling (2017) and recurrent neural structures, e.g., GRU Cho et al. (2014). Some other approaches Han et al. (2021); Sun et al. (2021); Li et al. (2021) start from each LP query1 and traverse the temporal history in a TKG to search for the prediction answer. There also exist some methods, e.g., Zhu et al. (2021); Xu et al. (2023), that achieve forecasting based on the appearance of historical facts. Compared with embedding-based TKGF approaches, rule-based TKGF has still not been extensively explored. One popular rule-based TKGF method is TLogic Liu et al. (2022). It extracts temporal logical rules from TKGs and uses a symbolic reasoning module for LP. Based on it, ALRE-IR Mei et al. (2022) proposes an adaptive logical rule embedding model to encode temporal logical rules into rule representations. This makes ALRE-IR both a rule-based and an embedding-based method. Rule-based TKGF methods have strong ability in reasoning over zero-shot unseen entities connected by the seen relations, however, they are not able to handle unseen relations since the learned rules are strongly bounded by the observed relations.

Footnote 1: A TKG LP query is denoted as \((s,r,?,t)\) (object prediction query) or \((?,r,o,t)\) (subject prediction query).

Inductive Learning on TKGs.Inductive learning on TKGs refers to developing models that can handle the relations and entities unseen in the training data. Most of TKG inductive learning methods are based on few-shot learning, e.g., Ding et al. (2022); Zhang et al. (2019); Ding et al. (2023); Mirtaheri et al. (2021); Ding et al. (2023, 2023). They first compute inductive representations of newly-emerged entities or relations based on \(K\)-associated facts (\(K\) is a small number, e.g., 1 or 3), and then use them to predict other facts regarding few-shot elements. One limitation of these works is that the inductive representations cannot be learned without the \(K\)-shot examples, making them hard to solve the zero-shot problems. Different from few-shot learning methods, SST-BERT Chen et al. (2023) pre-trains a time-enhanced BERT Devlin et al. (2019) and proves its inductive power over unseen entities but has not shown its ability in reasoning ze 

recent work MTKGE (Chen et al., 2023b) is able to concurrently deal with both unseen entities and relations. However, it requires a support graph containing a substantial number of data examples related to the unseen entities and relations, which is far from the zero-shot setting.

TKG Reasoning with Language Models.Recently, more and more works have introduced LMs into TKG reasoning. SST-BERT pre-trains an LM on a corpus of training TKGs for fact reasoning. ECOLA Han et al. (2023) aligns facts with additional fact-related texts and enhances TKG reasoning with BERT-encoded language representations. PPT Xu et al. (2023a) converts TKGF into the pre-trained LM masked token prediction task and finetunes a BERT for TKGF. Apart from them, one recent work Lee et al. (2023) explores in-context learning (ICL) Brown et al. (2020) with LLMs to predict future facts without finetuning. Another recent work GenTKG Liao et al. (2023) finetunes Llama2-7B Touvron et al. (2023), and let it directly generate the LP answer in TKGF.

Although previous works have shown success of LMs in TKG reasoning, they have limitations: (1) None of them has studied whether LMs, in particular LLMs, can be used to better reason zero-shot relations. (2) By only using ICL, LLMs are beaten by traditional TKGF methods in performance Lee et al. (2023). The performance can be greatly improved by finetuning LLMs Liao et al. (2023), but finetuning LLMs requires huge computational resources. (3) Since LMs are pre-trained with a huge corpus originating from diverse information sources, it is inevitable that they have already seen the world knowledge before they are used to solve TKG reasoning tasks. Most popular TKGF benchmarks are constructed with the facts before 2020 (ICEWS14/18/05-15 Jin et al. (2020)). The facts inside are based on the world knowledge before 2019, which means LMs might have encountered them in their training corpus, posing a threat of information leak to the LM-driven TKG reasoning models. To this end, we (1) draw attention to studying the impact of LLMs on zero-shot relational learning in TKGs; (2) make a compromise between performance and computational efficiency by not finetuning LMs or LLMs but adapting the LLM-provided semantic information to non-LM-based TKGF methods; (3) construct new benchmarks whose facts are all happening from 2021 to 2023, which avoids the threat of information leak when we utilize T5-11B that was released in 2020.

### Definitions and Task Formulation

**Definition 1 (Tkg).** Let \({\cal E}\), \({\cal R}\), \({\cal T}\) denote a set of entities, relations and timestamps, respectively. A TKG \({\cal G}=\{(s,r,o,t)\}\subseteq{\cal E}\times{\cal R}\times{\cal E}\times{\cal T}\) is a set of temporal facts where each fact is represented with a fact quadruple \((s,r,o,t)\).

**Definition 2 (Tkg Forecasting).** Assume we have a ground truth TKG \({\cal G}_{gt}\) that contains all the true facts. Given an LP query \((s_{q},r_{q},?,t_{q})\) (or \((o_{q},r_{q},?,t_{q})\)), TKGF requires the models to predict the missing object \(o_{q}\) (or subject \(s_{q}\)) based on the facts observed before the query timestamp \(t_{q}\), i.e., \({\cal O}=\{(s,r,o,t_{i})\in{\cal G}_{gt}|t_{i}<t_{q}\}\).

**Definition 3 (Zero-Shot TKG Forecasting).** Assume we have a ground truth TKG \({\cal G}_{gt}\subseteq{\cal E}\times{\cal R}\times{\cal E}\times{\cal T}\), where \({\cal R}\) can be split into seen \({\cal R}_{se}\) and unseen \({\cal R}_{un}\) relations (\({\cal R}={\cal R}_{se}\cup{\cal R}_{un},{\cal R}_{se}\cap{\cal R}_{un}=\emptyset\)). Given an LP query \((s_{q},r_{q},?,t_{q})\) (or \((o_{q},r_{q},?,t_{q})\)) whose query relation \(r_{q}\in{\cal R}_{un}\), models are asked to predict the missing object \(o_{q}\) (or subject \(s_{q}\)) based on the facts \({\cal O}=\{(s,r_{i},o,t_{i})\in{\cal G}_{gt}|t_{i}<t_{q},r_{i}\in{\cal R}_{se}\}\) containing seen relations and happening before \(t_{q}\).

## 3 zrlLM

zrlLM is coupled with TKGF models to enhance zero-shot ability. It uses GPT-3.5 to generate enriched relation descriptions (ERDs) based on the relation texts provided by TKG datasets. It then inputs the ERDs into the encoder of T5-11B and aligns its output to TKG embedding space. zrlLM also employs a relation history learner (RHL) to capture the temporal relation patterns based on the LLM-based relation representations, which further promotes embedding space alignment. See Fig. 1 for illustration of zrlLM-enhanced TKGF models.

### Represent KG Relations with LLMs

Generate Text Representations with ERDs.We generate text representations with T5-11B based on the textual descriptions of KG relations. Since the relation texts provided by TKG datasets are short and concise, we use GPT-3.52 to enrich them for more comprehensive semantics. Our prompt for description enrichment is depicted in Fig. 2. For each relation, we treat the combination of its relation text and LLM-generated explanation as its ERD. See Table 1 for two enrichment examples.

 

We then input the ERDs into T5-11B. T5 is with an encoder-decoder architecture, where its encoder can be taken as a module that helps to understand the text input and the decoder is solely used for text generation. We take the output of T5-11B's encoder, i.e., the hidden representations, for our downstream task. Note that although ERDs are produced by GPT-3.5 who is trained with the corpus until the end of 2021, the representations used for TKGF are generated only with T5-11B, preventing information leak. Also, through our prompt, GPT-3.5 does not know our underlying task of TKGF. We manually check the ERDs generated by GPT-3.5 and make sure that GPT-3.5 generates relation explanations solely from the semantic perspective and no world knowledge is contained in its output.

Align Text Representations to TKG Embedding Space.For each KG relation \(r\), the T5-generated text representation is a parameter matrix \(\bar{\mathbf{H}}_{r}\in\mathbb{R}^{L\times d_{w}}\). \(L\) is the length of the Transformers Vaswani et al. (2017) in T5 and \(d_{w}\) is the embedding size of each word output from T5 encoder. The \(l^{\text{th}}\) row in \(\bar{\mathbf{H}}_{r}\) is the T5 encoded hidden representation \(\mathbf{w}_{l}\in\mathbb{R}^{d_{w}}\) of the \(l^{\text{th}}\) word in the enriched description. To align \(\bar{\mathbf{H}}_{r}\) to an embedding-based TKGF model, we first use a multi-layer perceptronMLP to map each \(\mathbf{w}_{l}\) to the dimension of the TKGF model's relation representation.

\[\mathbf{w}_{l}^{\prime}=\text{MLP}(\mathbf{w}_{l}),\text{where}~{}\mathbf{ w}_{l}^{\prime}\in\mathbb{R}^{d}. \tag{1}\]

Then we learn a representation of \(r\)'s ERD \(\bar{\mathbf{h}}_{r}\) using a GRU.

\[\begin{split}&\bar{\mathbf{h}}_{r}^{(l)}=\text{GRU}(\mathbf{w}_{ l}^{\prime},\bar{\mathbf{h}}_{r}^{(l-1)});~{}\bar{\mathbf{h}}_{r}^{(0)}= \mathbf{w}_{0}^{\prime},\\ &\bar{\mathbf{h}}_{r}=\bar{\mathbf{h}}_{r}^{(L-1)}.\end{split} \tag{2}\]

\(l\in[1,L-1]\). \(\bar{\mathbf{h}}_{r}\) contains semantic information from ERD, and therefore, we can view it as an LM-based relation representation. We substitute the relation representations of TKGF models with LM-based representations for semantics integration. Note that we fix the values of every \(\bar{\mathbf{H}}_{r}\) to keep the LLM-provided semantic information intact. This is because we do not want the relation representations to lay excessive emphasis on the training data where zero-shot relations never appear. We want the models to maximally benefit from the semantic information for better generalization power. The textual descriptions of the relations with close meanings will show similar semantics. Since for each relation \(r\), \(\bar{\mathbf{H}}_{r}\) is generated based on t

Figure 1: Illustration of zrLLM-enhanced TKGF models. RHL-related components are marked in blue. RHL works differently in training and evaluation. During training, since we know both entities (\(s,o\) in 1a) in the training fact, we can find the ground truth historical relations between them over time. We train a history prediction network (HPN) that aims to generate the relation history between two entities given their current relation (\(r\)). During evaluation, we directly use the trained HPN to infer the relation history. See Sec. 3 for details.

\begin{table}
\begin{tabular}{c c} \hline
**KG Relations Test** & **Extended Relations Divertions** \\ \hline Design in registration & Engrist in registration. This includes a willingness to participate in discussions or dialogue environments such as transferable approaches or techniques or techniques. In practice or reliance. These policies require redundant or irrelevant entity’s actions, picking, or behavior. It is a form of opposing aspect or adjustments. \\ \hline Prior \# address & Prints or enhance. This applies some redundant or irrelevant entity’s actions, picking, or behavior. It is a form of upcoming aspects of abstractions. \\ \hline \end{tabular}
\end{table}
Table 1: Relation description enrichment examples.

 

\(r\)'s ERD, the relations with close meanings will naturally lead to highly correlated text representations, building connections on top of the natural language space regardless of the observed TKG data.

### Improving Text-to-Graph Alignment with Relation History Learner

As the relationship between two entities evolves through time, it follows certain temporal patterns. For example, the fact (_China_, _Sign formal agreement_, _Nicaragua_, 2022-01-10) happens after (_China_, _Grant diplomatic recognition_, _Nicaragua_, 2022-01-04), implying that an agreement will be signed after showing diplomatic recognition. These temporal patterns are entity-agnostic and can reflect the dynamic relationship between any two entities over time. To this end, we develop RHL, aiming to capture such patterns. RHL leverages the LLM-based relation representations for pattern modeling, which further promotes the alignment between the text and TKG embedding spaces.

Assume we have a training fact \((s,r,o,t)\), we search for the historical facts \({\mathcal{G}^{<t}_{s,o}}\) containing \(s\) and \(o\) before \(t\), and group these facts according to their timestamps, i.e., \({\mathcal{G}^{<t}_{s,o}=\{{\mathcal{G}^{0}_{s,o}},...,{\mathcal{G}^{t-1}_{s, o}}\}}\). The searched facts with the same timestamp are put into the same group. For each group, we pick out the relations of all its facts and form a relation set, e.g., \({\mathcal{R}^{0}_{s,o}}\) is derived from \({\mathcal{G}^{0}_{s,o}}\). \(s\) and \(o\)'s relationship at \(t_{i}\) (\(t_{i}\in[0,t-1]\)) is computed with an aggregator

\[\mathbf{h}^{t_{i}}_{s,o}=\sum_{m}a_{m}\tilde{\mathbf{h}}_{r_{m}};~{}a_{m}= \mathrm{softmax}(\bar{\mathbf{h}}^{T}_{r_{m}}\text{MLP}_{\text{agg}}(\tilde{ \mathbf{h}}_{r})). \tag{3}\]

\(r_{m}\in{\mathcal{R}^{t}_{s,o}}\) denotes a relation bridging \(s\) and \(o\) at \(t_{i}\). If \({\mathcal{R}^{t}_{s,o}}=\emptyset\), we set \(\mathbf{h}^{t_{i}}_{s,o}\) to a dummy embedding \(\mathbf{h}_{\text{dum}}\). To capture the historical relation dynamics, we use another GRU, i.e., \(\text{GRU}_{\text{RHL}}\).

\[\begin{split}&\mathbf{h}^{t_{i}}_{\text{hist}}=\text{GRU}_{ \text{RHL}}(\mathbf{h}^{t_{i}}_{s,o},\mathbf{h}^{t_{i}-1}_{\text{hist}});~{} \mathbf{h}^{0}_{\text{hist}}=\mathbf{h}^{0}_{s,o},\\ &\mathbf{h}_{\text{hist}}=\mathbf{h}^{t-1}_{\text{hist}}.\end{split} \tag{4}\]

\(\mathbf{h}_{\text{hist}}\) is taken as the encoded relation history until \(t-1\). Note that during evaluation, TKGF asks models to predict the missing object of each LP query \((s_{q},r_{q},?,t_{q})\), which means we do not know which two entities should be used for historical fact searching3. To solve this problem, during training, we train another history prediction network (HPN) that aims to directly infer the relation history given the training fact relation \(r\).

Footnote 3: We can indeed couple \(s_{q}\) with every candidate entity \(e\in\mathcal{E}\) and search for their historical facts. But it requires huge computational resources and greatly harms model’s scalability.

\[\tilde{\mathbf{h}}_{\text{hist}}=\alpha\text{MLP}_{\text{hist}}(\bar{ \mathbf{h}}_{r})+\bar{\mathbf{h}}_{r}. \tag{5}\]

Here, \(\alpha\) is a hyperparameter scalar and \(\text{MLP}_{\text{hist}}\) is an MLP. \(\tilde{\mathbf{h}}_{\text{hist}}\) is the predicted relation history given \(r\). Since we want \(\tilde{\mathbf{h}}_{\text{hist}}\) to represent the ground truth relation history, we use a mean square error (MSE) loss to constrain it to be close to \(\mathbf{h}_{\text{hist}}\).

\[\mathcal{L}_{\text{hist}}=\text{MSE}(\tilde{\mathbf{h}}_{\text{hist}},\mathbf{ h}_{\text{hist}}). \tag{6}\]

In this way, during evaluation, we can directly use Eq. 5 to generate a meaningful \(\tilde{\mathbf{h}}_{\text{hist}}\) for further computation. Given \(\tilde{\mathbf{h}}_{\text{hist}}\), we do one more step in \(\text{GRU}_{\text{RHL}}\) to capture the \(r\)-related relation pattern.

\[\mathbf{h}_{\text{pat}}=\text{GRU}_{\text{RHL}}(\bar{\mathbf{h}}_{r},\tilde{ \mathbf{h}}_{\text{hist}}). \tag{7}\]

\(\mathbf{h}_{\text{pat}}\) can be viewed as a hidden representation containing comprehensive information of temporal relation patterns. Inspired by TuckER Balazevic et al. (2019), we compute an RHL-based score for the training target \((s,r,o,t)\) as

\[\phi((s,r,o,t))=\mathcal{W}\times_{1}\mathbf{h}_{(s,t)}\times_{2}\mathbf{h}_{ \text{pat}}\times_{3}\mathbf{h}_{(o,t)}, \tag{8}\]

where \(\mathcal{W}\in\mathbb{R}^{d\times d\times d}\) is a learnable core tensor and \(\times_{1},\times_{2},\times_{3}\) are three operators indicating the tensor product in three different modes (details in Balazevic et al. (2019)). \(\mathbf{h}_{(s,t)}\) and \(\mathbf{h}_{(o,t)}\) are the time-aware entity representations of \(s\) and \(o\) computed by TKGF model, respectively. RHL-based score can be viewed as measuring how much two entities match the relation pattern generated by the relation history. We couple this score with the score computed by the original TKGF model \(\phi^{\prime}((s,r,o,t))\) and use the total score for LP.

\[\phi_{\text{total}}((s,r,o,t))=\phi^{\prime}((s,r,o,t))+\gamma\phi((s,r,o,t))\,. \tag{9}\]

\(\gamma\) is a hyperparameter. RHL enables models to make decisions by additionally considering the temporal relation patterns. Note that patterns are captured with LLM-empowered relation representations that contain rich semantic information. This guarantees RHL to generalize well to zero-shot relations. See App. I for explanations.

### Parameter Learning and Evaluation

We let zrLLM be co-trained with TKGF model. Assume \(f\) is a TKGF model's loss function, e.g., cross-entropy, where \(f\) takes a fact quadruple's 

score computed by model's score function \(\phi^{\prime}\) and returns a loss for this fact. We input the quadruple score computed with Eq. 9 into \(f\) to let TKGF models better learn the parameters in RHL.

\[\mathcal{L}_{\text{TKGF}}=\tfrac{1}{|\mathcal{G}_{\text{train}}|}\sum_{\lambda \in\mathcal{G}_{\text{train}}}f(\phi_{\text{total}}(\lambda)), \tag{10}\]

where \(\lambda\) denotes a fact quadruple \((s,r,o,t)\in\mathcal{G}_{\text{train}}\) in the training set \(\mathcal{G}_{\text{train}}\). Besides, we also employ an additional binary cross-entropy loss \(\mathcal{L}_{\text{RHL}}\) directly on the RHL-based score

\[\begin{split}&\mathcal{L}_{\text{RHL}}=\frac{1}{N}\sum_{\lambda }\sum_{e\in\mathcal{E}}\mathcal{L}_{\text{RHL}}^{\lambda,e};\\ &\mathcal{L}_{\text{RHL}}^{\lambda,e}=-y_{\lambda^{\prime}}\log( \phi(\lambda^{\prime}))-(1-y_{\lambda^{\prime}})\log(1-\phi(\lambda^{\prime})) .\end{split} \tag{11}\]

\(N=|\mathcal{G}_{\text{train}}|\times|\mathcal{E}|\). \(\lambda^{\prime}\) is a perturbed fact by switching the object of \(\lambda\) to any \(e\in\mathcal{E}\) and \(y_{\lambda^{\prime}}\) is its label. If \(\lambda^{\prime}\in\mathcal{G}_{\text{train}}\), then \(y_{\lambda^{\prime}}=1\), otherwise \(y_{\lambda^{\prime}}=0\). Finally, we define the total loss \(\mathcal{L}_{\text{total}}\) as

\[\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{TKGF}}+\mathcal{L}_{\text{hist} }+\eta\mathcal{L}_{\text{RHL}}. \tag{12}\]

\(\eta\) is a hyperparameter deciding \(\mathcal{L}_{\text{RHL}}\)'s magnitude. Given our loss, we can also view RHL as a module that does a subtask during training. The subtask is to leverage the relation patterns encoded solely with LLM-based relation representations to perform TKG forecasting, which is parallel to the pipeline of the original TKGF model. This subtask training process helps to improve the embedding space alignment between text and graph representations. During evaluation, for each LP query \((s_{q},r_{q},?,t_{q})\), we compute scores \(\{\phi_{\text{total}}((s_{q},r_{q},e,t_{q}))\}|e\in\mathcal{E}\}\) and take the entity with maximum score as the predicted answer. We provide algorithms of training and evaluation in App. D.

## 4 Experiments

We give details of our new zero-shot TKGF datasets in Sec. 4.1. In Sec. 4.3, we (1) do a comparative study to show how zrLLM improves TKGF models, (2) do ablation studies, (3) compare zrLLM with recent LM-enhanced TKGF models, and (4) do a case study to prove RHL's effectiveness. The implementation code and our proposed zero-shot datasets are in the following page: [https://github.com/ZifengDing/zrLLM](https://github.com/ZifengDing/zrLLM)

### Datasets for Zero-Shot TKGF

As discussed in Sec. 2.1, LM-enhanced TKGF models experience the risk of information leak. To exclude this concern, we construct new benchmark datasets on top of the facts happening after the publication date of T5-11B. We first construct two datasets ICEWS21-zero and ICEWS22-zero based on the Integrated Crisis Early Warning System (ICEWS) (Boschee et al., 2015) KB. ICEWS21-zero contains the facts happening from 2021-01-01 to 2021-08-31, while all the facts in ICEWS22-zero happen from 2022-01-01 to 2022-08-31. Besides, we also construct another dataset ACLED-zero based on another KB: The Armed Conflict Location & Event Data Project (ACLED) (Raleigh et al., 2010). Facts in ACLED-zero take place from 2023-08-01 to 2023-08-31. All the facts in all three datasets are based on social-political events described in English.

Inspired by (Mitraheri et al., 2021), our dataset construction process consists of the following steps. (1) For each dataset, we first collect all the facts within the time period of interest from the associated KB and then sort them in the temporal order. (2) Then we split the collected facts into two splits, where the first split contains the facts for model training and the second one has all the facts for evaluation. Any fact from the evaluation split happens later than the maximum timestamp of all the facts from the training split. Since we are studying zero-shot relations, we exclude the facts in the evaluation split whose entities do not appear in the training split, to avoid the potential impact of unseen entities. (3) We compute the frequencies of all relations in the evaluation split, and set a frequency threshold (40 for ACLED-zero and ICEWS21-zero, 60 for ICEWS22-zero). (4) We take each relation whose frequency is lower than the threshold as a zero-shot relation, and treat every fact containing it in the evaluation split as zero-shot evaluation data \(\mathcal{G}_{\text{test}}\). We exclude the facts associated with zero-shot relations from the training split to ensure that models cannot see these relations during training, and take the rest as the training set \(\mathcal{G}_{\text{train}}\). The rest of facts in the evaluation split are taken as the regular evaluation data \(\mathcal{G}_{\text{valid}}\). We do validation over \(\mathcal{G}_{\text{valid}}\) and test over \(\mathcal{G}_{\text{test}}\) because we want to study how models perform over zero-shot relations when

\begin{table}
\begin{tabular}{l c c c c c c c c c} Dataset & \(|\mathcal{I}|\) & \(|\mathcal{II}|\) & \(|\mathcal{III}|\) & \(|\mathcal{III}|\) & \(|\mathcal{III}|\) & \(|\mathcal{III}|\) & \(|\mathcal{III}|\) & \(|\mathcal{III}|\) & \(|\mathcal{III}|\) \\ \hline ACED-zero & 601 & 22 & 79 & 11 & 9 & 14 & 218.2 & 931 & 146 \\ ICWWS21-zero & 18,205 & 253 & 181 & 62 & 139 & 128 & 247.964 & 77.195 & 1.395 \\ ICWWS22-zero & 999 & 248 & 181 & 62 & 93 & 155 & 171.013 & 47.784 & 1.956 \\ \hline \end{tabular}
\end{table}
Table 2: Dataset statistics. Dataset timestamps consist of both training and evaluation timestamps, i.e., \(\mathcal{T}=\mathcal{T}_{\text{train}}\cup\mathcal{T}_{\text{eval}}\), \(\mathcal{T}_{\text{train}}\cap\mathcal{T}_{\text{real}}=\emptyset\), max\((\mathcal{T}_{\text{train}})<\text{min}(\mathcal{T}_{\text{eval}})\).

 

they reach the best performance over seen relations. See Table 2 and App. B for dataset statistics.

### Experimental Setup

**Training and Evaluation for Zero-Shot TKGF.** All TKGF models are trained on \(\mathcal{G}_{\text{train}}\). We take the model checkpoint achieving the best validation result on \(\mathcal{G}_{\text{valid}}\) as the best model checkpoint, and report their test result on \(\mathcal{G}_{\text{test}}\) to study the zero-shot inference ability. To keep zero-shot relations "always unseen" during the whole test process, we constrain all models to do LP only based on the training set as several popular TKGF methods, e.g., RE-GCN Zhu et al. (2021). Some TKGF models, e.g., TiRGN Li et al. (2022), allow using the ground truth TKG data until the LP query timestamp, including the facts in evaluation sets. This will violate the zero-shot setting because every unseen relation will occur multiple times in the evaluation data and is no longer zero-shot after models observe any fact of it. We prevent them from observing evaluation data to maintain the zero-shot setting. See App. C.5 for explanation. Note that \(\mathcal{G}_{\text{valid}}\) and \(\mathcal{G}_{\text{test}}\) share the same time period. This is because **we want to make sure that zrLLM can enhance zero-shot reasoning and simultaneously maintain TKGF models' performance on the facts with seen relations. Improving zero-shot inference ability at the cost of sacrificing too much performance over seen relations is undesired.**

Baselines and Evaluation Metrics.We consider seven recent embedding-based TKGF methods as baselines, i.e., CyGNet Zhu et al. (2021), TANGOTuckER/Distmult Han et al. (2021), RE-GCN Li et al. (2022), TiRGN Li et al. (2022), CENET Xu et al. (2023b) and RETA Liu et al. (2023). We couple them with zrLLM and show their improvement in zero-shot relational learning on TKGs (implementation details in App. C). We employ two evaluation metrics, i.e., mean reciprocal rank (MRR) and Hits@1/3/10. See App. E for detailed definitions. As suggested in Gastinger et al. (2023), we use the time-aware filtering setting Han et al. (2021a) for fairer evaluation.

### Comparative Study and Further Analysis

Comparative Study.We report the LP results of all baselines and their zrLLM-enhanced versions in Table 3. We have two findings: (1) zrLLM greatly helps TKGF models in forecasting the facts with unseen zero-shot relations. (2) In most cases, zrLLM even improves models in predicting the facts with seen relations. The zrLLM-enhanced models whose performance drops over seen relations still achieve better overall performance. These findings prove that embedding-based TKGF models benefit from the semantic information extracted from LLMs, especially when they are dealing with zero-shot relations.

Ablation Study.We conduct ablation studies from three aspects. (1) First, we directly input the dataset-provided relation texts into T5-11B encoder, ignoring the relation explanations generated by GPT-3.5. From Table 4 (-ERD), we observe that in most cases, models' performance drops on the facts with both seen and zero-shot relations, which proves the usefulness of ERDs. (2) Next, we remove the RHL from all zrLLM-enhanced models. From Table 4 (-RHL), we find that all the considered TKGF models can benefit from RHL, especially CENET. (3) We switch T5-11B to T5

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c c} \hline \multicolumn{16}{c}{**Dataset**} & \multicolumn{9}{c}{**ICKSV2-27**} \\ \cline{2-16}  & \multicolumn{3}{c}{Zero-Shot Relations} & \multicolumn{3}{c}{Seen Relations} & Overall & \multicolumn{3}{c}{Zero-Shot Relations} & \multicolumn{3}{c}{Seen Relations} & \multicolumn{3}{c}{Credit} & \multicolumn{3}{c}{Zone-Shot Relations} & \multicolumn{3}{c}{Sceen Relations} & Overall \\ \multirow{-2}{*}{Model} & 388k & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 & Host0 & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 & Host1 & Host0 \\ \hline CyGNet & 0.47 & 0.39 & **0.71** & 0.65 & 0.03 & 0.71 & 0.17 & 0.04 & 0.20 & 0.24 & 0.16 & 0.20 & 0.22 & 0.21 & 0.09 & 0.59 & 0.18 & 0.50 & 0.31 \\ CyGNet & **0.53** & **0.43** & **0.47** & **0.57** & **0.54** & **0.49** & **0.72** & **0.21** & **0.43** & **0.43** & **0.26** & **0.16** & **0.47** & **0.26** & **0.14** & **0.47** & **0.38** & **0.47** & **0.38** & **0.20** & **0.56** & **0.34** \\ FANCOT & 0.65 & 0.01 & 0.04 & 0.74 & 0.01 & 0.00 & 0.61 & 0.07 & 0.01 & 0.01 & 0.32 & **0.59** & 0.19 & **0.60** & **0.27** & 0.00 & 0.01 & **0.01** & **0.03** & 0.20 & 0.57 & 0.35 \\ Taxonomy+, **0.52** & **0.39** & **0.54** & **0.77** & **0.72** & **0.69** & **0.50** & **0.26** & **0.12** & **0.13** & **0.39** & 0.20 & 0.20 & 0.16 & 0.46 & **0.29** & **0.39** & **0.18** & **0.77** & **0.33** & **0.21** & **0.58** & **0.58** & **0.362** \\ TanNOO & 0.61 & 0.03 & 0.09 & **0.47** & **0.77** & **0.01** & **0.67** & 0.07 & 0.01 & 0.02 & 0.05 & 0.26 & 0.26 & **0.17** & 0.49 & 0.39 & 0.24 & 0.01 & 0.01 & 0.02 & 0.03 & **0.38** & **0.22** & 0.50 & 0.37 \\ TAMNOO & **0.61** & **0.34** & **0.78** & **0.78** & 0.78 & 0.08 & 0.07 & 0.72 & 0.02 & 0.02 & 0.02 & 0.26 & 0.17 & 0.49 & 0.45 & 0.24 & 0.18 & 0.08 & 0.38 & 0.22 & 0.59 & 0.38 \\ SECON & 0.41 & 0.32 & 0.71 & 0.76 & **0.63** & 0.63 & 0.63 & 0.20 & 0.19 & 0.17 & 0.27 & 0.15 & 0.46 & 0.26 & 0.26 & 0.16 & 0.16 & 0.16 & 0.16 & 0.20 & 0.57 & 0.35 \\ FG-CCN & **0.52** & **0.59** & **0.59** & **0.78** & **0.74** & **0.50** & **0.63** & **0.76** & **0.24** & **0.17** & **0.04** & **0.20** & **0.20** & **0.18** & **0.66** & **0.29** & **0.34** & **0.19** & **0.35** & **0.37** & **0.24** & **0.57** & **0.36** \\ TRGD & 0.73 & 0.78 & 0.74 & **0.57** & **0.63** & **0.58** & **0.78** & **0.19** & 0.19 & 0.01 & 0.03 & 0.75 & 0.15 & 0.47 & 0.27 & 0.23 & 0.19 & 0.77 & 0.32 & 0.19 & 0.77 & 0.32 & 0.22 & 0.57 & 0.55 \\ TEGCs & **0.54** & **0.46** & **0.76** & **0.74** & **0.67** & 0.67 & 0.65 & **0.77** & **0.21** & **0.13** & **0.10** & **0.30** & **0.40** & **0.27** & **0.18** & **0.35** & **0.64** & **0.27** & **0.33** & **0.20** & **0.20** & **0.56** & **0.39** & **0.57** & **0.36** \\ NPTIA & **0.49** & **0.30** & **0.70** & **0.72** & **0.71** & **0.54** & **0.75** & **0.55** & **0.77** & **0.21** & **0.30** & **0.40** & **0.27** & **0.18** & **0.33** & **0.20** & **0.20** & **0.56** & **0.39** & **0.20** & **0.57** & **0.35** \\ NPTIA & **0.57** & **0.49** & **0.30** & **0.78** & **0.73** & **0.93** & **0.74** & **0.54** & & & & & & & & & & & & & & & & & & \\ 

3B to see the impact of LM size on zrLLM. We observe from Table 4 that decreasing the size of T5 harms model performance. This proves that using larger scale LMs can provide semantic information of higher quality, and can be more beneficial to downstream TKGF (whether zero-shot or not).

#### Compare with Previous LM-Enhanced Model.

We benchmark two recent LM-enhanced TKGF models PPT Xu et al. (2023a) and ICL + GPTNeoX-20B Lee et al. (2023); Black et al. (2022) (Table 5). PPT finetunes BERT for TKGF. We find that although PPT achieves strong zero-shot results, it is beaten by several zrLLM-enhanced models. This proves that aligning language space to TKGF is helpful for zero-shot relational learning and LMs with larger size can be more contributive. ICL shows inferior results. This proves that without finetuning or alignment, LLMs are unable to optimally solve TKGF. zrLLM not only benefits from a large LM but also enables efficient alignment from language to TKG embedding space, which leads to superior performance.

Case Study of RHLWe do a case study to show: (1) RHL's HPN is able to capture ground truth relation history (GTH). (2) By capturing temporal relation patterns, RHL helps for better zero-shot TKGF. We ask zrLLM-enhanced CENET to predict the missing object of the test query \(q=(s_{q},r_{q},?,t_{q})=\text{(United States, Reduce or stop military assistance, ?, 2021-07-03) (answer is \(o_{q}=\textit{African Union}\)) taken from ICEWS21-zero. The GTH of \(s_{q}\) and \(o_{q}\) (Fig. 2(a), left) shows a pattern indicating their recent worsening relationship. It can serve as a clue in LP over \(q\) because it can be viewed as a "cause" to the query relation \(r_{q}\) which also implies a negative relationship. In other words, the entities with a worsening historical relationship are more likely to be connected with a relation showing their bad relationship currently. Since RHL uses HPN to infer GTH during test, we wish to study whether HPN can achieve reasonable inference to support LP. Based on GTH, we first change all three relations on 2021-06-17 to randomly sampled positive relations seen in the training data and form a changed history 1 (CH1, Fig. 2(a), middle). Then we further modify the relations on 2021-06-24 in the same way and form a changed history 2 (CH2, Fig. 2(a), right). We use Eq. 4 to encode GTH, CH1, CH2, and visualize them together with the predicted history (PRH) computed with HPN

Figure 3: (a) Ground truth and changed relation histories between _United States_ and _African Union_. Changed relations are marked in red. Only the histories nearest to 2021-07-03 are shown. (b) t-SNE of encoded GTH, CH1, CH2 (computed with Eq. 4), and predicted history PRH. Numbers beside dashed lines denote point distances (L2 norm). (c) Ground truth relation histories between _United States_ and _Afghanistan_.

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Datasets** & \multicolumn{2}{c}{**ACLED-zero**} & \multicolumn{2}{c}{**ICEWESI21-zero**} & \multicolumn{2}{c}{**ICEWSS22-zero**} \\  & MRR & & \multicolumn{2}{c}{MRR} & \multicolumn{2}{c}{MRR} & \multicolumn{2}{c}{MRR} \\
* & Zero & Seen & Overall & Zeno & Seen & Overall & Zeno & Seen & Overall \\ \hline CyCreate & **85.3*
* & 0.751 & **0.723*
* & **0.920*
* & **0.820*
* & **0.254*
* & **0.286*
* & **0.315*
* & **0.341*
* \\ -EDRD & 0.502 & 0.748 & 0.716 & 0.198 & 0.252 & 0.251 & 0.250 & 0.314 & 0.311 \\ -RHL & 0.503 & **0.752*
* & 0.720 & 0.199 & 0.256 & 0.256 & 0.258 & 0.257 & 0.296 \\ T5-3B & 0.517 & **0.752*
* & 0.711 & 0.240 & 0.202 & 0.252 & **0.315*
* & 0.313 \\ \hline FANCOT-0 & 0.525 & **0.775*
* & **0.743*
* & **0.216*
* & **0.380*
* & **0.279*
* & **0.326*
* & **0.363*
* & **0.362*
* \\ EDD & 0.533 & 0.727 & 0.741 & 0.241 & **0.200*
* & **0.279*
* & 0.320 & 0.362 & 0.360 \\ -RHL & 0.506 & 0.755 & 0.740 & 0.213 & 0.277 & 0.276 & 0.309 & **0.363*
* & 0.361 \\ -YS & **0.544*
* & 0.771 & 0.420 & 0.274 & 0.273 & 0.323 & 0.359 & 0.358 \\ \hline Taxonomy-CD & **0.49*
* & **0.760*
* & **0.725*
* & **0.212*
* & **0.263*
* & **0.247*
* & **0.311*
* & **0.350*
* & **0.348*
* \\ -ERD & **0.491*
* & 0.702 & 0.675 & 0.205 & 0.267 & 0.266 & 0.258 & 0.328 & 0.326 \\ Bull & 0.032 & 0.725 & 0.693 & 0.724 & 0.224 & 0.292 & 0.324 & 0.324 & 0.321 \\ TS-3B & 0.490 & 0.701 & 0.674 & 0.234 & 0.223 & 0.222 & 0.308 & 0.284 & 0.285 \\ \hline RECE-CCB & **0.529*
* & **0.731*
* & **0.705*
* & **0.214*
* & **0.280*
* & **0.279*
* & **0.324*
* & **0.357*
* & **0.366*
* \\ RD & 0.482 & 0.730 & 0.730 & 0.699 & 0.211 & 0.277 & 0.266 & 0.354 & 0.352 \\ -RRIL & 0.519 & 0.726 & 0.699 & 0.213 & 0.277 & 0.276 & 0.317 & 0.350 & 0.349 \\ T5-3B & 0.504 & 0.721 & 0.693 & 0.021 & 0.259 & 0.283 & 0.301 & 0.354 & 0.352 \\ \hline TRGN+ & **0.548*
* & **0.754*
* & **0.727*
* & **0.221*
* & **0.279*
* & **0.278*
* & **0.333*
* & **0.353*
* & **0.352*
* \\ -ERD & 0.480 & 0.747 & 0.713 & 0.211 & 0.275 & 0.274 & 0.282 & **0.353*
* & 0.350 \\ -RHL & 0.515 & 0.752 & 0.721 & 0.251 & 0.275 & 0.276 & 0.320 & 0.350 & 0.349 \\ TSS-3B & 0.498 & 0.749 & 0.717 & 0.208 & 0.271 & 0.270 & 0.325 & 0.345 & 0.344 \\ \hline LETTA & **0.557*
* & **0.783*
* & **0.754*
* & & & & **0.3313*
* & **0.358*
* & **0.357*
* \\ -BD & 0.517 & 0.777 & 0.744 &
* & & & & 0.3203 & 0.354 & 0.352 \\ -RHL & 0.529 & 0.782 & 0.749 & & & & & 0.3138 & 0.357 & 0.355 \\ T5-3B & 0.512 & 0.776 & 0.742 & & & & & 0.3303 & 0.353 & 0.352 \\ \hline CINETF & **0.691*
* & **0.779*
* & **0.756*
* & **0.335*
* & **0.396*
* & **0.396*
* & **0.564*
* & **0.571*
* & **0.570*
* \\ EBD & 0.535 & 0.777 & 0.101 & 0.321 & 0.374 & 0.373 & 0.352 & 0.570 & 0.564 \\ RHL & 0.445 & 0.754 & 0.714 & 0.212 & 0.230 & 0.289 & 0.295 & 0.370 & 0.367 \\ T5-3B & 0.568 & 0.736 & 0.714 &
 

by using t-SNE (van der Maaten and Hinton, 2008) in Fig. 3b. We find that PRH is the closest to GTH and CH1 is closer than CH2 to GTH. The reason why CH2 is much farther from GTH is that CH2 changes more negative relations to positive, greatly changing the semantic meaning stored in GTH. CH1 only introduces changes on 2021-06-17, making it less deviated from GTH. HPN takes the \(r_{q}\) and can keep PRH close to GTH, making zrLLM able to maximally capture the temporal patterns indicated by GTH, while preventing the scalability problem incurred by searching relation histories of all candidate entities. By using RHL, the zrLLM-enhanced CENET can correctly predict \(o_{q}\), while the model without RHL takes \(o^{\prime}=Afghanistan\) as the predicted answer. We present the nearest GTH between \(s_{q}\) and \(o^{\prime}\) in Fig. 3c and find that it indicates a positive relationship which is unlikely to cause \(r_{q}\) right after. During training, RHL learns patterns and matches entity pairs with them (Eq. 8). This enables RHL to exclude the entities that do not fit into the learned patterns from the answer set and make more accurate predictions.

## 5 Conclusion

We study zero-shot relational learning in TKGF and design an LLM-empowered approach, i.e., zrLLM. zrLLM extracts the semantic information of KG relations from LLMs and introduces it into TKG representation learning. It also uses an RHL module to capture the temporal relation patterns for better reasoning, and meanwhile promote the embedding space alignment between text and TKGs. We couple zrLLM with several embedding-based TKGF models and find that zrLLM provides huge help in forecasting the facts with zero-shot relations, and moreover, it maintains models' performance over seen relations.

## 6 Limitations

Our limitations can be summarized as follows. First, zrLLM is developed only for enhancing embedding-based TKG forecasting methods. It is not directly applicable to the rule-based methods, e.g., TLogic. Besides, relation history learner inevitably increases model's training and evaluation time since relation patterns are learned with GRUs where recurrent computations are performed along the time axis. More GPU memory is also required for storing relation histories. This hinders the efficiency of zrLLM-enhanced models compared with the original baselines. In the future, we will explore how to generalize our proposed method to rule-based models and try to improve model efficiency. We will also try to experiment zrLLM on more TKG forecasting methods and study whether we can benefit more of them.

## Acknowledgments

This work has been partially funded by the Munich Center for Machine Learning and supported by the Federal Ministry of Education and Research and the State of Bavaria. This work has also been supported by the German Federal Ministry for Economic Affairs and Climate Action (BMWK) as part of the project CoyPu under grant number 01MK21007K. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Bo Xiong. Bo Xiong has also been partially funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2075 - 390740016, the Stuttgart Center for Simulation Science (SimTech), and the Bundesministerium fur Wirtschaft und Energie (BMWi), grant aggrement No. 01MK20008F.

## References

* Balazevic et al. (2019) Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2019. Tucker: Tensor factorization for knowledge graph completion. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 5184-5193. Association for Computational Linguistics.
* Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-neox-20b: An open-source autoregressive language model. _CoRR_, abs/2204.06745.
* Boschee et al. (2015) Elizabeth Boschee, Jennifer Lautenschlager, Sean O'Brien, Steve Shellman, James Starz, and Michael Ward. 2015. ICEWS Coded Event Data.
* Brown et al. (2018) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, 

Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
* Chen et al. (2023a) Zhongwu Chen, Chengjin Xu, Fenglong Su, Zhen Huang, and Yong Dou. 2023a. Incorporating structured sentences with time-enhanced BERT for fully-inductive temporal relation prediction. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pp. 889-899. Cited by: SS5.
* 4 May 2023, pp. 2433-2443. External Links: Link, Document Cited by: SS5.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1724-1734. External Links: Link, Document Cited by: SS5.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-4186. External Links: Link, Document Cited by: SS5.
* Z. Ding, B. He, J. Wu, Y. Ma, Z. Han, and V. Tresp (2023)Learning meta-representations of one-shot relations for temporal knowledge graph link prediction. In International Joint Conference on Neural Networks, IJCNN 2023, Gold Coast, Australia, June 18-23, 2023, pp. 1-10. External Links: Link, Document Cited by: SS5.
* Z. Ding, Z. Li, R. Qi, J. Wu, B. He, Y. Ma, Z. H. Ma, Z. Meng, S. Chen, R. Liao, Z. Han, and V. Tresp (2023b)Forecasttkgquestions: a benchmark for temporal question answering and forecasting over temporal knowledge graphs. In ISWC, Vol. 14265 of Lecture Notes in Computer Science, Vol. 541-560, Springer, pp. 541-560. External Links: ISBN 978-3- 

Julien Leblay and Melisachew Wudage Chekol. 2018: Deriving validity time in knowledge graph. In _Companion of the The Web Conference 2018 on The Web Conference 2018, WWW 2018, Lyon , France, April 23-27, 2018_, pages 1771-1776. ACM.
* Lee et al. (2023) Dong-Ho Lee, Kian Ahrabian, Woojeong Jin, Fred Morstatter, and Jay Pujara. 2023. Temporal knowledge graph forecasting without knowledge using context learning. _CoRR_, abs/2305.10613.
* Li et al. (2022) Yujia Li, Shiliang Sun, and Jing Zhao. 2022. Tirgn: Time-guided recurrent graph network with local-global historical patterns for temporal knowledge graph reasoning. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022_, pages 2152-2158. ijcai.org.
* Li et al. (2021a) Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhuo Wang, and Xueqi Cheng. 2021a. Search from history and reason for future: Two-stage reasoning on temporal knowledge graphs. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 4732-4743. Association for Computational Linguistics.
* Li et al. (2021b) Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. 2021b. Temporal knowledge graph reasoning based on evolutional representation learning. In _SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_, pages 408-417. ACM.
* Liao et al. (2023) Ruotong Liao, Xu Jia, Yunpu Ma, and Volker Tresp. 2023. GenTKG: Generative forecasting on temporal knowledge graph. In _Temporal Graph Learning Workshop @ NeurIPS 2023_.
* Liu et al. (2023) Kangzheng Liu, Feng Zhao, Guandong Xu, Xianzhi Wang, and Hai Jin. 2023. RETIA: relation-entity twin-interact aggregation for temporal knowledge graph extrapolation. In _39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, CA, USA, April 3-7, 2023_, pages 1761-1774. IEEE.
* March 1, 2022_, pages 4120-4127. AAAI Press.
* Ma et al. (2023) Ruizin Ma, Biao Mei, Yunlong Ma, Hongyan Zhang, Meihong Liu, and Liang Zhao. 2023. One-shot relational learning for extrapolation reasoning on temporal knowledge graphs. _Data Min. Knowl. Discov._, 37(4):1591-1608.
* Mei et al. (2022) Xin Mei, Libin Yang, Xiaoyan Cai, and Zuowei Jiang. 2022. An adaptive logical rule embedding model for inductive reasoning over temporal knowledge graphs. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 7304-7316. Association for Computational Linguistics.
* Mirtaheri et al. (2021) Mehrnoods Mirtaheri, Mohammad Rostami, Xiang Ren, Fred Morstatter, and Aram Galstyn. 2021. One-shot learning for temporal knowledge graphs. In _3rd Conference on Automated Knowledge Base Construction, AKBC 2021, Virtual, October 4-8, 2021_.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykthan Tejani, Sasank Chilamkurthy, Benoi Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 8024-8035.
* Petroni et al. (2019) Fabio Petroni, Tim Rocktaschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 2463-2473. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Raleigh et al. (2010) Clionadh Raleigh, rew Linke, Havard Hegre, and Joakim Karlsen. 2010. Introducing acled: An armed conflict location and event dataset. _Journal of Peace Research_, 47(5):651-660.
* Saxena et al. (2021) Apooryx Saxena, Soumen Chakrabarti, and Partha P. Talukdar. 2021. Question answering over temporal knowledge graphs. In _ACL/IJCNLP (1)_, pages 6663-6676. Association for Computational Linguistics.
* Sun et al. (2021) Haohai Sun, Jialu Zhong, Yunpu Ma, Zhen Han, and Kun He. 2021. TimeTraveler: Reinforcement learning for temporal knowledge graph forecasting. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 

8306-8319, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of Machine Learning Research_, 9(86):2579-2605.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Wang et al. (2022) Ruijie Wang, Zheng Li, Dachun Sun, Shengzhong Liu, Jinning Li, Bing Yin, and Tarek F. Abdelzaher. 2022. Learning to sample and aggregate: Few-shot reasoning over temporal knowledge graphs. In _NeurIPS_.
* Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. _Trans. Mach. Learn. Res._, 2022.
* Xu et al. (2023a) Wenjie Xu, Ben Liu, Miao Peng, Xu Jia, and Min Peng. 2023a. Pre-trained language model with prompts for temporal knowledge graph completion. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 7790-7803. Association for Computational Linguistics.
* Xu et al. (2023b) Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2023b. Temporal knowledge graph reasoning with historical contrastive learning. In _Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023_, pages 4765-4773. AAAI Press.
* Yang et al. (2015) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Zhang et al. (2019) Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. Quaternion knowledge graph embeddings. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 2731-2741.
* Zhu et al. (2021) Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan Cheng, and Yan Zhang. 2021. Learning from history: Modeling temporal knowledge graphs with sequential copy-generation networks. In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_, pages 4732-4740. AAAI Press.

## Appendix A Detailed Illustration of Prompt for GPT-3.5

We give a detailed illustration of our prompt for producing ERDs with GPT-3.5 in Fig. 4. For every batch of \(n\) relations, we incorporate their dataset-provided texts into our prompt to generate their enriched descriptions.

## Appendix B Further Details of Zero-Shot Datasets

For each dataset, we provide the distribution of all zero-shot relations' frequncies in Fig. 5. We take the relations with lowest frequencies as zero-shot relations when we construct datasets, following previous few-shot relational TKG learning frameworks, e.g., OAT Mirtaheri et al. (2021) and MOST Ding et al. (2023a). The proportion of zero-shot relations for each dataset is high. 14 out of 23; 123 out of 253; 155 out of 248 relations in ACLED-zero; ICEWS21-zero; ICEWS22-zero are zero-shot relations. This ensures the diversity of relation types in test sets.

## Appendix C Implementation Details

All experiments are implemented with PyTorch Paszke et al. (2019) on a server equipped with an AMD EPYC 7513 32-Core Processor and a single NVIDIA A40 with 48GB memory. All the experimental results are the average of three runs with different random seeds.

### Baseline Implementation Details

Our baselines are all based on neural networks rather than pure score function-based (e.g., TTransE Leblay and Chekol (2018)). This is because the most popular and recent TKGF methods all leverage neural networks to gain the forecasting ability and it is hard for pure score function-based methods to achieve that solely with geometric embeddings. The implementation details of each TKGF baseline is as follows 

* **CyGNet.** We use the official code of CyGNet4. We search hyperparameters of baseline CyGNet following Table6. The best hyperparameters are marked as bold. For each dataset, we do 4 trials to try different hyperparameter settings. We run 5 epochs for each trail and take the one with the best validation result as the best hyperparameter setting. Footnote 3: [https://github.com/CunchaoZ/CyGNet](https://github.com/CunchaoZ/CyGNet)

* **TANGO-TuckER/Distmult.** We use the official code of TANGO5. We search hyperparameters of baseline TANGO-TuckER/Distmult following Table7. The best hyperparameters are marked as bold. For each dataset, we do 6 (TANGO-TuckER) and 9 (TANGO-Distmult) trials to try different hyperparameter settings. We run 10 epochs for each trail and take the one with the best validation result as the best validation result as the best hyperparameter setting. Footnote 5: [https://github.com/TemporalKGTear/TANGO](https://github.com/TemporalKGTear/TANGO)

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Dataset** & **ACLED-zero** & **ICEW821-zero** & **ICEWS22-zero** \\
**Hyperparameter** & CyGNet & CyGNet & CyGNet \\ \hline Embedding Size & \{100, **200**\} & \{100, **200**\} & \{100, **200**\} \\ Alpha [Alg. go in (Liu et al., 2021)] & \{**02.0**\} & \{**02.0**\} & \{**02.0**\} \\ \hline \hline \end{tabular}
\end{table}
Table 6: CyGNet hyperparameter searching strategy.

Figure 4: Prompting GPT-3.5 for ERDs. The green texts are the short relation texts provided in the original datasets. The orange texts are the generated relation explanations from GPT-3.5.

Figure 5: Zero-shot Relation frequency on all zero-shot TKGF datasets. Horizontal axis denotes the appearance times, i.e., frequency. Vertical axis denotes the number of relations.

 

TiRGN following Table 9. The best hyperparameters are marked as bold. For each dataset, we do 12 trials to try different hyperparameter settings. We run 10 epochs for each trail and take the one with the best validation result as the best hyperparameter setting.
* **RETIA.** We use the official code of RETIA8. We search hyperparameters of baseline RETIA following Table 10. The best hyperparameters are marked as bold. For each dataset, we do 4 trials to try different hyperparameter settings. We run 10 epochs for each trail and take the one with the best hybridization result as the best hyperparameter setting. Footnote 8: [https://github.com/CGCL-codes/RETIA](https://github.com/CGCL-codes/RETIA)

The hyperparameters not discussed above follow the settings reported in the original papers.

### zrLLM Implementation Details

We fix the hyperparameters searched from the baselines and additionally search zrLLM-specific hyperparameters for zrLLM-enhanced models. The hyperparameter searching strategy and the best hyperparameter settings regarding the zrLLM-enhanced baselines are reported in Table 12. Note that \(\gamma\) can be either a learnable parameter or a fixed scalar. When \(\gamma\) is not fixed, \(\gamma\) Value means the initialized parameter value during training. For each zrLLM-enhanced model, in each dataset, we do 24 trials to try different hyperparameter settings. We run 7 epochs for each trail and take the one with the best validation result as the best hyperparameter setting.

### Implementation Details of PPT and Icl

We use the official code of PPT10 and ICL11. For PPT, we use the default hyperparameter setting used for ICEWS14 when we implement it on all our new datasets. Since PPT only explores object entity prediction in its original implementation, we add the subject entity prediction part and report the overall result. We achieve subject prediction by first deriving the inverse relation texts for each relation in each TKG dataset, e.g., use _Inversed Reduce or stop military assistance_ to represent the inverse relation of the relation _Reduce or stop military assistance_, and then turning each subject prediction query \((?,r_{q},o_{q},t_{q})\) to an object prediction query \((o_{q},r^{-1}_{q},?,t_{q})\), where \(r^{-1}_{q}\) stands for the inverse relation of \(r_{q}\). For ICL, we use the lexical-based prompt because we are dealing with zero-shot relations where text information is important. We also employ the unidirectional entity-focused history, which achieves best results on ICEWS14 as reported in ICL's original paper. We use the default history length of 20 for all datasets.

Footnote 11: [https://github.com/usc-isi-i2/isi-kkg-icl](https://github.com/usc-isi-i2/isi-kkg-icl)

### Computational Resource Usage

We report the computational resources for all zrLLM-enhanced models and PPT in Table 13. Training time denotes the period of time a model requires to reach its best validation performance. PPT requires extremely long time for sampling and thus has high time consumption. Note that zrLLM loads T5 to generate LM-based relation representations. This process takes a substantial amount of GPU memory. However, in our work, we store the output of T5's encoder as saved parameters and use them in downstream zero-shot TKGF with any

\begin{table}
\begin{tabular}{p{65.4pt} p{34.1pt} p{34.1pt} p{34.1pt}} \hline \hline
**Dataset** & **ACLED-arre** & **ICKWS21-zero** & **ICKWS22-error** \\
**Hyperparameter** & RETA & RETA & RETA \\ \hline Embedding Size & \((100,200)\) & \((100,200)\) & \((100,200)\) \\ History Length & \((3,9)\) & \((3,9)\) & \((3,9)\) \\ \hline \end{tabular}
\end{table}
Table 10: RETIA hyperparameter searching strategy.

\begin{table}
\begin{tabular}{p{65.4pt} p{34.1pt} p{34.1pt} p{34.1pt}} \hline \hline
**Dataset** & **ACLED-arre** & **ICKWS21-zero** & **ICKWS22-error** \\
**Hyperparameter** & CENT & CENT & CENT \\ \hline Embedding Size & \([100,200]\) & \((100,200)\) & \((100,200)\) \\ History Length & \((3,9)\) & \((3,9)\) & \((3,9)\) \\ \hline \hline \end{tabular}
\end{table}
Table 8: RE-GCN hyperparameter searching strategy.

\begin{table}
\begin{tabular}{p{65.4pt} p{34.1pt} p{34.1pt} p{34.1pt}} \hline \hline
**Dataset** & **ACLED-arre** & **ICKWS21-zero** & **ICKWS22-error** \\
**Hyperparameter** & CBETA & CBETA & CBETA \\ \hline Embedding Size & \([100,200]\) & \((100,200)\) & \((100,200)\) \\ History Length & \((3,9)\) & \((3,9)\) & \((3,9)\) \\ \hline \hline \end{tabular}
\end{table}
Table 11: CENET hyperparameter searching strategy.

\begin{table}
\begin{tabular}{p{65.4pt} p{34.1pt} p{34.1pt} p{34.1pt}} \hline \hline
**Dataset** & **ACLED-arre** & **ICKWS21-zero** & **ICKWS21-zero** \\
**Hyperparameter** & CBETA & CBETA & CBETA \\ \hline Embedding Size & \([100,200]\) & \((100,200)\) & \((100,200)\) \\ History Length & \((3,9)\) & \((3,9)\) & \((3,9)\) \\ \hline \hline \end{tabular}
\end{table}
Table 9: TiRGN hyperparameter searching strategy.

 

zrLLM-enhanced model. This prevents from high memory demand during model training and evaluation. We use Fig. 6 to illustrate the direct comparison among zrLLM-enhanced models and PPT regarding their required computational resources during training.

ICL loads GPT-NeoX-20B that requires huge memory consumption. We use two NVIDIA A40 for all its experiments. Since ICL does not require training, we only report its validation and test time here. For ACLED-zero, GPU memory usage is 90,846 MB. Validation time is 0.63 h and test time is 0.12 h. For ICEWS21-zero, GPU memory usage is 90,868 MB. Validation time is 35.48 h and test time is 0.82 h. For ICEWS22-zero, GPU memory usage is 91,458 MB. Validation time is 22.98 h and test time is 1.15 h.

### Zero-Shot Evaluation Setting Explanation

To keep zero-shot relations "always unseen" during the whole evaluation process, we constrain all models to do LP only based on the training set. Among all TKGF models, TANGO, RE-GCN, TiRGN and RETIA use recurrent neural structures to model historical TKG information from a short sequence of timestamps prior to the prediction timestamp. We constrain them to only use the latest training data, i.e., from \(t_{\text{train\_max}}-k\) to \(t_{\text{train\_max}}\), to encode historical information during evaluation. \(k\) is the considered history length and \(t_{\text{train\_max}}=\text{max}(\mathcal{T}_{\text{train}})\) is the maximum timestamp in the training data. For CyGNet and CENET, they have originally met our restriction of not observing any ground truth evaluation data during evaluation, and thus can be directly implemented in our zero-shot setting. Another point worth noting is that RHL requires ground truth relation history. We restrict zrLLM to only capture the relation history across the whole training time period to prevent from exposing zero-shot relations during evaluation.

## Appendix D Algorithm

We provide algorithms to show the whole process of using zrLLM to enhance TKGF models. First, zrLLM generates LLM-based relation representations by using GPT-3.5 and T5-11B (Algorithm 1). Then we train zrLLM jointly with TKGF baseline models (Algorithm 2). The trained models are then used for evaluation (Algorithm 3).

``` Input: Relations \(\mathcal{R}\), relation text of all relations provided by the TKGF dataset Text 

(the definition is similar for \({\cal G}_{\text{valid}}\)). Hits@1/3/10 denote the proportions of the predicted links where ground truth missing entities are ranked as top 1, top3, top10, respectively. As explored and suggested in Gastinger et al. (2023), we also use the time-aware filtering setting proposed in Han et al. (2021a) for fairer evaluation.

## Appendix F Complete Comparative Study Results

We report the complete results of comparative study in Table 14 and 15.

## Appendix G Complete Ablation Study Results

We report the complete ablation study results in Table 16.

## Appendix H Complete Results of Previous LM-Enhanced TKGF Model

We report the complete results of previous LM-enhanced TKGF models in Table 14 and 15.

 



## Appendix J Failure Case Discussion

From Table 4, we observe several failure cases when the complete zrLLM is implemented, e.g., (1) TANGO-T+ without ERDs show a slightly better zero-shot result on ACLED-zero compared with the complete TANGO-T+; (2) TANGO-T+ does not witness an improvement over the seen relations on ICEWS21-zero compared with TANGOT+ without RHL. We attribute such failure cases to the characteristics of the considered TKGF models. As highlighted in Sec. 4.2, our goal is to use zrLLM to enhance TKGF model performance over zero-shot relations while maintaining strong performance over seen relations. By carefully comparing the overall performance of zrLLM-enhanced models with their ablated variants, e.g., -ERD, we find that the complete version of zrLLM with ERDs, RHL and T5-11B can always achieve the best overall performance, which aligns to our motivation. The small number of failure cases caused by several baseline TKGF methods cannot overturn the merit brought by the modules of zrLLM.

## Appendix K Related Work Details

Traditional TKG Forecasting Methods.As discussed in Sec. 1, traditional TKGF methods are trained to forecast the facts containing the KG relations (and entities) seen in the training data, regardless of the case where zero-shot relations (or entities) appear as new knowledge arrives12. These methods can be categorized into two types: embedding-based and rule-based. Embedding-based methods learn hidden representations of KG relations and entities (some also learn time representations), and perform link forecasting by inputting learned representations into a score function for computing scores of fact quadruples. Most existing embedding-based methods, e.g., Jin et al. (2020); Han et al. (2021); Li et al. (2021); Liu et al. (2023), learn evolutional entity and relation representations by jointly employing graph neural networks Kipf and Welling (2017) and recurrent neural structures, e.g., GRU Cho et al. (2014). Historical TKG information are recurrently encoded by the models to produce the temporal sequence-aware evolutional representations for future prediction. Some other approaches Han et al. (2021); Sun et al. (2021); Li et al. (2021) start from each LP query and traverse the temporal history in a TKG to search for the prediction answer. Apart from them, CyGNet Zhu et al. (2021) achieves forecasting purely based on the appearance of historical facts.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c} \hline \multirow{2}{*}{**Datasets**} & \multicolumn{6}{c}{**ICEWS21-zero**} \\  & \multicolumn{3}{c}{Zero-Shot Relations} & \multicolumn{3}{c}{Seen Relations} & Overall & \multicolumn{3}{c}{Zero-Shot Relations} & \multicolumn{3}{c}{Seen Relations} & Overall \\ \cline{2-19}
**Model** & MRR & H@1 & H@3 & H@1 & H@3 & H@1 & H@3 & H@1 & H@3 & H@1 & H@3 & H@1 & H@3 & H@1 & H@3 & H@1 & H@1 & H@3 & H@1 & H@3 & H@1 & H@1 & H@3 & H@1 & H@1 & H@3 & H@1 & H@1 & H@3 &

Another recent work CENET Xu et al. (2023b) trains contrastive representations of LP queries to identify highly correlated entities in either historical or non-historical facts. Compared with the rapid advancement in developing embedding-based TKGF methods, rule-based TKGF has still not been extensively explored. One popular rule-based TKGF method is TLogic Liu et al. (2022). It extracts temporal logic rules from TKGs and uses a symbolic reasoning module for LP. Based on it, ALRE-IR Mei et al. (2022) proposes an adaptive logical rule embedding model to encode temporal logical rules into rule representations. This makes ALRE-IR both a rule-based and an embedding-based method. Experiments in TLogic and ALRE-IR have proven that rule-based TKGF methods have strong ability in reasoning over zero-shot unseen entities connected by the seen relations, however, they are not able to handle unseen relations since the learned rules are strongly bounded by the observed relations. In our work, we implement zrLLM on embedding-based TKGF models because (1) embedding-based methods are much more popular; (2) zrLLM utilizes LLM to generate relation representations, which is more compatible with embedding-based methods.

Inductive Learning on TKGs.Inductive learning on TKGs has gained increasing interest. It refers to developing models that can handle the relations and entities unseen in the training data. TKG inductive learning methods can be categorized into two types. The first type of works focuses on reasoning over unseen entities Ding et al. (2022); Wang et al. (2022); Ding et al. (2023c); Chen et al. (2023a), while the second type of methods aims to deal with the unseen relations Mirtaheri et al. (2021); Ding et al. (2023a); Ma et al. (2023). Most of inductive learning methods are based on few-shot learning (e.g., FILT Ding et al. (2022), MetaTKGR Zhang et al. (2019), FITCARL Ding et al. (2023c), OAT Mirtaheri et al. (2021), MOST Ding et al. (2023a) and OSLT Ma et al. (2023)). They first compute inductive representations of newly-emerged entities or relations based on \(K\)-associated facts (\(K\) is a small number, e.g., 1 or 3) observed during inference, and then use them to predict the facts regarding few-shot elements. One limitation of these works is that the inductive representations cannot be learned without the \(K\)-shot examples, making them hard to solve the zero-shot problems. Different from few-shot learning methods, SST-BERT Chen et al. (2023a) pre-trains a time-enhanced BERT Devlin et al. (2019) for TKG reasoning. It achieves inductive learning over unseen entities but has not shown its ability in reasoning zero-shot relations. Another recent work MTKGE Chen et al. (2023b) is able to concurrently deal with both unseen entities and relations. However, it requires a support graph containing a substantial number of data examples related to the unseen entities and relations, which is far from the zero-shot problem that we focus on.

TKG Reasoning with Language Models.Recently, more and more works have introduced LMs into TKG reasoning. SST-BERT Chen et al. (2023a) generates a small-scale pre-training corpus

\begin{table}
\begin{tabular}{l l c c c c c c c c c c c c c c c c c c} \hline \multicolumn{14}{l}{**Dataset**} & \multicolumn{6}{c}{**XUB-250**} \\  & \multicolumn{3}{c}{Zero-Shot Relations} & \multicolumn{3}{c}{See Sections} & Overall & Zero Star-Reliates & Score & Recall & Accuracy & Recall & Recall & Recall & Recall & Recall & Recall & Recall & Recall & Recall & Recall \\ \cline{2-14}
**Model** & 859k 

based on the training TKGs and pre-trains an LM for encoding TKG facts. The encoded facts are then fed into a scoring module for LP. ECOLA (Han et al., 2023) aligns facts with additional fact-related texts and proposes a joint training framework that enhances TKG reasoning with BERT-encoded language representations. PPT Xu et al. (2023a) converts TKGF into the pre-trained LM masked token prediction task and finetunes a BERT for TKGF. It directly input TKG facts into the LM for answer prediction. Apart from them, one recent work Lee et al. (2023) explores the possibility of using incontext learning (ICL) Brown et al. (2020) with LLMs to make predictions about future facts without fintuning. Another recent work GenTKG Liao et al. (2023) finetunes an LLM, i.e., Llama2-7B Touvron et al. (2023), and let the LLM directly generate the LP answer in TKGF. It mines temporal logical rules and uses them to retrieve historical facts for prompt generation.

Although the above-mentioned works have shown success of LMs in TKG reasoning, they have limitations: (1) None of these works has studied whether LMs can be used to better reason the zero-shot relations. (2) By only using ICL, LLMs are beaten by traditional TKG reasoning methods in performance Lee et al. (2023). The performance can be greatly improved by finetuning LLMs (as in GenTKG Liao et al. (2023)), but finetuning LLMs requires huge computational resources. (3) Since LMs, e.g., BERT and Llama2, are pre-trained with a huge corpus originating from diverse information sources, it is inevitable that they have already seen the world knowledge before they are used to solve TKG reasoning tasks. Most popular TKGF benchmarks are extracted from the TKGs constructed before 2020, e.g., ICEWS14, ICEWS18 and ICEWS05-15 Jin et al. (2020). The facts inside are based on the world knowledge before 2019, which means LMs might have encountered them in their training corpus, posing a threat of information leak to the LM-driven TKG reasoning models. To this end, we (1) draw attention to studying the impact of LMs on zero-shot relational learning in TKGs; (2) make a compromise between performance and computational efficiency by not fintuning LMs or LLMs but adapting the LLM-provided semantic information to non-LM-based TKGF methods; (3) construct new benchmarks where the facts are all happening from 2021 to 2023, which avoids the possibility of information leak when we utilize T5-11B that was released