Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning
Authors: Dong-Ho Lee
Kian Ahrabian
Woojeong Jin
Fred Morstatter
Jay Pujara
Venue: EMNLP 2023 Main (2023)
Keywords: Temporal Knowledge Graph
In-context Learning
Large Language Model

Abstract: Temporal knowledge graph (TKG) forecasting benchmarks challenge models to predict future facts using knowledge of past facts.
In this paper, we develop an approach to use in-context learning (ICL) with large language models (LLMs) for TKG forecasting.
Our extensive evaluation compares diverse baselines, including both simple heuristics and state-of-the-art (SOTA) supervised models, against pre-trained LLMs across several popular benchmarks and experimental settings.
We observe that naive LLMs perform on par with SOTA models, which employ carefully designed architectures and supervised training for the forecasting task, falling within the (-3.6\%, +1.5\%) Hits@1 margin relative to the median performance.
To better understand the strengths of LLMs for forecasting, we explore different approaches for selecting historical facts, constructing prompts, controlling information propagation, and parsing outputs into a probability distribution.
A surprising finding from our experiments is that LLM performance endures ($\pm$0.4\% Hit@1) even when semantic information is removed by mapping entities/relations to arbitrary numbers, suggesting that prior semantic knowledge is unnecessary; rather, LLMs can leverage the symbolic patterns in the context to achieve such a strong performance. 
Our analysis also reveals that ICL enables LLMs to learn irregular patterns from the historical context, going beyond frequency and recency biases

Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning
Dong-Ho Lee; Kian Ahrabian', Woojeong Jin, Fred Morstatter, Jay Pujara
Department of Computer Science and Information Sciences Institute University of Southern California
(dongho. lee, ahrabian, woojeong. jiapoonnnnnjnneeee jin]euscc.eeeeee edu, (fredmors, jjpijarrs,jjjaaaaa jpujara)@isi. edu
Abstract
Which team will win the Super Bowl in 2023?
Temporal knowledge graph (TKG) forecasting benchmarks challenge models to predict future facts using knowledge of past facts. In this pa- per, develop approach to use in-context learning (ICL.) with large language models (LLMs) for TKG forecasting. Our extensive evaluation compares diverse baselines, includ- ing both simple heuristics and state-of-the-arr (SOTA) supervised models, against pre- pre-trained LLMs across several popular benchmarks and experimental settings. We observe that naive LLMs perform par with SOTA models, which employ carefully designed architectures and supervised training for the forecasting task, falling within the (-3.6% +1.5%) Hits@ mar- gin relative to the median performance. To bet- ter understand the strengths of LLMs for fore- casting, we explore different approaches for selecting historical facts, constructing prompts, controlling information propagation, and pars- ing outputs into probability distribution. A surprising finding from our experiments is that LLM performance endures (1.0.4% Hit@1 I) even when semantic information is removed by mapping entities/relations to arbitrary num- bers, suggesting that prior semantic knowledge is unnecessary; rather, LLMs can leverage the symbolic patterns in the context achieve such a strong performance. Our analysis also reveals that ICL enables LLMs to learn irregular pat- terss from the historical context, going beyond frequency and recency b
2000:
[Superbowl, Champion, St Louis]
2001:
[Superbowl, Champion, Bal Baltieere] 2002: [Superbowl, Champion, New England]
2003:
[Superbowl, Champion, Tampa Bay]
...
2019: [Supezbowl, Champion,
England]
2020: [Superbowl, Champion, Kansas City]
2021: [Superbowl, Champion, Tampa Bay]
2022: [Superbowl, Champion, Los Angeles]
2023:
[Superbowl, Champion,
Model
Params.
Prediction
G.T Kank
EleutberAl apt-j-Bb
6B
Los Angeles
EleutherAl apt-nocxx200
20B
Kansus
OpenAl text-ada-0e1
350M
Carolina
OpaAA text-bitlugr-000
1.38
Indiunago
>-
OpesAA text-curse-6a1
6.78
New England
>5
OpenAl toxt-dacinci-083
1758
TBD
25
OpmnA 3.5 Surbo
Sorry, cannot prodict future
Table 1: Model predictions for an example struc- tured prompt. Given a structured prompt (top), large language models forecast the missing fact using next token prediction (bottom). G.T Rank indicates the rank of the ground truth response ("Kansas Ciry") within the token probability distribution.
of knowledge (Leblay and Chekol, 2018; Gareía- Durán et al., 2018; Goel et al., 2020; Lacroix et al., 2020). Recent endeavors in the area of TKGs have been focused on predicting future missing links (i.e., forecasting), given a query quadruple Pj. ?,,?,ty) tr)) and a set of associated historical facts Eg = {(s,p, t) < [T} S & (Gastinger et al., 2022). An illustrative example of this task is the question "Which team will win the Super Bowl in 2023?" that can be expressed as q = (Su- per bowl, Champion, ?, 2023) and Eg = ((Super bowl, Champion, Los Angeles, 2022), (Super bowl, Champion, Tampa Bay, 2021), ...} (See Table 1). The ultimate objective is to identify the most suit- able entity among all the football teams & (e.g., St Louis, Baltimore) to fill the missing field.
Ps,,
1 Introduction
Knowledge Graphs (KGs) are prevalent resources for representing real-world facts in a structured way. While traditionally, KGs have been utilized for rep- resenting static snapshots of "current" knowledge, recently, temporal KGs (TKGs) have gained popu- larity to preserve the complex temporal dynamics
Prior research on TKG forecasting has been primarily focused on developing supervised ap-
Authors contributed equally. http:////gthuuubooo/m/sssss--iie

---

proaches such employing graph neural networks model interrelationships among entities and relations (Jin et al., 2020; et al., 2021; Han et al., (b,), using reinforcement learning tech- niques (Sun et al., 2021), and utilizing logical rules (Zhu et al., 2021; Liu et al., 2022). How- ever, these techniques have prominent limitations, including the need for large amounts of training data that include thorough historical information for the entities. Additionally, model selection is a computationally expensive challenge as the state- of-the-art approach differs for each dataset.
+1.5%) Hits@1 margin, relative the median ap- proach for each dataset; (2) LLMs perform almost identically when we replace entities' and relations' lexical names with numerically mapped indices, suggesting that the prior semantic knowledge is not critical factor for achieving such high perfor- mance; and (3) LLMs outperform the best heuristic rule-based baseline on each dataset (i.e., the most frequent the most recent, given the historical context) by (+10%, +28%) Hits@ relative margin, indicating that they do not simply select the output using frequency or recency biases in ICL (Zhao et al., 2021).
In this paper, we develop a TKG forecasting approach by casting the task as an in-context learn- ing (ICL) problem using large language models (LLMs). ICL refers to the capability of LLMs to learn and perform an unseen task efficiently when provided with few examples of input-label pairs in the prompt (Brown al., 2020). Prior works on ICL usually leverage few-shot demonstrations, where uniform number of examples are provided for each label to solve a classification task (Min et al., 2022; Wei et al., 2023). In contrast, our work investigates what the model learns from ir- regular patterns of historical facts in the context, We design three-stage pipeline to control (1) the background knowledge selected for context, (2) the prompting strategy for forecasting, and (3) decod- ing the output into prediction. The first stage uses the prediction query to retrieve a set of relevant past facts from the TKG that can used as context (Section 3. The second stage transforms these contextual facts into lexical prompt representing the prediction task (Section 3.3). The third stage de- codes the output of the LLM into probability dis- tribution over the entities and generates response the prediction query (Section 3.4). Our exper- imental evaluation performs competitively across a diverse collection of TKG benchmarks without requiring the consumnnn supervised training, or custom-designed architectures.
Problem Formulation
In-Context Learning. ICL is an emergent capa- bility of LLMs that aims to induce a state in the model to perform a task by utilizing contextual input-label examples, without requiring changes to its internal parameters (Brown al., 2020). Formally, in ICL for classification, prompt is constructed by linearizing a few input-output pair examples y;) from the training data. Subse- quently, when a new test input text X is provided, ICL generates the output Yiest ~ PL test X1, Yı...., Xk, Xies) where ~ refers to decod- ing strategy.
Temporal Knowledge Graph Forecasting, For- mally, a TKG, G (V, R, E, is comprised of set of entities V. relations R, facts E, and timestamps T. Moreover, since time sequen- tial, G can be split into a sequence of time-stamped snapshots, g = [91, G2...., Gr,,..,............. G.....). where each snapshot, Gt = (,,R,E, contains the facts at specific point in time t. Each fact f E & is a quadruple (s, p, o, where s, E V, p E R, and T. The TKG forecasting task involves pre- dicting temporally conditioned missing entity in the future given a query quadruple, (?, (?,P. o, t) or (s,p,?. p, l), and previous graph snapshots G1.t-1 {9,, G2,... .eg....G. G Here, the prediction typically involves ranking each entity's assigned score.
We present extensive experimental results on common TKG benchmark datasets such as WIKI (Leblay and Chekol, 2018), YAGO (Mahdis- oltani et al., 2014), and ICEWS (García-Durán et al., 2018; Jin et al., 2020). Our findings follows: (1) LLMs demonstrate the ability to make predictions about future facts using ICL with- out requiring any additional training. Moreover, these models show comparable performance to supervised approaches, falling within the (-3.6%,
In-context Learning for Temporal Knowledge Graph Forecasting
In this work, we focus on 1) modeling appropriate history &, for given query quadruple q, 2) con- verting {Eg. 9} into prompt 0g. and 3) employing ICL get prediction y, PLLM(y, 0g)) in a zcro-shot manner. Here, the history E, is mod- eled on the facts from the previous graph snapshots

---

G 1 1 .2...,,.. Ge 1). and we employ to- ken probabilities for Yq get ranked scores of candidate entities in zero-shot manner. In the rest of this section, we study history modeling strate- gies (Sec 3. response generation approaches (Sec 3.2), prompt construction templates (Sec 3.3), and common prediction settings (Sec 3,4).
Category
Prompt
2000: [Superbowl, Champion. Louis]
Lexical (C)
2001: [Superbowl, Chumpion, 1. Baltimore]
...
2023: [Superbowl, Champion.
2000: [0, O. 2001: [0,
Index I(-)
...
2023: 10.0.
Table 2: Prompt Example.
3.1 History Modeling
To model the history Eg. we filter facts that the known entity or relation the query has been involved in. Specifically, given the query quadruple
3.3 Prompt Construction
Given the history &g and query q, we construct a prompt using a pre-defined template 8. Specifically, given the query quadruple = (s, p, ?. under the object entity prediction setting. we present two versions of the template 0 with varying levels of information. Our assumption is that each entity relation has indexed I(.) (e.g., 0) and lexical C() (e.g. Superbowl) form (See Table 2).
(s, p, ?, under the object entity prediction setting, we experiment with two different aspects of historical facts:
Entity includes past facts that contain 8, e.g. all historical facts related to Super- bowl. In contrast, Pair includes past facts that contain both and p. e.g., a list of (Superbowl, Champion, Year) as shown in Table 1.
Entity vs. Pair.
Index. Index displays every fact, (Ssr Spr Sor ft) using the (f,, ((fo),,n.... f I(fo)] tem- plate where fs: E V. fp E R. St T, nf. denotes an incrementally assigned numerical label (i.e., indirect logit), and I is a mapping from enti- ties to unique indices. For example, in Table 1, we can use the following mappings are for the entities and relations. respectively: (Superbowl + 0. St Louis -> 1, Baltimore 22 and |Champion -> 0). The query q is then represented as "t:[Z(s). I(p).". concatenated to the end of the prompt. For subject entity prediction, we follow the same procedure from the other side.
Unidirectional vs. Bidirectional. Unidirec- tional includes past facts FF wherein s (Entity) (s,p) (Pair) in the same position as g (e.g., Unidirectional Pair s and p served as subject and predicate in f c F). Bidirectional includes past facts F wherein (Entity) or (s, (Pair) appear in any valid position (e.g.. Bidirec- tional & Entity ty served subject or object F). As example of the Bidirectional setting, given q = (Superbowl, Champion, 2023),
we include (Kupp, Played, Superhowl, 2022)
because s (i.e., Superbowl) present as the object in f. Moreover, the Bidirectional setting, to preserve the semantics of the facts in the Eg. we transform the facts where s appears an object by 1) swapping the object and subject and 2) replac- ing the relation with its uniquely defined inverse relation (e.g., (fs, Spr for fi) (Sor Sp fs. ft)).
Lexical Lexical follows the same process as Index but uses lexical form [()) of entity and rela- tion. Each fact in fp. ft) E & is represented as "f:: is represented as "t:[C(s), C(p)," concatenated to the end of the prompt.
L(fo) and the query
3.4 Prediction Setting
3.2 Response Generation
All the historical facts in the datasct are split into three subsets, D Dvalid, and Diest, based on the chronological order with train valid < test. Given this split, during the evaluation phase, the TKG forecasting task requires models to predict over Dies under the following two settings:
Given a prompt lg. we pass to an LLM to ob- tain the next token probabilities. Then, we use the obtained probabilities to get a ranked list of enti- ties. However, obtaining scores for entities based on these probabilities is challenging they may be composed of several tokens. To address this challenge, we utilize a mapped numerical label as an indirect logit to estimate their probabilities (Lin et al., 2022).
Single Step. In this setting, for each test query, the model provided with ground truth facts from past timestamps in the test period. Hence, after making predictions for a test query a specific

---

# of Facts
Model Fumily
Model Name
Params
Instruction-tuno
Dalasel
# Ents
# Rels
Interval
Train
Valid
Test
GPTZ
gpt2
124M
X
WIKI
12,554
24
$39,286
67,538
63,110
year
got2-mediun
355M
X
YAGO
10,623
10
161,540
19,523
20,026
1 year
got2-large
774M
X
8p12-x1
ICEWS14
6,869
230
74,845
8,514
7,371
day
ICEWS18
23,033
256
373,018
45,995
49,985
day
GPT-J
got-j-80
6B
X
ACI-DDCD222
243
6
1,788
216
222
day
GPT-NeoX
gpt-neox-28b
208
X
InstresGPPT
gpx-3.5-turbo
Table 3: Data statistics. Each dataset consists of enti- ties, relations, and historical facts, with the facts within the same time interval identified by the same timestamp. The facts are divided into three subsets based on time, where train < valid test.
Table 4: Language Models used in the paper. Exact model size of gpt-3. 3-5tbbb is unknown.
4.2 Evaluation
timestamp, the ground truth fact for that query is added to the history before moving to the test queries in the next timestamp.
We evaluate the models on well-known metrics for link prediction: Hits@ k, with k = 1,3, 10. Follow- ing (Gastinger et al., 2022), we report our results in two evaluation settings: 1) Raw retrieves the sorted scores of candidate entities for given query quadruple and calculates the rank of the correct entity; and 2) Time-aware filter also retrieves the sorted scores but removes the entities that are valid predictions before calculating the rank, preventing them from being considered errors. To illustrate, if the test query is (NBA, Clinch Playoff, ?, 2023) and the true answer is Los Angeles Lakers, there may exist other valid predictions such (NBA, Clinch Playoff, Milwaukee Bucks, 2023) (NBA, Clinch Playoff, Boston Celtics, 2023). In such cases, the time-aware filter removes these valid predictions, allowing for accurate determination of the rank of the "Los Angeles Lakers." In this paper, we present performance with the time-aware filter.
Multi Step. In this setting, the model is not pro- vided with ground truth facts from past timestamps in the test period and has to rely on its noisy pre- dictions. Hence, after making predictions for a test query in specific timestamp, instead of the ground truth fact for that query, we add the pre- dicted response the history before moving to the test queries the next timestamp. This setting is considered more difficult as the model forced to rely on its own noisy predictions, which can lead to greater uncertainty with each successive timestamp.
4 Experimental Setup
4.1 Datasets
4.3 Models.
For our experiments, use the WIKI (Leblay and Chekol, 2018), YAGO (Mahdisoltani et al., 2014), ICEWS14 (Garefa-Durán et al., 2018), and ICEWS18 (Jin et al., 2020) benchmark datasets with the unified splits introduced in previous stud- ies (Gastinger et al., 2022). Additionally, we ex- tract a new temporal forecasting dataset from the Armed Conflict Location & Event Data Project (ACLED) project" which provides factual data of crises in particular region. We specifically focus on incidents of combat and violence against civil- ians Cabo Delgado from January 1900 to March 2022, using data from October 2021 to March 2022 as our test set. This dataset aims to investigate whether LLMs leverage prior semantic knowledge to make predictions and how effective they are when deployed in real-world applications. Table presents the statistics of these datasets.
As shown in Table 4, we perform experiments on four language model families. Among those, three are open-sourced: GPT2 (Radford et al.. 2019), GPT-J (Wang, 2021), and GPT-NeoX (Black et al.. 2022). All models employ the GPT-2 byte level BPE tokenizer (Radford et al., 2019) with nearly identical vocabulary size. In addition, we use the 3. 3 5-turbo model to analyze the per- formance of the instruction- -undd models. How- ever, we do not directly compare this model to other models in terms of size since the actual model size is unknown. As for the TKG baselines, (i.e,, RE-Net (Jin et al., 2020), RE-GCN (Li et al., 2021), TANGO (Han et al., 2021b), xERTE (Han (Sun al., 2021), CyGNet (Zhu et al., 2021), and TLogic (Liu al., 2022)), we report the numbers presented in prior research (Gastinger et al., 2022). Appendix A.4 provides more details on baseline models.
et al., 2021a),
Phtps://data.humma

---

Single-Step
Train
YAGO
WIKI
ICRAS114
ICEWS18
ACLED-CD22
H69
HR8
H
H
HP:10
HRR
H@1
H
HGP
H88
H683
10
RE-DON
0.787
0.842
0.884
0.747
0.817
0.846
0.313
0.473
0.626
0.223
0.367
0.525
0.446
0.545
0.608
KERTE
0.842
0.902
0.912
0.703
0.785
0.501
0.330
0.454
0.570
0.209
0.335
0.462
0.320
0.445
0.497
TLogic
0.740
0.759
0.791
0.786
0,860
0.570
0.332
0.476
0.602
0.204
0.336
0.480
0.009
0.045
0.091
0.590
0.677
0,514
0.527
0.272
0.408
0.550
0.191
0.462
0.327
0.482
Tänetraveler
0.845
0.908
0.912
0.751
0.820
0.830
0.319
0.454
0.575
0.212
0.325
0.439
0.240
0.315
0.457
GPT-Neok Neok (Entity)
x
0.784
0.591
0,927
0.694
0.804
0.844
0.324
0,460
0,565
0.192
0.313
0.414
0.324
0,492
0.604
GPT-MooX (Pain)
0.787
0.926
0,812
0.847
0.297
0.408
0.482
0.196
0.307
0.402
0.317
0,440
Sulti-Step
Train
YAGO
WIKI
ICEWS18
ACLED-CD22
Har
1869
1148 10
H@1
3183
H6:10
H363
H6 10
H@01
H66
H6 10
1681
H363
166 10
RE-SON
0.717
0.812
0.648
0.678
0.278
0.421
0.475
0.464
RE-Net
0.534
0.613
0.862
0.472
0.507
0.530
0.278
0.408
0.549
0.184
0.314
0.461
0.238
0.445
0.563
CyGNet
0.613
0.742
0,834
0.525
0.624
0.675
0.266
0.402
0.545
0.166
0.295
0.444
0.408
0,500
0,588
TLogie
0.631
0.705
0.715
0.613
0,663
0.682
0.265
0.395
0.531
0.155
0.272
0.112
0.009
0.045
0.091
GPT-NeoX (Entity)
X
0.685
0.793
0.840
0.543
0.622
0.655
0.247
0.363
0.471
0.136
0.224
0.321
0.319
0.417
0.500
GPT-Neox (Pair)
X
0.688
0.793
0.839
0.570
0.625
0.652
0.236
0.524
0.395
0.155
0.246
0.331
0.289
0.410
0,464
Table 5: Performance (Hits @ K) comparison between supervised models and ICI. for single-step (top) and multi-
step (bottom) prediction. The first group in each table consists of supervised models, whereas the second group consists of ICL models, i.e., GPT-NeoX with a history length of 100. The best model for each dataset the first group is shown in bold, and the second best is underlined.
ICEWS14
ICEWS18
ICEWS14
ICEWS18
Single-Step
Multi-Step
H@1
H@3
H@10
H@1
H@3
H@
H@8
H
H6
H61
H68
H6 10
frequency
0.243
0.387
0.532
0.141
0,265
0.409
frequency
0,222
0.349
0.460
0.121
0,207
0.307
recency
0,228
0.387
0.536
0.120
0.242
0.403
recendy
0.151
0,268
0.423
0.044
0.266
GPT-NeoX (Fociry)
0.324
0.460
0.565
0.192
0.313
0.414
GPT-heoX (Fatity)
0.247
0.363
0.471
0.136
0.224
0.321
GPT-NeoX (Pair)
0,297
0.408
0.482
0.196
0.307
0.402
GPT-NeoX (Pait)
0,324
0,395
0.155
(a) Single-step
(b) Multi-step
Table 6: Performance (Hits@ K) with rule-based predictions. The best model for each dataset is shown in bold.
4.4 ICL Implementation Details.
the rank of the actual value within the predictions
is assigned value of 100, which is considered incorrect according to our evaluation metric. For instruction-ture model, we use the manual curated system instructions in Appendix A.3.
We implement our frameworks using py- Torch (Paszke et al., 2019) and Huggingface (Wolf et al., 2020). We first collate the facts f E Drest based on the identical test query to eliminate any repeated inference. To illustrate, suppose there exist two facts in the test set denoted as (s. p, a, t) and (s, p. t) in the object prediction scenario. We consolidate these facts into (s, p. [a, b).t) t) and forecast only one for (s,p, p. t). Subsequently, we proceed to generate an output for each test query with history by utilizing the model, obtaining the probability for the first generated token in a greedy approach, and sorting the probability. The outputs are deterministic for every iteration. We retain the numerical tokens corresponding to the numerical label n that was targeted, selected from the top 100 probability tokens for each test query. To facilitate multi-step prediction, we incorporate the top-k predictions of each test query as supplementary reference history. In this paper, we present results with A- 1. It is important acknowledge that the prediction may contain minimal or no numerical tokens as result of inadequate in-context learning. This can lead to problems when evaluating based metrics. To mitigate this, we have established a protocol where
5 Experimental Results
5.1 In-context learning for TKG Forecasting
In this section, we present multifaceted perfor- mance analysis of ICL under Index & Unidirec- tion prompt strategy for both Entity and Pair history.
Q1: How well does ICL fare against the super- vised learning approaches? We present a com- parative analysis of the top-performing ICL model against established supervised learning methodolo- gies for TKG reasoning, which are mostly based on graph representation Icarning. As evident from the results in Table 5, GPT-NeoX with history length of 100 shows comparable performance to super- vised learning approaches, without any fine-tuning on any TKG training dataset.
Q2: How do frequency and recency biases affect ICL's predictions? To determine the extent to which LLMs engage in pattern analysis, beyond simply relying on frequency and recency biases,

---

Singlestep, Unidirection, Entity, Length
ICEWS14
Single-Step
Prompt
Original
0.7
H@!
Time-removed
0.6
Time-removed
-et-3.5-turbo index
0.15
0.5
gpt-3.5-turbe
lexical
0.1858
0.4
0.3
Table 7: Performance (Hits@) 1) between index and lexical for 3pt-3.5-turme
0.2
YAGO
WikI
ICEWS14
ICEWS18
Figure I: Performance (Hit@) I) with and without time, and with shuffling
Q4: How does instruction-tuning affect ICL's performance? To investigate the impact of instruction-tuniin
wC run a comparative analysis between GPT-NeoX and heuristic-rules (i.e., frequency & recency) on the ICEWS14 dataset, with history length set to 100. frequency identifies the target that appears most frequently in the provided history while re- cency selects the target associated with the most recent fact in the provided history. The reason for our focus on ICEWS is that each quadruple repre- sents a single distinct event time. contrast, the process of constructing YAGO and WIKI involves converting durations to two timestamps to display events across the timeline. This step has resulted in recency heuristics outperforming all of the ex- isting models, showcasing the shortcoming of ex- isting TKG benchmarks (See Appendix A.5). The experimental results presented in Table 6 demon- strate that ICL exhibits superior performance to rule-based baselines. This finding suggests that ICL does not solely rely on specific biases to make predictions, but rather it actually learns more so- phisticated patterns from historical data.
ICL, employ the gpt- 3. 5urboo model with manually curated system instruction detailed in Appendix 4.4. Since the size of this model not publicly disclosed, is challenging to make direct comparisons with other models featured in this paper. Moreover, since this model does not provide output probabilities, we are only able to report the Hit@) metric.
Table 7 showcases that the performance of the lexical prompts exceeds that of the index prompts by 0.024, suggesting that instruction tumnd models can make better use of semantic priors. This behavior is different from the other foundation LLMs, where the performance gap between the two prompt types was insignificant (See Figure 4 (a)).
Q5: How does history length affect ICL's per- formance? To evaluate the impact of the history length provided the prompt, we conduct a set of experiments using varying history lengths. For this purpose, we use the best performing prompt format for each benchmark, i.e., Enti ty for WIKI, YAGO, ICEWS18, and Pair for ICEWS14. Our results, shown in Figure 2, indicate consistent improvement performance as the history length increases. This suggests that the models learn bet- ter as additional historical facts are presented. This observation connected to few-shot learning other domains, where performance improves as the number of examples per label increases. However, in our case, the historical patterns presented in the prompt do not explicitly depict the input-label map- ping but rather aid in inferring the next step.
Q3: How does ICL use the sequential and tempo- ral information of events? To assess the ability of LLMs to comprehend the temporal information of historical events, we compare the performance of prompts with and without timestamps. Specifically, we utilize the original prompt format, "fa, I(fr), f.- I(fo)I", and the time-removed prompt I(f.)l", make the comparison (See Appendix A.2). Additionally, we shuffle the historical facts in the time-removed prompt format to see how the model is affected by the corruption of sequential information. Figure shows that the absence of time reference can lead to deterioration in performance, while the ran- dom arrangement of historical events may further exacerbate this decline in performance. This obser- vation implies that the model has the capability to forecast the subsequent event by comprehending the sequential order events.
format, "II(f.),I(f));
Q6: What is the relation between ICL's perfor- mance and model size? Here, we analyze the connection between model size and performance. Our results, presented in Figure 3, conform to the expected trend of better performance with larger models. This finding aligns with prior works show- ing the scaling law of in-context learning perfor- mance. Our findings are still noteworthy since they

---

0.9
Sigle Stop
0.9
Mutti Step
Single Stop
0.0
MATR1 Stop
1A60
0.7
0.7
1950
0.7
0.5
0.3
0.5
0.3
0.2
ICEW514
ICEVS18
0.3
ICEWS14
0.3
CEWSS
0.1
0.2
0.1
0/0
3338
774M
1:38
200
Figure 2: Performance (Hit@) @ 1) adheres the scaling law based on the history length.
Figure 3: Performance (Hit@) 1) adheres to the scaling law based on the model size.
show how scaling model size facilitate more powerful pattern inference for forecasting tasks.
Entity vs. Pair Finally, we examine the impact of the bistory retrieval query on performance. Our hypothesis posits that when the query is limited to single entity,
5.2 Prompt Construction for TKG Forecasting
incorporate more diverse his- torical facts. Conversely, when the query is pair, can acquire a more focused of historical facts related the query. Our results (Figure 4 (c)) indi- cate that the performance of the model is dependent on the type of data being processed. Specifically, the WTKI and ICEWS18 benchmarks perform bet- ter when the query is focused on the entity, as a broader range of historical facts is available. In contrast, the ICEWS14 benchmark performs better when the query is focused pairs, as the historical facts present a more focused pattern.
To determine the most effective prompt variation, set of experiments on all prompt variations, using GPT-J (Wang, 2021) and under the single- step setting. Comprehensive results for prompt variations can be found in Appendix A.5.
Index vs. Lexical Our first analysis compares the performance of index and lexical prompts. This investigation aims to determine whether the model relies solely on input-label mappings or if it also incorporates semantic priors from pre-training to make predictions. Our results (Figure 4 (a)) show that the performance is almost similar (tM4e - on average) across the datasets. This finding is aligned with previous studies indicating that foundation models depend more on input-label mappings and are minimally impacted by semantic priors (Wei al., 2023).
6 Related Works
Event Forecasting. Forecasting is complex task that plays crucial role in macisoon-making and safety across various domains (Hendrycks et al., 2021). To tackle this challenging task, researchers have explored various approaches, including sta- tistical and judgmental forecasting (Webby and O'Connor, 1996; Armstrong, 2001; Zou et al.. 2022). Statistical forecasting involves leveraging probabilistic models (Hyndman and Khandakar, 2008) neural networks (Li et al., 2018; Sen al., 2019) to predict trends over time-series data, While this method works well when there are many past observations and minimal distribution shifts, it is limited to numerical data and may not capture the underlying causal factors and dependencies that affect the outcome. On the other hand, judgmen- tal forecasting involves utilizing diverse sources of information, such news articles and exter- nal knowledge bases, reason and predict future events. Recent works have leveraged language models to enhance reasoning capabilities when ana- lyzing unstructured text data to answer forecasting inquiries (Zou et al., 2022; Jin et al., 2021).
Unidirectional vs. Bidirectional We next ana- lyze how the relation direction in the history model- ing impacts the performance. This analysis aims to ascertain whether including historical facts, where the query entity pair appears in any position, can improve performance by offering diverse array of historical facts. Our results (Figure 4 (b)) show that there is a slight decrease in performance when Bidirections ional history is employed, with a signif- icant drop in performance observed particularly in the ICEWS benchmarks. These observations may be attributed to the considerably more significant number of entities placed in both subject and ob- ject positions in ICEWS benchmarks than YAGO and WIKI benchmarks (See Appendix A.5). This finding highlights the necessity of having robust constraints on the historical data for ICL to com- prehend the existing pattern better.

---

Unidirection, Enttyy Histery
Index.
Index. History Length
Unidiaction Bidlrection
Enktty
Lokicl
0.6
Pair
0.4
0.3
0.
0.1
YAGO
Wik
KEWS14 ICEWS18
YAGO
WKI
ICESSI4 ICEW/188
(a) Index vs. Lexical
(b) Unidirection Bidirection
(c) Entity vs. Pair
Figure 4: Performance (Hit@1) 1) Analysis on Prompt Variation. The comparable performance exhibited by both the Index and Lexical models indicates that these models rely heavily on learning patterns and are less influenced by semantic priors. Moreover, the Unidirectional model typically outperforms the Bidirection ional model, suggesting that the robust constraints historical data enable the model to comprehend observed patterns better. Finally, the performance of the Entity and Pair models varies depending on the dataset.
Temporal Knowledge Graph. Temporal knowl- edge graph (TKG) reasoning models are commonly employed in two distinct settings, namely interpo- lation and extrapolation, based on the facts avail- able from to to 4. (1) Interpolation aims to pre- dict missing facts within this time range from 6o to In and recent works have utilized embedding- based algorithms to learn low- dinddimnsiooall repre- sentations for entities and relations date facts (Leblay and Chekol, 2018; Gareia-Durán et al., 2018; Goel al., 2020; Lacroix al., 2020); (2) Extrapolation aims to predict future facts be- yond fn- Recent studies have treated TKGs as a sequence of snapshots, each containing facts corre- sponding to a timestamp i,, and proposed solutions by modeling multi-relational interactions among entities and relations over these snapshots using graph neural networks (Jin et al., 2020; Li al., 2021; Han et al., 2021b,a), reinforcement learn- ing (Sun et al., 2021) or logical rules (Zhu al., 2021; Liu et al., 2022). In our work, we focus on the extrapolation setting.
edge by examining their correlation with training examples (Min et al., 2022; Razeghi et al., 2022; Xie et al., 2022), data distribution (Chan al., 2022), and language compositionallity (Hahn and Goyal, 2023) in the pre-training corpus. Other recent works show that LLMs can actually learn input-label mappings from in-context examples by showing the transformer models trained specific linear function class is actually predicting accu- rately on new unseen linear functions (Garg et al., 2022). More recently, there is a finding that large- enough models can still do ICL using input-label mappings when semantic prior knowledge not available (Wei et al., 2023).
candi-
Conclusion
In this paper, examined the forecasting capabil- ities in-context learning in large language mod- cls. To this end, we experimented with temporal knowledge graph forecasting benchmarks. We pre- sented a framework that converts relevant histor- ical facts into prompts and generates ranked link predictions through token probabilities. Our exper- imental results demonstrated that without any fine- tuning and only through ICL, LLMs exhibit com- parable performance to current supervised TKG methods that incorporate explicit modules to cap- ture structural and temporal information. We also discovered that using numerical indices instead of entity/relation names does not significantly affect the performance, suggesting that prior semantic knowledge is not critical for overall performance. Additionally, our analysis indicated that ICL helps the model learn irregular patterns from historical facts, beyond simply making predictions based on the most common or the most recent facts the given context. Together, our results and analyses
In-context Learning. In-context learning (ICL) has enabled LLMs accomplish diverse tasks in few-shot manner without needing parameter adjust- ments (Brown et al., 2020: Chowdhery et al., 2022). In order to effectively engage in ICL,, models can leverage semantic prior knowledge to accurately predict labels following the structure of in-context exemplars (Min et al., 2022; Razeghi et al., 2022; Xic et al., 2022; Chan et al., 2022; Hahn and Goyal, 2023), and learn the input-label mappings from the in-context examples presented (Wei et al., 2023), To understand the mechanism of ICL, recent stud- ies have explored the ICL capabilities LLMs with regards to the impact of semantic prior knowl-

---

demonstrated that ICL can be valuable tool for predicting future links using historical patterns, and also prompted further inquiry into the potential of ICL for additional capabilities.
2022. Gpt-neox-20b; An souree autoregressive language model. arXiv preprint ar arXiv:2204.06745
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-9911
8 Limitations
There are certain limitations to our experiments. First, computing resource constraints restrict our experiments to small-scale open-source models. Second, our methodologies have constraints re- garding models where the tokenizer vocabulary comprises solely of single-digit numbers as tokens, such as LLAMA (Touvron et al., 2023). The per- formance of such models exhibits a similar trend in terms of scaling law concerning model size and his- tory length, but these models demonstrate inferior performance compared to other models of the same model size. Third, our methodologies have certain limitations with respect link prediction settings. While real-world forecasting can be performed in the transductive setting, where the answer can be an unseen history, our approach constrained to the inductive setting, where the answer must be one of the histories observed. There are further directions that can be pursued. The first is to explore trans- ductive extrapolation link prediction using LLMs. The second is to analyze the effects of fine-tuning on the results. Lastly, there is the opportunity to investigate the new capabilities of ICL.
Stephanie C.Y. Chan. Adam Santoro, Andrew Kyle Lampinen, Jane X Wang. Aaditya K Singh, Pierre Harvey Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. In Ad- vances in Neural Information Processing Systems.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint MXXi:2204.023111
Alberto Garefa- -uránn, Sebastijan Dumančić, and Math- ias Niepert. 2018. Learning sequence encoders for temporal knowledge graph completion. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 48 4816-4821, Brussels, Belgium. Association for Computational Linguistics.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gre- gory Valiant. 2022. What can transformers learn in-context? case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598.
Julia Gastinger, Timo Sztyler, Lokesh Sharma, and Anett Schuelke. 2022. On the evaluation of meth- ods for temporal knowledge graph forecasting. In NeurlPS 2022 Temporal Graph Learning Workshop.
9 Acknowledgement
This work was funded in part by the Defense Advanced Research Projects Agency (DARPA) and Army Research Office (ARO) under Con- tract No. 191INFF11-C-O0 HIR00112201006 and with support from the Ke- ston Exploratory Research Award and Amazon.
Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker. and Pascal Poupart. 2020. Diachromic embedding for temporal knowledge graph completion. In Proceed- ings of the AAAI conference on artificial intelligence, volume 34, pages 3988-3995.
and Contract No.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ARO or the U.S. Government.
Michael Hahn and Navin Goyal. 2023, A theory of emergent in-context learning as implieit structure induction. arXiv preprint arXiv:2303.073
Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. 2021a. Explainable subgraph reasoning for forecast- ing on temporal knowledge graphs. In International Conference on Learning Representations.
References
Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp. 2021b. Learning neural ordinary equa- tions for forecasting future links on temporal knowl- edge graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 8352-8364, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Jon Scott Armstrong. 2001. Principles of forecasting: a handbook for researchers and practitioners, vol- ume 30. Springer.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al.

---

Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. 2021. Unsolved problems in ml safety. arXiv preprint arXiv:2109, 1.396.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe. Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 11048-1 1064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Rob Hyndman and Yeasmin Khandakar. 2008. Auto- matic time series forecasting: the forecast package for r. Journal of statistical software, 27:1-22.
Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. 2021. ForecastQA: A question answering challenge for event forecasting with temporal text data, In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational I Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 4636- 4650, Online. Association for Computational Lin- guistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner. Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning li- brary. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- fion Processing Systems 2019, NeurIPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada, pages 8024 8035.
Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. 2020. Recurrent event network: Autoregressive struc- ture inferenceover temporal knowledge graphs. In Pruceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6669 6683, Online. Association for Computa- tional Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Timothée Lacroix, Guillaume Obozinski, and Nicolas Usunier. 2020. Tensor decompositions for temporal knowledge base completion. In International Con- ference on Learning Representations.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner. and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguisties.
Julien Leblay and Melisachew Wudage Chekol. 2018. Deriving validity time in knowledge graph. In Com- panion proceedings of the the web conference 2018 pages 1771-1776.
Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. 2019. Think globally, act locally: A deep neural network approach high-dimensional time series forecasting. Advances in neural information processing systems, 32.
Yaguang Li. Rose Yu. Cyrus Shahabi, and Yan Liu. 2018. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In Interna- tional Conference on Learning Representations.
Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang. and Xueqi Cheng. 2021. Temporal knowledge graph reason- ing based on evolutional representation learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 408-417.
Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. 2021. TimeTraveled Reinforcement learn- ing for temporal knowledge graph forecasting. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8306 8319, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 13777
Yushan Liu, Yunpu Ma. Marcel Hildebrandt, Mitchell Joblin, and Volker Tresp. 2022. Tlogic: Temporal logical rules for explainable link forecasting on tem- poral knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4120-4127.
Ben Wang. 2021. Mrsh--Transormme-JJA Model- Parallel Implementation of Transformer Lan- guage Model with JAX. https://github. com/
Farzaneh Mahdisoltani, Joanna Biega, and Fabian Suchanek. 2014. Yago3: A knowledge hase from multilingual wikipedias. In 7th biennial conference on innovative data systems research. CIDR Confer- ence.
Richard Webby and Marcus O'Connor. 1996. Judge- mental and statistical time series forecasting: view of the literature. International Journal of fore- casting, 12

---

Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. arXiv preprinl arXiv:2303.03845
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ar natural language processing. Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning implicit bayesian inference. In Interna- tional Conference on Learning Representations.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing sho-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR.
Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan Cheng. and Yan Zhang. 2021. Learning from history: Modeling temporal knowledge graphs with sequen- tial copy-generation networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 4732-4740.
Andy Zou, Tristan Xiao, Ryan Jia, Kwon, Mantas Mazeika, Richard Li. Dawn Song. Jacob Steinhardt, Owain Evans, and Dan Hendrycks. 2022. Forecast- ing future world events with neural networks. In siith Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track.

---

A Appendix
A.4 Baseline Models
A.1 Prompt Example
RE-Net (Jin et al., 2020) leverages an auto- regressive architecture that employs two-step pro- cess for learning temporal dependency from a se- quence of graphs and local structural dependency from the vicinity. The model represents the likeli- hood of a fact occurring as probability distribution that is conditioned on the sequential history of past snapshots.
Given the test query at timestamp 571, prompt ex- amples for Index and Lexical are shown in Fig- ure 5. Here, we assume the entity dictionary con- tains "Islamist Militia (Mozambique)" as index O, "Meluco" as 10, "Namatil" 36, "Muatide" 53, "Limala" as 54, and "Nacate" as 55, while relation dictionary contains "Battles" as index and "Vi- olence against civilians" as 4. Also, the history setting is unidirectional entity setting where the history length is set to 5.
RE-GCN (Li et al., 2021) also employs auto- regressive architecture while it utilizes multi-layer relation-aware GCN on each graph snapshot to cap- ture the structural dependencies among concurrent facts. Furthermore, the static properties of entities such as entity types, are also incorporated via a static graph constraint component to obtain better entity representations.
su8 (I, 1.35)
Sui (Kaanie Milita (Nnomttrre) Vindenoe againe civllins,1 1 Nocit] Metsee
598
sW Dianie Milis (nnomtipue Bettes Kanaat]
599 00.1..054
Sot [aseman Niita Nooumbaaaa Buttia Lenal))
mI [L. 1.4.55)
mi (icansic Ailita (Nnomttaree, Bottes Miatidy
(Muomitipee,
dg Lesial
Figure 5: Prompt examples for Index and Lexical settings.
TANGO (Han et al., 2021b) employs auto- regressive architecture as well but the use of continuous-time embedding encoding tempo- ral and structural information is a distinguishing feature of the proposed method, as opposed to RE- Net (Jin et al., 2020) (Li et al., 2021) and RE-GCN which operate on a discrete level with regards to time.
A.2 Prompt Example for Analysis
To assess the ability of LLMs comprehend the sequential information of historical events, we com- pare the performance of prompts with and without timestamps (See Scction 5.1 Q3). Figure shows the prompt examples for time-removed and shuf- fled version of prompts.
XERTE (Han et al., 2021a) employs an atten- tion mechanism that can effectively capture the relevance of important aspects by selectively fo- cusing on them. It employs sequential reasoning approach over local subgraphs. This process begins with the query and iteratively selects relevant edges of entities within the subgraph, subsequently prop- agating attention along these edges. After multiple rounds of expansion, the final subgraph represents the interpretable reasoning path towards the pre- dicted outcomes.
968 2.30 599 (1, 0.
3. 30
4. 38
1.3.0.55 54)
15 1.1.4 83
(6)
Figure 6: Prompt examples for time-removed and shuffled version.
A.3 System Instruction for Instruction-tuned models.
Travelr (Sun et al., 2021) employs rein- forcement learning for forecasting. The approach involves the use of an agent that navigates through historical knowledge graph snapshots, commenc- ing from the query subject node. Thereafter, it se- quentially moves to a new node by leveraging tem- poral facts that are linked to the current node, with the ultimate objective of halting at the answer node. To accommodate the issue of unseen-timestamp. the approach incorporates a relative time encod- ing function that captures time-related information when making decisions.
For the instruciion-moder we use the manual cu- rated system instructions to provide task descrip- tions and constraint the output format as follow:
You nust be able to correctly predict the next (object_label) from a given text consisting of mul tiple quadruplets in the form of "[time): ((sume):((uujject), (relation), (abject_label). [object)]" and the query in the form of "(time): ((sujject, (relation), in the end.
You must generate only the single number for [object_label] without any explanation.

---

CyGNet (Zhu et al., 2021) leverages the statis- tical relevance of historical facts, acknowledging the recurrence events in the temporal knowl- edge graph datasets. It incorporates two inference modes, namely Copy and Generation. The Copy mode determines the likelihood of the query being repetition of relevant past facts. On the other hand, the Generation mode estimates the probabil- ity of each potential candidate being the correct prediction, using a linear classifier. The final fore- cast is obtained by aggregating the outputs of both modes.
TLogic (Liu et al., 2022) mines cyclic temporal logical rules by extracting temporal random walks from graph. This process involves the extraction of temporal walks from the graph, followed by lift to a more abstract, semantic level, resulting in the derivation of temporal rules that can generalize to new data. Subsequently, the application these rules generates answer candidates, with the body groundings in the graph serving as explicit and easily comprehensible explanations for the results obtained.
A.5 Full Experimental Results

---

YAGO
WIKI
ICEWS14
ICEWS18
ACLED
Prompt
History
H
H:93
H0:
H681
H 10
H61
H
10
H68
H@B
H 10
H68
H
H 10
Unidirectional Entiy
0.850
0.775
Todex
Unidirectional Pair
0.778
0.880
0.904
0.646
0.731
0.777
0.294
0.400
0.471
0.187
0.254
0.385
0.351
0:457
01.64
Tndex
Ridircetional Eatity
0.778
0.879
0.607
0.773
0:404
0.142
0.245
0.382
B:497
Index
Bidirectional Pair
0.780
0.879
0,904
0.647
0.733
0.777
0.291
0.398
0.471
0.185
0.291
0.384
0.324
0.455
0.564
Lextical
Unidiractionl Fatity
0.777
0.874
0,904
0.607
0.722
0.775
0.285
0.406
0.532
0.149
0.257
0.394
0.293
0.400
06001
Lexical
Unidirectioal Pair
0.781
0.874
0.645
0.381
0.426
0.549
Lexical
Didirectional Emity
0.773
0.872
0.901
0.601
0.714
0.771
0.270
0.399
0.526
0.141
0.238
0.371
0.343
0.464
0.601
Lexical
Bidirectional
0.777
0.182
Table 8: Performance Comparison (Hits @ k) in single-step inference with time-aware filter.
YAGO
WIKI
ICEWS14
ICEWSI8
ACLED
Prompt
History
H&1
HG3
HG:10
H6B
H6: 10
H60 1
H
HG:10
H661
H@B
H6:00
HG0
H
H6: 10
Index
Unkdireztional Entily
0,750
0.587
0.303
0.435
Index
Unidinectional Pair
0.694
0.790
0.520
0,518
0.567
0.596
0.228
0.315
0.381
0.143
0.224
0.298
0.317
0,400
0.438
Bidirectional Eminty
0.970
0.580
0.178
Index
Bidirectional Pair
0.694
0,750
0.800
0.518
0,567
0.595
0.226
0.313
0.379
0.142
0,225
0.298
0.308
0.400
0448
Lexical
Unidirectional Entity
0.697
0.789
0.800
0.487
0.557
0.588
0.218
0.306
0.399
0.102
0.161
0.223
0.341
0.428
0.481
Lexical
Unidireciooal Fair
0.698
0.789
0.820
0.524
0.570
0.597
0.230
0.317
0.352
0.143
0.226
0.299
0.303
0.393
0.445
Lexical Bidirectiona Emity
0.693
0.789
0.800
0.479
0.547
0.579
0.178
0.255
0.327
0.091
0.122
0. 0.168
0.338
0.428
0-481
Lexical
Bidirectional
0.596
0.212
Table 9: Performance Comparison (Hits@k) in multi-step inference with time-aware filter.
YAGO
WIKI
ICEWS14
ICEWS18
Single-Step
I@@3
H@1
H@3
H@@
H@
H@3
H@ 10
H@@1
H@@
H@@0
frequency
0.766
0.859
0.921
0.549
0.712
0.818
0.243
0.387
0.532
0.141
0.265
0.409
recency
0.886
0.927
0.928
0.701
0.831
0.849
0.228
0.387
0.536
0.120
0.242
0.403
GPT-NeoX (Entity)
0.784
0.891
0.927
0.694
0.804
0.844
0.324
0.460
0.565
0.192
0.414
GPT-NeoX (Pair)
0.787
0.892
0.926
0.721
0.812
0.847
0.297
0.408
0.482
0.196
0.307
0.402
Table 10: with time-aware filter.
Performance Comparison (Hits@k) between rule-based prediction and ICI. in single-step inference
YAGO
WIKI
ICEWS14
ICEWS18
Single-Step
H@l
H@3
H@ 10
H@1 1
H@3
H@ 10
H@]
H@3
H@ 10
H@1
H@3
H@10
Frequency
0.691
0.789
0.837
0.484
0.603
0.652
0.222
0.349
0.460
0.121
0.207
0.307
recency
0.785
0.840
0.842
0.540
0.637
0.661
0.151
0.268
0.423
0.074
0.149
0.266
GPT-NeoX (Entity)
0.686
0.793
0.840
0.543
0.622
0,655
0.247
0.363
0.471
0.136
0.224
0.321
GPT-NeoX (Pair)
0.688
0.793
0.839
0.570
0.625
0.652
0.236
0.324
0.395
0.155
0.245
0.331
Table 11: Performance Comparison (Hits@kk) between rule-based prediction and ICL in multi-step inference with time-aware filter.


Reviews:
[{"review_id": "6CRkU5rAsl", "reviewer": "Reviewer_MDqV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our work. Thank you for taking the time to provide us with valuable insights. Here are our comments on your feedback.\n\n#### **R1. `It may be unfair to supervised models because test triples may exist in textual form during the pretraining phase of LLMs.`**\nWe agree with your opinion, which is why we carried out experiments using the `index` prompt, where entities are mapped to random numerical values (refer to Table 2). The main experiments, as stated in line 354, are conducted using this prompt to prevent such information leaks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, this a clearly written paper describing a novel application of in-context learning for TKG forecasting. The experiments are extensive, and the results demonstrate the promise of leveraging LLMs for  knowledge graph reasoning tasks.", "reasons_to_reject": "It may be unfair to supervised models because test triples may exist in textual form during the pretraining phase of LLMs. We do not know whether LLMs learned historical information for reasoning or just remembered knowledge.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an interesting approach for temporal knowledge graph (TKG) forecasting using in-context learning with large language models (LLMs). The main contributions are: 1) proposing a pipeline with history modeling, prompt construction, and output decoding stages to enable LLMs to make TKG predictions in a zero-shot setting. 2) Comparing LLMs capabilities against strong supervised baselines on benchmark datasets. The results show LLMs are competitive without any training. 3) Analyzing different factors like sequential and temporal information of events, model size, history length, etc, that impact the effectiveness of LLMs for TKG forecasting.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sg3uDlGFdA", "reviewer": "Reviewer_rjDk", "summary": "", "questions": "(1) The #Params of gpt-3.5-turbo is reported.\n(2) Is there too much sample in in-context learning? Will some of it be deleted?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our work. Thank you for taking the time to provide us with valuable insights. Here are our comments on your feedback.\n\n#### **R1. `The method seems to only consider the 1-hop neighbors of the query, ignoring the long-range neighbor structures.`**\nYes, our approach only considers the 1-hop neighbors of the query as history during inference. However, even with this disadvantage, it achieves comparable results to other GNN-based approaches that use multi-hop neighbors. Our paper also demonstrates that widely-used TKG benchmark datasets present a weak forecasting problem (e.g., WIKI and YAGO) that can be solved with 1-hop neighbors. We will more strenuously articulate this point.\n\n#### **R2. Rationale behind `multi step` setting**\nThe multi-step setting is crucial and presents a greater challenge as models can only rely on their own forecasts, leading to an accumulation of uncertainty with each additional forecasted timestamp. Additionally, there are real-world scenarios where the multi-step setting is employed. For instance, to predict the World Cup winner before all the game rounds start, people make predictions by forecasting the result of each round.\n\n#### **Q1. # Parameters of `gpt-3.5-turbo`**\nThe exact number of parameters for the `gpt-3.5-turbo` model has not been officially disclosed by OpenAI; hence, the reason for exclusion.\n\n#### **Q2. `Is there too many samples in in-context learning? Will some of it be deleted?`**\nIn the past, in-context learning has primarily been applied to general NLP tasks that do not require a large number of examples (e.g., 4-shot: 4 examples per each label). However, this study aims to investigate whether in-context learning can be employed to capture complex patterns in sequences of numeric or arbitrary tokens. To achieve this, we provide the model with an extensive history of numeric tokens, allowing it to implicitly learn the patterns and generate a forecast for completing the sequence. Therefore, we can provide numerous samples as long as they adhere to the maximum sequence length of the model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) It seems the first study that introduces LLM for TKG forecasting.\n(2) The paper gives some perspective findings that can accelerate the development of technology in the community.\n(3)  Very detailed experiments completed in Section 5.1 and 5.2.", "reasons_to_reject": "(1) The method seems to only consider the 1-hop neighbors of the query, ignoring the long-range neighbor structures. The rich semantic information in the TKG is not used. The proposed method is too simple and direct.\n(2) Why three is \"Multi Step\" setting in the experiment? This doesn't seem to be in line with reality. In reality, intermediate events are generally not predicted, but rather the target query is directly predicted.", "questions_to_authors": "(1) The #Params of gpt-3.5-turbo is reported.\n(2) Is there too much sample in in-context learning? Will some of it be deleted?", "paper_topic_and_main_contributions": "The paper utilizes LLMs to conduct temporal knowledge graph (TKG) forecasting through in-context learning. It seems the first study that introduces LLM for TKG forecasting.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f6gr1OAup2", "reviewer": "Reviewer_KePy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our work. Thank you for taking the time to provide us with valuable insights. Here are our comments on your feedback.\n\n#### **R1. `Experiments are limited to small-scale open-source models due to computing resource constraints`**\n\nWe were also concerned about this limitation, so we conducted additional experiments with other model variants (some larger) after the submission. These additional results (Hit@1) can be included in the camera-ready version:\n\n| Model | # parameters    | YAGO | WIKI  |   ICEWS  | ICEWS18   |\n| ------ | ------ | ------ | ------ | ------ | ------ |\n| OPT  |  13b | 0.7648 |  0.6694  |  0.8149   | 0.1774 | \n| OPT  |  66b         | 0.7758  | 0.6841  |  0.8191  | 0.1860 | \n\nIt is important to note that a direct performance comparison of different model variants (GPT-J, GPT-NeoX, OPT, etc.) is challenging due to the influence of pre-training strategy and data distribution on the capabilities of in-context learning [1,2]. Nonetheless, our experimental results show that the models within the same variants adhere to the scaling law.\n\n[1] Data distributional properties drive emergent in-context learning in transformers., Chan et al., 2022 \\\n[2] A theory of emergent in-context learning as implicit structure induction., Hanh and Goyal., 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include its thorough evaluation of diverse baselines and state-of-the-art models for TKG forecasting, as well as its exploration of the strengths and weaknesses of large language models (LLMs) for this task. The paper also proposes a new evaluation methodology and introduces a new dataset for TKG forecasting, which can be valuable resources for future research in this area. \n\nIf this paper were to be presented at the conference or accepted into Findings, the main benefits to the NLP community would be the insights it provides into the use of LLMs for TKG forecasting, as well as the new evaluation methodology and dataset it proposes. This can help researchers to better understand the strengths and weaknesses of different models for TKG forecasting and to develop more effective approaches for this task. Additionally, the paper's findings can have practical applications in areas such as natural language generation and question answering, which rely on accurate predictions of future events.", "reasons_to_reject": "One potential weakness of this paper is that the experiments are limited to small-scale open-source models due to computing resource constraints. This may limit the generalizability of the findings to larger models or real-world applications. Additionally, the paper's methodologies have certain limitations with respect to link prediction settings, which may affect the applicability of the proposed evaluation methodology to other TKG forecasting tasks. \n\nThe main risks of having this paper presented at the conference or accepted into Findings are minimal, as the paper's contributions are valuable and the weaknesses are relatively minor. However, it is important to note that the findings of this paper should be interpreted in the context of the limitations of the experiments and methodologies used.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about Temporal Knowledge Graph (TKG) Forecasting Using In-Context Learning. The paper addresses the problem of forecasting future events in a TKG, which is a graph that represents temporal relations between entities. The main contribution of the paper is to explore the use of large language models (LLMs) for TKG forecasting and compare them to diverse baselines and state-of-the-art models. The paper makes several contributions towards a solution to this problem, including proposing a new evaluation methodology for TKG forecasting, introducing a new dataset for TKG forecasting, and conducting experiments to compare the performance of LLMs to other models. The paper also provides insights into the strengths and weaknesses of LLMs for TKG forecasting, which can inform future research in this area.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel approach for temporal knowledge graph forecasting, based on in-context learning with a large language model. The paper makes several contributions towards this problem, including proposing an evaluation methodology, introducing a new dataset, and using those to compare the performance of LLMs to other models.

The authors further provide extensive experiments with their proposed in context learning approach, including on older datasets, showing that their approach is competitive with more complex, supervised systems – even when entity names are replaced with random numbers. This will provide a fruitful starting point for further research.
Decision: Accept-Main