RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Conclusion
In this paper, we examined the forecasting capabilities of in-context learning in large language models.
To this end, we experimented with temporal knowledge graph forecasting benchmarks.
We presented a framework that converts relevant historical facts into prompts and generates ranked link predictions through token probabilities.
Our experimental results demonstrated that without any fine-tuning and only through ICL, LLMs exhibit comparable performance to current supervised TKG methods that incorporate explicit modules to capture structural and temporal information.
We also discovered that using numerical indices instead of entity/relation names does not significantly affect the performance, suggesting that prior semantic knowledge is not critical for overall performance.
Additionally, our analysis indicated that ICL helps the model learn irregular patterns from historical facts, beyond simply making predictions based on the most common or the most recent facts in the given context.
Together, our results and analyses demonstrated that ICL can be a valuable tool for predicting future links using historical patterns, and also prompted further inquiry into the potential of ICL for additional capabilities.
Limitations
There are certain limitations to our experiments.
First, computing resource constraints restrict our experiments to small-scale open-source models.
Second, our methodologies have constraints regarding models where the tokenizer vocabulary comprises solely of single-digit numbers as tokens, such as LLAMA Touvron et al. ( 2023 ) .
The performance of such models exhibits a similar trend in terms of scaling law concerning model size and history length, but these models demonstrate inferior performance compared to other models of the same model size.
Third, our methodologies have certain limitations with respect to link prediction settings.
While real-world forecasting can be performed in the transductive setting, where the answer can be an unseen history, our approach is constrained to the inductive setting, where the answer must be one of the histories observed.
There are further directions that can be pursued.
The first is to explore transductive extrapolation link prediction using LLMs.
The second is to analyze the effects of fine-tuning on the results.
Lastly, there is the opportunity to investigate the new capabilities of ICL.
Acknowledgement
This work was funded in part by the Defense Advanced Research Projects Agency (DARPA) and Army Research Office (ARO) under Contract No. W911NF-21-C-0002 and Contract No. HR00112290106, and with support from the Keston Exploratory Research Award and Amazon.

The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ARO or the U.S. Government.
A.1 Prompt Example
Given the test query at timestamp 571, prompt examples for Index and Lexical are shown in Figure 5 .
Here, we assume the entity dictionary contains “Islamist Militia (Mozambique)” as index 0, “Meluco” as 10, “Namatil” as 36, “Muatide” as 53, “Limala” as 54, and “Nacate” as 55, while relation dictionary contains “Battles” as index 1 and “Violence against civilians” as 4.
Also, the history setting is unidirectional entity setting where the history length is set to 5.
A.2 Prompt Example for Analysis
To assess the ability of LLMs to comprehend the sequential information of historical events, we compare the performance of prompts with and without timestamps (See Section 5.1 Q3).
Figure 6 shows the prompt examples for time-removed and shuffled version of prompts.
A.3 System Instruction for Instruction-tuned models.
For the instruction-model, we use the manual curated system instructions to provide task descriptions and constraint the output format as follow:

You must be able to correctly predict the next {object_label} from a given text consisting of multiple quadruplets in the form of "{time}:[{subject}, {relation}, {object_label}. {object}]" and the query in the form of "{time}:[{subject}, {relation}," in the end. You must generate only the single number for {object_label} without any explanation.
RE-Net Jin et al. ( 2020 )
leverages an auto-regressive architecture that employs a two-step process for learning temporal dependency from a sequence of graphs and local structural dependency from the vicinity.
The model represents the likelihood of a fact occurring as a probability distribution that is conditioned on the sequential history of past snapshots.
RE-GCN Li et al. ( 2021 )
also employs auto-regressive architecture while it utilizes multi-layer relation-aware GCN on each graph snapshot to capture the structural dependencies among concurrent facts.
Furthermore, the static properties of entities such as entity types, are also incorporated via a static graph constraint component to obtain better entity representations.
TANGO Han et al. ( 2021b )
employs auto-regressive architecture as well but the use of continuous-time embedding in encoding temporal and structural information is a distinguishing feature of the proposed method, as opposed to RE-Net Jin et al. ( 2020 ) Li et al. ( 2021 ) and RE-GCN which operate on a discrete level with regards to time.
xERTE Han et al. ( 2021a )
employs an attention mechanism that can effectively capture the relevance of important aspects by selectively focusing on them.
It employs a sequential reasoning approach over local subgraphs. This process begins with the query and iteratively selects relevant edges of entities within the subgraph, subsequently propagating attention along these edges.
After multiple rounds of expansion, the final subgraph represents the interpretable reasoning path towards the predicted outcomes.
TimeTraveler Sun et al. ( 2021 )
employs reinforcement learning for forecasting.
The approach involves the use of an agent that navigates through historical knowledge graph snapshots, commencing from the query subject node.
Thereafter, it sequentially moves to a new node by leveraging temporal facts that are linked to the current node, with the ultimate objective of halting at the answer node.
To accommodate the issue of unseen-timestamp, the approach incorporates a relative time encoding function that captures time-related information when making decisions.
CyGNet Zhu et al. ( 2021 )
leverages the statistical relevance of historical facts, acknowledging the recurrence of events in the temporal knowledge graph datasets.
It incorporates two inference modes, namely Copy and Generation.
The Copy mode determines the likelihood of the query being a repetition of relevant past facts.
On the other hand, the Generation mode estimates the probability of each potential candidate being the correct prediction, using a linear classifier.
The final forecast is obtained by aggregating the outputs of both modes.
TLogic Liu et al. ( 2022 )
mines cyclic temporal logical rules by extracting temporal random walks from a graph.
This process involves the extraction of temporal walks from the graph, followed by a lift to a more abstract, semantic level, resulting in the derivation of temporal rules that can generalize to new data.
Subsequently, the application of these rules generates answer candidates, with the body groundings in the graph serving as explicit and easily comprehensible explanations for the results obtained.
Figure 1: Performance (Hit@1) with and without time, and with shuffling; Performance (Hit@1) with and without time, and with shuffling
Figure 2: Performance (Hit@1) adheres to the scaling law based on the history length.; Performance (Hit@1) adheres to the scaling law based on the history length.
Figure 3: Performance (Hit@1) adheres to the scaling law based on the model size.; Performance (Hit@1) adheres to the scaling law based on the model size.
Figure 4: Performance (Hit@1) Analysis on Prompt Variation. The comparable performance exhibited by both the Index and Lexical models indicates that these models rely heavily on learning patterns and are less influenced by semantic priors.
Moreover, the Unidirectional model typically outperforms the Bidirectional model, suggesting that the robust constraints on historical data enable the model to comprehend observed patterns better.
Finally, the performance of the Entity and Pair models varies depending on the dataset.; Performance (Hit@1) Analysis on Prompt Variation. The comparable performance exhibited by both the Index and Lexical models indicates that these models rely heavily on learning patterns and are less influenced by semantic priors.
Moreover, the Unidirectional model typically outperforms the Bidirectional model, suggesting that the robust constraints on historical data enable the model to comprehend observed patterns better.
Finally, the performance of the Entity and Pair models varies depending on the dataset.
Figure 5: Prompt examples for Index and Lexical settings.; Prompt examples for Index and Lexical settings.
Figure 6: Prompt examples for time-removed and shuffled version.; Prompt examples for time-removed and shuffled version.
Model predictions for an example structured prompt . Given a structured prompt ( top ), large language models forecast the missing fact using next token prediction ( bottom ). G.T Rank indicates the rank of the ground truth response (“ Kansas City ”) within the token probability distribution.
Prompt Example.
Data statistics. Each dataset consists of entities, relations, and historical facts, with the facts within the same time interval identified by the same timestamp. The facts are divided into three subsets based on time, where train < valid < test.
Language Models used in the paper. Exact model size of gpt-3.5-turbo is unknown.
Performance (Hits@K) comparison between supervised models and ICL for single-step (top) and multi-step (bottom) prediction.
The first group in each table consists of supervised models, whereas the second group consists of ICL models, i.e., GPT-NeoX with a history length of 100.
The best model for each dataset in the first group is shown in bold , and the second best is underlined .
Performance (Hits@K) with rule-based predictions. The best model for each dataset is shown in bold .
Performance (Hits@1) between index and lexical for gpt-3.5-turbo .
Performance Comparison (Hits@k) in single-step inference with time-aware filter.
Performance Comparison (Hits@k) in multi-step inference with time-aware filter.
Performance Comparison (Hits@k) between rule-based prediction and ICL in single-step inference with time-aware filter.
Performance Comparison (Hits@k) between rule-based prediction and ICL in multi-step inference with time-aware filter.