RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
In-Context Learning.
ICL is an emergent capability of LLMs that aims to induce a state in the model to perform a task by utilizing contextual input-label examples, without requiring changes to its internal parameters Brown et al. ( 2020 ) .
Formally, in ICL for classification, a prompt is constructed by linearizing a few input-output pair examples (\mathbf{x}_{i},\mathbf{y}_{i}) from the training data.
Subsequently, when a new test input text \mathbf{x}_{\text{test}} is provided, ICL generates the output \mathbf{y}_{\text{test}}\sim\mathcal{P}_{\text{LLM}}(\mathbf{y}_{\text{test}}\mid\mathbf{x}_{1},\mathbf{y}_{1},\ldots,\mathbf{x}_{k},\mathbf{y}_{k},\mathbf{x}_{\text{test}}) where \sim refers to decoding strategy.
Temporal Knowledge Graph Forecasting.
Formally, a TKG, \mathcal{G}=\left(\mathcal{V},\mathcal{R},\mathcal{E},\mathcal{T}\right) , is comprised of a set of entities \mathcal{V} , relations \mathcal{R} , facts \mathcal{E} , and timestamps \mathcal{T} .
Moreover, since time is sequential, \mathcal{G} can be split into a sequence of time-stamped snapshots, \mathcal{G}=\left\{\mathcal{G}_{1},\mathcal{G}_{2},\ldots,\mathcal{G}_{t},\ldots\right\} , where each snapshot, \mathcal{G}_{t}=\left(\mathcal{V},\mathcal{R},\mathcal{E}_{t}\right) , contains the facts at a specific point in time t .
Each fact f\in\mathcal{E}_{t} is a quadruple \left(s,p,o,t\right) where s,o\in\mathcal{V} , p\in\mathcal{R} , and t\in\mathcal{T} .
The TKG forecasting task involves predicting a temporally conditioned missing entity in the future given a query quadruple, \left(?,p,o,t\right) or \left(s,p,?,t\right) , and previous graph snapshots \mathcal{G}_{1:t-1}=\left\{\mathcal{G}_{1},\mathcal{G}_{2},\ldots,\mathcal{G}_{t-1}\right\} .
Here, the prediction typically involves ranking each entity’s assigned score.
In-context Learning for Temporal Knowledge Graph Forecasting
In this work, we focus on 1) modeling appropriate history \mathcal{E}_{q} for a given query quadruple q , 2) converting \{\mathcal{E}_{q},q\} into a prompt \theta_{q} , and 3) employing ICL to get prediction \mathbf{y}_{q}\sim\mathcal{P}_{\text{LLM}}(\mathbf{y}_{q}\mid\theta_{q}) in a zero-shot manner.
Here, the history \mathcal{E}_{q} is modeled on the facts from the previous graph snapshots \mathcal{G}_{1:t-1}=\left\{\mathcal{G}_{1},\mathcal{G}_{2},\ldots,\mathcal{G}_{t-1}\right\} , and we employ token probabilities for \mathbf{y}_{q} to get ranked scores of candidate entities in a zero-shot manner.
In the rest of this section, we study history modeling strategies (Sec 3.1 ), response generation approaches (Sec 3.2 ), prompt construction templates (Sec 3.3 ), and common prediction settings (Sec 3.4 ).
History Modeling
To model the history \mathcal{E}_{q} , we filter facts that the known entity or relation in the query q has been involved in.
Specifically, given the query quadruple q=\left(s,p,?,t\right) under the object entity prediction setting, we experiment with two different aspects of historical facts:
Entity vs. Pair.
Entity includes past facts that contain s , e.g., all historical facts related to Superbowl .
In contrast, Pair includes past facts that contain both s and p , e.g., a list of ( Superbowl, Champion, Year ) as shown in Table 1 .
Unidirectional vs. Bidirectional.
Unidirectional includes past facts \mathcal{F} wherein s ( Entity ) or (s,p) ( Pair ) is in the same position as it is in q ( e.g., Unidirectional & Pair – s and p served as subject and predicate in f\in\mathcal{F} ). Bidirectional includes past facts \mathcal{F} wherein s ( Entity ) or (s,p) ( Pair ) appear in any valid position ( e.g., Bidirectional & Entity – s served as subject or object in f\in\mathcal{F} ).
As an example of the Bidirectional setting, given q= ( Superbowl , Champion , ?, 2023), we include f= ( Kupp , Played , Superbowl , 2022) because s ( i.e., Superbowl ) is present as the object in f .
Moreover, in the Bidirectional setting, to preserve the semantics of the facts in the \mathcal{E}_{q} , we transform the facts where s appears as an object by 1) swapping the object and subject and 2) replacing the relation with its uniquely defined inverse relation ( e.g., (f_{s},f_{p},f_{o},f_{t})\rightarrow(f_{o},f_{p}^{-1},f_{s},f_{t}) ).
Response Generation
Given a prompt \theta_{q} , we pass it to an LLM to obtain the next token probabilities.
Then, we use the obtained probabilities to get a ranked list of entities.
However, obtaining scores for entities based on these probabilities is challenging as they may be composed of several tokens.
To address this challenge, we utilize a mapped numerical label as an indirect logit to estimate their probabilities Lin et al. ( 2022 ) .
Prompt Construction
Given the history \mathcal{E}_{q} and query q , we construct a prompt using a pre-defined template \theta .
Specifically, given the query quadruple q=\left(s,p,?,t\right) under the object entity prediction setting, we present two versions of the template \theta with varying levels of information.
Our assumption is that each entity or relation has an indexed \mathcal{I}(\cdot) ( e.g., 0) and a lexical \mathcal{L}(\cdot) ( e.g., Superbowl ) form (See Table 2 ).
Index.
Index displays every fact, (f_{s},\allowbreak f_{p},\allowbreak f_{o},\allowbreak f_{t})\allowbreak\in\mathcal{E} using the “ f_{t} :[ \mathcal{I}(f_{s}),\mathcal{I}(f_{p}),\allowbreak n_{f_{o}}.\allowbreak\ \mathcal{I}({f_{o}}) ]” template where f_{s},f_{o}\in\mathcal{V} , f_{p}\in\mathcal{R} , f_{t}\in\mathcal{T} , n_{f_{o}} denotes an incrementally assigned numerical label ( i.e., indirect logit), and \mathcal{I} is a mapping from entities to unique indices.
For example, in Table 1 , we can use the following mappings are for the entities and relations, respectively: { Superbowl \rightarrow 0, St Louis \rightarrow 1, Baltimore \rightarrow 2} and { Champion \rightarrow 0}.
The query q is then represented as “ t :[ \mathcal{I}(s) , \mathcal{I}(p) ,” , concatenated to the end of the prompt.
For subject entity prediction, we follow the same procedure from the other side.
Lexical.
Lexical follows the same process as Index but uses lexical form \mathcal{L}(\cdot) of entity and relation.
Each fact in (f_{s},f_{p},f_{o},f_{t})\in\mathcal{E} is represented as “ f_{t} :[ \mathcal{L}(f_{s}),\mathcal{L}(f_{p}),n_{f_{o}} . \mathcal{L}(f_{o}) ]” and the query q is represented as “ t :[ \mathcal{L}(s),\mathcal{L}(p), ”, concatenated to the end of the prompt.
Prediction Setting
All the historical facts in the dataset are split into three subsets, \mathcal{D}_{\text{train}} , \mathcal{D}_{\text{valid}} , and \mathcal{D}_{\text{test}} , based on the chronological order with train < valid < test.
Given this split, during the evaluation phase, the TKG forecasting task requires models to predict over \mathcal{D}_{\text{test}} under the following two settings:
Single Step.
In this setting, for each test query, the model is provided with ground truth facts from past timestamps in the test period.
Hence, after making predictions for a test query in a specific timestamp, the ground truth fact for that query is added to the history before moving to the test queries in the next timestamp.
Multi Step.
In this setting, the model is not provided with ground truth facts from past timestamps in the test period and has to rely on its noisy predictions.
Hence, after making predictions for a test query in a specific timestamp, instead of the ground truth fact for that query, we add the predicted response to the history before moving to the test queries in the next timestamp.
This setting is considered more difficult as the model is forced to rely on its own noisy predictions, which can lead to greater uncertainty with each successive timestamp.
Datasets
For our experiments, we use the WIKI Leblay and Chekol ( 2018 ) , YAGO Mahdisoltani et al. ( 2014 ) , ICEWS14 García-Durán et al. ( 2018 ) , and ICEWS18 Jin et al. ( 2020 ) benchmark datasets with the unified splits introduced in previous studies Gastinger et al. ( 2022 ) .
Additionally, we extract a new temporal forecasting dataset from the Armed Conflict Location & Event Data Project (ACLED) project 2 2 2 https://data.humdata.org/organization/acled which provides factual data of crises in a particular region.
We specifically focus on incidents of combat and violence against civilians in Cabo Delgado from January 1900 to March 2022, using data from October 2021 to March 2022 as our test set.
This dataset aims to investigate whether LLMs leverage prior semantic knowledge to make predictions and how effective they are when deployed in real-world applications.
Table 3 presents the statistics of these datasets.
Evaluation
We evaluate the models on well-known metrics for link prediction: Hits@ k , with k=1,3,10 .
Following (Gastinger et al., 2022 ) , we report our results in two evaluation settings:
1) Raw retrieves the sorted scores of candidate entities for a given query quadruple and calculates the rank of the correct entity; and 2) Time-aware filter also retrieves the sorted scores but removes the entities that are valid predictions before calculating the rank, preventing them from being considered errors.
To illustrate, if the test query is (NBA, Clinch Playoff, ?, 2023) and the true answer is Los Angeles Lakers, there may exist other valid predictions such as (NBA, Clinch Playoff, Milwaukee Bucks, 2023) or (NBA, Clinch Playoff, Boston Celtics, 2023).
In such cases, the time-aware filter removes these valid predictions, allowing for accurate determination of the rank of the “Los Angeles Lakers.”
In this paper, we present performance with the time-aware filter.
Models.
As shown in Table 4 , we perform experiments on four language model families.
Among those, three are open-sourced:
GPT2 Radford et al. ( 2019 ) , GPT-J Wang ( 2021 ) , and GPT-NeoX Black et al. ( 2022 ) .
All models employ the GPT-2 byte level BPE tokenizer Radford et al. ( 2019 ) with nearly identical vocabulary size.
In addition, we use the gpt-3.5-turbo model to analyze the performance of the instruction-tuned models.
However, we do not directly compare this model to other models in terms of size since the actual model size is unknown.
As for the TKG baselines, ( i.e., RE-Net Jin et al. ( 2020 ) , RE-GCN Li et al. ( 2021 ) , TANGO Han et al. ( 2021b ) , xERTE Han et al. ( 2021a ) , TimeTraveler Sun et al. ( 2021 ) , CyGNet Zhu et al. ( 2021 ) , and TLogic Liu et al. ( 2022 ) ), we report the numbers presented in prior research Gastinger et al. ( 2022 ) .
Appendix A.4 provides more details on baseline models.

(a) Single-step

(b) Multi-step
ICL Implementation Details.
We implement our frameworks using PyTorch Paszke et al. ( 2019 ) and Huggingface Wolf et al. ( 2020 ) .
We first collate the facts f\in\mathcal{D}_{test} based on the identical test query to eliminate any repeated inference.
To illustrate, suppose there exist two facts in the test set denoted as \left(s,p,a,t\right) and \left(s,p,b,t\right) in the object prediction scenario.
We consolidate these facts into \left(s,p,[a,b],t\right) and forecast only one for \left(s,p,?,t\right) .
Subsequently, we proceed to generate an output for each test query with history by utilizing the model, obtaining the probability for the first generated token in a greedy approach, and sorting the probability.
The outputs are deterministic for every iteration.
We retain the numerical tokens corresponding to the numerical label n that was targeted, selected from the top 100 probability tokens for each test query.
To facilitate multi-step prediction, we incorporate the top- k predictions of each test query as supplementary reference history.
In this paper, we present results with k=1 .
It is important to acknowledge that the prediction may contain minimal or no numerical tokens as a result of inadequate in-context learning.
This can lead to problems when evaluating rank-based metrics.
To mitigate this, we have established a protocol where the rank of the actual value within the predictions is assigned a value of 100, which is considered incorrect according to our evaluation metric.
For instruction-tuned model, we use the manual curated system instructions in Appendix A.3 .
In-context learning for TKG Forecasting
In this section, we present a multifaceted performance analysis of ICL under Index & Unidirection prompt strategy for both Entity and Pair history.
Q1: How well does ICL fare against the supervised learning approaches?
We present a comparative analysis of the top-performing ICL model against established supervised learning methodologies for TKG reasoning, which are mostly based on graph representation learning.
As evident from the results in Table 5 , GPT-NeoX with a history length of 100 shows comparable performance to supervised learning approaches, without any fine-tuning on any TKG training dataset.
Q2: How do frequency and recency biases affect ICL’s predictions?
To determine the extent to which LLMs engage in pattern analysis, beyond simply relying on frequency and recency biases, we run a comparative analysis between GPT-NeoX and heuristic-rules ( i.e., frequency & recency ) on the ICEWS14 dataset, with history length set to 100. frequency identifies the target that appears most frequently in the provided history while recency selects the target associated with the most recent fact in the provided history.
The reason for our focus on ICEWS is that each quadruple represents a single distinct event in time.
In contrast, the process of constructing YAGO and WIKI involves converting durations to two timestamps to display events across the timeline.
This step has resulted in recency heuristics outperforming all of the existing models, showcasing the shortcoming of existing TKG benchmarks (See Appendix A.5 ).
The experimental results presented in Table 6 demonstrate that ICL exhibits superior performance to rule-based baselines.
This finding suggests that ICL does not solely rely on specific biases to make predictions, but rather it actually learns more sophisticated patterns from historical data.
Q3: How does ICL use the sequential and temporal information of events?
To assess the ability of LLMs to comprehend the temporal information of historical events, we compare the performance of prompts with and without timestamps.
Specifically, we utilize the original prompt format, “ f_{t} :[ \mathcal{I}(f_{s}),\allowbreak\mathcal{I}(f_{r}),\allowbreak n_{f_{o}}.\ \mathcal{I}(f_{o}) ]”, and the time-removed prompt format, “[ \mathcal{I}(f_{s}),\mathcal{I}(f_{r}),n_{f_{o}}.\ \mathcal{I}(f_{o}) ]”, make the comparison (See Appendix A.2 ).
Additionally, we shuffle the historical facts in the time-removed prompt format to see how the model is affected by the corruption of sequential information.
Figure 1 shows that the absence of time reference can lead to a deterioration in performance, while the random arrangement of historical events may further exacerbate this decline in performance.
This observation implies that the model has the capability to forecast the subsequent event by comprehending the sequential order of events.
Q4: How does instruction-tuning affect ICL’s performance?
To investigate the impact of instruction-tuning on ICL, we employ the gpt-3.5-turbo model with manually curated system instruction detailed in Appendix 4.4 .
Since the size of this model is not publicly disclosed, it is challenging to make direct comparisons with other models featured in this paper.
Moreover, since this model does not provide output probabilities, we are only able to report the Hit@1 metric.

Table 7 showcases that the performance of the lexical prompts exceeds that of the index prompts by 0.024, suggesting that instruction-tuned models can make better use of semantic priors.
This behavior is different from the other foundation LLMs, where the performance gap between the two prompt types was insignificant (See Figure 4 (a)).
Q5: How does history length affect ICL’s performance?
To evaluate the impact of the history length provided in the prompt, we conduct a set of experiments using varying history lengths.
For this purpose, we use the best performing prompt format for each benchmark, i.e., Entity for WIKI, YAGO, ICEWS18, and Pair for ICEWS14.
Our results, as shown in Figure 2 , indicate a consistent improvement in performance as the history length increases.
This suggests that the models learn better as additional historical facts are presented.
This observation is connected to few-shot learning in other domains, where performance improves as the number of examples per label increases.
However, in our case, the historical patterns presented in the prompt do not explicitly depict the input-label mapping but rather aid in inferring the next step.
Q6: What is the relation between ICL’s performance and model size?
Here, we analyze the connection between model size and performance.
Our results, as presented in Figure 3 , conform to the expected trend of better performance with larger models.
This finding aligns with prior works showing the scaling law of in-context learning performance.
Our findings are still noteworthy since they show how scaling model size can facilitate more powerful pattern inference for forecasting tasks.
Prompt Construction for TKG Forecasting
To determine the most effective prompt variation, we run a set of experiments on all prompt variations, using GPT-J Wang ( 2021 ) and under the single-step setting.
Comprehensive results for prompt variations can be found in Appendix A.5 .
Index vs. Lexical
Our first analysis compares the performance of index and lexical prompts.
This investigation aims to determine whether the model relies solely on input-label mappings or if it also incorporates semantic priors from pre-training to make predictions.
Our results (Figure 4 (a)) show that the performance is almost similar ( \pm 4e-3 on average) across the datasets.
This finding is aligned with previous studies indicating that foundation models depend more on input-label mappings and are minimally impacted by semantic priors Wei et al. ( 2023 ) .
Unidirectional vs. Bidirectional
We next analyze how the relation direction in the history modeling impacts the performance.
This analysis aims to ascertain whether including historical facts, where the query entity or pair appears in any position, can improve performance by offering a diverse array of historical facts.
Our results (Figure 4 (b)) show that there is a slight decrease in performance when Bidirectional history is employed, with a significant drop in performance observed particularly in the ICEWS benchmarks.
These observations may be attributed to the considerably more significant number of entities placed in both subject and object positions in ICEWS benchmarks than YAGO and WIKI benchmarks (See Appendix A.5 ).
This finding highlights the necessity of having robust constraints on the historical data for ICL to comprehend the existing pattern better.

(a) Index vs. Lexical \foreachitem \z ∈ \clarg []

\z

(b) Unidirection vs. Bidirection \foreachitem \z ∈ \clarg []

\z

(c) Entity vs. Pair \foreachitem \z ∈ \clarg []

\z
Entity vs. Pair
Finally, we examine the impact of the history retrieval query on performance.
Our hypothesis posits that when the query is limited to a single entity, we can incorporate more diverse historical facts.
Conversely, when the query is a pair, we can acquire a more focused set of historical facts related to the query.
Our results (Figure 4 (c)) indicate that the performance of the model is dependent on the type of data being processed.
Specifically, the WIKI and ICEWS18 benchmarks perform better when the query is focused on the entity, as a broader range of historical facts is available.
In contrast, the ICEWS14 benchmark performs better when the query is focused on pairs, as the historical facts present a more focused pattern.
Event Forecasting.
Forecasting is a complex task that plays a crucial role in decision-making and safety across various domains Hendrycks et al. ( 2021 ) .
To tackle this challenging task, researchers have explored various approaches, including statistical and judgmental forecasting Webby and O’Connor ( 1996 ); Armstrong ( 2001 ); Zou et al. ( 2022 ) . Statistical forecasting involves leveraging probabilistic models Hyndman and Khandakar ( 2008 ) or neural networks Li et al. ( 2018 ); Sen et al. ( 2019 ) to predict trends over time-series data.
While this method works well when there are many past observations and minimal distribution shifts, it is limited to numerical data and may not capture the underlying causal factors and dependencies that affect the outcome.
On the other hand, judgmental forecasting involves utilizing diverse sources of information, such as news articles and external knowledge bases, to reason and predict future events.
Recent works have leveraged language models to enhance reasoning capabilities when analyzing unstructured text data to answer forecasting inquiries Zou et al. ( 2022 ); Jin et al. ( 2021 ) .
Temporal Knowledge Graph.
Temporal knowledge graph (TKG) reasoning models are commonly employed in two distinct settings, namely interpolation and extrapolation , based on the facts available from t_{0} to t_{n} .
(1) Interpolation aims to predict missing facts within this time range from t_{0} to t_{n} , and recent works have utilized embedding-based algorithms to learn low-dimensional representations for entities and relations to score candidate facts Leblay and Chekol ( 2018 ); García-Durán et al. ( 2018 ); Goel et al. ( 2020 ); Lacroix et al. ( 2020 ) ;
(2) Extrapolation aims to predict future facts beyond t_{n} .
Recent studies have treated TKGs as a sequence of snapshots, each containing facts corresponding to a timestamp t_{i} , and proposed solutions by modeling multi-relational interactions among entities and relations over these snapshots using graph neural networks Jin et al. ( 2020 ); Li et al. ( 2021 ); Han et al. ( 2021b , a ) , reinforcement learning Sun et al. ( 2021 ) or logical rules Zhu et al. ( 2021 ); Liu et al. ( 2022 ) .
In our work, we focus on the extrapolation setting.
In-context Learning.
In-context learning (ICL) has enabled LLMs to accomplish diverse tasks in a few-shot manner without needing parameter adjustments Brown et al. ( 2020 ); Chowdhery et al. ( 2022 ) .
In order to effectively engage in ICL, models can leverage semantic prior knowledge to accurately predict labels following the structure of in-context exemplars Min et al. ( 2022 ); Razeghi et al. ( 2022 ); Xie et al. ( 2022 ); Chan et al. ( 2022 ); Hahn and Goyal ( 2023 ) , and learn the input-label mappings from the in-context examples presented Wei et al. ( 2023 ) .
To understand the mechanism of ICL, recent studies have explored the ICL capabilities of LLMs with regards to the impact of semantic prior knowledge by examining their correlation with training examples Min et al. ( 2022 ); Razeghi et al. ( 2022 ); Xie et al. ( 2022 ) , data distribution Chan et al. ( 2022 ) , and language compositionality Hahn and Goyal ( 2023 ) in the pre-training corpus.
Other recent works show that LLMs can actually learn input-label mappings from in-context examples by showing the transformer models trained on specific linear function class is actually predicting accurately on new unseen linear functions Garg et al. ( 2022 ) .
More recently, there is a finding that large-enough models can still do ICL using input-label mappings when semantic prior knowledge is not available Wei et al. ( 2023 ) .