Towards Explainable Temporal Reasoning in Large Language Models: A Structure-Aware Generative Framework
Authors: Zihao Jiang, Ben Liu, Miao Peng, Wenjie Xu, Yao Xiao, Zhenyan Shan, Min Peng
Accepted at: acl25

Towards Explainable Temporal Reasoning in Large Language Models: A Structure-Aware Generative Framework

Zihao Jiang1*, Ben Liu1*, Miao Peng2, Wenjie Xu1, Yao Xiao1, Zhenyan Shan1, Min Peng1†
1School of Computer Science, Wuhan University, China
2The Hong Kong University of Science and Technology (Guangzhou)
{jiangzihao,liuben123,vingerxu,y.xiao,bbcavendish,pengm}@whu.edu.cn
mpeng885@connect.hkust-gz.edu.cn

Abstract

While large language models (LLMs) show great potential in temporal reasoning, most existing work focuses heavily on enhancing performance, often neglecting the explainable reasoning processes underlying the results. To address this gap, we introduce a comprehensive benchmark covering a wide range of temporal granularities, designed to systematically evaluate LLMs’ capabilities in explainable temporal reasoning. Furthermore, our findings reveal that LLMs struggle to deliver convincing explanations when relying solely on textual information. To address challenge, we propose GETER, a novel structure-aware generative framework that integrates Graph structures with text for Explainable TEmporal Reasoning. Specifically, we first leverage temporal knowledge graphs to develop a temporal encoder that captures structural information for the query. Subsequently, we introduce a structure-text prefix adapter to map graph structure features into the text embedding space. Finally, LLMs generate explanation text by seamlessly integrating the soft graph token with instruction-tuning prompt tokens. Experimental results indicate that GETER achieves state-of-the-art performance while also demonstrating its effectiveness as well as strong generalization capabilities. Our dataset and code are available at https://github.com/carryTatum/GETER.

Figure 1: An illustration of existing temporal reasoning works highlights the lack of focus on explanations behind the reasoning. Meanwhile, LLMs often struggle to generate convincing answers due to hallucinations.

depend on temporal data, including search engine recommendations (Bogina et al., 2023) and news article aggregation (Wu et al., 2025).

Recently, large language models (LLMs) have demonstrated remarkable performance in tackling complex tasks (Wei et al., 2022; Huang and Chang, 2023; OpenAI, 2023; Peng et al., 2025; Liu et al., 2025b). Building on this success, recent studies have increasingly focused on exploring the TR capabilities of LLMs. These works primarily adopt general approaches to evaluate and enhance the TR capabilities of LLMs. For instance, Tan et al. (2023) and Wei et al. (2023) design time-sensitive queries to benchmark LLMs, while Wang and Zhao (2024) and Chu et al. (2024) extend these efforts by using prompting strategies like in-context learning (ICL) and Chain-of-Thought (CoT) reasoning for comprehensive evaluation. Furthermore, Lee et al. (2023) and Xia et al. (2024) employ ICL with prompts containing intermediate reasoning steps to guide models, while Liao et al. (2024) and Luo et al. (2024) adopt fine-tuning methods, training LLMs on reasoning process texts to enable them to produce accurate answers.

Although existing methods have explored LLMs’

1 Introduction

Temporal reasoning (TR) is a fundamental cognitive skill essential for understanding complex tasks like planning and causal relation discovery (Xiong et al., 2024). In natural language processing (NLP), temporal reasoning refers to a model’s capability to effectively comprehend, represent, and predict time-sensitive contexts (Yang et al., 2024b). This capability is critical for real-world applications that
potential in temporal reasoning, they exceedingly focus on improving performance, often overlooking the explainable reasoning processes behind the results, as illustrated in Figure 1(a). The study of explainable temporal reasoning is crucial, as it promotes transparency, enhances effectiveness, and fosters trust in understanding temporal dynamics. Moreover, with their impressive semantic understanding and generation capabilities, LLMs are uniquely positioned to address the challenges of explainable reasoning (Wang et al., 2023; Ma et al., 2024), as they can generate flexible, human-readable reasoning processes. Therefore, we posit the following research question to guide our study:
Can LLMs effectively make accurate predictions and clearly explaining their reasoning processes in complex temporal reasoning scenarios?

To address this challenge, we propose the ETR benchmark, a comprehensive benchmark for explainable temporal reasoning. Specifically, ETR consists of five datasets covering a wide range of temporal granularities (minutes, days, and years). Each instance is represented as a triple of <query text, reasoning chains text, explanation text> where the query and related reasoning chains are derived from Temporal Knowledge Graphs (TKGs). The explanation text is synthesized using GPT-4o (OpenAI, 2023) with constrained generation prompt protocols, taking the query and reasoning chains as input. The resulting explanation text effectively integrates both the original gold prediction and the underlying reasoning processes. ETR aims to challenge LLMs not only to predict future events from the given reasoning chains text but also to generate explanations of their reasoning processes.

Building on this benchmark, we identify that the key to achieving explainable temporal reasoning lies in enabling LLMs to capture structured patterns that reflect the relationships and dynamics between events over time. As shown in Figure 1(b), our findings reveal that LLMs struggle to deliver convincing explanations when relying solely on textual information, a challenge (e.g. hallucinations) also highlighted in previous analyses (He et al., 2024; Liu et al., 2025a). To address this challenge, we propose a novel structure-aware generative framework GETER, which advances explainable temporal reasoning by effectively bridging the gap between graph structures and text. Specifically, we leverage TKGs to develop a temporal encoder that captures structural information. Subsequently, the encoder converts the query and reasoning chains into a soft graph token, which is then mapped into the LLM’s text space via a lightweight adapter. Finally, LLM can generate explanation text by integrating the soft graph token with instruction-tuning prompt tokens, seamlessly combining structural and contextual semantic information. Experimental results show that our proposed GETER achieves state-of-the-art performance. In summary, the contributions of this paper are as follows:

• We introduce ETR, a comprehensive benchmark covering a wide range of temporal granularities for systematically evaluating LLMs’ explainable temporal reasoning.

• To bridge the gap between graph structures and text, we propose GETER, a novel structure-aware generative framework which leverages a lightweight structure-text adapter to enhance LLMs’ explainable temporal reasoning capabilities.

• Our GETER achieves state-of-the-art performance on five datasets using widely-used LLMs, demonstrating the superiority of our model. Further experiments reveal the effectiveness and strong generalization ability of GETER.

2 Related Work

2.1 LLMs for Temporal Reasoning

With the rapid advancement of LLMs, research has increasingly focused on evaluating and enhancing their temporal reasoning capabilities. Existing studies primarily leverage the parametric knowledge of LLMs to assess and improve performance. For instance, several studies (Tan et al., 2023; Wei et al., 2023) design time-sensitive queries to benchmark LLMs, while others (Wang and Zhao, 2024; Chu et al., 2024) extend these efforts to diverse temporal reasoning tasks using general evaluation methods. Additionally, some methods (Lee et al., 2023; Xia et al., 2024) utilize in-context learning by providing prompts with demonstrations of intermediate reasoning steps to guide the model, whereas fine-tuning methods (Liao et al., 2024; Luo et al., 2024) train LLMs on reasoning texts to enable them to generate accurate final answers. Despite these advancements, most efforts focus on improving performance through parametric knowledge, with limited emphasis on explanation.
2.2 Explainable Temporal Reasoning

In temporal reasoning tasks, explainability is crucial for ensuring transparency, trust, and reliability. Existing works for explainable temporal reasoning primary fall into two categories: logic rule-based methods and reinforcement learning-based methods. Logic rule-based methods (Liu et al., 2022b; Lin et al., 2023; Mei et al., 2022) ensure explainability through explicit rule templates but struggle to balance generalization and explainability in complex scenarios. Reinforcement learning-based methods (Han et al., 2021; Sun et al., 2021) construct reasoning paths guided by predefined reward mechanisms. However, their explainability is limited by the implicit nature of their decision-making processes. In contrast, LLMs offer unique advantages for explainable reasoning by leveraging semantic understanding and generation capabilities (Tan et al., 2023, 2024), enabling more flexible and human-readable reasoning processes. While Yuan et al. (2024) conduct a preliminary exploration of LLM explainability, their work overlooks finer-grained temporal dimensions evaluation and fails to enhance LLMs through the integration of temporal graph features.

3 Proposed ETR Benchmark

3.1 Problem Definition

Temporal Knowledge Graphs (TKGs) \( \mathcal{G} \) are represented as a sequence of KGs (\( \mathcal{G}_0, \mathcal{G}_1, \ldots, \mathcal{G}_l \)) arranged by timestamp \( t \). Let \( \mathcal{G} = (\mathcal{E}, \mathcal{R}, \mathcal{F}) \) be a TKG instance, where \( \mathcal{E}, \mathcal{R}, \mathcal{F} \) represent the set of entities, relations and facts, respectively. Each fact can be represented as a quadruple \( (e_s, r, e_o, t) \in \mathcal{F} \), where subject and object \( e_s, e_o \in \mathcal{E} \), relation \( r \in \mathcal{R} \). Explainable temporal reasoning aims to challenge LLMs to predict future events based on reasoning chains and generate explanations of their reasoning. Formally, given reasoning chains \( \mathcal{C} \) consisting of facts \( \mathcal{F}_{[t_q-w, t_q]} \), the task is to predict the probability that a query \( q \) will occur at future time \( t_q \), where \( w \) is the window size. Based on this probability, the model classifies \( q \) into one of three categories: "Yes", "No", or "Unsure", and generates an explanation for its prediction. The prediction and explanation together form the final output \( A \). To train and evaluate the model, we define two types of instances: training instances \( \mathcal{T}_{train} \) and test instances \( \mathcal{T}_{test} \). These instances follow the extrapolation condition (Jin et al., 2020), where the training time (\( t_{train} \)) strictly precedes the test time (\( t_{test} \)), i.e., \( t_{train} < t_{test} \). Each instance \( \mathcal{T}_i \) consists of the following components: the query text \( Q_i \), the input reasoning chains text \( C_i \), and explanation text \( A_i \), formally defined as: \( \mathcal{T}_i = \{Q_i, C_i, A_i\} \).

3.2 Pipeline

As illustrated in Figure 2, we present ETR, a comprehensive benchmark for Explainable Temporal Reasoning. To accomplish this goal, we extract reasoning chains for each query and generate explanation text using GPT-4o. Additionally, we sample negative and neutral examples in a similar manner to provide a thorough evaluation of the LLMs. The detailed construction process is outlined as follows.

3.2.1 Reasoning Chains Text Construction

To construct reasoning chains text, given a query \( q = (e_s, r, e_o, t_q) \), we extract the graph reasoning chains \( \mathcal{C}(e_s, e_o) \) associated with entities \( e_s \) and \( e_o \) using a breadth-first search (BFS) methods (Jiang et al., 2023). The extraction process considers reasoning chains occurring within the time interval \([t_q - w, t_q)\) and is formalized as follows:

\[
\mathcal{C}(e_s, e_o) \leftarrow \bigwedge_{i=1}^{l} (E_i, R_i, E_{i+1}, T_i),
\]

where \( E_1 = e_s, E_{l+1} = e_o, l \in \{1, 2\} \) denotes the path length. Here, \( E_i \) represents the entity, \( R_i \) denotes the relation, and \( T_i \) is the corresponding timestamp. Once these reasoning chains \( \mathcal{C}(e_s, e_o) \) are extracted, they are converted into natural language sentences to form the input text \( C_i \).

3.2.2 Explanation Generation

Based on the query \( q = (e_s, r, e_o, t_q) \) and reasoning chains \( \mathcal{C}(e_s, e_o) \), we employ a template to generate an initial explanation text \( A_i \) as follows:

We predict that [e_s] [r] [e_o] will happen on [t_q]. Here are the reasoning steps: \( \mathcal{C}(e_s, e_o) \).

However, not all reasoning chains can adequately justify the occurrence of the given query, and the template-generated explanation text often exhibits issues such as incoherence, unnatural flow, and insufficient logical consistency, ultimately failing to provide a clear and compelling rationale. To address these limitations, we employ GPT-4o to enhance the quality of the final explanations \( A_i \), guided by the prompt provided in Appendix A.1
Figure 2: The pipeline of generating ETR benchmark.

<table>
  <tr>
    <th>Dataset</th>
    <th>Time Granularity</th>
    <th>Type</th>
    <th>|Pos. |</th>
    <th>|Neg. |</th>
    <th>|Neu. |</th>
    <th>Total</th>
  </tr>
  <tr>
    <td rowspan="2">ICEWS14</td>
    <td rowspan="2">1 day</td>
    <td>Train</td>
    <td>5000</td>
    <td>4800</td>
    <td>4500</td>
    <td>14300</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>800</td>
    <td>700</td>
    <td>600</td>
    <td>2100</td>
  </tr>
  <tr>
    <td rowspan="2">ICEWS05-15</td>
    <td rowspan="2">1 day</td>
    <td>Train</td>
    <td>4500</td>
    <td>4400</td>
    <td>4200</td>
    <td>13100</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>720</td>
    <td>680</td>
    <td>660</td>
    <td>2060</td>
  </tr>
  <tr>
    <td rowspan="2">ICEWS18</td>
    <td rowspan="2">1 day</td>
    <td>Train</td>
    <td>4400</td>
    <td>4200</td>
    <td>4000</td>
    <td>12600</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>750</td>
    <td>700</td>
    <td>650</td>
    <td>2100</td>
  </tr>
  <tr>
    <td rowspan="2">GDELT</td>
    <td rowspan="2">15 minutes</td>
    <td>Train</td>
    <td>4800</td>
    <td>4600</td>
    <td>4400</td>
    <td>13800</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>800</td>
    <td>700</td>
    <td>650</td>
    <td>2150</td>
  </tr>
  <tr>
    <td rowspan="2">WIKI</td>
    <td rowspan="2">1 year</td>
    <td>Train</td>
    <td>2482</td>
    <td>2504</td>
    <td>2342</td>
    <td>7328</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>347</td>
    <td>286</td>
    <td>316</td>
    <td>949</td>
  </tr>
</table>

Table 1: Statistics of the ETR benchmark. |Pos.|, |Neg.|, and |Neu.| denote the number of positive, negative, and neutral samples, respectively.

3.2.3 Negative and Neutral samples

To evaluate the ability of LLMs in explainable temporal reasoning, particularly in inferring logical correlations between the queries and historical facts, we introduce negative and neutral samples. Negative samples are used to test the model’s ability to reject logically inconsistent or counterfactual scenarios, while neutral samples assess its capacity to infer uncertainty and ambiguity in scenarios with insufficient evidence.

Negative Samples. Negative samples represent counterfactual queries. To achieve this goal, we modify the positive query quadruple \( q = (e_s, r, e_o, t_q) \) by replacing \( o \) with a different entity \( o' \), resulting in \( q' = (e_s, r, e'_o, t_q) \), where \( q' \notin \mathcal{F} \). This creates a hard negative sample that introduces factual inconsistencies. Additionally, we derive negative sample reasoning chains \( \mathcal{C}(e_s, e'_o) \) as defined in Equation 1. Following a similar process for positive samples, we design the corresponding prompt for GPT-4o, detailed in Appendix A.2.

Neutral Samples. In neutral samples, LLMs are expected to predict "unsure" for the query, as the reasoning chain lacks sufficient evidence to support or refute it. To construct these samples, we replace the positive query relation \( q = (e_s, r, e_o, t_q) \) with \( q'' = (e_s, r', e_o, t_q) \), where \( r' \) is a semantically neutral relation to \( r \) and \( q'' \notin \mathcal{F} \). The neutral relation \( r' \) is identified using a Natural Language Inference (NLI) model (He et al., 2023), which classifies relationships into entailment, contradiction, and neutral. We select \( r' \) as neutral only if the NLI model assigns \( P(\text{neutral}) > \tau \), where \( \tau \) is a predefined threshold. The reasoning chains for neutral samples, \( \mathcal{C}(e_s, e_o) \), are consistent with those of positive samples. Details of the GPT-4o prompt are provided in Appendix A.3.

3.3 Benchmark Summary and Evaluation

As summarized in Table 1, the proposed benchmark covers a wide range of temporal granularities. To achieve this goal, we use five widely adopted temporal knowledge graph reasoning datasets: ICEWS14 (García-Durán et al., 2018), ICEWS18 (Han et al., 2021), ICEWS05-15 (García-Durán et al., 2018)), GDELT (Liao et al., 2024), and WIKI (Leblay and Chekol, 2018). To ensure the quality of the dataset, we filter out invalid answers and conduct human evaluation. Further details refer to Appendix A.5.
Figure 3: The overall framework of GETER. To bridge the gap between graph and text, we leverage TKGs to train a temporal encoder that captures structural information. Subsequently, the query and reasoning chains are encoded into a soft graph token, which is mapped into the text embedding space through a lightweight adapter. Finally, the target explanation text is generated using the soft graph token and related instruction tuning prompt tokens.

4 Methodology

In this section, we present GETER, a novel structure-aware generative framework that integrates Graph structures with text for Explainable TEmporal Reasoning. The overall architecture of our proposed model is illustrated in Figure 3. Specifically, we first leverage a temporal encoder (Sec. 4.1) to obtain structural embeddings for both entities and relations. Subsequently, we introduce a structure-text prefix adapter as described in Sec. 4.2 to map graph structure features into the text embedding space. Finally, we apply an instruction-tuning strategy (Sec. 4.3) to effectively adapt the model to the explainable temporal reasoning task.

4.1 Indexing

We aim to harness the semantic understanding and temporal reasoning capabilities of LLMs for the explainable temporal reasoning task. However, relying solely on LLMs within a text-based prediction framework to infer correlations between queries and reasoning chains inevitably neglects the structural information in the TKG \( \mathcal{G} \). To address this, we first employ a temporal encoder (TKG model), such as RE-GCN (Li et al., 2021), which utilizes the message-passing mechanism of GNNs to effectively capture structural patterns, to generate the structural representation \( s_n \):

\[
s_n = TemporalEncoder(x_n|\mathcal{G}) \in \mathbb{R}^{d_s},
\]

where \( x_n \) represents the initialized embedding of entity or relation \( n \), and \( d_s \) denotes the dimension of the structural embedding. In this way, we get entity embedding matrix \( \mathbf{E} \in \mathbb{R}^{|\mathcal{E}| \times d_s} \) and relation embedding matrix \( \mathbf{R} \in \mathbb{R}^{|\mathcal{R}| \times d_s} \), respectively.

4.2 Structure-Text Adapter

To effectively integrate structure-based embeddings of entities and relations with textual information, we propose a soft prompt strategy that combines structural and textual features in a contextualized manner. Specifically, given the query \( q = (e_s, r, e_o, t) \) and reasoning chains \( \mathcal{C}(e_s, e_o) \), we compute the representation of the query and reasoning chains via parameter-free message passing on the encoded structural features. The resulting graph representation is then projected into the embedding space of LLMs using a trainable projection matrix \( \mathbf{W}_p \in \mathbb{R}^{3d_s \times d_x} \), as follows:

\[
\mathbf{S}_{\mathcal{C}(e_s, e_o)} = \sum_{(e'_s, r', e'_o) \in \mathcal{C}(e_s, e_o)} (\mathbf{e}'_s \| \mathbf{r}' \| \mathbf{e}'_o),
\]
\[
\mathbf{S}_{graph} = \mathbf{W}_p \cdot \frac{\mathbf{S}_{\mathcal{C}(e_s, e_o)} + \mathbf{S}_q}{|\mathcal{C}(e_s, e_o)| + 1},
\]

where \( \| \) denotes concatenation, \( \mathbf{S}_q = (\mathbf{e}_s \| \mathbf{r} \| \mathbf{e}_o) \), \( \mathbf{S}_{graph} \) is the projected graph representation, and
\( d_e \) denotes the dimension of embedding space of LLMs. \( e_s' \in \mathbb{R}^{1 \times d_e} \), \( r' \in \mathbb{R}^{1 \times d_e} \), and \( o' \in \mathbb{R}^{1 \times d_e} \) are the embeddings of the subject entity, relation, and object entity, respectively. This straightforward linear mapping is adopted due to its proven effectiveness in aligning graph-based and textual representations (He et al., 2024; Liu et al., 2025a).

4.3 Instruction Tuning Strategy
The instruction tuning process is designed to adapt the reasoning behavior of the LLM to align with the specific constraints and requirements of the explainable temporal reasoning task. To facilitate the generation of the target explainable text, we provide the corresponding query text \( Q \) and reasoning chains text \( C(e_s, e_o) \) as inputs to the LLM, which produce their textual representations, denoted as \( X = X_Q + X_C \). Let \( X \in \mathbb{R}^{|X| \times d_e} \) represent the textual content embeddings of the input, where \( |X| \) denotes the token length of \( X \). The final input to the LLM is constructed by concatenating the soft graph token embeddings \( S_{graph} \) (as described in Sec. 4.2) with the textual embedding, expressed as \( X' = S_{graph} \| X \). Lastly, our optimization objective is to maximize the likelihood of generating the target explanation text \( Y_A \):

\[
P(Y_A|X', X_I) = \prod_{j=1}^L P_\theta(y_j|X', X_I, Y_{<j}),
\]

where \( X_I \) denotes the representation of instruction tokens, \( L \) is the token length of the target explanation text, and \( Y_{<j} \) represents the prefix of the missing explanation text sequence \( Y_A \) up to position \( j-1 \). Considering the overhead of updating all parameters in LLMs, we adopt Low-Rank Adaptation (LoRA) technique (Hu et al., 2022) for its effectiveness (Liu et al., 2022a). The example of instruction data can be seen in Appendix A.4.

5 Experiments

5.1 Experiments Setup
Baselines. We evaluate our benchmark with four representative LLMs: GPT-4o (OpenAI, 2023), Llama3-8B-Instruct (Dubey et al., 2024), Qwen2.5-7B-Instruct (Yang et al., 2024a), and Mistral-7B-Instruct-v0.3 (Jiang et al., 2024). For our framework, we adopt open-source LLMs as backbones and use RE-GCN (Li et al., 2021) as temporal encoder. Implementation details refer to Appendix B. Furthermore, performance comparisons with four graph-based methods (RE-GCN, CEN (Li et al., 2022), CENET (Xu et al., 2023), and SiMFy (Liu et al., 2023)) are presented in Appendix C.1.

Metrics. We evaluate explainable temporal reasoning capabilities of models in two aspects: prediction and explanation. For prediction, we report precision, recall, and F1 scores. For explanation, we employ BLEU (Papineni et al., 2002) (4-gram), ROUGE (Lin, 2004) (ROUGE-L), METEOR (Banerjee and Lavie, 2005), and BertScore (Zhang et al., 2020) to measure the similarity between model-generated explanations and the ground truth in the test set.

5.2 Main results
In our experiments, we compare GETER with two model configurations: 1) Inference-only (zero-shot): Utilizing a frozen LLM to generate explanations directly without any additional training. 2) Tuned-only: Fine-tuning the LLM using LoRA to enhance its performance on the task. Table 2 presents the prediction results, while Table 3 summarizes the explanation results. Overall, GETER demonstrates consistent and significant improvements across most metrics on both datasets, highlighting the effectiveness of the proposed approach. Further comparisons with graph-based methods are provided in Appendix C.1.

Prediction Results. Table 2 reports the prediction evaluation metrics for each LLM. The results show that both the Tuned-only setting and GETER methods significantly outperform Inference-only setting methods. This performance gap arises because fine-tuning allows models to better capture task-specific temporal patterns and improve logical consistency. Notably, GETER with Mistral demonstrates substantial improvements of 97.95%, 95.55%, and 101.58% in overall F1 scores compared to the best-performing Inference-only model GPT-4o. Furthermore, compared to Tuned-only methods, GETER with Mistral achieves overall F1 score improvements of 11.10%, 10.71%, and 7.54% across the three datasets. These results further underscore that GETER can effectively leverage the structural information of TKGs to enhance its explainable temporal reasoning capabilities.

Explanation Results. Table 3 presents the evaluation metrics for explanation generation. GETER demonstrates remarkable improvements across all key metrics. Specifically, compared to GPT-4o, GETER with Mistral achieves substantial enhancements in BLEU-4 scores across the three datasets,
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Types</th>
    <th colspan="4">Datasets ICEWS14</th>
    <th colspan="4">GDELT</th>
    <th colspan="4">ICEWS05-15</th>
  </tr>
  <tr>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>zero-shot w/o chains text</td>
    <td>53.13</td>
    <td>20.12</td>
    <td>12.95</td>
    <td>30.61</td>
    <td>19.08</td>
    <td>43.78</td>
    <td>25.50</td>
    <td>29.06</td>
    <td>55.45</td>
    <td>26.33</td>
    <td>15.47</td>
    <td>33.03</td>
  </tr>
  <tr>
    <td></td>
    <td>zero-shot</td>
    <td>60.10</td>
    <td>9.54</td>
    <td>48.56</td>
    <td>39.95</td>
    <td>42.74</td>
    <td>37.16</td>
    <td>29.21</td>
    <td>36.83</td>
    <td>61.63</td>
    <td>11.89</td>
    <td>47.16</td>
    <td>40.58</td>
  </tr>
  <tr>
    <td>Llama3-8B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>21.69</td>
    <td>27.11</td>
    <td>35.42</td>
    <td>27.42</td>
    <td>1.95</td>
    <td>33.13</td>
    <td>39.44</td>
    <td>23.44</td>
    <td>11.75</td>
    <td>28.98</td>
    <td>39.41</td>
    <td>26.30</td>
  </tr>
  <tr>
    <td></td>
    <td>zero-shot</td>
    <td>56.51</td>
    <td>10.20</td>
    <td>26.20</td>
    <td>26.70</td>
    <td>53.48</td>
    <td>15.62</td>
    <td>29.47</td>
    <td>33.90</td>
    <td>57.14</td>
    <td>17.50</td>
    <td>14.03</td>
    <td>30.24</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>62.27</td>
    <td>36.98</td>
    <td>48.17</td>
    <td>49.81</td>
    <td>61.94</td>
    <td>7.18</td>
    <td>69.14</td>
    <td>46.29</td>
    <td>65.56</td>
    <td>58.36</td>
    <td>88.02</td>
    <td>57.47</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>74.60</td>
    <td>65.64</td>
    <td>75.62</td>
    <td>71.90</td>
    <td>62.86</td>
    <td>28.57</td>
    <td>78.86</td>
    <td>56.42</td>
    <td>71.51</td>
    <td>71.42</td>
    <td>88.56</td>
    <td>78.66</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>75.07</td>
    <td>67.38</td>
    <td>81.15</td>
    <td>74.25</td>
    <td>62.62</td>
    <td>68.74</td>
    <td>88.73</td>
    <td>72.51</td>
    <td>78.58</td>
    <td>75.95</td>
    <td>91.48</td>
    <td>81.84</td>
  </tr>
  <tr>
    <td>ΔImprove</td>
    <td>6.68%</td>
    <td>16.05%</td>
    <td>19.36%</td>
    <td>13.20%</td>
    <td>-0.38%</td>
    <td>140.54%</td>
    <td>12.95%</td>
    <td>28.49%</td>
    <td>10.18%</td>
    <td>46.70%</td>
    <td>22.96%</td>
    <td>24.26%</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>23.61</td>
    <td>42.54</td>
    <td>14.73</td>
    <td>27.39</td>
    <td>11.27</td>
    <td>44.92</td>
    <td>19.81</td>
    <td>24.81</td>
    <td>11.71</td>
    <td>39.45</td>
    <td>15.82</td>
    <td>29.17</td>
  </tr>
  <tr>
    <td></td>
    <td>zero-shot</td>
    <td>53.08</td>
    <td>45.32</td>
    <td>11.41</td>
    <td>38.59</td>
    <td>22.22</td>
    <td>48.23</td>
    <td>1.21</td>
    <td>24.34</td>
    <td>40.81</td>
    <td>48.32</td>
    <td>1.75</td>
    <td>30.78</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>62.82</td>
    <td>58.59</td>
    <td>71.97</td>
    <td>64.03</td>
    <td>31.28</td>
    <td>52.11</td>
    <td>12.41</td>
    <td>32.36</td>
    <td>55.33</td>
    <td>68.65</td>
    <td>85.89</td>
    <td>69.52</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>74.60</td>
    <td>65.64</td>
    <td>75.62</td>
    <td>71.90</td>
    <td>22.39</td>
    <td>56.61</td>
    <td>66.79</td>
    <td>46.95</td>
    <td>66.83</td>
    <td>70.95</td>
    <td>84.09</td>
    <td>73.72</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>76.41</td>
    <td>74.61</td>
    <td>84.49</td>
    <td>78.12</td>
    <td>63.77</td>
    <td>70.06</td>
    <td>88.42</td>
    <td>73.27</td>
    <td>78.23</td>
    <td>72.95</td>
    <td>89.90</td>
    <td>80.23</td>
  </tr>
  <tr>
    <td>ΔImprove</td>
    <td>2.43%</td>
    <td>13.66%</td>
    <td>11.73%</td>
    <td>8.65%</td>
    <td>18.48%</td>
    <td>23.77%</td>
    <td>32.39%</td>
    <td>56.04%</td>
    <td>17.06%</td>
    <td>2.82%</td>
    <td>6.91%</td>
    <td>8.38%</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>3.65</td>
    <td>39.44</td>
    <td>46.24</td>
    <td>27.81</td>
    <td>5.32</td>
    <td>40.64</td>
    <td>23.50</td>
    <td>22.39</td>
    <td>7.56</td>
    <td>33.21</td>
    <td>46.10</td>
    <td>28.37</td>
  </tr>
  <tr>
    <td></td>
    <td>zero-shot</td>
    <td>22.94</td>
    <td>41.04</td>
    <td>37.24</td>
    <td>29.75</td>
    <td>9.16</td>
    <td>27.61</td>
    <td>32.32</td>
    <td>70.91</td>
    <td>22.64</td>
    <td>40.83</td>
    <td>36.27</td>
    <td>39.56</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>3.45</td>
    <td>16.52</td>
    <td>16.71</td>
    <td>6.15</td>
    <td>23.8</td>
    <td>13.41</td>
    <td>17.03</td>
    <td>56.98</td>
    <td>2.77</td>
    <td>12.88</td>
    <td>10.53</td>
    <td>58.28</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>27.37</td>
    <td>39.71</td>
    <td>49.44</td>
    <td>80.16</td>
    <td>18.12</td>
    <td>37.05</td>
    <td>35.92</td>
    <td>77.51</td>
    <td>27.59</td>
    <td>59.63</td>
    <td>48.50</td>
    <td>80.17</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>40.54</td>
    <td>52.54</td>
    <td>53.87</td>
    <td>84.75</td>
    <td>34.46</td>
    <td>54.51</td>
    <td>81.75</td>
    <td>83.62</td>
    <td>45.98</td>
    <td>57.27</td>
    <td>58.16</td>
    <td>86.39</td>
  </tr>
  <tr>
    <td>ΔImprove</td>
    <td>3.61%</td>
    <td>13.89%</td>
    <td>14.63%</td>
    <td>11.10%</td>
    <td>1.21%</td>
    <td>25.11%</td>
    <td>8.98%</td>
    <td>10.71%</td>
    <td>8.84%</td>
    <td>6.30%</td>
    <td>7.36%</td>
    <td>7.54%</td>
  </tr>
</table>

Table 2: F1 scores (%) of each model on the ICEWS14, GDELT, and ICEWS05-15 test instances. "Overall" represents the weighted average F1 score. w/o chains text refers to the absence of reasoning chain input for LLMs. The best-performing results are highlighted in bold. ΔImprove represents the relative improvements of GETER compared to Tuned-only methods. Additional datasets and detailed prediction results are provided in Appendix E.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Types</th>
    <th colspan="3">Datasets ICEWS14</th>
    <th colspan="3">GDELT</th>
    <th colspan="3">ICEWS05-15</th>
  </tr>
  <tr>
    <th>BLEU-4 rougeL</th>
    <th>METEOR</th>
    <th>BertScore(F1)</th>
    <th>BLEU-4 rougeL</th>
    <th>METEOR</th>
    <th>BertScore(F1)</th>
    <th>BLEU-4 rougeL</th>
    <th>METEOR</th>
    <th>BertScore(F1)</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>zero-shot w/o chains text</td>
    <td>10.78</td>
    <td>28.32</td>
    <td>31.14</td>
    <td>68.16</td>
    <td>9.55</td>
    <td>21.30</td>
    <td>26.84</td>
    <td>64.73</td>
    <td>10.74</td>
    <td>23.63</td>
    <td>30.94</td>
    <td>68.00</td>
  </tr>
  <tr>
    <td></td>
    <td>zero-shot</td>
    <td>22.94</td>
    <td>41.04</td>
    <td>37.24</td>
    <td>29.75</td>
    <td>9.16</td>
    <td>27.61</td>
    <td>32.32</td>
    <td>70.91</td>
    <td>22.64</td>
    <td>40.83</td>
    <td>36.27</td>
    <td>39.16</td>
  </tr>
  <tr>
    <td>Llama3-8B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>4.35</td>
    <td>16.52</td>
    <td>16.71</td>
    <td>6.15</td>
    <td>23.8</td>
    <td>13.41</td>
    <td>17.03</td>
    <td>56.98</td>
    <td>2.77</td>
    <td>12.88</td>
    <td>10.53</td>
    <td>58.28</td>
  </tr>
  <tr>
    <td></td>
    <td>zero-shot</td>
    <td>27.37</td>
    <td>39.71</td>
    <td>49.44</td>
    <td>80.16</td>
    <td>18.12</td>
    <td>37.05</td>
    <td>35.92</td>
    <td>77.51</td>
    <td>27.59</td>
    <td>59.63</td>
    <td>48.50</td>
    <td>80.17</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>39.21</td>
    <td>59.06</td>
    <td>54.03</td>
    <td>84.28</td>
    <td>34.32</td>
    <td>54.84</td>
    <td>51.49</td>
    <td>83.75</td>
    <td>42.98</td>
    <td>54.50</td>
    <td>56.65</td>
    <td>85.45</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>40.54</td>
    <td>52.54</td>
    <td>53.87</td>
    <td>84.75</td>
    <td>34.46</td>
    <td>54.51</td>
    <td>81.75</td>
    <td>83.62</td>
    <td>45.98</td>
    <td>57.27</td>
    <td>58.16</td>
    <td>86.39</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>39.21</td>
    <td>59.06</td>
    <td>54.03</td>
    <td>84.28</td>
    <td>34.32</td>
    <td>54.84</td>
    <td>51.49</td>
    <td>83.75</td>
    <td>42.98</td>
    <td>54.50</td>
    <td>56.65</td>
    <td>85.45</td>
  </tr>
  <tr>
    <td>ΔImprove</td>
    <td>0.48%</td>
    <td>0.04%</td>
    <td>0.32%</td>
    <td>0.19%</td>
    <td>29.54%</td>
    <td>15.76%</td>
    <td>14.48%</td>
    <td>4.58%</td>
    <td>2.69%</td>
    <td>2.11%</td>
    <td>1.78%</td>
    <td>0.40%</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>7.17</td>
    <td>19.40</td>
    <td>24.27</td>
    <td>65.46</td>
    <td>4.89</td>
    <td>18.20</td>
    <td>25.78</td>
    <td>63.60</td>
    <td>7.24</td>
    <td>19.29</td>
    <td>23.26</td>
    <td>65.10</td>
  </tr>
  <tr>
    <td></td>
    <td>zero-shot</td>
    <td>9.19</td>
    <td>25.36</td>
    <td>25.70</td>
    <td>71.63</td>
    <td>7.46</td>
    <td>27.96</td>
    <td>25.99</td>
    <td>70.43</td>
    <td>7.95</td>
    <td>27.40</td>
    <td>23.60</td>
    <td>70.73</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>28.17</td>
    <td>40.22</td>
    <td>45.20</td>
    <td>80.12</td>
    <td>17.15</td>
    <td>36.89</td>
    <td>34.52</td>
    <td>75.71</td>
    <td>28.60</td>
    <td>40.52</td>
    <td>45.76</td>
    <td>80.39</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>38.10</td>
    <td>53.84</td>
    <td>54.70</td>
    <td>81.03</td>
    <td>18.22</td>
    <td>38.08</td>
    <td>37.45</td>
    <td>78.76</td>
    <td>28.26</td>
    <td>40.13</td>
    <td>45.96</td>
    <td>80.45</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>40.21</td>
    <td>51.84</td>
    <td>54.90</td>
    <td>84.65</td>
    <td>32.18</td>
    <td>53.27</td>
    <td>49.06</td>
    <td>82.83</td>
    <td>45.07</td>
    <td>54.88</td>
    <td>57.70</td>
    <td>86.13</td>
  </tr>
  <tr>
    <td>ΔImprove</td>
    <td>3.61%</td>
    <td>2.03%</td>
    <td>4.33%</td>
    <td>7.59%</td>
    <td>4.04%</td>
    <td>1.97%</td>
    <td>3.77%</td>
    <td>6.07%</td>
    <td>4.74%</td>
    <td>3.52%</td>
    <td>3.14%</td>
    <td>7.07%</td>
  </tr>
</table>

Table 3: The semantic similarity performance (%) of each model on the ICEWS14, GDELT, and ICEWS05-15 test instances. w/o chains text refers to the absence of reasoning chain input for LLMs. The best-performing results are highlighted in bold. Additional dataset explanation results are presented in Appendix E.

with gains of 75.28%, 251.31%, and 99.07%, respectively. These results highlight GETER's ability to leverage high-quality fine-tuning datasets to enhance explainable temporal reasoning capabilities.

<table>
  <tr>
    <th>No.</th>
    <th>Model</th>
    <th>ICEWS14</th>
    <th>GDELT</th>
    <th>ICEWS05-15</th>
  </tr>
  <tr>
    <td>1</td>
    <td>GETER</td>
    <td><b>79.08</b></td>
    <td><b>72.02</b></td>
    <td><b>81.80</b></td>
  </tr>
  <tr>
    <td>2</td>
    <td>GETER w/o STA</td>
    <td>71.18<sup>(17.90)</sup></td>
    <td>65.05<sup>(6.97)</sup></td>
    <td>76.07<sup>(15.78)</sup></td>
  </tr>
  <tr>
    <td>3</td>
    <td>GETER w/o RCT</td>
    <td>72.05<sup>(17.03)</sup></td>
    <td>68.89<sup>(13.13)</sup></td>
    <td>77.82<sup>(13.98)</sup></td>
  </tr>
  <tr>
    <td>4</td>
    <td>GETER w/o (STA & RCT)</td>
    <td>66.79<sup>(12.29)</sup></td>
    <td>47.80<sup>(24.22)</sup></td>
    <td>61.95<sup>(19.85)</sup></td>
  </tr>
</table>

Table 4: Ablation study of GETER with Mistral on ICEWS14, GDELT, and ICEWS05-15 datasets using overall F1 scores (%). STA denotes structure-text adapter, while RCT denotes reasoning chains text.

5.3 Ablation Study

In this subsection, we conduct an ablation study to investigate the individual contributions of different components in GETER. The results for various variants are presented in Table 4, indicating that all modules are essential, as removing any of them leads to a decline in performance. Notably, to validate the usefulness of the structural information provided by GETER, we directly removed the structure-text adapter from the model (Line 2). This ablation results in overall F1 score reductions of 11.10%, 10.71%, and 7.53% across the three datasets, respectively. These results demonstrate that the soft graph token with lightweight adapter can effective capture the structural characteristics
for the query. Additionally, as shown in Line 3 of Table 4, removing the reasoning chains text leads to a significant performance decline, with F1 scores dropping by 9.76%, 10.71%, and 5.11% across the three datasets, respectively. This result highlights the importance of reasoning chains text, as they provide sequenced evidence that enriches the contextual background. Furthermore, we observe that GETER scheme significantly outperforms the base model that directly adopts instruction tuning (Line 4). This demonstrates the effectiveness of GETER, which combine structural and contextual semantic information to activate and harness the LLM’s capability for explainable temporal reasoning.

5.4 Discussion

In this subsection, we conduct further analysis of the impact of different temporal encoders, the influence of MLP depth, and the effect of various reasoning chain serialization formats on the model’s performance. All experiments are conducted using Mistral for its superior performance. Additionally, we present a complexity analysis in Appendix C.2 and a case study in Appendix D to further highlight the advantages of our proposed method.

Q1: What is the impact of different temporal encoders on GETER’s performance? To evaluate the impact of different temporal encoders, we also integrate CEN, CENET, and SiMFy into the our framework, as described in 5.1. The performance comparison is illustrated in Figure 4. The results demonstrate that GETER achieves consistently high performance across two datasets when paired with any of the three temporal encoders, significantly outperforming methods that rely solely on LoRA. These findings demonstrate that GETER is robust to variations in temporal encoders. Details about temporal encoders refer to Appendix B.1.

Figure 4: Comparison of GETER with different temporal encoders on the ICEWS14 and GDELT datasets in terms of overall F1 scores (%).

![Comparison of GETER with different temporal encoders on the ICEWS14 and GDELT datasets in terms of overall F1 scores (%)](page_186_1042_1077_312.png)

Figure 5: MLP depth comparison on ICEWS14 and GDELT datasets in terms of overall F1 scores (%).

![MLP depth comparison on ICEWS14 and GDELT datasets in terms of overall F1 scores (%)](page_1012_1042_377_246.png)

<table>
  <tr>
    <th>Model</th>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td><i>GETER</i> (paths order)</td>
    <td>77.45</td>
    <td>75.73</td>
    <td>85.15</td>
    <td>79.08</td>
  </tr>
  <tr>
    <td>descending order</td>
    <td><b>80.53</b></td>
    <td>76.00</td>
    <td><b>86.34</b></td>
    <td><b>80.68</b></td>
  </tr>
  <tr>
    <td>ascending order</td>
    <td>77.72</td>
    <td><b>77.52</b></td>
    <td>86.04</td>
    <td>80.03</td>
  </tr>
  <tr>
    <td>random order</td>
    <td>75.02</td>
    <td>76.31</td>
    <td>82.45</td>
    <td>77.57</td>
  </tr>
</table>

Table 5: Performance (F1 (%)) of GETER with different reasoning chain formats on the ICEWS14 dataset.

Q2: How does the depth of the MLP affect GETER’s performance? GETER uses a one-layer MLP to map the graph structure feature into the text embedding space. To investigate whether deeper neural structures improves performance, we conduct experiments to replace the one-layer MLP with deeper variants. The results on the ICEWS14 and GDELT datasets are presented in Figure 5. We can observe that increasing model complexity has minimal impact on performance. This is likely because deeper structures fail to capture evolving structural information more effectively.

Q3: What is the effect of different reasoning chain text formats on GETER’s performance? We further investigate how GETER utilizes reasoning chain text, which provides contextualized background information for queries. Specifically, we evaluate three different serialization formats based on the timestamp of quadruples: ascending, descending, and random. As shown in Table 5, the model achieves the best performance with the descending order format. Surprisingly, even with random serialization, GETER still maintains competitive performance. This is attributed to the structured adapter in GETER, which effectively couple structure and text information in a contextualized manner. These findings further highlight the robustness and adaptability of our proposed GETER.
6 Conclusion

We introduce a comprehensive benchmark covering a wide range of temporal granularities for systematically evaluating LLMs’ explainable temporal reasoning. To address the challenge of LLMs struggling to deliver convincing explanations, we propose a novel structure-aware generative framework GETER, which effectively bridges the gap between graph structures and text by through a lightweight structure-text adapter. Extensive experiments validate the effectiveness and robustness of our proposed GETER.

Limitations

GETER can effectively activate and harness the explainable reasoning ability of LLMs by incorporate the graph structural information into the LLMs. However, the extremely large number of parameters in LLMs makes fine-tuning them resource-intensive. At the same time, LLMs are notoriously slow at decoding during inference. In our experiment, we use DeepSpeed (Rajbhandari et al., 2020) to accelerate training and inference. Additionally, some reasoning chains may introduce noisy text, which could negatively affect explainable temporal reasoning performance.

Ethics Statement

In developing this explainable temporal reasoning benchmark, all data used in this study are publicly available and do not pose any privacy concerns. Additionally, we have carefully considered ethical issues and limitations commonly associated with large language models. Nonetheless, we acknowledge that, despite our best efforts, the benchmark may still contain gaps or unintended biases. To mitigate this, the source data has been meticulously curated to ensure diversity and minimize potential biases. Through rigorous design and testing processes, we strive to uphold ethical AI principles while advancing research in temporal reasoning.

Acknowledgments

We would like to thank all the anonymous reviewers and area chairs for their comments. This research is supported by National Natural Science Foundation of China (U23A20316) and founded by Joint&Laboratory on Credit Technology.

References

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In IEEvaluation@ACL, pages 65–72. Association for Computational Linguistics.

Veronika Bogina, Tsvi Kuffik, Dietmar Jannach, Mária Bieliková, Michal Kompan, and Christoph Trattner. 2023. Considering temporal aspects in recommender systems: a survey. User Model. User Adapt. Interact., 33(1):81–119.

Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models. In ACL (1), pages 1204–1228. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Al-lonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibou, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Milon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Bilkock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasudenh Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.

Alberto García-Durán, Sebastijan Dumancic, and Mathias Niepert. 2018. Learning sequence encoders for temporal knowledge graph completion. In Proc. of EMNLP, pages 4816–4821.

Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. 2021. Explainable subgraph reasoning for forecasting on temporal knowledge graphs. In ICLR. Open-Review.net.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. CoRR, abs/2402.07630.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In ACL (Findings), pages 1049–1065. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamber, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mix-trial of experts. CoRR, abs/2401.04088.

Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Yaliang Li, and Ji-Rong Wen. 2023. Reasoninglm: Enabling structural subgraph reasoning in pre-trained language models for question answering over knowledge graph. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 3721–3735. Association for Computational Linguistics.

Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. 2020. Recurrent event network: Autoregressive structure inferenceover temporal knowledge graphs. In Proc. of EMNLP, pages 6669–6683.

Julien Leblay and Melisachew Wudage Chekol. 2018. Deriving validity time in knowledge graph. In Companion of the The Web Conference 2018 on The Web Conference 2018, WWW 2018, Lyon , France, April 23-27, 2018, pages 1771–1776. ACM.

Dong-Ho Lee, Kian Ahbarian, Woojeong Jin, Fred Morstatter, and Jay Pujara. 2023. Temporal knowledge graph forecasting without knowledge using in-context learning. In EMNLP, pages 544–557. Association for Computational Linguistics.

Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, and Xueqi Cheng. 2022. Complex evolutionary pattern learning for temporal knowledge graph reasoning. In ACL (2), pages 290–296. Association for Computational Linguistics.

Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. 2021. Temporal knowledge graph reasoning based on evolutional representation learning. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 408–417. ACM.

Rutong Liao, Xu Jia, Yangzhe Li, Yunpu Ma, and Volker Tresp. 2024. Gentkg: Generative forecasting on temporal knowledge graph with large language models. In NAACL-HLT (Findings), pages 4303–4317. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Qika Lin, Jun Liu, Rui Mao, Fangzhi Xu, and Erik Cambria. 2023. TECHS: temporal logical graph networks for explainable extrapolation reasoning. In ACL (1), pages 1281–1293. Association for Computational Linguistics.

Ben Liu, Jihai Zhang, Fangquan Lin, Cheng Yang, and Min Peng. 2025a. Filter-then-generate: Large language models with structure-text adapter for knowledge graph completion. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pages 11181–11195. Association for Computational Linguistics.

Ben Liu, Jihai Zhang, Fangquan Lin, Cheng Yang, Min Peng, and Wotao Yin. 2025b. Symagent: A neural-symbolic self-learning agent framework for complex reasoning over knowledge graphs. In WWW, pages 98–108. ACM.

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022a. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Yushan Liu, Yunpu Ma, Marcel Hildebrandt, Mitchell Joblin, and Volker Tresp. 2022b. Tlogic: Temporal logical rules for explainable link forecasting on temporal knowledge graphs. In AAAI, pages 4120–4127. AAAI Press.

Zhengtao Liu, Lei Tan, Mengfan Li, Yao Wan, Hai Jin, and Xuanhua Shi. 2023. Simfy: A simple yet effective approach for temporal knowledge graph reasoning. In EMNLP (Findings), pages 3825–3836. Association for Computational Linguistics.
Rulin Luo, Tianle Gu, Haoling Li, Junzhe Li, Zicheng Lin, Jiayi Li, and Yujiu Yang. 2024. Chain of history: Learning and forecasting with llms for temporal knowledge graph completion. CoRR, abs/2401.06072.

Qiyao Ma, Xubin Ren, and Chao Huang. 2024. Xrec: Large language models for explainable recommendation. In EMNLP (Findings), pages 391–402. Association for Computational Linguistics.

Xin Mei, Libin Yang, Xiaoyan Cai, and Zuowei Jiang. 2022. An adaptive logical rule embedding model for inductive reasoning over temporal knowledge graphs. In EMNLP, pages 7304–7316. Association for Computational Linguistics.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318. ACL.

Miao Peng, Nuo Chen, Zongrui Suo, and Jia Li. 2025. Rewarding graph reasoning process makes llms more generalized reasoners. CoRR, abs/2503.00845.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20. IEEE/ACM.

Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. 2021. Timetraveler: Reinforcement learning for temporal knowledge graph forecasting. In EMNLP (1), pages 8306–8319. Association for Computational Linguistics.

Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. Towards benchmarking and improving the temporal reasoning capability of large language models. In ACL (1), pages 14820–14835. Association for Computational Linguistics.

Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2024. Towards robust temporal reasoning of large language models via a multi-hop QA dataset and pseudo-instruction tuning. In ACL (Findings), pages 6272–6286. Association for Computational Linguistics.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In ICLR. OpenReview.net.

Yuqing Wang and Yun Zhao. 2024. TRAM: benchmarking temporal reasoning for large language models. In ACL (Findings), pages 6389–6415. Association for Computational Linguistics.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022.

Yifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, and Kang Liu. 2023. Menatqa: A new dataset for testing the temporal comprehension and reasoning abilities of large language models. In EMNLP (Findings), pages 1434–1447. Association for Computational Linguistics.

Weiqi Wu, Shen Huang, Yong Jiang, Pengjun Xie, Fei Huang, and Hai Zhao. 2025. Unfolding the headline: Iterative self-questioning for news retrieval and timeline summarization. CoRR, abs/2501.00888.

Yuwei Xia, Ding Wang, Qiang Liu, Liang Wang, Shu Wu, and Xiaoyu Zhang. 2024. Chain-of-history reasoning for temporal knowledge graph forecasting. In ACL (Findings), pages 16144–16159. Association for Computational Linguistics.

Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. Large language models can learn temporal reasoning. In ACL (1), pages 10452–10470. Association for Computational Linguistics.

Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2023. Temporal knowledge graph reasoning with historical contrastive learning. In AAAI, pages 4765–4773. AAAI Press.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao-ran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. 2024a. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Wanqi Yang, Yanda Li, Meng Fang, and Ling Chen. 2024b. Enhancing temporal sensitivity and reasoning for time-sensitive question answering. In EMNLP (Findings), pages 14495–14508. Association for Computational Linguistics.

Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. Back to the future: Towards explainable temporal reasoning with large language models. In Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024, pages 1963–1974. ACM.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In ICLR. OpenReview.net.

A Benchmark Details

A.1 Prompt for Generating Explanations of Positive Samples

Prompt for Positive Samples’ Explanation

Given the following text: “we predict that [e_a] [r] [e_o] will happen on [t_q]. Here are the reasoning steps: \( \mathcal{C}(e_s, e_o) \).” Please revise the provided text to ensure that the prediction aligns with the reasoning steps. Expand the explanation of each reasoning step to make the text more coherent and readable. If necessary, add additional reasoning steps to clarify the logic. The output should be a single, concise paragraph without bullet points, ensuring clarity and logical consistency.

A.2 Prompt for Generating Explanations of Negative Samples

Prompt for Negative Samples’ Explanation

Given the following text: “It is plausible that [e_a] [r'] [e_o] will not happen on [t_q]. Here are the reasoning steps: \( \mathcal{C}(e_s, e_o) \).” Please revise the provided text to ensure that the prediction aligns with the reasoning steps. Expand the explanation of each reasoning step to make the text more coherent and readable. If necessary, add additional reasoning steps to clarify the logic. The output should be a single, concise paragraph without bullet points, ensuring clarity and logical consistency.

A.3 Prompt for Generating Explanations of Neutral Samples

Prompt for Neutral Samples’ Explanation

Given the following text: “It is unsure that [e_a] [r'] [e_o] will happen on [t_q]. Here are the reasoning steps: \( \mathcal{C}(e_s, e_o) \).” Please revise the provided text to ensure that the prediction aligns with the reasoning steps. Expand the explanation of each reasoning step to make the text more coherent and readable. If necessary, add additional reasoning steps to clarify the logic. The output should be a single, concise paragraph without bullet points, ensuring clarity and logical consistency.

A.4 Example Prompt for Instruction Tuning

Here is an example of an instruction tuning prompt for the query: (Police (Australia), Make an appeal or request, Citizen (Australia), 2014-03-12).

Example Prompt for Instruction Tuning

Instruct: Given the following document, is it plausible that Police (Australia) will Make an appeal or request Citizen (Australia) on 2014-03-12? Please answer yes, no, or unsure then explain your decision.

Input: Police (Australia) Appeal for aid Citizen (Australia) 2014-02-10. Police (Australia) Arrest, detain, or charge with legal action Citizen (Australia) 2014-02-25. Police (Australia) Arrest, detain, or charge with legal action Citizen (Australia) 2014-03-04. ..... Police (Australia) Arrest, detain, or charge with legal action Criminal (Australia) on 2014-02-27. Criminal (Australia) Sexually assault Citizen (Australia) on 2014-03-08......

Output: Yes. Based on the information provided, it is plausible that the Police in Australia will make an appeal or request to citizens on March 12, 2014. This prediction is supported by a pattern of interactions between the police and citizens leading up to this date. The police have previously made appeals or requests to citizens on several occasions, including February 23, February 26, and March 11, 2014. Additionally, the police have been actively involved in various actions such as arrests, detentions, and investigations, indicating a heightened level of engagement with the public. The police have also been involved in more severe actions, such as using conventional military force and fighting with small arms, which suggests a context of ongoing tension or conflict. Given this backdrop, it is reasonable to anticipate that the police might continue their efforts to communicate with citizens, possibly to seek cooperation or provide updates on ongoing situations. Therefore, considering the frequency and nature of past interactions, it is likely that the police will make another appeal or request to citizens on March 12, 2014.

A.5 Benchmark Summary and Evaluation

The statistical details of the source data used to construct the benchmark are provided in Table 6. The data consist of three sources: the Integrated Crisis Early Warning System (ICEWS), the Global Database of Events, Language, and Tone (GDELT), and Wikipedia (WIKI). Specifically, the
<table>
  <tr>
    <th>Datasets</th>
    <th>#train</th>
    <th>#valid</th>
    <th>#test</th>
    <th>#entity</th>
    <th>#relations</th>
    <th>time gap</th>
  </tr>
  <tr>
    <td>ICEWS14</td>
    <td>7854</td>
    <td>8514</td>
    <td>7371</td>
    <td>7128</td>
    <td>230</td>
    <td>1 day</td>
  </tr>
  <tr>
    <td>ICEWS18</td>
    <td>373018</td>
    <td>45995</td>
    <td>49545</td>
    <td>23033</td>
    <td>256</td>
    <td>1 day</td>
  </tr>
  <tr>
    <td>ICEWS05-15</td>
    <td>368868</td>
    <td>46302</td>
    <td>46159</td>
    <td>10094</td>
    <td>251</td>
    <td>1 day</td>
  </tr>
  <tr>
    <td>GDELT</td>
    <td>79519</td>
    <td>9957</td>
    <td>9715</td>
    <td>5850</td>
    <td>238</td>
    <td>15 minutes</td>
  </tr>
  <tr>
    <td>WIKI</td>
    <td>539286</td>
    <td>67538</td>
    <td>63110</td>
    <td>12354</td>
    <td>24</td>
    <td>1 year</td>
  </tr>
</table>

Table 6: Dataset statistics.

ICEWS14 dataset includes events from 2014, the ICEWS18 dataset includes events from 2018, and the ICEWS05-15 dataset spans events from 2005 to 2015. The GDELT dataset records events at 15-minute intervals, while WIKI consists of Wikidata knowledge bases that store factual information with a time interval of one year. To ensure the quality and reliability of our dataset, we recruited three volunteers to evaluate the benchmark. Each volunteer assessed 200 randomly selected examples from the dataset. They were instructed to perform two key evaluations, assigning scores on a scale of 1 to 3 based on the following criteria:

Explanation Text Quality (1-3):

• 1 - The explanation is unclear, incoherent, or unreasonable.

• 2 - The explanation is somewhat clear and reasonable but lacks coherence or completeness in certain aspects.

• 3 - The explanation is clear, coherent, and fully reasonable.

Overall Consistency (1-3):

• 1 - The query text, reasoning chain, and explanation text are inconsistent or logically disconnected.

• 2 - There is partial consistency among the query text, reasoning chain, and explanation text, but logical gaps remain.

• 3 - The query text, reasoning chain, and explanation text are fully consistent and logically aligned.

The results of the human evaluation, as shown in Table 7, demonstrate a high level of accuracy and reliability in our benchmark generation process.

B Implementation Details

B.1 Baselines

Below, we provide brief introductions to the LLMs used in our methods:

<table>
  <tr>
    <th>Volunteer</th>
    <th>Explanation Text Quality</th>
    <th>Overall Consistency</th>
  </tr>
  <tr>
    <td>Volunteer 1</td>
    <td>2.80</td>
    <td>2.78</td>
  </tr>
  <tr>
    <td>Volunteer 2</td>
    <td>2.74</td>
    <td>2.79</td>
  </tr>
  <tr>
    <td>Volunteer 3</td>
    <td>2.86</td>
    <td>2.89</td>
  </tr>
</table>

Table 7: Average scores for Explanation Text Quality and Overall Consistency by Volunteers.

• *GPT-4o* (OpenAI, 2023) is a large language model developed by OpenAI, representing an advanced iteration of the GPT series. It is known for its strong generalization capabilities across a wide range of natural language processing tasks, including reasoning, generation, and instruction-following.

• *Llama-3.1-8B-Instruct* (Dubey et al., 2024) is an instruction-tuned version of the Llama3 series, with 8 billion parameters. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.

• *Qwen2.5-7B-Instruct* (Yang et al., 2024a) is the latest series of Qwen large language models. It focuses on optimizing performance for instruction-based tasks.

• *Mistral-7B-Instruct-v0.3* (Jiang et al., 2024) is a 7-billion-parameter instruction-tuned model with an extended 32,768-token vocabulary, v3 tokenizer support, and function calling capabilities for improved task performance.

We also introduce the graph-based methods (temporal encoders) utilized in our methods:

• *RE-GCN* (Li et al., 2021) proposes a recurrent evolution module based on relational GNNs to obtain embeddings that contain dynamic information for entities and relations.

• *CEN* (Li et al., 2022) uses a length-aware Convolutional Neural Network(CNN) to handle evolutional patterns of different lengths via an easy-to-difficult curriculum learning strategy.

• *CENET* (Xu et al., 2023) aims to learn a robust distribution over the entire entity set and identify significant entities by leveraging both historical and non-historical dependencies within a contrastive learning framework.

• *SiMFy* (Liu et al., 2023) is a straightforward method that combines MLP and historical frequency to model the temporal events.
B.2 Hyperparameters
We set the window size \( w \) to 30 and the threshold \( \tau \) to 0.7 for constructing our benchmark. During training, the RE-GCN module is kept frozen, and LoRA is employed to fine-tune the model. The structural embedding size \( d_s \) is set to 512, while the textual embedding size \( d_x \) retains the original hidden layer dimensions of each LLM. The detailed hyperparameters used during training and inference are provided in Table 8. For optimization, we enable DeepSpeed ZeRO stage3.1. All models are trained and evaluated on 2 Nvidia A800 GPUs, each with 80GB of memory.

<table>
  <tr>
    <th>Name</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>lora r</td>
    <td>16</td>
  </tr>
  <tr>
    <td>lora alpha</td>
    <td>32</td>
  </tr>
  <tr>
    <td>lora dropout</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>lora target modules</td>
    <td>(q, k, v, o, down, up, gate) proj</td>
  </tr>
  <tr>
    <td>cutoff len</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>epochs</td>
    <td>3</td>
  </tr>
  <tr>
    <td>per device batch size</td>
    <td>6</td>
  </tr>
  <tr>
    <td>gradient accumulation steps</td>
    <td>1</td>
  </tr>
  <tr>
    <td>learning rate</td>
    <td>3e - 4</td>
  </tr>
  <tr>
    <td>weight decay</td>
    <td>1e - 5</td>
  </tr>
  <tr>
    <td>warm ratio</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>lr scheduler type</td>
    <td>cosine</td>
  </tr>
  <tr>
    <td>num return sequences</td>
    <td>10</td>
  </tr>
  <tr>
    <td>projection layers</td>
    <td>1</td>
  </tr>
</table>

Table 8: Detailed hyperparameters used in our paper.

C Additional Comparative Study Results

C.1 Comparison with Graph-based Methods
To provide a comprehensive comparison, we also evaluate four state-of-the-art graph-based methods (REGCN, CEN, CENET, and SiMFy) in comparison with our method on the task. Specifically, for the query \((e_s, r, e_o, t_q)\), we utilize an MLP to adapt to our task, as defined below:

\[
P = W_{query}(e_s \parallel r \parallel e_o)
\]

where \( \parallel \) denotes the concatenation operation, \( P \in \mathbb{R}^3 \), \( e_s \in \mathbb{R}^{1 \times d_s} \), \( r \in \mathbb{R}^{1 \times d_r} \), and \( e_o \in \mathbb{R}^{1 \times d_o} \). Here, \( W_{query} \in \mathbb{R}^{3 \times 3d_s} \) is a learnable weight matrix, and \( d_s \) represents the embedding dimension.

The prediction results are presented in Table 9 through Table 11. We can observe that GETER significantly outperforms existing graph-based in terms of prediction results. Furthermore, our approach provides human-readable inference processes, ensuring greater interpretability. In contrast, the intrinsic property of these graph-based methods is that they are **black-box models**, inherently lacking explainability and unable to generate explanation text. The detailed results of the prediction experiments are summarized in Table 20.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">ICEWS14</th>
    <th colspan="4">GDELT</th>
  </tr>
  <tr>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>REGCN</td>
    <td>52.76</td>
    <td>55.00</td>
    <td>75.38</td>
    <td>59.97</td>
    <td>57.62</td>
    <td>59.39</td>
    <td>84.16</td>
    <td>66.22</td>
  </tr>
  <tr>
    <td>CEN</td>
    <td>61.25</td>
    <td>53.01</td>
    <td>76.24</td>
    <td>62.79</td>
    <td>60.29</td>
    <td>61.42</td>
    <td>86.93</td>
    <td>68.71</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>55.03</td>
    <td>60.82</td>
    <td>78.27</td>
    <td>63.60</td>
    <td>61.34</td>
    <td>62.71</td>
    <td>87.86</td>
    <td>69.84</td>
  </tr>
  <tr>
    <td>SiMFy</td>
    <td>53.40</td>
    <td>63.03</td>
    <td>78.92</td>
    <td>63.90</td>
    <td>63.30</td>
    <td>60.91</td>
    <td>88.23</td>
    <td>70.06</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>77.45</td>
    <td>75.73</td>
    <td>85.15</td>
    <td>79.08</td>
    <td>61.29</td>
    <td>68.92</td>
    <td>88.59</td>
    <td>72.02</td>
  </tr>
</table>

Table 9: F1 scores (%) of different graph-based methods on ICEWS14 and GDELT datasets.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">ICEWS05-15</th>
    <th colspan="4">ICEWS18</th>
  </tr>
  <tr>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>REGCN</td>
    <td>65.10</td>
    <td>63.53</td>
    <td>83.57</td>
    <td>70.50</td>
    <td>62.13</td>
    <td>58.81</td>
    <td>81.53</td>
    <td>67.03</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>67.29</td>
    <td>64.17</td>
    <td>89.44</td>
    <td>73.36</td>
    <td>61.99</td>
    <td>64.45</td>
    <td>84.90</td>
    <td>69.90</td>
  </tr>
  <tr>
    <td>CEN</td>
    <td>63.27</td>
    <td>65.57</td>
    <td>86.88</td>
    <td>71.59</td>
    <td>59.82</td>
    <td>60.86</td>
    <td>79.19</td>
    <td>68.16</td>
  </tr>
  <tr>
    <td>SiMFy</td>
    <td>67.61</td>
    <td>66.95</td>
    <td>89.14</td>
    <td>74.29</td>
    <td>60.88</td>
    <td>62.61</td>
    <td>82.34</td>
    <td>68.10</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>78.94</td>
    <td>76.48</td>
    <td>90.38</td>
    <td>81.80</td>
    <td>75.61</td>
    <td>75.94</td>
    <td>87.51</td>
    <td>79.40</td>
  </tr>
</table>

Table 10: F1 scores (%) of different graph-based methods on ICEWS05-15 and ICEWS18 datasets.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">WIKI</th>
  </tr>
  <tr>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>REGCN</td>
    <td>75.27</td>
    <td>70.38</td>
    <td>77.65</td>
    <td>74.59</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>76.11</td>
    <td>77.06</td>
    <td>83.51</td>
    <td>78.86</td>
  </tr>
  <tr>
    <td>CEN</td>
    <td>74.36</td>
    <td>76.04</td>
    <td>82.25</td>
    <td>77.49</td>
  </tr>
  <tr>
    <td>SiMFy</td>
    <td>79.03</td>
    <td>77.39</td>
    <td>81.53</td>
    <td>79.40</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>99.28</td>
    <td>94.49</td>
    <td>96.19</td>
    <td>96.81</td>
  </tr>
</table>

Table 11: F1 scores (%) of different graph-based methods on the WIKI dataset.

![Comparison of training time between GETER and the LoRA fine-tuning method. The Y-axis represents the training time (hours).](page_1012_1342_388_246.png)

Figure 6: Comparison of training time between GETER and the LoRA fine-tuning method. The Y-axis represents the training time (hours).

C.2 Complexity Analysis of GETER
LLM applications often face challenges related to high computational costs due to the large number of model parameters. Specifically, for our method
GETER, during the training and inference stages, the complexity is \( O(|L_1|^2 \cdot |L_2|) \) for the input-answer pair, where \( |L_1| \) represents the length of the input text and \( |L_2| \) represents the length of the answer. Considering the costs, we leverage Low-Rank Adaptation (LoRA) and DeepSpeed to accelerate both training and inference. Additionally, for a clearer comparison, we present the training time of GETER against LoRA fine-tuning methods across five datasets in Figure 6. The results demonstrate that incorporating graph tokens into LLM fine-tuning introduces minimal additional time costs compared to simple LoRA fine-tuning. Furthermore, given the significant performance improvements achieved by our method, as detailed in Section 5.2, we consider these additional costs to be negligible.

D Case Study

In this section, we present a case study to highlight the differences in responses among Inference-only method, Tuned-only method, and GETER. Specifically, we analyze the following positive query: (Police (Australia), Engage in material cooperation, Citizen (Australia), 2014-11-16), where the expected label is "Yes". As shown in Table 12, Inference-only method fail to capture the subtle cooperative signals embedded within the document (highlighted in orange), instead focusing primarily on dominant antagonistic actions, such as arrests and accusations, which result in incorrect negative predictions. While Tuned-only method can observe cooperative signals and demonstrate an improved ability to incorporate the temporal aspects of events, they struggle to fully model the interplay between cooperative and antagonistic actions (highlighted in blue), leading to comparable negative predictions. In contrast, GETER effectively captures the evolving patterns of event relationships and cooperative signals (highlighted in red). By leveraging explicit cues, such as requests and expressed intentions to cooperate, GETER not only predicts a positive outcome accurately but also provides the correct explanation.

E Full Experimental Results

The prediction results for the ICEWS18 and WIKI datasets are summarized in Table 13, while the explanation results are detailed in Table 14. Notably, GETER demonstrates consistent and significant improvements across most metrics on these two datasets, underscoring its robustness and superior performance in complex scenarios. Compared to Tuned-only methods, GETER combined with Mistral achieves overall F1 score improvements of 16.42% and 10.35% on the respective datasets. Additionally, the detailed prediction results for all five datasets are comprehensively summarized in Table 15 through Table 20.
Instruction: Given the following document, is it plausible that Police (Australia) will Engage in material cooperation Citizen (Australia) on 2014-11-16? Please answer yes, no, or unsure then explain your decision.

Input: Police (Australia) Arrest, detain, or charge with legal action Citizen (Australia) 2014-11-12. Police (Australia) Arrest, detain, or charge with legal action Citizen (Australia) 2014-11-13. Police (Australia) Arrest, detain, or charge with legal action Citizen (Australia) 2014-11-14. Police (Australia) Accuse Citizen (Australia) 2014-11-02. Police (Australia) Accuse Citizen (Australia) 2014-11-09. Police (Australia) Accuse Citizen (Australia) 2014-11-10. Police (Australia) Make an appeal or request Citizen (Australia) 2014-11-12. Police (Australia) Make an appeal or request Citizen (Australia) 2014-11-13. Police (Australia) Make an appeal or request Citizen (Australia) 2014-11-14. Police (Australia) fight with small arms and light weapons Citizen (Australia) 2014-11-09. Police (Australia) fight with small arms and light weapons Citizen (Australia) 2014-11-12. Police (Australia) fight with small arms and light weapons Citizen (Australia) 2014-11-14. Police (Australia) Use conventional military force Citizen (Australia) 2014-10-22. Police (Australia) Use conventional military force Citizen (Australia) 2014-10-24. Police (Australia) Use conventional military force Citizen (Australia) 2014-11-10. Police (Australia) Investigate Citizen (Australia) 2014-11-03. Police (Australia) Investigate Citizen (Australia) 2014-11-04. Police (Australia) Investigate Citizen (Australia) 2014-11-09. Police (Australia) Express intent to meet or negotiate Citizen (Australia) 2014-10-24. Police (Australia) Express intent to meet or negotiate Citizen (Australia) 2014-11-11. Police (Australia) Express intent to meet or negotiate Citizen (Australia) 2014-11-14. Police (Australia) Criticize or denounce Citizen (Australia) 2014-10-28. Police (Australia) Criticize or denounce Citizen (Australia) 2014-11-03. Police (Australia) Confiscate property Citizen (Australia) 2014-10-30. Police (Australia) Investigate human rights abuses Citizen (Australia) 2014-10-30. Police (Australia) Appeal for intelligence Citizen (Australia) 2014-11-04. Police (Australia) Reject Citizen (Australia) 2014-11-07. Police (Australia) Abduct, hijack, or take hostage Citizen (Australia) 2014-11-09. Police (Australia) Physically assault Citizen (Australia) 2014-11-13. Police (Australia) Impose restrictions on political freedoms Citizen (Australia) 2014-11-14. Police (Australia) Return, release person(s) Citizen (Australia) 2014-11-14. Police (Australia) Arrest, detain, or charge with legal action Criminal (Australia) on 2014-10-23. Criminal (Australia) Engage in symbolic act Citizen (Australia) on 2014-11-03. Police (Australia) Arrest, detain, or charge with legal action Criminal (Australia) on 2014-10-23. Criminal (Australia) Sexually assault Citizen (Australia) on 2014-11-05. Police (Australia) Accuse Criminal (Australia) on 2014-10-22. Criminal (Australia) Engage in symbolic act Citizen (Australia) on 2014-11-03. Police (Australia) Accuse Criminal (Australia) on 2014-10-22. Criminal (Australia) Sexually assault Citizen (Australia) on 2014-11-05. Police (Australia) Demand Lawyer/Attorney (Australia) on 2014-10-31. Lawyer/Attorney (Australia) Arrest, detain, or charge with legal action Citizen (Australia) on 2014-11-03. Police (Australia) Demand Lawyer/Attorney (Australia) on 2014-10-31, Lawyer/Attorney (Australia) Accuse Citizen (Australia) on 2014-11-10. Police (Australia) Demand Lawyer/Attorney (Australia) on 2014-10-31, Lawyer/Attorney (Australia) Threaten Citizen (Australia) on 2014-11-05. Police (Australia) Confiscate property Criminal (Australia) on 2014-10-23. Criminal (Australia) Engage in symbolic act Citizen (Australia) on 2014-11-03. Police (Australia) Confiscate property Criminal (Australia) on 2014-10-23. Criminal (Australia) Sexually assault Citizen (Australia) on 2014-11-05.

Label output: Yes. Based on the information provided, it is plausible that the Police in Australia will engage in material cooperation with Australian citizens on November 16, 2014. This prediction is supported by a series of interactions between the police and citizens leading up to this date. The police have been actively involved in various legal actions, such as arresting, detaining, or charging citizens on multiple occasions from November 12 to November 14, 2014. Additionally, they have made appeals or requests to citizens during the same period, indicating a willingness to communicate and potentially collaborate. Despite instances of conflict, such as accusations, criticism, and even physical confrontations, there are also signs of negotiation and intent to meet, as evidenced by the police expressing their intent to negotiate on November 11 and November 14, 2014. Furthermore, the police have returned or released individuals on November 14, suggesting a move towards resolution and cooperation. These actions, combined with the broader context of police investigations and appeals for intelligence, suggest a complex relationship that could lead to material cooperation on November 16, 2014.
<table>
  <tr>
    <th>Methods</th>
    <th>Labels | Output</th>
  </tr>
  <tr>
    <td>Inference-only</td>
    <td>x<br>Based on the provided document, it is not plausible that Police (Australia) will engage in material cooperation with the Citizen (Australia) on 2014-11-16. The document shows a series of actions by the Police (Australia) that are generally antagonistic towards the Citizen (Australia), including arrests, detentions, charges, accusations, confiscation of property, physical assault, and investigations into human rights abuses. These actions suggest a conflictual relationship rather than cooperation. Furthermore, there is no indication in the document of any intention by the Police (Australia) to engage in cooperation with the Citizen (Australia) on the specific date of 2014-11-16.</td>
  </tr>
  <tr>
    <td>Tuned-only</td>
    <td>x<br>No. Based on the information provided, it is predicted that Police (Australia) will not engage in material cooperation with Citizen (Australia) on 2014-11-16. This prediction is supported by a series of events leading up to this date, which indicate a pattern of conflict and adversarial interactions between the police and citizens. In the days preceding 2014-11-16, the police have been involved in multiple actions against citizens, including arrests, detentions, and charges on 2014-11-12, 13, and 14. Additionally, there have been accusations against citizens on 2014-11-2, 9, and 16, as well as instances of police using force and military actions against Citizens on various dates. The police have also been involved in investigations, criticisms, and confiscations, further indicating a tense relationship. Despite some expressions of intent to negotiate or meet, the overall pattern of behavior suggests a lack of cooperative engagement. Given this context of ongoing conflict and legal actions, it is reasonable to conclude that material cooperation between the police and citizens is unlikely to occur on 2014-11-16.</td>
  </tr>
  <tr>
    <td>GETER (ours)</td>
    <td>✓<br>Yes. Based on the sequence of events outlined in the document, it is plausible to predict that the Police in Australia will engage in material cooperation with Australian citizens on November 16, 2014. This prediction is supported by a series of interactions between the police and citizens leading up to this date. From November 12 to November 14, the police were actively arresting, detaining, or charging citizens with legal actions, indicating a period of heightened enforcement. However, alongside these actions, the police also made several appeals and requests to citizens on the same dates, suggesting a willingness to engage in dialogue or seek assistance. Additionally, the police expressed an intent to meet or negotiate with citizens on multiple occasions, including November 11 and November 14, which implies a shift towards more collaborative engagement. Despite the confrontational actions, such as the use of force and accusations, the police also demonstrated a capacity for reconciliation by returning or releasing individuals on November 14. This dual approach of enforcement and negotiation, combined with the police’s expressed intent to meet and cooperate, supports the likelihood of material cooperation occurring on November 16, 2014.</td>
  </tr>
</table>

Table 12: Case comparisons between GETER and other methods. While Tuned-only method demonstrate an improved ability to handle the temporal aspects of events (highlighted in blue), they still resulting in negative predictions. In contrast, GETER leverages temporal graph structures to model the evolving patterns of event relationships and effectively identifies cooperative signals (highlighted in red), enabling more accurate predictions.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Types</th>
    <th colspan="4">ICEWS18</th>
    <th colspan="4">WIKI</th>
  </tr>
  <tr>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td rowspan="2">GPT-4o</td>
    <td>zero-shot w/o chains text</td>
    <td>51.64</td>
    <td>36.61</td>
    <td>24.79</td>
    <td>38.32</td>
    <td>69.5</td>
    <td>53.45</td>
    <td>17.77</td>
    <td>47.45</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>60.33</td>
    <td>23.78</td>
    <td>40.72</td>
    <td>42.08</td>
    <td>61.94</td>
    <td>37.44</td>
    <td>40.88</td>
    <td>47.54</td>
  </tr>
  <tr>
    <td rowspan="2">Llama3-8B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>7.68</td>
    <td>24.39</td>
    <td>38.95</td>
    <td>22.93</td>
    <td>48.31</td>
    <td>54.39</td>
    <td>66.46</td>
    <td>52.44</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>55.12</td>
    <td>18.81</td>
    <td>9.14</td>
    <td>28.79</td>
    <td>51.76</td>
    <td>26.43</td>
    <td>1.26</td>
    <td>27.31</td>
  </tr>
  <tr>
    <td rowspan="2">LoRA w/o chains text</td>
    <td>57.47</td>
    <td>47.14</td>
    <td>56.30</td>
    <td>53.66</td>
    <td>84.08</td>
    <td>70.67</td>
    <td>83.36</td>
    <td>79.80</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>62.30</td>
    <td>46.24</td>
    <td>66.46</td>
    <td>58.23</td>
    <td>88.59</td>
    <td>73.29</td>
    <td>81.36</td>
    <td>81.57</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>75.78</td>
    <td>74.09</td>
    <td>87.53</td>
    <td>78.85</td>
    <td>98.99</td>
    <td>90.58</td>
    <td>91.00</td>
    <td>93.79</td>
  </tr>
  <tr>
    <td>ΔImprove</td>
    <td>21.64% 60.24% 31.70% 35.41%</td>
    <td></td>
    <td></td>
    <td></td>
    <td>11.74% 23.59% 11.85% 14.98%</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Qwen2.5-7B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>30.94</td>
    <td>40.53</td>
    <td>25.13</td>
    <td>32.34</td>
    <td>43.51</td>
    <td>53.31</td>
    <td>7.72</td>
    <td>34.54</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>44.22</td>
    <td>48.67</td>
    <td>10.92</td>
    <td>35.40</td>
    <td>46.46</td>
    <td>47.84</td>
    <td>2.47</td>
    <td>32.23</td>
  </tr>
  <tr>
    <td rowspan="2">LoRA w/o chains text</td>
    <td>45.82</td>
    <td>59.83</td>
    <td>66.27</td>
    <td>56.82</td>
    <td>87.16</td>
    <td>80.29</td>
    <td>87.00</td>
    <td>85.04</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>69.68</td>
    <td>60.54</td>
    <td>63.21</td>
    <td>64.48</td>
    <td>88.65</td>
    <td>78.58</td>
    <td>87.36</td>
    <td>85.19</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>74.77</td>
    <td>74.41</td>
    <td>86.79</td>
    <td>78.37</td>
    <td>97.32</td>
    <td>93.33</td>
    <td>94.01</td>
    <td>95.02</td>
  </tr>
  <tr>
    <td>ΔImprove</td>
    <td>7.31% 22.91% 37.28% 21.55%</td>
    <td></td>
    <td></td>
    <td></td>
    <td>9.78% 18.77% 7.61% 11.54%</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Mistral-7B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>1.06</td>
    <td>34.23</td>
    <td>47.64</td>
    <td>26.53</td>
    <td>35.81</td>
    <td>49.71</td>
    <td>55.40</td>
    <td>46.52</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>4.14</td>
    <td>33.06</td>
    <td>41.58</td>
    <td>25.37</td>
    <td>62.98</td>
    <td>44.44</td>
    <td>41.89</td>
    <td>50.37</td>
  </tr>
  <tr>
    <td rowspan="2">LoRA w/o chains text</td>
    <td>58.07</td>
    <td>55.27</td>
    <td>74.46</td>
    <td>62.21</td>
    <td>84.94</td>
    <td>77.82</td>
    <td>83.08</td>
    <td>82.18</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>64.22</td>
    <td>64.63</td>
    <td>76.63</td>
    <td>68.20</td>
    <td>89.29</td>
    <td>86.61</td>
    <td>87.04</td>
    <td>87.73</td>
  </tr>
  <tr>
    <td>GETER</td>
    <td>75.61</td>
    <td>75.94</td>
    <td>87.51</td>
    <td>79.40</td>
    <td>99.28</td>
    <td>94.49</td>
    <td>96.19</td>
    <td>96.81</td>
  </tr>
  <tr>
    <td>ΔImprove</td>
    <td>17.74% 17.50% 14.20% 16.42%</td>
    <td></td>
    <td></td>
    <td></td>
    <td>11.19% 9.10% 10.51% 10.35%</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 13: F1 scores (%) of each model on the ICEWS18 and WIKI test instances. "Overall" represents the weighted average F1 score. w/o chains text refers to the absence of reasoning chain input for LLMs. The best-performing results are highlighted in bold. ΔImprove represents the relative improvements of GETER compared to Tuned-only methods.
Table 14: The semantic similarity performance (%) of each model on the ICEWS18 and WIKI test instances. w/o chains text refers to the absence of reasoning chain input for LLMs. The best-performing results are highlighted in bold.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Types</th>
    <th colspan="3">ICEWS18</th>
    <th>WIKI</th>
  </tr>
  <tr>
    <th>BLEU-4</th>
    <th>rougeL</th>
    <th>METEOR</th>
    <th>BertScore (F1)</th>
    <th>BLEU-4</th>
    <th>rougeL</th>
    <th>METEOR</th>
    <th>BertScore (F1)</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>zero-shot w/o chains text<br>zero-shot</td>
    <td>9.33<br>14.84</td>
    <td>22.67<br>31.16</td>
    <td>29.87<br>37.47</td>
    <td>67.48<br>72.98</td>
    <td>13.25<br>25.98</td>
    <td>28.18<br>41.77</td>
    <td>36.65<br>45.52</td>
    <td>69.10<br>78.69</td>
  </tr>
  <tr>
    <td rowspan="5">Llama3-8B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>4.10</td>
    <td>15.85</td>
    <td>16.20</td>
    <td>61.14</td>
    <td>9.39</td>
    <td>25.41</td>
    <td>27.95</td>
    <td>66.88</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>10.01</td>
    <td>29.52</td>
    <td>27.19</td>
    <td>70.01</td>
    <td>14.67</td>
    <td>36.67</td>
    <td>33.43</td>
    <td>75.85</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>23.55</td>
    <td>35.95</td>
    <td>42.54</td>
    <td>78.02</td>
    <td>48.99</td>
    <td>63.53</td>
    <td>63.08</td>
    <td>87.13</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>37.33</td>
    <td>49.18</td>
    <td>53.05</td>
    <td>83.58</td>
    <td>52.09</td>
    <td>65.27</td>
    <td>66.67</td>
    <td>87.99</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td><b>40.39</b></td>
    <td><b>52.12</b></td>
    <td><b>54.85</b></td>
    <td><b>84.60</b></td>
    <td><b>55.52</b></td>
    <td><b>68.06</b></td>
    <td><b>69.16</b></td>
    <td><b>88.77</b></td>
  </tr>
  <tr>
    <td></td>
    <td><b>ΔImprove</b></td>
    <td><b>8.20%</b></td>
    <td><b>9.58%</b></td>
    <td><b>3.39%</b></td>
    <td><b>1.22%</b></td>
    <td><b>6.59%</b></td>
    <td><b>4.28%</b></td>
    <td><b>3.73%</b></td>
    <td><b>0.89%</b></td>
  </tr>
  <tr>
    <td rowspan="5">Qwen2.5-7B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>7.02</td>
    <td>19.52</td>
    <td>30.09</td>
    <td>65.92</td>
    <td>7.33</td>
    <td>21.46</td>
    <td>34.56</td>
    <td>66.38</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>10.46</td>
    <td>27.97</td>
    <td>26.80</td>
    <td>71.77</td>
    <td>20.21</td>
    <td>36.19</td>
    <td>41.52</td>
    <td>77.61</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>25.50</td>
    <td>37.61</td>
    <td>42.91</td>
    <td>78.56</td>
    <td>51.84</td>
    <td>65.09</td>
    <td>65.59</td>
    <td>87.79</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>37.49</td>
    <td>49.61</td>
    <td>52.07</td>
    <td>83.60</td>
    <td>53.57</td>
    <td>67.19</td>
    <td>67.23</td>
    <td>88.59</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td><b>38.99</b></td>
    <td><b>50.70</b></td>
    <td><b>53.79</b></td>
    <td><b>84.17</b></td>
    <td><b>55.00</b></td>
    <td><b>67.49</b></td>
    <td><b>70.00</b></td>
    <td><b>88.99</b></td>
  </tr>
  <tr>
    <td></td>
    <td><b>ΔImprove</b></td>
    <td><b>4.00%</b></td>
    <td><b>2.20%</b></td>
    <td><b>3.30%</b></td>
    <td><b>0.68%</b></td>
    <td><b>2.67%</b></td>
    <td><b>0.45%</b></td>
    <td><b>4.12%</b></td>
    <td><b>0.45%</b></td>
  </tr>
  <tr>
    <td rowspan="5">Mistral-7B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>7.60</td>
    <td>19.43</td>
    <td>29.38</td>
    <td>65.87</td>
    <td>11.41</td>
    <td>26.23</td>
    <td>31.64</td>
    <td>67.79</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>9.74</td>
    <td>29.00</td>
    <td>26.00</td>
    <td>71.95</td>
    <td>21.25</td>
    <td>40.05</td>
    <td>41.43</td>
    <td>77.27</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>25.46</td>
    <td>37.62</td>
    <td>42.91</td>
    <td>78.67</td>
    <td>51.58</td>
    <td>66.32</td>
    <td>65.29</td>
    <td>87.96</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>36.96</td>
    <td>49.12</td>
    <td>51.70</td>
    <td>83.38</td>
    <td>52.61</td>
    <td>65.40</td>
    <td>66.80</td>
    <td>87.97</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td><b>39.64</b></td>
    <td><b>51.62</b></td>
    <td><b>54.04</b></td>
    <td><b>84.37</b></td>
    <td><b>54.96</b></td>
    <td><b>67.74</b></td>
    <td><b>69.17</b></td>
    <td><b>88.92</b></td>
  </tr>
  <tr>
    <td></td>
    <td><b>ΔImprove</b></td>
    <td><b>7.25%</b></td>
    <td><b>5.09%</b></td>
    <td><b>4.53%</b></td>
    <td><b>1.19%</b></td>
    <td><b>4.47%</b></td>
    <td><b>3.58%</b></td>
    <td><b>3.55%</b></td>
    <td><b>1.08%</b></td>
  </tr>
</table>

Table 15: Precision (%), Recall (%), and F1 scores (%) for each model on the ICEWS14 dataset.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Types</th>
    <th colspan="3">Positive</th>
    <th colspan="3">Negative</th>
    <th colspan="3">Neutral</th>
    <th colspan="3">Overall</th>
  </tr>
  <tr>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>zero-shot w/o chains text<br>zero-shot</td>
    <td>41.89<br>58.53</td>
    <td>72.62<br>67.15</td>
    <td>53.13<br>60.10</td>
    <td>24.63<br>33.05</td>
    <td>16.86<br>5.97</td>
    <td>20.02<br>4.56</td>
    <td>23.08<br>30.78</td>
    <td>9.00<br>48.56</td>
    <td>12.95<br>43.91</td>
    <td>30.76<br>39.45</td>
    <td>35.86<br>39.95</td>
  </tr>
  <tr>
    <td rowspan="5">Llama3-8B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>41.94</td>
    <td>16.26</td>
    <td>21.69</td>
    <td>30.23</td>
    <td>24.57</td>
    <td>27.11</td>
    <td>26.20</td>
    <td>54.67</td>
    <td>35.42</td>
    <td>33.54</td>
    <td>29.38</td>
    <td>27.42</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>40.59</td>
    <td>9.00</td>
    <td>56.51</td>
    <td>21.17</td>
    <td>6.71</td>
    <td>10.20</td>
    <td>44.44</td>
    <td>3.33</td>
    <td>6.20</td>
    <td>38.62</td>
    <td>35.22</td>
    <td>26.70</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>54.16</td>
    <td>73.25</td>
    <td>62.27</td>
    <td>35.28</td>
    <td>38.86</td>
    <td>39.68</td>
    <td>82.59</td>
    <td>34.00</td>
    <td>48.17</td>
    <td>55.99</td>
    <td>50.57</td>
    <td>49.81</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>66.08</td>
    <td>75.25</td>
    <td>70.37</td>
    <td>56.75</td>
    <td>59.43</td>
    <td>58.06</td>
    <td>78.73</td>
    <td>59.83</td>
    <td>67.99</td>
    <td>65.59</td>
    <td>65.57</td>
    <td>65.59</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td><b>71.62</b></td>
    <td><b>78.87</b></td>
    <td><b>75.07</b></td>
    <td><b>66.90</b></td>
    <td><b>67.86</b></td>
    <td><b>67.38</b></td>
    <td><b>88.41</b></td>
    <td><b>75.00</b></td>
    <td><b>81.15</b></td>
    <td><b>74.85</b></td>
    <td><b>74.10</b></td>
    <td><b>74.25</b></td>
  </tr>
  <tr>
    <td rowspan="5">Qwen2.5-7B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>41.51</td>
    <td>16.50</td>
    <td>23.61</td>
    <td>32.18</td>
    <td>62.71</td>
    <td>42.54</td>
    <td>17.94</td>
    <td>12.50</td>
    <td>14.73</td>
    <td>31.67</td>
    <td>30.76</td>
    <td>27.39</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>55.79</td>
    <td>50.62</td>
    <td>53.08</td>
    <td>34.79</td>
    <td>65.00</td>
    <td>43.52</td>
    <td>57.58</td>
    <td>6.33</td>
    <td>11.41</td>
    <td>49.30</td>
    <td>42.76</td>
    <td>38.59</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>68.55</td>
    <td>59.25</td>
    <td>62.82</td>
    <td>51.23</td>
    <td>68.43</td>
    <td>58.59</td>
    <td>83.33</td>
    <td>63.33</td>
    <td>71.97</td>
    <td>66.35</td>
    <td>63.48</td>
    <td>64.03</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>74.32</td>
    <td>74.88</td>
    <td>74.60</td>
    <td>61.58</td>
    <td>70.29</td>
    <td>65.64</td>
    <td>83.64</td>
    <td>69.00</td>
    <td>75.62</td>
    <td>72.73</td>
    <td>71.67</td>
    <td>71.90</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td><b>81.56</b></td>
    <td><b>71.88</b></td>
    <td><b>76.41</b></td>
    <td><b>68.84</b></td>
    <td><b>81.43</b></td>
    <td><b>74.61</b></td>
    <td><b>86.95</b></td>
    <td><b>82.17</b></td>
    <td><b>84.49</b></td>
    <td><b>78.86</b></td>
    <td><b>78.00</b></td>
    <td><b>78.12</b></td>
  </tr>
  <tr>
    <td rowspan="5">Mistral-7B-Instruct</td>
    <td>zero-shot w/o chains text</td>
    <td>68.18</td>
    <td>1.87</td>
    <td>3.65</td>
    <td>38.38</td>
    <td>40.57</td>
    <td>39.44</td>
    <td>33.63</td>
    <td>75.00</td>
    <td>46.44</td>
    <td>48.38</td>
    <td>35.67</td>
    <td>28.71</td>
  </tr>
  <tr>
    <td>zero-shot</td>
    <td>55.56</td>
    <td>13.75</td>
    <td>22.04</td>
    <td>27.43</td>
    <td>27.86</td>
    <td>27.64</td>
    <td>30.65</td>
    <td>60.83</td>
    <td>40.76</td>
    <td>39.06</td>
    <td>31.90</td>
    <td>29.26</td>
  </tr>
  <tr>
    <td>LoRA w/o chains text</td>
    <td>77.89</td>
    <td>46.25</td>
    <td>58.04</td>
    <td>56.13</td>
    <td>78.43</td>
    <td>65.44</td>
    <td>77.13</td>
    <td>83.17</td>
    <td>80.03</td>
    <td>70.42</td>
    <td>67.52</td>
    <td>66.79</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>72.56</td>
    <td>73.38</td>
    <td>72.96</td>
    <td>60.66</td>
    <td>73.57</td>
    <td>66.49</td>
    <td>87.56</td>
    <td>64.50</td>
    <td>74.28</td>
    <td>72.88</td>
    <td>70.90</td>
    <td>71.18</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td><b>83.62</b></td>
    <td><b>72.12</b></td>
    <td><b>77.45</b></td>
    <td><b>69.23</b></td>
    <td><b>83.57</b></td>
    <td><b>75.73</b></td>
    <td><b>87.79</b></td>
    <td><b>82.67</b></td>
    <td><b>85.15</b></td>
    <td><b>80.02</b></td>
    <td><b>78.95</b></td>
    <td><b>79.08</b></td>
  </tr>
</table>

Table 16: Precision (%), Recall (%), and F1 scores (%) for each model on the GDELT dataset.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Types</th>
    <th colspan="3">Positive</th>
    <th colspan="3">Negative</th>
    <th colspan="3">Neutral</th>
    <th colspan="3">Overall</th>
  </tr>
  <tr>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>zero-shot w/o chains text<br>zero-shot</td>
    <td>46.72</td>
    <td>68.19</td>
    <td>55.45</td>
    <td>25.52</td>
    <td>27.21</td>
    <td>26.33</td>
    <td>25.70</td>
    <td>11.06</td>
    <td>15.47</td>
    <td>32.99</td>
    <td>36.36</td>
    <td>33.03</td>
  </tr>
  <tr>
    <td>Llama3-8B-Instruct</td>
    <td>zero-shot w/o chains text<br>zero-shot<br>LoRA w/o chains text<br>LoRA<br>GETER</td>
    <td>42.98<br>41.62<br>51.29<br>47.92<br>73.37</td>
    <td>6.81<br>91.11<br>91.25<br>68.08<br>84.58</td>
    <td>11.75<br>57.14<br>65.86<br>71.32<br>58.38</td>
    <td>31.60<br>25.28<br>51.73<br>40.86<br>83.84</td>
    <td>26.76<br>13.38<br>30.74<br>40.16<br>99.41</td>
    <td>28.98<br>17.50<br>38.56<br>51.77<br>97.00</td>
    <td>29.20<br>44.35<br>93.87<br>61.46<br>91.00</td>
    <td>60.61<br>8.33<br>53.33<br>49.24<br>97.87</td>
    <td>39.41<br>14.03<br>68.02<br>74.40<br>91.48</td>
    <td>34.81<br>37.10<br>59.13<br>67.48<br>82.48</td>
    <td>30.63<br>38.93<br>59.13<br>67.48<br>81.84</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>zero-shot w/o chains text<br>zero-shot<br>LoRA w/o chains text<br>LoRA<br>GETER</td>
    <td>46.63<br>60.88<br>74.20<br>84.18<br>71.68</td>
    <td>24.03<br>50.69<br>42.50<br>55.42<br>86.11</td>
    <td>30.80<br>40.81<br>55.33<br>66.83<br>78.23</td>
    <td>54.85<br>33.93<br>56.24<br>62.07<br>84.36</td>
    <td>39.45<br>38.53<br>88.09<br>82.79<br>64.26</td>
    <td>18.53<br>23.08<br>49.49<br>82.92<br>72.95</td>
    <td>15.84<br>0.91<br>85.80<br>84.09<br>89.10</td>
    <td>15.82<br>39.75<br>74.95<br>76.48<br>81.34</td>
    <td>32.07<br>39.89<br>70.39<br>74.03<br>80.23</td>
    <td>30.97<br>38.59<br>70.39<br>73.72<br>80.23</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct</td>
    <td>zero-shot w/o chains text<br>zero-shot<br>LoRA w/o chains text<br>LoRA<br>GETER</td>
    <td>61.70<br>59.30<br>75.97<br>77.44<br>75.67</td>
    <td>4.03<br>30.42<br>70.81<br>68.19<br>82.50</td>
    <td>7.65<br>17.73<br>78.10<br>72.53<br>78.94</td>
    <td>34.78<br>35.97<br>77.39<br>68.77<br>82.85</td>
    <td>31.76<br>25.44<br>38.12<br>75.44<br>71.03</td>
    <td>32.32<br>28.80<br>39.12<br>71.95<br>76.48</td>
    <td>33.98<br>46.13<br>62.79<br>82.94<br>88.29</td>
    <td>71.67<br>70.65<br>95.61<br>84.18<br>90.38</td>
    <td>43.93<br>45.25<br>68.84<br>76.34<br>82.08</td>
    <td>34.85<br>37.52<br>65.83<br>76.12<br>81.80</td>
  </tr>
</table>

Table 17: Precision (%), Recall (%), and F1 scores (%) for each model on the ICEWS05-15 dataset.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Types</th>
    <th colspan="3">Positive</th>
    <th colspan="3">Negative</th>
    <th colspan="3">Neutral</th>
    <th colspan="3">Overall</th>
  </tr>
  <tr>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>zero-shot w/o chains text<br>zero-shot</td>
    <td>47.43<br>53.09</td>
    <td>56.67<br>69.87</td>
    <td>51.64<br>60.33</td>
    <td>38.20<br>34.51</td>
    <td>35.14<br>18.14</td>
    <td>36.61<br>23.78</td>
    <td>26.79<br>38.12</td>
    <td>23.08<br>43.69</td>
    <td>24.79<br>40.72</td>
    <td>37.96<br>42.26</td>
    <td>39.10<br>44.52</td>
    <td>38.32<br>42.08</td>
  </tr>
  <tr>
    <td>Llama3-8B-Instruct</td>
    <td>zero-shot w/o chains text<br>zero-shot<br>LoRA w/o chains text<br>LoRA<br>GETER</td>
    <td>25.19<br>39.37<br>75.20<br>74.41<br>72.42</td>
    <td>4.53<br>91.87<br>57.33<br>51.51<br>79.47</td>
    <td>7.68<br>55.12<br>57.33<br>62.30<br>75.78</td>
    <td>31.25<br>33.09<br>39.33<br>43.43<br>78.87</td>
    <td>20.00<br>13.14<br>38.12<br>35.87<br>74.09</td>
    <td>24.39<br>18.81<br>48.77<br>48.00<br>80.06</td>
    <td>27.82<br>45.83<br>70.65<br>87.08<br>88.00</td>
    <td>64.92<br>5.08<br>56.30<br>87.53<br>87.53</td>
    <td>38.95<br>9.14<br>56.30<br>87.53<br>87.53</td>
    <td>28.02<br>39.28<br>61.44<br>79.10<br>79.10</td>
    <td>28.38<br>38.76<br>52.90<br>78.90<br>78.90</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>zero-shot w/o chains text<br>zero-shot<br>LoRA w/o chains text<br>LoRA<br>GETER</td>
    <td>43.98<br>50.81<br>72.00<br>69.50<br>74.87</td>
    <td>23.87<br>35.73<br>33.60<br>49.87<br>74.77</td>
    <td>30.94<br>44.22<br>45.82<br>69.68<br>74.77</td>
    <td>34.15<br>35.55<br>45.02<br>52.46<br>78.62</td>
    <td>49.86<br>41.74<br>89.14<br>51.70<br>76.80</td>
    <td>40.53<br>38.67<br>93.21<br>50.62<br>84.81</td>
    <td>25.44<br>35.29<br>92.31<br>62.31<br>88.75</td>
    <td>25.54<br>6.46<br>66.27<br>63.21<br>89.42</td>
    <td>25.13<br>43.49<br>69.29<br>64.63<br>86.79</td>
    <td>33.45<br>40.48<br>57.71<br>68.35<br>78.29</td>
    <td>32.34<br>35.40<br>56.82<br>64.48<br>78.37</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct</td>
    <td>zero-shot w/o chains text<br>zero-shot<br>LoRA w/o chains text<br>LoRA<br>GETER</td>
    <td>57.14<br>69.57<br>69.52<br>73.41<br>75.36</td>
    <td>0.53<br>2.13<br>49.87<br>57.07<br>75.87</td>
    <td>1.06<br>4.14<br>58.07<br>64.22<br>75.61</td>
    <td>35.71<br>35.04<br>62.68<br>62.80<br>73.98</td>
    <td>32.86<br>31.29<br>49.43<br>66.57<br>78.00</td>
    <td>34.23<br>30.10<br>55.27<br>70.45<br>75.94</td>
    <td>34.51<br>30.10<br>61.19<br>80.61<br>90.61</td>
    <td>76.92<br>67.23<br>95.08<br>84.62<br>84.62</td>
    <td>47.64<br>41.58<br>64.66<br>68.46<br>87.51</td>
    <td>42.99<br>34.20<br>56.53<br>66.46<br>79.62</td>
    <td>34.95<br>32.00<br>53.00<br>63.71<br>79.29</td>
  </tr>
</table>

Table 18: Precision (%), Recall (%), and F1 scores (%) for each model on the ICEWS18 dataset.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Types</th>
    <th colspan="3">Positive</th>
    <th colspan="3">Negative</th>
    <th colspan="3">Neutral</th>
    <th colspan="3">Overall</th>
  </tr>
  <tr>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>zero-shot w/o chains text<br>zero-shot</td>
    <td>66.93<br>52.51</td>
    <td>72.33<br>75.50</td>
    <td>69.53<br>61.94</td>
    <td>40.85<br>33.80</td>
    <td>47.77<br>37.44</td>
    <td>53.45<br>48.44</td>
    <td>93.94<br>88.42</td>
    <td>9.81<br>26.58</td>
    <td>17.77<br>40.88</td>
    <td>68.07<br>58.83</td>
    <td>53.00<br>49.10</td>
    <td>47.45<br>47.54</td>
  </tr>
  <tr>
    <td>Llama3-8B-Instruct</td>
    <td>zero-shot w/o chains text<br>zero-shot<br>LoRA w/o chains text<br>LoRA<br>GETER</td>
    <td>68.98<br>39.23<br>91.35<br>96.25<br>99.13</td>
    <td>37.18<br>76.08<br>77.89<br>82.06<br>98.85</td>
    <td>48.31<br>51.76<br>84.08<br>85.89<br>98.99</td>
    <td>51.74<br>27.01<br>67.83<br>70.98<br>95.89</td>
    <td>57.34<br>25.87<br>73.76<br>75.75<br>93.33</td>
    <td>54.39<br>24.63<br>70.67<br>73.29<br>90.58</td>
    <td>47.19<br>100.00<br>78.48<br>75.95<br>96.13</td>
    <td>66.46<br>0.63<br>88.89<br>87.59<br>86.39</td>
    <td>56.53<br>1.26<br>83.36<br>81.36<br>91.00</td>
    <td>53.00<br>55.78<br>80.31<br>81.88<br>94.14</td>
    <td>52.44<br>35.73<br>80.31<br>80.31<br>93.78</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>zero-shot w/o chains text<br>zero-shot<br>LoRA w/o chains text<br>LoRA<br>GETER</td>
    <td>52.44<br>73.29<br>91.93<br>95.30<br>93.42</td>
    <td>37.18<br>34.01<br>82.86<br>94.75<br>97.32</td>
    <td>43.51<br>46.46<br>87.16<br>88.65<br>97.32</td>
    <td>37.83<br>32.69<br>76.22<br>85.31<br>96.28</td>
    <td>90.21<br>89.16<br>84.82<br>72.84<br>90.56</td>
    <td>53.31<br>47.84<br>80.29<br>78.58<br>93.33</td>
    <td>61.90<br>50.00<br>85.76<br>86.39<br>93.71</td>
    <td>4.11<br>1.27<br>88.27<br>88.35<br>94.01</td>
    <td>7.72<br>2.47<br>87.00<br>87.36<br>90.55</td>
    <td>57.62<br>53.30<br>85.14<br>84.93<br>95.05</td>
    <td>45.12<br>39.73<br>85.25<br>86.02<br>95.02</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct</td>
    <td>zero-shot w/o chains text<br>zero-shot<br>LoRA w/o chains text<br>LoRA<br>GETER</td>
    <td>73.87<br>75.20<br>91.93<br>93.66<br>98.86</td>
    <td>23.63<br>54.18<br>82.86<br>85.30<br>99.71</td>
    <td>35.81<br>48.28<br>87.16<br>89.29<br>99.28</td>
    <td>54.85<br>38.95<br>71.45<br>82.52<br>99.61</td>
    <td>45.91<br>47.44<br>80.29<br>86.16<br>94.99</td>
    <td>42.26<br>41.69<br>87.04<br>86.08<br>92.67</td>
    <td>80.38<br>41.69<br>87.04<br>88.03<br>100.00</td>
    <td>55.40<br>42.09<br>87.04<br>87.88<br>96.19</td>
    <td>56.72<br>31.53<br>82.45<br>87.86<br>96.84</td>
    <td>49.10<br>42.92<br>82.45<br>87.73<br>96.81</td>
  </tr>
</table>

Table 19: Precision (%), Recall (%), and F1 scores (%) for each model on the WIKI dataset.
<table>
  <tr>
    <th rowspan="2">Datasets</th>
    <th rowspan="2">Methods</th>
    <th colspan="3">Positive</th>
    <th colspan="3">Negative</th>
    <th colspan="3">Neutral</th>
    <th colspan="3">Overall</th>
  </tr>
  <tr>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="5">ICEWS14</td>
    <td>REGCN</td>
    <td>58.02</td>
    <td>48.38</td>
    <td>52.76</td>
    <td>48.89</td>
    <td>62.86</td>
    <td>55.00</td>
    <td>80.11</td>
    <td>71.17</td>
    <td>75.38</td>
    <td>61.29</td>
    <td>59.71</td>
    <td>59.97</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>70.51</td>
    <td>45.12</td>
    <td>55.03</td>
    <td>53.03</td>
    <td>71.29</td>
    <td>60.82</td>
    <td>75.43</td>
    <td>81.33</td>
    <td>78.27</td>
    <td>66.09</td>
    <td>64.19</td>
    <td>63.60</td>
  </tr>
  <tr>
    <td>CEN</td>
    <td>60.29</td>
    <td>62.25</td>
    <td>61.25</td>
    <td>56.05</td>
    <td>50.29</td>
    <td>53.01</td>
    <td>73.53</td>
    <td>79.17</td>
    <td>76.24</td>
    <td>62.66</td>
    <td>63.10</td>
    <td>62.79</td>
  </tr>
  <tr>
    <td>SiMFy</td>
    <td>68.84</td>
    <td>43.63</td>
    <td>53.40</td>
    <td>53.43</td>
    <td>76.86</td>
    <td>63.03</td>
    <td>79.86</td>
    <td>78.00</td>
    <td>78.92</td>
    <td>66.85</td>
    <td>64.52</td>
    <td>63.90</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td>83.62</td>
    <td>72.12</td>
    <td>77.45</td>
    <td>69.23</td>
    <td>83.57</td>
    <td>75.73</td>
    <td>87.79</td>
    <td>82.67</td>
    <td>85.15</td>
    <td>80.02</td>
    <td>78.95</td>
    <td>79.08</td>
  </tr>
  <tr>
    <td rowspan="5">GDELT</td>
    <td>REGCN</td>
    <td>61.93</td>
    <td>53.87</td>
    <td>57.62</td>
    <td>55.61</td>
    <td>63.71</td>
    <td>59.39</td>
    <td>84.05</td>
    <td>84.31</td>
    <td>84.18</td>
    <td>66.56</td>
    <td>66.28</td>
    <td>66.22</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>65.58</td>
    <td>57.63</td>
    <td>61.34</td>
    <td>59.39</td>
    <td>66.43</td>
    <td>62.71</td>
    <td>87.05</td>
    <td>88.92</td>
    <td>87.98</td>
    <td>70.05</td>
    <td>69.95</td>
    <td>69.84</td>
  </tr>
  <tr>
    <td>CEN</td>
    <td>64.80</td>
    <td>56.37</td>
    <td>60.29</td>
    <td>57.21</td>
    <td>66.29</td>
    <td>61.42</td>
    <td>87.40</td>
    <td>86.46</td>
    <td>86.93</td>
    <td>69.16</td>
    <td>68.70</td>
    <td>68.71</td>
  </tr>
  <tr>
    <td>SiMFy</td>
    <td>63.11</td>
    <td>63.50</td>
    <td>63.30</td>
    <td>60.39</td>
    <td>61.43</td>
    <td>60.91</td>
    <td>89.42</td>
    <td>87.08</td>
    <td>88.23</td>
    <td>70.18</td>
    <td>69.95</td>
    <td>70.06</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td>76.21</td>
    <td>51.25</td>
    <td>61.29</td>
    <td>58.15</td>
    <td>84.57</td>
    <td>68.92</td>
    <td>92.76</td>
    <td>84.77</td>
    <td>88.59</td>
    <td>75.33</td>
    <td>72.23</td>
    <td>72.02</td>
  </tr>
  <tr>
    <td rowspan="5">ICEWS05-15</td>
    <td>REGCN</td>
    <td>66.96</td>
    <td>63.33</td>
    <td>65.10</td>
    <td>65.57</td>
    <td>61.62</td>
    <td>63.53</td>
    <td>79.05</td>
    <td>88.64</td>
    <td>83.57</td>
    <td>70.38</td>
    <td>70.87</td>
    <td>70.50</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>65.27</td>
    <td>69.44</td>
    <td>67.29</td>
    <td>65.30</td>
    <td>63.09</td>
    <td>64.17</td>
    <td>91.05</td>
    <td>87.88</td>
    <td>89.44</td>
    <td>73.54</td>
    <td>73.25</td>
    <td>73.36</td>
  </tr>
  <tr>
    <td>CEN</td>
    <td>67.45</td>
    <td>59.58</td>
    <td>63.27</td>
    <td>61.33</td>
    <td>70.44</td>
    <td>65.57</td>
    <td>88.02</td>
    <td>85.76</td>
    <td>86.88</td>
    <td>72.02</td>
    <td>71.55</td>
    <td>71.59</td>
  </tr>
  <tr>
    <td>SiMFy</td>
    <td>69.03</td>
    <td>66.25</td>
    <td>67.61</td>
    <td>65.45</td>
    <td>68.53</td>
    <td>66.95</td>
    <td>89.35</td>
    <td>88.94</td>
    <td>89.14</td>
    <td>74.36</td>
    <td>74.27</td>
    <td>74.29</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td>75.67</td>
    <td>82.50</td>
    <td>78.94</td>
    <td>82.85</td>
    <td>71.03</td>
    <td>76.48</td>
    <td>88.29</td>
    <td>92.58</td>
    <td>90.38</td>
    <td>82.08</td>
    <td>81.94</td>
    <td>81.80</td>
  </tr>
  <tr>
    <td rowspan="5">ICEWS18</td>
    <td>REGCN</td>
    <td>59.34</td>
    <td>65.20</td>
    <td>62.13</td>
    <td>62.82</td>
    <td>55.29</td>
    <td>58.81</td>
    <td>80.91</td>
    <td>82.15</td>
    <td>81.53</td>
    <td>67.18</td>
    <td>67.14</td>
    <td>67.03</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>65.38</td>
    <td>58.93</td>
    <td>61.99</td>
    <td>62.58</td>
    <td>66.43</td>
    <td>64.45</td>
    <td>82.97</td>
    <td>86.92</td>
    <td>84.90</td>
    <td>69.89</td>
    <td>70.10</td>
    <td>69.90</td>
  </tr>
  <tr>
    <td>CEN</td>
    <td>60.60</td>
    <td>59.07</td>
    <td>59.82</td>
    <td>59.38</td>
    <td>62.43</td>
    <td>60.86</td>
    <td>80.25</td>
    <td>78.15</td>
    <td>79.19</td>
    <td>66.28</td>
    <td>66.10</td>
    <td>66.16</td>
  </tr>
  <tr>
    <td>SiMFy</td>
    <td>64.71</td>
    <td>57.47</td>
    <td>60.88</td>
    <td>61.29</td>
    <td>64.00</td>
    <td>62.61</td>
    <td>79.23</td>
    <td>85.69</td>
    <td>82.34</td>
    <td>68.07</td>
    <td>68.38</td>
    <td>68.10</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td>75.36</td>
    <td>75.87</td>
    <td>75.61</td>
    <td>73.98</td>
    <td>78.00</td>
    <td>75.94</td>
    <td>90.61</td>
    <td>84.62</td>
    <td>87.51</td>
    <td>79.62</td>
    <td>79.29</td>
    <td>79.40</td>
  </tr>
  <tr>
    <td rowspan="5">WIKI</td>
    <td>REGCN</td>
    <td>81.76</td>
    <td>69.74</td>
    <td>75.27</td>
    <td>72.32</td>
    <td>68.53</td>
    <td>70.38</td>
    <td>70.94</td>
    <td>85.76</td>
    <td>77.65</td>
    <td>75.31</td>
    <td>74.71</td>
    <td>74.59</td>
  </tr>
  <tr>
    <td>CENET</td>
    <td>77.95</td>
    <td>74.35</td>
    <td>76.11</td>
    <td>77.19</td>
    <td>76.92</td>
    <td>77.06</td>
    <td>81.38</td>
    <td>85.76</td>
    <td>83.51</td>
    <td>78.86</td>
    <td>78.93</td>
    <td>78.86</td>
  </tr>
  <tr>
    <td>CEN</td>
    <td>77.50</td>
    <td>71.47</td>
    <td>74.36</td>
    <td>78.44</td>
    <td>73.78</td>
    <td>76.04</td>
    <td>77.22</td>
    <td>87.97</td>
    <td>82.25</td>
    <td>77.69</td>
    <td>77.66</td>
    <td>77.49</td>
  </tr>
  <tr>
    <td>SiMFy</td>
    <td>78.25</td>
    <td>79.83</td>
    <td>79.03</td>
    <td>82.03</td>
    <td>73.43</td>
    <td>77.39</td>
    <td>78.76</td>
    <td>84.49</td>
    <td>81.53</td>
    <td>79.56</td>
    <td>79.45</td>
    <td>79.40</td>
  </tr>
  <tr>
    <td><b>GETER</b></td>
    <td>98.86</td>
    <td>99.71</td>
    <td>99.28</td>
    <td>99.61</td>
    <td>89.86</td>
    <td>94.49</td>
    <td>92.67</td>
    <td>100.00</td>
    <td>96.19</td>
    <td>97.02</td>
    <td>96.84</td>
    <td>96.81</td>
  </tr>
</table>

Table 20: Precision (%), Recall (%), and F1 scores (%) for each graph-based model across different datasets. "Overall" represents the weighted average F1 score.

Reviewer paper summaries: This paper introduces a novel framework named GETER aimed at improving the explainability of temporal reasoning in large language models (LLMs). The authors first propose a comprehensive benchmark (ETR) covering various temporal granularities to systematically evaluate LLMs' capabilities in explainable temporal reasoning.

This paper focuses on explainable temporal reasoning (ETR) using Large Language Models (LLMs). First, they propose a new ETR Benchmark across different granularities, with positive, negative, and neutral samples. Furthermore, a model framework called GETER is proposed for the task, where Temporal knowledge graphs (TKGs) are incorporated via a temporal encoder and structure-text adapter. The model is evaluated using LLaMA, Qwen, and Mistral LLMs and shows strong performance in accuracy (F1 score) and explanation quality (BLEU, ROUGE, METEOR, BERTScore) across multiple datasets.
Summary of strengths: -- GETER effectively bridges the gap between graph structures and text. By leveraging temporal knowledge graphs and a structure-text adapter, the framework enhances the explainability of LLMs' reasoning processes. 

-- The experimental results demonstrate that GETER achieves good performance on multiple datasets. The framework not only improves prediction accuracy but also significantly enhances the quality of generated explanations.

- focus on the understudied field of explainable temporal reasoning,
- propose a well-constructed benchmark, with human-verified GPT-4o generated explanations,
- propose a framework GETER that integrates a temporal encoder and a structure-text adapter for performance enhancement,
- include detailed experiments across 5 dataset, 3 llm backbones, and multiple model and setting variants
Summary of weaknesses: -- The novelty of the paper is overclaimed. ETR is proposed as a benchmark. Similar strategies and datasets have also been employed in other knowledge graph (KG) or temporal knowledge graph (TKG) research efforts.

-- In the experimental section, the authors solely present results for GETER. To ensure a thorough and balanced evaluation, it is essential to include other comparable reasoning methods for knowledge graphs (KGs) or temporal knowledge graphs (TKGs).

-- The paper mentions that not all reasoning chains are beneficial for a given query and some may introduce noise. This suggests that the framework might benefit from a mechanism to filter or prioritize reasoning chains based on their relevance. The current approach of using all reasoning chains indiscriminately could potentially degrade performance in complex scenarios.

- no analysis / illustration why GPT-4o performs bad in the task but still reasonable to be used to generate ground truth,
- why don’t you report results on datasets with different time granularity in main body given that multiple temporal granularities is one of the highlight,
- what about accuracy performance comparisons to other sota graph-based / embedding-based methods and other llm-based methods,
- what about the complexity of incorporating the graph embedding module
- why traditional benchamrks results are not provided and compared in tables? (at least in appendix could be helpful). Authors only put LLM-based results in main table and compared mostly with different variations of themself, why?
- authors could also discuss complexity or runtime in comparsion with traditional embedding based methods. E.g., if with LLM, the training time could be very expensive.
- the authors could also provide sample efficiency as in GenTKG, since in GenTKG, 16 samples could already reach a comparable results. However, due to the adapter (many parameters) in this model, sample efficiency could be a huge issue.
Comments, suggestions and typos: Can the authors elaborate on potential strategies for filtering or prioritizing reasoning chains based on their relevance to the query? 

Could the authors provide a more detailed analysis of potential biases in the ETR benchmark? Specifically, how might these biases affect the model's predictions and explanations, and what steps can be taken to further mitigate them?

- missing legends in figure 4 and figure 5
- Many very first attempts on temporal KGs representation learning [1, 2] and concurrent benchmarks [3] are totally ignored or not compared at all. A comprehensive literature search is necessary. [1] Embedding models for episodic knowledge graphs. [2] Graph hawkes neural network for forecasting on temporal knowledge graphs [3] Learning neural ordinary equations for forecasting future links on temporal knowledge graphs
Overall assessment scores: 2.5

3