ETRQA: A Comprehensive Benchmark for Evaluating Event Temporal Reasoning Abilities of Large Language Models
Authors: Sigang Luo, Yinan Liu, Dongying Lin, Yingying Zhai, Bin Wang, Xiaochun Yang, Junpeng Liu
Accepted at: acl25

ETRQA: A Comprehensive Benchmark for Evaluating Event Temporal Reasoning Abilities of Large Language Models

Sigang Luo1*, Yinan Liu1*, Dongying Lin1, Yingying Zhai1†,
Bin Wang1,3, Xiaochun Yang1,3, Junpeng Liu2
1School of Computer Science and Engineering, Northeastern University, Shenyang, China
2The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
3National Frontiers Science Center for Industrial Intelligence and Systems optimization, Northeastern University, Shenyang, China
2301925@stu.neu.edu.cn, liuyinan@cse.neu.edu.cn, 2472128@stu.neu.edu.cn
{zyy, binwang, yangxc}@mail.neu.edu.cn, junpengliu@hkust-gz.edu.cn

Abstract

Event temporal reasoning (ETR) aims to model and reason about the relationships between events and time, as well as between events in the real world. Proficiency in ETR is a significant indicator that a large language model (LLM) truly understands the physical world. Previous question-answering datasets available for evaluating the ETR ability lack a systematic taxonomy and pay limited attention to compound questions. In this paper, we propose a unified taxonomy for event temporal questions and construct a comprehensive benchmark ETRQA, to evaluate the ETR abilities of LLMs based on this taxonomy. ETRQA not only inherits and expands the evaluation content of existing datasets but also contains multiple categories of compound questions. We evaluate two leading LLM series, Llama and Qwen, on ETRQA across various settings. Our experimental results indicate that large-scale LLMs exhibit certain ETR abilities. Yet they do not perform well in solving specific types of reasoning tasks, including reasoning involving time spans, reasoning for compound questions, and reasoning with fine temporal granularity. Additionally, we hope ETRQA can benefit the temporal reasoning research community for future studies.1

1 Introduction

Recently, large language models (LLMs) have demonstrated impressive performance in numerous reasoning tasks (Qiao et al., 2023; Huang and Chang, 2023; Zhao et al., 2024; Chang et al., 2024). However, it has been observed that they still underperform in a fundamental aspect of human cognition, namely temporal reasoning (Chu et al., 2024; Wang and Zhao, 2024). Temporal reasoning is typically categorized into three levels: symbolic, commonsense, and event temporal reasoning (Chu et al., 2024). Event temporal reasoning (ETR) is the most comprehensive of these levels, as it not only encompasses the understanding and application of the first two types of reasoning but also involves modeling and reasoning about the relationships between events and time, as well as between events in the real world (Tan et al., 2023). The proficient ETR ability is an important indicator that an LLM truly grasps the physical world. Therefore, we focus on the task of ETR.

Most question-answering datasets available for evaluating the ETR ability are constructed using temporal knowledge graphs. They generally generate questions based on facts that evolve over time (Saxena et al., 2021; Chen et al., 2021; Tan et al., 2023; Chen et al., 2023; Tan et al., 2024; Fatemi et al., 2024). For example, the question “What team did LeBron James play for in 2009?” is based on the facts that LeBron James has played for different teams over various periods. Each fact can be regarded as a temporal event, and we refer to such questions as event temporal questions in this paper. To answer such questions, LLMs need to perform ETR. However, there exist two issues in these previous datasets: (1) due to the lack of a systematic taxonomy, existing datasets are often designed with several empirically defined question types, which results in incomplete evaluations, as shown in Table 1; (2) existing datasets rarely focus on compound questions, which are critical for evaluating the ETR abilities of LLMs.

To address the above issues, we first establish a unified taxonomy based on previous studies. This taxonomy categorizes event temporal questions by question composition and answer type, providing a standardized reference that can be widely adopted. Based on this taxonomy, we propose a question-answering dataset for ETR, named ETRQA. ETRQA inherits the evaluation content of existing datasets while expanding upon them
through a more comprehensive coverage and examination of question composition and answer type. Specifically, ETRQA contains multiple types of compound questions, which make the dataset more complex and challenging. For instance, to solve the compound question “What was the team that LeBron James played for the longest time from 2009 to 2015?”, LLMs need to not only understand the temporal constraint “from 2009 to 2015” but also compare the durations of events to find “for the longest time” within that range. It is clear that such compound questions are well-suited for evaluating the comprehensive ETR abilities of LLMs.

To evaluate the ETR abilities of LLMs, we adopt two industry-leading LLM series: the Llama series (Grattafiore et al., 2024) and the Qwen series (Yang et al., 2025) on ETRQA. The latest models, Llama-3.3-70B-Instruct and Qwen2.5-72B-Instruct, have demonstrated competitive performance, matching or even surpassing GPT-4o (Hurst et al., 2024) on many benchmarks, making them highly representative. We show the effects of prompting strategies, instruction tuning, model sizes, and general reasoning distillation in the task of ETR via LLMs. Furthermore, we analyze and discuss the difficulties and challenges LLMs face. The experimental results show that: (1) large-scale LLMs demonstrate a certain level of ETR abilities compared with small-scale LLMs, but there is still room for improvement; (2) careful thought before answering and post-training techniques (e.g., instruction tuning and general reasoning distillation) are crucial for enhancing the models’ ETR abilities; (3) LLMs struggle with reasoning involving time spans and show a decline in performance when reasoning for compound questions and reasoning with fine temporal granularity.

Our contributions can be summarized as follows:

(1) We propose a unified taxonomy for event temporal questions, providing a standardized reference that can be widely adopted.

(2) By expanding the evaluation content of existing datasets and systematically designing multiple types of compound questions, we construct a comprehensive and extensive benchmark for ETR evaluation.

(3) We conduct a quantitative evaluation of two industry-leading LLM series. Our analysis examines the effects of prompting strategies, instruction tuning, model sizes, and general reasoning distillation, offering valuable insights into the challenges and potential improvements in ETR.

2 A Taxonomy of Event Temporal Questions

The proposed taxonomy of event temporal questions, shown in Table 1, consists of two aspects: (1) question composition and (2) answer type.

2.1 Question Composition

An event temporal question can either be a non-compound question that contains only a temporal constraint or a temporal comparison, or a compound question that includes both a temporal constraint and a temporal comparison.

Temporal constraint. A temporal constraint limits the time range of a question. It consists of a temporal signal and a temporal expression. For example, in the temporal constraint “in 2009”, the temporal signal is “in” and the temporal expression is “2009”. Inspired by previous work (Jia et al., 2018; Chen et al., 2021), we identify four types of temporal signals: DURING, IN, BEFORE, and AFTER. The typical trigger words for these temporal signal types are as follows: (1) DURING: “during” and “from ... to ...”; (2) IN: “in”; (3) BEFORE: “before” and “prior to”; (4) AFTER: “after” and “following”.

Temporal constraints can be divided into three categories based on the form of the temporal expression: (1) explicit temporal constraint denotes the time range that can be directly determined without additional context or computation (e.g., “in 2009” and “from 2009 to 2015”); (2) implicit temporal constraint requires determining the time range based on specific temporal events (e.g., “in the time when the Nuggets won the NBA championship” and “during the period when James was playing for the Heat”); (3) relative temporal constraint (Tan et al., 2024) involves calculating the target time based on a reference time and the specified time span to determine the time range (e.g., “in the second year before 2001” is equal to “in 1999” and “during the 2 years before 2002” is equal to “from 2000 to 2001”).

Explicit and implicit temporal constraints involve all four temporal signal types, while relative temporal constraints involve only the temporal signal types DURING and IN, as they require considering both the reference and target times.

Temporal comparison. Temporal events can be compared from two aspects (Xiong et al., 2024): (1) Order and (2) Duration. Order involves determining the chronological order of events. For example,
<table>
  <tr>
    <th>Dataset</th>
    <th>CronQuestions (Saxena et al., 2021)</th>
    <th>TimeQA (Chen et al., 2021)</th>
    <th>TempReason (Tan et al., 2023)</th>
    <th>MultiTQ (Chen et al., 2023)</th>
    <th>Complex-TR (Tan et al., 2024)</th>
    <th>ToT (Faiemi et al., 2024)</th>
    <th>ETRQA</th>
  </tr>
  <tr>
    <th colspan="8">Question Composition</th>
  </tr>
  <tr>
    <td rowspan="3">Temporal constraint</td>
    <td>Explicit<br><i>DURING IN BEFORE/AFTER</i></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Implicit<br><i>DURING IN BEFORE/AFTER</i></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Relative<br><i>DURING IN</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal comparison</td>
    <td>Order<br>Duration</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Compound</td>
    <td>Explicit + Order<br>Implicit + Order<br>Relative + Order<br>Explicit + Duration<br>Implicit + Duration<br>Relative + Duration</td>
    <td></td>
    <td></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <th colspan="8">Answer Type</th>
  </tr>
  <tr>
    <td>Entity</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Time</td>
    <td>Time point<br>Time interval<br>Time span</td>
    <td>✓</td>
    <td></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Numeric</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 1: Summary of event temporal question-answering datasets according to our proposed taxonomy.

in the question “What was the team that LeBron James played for the first time?”, the ordinal term ‘first’ indicates a comparison of order. Duration involves comparing the time span of events. For example, in the question “What was the team that LeBron James played for the longest time?”, the term “longest” indicates a comparison of durations.

**Compound.** Temporal constraints and temporal comparisons can appear independently in questions or be combined to form more complex questions (Chen et al., 2023), an important category that is often overlooked by existing datasets. We introduce 14 types of compound questions by combining three kinds of temporal constraints excluding the temporal signal type \( IN \) and two kinds of temporal comparisons, as shown in Table 1. Note that \( IN \) confines the time range to a single point, making any comparison impossible. Therefore, compound questions involve performing a temporal comparison on temporal events that satisfy the temporal constraint, which are used to better evaluate the comprehensive ETR ability.

**2.2 Answer Type**

The type of answer is determined by the question word or phrase. We categorize the answers to event temporal questions into three types: (1) Entity involves identifying entities using question words like “Who”. (2) Time is relevant for time-related questions, including time points (e.g., 2001) using question words like “When”, time intervals (e.g., between 2000 and 2001) using question phrases like “During which period”, and the time span (e.g., 2 years) using question phrases like “How long”. (3) Numeric involves counting entities, time points, or time intervals using question phrases like “How many”, and is relevant for statistical questions.

**3 Dataset Construction**

**3.1 Temporal Event Preprocessing**

We use the Wikidata (Vrandečić and Krötzsch, 2014) dump of September 2024 as the data source for extracting temporal events. Temporal events can be classified into two categories based on different annotating ways: (1) temporal interval events and (2) temporal point events. A temporal interval event can be represented as a quintuple \( \langle s, r, o, t_s, t_e \rangle \), where \( s \) is a subject, \( r \) is a relation, \( o \) is an object, \( t_s \) is its start time and \( t_e \) is its end time. A temporal point event can be represented as a quadruple \( \langle s, r, o, t_p \rangle \), where \( t_p \) is the event time. We select 16 relations that represent temporal interval events and 6 relations for temporal point events, as shown in Appendix A.2. We then extract all corresponding temporal events and group them by subject and relation to form temporal event groups. These include the temporal interval event group \( \{ \langle s, r, o_i, t_{s_i}, t_{e_i} \rangle \mid i = 1, 2, \ldots, N \} \) and the temporal point event group \( \{ \langle s, r, o_i, t_{p_i} \rangle \mid i = 1, 2, \ldots, N \} \), where \( N \) represents the number of events. Within these groups, we remove those containing only a single event or where the events involve the same object to avoid pseudo-temporal rea-
soning (Chen et al., 2022). Temporal event groups are the basis for constructing event temporal questions since they reflect the evolving process of facts over time. For example, the temporal event group representing LeBron James playing for different teams during different periods can lead to event temporal questions like “What team did LeBron James play for in 2009?”. Additionally, Wikidata annotates time with three levels of temporal granularity: year, month, and day. We preserve this feature, so our dataset also involves multi-granularity temporal reasoning (Chen et al., 2023). We note that within a temporal event group, the temporal granularity of different events may be inconsistent. Therefore, we standardize the granularity within each group to a unified level. For example, if a temporal event group contains both year and month granularities, we standardize it to the year granularity.

3.2 Event Temporal Question Context Construction

Considering the potential data leakage issue with LLMs (Fatemi et al., 2024; Xiong et al., 2024), where prior knowledge of constructed questions may be implicitly contained within model training, it is possible for LLMs to provide direct answers without any reasoning. For example, when asked “What was the team that LeBron James played for the longest time?”, if the training corpus includes the information “LeBron James played the longest for the Cavaliers,” the LLM may deliver the answer directly. Therefore, we evaluate LLMs in an anonymized open-book question answering setting, in which the LLM is provided an anonymized context to answer the anonymized question. This shifts the evaluation focus to the LLM’s ability to understand the context and perform the reasoning process itself. Specifically, we anonymize the temporal event group by mapping and replacing the specific entities in the subject and object positions of all temporal events with anonymized entities denoted as \( E^* \), where * denotes a number. The anonymized temporal event group is textualized to serve as anonymized context. The questions are constructed based on anonymized temporal event groups, effectively avoiding data leakage issues. These contexts vary in length, time coverage, co-temporal types (Su et al., 2024), temporal event types, and relation composition (i.e., contexts involve two types of relations under implicit temporal constraints), thereby providing diversity in reasoning at the contextual level. Templates for textualizing the structured temporal events and the example of an anonymized context can be found in Appendix A.1 and Appendix B, respectively.

3.3 QA Pairs Construction

The automated construction of QA pairs is achieved through question template filling and rule-based answer acquisition. We have designed a total of 566 question templates for different relations. In designing these templates, we ensure comprehensive coverage of the taxonomy for event temporal questions. Details regarding the design of the question templates can be found in Appendix A.2. Each of these templates has a corresponding rule for obtaining answers, which can be executed on structured temporal event groups. For instance, given a temporal event group denoting the different positions E1 held during various periods, a compound question template like “What was the {ord} position {s} held {tc}?” could be filled. Here, the placeholder {ord} is for ordinal terms indicating a comparison of order, {s} is for the subject, and {tc} is for the temporal constraint. An instance might be “What was the first position E1 held after 2002?”. The filled temporal constraint (i.e., after 2002) needs to ensure that there are multiple events satisfying the constraint, and then the answer is derived by performing a comparison of order (i.e., determining the “first” event) among the candidate events. For the same context, a single question template can generate multiple questions with similar reasoning processes (e.g., by sampling different times within temporal constraints). To prevent the dataset from containing redundant questions involving similar reasoning over the same context, we limit each template to produce only one question per context.

3.4 Dataset Statistics

Following Chen et al. (2023), we divide the Wikidata dump into training, development, and test splits, ensuring there is no overlap of entities among them. For each split, we conduct temporal event preprocessing, construct context and QA pairs, and perform sampling. As a result, we obtain a dataset with an approximate ratio of 8:1:1 for training, development, and test sets, totaling 160k questions. Detailed statistics are shown in Table 2.

3.5 Quality Check

Following Fatemi et al. (2024), we conduct multiple rounds of quality checks on the dataset. Specif-
ically, considering the time cost, we sample 545 QA pairs and their corresponding contexts from the test set, covering various question templates without accounting for the different relations involved in these templates. We perform manual reviews, primarily focusing on: (1) answer accuracy; (2) potential format, grammatical, and semantic errors; (3) question ambiguity. This process repeats until no further issues are identified in the dataset.

<table>
  <tr>
    <th rowspan="2">Question Composition</th>
    <th colspan="1">Train</th>
    <th colspan="1">Dev</th>
    <th colspan="1">Test</th>
  </tr>
  <tr>
    <td>Explicit</td>
    <td>17,505</td>
    <td>2,308</td>
    <td>2,384</td>
  </tr>
  <tr>
    <td rowspan="4">Non-compound</td>
    <td>Implicit</td>
    <td>27,691</td>
    <td>3,820</td>
    <td>4,012</td>
  </tr>
  <tr>
    <td>Relative</td>
    <td>7,812</td>
    <td>1,024</td>
    <td>1,073</td>
  </tr>
  <tr>
    <td>Order</td>
    <td>4,916</td>
    <td>648</td>
    <td>658</td>
  </tr>
  <tr>
    <td>Duration</td>
    <td>2,400</td>
    <td>300</td>
    <td>300</td>
  </tr>
  <tr>
    <td rowspan="6">Compound</td>
    <td>Explicit + Order</td>
    <td>14,248</td>
    <td>1,858</td>
    <td>1,907</td>
  </tr>
  <tr>
    <td>Implicit + Order</td>
    <td>24,311</td>
    <td>3,304</td>
    <td>3,425</td>
  </tr>
  <tr>
    <td>Relative + Order</td>
    <td>4,916</td>
    <td>648</td>
    <td>658</td>
  </tr>
  <tr>
    <td>Explicit + Duration</td>
    <td>7,200</td>
    <td>900</td>
    <td>900</td>
  </tr>
  <tr>
    <td>Implicit + Duration</td>
    <td>12,820</td>
    <td>1,737</td>
    <td>1,759</td>
  </tr>
  <tr>
    <td>Relative + Duration</td>
    <td>2,400</td>
    <td>300</td>
    <td>300</td>
  </tr>
  <tr>
    <th rowspan="3">Answer Type</th>
    <th colspan="1">Entity</th>
    <td>34,161</td>
    <td>4,496</td>
    <td>4,621</td>
  </tr>
  <tr>
    <td>Time</td>
    <td>22,141</td>
    <td>3,047</td>
    <td>3,241</td>
  </tr>
  <tr>
    <td>Time interval</td>
    <td>23,772</td>
    <td>3,147</td>
    <td>3,186</td>
  </tr>
  <tr>
    <td>Time span</td>
    <td>19,657</td>
    <td>2,581</td>
    <td>2,593</td>
  </tr>
  <tr>
    <td>Numeric</td>
    <td>26,488</td>
    <td>3,576</td>
    <td>3,735</td>
  </tr>
  <tr>
    <th>Total</th>
    <td>126,219</td>
    <td>16,847</td>
    <td>17,376</td>
  </tr>
</table>

Table 2: Dataset statistics of ETRQA

4 Experiments

4.1 Experimental Setting

Evaluation Metrics. Accuracy is used as the evaluation metric. For multi-answer cases, it is measured using set accuracy (Zhong et al., 2023), which considers a prediction correct only if the predicted set matches the ground truth answer set exactly.

LLMs for Evaluation. We evaluate ETR abilities of two industry-leading LLM series: the Llama series (Grattafiori et al., 2024) and the Qwen series (Yang et al., 2025), which include models of varying sizes, as well as both base and instruction-tuned versions. Additionally, DeepSeek releases general reasoning-enhanced versions of the Llama and Qwen models. These models are fine-tuned using 800k samples curated with DeepSeek-R1 (Guo et al., 2025), achieving significant improvements in reasoning performance. Therefore, we select them to evaluate whether general reasoning distillation contributes to enhancing ETR abilities. We run these models without quantization on four A800-80G GPUs, with the temperature parameter set to 0 to ensure reproducibility.

Setting Details. We evaluate the LLMs under four common prompting strategies: zero-shot, few-shot (Brown et al., 2020), zero-shot CoT (Kojima et al., 2022), and few-shot CoT (Wei et al., 2022). In the zero-shot setting, LLMs are required to provide the answer directly based on the context. In the few-shot setting, additional examples are provided to help LLMs understand the task. These examples are sampled from the training set. In the zero-shot CoT setting, we add the phrase “Let’s think step by step.” as a CoT trigger after the question and prompt the LLM to provide the final answer using “The final answer is:”. In the few-shot CoT setting, CoT examples are provided to guide step-by-step reasoning. These CoT examples are automatically annotated using a rule-based program. We evaluate the general reasoning enhanced Llama and Qwen models only under the zero-shot CoT setting, as they are fine-tuned with long CoT data and, in practice, do not follow the reasoning patterns of the examples we provide in the few-shot CoT setting. All prompt templates can be found in Appendix C.

4.2 Overall Performance

We conduct the evaluation based on question composition. The experimental results of the base models, instruction-tuned models, and general reasoning enhanced models on the ETRQA test set are presented in Appendix D, Table 3 and Table 4, respectively.

Effect Analysis of Prompting Strategy. To verify the effect of different prompting strategies, we evaluate the overall accuracy of the instruction-tuned models and base models under different prompting strategies. From the results shown in Figure 1, we can see all models achieve their best performance under the few-shot CoT setting. The best-performing model is Llama-3.3-Instruct, with an overall accuracy of 78.7%, demonstrating that LLMs possess a certain level of ETR ability, though there is still room for improvement. In the zero-shot CoT setting, the base and small-scale models show a significant decline in performance compared to the instruction-tuned large-scale models, revealing the inherent limitations of these models in ETR. Furthermore, we observe consistent performance improvements for base and small-scale models when moving from zero-shot CoT to few-shot CoT, as explicit reasoning examples help com-
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="5">Non-Compound</th>
    <th colspan="5">Compound</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>E.</th><th>I.</th><th>R.</th><th>O.</th><th>Avg.</th>
    <th>E. + O.</th><th>I. + O.</th><th>R. + O.</th><th>E. + D.</th><th>I. + D.</th><th>R. + D.</th><th>Avg.</th>
  </tr>
  <tr>
    <th colspan="13">Llama-3.1-8B-Instruct</th>
  </tr>
  <tr>
    <td>ZS</td><td>36.6</td><td>27.4</td><td>24.9</td><td>46.8</td><td>15.3</td><td>30.8</td><td>35.4</td><td>26.0</td><td>26.3</td><td>16.0</td><td>14.0</td><td>12.0</td><td>24.2</td><td>27.4</td>
  </tr>
  <tr>
    <td>FS</td><td>39.4</td><td>33.7</td><td>31.3</td><td>55.8</td><td>17.3</td><td>36.1</td><td>34.8</td><td>31.1</td><td>28.0</td><td>19.3</td><td>19.6</td><td>11.3</td><td>27.5</td><td>31.7</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>27.1</td><td>17.6</td><td>16.1</td><td>52.4</td><td>32.0</td><td>23.3</td><td>30.2</td><td>21.8</td><td>23.3</td><td>21.6</td><td>17.5</td><td>16.7</td><td>22.7</td><td>23.0</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>58.6</td><td>56.0</td><td>50.2</td><td>66.0</td><td>35.0</td><td>56.0</td><td>52.5</td><td>48.5</td><td>41.6</td><td>34.3</td><td>31.4</td><td>15.3</td><td>43.0</td><td>49.3</td>
  </tr>
  <tr>
    <th colspan="13">Llama-3.1-70B-Instruct</th>
  </tr>
  <tr>
    <td>ZS</td><td>57.5</td><td>45.8</td><td>38.6</td><td>73.9</td><td>30.0</td><td>49.8</td><td>55.3</td><td>52.4</td><td>40.7</td><td>33.8</td><td>33.3</td><td>25.7</td><td>45.7</td><td>47.7</td>
  </tr>
  <tr>
    <td>FS</td><td>59.7</td><td>54.4</td><td>42.3</td><td>80.9</td><td>33.3</td><td>55.7</td><td>58.2</td><td>55.9</td><td>40.6</td><td>32.9</td><td>34.8</td><td>24.7</td><td>47.7</td><td>51.6</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>83.9</td><td>78.7</td><td>67.6</td><td>85.1</td><td>63.7</td><td>78.7</td><td>77.1</td><td>73.6</td><td>58.1</td><td>64.6</td><td>58.7</td><td>47.0</td><td>68.5</td><td>73.4</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>82.6</td><td>80.9</td><td>74.3</td><td>84.7</td><td>59.0</td><td>80.0</td><td>81.4</td><td>78.5</td><td>68.5</td><td>63.8</td><td>60.4</td><td>51.3</td><td>72.4</td><td>76.1</td>
  </tr>
  <tr>
    <th colspan="13">Llama-3.3-70B-Instruct</th>
  </tr>
  <tr>
    <td>ZS</td><td>58.6</td><td>47.9</td><td>36.6</td><td>75.8</td><td>30.3</td><td>51.1</td><td>55.2</td><td>52.6</td><td>40.6</td><td>31.1</td><td>28.2</td><td>19.0</td><td>44.2</td><td>47.5</td>
  </tr>
  <tr>
    <td>FS</td><td>56.0</td><td>52.2</td><td>37.9</td><td>81.9</td><td>29.3</td><td>52.9</td><td>57.3</td><td>56.4</td><td>42.9</td><td>29.6</td><td>33.0</td><td>22.3</td><td>47.2</td><td>50.0</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td><b>87.2</b></td><td><b>82.9</b></td><td><b>72.5</b></td><td><b>86.8</b></td><td><b>68.0</b></td><td><b>82.5</b></td><td><b>81.0</b></td><td><b>80.0</b></td><td><b>63.7</b></td><td><b>69.2</b></td><td><b>63.8</b></td><td><b>46.0</b></td><td><b>73.6</b></td><td><b>77.9</b></td>
  </tr>
  <tr>
    <td>FS CoT</td><td>82.9</td><td>82.3</td><td>75.6</td><td>88.9</td><td>67.0</td><td>81.6</td><td>84.2</td><td>82.0</td><td>73.6</td><td>67.9</td><td>63.8</td><td>58.3</td><td>76.1</td><td>78.7</td>
  </tr>
  <tr>
    <th colspan="13">Qwen2.5-7B-Instruct</th>
  </tr>
  <tr>
    <td>ZS</td><td>33.7</td><td>28.3</td><td>21.2</td><td>46.5</td><td>18.0</td><td>30.0</td><td>35.1</td><td>25.3</td><td>26.3</td><td>18.9</td><td>15.7</td><td>12.0</td><td>24.5</td><td>27.2</td>
  </tr>
  <tr>
    <td>FS</td><td>36.2</td><td>32.7</td><td>24.0</td><td>53.8</td><td>23.3</td><td>33.9</td><td>37.5</td><td>28.9</td><td>30.7</td><td>19.6</td><td>20.6</td><td>13.3</td><td>27.8</td><td>30.7</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>54.0</td><td>42.9</td><td>40.4</td><td>62.2</td><td>37.0</td><td>47.0</td><td>52.8</td><td>41.5</td><td>36.3</td><td>33.9</td><td>28.1</td><td>17.3</td><td>39.3</td><td>43.0</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>57.8</td><td>50.1</td><td>46.6</td><td>66.3</td><td>34.3</td><td>52.5</td><td>57.6</td><td>49.4</td><td>40.4</td><td>34.9</td><td>30.9</td><td>30.3</td><td>44.7</td><td>48.5</td>
  </tr>
  <tr>
    <th colspan="13">Qwen2.5-14B-Instruct</th>
  </tr>
  <tr>
    <td>ZS</td><td>46.4</td><td>37.6</td><td>31.3</td><td>64.1</td><td>23.3</td><td>40.9</td><td>41.1</td><td>36.2</td><td>34.5</td><td>25.8</td><td>21.9</td><td>19.0</td><td>32.7</td><td>36.6</td>
  </tr>
  <tr>
    <td>FS</td><td>47.7</td><td>41.5</td><td>36.6</td><td>63.5</td><td>15.7</td><td>43.4</td><td>39.2</td><td>34.2</td><td>31.6</td><td>15.9</td><td>16.3</td><td>11.7</td><td>29.0</td><td>36.0</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>59.0</td><td>55.0</td><td>53.5</td><td>75.4</td><td>53.3</td><td>57.5</td><td>63.0</td><td>59.0</td><td>49.8</td><td>50.1</td><td>45.2</td><td>36.0</td><td>54.8</td><td>56.1</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>58.8</td><td>58.1</td><td>49.8</td><td>76.1</td><td>53.7</td><td>58.5</td><td>68.1</td><td>64.3</td><td>52.1</td><td>52.8</td><td>45.3</td><td>35.3</td><td>58.3</td><td>58.4</td>
  </tr>
  <tr>
    <th colspan="13">Qwen2.5-32B-Instruct</th>
  </tr>
  <tr>
    <td>ZS</td><td>59.0</td><td>45.5</td><td>43.0</td><td>66.4</td><td>28.0</td><td>50.0</td><td>49.5</td><td>43.4</td><td>38.3</td><td>30.1</td><td>26.1</td><td>23.0</td><td>38.9</td><td>44.3</td>
  </tr>
  <tr>
    <td>FS</td><td>59.2</td><td>47.7</td><td>40.2</td><td>73.9</td><td>24.3</td><td>51.2</td><td>49.1</td><td>42.5</td><td>34.5</td><td>27.8</td><td>25.9</td><td>18.0</td><td>37.8</td><td>44.3</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>79.9</td><td>76.3</td><td>73.7</td><td>86.6</td><td>58.0</td><td>77.1</td><td>73.3</td><td>71.9</td><td>62.3</td><td>57.4</td><td>56.0</td><td>49.7</td><td>66.2</td><td>71.5</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>83.0</td><td>76.3</td><td>77.1</td><td>79.2</td><td>71.3</td><td>78.4</td><td>80.1</td><td>74.7</td><td>70.5</td><td>66.2</td><td>57.9</td><td>56.3</td><td>70.8</td><td>74.5</td>
  </tr>
  <tr>
    <th colspan="13">Qwen2.5-72B-Instruct</th>
  </tr>
  <tr>
    <td>ZS</td><td>56.0</td><td>47.3</td><td>40.2</td><td>73.1</td><td>23.0</td><td>50.0</td><td>49.8</td><td>44.7</td><td>39.4</td><td>28.4</td><td>24.6</td><td>18.3</td><td>38.9</td><td>44.3</td>
  </tr>
  <tr>
    <td>FS</td><td>58.6</td><td>51.0</td><td>42.7</td><td>81.3</td><td>36.3</td><td>53.9</td><td>55.7</td><td>48.0</td><td>41.0</td><td>32.0</td><td>28.0</td><td>23.0</td><td>42.8</td><td>48.2</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>84.1</td><td>81.7</td><td>76.2</td><td>88.1</td><td>66.0</td><td>81.6</td><td>74.6</td><td>73.5</td><td>67.3</td><td>59.9</td><td>58.7</td><td>50.7</td><td>68.3</td><td>74.7</td>
  </tr>
  <tr>
    <td>FS CoT</td><td><b>84.7</b></td><td><b>80.7</b></td><td><b>79.3</b></td><td><b>88.4</b></td><td><b>75.0</b></td><td><b>82.0</b></td><td><b>80.4</b></td><td><b>76.7</b></td><td><b>73.6</b></td><td><b>65.9</b></td><td><b>56.6</b></td><td><b>54.7</b></td><td><b>71.5</b></td><td><b>76.6</b></td>
  </tr>
</table>

Table 3: Experimental results of instruction-tuned models under four common prompting strategies. The abbreviations E., I., R., O., D. refer to Explicit, Implicit, Relative, Order, and Duration respectively. The results in bold (resp. underline) denote the best (resp. second) results.

pensate for their limited reasoning abilities. In contrast, instruction-tuned large-scale models already possess considerable reasoning abilities, so for simple, non-compound questions, few-shot CoT does not always outperform zero-shot CoT, since the effectiveness of few-shot CoT depends on factors such as the quality, complexity, order, and number of the provided examples. However, for more complex, compound questions, few-shot CoT still leads to better performance in most cases, indicating that these models still need to further learn the reasoning patterns required for solving compound questions.

Providing explicit step-by-step reasoning via CoT generally enhances model performance compared to direct input-output prompting strategies in zero-shot and few-shot settings. However, this influence can be inconsistent (Chu et al., 2024). For example, Llama-3.1-8B performs worse in the zero-shot CoT setting compared to the zero-shot and few-shot settings. We provide the error analysis in Appendix E.

Effect Analysis of Instruction Tuning. As shown in Figure 1, for each LLM, its instruction-tuned version generally outperforms the corresponding base model. Moreover, we observe that providing examples helps narrow the performance gap between the base and instruction-tuned versions of the same LLM. Specifically, the gap is smaller in the few-shot setting than in the zero-shot setting. A similar trend is observed in the CoT setting, where the performance gap is narrower under few-shot CoT compared to zero-shot CoT. These findings suggest that providing examples can partially mitigate
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="5">Non-Compound</th>
    <th colspan="6">Compound</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>E.</th><th>I.</th><th>R.</th><th>O.</th><th>D.</th>
    <th>Av.</th><th>E. + O.</th><th>I. + O.</th><th>R. + O.</th><th>E. + D.</th><th>I. + D.</th><th>R. + D.</th><th>Avg.</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B†</td>
    <td>76.1</td><td>63.8</td><td>55.5</td><td>73.9</td><td>29.3</td><td>65.8</td><td>61.6</td><td>55.5</td><td>51.5</td><td>32.0</td><td>31.8</td><td>21.7</td><td>48.4</td><td>56.8</td>
  </tr>
  <tr>
    <td>Llama-3.3-70B-Instruct†</td>
    <td><b>94.0</b></td><td><b>90.3</b></td><td><b>84.1</b></td><td><b>87.7</b></td><td><b>59.3</b></td><td><b>89.2</b></td><td><b>85.5</b></td><td><b>82.2</b></td><td><b>76.7</b></td><td><b>61.3</b></td><td><b>65.0</b></td><td><b>39.7</b></td><td><b>75.6</b></td><td><b>82.2</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B†</td>
    <td>65.6</td><td>53.0</td><td>49.4</td><td>70.4</td><td>26.0</td><td>56.5</td><td>57.2</td><td>49.1</td><td>38.8</td><td>28.0</td><td>26.7</td><td>15.7</td><td>42.4</td><td>49.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B†</td>
    <td>86.2</td><td>84.1</td><td>77.4</td><td>85.7</td><td>43.7</td><td>82.5</td><td>76.9</td><td>74.7</td><td>66.3</td><td>47.8</td><td>54.9</td><td>33.7</td><td>66.5</td><td>74.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B†</td>
    <td><b>92.8</b></td><td><b>89.6</b></td><td><b>86.8</b></td><td><b>85.7</b></td><td><b>51.7</b></td><td><b>88.5</b></td><td><b>78.7</b></td><td><b>77.7</b></td><td><b>71.9</b></td><td><b>55.2</b></td><td><b>59.4</b></td><td><b>40.0</b></td><td><b>70.4</b></td><td><b>79.2</b></td>
  </tr>
</table>

Table 4: Experimental results of general reasoning enhanced models in the zero-shot CoT setting. † denotes that the model has undergone general reasoning distillation. The abbreviations E., I., R., O., D. refer to Explicit, Implicit, Relative, Order, Duration. The results in bold (resp. underline) denote the best (resp. second) results.

the issue where base models, lacking instruction tuning, struggle with ETR.

Effect Analysis of Model Size. As shown in Figure 2, we evaluate how the scale of LLMs affects ETR performance using four model sizes (i.e., 7B, 14B, 32B, and 72B) of the Qwen instruction-tuned models under different prompting strategies. It can be observed that regardless of the prompting strategy used, performance improves as model size increases. However, compared to the performance improvement when scaling from 7B to 14B and from 14B to 32B, the improvement from 32B to 72B is much smaller, which demonstrates that further expansion will yield limited performance improvements of the model after its size increases to a certain extent. Therefore, it is necessary to explore other methods to further enhance the model’s ETR ability.

Effect Analysis of General Reasoning Distillation. As shown in Table 4, general reasoning distillation significantly enhances the models’ ETR abilities, especially for small-scale models. The best-performing model is the general reasoning enhanced Llama-3.3-70B-Instruct, with an overall accuracy of 82.2%, showing a 3.5% improvement over its underlying model, Llama-3.3-Instruct. However, these models tend to overthink (Chen et al., 2025) when dealing with questions involving duration comparisons, leading to poor performance on such questions. Specifically, even with the maximum output token limit set to 2,048, these models fail to provide an answer within a reasonable number of reasoning steps.

4.3 Challenges in Event Temporal Reasoning

Compound questions are more challenging for existing LLMs. When handling compound and non-compound questions, the models exhibit a noticeable performance gap. The best performance on non-compound questions is achieved by the general reasoning enhanced Llama-3.3-70B-Instruct, with an average accuracy of 89.2%. In contrast, the best performance on compound questions is achieved by Llama-3.3-70B-Instruct, with an average accuracy of 76.1%, marking a difference of 13.1%, which highlights the greater challenge posed by compound questions and indicates that LLMs’ multi-step ETR abilities still need to be enhanced.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Entity</th>
    <th colspan="2">Time</th>
    <th rowspan="2">Numeric</th>
  </tr>
  <tr>
    <th>Time point</th><th>Time interval</th><th>Time span</th>
  </tr>
  <tr>
    <td>Llama-3.3-70B-Instruct†</td>
    <td>88.6</td><td>93.4</td><td>78.7</td><td>47.1</td><td>91.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B†</td>
    <td><b>88.6</b></td><td>88.2</td><td>70.8</td><td>42.6</td><td>92.3</td>
  </tr>
  <tr>
    <td>Llama-3.3-70B-Instruct</td>
    <td>84.2</td><td>87.2</td><td>71.4</td><td>56.5</td><td>82.6</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>81.9</td><td>86.5</td><td>68.5</td><td>40.8</td><td>84.6</td>
  </tr>
</table>

Table 5: Performance of models on different answer types. † denotes that the model has undergone general reasoning distillation. The the best results are in bold.

LLMs struggle with reasoning involving time spans. We observe that all models demonstrate weaker performance when handling questions involving relative temporal constraints or duration comparisons, particularly when both are compounded. In such compounded cases, Llama-3.3-70B-Instruct model performs the best, yet it only achieves an accuracy of 58.3%. Both relative temporal constraints and duration comparisons are related to the understanding and computation of time spans. Additionally, as shown in Table 5, we analyze the models’ performance across different answer types. When questions require a precise time span as an answer, the models also show poor performance. Therefore, a major focus for improving ETR abilities in LLMs is enhancing their abilities to understand and accurately compute time spans.

LLMs’ performance declines when reasoning with fine temporal granularity. We analyze the models’ performance across different temporal granularities, as shown in Table 6. The models perform best at the year level, but their performance declines significantly at the month and day levels. The two instruction-tuned models exhibit
Figure 1: Overall performance of base models and instruction-tuned models on ETRQA under four common prompting strategies.

Figure 2: Overall performance of different model sizes in the Qwen series on ETRQA under four common prompting strategies.

<table>
  <tr>
    <th>Method</th>
    <th>Year</th>
    <th>Month</th>
    <th>Day</th>
  </tr>
  <tr>
    <td>Llama-3.3-70B-Instruct†</td>
    <td>89.8</td>
    <td>80.2</td>
    <td>76.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B†</td>
    <td>88.0</td>
    <td>77.4</td>
    <td>71.7</td>
  </tr>
  <tr>
    <td>Llama-3.3-70B-Instruct</td>
    <td>82.7</td>
    <td>75.1</td>
    <td>75.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>81.4</td>
    <td>71.1</td>
    <td>71.3</td>
  </tr>
</table>

Table 6: Performance of models on different temporal granularities. † denotes that the model has undergone general reasoning distillation. The results in bold denote the best results.

similar performance at the month and day levels, while the two general reasoning enhanced models show a clear trend of decline. Additionally, it is observed that the general reasoning enhanced Llama-3.3-70B-Instruct improves its accuracy by 7.1% at the year level, 5.2% at the month level, and only 0.6% at the day level compared to its underlying model, Llama-3.3-70B-Instruct. This is because general reasoning distillation often focuses on enhancing mathematical abilities, which aids in time calculations and comparisons. However, at the day level, additional temporal commonsense, such as the different number of days in each month and the occurrence of leap years, also plays a role. Therefore, it is still necessary to strengthen the models’ ETR abilities at finer temporal granularities.

5 Related Work

Early temporal NLP research primarily focuses on temporal information extraction tasks, leading to the development of early evaluation benchmarks such as TempEval (Uzzaman et al., 2014). These efforts are mainly centered around extracting events, times, and temporal relations from text. However, with advancements in language models, particularly large language models, attention gradually shifts to the more challenging task of temporal reasoning. This shift leads to the emergence of numerous temporal reasoning datasets. Among them, comprehensive benchmarks such as TRAM (Wang and Zhao, 2024) and TimeBench (Chu et al., 2024) represent a major leap forward in temporal reasoning evaluation. These benchmarks consolidate various existing temporal reasoning datasets to facilitate a thorough evaluation of temporal reasoning abilities. TimeBench categorizes temporal reasoning into three levels: symbolic (Thukral et al., 2021; Tan et al., 2023), commonsense (Zhou et al., 2019; Qin et al., 2021; Virgo et al., 2022; Zhang and Wan, 2023), and event temporal reasoning. Our work focuses on the third category, which involves not only understanding and applying the first two types of reasoning but also modeling and reasoning about the relationships between events and time, as well as between events.

Many existing datasets can be used to evaluate event temporal reasoning abilities. Some of these datasets are specifically designed to evaluate the temporal reasoning abilities of language models. TimeQA (Chen et al., 2021) is an early dataset
used to evaluate a model’s ability to understand and reason about time-evolving facts. TempReason (Tan et al., 2023) evaluates temporal reasoning across three levels: time-time, time-event, and event-event relations. Its follow-up work, Complex-TR (Tan et al., 2024), emphasizes the importance of multi-answer and multi-hop (i.e., relative time constraints) temporal reasoning. These three datasets share a common limitation: the question types are limited, and the complexity of reasoning primarily arises from external contexts (e.g., Wikipedia). This makes it difficult to pinpoint the specific challenges that models face in event temporal reasoning. TGQA (Xiong et al., 2024) addresses potential data leakage in evaluation by anonymizing data through replacing entities with fictional ones. However, ToT (Fatemi et al., 2024) suggests that this method may introduce incorrect claims and adopts a simpler, non-semantic anonymization approach, while extending the context to more complex graph structures. Despite this, both datasets define question types empirically, lacking a systematic taxonomy. ComplexTempQA (Gruber et al., 2024) divides temporal questions into three categories: attribute, comparison, and counting questions. However, some of the question types it defines involve mathematical reasoning, which makes it unsuitable for independently evaluating temporal reasoning. Additionally, several time question-answering datasets built for temporal knowledge graph question answering (TKGQA) (Jia et al., 2018; Saxena et al., 2021; Jia et al., 2021; Neelam et al., 2022; Chen et al., 2023; Zhang et al., 2024), table question answering (Gupta et al., 2023), or heterogeneous source question answering (Jia et al., 2024) can also be used for evaluation, but they share similar limitations as the aforementioned datasets.

To address the aforementioned limitations, we conduct an evaluation based on event temporal questions and establish a unified taxonomy. Furthermore, we introduce more challenging compound questions to further increase the complexity and diversity of reasoning. Based on this taxonomy, we propose the ETRQA benchmark to comprehensively evaluate event temporal reasoning abilities.

6 Potential impact

We believe that enhancing event temporal reasoning abilities is crucial for applying large language models (LLMs) to various real-world tasks related to temporal events, such as event timeline summarization (Song et al., 2025). Our work will have an impact on temporal reasoning in the following areas:

Dataset For the unified taxonomy of event temporal questions that we propose, we use it to guide the creation of more comprehensive question templates and the process of question generation. In practice, datasets can also be constructed through crowdsourcing or LLM synthesis, and the taxonomy we present will serve as a key guide for both of these approaches.

Model Training and Evaluation Our experimental results show that LLMs still do not perform well in solving certain types of reasoning tasks. Therefore, our dataset can be introduced during the model’s pre-training or post-training phases to enhance its learning of event temporal reasoning. Furthermore, our dataset can not only be used for a comprehensive evaluation of LLMs’ event temporal reasoning abilities but also to assess the impact of specific ability enhancements (e.g., mathematical reasoning) on the model’s performance in event temporal reasoning tasks.

Retrieval-Enhanced Generation Applying event temporal reasoning to real-world scenarios often requires integrating context retrieval. Our paper presents a novel approach for retrieval-enhanced event temporal reasoning research, specifically decoupling reasoning from retrieval. Our experimental results provide the upper bound for the event temporal reasoning abilities of some mainstream models, and when these models are combined with retrieval systems, the key objective will be to optimize the retrieval system to help the models achieve this upper bound.

7 Conclusion

To comprehensively evaluate the event temporal reasoning abilities of LLMs, we develop a unified taxonomy for event temporal questions and propose a comprehensive question-answering benchmark named ETRQA, by inheriting and expanding the evaluation content of existing datasets, as well as designing various types of compound questions. We conduct a thorough evaluation of two industry-leading LLM series on ETRQA. The experimental results indicate that the ETR abilities of LLMs still have room for improvement, particularly in reasoning involving time spans, reasoning for compound questions, and reasoning with fine temporal granularity.
Limitations

Our work still has some limitations as follows: (1) The questions are constructed using manual templates, which may lack sufficient linguistic diversity. Therefore, we plan to incorporate human paraphrasing or utilize paraphrases generated by LLMs in the future. (2) The context of the questions is relatively simplistic. In real-world scenarios, contexts feature more diverse expressions of event and time relationships. For example, “LeBron James won the NBA MVP in 2008, and he won again a year later.” In the future, we aim to generate more varied contexts based on rules or by using LLMs, which will further challenge the performance of LLMs. (3) Due to cost considerations, we have not yet evaluated closed-source models. However, considering the rapidly narrowing gap between open-source and closed-source models (Guo et al., 2025), evaluating industry-leading open-source models can reflect the current performance levels of LLMs.

Acknowledgments

The work is partially supported by the National Key Research and Development Program of China (2024YFF0617702), the National Natural Science Foundation of China (Nos.U22A2025, 62402097, 62232007, U23A20309), the Joint Funds of Natural Science Foundation of Liaoning Province (2023-BSBA-132), and 111 Project (No.B16009).

References

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6–12, 2020, virtual.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15(3).

Wenhui Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive questions. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.

Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhusheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. Do not think that much for 2+3=? on the overthinking of ol-like llms. Preprint, arXiv:2412.21187.

Ziyang Chen, Jinzhi Liao, and Xiang Zhao. 2023. Multigranularity temporal question answering over knowledge graphs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11378–11392, Toronto, Canada. Association for Computational Linguistics.

Ziyang Chen, Xiang Zhao, Jinzhi Liao, Xinyi Li, and Evangelos Kanoulas. 2022. Temporal knowledge graph question answering via subgraph reasoning. Know.-Based Syst., 251(C).

Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. TimeBench: A comprehensive evaluation of temporal reasoning abilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1204–1228, Bangkok, Thailand. Association for Computational Linguistics.

Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2024. Test of time: A benchmark for evaluating llms on temporal reasoning. Preprint, arXiv:2406.09170.

Aaron Grattafori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Raphael Gruber, Abdelrahman Abdallah, Michael Färber, and Adam Jatowt. 2024. Complextempqa: A large-scale dataset for complex temporal question answering. Preprint, arXiv:2406.04866.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Vivek Gupta, Pranshu Kandoi, Mahek Vora, Shuo Zhang, Yujie He, Ridho Reinanda, and Vivek Srikumar. 2023. TempTabQA: Temporal question answering for semi-structured tables. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2431–2453, Singapore. Association for Computational Linguistics.
Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.

Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, et al. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276.

Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jan-nik Strötgen, and Gerhard Weikum. 2018. Tem-pquestions: A benchmark for temporal question answering. In Companion Proceedings of The Web Conference 2018, WWW ’18, page 1057–1062, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.

Zhen Jia, Philipp Christmann, and Gerhard Weikum. 2024. Tiq: A benchmark for temporal question answering with implicit time constraints. In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, page 1394–1399, New York, NY, USA. Association for Computing Machinery.

Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex temporal question answering on knowledge graphs. CIKM ’21, page 792–802, New York, NY, USA. Association for Computing Machinery.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Sumit Neelam, Udit Sharma, Hima Karanam, Shajith Ikbai, Pavan Kapanipathi, Ibrahim Abdelaziz, Nandana Mihindukulasooriya, Young-Suk Lee, Santosh Srivastava, Cezar Pendus, Saswati Dana, Dinesh Garg, Achille Fokoue, G P Shrivatsa Bhargav, Dinesh Khandelwal, Srinivas Ravishankar, Sairam Rajajada, Maria Chang, Rosario Uceda-Sosa, Salim Roukos, Alexander Gray, Guilherme Lima, Ryan Riegel, Francois Luus, and L Venkata Subramaniam. 2022. A benchmark for generalizable and interpretable temporal question answering over knowledge bases. Preprint, arXiv:2201.05793.

Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2023. Reasoning with language model prompting: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5368–5393, Toronto, Canada. Association for Computational Linguistics.

Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Maanal Faruqui. 2021. TI-MEDIAL: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7066–7076, Online. Association for Computational Linguistics.

Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021. Question answering over temporal knowledge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6663–6676, Online. Association for Computational Linguistics.

Jiayu Song, Mahmud Akhter, Dana Atzil Slonim, and Maria Liakata. 2025. Temporal reasoning for timeline summarisation in social media. Preprint, arXiv:2501.00152.

Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, and Min Zhang. 2024. Living in the moment: Can large language models grasp co-temporal reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13014–13033, Bangkok, Thailand. Association for Computational Linguistics.

Qingyu Tan, Hewei Tou Ng, and Lidong Bing. 2023. Towards benchmarking and improving the temporal reasoning capability of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14820–14835, Toronto, Canada. Association for Computational Linguistics.

Qingyu Tan, Hewei Tou Ng, and Lidong Bing. 2024. Towards robust temporal reasoning of large language models via a multi-hop QA dataset and pseudo-instruction tuning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6272–6286, Bangkok, Thailand. Association for Computational Linguistics.

Shivin Thukral, Kunal Kukreja, and Christian Kavouras. 2021. Probing language models for understanding of temporal expressions. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 396–406, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Naushad Uzzaman, Hector Llorens, James Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. 2014. Tempeval-3: Evaluating events, time expressions, and temporal relations. Preprint, arXiv:1206.5333.

Felix Virgo, Fei Cheng, and Sadao Kurohashi. 2022. Improving event duration question answering by
leveraging existing temporal information extraction data. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4451–4457, Marseille, France. European Language Resources Association.

Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledgebase. Commun. ACM, 57(10):78–85.

Yuqing Wang and Yun Zhao. 2024. TRAM: Benchmarking temporal reasoning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6389–6415, Bangkok, Thailand. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichtler, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. Large language models can learn temporal reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10452–10470, Bangkok, Thailand. Association for Computational Linguistics.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, et al. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Tingyi Zhang, Jian Wang, Zhixu Li, Jianfeng Qu, An Liu, Zhigang Chen, and Hongping Zhi. 2024. MusTQ: A temporal knowledge graph question answering dataset for multi-step temporal reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11688–11699, Bangkok, Thailand. Association for Computational Linguistics.

Yunxiang Zhang and Xiaojun Wan. 2023. Situatedgen: Incorporating geographical and temporal contexts into generative commonsense reasoning. In Advances in Neural Information Processing Systems, volume 36, pages 67355–67373. Curran Associates, Inc.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingjian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2024. A survey of large language models. Preprint, arXiv:2303.18223.

Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettlemoyer. 2023. RoMQA: A benchmark for robust, multi-evidence, multi-answer question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7055–7067, Singapore. Association for Computational Linguistics.

Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363–3369, Hong Kong, China. Association for Computational Linguistics.
A Templates

A.1 Context Templates
In Table 7, we show example templates for textualizing the structured temporal event. In the templates, the placeholders \(\{s\}\) and \(\{o\}\) are used to fill in the subject and object, respectively. The placeholders \(\{t_s\}\) and \(\{t_e\}\) are used to fill in the start and end times of a temporal interval event, respectively, while \(\{t_p\}\) is used to fill in the event time of a temporal point event.

A.2 Question Templates
In designing question templates, besides ensuring comprehensive coverage of the taxonomy of event temporal questions, we also take the following into consideration:

• We create templates for both types of temporal event groups (i.e., temporal interval event groups and temporal point event groups) to cover the two common ways of annotating temporal events.

• To further enhance the diversity of questions, we consider the different focuses that questions might have on temporal event groups and design templates accordingly: (1) questions that focus on the entire temporal event group, such as “What was the position that E1 held for the longest time?”; (2) questions that focus on specific events within a temporal event group sharing the same object, such as “When did E1 receive the award of E2 for the first time?”; (3) for temporal interval event groups, questions can emphasize either the start or the end, thereby focusing reasoning solely on the start or end time of the events, such as “What distinct positions did E1 begin holding after 2002?”.

We design question templates for the representative relations in Wikidata shown in Table 8. In Table 10, we use the relation “position held” to present examples of question templates for questions based on temporal interval event groups. In Table 11, we use the relation "award received" to present examples of question templates for questions based on temporal point event groups. In the templates, the placeholders \(\{s\}\) and \(\{o\}\) are used to fill in the subject and object, respectively. The placeholder \(\{tc\}\) is used to fill in the time constraint, the placeholder \(\{ord\}\) is used to fill in ordinal terms, indicating a comparison of order, and the placeholder \(\{dur\}\) is used to fill in terms like “longest”, indicating a comparison of durations.

For each designed template, we perform grammatical and semantic checks as well as ambiguity resolution. The primary strategy for resolving ambiguity is to add additional clarifications within the question template.

B Dataset Sample
Each sample in the dataset includes an anonymized context and a QA pair constructed based on the anonymized temporal event groups. This anonymized context is generated by textualizing anonymized temporal event groups. A sample is shown in Figure 3.

C Prompts
The prompt templates for four common prompting strategies used in the experiments are provided. Figure 4 is the zero-shot prompt template, Figure 5 is the few-shot prompt template, Figure 6 is the zero-shot CoT prompt template, and Figure 7 is the few-shot CoT prompt template. In the prompts, we include instructions indicating the answer format, such as “The answer is a time interval, format the interval as (start_time, end_time),” to ensure that the LLMs provide answers that can be interpreted by the evaluation program.

D Base Models Experimental Results
The experimental results of the base model on the ETRQA test set are presented in Table 12.

E Error Analysis
To understand why the base and small-scale models in the Llama series exhibit inconsistent behavior under the zero-shot CoT setting, particularly a significant performance drop, while other models tend to show steady improvement, we conduct an error analysis. We examine the responses of Llama-3.1-8B, Llama-3.1-70B, Qwen2.5-7B, and Qwen2.5-72B, including both base and instruction-tuned versions. We categorize the errors into four types: (1) Incorrect Answer: The response contains an incorrect answer; (2) Incomplete Answer: In cases where multiple answers are expected, the model fails to return the full set of correct answers; (3) Uncertainty Error: The model does not provide an answer, either because it believes there is none or it is uncertain; (4) Instruction-Following Error:
The model fails to follow the instruction to include the phrase “The final answer is:”, making it difficult to extract the final answer.

As shown in Table 9, the base and small-scale models in the Llama series tend to exhibit more instruction-following errors under the zero-shot CoT setting, while the Qwen series exhibits fewer such errors. This discrepancy may be attributed to differences in prompt robustness across models. These observations suggest that the ability to reliably follow instructions plays an important role in supporting the reasoning performance of language models.
Context:
E1 began working for the employer of E4 on 01 September 2000.
E1 finished working for the employer of E4 on 31 August 2001.
E1 began working for the employer of E5 on 01 September 2001.
E1 finished working for the employer of E5 on 31 August 2005.
E1 worked for the employer of E3 from 01 September 2005 to 30 May 2009.
E1 began working for the employer of E3 on 01 June 2009.
E1 finished working for the employer of E3 on 28 September 2010.
E1 worked for the employer of E3 from 01 September 2011 to 24 March 2012.
E1 began working for the employer of E2 on 01 May 2013.
E1 finished working for the employer of E2 on 26 May 2016.
E1 worked for the employer of E2 from 01 January 2017 to 31 May 2019.
E1 worked for the employer of E6 from 01 November 2021 to 30 July 2022.

Question:
What was the employer that E1 worked for for the second shortest time? Considering all periods during which E1 worked for it.
Answer:
E4

Figure 3: A sample from ETRQA dataset.

<table>
  <tr>
    <th>Answer the question based on the context:</th>
  </tr>
  <tr>
    <td>{context}</td>
  </tr>
  <tr>
    <td>{answer format instruction}</td>
  </tr>
  <tr>
    <td><b>Please provide the answer directly, without explanation.</b></td>
  </tr>
  <tr>
    <td><b>Question:</b> {question}</td>
  </tr>
  <tr>
    <td><b>Answer:</b></td>
  </tr>
</table>

Figure 4: Zero-shot prompt template

<table>
  <tr>
    <th>Example:</th>
  </tr>
  <tr>
    <td><b>Answer the question based on the context:</b></td>
  </tr>
  <tr>
    <td>{context}</td>
  </tr>
  <tr>
    <td>{answer format instruction}</td>
  </tr>
  <tr>
    <td><b>Please provide the answer directly, without explanation.</b></td>
  </tr>
  <tr>
    <td><b>Question:</b> {question}</td>
  </tr>
  <tr>
    <td><b>Answer:</b> {answer}</td>
  </tr>
  <tr>
    <td>...</td>
  </tr>
  <tr>
    <td><b>Refer to the examples above.</b></td>
  </tr>
  <tr>
    <td><b>Answer the question based on the context:</b></td>
  </tr>
  <tr>
    <td>{context}</td>
  </tr>
  <tr>
    <td>{answer format instruction}</td>
  </tr>
  <tr>
    <td><b>Please provide the answer directly, without explanation.</b></td>
  </tr>
  <tr>
    <td><b>Question:</b> {question}</td>
  </tr>
  <tr>
    <td><b>Answer:</b></td>
  </tr>
</table>

Figure 5: Few-shot prompt template
Answer the question based on the context:
{context}
{answer format instruction}
Please think step by step, and at the end of your reasoning, use “The final answer is: ” followed by the answer in the correct format, then stop reasoning.
Question: {question}
Let’s think step by step.

Figure 6: Zero-shot CoT prompt template

Example:
Answer the question based on the context:
{context}
{answer format instruction}
Please think step by step, and at the end of your reasoning, use ’The final answer is:’ followed by the answer in the correct format, then stop reasoning.
Question: {question}
Let’s think step by step.
{CoT}
The final answer is: {answer}

...

Refer to the examples above.
Answer the question based on the context:
{context}
{answer format instruction}
Please think step by step, and at the end of your reasoning, use ’The final answer is:’ followed by the answer in the correct format, then stop reasoning.
Question: {question}
Let’s think step by step.

Figure 7: Few-shot CoT prompt template

<table>
  <tr>
    <th>Temporal Event Type</th>
    <th>Example Template</th>
    <th>Example</th>
  </tr>
  <tr>
    <td rowspan="3">Temporal interval event</td>
    <td>{s} began holding the position of {o} in/on {t_s}.</td>
    <td>E1 began holding the position of E2 in 2002.</td>
  </tr>
  <tr>
    <td>{s} finished holding the position of {o} in/on {t_e}.</td>
    <td>E1 finished holding the position of E2 in 2005.</td>
  </tr>
  <tr>
    <td>{s} held the position of {o} from {t_s} to {t_e}.</td>
    <td>E1 held the position of E2 from 2002 to 2005.</td>
  </tr>
  <tr>
    <td>Temporal point event</td>
    <td>{s} received the award of {o} in/on {t_p}.</td>
    <td>E1 received the award of E2 in 2002.</td>
  </tr>
</table>

Table 7: Example templates for textualizing the structured temporal event.
<table>
  <tr>
    <th>Temporal Event Type</th>
    <th>Wikidata Id</th>
    <th>Relation</th>
  </tr>
  <tr>
    <td rowspan="15">Temporal interval event</td>
    <td>P39</td>
    <td>position held</td>
  </tr>
  <tr><td>P108</td><td>employer</td></tr>
  <tr><td>P106</td><td>occupation</td></tr>
  <tr><td>P54</td><td>member of sports team</td></tr>
  <tr><td>P102</td><td>member of political party</td></tr>
  <tr><td>P463</td><td>member of</td></tr>
  <tr><td>P27</td><td>country of citizenship</td></tr>
  <tr><td>P937</td><td>work location</td></tr>
  <tr><td>P551</td><td>residence</td></tr>
  <tr><td>P69</td><td>educated at</td></tr>
  <tr><td>P6087</td><td>coach of sports team</td></tr>
  <tr><td>P1308</td><td>position holder</td></tr>
  <tr><td>P286</td><td>head coach</td></tr>
  <tr><td>P6</td><td>head of government</td></tr>
  <tr><td>P35</td><td>head of state</td></tr>
  <tr><td>P488</td><td>chairperson</td></tr>
  <tr>
    <td rowspan="6">Temporal point event</td>
    <td>P166</td>
    <td>award received</td>
  </tr>
  <tr><td>P1411</td><td>nominated for</td></tr>
  <tr><td>P512</td><td>academic degree</td></tr>
  <tr><td>P2522</td><td>competition won</td></tr>
  <tr><td>P793</td><td>significant event</td></tr>
  <tr><td>P410</td><td>military or police rank</td></tr>
</table>

Table 8: Representative relations for temporal interval events and temporal point events in Wikidata

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Error Type</th>
    <th rowspan="2">Total Errors</th>
  </tr>
  <tr>
    <th>Incorrect Answer</th>
    <th>Incomplete Answer</th>
    <th>Uncertainty Error</th>
    <th>Instruction-Following Error</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>66.29%</td>
    <td>2.66%</td>
    <td>0.02%</td>
    <td>31.03%</td>
    <td>14,306</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>37.76%</td>
    <td>3.03%</td>
    <td>0.08%</td>
    <td>59.13%</td>
    <td>13,383</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>71.89%</td>
    <td>7.34%</td>
    <td>1.41%</td>
    <td>19.36%</td>
    <td>9,577</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>78.91%</td>
    <td>8.91%</td>
    <td>0.54%</td>
    <td>11.64%</td>
    <td>4,614</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>83.67%</td>
    <td>6.80%</td>
    <td>0.45%</td>
    <td>9.08%</td>
    <td>11,710</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>89.84%</td>
    <td>8.91%</td>
    <td>0.35%</td>
    <td>0.90%</td>
    <td>9,898</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>85.65%</td>
    <td>6.42%</td>
    <td>0.46%</td>
    <td>7.47%</td>
    <td>7,994</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>91.11%</td>
    <td>8.59%</td>
    <td>0.18%</td>
    <td>0.11%</td>
    <td>4,388</td>
  </tr>
</table>

Table 9: Error analysis of different models under the zero-shot CoT setting. We report the total number of errors for each model and the proportion of each error type.
<table>
  <tr>
    <th>Question Composition</th>
    <th>Example Template</th>
    <th>Example Question</th>
    <th>Answer Type</th>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint</td>
    <td>What distinct positions did {s} hold {tc}?</td>
    <td>What distinct positions did E1 hold after 2002?</td>
    <td>Entity (multiple)</td>
  </tr>
  <tr>
    <td>How many distinct positions did {s} hold {tc}?</td>
    <td>How many distinct positions did E1 hold after 2002?</td>
    <td>Numeric</td>
  </tr>
  <tr>
    <td rowspan="3">Order</td>
    <td>What was the {ord} position {s} held?</td>
    <td>What was the first position E1 held?</td>
    <td>Entity</td>
  </tr>
  <tr>
    <td>During which period did {s} hold the position for the {ord} time?</td>
    <td>During which period did E1 hold the position for the first time?</td>
    <td>Time interval</td>
  </tr>
  <tr>
    <td>How long did {s} hold the position for the {ord} time?</td>
    <td>How long did E1 hold the position for the first time?</td>
    <td>Time span</td>
  </tr>
  <tr>
    <td rowspan="3">Duration</td>
    <td>What was the position that {s} held for the {dur} time? Considering all periods during which {s} held it.</td>
    <td>What was the position that E1 held for the longest time? Considering all periods during which E1 held it.</td>
    <td>Entity</td>
  </tr>
  <tr>
    <td>During which period did {s} hold the position for the {dur} time? List all periods during which {s} held it.</td>
    <td>During which period did E1 hold the position for the longest time? List all periods during which E1 held it.</td>
    <td>Time interval (multiple)</td>
  </tr>
  <tr>
    <td>What was the total duration {s} held the position for the {dur} time? Summing all periods during which {s} held it.</td>
    <td>What was the total duration E1 held the position for the longest time? Summing all periods during which E1 held it.</td>
    <td>Time span</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint + Order</td>
    <td>What was the {ord} position {s} held {tc}?</td>
    <td>What was the first position E1 held after 2002?</td>
    <td>Entity</td>
  </tr>
  <tr>
    <td>During which period did {s} hold the position for the {ord} time {tc}? only counting the portions {tc}.</td>
    <td>During which period did E1 hold the position for the first time after 2002? Only counting the portions after 2002.</td>
    <td>Time interval</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint + Order</td>
    <td>How long did {s} hold the position for the {ord} time {tc}? only counting the portions {tc}.</td>
    <td>How long did E1 hold the position for the first time after 2002? Only counting the portions after 2002.</td>
    <td>Time span</td>
  </tr>
  <tr>
    <td>What was the position that {s} held for the {dur} time {tc}? Considering all periods during which {s} held it but only counting the portions {tc}.</td>
    <td>What was the position that E1 held for the longest time after 2002? Considering all periods during which E1 held it but only counting the portions after 2002.</td>
    <td>Entity</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint + Duration</td>
    <td>During which period did {s} hold the position for the {dur} time {tc}? List all periods during which {s} held it but only counting the portions {tc}.</td>
    <td>During which period did E1 hold the position for the longest time after 2002? Considering all periods during which E1 held it but only counting the portions after 2002.</td>
    <td>Time interval (multiple)</td>
  </tr>
  <tr>
    <td>What was the total duration {s} held the position for the {dur} time {tc}? Summing all periods during which {s} held it but only counting the portions {tc}.</td>
    <td>What was the total duration E1 held the position for the longest time after 2002? Summing all periods during which E1 held it but only counting the portions after 2002.</td>
    <td>Time span</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint + Duration</td>
    <td>What distinct positions did {s} begin holding {tc}?</td>
    <td>What distinct positions did E1 begin holding after 2002?</td>
    <td>Entity (multiple)</td>
  </tr>
  <tr>
    <td>How many distinct positions did {s} begin holding {tc}?</td>
    <td>How many distinct positions did E1 begin holding after 2002?</td>
    <td>Numeric</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint + Order</td>
    <td>What was the {ord} position {s} began holding {tc}?</td>
    <td>What was the first position E1 began holding after 2002?</td>
    <td>Entity</td>
  </tr>
  <tr>
    <td>When did {s} begin holding the position for the {ord} time {tc}?</td>
    <td>When did E1 begin holding the position for the first time after 2002?</td>
    <td>Time point</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint + Order</td>
    <td>During which period did {s} hold the position of {o} {tc}? Considering all periods during which {s} held it but only counting the portions {tc}.</td>
    <td>During which period did E1 hold the position of E2 after 2002? Considering all periods during which E1 held it but only counting the portions after 2002.</td>
    <td>Time interval (multiple)</td>
  </tr>
  <tr>
    <td>How many times did {s} hold the position of {o} {tc}?</td>
    <td>How many times did E1 hold the position of E2 after 2002?</td>
    <td>Numeric</td>
  </tr>
  <tr>
    <td rowspan="2">Order</td>
    <td>During which period did {s} hold the position of {o} for the {ord} time {tc}?</td>
    <td>During which period did E1 hold the position of E2 for the first time?</td>
    <td>Time interval</td>
  </tr>
  <tr>
    <td>How long did {s} hold the position of {o} for the {ord} time?</td>
    <td>How long did E1 hold the position of E2 for the first time?</td>
    <td>Time span</td>
  </tr>
  <tr>
    <td rowspan="2">Duration</td>
    <td>During which period did {s} hold the position of {o} for the {dur} time {tc}?</td>
    <td>During which period did E1 hold the position of E2 for the longest time?</td>
    <td>Time interval</td>
  </tr>
  <tr>
    <td>How long did {s} hold the position of {o} for the {dur} time?</td>
    <td>How long did E1 hold the position of E2 for the longest time?</td>
    <td>Time span</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint + Order</td>
    <td>During which period did {s} hold the position of {o} for the {ord} time {tc}? only counting the portions {tc}.</td>
    <td>During which period did E1 hold the position of E2 for the first time after 2002? only counting the portions after 2002.</td>
    <td>Time interval</td>
  </tr>
  <tr>
    <td>How long did {s} hold the position of {o} for the {ord} time {tc}? only counting the portions {tc}.</td>
    <td>How long did E1 hold the position of E2 for the first time after 2002? only counting the portions after 2002.</td>
    <td>Time span</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint + Duration</td>
    <td>During which period did {s} hold the position of {o} for the {dur} time {tc}? only counting the portions {tc}.</td>
    <td>During which period did E1 hold the position of E2 for the longest time after 2002? only counting the portions after 2002.</td>
    <td>Time interval</td>
  </tr>
  <tr>
    <td>How long did {s} hold the position of {o} for the {dur} time {tc}? only counting the portions {tc}.</td>
    <td>How long did E1 hold the position of E2 for the longest time after 2002? only counting the portions after 2002.</td>
    <td>Time span</td>
  </tr>
  <tr>
    <td rowspan="2">Temporal constraint + Order</td>
    <td>When did {s} begin holding the position of {o} {tc}?</td>
    <td>When did E1 begin c E2 after 2002? List all if began multiple times.</td>
    <td>Time point (multiple)</td>
  </tr>
  <tr>
    <td>How many times did {s} begin holding the position of {o} {tc}?</td>
    <td>How many times did E1 begin holding the position of E2 after 2002?</td>
    <td>Numeric</td>
  </tr>
  <tr>
    <td rowspan="2">Order</td>
    <td>When did {s} begin holding the position of {o} for the {ord} time?</td>
    <td>When did E1 begin holding the position of E2 for the first time?</td>
    <td>Time point</td>
  </tr>
  <tr>
    <td>When did {s} begin holding the position of {o} for the {ord} time {tc}?</td>
    <td>When did E1 begin holding the position of E2 for the first time after 2002?</td>
    <td>Time point</td>
  </tr>
</table>

Table 10: Example question templates for questions based on temporal interval event groups.
<table>
  <tr>
    <th>Question Composition</th>
    <th>Example Template</th>
    <th>Example Question</th>
    <th>Answer Type</th>
  </tr>
  <tr>
    <td>Temporal constraint</td>
    <td>What awards did {s} receive {tc}? Don’t need deduplicate.</td>
    <td>What awards did E1 receive after 2002? Don’t need deduplicate.</td>
    <td>Entity (multiple)</td>
  </tr>
  <tr>
    <td>Temporal constraint</td>
    <td>How many awards did {s} receive {tc}? Don’t need deduplicate.</td>
    <td>How many awards did E1 receive after 2002? Don’t need deduplicate.</td>
    <td>Numeric</td>
  </tr>
  <tr>
    <td>Temporal constraint + Order</td>
    <td>What was the {ord} award {s} received {tc}?</td>
    <td>What was the first award E1 received after 2002?</td>
    <td>Entity</td>
  </tr>
  <tr>
    <td>Order</td>
    <td>When did {s} receive the award for the {ord} time?</td>
    <td>When did E1 receive the award for the first time?</td>
    <td>Time point</td>
  </tr>
  <tr>
    <td>Temporal constraint + Order</td>
    <td>When did {s} receive the award for the {ord} time {tc}?</td>
    <td>When did E1 receive the award for the first time after 2002?</td>
    <td>Time point</td>
  </tr>
  <tr>
    <td>Temporal constraint</td>
    <td>When did {s} receive the award of {o} {tc}?</td>
    <td>When did E1 receive the award of E2 after 2002?</td>
    <td>Time point (multiple)</td>
  </tr>
  <tr>
    <td></td>
    <td>List all received multiple times.</td>
    <td>List all received multiple times.</td>
    <td></td>
  </tr>
  <tr>
    <td>Temporal constraint</td>
    <td>How many times did {s} receive the award of {o} {tc}?</td>
    <td>How many times did E1 receive the award of E2 after 2002?</td>
    <td>Numeric</td>
  </tr>
  <tr>
    <td>Order</td>
    <td>When did {s} receive the award of {o} for the {ord} time?</td>
    <td>When did E1 receive the award of E2 for the first time?</td>
    <td>Time point</td>
  </tr>
  <tr>
    <td>Temporal constraint + Order</td>
    <td>When did {s} receive the award of {o} for the {ord} time {tc}?</td>
    <td>When did E1 receive the award of E2 for the first time after 2002?</td>
    <td>Time point</td>
  </tr>
</table>

Table 11: Example question templates for questions based on temporal point event groups.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="5">Non-Compound</th>
    <th colspan="6">Compound</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>E.</th><th>I.</th><th>R.</th><th>O.</th><th>D.</th>
    <th>Avg.</th>
    <th>E. + O.</th><th>I. + O.</th><th>R. + O.</th><th>E. + D.</th><th>I. + D.</th><th>R. + D.</th><th>Avg.</th>
  </tr>
  <tr>
    <th colspan="14">Llama-3.1-8B</th>
  </tr>
  <tr>
    <td>ZS</td><td>21.6</td><td>17.8</td><td>14.6</td><td>36.3</td><td>11.3</td><td>19.7</td><td>29.6</td><td>23.0</td><td>22.8</td><td>17.8</td><td>15.4</td><td>11.3</td><td>22.0</td><td>20.9</td>
  </tr>
  <tr>
    <td>FS</td><td>38.7</td><td>32.9</td><td>33.3</td><td>52.3</td><td>17.7</td><td>35.6</td><td>38.5</td><td>33.3</td><td>31.2</td><td>21.1</td><td>20.5</td><td>12.3</td><td>29.8</td><td>32.6</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>18.6</td><td>12.7</td><td>12.8</td><td>33.0</td><td>17.0</td><td>16.1</td><td>25.3</td><td>20.0</td><td>16.7</td><td>15.2</td><td>14.7</td><td>12.7</td><td>19.1</td><td>17.7</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>48.1</td><td>46.8</td><td>46.0</td><td>65.7</td><td>30.7</td><td>48.0</td><td>45.4</td><td>44.1</td><td>39.7</td><td>26.8</td><td>26.8</td><td>18.0</td><td>38.0</td><td>42.8</td>
  </tr>
  <tr>
    <th colspan="14">Llama-3.1-70B</th>
  </tr>
  <tr>
    <td>ZS</td><td>51.8</td><td>41.6</td><td>36.7</td><td>61.4</td><td>19.7</td><td>44.7</td><td>43.7</td><td>38.5</td><td>32.1</td><td>24.3</td><td>23.6</td><td>15.0</td><td>34.0</td><td>39.2</td>
  </tr>
  <tr>
    <td>FS</td><td>53.3</td><td>48.0</td><td>36.3</td><td>78.7</td><td>35.0</td><td>49.9</td><td>54.2</td><td>51.7</td><td>45.1</td><td>31.3</td><td>28.0</td><td>17.3</td><td>43.9</td><td>46.8</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>46.5</td><td>48.4</td><td>42.7</td><td>70.4</td><td>30.7</td><td>48.2</td><td>51.5</td><td>48.8</td><td>35.4</td><td>28.2</td><td>30.3</td><td>19.7</td><td>41.7</td><td>44.9</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>78.6</td><td>73.7</td><td>70.7</td><td>80.2</td><td>60.3</td><td>74.7</td><td>76.6</td><td>72.0</td><td>65.0</td><td>53.2</td><td>52.0</td><td>43.3</td><td>65.7</td><td>70.1</td>
  </tr>
  <tr>
    <th colspan="14">Qwen2.5-7B</th>
  </tr>
  <tr>
    <td>ZS</td><td>33.4</td><td>27.0</td><td>25.3</td><td>44.4</td><td>13.3</td><td>29.5</td><td>32.9</td><td>23.5</td><td>24.6</td><td>16.8</td><td>16.1</td><td>11.7</td><td>23.0</td><td>26.2</td>
  </tr>
  <tr>
    <td>FS</td><td>39.1</td><td>33.0</td><td>29.3</td><td>50.8</td><td>22.7</td><td>35.3</td><td>37.9</td><td>31.0</td><td>29.2</td><td>20.0</td><td>21.0</td><td>13.0</td><td>28.6</td><td>31.9</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>38.0</td><td>29.0</td><td>33.0</td><td>52.0</td><td>28.0</td><td>33.8</td><td>40.8</td><td>34.0</td><td>26.9</td><td>25.9</td><td>23.5</td><td>16.7</td><td>31.5</td><td>32.6</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>52.9</td><td>43.1</td><td>43.3</td><td>59.0</td><td>29.0</td><td>46.7</td><td>50.9</td><td>42.8</td><td>36.2</td><td>30.1</td><td>26.6</td><td>24.0</td><td>39.0</td><td>42.7</td>
  </tr>
  <tr>
    <th colspan="14">Qwen2.5-14B</th>
  </tr>
  <tr>
    <td>ZS</td><td>42.7</td><td>32.7</td><td>30.1</td><td>51.7</td><td>18.3</td><td>36.2</td><td>39.3</td><td>31.8</td><td>27.7</td><td>24.0</td><td>21.3</td><td>16.3</td><td>29.7</td><td>32.8</td>
  </tr>
  <tr>
    <td>FS</td><td>46.8</td><td>37.9</td><td>34.7</td><td>61.2</td><td>19.3</td><td>41.2</td><td>42.7</td><td>36.9</td><td>30.9</td><td>25.1</td><td>22.8</td><td>16.3</td><td>33.0</td><td>37.0</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>54.7</td><td>42.1</td><td>42.9</td><td>72.3</td><td>44.0</td><td>48.2</td><td>56.1</td><td>46.4</td><td>35.4</td><td>37.2</td><td>34.5</td><td>28.3</td><td>43.8</td><td>45.9</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>64.5</td><td>58.6</td><td>55.3</td><td>66.9</td><td>43.7</td><td>60.0</td><td>64.1</td><td>57.6</td><td>48.3</td><td>42.8</td><td>38.7</td><td>29.3</td><td>52.2</td><td>55.9</td>
  </tr>
  <tr>
    <th colspan="14">Qwen2.5-32B</th>
  </tr>
  <tr>
    <td>ZS</td><td>54.6</td><td>38.8</td><td>38.6</td><td>63.2</td><td>23.3</td><td>44.6</td><td>47.4</td><td>41.7</td><td>38.1</td><td>30.8</td><td>25.5</td><td>20.7</td><td>37.7</td><td>41.0</td>
  </tr>
  <tr>
    <td>FS</td><td>57.3</td><td>44.8</td><td>40.6</td><td>70.4</td><td>21.3</td><td>49.0</td><td>48.0</td><td>41.6</td><td>35.3</td><td>28.1</td><td>24.4</td><td>17.3</td><td>36.9</td><td>42.8</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>73.4</td><td>63.6</td><td>61.5</td><td>86.2</td><td>50.7</td><td>67.4</td><td>69.4</td><td>63.6</td><td>52.3</td><td>49.9</td><td>50.0</td><td>41.7</td><td>59.2</td><td>63.2</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>76.0</td><td>72.3</td><td>71.8</td><td>77.5</td><td>57.0</td><td>73.1</td><td>77.2</td><td>71.0</td><td>64.6</td><td>58.6</td><td>52.1</td><td>44.3</td><td>66.0</td><td>69.5</td>
  </tr>
  <tr>
    <th colspan="14">Qwen2.5-72B</th>
  </tr>
  <tr>
    <td>ZS</td><td>57.7</td><td>46.0</td><td>43.2</td><td>67.0</td><td>22.0</td><td>49.8</td><td>48.9</td><td>42.5</td><td>35.4</td><td>27.4</td><td>25.1</td><td>19.0</td><td>37.6</td><td>43.5</td>
  </tr>
  <tr>
    <td>FS</td><td>61.2</td><td>48.5</td><td>46.7</td><td>74.2</td><td>25.0</td><td>53.0</td><td>51.9</td><td>46.6</td><td>37.8</td><td>30.3</td><td>27.2</td><td>19.7</td><td>40.7</td><td>46.7</td>
  </tr>
  <tr>
    <td>ZS CoT</td><td>64.4</td><td>55.1</td><td>43.2</td><td>76.7</td><td>45.3</td><td>57.6</td><td>56.3</td><td>57.4</td><td>41.3</td><td>40.7</td><td>43.6</td><td>29.7</td><td>50.6</td><td>54.0</td>
  </tr>
  <tr>
    <td>FS CoT</td><td>79.1</td><td>75.1</td><td>73.0</td><td>81.3</td><td>63.0</td><td>76.0</td><td>78.8</td><td>73.1</td><td>67.8</td><td>58.3</td><td>54.2</td><td>46.3</td><td>67.8</td><td>71.8</td>
  </tr>
</table>

Table 12: Experimental results of base models under four common prompting strategies. The abbreviations E., I., R., O., D. refer to Explicit, Implicit, Relative, Order, and Duration respectively. The results in bold (resp. underline) denote the best (resp. second) results.

Reviewer paper summaries: This paper introduces ETRQA, a comprehensive benchmark for evaluating the event temporal reasoning (ETR) abilities of large language models (LLMs). The dataset systematically addresses limitations in prior temporal QA datasets by proposing a unified taxonomy covering question composition (explicit/implicit/relative temporal constraints, order/duration comparisons, and compound questions) and answer types (entity, time, numeric). Constructed from Wikidata, ETRQA includes 160k anonymized questions derived from 22 temporal relations (16 interval, 6 point events), with multi-granularity time reasoning (year/month/day). Experiments on Llama and Qwen series reveal that while LLMs exhibit basic ETR capabilities, they struggle with compound questions (e.g., combining constraints and comparisons), fine-grained temporal reasoning (e.g., month/day calculations), and duration-based tasks. Key findings highlight the need for improved multi-step reasoning, temporal commonsense integration, and specialized training strategies. The work provides a standardized framework for advancing temporal reasoning research in LLMs.

The paper introduces a comprehensive temporal reasoning dataset. The dataset was created through taxonomy analysis of previous papers. Two LLM models, Llama and Qwen, were used for evaluation. The authors provide a detailed analysis of LLMs' performance on their dataset.
Summary of strengths: ### 1. Unified Taxonomy for Temporal Reasoning

The paper introduces a systematic taxonomy for event temporal questions, categorizing them by question composition (explicit/implicit/relative constraints, order/duration comparisons, and compound questions) and answer types (entity, time, numeric). This framework addresses the lack of standardized evaluation in prior datasets.

### 2. Comprehensive Benchmark Design

  ETRQA expands on existing datasets by including compound questions (e.g., combining temporal constraints with duration/order comparisons), which require multi-step reasoning. This design better evaluates the integrated temporal reasoning abilities of LLMs, a gap in previous work.

### 3. Multi-Granularity Temporal Reasoning

  The dataset preserves year/month/day-level granularity in time annotations, enabling fine-grained temporal reasoning (e.g., handling leap years or varying month lengths). This adds complexity and realism to the evaluation.

### 4. Anonymization for Fair Evaluation

  By replacing entities with placeholders (e.g., E1, E2), the dataset ensures models cannot rely on pre-trained knowledge, focusing evaluation on context-based reasoning rather than memorization.

- The authors provide a structured taxonomy which allows easier comparison of previously published temporal reasoning datasets.
- The constructive dataset includes various temporal question types based on the taxonomy of temporal questions.
- The authors provide thorough analysis of LLMs' performance.
Summary of weaknesses: ### 1. Homogenization in the dataset's reasoning types.

Since the authors clearly defined the type of event temporal reasoning and used templates to standardize and determine each task format,and also performed anonymization,the dataset may face a homogenization problem. For example, given their contexts, Question 1"Which company did E1 work for in 2021?"and Question 2"Which company did E2 work for in 1999?" test the exact same capability of the model. Moreover, due to the limited number of task types and event relationship types, the dataset's homogenization may be quite severe. This issue will further lead to the test set not being sufficiently out-of-distribution(OOD) compared to the training set.

### 2. The authors lack further exploration of the current LLM's deficiencies in event temporal reasoning.

The authors evaluated the performance of current LLMs based on ETRQA in overall and specific dimensions, but these performances did not delve into the causes of the problems, such as certain aspects of the LLM's training or the data that might be the reasons for these issues, which is an important significance of a benchmark paper (the authors did mention one point that the base model is inferior to the instruct model, but this conclusion is universal and already known).

### 3. The authors did not propose methods to enhance the model's ETR capability, nor did they reveal potential directions for improvement.

As a benchmark paper, the authors need to propose methods or explain the directions for improving this task.

### 4. The relationship with MultihopQA.

The authors proposed an interesting compound ETR setting, but they need to explore its relationship with MultihopQA. If the additional capability requirements of compound ETR compared to ETR are the same as those of MultihopQA compared to SinglehopQA (I mean the incremental part), it would weaken the contribution of this paper,as the latter has already been extensively studied.

- The authors reported performance of two open sourced models, Llama and Qwen. Including other models may better justify the dataset usability. It would be great to include the evaluation results with DeepSeek-R1 or other recent commercial models such as GPT 4.5.
- The dataset includes synthetic QA pairs only. These were generated through question template filling and rule-based answer acquisition.
Comments, suggestions and typos: 1. Is the QA construction process automatic or manual? I don't find the details of this part in paper. If it's automatic, how to ensure the quality of the answer.

The DeepSeek-R1 results will be particularly insightful, given its reputation for excelling in complex reasoning. We're eager to see how it performs in this specific evaluation.
Overall assessment scores: 2.5

2.5
Metareview score: 3
Metareview: The paper introduces ETRQA, a new benchmark designed to evaluate Event Temporal Reasoning (ETR) abilities in large language models. By addressing gaps in existing datasets, the authors propose a unified taxonomy that covers various aspects of temporal reasoning—from explicit and implicit constraints to relative comparisons and compound questions. ETRQA is constructed using data extracted from Wikidata and generated via a template-based approach with entity anonymization, thereby emphasizing reasoning over memorization.

Experimental evaluations on models from the Llama and Qwen series reveal that—even after instruction tuning and general reasoning distillation—current LLMs still struggle with compound questions, fine-grained temporal granularity, and certain relative temporal reasoning tasks. While the benchmark demonstrates a comprehensive assessment and provides useful insights into the ETR capabilities of LLMs, several issues remain to be clarified. For instance, the paper raises questions about why, in some scenarios, the FS CoT (few-shot chain-of-thought) approach for Llama-3.3-70B-Instruct underperforms compared to the zero-shot variant, and why Llama-3.3-70B-Instruct appears to lag behind Qwen2.5-32B on relative temporal questions. Moreover, inconsistencies in performance among Llama-3.1-8B series models observed in Figure 1 need further explanation.

Overall, the work makes a timely contribution by proposing a structured and systematic evaluation framework for event temporal reasoning. However, further analysis and explanation of experimental discrepancies, as well as additional details on the automated QA template design and quality control procedures, would strengthen the paper’s findings.
Summary of reasons to publish: - Proposes a unified taxonomy and systematic evaluation framework for event temporal reasoning.

- Addresses important gaps by incorporating compound and multi-granularity temporal questions.

- Provides extensive empirical evaluation across different model families and settings.