ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events
Authors: Duygu Sezen Islakoglu, Jan-Christoph Kalo
Accepted at: acl25

ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events

Duygu Sezen Islakoglu*
Utrecht University, Netherlands
d.s.islakoglu@uu.nl

Jan-Christoph Kalo*
University of Amsterdam, Netherlands
j.c.kalo@uva.nl

Abstract
Large Language Models (LLMs) still face significant challenges in reasoning and arithmetic. Although temporal reasoning has raised increasing research attention, comprehensive testing of Allen’s interval relations (e.g., before, after, during) –a fundamental framework for temporal relationships– remains underexplored. To fill this gap, we present ChronoSense, a new benchmark for evaluating LLMs’ temporal understanding. It includes 16 tasks, identifying the Allen relation between two temporal events and temporal arithmetic. We assess the performance of seven recent LLMs. The results indicate that models handle Allen relations, even symmetrical ones, quite differently. Moreover, the findings suggest that the models may rely on memorization to answer time-related questions. Overall, the models’ low performance highlights the need for improved temporal understanding in LLMs. Our dataset and the source code are available at https://github.com/duyguislakoglu/chronosense.

1 Introduction
Large Language Models (LLMs) have demonstrated remarkable proficiency across various tasks in NLP. Despite these advancements, significant challenges persist in areas such as reasoning, arithmetic (BIG-bench authors, 2023), and working with numerical values (Wei et al., 2022). These limitations affect their performance in temporal reasoning and numerical arithmetic.

Recent research has shown a growing interest in evaluating the temporal reasoning capabilities of LLMs. Efforts have focused on event ordering, comparing temporal events, temporal question answering, and event forecasting (Chu et al., 2023). However, a notable gap remains: the comprehensive testing of Allen’s intervals, one of the most fundamental temporal reasoning frameworks that have been in use for over 30 years (Allen, 1989).

<table>
  <tr>
    <th></th>
    <th>Equals(A,B), equals(B,A)</th>
  </tr>
  <tr>
    <td>before(A,B), after(B,A)</td>
    <td></td>
  </tr>
  <tr>
    <td>overlaps(A,B), overlapped-by(B,A)</td>
    <td></td>
  </tr>
  <tr>
    <td>contains(A,B), during(B,A)</td>
    <td></td>
  </tr>
  <tr>
    <td>started-by(A,B), starts(B,A)</td>
    <td></td>
  </tr>
  <tr>
    <td>finished-by(A,B), finishes(B,A)</td>
    <td></td>
  </tr>
  <tr>
    <td>meets(A,B), met-by(B,A)</td>
    <td></td>
  </tr>
</table>

Figure 1: 13 Allen relations between two intervals, covering all combinations.

Allen’s intervals provide a formal structure for representing temporal relationships between events, defining thirteen possible relations between time intervals. Despite its importance, existing benchmarks cover only subsets of these relations. We demonstrate these 13 relations in Figure 1.

To illustrate our task, consider the following example: In Figure 2, the first event is the fourth cholera pandemic which occurred between 1863 and 1875, while World War II occurred between 1939 and 1945. In our prompt, we list these two events with their names and respective start and end years and then ask a True/False question about one of the 13 Allen relations. For example, we ask the LLM whether the fourth cholera pandemic happened “before” World War II.

While such tasks are straightforward for humans, they pose considerable difficulty for LLMs due to the need to compare numerical values accurately. Our research focuses on reasoning about time intervals and assessing how models perform on temporal understanding tasks. We also incorporate three time arithmetic tasks to challenge the models further.

Our contributions can be summarized as follows:
• We present a comprehensive evaluation of LLMs’ performance on temporal reasoning tasks using our ChronoSense benchmark. Our evaluation spans Allen relations and temporal arithmetic tasks across 0-shot, few-shot, and chain-of-thought (CoT) prompting scenarios.
• We demonstrate the effectiveness of few-shot and CoT prompting in improving LLM performance, especially on temporal arithmetic tasks that require step-by-step reasoning.
• We investigate the influence of memorization on LLMs’ ability to perform temporal reasoning tasks, especially when models encounter real-world event names that might have been part of pre-training data.

2 Preliminaries

Allen’s Interval Algebra. Allen’s interval algebra (IA) (Allen, 1989) provides 13 different relations between two intervals. As illustrated in Figure 1, these relations are "Equals", "Before", "After", "Overlaps", "Overlapped-by", "Contains", "During", "Started-by", "Starts", "Finished-by", "Finishes" "Meets" and "Met-by". These relations are mutually exclusive and cover all possible temporal relationships between two intervals. IA serves as a base for artificial intelligence and has been used in many applications (Janhunen and Sioutis, 2019). Although it is not the focus of this study, it allows deriving new facts. For instance, through transitivity, if Event \( e_1 \) happens before Event \( e_2 \), and Event \( e_2 \) happens before Event \( e_3 \), then Event \( e_1 \) happens before Event \( e_3 \). Therefore, correctly identifying the relationships between intervals is essential to support this type of reasoning.

3 ChronoSense Dataset

We create an event-centric dataset, named ChronoSense\(^1\). This dataset is designed to diagnose how well LLMs comprehend temporal events and the relationships between them, as illustrated in Figure 2. ChronoSense contains True/False questions that include different temporal dimensions. It features two types of questions: (1) Allen questions (requiring models to determine the Allen relation of two time intervals) and (2) temporal arithmetic tasks focused on a single event (challenging models to draw conclusions based on explicit time information). We set the time granularity to years for both question types. The prompts used in ChronoSense can be seen in Table 3 in Appendix A.

**Question Type 1:** **Comparing Two Temporal Events with Allen Relations.** We extract real event pairs from the Wikidata (Vrandečić and Krötzsch, 2014) (Section A.1). Similar to (Yang et al., 2023), every test instance in our dataset is in (Context, Hypothesis, Correctness) format. *Context* introduces the events and explicitly states the time periods when the events have occurred (e.g. *The event 'fourth cholera pandemic' occurred between year 1863 and year 1875. The event 'World War II' occurred between year 1939 and year 1945.*). *Hypothesis* verbalizes an Allen relation in natural language (e.g. *Did 'fourth cholera pandemic' occur before 'World War II' without any overlap between the two events? Answer True or False.*). *Correctness* is True if *Hypothesis* describes the temporal relationship between these two events correctly and False otherwise (e.g. *True* for the example above.).

**Question Type 2:** **Temporal Arithmetic With A Single Event.** To get insights into models’ ability to perform temporal arithmetic, we also include temporal arithmetic questions in *ChronoSense*. *Context* introduces a single event and explicitly states the time information and a temporal feature such as its duration or frequency (e.g. *'Event A' first occurred in year 1909. 'Event A' occurs every 12 years.*). *Hypothesis* is a statement that is not covered in *Context* and requires arithmetic calculations to verify (e.g. *Did 'Event A' occur again in the year 1921? Answer True or False.*). *Correctness* is True if *Hypothesis* matches with the calculations based on the *Context* and False otherwise (e.g. *True* for the example above).

\(^1\)The dataset will be released under the CC BY 4.0 license.

![An example for comparing two temporal events with LLMs.](page_184_120_496_312.png)

Figure 2: An example for comparing two temporal events with LLMs.
The temporal arithmetic questions cover three different aspects. End Timepoint focuses on the duration of an event and requires models to determine the end time based on the given start time and duration. Next Occurrence focuses on the frequency of events and challenges models to calculate when an event occurs again based on a given frequency. Intermediate Timepoint, which is novel to this work, challenges models to infer whether an event was happening between its start and end time by asking if it happened at a certain year in time. Due to the limited number of events with frequency from Wikidata, we synthetically create these questions. Therefore, the events do not have event names, but rather we name them as "Event A". For each question, we create a negative sample by creating a wrong Hypothesis (e.g. by changing the next occurrence year in the previous example from 1921 to 1950.).

Different event abstraction levels. For Allen questions, we have an abstract version of each question where we hide the names of the events by replacing them with letters such as "Event A" and "Event B". This setting allows us to see how the memorization affects LLM’s performance by comparing the abstract versions with the original versions (where we have event names).

Different prompts for questions. There are multiple ways to ask a question, so we create two different additional prompts for each question to understand the effect of the prompt. All prompts can be seen in Table 3 and Table 10 in the Section A.

Negative samples. To evaluate the robustness of the LLM’s predictions, we generate negative examples for each data instance (detailed in A.1.1). Therefore, the Correctness value is "True" in 50% of the data instances, and "False" in the other half.

Dataset statistics. For each Allen relation and each temporal arithmetic question, ChronoSense has 4,000 training samples, 500 validation samples, and 500 test samples.

4 Experiments

We evaluate the performance of various LLMs on a task framed as binary classification. Specifically, the models are tasked with answering True or False to a set of prompts on temporal reasoning. We evaluate the accuracy of the models, where we have a random chance accuracy of 50%. We compare the following LLMs in our experiments: Gemma2-9b-it2, GPT-4o (gpt-4o-2024-05-13)3, GPT-4o-mini (gpt-4o-mini-2024-07-18)4, Meta-Llama-3.1-8B-Instruct5, Mistral-7B-Instruct-v0.26 (Jiang et al., 2023), Mixtral-8x7B-Instruct-v0.17 (Jiang et al., 2024), Phi-3-mini-128k-instruct8. Each model can generate up to 64 new tokens for an answer; however, in the chain-of-thought (CoT) setting, the maximum token limit is increased to 512 to provide more space for reasoning. For both question types (Allen and temporal arithmetic), we report on different settings: 0-shot, 1-shot, 3-shot, and Chain-of-Thought (CoT) prompting. For CoT experiments, we add a "Let’s think step by step." sentence to the original prompts (Table 3). This follows the idea introduced in (Kojima et al., 2022). For Allen questions, we also report on abstract versions in which we remove the real event names. As mentioned in Section 3, the temporal arithmetic questions are all in the abstract setting.

We report the averaged results in Table 1. The complete experimental results, including the experiments on individual Allen relations, can be found in A.2. Moreover, in Table 2, we zoom in and report the 0-shot performance on individual Allen relations for three models. We include qualitative examples of failure cases in Section A.5, and provide an analysis of different prompt variants in Section A.3.

General Findings. (1) The models exhibit low performance and lack consistency on ChronoSense questions across the experiments, given the fact that the random prediction would lead to 0.50 accuracy. This suggests the need for improvements in temporal understanding in LLMs. (2) Few-shot and CoT settings are helpful for most models for Allen questions. Despite these improvements, the tasks remain challenging, as several models still have an accuracy below 0.60. (3) Arithmetic questions are typically more challenging than Allen relations in both zero-shot and few-shot settings. For these questions, the few-shot setting only im-

2https://huggingface.co/google/gemma-2-9b-it
3https://platform.openai.com/docs/models/gpt-4o
4https://platform.openai.com/docs/models/gpt-4o-mini
5https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct
6https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
7https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
8https://huggingface.co/microsoft/Phi-3-mini-128k-instruct
<table>
  <tr>
    <th>Type</th>
    <th>Setting</th>
    <th>Gemma2.9B-it</th>
    <th>GPT-4o</th>
    <th>GPT-4o-mini</th>
    <th>Llama3.18B</th>
    <th>Mistral-7B</th>
    <th>Mixtral-8x7B</th>
    <th>Phi-3-mini</th>
  </tr>
  <tr>
    <td rowspan="4">Allen</td>
    <td>0-shot</td>
    <td>0.09*</td>
    <td>0.87</td>
    <td>0.72</td>
    <td>0.13*</td>
    <td>0.50</td>
    <td>0.54</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>0.75</td>
    <td>0.93</td>
    <td>0.75</td>
    <td>0.01*</td>
    <td>0.47</td>
    <td>0.56</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>0.26*</td>
    <td>0.95</td>
    <td>0.78</td>
    <td>0.01*</td>
    <td>0.49</td>
    <td>0.58</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>0.75</td>
    <td>0.65</td>
    <td>0.69</td>
    <td>0.75</td>
    <td>0.51</td>
    <td>0.57</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td rowspan="4">Arithmetic</td>
    <td>0-shot</td>
    <td>0.76</td>
    <td>0.55</td>
    <td>0.60</td>
    <td>0.48</td>
    <td>0.36*</td>
    <td>0.35</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>1-shot</td>
    <td>0.71</td>
    <td>0.50*</td>
    <td>0.49</td>
    <td>0.19</td>
    <td>0.43</td>
    <td>0.50</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>3-shot</td>
    <td>0.54</td>
    <td>0.51</td>
    <td>0.39*</td>
    <td>0.10*</td>
    <td>0.47</td>
    <td>0.64</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>0.94</td>
    <td>0.99</td>
    <td>0.99</td>
    <td>0.92</td>
    <td>0.70</td>
    <td>0.75</td>
    <td>0.98</td>
  </tr>
</table>

Table 1: The average performance comparison between different settings on two different question types in ChronoSense. (*) indicates the models that perform poorly due to producing a high number of unclear answers (≥ 250) in the majority of the tasks. These models fail to follow the instruction by not answering with "True" or "False" as required.

<table>
  <tr>
    <th>Allen Relation</th>
    <th>GPT-4o</th>
    <th>Mixtral-8x7B</th>
    <th>Phi-3-mini</th>
  </tr>
  <tr>
    <td>After</td>
    <td>0.956</td>
    <td>0.78</td>
    <td>0.566</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>0.914</td>
    <td>0.902</td>
    <td>0.758</td>
  </tr>
  <tr>
    <td>Contains</td>
    <td>0.884</td>
    <td>0.472</td>
    <td>0.652</td>
  </tr>
  <tr>
    <td>During</td>
    <td>0.878</td>
    <td>0.512</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Equals</td>
    <td>0.69</td>
    <td>0.336</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Finished-By</td>
    <td>0.926</td>
    <td>0.398</td>
    <td>0.486</td>
  </tr>
  <tr>
    <td>Finishes</td>
    <td>0.908</td>
    <td>0.43</td>
    <td>0.492</td>
  </tr>
  <tr>
    <td>Meets</td>
    <td>0.91</td>
    <td>0.74</td>
    <td>0.488</td>
  </tr>
  <tr>
    <td>Met-By</td>
    <td>0.864</td>
    <td>0.594</td>
    <td>0.494</td>
  </tr>
  <tr>
    <td>Overlapped-By</td>
    <td>0.842</td>
    <td>0.476</td>
    <td>0.786</td>
  </tr>
  <tr>
    <td>Overlaps</td>
    <td>0.884</td>
    <td>0.43</td>
    <td>0.648</td>
  </tr>
  <tr>
    <td>Started-By</td>
    <td>0.896</td>
    <td>0.578</td>
    <td>0.474</td>
  </tr>
  <tr>
    <td>Starts</td>
    <td>0.846</td>
    <td>0.442</td>
    <td>0.492</td>
  </tr>
</table>

Table 2: 0-shot setting results for GPT-4o, Mixtral-8x7B, and Phi-3-mini on 13 Allen relations.

proves Mistral-7B and Mixtral-8x7B models. However, CoT prompting enhances model performance on arithmetic questions across all models. This is expected as these questions require step-by-step reasoning. (4) When averaged over models, some Allen relations are easier and some are more challenging for the models. First, "Before" and "After" are easier than other relations in all experiments, with one exception. This is expected as these relations are the most frequently used phrases among others. This may also indicate that the models are better at detecting relations that do not contain any overlap. Second, "Equals" is the hardest relation in zero-shot and CoT settings, and "Finishes" is the hardest for few-shot and abstract settings. The questions for both relations require checking whether the endpoints of events are the same. (5) The models do not perform similarly for symmetrical Allen relations. For instance, despite their symmetric nature, the averaged model performance for "Before" is higher than for "After" and "Meets" is higher than "Met-by". Similarly, "Contains", "Finished-by" and "Overlaps" are easier than their symmetrical relations ("During", "Finishes" and "Overlapped-by") with one exception. (6) The abstract versions are more challenging for most of the models. Models may rely on memorization to answer temporal understanding questions for the events included in the pre-training data. In other words, the implicit knowledge from pre-training can influence their performance on temporal understanding. (7) As illustrated in Section A.5, the types of model failures include: confusion between start and end years, incorrect reasoning, calculation errors (including extra calculations), incorrect conclusions despite correct explanations, and confusion caused by temporal granularity.

"Finished-by" and "Overlaps" are easier than their symmetrical relations ("During", "Finishes" and "Overlapped-by") with one exception. (6) The abstract versions are more challenging for most of the models. Models may rely on memorization to answer temporal understanding questions for the events included in the pre-training data. In other words, the implicit knowledge from pre-training can influence their performance on temporal understanding. (7) As illustrated in Section A.5, the types of model failures include: confusion between start and end years, incorrect reasoning, calculation errors (including extra calculations), incorrect conclusions despite correct explanations, and confusion caused by temporal granularity.

5 Related Work

Temporal reasoning has been extensively studied in NLP (Terenziani, 2009; Sanampudi and Kumari, 2010) and QA over temporal knowledge graphs (Dhingra et al., 2022; Zhao et al., 2022; Saxena et al., 2021; Chen et al., 2021; Jia et al., 2018a,b, 2021). A new line of work focuses on LLMs’ temporal knowledge and reasoning. TimeBench (Chu et al., 2023) covers abstract temporal expressions, commonsense reasoning, and event relationships. Other benchmarks include those by (Jain et al., 2023) for commonsense-based temporal tasks and TimeLlama (Yuan et al., 2023) for event forecasting. TGQA (Xiong et al., 2024) evaluates synthetic temporal QA but only covers three simple event relations. TRACIE (Zhou et al., 2021) assesses reasoning over implicit events, while TEMPREASON (Tan et al., 2023a) probes three levels of temporal understanding but primarily focuses on factual recall. TRAM (Wang and Zhao, 2023) includes event relations from (UzZaman et al., 2013) but lacks explicit events. (Tan
et al., 2023b) has temporal arithmetic but it is event-independent. LTLBench (Tang and Belle, 2024) uses linear temporal logic to model the temporal relationships between events. Test of Time (Fatemi et al., 2024) creates a synthetic dataset to isolate temporal reasoning. Recent works on event ordering include TDDiscourse (Naik et al., 2019), which classifies implicit event relations overlapping with Allen’s framework. Datasets from (Vashishtha et al., 2020) focus on event ordering and duration, while TORQUE (Ning et al., 2020) presents a reading comprehension dataset to investigate the temporal ordering of events but lacks explicit start and end times. Despite the variety of benchmarks, none covers all 13 of Allen’s interval relations.

6 Conclusion

We introduce ChronoSense, a diagnostic dataset designed to assess LLMs’ ability to compare event timelines using Allen relations and perform temporal arithmetic. We show that models frequently struggle with these tasks and may rely on memorization rather than reasoning. This raises critical concerns about their reliability in applications such as historical analysis, legal AI, and medical timelines. Future research should focus on improving LLMs’ temporal reasoning capabilities, integrating temporal constraint-based reasoning, and analyzing multi-event comparisons.

7 Limitations

Our work has some limitations regarding the dataset and the evaluation. Concerning the dataset, some Wikidata events have ambiguous names that may mislead the model, e.g., an exhibition event named after a painter, which may not clearly indicate a temporal event to the model. On the evaluation side, our study involves a relatively small selection of models and some closed-source models (e.g. GPT-4o). Moreover, although we test 3 different prompt versions per task, we acknowledge that the prompt content may influence the model’s performance. Lastly, we truncate the LLM outputs when they exceed the maximum token lengths. This potentially omits some of the correct answers and leads to lower accuracy scores for the respective models.

8 Ethics Statement

Our dataset, which sources events from Wikidata, inherently carries the risk of containing incorrect information. This could unintentionally propagate misinformation. While our script filters out data points containing certain triggering keywords, some event names may still include inappropriate or harmful content. This does not reflect the views or opinions of the authors. Moreover, the data points in ChronoSense do not represent individuals but rather events categorized as instances or subclasses of "occurrence" 9. However, some events include the names of individuals, such as exhibitions named after artists. Furthermore, we acknowledge the environmental impact associated with LLMs. Although our study only utilizes pre-trained models, inference with these models still demands significant computational resources.

References

James F. Allen. 1989. Maintaining Knowledge about Temporal Intervals, page 361–372.

BIG-bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.

Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive questions. Preprint, arXiv:2108.06314.

Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2023. Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models. ArXiv, abs/2311.17667.

Bhuvan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257–273.

Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2024. Test of time: A benchmark for evaluating llms on temporal reasoning. Preprint, arXiv:2406.09170.

Raghav Jain, Daivik Sojitra, Arkadeep Acharya, Sriparna Saha, Adam Jatowt, and Sandipan Dandapat. 2023. Do language models have a common sense regarding time? revisiting temporal commonsense reasoning in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6750–6774.

9 https://www.wikidata.org/wiki/Q1190554
Tomi Janhunen and Michael Sioutis. 2019. Allen’s interval algebra makes the difference. Preprint, arXiv:1909.01128.

Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jan-nik Strötgen, and Gerhard Weikum. 2018a. Tem-pquestions: A benchmark for temporal question answering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1057–1062.

Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jan-nik Strötgen, and Gerhard Weikum. 2018b. Tequila: Temporal question answering over knowledge bases. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM ’18, page 1807–1810.

Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex temporal question answering on knowledge graphs. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 792–802.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lamplé, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi-anna Lengyel, Guillaume Bour, Guillaume Lamplé, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mix-tral of experts. Preprint, arXiv:2401.04088.

Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.

Aakanksha Naik, Luke Breitfeller, and Carolyn Rose. 2019. TDDiscourse: A dataset for discourse-level temporal ordering of events. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 239–249.

Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1158–1172.

Suresh Kumar Sanampudi and G Vijaya Kumari. 2010. Temporal reasoning in natural language processing: A survey. International Journal of Computer Applications, 1(4):68–72.

Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021. Question answering over temporal knowledge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6663–6676.

Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023a. Towards benchmarking and improving the temporal reasoning capability of large language models. In Annual Meeting of the Association for Computational Linguistics.

Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023b. Towards benchmarking and improving the temporal reasoning capability of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14820–14835.

Weizhi Tang and Vaishak Belle. 2024. Ltlbench: Towards benchmarks for evaluating temporal logic reasoning in large language models. ArXiv, abs/2407.05434.

Paolo Terenziani. 2009. Qualitative Temporal Reason-ing, pages 2225–2229.

Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 task 1: TempEval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (*SEM). Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 1–9.

Siddharth Vashishtha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White. 2020. Temporal reasoning in natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4070–4078.

Denny Vrandečić and Markus Krötzsch. 2014. Wiki-data: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.

Yuqing Wang and Yun Zhao. 2023. Tram: Benchmarking temporal reasoning for large language models. ArXiv, abs/2310.00835.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. Large language models can learn temporal reasoning. Preprint, arXiv:2401.06853.

Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. 2023. Logical reasoning over natural language as knowledge representation: A survey. Preprint, arXiv:2303.12023.
Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2023. Back to the future: Towards explainable temporal reasoning with large language models. Preprint, arXiv:2310.01074.

Ruilin Zhao, Feng Zhao, Guandong Xu, Sixiao Zhang, and Hai Jin. 2022. Can language models serve as temporal knowledge bases? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2024–2037.

Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2021. Temporal reasoning on implicit events from distant supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1361–1371.
A Appendix

A.1 Allen Question Generation
To generate the Allen questions, we take the following steps:

1. We extract real-world event pairs from Wikidata (Vrandečić and Krötzsch, 2014) via SPARQL. The used Wikidata content is licensed under CC0 10.
2. We determine the valid Allen relation for this event pair by comparing the time intervals of these events.
3. In order to map these relations into text, we verbalize each Allen relation using the prompts as depicted in Table 3.

A.1.1 Negative Samples For Allen Questions
For the positive samples, we put the correct Allen relations to the Hypothesis and set the Correctness as True. However, for negative samples, we choose another Allen relation (e.g. choosing the "Meets" relation instead of "Before") and set the Correctness to False. However, since we set the time granularity as years instead of days, generating negative samples for Allen relations presents certain challenges. For example, the "Equals" relation requires that both the start and end points of two events match exactly. When we create a negative sample for "Equals", we cannot use the "Contains" relation. This is because the second event could start later and end earlier than the first event, even if the years are the same. Since the exact days/dates of the events are not known, the information provided in the context will be ambiguous. To address this issue, we exclude such problematic relations from the pool of candidate relations during negative sampling.

Below we provide a list of Allen relations along with the Allen relations that are excluded from its negative sample candidates to avoid such inconclusive cases.

• "Equals": "Overlaps", "Contains", "During",
  "Overlapped-By", "Started-By", "Starts",
  "Finished-By", "Finishes"
• "Started-By": "Contains", "Overlapped-By"
• "Starts": "Overlaps", "During"
• "Finished-By": "Overlaps", "Contains"
• "Finishes": "During", "Overlapped-By"
• "Meets": "Before", "Overlaps"
• "Met-By": "Overlapped-By", "After"

10https://www.wikidata.org/wiki/Wikidata:Licensing

A.2 Detailed Results
For Allen questions, we report the 0-shot, 1-shot, 3-shot, and Chain-of-Thought results in Table 4, Table 5, Table 6, and Table 7. Moreover, Table 8 includes the results for the abstract setting, where we replace the actual event names with abstract names such as "Event A" and "Event B". Table 9 reports the results of the 0-shot, few-shot, and chain-of-thought for temporal arithmetic questions (End Timepoint, Intermediate Timepoint and Next Occurrence).

A.3 Different Prompt Variants
ChronoSense has different prompt variants for each question type. The templates for prompt variants can be seen in Table 10. In order to show the effect of different prompts, we report the average accuracy values with standard deviation across three prompt variants in Table 11. Although there are cases with high standard deviation, we do not observe a relation that has consistently high values.

A.4 Computational Budget
We ran all experiments using HuggingFace on a single Nvidia H100 - 80GB or via the OpenAI API. None of the experiments per model took longer than 24 hours. The experiments via the OpenAI API caused costs of less than 100$.

A.5 Qualitative Results For Failure Cases
In this section, we present illustrative qualitative examples. Model outputs for selected questions are shown in Table 12. As illustrated in the table, model failures occur for several reasons. These include confusion between start and end years (Example #1), incorrect reasoning (Examples #3 and #6), and calculation errors or extra calculations (Examples #4 and #5). In some cases, the model produces incorrect answers despite providing a correct explanation (Example #2). Errors can also result from temporal granularity, as seen in GPT-4o’s response in an "Equals" question: *The information provided only states that both events occurred between 2016 and 2017. It does not specify the exact start and end dates for each event, so we cannot conclude that they began and ended in the same years.*
<table>
  <tr>
    <th>Type</th>
    <th>Question</th>
    <th>Template</th>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Equals</td>
    <td>Did 'Event A' begin in the same year as 'Event B' and end in the same year as 'Event B'? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Before</td>
    <td>Did 'Event A' occur before 'Event B', without any overlap between the two events? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>After</td>
    <td>Did 'Event A' occur after 'Event B', without any overlap between the two events? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Meets</td>
    <td>Did 'Event A' begin before 'Event B' and end before 'Event B' ended, with some overlap between the two events? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Overlaps</td>
    <td>Did 'Event A' begin before 'Event B' and end before 'Event B' ended, with some overlap between the two events? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Overlapped-By</td>
    <td>Did 'Event B' begin before 'Event A' and end before 'Event A' ended, with some overlap between the two events? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Contains</td>
    <td>Did 'Event A' begin before 'Event B' began and end after 'Event B' ended, entirely containing 'Event B'? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>During</td>
    <td>Did 'Event A' begin after 'Event B' began and end before 'Event B' ended, being entirely contained within 'Event B'? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Started-By</td>
    <td>Did 'Event B' begin in the same year as 'Event A', but end before 'Event A' ended? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Started-By</td>
    <td>Did 'Event B' begin in the same year as 'Event A', but end before 'Event A' ended? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Finished-By</td>
    <td>Did 'Event B' begin after 'Event A' began and end in the same year as 'Event A'? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Finishes</td>
    <td>Did 'Event A' begin after 'Event B' began and end in the same year as 'Event B'? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Meets</td>
    <td>Did 'Event A' end in the same year as 'Event B' began? Answer True or False.</td>
  </tr>
  <tr>
    <td>Allen</td>
    <td>Met-by</td>
    <td>Did 'Event B' end in the same year as 'Event A' began? Answer True or False.</td>
  </tr>
  <tr>
    <td>Arithmetic</td>
    <td>End timepoint</td>
    <td>Did 'Event A' occur again in the year [start-duration]? Answer True or False.</td>
  </tr>
  <tr>
    <td>Arithmetic</td>
    <td>Next occurrence</td>
    <td>Did 'Event A' occur again in the year [next-occurrence]? Answer True or False.</td>
  </tr>
  <tr>
    <td>Arithmetic</td>
    <td>Intermediate timepoint</td>
    <td>Was 'Event A' happening in the year [intermediate]? Answer True or False.</td>
  </tr>
</table>

Table 3: Templates used in ChronoSense.

<table>
  <tr>
    <th></th>
    <th>Gemma2-9B-it</th>
    <th>GPT-4o</th>
    <th>GPT-4o-mini</th>
    <th>Llama3.1-8B</th>
    <th>Mistral-7B</th>
    <th>Mixtral-8x7B</th>
    <th>Phi-3-mini</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>After</td>
    <td>0.28*</td>
    <td>0.956</td>
    <td>0.882</td>
    <td>0.136*</td>
    <td>0.918</td>
    <td>0.78</td>
    <td>0.566</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>0.458*</td>
    <td>0.914</td>
    <td>0.838</td>
    <td>0.182</td>
    <td>0.896</td>
    <td>0.902</td>
    <td>0.758</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Contains</td>
    <td>0.064*</td>
    <td>0.884</td>
    <td>0.728</td>
    <td>0.07*</td>
    <td>0.466</td>
    <td>0.472</td>
    <td>0.652</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>During</td>
    <td>0.096*</td>
    <td>0.878</td>
    <td>0.668</td>
    <td>0.142*</td>
    <td>0.476</td>
    <td>0.512</td>
    <td>0.49</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Equals</td>
    <td>0.012*</td>
    <td>0.69</td>
    <td>0.53</td>
    <td>0.166*</td>
    <td>0.102</td>
    <td>0.336</td>
    <td>0.54</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>Finished-By</td>
    <td>0.026*</td>
    <td>0.926</td>
    <td>0.786</td>
    <td>0.108*</td>
    <td>0.454</td>
    <td>0.398</td>
    <td>0.486</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Finishes</td>
    <td>0.03*</td>
    <td>0.908</td>
    <td>0.602</td>
    <td>0.134*</td>
    <td>0.416</td>
    <td>0.43</td>
    <td>0.492</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>Meets</td>
    <td>0.108*</td>
    <td>0.91</td>
    <td>0.782</td>
    <td>0.146*</td>
    <td>0.492</td>
    <td>0.74</td>
    <td>0.488</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Met-By</td>
    <td>0.06*</td>
    <td>0.864</td>
    <td>0.73</td>
    <td>0.16*</td>
    <td>0.496</td>
    <td>0.594</td>
    <td>0.494</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Overlapped-By</td>
    <td>0.042*</td>
    <td>0.842</td>
    <td>0.706</td>
    <td>0.122*</td>
    <td>0.462</td>
    <td>0.476</td>
    <td>0.786</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Overlaps</td>
    <td>0.044*</td>
    <td>0.884</td>
    <td>0.708</td>
    <td>0.12*</td>
    <td>0.476</td>
    <td>0.43</td>
    <td>0.648</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Started-By</td>
    <td>0.03*</td>
    <td>0.896</td>
    <td>0.754</td>
    <td>0.12*</td>
    <td>0.424</td>
    <td>0.578</td>
    <td>0.474</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Starts</td>
    <td>0.042*</td>
    <td>0.846</td>
    <td>0.748</td>
    <td>0.156*</td>
    <td>0.45</td>
    <td>0.442</td>
    <td>0.492</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>0.09*</td>
    <td>0.87</td>
    <td>0.72</td>
    <td>0.13*</td>
    <td>0.50</td>
    <td>0.54</td>
    <td>0.56</td>
    <td></td>
  </tr>
</table>

Table 4: 0-shot setting results on 13 Allen questions with explicit event names. (*) indicates a high number of unclear answers (\( \geq 250 \)).

<table>
  <tr>
    <th></th>
    <th>Gemma2-9B-it</th>
    <th>GPT-4o</th>
    <th>GPT-4o-mini</th>
    <th>Llama3.1-8B</th>
    <th>Mistral-7B</th>
    <th>Mixtral-8x7B</th>
    <th>Phi-3-mini</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>After</td>
    <td>0.922</td>
    <td>0.966</td>
    <td>0.898</td>
    <td>0.006*</td>
    <td>0.846</td>
    <td>0.85</td>
    <td>0.882</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>0.976</td>
    <td>0.964</td>
    <td>0.912</td>
    <td>0.008*</td>
    <td>0.908</td>
    <td>0.886</td>
    <td>0.964</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>Contains</td>
    <td>0.824</td>
    <td>0.972</td>
    <td>0.794</td>
    <td>0.014*</td>
    <td>0.448</td>
    <td>0.526</td>
    <td>0.594</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>During</td>
    <td>0.716</td>
    <td>0.95</td>
    <td>0.744</td>
    <td>0.014*</td>
    <td>0.432</td>
    <td>0.596</td>
    <td>0.444</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Equals</td>
    <td>0.604</td>
    <td>0.872</td>
    <td>0.768</td>
    <td>0.056*</td>
    <td>0.316</td>
    <td>0.578</td>
    <td>0.612</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Finished-By</td>
    <td>0.784</td>
    <td>0.984</td>
    <td>0.754</td>
    <td>0.01*</td>
    <td>0.408</td>
    <td>0.522</td>
    <td>0.664</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Finishes</td>
    <td>0.464</td>
    <td>0.944</td>
    <td>0.518</td>
    <td>0.006*</td>
    <td>0.358</td>
    <td>0.472</td>
    <td>0.416</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Meets</td>
    <td>0.91</td>
    <td>0.932</td>
    <td>0.88</td>
    <td>0.022*</td>
    <td>0.442</td>
    <td>0.494</td>
    <td>0.478</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Met-By</td>
    <td>0.65</td>
    <td>0.864</td>
    <td>0.646</td>
    <td>0.022*</td>
    <td>0.43</td>
    <td>0.484</td>
    <td>0.466</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Overlapped-By</td>
    <td>0.686</td>
    <td>0.812</td>
    <td>0.642</td>
    <td>0.006*</td>
    <td>0.356</td>
    <td>0.482</td>
    <td>0.626</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>Overlaps</td>
    <td>0.84</td>
    <td>0.92</td>
    <td>0.576</td>
    <td>0.01*</td>
    <td>0.472</td>
    <td>0.55</td>
    <td>0.574</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Started-By</td>
    <td>0.602</td>
    <td>0.99</td>
    <td>0.856</td>
    <td>0.008*</td>
    <td>0.37</td>
    <td>0.46</td>
    <td>0.394</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Starts</td>
    <td>0.884</td>
    <td>0.936</td>
    <td>0.856</td>
    <td>0.004*</td>
    <td>0.418</td>
    <td>0.506</td>
    <td>0.56</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>0.75</td>
    <td>0.93</td>
    <td>0.75</td>
    <td>0.01*</td>
    <td>0.47</td>
    <td>0.56</td>
    <td>0.59</td>
    <td></td>
  </tr>
</table>

Table 5: 1-shot setting results on Allen questions with explicit event names. (*) indicates a high number of unclear answers (\( \geq 250 \)).
<table>
  <tr>
    <th></th>
    <th>Gemma2-9B-it</th>
    <th>GPT-4o</th>
    <th>GPT-4o-mini</th>
    <th>Llama3.1-8B</th>
    <th>Mistral-7B</th>
    <th>Mistral-8x7B</th>
    <th>Phi-3-mini</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>After</td>
    <td>0.318</td>
    <td>0.974</td>
    <td>0.892</td>
    <td>0.008*</td>
    <td>0.734</td>
    <td>0.798</td>
    <td>0.866</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>0.384</td>
    <td>0.966</td>
    <td>0.928</td>
    <td>0.014*</td>
    <td>0.768</td>
    <td>0.944</td>
    <td>0.954</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Contains</td>
    <td>0.222*</td>
    <td>0.984</td>
    <td>0.832</td>
    <td>0.014*</td>
    <td>0.45</td>
    <td>0.504</td>
    <td>0.512</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>During</td>
    <td>0.218*</td>
    <td>0.978</td>
    <td>0.774</td>
    <td>0.008*</td>
    <td>0.46</td>
    <td>0.534</td>
    <td>0.42</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Equals</td>
    <td>0.364</td>
    <td>0.948</td>
    <td>0.866</td>
    <td>0.046*</td>
    <td>0.414</td>
    <td>0.52</td>
    <td>0.862</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Finished-By</td>
    <td>0.23*</td>
    <td>0.98</td>
    <td>0.7</td>
    <td>0.004*</td>
    <td>0.462</td>
    <td>0.512</td>
    <td>0.668</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Finishes</td>
    <td>0.128*</td>
    <td>0.972</td>
    <td>0.558</td>
    <td>0.008*</td>
    <td>0.402</td>
    <td>0.414</td>
    <td>0.452</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Meets</td>
    <td>0.484</td>
    <td>0.956</td>
    <td>0.924</td>
    <td>0.012*</td>
    <td>0.464</td>
    <td>0.722</td>
    <td>0.624</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Met-By</td>
    <td>0.264*</td>
    <td>0.89</td>
    <td>0.69</td>
    <td>0.018*</td>
    <td>0.468</td>
    <td>0.558</td>
    <td>0.494</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Overlapped-By</td>
    <td>0.194*</td>
    <td>0.804</td>
    <td>0.63</td>
    <td>0.014*</td>
    <td>0.454</td>
    <td>0.392</td>
    <td>0.732</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>Overlaps</td>
    <td>0.152*</td>
    <td>0.944</td>
    <td>0.518</td>
    <td>0.002*</td>
    <td>0.474</td>
    <td>0.576</td>
    <td>0.74</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Started-By</td>
    <td>0.258*</td>
    <td>0.998</td>
    <td>0.934</td>
    <td>0.004*</td>
    <td>0.442</td>
    <td>0.54</td>
    <td>0.584</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>Starts</td>
    <td>0.286*</td>
    <td>0.98</td>
    <td>0.916</td>
    <td>0.0*</td>
    <td>0.446</td>
    <td>0.576</td>
    <td>0.78</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>0.26*</td>
    <td>0.95</td>
    <td>0.78</td>
    <td>0.01*</td>
    <td>0.49</td>
    <td>0.58</td>
    <td>0.66</td>
    <td></td>
  </tr>
</table>

Table 6: 3-shot setting results on Allen questions with explicit event names. (*) indicates a high number of unclear answers (≥ 250).

<table>
  <tr>
    <th></th>
    <th>Gemma2-9B-it</th>
    <th>GPT-4o</th>
    <th>GPT-4o-mini</th>
    <th>Llama3.1-8B</th>
    <th>Mistral-7B</th>
    <th>Mistral-8x7B</th>
    <th>Phi-3-mini</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>After</td>
    <td>0.922</td>
    <td>0.778</td>
    <td>0.822</td>
    <td>0.81</td>
    <td>0.928</td>
    <td>0.808</td>
    <td>0.87</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>0.956</td>
    <td>0.766</td>
    <td>0.848</td>
    <td>0.894</td>
    <td>0.918</td>
    <td>0.934</td>
    <td>0.932</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Contains</td>
    <td>0.748</td>
    <td>0.666</td>
    <td>0.766</td>
    <td>0.682</td>
    <td>0.41</td>
    <td>0.676</td>
    <td>0.798</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>During</td>
    <td>0.846</td>
    <td>0.684</td>
    <td>0.804</td>
    <td>0.732</td>
    <td>0.434</td>
    <td>0.636</td>
    <td>0.644</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Equals</td>
    <td>0.452</td>
    <td>0.596</td>
    <td>0.672</td>
    <td>0.482</td>
    <td>0.248</td>
    <td>0.406</td>
    <td>0.576</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>Finished-By</td>
    <td>0.69</td>
    <td>0.58</td>
    <td>0.446*</td>
    <td>0.756</td>
    <td>0.444</td>
    <td>0.512</td>
    <td>0.794</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>Finishes</td>
    <td>0.658</td>
    <td>0.584</td>
    <td>0.502</td>
    <td>0.766</td>
    <td>0.436</td>
    <td>0.46</td>
    <td>0.852</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>Meets</td>
    <td>0.812</td>
    <td>0.788</td>
    <td>0.818</td>
    <td>0.828</td>
    <td>0.468</td>
    <td>0.586</td>
    <td>0.87</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>Met-By</td>
    <td>0.69</td>
    <td>0.768</td>
    <td>0.814</td>
    <td>0.81</td>
    <td>0.532</td>
    <td>0.512</td>
    <td>0.85</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Overlapped-By</td>
    <td>0.746</td>
    <td>0.542</td>
    <td>0.66</td>
    <td>0.66</td>
    <td>0.486</td>
    <td>0.468</td>
    <td>0.552</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Overlaps</td>
    <td>0.738</td>
    <td>0.504</td>
    <td>0.712</td>
    <td>0.726</td>
    <td>0.598</td>
    <td>0.488</td>
    <td>0.538</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>Started-By</td>
    <td>0.68</td>
    <td>0.602</td>
    <td>0.528</td>
    <td>0.8</td>
    <td>0.426</td>
    <td>0.526</td>
    <td>0.75</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>Starts</td>
    <td>0.828</td>
    <td>0.662</td>
    <td>0.586</td>
    <td>0.84</td>
    <td>0.408</td>
    <td>0.496</td>
    <td>0.818</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>0.75</td>
    <td>0.65</td>
    <td>0.69</td>
    <td>0.75</td>
    <td>0.51</td>
    <td>0.57</td>
    <td>0.75</td>
    <td></td>
  </tr>
</table>

Table 7: Chain-of-Thought setting results on Allen questions with explicit event names. (*) indicates a high number of unclear answers (≥ 250).

<table>
  <tr>
    <th></th>
    <th>Gemma2-9B-it</th>
    <th>GPT-4o</th>
    <th>GPT-4o-mini</th>
    <th>Llama3.1-8B</th>
    <th>Mistral-7B</th>
    <th>Mistral-8x7B</th>
    <th>Phi-3-mini</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>After</td>
    <td>0.206*</td>
    <td>0.948</td>
    <td>0.87</td>
    <td>0.076*</td>
    <td>0.448*</td>
    <td>0.672</td>
    <td>0.498</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>0.498*</td>
    <td>0.918</td>
    <td>0.82</td>
    <td>0.152*</td>
    <td>0.458*</td>
    <td>0.846</td>
    <td>0.742</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Contains</td>
    <td>0.142*</td>
    <td>0.848</td>
    <td>0.666</td>
    <td>0.038*</td>
    <td>0.2</td>
    <td>0.38</td>
    <td>0.85</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>During</td>
    <td>0.286*</td>
    <td>0.806</td>
    <td>0.512</td>
    <td>0.062*</td>
    <td>0.202*</td>
    <td>0.33</td>
    <td>0.496</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>Equals</td>
    <td>0.0*</td>
    <td>0.448</td>
    <td>0.382</td>
    <td>0.522</td>
    <td>0.22*</td>
    <td>0.016*</td>
    <td>0.692</td>
    <td>0.32</td>
  </tr>
  <tr>
    <td>Finished-By</td>
    <td>0.024*</td>
    <td>0.852</td>
    <td>0.736</td>
    <td>0.074*</td>
    <td>0.182*</td>
    <td>0.358*</td>
    <td>0.5</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>Finishes</td>
    <td>0.026*</td>
    <td>0.708</td>
    <td>0.58</td>
    <td>0.08*</td>
    <td>0.054*</td>
    <td>0.184*</td>
    <td>0.492</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>Meets</td>
    <td>0.602</td>
    <td>0.938</td>
    <td>0.716</td>
    <td>0.47</td>
    <td>0.404</td>
    <td>0.4</td>
    <td>0.5</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Met-By</td>
    <td>0.062*</td>
    <td>0.752</td>
    <td>0.682</td>
    <td>0.126*</td>
    <td>0.306</td>
    <td>0.426</td>
    <td>0.494</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Overlapped-By</td>
    <td>0.008*</td>
    <td>0.552</td>
    <td>0.46</td>
    <td>0.034*</td>
    <td>0.188*</td>
    <td>0.216</td>
    <td>0.796</td>
    <td>0.32</td>
  </tr>
  <tr>
    <td>Overlaps</td>
    <td>0.014*</td>
    <td>0.77</td>
    <td>0.508</td>
    <td>0.032*</td>
    <td>0.218*</td>
    <td>0.292*</td>
    <td>0.926</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>Started-By</td>
    <td>0.008*</td>
    <td>0.884</td>
    <td>0.738</td>
    <td>0.134*</td>
    <td>0.098*</td>
    <td>0.24</td>
    <td>0.496</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Starts</td>
    <td>0.1*</td>
    <td>0.844</td>
    <td>0.67</td>
    <td>0.106*</td>
    <td>0.09*</td>
    <td>0.218*</td>
    <td>0.498</td>
    <td>0.36</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>0.15*</td>
    <td>0.78</td>
    <td>0.64</td>
    <td>0.14*</td>
    <td>0.23*</td>
    <td>0.35</td>
    <td>0.61</td>
    <td></td>
  </tr>
</table>

Table 8: 0-shot setting results on Allen questions with the abstract event names. (*) indicates a high number of unclear answers (≥ 250).
<table>
  <tr>
    <th></th>
    <th>Gemma2-9B-it</th>
    <th>GPT-4o</th>
    <th>GPT-4o-mini</th>
    <th>Llama3.1-8B</th>
    <th>Mistral-7B</th>
    <th>Mistral-8x7B</th>
    <th>Phi-3-mini</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>End-Timepoint (0-shot)</td>
    <td>0.67</td>
    <td>0.552</td>
    <td>0.652</td>
    <td>0.462</td>
    <td>0.558</td>
    <td>0.456</td>
    <td>0.604</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Intermediate-Timepoint (0-shot)</td>
    <td>0.938</td>
    <td>1.0</td>
    <td>0.996</td>
    <td>0.878</td>
    <td>0.452*</td>
    <td>0.468</td>
    <td>0.994</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>Next-Occurrence (0-shot)</td>
    <td>0.678</td>
    <td>0.126*</td>
    <td>0.158*</td>
    <td>0.104*</td>
    <td>0.082*</td>
    <td>0.128*</td>
    <td>0.432</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>Average (0-shot)</td>
    <td>0.76</td>
    <td>0.55</td>
    <td>0.60</td>
    <td>0.48</td>
    <td>0.36*</td>
    <td>0.35</td>
    <td>0.67</td>
    <td></td>
  </tr>
  <tr>
    <td>End-Timepoint (1-shot)</td>
    <td>0.588</td>
    <td>0.446*</td>
    <td>0.404</td>
    <td>0.156</td>
    <td>0.44</td>
    <td>0.528</td>
    <td>0.556</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Intermediate-Timepoint (1-shot)</td>
    <td>0.984</td>
    <td>1.0</td>
    <td>0.988</td>
    <td>0.392</td>
    <td>0.518</td>
    <td>0.584</td>
    <td>0.872</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>Next-Occurrence (1-shot)</td>
    <td>0.578</td>
    <td>0.062*</td>
    <td>0.1*</td>
    <td>0.038</td>
    <td>0.336</td>
    <td>0.394</td>
    <td>0.522</td>
    <td>0.29</td>
  </tr>
  <tr>
    <td>Average (1-shot)</td>
    <td>0.71</td>
    <td>0.50*</td>
    <td>0.49</td>
    <td>0.19</td>
    <td>0.43</td>
    <td>0.50</td>
    <td>0.65</td>
    <td></td>
  </tr>
  <tr>
    <td>End-Timepoint (3-shot)</td>
    <td>0.588</td>
    <td>0.494</td>
    <td>0.172*</td>
    <td>0.076*</td>
    <td>0.358</td>
    <td>0.614</td>
    <td>0.352</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Intermediate-Timepoint (3-shot)</td>
    <td>0.972</td>
    <td>1.0</td>
    <td>0.998</td>
    <td>0.252*</td>
    <td>0.664</td>
    <td>0.8</td>
    <td>0.476</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>Next-Occurrence (3-shot)</td>
    <td>0.082*</td>
    <td>0.054*</td>
    <td>0.008*</td>
    <td>0.0*</td>
    <td>0.414</td>
    <td>0.534</td>
    <td>0.31</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>Average (3-shot)</td>
    <td>0.54</td>
    <td>0.51</td>
    <td>0.39*</td>
    <td>0.10*</td>
    <td>0.47</td>
    <td>0.64</td>
    <td>0.37</td>
    <td></td>
  </tr>
  <tr>
    <td>End-Timepoint (CoT)</td>
    <td>0.992</td>
    <td>0.978</td>
    <td>0.978</td>
    <td>0.988</td>
    <td>0.798</td>
    <td>0.92</td>
    <td>0.996</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>Intermediate-Timepoint (CoT)</td>
    <td>0.978</td>
    <td>0.998</td>
    <td>0.998</td>
    <td>0.972</td>
    <td>0.542</td>
    <td>0.566</td>
    <td>0.984</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>Next-Occurrence (CoT)</td>
    <td>0.874</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>0.82</td>
    <td>0.768</td>
    <td>0.788</td>
    <td>0.962</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>Average (CoT)</td>
    <td>0.94</td>
    <td>0.99</td>
    <td>0.99</td>
    <td>0.92</td>
    <td>0.70</td>
    <td>0.75</td>
    <td>0.98</td>
    <td></td>
  </tr>
</table>

Table 9: The results on all temporal arithmetic questions in 0-, 1-, and 3-shot settings, as well as using CoT prompting. (*) indicates a high number of unclear answers (\( \geq 250 \)).
<table>
  <tr>
    <th>Question</th>
    <th>Prompt alternative 1</th>
    <th>Prompt alternative 2</th>
  </tr>
  <tr>
    <td>Equals</td>
    <td>Does ‘Event A’ have identical start and end years as ‘Event B’? Answer True or False.</td>
    <td>Are the starting and ending years of ‘Event A’ and ‘Event B’ the same? Answer True or False.</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>Is it true that ‘Event A’ took place completely before ‘Event B’? Answer True or False.</td>
    <td>Can it be confirmed that ‘Event A’ completely preceded ‘Event B’? Answer True or False.</td>
  </tr>
  <tr>
    <td>After</td>
    <td>Is it true that ‘Event A’ took place completely after ‘Event B’? Answer True or False.</td>
    <td>Can it be confirmed that ‘Event A’ completely succeeded ‘Event B’? Answer True or False.</td>
  </tr>
  <tr>
    <td>Overlaps</td>
    <td>Does ‘Event A’ overlap with ‘Event B’ by starting before and ending during it? Answer True or False.</td>
    <td>Is there a period where ‘Event A’ and ‘Event B’ overlapped, with ‘Event A’ starting and ending first? Answer True or False.</td>
  </tr>
  <tr>
    <td>Overlapped-By</td>
    <td>Does ‘Event A’ overlap with ‘Event B’ by starting after and ending after it? Answer True or False.</td>
    <td>Is there a period where ‘Event A’ and ‘Event B’ overlapped, with ‘Event A’ starting and ending last? Answer True or False.</td>
  </tr>
  <tr>
    <td>Contains</td>
    <td>Does ‘Event A’ fully enclose ‘Event A’, starting before and ending after ‘Event B’? Answer True or False.</td>
    <td>Does the time interval of ‘Event A’ contain the time interval of ‘Event B’? Answer True or False.</td>
  </tr>
  <tr>
    <td>During</td>
    <td>Is ‘Event A’ fully enclosed by ‘Event B’, starting and ending within ‘Event B’s duration? Answer True or False.</td>
    <td>Can ‘Event A’ be considered to occur entirely during ‘Event B’, from start to finish? Answer True or False.</td>
  </tr>
  <tr>
    <td>Started-By</td>
    <td>Does ‘Event A’ have the same starting year as ‘Event B’ but finish later? Answer True or False.</td>
    <td>Did ‘Event A’ start in the same year as ‘Event B’ yet end later? Answer True or False.</td>
  </tr>
  <tr>
    <td>Starts</td>
    <td>Does ‘Event A’ have the same starting year as ‘Event B’ but finish earlier? Answer True or False.</td>
    <td>Did ‘Event A’ start in the same year as ‘Event B’ yet end sooner? Answer True or False.</td>
  </tr>
  <tr>
    <td>Finished-By</td>
    <td>Does ‘Event A’ start before the start of ‘Event B’ and finish in the same calendar year? Answer True or False.</td>
    <td>Is ‘Event A’ starting earlier than ‘Event A’ and concluding within the same year? Answer True or False.</td>
  </tr>
  <tr>
    <td>Finishes</td>
    <td>Does ‘Event A’ start after the start of ‘Event B’ and finish in the same calendar year? Answer True or False.</td>
    <td>Is ‘Event A’ starting later than ‘Event B’ and concluding within the same year? Answer True or False.</td>
  </tr>
  <tr>
    <td>Meets</td>
    <td>Is the end of ‘Event A’ coinciding with the start of ‘Event B’ in the same year? Answer True or False.</td>
    <td>Does the end of ‘Event A’ align with the beginning of ‘Event B’ within the same year? Answer True or False.</td>
  </tr>
  <tr>
    <td>Met-by</td>
    <td>Is the start of ‘Event A’ coinciding with the end of ‘Event B’ in the same year? Answer True or False.</td>
    <td>Does the beginning of ‘Event A’ align with the end of ‘Event B’ within the same year? Answer True or False.</td>
  </tr>
  <tr>
    <td>End timepoint</td>
    <td>Is the conclusion of ‘Event A’ marked within the year [start+duration]? Answer True or False.</td>
    <td>Can it be confirmed that ‘Event A’ finished in the year [start+duration]? Answer True or False.</td>
  </tr>
  <tr>
    <td>Next occurrence</td>
    <td>Is a recurrence of ‘Event A’ expected in the year [next-occurrence]? Answer True or False.</td>
    <td>Can we anticipate another instance of ‘Event A’ in the year [next-occurrence]? Answer True or False.</td>
  </tr>
  <tr>
    <td>Intermediate timepoint</td>
    <td>During the year [intermediate], was ‘Event A’ in progress? Answer True or False.</td>
    <td>In the year [intermediate], can it be verified that ‘Event A’ was active? Answer True or False.</td>
  </tr>
</table>

Table 10: The different prompt variants used in ChronoSense.
<table>
  <tr>
    <th></th>
    <th>Gemma2-9b-it</th>
    <th>GPT-4o</th>
    <th>GPT-4o-mini</th>
    <th>Llama3.1-8B</th>
    <th>Mistral-7B</th>
    <th>Mistral-8x7B</th>
    <th>Phi-3-mini</th>
  </tr>
  <tr>
    <td>After</td>
    <td>0.36 ± 0.09</td>
    <td>0.95 ± 0.00</td>
    <td>0.82 ± 0.15</td>
    <td>0.18 ± 0.04</td>
    <td>0.65 ± 0.23</td>
    <td>0.67 ± 0.18</td>
    <td>0.55 ± 0.05</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>0.63 ± 0.17</td>
    <td>0.95 ± 0.04</td>
    <td>0.93 ± 0.08</td>
    <td>0.47 ± 0.25</td>
    <td>0.87 ± 0.06</td>
    <td>0.93 ± 0.03</td>
    <td>0.82 ± 0.15</td>
  </tr>
  <tr>
    <td>Contains</td>
    <td>0.13 ± 0.06</td>
    <td>0.92 ± 0.03</td>
    <td>0.60 ± 0.16</td>
    <td>0.13 ± 0.05</td>
    <td>0.48 ± 0.01</td>
    <td>0.50 ± 0.04</td>
    <td>0.67 ± 0.21</td>
  </tr>
  <tr>
    <td>During</td>
    <td>0.13 ± 0.04</td>
    <td>0.87 ± 0.02</td>
    <td>0.50 ± 0.15</td>
    <td>0.17 ± 0.02</td>
    <td>0.48 ± 0.01</td>
    <td>0.49 ± 0.02</td>
    <td>0.49 ± 0.01</td>
  </tr>
  <tr>
    <td>Equals</td>
    <td>0.08 ± 0.11</td>
    <td>0.81 ± 0.11</td>
    <td>0.41 ± 0.13</td>
    <td>0.30 ± 0.13</td>
    <td>0.10 ± 0.01</td>
    <td>0.42 ± 0.11</td>
    <td>0.51 ± 0.03</td>
  </tr>
  <tr>
    <td>Finished-By</td>
    <td>0.06 ± 0.03</td>
    <td>0.79 ± 0.25</td>
    <td>0.67 ± 0.30</td>
    <td>0.16 ± 0.05</td>
    <td>0.49 ± 0.08</td>
    <td>0.45 ± 0.05</td>
    <td>0.48 ± 0.01</td>
  </tr>
  <tr>
    <td>Finishes</td>
    <td>0.06 ± 0.03</td>
    <td>0.89 ± 0.02</td>
    <td>0.63 ± 0.06</td>
    <td>0.16 ± 0.02</td>
    <td>0.45 ± 0.03</td>
    <td>0.49 ± 0.06</td>
    <td>0.52 ± 0.04</td>
  </tr>
  <tr>
    <td>Meets</td>
    <td>0.18 ± 0.06</td>
    <td>0.90 ± 0.03</td>
    <td>0.75 ± 0.04</td>
    <td>0.20 ± 0.05</td>
    <td>0.48 ± 0.02</td>
    <td>0.57 ± 0.15</td>
    <td>0.48 ± 0.00</td>
  </tr>
  <tr>
    <td>Met-By</td>
    <td>0.10 ± 0.04</td>
    <td>0.81 ± 0.05</td>
    <td>0.66 ± 0.06</td>
    <td>0.19 ± 0.03</td>
    <td>0.49 ± 0.01</td>
    <td>0.52 ± 0.07</td>
    <td>0.47 ± 0.03</td>
  </tr>
  <tr>
    <td>Overlapped-By</td>
    <td>0.08 ± 0.04</td>
    <td>0.68 ± 0.19</td>
    <td>0.64 ± 0.15</td>
    <td>0.16 ± 0.04</td>
    <td>0.47 ± 0.01</td>
    <td>0.46 ± 0.01</td>
    <td>0.57 ± 0.18</td>
  </tr>
  <tr>
    <td>Overlaps</td>
    <td>0.08 ± 0.03</td>
    <td>0.85 ± 0.04</td>
    <td>0.78 ± 0.09</td>
    <td>0.16 ± 0.04</td>
    <td>0.49 ± 0.02</td>
    <td>0.50 ± 0.06</td>
    <td>0.59 ± 0.11</td>
  </tr>
  <tr>
    <td>Started-By</td>
    <td>0.09 ± 0.05</td>
    <td>0.93 ± 0.03</td>
    <td>0.74 ± 0.09</td>
    <td>0.17 ± 0.05</td>
    <td>0.46 ± 0.03</td>
    <td>0.54 ± 0.08</td>
    <td>0.64 ± 0.14</td>
  </tr>
  <tr>
    <td>Starts</td>
    <td>0.09 ± 0.05</td>
    <td>0.86 ± 0.01</td>
    <td>0.79 ± 0.06</td>
    <td>0.18 ± 0.02</td>
    <td>0.45 ± 0.01</td>
    <td>0.47 ± 0.02</td>
    <td>0.51 ± 0.08</td>
  </tr>
  <tr>
    <td>End-Timepoint</td>
    <td>0.55 ± 0.19</td>
    <td>0.58 ± 0.10</td>
    <td>0.58 ± 0.14</td>
    <td>0.39 ± 0.19</td>
    <td>0.41 ± 0.24</td>
    <td>0.50 ± 0.06</td>
    <td>0.60 ± 0.01</td>
  </tr>
  <tr>
    <td>Intermediate-Timepoint</td>
    <td>0.64 ± 0.26</td>
    <td>1.00 ± 0.00</td>
    <td>0.98 ± 0.02</td>
    <td>0.58 ± 0.32</td>
    <td>0.29 ± 0.22</td>
    <td>0.25 ± 0.20</td>
    <td>0.96 ± 0.03</td>
  </tr>
  <tr>
    <td>Next-Occurrence</td>
    <td>0.56 ± 0.11</td>
    <td>0.18 ± 0.07</td>
    <td>0.17 ± 0.05</td>
    <td>0.14 ± 0.09</td>
    <td>0.16 ± 0.07</td>
    <td>0.15 ± 0.03</td>
    <td>0.48 ± 0.04</td>
  </tr>
</table>

Table 11: The mean accuracy and standard deviation values for three prompt variants.

<table>
  <tr>
    <th>#</th>
    <th>Model & Setting</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td>1</td>
    <td>GPT-4o-mini<br>Finishes<br>0-shot<br>False</td>
    <td>"The event 'Unpacking Europe' occurred between year 2001 and year 2002. The event 'Intercontinental Handball Cup' occurred between year 1998 and year 2002. Did 'Unpacking Europe' end in the same year as 'Intercontinental Handball Cup' began? Answer True or False."</td>
    <td>"True. The event 'Unpacking Europe' ended in 2002, and the 'Intercontinental Handball Cup' began in 1998. Therefore, 'Unpacking Europe' ended in the same year that 'Intercontinental Handball Cup' began."</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Phi-3-mini<br>Finished-By<br>0-shot<br>True</td>
    <td>"The event 'Estonian War of Independence' occurred between year 1918 and year 1920. The event 'Syrian National Congress' occurred between year 1919 and year 1920. Did 'Estonian War of Independence' begin before 'Syrian National Congress'? Answer True or False."</td>
    <td>"False. The Estonian War of Independence began in 1918 and ended in 1920, while the Syrian National Congress took place in 1919. Although both events ended in the same year, the Estonian War of Independence began before the Syrian National Congress."</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Llama3.1-8B<br>Before<br>CoT<br>False</td>
    <td>"The event 'War of the Generals' occurred between year 1911 and year 1912. The event 'Black Power Revolution' occurred between year 1968 and year 1970. Did 'War of the Generals' occur after 'Black Power Revolution' without any overlap between the two events? Answer True or False."</td>
    <td>"To determine if 'War of the Generals' occurred after 'Black Power Revolution' without any overlap, let's analyze the information step by step. 1. 'Black Power Revolution' occurred between 1968 and 1970. 2. 'War of the Generals' occurred between 1911 and 1912. Therefore, 'War of the Generals' occurred after 'Black Power Revolution' and there was no overlap between the two events. The answer is True."</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Mistral-7B<br>End-Timepoint<br>0-shot<br>True</td>
    <td>"Event A' started in 1948. 'Event A' took exactly 39 years. Did 'Event A' end in the year 1987? Answer True or False."</td>
    <td>"False. If 'Event A' started in 1948 and took exactly 39 years, then it would have ended in the year 1987 plus 39 years, which is 2026."</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Mistral-7B<br>Next-occurrence<br>1-shot<br>False</td>
    <td>"Event A' first occurred in year 1773. 'Event A' occurs every 5 years. Did 'Event A' occur again in the year 1779? Answer True or False."</td>
    <td>"Answer:True. If 'Event A' occurs every 5 years, then it would have occurred again in the year 1779 (which is 5 years after 1773)."</td>
  </tr>
  <tr>
    <td>6</td>
    <td>Mistral-8x/B<br>Next-occurrence<br>CoT<br>True</td>
    <td>"Event A' first occurred in year 1555. 'Event A' occurs every 6 years. Did 'Event A' occur again in the year 1561? Answer True or False."</td>
    <td>"To determine if 'Event A' occurred again in the year 1561, we need to see if 1561 is a multiple of 6, as we know 'Event A' happens every 6 years. 1. Multiply 6 by 6 to get 36. 2. Divide 1561 by 6: 1561 / 6 = ~260.1. Since x is a whole number (6 times 256 equals 1536), 'Event A' does not occur in the year 1561. The answer is False."</td>
  </tr>
</table>

Table 12: Qualitative examples for failure cases.

Reviewer paper summaries: The paper introduces a novel temporal interval corpus grounded in Allen's interval relations. This corpus can be utilized to evaluate the capabilities of LLMs in comprehending temporal interval relations, which comprising several tasks.
The author evaluates the performance of current llms concerning Allen's interval relations, providing valuable insights.
    
Overall, this article addresses the shortcomings of previous temporal interval datasets by proposing a more comprehensive, diverse, and unbiased dataset, further advancing the development of the field of temporal reasoning.
Summary of strengths: Understanding temporal intervals is an important aspect of temporal reasoning. Although some previous studies have proposed benchmarks based on Allen's temporal intervals, they have fallen behind the current demands of the era of LLMs.

Compared to previous temporal interval datasets, this article makes the following improvements:

- Earlier datasets focused on a subset of temporal reasoning, while this article covers the complete set of Allen relations.

- The dataset introduces arithmetic tasks to evaluate the model's abstract understanding and reasoning capabilities regarding time.

- The article designs an abstract version of the dataset that removes specific entities from the questions to eliminate biases caused by the model's internal memory, allowing for a more accurate assessment of the model's temporal reasoning abilities.
Summary of weaknesses: Although the dataset presented in the article addresses the shortcomings of chronological datasets, the task may seem somewhat outdated in the era of large language models. (Note that this does not imply that the dataset introduced in this article is without value. On the contrary, it adapts traditional tasks to the era of large language model reasoning.)

The number of labeled samples in the article may be relatively small, with only 4,000 for training, 500 for validation, and 50 for testing. Such a limited number of test samples could introduce randomness into the evaluation.
Comments, suggestions and typos: Is it possible for the authors to adjust the distribution of the dataset?  
For example, in the age of llms, reasoning without training has become a common evaluation paradigm. Given this, placing 4,000 labeled samples into the training set might be seen as somewhat inefficient. Allocating a greater number of samples to the test set could enhance future data utilization and ensure a fairer assessment.
Overall assessment scores: 3