Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models
Authors: Raghav Jain
Daivik Sojitra
Arkadeep Acharya
Sriparna Saha
Adam Jatowt
Sandipan Dandapat
Venue: EMNLP 2023 Main (2023)
Keywords: Benchmarking
Temporal Reasoning
LLMs

Abstract: Temporal reasoning represents a vital component of human communication and understanding, yet remains an underexplored area within the context of Large Language Models (LLMs). Despite LLMs demonstrating significant proficiency in a range of tasks, a comprehensive, large-scale analysis of their temporal reasoning capabilities is missing. Our paper addresses this gap, presenting the first extensive benchmarking of LLMs on temporal reasoning tasks. We critically evaluate 8 different LLMs across 6 datasets using 3 distinct prompting strategies.  Additionally, we broaden the scope of our evaluation by including in our analysis 2 Code Generation LMs. Beyond broad benchmarking of models and prompts, we also conduct a fine-grained investigation of performance across different categories of temporal tasks. We further analyze the LLMs on varying temporal aspects, offering insights into their proficiency in understanding and predicting the continuity, sequence, and progression of events over time. Our findings reveal a nuanced depiction of the capabilities and limitations of the models within temporal reasoning, offering a comprehensive reference for future research in this pivotal domain.

Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models
Raghav Jain', Daivik Sojitra', Arkadeep Acharya', Sriparna Saha',
Adam Jatowt?, and Sandipan Dandapat
Depaarmentt of Computer Science and Engineering, Indian Institute of Technology Patna University of Innsbruck, Austria
Microsoft, India
@ighavjain166@@maill com
Abstract
2021). Temporal reasoning extends beyond the mere awareness of time requires a nuanced un- derstanding of time's relation to events and actions. For instance, humans inherently know that a vaca- tion usually lasts longer than a walk and occurs less frequently. Therefore, the quest to enhance tempo- ral reasoning within Al systems is of paramount importance. A language model with robust under- standing of temporal context is primed to perform better on downstream Natural Language Process- ing (NLP) tasks such as storytelling (Mostafazadeh et al., 2016), natural language inference (Hosokawa et al., 2023), timeline understanding (Steen and Markert, 2019). and user status tracking (Xia and Qi, 2022). The integration of temporal reasoning not only enhances the nuances of these applications but also holds the potential significantly improve the overall performance of Al systems.
Temporal reasoning represents vital compo- nent of human communication and understand- ing, yet remains an underexplored area within the context of Large Language Models (LLMs). Despite LLMs demonstrating significant pro- ficiency in range of tasks, comprehensive, large-scale analysis of their temporal reasoning capabilities is missing. Our paper addresses this gap, presenting the first extensive bench- marking of LLMs on temporal reasoning tasks. We critically evaluate 8 different LLMs across 6 datasets using 3 distinct prompting strate- gies. Additionally, we broaden the scope our evaluation by including
analysis 2 Code Generation LMs. Beyond broad benchmarking models and prompts, we also conduct a fine- grained investigation of performance across dif- ferent categories of temporal tasks. We fur- ther analyze the LLMs on varying temporal aspects, offering insights into their proficiency understanding and predicting the continu- ity, sequence, and progression of events over time. Our findings reveal nuanced depiction the capabilities and limitations the models within temporal reasoning, offering compre- hensive reference for future research in this pivotal domain.
Large Language Models (LLMs) (Zhao et al., 2023) have demonstrated remarkable capabilities in a variety of tasks, ranging from commonsense reasoning (Li et al., 2022) to arithmetic problem- solving (Imani et al., 2023). Despite the abun- dance of studies benchmarking LLMs on these tasks, there is a conspicuous absence of compre- hensive, large-scale analysis focusing on bench- marking the models temporal reasoning. Since temporal reasoning represents a crucial aspect of human comprehension, influencing our interpreta- tion and response to a myriad of scenarios, hence the lack of large-scale benchmarking for LLMs on temporal reasoning tasks is a significant gap in our understanding of these models' capabilities. Acknowledging this considerable gap, we have un- dertaken the first extensive benchmarking of LLMs temporal reasoning tasks. (1) Our comprehen- sive analysis encompasses 6 datasets, leveraging different language models. The language mod- els have been tested through different prompting strategies, aiming to explore the breadth and depth of their temporal reasoning proficiency. Moreover,
1 Introduction
Temporal reasoning (Allen, 1983; Wenzel and Ja- towt, 2023) stands as a fundamental pillar of hu- man communication and understanding, acting as a guiding force our interpretation of events and narratives. Comprehending natural language in- volves a deep understanding of time and its facets, which include the duration, sequence, and fre- quency of events. Within the realm of Natural Language Understanding (NLU), the ability to rea- with temporal information has emerged significant area of research. This exploration has witnessed substantial strides in recent years, with numerous researchers contributing to this growing body of knowledge (Zhou et al., 2019; Qin et al.,

---

we have also included 2 Code Generation LMs, further broadening the spectrum of our analysis. (2) In addition to our broad benchmarking efforts, we have conducted a fine-grained analysis of the performance of these models across different cate- gories of temporal reasoning including estimation of event duration, event order, event frequency, sta- tionarity of events and typical time of events. (3) We further analyze the LLMs on varying tempo- ral aspects, offering insight into their proficiency in understanding and predicting the continuity, se- quenee, and progression events over time. In particular, we investigate the following re- search questions: What is the general performance of LLMs in Temporal Commonsense Reasoning?
2021). The Temporal Knowledge Graph Comple- tion (TKGC) domain has been exploring temporal reasoning within knowledge graphs (Dhingra et al., 2022; Jang et al., 2023). Overall, contemporary research has exhibited a notable expansion in tem- poral reasoning studies in natural language under- standing (Wenzel and Jatowt, 2023).
Benchmarking LLMs: The proficiency of LLMs has been notably illustrated across various tasks, yet their exact potential and constraints remain somewhat ambiguous. Recent studies have made strides in scrutinizing the performance of LLMs in diverse scenarios and tasks. For instance, Asai et al. (2023) and Ahuja et al. (2023) have con- ducted extensive benchmarking of three LLMs on cross-lingual and multilingual tasks, respectively. In addition, Wadhwa et al. (2023) performed assessment of two LLMs' capabilities on relation extraction tasks. Yang et al. (2023) carried out benchmarking of ChatGPT in the context men- tal health issues. Furthermore, Nay et al. (2023), conducted comparative analyses of ChatGPT and GPT-44 regarding their performances on legal tax problems. In essence, the latest research show- cases an escalating trend in probing the potential applications of LLMs across a variety of domains, languages, and tasks.
Are the models proficient across all different temporal tasks?
Which temporal commonsense tasks present the greatest challenges?
Does the ambiguity in temporal expressions affect model performance?
How do models perform when they need to rea- son about long time frames, multiple events, or over past and future events?
2 Related Works
In conclusion, despite comprehensive research on LLMs benchmarking in diverse contexts, their proficiency in a temporal common sense remains largely unexplored. This area reveals a necessity for a systematic evaluation of LLMs' understand- ing and reasoning within the temporal domain.
Temporal Reasoning and Understanding: Re- cent years have witnessed a significant rise in in- terest in evaluating models' temporal understand- ing, Key contributions this field have been made through the introduction of datasets explicitly de- signed to assess and improve the temporal under- standing of models. Recent work by Thukral et al. (2021) and Hosokawa et al. (2023) have created Natural Language Inference (NLI) datasets to as- sess pretrained models' understanding of typical common-sent sense temporal expressions, encompass- ing concepts such as containment and verification of the state of events. To probe models' common sense, researchers formulated TimeDial (Qin et al., 2021) and MC-TACO (Zhou et al., 2019), which contain a diverse array of situations and tempo- ral expressions. The recent past has also seen the proposal of several QA datasets that are sensitive to time (Chen et al., 2021). Furthermore, the re- cent research in temporal reasoning has focused on developing time-aware training and representa- tion strategies for language models (Wang et al., 2023; Cole et al., 2023; Kimura et al., 2021; Zhou et al., 2020; Kimura et al., 2022; Saxena et al.,
Benchmark Setup
In the following sections, we provide the details of the datasets, tasks, prompting techniques, and language models used in our research study. The datasets and tasks primarily pertain to temporal reasoning tasks, requiring the models to display an understanding and reasoning in time-sensitive contexts and situations. In terms of language mod- els, we examine diverse set of models pretrained with different strategies, including both standard and Code Generation LMs.
3.1 Datasets and Tasks
We have employed the following datascts that are related to temporal reasoning tasks:
MC-TACO (Zhou et al., 2019): Given context, a question, and candidate response, the objective is to determine whether the candidate answer is "yes" (plausible) or "no" (implausible). The dataset fo-

---

Dataset
Task Deseription
Output
Evaluation Metrie
Temporal Reasoning
MC-TACO
Binary Classification
Yes/No
Acc and weighted F1
ED, EO. F.
TimeDial
Binary Classification
Yes/No
Acc and weighted F1
TT
TNLJ
Natural Language Inference
Binary Clussific
SupporthnwalidtteNennnn
Acc and weighted Fl
S
Wikillow
Classification
Yes/No
Acc and weighted 1-1
E0
BIG-bench
Multi-Cluss Classification
Correct Option number
Acc and weighted F-1
EO
TimeQA
Question Answering
Answer string
EM and F1
TT
Table 1: Datasets Summary (ED: Event Duration, EO: Event Ordering, F: Frequency, S: Stationarity, TT: Typical
Time, Ace: Accuracy, EM: Exact Match)
cuses on assessing the plausibility of the answer within the temporal context provided.
Table summarizes the datasets gives information the types of their temporal commonsense reasoning (ef. the last column), and the characteristics of their tasks, the format of the output, and the evaluation metrics applied.
and
TimeDial (Qin al., 2021): Dataset of multiple- choice cloze task featuring over 1.1K carefully curated dialogues. The dialogues require an un- derstanding temporal commonsense concepts interwoven with the presented events.
3.2 Prompting Techniques In the context of our research, undertake comprehensive examination of the following in-context learning methods across various models: Zero-shot Prompting: Zero-shot prompting is the most basic form of prompting. It is simply showing the model prompt without examples and asking it to generate response. The zero-shot prompt can be represented as P = Sprompt (TD)) D; press) where TD corresponds to the task description, Ptest refers to the test example, and fprompt is function transforming the data into natural language prompt.
TNLI (Hosokawa et al., 2023): Dataset for novel task known as Temporal Natural Language Infer- ence (TNL/). In this task, the model has to ascertain the validity of textual content by using additional associated content as corroborating evidence.
WikiHow (Zhang et al., 2020): Given a goal and a number of steps, a system has to determine the steps are in the correct temporal order.
BIG-bench (Srivastava et al., 2023): Provided with sequence of finished events, each with its defined timeframe, the model needs to determine when individual might have been available for an scheduled activity. While both BIG-bench and Wik- iHow encompass various other reasoning tasks, we specifically focused only on temporal reasoning subtasks.
This technique involves presenting the model with two or more instances, known as few-shot prompting. For each label in the dataset, examples are selected randomly; in our case, single example is chosen for each label. Few-shot prompt be represented P Iprompe (Mi, Mi) Ftest) where (24.8)) symbolizes randomly picked sample from the dataset, and n denotes the number of examples Chain-of-Thouuhh (CoT) Prompting (Wei et al., 2023): This is a recently introduced prompting technique that facilitates the LLMs to elucidate their thought process. The core idea of CoT lies in presenting the LLM with few-shot examples that incorporate explanations of the reasoning process. The CoT prompt
Few-shot Prompting:
TimeQA (Chen et al., 2021): This dataset com- prises a series of time-sensitive question-answe answer pairs. Answering these questions involves under- standing and reasoning within longer context that requires temporal comprehension.
The above datasets (refer to Appendix B.4 for examples of each dataset) cover most of the tem- poral commonsense reasoning styles according to the categorization proposed by Zhou al. (2019) (refer to Appendix B.I for detailed description and examples of each task):
be represented as P fprompt (T (It y/s Ri)n; Itest) where R; stands for the rationale associated with each few-shot example (I,,
Event Duration (ED): reasoning about event du- rations.
DTD;
Event Ordering (EO): reasoning about the typi- cal sequence of events.
Code Prompts (Zhang et al., 2023): The tech- nique of using code-like structures (for example, Python) to prompt Code Generation LM for natural language tasks has been found to enhance performance. The code prompt can be represented
Frequency (F): reasoning about the frequeney of event occurrences.
Stationarity (S): reasoning about the length of state persistence.
Typical Time (TT): reasoning about the specific timing of events.
Please note that we didn't 't perform Few-shot on TimeQA datasct hccause of the context length limit of L.Mss

---

Language Model
Params
Architecture
Type
Few-Shot Zero-Shot
Col
Code-Prompt
GPT-1
6B
Autoregressive Decoder only
Base
GPT Neo
1.3B
Autoregressive Decoder only
Base
LLaMA
7B
Autoregressive Decoder only
Base
OPT
350M
Autoregressive Decoder only
Base
BLOOMZ
560M
Autoregressive Decoder only
SIFT
Dolly
3B
Autoregressive Decoder only
SIFT
FLAN-T5
780M
Encoder- Decrreer
SIFT
SantaCoder
1.18
Autoregressive Decoder only
Basc
CodeGen2
2B
Encoder-Decoder Decoder
Base
GPT-3.5
RLHF
Table 2: Characteristics of Different LLMs employed this study. Basc denotes standard pre-training strategics,
SIFT means Supervised Instruction Fine Tuning and RLHF means Reinforcement Learning from Human Feedback
as P- fende (TD; Frest) where fuxte denotes the function that translates a natural language prompt into code representation, wherein instructions and input samples are given as variables with relevant and meaningful names, enriched by comments that describe their purpose and provide an overarching task description. We refer readers to Appendix B.3 for the exact prompts used. Note that scope of the paper does not include many possible advances of the aforementioned prompt like dynamic prompting (Liu et al., 2021), Auto-CoT (Zhang et al., 2022b) which can be explored in future research.
different LLMs in various temporal tasks. Finally, we turn our attention to analyzing different tem- poral characteristics of the best- rest-performing LLMs from our previous assessments (Sec. 4.3).
4.1 Model and Prompt-based Analysis
Strong performance of GPT-3.5 3.5 and FLAN-T5.
Table 3 illustrates the superior performance of GPT-3.5, especially in few-shot and zero-shot learning tasks in datasets such as MC-TACO and TNLI, showcasing its good generalization ability and vast intrinsic knowledge. This is further im- proved by the CoT prompting strategy and the RLHF training strategy, highlighting GPT-3.5% robust in-context learning. On the other hand, FLAN-T5, despite being older model when com- pared to the other instruction-tuned models, deliv- ers strong performance closely following GPT-3.5 in few-shot learning tasks and even surpasses zero-shot learning on the TimeQA dataset. This can be attributed to the inherent strength of base T5 models which have been trained with 1 trillion to- kens and leverage the extensive C4 dataset (Raffel et al., 2019). BLOOMZ also performs better than the other instruction-tuned decoder model(Dolly) was trained a cross-lingual mixture of tasks (xP3)) spanning dozens of languages. This exposes the model to far higher diversity during pretrain- ing compared to Dolly, which was trained only on English data from a single company's employees. Previous studies have also shown (Tanwar et al., 2023) that multilingual LLMss are better at instruc- tion following and in in-context learning. LLaMA per- forms better than other base autoregressive models as LLaMa has been trained on much more larger and diverse dataset as well as training objectives, com- pared to other base autoregressive models. More- over, the influence of the CoT prompting strategy
3.3 Language Models In this work, we evaluate a set of diverse mod- cls pre-trained with different strategies (cf. Table 2): (1) Models for In-Context Learning: We experiment with set of diverse models; Large au- toregressive models- GPT-J,2 GPT Neo, LLaMA (Touvron et al., 2023), and OPT (Zhang et al., 2022a), Supervised Instruction Finetuned mod- els - FLAN-T5 (Chung et al., 2022), BLOOMZ (Muen- nighoff et al., 2022), Dolly,* and RLHF model G.T-3.5. (2) Code Generation LMs: Addition- ally, we explore the use of Code Generation LM gauge their effectiveness in handling temporal reasoning tasks. For this purpose, we have utilized the following models: (a) SantaCoder (Allal et al., 2023) and (b) CodeGen2 (Nijkamp et al., 2023).
4 Results
In the following subsection, we initiate our discus- sion by providing a succinct overview of the per- formance of various LLMs, code generation LMs, and diverse prompting techniques across multiple datasets. We then advance to more in-depth explo- ration in Sec. 4.2, scrutinizing the performance of
http://ggttuuu..oo//nnngooooo/ mesh- jaajrrrrrrforrrjja
httpps:/huggingtt aot.a//./aa.baccke..... dolly-v2-3b
"https://huggingfacee http://iiinaaacccc/aaaaaaaeeaeee cot/caaasstt///iiccceeeeee
xP3

---

MC-TACO
TNLJ
TimeDinl
WikiHow
BIG-bench
TimeQA
Few-shot
Zero-shot
Few-shot
Zero-shot
Few-shot
Zero-shot
Fewsshot
Zero-shot
Few-shot
Zero-shot
Zeru-shot
GPT-3.5
0.8
(1.5%)
0.62
0.5 (112%)
0.65
11,8 (-0%)
0.55
(1.6%)
0.25
0.26 (113))
D.1
FLAN-T5
0.7
(144))
0.39
0.37 (125%)
0.54
0.54 (=0R)
0.45
0.45 (=0%)
0.16
0.34 (U22))
BLO0MZ
0.59
0.63
0.34
0.31 (1.3%)
0.49
0.46 (J.3%)
0.53
0.46 (1.%%)
0.28
0.28 (-066)
Dolly
0.45
0.52
(178)
0.42
0.32 (310%)
0.5
0.49 ((18)
0.45
(UT9)
0.23
0.27 (1446)
GPT-J
0.59
0.37
0.34
(115)
0.45
(J13))))
0.46
0.45 (U.1%)
GPT Neo
0.59
0.37 (U22%)
0.32
(119))
0.49
0,48 ((11))
0.47
(U,9))
0.26
0.27
(1118)
LLaMA
0.65
0.65 (-00)
0.32
0.32 (--0)
0.5
0.35
(J155%)
0.58
0.54
(U1%)
0.28
0JJ8 (006)
0.18
0.1
0.65
0.45 (1,009)
0.33
(125))
(J11))
0.46
(139))
(U1%)
Table 3: Performance of eight Large Language Models on six datasets, analyzed under two different prompting
strategies. "Ace" stands for Accuracy and "EM" corresponds to Exact Match. The percentage changes in accuracy
performance between Zero-shot and Few-shot prompting are indicated in parentheses. (- (-uggesss the data instances exceed the LLM's context limit; hence results cannot determined.)
JNIJ
MEC-TACO
TimeDial
BiG-dench
ZimeçAA
Midel
Aec
Aec
Avc
Ace
Aec
EM
Model
Aic
Acc
Acc
Auc
Auc
EM
082 (p25)
0.46 (3163)
(p-9)
0.54 1319)
0A77 (1428)
ILI8
GPT-3.5
0.49
(U18))
0.5
0:48
Santaloder
03
0.36
03
0.45
0.27
BLOOKE
0.45 (114%)
(726)
0.53
(195)
Ondaeen?
0.61
0.35
0.5
0.45
0.25
0.F
Delly
((36))3))
(L34
(.5
0.47
(12%)
0.29 (166)
Table 5: Performance of Code Generation LMs with
Table 4: Performance of Instruction Tuned LLMs with
Code prompts in Zero-shot setting. ( (uuggests the data instances exceed the LLM''s context limit; hence results cannot be determined.)
CoT prompting strategy. The percentage changes in ac-
curacy performance between CoT and Few-shot prompt- ing (from Table 3) are indicated in parentheses.
on all datasets and tasks as shown in Table Our experiments align with the findings of Zhang et al. (2023), who demonstrate that code prompts do not surpass the performance of text prompts. Appendix B.5 provides detailed results with FI scores.
(Table 4) over different models varies; it signifi- cantly improves complex temporal reasoning tasks like BIG-bench, while the enhancement in others, such as WikiHow, is similar across other settings. However, TNLI is characterized by inconsistent performance under the CoT setup.
4.2 Temporal Task-based Analysis
Strong performance of LLMs on event fre- quency, and duration tasks.
Code Generation LMs are not temporal com- monsense reasoners.
The heatmap illustrated in Figure 1 provides a visualization of the performance of various LLMs across distinct prompting settings when applied to the MC-TACO datasets and their respective fine- grained temporal task categories (ED, EO, F, S, TT), Models predominantly perform well on tasks asso- ciated with event duration, with GPT-3. 5 taking the lead in accuracy and Fl scores, followed closely by FLAN-T5. Other models like Dolly, GPT-J, GPT Neo, LLaMA, and OPT demonstrate mixed re- sults. However, BLOOMZ's zero-shot capabilities align well with FLAN-T5 and GPT-3. 5. On tasks re- lated to event frequency, GPT-3. maintains strong performance, while FLAN-T5 experiences slight drop but still presents impressive performance.
Table 5 shows a performance analysis of vari- ous models across different tasks, specifically fo- cusing on code generation language models and their ability to reason with temporal commonsense. Previous studies (Madaan et al., 2022) have high- lighted the superiority of Code Generation LMs in reasoning and commonsense tasks compared to general-purpose LMs. However, upon reviewing the results in Table 5, it becomes evident that the code generation LMs, SantaCoder and CodeGen2, struggle as temporal commonsense reasoners. They encounter difficulties across multiple datasets, in- cluding MC-TACO, TNLI, TimeDial, and WikiHow, where understanding and reasoming about tempo- ral aspects are crucial. Similarly, GPT-3.5 with code prompts also exhibits limited performance in temporal commonsense reasoning, reflected by its relatively lower scores in tasks like TimeDial and BIG-bench. Although GPT-3.5 outperforms the other code generation LMs, its performance still falls short compared to normal text prompts
Mixed performance on event ordering tasks.
Performance varies across models on the 'Event Ordering' task in the MC-TACO dataset, with GPT-3.5 and FLAN-T5 leading and others like LLaMA and OPT showing declines, indicating chal-

---

lenges with event ordering (Figure 1). However, comparison with other datasets like WikiHow and BIG-bench reveals lower performance, likely due to the increased complexity of these tasks. The improvement in results when models are combined with the CoT prompting technique suggests that more complex event-ordering tasks demand greater reasoning abilities from LLMs.
4.3 Temporal Aspect-based Analysis
Reasoning about future events is more diffi- cult than about past events.
We used the TimeDial dataset and identified 'Past' and 'Future' events based verb tenses using SpaCy toolkir Manual verification ensured the accuracy post automatic classification. We re- tained 200 instances each for both categories enable an effective performance comparison (an ex- ample of such an instance is shown in Figure 2(a)). Figure 3(a) highlights a consistent trend among models, indicating a slight drop in performance when reasoning about the Future events compared to the Past events. All models show trend of higher accuracy in reasoning about the past com- pared to the future. This disparity may be attributed to the model's extensive training on past events or scenarios, which is more likely to be found in the training data, providing it with a richer database to draw from when making predictions about the past. Joho et al. (2015, 2013) also showed that users struggle more with finding information about the future than one about the past when using search engines and that search for future content is rela- tively common. In summary, while these models are adept at temporal reasoning, their performance slightly drops when dealing with future events.
Performance drop on understanding event temporal states.
There is notable decrease in performance across all models in the 'Stationarity" task, high- lighting the difficulty in assessing the temporal states of events, such as whether events or situa- tions remain constant over time (Figure 1). Remark- ably, BLOOMZ demonstrates strong performance in the zero-shot configuration, almost on par with GPT-3.5, 5, suggesting that it has a specific strength in identifying event stationarity. Furthermore, re- ferring to Table it is evident that even when using the CoT prompting strategy, Language Mod- els struggle with other tasks related to Stationarity, such as ones in TNL/ dataset (TNLJ dataset requires understanding the states of events). This points to- wards an inherent challenge faced by these models in grasping and reasoning over concepts of the tem- poral stability of events.
LLMs perform better on temporal reasoning over longer timeframes.
LLMs struggle with specific event timings.
The "Typical Time' task stands out the most demanding for all the models, emphasizing the intricate nature of predicting and reasoning over typical event timings (Figure 1). While GPT-3.5 5 continues to perform best in this category, its lead over other models is significantly reduced com- pared to other tasks. The performance gap between GPT-3.5 and other models such as FLAN-T5 and BLOOMZ is notably narrower in the zero-shot con- figuration, indicating less dominance by GPT-3.5 on this task. A substantial performance decline is also observed other models, further underlin- ing the task's complexity. A reference to Table 3 reinforces this conclusion of LLMs struggling with exact timings, as it reveals that all Language Models struggle not only with the 'Typical Time' task on MC-TACO, but also with other time-related tasks, such TimeQA. This indicates broader challenge for L.LMs in reasoning over specific time periods. Readers can refer to Appendix B.7 for both Fl and accuracy results.
To construct the dataset for this experiment, we scrutinized the TimeDial dataset. Each instance was classified based on the duration specified by the correct label associated with its context. This may lead to the creation of categories: seconds, minutes, hours, day, and durations exceeding a day. After manual verification for accuracy, we sampled 200 instances from each category (Fig- ure 2(b)). Figure 3(b) offers a comparative analy- sis of models' performance when reasoning about events that occur over varying time frames. The emerging pattern highlights that the models tend to perform better when dealing with longer time frames compared to shorter ones. GPT-3.5 pro- vides the clearest representation of this trend, start- ing with a relatively lower accuracy for events that transpire over seconds. However, its accuracy in- creases as the duration of events extends to minutes, hours, and days. Impressively, GPT-3.5 3.5 reaches its
'https://spacy.tt

---

OP 43
331
6:41
1:44
0.77
Ei
1:17
6
EdB
0.02
0200 0000 L002 0702
IRCOM
2.66
1:41
0.33
6.4
A11
H00
8.43
5.
9.77
6.1
0.1
DB
8.45
1:35
6.75
0.74
5.18
0.6
MAMA
0:3
0.5
8.33
0.23
3.45
8:30
8/6
9.45
143
197
3.1
575
6.45
9.47
0.33
145
0.81
OFT
6,32
off
8.37
6.21
6.1
PE40
M64
9/11
16.
PE4D
MC40
99/
HC41
MC42
HEAD
PC4
HCll
(a) Heatmap for Accuracy of LLMs in (b) Heatmap for Accuracy of LLMs (c) Heatmap for Accuracy of Few-shot setting
Zero-shot setting
LLMs in CoT setting
Figure 1: Performance of LLMs on MC-TACO dataset and it's fine- tieegrained temporal task categories on Few-shot settings. Y-axis represents different models and X-axis represents different temporal tasks (MC: MC-TACO)
Dataset
MC-TACO
TNLI
TimeDial
WikiHow
BIG-bench
TimeQA
Acc
Acc
Acc
Acc
Acc
EM
Human Baselines
0.75
0.82
0.97
0.97
1.0
0.9
Baseline Fine-Tuned Models
0.64
0.878
0.748
0.801
0.55
Table 6: Performance comparison of Fine-Tuned Models and Human Baselines across Various Datasets. indicates that the BIG-bench dataset does not have a training set, so no model can be fine-tuned.
Houu vvname aasa had -1AAB0- ooca hince nacking
Answer:
Label:
lengths. In Figure 3(c), we observe that the con- text length increases the performance of the models tends to decrease (with the exception of FLAN-T5). GPT-3.5 5 shows decline in performance as the context length increases from 0-200 to 400-600, indicating a possible difficulty in handling long contextual information.
Past
in office Stanghel may'
Yeor
Yes
Reasoning
lorkad fiiggn
your ago
Future
Whatls this cvidt nolice for?
Answer: Label:
Reasoning
naßce you receled a
notice
30 Days
Yes
You can't throw me
street
You bome 30 dlase au caich on your rock, ofal will yeu
Ecample of past reasoring va future reasaning
Affow are the hitnn coing vi rippee"
Label:
TimeeFrame
Second
Bill seypleses with the parbemancaa Teeny
100 neters HAMS-- That's
12 sec
Yes
More than
A-ow much doos pool his nolorttu
Answer:
Label:
Day TimeFrama B. 4M 300, for days 800
Days
Yes
LLMs struggle with exact temporal expres- sions compared to ambiguous ones.
[|]
day
Figure 2: Dataset example for temporal aspect analysis
We created specialized dataset from MC-TACO leveraging the combination of HeidelTime (Ströt- gen and Gertz, 2010) and manual extraction. This allowed us distinguish between instances con- taining 'Exact Timings', such as specific numerical expressions of time, day names, and month names, and 'Ambiguous Temporal Expressions', like in the meantime, after few days, and meanwhile (exam- ples shown in Figure 4(a)). We then performed a manual verification of these instances, maintaining balanced collection of 200 instances each from the categories of exact and ambiguous temporal expressions. The comparative performance analy- sis (Figure 5(a)) across diverse temporal reasoning tasks highlights a persistent challenge the models face when dealing with 'Exact Timings' such as numerical values, day names, or month names, opposed to 'ambiguous temporal expressions'. The models may encounter difficulties in associating specific timings with their implications for the oc- currence or sequence of events, indicating poten- tial challenge in comprehending and connecting exact temporal information. It underscores key
peak performance when reasoning about events that last more than a day. This upward trajectory suggests that GPT-3.5 5 is more adept handling the intricacies involved in reasoning about longer- time frame events. Similarly, FLAN-T5 shows an improvement in performance when moving from shorter to longer time frames, although its trajec- tory is not that consistent. LLaMA, despite some variation, also seems to perform better with longer- duration events. These findings could suggest that models are better equipped to handle the complexi- ties and nuances involved in reasoning about longer tineframe events.
LLMs have difficulty with temporal reason- ing over longer context.
We divided the TimeDial dataset into three cate- gories based on context length: 0-200 words, 200- 400 words, and 400-600 words. We then selected 200 instances randomly from each category, ef- fectively creating a dataset with diverse context

---

0206
233-900
54
01
Performance of I.I. across Past Performance of LL across different timeframes
Performance of L.Ms dif- ferent context length
and Future events
Figure 3: Bar Plots indicating the performance of LLMs across different temporal aspects in few-shot setting. X-axis represents the different L.Mss, and Y-axis represents performance accuracy
Ambiguous
Contect nonccwatted
Questioo: Now ong Scheodinger
understanding the persistence of states and the typ- ical sequence of events, can become more complex and challenging when multiple events involved. In the 'Typical Time' task, observe that all mod- els generally perform better or at least equally well when reasoning about single events, perhaps indica- tive of their difficulties when attempting to com- prehend specific timings associated with multiple events.
Exprossion
Temporal
heavcy crlccaed 1317 Erwin
Schnuciigee others.
odjects somubelersore
poeusikenns
oppass
Exact
socurity
How King
Arnove:
Label:
Temporal
sccurity
Ves
Expression
rg agendos.
sorup
(0) Dataset Example Exact Artiiguous TE
Single Event
Contrnt: Beedin arm gining
Quesitie:: tine Sky was pross confirenne
Lobel:
pross conforince about toir
You
new film Sing cal Mystory Tour
Muttiple Event
Contoct
Quaritioon
1902, and mother ded 1513.
Danerr
do Dateset Example Single Event Muliiplo Evant
Figure 4: Dataset examples for temporal aspect analysis
5 Comparison with Fine- Tuned models and Human Performance
area of struggle for these models effective rea- soning over specific time periods across different tasks. This may be attributed to the higher preva- lence of ambiguous events as compared to precise ones, largely because they are utilized more fre- quently in a variety scenarios. Readers can refer to Appendix B.6 for detailed results of this section.
Human Evaluation: Human evaluations were car- ried out using 100 random samples from the TNLI, BIG-bench, and TimeQA datasets, assessed by three in-house annotators. For other datasets, relied human baselines provided in the original pa- pers. Annotators' accuracy was gauged by compar- ing their responses the ground truth, establish- ing human benchmarks for performance (Table 6). For TimeDial, WikiHow, BIG-bench, and TimeQA tasks, human proficiency substantially outperforms all LLMs. For example, humans scored 0.97 on TimeDial, whereas the best LLM, GPT-3. only achieved 0.65, The gap is further widened in tasks like WikiHow, where humans scored 0.975. In the TNLI dataset, even the top-performing LLM. GPT-3.5, 5, falls short of the human baseline score (0.82 vs. 0.62), indicating lingering challenges for LLMs in this task. In contrast, GPT-3.5 matches human performance on the MC-TACO dataset in a few-shot setting, both scoring 0.8, suggesting that under certain conditions, latest LLMSs can achieve human-level capabilities.
L.MSs struggle with understanding the states and orders of multiple events.
We manually dissected the MC-TACO dataset due to the inherent complexity of accurately iden- tifying single and multiple temporal events within instances. We systematically curated two groups from each category: instances with single and mul- tiple temporal events. To maintain uniformity, we selected 200 instances from each group. as shown in Figure 4(b). Figure 5(b) compares model per- formance across tasks for single versus multiple temporal events. In the context of 'Event Duration' and 'Frequency', all models perform better when reasoning about multiple events compared to single events. On the other hand, the 'Stationarity" and 'Event Ordering' tasks display a different trend, where all models perform better when reasoning about single events as opposed to multiple events. This could indicate that these tasks, which require
Fine Tuned Model Baselines: Table 6 also in- cludes evaluation results for the best-performing fine-tuned models from the papers introducing each dataset. Key findings are as follows: In the MC- TACO dataset, the baseline model scored 0.64.

---

GF33.3
FLANTS
LLMA
ED
B
(a) L.LMs' performance when dealing with exact and amhiguous tem- poral expressions across temporal tasks (TE: Temporal Expression).
6P14.
FLANTS
LLAMA
ED
Mucppl
(b) Performance of LLMs when processing both single-event and multiple-event scenarios across temporal tasks.
Figure Radar plots comparing accuracy of LLMs across different temporal aspects in few-shot setting
However, LLMs like GPT-3. and F5 signif- icantly outperformed it in both few-shot and zero- shot scenarios. For TNLI, the fine-tuned baseline model achieved 0.878, outperforming all LLMs. Notably, GPT-3. the top LLM, scored only 0.62 in few-shot settings. This is because the baseline was fine-tuned on TNL/ data and leveraged exter- nal commonsense knowledge. On TimeDial and Wikillow, the baseline models scored 0.748 and 0,801, respectively. GPT-3. 5 did not surpass the baseline. The fine-tuned mod- els excel due task-specific optimizations. For TimeQA, with baseline score of 0.55, FLAN-T5 was the closest among LLMs with an EM score of 0.4. The baseline's higher performance is attributed to its use Retrieval Augmented Generation (FiD), allowing it to handle the dataset's long context.
challenges, our study contributes to a better un- derstanding of the capabilities and limitations of LLMs in temporal reasoning tasks.
Acknowledgement: Dr. Sripamn Saha gratefully acknowledges the Young Faculty Research Fellow- ship (YFRF) Award, supported by Visvesvaraya Ph.D. Scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Dig- ital India Corporation (formerly Media Lab Asia) for carrying out this research.
led among LLMs but
Limitations
There are several limitations of our study that should be acknowledged. Firstly, in terms of prompt selection, our testing was limited few prompting strategies, and there exist numer- ous other techniques and variations that could be explored. Therefore, the generalizability of our findings to different prompt settings may be con- strained. Secondly, our evaluation of models was primarily focused on open-source models and cov- ered only one closed model (GPT-3.5), We did not include closed models like PALM. In this study, covered only temporal commonsense reason- ing. Yet. we acknowledge that there are various other temporal tasks that were not covered, such timeline summarization and temporal information retrieval.
6 Conclusion
this paper, we aimed to bridge a critical knowl- edge gap by conducting a comprehensive bench- marking of LLMs on temporal reasoning tasks. Our thorough analysis has shed light on certain limitations in the ability of LLMs to reason tempo- rally. Specifically, we have identified areas where LLMs struggle, such comprehending the tempo- ral states of events, accurately reasoning over pre- cise timings, managing multiple temporal events, and predicting future events. By highlighting these

---

References
Hideo Joho, Adam Jatowt, and Roi Blanco. 2015. Tem- poral information searching behaviour and strategies. Information Processing Management, (6):8834 850.
Kabir Ahuja, Harshita Diddee, Rishav Hada, Milli- cent Ochieng. Krithika Ramesh, Prachi Jain, Ak- shay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. Mega: Multilingual evaluation of generative ai.
Hideo Joho, Adam Jatowt, and Blanco Roi. 2013. A sur- vey of temporal web search experience. In Proceed- ings of the 22nd International Conference on World Wide Web, WWW "13 Companion, page 1101 108, New York, NY, USA. Association for Computing Machinery.
Loubna Ben Allal, Raymond Li. Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, al. 2023. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.0399)
Mayuko Kimura, Lis Kanashiro Pereira, and Ichiro Kobayashi. 2021. Towards language model for temporal commonsense reasoning. Proceedings of the Student Research Workshop Associated with RANLP 2021, pages 78 78-84, Online. INCOMA Ltd.
James F. Allen. 1983. Maintaining knowledge about temporal intervals. Commun, ACM, 26(1 1):82-883.
Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannanch Hajishirzi. 2023. Buffet: Benchmarking large language models for few-shot cross-lingual transfer.
Mayuko Kimura. Lis Kanashiro Pereira, and Ichiro Kobayashi. 2022. Toward building language model for understanding temporal commonsense. In Pro- ceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference Natural Language Processing: Student Research Workshop, pages 17-24.
Wenhu Chen, Xinyi Wang, and William Yang Wang- 2021. A dataset for answering time-sensitive ques- tions.
Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoff- mann, Cyprien de Masson d' Autume, Phil Blunsom, and Aida Nematzadch. 2022. A systematic investiga- tion of commonsense knowledge in large language models.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling mnstruction-iietuneed language mod- els.
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint urXiv:2101.066:
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang. and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. arXiv preprint arXiv:2210.07128.
Jeremy R. Cole, Aditi Chaudhary, Bhuwan Dhingra, and Partha Talukdar, 2023. Salient span masking for temporal understanding.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Fisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-Aware Language Models as Temporal Knowledge Bases. Transac- tions of the Association for Computational Linguis-
Niklas Muennighoff. Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint arXiv:221 1.0078.6...
tics, 10:257-273.
Taishi Hosokawa. Adam Jatowt, and Kazunari Sugiyama. 2023. Temporal Natural Language Infer- ence: Evidence- -assed Evaluation of Temporal Text Validity, pages 441-458
John Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi, and Jungo Kasai. 2023. Large language models as tax attorneys: A case study in legal capabilities emergence.
Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05339
Joel Jang. Seonghyeon Ye. Changho Lee, Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Sco. 2023. Temporalwiki: A lifelong bench- mark for training and evaluating ever-evolving lan- guage models.
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Sil- vio Savarese, and Yingbo Zhou. 2023. Codegen2: Lessons for training Ilms on programming and natu- ral languages. arXiv preprint.

---

Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He. Yejin Choi, and Manaal Faruqui. 2021. Time- Dial: Temporal Commonsense Reasoning in Dialog. In Proc. of ACL.
Georg Wenzel and Adam Jatowt. 2023. An overview of temporal commonsense reasoning and acquisition. CoRR, abs/2308.0000)
Xiaona Xia and Wanxue Qi. 2022. Temporal tracking and early warning of multi semantic features of lear- ing behavior. Computers and Education: Artificial Intelligence, 3:100045.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. arXiv e-prints.
Kailai Yang. Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with chatgpt.
Apoory Saxena, Soumen Chakraharti, and Partha Taluk- dar. 2021. Question answering over temporal knowl- edge graphs.
Li Zhang, Liam Dugan, Hainiu Xu, and Chris Callison- Burch. 2023. Exploring the curious case of code prompis.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, and multiple authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models.
Li Zhang, Qing Lyu, and Chris Callison-Buree 2020. Reasoning about goals, steps, and temporal ordering with WikiHow. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 4630-4639, Online. As- sociation for Computational Linguistics.
Julius Steen and Katja Markert. 2019. Abstractive time- line summarization. In Proceedings of the 2nd Work- shop on New Frontiers in Summarization, pages 21- 31. Hong Kong, China. Association for Computa- tional Linguistics.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022a. Opt: Open mr-itrieeed transformer language models.
Jannik Strötgen and Michael Gertz. 2010. HeidelTime: High quality rule-based extraction and normaliza- tion of temporal expressions. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 321-324, Uppsala. Sweden. Association for Computational Linguistics.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompt- ing in large language models. arXiv preprint arX1v:2210.03493.
Eshaan Tanwar, Manish Borthakur, Subhabrata Dutta, and Tanmoy Chakraborty. 2023. Multilingual 1lss are better coos-linguual in in-context learners with align- ment.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong. Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Nie, and Ji-Rong Wen. 2023. A survey of large language models.
Shivin Thukral, Kunal Kukreja, and Christian Kavouras. 2021. Probing language models for understanding of temporal expressions. In Proceedings of the Fourth BlackboxNLP Workshop on Anulyzing and Interpret- ing Neural Networks for NLP, pages 396-406, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "going on a vacation" takes longer than "go- ing for walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363-3369, Hong Kong, China, Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Somin Wadhwa, Silvio Amir, and Byron C. Wallace. 2023. Revisiting relation extraction in the era of large language models.
Ben Zhou, Kyle Richardson, Qiang Ning. Tushar Khot, Ashish Sabharwal, and Dan Roth. 2020. Temporal reasoning on implicit events from distant supervision. arXiv preprint 2010
Jiexin Wang, Adam Jatowt, Masatoshi Yoshikawa, and Yi Cai. 2023. Bitimebert: Extending pre- pre-trained lan- guage representations with bi-temporal information.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.

---

Appendix
our study, Still, in the interest of transparency and further research, we commit to publicly sharing the manually curated data utilized our experiments upon acceptance of our work. We believe that this does not only contribute to the reproducibility of our findings but also fosters research advancements in this field.
Frequently Asked Questions (FAQs)
* What was the rationale behind selecting these specific LLMs?
Our primary objective for utilizing a diverse set of Large Language Models (LLMs) in the analysis of temporal commonsense was to ensure wide representation of different models. This includes those ranging from au- toregressive decoder-only models encoder- decoder models, as well as instruction-tun -uuee models. We selected GPT-3.5 as is among the top-performing LLMs at present. How- ever, given its proprietary nature, we comple- mented it with the newly launched instruction- tuned LLMs such as BLOOMZ and Dolly. A unique aspect of these models is that Chat- GPT was trained with Reinforcement Learn- ing from Human Feedback (RLHF) approach, while BLOOMZ and Dolly used supervised in- struction fine-tuning. We added FLAN-T5 to our roster due to its distinct encoder-decoder architecture, which offers a contrast to other instruction-tuned LLMs. The inclusion of models like LLama, OPT, GPT-J, GPT-Neo served exhibit the capabilities of an autore- gressive decoder-only model. Furthermore. we integrated two code generation Language Models to contrast their performance in tem- poral commonsense reasoning with the gen- eral purpose LLMs.
* What is the rationale behind selecting TACO and TimeDial as the founda- tional datasets for the Temporal-Aspect based analysis section?
Our choice to utilize MC-TACO and Dial as the base datasets for the Temporal-Aspect based analysis section was primarily driven by their significant size and well-established reputation in the research community. MC- TACO provides a vast array of approximately 9,441 data samples, while TimeDial furnishes an additional 5,784 examples. The wealth of data they offer surpasses most other datasets the field. In terms of research queries like analyzing performance with varying context lengths, past versus future reasoning capabil- ities, and performance across different time- frames, TimeDial was particularly suitable choice. Its emphasis on timing- based contexts rendered it a valuable resource. Morcover, for specialized inquíries into ambiguous tem- poral expression reasoning and single versus multiple event reasoning, MC-TACO proved indispensable It presents a broad spectrum of diverse examples catering to these specific areas of interest. Overall, these datasets, by virtuc of their depth, diversity, and relevance, enable robust, comprehensive, and nuanced analyses. They represent reliable sources for generating insights into temporal-aspect based reasoning with large language models.
* Who was responsible for manually validating the datasets used in the Temporal-Aspect based analysis sec- tion?
We implemented heuristic and lagger-based techniques categorize our dataset into dif- ferent classes, and as authors, we also person ally conducted manual verification to elimi- nate any potential misclassification. However, it is important to highlight that these sam- ples were sourced from well-established and trusted datasets in the community: MC-TACO and TimeDial. Given their standardization and wide acceptance in the field, we did not find necessary to compute Intor-Annotator Agree- ment (IAA) scores. We are confident in the integrity and relevance of these datasets for
What was the reason behind choosing only three prompting strategies for anal- ysis?
In this study, focused on utilizing threc widely recognized prompting strategies: Few- shot, Zero-shot, and CoT prompting. These strategies were chosen due to their established prominence in the field and their frequent in- clusion in related analysis and benchmarking

---

works (Wadhwa al., 2023; Nay et al., 2023), By employing these standard techniques, we ensured consistency with existing literature and provided a solid foundation for compar- ison and benchmarking. While we acknowl- edge that there have been recent advancements and emerging techniques beyond the scope of this study, we aim to emphasize that our research serves starting point in explor- ing these methods for temporal commonsense tasks. Our intention is to pave the way for future investigations that delve into the latest developments in prompting strategies, thus enabling a more comprehensive understand- ing of their effectiveness in temporal com- monsense analysis. We remain committed to staying abreast of the evolving landscape of prompting techniques and incorporating them in future endeavors to expand the breadth and depth of our findings.
top_p=1, and top_ k-5, fixed across experiments. as commonly adopted values in prior work.
B.3 Prompts
Prompt samples for all the datasets.
B.3.1 Prompts for MC-TACO
Few-shot Prompt for MC-TACO
Given the passage. the question, the candidate the task to determine whether the candidate answer plausible ('yes") or not ("no").
Passage: Durer's father died in 1502, and his mother died 1513.
B Supplementary Material
Question: How long his mother ill? answer: six centuries
This section provides supplementary material in the form of additional results, implementation details, etc. to bolster the reader's understanding of the concepts presented in this work.
Response:No
###
Given the passage, the question, and the candidate answer, the task is to determine whether the candidate answer plausible ("yes") or not ("no"). Passage: Durer's father died 1502, and his mother died in 1513.
B.1 Dataset Example for Temporal Task based Analysis
Question: How long was his mother Ill?
answer: months
Response:Yes ###
Event Duration (ED): This task necessitates reasoning about event durations.
Given the passage. the question, the candidate answer, the task determine whether the candidate answer plausible ('yes") or not ("no"). Passage: Durer's father died in 1502, and his mother died 1513.
Event Ordering (EO): This task calls for soning about the typical sequence of events.
Question: How long his mother ill?
answer: 3 minutes Response:No
Frequency (F): This task requires reasoning about the frequency of event occurrences.
Given the passage, the question, and the candidate answer, the task is to determine whether the candidate answer plausible ("yes") or not ("no")
Stationarity (S): This task demands reason- ing about the length of state persistance.
Passage: (Passage)
Question: (Question)
Answer: (Answer)
Response:
Typical Time (TT): This task needs reason- ing about the specific timing of events.
Table 7 contains examples for each of these cate- gories.
Zero-shot Prompt MC-TACO
Given the passage, the question, and the candidate answer, the task to determine whether the candidate answer plausible ('yes") ("no"). Passage: (Passage)
B.2 Experimental Setup
Operating System: Ubuntu 18.04.5 LTS
RAM: 220GB
Question: [Question)
Answer: (Answer)
GPU: NVIDIA GeForce RTX 3090 (24 GB) Python: 3.10.11
Return your Response in Yes No only.
Response:
Hyperparameters: We used 1emperature=0.55

---

Context
Question
Answer
Label
Temporal Task
Safti admits his love for Edwina to Lord Esketh, who is now sympathetic toward this good man's plight.
How long has Safti been year in love with Edwina?
yes
Event Duration
Durer's father died in 1502, and his mother died in 1513.
How long was his mother ill?
she was ill for 30 sec- onds
Event Duration
What we call the lab-to- fab time should be close to as possible, Kelly said.
What happened after Kelly spoke?
many people agreed
yes
Event Ordering
Tim knew if the bike was going to in any of presents was go- ing to be in this box.
After found box. what happened?
he tossed away
no
Event Ordering
Most of us have seen How often does it rain the summer?
couple month
times
every yes
Frequency
steam rising off' a wet road after a summer rainstorm.
The organization has al- ready lost some staff through attrition and has turned away some cases, she said.
How often does the or- ganization turn away cases?
always
Frequency
A thwarted Mongol invasion in 1274 weak- ened the Kamakura regime.
Is the invasion still tak- ing place today?
yes
Stationarity
Tony Ally like play other games like hopscotch or jump rope but that day they joined game tag.
Do Tony and Ally still no enjoy jump rope?"
Stationarity
Johnson is a justice on
When did Johnson ar- rive to court?
9:00 AM
yes
Typical Time
California's 's Second Dis-
trict Court of Appeal.
She had it for a long time so is now dark
What time did she buy it?
1:00 AM
no
Typical Time
brown color.
Table 7: Examples from datasets for Temporal Task based Analysis

---

CoT Prompt [or MC-TACO
Code Prompt for MC-TACO
Given passage, question, and candidate answer, the task is determine whether the candidate
import westion__nnswrring
answer is plausible ("yes") not ("no"). Final Label will be yes or no only.
class Context:
"SStrictly return only "ø" or the question, based context and type of answer, task is to determine whether the given candidate answer is plausible ("1*) not ("0").'
Passage: Durer's father died 1502, his mother died in 1513.
given
Question: How long was his mother ill??
Answer: six centuries
Response: Let's think step step. Question is asking about the duration for which Durer's mother was ill and answer mentioned It is six centuries. It is not possible as any human can't for such long period So the answer is not plausible.
def i___i_...sslll......_______ (self, question, ceestion,context. The answer, type): question # The question self.answer = answer # answer self. type type type the answer
self. context
Final Label: No ###
Given the passage, question, and candidate
answer, the task is determine whether the candidate
def get_answer ingeewerinnggeeee (ssl.....ues...on ans cuestion_anseee
answer is plausible ('yes") or not ("no").
Passage: Durer's father died 1502, his mother
colf.context,
died in 1513.
self.answer, self. Lype)
Question: How long was his mother ilI?
Answer: 6 months
return ans #Strictly return 0/1 only
think step by step. Question asking about the duration for which Durer's mother was ill and answer mentioned it was 6 months. This can be possible as duration for illness as many illness lasts for such periods only.
Response:
context Context( context "()", question "()"
Final Label: Yes ###
answer "()"
type "()"
Given passage, question, and candidate
answer, the task is determine whether the candidate answer is plausible ('yes") or not ("no").
Lccnteext. L_a_swee
= formatt (question, context, answer, type)
Passage: Durer's father died in 1502, and his mother
died in 1513.
Question: How long was his mother ill?
Answer: 3 minutes
Response: Let's think step by step. Question asking about the duration for which Durer's mother and
answer mentioned it was 3 minutes which can't be possible as no iliness last for such small time period. Label: ##4
Given the passage, the question, and the candidate answer, the task is to determine whether the candidate answer plausible ("yes") not ("no").
Passage: (Passage)
Question: (Question)
Answer: (Answer)
Response: Let's think step by step.

---

B.3.2 Prompts for TimeDial
CoT Prompt for TimeDial
You are given a corversation between two persons. Each conversation has fill-n-theebbaank them which is represented by <MASK>> token. You are also given a candidate answer for that <MASK> token. Your task determine whether the candidate
is plausible (Yes) or not (No) by first generating first reasoning and then final label as Yes or No.
Conversation:
A:We need take the accounts system offline carry out the upgrade But don't worry it won't cause too much inconvenience We're going to do it over the weekend.
Few-shot Prompt for TimeDial
You are given conversation between two persons. Each conversation has fillinnthhe-blank in them which is represented by a <MASK> token. You are also given a candidate answer for that <MASK> token. Your task is determine whether the candidate plausible (Yes) or not (No). Return your answer in Yes and No only.
B: How long will the system be down for?
A: We'll be taking everything offline in about two hours time
be down for a minimum twelve hours If everything goes according to plan should up again by 6 pm Saturday.
B: That's fine. We've allowed <MASK> to be on the side
Conversation:
A:We need to take the accounts system offline carry out the upgrade. But don't worry, it won't cause too much inconvenience. We're going to do over the weekend.
Answer: forty-eight hours
Label: Let's think step by step. The conversation
seems to be taking place in an office environment speaker says that they will systems down for 12 hrs over the weekend for maintenance. and speaker says they are allowed <mask> hrs without systems and answer here says mask should weekend
B: How long will the system be down for? A: We'll be taking everything offline in about two hours time It'll down
minimum of twelve hours everything goes according plan. should be up again by 6 pm on Saturday.
to correct which means for 48 hrs they don't need to work on these systems.
This
B: That's fine We' ve allowed =MASK> to be on the
safe
Final Label:Yes
Answer: firtt- hours Label:Yes
###
Conversation:
###
A:Mr. Emory, I I'd like to take this afternoon if it's right with you.
Conversation:
A:Mr Emory l'd like to take this afternoon off it's
B: But Steven you've called in sick 3 times during
all right with you.
<MASK>.
B: But Steven you've called sick times during
know Mr. Emory . sorry. But really need to the doctor this afternoon. feel dizzy and can't concentrate on work.
<MASK>.
A: know. Mr Emory I'm sorry.. But really need to see the doctor this afternoon. feel dizzy and can't concentrate my work.
B: All right,, then But don't forget to bring a doctors note tomorrow.
B: All right, then But don't forget bring doctors note tomorrow.
A: OK, thank you
Answer: last 15 seconds
A: OK, thank you
Label: Let's think step by step. The conversation seems be taking place in office environment
Answer: last 15 seconds
Label:No
where speaker A is asking for a leave to speaker B and Speaker said that he already took leaves dur- ing <MASK>
###lt Conversation: (Conversation)
answer here says its 15 seconds which is possible leaves are taken during week a month. That's why this answer is incorrect.
period.
Answer: (Answer)
Label:
Final Label:No
####
Conversation: (Conversation)
Answer: (Answer)
Label: Let's think step by step.
Zero-shot Prompt for TimeDial
You are given conversation between two persons. Each conversation has a fillin-theeblankk in them which represented by <MASK> token. given candidate answer for that <MASK> token. Your task is to determine whether the candidate answer is plausible (Yes) or not (No). Striclly return your answer Yes and No only.
Conversation: (Conversation) Answer: (Answer)
Label:

---

Code Prompt for TimeDial
Zero-shot Prompt for TNLI
inport Fill_blane [ill__lann_____ blank_from cron_conversation
You are given two sentences, Sentence tence where Sentence tence 2 is premise sentence. The task is to assign one of the following three classes to Sentence based the inference using the content of Sentence 2. The labels are Support, Invalidate, and Neutral. The SUP- PORT class means that Sentence is still valid given the information Sentence 2. The INVALIDATE class, the other hand, be valid in view of Sentence 2. The third one, Neutral class, indicates that the situation evidence is not con- clusive clear, and cannot verify the validity of hypothesis.
Sen- hypothesis, and Sen-
class Conversation:
"Conversation between two persons given each conversation
fill-in sillin-thee the-blank in them which is represented by <MASK> taken.
You are also given candidate answer token. Determine whether the is plausible ("1") not ("0*). Strictly return "ø" or "1" only."".
for that
candidate
that Sentence ceased to
Liiiit (ssl conversation, answer): self. conversation conversation # Converstion between two persons self.answer answer answer
Sentence 1: (Sentence 1)
Sentence 2: (Sentence 2)
the fill
blank
Label:
def (st_ansuer(sn1f)):
ans fill__llnn__fee..
self. conversation, anf.ansser)
return ans astrictly return 8/1 only
CoT Prompt for TNLI
conversation Conversation( conversation = "()", "()"
You are given two sentences, Sentence 1, and Sen- tence where Sentence tence is premise sentence. The task assign one of the following three classes Sentence based on the inference using the content of Sentence 2. The are Support, Invalidate, and Neutral. The SUP- PORT class means that Sentence is still valid given the information in Sentence 2. The INVALIDATE class, on other hand, means that Sentence ceased to valid view Sentence The third one, Neutral class, indicates that the situation evidence is not con- clusive or clear, and we cannot verify the validity of the hypothesis.
hypothesis. Sen-
assert (converssa ion, get.nngge.ae...... tnsserr (conversation answer)
B.3.3
Prompt for TNLI
Sentence group of people sing and dance at concert
Sentence 2: A group of people going to take rest. Label: Let's think step by step. In Sentence it mentioned that group people are taking rest. This implies that they won't perform any activity. But in Sen- tence mentioned that group people are singing dancing concert. based on information from Sentence it is not possible as they are taking rest. So the final label should be Invalidate.
Few-shot Prompt for TNLI
You given sentences, Sentence 1, and tence 2 where Sentence 1 is a hypothesis. and Sen- tence 2 is premise sentence. The task is to assign one following three classes to Sentence based the inference using content Sentence The labels are Support, Invalidate, and Neutral. The Sup- port class means that Sentence is still valid given information in Sentence 2. Invalidate class, means that Sentence ceased
Invalddne
###
Sentence 1: female is scrambling eggs in a bowl. Sentence 2: Eggs are scrambled in a bowl.
Label: Let's think step by step. In Sentence mentioned that eggs scrambled in bowl. How- ever, in Sentence a female is scrambling eggs in a bowl which supports the statement Sentence that eggs are scrambled. the final label should be Support.
the other be valid in view of Sentence 2. The third one, Neutral class, indicates that situation evidence not con- clusive clear, hypothesis.
cannot verify validity the
Sentence 1: female is scrambling eggs in bowl. Sentence Eggs are scrambled bowl. Label: Support
Sappl:
The horses race the dirt track while their riders urge them on.
Sentence 1:
###
Sentence 1: A group of people sing and dance concert.
Sentence 2: Most people enjoy watching horse racing. Label: Let's think step step. In Sentence 2,it mentioned that Most people enjoy watching horse ing. However, in Sentence 1, The horses race on the dirt track while their riders urge them on. Both these statements neither supporting each other in- validating each other. So the final label should be Neutral.
Sentence group of people going to take rest.
Label: Invalidate
###
The horses their riders urge them on.
Sentence 1:
the dirt track while
Sentence 2: Most people enjoy watching horse racing.
Label: Neutral
Label:Neutral
###
#####
Sentence 1: (Sentence 1)
Sentence 1: (Sentence 1)
Sentence 2: (Sentence 2)
Sentence 2: (Sentence 2)
Label:
Label: Let's think step step.

---

Code Prompt for TNLI
CoT Prompt for Wikillow
neuralnli
You are given goal and steps to accomplish that goal. Your task to determine whether the steps in right order (Yes) or not (No). Return your answer in Yes and No only.
class (aturralLnnuaggnnneeeeee)))
1*1 function to answer the natural language and hypothesis.
inferrence task
Goal:How Select Dog Bed Understanding Differ- ent Types of Beds
def _init _selit_sselff) model neuralnli()
Steps:Buy mat for the easiest solution. Pick a pillow bed large dog. Select a donut if your dog likes feel secure. Purchase nest bed for cuddling comfort. Buy bolster lype bed if your dog is a leaner. Look for cave-style bed your dog likes to burrow. Consider hammock
def Forward(self, premise, hypothesis): answer (premis.
hypothesis){ annswr'
for ease of cleaning.
return answer
Answer: Let's think step by step. The correct order should be: Look for cave-style bed if your dog likes burrow, Select a donut bed your dog likes feel secure. Purchase
nli_model NeturallanguaglInffreeeeo
bed cuddling comfort. Buy a polster-type bed if your dog leaner. Consider hammock bed for ease of cleaning. Buy mat for the easiest solution. Pick pillow this sequence not match with given steps, the final answer is No. Answer: No
promise = "("" hypothesis "(2-
large dog.
IInvalidate, Supporit, Neutral?"
answer nli_model. forarr.nnredddrreenisse hypothesis)
assert answer = format (statemmnttt stateeemt)))
Goal: Get Married in Oregon Planning Wed- ding Ceremony
Sleps:Decide
type of ceremony. Choose Hire wedding vendors. Confirm the date with vendors and officiants. Make linal payments.
Answer: Let's think step by step. The correct order should be: To get married, one need to decide ceremony. Then choose season. Then hire wed- ding vendor for organization. Confirm and finalize date. Then make the final payments. As this sequence the match given steps, so final answer Yes.
B.3.4 Prompts for WikiHow dataset
Few-shot Prompt for WikiHow
Answer: Yes
You given goal steps accomplish that goal. Your task is to determine whether the steps are right order (Yes) or not (No). Return your answer as Yes and No only.
###
Goal: (Goal)
Steps: (Steps)
Answer: Let's think step by step.
Goal: How to Select Dog Bed Understanding Dif- ferent Types of Beds
Steps: Buy a mat for the easiest solution. Pick pillow bed for large dog. Select donut bed your dog likes to feel secure. Purchase nest bed for cuddling comfort. Buy bolster-type bed your dog is leaner. if your dog likes burrow. Consider hammock bed for ease cleaning. Answer: No ###
Look cave-sty cave-style
Code Prompt for Wikiflow
Goal: How to Married Oregon Planning a
import order_steps
Wedding Ceremony
Steps: Decide on the type of ceremony. Choose a season. Hire wedding vendors. Confirm the date with vendors and officiants. Make final payments.
class Event:
1 "Given goal steps to achi determine
whether the steps in right order not. Return Yes if right order and No if order is wrong,
Answer: Yes ####
Goal: (Goal)
def tself, goal, steps): self. goal goal
(self,
Steps: (Steps)
Answer:
The goal that someone is trying accomplish" "1
self, steps = stops All the stops
Zero-shot Prompt for WikiHow
def get_order_of get_order_of_stepssce _ssp__ssssssssss)
Output Binary response Yes no return srder_stepssself goal, self stlf.steps)
You are given goal and steps to accomplish that goal. Your task is to determine whether the steps are in right order (Yes) or not (No). Return your answer Yes and No only.
event Event(
goal "(goal)"
Goal: (Goal)
steps "(steps)"
Steps: (Steps)
(event, get_order_of -toops == cfim-suffix> format (toa),
Answer:

---

B.3.5 Prompts for BIG-bench
CoT Prompt for BIG-bench
Task descriplion: Answer questions about which times certain events could have occurred.
Q: Today, Emily went to the museum. Between what times could they have gone?
We know that: Emily woke up pm Elizabeth saw Emily reading at the library from 2pm to 4pm. Jessica saw Emily watching a movie at the theater from 4pm 6pm. Leslle saw Emily waiting the airport from 5pm 6pm. William Emily buying clothes at the mail from 6pm to 7pm. The museum was closed after 7pm.
Few-shot Prompt for BIG-bench
museum. Between what times could they have gone?
Today, Emily went
Between times could Emily gone museum?
We know that: Emily wake at 1pm. Elizabeth saw Emily reading at library from 2pm to 4pm. Jessica saw Emily watching movie the theater from 4pm to 5pm. Leslle saw Emily waiting at the airport from 5pm to 6pm. William saw Emily buying clothes at the mall from 6pm 7pm. The museum was closed after 7pm.
Options:
(A) 1pm 2pm
(B) 6pm 7pm
(C) 5pm 6pm
(D) 2pm to 4pm
Answer: Let's think step by step. Wake-up time: 1mm.
Between what times could Emily have gone to the museum?
pm 2pm: free. 2pm-4pm: reading the library. 4pm- 5pm: watching a movie at the theater. 5pm-6pt 6om: wait- ing at the airport. 6pm-7pm: buying clothes at the mall. The museum closure time: 7pm. The only when Emily could have gone to the museum was pm to 2pm. So the answer is (A).
Options:
(A) 1pm to 2pm
(B) 6pm to 7pm
(C) 5pm to 6pm
(D) 2pm to 4pm
Answer: (A) ###
Strictly return the correct option which means return the letter of choice only
Input: (Input)
Ans:A ###
Answer: Let's think step by step.
Q: Today, Tiffany went to the beach. Between what times could they have gone?
Betty Tiffany getting coffee the cafe from 5am to 6am. Jessica saw Tiffany working at the office from 6am to 9am. saw Tiffany stretching at a yoga studio rooftop
know that: Tiffany woke
Code Prompt for BJG-bench
9am 12pm. Sean saw Tiffany sitting from 12pm to 2pm. Sarah saw Tiftany playing tennis at the tennis court from 2pm to 3pm. The beach was closed after 4pm.
import Schedul ing. quest.queeeeeee...e.. ion_answerinn
Context:
'ccoooe the right option number for the question depending on the context"
Between what times could Tiffany have gone the beach?
def __init_..ssttt question, ceassiioncontex... optton)::
Options:
context The context self fulffquestion question # The question options options options
9am to 12pm
(B) 12pm to 2pm
(C) 5am to 6am
(D) 3pm to 4pm
def (sell):::: answer low.tinggetionaaaneeee
Strictly return the correct option which means return the letter of choice only
Ans:D
self. questionn self. sal..context,sel. sel.text.elf..otiion. options) return answer
####t
input: (input)
Strictly return the correct option which means return
letter of choice only
context Context context "()", question "("" Options "()"
Ans:
assert Sesteettccntext.gee. setponttexx.get_asseeeee "er.fommttcistSSSiiinnnCCnnteee Qusstiio, optionss)
Zero-shot Prompt for BIG-bench
B.3.6 Prompts for TimeQA
Task description: Answer questions about which times certain events could have occurred Always return op- tion letter the end. There won't be any case when will
Zero-shot Prompt TimeQA
the options. Relurn the correct option only B, C or D. Input: (Input)
Answer the question based on the context. Context: (Context)
Question: (Question)
Ans:
Answer:.

---

CoT Prompt [or TA
B.4.1 MC-TACO
Answer the question based on context. Answer will be in the context. First generate reasoning and then provide final answer in a new line as Final Answer:
Context: (Context)
Question: (Question)
Answer: Let's think step by step.
MC-TACO dataset instance examples
Context:
Durer's father died in 1502, and his mother died in 1513.
Question: How long was his mother ill? Answer: she was ill for seconds
Label: no
Temporal Reasoning: Event Duration
Code Prompt for TA
Safti admits his love for Edwina to Lord Esketh, who is now sympathetic toward this good man's plight.
Context:
import Question__anserrnng
class Context:
"Return answer for this question based context'"
Question: Has Safti always been in love with Edwina?
def www.iiiittsll.ffeeooooooooeleeeo (self, contextttooxtt. self.context context The context self self.question- question question
Answer: no this a new thing
Label: yes
Temporal Reasoning: Stationarity
def get_answer(self): ans question_answering(e)) context, self. question)
The next evening, she arrived with stack of glistening stopboxes contain- ing sushi, sashimi, oysters in their shells, and Terran vegetables fresh plucked from their hydroponic beds.
Context:
return ans
context Context( context "()", question "()
assert(context..ett.co gesee ==, (contttt.
Question: At what time did she arrive?
question)
Answer: 6:00 PM
Label: yes
Temporal Reasoning: Typical Time
Context: The CIA now estimates that it cost al Qaeda about $30 million per year to sustain activities before 9/11 and that this money was raised almost entirely through donations.
Question: What happened to al Qaeda's finances after 9/11?
Answer: they were dealt a big blow Label: yes
B.4 Dataset Samples
Temporal Reasoning: Event Ordering
Context: This is an astonishing new record for a coin, he said.
Question: How often are new records es- tablished?
Answer: three times an second
Label: no
Example instances for all the dataset present this study.
Temporal Reasoning: Frequency

---

Goal: How to Buy a Used Sailboat Engine B.4.2 TimeDial
B.4.3 TNLI
TNL./ dataset instance examples
The woman wearing the pink jacket has thrown a Frisbee for the dog to catch.
Sentence 1:
Sentence 2:
The dog falling into a lake trying to catch the frisbee. Label: Support
Sentence 1: A young boy skipping down tennis court in absolute glee.
Sentence 2: He is now at a volleyball court. Label: Invalidate
TimeDial dataset instance examples
Sentence 1: A man sitting on sidewalk with shirt over his head.
Conversation: A:We need to take the ac- counts system offline carry the up- grade But don't worry, it won't 't cause too much inconvenience. We're going do it over the weekend.
Sentence 2: Most people prefer sitting to standing.
Label: Neutral
B: How long will the system be down for ? A: We'll be taking everything offline in about two hours time.. It'll be down for minimum of twelve hours If everything goes according to plan, it should be up again by 6 pm on Saturday
B.4.4 WikiHow
WikiHow dataset instance examples
B: That's fine. We've allowed <MASK> on the safe side.
Answer: forty-eight hours
Steps: Steer clear of rare or very old en- gines unless you're re certain there's an ad- equate supply of parts. Do the Smoke Test: healthy diesels make small amounts white cold
Label:
Conversation: A:Excuse me, Miss B: Yes May I help you?
of black smoke with starts. Check for fuel leaks and working bilge blower in gasoline engines, Before the seller cranks the engine, check see is already warm.
A: I'm a graduate student here in mathemat- ics I've just come from China and I've never used western library before I'll be here for <MASK> so I'd like to learn to use the library as efficiently as possible.. wonder if someone might have time to show me around.
Ordered?: 0
Goal: How to Breed Alpacas Encouraging Reproduction
B: I'd be very glad to show you around but
I'm very busy right now Could you come back about 3 thirty?
Steps: Expose the breeding male to the fe- male. Induce ovulation in the female al- paca. Place the male and female alpaca in the breeding pen. Separate the alpacas the female is not receptive. Wait a week or two after copulation to re-mate alpacas. Ordered?: 1
A: Sure thirty this afternoon.
Good. See you later
A: Thank you. Good-bye.
Answer: decades
Label: 0

---

B.4.5 BIG-bench
B.4.6 TimeQA
TimeQA dataset instance examples
Context: HMAS Wollongong ( (J172) HMAS Wollongong (J172). named for the city Wollongong, New South Wales wwas one of 60 s constructed during World War II and one of 20 built for the Admi- ralty but manned by personnel of and com- missioned into the Royal Australian Navy RAN ) Design and construction In 1938 the Australian Commonwealth Naval Board ( (ACNB) identified the need for general purpose local defence vessel capable of both anti-submarine and warfare duties while easy to construct and operate The vessel was initially envisaged as having a displacement of approximately 500 tons speed of at least, and a range of The oppor- tunity build prototype in the place of cancelled Bar-class boom defence vessel saw the proposed design increased ton vessel, with a top speed, and a range of armed with 4-inch gun equipped with asdic and able to fitted with either depth charges or minesweeping equipment depending on the planned operations al- though closer in size to sloop than local defence vessel, the resulting increased pabilities were accepted due to advantages over British-designed mine warfare and anti- submarine vessels.
BIG-bench dataset instance examples
Today, James went to the beach. Be- tween what times could they have gone? We know that: James woke up at 5am. Sean James walking towards the Statue of Liberty from 5am 6am. Michael saw James driving to the water park from 6am to 7am. Anthony saw James reading at the brary from 7am to 3pm. William saw James getting a coffee at the cafe from 4pm to 9pm. The beach was closed after 9pm.
Input:
680-
Between what times could James have gone to the beach?
Options:
(A) 7am to 3pm
(B) Sam to 6am
(C) 4pm to 9pm
(D) 3pm to 4pm
Answer: (D)
Input: Today, David went to the art studio.
Between what times could they have gone? We know that: David woke up at 5am. Linda saw David watching a movie the theater from Sam 7am. James saw David buying lunch at the deli from 9am to 10am. Mary saw David buying phone at the elec- tronics store from 10am to I lam. Leslie David driving to the water park from 11am to 2pm. Jessica saw David buying bike the bike shop from 2pm to 7pm. The studio was closed after 7pm.
Question: Which Navy operated the war- ship HMAS Wollongong from 1950 to 1951?
Answer: Indonesian Navy
B.5 Detailed Tables for Model and Prompt based Analysis
Table 8 compares the performance of eight Large Language Models on six datasets, analyzed under two different prompting strategies across F1 score and accuracy. Table 9 compares the performance of Instruction Tuned LLMs with CoT prompting strat- egy across both Fl and Accuracy, Table 10 com- pares the performance of Code Generation LMs with Code prompts across all datasets on both Fl and accuracy.
Between what times could David have gone to the art studio?
Options:
(A) 7am to 9am
(B) 2pm to 7pm
(C) 5am to 7am
(D) 11 2pm
Answer: (A)

---

MC-TACO
TNLI
TimeDiol
Wikilfow
BRG beach
Timeeit
Pow-shed
Zerussot
Few-shol
Zano-shot
Pew-chot
Zeru-shtt
Pew-shol
Zero-suu
Fow-sbol
Zunoshul
Zurovahot
GPT-3.5
8800
(1.5)
8622662
0.50049
(81298)
B459.64
0ASHU65 (-016)
0.55A151
0.480U45 (1 (,,0)
0.340.24
0.20/025 (1155)
FLA-15
0.30.74
0.740.71
(148)
0.290.3
0,33/125 (J28))
0.440.48
0450.3
0,550 28 (-05))
0.161.1
0,140.13
0.49.5
0.340
0466644
0.53041
0.280
1C280.12
DAN047
(9)
(3TPP))
(6 (4)
DA400
6P1-3
0.39816
0,390 N 226)
0.100
0,330.16
0,350.33
(4 126)
0.264022
0.260.34 (-14)
GPT
0,590158
0.520125
(11%)
0,430035
0,26-1113
0.290.12
05560.55
0.000
0.520221
0.320.36
(m09))
(59)))
0.548.53 (8)9)
(1109)
OP1
5.600.54
(1339))
0.33M.15
0,350.26
0.50.56
0.490033 (818))
0460.3
0.490L45 (15%)
0.540.
0.300.0(U15)
Table The performance of eight Large Language Models on six datasets, analyzed under two different prompting strategies. "Ace" stands for Accuracy, "FI" signifies Weighted-FI score, and "EM" corresponds to Exact Match. The percentage changes in accuracy performance between zero-shot and Few-shot prompting are indicated in parentheses.
MC-Taco
TNLI
TimeDial
WikilHw
BIG-bench
TimeQA
Model
Acc/F1
Acc/F1
Acc/F1
Acc/F1
Acc/F1
EM/FI
GPT-3.5
0.82/0.82 (1/2%)
0.46/0.45 (616%)
0.770.69 (115%)
0.54N.53 (11%))
0.67/0.67 (1/42%)
0.15/0.22
FLAN-T5
0.73/0.74 (133%)
0.38.0.3 (U.)))
0.54/0.48 (0%)
0,48/0.46 (113%)
0.2.1/0.11 (115%)
0.05/0.15
BLOONZ
0.45/0.45 (1) 14%)
0. 0,36/0.29 (114%)
0.53/0.52 (1145)
0.5/0.48 (4.3%)
0.3/0.22 (124%)
Dolly
0.48/0.48 (13%)
0.34/0.2 (U8%)
0,5/0.47 (0%)
0.47/0.37 (12%)
0.29/0.21 (116%)
0/0.15
Table Performance of Instruction Tuned LLMs with CoT prompting strategy. The percentage changes in accuracy performance between CoT and Few-shot prompting (from Table 8) are indicated in parentheses.
B.6 Detailed Table for Temporal Aspect Based Analysis
Table 11 compares the performance of LLMs across different temporal expressions. Table 12 compares the performance of LLMs across past and future reasoning events. Table 13 compares the performance of LLMs across different time- frames. Table 14 compares the performance of LLMs across multiple events and a single event. Ta- ble 15 compares the performance of LLMs across different context lengths.
B.7 Detailed Table for Temporal Task-based Analysis
Table 16 compares the performance of eight Large Language Models MC-TACO dataset and its fine-grained temporal task categories across both Fl and accuracy metrics.

---

MC-TACO
FNLI
TimeDial
WikiHow
BIG-hench
TimeQA
Model
Acc/F1
Acc/F1
Ace/F1
Acc/F1
Acc/F1
EM/FI
GPT-3.5 5
0,5/0.5
0,45/0.41
0.49/0.43
0.45/0.29
0.29/0.29
0.1/0.13
LanteCr
0.5/0.51
0.360.29
0.5/0.5
0.45/0.29
0.27/0.19
CodeGen2
0.61/0.6
0.35/0.25
0.5/0.41
0.45/0.29
0.25/0.19
0.1/0.13
Table 10: Performance Code Generation LMs with Code prompts in zero-shot setting
ED
EO
TT
Model
Exact
Ambiguous
Exact
Ambiguous
Exact
Ambiguous
Exact
Ambiguous
Exact
Ambiguous
GPT-3.5
0.87
0.89
0.84
0.85
0.83
0.85
0.76
0.79
0.62
0.81
FLAN
0.81
0.85
0.7
0.73
0.77
0.79
0.79
0.77
LLaM
0.75
0.74
0.5
0.54
0.41
0.37
Table 11: Performance of LLMs in accuracy across different temporal expressions in few-shot setting (ED; Event Duration, EO: Event Ordering, Frequency, S: Stationarity, TT: Typical Time)
Model
Past
Future
GPT-3.5
0.62
0.60
FLAN-TS
0.55
0.50
LLaMA
0.52
0.50
Table 12: Performance of LLMs in accuracy on past future based reasoning tasks in few-shot setting
Model
Seconds
Minutes
Hours
Day
More than Day
GPT-3.5
0.5
0.65
0.6
0.69
0.7
FLAN-T5
0.43
0.48
0.55
0.5
0.56
LLaMA
0.43
0.48
0.52
0.55
0.5
Table 13: Performance of LLMs in accuracy across different Timeframes in few-shot setting
ED
EO
TT
Model
ME
SE
ME
SE
ME
SE
ME
SE
ME
SE
GPT-3.5
0.91
0.82
0.75
0.88
0.88
0.7
0.8
0.68
0.68
FLAN-T5
0.85
0.8
0.57
0.75
0.89
0.79
0.5
0.72
0.58
0.6
LLaMA
0.72
0.41
0.5
0.5
0.78
0.6
0.4
0.62
0.44
0.54
Table 14: Performance of LLMs accuracy across single and multiple events in few-shot setting. (ME: Multiple events, SE: Single Events, ED: Event Duration, EO: Event Ordering. F: Frequency, S: Stationarity, TT: Typical Time)
Model
0-200
200-400
400-600
GPT-3.5
0.63
FLAN-T5
0.53
0.55
0.55
LLaMA
0.51
0.48
0.46
Table 15: Performance L.MMs in accuracy across different context length in few-shot setting

---

F1
0.60
0.2 053 0.02 0.56 0.5 0.51
0.5 0.5
ACC
0:71
048
1255
032 051 0.53 0.
1631
11.8
0.41
MCTI
ACC
0402
0.5 045
11.
0,50
FI
0.09 007
033 053
029
042
ACC
035
0.54
036
0.76 6.55
0.33 0.11 0,36 0.36 0.76 0.76 0.79 0.73 0.79 0.25 0,36 0.73 03 0.69 0.68 0.06 0.2 06
051
CUT
ARe
048 061 060 0.33
FL
0.56 055 042 (L.)
0.40 0:44 0L30
0.54 0.45
MCS
25
ARe
ON 0.85 0.5 0.77 0.77 0.69
II
03
0.49
045
Arc
0.57
ARE H
0.
045
0.8
0.55
02
0.31 0225
0.73 002
MGP
ARE
0.76 0.84 0.4 0.84 082 0.32
0.83
0.52
0664 0.S
HI
0.50
062
0.64
Aet
0.49
BAS
0643
0.71
HI
048
ARE
0.52
MC-B0 DD
HI
0.50 0.50 0.55 0.55
0.38
03
04
ARE
0,57 0.00 0,83 0.82 0.77
0.77
0,53
061 061
0.51
0.08
ARE
0.59
0522
0.54
0.52
II
0.5 0777 05 0.78 0.78 0.77
052 0.66 0448
ARE
EI
05 0ZT
0.76
0.58 06 0:32 025
006 1.29 0.99
073
0.22
MC-22
ARE
0.00
EI
085
046
005
AEE
085
045
0.66
0.33
0.2
071 053 0.74
0.48
ARD
032 0882
0455
044
FE
055
11.77 022
1137 0:2
0.53
MC
Z8
ARC
0175
0.34
0.52
F1
08
0.34
0.47
06
0.58
055
0.54
Axe
ULS
05
045
0.59
0.59
Model
GFT-3.5
FLO0-T5
Bolly
6PT-3
CFT Nba
LLO%
Table 16: Performance of eight Large Language Models MC-TACO dataset and it's fine-grained temporal task
categories (MC: MC-TACO, FS: Few-Shot, ZS: Shro-Shot)


Reviews:
[{"review_id": "8MTS9sy1w6", "reviewer": "Reviewer_izhz", "summary": "", "questions": "Question A: Can the authors provide more detail on the differences in performance among the individual models? Were there specific architectures or training protocols that produced better temporal reasoning abilities?\nQuestion B: Is there a reason the authors did not include human baselines in their evaluation? How do they believe the models would perform compared to human-level performance on the same tasks?\nQuestion C: Can the authors comment on the possibility of extending this work to other languages? What challenges and considerations might be involved in such an endeavor?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable review.\n\nSome weaknesses of the paper are: \n\n**Weakness 1: Lack of detailed discussion on individual model performance: While the paper provides an overview of the performance of large language models, it needs to delve into why individual models performed the way they did. For instance, it would be helpful to understand the architectural or training differences that led to the observed performance variations among the models.**\n\nResponse 1: We agree that digging deeper into the factors driving each model's strengths and weaknesses would provide useful insights. In the camera-ready version, we will expand the discussion to include the following dedicated analysis of architectural and training differences that may explain observed performance patterns. We assure the reviewer that we will also add one more column to Table 2 (“Characteristics of Different LLMs employed in this study”) in our paper. Our responses to the above comment are given below.\n\n   (a) Why GPT-3.5 Performs Best: GPT-3.5 excels due to its human-annotated instruction-following dataset from API prompts and the use of RLHF. After initial unsupervised and supervised training, human feedback refines its performance, enhancing language generation. \n\n  (b) Why FLAN-T5 Performs Best Among Open Source Models: Despite being older, FLAN-T5 showcases strong performance, even surpassing GPT-3.5 in some tasks, thanks to the extensive training data it received from the C4 dataset ((https://huggingface.co/datasets/c4)). \n\n  (c) Why BLOOMZ Performs Better than Other Instruction-Tuned Decoder Models (Dolly): BloomZ was trained on a cross-lingual mixture of tasks (xP3 - https://huggingface.co/datasets/bigscience/xP3) spanning dozens of languages. This exposes the model to far higher diversity during pretraining compared to Dolly, which was trained only on English data from a single company's employees. Previous studies [1] also shown that multilingual LLMs are better at instruction following and in-context learning. \n\n  [1] Tanwar, Eshaan, et al. \"Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment.\" arXiv preprint arXiv:2305.05940 \n   (2023)\n\n\n  (d) Why LLaMA Performs Better than Other Base Autoregressive Models: LLaMa has been trained on much more larger and diverse dataset as well as training objectives, compared to other base autoregressive models. LLaMa has been trained on a mixture of following datasets: (1) English CommonCrawl, (2) C4 dataset, (3) public Github code, (4) Wikipedia, (5) Gutenberg and Books, (6) Arxiv, and (7) Stack Exchange. Compared to this, OPT model has been trained on BookCorpus and Stories subsets of the RoBERTa corpus, the Pile dataset, and Pushshift.io corpus. GPT-J and GPT-Neo were trained only on The Pile Dataset.\n\nAs mentioned above, we will include the above explanations in the camera-ready version of the paper.\n\n\n**Weakness 2: Lack of performance comparison with human baselines: It would have been beneficial to compare the models' performance against human baselines. This would clearly explain how far or close LLMs are to human-level performance in temporal reasoning tasks.**\n\nResponse 2:  In our initial submission, we did not conduct additional human studies, primarily because published human results were only available for the MC-TACO, WikiHow, and TimeDial dataset itself. However, per your recommendation, we have now conducted human evaluations on 100 samples from each dataset. We hired 3 human annotators affiliated with our institution. Each annotator was given 100 randomly selected samples from the TNLI, BIG-bench, and TimeQA test sets. The annotators were instructed to answer the questions associated with the provided samples, ensuring they comprehensively understood the context and nuances of each sample. Post annotation, we computed the accuracy and F1 scores for the annotator's responses by comparing them against the ground truth. This enabled us to objectively gauge the human benchmark on the datasets.\nTo the best of our efforts to make the paper more thorough and experimentally rigorous, we have thus added human accuracy results in the table below:\n \n|Dataset | MC-Taco | TNLI* | TimeDial | WikiHow | BIG-bench* | TimeQA* |\n| ----------- | ----------- | ----------- | ----------- |----------- | ----------- |----------- |\n|Human  | 0.75         | 0.82  | 0.97         | 0.975     | 1.0                | 0.9.         |\n|GPT-3.5 Few Shot | 0.8         | 0.62  | 0.65         | 0.55     | 0.25              | 0.1         |\n\n* \\* indicates the human baseline that we performed on 100 randomly selected samples from each dataset. (this table will be added to the paper)\n\n*Discussion of the comparison of LLM performance against human performance*:\n\nThe comparative analysis of the performances of various Large Language Models (LLMs) (Table 3 in our paper) against the performance of human annotators on six different datasets (results shown in the above table) presents some intriguing insights. \n\n* Firstly, for tasks such as TimeDial, WikiHow, BIG-bench, and TimeQA, no LLM comes close to matching human proficiency. For instance, while humans score 0.97 on TimeDial, the best-performing LLM, GPT-3.5, only reaches 0.65 in its best scenario. This gap is even more pronounced in the WikiHow task, where the human baseline has a score of 0.975. For BIG-bench and TimeQA, human baseline also outperforms all models by a considerable margin with human baseline having a perfect score and 0.9 respectively. \n* Secondly, in the TNLI dataset, even the top-performing LLM lags behind human baseline scores. The human baseline had a score of 0.82, while the nearest LLM, GPT-3.5, managed only 0.62. This suggests that this task presents challenges that LLMs are still struggling with. \n* However, there is a silver lining in the MC-TACO dataset. Here, GPT-3.5 manages to surpass human performance in a few shot setting, with both scoring 0.8. This shows that in certain situations, some of the latest LLMs can indeed match human-level capabilities.\n\nWe would like to assure the reviewer that we will add these results as an additional row in Table 3, as well as the above discussion in Section 4.1 in camera-ready version.\n\n**Weakness 3: Absence of cross-lingual evaluation: The analysis is limited to English-language tasks. Since LLMs are often used for tasks involving multiple languages, evaluating temporal reasoning abilities in other languages would have made the study more comprehensive.**\n\nResponse 3: We respectfully note that multilingual datasets for temporal reasoning simply do not yet exist to enable such analysis. As the recent survey by Wenzel and Jatowt (2023) [1] shows, all the existing benchmarks cover solely the English language. While we agree that expanding to other languages is an important future direction, at present, it is not feasible given the lack of supporting data resources. \nHowever, we would also like to acknowledge some challenges that LLMs might facing when tested on multilingual and cross lingual datasets:\n* Differences in temporal expressions - The way time concepts are expressed varies across languages (e.g. grammatical distinctions like tense or aspect).\n* Cultural variations - How people interpret and reason about events can differ across cultures, which may affect model capabilities when expanding geographically.\n\n[1] Wenzel, Georg, and Adam Jatowt. \"An Overview Of Temporal Commonsense Reasoning and Acquisition.\" arXiv preprint arXiv:2308.00002 (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Key strengths of the paper include:\n\n- Depth of analysis: This paper's systematic and in-depth analysis stands out. Using various datasets and a wide range of LLMs provides a robust and comprehensive evaluation of temporal reasoning capabilities. Furthermore, the authors break down tasks into event duration, structure complexity, and timing precision, allowing for a nuanced understanding of LLM performance in temporal reasoning.\n\n- Positioning for future research: The paper underlines the limitations and strengths of current LLMs and outlines potential directions for future research. This could significantly benefit the NLP community by guiding efforts to improve LLMs' temporal reasoning capabilities, a critical aspect of natural language understanding.\n\n- Accessibility of results: The authors have done a commendable job in presenting their findings in a clear, comprehensible manner, making them accessible to a broad spectrum of the NLP community.", "reasons_to_reject": "The concerns were fully addressed and discussed by the authors.", "questions_to_authors": "Question A: Can the authors provide more detail on the differences in performance among the individual models? Were there specific architectures or training protocols that produced better temporal reasoning abilities?\nQuestion B: Is there a reason the authors did not include human baselines in their evaluation? How do they believe the models would perform compared to human-level performance on the same tasks?\nQuestion C: Can the authors comment on the possibility of extending this work to other languages? What challenges and considerations might be involved in such an endeavor?", "paper_topic_and_main_contributions": "Problem or Question Addressed:\n\nThe paper focuses on understanding how well LLMs perform on temporal reasoning tasks. Temporal reasoning refers to the cognitive processes and mechanisms humans use to understand and think about time. The main concern is to understand if LLMs can accurately reason about the timing, sequence, and duration of events, predict future occurrences, and discern between past and future events.\n\nMain Contributions:\n\n- Performance benchmarking of LLMs: The paper provides a comprehensive benchmarking of LLMs on a variety of temporal reasoning tasks. This involves testing models on tasks like understanding event stationarity, predicting typical timings of events, and discerning exact timings.\n\n- Identification of strengths and weaknesses: The study identifies specific strengths and weaknesses of different LLMs, like GPT-3.5, BLOOMZ, and FLAN-T5. \n\n- Handling of temporal expressions: LLMs find reasoning over exact temporal expressions (e.g., specific dates or times) more challenging than ambiguous ones (e.g., \"in a while\" or \"soon\").\n\n- Analysis of single vs. multiple temporal events: The authors found that tasks related to understanding the states and orders of events become more complex for models when there are multiple events involved as opposed to just a single event.\n\n- Limitations and future directions: The paper not only provides insights into the capabilities of LLMs but also outlines the limitations of the study, laying the foundation for future research in this domain.\n\nIn summary, the paper bridges an essential knowledge gap in the current understanding of LLMs’ capabilities in temporal reasoning. By highlighting their strengths and weaknesses, the study contributes significantly to the ongoing discourse on enhancing the reasoning capacities of these models.", "excitement": 5, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lbT7tL1Dhw", "reviewer": "Reviewer_topD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable review.\n\n**Weakness 1: One key weakness is the paper fails to put into perspective the performance of the different LLMs on temporal reasoning with regard to the best-performing fine-tuned model on each of these datasets. A quick glance at TNLI reveals the best model performance on the dataset was significantly higher (0.878 vs 0.62). Lack of this baseline makes it difficult to ascertain the extent of the reasoning capabilities of these LLMs. Furthermore, the original MC-TACO dataset was a QA dataset and used EM/F1 to measure the models' performance. Hence converting it into a binary classification task and using Acc as the metric reveals that most of these LLMs perform on par with random choice. Considering this insight, the paper seems to exaggerate the prowess of these LLMs much more than they should be given credit.**\n\nResponse 1: In the following table, we include the evaluation results from the best-performing fine-tuned models originally reported in the publication introducing each dataset.\n\n| Dataset|MC-Taco|TNLI|TimeDial|WikiHow|BIG-bench|TimeQA|\n| ----------- | ----------- |----------- |----------- |----------- |----------- |----------- |\n|Baseline Fine-Tuned Models|0.64|0.878|0.748|0.801|-|0.55|\n\n* \\- indicates that the BIG-bench dataset does not have a training set, so no model can be fine-tuned.\n\n*Comparison of LLM performance with Fine-tuned Models:*\nThe performance comparison of Large Language Models (LLMs) on various temporal reasoning tasks (Table 3 in our paper) against the fine-tuned models (see the above table) reveals some interesting findings. \n\n* First and foremost, when looking at the MC-TACO dataset, it is evident that while the baseline achieves an accuracy of 0.64, many LLMs, notably GPT-3.5 and FLAN-T5, outperform it significantly in both few-shot and zero-shot settings. For the TNLI dataset, where the baseline model registers an accuracy of 0.878, most LLMs trail behind, with even the best performer, GPT-3.5, only managing a 0.62 in the few-shot scenario. This can be attributed to the fact that this TNLI-based fine-tuned model is both fine-tuned on its training data and is also using external commonsense knowledge as compared to LLMs which are working purely in zero and few shot settings. \n* In the TimeDial and WikiHow datasets, the corresponding baseline models exhibit a performance of 0.748 and 0.801, respectively. Here again, GPT-3.5 leads the pack among LLMs but does not significantly surpass the baseline in either dataset. Again these models outperform LLMs purely because of the task-specific fine-tuning. \n* Lastly, for TimeQA, with a baseline performance of 0.55, only FLAN-T5 comes close to this mark with an exact match (EM) score of 0.4. This can be attributed to the fact that the baseline method for TimeQA uses a form of Retrieval augmented generation, FiD, thanks to which this technique was able to handle the long context of the dataset as compared to LLMs. \n\nWe would like to assure the reviewer that we will add these results as an additional row in Table 3 in the paper, as well as the above discussion will be incorporated into Section 4.1 in the camera-ready version of the paper.\n\nWe would also like to clarify the distinction between the original MC-TACO setup and our approach. In the original MC-TACO evaluation, the EM (Exact Match) metric requires labeling every answer for a given question correctly. This inherently makes the evaluation more challenging because a single error in any of the answers for a question leads to zero credit for that question. For ease of comparison, and in alignment with several previous works in the field [1][2], we chose to treat this as a classification problem and thus utilized accuracy as our primary evaluation metric. We believe this approach offers a clearer, direct measure of performance on individual instances. We also showed F1 results for this task in Table 7 in Appendix B.5\n\n On the point of potentially exaggerating the prowess of the LLMs, we respectfully disagree with this perspective. Our intention throughout the paper has been to provide a balanced view of the capabilities of LLMs, emphasizing both their strengths and weaknesses. To further clarify, we do not claim LLMs are capable of solving all temporal reasoning problems perfectly. Our goal is to benchmark performance across a spectrum of tasks to reveal where capabilities are strong versus lacking. The limitations we outline, like struggling with precise time expressions or future event forecasting, emphasize that significant work remains to achieve human-level temporal commonsense reasoning. Furthermore, it's important to underscore the practical value of having a single LLM that is proficient across various temporal commonsense tasks rather than fine-tuning and then applying different models for each different case.\n\n[1] Guan, Xin, et al. \"Multi-hop Commonsense Knowledge Injection Framework for Zero-Shot Commonsense Question Answering.\" arXiv preprint arXiv:2305.05936 (2023).\n\n[2] Bian, Ning, et al. \"Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models.\" arXiv preprint arXiv:2303.16421 (2023).\n\n**Missing References:\nSaxena, A., Chakrabarti, S. and Talukdar, P., Question Answering Over Temporal Knowledge Graphs. \nNaik, A., Breitfeller, L. and Rose, C., 2019, September. TDDiscourse: A dataset for discourse-level temporal ordering of events. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue (pp. 239-249).**\n\nResponse: Thank you for pointing out the missing references. We will correct this in the future version by adding these references and discussing the connections to our work where appropriate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A comprehensive survey of model performance of 8 LLMs with 3 different prompting techniques over 6 datasets.\n2. Well-devised out stress-tests to capture the model performance across different temporal reasoning categories.", "reasons_to_reject": "1. One key weakness is the paper fails to put into perspective the performance of the different LLMs on temporal reasoning with regard to the best-performing fine-tuned model on each of these datasets. A quick glance at TNLI reveals the best model performance on the dataset was significantly higher (0.878 vs 0.62). Lack of this baseline makes it difficult to ascertain the extent of the reasoning capabilities of these LLMs. Furthermore, the original MC-TACO dataset was a QA dataset and used EM/F1 to measure the models' performance. Hence converting it into a binary classification task and using Acc as the metric reveals that most of these LLMs perform on par with random choice. Considering this insight, the paper seems to exaggerate the prowess of these LLMs much more than they should be given credit.\n2. Experiment details of prompting, such as the impact of temperature, number of runs for each experiment, and number of few-shot examples chosen, are missing from the work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a comprehensive survey of LLMs on the task of temporal commonsense reasoning; they cover 8 LLMs and try out 3 different prompting techniques, including code-based prompting to evaluate the models' performance on 6 datasets. They also deep dive into the model performance across different categories of temporal tasks and also investigate model performance in terms of reasoning over longer time frames, multiple events, and past/ future events, amongst others.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Saxena, A., Chakrabarti, S. and Talukdar, P., Question Answering Over Temporal Knowledge Graphs.\n\nNaik, A., Breitfeller, L. and Rose, C., 2019, September. TDDiscourse: A dataset for discourse-level temporal ordering of events. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue (pp. 239-249)."}}, {"review_id": "J9SrjgXi8y", "reviewer": "Reviewer_Kfj1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable review.\n\n**Weakness 1: The scope of the evaluated models is limited. Because the evaluation method is zero or few shot, the authors should evaluate more models that have 10-60B parameters. The latest GPT-4 should also be evaluated for us to gain a better understanding of LM's ability of temporal reasoning.**\n\nResponse 1: We appreciate your thoughtful review of our work and your suggestions to expand the scope of model evaluations to include larger models like GPT-4, as well as models with a wider range of hyperparameters. We understand the importance of a comprehensive evaluation, and we would like to address your concerns as follows:\n*We acknowledge the importance of evaluating the latest models, such as GPT-4, to gain a deeper understanding of their temporal reasoning capabilities. However, it's important to note that conducting experiments with GPT-4 is considerably more resource-intensive. As our lab is located in a developing country with limited funding allocation for research, the associated costs unfortunately pose hard restrictions. To provide context, our experiments with GPT-3.5 cost approximately 200 USD. We estimate the GPT-4 compute would be over 2000 USD given it is 10x more expensive, which we simply cannot afford at this time. Moreover, the broader research community and the conference organizers have been emphasizing the adoption and evaluation of open-source models. The rationale is to ensure reproducibility and democratize access to findings. There is a prominent stance advocating open, reproducible research while discouraging or prohibiting non-open-source models, as evident in discussions like https://hackingsemantics.xyz/2023/closed-baselines/. Such positions underscore the call for transparent models that can be openly and freely studied, rather than proprietary systems.\n\n* We acknowledge the importance of assessing models with varying parameter sizes. Unfortunately, our current GPU infrastructure cannot accommodate models with more than 7B parameters. As a result, we were constrained to this parameter threshold in our evaluation. Our efforts were directed towards leveraging the resources we had to produce meaningful and reliable results within our technological constraints.\n\n* We acknowledge the importance of assessing models with varying parameter sizes. Unfortunately, our current GPU infrastructure cannot accommodate models with more than 7B parameters. As a result, we were constrained to this parameter threshold in our evaluation. Our efforts were directed towards leveraging the resources we had to produce meaningful and reliable results within our technological constraints.\n* Despite these resource constraints, we devoted considerable effort to ensuring that our study encompassed as much diversity and comprehensiveness as our budget and resources would permit. We executed over 400 experiments in total, aiming to provide a broad and representative evaluation of the models within our capabilities. Our intention was to offer valuable insights within the scope of our means, even though some models and parameter sizes were beyond our reach.\n\n**Weakness 2: The authors should include more temporal relation datasets, because MC-TACO only considers a small set of temporal ordering, and it has been shown that general temporal relation (e.g., TRACIE from Zhou et al. 2020) is very challenging to LMs.**\n\nResponse 2: In response to your points, we would like to address them as follows: \n\n* TRACIE Dataset Inclusion: We acknowledge the significance of the TRACIE dataset in evaluating temporal reasoning abilities in language models. However, we regret not being able to include the TRACIE dataset in our analysis. This decision was primarily driven by the dataset's lack of a publicly available test set (https://github.com/allenai/aristo-leaderboard/tree/master/tracie/data). While the training data and labels are accessible, the absence of a public test set makes it challenging to conduct a comprehensive evaluation that adheres to transparent and reproducible research standards. \n* Instead, we included the TNLI dataset which is similar to TRACIE in evaluating natural language inference over temporal relations. Much like the TRACIE dataset, TNLI offers a valuable perspective on temporal reasoning through natural language inference, thereby enhancing the breadth of tasks covered in our study.\n* Diverse Temporal Tasks: We have taken great care to encompass a wide range of temporal tasks in our dataset selection, as highlighted in the Table 1 in paper. The various tasks, such as binary classification, natural language inference, question answering, and multi-class classification, offer a comprehensive assessment of temporal reasoning abilities. We believe that this diversity enriches our analysis and provides a holistic understanding of the strengths and limitations of language models in handling temporal relations.\n\n**Weakness 3: The conclusions taken from the experiments are not thorough enough, there is not much useful takeaways that future work can benefit from.**\n\nResponse 3: We respectfully disagree with the perspective that our paper lacks thorough conclusions. Our intent was to provide detailed insights into the performance of various language models on temporal reasoning tasks. We thus would like to provide a clearer articulation of our conclusions and their implications:\n\n*  Our experiments have revealed the robust performance of GPT-3.5 and FLAN-T5 in temporal reasoning tasks.\n    * Implications: (1) Future work could prioritize these models for applications requiring robust temporal commonsense reasoning. (2) Most of the current LLM works focus on arithmetic and commonsense tasks but this shows that many LLMs struggle with temporal reasoning tasks underlining the need to add these tasks as essential experiments for every LLM.\n\n* We have observed that Code Generation LMs do not excel as temporal commonsense reasoners.\n    * Implications: While many studies have demonstrated that code generation LLMs are superior commonsense reasoners compared to general-purpose LLMs, this does not appear to hold true for temporal commonsense. Researchers should exercise caution when selecting Code Generation LLMs for tasks focused on temporal reasoning.\n\n* Language models exhibit strong performance on event frequency and duration tasks.\n    * Implications: These tasks can serve as baseline tests for upcoming models to gauge basic temporal reasoning proficiency.\n\n* There is a mixed performance on event ordering tasks, indicating areas where improvement is needed.\n    * Implications: Recognizing and addressing the shortcomings in event ordering tasks can lead to the development of LLMs that better understand narratives, timelines, and event sequences. This improvement can have a cascading effect on applications like story generation, timeline extraction, and chronological data interpretation, making them more reliable and robust.\n\n* Challenges arise in understanding event temporal states, leading to performance drops.\n    * Implications: Researchers should focus on expanding and refining training datasets to include a diverse range of temporal states and contexts. Additionally, specialized training modules or auxiliary tasks can be introduced to specifically target understanding and differentiating between various temporal states.\n\n* Our analysis highlights that LLMs struggle with specific event timings.\n    * Implications: (1) Temporal reasoning research may need to focus efforts on incorporating more explicit timing data and signals into models. The use of temporal markers, timings, and calendars in training data might provide better cues for models to grasp specific timings. (2) By bolstering an LLM's capability to discern specific event timings, models could become more adept in applications like event scheduling, time-sensitive data analysis, and precise temporal reporting.\n\n* Reasoning about future events proves more challenging than reasoning about past events.\n    * Implications: (1) Evaluation sets may need to better balance past and future to avoid overestimating abilities. (2) Future work could focus on incorporating future forecasting objectives during pretraining, such as predicting subsequent events and fine-tuning on future-focused datasets requiring forward prediction beyond current states.\n\n* Language models tend to perform better in temporal reasoning over longer time frames.\n    * Implications: (1) This suggests that models can build up temporal context given sufficient history, but may struggle with sparse events. (2) Model architectures can explore separate encoding pathways for long-range versus short-range timing data. (3) Prompting strategies that efficiently provide background temporal context could aid short-context reasoning.\n\n* However, they encounter difficulties in temporal reasoning when dealing with extended contexts.\n    * Implications: (1) This suggests that there are limitations in tracking temporal flows over lengthy histories. (2) Performance could degrade for applications requiring long-term temporal dependencies. (3) Evaluation benchmarks should explicitly target multi-paragraph or chapter-length temporal reasoning. (4) Model architectures can explore strategies like memory components to help retain long-range temporal state.\n\n* Precise temporal expressions pose a challenge compared to ambiguous ones.\n   * Implications: Pretraining objectives could mix explicit and implicit temporal references. Evaluation sets should include both precise and ambiguous temporal language.\n\n* Understanding the states and orders of multiple events emerges as a challenge for LLMs.\n    * Implications: There is a need to develop multi-event temporal benchmark tasks requiring modeling interdependencies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is the first work to focus on temporal reasoning and evaluate several models on several baselines.", "reasons_to_reject": "1) The scope of the evaluated models is limited. Because the evaluation method is zero or few shot, the authors should evaluate more models that have 10-60B parameters. The latest GPT-4 should also be evaluated for us to gain a better understanding of LM's ability of temporal reasoning. \n2) The authors should include more temporal relation datasets, because MC-TACO only considers a small set of temporal ordering, and it has been shown that general temporal relation (e.g., TRACIE from Zhou et al. 2020) is very challenging to LMs. \n3) The conclusions taken from the experiments are not thorough enough, there is not much useful takeaways that future work can benefit from.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper evaluates several language models on six temporal reasoning benchmarks. Following Zhou et al's setting, the authors employ five temporal reasoning categories and separately evaluate and analyze different categories. This paper has proposed some conclusions on different models' ability of temporal reasoning based on experiments and analysis.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an extensive analysis of temporal reasoning capabilities of LLMs. Most reviewers agreed that the paper presented a clear and thorough analysis of an interesting and important category of reasoning. The authors responded to reviewers' questions with very thorough responses and additional results that inspire confidence that they will update the next draft to include these additional requested details and results, which will make the paper even stronger. Main concerns raised were that the paper doesn't include an analysis of a 10B+ parameter LM, and that it should include evaluation on the TRACIE dataset. The missing TRACIE eval was due to a misunderstanding where the authors believed that there was no public test set. However, they claimed to be running the experiments and plan to include these results in their next draft. The authors also justify the lack of 10B+ model by citing the cost, which I think is reasonable; while including larger models would be nice, they are not required for the soundness of the paper, and I believe the current set of results will already be of interest to the community.
Decision: Accept-Main