MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models
Authors: Yifan Wei
Yisong Su
Huanhuan Ma
Xiaoyan Yu
Fangyu Lei
Yuanzhe Zhang
Jun Zhao
Kang Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: Temporal Reasoning
Large Language Models

Abstract: Large language models (LLMs) have shown nearly saturated performance on many natural language processing (NLP) tasks. As a result, it is natural for people to believe that LLMs have also mastered abilities such as time understanding and reasoning. However, research on the temporal sensitivity of LLMs has been insufficiently emphasized. To fill this gap, this paper constructs Multiple Sensitive Factors Time QA (MenatQA), which encompasses three temporal factors (scope factor, order factor, counterfactual factor) with total 2,853 samples for evaluating the time comprehension and reasoning abilities of LLMs. This paper tests current mainstream LLMs with different parameter sizes, ranging from billions to hundreds of billions. The results show most LLMs fall behind smaller temporal reasoning models with different degree on these factors. In specific, LLMs show a significant vulnerability to temporal biases and depend heavily on the temporal information provided in questions. Furthermore, this paper undertakes a preliminary investigation into potential improvement strategies by devising specific prompts and leveraging external tools. These approaches serve as valuable baselines or references for future research endeavors.

MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models
Yifan Weil.2, Yisong Su Huanhuan Ma², Xiaoyan Yu Fangyu Lei
Yuanzhe Zhang 1.2 Jun Zhao Kang Liu
The Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA University of Chinese Academy of Sciences, FFuzhou University *Bejjing Institute of Technology, 'Shanghai Artificial Intelligence Laboratory ac.
(weiyifan2021, mahuanhuan2021, 1ei fangyu22222@iaaa....m ac. cn, 2210200420fzu edu.cucc {xiaoyan. yu, yzzhang, jzhao, kliujenlpr.... ia ac cn
Abstract
Context [1] Twittrr created by ack Dorsey, Noah Glass, Biz Stone, and Evan Williams in March 2006 and launched in July of year. [2]On October 16, 2008, Evan Williams be- came the CEO, and Dorsey became the chairman of the com- pany. [3] ack Dorsey rejoined Twitter in March 2011 as Exe- cutive Chief of Product Development. [4] In une 2020, Twitter announced that Pttrick Pichette wauld succeed Omid Korde- chairman. [5] November 2021, ack Dorsey stepped
Large language models (LLMs) have shown nearly saturated performance on many natural language processing (NLP) tasks. As a result, it is natural for people to believe that LLMs have also mastered abilities such as time un- derstanding and reasoning. However, research the temporal sensitivity of LLMs has been insufficiently emphasized. To fill this gap, this paper constructs Multiple Sensitive Factors Time QA (MenatQA), which encompasses three temporal factors (scope factor, order fac- tor, counterfactual factor) with total 2,853 sam- ples for evaluating the time comprehension and reasoning abilities of LLMs. This paper tests mainstream LLMs with different param- eter sizes, ranging from billions to hundreds of billions. The results show most LLMs fall be- hind smaller temporal reasoning models with different degree on these factors. In specific, LLMs show significant vulnerability to tem- poral biases and depend heavily the tempo- ral information provided in questions. Further- more, this paper undertakes a preliminary inves- tigation into potential improvement strategies by devising specific prompts and leveraging external tools. These approaches able baselines or references for future research endeavors.
down as CEO and replaced by Parag Agrawal, the chief technology officer. [6] On October 27, 2022. business magnate Elon Musk acquired Twitter for US$44 billion, gaining control of the platform. [7] On May 12, 2023, Musk announced that he will resign as CEO of Twitter in approximately six weeks.
Multiple Sensitive Factors Time QA Scope Factor:
Who was the CEO of Twitter from May 2013 2020
Order Factor:
Shuffle the order of the sentences [1-7] in the context.
Counterfactual Factor:
W was the CEO of Twitter from March 2011 to July 2022, If
ack Dorsey stepped down as CEO in November 2022
Answer: ack Dorsey
Yellow front indicates the time specifiers of events in the context. Scope Factor refers to the time specifiers would be different between the question and the given context. Order Factor is where the complete events in the context are shuffled in chronological order. Counterfactual Factor a question with hypothetical propositions.
Figure 1:
valu-
1 Introduction
capture the temporal scope and interval of events in a given context, is yet seldomly explored. This ability is particularly important and necessary in many downstream tasks, such as Question Answer- ing (QA), due to the inconsistency of answers in real events across time ranges. For example, as shown in Figure 1, given the context "From March 201 I to November 2021, Jack Dorsey rejoined Twit- ter CEO, and In November 2021 Jack Dorsey stepped down as CEO and was replaced by Chief Technology Officer Parag Agrawal", the answer the question "Who was the CEO of Twitter from year A to year B?" could be either "Jack Dorsey" or "Parag Agrawal", depending on the time pe-
Recent Large Language Models (LLMs; Zeng et al. 2022; Touvron al. 2023; Zhang et 2022) such as GPT-4 (OpenAl, 2023) pretrained on a vast amount of text corpus have achieved nearly sat- urated performance on most Natural Language Pro- cessing (NLP) tasks. Meanwhile, plenty of works have evaluated the reasoning abilities of LLMs on several tasks, such numerical reasoning (Chen al., 2022), logical reasoning (Saparov and He. 2022), counterfactual reasoning (Li et al., 2023), and multi-hop reasoning (Lei et al., 2023; Ma et al., 2023). However, the temporal reasoning ability of LLMs, which refers to the capacity of a model to

---

riod (Lyear A, year BJ) in the question.
the temporal reasoning ability of the LLMs. In de- tail, the MenatQA dataset contains 2,853 samples. which are partitioned into 1.448 samples for the scope type, 857 samples for the order type, and 548 samples for the counterfactual type, respec- tively.
To verify the temporal reasoning ability of mod- els, a few datasets have been proposed. Situat- edQA (Zhang and Choi, 2021) focused on how answers vary according to different extra- sataa-inguuistic contexts, such as, when questions are asked. Re- alTime QA (Kasai et al., 2022) was proposed to answer questions where real-time news was served the contexts. Recently, TimcQA (Chen et al., 2021) was proposed for time-sensitive question an- swering, particularly for the temporal scope factor. In TimeQA, the time specifiers would be inconsis- tent between the question and the given context. As shown in Figure 1, a time specifier in "Who was the CEO of Twitter from May 2013 to 2020?" is 2020, but the time specifier of the correct an- swer in the context is November 2021. As result, the system needs more powerful abilities of time understanding and temporal reasoning.
Based on the proposed MenatQA, serveral main- stream models are evaluated, including the SOTA temporal reasoning models (BigBird (Zaheer al., 2020) and FiD (Izacard and Grave, 2020)), and cur- rent typical large language models such as LLAMA (Touvron et al., 2023), OPT (Zhang et al., 2022) and GPT-3.5 (gpt-3.5-turbo; Brown et al. 2020). The experimental results demonstrate the majority of LLMs perform poorly on our MenatQA dataset. It indicates potential deficiency in LLMs' com- prehension of temporal concepts. Moreover, to enhance the temporal reasoning ability of LLMs. especially for aforementioned scope factor, order factor, and counterfactual factor, this paper pro- poses some preliminary investigations, such as de- signing specific prompts and tool learning. These approaches will serve as baselines in MenatQA and can be used as benchmark for future research. Our main contributions are summarized as fol- lows:
Nevertheless, there are more temporal reason- ing abilities (factors) that need to be verified but are usually neglected, besides the identified tem- poral scope factor in TimeQA. The first is order factor. For example. "[On October 16, 2008J, Evan Williams became the CEO, and Dorsey be- came the chairman of the company. Jack Dorsey re- joined Twitter in [March 2011] Executive Chief of Product Development". In this example, the chronological sequence of events is laid out by the given time specifiers, illuminating the progression of roles within the company. Consequently, recog- nizing the chronological order of events is a funda- mental ability and typically assessed in evaluations concerning time understanding and reasoning.
We present a new dataset named Multiple Sensitive Factors Time QA (MenatQA). This is the first dataset containing multiple time- sensitive factors that can be used as an evalua- tion benchmark for assessing the time under- standing and reasoning abilities of LLMs.
We evaluate the performance of current LLMs on three temporal factors, revealing their high susceptibility to temporal biases and their re- liance on specific temporal information given questions for reasoning about time.
The second is counterfactual factor. Questions with temporal assumptions greatly escalate the dif- ficulty of temporal reasoning. Answering such questions may require additional information or counterfactual thinking of models. For example, "Who was the CEO of Twitter from March 201 to July 2022, if Jack Dorsey stepped down as CEO in November 2022?". LLMs should be able to understand that "Jack Dorsey was still the CEO of Twitter from March 2011 to November 2022". Obviously, answering such types of questions is another form test temporal reasoning ability.
We provide preliminary investigations to op- timize temporal reasoning ability of LLMs, which can be used as baseline to inspire the future research.
The MenatQA Dataset
2.1 Dataset Construction
To facilitate the development of research around the aforementioned problems, this paper proposes a new dataset, Multiple Sensitive Factors Time QA (MenatQA). which encompasses the above three temporal sensitivity factors and is used to evaluate
Data collection. We construct MenatQA based on TimeQA (Chen et al., 2021) dataset. Only time questions that are accompanied with a golden con- text and a detailed time scope of event are col- lected. To extract the relevant time scope, correct

---

Type
Questions
Answerable
Unanswerable
Doc-Token
#Questioo-Tooee
Scope
1448
1316
132
227
16
Order
857
669
188
264
Counterfactual
548
462
86
227
30
Total
2853
2447
406
Table 1:
The dataset provides statistics for different types of factors. #Doc-Tokee and #Question Tokke represent the average number of tokens within the document and question, respectively. This paper counts the number of tokens using GPT-2 tokenizer, which the same tokenizer as ChatGPT.
answers, annotated paragraphs, and golden context from documents, we develop a script that utilizes JSON syntax for accurate identification.
tion goes beyond the context and requires imagination to connect the context and the hypothetical question (Li et al., 2023; Tang et al., 2023). Counterfactual questions consist of question ("Who was the CEO of Twitter from March 2011 July 2022?"), alongside a premise that contradicts the given context ("If Jack Dorsey stepped down as CEO November 2022"). Based on this premise, an imaginary consequence of the counterfac- tual question yields "Jack Dorsey", as shown in Figure 1. Inspired by previous work on constructing counterfactual samples (Li al., 2022), we ask the annotators to imagine a tem- poral hypothesis that contradicts the context (e.g., changes in years). Then constructing a "if" question based on the hypothesis, whilc providing the correct answer. To ensure the diversity of phrasing, annotators are free to generate various phrasing of the assumption, and there is restriction the position of the assumption.
Data annotation. We represent the time factor three types: scope factor, order factor, counter- factual factor. The detailed information about the annotation can be found in the Appendix A.2.
The definition of the scope factor refers to the time scopes that are relevant to the ques- tion (e.g.. "From 2011 to 2021"). Specially, the scope type includes two types of ques- tions: extraction and reasoning. The extrac- tion questions originate from TimeQA!, and the reasoning questions can be obtained by adding more fine-grained information such as months (e.g.. "From March 201 1 to November 2021"), "), narrowing the time range (e.g.. "From 2013 to 2020"), or expanding the time range (e.g., "From 2008 to 2021"). In detail, the reasoning type questions are addressed using OpenAl's text-davinci-003 API, employing few-shot learning alter the temporal inter- vals mentioned in the questions. Subsequently, we provide both the original and altered ques- tions to the three annotators, requesting them to provide answers to the altered questions based on the contextual information.
2.2 Dataset Statistics
Key statistics. The MenatQA dataset contains 2853 time-sensitive factor samples, which are parti- tioned into the scope type, order type and counter- factual type corresponding to 1448, 857 and 548 samples. The main statistical data for factors shown in Table To address the issue of potential illusory outputs in LLMs, introducing unanswer- able questions serves as an effective means to as- sess their understanding of temporal knowledge. MenatQA, we find that there are only 85.7% of the questions are answerable questions, while 14.2% are unanswerable questions.
The order factor pertains the chronologi- cal sequence of events in the context. Typ- ically, the descriptive information on each Wikipedia page is written in chronological order, as shown in the context in Figure 1. We asked three annotators to read the context, identify different events based time, and then shuffle the order of these events in the context.
Specially, the scope type includes two types of questions: reasoning and extraction, with 450 and 998 samples, respectively. The extraction type refers questions where the corresponding time specifier can be directly found in the context, while the reasoning type refers to questions where there
The counterfactual factor refers to hypotheti- cal propositions about time, where the assump-
We adopt the Fasy-Mode version of TimeQA, which only involves extraction type questions.

---

MenatQA
et al., 2022). LLAMA (7B and 13B) (Touvron al.. 2023), ChatGPT (gpt-3.5-turbo: Brown al. 2020). The detailed information about models can be found in the Appendix A.3.6.
17%
3.3 Results and Analysis
52h
31N
In this section, identify weaknesses in LLMs with respect three temporal factors by analyzing the differences among various models.
Scope
Order
Counterfacutal
To validate the impact of various time-sensitive factors that were overlooked in previous works the temporal reasoning ability of large language models,
Figure 2: Statistics on the types of time-sensitive factors in the MenatQA dataset.
the performance aforementioned LLMs under three time factors on MenatQA, shown Table 2. In order to further comprehen- sively analyze the susceptibility of large language models to temporal biases, we compare the perfor- mance of LLMs on extraction and reasoning ques- tions in the scope factor of MenatQA, shown in Table 3. Based on the results,
is discrepancy between the time in the context and the question. The proportion of time factor types is shown in Figure These statistics indicate that MenatQA exhibits rich diversity in question distribution. The average length of questions in MenatQA is 20.71 words, while the context con- sists on average of 238.16 16 words, demonstrating their rich vocabulary. For more detailed statistical data, please refer to Appendix A.1.
find that: Firstly, analyzing the results Table 2, it can be observed that LLMs display varying sensitivities towards different time factors. Notably, the coun- terfactual and scope factors exert the most signifi- cant impact on LLMs, shown by the highlighted sections with yellow background in the table. Ad- ditionally, not all LLMs outperform FiD on every type of factor. For instance, when evaluating the performance of GPT-3.5-turi on the counterfac- tual factor, it fails to surpass FiD, with F1 and EM scores of 34.69 and 27.66, respectively. These scores are significantly lower than the correspond- ing results achieved by FiD (F1: 45.79, EM: 34.03). Besides, none of the other LLMSs demonstrate supe- riority over FiD acToss all temporal factors, except for LLama-13B. conclusion. LLMs still have limitations in effectively processing implicit tem- poral information, as indicated by their inadequate performance and sensitivity to different temporal factors. Therefore, more research is needed to en- hance the temporal understanding and reasoning capabilities of LLMs.
3
The Performance of LLMs on MenatQA
3.1 Task Definition
We focus on time-sensitive question answering tasks. The input of these tasks is formulated as (c.q) q) for free-form generation tasks, where e is the context and q the question. The desired output is either span from the context or "unanswerable" text.
3.2 Baselines
In this section, we introduce the temporal reason- ing models and currently popular large language models. These serve as the main evaluation back- bone for MenatQA, enabling assess the per- formance of mainstream large language models on three types of temporal factors.
The baselines in our experiments include: Big-
Secondly. in extraction type questions, the major- ity of LLMs (i.e., ChatGLM-6B, Bloom-7BI. OPT- Series and GPT-Series) cannot achieve satisfactory outcomes when compared with temporal reasoning models (i.e., BigBird and Fid), as shown in Table 3. The weakness of LLMs in temporal reasoning is more prominent in reasoning type questions, where all LLMs exhibit varying degrees of performance decline compared to extraction type questions. This
Bird (Zaheer et al., 2020) and FiD2 (Izacard and Grave, 2020), ChatGLM (6B) (Zeng et al., 2022),
BLOOM (7.1B) (Scao al., 2022), GPT-J (6B)
(Wang and Komatsuzaki, 2021), GPT-NEOX (20B)
(Black et al., 2022), OPT (6.7B and 13B) (Zhang
'Especially, We use the versions of BigBird and FiD that have heen finc-tuned on the Natural Questions (NQ: Kwiatkowski et al. 2019) dataset.

---

Me w/scope
MenatQA w/order
MenatQA w/counterfactual
MenatQA w/all factors
Method
Fl
EM
F1
EM
F1
EM
F1
EM
Temporal Reasoning Models
Big)ird(NQ)
35.31
23.72
41.84
29.62
40.84
28.22
35,81
24.22
FiD(NQ)
41.03
30.63
45.85
33.73
45.79
34.03
36.12
26.32
Large Language
Models
ChatGLM-6B
21.49
6.11
23.99
6.01
19.20
3.61
13.77
3.81
Bloom-7BI
25.69
13.33
32.34
16.94
29.34
12.62
24.32
9.62
GPT-J-6B
34.56
24.95
39.40
27.86
45.64
33.07
37.84
25.85
GPT-Neox-20B
36.23
25.55
41.61
27.86
45.73
30.56
37.81
24.55
OPT-6.7B
29.14
19.84
35.79
24.35
32.77
20.54
27.27
16.13
OРT-13B
37.30
26.85
45.17
32.06
42.32
25.95
35.93
21.94
LLama-7B
45.71
34.47
51.93
37.17
49.78
33.97
37.53
23.45
LLama-133
52.13
41.58
64.45
52.40
53.56
41.48
39.80
28.66
GPT-3.5-turbo
47.34
37.78
51.20
38.38
34.69
27.66
27.99
24.45
Table 2: The performance of models each time-sensitive factor in the MenatQA dataset. Bold scores indicate superior performance compared to FiD. The factor with the most significant impact (lowest performance) on individual model is highlighted with yellow background color.
Scope factor in MenatQA
mance.
Method
Extraction
Reasoning
Fl
Fl
EM
4 Simple Investigations for Improvement
BigBird(NQ))
43.38
30.13
35.31
23.72
In order handle the three types of time factors in MenatQA, this paper proposes scope prompt- ing, counterfactual prompting and rerank prompt- ing methods under the zero-shot settings. Since the scope prompting method not universal (e.g.. it causes the EM score of GPT-3.5-turbo to drop from 37.78 to 31.36, as shown in Table 4), this paper explores tool learning and designs time comparison tool specifically to address the scope factor questions.
FiD(NQ)
48.77
36.73
41.03
30.63
LLMs
ChatGLM-6B
24.97
6.21
21.49
6.11
Bloom-7BI
31.90
17.33
25.69
13.33
GPT-J-6B
38.86
27.86
34.56
24.95
GPT-Neox- 20B
43.71
30.76
36.23
25.55
OPT-6.7B
35.50
24.65
29.14
19.84
OРT-13B
45.10
32.46
37.30
26.85
LLama-7B
57.09
43.09
45.71
34.47
LLama-13B
65.55
53.41
52.13
41.58
GPT-3.5-turbe
52.52
39.08
47.34
37.78
4.1
Specific Prompts for Three Temporal Factors
Table 3: The performance of LLMs extraction and reasoning questions in the scope factor of MenatQA. Bold Scores indicate a higher performance than FiD.
Base Prompt To evaluate the temporal reasoning performance LLMSs in the zero-shot setting, this paper uses the Base Prompt:
finding proves that LLMs are highly susceptible temporal biases, and their ability reason about time relies on the specific temporal infor- mation provided the question.
Base prompt
Instruction:
Finally, larger parameter sizes generally lead to stronger temporal reasoning ability in the same series of LLMs. (i.e., LLama-7B & LLama- 13B; and OPT-6.7B & ОPT-13B). This conclusion is consistent with previous works (Zhong et al., 2021; Wei et al., 2022) that LLMs with a larger number of parameters tend to exhibit better perfor-
Get answers the question based on the context. where answers derived from substrings in context or categorized as "unanswerable"
Context: (c) Question: (q) Answer:
Scope Prompt Following the way humans answer time scope questions, we first identify the start and

---

end time specifiers of the events in the context, and then compare the time in the question with the time interval of the corresponding event, so as to achieve temporal reasoning by comparing two time scopes. The scope prompting template is as follows:
scenarios, and represents the main question of the original question. Specially, the instruction setting in the counterfactual prompt is consistent with the base prompt.
4.2 Tool Learning for Temporal Scope Factor
Tools provide domain-specific knowledge and ca- pabilities. By leveraging tools to address the weak- nesses of LLMss in tasks that go beyond the realm of pure natural language, such as arithmetic calcu- lation (Wei et al., 2023) and table-based question answering (Lei et al., 2022), we can effectively bridge the gap between language understanding and task specific requirements, enabling LLMs to excel in wider range of applications beyond tradi- tional NLP tasks. Time Comparison Tool This paper follows the REACT (Yao et al., 2022), which prompts an LLM to generate reasoning texts that break down com- plex problems into intermediate steps, and action texts that allocate NLP tools for solving these steps. One example is that LLM can make a decision based on real-time problems to call search engine and gather the latest internet information that is not present in the pre-training corpus, and return the user. Inspired by the efficacy of reasoning and acting with LLMs and NLP tools, we explore the integration time comparison tool with LLMs. In our setting, we build our time comparison tool based on the langchain framework. By compar- ing whether the event mentioned in the question falls within the temporal scope corresponding the events in the context, this approach helps LLMs understand temporal scope knowledge, as shown in Figure 8.
Scope prompt
Instruction:
Get answers for the question based on the context. If the time interval when the event mentioned in question occurred the context, the answer is the span the context. Else output "unanswerable"
Context: [e} Question: {q} Answer:
Counterfactual Prompt In this paper, propose transform the context to narrator's statement and the question enquire about the narrator's opinion this statement (Zhou al., 2023). Our method is motivated by our own cognitive process for answering different types of questions. The counterfactual prompting template is as follows:
Counterfactual prompt
Instruction: [Instruction]
Eric reads an article as follows: {c} Eric imagines
counterfactuals that {h}
Question: {4] Eric's imagination
Answer:
Rerank Prompt In real-world scenarios, numer- ical information such as years often appears in different sentences in the text. For example, the recording of events is usually in chronological or- der, and the time specifier is used to distinguish different events. Therefore, we use the year infor- mation in the sentences to reorder the chronological sequence of multiple events. The rerank prompting template as follows:
Experimental Results
As shown in Table and Table 5, our observations indicate that utilizing special prompting methods and a tool learning method for three temporal fac- tors can enhance the performance of LLMs..4
Effect of the Scope Prompt We present results in Table 4, where the section with yellow back- ground represents the effect of the scope prompt- ing method. The scope prompting method im- proves performance over LLama-7B and LLama- 13B (+1.10 and +1.41 on EM metrics). However,
Rerank prompt
Instruction:
Sort the sentences in the context by year, output the
sorted context:
Context: (c)
Sorted Context:
hhup:://itthub com/hwehase 17lagghaai
"We select LLMs with billions (LLama-7B), tens of bil- lions (L.Lama-1BB) hundreds of billions (GPT-3.5-turoo) pa- rameter scales as baselines for the proposed solutions in this paper.
all of the above prompting templates, where c denotes the context, represents the hypothetical

---

MenatQA w/scope
MenatQA
MenatQA w/counterfactual
MenatQA
Method
w/order
w/all factors
F1
EM
F1
EM
F1
EM
F1
EM
LLama-7B 7B
45.71
34.47
51.93
37.17
49.78
33.97
37.53
23.45
LLama-7Bh + Prompts
46.59
35.57
53.57
38.34
56.21
42.48
46.34
34.07
LLama-134
52.13
41.58
64.45
52.40
53.56
41.48
39.80
28.66
LLama- 13B Prompts
51.72
42.99
64.47
52.
60.55
49.90
48.37
38.98
GPT-3.5-turbo
47.34
37.78
51.20
38.38
34.69
27.66
27.99
24.45
GPT-3.5-turbo + Prompts
42.41
31.36
51.66
38.61
42.69
33.87
36.72
30.76
Table The effect of the various prompting methods, where the scope prompt, order prompt, and counterfactual prompt are represented by the background colors, Blue, Green and Red respectively, Notably, the Orange background color used to indicate simultaneous use the scope prompt, order prompt and counterfactual prompt.
MenatQA
Method
Scope Factor
All Factors
F1
EM
F1
EM
LLama-7B
45.71 (0.00)
34.47 (0.00)
37.53 (0.00)
23.45 (0.00)
LLama- prompts
46.59 (0.88)
35.57 (1.10)
46.34 (8.81)
34.07 (10.62)
LLama-7B tool + prompts
46.90 (1.19)
35.37 (0.90)
44.67 (7.14)
32.67 (9.22)
LLama-13B
52.13 (0.00)
41.58 (0.00)
39.80 (0.00)
28.66 (0.00)
LLama-138 + prompts
51.72 (-0.41)
42.99 (1.41)
48.37 (8.57)
39.98 (10.32)
LLama-13B + tool + prompts
65.85 (13.72)
55.03 (13.45)
61.06 (21.26)
51.80 (23.14)
GPT-3.5-turbo
47.34 (0.00)
37.78 (0.00)
27.99 (0.00)
24.45 (0.00)
GPT-3.5-turo prompts
42.41 (-4.93)
31.36 (-6.42)
36.72 (8.73)
30.76 (6.31)
GPT-3.5-turbo tool + prompts
47.71 (0.37)
37.58 (-0.20)
38.65 (10.66)
32.87 (8.42)
Table table shows a comparison between time comparison tool and the scope prompt on the scope factor and all factors. In brackets, the differences from scores compared to the original L.MSs.
it does not do as well on GPT-3.5-turbo, which significantly reduces the EM score (-6.42).
every word the context verbatim but rather tend to reorganize their language output, as shown A.4.
Effect of the Counterfactual Prompt Based on the results in Table 4, we can find that the counter- factual prompt exhibits the greatest improvement in LLMs compared the other methods, with average increase of 7.71 in EM score. This indicates that transforming counterfactual events into the perspective of others can effectively assist LLMs in achieving counterfactual temporal associ- ations and reasoning.
Effect of the Time Comparsion Tool One the one hand, the experimental results in Table 5 indi- cate that the time comparison tool has stronger bustness compared the scope prompting method, with similar performance on LLama-7B, and the time comparison tool does not cause drastically performance degradation on GPT-3.5-turbo, unlike the scope prompting method. Besides, the time comparison tool significantly improved the perfor- mance on LLaaa- 13B, these results demonstrate that the tool is more suitable for LLMs with larger parameters address time scope questions com- pared to the scope prompting method. On the other hand, the performance difference between L.Lama- 7B and LLama- 13B shows that LLMs with larger parameter sizes have a stronger capacity for utiliz-
Effect of the Rerank Prompt Compared to the highlighted sections with yellow background in Table 4, it can be observed that the use of the rerank prompt exhibits only minor improvement in the order factor, possibly due to the loss of information in the sorted context. We conduct an evaluation of the quality of the reordered context, and the results reveal that LLMs are not inclined to output

---

ing tools. However, the performance of 5 turbo do not improve, possibly due its incorrect understanding of the temporal feedback provided by the tool and the limited impact of the scope factor (e.g., EM metrics from 39.08 to 37.78), as shown in Table 3.
extraction and time-sensitive reinforcement learn- ing framework to improve the temporal reasoning capability of large language models. However, they only evaluated three models, T5-Large (780M), Flan-T5-Large (780M), and GPT-3.5-turbo (175B), and mainly focused on using fine-tuning to improve the time reasoning ability of LarLgage and Flan-TS- Large. Besides, the fine-tuning based improvement methods are not applicable large language mod- els, such as OPT- 175B. Our work aims to evaluate the time reasoning capability of current mainstream LLMs on three time-sensitive factors, and conducts preliminary investigations to improve the current LLMs on different time factors by designing vari- ous specific prompts and tool learning.
6 Related Work
There have been plenty of works to tackle the tem- poral reasoning task. Zhang and Choi (2021) was introduced to tackle open-domain time-sensitive question answering, with a particular emphasis on analyzing how answers differ based on extra- linguistic factors, such as the time of inquiry. Kasai et al. (2022) extended time question answering to scenarios where real-time news serves as context, and requested the model to retrieve the latest tempo- ral evidence to answer the question. StreamingQA (Liska al., 2022) introduced the first QA dataset and task for studying adaptation to new informa- tion over time in open and close-book settings with temporally non-overlapping training and evalua- tion sets. TimeQA (Chen et al., 2021) built the first dataset to investigate whether existing models can understand time-sensitive facts.
7 Conclusion
In this paper, propose question answering dataset named Multiple Sensitive Factors Time QA (MenatQA). the first dataset containing multiple time-sensitive factors that can be used as an evaluation benchmark for assessing the time un- derstanding and reasoning abilities of LLMs. We find that most LLMs fall behind smaller temporal reasoning models with different degree on three factors. Moreover, the parameter size of LLMs substantially influences their capacity for tempo- ral reasoning. LLMs also demonstrate a signifi- cant vulnerability to temporal biases and depend heavily on the precise temporal information pro- vided in questions when reasoning about time. Fi- nally, we conduct some preliminary investigations into improving the current LLMs' performance on the three temporal factors by utilizing prompting method and tool learning method, which could be potential avenues for future research.
There are a few major differences between the aforementioned works and MenatQA: : 1) MenatQA encompasses various temporal factors, such as the scope factor, order factor, and counterfactual fac- tor, involving significant amount of reasoning about implicit temporal information. This aspect of temporal reasoning ability, which is neglected by previous works, is the most important. 2) MenatQA is not only the first dataset designed specifically for evaluating the time understanding and reasoning ca- pabilities of LLMs, but also provides some simple optimization methods and baseline comparisons, which offer valuable references for evaluating the time reasoning of LLMs in the future. 3) Consid- ering the existence of hallucinations in generative models, we introduce unanswerable types to pe- nalize the illusory outputs of LLMs in MenatQA. These unanswerable type questions are impossible for humans to answer as well, and enable a genuine assessment of whether LLMs truly grasp temporal knowledge.
Limitations
The 2853 samples in MenatQA can only be used as a test set for evaluating LLMs, and the data size is not sufficient for fine-tuning the models. However, this limitation can be mitigated by utilizing previ- ous temporal reasoning datasets. The improvement solutions proposed in this paper, including the time comparison tool, scope prompt, rerank prompt, and counterfactual prompt, cannot be used a com- plete and mature framework for LLMs. Instead, they represent a preliminary investigation aimed
One concurrent work (published 15 Jun 2023) similar to ours (Tan et al., 2023), which proposed a comprehensive probing dataset TEMPREASON to evaluate the temporal reasoning capability of lan- guage models. They also proposed a temporal span
*The dataset and code are released in https: //gi thub. com/weiyi fanl 023/MenatQQ

---

improving the LLMs' performance in time rea- soning. Due to hardware limitations, we do not evaluate LLMs that require loading weights with a scale of more than 20B in the tens of billions parameter range.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lec, al. 2019. Natural questions: benehmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Acknowledgements
Fangyu Lei. Shizhu He, Xiang Li, Jun Zhao, and Kang Liu. 2022. Answering numerical reasoning ques- tions in table-text hybrid contents with graph-based encoder and tree-based decoder. arXiv preprint arXiv:2209007622
This work was supported by the National Key R&D Program of China (2022ZD0100503) and the National Natural Science Foundation of China (No.62276264). This research was also supported by Meituan. We thank Kang Liu, Yuanzhe Zhang, Yifan Wei and Xiaoyan Yu for helpful discussions of the methods used in this paper and the conclu- sions we reach from our experiments. Additionally, we thank Jun Zhao, Yisong Su, Huanhuan Ma and Fangyu Lei for feedback on writing and presen- tation of results. This work was conducted while Xiaoyan Yu and Yisong Su were an intern student at Institute of automation, Chinese academy of sci- ence.
Fangyu Lei, Xiang Li, Yifan Wei, Shizhu He, Yiming Huang, Jun Zhao, and Kang Liu. 2023. S3HQA: A three-stage approach for multi- hop text-table hybrid question answering. In Proceedings of the 6 An- nual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 1731- 1740, Toronto, Canada. Association for Computa- tional Linguistics.
Jiaxuan Li, Lang Yu, and Allyson Ettinger. 2023, Coun- terfactual reasoning: Testing language models' un- derstanding of hypothetical scenarios. arXiv preprint arXiv:2305.165:
Moxin Li. Fuli Feng. Hanwang Zhang, Xiangnan He. Fengbin Zhu, and Tat-Seng Chua. 2022. Learning to imagine: Integrating counterfactual thinking in neural discrete reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 57-69.
References
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint ar arXiv:2204.06745.
Adam Liska, Tomas Koeisky, Elena Gribovskaya, Tay- fun Terzi. Eren Sezener. Devang Agrawal, Autume Cyprien De Masson, Tim Scholtes, Manzil Zaheer, Susannah Young, et al. 2022. Streamingqa: A bench- mark for adaptation to new knowledge over time in question answering models. In International Con- ference on Machine Learning, pages 13604-13622. PMLR
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Huanhuan Ma, Weizhi Xu, Yifan Wei, Liuji Chen, Liang Wang, Qiang Liu, Shu Wu, and Liang Wang. 2023, Ex-fever: A dataset for multi-hop explainable fact verification.
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint arXiv:2211. 125.88888.
OpenAI. 2023. Gpt-4 technical report.
Wenhu Chen, Xinyi Wang, and William Yang Wang- 2021. A dataset for answering time-sensitive ques- tions. arXiv preprint 1rxiv:2108.06314
Abulhair Saparov and He He. 2022. Language models are greedy reasoners: A systematic formal analysis of shain-of-thoughe arXiv preprint arXiv:22 01240.
Gautier Izacard and Edouard Grave. 2020. Leverag- ing passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 22
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi. Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime qa: What's the answer right now? arXiv preprini arXiv:2207.15 13332.
Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. Towards benchmarking and improving temporal reasoning capability of large language models.

---

Tianyi Tang, Yushuo Chen, Yifan Du Junyi Li. Wayne Xin Zhao, and Ji- Rong Wen. 2023. Learn- ing to imagine: Visually augmentye natural language generation. arXiv preprint arXiv:2305. 16944.
A
Appendix
A.1 Data statisties
In MenatQA, the order factor be combined with other counterfactual and scope factors. Specif- ically, the scope factor type can be further classified into granularity operation, contraction operation, and expansion operation, as shown in section 2.1. We calculated the proportions of different question types under ordered and unordered contexts, as shown in Figure 3 and Figure 4. Additionally, we also calculated the proportions of answerable and unanswerable question types, and the results are shown in Figure and Figure 6.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 131.:2232..39771.
Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-ób; A 6 billion parameter autoregressive language model.
Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud. Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682
(a) Order
Yifan Wei, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, and Kang Liu. 2023. Multi-view graph representation learning for answering hybrid numerical reasoning question. arXiv preprint WxXv::3305.03488
38%
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint Xrx: 10.03629.
26%
13%
Granualrity
Contraction
Expansion
Counterfactual
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33: 13:1133-17297,
Figure order statistic
(b) Disorder
Aohan 7eng, Xiao Liu, Zhengxiao Du Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-132 1300: An open bilingual pre-trained model. arXiv preprint arxiv:2210.02414.
16%
14h
56%
Michael JQ Zhang and Funsol Choi. 2021. Situatedqa: Incorporating extra-linguistic contexts into qa. arXiv preprint arXiv:2109.0611
12%
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprini
Granualrity
Contraction
Expansion
Counterfactual
Figure 4: disorder statistic
Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. 2021. Are larger pretrained language models uniformly better? comparing performance at the instance level. arXiv preprint 2rXiv:2105. 06020.
A.2 Data annotation
We recruit college students majoring in English- related fields and adopt the quality control ap- proaches of annotator training and two-round vali- dation to ensure the quality of MenatQA.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. Context-faithful prompt- ing for large language models. arXiv preprint arXiv:2303.1132 13315.
Considering the input length limitation of LLMs, we set the maximum number of documents

---

(c) Answerable
LLama-7B, LLama-13B, and GPT-3.5-turbo the LLMs, and version 0.0. 166 of the Langchain frame- work was used to implement the Time Comparison Tool.
20%
A.3.2 Zero-Shot Setting
59%
All of our experiments were conducted under the zero-shot setting based on the base prompt, shown in Base Prompt, and all the LLMs used in experiments can be downloaded from the official website of Hugging Face.
Granualrity
Expansion
Figure 5: answerable statistic
A.3.3 Extraction and Reasoning Questions
(d) Unanswerable
In section 3.3, to validate sensitivity LLMs to various time factors, only reasoning type questions were used in the scope factor, and extraction type questions were excluded. Sepcifically, extraction type questions originate from the TimeQA easy mode version, where the time points mentioned in the questions are explicitly present in the con- text. On the other hand, reasoning type questions involve time points that cannot be directly found in the context and require inference to obtain the answers. Moreover, reasoning type questions can be further classified into granularity questions, con- traction questions, and expansion questions. These categories align with the classification in previ- ous works, and therefore, further diseussion needed.
55%
Granualrity
Contraction
Expansion
Counterfactual
Figure 6: unanswerable statistic
TimeQA to 5, which already includes the gold ev- idence. Any other documents exceeding the max- imum number will be filtered out. In our Closed Book QA setting, there is no need to set up re- triever to search for relevant documents. We en- sure that all answers come from the context which provide based on the gold paragraph field given in TimeQA's annotation documents. Taking into account the phenomenon of knowledge conflicts. we restrict the temporal scope of the questions to before 2021. This measure ensures that the con- text from Wikipedia pages appears the pretrain- ing corpus of LLMs, thereby aligning the parame- ter knowledge of LLMs with the external context knowledge.
A.3.4 Baselines Setting
Based on the Table 2, we choose the best LLMs with parameter sizes at the billion scale (e.g., LLama-7B), tens of billions scale (e.g.. LLama- 13B), and hundreds of billions scale (e.g., GPT-3.5- turbo) as baseline models to evaluate the effective- of our proposed enhancement methods (e.g., Time Comparison Tool). To ensure that the pre- dictions are consistent, we used the GPT-3.5-turbo- 0301 version of ChatGPT.
A.3.5 Parameter Setting
We generate scope factor type data using the prompt shown in Figure scope factor be further divided into three operations: granularity, contraction, and expansion.
We use InstructGPT (gpt-3.5-turbo) as the frozen LLM, with temperature set to 0.0 and nucleus sampling set to I and represents the number of chat completion options generated for each input prompt, which is 1. The hyperparameter set- tings for other LLMs the same as above. We selected the EM metric as our primary evaluation metric to measure the performance of LLMs, and report performance averaged over 3 runs.
A.3 Experimental Setup
A.3.1 Time Comparison Tool
The workflow diagram for using the time compari- son tool is shown in Figure In this paper, we used

---

Few-shot prompt
Narrow down the time scope in the question, and then update the question: Question: Which employer did Philip Alston work for from 1996 to 2001?
Updated Question: Which employer did Philip Alston work for from 1997 to 2000?
Expand the time scope of the question, and then update the question:
Question: Which employer did Philip Alston work for from 1996 to 2001?
Updated Question: Which employer did Philip Alston work for from 1994 to 2002?
Add monthly information to the time scope of the question, and then update the question:
Question: Which employer did Philip Alston work for from 1996 to 2001?
Updated Question: Which employer did Philip Alston work for from August 1996 to July 2001?
Figure 7: few-shot construction prompt
A.3.6 Baseline models
state-of-the-att models using publicly avail- able datasets exclusively, without resorting proprietary and inaccessible datasets. In particular, LLAMA-13B outperforms GPT-3 (175B) on most benchmarks.
The models used in this paper follows:
BigBird and FiD use 12 layers of encoder and decoder with 12 attention heads based on HugginFace Transformer.
ChatGPT (gpt-3.5-turbo), the most capable and cost effective model in the GPT-3.5 fam- ily is gpt-3.5-turbo which is optimized for chat but works well for traditional comple- tions tasks as well, and openai recommends using g-t-3.5-turbo over the other GPT-3.5 models because of its lower cost.
ChatGLM (6B), ChatGLM-6B is an open bilingual language model based on General Language Model (GLM) framework, with 6.2 billion parameters. ChatGLM-6B uses tech- nology similar to ChatGPT, optimized for Chi- nese QA and dialogue.
BLOOM (7.1B), BLOOM model is a large dccoder-only language model pretrained for around 350 billion tokens with an architecture similar to GPT-3.
A.4 Case Study
Compared to other LLMs, GPT-3.5-turke tends to classify questions that require temporal reasoning (where the time period is not directly mentioned in the context) unanswerable, leading to incor- rect outputs of "unanswerable". On the other hand, other LLMs such LLama are lacking in their abil- ity reject answering, making them more likely to classify the questions answerable. Therefore, the impact of the counterfactual type on LLama is smaller compared to GPT-3.5-turie
GPT-J (6B), an auto-regressive text genera- tion model trained on the Pile with 6 billion parameters.
GPT-NEOX (20B), 20 billion parameter auto-regressive language model trained on the Pile.
Here are the possible reasons we have inferred:
OPT (6.7B and 13B), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters.
I) LLMS exhibit performance degradation due to scope factors, the time involved in the questions does appear directly in the context and requires model inference derive the answer. This type is more challenging compared to extractive questions, as shown in Table 3. 2) To the best of our knowl- edge, we are the first to introduce counterfactual
LLAMA (7B and 13B). collection of foun- dation language models ranging from 7B to 65B parameters, and it is possible to train

---

Question:
Thought: need use time tool to compare temporal scopes
Who was the CEO of Twitter from May 2013 to 2020?
Time Tool
Instruction
Action: Call API
LLMs
Context
Human
Aligning Temporal Information
The answer is Jack Dorsey
Scope Q
Event1
Event2
Event3
Event4
Year A
Year B
Year C
Year
Year E
The overall process of the Time Comparison Tool. The Time Comparison Tool is used to determine whether the events in the question belong to the corresponding time range of the events in the context. Specially, Scope Q refers to the blue temporal information involved in the question, and the timeline represents the events that appear in the context and their corresponding time ranges.
Figure 8:
questions related to time, which LLMs have not encountered in their pretraining corpus, resulting in poor performance these counterfactual ques- tions. 3) Furthermore, in the order type, since the time relevant the question appears directly in the context, LLMs can directly extract the answer from the context, resulting in the least performance degradation on this factor.
A.5 Sample Introduction
In this work, unanswerable questions refer to ques- tions that cannot be inferred from the temporal in- formation in the context within given time period. As shown in Figure able question is "who was the CEO of Twitter from 2005 to March 2006?". In cases like this, where the time period mentioned the question is not addressed the context, it is not possible to infer and provide an answer. Therefore, the appropriate output would be marked as "unanswerable." The sample results of the rerank prompt are shown in Figure 9.
example of unanswer-

---

Instruction: Sort the sentences in the context by year, output the sorted context:
Context: Attaphol began his career as a player at Thai Port FC Authority of Thailand 1985. His time from 1991 to 1994 was marked by less success than in his first stay at Port Authority. From 1994 to 1996 he played for Pahang again and this time. In his first year, he won his first championship with the club. He played for the club until 1989 and 1987 also won the Queens Cup. He then moved to Malaysia for two seasons for Pahang FA, then return to Thailand to his former club.
Sorted Context:
Sorted Context:
Attaphol began his career as player at Thai Port FC Authority of Thailand in 1985.
He played for the club until 1989 and in 1987 also won the Queens Cup.
From 1991 to 1994 marked by less success than in his first stay at Port Authority.
He then moved to Malaysia for two seasons for Pahang FA, then returned to Thailand to his former club.
From 1994 to 1996, he played for Pahang again and this time, in his first year, he won his first championship with the club.
Figure 9: Rerank case on MenatQA using rerank prompt.


Reviews:
[{"review_id": "x2MX5BlcWe", "reviewer": "Reviewer_LWDP", "summary": "", "questions": "A. Time understanding and reasoning is related to the common sense of month order and math computations. I'm curious whether adding instructions regarding month order helps, and whether Chain of Thought prompting helps the model.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1**: Although the creation and analysis of the dataset are commendable, it would be beneficial if the authors could further analyze when and why the model makes mistakes, for now, it doesn't significantly advance our understanding beyond existing knowledge.\n\n- Thanks for your valuable suggestion. We agree that the analysis of \"when and why the model makes mistakes\" will enhance the overall comprehensiveness of our research. Regrettably, **it is unfeasible to provide additional analysis on the fundamental reasons behind LLMs, as it involves factors such as the structure of different LLMs, pretraining methods, and pretraining corpora**. However, we will provide preliminary results through the case study for further analysis in the revision.\n\n\n - Compared to other LLMs, GPT-3.5-turbo **tends to classify questions that require temporal reasoning** (where the time period is not directly mentioned in the context) **as unanswerable**, leading to incorrect outputs of \"unanswerable\". On the other hand, other LLMs such as LLama are **lacking in their ability to reject answering**, making them more likely to classify the questions as answerable. Therefore, the impact of the counterfactual type on LLama is smaller compared to GPT-3.5-turbo.\n\n- Here are the possible reasons we have inferred: \n    1.  LLMs exhibit performance degradation due to scope factors, **as the time involved in the questions does not appear directly in the context** and **requires model inference to derive the answer**. This type is more challenging compared to extractive questions, as shown in Table 3.  \n    2. To the best of our knowledge, we are the first to introduce **counterfactual questions related to time**, which **LLMs have not encountered in their pretraining corpus**, resulting in poor performance on these counterfactual questions. \n    3. Furthermore, in the order type, **since the time relevant to the question appears directly in the context**, LLMs can directly extract the answer from the context, resulting in the least performance degradation on this factor.\n\nTo validate the aforementioned reasoning, we can conduct additional experiments by fine-tuning the FID model  on three time-sensitive factors. This will help verify if the observed results are due to pretraining bias. To procure a more comprehensive understanding and gain advanced insights,  we will continue to investigate this matter in subsequent research endeavors.\n\n**Question A**: whether adding instructions regarding month order helps, and whether Chain of Thought prompting helps the model?\n\n1. **We believe that adding instructions regarding month order would not be effective.**  In the case of GPT-3.5-turbo, simple experiements show that it possesses commonsense knowledge regarding the order of months in a year. Consequently, we believe that adding instructions specifically addressing the month order would not yield effective results. Furthermore, in the experiments of Time Comparison Tool (Section 4.2), we used a Python script to compare the time information (Year-Month) extracted by LLMs, **ensuring the correctness of time comparison**. However, the results indicate no improvement (Table 5), which suggests that adding instructions regarding the month order provide no benefit.\n\n\n\n2. **We believe that adding CoT-based prompts will be helpful for improving temporal reasoning of LLMs**. However, considering **the limitations of context length and LLMs' input length**, our experiments were conducted **in a fair zero-shot setting**. This idea is worthy of further exploration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed dataset might be useful to inspire further studies regarding time comprehension and reasoning abilities. The experiments and analyses are comprehensive. The proposed prompt designs and including time comparison tools work well.", "reasons_to_reject": "Although the creation and analysis of the dataset are commendable, it would be beneficial if the authors could further analyze when and why the model makes mistakes, for now, it doesn't significantly advance our understanding beyond existing knowledge.", "questions_to_authors": "A. Time understanding and reasoning is related to the common sense of month order and math computations. I'm curious whether adding instructions regarding month order helps, and whether Chain of Thought prompting helps the model.", "paper_topic_and_main_contributions": "This paper presents a new dataset for testing the time comprehension and reasoning abilities of large language models (LLMs), they consider three temporal factors, i.e., scope factor, order factor, and counterfactual factor. After testing multiple models on this dataset, they find that LLMs may fall behind smaller temporal reasoning models. Also, scope factor and counterfactual factor generally impact more on LLMs, and LLMs struggle more with reasoning compared with extractions. This paper also proposes different prompting methods and a tool learning method to improve the model performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7gbTJ5WIFf", "reviewer": "Reviewer_2UnL", "summary": "", "questions": "1. TimeQA is developed for time-sensitive question answering, specifically targeting the temporal scope factor. On the other hand, this paper introduces MenatQA, which also emphasizes the scope factor. So, what are the distinctions between the two datasets?\n\n2. The authors mention that they utilize a two-round validation process to ensure the quality of MenatQA. However, the specific details regarding this two-round validation are not adequately explained.\n\n3. It would be beneficial to provide some examples of unanswerable questions.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1**: What is the rationale behind selecting these three types of factors, and how do they relate to each other?\n\n- **The rationale behind selecting these three types of factors is detailed in the 3rd paragraph in Section 1.** For your better understanding, here is the rationale behind each factors:\n  * Scope factor: time-related questions require inference of time that is not explicitly stated in the context, infering such scope is important for temporal reasoning.\n  * Order factor: the description of events typically follows a chronological order, recognizing such order is a fundamental ability concerning temporal reasoning.\n  * Counterfactual factor: questions with temporal assumptions greatly escalate the difficulty of temporal reasoning, answering such questions require counterfactual thinking of models.\n\n- They are all important factors in temporal reasoning, each fulfilling distinct roles and serving specific purposes. While **they may not necessarily be interrelated**, they collectively contribute to the overall process of temporal reasoning.\n\n**Weakness 2**: Since different prompts may result in varying performance outcomes, it is essential to discuss how to design prompts effectively.\n\n- The main contribution of the paper is to propose a new dataset to verify the temporal understanding and reasoning abilities of LLMs on three factors. We just proposed three simple prompts to improve such abilities of LLMs on the proposed dataset，which are very simple and could be regarded as the baselines for future researches. But we agree with you that designing effective prompts is very important for improving performance. We will leave it as the next research problem. Thanks for your suggestions.\n\n**Weakness 3**: The analysis of experimental results regarding the poor performance of GPT-3.5-turbo is insufficient.\n\nThanks for your valuable suggestion. We agree that the reason of the poor performance of GPT-3.5-turbo should be explored. We believe this issue is the same as mentioned by Reviewer LWDP, and **please see response to Reviewer 1**.\n\n- Regarding the reasons behind the poor temporal reasoning capabilities of LLMs, we conducted relevant experiments for analysis. For example, in the case study, we randomly sampled 100 outputs from GPT-3.5-turbo. We found that the use of scope prompts led to subpar performance of GPT-3.5-turbo. This was because **the scope prompt made GPT-3.5-turbo more inclined to consider scope type questions as unanswerable**, resulting in incorrect outputs labeled as \"unanswerable.\"\n\nWe apologize for the lack of detailed discussions in the experiments. We will provide supplements to the case study section in subsequent revisions to address this issue. Furthermore, **it is unfeasible to delve deeper into the underlying reasons due to the fact that GPT-3.5-turbo has not been open-sourced**.\n\n\n**Question 1**:  what are the distinctions between TimeQA and MenatQA ?\n\n- While both TimeQA and MenatQA are temporal reasoning datasets, they have distinct objectives. MenatQA is designed to assess the temporal reasoning ability of LLMs. \n- Compared with MenatQA, TimeQA only addressed the scope type. Specifically MenatQA focused not only on the scope type, but also on the counterfactual and order types. Even though on the scope type, MenatQA further provides more detailed categorization such as granularity, contraction, and expansion. By contrast, in TimeQA, these distinctions are not explicitly made, except for the mention of granularity (month) as a specific case.\n\n\n**Question 2**: The specific details regarding this two-round validation are not adequately explained.\n\n- The two-round validation process, as **outlined in Appendix A.2**, follows the same procedure to other works involving data annotation. To **provide a clearer understanding**, here is a general overview of the two-round validation process:\n     * Round 1: Annotators are trained on the annotation guidelines and provided with sample data, and then independently annotate a subset of the dataset based on the guidelines. A quality inspector checks for consistency, adherence to guidelines, and potential errors or ambiguities. Any discrepancies or issues are identified and discussed with the annotators. \n     * Round 2: Annotators revise and refine their annotations based on the feedback and clarifications received. The quality inspector compares the Round 1 and Round 2 annotations, looking for consistency and improvements. In case of discrepancies or disagreements, further discussions and consensus-building take place.\n\n**Question 3**: It would be beneficial to provide some examples of unanswerable questions.\n\n+ In this work, unanswerable questions refer to questions that cannot be inferred from the temporal information in the context within a given time period. \n+ As shown in Figure 1, an example of unanswerable question is \"who was the CEO of Twitter from 2005 to March 2006?\". In cases like this, where **the time period mentioned in the question is not addressed in the context, it is not possible to infer and provide an answer**. Therefore, the appropriate output would be marked as \"unanswerable.\"\n+ We will add examples of unanswerable questions to the appendix in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Evaluating the abilities of LLMs on Temporal Comprehension and Reasoning is a meaningful task.\n", "reasons_to_reject": "1. This paper aims to evaluate the performance of current LLMs on different temporal factors and select three types of factors, including cope, order, and counterfactual. What is the rationale behind selecting these three types of factors, and how do they relate to each other?\n\n2. More emphasis should be placed on prompt design. This paper introduces several prompting methods to address issues in MenatQA. Since different prompts may result in varying performance outcomes, it is essential to discuss how to design prompts effectively.\n\n3. The analysis of experimental results is insufficient. For instance, the authors only mention that the scope prompting method shows poor performance on GPT-3.5-turbo, but they do not provide any analysis of the underlying reasons behind this outcome.\n", "questions_to_authors": "1. TimeQA is developed for time-sensitive question answering, specifically targeting the temporal scope factor. On the other hand, this paper introduces MenatQA, which also emphasizes the scope factor. So, what are the distinctions between the two datasets?\n\n2. The authors mention that they utilize a two-round validation process to ensure the quality of MenatQA. However, the specific details regarding this two-round validation are not adequately explained.\n\n3. It would be beneficial to provide some examples of unanswerable questions.\n", "paper_topic_and_main_contributions": "This paper aims to assess the performance of current Language Models (LLMs) on Temporal Comprehension and Reasoning.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hBY7yZRgtW", "reviewer": "Reviewer_7Azk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness:**  Not much technical innovation. This resource paper is more suitable to be accepted into Findings.\n\n\n- In this paper, we found that existing LLMs lag behind in temporal reasoning ability. We propose the MenatQA dataset as a benchmark specifically desingned to **evaluate LLMs' temporal reasoning ability**. To the best of our knowledge, **we are the first to introduce** multiple factors for temporal reasoning of LLMs. On the **technical front**, we propose simple yet effective method that can **serve as future baselines** for evaluating temporal reasoning capabilities of LLMs.\n\n- Our technical innovation is two-fold. First, we introduce a time comparison tool and employ tool learning techniques specifically tailored for temporal reasoning tasks. The design of tools for LLMs and the instruction of LLMs on their usage represent a hot research area. Our work **not only fills the gap in tool learning for such tasks** but also **provides LLMs with a practical temporal reasoning tool that can be readily employed**. Second,  we designed prompt methods specifically tailored for temporal reasoning, targeting three different time factors. This is **the first prompt-based method for temporal reasoning tasks**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This resource paper propose a useful dataset to evaluate the temporal reasoning ability of LLMs. They conducted preliminary investigations to optimise temporal reasoning ability of LLMs, which can be used as baseline to inspire the future research. I suggest this paper should be accepted into Findings.", "reasons_to_reject": "Not much technical innovation. This resource paper is more suitable to be accepted into Findings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Temporal reasoning ability is an important ability of LLMs. Existing dataset mainly considers the temporal scope factor but neglecting other temporal reasoning abilities (factors). This paper propose a new dataset named MenatQA which is to evaluate the time comprehension and reasoning abilities of LLMs. This dataset contains three temporal factors (scope factor, order factor, counterfactual factor) with total 2,853 samples. Based on the proposed MenatQA, the conducted experiments on several mainstream models by using specific prompts and tool learning. The results demonstrate that the majority of LLMs perform poorly on the MenatQA dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new dataset, MenatQA, for testing the time comprehension and reasoning abilities of large language models (LLMs). The reviewers appreciate the comprehensive experiments and analyses, and the proposed prompt designs. However, they suggest that the authors could further analyze when and why the model makes mistakes, the rationale behind a few design choices, limitations in technical innovations. The authors' rebuttal highlights the value of the work, provides some explanation for the the experimental design, and explains some limitations, but there are still some opaqueness in the analysis and also limited novelty.
Decision: Accept-Findings