Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning? (2024)

Abstract: Temporal reasoning is fundamental for large language models (LLMs) to comprehend the world. Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections. In this paper, we introduce COTEMPQA, a comprehensive co-temporal Question Answering (QA) benchmark containing four co-temporal scenarios (Equal, Overlap, During, Mix) with 4,748 samples for evaluating the co-temporal comprehension and reasoning abilities of LLMs. Our extensive experiments reveal a significant gap between the performance of current LLMs and human-level reasoning on COTEMPQA tasks. Even when enhanced with Chain of Thought (CoT) methodologies, models consistently struggle with our task. In our preliminary exploration, we discovered that mathematical reasoning plays a significant role in handling co-temporal events and proposed a strategy to boost LLMs’ co-temporal reasoning from a mathematical perspective. We hope that our COTEMPQA datasets will encourage further advancements in improving the co-temporal reasoning capabilities of LLMs. Our code is available at https://github. com/zhaochen0110/Cotempqa.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13014–13033 August 11-16, 2024 ©2024 Association for Computational Linguistics
Temporal reasoning is fundamental for large language models (LLMs) to comprehend the world. Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections. In this paper, we introduce COTEMPQA, a comprehensive co-temporal Question Answering (QA) benchmark containing four co-temporal scenarios (Equal, Overlap, During, Mix) with 4,748 samples for evaluating the co-temporal comprehension and reasoning abilities of LLMs. Our extensive experiments reveal a significant gap between the performance of current LLMs and human-level reasoning on COTEMPQA tasks. Even when enhanced with Chain of Thought (CoT) methodologies, models consistently struggle with our task. In our preliminary exploration, we discovered that mathematical reasoning plays a significant role in handling co-temporal events and proposed a strategy to boost LLMs’ co-temporal reasoning from a mathematical perspective. We hope that our COTEMPQA datasets will encourage further advancements in improving the co-temporal reasoning capabilities of LLMs. Our code is available at https://github. com/zhaochen0110/Cotempqa.

heading: 1 Introduction
text: Recent advanced Large Language Models (LLMs) like GPT-4 (OpenAI, 2023) have shown impressive capabilities in understanding, generating, and reasoning about natural language (Wei et al., 2022a; Zhao et al., 2023; Chang et al., 2023). Despite their advancements, these models fall short in mastering
*Work was done during the internship at Shanghai AI lab. †Juntao Li is the Corresponding Author.
temporal reasoning (Chu et al., 2023), which is fundamental for humans to comprehend the world and distinguish daily events (Chen et al., 2021; Su et al., 2023; Tan et al., 2023), requiring a complex integration of capabilities, involving implicit arithmetic calculations (Zhu et al., 2023a), understanding logical implications (Wei et al., 2022c), and leveraging extensive world knowledge (Chu et al., 2023).
Current studies in temporal reasoning mainly focus on time-sensitive question-answering (TSQA). Chen et al. (2021) first introduced the TIMEQA datasets, constructing time-evolving facts for a given subject and formulating questions based on the specific timestamp within the evolutionary facts. TEMPLAMA (Dhingra et al., 2022) extracted structured facts from the Wikidata Knowledge Base (Vrandečić and Krötzsch, 2014) for closedbook TSQA. Furthermore, TEMPREASON (Tan et al., 2023) translated explicit temporal expressions into the implicit event information within questions, offering a more comprehensive evalu-
13014
ation framework of TSQA. Given the fact “Elon musk held the position of Tesla’s chairman from 2004 to 2018”, the models are tasked with accurately interpreting and responding to time specifiers in the questions, i.e., “Which position did Elon Musk hold in 2005?” in TIMEQA (Chen et al., 2021) or “Which position did Elon Musk held before he worked for OpenAI?” in TEMPREASON (Tan et al., 2023).
The datasets mentioned above provide a straightforward way to evaluate LLMs’ capabilities in temporal reasoning. However, as LLMs evolve, there is an urgent need to evaluate their proficiency in more realistic scenarios. As shown in Figure 1, the reality might present a more intricate and multifaceted nature, involving concurrent events and complex temporal interconnections over time (UzZaman et al., 2012). Current datasets mainly question single or isolated events and might not fully reflect the realistic temporal characteristics. Therefore, we create the Co-Temporal QA (COTEMPQA) datasets to complement existing corpora by focusing on the concurrent nature of time and cotemporal relations in real-world situations. Experiments conducted on both closed-book and openbook QA settings across 14 large language models reveal that even the advanced model GPT-4 is well below a satisfactory co-temporal reasoning performance. Specifically, GPT-4 achieves an overall score of 54.7, and the best open-source LLM is 30.1, which significantly falls behind the human performance of 92.8. We also observe that the representative reasoning enhancement strategies, e.g.,
Chain-of-Thought (CoT) (Wei et al., 2022b), fail to consistently improve and even reduce the temporal reasoning capabilities of LLMs in some scenarios.
Throughout the investigation on our COTEMPQA, we observed that mathematical reasoning plays a crucial role in handling co-temporal events. Building on this insight, we propose a simple but effective MATH-REASONING CoT (MR-COT) strategy to boost the co-temporal reasoning capability of LLMs, achieving a remarkable 10.8 point improvement over existing baselines. However, it is important to note that there remains a nonnegligible gap between the performance of our proposed MR-COT and human-level reasoning in handling complex, concurrent temporal relations. We hope our research could inspire more great works to improve the co-temporal ability of LLMs.

heading: 2 The COTEMPQA Datasets

heading: 2.1 The Taxonomy of Co-temporal Relations
text: Co-temporal relations are fundamental to understanding how events interconnect in time. These relationships highlight when facts or events happen simultaneously, which can be categorized into three distinct types (Pustejovsky et al., 2003), as shown in Table 2. Each of them represents a unique manner, whether events coincide with or overlap with each other in the temporal aspect. We divide these relations into four different scenarios below:
• Equal: Facts occur simultaneously, representing
a strict co-temporal relationship, occurring at the same time without duration differences.
• Overlap: Facts partially coincide in time. This scenario is common in real-world settings, where facts often intersect for a part of their duration.
• During: One fact is entirely contained within the timeline of another, reflecting a more complex interaction of timelines.
• Mix: A combination of the three types above. This category is particularly challenging as it involves the complexity and variability of realworld temporal relationships, necessitating a comprehensive level of co-temporal reasoning.

heading: 2.2 Structuring Temporal Facts
text: We utilize the Wikidata (Vrandečić and Krötzsch, 2014) dump of September 20, 2023 as our knowledge source for extracting time-dependent facts. Following Dhingra et al. (2022) and Tan et al. (2023), we focus on nine time-sensitive entity relations (Cheng et al., 2021; Gu et al., 2022; Qu et al., 2023) and keep a maximum of 2,000 subjects for each relation type. To structure the information, we transform the knowledge triples and qualifiers into a quintuplet format of (s, r, o, ts, te), where s is the subject, r is the relation, o is the object, ts and te are the start time and end time. We group all the temporal facts by subject, denoted as S = {(s, ri, oi, tsi , tei)|i ∈ 1 . . . N}, where N is the number of facts within a group. We keep the groups that contain three or more temporal facts.

heading: 2.3 Extracting Co-temporal Facts
text: Building on our approach to structuring timedependent facts from Wikidata, we compare the timestamps of different facts to identify overlaps. Each fact fi and its co-temporal counterpart fj are represented as a triple, with fi = {si, ri, oi} and fj = {sj , rj , oj}. S = {si, sj}, R = {ri, rj}, O = {oi, oj} are the sets of subjects, relations, and objects within the co-temporal fact pairs (fi, fj), respectively. We categorize fact pairs into five scenarios based on the consistency or variation of (S,R,O), involving (S,R,O), (S,R,O), (S,R,O), (S,R,O), (S,R,O), where an overline indicates a change in the specific set. For instance, (S,R,O) represents the scenario where the subjects and relations are constant while the objects differ between fi and fj . We exclude the scenarios (S,R,O) and (S,R,O) since it is unrealistic for
Algorithm 1 Identifying Co-temporal Facts 1: Input: Set of facts F , each fact as (s, r, o, ts, te) 2: Output: Set of co-temporal facts with their minimum
temporal units 3: function MINMAXTIME(fi, fj) 4: (si, ri, oi, tsi , tei)← fi 5: (sj , rj , oj , tsj , tej )← fj 6: start← max(tsi, tsj) 7: end← min(tei, tej) 8: Tmin ← (start, end) 9: if start ≤ end then return Tmin
10: else return None 11: end if 12: end function 13: R← empty set ▷ R is the set of co-temporal facts 14: for each fi in F do 15: for each fj in F where fi ̸= fj do 16: Tmin ← MINMAXTIME(fi, fj) 17: if Tmin is not None then 18: R← R ∪ {(fi, fj , Tmin)} 19: end if 20: end for 21: end for 22: return R
the same subject and object to have different relationships, or for the same object to have the same relationship with different subjects concurrently. The detailed illustrations are shown in Appendix B. Taken (S,R,O) as an example, we detail the extraction of co-temporal facts in the MINMAXTIME function (lines 3-11) from Algorithm 1. This framework identifies the complex co-temporal relations between events, allowing for a more intuitive understanding of how multiple events and states are interrelated in the temporal dimension.

heading: 2.4 QA Pairs Construction
text: Upon identifying co-temporal facts (fi, fj , Tmin), we construct the query Q by the condition fact fcond and the query fact fquery. fcond is selected from the intersection fact in (fi, fj), while fquery is the other fact in the pair. To control the correlation between fcond and fquery, we manually predefined 17 types of relevant relation pairs and constructed questions for the object by these question templates, which can be found in Table 11. By predefining these pairs, we align them logically and ensure the extracted facts are contextually interconnected. Based on the temporal relations identified through Tmin, we categorize the tasks into four distinct classes: Equal, Overlap, During, and Mix. Since multiple events can happen simultaneously in real life, we aggregate all valid answers for query Q to a set to avoid questions having several correct answers. As detailed in Table 3, the average number of our answers within the question is 1.42.

heading: 3 The Performance of LLMs on COTEMPQA

heading: 3.1 Experimental Setup
text: We investigate the co-temporal reasoning abilities of large language models within two problem settings: (1) Closed-Book QA (CBQA) is widely recognized task format in time-sensitive QA research (Dhingra et al., 2022; Liska et al., 2022; Tan et al., 2023). In this setting, the language model is given only the question and tasked with generating the answer without relying on external natural language texts. The primary challenge here involves the retention and temporal reasoning of knowledge pertinent to the question. (2) In the Open-Book QA (OBQA) setting, we provide all the relevant temporal facts within the group S = {(s, ri, oi, tsi , tei)|i ∈ 1...N} in a structured format directly into the prompt, which is in contrast to previous studies (Chen et al., 2021; Wei et al., 2023) that utilized Wikipedia as the knowledge base. This process shifts the evaluation’s emphasis towards the reasoning process itself, thereby minimizing the influence of the model’s inherent factual extraction capabilities on the outcomes (Tan et al., 2023; Chu et al., 2023). Here, the language models need to provide all possible answers within the concurrent timeframe.

heading: 3.2 LLMs for Evaluation
text: We perform comprehensive experiments on 14 representative large language models including (1) ChatGPT (Ouyang et al., 2022) ChatGPT is a chat model aligned through Supervised Finetuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). GPT-4 is an upgraded version of ChatGPT with enhanced reasoning capabilities, making it the most powerful LLM. Since the model is constantly updated, we used the gpt3.5-turbo-0613 and gpt4-0613 for consistent evaluation. (2) LLaMA2 (Touvron et al., 2023) LLaMA2 is one of the most popular opensource foundation models trained on 2T tokens with efficient group query attention (Ainslie et al., 2023). (3) Code-LLaMA (Roziere et al., 2023) Code-LLaMA models is a code generation model built on LLaMA2 and further trained on 500B tokens of code. (4) WizardMath (Luo et al., 2023a) WizardMath is also built on LLaMA2 and further trained on their proposed Reinforcement Learning from Evol-Instruct Feedback (RLEIF) (Xu et al., 2023) to enhance the mathematical reason-
ing abilities of LLaMA2. (5) WizardCoder (Luo et al., 2023b) WizardCoder, similar to WizardMath, adapts the RLEIF method to the domain of code. The implementation details of our experiments are shown in Appendix A.

heading: 3.3 Evaluation Metrics
text: Prior works followed the SQuAD benchmark’s evaluation protocol (Rajpurkar et al., 2016), using exact match (EM) and token-level F1 score. These metrics calculate the highest scores across all references, tending to overestimate performance in task settings involving questions with multiple possible answers. Following Zhong et al. (2022), we adopt a stricter Acc. score, where a prediction is correct only if it aligns with all the gold answers for a question. Additionally, we also evaluate our methods by answer-level F1 score (F1), which is a stricter metric compared to token-level F1 score.

heading: 3.4 Results and Analysis
text: The main results are shown in Table 4. We report human performance to serve as an upper bound. From the results, we can observe:
LLMs partially grasp co-temporal reasoning Our analysis reveals that, despite GPT-4 exhibiting the best performance among all LLMs, there is still a considerable disparity compared to human performance (54.7 vs. 92.8), indicating significant potential for further improvement in co-temporal reasoning. We also discover that models exhibit different reasoning capabilities in different co-temporal scenarios. Take GPT-4 for further illustration, in the simple co-temporal reasoning task, i.e., the Equal scenario, GPT-4 demonstrates strong performance, achieving a 92.7 score overall. However, its performance significantly declines in more complex scenarios. Specifically, in the Overlap category, GPT4’s accuracy falls to 59.4, decreasing further to 50.1 in the During category. In the most challenging category, Mix, which combines various temporal relations, GPT-4’s performance drops to 45.0. We provide a case study to explain the varying difficulties of scenarios and model performances in Appendix C. As shown in Table 7, the concurrent characteristics in the Equal scenario are relatively obvious compared with the Overlap and During scenarios. Furthermore, the Mix scenario has more than one answer and involves reasoning with multiple co-temporal relations, which makes it the most challenging compared to other scenarios.
CBQA is more challenging for LLMs LLMs exhibit significantly weaker performance in the CBQA compared to the OBQA, as reflected in the GPT-4’s performance (14.5 vs. 54.7). Interestingly, GPT-4 is outperformed by GPT-3.5 in CBQA. Our error analysis indicates that GPT-4 often responds with “uncertain" when unsure, unlike GPT-3.5, which tends to provide direct answers. This discovery is also found in previous works (OpenAI, 2023; Wei et al., 2023). This characteristic hinders GPT-4’s effectiveness in co-temporal CBQA, where precise answers are needed. While constructing our datasets, we concentrated on the top 2,000 subjects for each relationship type. These subjects are typically well-covered in pre-training stages, as Wikipedia is a significant part of their training data (Touvron et al., 2023). Despite this prior exposure, LLMs’ reduced capability in CBQA underscores the need to enhance the co-temporal reasoning abilities of LLMs, empowering them to comprehend and reason about concurrent events.
Different aspects of capability benefit cotemporal reasoning differently Notably, models specialized in mathematical reasoning (e.g., WizardMath-70B) show significant improvements in co-temporal reasoning, scoring 30.1, compared to the foundational LLaMA-70B model’s 22.2 and CodeLLaMA-34B’s 20.0. This improvement indicates a strong correlation between the skills utilized in math and those required for understanding
and interpreting complex temporal relationships. Although WizardMath is the overall best model among the baseline, we also observe its reduced effectiveness in the Mix scenario compared with others. By further investigation, questions have multiple answers in the Mix scenario. WizardMath tends to return a single response rather than enumerating all possible answers, causing higher precision but lower recall in contrast to other models (i.e., LLaMA, CodeLLaMA). We provide further experimental results and analysis in Appendix D.
In Appendix E, we provide a detailed error analysis to help understand the limitations of current models and guide future improvements in cotemporal reasoning capabilities.

heading: 3.5 Data Analysis
text: In Section 2.3, we categorize co-temporal facts into five scenarios. Building on this classification, this section delves into investigating how various types of fact elements influence LLMs’ ability to perform co-temporal reasoning. To ensure fairness in our experiments, we excluded questions with multiple answers and standardized the number of questions across all co-temporal relations. Figure 2 illustrates GPT-4’s performance with various element types. Additional results concerning different LLMs are presented in Table 10, and results consistently align with the findings shown below:
performance of each element type is bold.
The influence of triple element types As observed in Figure 2a, the complexity of co-temporal reasoning for models increases with the number of changing elements. Among the scenarios, (S,R,O), (S,R,O) are particularly challenging compared to others. It indicates that LLMs encounter significant challenges when dealing with scenarios of high complexity, where multiple elements undergo simultaneous changes. The analysis below further investigates which elements present the most significant challenges to co-temporal reasoning capabilities.
The influence of triplet element types In the left part of Figure 2b, we observe a notable decline in the model’s performance, i.e., 22.5 point decrease in Acc. and 24.4 in F1 when it engages in reasoning involving multiple subjects. These findings highlight models’ inherent difficulty when processing information from multiple concurrent subjects. While the reasoning process for handling multiple subjects shares similarities with single-subject sce-
narios, real-world situations are inherently more complex and variable. The model is required to integrate information across different subjects and understand complex relationships that extend beyond a single domain or context. On the other hand, in the right part of Figure 2b, we discover that the difference in the fact relation does not significantly impact the abilities of LLMs in co-temporal reasoning. This is attributed to the fact that relationship changes are not as complex as those involving multiple subjects in real-world scenarios (Huang and Chang, 2023), making them less challenging for the models’ capabilities.

heading: 4 Making Language Models Better COTEMPQA Responders
text: Previous research has demonstrated that the Chainof-Thought (CoT) enables models to process complex reasoning tasks, such as mathematical and logical reasoning, in a step-by-step manner (Wei et al., 2022b). Motivated by this, we explore the application of CoT prompting to improve the capa-
Question While Klaus Fuchs was working for Atomic Energy Research Establishment, which employer did Alexander Oppenheim work for during the identical time period?
Context
Alexander Oppenheim works for National University of Singapore from 1949 to 1965. Alexander Oppenheim works for University of Malaya from 1949 to 1965. Klaus Fuchs works for Atomic Energy Research Establishment from 1946 to 1950. ......
Gold Answer National University of Singapore and University of Malaya GPT-4 Answer: National University of Singapore
+ CoT
Reasoning: Klaus Fuchs worked for Atomic Energy Research Establishment from 1946 to 1950. Currently, Alexander Oppenheim worked for University of Ghana from 1968 to 1973. However, this does not overlap with the time Klaus Fuchs was at the Atomic Energy Research Establishment. Answer: no information
+ Fs Answer: University of Malaya
+ Fs&CoT
Reasoning: Klaus Fuchs worked for Atomic Energy Research Establishment from 1946 to 1950. During this same period, Alexander Oppenheim worked for National University of Singapore, as his tenure with the university spanned from 1949 to 1965. Answer: National University of Singapore
+ Fs&Mr-CoT The datetime can be formed as ((1946, None, None), (1950, None, None)). The content provided and related to the question can be structured as: (University of Malaya, (1949, None, None), (1965, None, None)) (National University of Singapore, (1949, None, None), (1965, None, None)) Given the ((1946, None, None), (1950, None, None)), compared with all contents related, we find that [(1949,None,None)− (1965,None,None)] ∩ [(1946,None,None)− (1950,None,None)] ̸= ∅ Reasoning: Klaus Fuchs worked for Atomic Energy Research Establishment from 1946 to 1950. Answer: National University of Singapore and University of Malaya
Table 6: Example inputs and outputs of GPT-4 with Zero-shot CoT (COT) prompting, Few-shot (FS) prompting, Few-shot CoT (FS&COT) prompting and Few-Shot Mr-CoT (FS&MR-COT). Answers highlighted in blue are correct, whereas thoses marked in red are incorrect.
bilities of co-temporal reasoning in this section.

heading: 4.1 Math-reasoning CoT (MR-COT)
text: As indicated in Table 4, our research uncovers a mathematically oriented reasoning framework that can enrich the LLMs’ understanding and handling of co-temporal reasoning tasks. Specifically, the WizardMath-70B model scores 30.1 overall, notably higher than the foundational LLaMA70B model’s score of 22.2. In light of this finding, we propose a MATH-REASONING (MR-COT) instruction-based prompting, which can be used together with in-context learning and chain-ofthought prompting. As demonstrated in the bottom of Table 6, our framework consists of three steps: (1) establish the key datetime, (2) structure the relevant timeline, and (3) mathematically identify the overlap. This prompt aims to guide the LLMs towards approaching temporal reasoning problems through a mathematical perspective, aligning their problem-solving processes more closely with mathematical logic and principles.

heading: 4.2 Experimental Setup
text: We launch experiments under both zero-shot and few-shot settings. In the zero-shot CoT scenario, we use Let’s think step by step (Kojima et al., 2022)
after questions as the reasoning trigger. In contrast, the few-shot setting provides the model with several question-answer pairs as initial demonstrations. Specifically, for the few-shot CoT scenario, we manually create rationales for each task, which are used as demonstrations to guide the model in step-by-step reasoning. Further details on the instructions and demonstrations are available from Figure 3 to Figure 12 in Appendix F.

heading: 4.3 Results and Analysis
text: The results are presented in Table 5, and the output of GPT-4 to a range of prompts under different settings are shown in Table 6. From these tables, we can discover the following insights:
Inconsistency in the impact of existing CoT prompts on GPT-4 In the zero-shot scenario, improvements were inconsistent, with a notable 5.5 performance increase in the Mix task and a 3.8 decrease in the Equal task under the OBQA setting. This suggests that the impact of CoT prompts varies significantly based on the task type. Moreover, GPT-4 demonstrates an overall decline in performance on both CBQA and OBQA when complemented with CoT. In the few-shot scenario, while overall improvements exist due to CoT prompts,
these are relatively modest, amounting to an average performance enhancement of 0.8 in OBQA. All results indicate that while existing CoT prompts can be beneficial, their effectiveness is nuanced and task-dependent.
Superiority of our proposed MR-COT Our method demonstrates significant superiority over existing reasoning enhancement strategies. Notably, MR-COT significantly enhances performance on the more challenging tasks, yielding improvements of 14.6, 11.4, and 13.5 on the tasks Overlap, During, and Mix, respectively in the OBQA setting. In the closed-book scenario, which is typically more challenging to improve, our method still achieves a 1.3 enhancement. However, it is observed that our method has a moderate effect on the Equal setting. We hypothesize that this is because this task is simple enough and does not require the additional complexity of mathematical reasoning. In such cases, this added complexity could be counterproductive. Despite these advancements, there is still a considerable gap compared to human-level reasoning, indicating the need for more effective methods to improve the model’s co-temporal reasoning abilities.

heading: 5 Related Work

heading: 5.1 Temporal Reasoning Benchmarks
text: Temporal reasoning in natural language processing has seen significant advancements over the years. Early benchmarks, such as TimeBank (PUSTEJOVSKY, 2003), and TempEval-3 (UzZaman et al., 2012), lay the foundational work in this domain. They primarily focused on understanding temporal relationships between events in text, offering a preliminary framework for analyzing time in language models. However, recent years have witnessed a significant surge in developing time-sensitive question-answering datasets. These newer datasets, including MC-TACO (Zhou et al., 2019), SituatedQA (Zhang and Choi, 2021), TimeQA (Chen et al., 2021), TempLAMA (Dhingra et al., 2022), StreamingQA (Liska et al., 2022), RealtimeQA (Kasai et al., 2022), TempREASON (Tan et al., 2023) and Menatqa (Wei et al., 2023), represent a more nuanced approach to temporal reasoning. These datasets challenge models to answer questions grounded in specific times or events, thereby testing the models’ ability to comprehend and reason with temporal information more dynamically. The introduction of bench-
marks such as TRAM (Wang and Zhao, 2023) and TimeBench (Chu et al., 2023) marks a significant advancement, providing crucial platforms for temporal reasoning research. Despite these advancements, there has been a noticeable gap in exploring the concurrent nature of temporal events. Previous research has primarily focused on individual events or sequences of events in isolation, overlooking the complexity of scenarios where multiple events cooccur or interact over the same period. Our work aims to fill this gap by being the first to explore the concurrent nature of temporal events.

heading: 5.2 Temporal Reasoning over LLMs
text: To enhance the temporal reasoning capabilities of language models, previous methods either rely heavily on knowledge graphs to rank entities that satisfy the time-related queries (Han et al., 2021; Mavromatis et al., 2022; Liang et al., 2022; Chen et al., 2023; Huang et al., 2024) or are strictly dependent on the continual pre-training to strengthen models’ abilities in certain temporal aspects (Su et al., 2022; Tan et al., 2023; Yuan et al., 2023). The evolution of LLMs has demonstrated impressive ability in complex reasoning tasks (Chen, 2023), such as mathematical reasoning (Mishra et al., 2022) and logic reasoning (liu et al., 2023). In light of these advancements, recent methods shift towards a program-aided approach (Gao et al., 2023) to improve the performance of time-sensitive tasks, employing Python code as an intermediate logical step instead of natural language (Li et al., 2023). This method, while effective, relies heavily on external tools (Zhu et al., 2023b) and does not fully leverage the inherent capabilities of LLMs (Brown et al., 2020). The results reveal that existing LLMs, even with advanced strategies like Chain of Thought (Wei et al., 2022b), demonstrate limited efficacy in addressing the complexities inherent in co-temporal reasoning tasks. Meanwhile, our research highlights the significant role of mathematical abilities in co-temporal reasoning, offering a direction for future methodologies.

heading: 6 Potential impact
text: We believe that it is crucial to enable LLMs to understand events that occur simultaneously or overlap in time. The potential impact of improved cotemporal reasoning on downstream applications includes the following:
• Question Answering: Advanced co-temporal
reasoning capabilities enable models to better handle questions involving concurrent events, leading to more accurate responses that reflect a deeper understanding of temporal relationships between events.
• Event Understanding: Enhanced co-temporal reasoning improves the model’s ability to comprehend concurrent events, leading to more accurate identification of event relationships and temporal sequences.
• Timeline Generation: Improved co-temporal reasoning facilitates more precise timeline generation by better understanding the relationships between events, resulting in more coherent and accurate timelines.

heading: 7 Conclusion
text: In this paper, we propose the COTEMPQA datasets to facilitate the investigation of under-explored cotemporal reasoning problems for large language models. Extensive experiments have shown a significant gap between existing advanced LLMs and human-level performance, even with the enhancement of reasoning approaches. We also discover that mathematical reasoning is crucial for understanding co-temporal events and propose a mathbased strategy to improve LLMs’ co-temporal reasoning. Reasoning on concurrent and intricate temporal relations remains an open research question, and we hope more enhancement to develop upon our COTEMPQA datasets.
Limitations
There are still some limitations in our work, which are listed below:
• While the COTEMPQA is comprehensive, with 4,748 samples, a larger dataset could provide more robust evaluation and training opportunities. As our dataset construction pipeline is adaptable to more data sources besides Wikidata, we will continuously expand our dataset and explore model training in future versions.
• For our open-book QA setting, we directly provide the subject’s relevant facts in a structured format in the prompt. Recent work shows that LLM’s performance in context-based reasoning was significantly weaker than in the former (Chu et al., 2023). In the future, we will employ some retrieval tools to construct prompts with more contextually rich information sources.
• We evaluate the co-temporal reasoning capabilities from the perspective of task performance. However, a more direct approach could involve analyzing how the model’s neurons and hidden states are triggered (Zhang et al., 2023). This limitation is not unique to our study and is common in most evaluations of Large Language Models.

heading: Acknowledgement
text: We want to thank all the anonymous reviewers for their valuable comments. This work was supported by the National Science Foundation of China (NSFC No. 62206194), the Priority Academic Program Development of Jiangsu Higher Education Institutions, the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), and Supercomputing Center in Yancheng, Grant No. 20231001.

heading: B The Details of Co-temporal Extraction
text: Building on our approach to structuring timedependent facts from Wikidata, we delve into extracting co-temporal facts by identifying overlaps in the timestamps of different facts. Specifically, we compare a given fact fi = (si, ri, oi, tsi , tei) with another fact fj in five distinct scenarios:
B.1 Scenario 1: (S,R,O) Original Fact: fi as defined above. Compared Fact: fj = (si, ri, oj , tsj , tej ). Explanation: In this scenario, the subject si and relation ri remain constant, indicating the same subject in the same type of relationship. However, the object changes, where the subject is related to different objects during co-temporal periods. Template: While <subject1> was holding the position of <object1>, which position did <subject1> hold during the same time span?
B.2 Scenario 2: (S,R,O) Original Fact: fi as above. Compared Fact: fj = (si, rj , oj , tsj , tej ). Explanation: Here, the subject si stays constant while the relation and the object change. This scenario is crucial for identifying instances where a single subject is involved in different relationships with different objects concurrently. Template: While <subject1> was holding the position of <object1>, which political party did <subject1> belong to simultaneously?
B.3 Scenario 3: (S,R,O) Original Fact: fi as above. Compared Fact: fj = (sj , ri, oi, tsj , tej ). Explanation: This scenario reflects cases where the relationship and object remain constant, but the subject changes. It suggests different subjects simultaneously having the same type of
1https://platform.openai.com/
relationship with the same object. Template: While <subject1> was holding the position of <object1>, who also held the position of <object1> concurrently?
B.4 Scenario 4: (S,R,O) Original Fact: fi as above. Compared Fact: fj = (sj , ri, oj , tsj , tej ). Explanation: Only the relationship remains constant in this case, while both the subject and the object change. This scenario signifies instances where different subjects share a common relationship with different objects concurrently. Template: While <subject1> was playing for <object1>, which team did <subject2> play for within the same time interval?
B.5 Scenario 5: (S,R,O) Original Fact: fi as above. Compared Fact: fj = (sj , rj , oj , tsj , tej ). Explanation: This scenario represents completely distinct facts that overlap in time, with all quintuplet elements changing. Template: While <subject1> was holding the position of <object1>, which employer did <subject2> work for during the same time?

heading: C Case Study
text: Table 4 indicates that existing LLMs can effectively reason about straightforward concurrent events. However, they encounter difficulties in more complex tasks that require a deeper understanding and comprehension of co-temporal reasoning. In this section, we provide a further case study to show this difference. As shown in Table 7, the Equal scenario is more accessible for LLMs as their cotemporal time interval entirely overlap. Overlap and During scenarios present intricate temporal intersections, necessitating more implicit reasoning to understand the co-temporal relationships. It becomes more challenging to determine whether one time period intersects another (i.e., During and Overlap) compared to the straightforward identification in the Equal scenario. Additionally, the Mix scenario has several correct answers and contains various co-temporal relationships, which makes it the most challenging compared to other scenarios.

heading: D Further Analysis for the Mix Scenario
text: In this section, we provide further analysis for WizardMath’s reduced effectiveness in the Mix scenario by the case and experimental results. As shown in Table 8, CodeLLaMA and LLaMA prefer to provide all potential answers, but WizardMath only returns a signal alternative answer. WizardMath is trained to return the answer with the highest probability since preciseness and accuracy are required in mathematical reasoning (Lu et al., 2023). As questions have more than one correct answers in the Mix scenario, WizardMath score the highest precision (47.5%) and the lowest Recall (28.8%), leading to underperformance in this scenario.

heading: E Error Analysis
text: To better understand the mistakes made by models, we focused our investigation on the responses generated by GPT-4 under 0-shot CoT. We divide the errors into three categories:
• Incomplete answer errors refer to situations where the questions have multiple correct answers, but failing to return all of them.
• Uncertainty errors represent the models’ inability to extract the co-temporal relation from the context provided and refuse to response the question.
• Incorrect answer errors are characterized by the model cannot return the correct answers, which means the models are insufficient in cotemporal reasoning.
Our case-wise error analysis is shown in Table 9, “uncertainty errors” are the most frequent error type, accounting for 43.14%. We assume that the GPT-4 tends to provide relatively conservative responses and only returns answers when there is a certain level of confidence (Cheng et al., 2024). Future research needs to optimize the model’s framework and further enhance the capabilities of LLMs in co-temporal understanding and reasoning.

heading: F Prompts
text: The prompts and demonstrations can be found from Figure 3 to Figure 12.
W ik
iD at
a ID
K B
R el
at io
n Pa
ir s
# Q
ue ri
es Te
m pl
at e
P1 02
-P 10
2 po
lit ic
al pa
rt y
& po
lit ic
al pa
rt y
47 5
W hi
le <s
ub je
ct >
w as
a m
em be
ro f<
ob je
ct >,
w hi
ch po
lit ic
al pa
rt y
di d
<s ub
je ct
> be
lo ng
to w
ith in
th e
sa m
e tim
e in
te rv
al ?
P3 9-
P3 9
po si
tio n
he ld
& po
si tio
n he
ld 1,
01 7
W hi
le <s
ub je
ct >
w as
ho ld
in g
th e
po si
tio n
of <o
bj ec
t> ,w
hi ch
po si
tio n
di d
<s ub
je ct
> ho
ld du
ri ng
th e
sa m
e tim
e sp
an ?
P1 08
-P 10
8 em
pl oy
er &
em pl
oy er
76 8
W hi
le <s
ub je
ct >
w as
w or
ki ng
fo r<
ob je
ct >,
w hi
ch em
pl oy
er di
d <s
ub je
ct >
w or
k fo
rd ur
in g
th e
sa m
e tim
e pe
ri od ? P5 4- P5 4 m em be ro fs po rt s te am & m em be ro fs po rt s te am 20 4 W hi le <s ub je ct > w as pl ay in g fo r< ob je ct >, w hi ch te am di d <s ub je ct > pl ay fo ra tt he sa m e tim e? P6 9- P6 9 ed uc at ed at & ed uc at ed at 25 8 W hi le <s ub je ct > at te nd ed <o bj ec t> ,w hi ch sc ho ol w as <s ub je ct > at te nd in g du ri ng th e id en tic al tim e pe ri od ? P1 27 -P 12 7 ow ne d by & ow ne d by 75 W hi le <s ub je ct > w as ow ne d by <o bj ec t> ,w ho w as th e ow ne ro f< su bj ec t> co nc ur re nt ly ? P1 02 -P 39 po lit ic al pa rt y & po si tio n he ld 11 7 W hi le <s ub je ct > w as a m em be ro f< ob je ct >, w hi ch po si tio n di d <s ub je ct > ho ld si m ul ta ne ou sl y? P1 02 -P 10 8 po lit ic al pa rt y & em pl oy er 10 1 W hi le <s ub je ct > w as a m em be ro f< ob je ct >, w hi ch em pl oy er di d <s ub je ct > w or k fo rd ur in g th e sa m e tim e sp an ? P1 02 -P 69 po lit ic al pa rt y & ed uc at ed at 74 W hi le <s ub je ct > w as a m em be ro f< ob je ct >, w hi ch sc ho ol w as <s ub je ct > at te nd in g w ith in th e sa m e tim e in te rv al ? P3 9- P1 02 po si tio n he ld & po lit ic al pa rt y 42 0 W hi le <s ub je ct > w as ho ld in g th e po si tio n of <o bj ec t> ,w hi ch po lit ic al pa rt y di d <s ub je ct > be lo ng to du ri ng th e
sa m
e tim
e pe
ri od ? P3 9- P1 08 po si tio n he ld & em pl oy er 38 0 W hi le <s ub je ct > w as ho ld in g th e po si tio n of <o bj ec t> ,w hi ch em pl oy er di d <s ub je ct > w or k fo ra tt he sa m e tim e? P1 08 -P 39 em pl oy er & po si tio n he ld 12 5 W hi le <s ub je ct > w as w or ki ng fo r< ob je ct >, w hi ch po si tio n di d <s ub je ct > ho ld du ri ng th e id en tic al tim e pe ri od ? P1 08 -P 69 em pl oy er & ed uc at ed at 24 1 W hi le <s ub je ct > w as w or ki ng fo r< ob je ct >, w hi ch sc ho ol w as <s ub je ct > at te nd in g co nc ur re nt ly ? P5 4- P6 9 m em be ro fs po rt s te am & ed uc at ed at 77 W hi le <s ub je ct > w as pl ay in g fo r< ob je ct >, w hi ch sc ho ol w as <s ub je ct > at te nd in g si m ul ta ne ou sl y? P6 9- P1 02 ed uc at ed at & po lit ic al pa rt y 18 7 W hi le <s ub je ct > at te nd ed <o bj ec t> ,w hi ch po lit ic al pa rt y di d <s ub je ct > be lo ng to du ri ng th e sa m e tim e sp an ? P6 9- P3 9 ed uc at ed at & po si tio n he ld 95 W hi le <s ub je ct > at te nd ed <o bj ec t> ,w hi ch po si tio n di d <s ub je ct > ho ld w ith in th e sa m e tim e in te rv al ? P6 9- P1 08 ed uc at ed at & em pl oy er 13 4 W hi le <s ub je ct > at te nd ed <o bj ec t> ,w hi ch em pl oy er di d <s ub je ct > w or k fo rd ur in g th e sa m e tim e pe ri od ?
Ta bl
e 11
:T em
pl at
es us
ed fo
rc on
ve rt
in g
W ik
id at
a fa
ct s
in to
na tu
ra lq
ue st
io ns
.W e
m an
ua lly
pr ed
efi ne
17 ty
pe s
of K
B re
la tio
n pa
ir s
an d
en su
re th
es e
re la
tio ns
ar e
in te
rc on
ne ct ed lo gi ca lly an d co nt ex tu al ly .T ak in g th e “p os iti on he ld & em pl oy er ” re la tio n pa ir as an ex am pl e, un de rs ta nd in g th e ov er la p be tw ee n th e pe rio d w he n a pe rs on he ld a sp ec ifi c po si tio n an d th ei re m pl oy m en ta tt he sa m e or ga ni za tio n ca n pr ov id e va lu ab le in si gh ts in to ca re er pa tte rn s.

heading: Future Work
text: Future research needs to optimize the model’s framework and further enhance the capabilities of LLMs in co-temporal understanding and reasoning