Title: Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?
Benchmark: CoTempQA
Summary: This paper addresses the task of reasoning across intricate temporal interconnections and introduces CoTempQA as a comprehensive co-temporal question answering benchmark. Current LLMs exhibit significant deficiencies versus humans in co-temporal comprehension and reasoning, even with Chain of Thought. Mathematical reasoning is found to play a notable role in handling co-temporal events, and a strategy to boost co-temporal reasoning in LLMs which leverages this insight is proposed.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: co-temporal comprehension and reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: Temporal reasoning "is fundamental for humans to comprehend the world and distinguish daily events, requiring a complex integration of capabilities, involving implicit arithmetic calculations, understanding logical implications, and leveraging extensive world knowledge." Yet "reality might present a more intricate and multifaceted nature, involving concurrent events and complex temporal interconnections over time." Co-temporal reasoning focuses on "the concurrent nature of time and co-temporal relations in real-world situations".
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: 1. Take a question and generate the answer without relying on external texts.
2. Take a question and relevant temporal facts and generate the answer.
Task item definition: Each sample consists of the context, question, and target answer.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), Wikidata
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 4748
Dataset metadata available: Yes
Dataset metadata detail: # questions per mode, # subjects per mode, average number of facts per mode, average number of answers per mode
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: temporal mode
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Temporal
New taxonomy: General Capability / Temporal Reasoning