Title: Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models
Benchmark: TEMPREASON
Summary: This paper introduces TEMPREASON, a comprehensive probing dataset designed to evaluate the temporal reasoning capabilities of LLMs across three hierarchical levels (Basic, Advanced, Complex) grounded in Allen's interval algebra. The paper also proposes a novel learning framework involving temporal span extraction and time-sensitive reinforcement learning to enhance LLM temporal reasoning. Experiments show TEMPREASON is challenging for current LLMs, and the proposed framework effectively improves performance.
Contribution: Key contributions include: (1) Creating TEMPREASON, a large-scale probing dataset for temporal reasoning with questions categorized into three difficulty levels based on established temporal logic. (2) Proposing a novel two-stage framework (TempReasoning) specifically designed to improve LLM temporal reasoning. (3) Evaluating several state-of-the-art LLMs on TEMPREASON, identifying their limitations, particularly on more complex reasoning levels. (4) Demonstrating the effectiveness of the proposed TempReasoning framework in enhancing LLM performance on the benchmark.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Temporal reasoning capability of Large Language Models.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to understand and infer temporal relationships between events described in text. This includes recognizing the 13 basic temporal relations in Allen's interval algebra, composing these relations (e.g., transitivity), and performing multi-step deductive reasoning based on chains of temporal facts.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: To provide a systematic and less biased dataset for probing LLM temporal reasoning compared to previous datasets, and to facilitate the development of methods specifically aimed at improving this capability. 
Task definition: Temporal Reasoning QA: Given a textual context and a question targeting a specific temporal reasoning skill (Basic, Advanced, or Complex based on Allen's algebra), select the correct answer from multiple choices.
Task item definition: An instance includes a context passage, a question testing temporal reasoning, and several multiple-choice options, with one designated as the correct answer.
Task definition detail: Questions are designed to probe understanding of the 13 basic Allen relations, 22 advanced triple patterns, and complex deductive chains. Contexts are generally short and focused. Multiple-choice format is used.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: The dataset was created semi-automatically using templates based on the rules and compositions within Allen's interval algebra across three complexity levels. Generated QA pairs were then filtered for quality and naturalness by human crowdworkers on Amazon Mechanical Turk.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Uses controlled contexts and templated questions to isolate and test specific temporal reasoning skills derived from Allen's algebra.
Dataset size: Total: 49,818 QA pairs. Test set: 9,964 examples.
Dataset size notes: Train: 34,872 examples. Dev: 4,982 examples.
Dataset metadata available: Yes
Dataset metadata detail: Temporal Reasoning Level (Basic/Advanced/Complex), Temporal Relation/Pattern Type.
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Response format detail: The model needs to select the index of the correct answer choice.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail:  Accuracy (percentage of correctly answered multiple-choice questions).
Metric aggregation: Simple Mean
Metric statistics: Accuracy (%)
Metric subscores: Yes
Metric subscores detail: Performance reported per reasoning level (Basic, Advanced, Complex). Analysis also performed across different QA settings (closed-book, open-book, reasoning QA).
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Dataset based on established Allen's interval algebra. Systematically covers Basic, Advanced, and Complex reasoning patterns. Human filtering via MTurk ensures data quality. Empirical results align with difficulty hierarchy (Basic > Advanced > Complex), supporting construct validity.
Taxonomy: Reasoning / Temporal
New taxonomy: General Capability / Temporal Reasoning
Procedural notes: Details the 3 reasoning levels based on Allen's algebra. Describes the semi-automatic generation process with templates. Details human filtering on MTurk. Describes the TempReasoning framework (TSE model based on LUKE, TSRL policy model based on Flan-T5-XL, reward function). Lists baseline models and evaluation setups.
Other notes: The benchmark's systematic structure based on Allen's interval algebra allows for fine-grained probing of temporal reasoning skills. The proposed TempReasoning framework demonstrates a targeted approach to improving this specific capability in LLMs.