Title: TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models
Benchmark: TIMEBENCH
Summary: TIMEBENCH is a comprehensive multi-task benchmark for evaluating large language models' temporal reasoning capabilities across symbolic, commonsense, and event-level reasoning. The benchmark evaluates various LLMs under different prompting conditions, revealing significant performance gaps compared to humans while providing detailed analyses of errors and scaling behaviors.
Contribution: This work unifies previously scattered temporal reasoning datasets under a coherent taxonomic framework with standardized evaluation metrics. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: temporal reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: TIMEBENCH focuses on a comprehensive evaluation of the temporal reasoning capabilities of large language models in challenging and complex scenarios. To achieve this goal, we summarize the difficulties and challenges faced in temporal reasoning, categorize them into three levels, and integrate diverse task formats to better align with the intricate nature of temporal reasoning  (Sec 2.1)
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: symbolic, commonsense, event relations
Task definition: The model is required to use temporal reasoning to assign the correct entailment label (premise-hypothesis pair), extract the correct short answer span, and identify all correct options from a 4-choice multi-select question, or generate sentences with temporal-keywords.
Task item definition: Context (+ temporal keywords) + question/premise (+ answer options) to predict answers. 
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task source detail: Examples are filtered and resampled while some keeping some filtered synthetic generation.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 19,000
Dataset size detail: 3 major categories, 10 tasks, 15 subtasks within the 19,000 instances.
Approximately 11k symbolic, 3.1K commonsense, 4.9k event relevant instances.
Dataset metadata available: Yes
Dataset metadata detail: Categories (symbolic/commonsense/event), type(free-form reading comprehension, natural language inference, generation, multi-select questions), human score
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: Mean, percentage
Metric subscores: Yes
Metric subscores detail: Subtask, Categories
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Analysis shortcomings (implicit reasoning, symbolic arithmetic)
Taxonomy: Reasoning / Temporal
New taxonomy: General Capability / Temporal Reasoning