Title: TempTabQA: Temporal Question Answering for Semi-Structured Tables
Benchmark: TempTabQA
Summary: This paper introduces TempTabQA, a new dataset designed for evaluating temporal question answering capabilities on semi-structured Wikipedia Infobox tables. The dataset includes over 11k QA pairs covering more than 90 domains. Experiments demonstrate that state-of-the-art models, including large language models, significantly underperform compared to humans, indicating the benchmark's difficulty and its potential to drive improvements in temporal reasoning.
Contribution: The main contributions are: (1) Defining the novel task of temporal QA over semi-structured tables. (2) Creating and releasing TempTabQA, a large, human-verified dataset specifically for this task, featuring diverse domains and complex temporal reasoning requirements. (3) Providing detailed analysis of the dataset's temporal reasoning challenges. (4) Benchmarking SOTA models (fine-tuned and LLMs via zero/few-shot prompting) and highlighting their limitations on this task.   

Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Temporal reasoning; Question answering over semi-structured tables with temporal constraints.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to answer natural language questions that require understanding and reasoning about temporal aspects (like dates, durations, ordering, implicit time references) based on information contained within semi-structured tables, such as Wikipedia Infoboxes.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: The benchmark was created to address the lack of focus on complex temporal reasoning in existing table QA datasets and to provide a challenging testbed for improving models' temporal understanding capabilities.
Task definition: Given a semi-structured table (e.g., Wikipedia Infobox) containing temporal information and a natural language question requiring temporal reasoning over the table's content, the task is to generate the correct answer.
Task item definition: An instance consists of a semi-structured Wikipedia Infobox table, a temporal question related to the table, and the ground-truth answer (usually a short phrase or number).
Task definition detail: Questions frequently involve mathematical operations on temporal concepts (e.g., calculating durations, counting events in a period) and require understanding implicit time references. The dataset spans over 90 distinct domains.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task source detail: Tables were selected from Wikipedia Infoboxes across >90 categories. MTurk workers drafted initial QA pairs following guidelines to ensure temporal complexity and diversity. Data was subsequently filtered and validated by expert NLP annotators.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The task models users querying structured summaries (Infoboxes) for specific information, often requiring temporal understanding.
Dataset size: Total Test set size = 2889 QA pairs (Head Test: 1851, Tail Test: 1038)
Dataset size notes: Train set: 7680 QA pairs. Dev set: 885 QA pairs.
Dataset metadata available: Yes
Dataset metadata detail: Table Domain/Category, Data Split (Train/Dev/Head Test/Tail Test).
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Response format detail: Answers are brief, often numerical or temporal values, either extracted or calculated from the table data.
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Meteor
Metric definition detail: F1 score, Exact Match (EM), ROUGE-1 (R1), ROUGE-2 (R2), and Meteor (MET)
Metric aggregation: Simple Mean
Metric statistics: F1, EM, R1, R2, MET
Metric subscores: Yes
Metric subscores detail: Results are reported separately for the Head Test and Tail Test sets. Performance is also broken down by question type (Wh-word), reasoning operation, implicit/explicit nature, and answer entity type. Category-specific analysis is also performed.
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Focused on tables with temporal data. 
Used MTurk with specific guidelines for complex temporal questions and linguistic diversity, including bias mitigation steps. 
Validation by 3 expert annotators per item in dev/test sets achieved high majority agreement (91-93%) and estimated human accuracy (~86%). 
Detailed statistical analysis of question complexity, temporal intervals, required operations, and answer types provided. 
Non-temporal/trivial questions were filtered.   
Taxonomy: Language Modelling / Updating
New taxonomy: General Application / Updating
Procedural notes: Details provided on crowdsourcing via MTurk (batches, payment, qualification, bonuses, quality control). 
The validation process using expert annotators is detailed. 
The Data filtering steps are described. 
The Table linearization method for models is explained. 
The Fine-tuning hyperparameters are listed.
Other notes: This work specifically targets the underexplored area of temporal reasoning over semi-structured tables. It demonstrates that even advanced LLMs struggle significantly with the complex temporal and numerical reasoning required, especially compared to human performance. The Head/Tail split provides insights into generalization capabilities.