Title: Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding
Benchmark: TCELongBench
Summary: TCELongBench assess LLMs’ ability to leverage temporal dynamics when understanding extensive texts. Experiments find that retrieval augmented generation and long-context modeling are fairly effective to handle such tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: complex event analysis, handling temporal dynamics, understanding extensive text
Phenomenon defined by authors: Yes
Phenomenon definition: "[Temporally complex events] consist of semantically related articles that together narrate the development of various entities over time... a TCE may span tens of news articles and then tens of thousands of tokens"
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The task is defined in three specific QA settings:
1. Finding and understanding evidence across numerous articles
2. Understanding the order of temporal sequences
3. Predicting future events based on historical data
Task item definition: Each sample consists of the following fields: question, answer choices, answer, ground truth, and shuffled answer choices, along with meta-data concerning sample ID and the sample generation process.
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 13124
Dataset size notes: Train: 63050; Validation: 13334
Dataset metadata available: Yes
Dataset metadata detail: question types, token counts, temporal duration
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: Metrics for the three different subtasks are provided, as well as results according to input length and input position.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context