Title: Marathon: A Race Through the Realm of Long Context with Large Language Models
Benchmark: Marathon
Summary: The paper presents the Marathon benchmark to evaluate comprehension and reasoning capabilities of LLMs over long texts. Marathon is used to assess SOTA LLMs and the efficacy of several existing long-context generation strategies.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-context comprehension and reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: "the capabilities of LLMs to comprehend long contexts"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: A long context is presented with a multiple-choice question.
Task item definition: Each sample is represented as the input context, question, and options.
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1530
Dataset metadata available: Yes
Dataset metadata detail: distribution of context lengths per task
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context