Title: BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Benchmark: BABILong
Summary: The BABILong benchmark tests language models’ ability to reason across facts distributed in extremely long documents in the reasoning setting, scattering relevant facts among less relevant natural text. The paper finds LLMs only effectively use less than 20% of the context in such settings, with reasoning complexity negatively impacting performance. Multiple methods including in-context reasoning, retrieval augmented generation, and context extension are applied to profile model capabilities in these long-context tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: language models’ ability to reason across facts distributed in extremely long documents
Phenomenon defined by authors: Yes
Phenomenon definition: "language models’ ability to reason across facts distributed in extremely long documents"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Perform one of 20 reasoning tasks (e.g., fact chaining, simple induction, deduction, counting, and handling lists/sets), generally presented in question format, given a long context with relevant and distracting articles.
Task item definition: A long-context input text, question, and the question's answer based on the input
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset metadata available: Yes
Dataset metadata detail: facts per task, relevant facts per task, reasoning task type
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric subscores detail: input length, task type, context size
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Advantages of the benchmark are compared versus existing related benchmarks based on design and correlation study, and the content of the benchmark and the relation between model performance and capability are analyzed.
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context