Title: Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
Benchmark: Loong
Summary: Loong is a long-context benchmark which aims to boost the realism of long-context capability evaluation by ensuring each document is relevant to the final answer, covering a range of context lengths and tasks. Various models are assessed on the benchmark, with RAG proving poor for improving performance.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-context understanding
Phenomenon defined by authors: Yes
Phenomenon definition: "long-context understanding in real-world multi-document scenarios"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: An input is provided with a task instruction or question, which the model must answer by leveraging *all* context documents.
Task item definition: Each sample consists of a question, instruction, documents, and answer, along with meta-data regarding sample index, task type, and level.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 1600
Dataset metadata available: Yes
Dataset metadata detail: length distribution, task type, avg tokens, language
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph), Structured response (e.g. valid JSON, API call alone)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: task type, input length
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context