Title: LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Benchmark: LongBench
Summary: LongBench is the first bilingual multi-task benchmark for long-context understanding. Benchmarking of open and closed source models suggests notable challenges for LLMs, with fine-tuning and scaled position embedding helping to improve long-context capabilities.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-context understanding
Phenomenon defined by authors: Yes
Phenomenon definition: "the ability to understand and reason over a long context"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a long context input and task instruction, produce an answer.
Task item definition: Each sample is represented in a standard format, consisting of the task input, context, ground truth answers, dataset source, language, ID, and meta-data including length and categories for classification tasks.
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt), Expert-annotated task examples (PhD students)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 4750
Dataset metadata available: Yes
Dataset metadata detail: avg length, data source, language, metric
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: context length, task type
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context