Title: Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks
Benchmark: Ada-LEval
Summary: Ada-LEval presents a length-adaptable benchmark for long-context understanding capabilities of LLMs, involving challenging questions for reliable evaluation and context lengths extending to the ultra-long setting. SOTA open and closed models are evaluated to demonstrate current limitations of LLMs in such settings.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Phenomenon defined by authors: No
Phenomenon definition: Context window is a notable factor in LLM performance and is critical to handling long texts. The effectiveness of LLMs in managing long text is still open for exploration and assessment.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: 1. Take in a long text and arrange the text segments in the correct order.
2. Choose the best answer from multiple candidate answers to a question based on a given long text.
Task item definition: Not provided, but generally the task samples consist of either a question and many sample answers, or a series of texts to be rearranged (per the task definition).
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: over 80k
Dataset metadata available: Yes
Dataset metadata detail: total samples per context length, max tokens, average number of tokens
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation), instruction following rate
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: context lengths from 2k to 16k
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Comparison with traditional long-context benchmarks such as GovReport demonstrate Ada-LEval requires more overall text understanding to complete.
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context