Title: L-Eval: Instituting Standardized Evaluation for Long Context Language Models
Benchmark: L-Eval
Summary: L-Eval presents a standardize evaluation suite for long-context language models consisting of 20 subtasks over long documents up to 200K tokens in length with diverse human-labeled query-response pairs. Evaluation metrics for long-context LLMs are compared for alignment with human judgment. Commercial and open-source LLMs are benchmarked.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-context capabilities
Phenomenon defined by authors: No
Phenomenon definition: N/A -- phenomenon is only defined indirectly through details of the setting for the work
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a long input context, answer a relevant question.
Task item definition: Each sample consists of an input document, potential instructions, ground truth outputs, data source, and evaluation metrics.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: LLM filtering is used for quality control.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset metadata available: Yes
Dataset metadata detail: avg tokens per input, max tokens per input, number of instructions per document, number of documents
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph), Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: data source, input length
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context