Title: M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models
Benchmark: M4LE
Summary: The paper introduces a comprehensive multi-range, multi-ability, multi-task, multi-domain benchmark for long context processing in LLMs. Analysis confirms LLMs struggle to handle long contexts, especially when multiple input spans are involved. Several long context methods are compared.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-context processing
Phenomenon defined by authors: Yes
Phenomenon definition: "processing instructions based on long sequences"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Identify single or multiple spans in a long context to use to respond to an instruction.
Task item definition: Each sample consists of the task description, input context, instruction, and response.
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 64800
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), Normalized score relative to GPT-3.5-Turbo-16K performance
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: context length, task type
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context