Title: LooGLE: Can Long-Context Language Models Understand Long Contexts?
Benchmark: LooGLE
Summary: The paper presents a long-context benchmark over recent (post-2022) documents with new questions in diverse domains. LooGLE assesses LLMs’ long dependency capabilities and finds poor performance even with long context window LLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-context processing and undersatnding
Phenomenon defined by authors: Yes
Phenomenon definition: enabling LLMs to "process, comprehend, or even learn from long-context textual information"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: An extremely long text paired with a task direction for a long- or short-dependency understanding task, namely summarization, timeline reordering, calculation, multiple information retrieval, comprehension and reasoning, question answering, or cloze.
Task item definition: Each task item consists of the input text, document title, QA pairs, and output.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 6448
Dataset metadata available: Yes
Dataset metadata detail: number of documents, avg # words, max # words, min # words, avg tokens, task type
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Human accuracy evaluation
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: task type, context length
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context
Procedural notes: https://github.com/bigai-nlco/LooGLE