Title: Attribute or Abstain: LLMs as Long Document Assistants
Benchmark: LAB
Summary: The authors introduce LAB, a 6‑task benchmark that evaluates whether LLMs reading single long documents can (i) answer or classify correctly, (ii) attribute each claim to explicit evidence spans, or (iii) abstain when the answer is absent. They compare five LLMs and five retrieval strategies, showing that “citation” (one‑shot answer + evidence generation) works best for large or fine‑tuned models, while post‑hoc evidence retrieval can help small models.
Contribution: - First systematic attribution benchmark in the long‑document (non‑RAG) setting.
- Curates six diverse datasets (science, law, government, Wikipedia) and adds synthetic evidence for GovReport.
- Analyses positional bias, input‑length effects, and the correlation between evidence quality and answer quality.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Attribution & abstention when answering from long documents
Phenomenon defined by authors: Yes
Phenomenon definition: "If an LLM finds the necessary information, it should provide a response and point to the evidence in the paper (attribute). If not, it should clearly communicate this (abstain). We investigate the capabilities of LLMs to fulfill these requirements, and the relation between response quality (i.e. correctness) and evidence quality (i.e. the relevance of the evidence to the response)."
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: For each instruction + long document, the model must produce either (a) a response with inline citations of evidence segment IDs, or (b) an explicit abstention.
Task item definition: A single row consists of instruction, full document text segmented, plus gold answer & gold evidence (or unanswerable flag).
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task source detail: The six component datasets are reused; GovReport evidence is added automatically with BM25; all others keep human annotations.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 13,394 (Test)
Dataset size notes: Train: 281k, and Validation: 10.7k
Dataset metadata available: Yes
Dataset metadata detail: domain, task‑type, doc length, evidence
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: ‑ Per‑metric means & confidence via single runs

‑ Spearman correlation (response/evidence vs position)

‑ Cohen’s κ for human IAA (0.74‑0.77)
Metric subscores: Yes
Metric subscores detail: per dataset, per approach, response vs evidence quality
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors double‑annotated 200 LLM outputs, achieved κ≈0.75, and used that set to pick the best attributability evaluator before large‑scale scoring.
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval