Title: Enabling LLMs to generate text with citations
Benchmark: ALCE
Summary: The paper introduces ALCE (Automatic LLMs’ Citation Evaluation), the first fully‑reproducible benchmark that evaluates how well large language‑model systems answer open questions while providing sentence‑level citations to supporting passages.  ALCE includes three citation‑focused QA datasets (ASQA, QAMPARI, ELI5), automatic metrics for fluency, factual correctness, and citation quality, and extensive experiments showing that even GPT‑4‑based systems remain citation‑incomplete roughly half the time.
Contribution: - First benchmark and codebase for end‑to‑end “answer‑with‑citations” evaluation.
- New automatic metrics (sentence‑level citation recall/precision via NLI, claim‑based correctness, MAUVE fluency) with demonstrated human correlation.
- Empirical study of prompting, retrieval, and reranking techniques, revealing limits of current LLMs and pointing to future work on better retrieval, long‑context reasoning, and multi‑source synthesis.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Answer generation with Citations
Phenomenon defined by authors: Yes
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given a natural‑language question and a large retrieval corpus, a system must retrieve passages, generate a multi‑sentence answer, and append bracketed citations after each informative statement, so that every claim is supported by the cited text.
Task item definition: One dataset row consists of (question, retrieval‑corpus); the model response is free‑form prose with inline numeric citations that refer to 100‑word corpus passages.
Task source: Modified from another benchmark (e.g. translation into another language)
Task source detail: No new questions are written; the authors re‑use/trim the dev splits of three existing QA datasets and pair them with Wikipedia or Common‑Crawl‑based corpora.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 3,000
Dataset metadata available: Yes
Dataset metadata detail: dataset name, question type, corpus, corpus size
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: Mean, and human–automatic correlation (Cohen’s Kappa coefficient) for validation.
Metric subscores: Yes
Metric subscores detail: Separate scores for fluency, correctness, citation recall, citation precision.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They test automatic scores against human judgements (Cohen’s kappa coefficient: 0.698 recall, 0.525 precision).
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval
Other notes: ALCE explicitly limits each statement to max three citations and passages are fixed‑length (≈100 words) to keep evidence concise and within LLM context windows.