Title: CiteME: Can Language Models Accurately Cite Scientific Claims?
Benchmark: CiteME
Summary: With thousands of new scientific papers published monthly, staying updated and accurately attributing claims is challenging. The CiteME benchmark evaluates the ability of large language models (LLMs) to identify cited papers in text excerpts from recent machine learning papers, highlighting a significant gap between human performance (69.7% accuracy) and LLMs (4.2-18.5% accuracy). Introducing CiteAgent, an autonomous system built on GPT-4o that searches and reads papers, bridges this gap by achieving 35.3% accuracy, moving towards better automatic verification of claims made by LMs.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Identify referenced papers in academic articles
Phenomenon defined by authors: Yes
Phenomenon definition: Abilities of LMs to correctly attribute scientific claims
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Read text excerpts that reference a single other paper, identify the referenced paper
Task item definition: excerpt, target paper title, target paper url, source paper title, source paper url, year, answer
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 130
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: no commands, search only, search and read
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Agents / 
New taxonomy: General Capability / Factuality