PaperBench: Evaluating AI's Ability to Replicate AI Research
Authors: name: Giulio Starace; name: Oliver Jaffe; name: Dane Sherburn; name: James Aung; name: Jun Shern Chan; name: Leon Maksin; name: Rachel Dias; name: Evan Mays; name: Benjamin Kinsella; name: Wyatt Thompson; name: Johannes Heidecke; name: Amelia Glaese; name: Tejal Patwardhan
Published: 2025-04-02 15:55:24

Abstract: We introduce PaperBench, a benchmark evaluating the ability of AI agents to
replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024
Spotlight and Oral papers from scratch, including understanding paper
contributions, developing a codebase, and successfully executing experiments.
For objective evaluation, we develop rubrics that hierarchically decompose each
replication task into smaller sub-tasks with clear grading criteria. In total,
PaperBench contains 8,316 individually gradable tasks. Rubrics are co-developed
with the author(s) of each ICML paper for accuracy and realism. To enable
scalable evaluation, we also develop an LLM-based judge to automatically grade
replication attempts against rubrics, and assess our judge's performance by
creating a separate benchmark for judges. We evaluate several frontier models
on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet
(New) with open-source scaffolding, achieves an average replication score of
21.0\%. Finally, we recruit top ML PhDs to attempt a subset of PaperBench,
finding that models do not yet outperform the human baseline. We
https://github.com/openai/preparedness{open-source our code} to
facilitate future research in understanding the AI engineering capabilities of
AI agents.