EXP-Bench: Can AI Conduct AI Research Experiments?
Authors: name: Patrick Tser Jern Kon, name: Jiachen Liu, name: Xinyi Zhu, name: Qiuyi Ding, name: Jingjia Peng, name: Jiarong Xing, name: Yibo Huang, name: Yiming Qiu, name: Jayanth Srinivasa, name: Myungjin Lee, name: Mosharaf Chowdhury, name: Matei Zaharia, name: Ang Chen

Automating AI research holds immense potential for accelerating scientific
progress, yet current AI agents struggle with the complexities of rigorous,
end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed
to systematically evaluate AI agents on complete research experiments sourced
from influential AI publications. Given a research question and incomplete
starter code, EXP-Bench challenges AI agents to formulate hypotheses, design
and implement experimental procedures, execute them, and analyze results. To
enable the creation of such intricate and authentic tasks with high-fidelity,
we design a semi-autonomous pipeline to extract and structure crucial
experimental details from these research papers and their associated
open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks
from 51 top-tier AI research papers. Evaluations of leading LLM-based agents,
such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial
capabilities: while scores on individual experimental aspects such as design or
implementation correctness occasionally reach 20-35%, the success rate for
complete, executable experiments was a mere 0.5%. By identifying these
bottlenecks and providing realistic step-by-step experiment procedures,
EXP-Bench serves as a vital tool for future AI agents to improve their ability
to conduct AI research experiments. EXP-Bench is open-sourced at
https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.