The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT
  Improvements
Authors: name: Bingchen Zhao, name: Despoina Magka, name: Minqi Jiang, name: Xian Li, name: Roberta Raileanu, name: Tatiana Shavrina, name: Jean-Christophe Gagnon-Audet, name: Kelvin Niu, name: Shagun Sodhani, name: Michael Shvartsman, name: Andrei Lupu, name: Alisia Lupidi, name: Edan Toledo, name: Karen Hambardzumyan, name: Martin Josifoski, name: Thomas Foster, name: Lucia Cipolina-Kun, name: Abhishek Charnalia, name: Derek Dunfield, name: Alexander H. Miller, name: Oisin Mac Aodha, name: Jakob Foerster, name: Yoram Bachrach

Rapid advancements in large language models (LLMs) have the potential to
assist in scientific progress. A critical capability toward this endeavor is
the ability to reproduce existing work. To evaluate the ability of AI agents to
reproduce results in an active research area, we introduce the Automated LLM
Speedrunning Benchmark, leveraging the research community contributions on the
NanoGPT speedrun, a competition to train a GPT-2 model in the shortest time.
Each of the 19 speedrun tasks provides the agent with the previous records
training script, optionally paired with one of three hint formats, ranging from
pseudocode to paper-like descriptions of the new records improvements. Records
execute quickly by design and speedrun improvements encompass diverse
code-level changes, ranging from high-level algorithmic advancements to
hardware-aware optimizations. These features make the benchmark both accessible
and realistic for the frontier problem of improving LLM training. We find that
recent reasoning LLMs combined with SoTA scaffolds struggle to reimplement
already-known innovations in our benchmark, even when given detailed hints. Our
benchmark thus provides a simple, non-saturated measure of an LLMs ability to
automate scientific reproduction, a necessary (but not sufficient) skill for an
autonomous research agent.