MLE-bench: Evaluating Machine Learning Agents on Machine Learning
  Engineering
Authors: name: Jun Shern Chan; name: Neil Chowdhury; name: Oliver Jaffe; name: James Aung; name: Dane Sherburn; name: Evan Mays; name: Giulio Starace; name: Kevin Liu; name: Leon Maksin; name: Tejal Patwardhan; name: Lilian Weng; name: Aleksander Mądry
Published: 2024-10-09 17:34:27

Abstract: We introduce MLE-bench, a benchmark for measuring how well AI agents perform
at machine learning engineering. To this end, we curate 75 ML
engineering-related competitions from Kaggle, creating a diverse set of
challenging tasks that test real-world ML engineering skills such as training
models, preparing datasets, and running experiments. We establish human
baselines for each competition using Kaggle's publicly available leaderboards.
We use open-source agent scaffolds to evaluate several frontier language models
on our benchmark, finding that the best-performing setup--OpenAI's o1-preview
with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in
16.9% of competitions. In addition to our main results, we investigate various
forms of resource scaling for AI agents and the impact of contamination from
pre-training. We open-source our benchmark code (github.com/openai/mle-bench/)
to facilitate future research in understanding the ML engineering capabilities
of AI agents.