AIDE: AI-Driven Exploration in the Space of Code
Authors: name: Zhengyao Jiang, name: Dominik Schmidt, name: Dhruv Srikanth, name: Dixing Xu, name: Ian Kaplan, name: Deniss Jacenko, name: Yuxiang Wu

Machine learning, the foundation of modern artificial intelligence, has
driven innovations that have fundamentally transformed the world. Yet, behind
advancements lies a complex and often tedious process requiring labor and
compute intensive iteration and experimentation. Engineers and scientists
developing machine learning models spend much of their time on trial-and-error
tasks instead of conceptualizing innovative solutions or research hypotheses.
To address this challenge, we introduce AI-Driven Exploration (AIDE), a machine
learning engineering agent powered by large language models (LLMs). AIDE frames
machine learning engineering as a code optimization problem, and formulates
trial-and-error as a tree search in the space of potential solutions. By
strategically reusing and refining promising solutions, AIDE effectively trades
computational resources for enhanced performance, achieving state-of-the-art
results on multiple machine learning engineering benchmarks, including our
Kaggle evaluations, OpenAI MLE-Bench and METRs RE-Bench.