MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline
Authors: name: Rushi Qiang, name: Yuchen Zhuang, name: Anikait Singh, name: Percy Liang, name: Chao Zhang, name: Sherry Yang, name: Bo Dai
Organization: MLE-Dojo

While Language Models (LMs) have made significant progress in automating
machine learning engineering (MLE), the acquisition of high-quality MLE
training data is significantly constrained. Current MLE benchmarks suffer from
low scalability and limited applicability because they rely on static, manually
curated tasks, demanding extensive time and manual effort to produce. We
introduce MLE-Smith, a fully automated multi-agent pipeline, to transform raw
datasets into competition-style MLE challenges through an efficient
generate-verify-execute paradigm for scaling MLE tasks with verifiable quality,
real-world usability, and rich diversity. The proposed multi-agent pipeline in
MLE-Smith drives structured task design and standardized refactoring, coupled
with a hybrid verification mechanism that enforces strict structural rules and
high-level semantic soundness. It further validates empirical solvability and
real-world fidelity through interactive execution. We apply MLE-Smith to 224 of
real-world datasets and generate 606 tasks spanning multiple categories,
objectives, and modalities, demonstrating that MLE-Smith can work effectively
across a wide range of real-world datasets. Evaluation on the generated tasks
shows that the performance of eight mainstream and cutting-edge LLMs on
MLE-Smith tasks is strongly correlated with their performance on carefully
human-designed tasks, highlighting the effectiveness of the MLE-Smith to
scaling up MLE tasks, while maintaining task quality.