FML-bench: A Benchmark for Automatic ML Research Agents Highlighting the
  Importance of Exploration Breadth
Authors: name: Qiran Zou, name: Hou Hei Lam, name: Wenhao Zhao, name: Yiming Tang, name: Tingting Chen, name: Samson Yu, name: Tianyi Zhang, name: Chang Liu, name: Xiangyang Ji, name: Dianbo Liu
Organization: National University of Singapore

Large language models (LLMs) have sparked growing interest in automatic
machine learning research agents. Among them, agents capable of autonomously
proposing ideas and conducting machine learning experiments are particularly
promising, as they maximize research automation and accelerate scientific
progress by iteratively refining ideas based on experimental results. However,
comprehensively evaluating such agents remains challenging. Existing benchmarks
tend to overemphasize engineering aspects while neglecting academic rigor,
creating barriers that obscure a clear assessment of an agent's scientific
capabilities in machine learning research. They also suffer from limited task
diversity, an overemphasis on application-oriented tasks over fundamental
research problems, and limited scalability to realistic research settings. To
address these limitations, we introduce FML-bench, a benchmark designed to
evaluate automatic machine learning research agents on 8 diverse and
fundamental machine learning research problems. It reduces coding burden,
emphasizes fundamental problems rather than specific use cases, offers high
task diversity, and is extensible to real-world machine learning GitHub
repositories. Furthermore, we present a unified evaluation framework with five
complementary metrics, designed to comprehensively assess agent performance on
our benchmark. We evaluate state-of-the-art automatic research agents on
FML-bench, and find that agents employing broad research exploration strategies
outperform those focusing on narrow but deep exploration. These findings
suggest that emphasizing the breadth of exploration may lead to more effective
research outcomes than focusing solely on incremental refinement. Our benchmark
is available at https://github.com/qrzou/FML-bench.