MLRC-Bench: Can Language Agents Solve Machine Learning Research
  Challenges?
Authors: name: Yunxiang Zhang, name: Muhammad Khalifa, name: Shitanshu Bhushan, name: Grant D Murphy, name: Lajanugen Logeswaran, name: Jaekyeom Kim, name: Moontae Lee, name: Honglak Lee, name: Lu Wang

Existing evaluation of large language model (LLM) agents on scientific
discovery lacks objective baselines and metrics to assess the viability of
their proposed methods. To address this issue, we introduce MLRC-Bench, a
benchmark designed to quantify how effectively language agents can tackle
challenging Machine Learning (ML) Research Competitions. Our benchmark
highlights open research problems that demand novel methodologies, in contrast
to recent benchmarks such as OpenAI's MLE-Bench (Chan et al., 2024) and METR's
RE-Bench (Wijk et al., 2024), which focus on well-established research tasks
that are largely solvable through sufficient engineering effort. Unlike prior
work, e.g., AI Scientist (Lu et al., 2024b), which evaluates the end-to-end
agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of
proposing and implementing novel research methods and evaluates them with newly
proposed rigorous protocol and objective metrics. Our curated suite of 7
competition tasks reveals significant challenges for LLM agents. Even the
best-performing tested agent (gemini-exp-1206 under MLAB (Huang et al., 2024a))
closes only 9.3% of the gap between baseline and top human participant scores.
Furthermore, our analysis reveals a misalignment between the LLM-judged
innovation and their actual performance on cutting-edge ML research problems.
MLRC-Bench is a dynamic benchmark, which is designed to continually grow with
new ML competitions to encourage rigorous and objective evaluations of AI's
research capabilities.