reviews: 
Comments: This paper introduces a new benchmark, “MLE-Bench”, with the goal of assessing AI agent’s abilities at machine learning engineering (MLE) tasks. The benchmark consists of 71 hand-selected Kaggle competitions from domains including image classification (38%), text classification (14%), and tabular tasks (13%) at a variety of difficulty levels. For each task, the agent is provided with a description of the task and a dataset. The agent is prompted to solve the task by writing code which is executed and evaluated analogously to the process of evaluating human Kaggle contestants’ submissions. Each agent is given up to 24 hours to iteratively improve its solution before evaluation.

The authors use MLE-Bench to evaluate several combinations of agent scaffolds and base language models,

- AIDE (with o1-preview, gpt-4o, llama-3.1-405b, claude-3-5-sonnet),
- ResearchAgent from MLAgentBench (with gpt-4o),
- CodeActAgent from OpenHands (with gpt-4o),

and compare the performance of these agents with a baseline derived from the performance of human Kaggle contestants. The evaluation metric is the percentage of tasks in which the agent would have received a Bronze, Silver, or Gold medal, had it actually participated in the respective Kaggle competition.

The authors show that both the choice of agent scaffold as well as the choice of language model has a significant effect on performance, and show that the best agent receives a medal in 17.3% of tasks.

The paper also includes experiments on the effect of hardware provided to the agents (0-2 GPUs) and the amount of time available to agents (up to 100 hours per task). Further, the authors evaluate whether the agent’s potential familiarity with a task (i.e. inclusion in its training data) affects performance. Finally, they analyze the agent’s code outputs for potential rule violations and plagiarism.
Strengths: The experimental results presented in the paper are interesting and provide a clearer picture on current AI agent’s abilities for MLE. The benchmark will be a useful contribution to future work on agent scaffolds and LLMs, as well as for evaluating current systems from AI safety and preparedness perspectives.

Some strengths worth highlighting:

1. The proposed benchmark is comprehensive, and covers many parts of ML engineering (preprocessing, model training,...) and task domains (image, text, tabular,..).
2. The main experiments are rigorous and demonstrate the usefulness of the proposed benchmark.
3. The evaluation protocol seems generally sensible and the reported metrics are useful for understanding the results, although some improvements could be made (see below).
Weaknesses: Please see the following suggestions that, in my opinion, would significantly improve this paper. Given some updates to the paper and clarifications to the questions in the next section, I would be happy to increase the score.

1. The authors acknowledge that their benchmark is very resource-intensive to run, requiring 1704 GPU-hours and >100M LLM tokens for a single seed (see Sec 6./Accessibility). Given that their main results used 16 and 36 seeds, this is clearly inaccessible to a large fraction of the research community. The paper would be improved if the authors could provide a lighter version of the benchmark (e.g. a subset of 5-10 tasks that reflect the main challenges of MLE) along with the metrics on this subset. This would not require running any additional experiments.
2. The aggregated results (across tasks, seeds, and time steps) provided in the paper are useful, but do not provide a full picture of the remaining open challenges in MLE and why the agents achieve good/bad levels of performance. It would be great if the authors could include more detailed and also qualitative results. In particular:
    1. A clearer analysis of which kinds of tasks agents perform well or badly on (e.g. split scores by complexity level and by task domain).
    2. The paper mentions raw per-task scores (Sec 2.2), although these do not seem to be included in the paper.
    3. The paper mentions that the authors analyzed agent transcripts/logs. It would be useful if these transcripts were provided (not necessarily in the paper, but in an external source)
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3