reviews: 
Comments: The authors present MLE-bench, a benchmark designed to evaluate AI agents’ capabilities in machine learning engineering. This benchmark is constructed from publicly available Kaggle challenges, with AI agent performance assessed based on the percentage achieving medal-level scores comparable to real human submissions. The paper releases the benchmark’s data and code, includes three open-source agents as baselines, and evaluates state-of-the-art foundational models using the AIDE agent.
Strengths: 1. Given recent advancements in AI agents for coding, engineering, and scientific discovery, the proposed benchmark serves as a highly valuable testbed for evaluating foundational models and agents in real-world machine learning engineering contexts.
1. The paper is clearly written and easy to follow, providing necessary details on both the dataset and technical implementation.
1. The study includes solid empirical evaluations across different agents, foundational models, scaling factors, and contamination issues, offering valuable insights into AI agents’ current capabilities and limitations in the field.
1. The authors provide a thorough discussion of the benchmark’s limitations and ethical considerations.
Weaknesses: ### Accessibility of the Benchmark

As discussed by the authors in Section 6 L505, the benchmark is costly to run. Based on my estimates, using the authors’ cost descriptions and current rates on OpenAI and Lambda Cloud, a single full evaluation with AIDE and o1-preview would require approximately 4,000 USD (around 2,600 USD for API queries and 1,300 USD for an A10 GPU server). Considering the costs for development and extensive experiments would be substantially higher than a single evaluation, this benchmark may be inaccessible to small to medium-scale academic labs.

The authors may consider providing a lighter dataset split to improve accessibility, similar to the approach in SWE-Bench.

### Evaluation Reliability and Contamination

One particular challenge in benchmarks for AI agents is establishing reliable detection for potential cheating behaviors (e.g., hacking the evaluation function or accessing private test data). I appreciate the authors for addressing this with a tool designed to flag such behaviors. However, Table 6 in Appendix A.3 indicates a high false positive rate, which may hinder practical reliability. The rate is sufficiently high that manually checking all flagged submissions would be demanding.

Additionally, while the authors provide a thorough discussion and empirical analysis of contamination issues, this remains a critical limitation for this and other similar benchmarks. For instance, Figure 5 shows GPT-4o familiarity scores above 0.4 across all problems. Does this suggest that these problems are included in the model’s training set? Also, the conclusions drawn from the correlation between familiarity and performance could be significantly impacted by confounders, such as problem difficulty. Furthermore, while the obfuscated dataset and plagiarism detection tools are commendable efforts, foundational models could still potentially recognize rephrased questions and apply high-level strategies from their memories, making such behaviors difficult to detect.

More discussions on those concerns could be helpful. However, it is worth noting that these issues reflect broader challenges in the field, and it would be unreasonable to expect any single paper to fully resolve them. The paper has made valuable contributions with its detailed discussions and insightful empirical results on these challenges.

### Comparison Between Human Medal Results

As noted by the authors in Section 6 (L497), the train-test splits used in MLE-bench differ from the original splits in Kaggle competitions. The authors state that they “ensure that the distributions… are similar by checking that the example submission scores similarly on both sets” (L156). However, it is unclear how these example submissions were created. If the same model training pipeline were applied to both the original training set and the modified training set (a subset of the original), one would typically expect lower performance on the latter due to reduced training data. Could the authors clarify the configuration of the example submission, and specifically, what comparisons were made and under which settings?

### Additional Questions

I have some further questions, outlined below, that may also be worth addressing in the paper.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4