reviews: 
Comments: The paper introduces MLE-bench, a comprehensive benchmark for assessing the performance of AI agents in machine learning engineering tasks. The benchmark draws upon real-world Kaggle competitions and measures success by comparing AI agent outcomes to human medal-winning submissions. The work makes a significant contribution by releasing both the benchmark infrastructure and accompanying code, providing three baseline implementations from open-source agents, and conducting systematic performance evaluations of cutting-edge language models via the AIDE framework.
Strengths: 1. The benchmark addresses a timely need in the field, providing an essential evaluation platform for assessing the capabilities of both foundational models and AI agents in practical machine learning engineering scenarios, particularly given the rapid progress in AI-driven coding and scientific research tools.
2. The presentation is well-organized and transparent, offering comprehensive documentation of dataset construction and implementation details that enhance reproducibility.
3. The experimental evaluation is thorough and multi-faceted, spanning various agent implementations, language models, model scales, and potential data leakage concerns, yielding actionable insights regarding the current state and limitations of AI agent performance.
4. The authors provide a comprehensive examination of benchmark limitations and raise important ethical considerations relevant to the work.
Weaknesses: ### Accessibility and Resource Requirements
The computational and financial barriers to using this benchmark are substantial. Based on cost analysis using current API pricing and cloud computing rates, executing a complete evaluation run with AIDE and o1-preview would likely exceed 4,000 USD (approximately 2,600 USD in API expenses and 1,300 USD for GPU server costs). Given that comprehensive research would involve substantially greater expenses, this creates a significant accessibility problem for resource-constrained research institutions and smaller academic teams. The authors might explore introducing a reduced-scope version of the benchmark comparable to existing practices in similar benchmarking efforts.

### Data Leakage Detection and Reliability
A key methodological concern involves the robustness of mechanisms designed to detect potential benchmark violations such as evaluation function manipulation or unauthorized access to test datasets. The detection system described by the authors addresses this issue, yet Table 6 demonstrates elevated false positive rates that undermine its practical applicability. The high number of incorrect flagging would necessitate extensive manual verification of all alerts, limiting its utility.

While the authors provide valuable discussion and analysis of data contamination, significant questions persist. The contamination analysis in Figure 5 reveals GPT-4o familiarity ratings exceeding 0.4 across all test cases, which raises questions about whether these problems appear in the model's training data. Moreover, the observed correlation between familiarity scores and performance metrics could be confounded by other variables such as problem complexity. Although the authors implement obfuscation and plagiarism detection methods, advanced language models may still recognize reformulated problems and leverage memorized high-level solution approaches in ways that current detection methods cannot identify. The paper would benefit from deeper exploration of these challenges, though it is important to note these represent systemic challenges in the field rather than deficiencies unique to this work.

### Train-Test Split Methodology
A methodological question concerns the train-test partitioning strategy employed in MLE-bench, which diverges from the original Kaggle competition splits. The authors claim distributional equivalence through baseline model performance, but the construction methodology for these baseline submissions remains ambiguous. Logically, applying identical model architectures and training procedures to both the original and modified (reduced) training sets should yield lower performance on the latter due to less available training data. The paper would benefit from explicit clarification regarding baseline creation methods and specification of the experimental parameters and conditions under which comparisons were conducted.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4