Kati8/ICLR_2023_2025 huggingface dataset:
authors: Jun Shern Chan
authors: Neil Chowdhury
authors: Oliver Jaffe
authors: James Aung
authors: Dane Sherburn
authors: Evan Mays
authors: Giulio Starace
authors: Kevin Liu
authors: Leon Maksin
authors: Tejal Patwardhan
authors: Aleksander Madry
authors: Lilian Weng
reviews: 
Comments: This paper constructs a benchmark for evaluating the capabilities of LLMs in automated data science tasks based on Kaggle competitions. This paper presents MLE-bench, consisting of 71 competitions covering diverse data modalities and task complexities. Rule-breaking detection and plagiarism detection are provided to prevent LLMs from generating undesired behaviors. Evaluations are conducted on both closed-source and open-source LLMs.
Strengths: 1. The proposed benchmark is quite challenging for current LLM agents.

2. A lot of factors are considered in the benchmark, such as time constraint, computation resource, etc.

3. The empirical evaluation is comprehensive.
Weaknesses: 1. My major concern lies in the test splits used in this benchmark, which cannot guarantee alignment with the private leaderboard score and thus leads to unfair comparison with data scientists from Kaggle. I checked several Kaggle competitions used in this benchmark and found that they supported “late submission” to provide the leaderboard score. Why MLE-bench does not choose to fully leverage this feature? Could you check whether the test splits used in MLE-bench align with the realistic leaderboard via this feature? If not, I think MLE-bench can only provide comparison among LLM agents rather than human data scientists from Kaggle.

2. As discussed in Line 501-504, there is an obvious discrepancy on available machine learning techniques between past Kagglers and modern LLMs. I suspect that the capabilities of LLM agents for winning a medal in MLE-bench also correlates to the year of the Kaggle competition. As such, the evaluation metric solely relying on whether an LLM agent can win a medal may lead to biased conclusion. Could you present complete results to show the effect of the competition starting year for the performance?

3. Also, could you provide empirical analyses on the effect of the competition complexity (as labeled high/medium/low) for the agent performance?

4. Could you present some successful cases and failed cases of o1-preview in MLE-bench?  The trajectories shown in Figure 2 are not complete enough to derive insightful findings.

5. How does plagiarism detection work? If the current competition is A, the plagiarism is detected in terms of merely notebooks in A or all the notebooks in competitions from MLE-bench?

6. As discussed in Line 505-509, I also think the current benchmark is too heavy for potential future research purposes. Maybe a light-weight version of MLE-bench can be considered as future work.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper introduces MLE-bench, which is a benchmark consisting of 71 Kaggle competitions for measuring how well AI agents perform at machine learning engineering. The benchmark includes human baselines using the publicly-available Kaggle leaderboards. The paper also includes benchmarking with various scaffolds and foundation models, as well as some analysis of possible issues like contamination from pre-training.
Strengths: Originality. To my knowledge, this benchmark is the first that is designed to evaluate ML engineering capabilities of AI agents. 

Quality. The paper has done a good-faith effort to benchmark using relevant models and mitigate potential issues (e.g., contamination and plagiarism). I liked the transparent discussion of limitations and potential issues. 

Clarity. I found the paper generally easy to follow. 

Significance. The introduction of this benchmark is quite timely given the interest in developing high-quality software engineering agents.
Weaknesses: 1. There seems to be an issue with Figure 2. I can only see a small snippet of the figure. 

2. I am concerned about the accessibility of this benchmark. As stated in section 6, it is a resource-intensive benchmark to run. If I understand the cost breakdown correctly, a single seed costs over $2500 to run (for the current prices of o1-preview). This is simply not feasible to university labs. I would suggest the authors provide two versions of the benchmark: one that is more accessible and one that is less accessible.

3. Given that the main contribution of this work is the benchmark, I think some of the experiments could be pushed to the appendix, whereas more details about the benchmark could be in the main body. For example, I’d like to see a more clear setup and rules section to make using the benchmark as easy as possible.

4. Given that this is a datasets and benchmarks submission, I wish the anonymized codebase was made available during submission. As a result, I am also reducing my confidence score.

5. There was a lot of discussion about splitting the competitions based on complexity, but I don’t see any presentation of the agent scores as a function of complexity. It feels strange to have this decomposition without using it in the later analysis.

6. Some of the selection criteria is clear (e.g., completed competition), but others are more qualitative (e.g., well-specified description), so it would be nice to see something a bit more detailed and systematic for those.

7. Some of the choices are not super clear. For example, what does “where sensible, we maintain the train/test split ratio” mean? (L157-158). Similarly, why was the headline metric chosen that way? Is this standard for Kaggle competitions? 

8. I would have liked to see examples of the generated code, potentially with an additional quality analysis. This analysis need not be extensive, even conducting the analysis on a randomly-selected output for one task would be interesting.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3