MLE-Smith: Scaling MLE Tasks with Automated Multi-agent Pipeline
Authors: Rushi Qiang
Yuchen Zhuang
Anikait Singh
Percy Liang
Chao Zhang
Sherry Yang
Bo Dai
Venue: ICLR 2026 Poster (2026)
Keywords: Machine Learning
LLM Agents
Primary area: datasets and benchmarks

Abstract: While Language Models (LMs) have made significant progress in automating machine learning engineering (MLE), the acquisition of high-quality MLE training data is significantly constrained. Current MLE benchmarks suffer from low scalability and limited applicability because they rely on static, manually curated tasks that demand extensive time and manual effort to produce.
We introduce MLE-Smith, a fully automated multi-agent pipeline, to transform raw datasets into competition-style MLE challenges through an efficient generate--verify--execute paradigm for scaling MLE tasks with verifiable quality, real-world usability and rich diversity. 
The proposed multi-agent pipeline in MLE-Smith drives structured task design and standardized refactoring, coupled with a hybrid verification mechanism that enforces strict structural rules and high-level semantic soundness. It further validates empirical solvability and real-world fidelity through interactive execution.
We apply MLE-Smith to 224 of real-world datasets and generates 606 tasks spanning multiple categories, objectives, and modalities, demonstrating that MLE-Smith can work 
effectively 
across a wide range of real-world datasets.
Evaluation on generated tasks shows that the performance of eight mainstream and cutting-edge LLMs on MLE-Smith tasks is strongly correlated with their performance on carefully human-designed tasks, highlighting the effectiveness of the MLE-Smith in scaling up MLE tasks while maintaining task quality.

Published as a conference paper at ICLR 2026
MLE-SMITH: SCALING MLE TASKS WITH AUTO- MATED MULTI-AGENT PIPELINE
Rushi Qiang', Yuchen Zhuang', Anikait Singh² Percy Liang", Chao Zhang', Sherry Yang". Bo Dail
Georgia Institute of Technology Stannfor University
ABSTRACT
While Language Models (LMs) have made significant progress automating machine learning engineering (MLE), the acquisition of high-quality MLE train- ing data is significantly constrained. Current MLE benchmarks suffer from low scalability and limited applicability because they rely on static, manually cu- rated tasks, demanding extensive time and manual effort to produce. We in- troduce MLE-Smi th, a fully automated multi-agent pipeline, to transform raw datasets into competition-style MLE challenges through an efficient generate- verify-execute paradigm for scaling MLE tasks with verifiable quality, real-world usability, and rich diversity. The proposed multi-agent pipeline in MLE- Smi th drives structured task design and standardized refactoring, coupled with hy- brid verification mechanism that enforces strict structural rules and high-level semantic soundness. further validates empirical solvability and real-world fi- delity through interactive execution. We apply MLE-Smith th to 224 of real-world datasets and generate 606 tasks spanning multiple categories, objectives, and modalities, demonstrating that range of real-world datasets. Evaluation on the generated tasks shows that the performance of eight mainstream and cutting-edge LLMs on MLE-Smit th tasks is strongly correlated with their performance on carefully human-designed tasks, highlighting the effectiveness of the MLE th to scaling up MLE tasks, while maintaining task quality.
th work effectively across wide
INTRODUCTION
Large Language Model (LLM) agents have demonstrated remarkable capabilities in automating complex coding and engineering domains (Chan et al., 2024; Qiang et al., 2025; Nathani et al., 2025: Jing et al., 2024; Yang et al., 2024; Jimenez et al., 2023), with machine learning engineering (MLE) emerging as a key frontier for evaluating the capability of models today. The development of sophisticated MLE agents, capable of autonomously handling tasks from data pre-processing to model tuning and deployment, promises to revolutionize scientific discovery and industrial ap- plications. However, evaluating and developing such agents poses significant challenge, due to the inherent complexity of MLE workflows, the need for domain-specific knowledge, and the it- erative, feedback-driven nature of real-world machine learning pipelines. Developing robust MLE agents, therefore, requires not only the design and implementation of agent frameworks but also the creation of holistic environments and benchmarks that support end-to-end experimentation and structured evaluation under truly real-world conditions, encompassing diverse task distributions.
Recent efforts have established valuable benchmarks and interactive environments for evaluating and training these agents (Huang et al., 2023; Jing et al., 2024; Chan et al., 2024; Qiang et al., 2025; Nathani et al., 2025). However, existing benchmarks such as MLE-Bench (Chan et al., 2024) and DS- DeSBennch (Jing al., 2024) and gym-like interactive environments such as MLE-Dojo (Qiang et al., 2025) and MLGym (Nathani et al., 2025) offer only static collections of tasks, and their con- struction remains heavily reliant extensive human curation. This manual effort stems from two main sources: (1) the competitions selected for inclusion in these benchmarks are often carefully designed by human experts, and (2) the benchmarks require substantial engineering work to adapt

---

Published as a conference paper at ICLR 2026
these competitions into a standardized format suitable for benchmarking. Such adaptation typically involves non-trivial engineering efforts such as the pre- pre-processing and splitting of data into train and test splits, along with implementing evaluation scripts and establishing scoring mechanism. In addition, the ambition to establish a comprehensive environment for evaluating and training MLE agents imposes further demands on the scale and diversity of available MLE tasks. The continued reliance on static, manually curated tasks restricts the diversity and realism of interaction scenarios and introduces a scalability bottleneck that impedes the rapid development and reliable assessment of next-generatin MLE agents. Thus, overcoming this limitation necessitates an automated frame- work that can continuously generate, verify, and evolve MLE tasks scale.
Building such framework for scaling MLE tasks presents formidable challenge: how can the framework rigorously validate the correctness and practical value of each newly generated task? Unlike conventional supervised datasets, an MLE benchmark must satisfy multiple intertwined cri- teria: (i) Structural integrity, ensuring that all associated components including data pre-processing scripts, file directory hicrarchies, and evaluation pipelines, must execute end-to-end without manual intervention, ensuring that the task is reproducible and computationally viable; (ii) Semantic sound- ness, confirming that the defined learning objective must be coherent, and the input-outpu structure must reflect the natural affordances and signals present in the source dataset, avoiding degenerate or trivial mappings; and (iii) Empirical solvability, demonstrating that the task should be non-trivial yet tractable-i.e. standard baseline agents must he able to achieve meaningful performance and ex- hibit stable improvement under reasonable training protocols. A failure on any of these dimensions undermines the utility of the task, preventing it from eliciting meaningful behavioral differences across agents or supporting their effective training and development in interactive settings.
To address these challenges, we present MLE-Smi Smith, a fully automated framework that transforms raw datasets into competition-style MLE tasks through scalable MLE-Smithh h is carefully designed to enforce structural integrity, semantic soundness, and empirical solvability by integrating multi-agent generation workflow, a robust hybrid verification mecha- nism, and an execution -based validation loop, as illustrated in Figure 1, which provides an overview of the end-to-end paradigm. The system features three specialized agents- Brainstormer, Designer, and Refactor-thal generate, concretize, and standardize task proposals in a modular, auditable manner. A persistent verification mechanism, combining both deterministic checks and agent- reviews, continuously ensures the correctness and coherence of tasks. Finally, each task is validated by interactive execution between a validation MLE agent and MLE environments, confirming that it supports end-to-end execution and delivers non-trivial signals on the performance of ML solu- tions. This principled pipeline ensures that each generated task is format-consistent, executable, and verifiable, while remaining practically meaningful for training and evaluating MLE agents.
pipeline.
We summarize our main contributions as follows:
A fully automated task generation framework. We propose MLE- th, the first end- to-end system that transforms raw datasets into competition-style machine learning engi- neering (MLE) tasks through a scalable generate-verify -eriertt-verff-eeeeetee execute pipeline. Unlike prior efforts that rely on static curation, MLE-Smiths th enables continuous generation of realistic and diverse MILE challenges at scale, without any human intervention.
A hybrid verification mechanism. To ensure the quality and utility of generated tasks, we design a multi-layer verification mechanism that combines static format validation, se- mantic alignment, and execution-based tests of empirical solvability. This hybrid stack enforces rigorous guarantees on task integrity, ensuring that each constructed challenge is well-structured, executable, and grounded in realistic machine learning scenarios.
A large-scale, diverse generated task suite. We apply MLE-Smi th to 224 woal-world datasets and produce 606 fully verified tasks spanning a wide spectrum of modalities (e.g., tabular, vision, time series), learning objectives (e.g.. classification, regression, ranking), and domains (e.g., healthcare, sports). Evaluation a representative subset of 50 tasks with eight cutting-edge LLMs reveals strong correlation with rankings of these LLMs human-curated benchmarks, demonstrating that ALE-Smmite th yields challenging, dis- criminative, and generalizable tasks suitable for evaluating and eventually training next- generation MLE agents.
2

---

Published as a conference paper at ICLR 2026
Electric Vehicle Specs
This datasel provides a collection of specifications and performance metrics for modern electrio vehicles (EVs). It is designed to support researchers, analysts, students, and developers working data science...
Regreesion performance
Hybrid Verification
MBEIRMBE metric
Classification on chest X-Ray Ray F1/Accuracy metric reviews Brainstorm BLEU ROUGE as metric
</>
Fills from Designer and Refactor
Generation
Assertions
rawl private! public/ metric.py prepare. Py
LLM Review
Task Objective
Pneumonia Chest
Target festurellabels
Hard format structure validation
Soft design
Cleaned chest X-ray images from
Data processing split Sample generation
semantic
pediatric patients (ages used
alignment
detect pneumonia (bacterial and virel)
and differentiate from normal cases.
Design
design Script implementation
Validate Code pam
The final quality review conducted by radiologist, ensuring accuracy...
Request Info
Execute
Code
dataset
Fake..cv cav (23502 take news articie)
Datasel separated in two files: 1.
ISOT Fake News Delect
NEWS
Refactor
metric.pp -olass Metrics prepare py dee prepare
private! -lest answer
public/ -data, submission
Test Agent
Dataset Info
MLE Env
Metric Score
Truc.csv (21417 true
Dataset columns: Title: article) news article; Text: body text of news article;
Error Messaga
File Read/Write
Shell Commands
Subject: subject of news article; Date:
Code Execution
publish date of article...
Toolset
Execation-based Verification
Figure I: ALE-Smith automatically generates competition-sypy style machine learning engineering (MLE) tasks from raw datasets through generate-verify exeeee paradigm.
RELATED WORKS
Agent Benchmarks and Environments. Recent efforts have introduced diverse suite of bench- marks and interactive environments for the evaluation and development of LLM-based agents across multiple domains, including software engineering (SWE) benchmarks (Jimenez et al., 2023; Pan et al., 2024a; Yang et al., 2024; Zhang et al., 2025: Zan et al., 2025; Alcithan et al., 2024) that test agents' ability to modify large codebases and repair real-world bugs, web navigation and browsing tasks (Chezelles et al., 2024; Zhou et al., 2023; Pan et al., 2024b; Levy et al., 2024; Wei et al., 2025; Wu et al., 2025; Yao et al., 2022) that evaluate agents' capacity to navigate complex websites or device interfaces, deep research settings (Du et al., 2025; Bosse et al., 2025; Phan et al., 2025) that require multi-step reasoning and information aggregation, general tool-use environments (Yao et al., 2024; Qin et al., 2023; Mialon et al., 2023; Liu et al., 2023; Luo et al., 2025) that probe agents' ability to orchestrate diverse tools and external resources, and studies of human agent collaboration in dynamic task scenarios (Shao et al., 2024). In the MLE domain, growing body of testbeds assesses agents on end-to-end workflows. For example, MLAGENT offers 13 curated MI_E tasks with baselines and performance thresholds, MLEE- BE-BENCH (Chan al., 2024) standardizes 75 Kaggle competitions for structured MLE evaluation, DS BENCH (Jing et al., 2024) includes 74 modeling tasks reflecting realistic data science processes, MLGYM (Nathani et al., 2025) provides Gym-style suite for AI research workflows, and MLE-DOJO (Qiang et al., 2025) scales to over 200 fully executable MLE tasks with step-wise interaction. While these MLE platforms advance realism and breadth, they remain limited by finite, manually curated task sets. In contrast, MLE-Smi -miitttt th proposes fully automated framework for scalable and high-quality MLE task generation, which allows for the continual generation of novel tasks in the MIE domain.
(Huang et al., 2023)
Automated Task Generation. Automated task generation has emerged as a promising direction for scaling agent evaluation and training. TASKCRAFT (Shi et al., 2025) creates scalable, multi-tool agentic tasks with execution traces via compositional extensions. AUTOCODEBENCI (Chou et al., 2025) generates high-difficulty, multilingual code problems with LLM-driven reverse synthesis and test validation. SWE-SMITH (Yang et al., 2025) synthesizes tens of thousands of bug- inducing software engineering tasks from real-world Python repositories. SELF-CHALLENGING (Zhou al., 2025) trains agents to generate and solve their own Code-as-Task problems with built-in verification, enabling high-quality self-supervised RI. SQLM (Chen et al., 2025) frames task generation as asymmetric self-play, where models propose and solve increasingly challenging problems without external data. MLE Smi th serves as the first automated framework for task generation in the MLE domain, paving the way for scalable agent evaluation and training on realistic, high-quality tasks.
3

---

Published as a conference paper at ICLR 2026
3 METHODS
MLE-Smi th automatically generates competition-style machine learning engineering (MLE) tasks from raw datasets (from sources such as Kaggle) through a generate-verine perify-execute paradigm. The pipeline couples (i) structured multi-agent generation that designs and generates feasible tasks in multiple directions, (ii) hybrid verification mechanism that enforces both hard structural con- straints and soft semantic criteria, and (iii) execution-based validation inside an interactive MLE environment to ensure empirical solvability and real-world validity. This sequential architecture is designed to balance the diversity of task proposals with strong guarantees on the structural correct- ness and downstream usability of generated MLE tasks.
3.1 MULTI-AGENT GENERATION WORKFLOW
MLE-Smith h employs three specialized agents that handoff generated artifacts in sequential pipeline augmented with controlled feedback loops to allow for upstream refinement. Each agent has access to useful domain tools, including file I/O, shell commands, code execution, and always generates outputs in a pre-defined, structured format amenable to automated verification. The mid- dle part of Figure 1 illustrates how these agents sequentially advance the pipeline and produce the corresponding deliverables.
Brainstormer: Given a dataset overview along with the toolset for in-depth, multi-round data ex- ploration, the Brainstormer enumerates set of candidate task formulations rather than single design, recognizing that a single dataset often supports multiple plausible learning objectives and modeling strategies. This diversity-aware generation allows the system to fully exploit the dataset's potential. The number of candidate tasks is adaptively determined by the Brainstormer based on the dataset's 's intrinsic properties and structural characteristics. A key principle is that all labels and fea- tures must be accurate and grounded in the data itself, either explicitly provided or deterministically derived, rather than synthetic or heuristically constructed. Each proposal specifies candidate pre- diction targets (classification labels, regression variables, sequence outputs), evaluation metrics (e.g., accuracy. macro-F1, RMSE, or domain-specifi scores), data utilization (e.g.. preprocessing, feature construction, label extraction) and justificatton that articulate the rationale and practical us- ability of the proposed design. Equipped with domain tools, the Brainstormer gains comprehensive and de-tepth insights, enabling it to generate grounded and valuable task proposals. By explicitly separating hypothesis generation from commitment, MLE ALE-Smi Smith preserves design optionality and encourages diversity without sacrificing feasibility.
Designer. For each candidate task formulation, the Designer is responsible for instantiating a fully specified machine learning engineering (MLE) task that can be executed end-to-end without manual intervention. This includes constructing 4 components necessary to define, prepare, and evaluate the task in a reproducible and verifiable manner: (i) preprocessing the raw dataset and producing deterministic training and test splits with appropriate label coverage and data integrity guarantees; (ii) defining input and output schemas that govern the structure of model predictions and evalua- tion targets; (iii) specifying the evaluation protocol and instantiating a fair, task-specific metric that captures performance with numerical stability; and (iv) generating the complete suite of auxiliary components, including task descriptions that summarize the problem setup, data usage, and evalua- tion strategy; preparation scripts that performs data preprocessing, splitting, and validation checks; structured sample submission files with randomized and valid predictions; evaluation scripts for sub- mission format validation and metric score calculation; and testing scripts to verify the correctness and consistency of the generated scripts.
Together with the original dataset, these artifacts form a complete, self-contained MLE task package that can be executed, evaluated, and iterated upon by agents in an interactive environment. Gener- ating multiple such packages in parallel allows for efficient exploration of diverse task designs and principled comparisons across candidate formulations.
The Refactor module standardizes all candidate task designs into a unified and well- specified format. We present the details of this structural task format in Appendix A.2. Rather than merely cleaning code or reorganizing files, this stage rewrites each task into shared, consistent schema that defines the preparation interface, input/output specifications, metric implementation, canonical file structure, and feedback reporting mechanism. We define set of conventions that
Refactor.
4

---

Published as a conference paper at ICLR 2026
govern the structure and semantics of valid tasks paired with verification routines that check confor- mance to these standards. By enforcing these common conventions while preserving task-specific logic, the Refactor ensures format consistency, cross-file coherence, and reliable execution. This unified representation enables downstream validation of structural correctness, streamlining auto- mated testing pipelines to verify whether each task executes end-to-end without intervention.
3.2 HYBRID VERIFICATION MECHANISM
To guarantee that every generated task is not only correct in terms of format but also semantically co- herent and practically solvable, we implement persistent Hybrid Verification Mechanism a multi- layered, multi-agen collaborative contract through the entire generate- veritte--eify-eeeee exieutexeeuteeeeee pipeline. This mechanism executes across stages and comprises three complementary verification strategies: de- terministic Assertions, model-mediated Reviews, and empirical Execution-based Validation.
Assertions (deterministic guards), Assertions encode mandatory structural constraints that are enforced through deterministic checks. These include validation existing files, directory layout, and compliance with structured schema for functions, classes, and scripts. Crucially, each assertion stage serves as a gatekeeper, ensuring that downstream modules can operate reliably without encoun- tering missing inputs malformed artifacts. Prior to Refactor, Assertions confirm the completeness and structural integrity of outputs from the Designer. As a representative example, Pre-Refactor Assertions may verify that the metric.py and prepare.py scripts execute correctly, and that both a Asser-
sample submission and a corresponding test answer are successfully created. tions enforce full conformance to the unified task schema, including function signatures, interface formats, and execution scripts. For instance, they may examine whether the entire directory satisfies the pre-defined, unified format as in Appendix A.2. These rigid checks not only eliminate syntactic and structural defects but also ensure that the task satisfies all requirements for automated down- stream execution. A task that successfully passes all assertions can be regarded as a fully structured and automation-ready MLE task, capable of rumning end-to-end without human intervention.
Reviews (semantic validation). Where assertions enforce formal correctness, Reviews evaluate the semantic quality and intent alignment of each task. Leveraging an LLM--bseed agent as the reviewer, this stage assesses the clarity of task descriptions, the appropriateness of metrics, and whether the setup encourages meaningful agent behavior over shorteut solutions. For example, Reviews may flag task descriptions that omit necessary information, or ones that leak ground truths, which would pass assertions but compromise semantic validity. Though non-deterministic, Reviews serve as a soft but crucial layer that guides refinement when rigid rules are insufficient.
Execution- based validation (empirical tractability). Beyond structural and semantic checks, a well-posed MLE task must also demonstrate empirical viability: it should admit learnable patterns, enable meaningful performance differentials, and support full-pipeline execution under realistic agentic interactions. To verify this, we introduce execution-based validation stage that runs the entire task within an interactive MLE environment. This stage leverages a coding agent with action budgets to simulate a typical MLE agent interaction process. The environment, based on MLE- Dojo (Qiang et al., 2025), exposes an API for retrieving task metadata, validating code, executing scripts, and evaluating submissions. This interface allows for transparency over the actions that the step-wise agent takes and provides fine-grained feedback on execution results and performance.
The environment monitors two key aspects of empirical validation: (i) realistic pipeline valida- tion, which ensures that the full pipeline, including data preparation, model training, evaluation and scoring, executes successfully without human assistance; and (ii) performance validation, which verifies that test agents achieve non-trivial predictive performance and that the evaluation metric exhibits sensitivity to method quality. Failures along either dimension are logged as structured de- fects and routed back into the verification mechanism, triggering either targeted refinement by the Refactor or Designer module or a re-execution of the corresponding stage. Positioned at the end of the generation pipeline, exccution-based validation ensures empirical solvability by running the full task pipeline and measuring non-trivial agent performance. It captures failure modes that escape earlier static semantic checks, serving the ultimate safeguard for real-world usability.
Taken together, the three layers of verification offer distinct but complementary guarantees: As- sertions ensure structural correctness, Reviews ensure semantic alignment, and Execution ensures real-world solvability and usability. Only tasks that satisfy all three criteria are retained as verified, high-quality MILE challenges suitable for automated benchmarking and agent development.
5

---

Published as a conference paper at ICLR 2026
Tabular (43,5%)
Classification (57.5) .%%)
Others (36.0%)
F1 Precisinn/ Recall (24.7 7%)
Natural Language (21.7%)
Regression (27.4%)
Entertainment/ Media/ Arts (21.2%)
(18.35) (18.3%)
Vislon-Image (33 8%%
Ranking (4.8%)
Healthcare Medicine (30.6%)
RMSE / MAE MSE (17.35 (17.3%)
Audio (9 9.6%)
Multillabel Classificatton (4.8%)
Retail/ E commerce (10.2%)
Custom Domain Metric (16.2%)
Time Serie 19. 596
Structured Prediction (3.1%)
Social Media, Web: (9.7%)
Others (15.2%)
Vision-Vtdeo (2.2%)
Generation (3 .%%)
Finance/ Econdmiics 7%)
Accurssy 64. (6.)
Cefiers
Others (1.0%)
Transpditation / Mobility (3.4%)
MAP (3.6%)
25%
10%
10%
15%
43%
12%
58%
10%
16%
10%
11%
22%
21%
17%
Domain, Modality, and Formulation Distribution of MLE-Smith generated tasks. From left to right, the panels show the distributions of modality, objective, domain, and metric, respec- tively. "Others" category aggregates all types whose individual proportions are relatively minor.
Figure 2:
Empirical Justification of Pipeline Components. We conduct comprehensive ablation study in Appendix B to empirically justify the necessity of each component in the multi-agent pipeline and verification mechanism. The results confirm that each component plays an essential and comple- mentary role in achieving an effective balance of cost, performance, and output quality.
4 AUTOMATED TASK GENERATION
ILE-Smith can operate seamlessly across datasets of diverse modalities, formats, and domains. To comprehensively evaluate the performance and capabilities of MLE-Smi th, we collect datasets from Kaggle, the most large-scale platform that hosts diverse, real-world machine-learning compe- titions and data resources. We sampled 224 datasets from those with high usability scores as the experimental corpus and generated 606 tasks from these 224 source datasets. We reserve a subset of 50 generated tasks to evaluate the quality of BILE-Smitae 1mith, by measuring the alignment of the performance of mainstream LLMs with the MLE-Dojo leaderboard.
4.1 AGENT AND ENVIRONMENT SETUPS
We use GPT-5 (OpenAI, 2025a) to serve as the backhone model for all of the agents in ML.E-Smi th. We a default temperature of 1.0 for GPT-5. We emphasize that the proposed multi-agent pipeline is compatible with any LLM. For each dataset, the Brainstormer Agent is allowed up to 30 steps of tool-call actions. Additionally, for each source dataset, the Brainstormer Agent allowed brain- storm at most 3 candidate task formulations. Then, for each candidate, both the Designer and Refac- tor Agents have at most retry times to pass all assertions. For every proposed task formulation, the Designer and Refactor are additionally allocated a separate budget of up to 30 steps to complete their respective processes. For the execution-based validation stage, we adapt MLE-Dojo and set up an interactive MIE environment with request_ in fo and execute_code interfaces, which re- spectively support retrieving task-related information and evaluating submissions. The environment provides step-wise, structured feedback to agents. We implement ReAct-style MLE Agent (Yao et al., 2023; Sun et al., 2023) with a budget of up to 10 steps to generate, debug, and execute code submissions to get valid metric scores.
4.2 STATISTICS OF GENERATED TASKS
Scale and Cost. MLE-Smith th produced total of 606 fully verified tasks across 224 distinct source datasets, demonstrating both scalability and efficiency of our proposed approach. On av- erage, each dataset yields 2.71 competition-style tasks with the end-to-end preparation time per task averaging 419.98 seconds, and per dataset averaging 1 133.20 seconds. This runtime excludes the execution- based verification stage, as this stage depends heavily on dataset/task characteristics, hardware configuration (GPU& & CPU), and the diversity of agent-generated code, exhibiting large variance. Here, the per-task execution time is typically below 600 seconds. The overall pipeline in- curred an average cost of $0.78 per task and $2. per dataset, including all the generation workflow and verification stages. This time required for automatic task generation is substantially lower than the manual cost of human experts authoring competition-sty/e tasks, and also significantly less than the engineering effort needed to localize and standardize Kaggle competitions into benchmark formats. Moreover, the execution-hasee verification stage is negligible when compared to the time

---

Published as a conference paper at ICLR 2026
it would take for human practitioners to solve task and achieve meaningful score. This consider- able efficiency in time strongly underscores the scalability of MLE-Sra S0mi h for large-scale machine learning engineering (MLE) task generation.
Domain, Modality, and Formulation Diversity. The generated tasks span a broad spectrum of real-world data modalities, target objectives, task domains and evaluation metrics. Figure il- lustrates the detailed distributions of generated tasks in these four aspects. Specifically, the task modalities of MLE-Smith generated tasks includes Tabular, Image, Video, Audio, Natural Lan- guage, Time Series, and other structured sources. Due to the characteristics of the source Kaggle datasets, tabular and natural language modalities appear more frequently. However, other modali- ties also constitute substantial portion of the generated tasks. The benchmark covers variety of formulations: while classification and regression are relatively common, it also includes ranking, multi-label classification, structured prediction, and generation tasks, offering diverse challenges for MLE agents. Compared to modality and objective, metric design tends to exhibit greater flexibility, as it not necessarily tied to the intrinsic properties of the dataset. Thus, MLE-Smi th naturally re- flects this flexibility. The benchmark employs a wide range of evaluation metrics, with F1, precision, and recall collectively accounting for 24.7%, followed by AUC/ROC (18.3%), RMSE/MAE/MSE (17.3%), and a notable portion of custom domain-specific metrics (16.2%). Other metrics, such as ranking- based measures like NDCG and MAP (3.6%), further contribute to the overall diversity, highlighting the pipeline's ability to support nuanced evaluation tailored to different task types.
Applicability Beyond Well-Structured Datasets. While many Kaggle datasets are relatively well-organized, they are not universally well-structured or pre-cleaned. To demonstrate the gen- erality of MLE-Smithe th, we additionally evaluate it on three categories of notably raw datasets: un- processed tabular data (no predefined features or labels), raw server logs, and raw scientific sensor data. In all cases, labels, and produces validated tasks, confirming its applicability beyond competition-reatt datasets. Full details and generated task descriptions are provided in Appendix E.
th autonomously organizes the data, defines appropriate features and
Agent-Wise Performance. For each candidate formulation proposed by the Brainstormer, both the Designer and Refactor components are allowed up to three retries, with a maximum step limit imposed for each attempt. For different datasets and formulations, the number of retries and steps used by the Designer and Refactor components is summarized by the following statistics. In over 99% of cases, the Designer succeeds on the first attempt and passes all assertion checks. Approxi- mately 92% of the time, it completes the task in no more than 15 steps, with the shortest successful case requiring only 8 steps. and none exceeding 26 steps. In contrast, the Refactor component re- quires more retries and tends to take more steps: around 6% of tasks are only completed successfully on the second attempt, and about 1 % require third. Across all tasks and formulations, Refactor consistently uses more than 13 steps, with the majority of tasks densely utilizing 15 to 22 steps. These results align with the intended roles and design of the agents: the Refactor typically requires more actions than the Designer, as it must read the provided examples, analyze how to standardize the code and file structure to meet the required specifications, and ultimately ensure all tests pass.
5 EXPERIMENTS: TASK EVALUATION
We evaluate whether the tasks generated by MLE-Smith faithfully reflect the difficulty and dis- criminative structure of real, human- tasks. We conduct a comprehensive evaluation of eight cutting-edge large language models (LLMs) on a curated benchmark of 100 machine learning engineering (MLE) tasks, which we refer to as the Combined set. This evaluation suite comprises 50 tasks from the original MLE-Dojo evaluation set Dojo set and 50 tasks automatically generated by MLE-Smi Smith set. Both subsets are designed to span diverse range of data modalities, application domains, and task formulations, providing a sufficiently diverse MLE testbed.
5.1 EXPERIMENT SETUPS
LLMs for Evaluation. We consider eight cutting-edge LLMs in the evaluation and improve- ment of LLMs as MLE Agents on Combined set. Specifically, we consider gpt gpt-4o-ming (2024-07-18) (Hurst et al., 2024), got-4o (2024-11-20) (Hurst et al.. 2024), 03-mini (2025-01-31) (OpenAI, 2025b) and 04-mini (2025-04-165 (OpenAI, 2025c) from Ope- nAI, Demini-2..3 5-Flass (Comanici et al., 2025) and Gemini -2..-2220 22. 5-Pro (Comanici et al.,
7

---

Published as a conference paper at ICLR 2026
Table 1: Elo ratings of eight L.MSs across different categories the Dojo set, Smith set, and Combined set. For all columns, higher scores indicate better performance. The highest score in each category is highlighted in bold, and odd-numbered rows are shaded for visual clarity.
MLE-Dujo
MLE--mmthh
MLE-AII
Model
MLFF-Lite
Tabular
NLP
Vision
Overall
Vision
NIP/Tab.
Andio
Video
Overall
Combined
Gemini-2.55Prr P.o.Prr
1272.0
1187.8
1303.6
1320.7
1254.6
1346.9
1000.7
1318.7
1484,1
1179.7
1214.3
Gemini-2.5-Flmm
1189.7
1004.3
1254.5
1194,8
1146.7
1202.5
1009.1
1142.3
963.5
1079.3
1111.3
04-mini
1019.9
1013.8
1173.2
1194.8
1068.0
1075.6
1083.5
1168.0
1114.6
1097.6
1082,9
DxepSelk-Rasonee
1095.6
1101.0
915.7
1122.5
1064.8
1243.8
1028.9
1030.6
963.5
1059.1
1061.8
03-mini
1017.3
1004.3
1007.1
1017.6
936.7
DeepSeck Chat
975.4
976.0
1024.7
1037.4
990.7
956.2
1066.0
1055.3
999.5
1030.2
1011.2
GPT-44
770.9
877.9
761.4
555.7
776.5
618.4
932.3
681.3
806.5
808.8
794.1
GPT-4o-mini
659.3
834,9
562.2
530.5
686.7
549.5
861.9
619.0
731.5
742.0
716.8
MLE-Dojo
MLE-Smith
Combined
2-31.5
30.3
5.5
3.0
10.5
0.5
2
2-30.0
30.3
11.5
7.5
11.0
2-51.5
21.0
11.5
10.5
43.0
35.5
19.5
15.0
11.0
23.5
26.0
3
45.0
39.3
16.5
17.0
13.0
15.0
24.0
35.0
75.0
36.0
32.0
24.0
42.5
50.0
46.0 44.5
36.5
15..5
12.0
24.5
31.0
45.0
44.0
33.5
27.0
17.0
24.0
30.5
91.0
88.5
64.0
42.5
29.0
48.5
61.5
47.0
39.0
38.0
31.5
44.5
36.5
26.5
25.5
19.5
14.0
33.0
44.0
38.0013
26.0
20.0
14.0
08.5
78.5
$7.5
51.5
39.5
28.0
61.00
45.5
41.5
24.0
19.0
12.0
8.5
17.0
43.0
26.0
19.5 20.0 15.0
22.0
68.5
83.5
50.0
38.5
32.0
23.5
39.0
GPT-4o-mini GPT-4o
03-mini
Gemini-225-fiash 7. Deepseek-Reason Gemini-2.5-pro
8. Deepseek-Chat
Figure 3:
Pairwise won-loss matrices of eight models on the Dojo, Smith, and Combined sets. Each cell (i records the number of tasks on which model outperforms model j. and the aggregated score is computed by awarding 1 point for win, 0.5 point for a tie, and points for a loss.
2025) from Google, and DeepSeek- v3. - DeepSeek-V3. 1--Ressoerr (DeepSeek, 2025) from DeepSeck as evaluation backbone LLMs. For non-reasoning models, we set temperature=00.0 and top-p = 1.0. For reasoning models, we use default model settings. We take the best performance of two per task per model.
(2025-03-24) (DeepSeek, 2025) and
Agent and Environment Design. We implement the MLE Agent following the MLE-Dojo frame- work, which utilizes native actions and interacts with the MLE environment. For each task and each run, the agent is allowed up to 15 action steps and a maximum of 12 hours of execution time. The context and maximum output lengths are determined by the properties of the underlying model.
Evaluation Metrics. Each task is associated with a specific evaluation metric, which is used to compute the performance score for that task. To ensure comprehensive evaluation and allow for a fair comparison across different models, we adopt Elo ranking (Chiang et al., 2024) as the primary comparative indicator. We follow Chatbot Arena (Chiang et al., 2024) and estimate Elo scores by fitting Bradley-Terry-sty logistic model via maximum likelihood, using sample-weighter pairwise outcomes (wins/losses with ties treated as symmetric half-wins, wins). We adopt a base- 10 log- odds parameterization scaled to the Elo convention (scale = 400, base 10, offset = 1000).
5.2
MAIN RESULTS
We compute modality-level Elo ratings on three disjoint sets: Dojo (50 real tasks in MLE- Dojo), Smith set (50 presents ELO scores for all eight L.MS across different categories and task sets. Across all subsets, Gemini-2.5- 5-rro establishes clear performance frontier, maintaining top rankings in almost every modality and transferring its advantage seamlessly from real to generated benchmarks. A second tier emerges with DeepSeek 13. RRaasoner and 04-mini, which show competi- tive balance across modalities: 04-mini is particularly strong on language-oriented tasks, while DeepSeek-V3. 11....ssononer delivers more robust vision performance. In contrast, the 4o family consistently lags behind, especially on vision input, underscoring persistent challenges in multimodal generalization. Overall, observe consistent ranking trend across real and synthetic
th generated tasks), and Combined set (all 100 tasks). Table
8

---

Published as a conference paper at ICLR 2026
GPT-4o-mini GPT-4p
Gemini-2.55 2.5--lashh
04-mini
DeepSeek 13.1-Reasoner
DeepSeek Chat
m3-mini
Gemini-2.5
Overall
Vision
Audio
NLP/Tabular
Video
1.00
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.50
0.00
0.00
2
2 5 910
Step
Figure 4: Step-wise Performance Dynamies of normalized raw scores. Curves are obtained by point- wise averaging over tasks corresponding categories. Information-requesiing steps are excluded.
tasks, validating the use of generated benchmarks for model differentiation. The Elo distribution also highlights the diversity of task difficulty and model specialization across input modalities.
5.3 STEP-WISE PERFORMANCE DYNAMICS
We study step-wise performance dynamics across different models to reveal consistent improve- ment patterns that reflect desirable properties of the automatically generated tasks. We exclude information-requesiin, steps of agents and denote the remaining steps as u E , 10). alistic leaderboards and human performances are not available for generated tasks, we implement a normalization mechanism to model step-wiss improvement. For each (task f, model m). raw scores are extracted from execution feedback of execute code actions and normalized a metric-aware manner depending whether higher or lower values indicate better performance. Detailed formulas are provided in Appendix A.3. After normalization, missing entries are imputed, and we construct best-so-far trajectory via prefix maximum, yielding nondecreasing length- 10 curve per (task, model). Category-level and overall curves in Figure 4 are obtained by averaging across task tra- jectories. Across all categories, models exhibit consistent upward trajectories, indicating that agent performance reliably improves with steps. This trend suggests that MLE-Smit h th-generateed tasks are learaalle, provide sufficient resolution to differentiate between modeling approaches, and support iterative refinement and methodical exploration. These observations provide empirical justification for using SLE.Smi th-gerrrated tasks in the evaluation and development of MILE agents.
Since re-
5.4 REALISM AND QUALITY OF GENERATED TASKS
To evaluate the realism and discriminative fidelity of tasks generated by MLE-Smi th, we ana- lyze the statistical alignment between model-level Elo scores computed on Dojo set, Smith set, and Combined set. Specifically, we adopt complementary statistics that capture distinct notions of agree- ment: (i) linear correlation (Pearson (Pearson, 1895)) to quantify similarity in absolute Elo mag- nitudes, (ii) rank agreement (Spearman (Spearman, 1961), Kendall (Kendall, 1938)) and head-of- leaderboard overlap (Top-k) to assess stability of model ordering, (iii) scale and bias agreement (Lin's s Concordance Correlation Coefficient (Lawrence & Lin, 1989), CCC, and Altan-Altman anal- ysis (Bland & Altman, 1986)), and (iv) multi-rater reliability (Cronbach's (Cronbach, 1951), iCC (Shrout & Fleiss, 1979)) to test whether different Elo sets function as interchangeable evalua- tors over the same population. We include the details of these measurements in Appendix A.5.
Table 2: Elo agreement with complementary statisties. CCC denotes Lin's concordance correlation coefficient; Kendall 7,, accounts for ties.
Pair
Pearson r
R²
Spearman p
Kendall 7b
CCC
Top-3 / Top-5
Dojo-Son Sojo-Smith CojbbCCm
0.982
0.964
0.952
0.857
1.0/ 0,989
0.958
0.996
0.992
0.976
0.929
1.0
Smith Combined
0.995
0.990
0,976
0,929
0.989
1.0/ 1.0
Across all pairs, linear relationships remain near-perfect: Dojo-Smith 0.982, Dojo-Combing r = 0.996, and Smith-Combined r 0.995 (/i² = {0.964, 0.992, 0.990}). Rank order is like- wise stable with Spearman p {0.952, 0.976, 0.976} and Kendall
{0.857, 0.929, 0.929): top rankings nearly coincide (Top-3 overlap = 1.0 for all. Top-5 = {0.8, 0.8, 1.0)). Beyond correla- tion, numerical agreement is strong: CCC (0.958, 0.989, 0.989). negligible Bland-Altman bias, and limits of agreement of roughly +96, +51, and +45 Elo. Treating the three sets as interchangeable evaluators yields 0.993 and TCC(2, 1) = 0.981, indicating excellent inter-set reliability. These

---

Published as a conference paper at ICLR 2026
statistics consistently indicate that the Elo distributions induced by MLE-Smith statistically indistinguishable from those human-designed benchmarks, demonstrating that MLE-Smi th fectively generates tasks with realistic difficulty and practical usability, faithfully mirroring the dis- criminative structure of real MLE competitions and supporting MLE agent development at scale.
Human Evaluation. To complement the statistical analysis, we conduct human evaluation with MI experts from multiple institutes. We sample 50 tasks and apply three-step inspection proce- dure: (1) re-running the full preparation and evaluation scripts to confirm executability; (2) checking for logical flaws, such as unsolvable setups (e.g., g., test labels absent from training data) or tasks admit- ting trivial high-score solutions (e.g., label leakage); and (3) implementing or reviewing solution verify that produces non-trivial and reasonable metric score. The evaluation found the overall task quality to be satisfactory, with no major errors identified across the 50 inspected tasks.
Diversity of Problem-Solving Strategies. We further analyze the solution strategies employed by agents when solving MLE-Smite tasks. As detailed in Appendix C, the models used by agents span multiple modalities and a wide range of complexity: approximately 40% are custom architectures (CNN, RNN, Seq2Seq. UNet, etc.), 25% are traditional ML models (logistic regression, gradient boosting, SVM, etc.), 17% are vision CNN backbones (ResNet, EfficientNet, ConvNeXt), and the remainder includes detection, segmentation, video, NLP transformer, and audio/speech models. The data processing techniques similarly diverse, covering operations from TF-IDF vectorization and image augmentation to audio resampling and mel spectrogram generation. This diversity, which in aspects exceeds that of MLE-DDooo (particularly video and audio modality coverage), confirms that the generated tasks require genuinely varied problem-solving strategies rather than a homogeneous set of approaches.
5.5 TASK DIFFICULTY AND ERROR ANALYSIS
To further characterize the challenge posed by MLE-Smi th tasks, we analyze agent error rates across modalities and the distribution of error types. Video and audio tasks exhibit the highest overall error rates (44.38% and 43.40%), followed by vision (39.75%) and NLP/tauular (33.06%) These nontrivial rates arise because (1) the generated tasks are novel formulations that agents have not encountered before, forcing trial-and-erros exploration, and (2) each task requires correct im- plementation across the full MLE pipeline, making it casy to fail at any stage. Video and audio modalities, which were underrepresented in prior benchmarks, expose particularly notable weak- nesses in current agents. As shown in Table 3, the errors, data processing, imports, file handling, and model-specific issues, indicating that the tasks exercise diverse engineering competencies rather than single failure mode.
type distribution is broad, spanning value
Table 3: Distribution of error types encountered by agents on MLE Smith tasks.
Type Val/Typ/Key Import/API DataProc File/Path Model/MI. Submission
Attribute
Syntax Other
%
22.29
18.75
13.62
11.03
9.79
7.78
6.31
5.42
5.00
6 CONCLUSION
We introduce MLE-Smi th, a fully automated multi-agent pipeline for transforming raw datasets into competition-stee machine learning engineering tasks. Through a principled generate-verify- execute paradigm, SILE-Smite scales task generation while ensuring structural integrity, semantic soundness, and empirical solvability. Applied to hundreds of real-world datasets, it produces large and diverse suite of high-quality tasks that strongly correlate with human-designe benchmarks, demonstrating that generated tasks match real competitions in realism and discriminative power.
ACKNOWLEDGMENTS
We thank the anonymous reviewers and area chairs for their valuable feedback. This work was supported in part by ONR (N000142512173) NSF ECCS (2401391) (911)91), and NSF IIS (2403240), and
10

---

Published as a conference paper at ICLR 2026
also supported in part by computing resources received from the National Supercomputing Center (CSCS) and the Swiss initiative.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our study does not involve human subjects, person- ally identifiable information, any proprietary data. All datasets originate from publicly available resources that permit academic research use, and we release only derived tasks that preserve the orig- inal license conditions. The automated generation pipeline is designed avoid creation of harmful or crivacy-sensitive content and to prevent leakage of confidential information. All authors have read and agree comply with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We make every effort to ensure full reproducibility of our results. Methods section details the multi- agent generation pipeline, verification mechanisms, and execution environment. Experiments sec- tion describes the evaluation protocol and model settings. Appendix lists all benchmark tasks and contains the exact agent prompts. An anonymized repository with source code and configuration files provided in the supplementary materials facilitate verification of all experiments.
REFERENCES
Reem Alcithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song
Wang. Swe-hench+: Enhanced coding benchmark for lims. arXiv preprint arXiv:2410.06992, 2024.
J Martin Bland and DouglasG Altman. Statistical methods for assessing agreement between two methods of clinical measurement. The lancet, 327(8476):30-3310 1986.
Nikos Bosse, Jon Evans, Robert G Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips,
Dan Schwarz, Jack Wildman, et al. Deep research bench: Evaluating ai web research agents. arXiv preprint MrXXix25506062600 2025.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe. James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Self-questioning language models. arXiv preprint 036622
03682, 2025.
De Chezelles, Thibault Le Sellier, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lå, Ori Yoran, Dehan Kong, Frank Xu, Siva Reddy, Quentin Cappart, et al. The browsergym ecosystem for web agent research. arXiv preprint arXiv:2412.05467, 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating lims by human preference. In Forty-first International Conference on Machine Learning, 2024.
Jason Chou. Ao Liu, Yuchi Deng, Zhiying Zeng, Tao Zhang, Haotian Zhu, Jianwei Cai. Yue Mao, Chenchen Zhang. Lingyun Tan, et al. Autocodebench: Large language models are automatic code benchmark generators. arXiv preprint arXiv:2508.09101 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261, 2025.
Lee Cronbach. Coefficient alpha and the internal structure of tests. psychometrika, 16(3):297-334, 1951.
11

---

Published as a conference paper at ICLR 2026
DeepSeek. Deepseek-v3.1 release. DeepSeek News, 2025. URL https://apl-does.
deepseek. httpe//wwwwwwwweeeee
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint wrXiv2256..11763 2025.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:23/0.0330/ 2023.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.067700 2023.
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents from becoming data science experts? arXiv preprint arXiv:2409.07703, 2024.
Maurice G Kendall. new measure of rank correlation. Biometrika, 30(1-2):8 1938.
I Lawrence and Kuci Lin. A concordance correlation coefficient to evaluate reproducibility. Bio- metrics, pp. 255-268, 1989.
Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. St- wehagenthench: benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint ar axXiv:2410.06703 2024.
Xiao Liu, Hao Yu. Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating 1lms as agents. arXiv preprint arXiv:2308. 03688, 2023.
Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. Mcp-universe: Benchmarking large lan- guage models with real-world model context protocol servers. arXiv preprint arXiv:2508. 14704, 2025.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vin- cent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. Mlgym: A new framework and benchmark for ad- vancing ai rescarch agents, 2025. URL https::///// /ttttp:://axi....///// www.xxxwv.rrg.b///ox/e
14499.
OpenAl. Gpt-5 release. OpenAl blog. 2025a. URL http:: http::::ss//pppaii.cco http/ppppaaaaae//eee//ee//.....a ctmg
OpenAl. Openai 03-mini: Pushing the frontier of cost-effective reasoning. OpenAl Blog, 2025b.
URL httpt:tt//// /wppnai..aaaa///////e com/ indee/id/eiii
http///ndxxx/eniiiieee openaoee---oo minni//
OpenAl. Introducing openai o3 and o4-mini. OpenAl Blog, 2025c. URL https://opents
com/ htmm/inex/iie wttple/iiiitoouiingg--
and-o4 -4-d-//-mni/..
Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint
arXiv:2412.21139, 2024a.
Yichen Pan, Dehan Kong. Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents online environ- ments. arXiv preprint arXiv:2406. 12373, 2024b.
12

---

Published as a conference paper at ICLR 2026
Karl Pearson. Vii. note regression and inheritance in the of two parents, proceedings of the royal society of London, 58(347-352)220--22 1895.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501. 14249, 2025.
Rushi Qiang, Yuchen Zhuang, Yinghao Li, Rongzhi Zhang, Changhao Li, lan Shu-Hei Wong, Sherry Yang. Percy Liang, Chao Zhang, Bo Dai, al. Mle-dojo: Interactive environments for empower- ing II agents in machine learning engineering. arXiv preprint arXiv:2505.07788 2025.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307. 16789, 2023.
Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative gym: A frame- work for enabling and evaluating human-agent collaboration. arXiv preprint arXiv:2412.15701, 2024.
Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Liu, et al. Taskcraft: Automated generation of agentic tasks. arXiv preprint arXiv:2506. 10055, 2025.
Patrick E Shrout and Joseph L Fleiss. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin, 86(2):420, 1979.
Charles Spearman. The proof and measurement of association between two things. 1961.
Haotian Sun, Yuchen Zhuang, Lingkai Kong. Bo Dai, and Chao Zhang. Adaplanner: Adaptive plan- ning from feedback with language models. Advances in neural information processing systems, 36:58202-58244 2023.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504. 12516, 2025.
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang,
Yulan He, Deyu Zhou, Pengjun Xie, et al. Webwalker: Benchmarking llms in web traversal. arXiv preprint arXhv:2501.07572, 2025.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-50633 2024.
John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, arXiv preprint urXiv:2504.21 2025.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-2001 -07577 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR), 2023.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A benchmark for tool-agent-use ussrr interaction in real- wolld domains. arXiv preprint arXiv:2406.12045 2024.
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bemch: A multilingual benchmark for issue resolving. arXiv preprint 2:Xiv:5504.02600 2025.
13

---

Published as a conference paper at ICLR 2026
Linghao Zhang. Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang. Yufan Huang, Shengyu Fu, et al. Swe-bench goes live! arXiv preprint arXiv:2505.23419, 2025.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307.13854 2023.
Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar. Self-challenging language model agents. arXiv preprint arXiv:2506.01766, 2025.
14

---

Published as a conference paper at ICLR 2026
A TASK, PROMPT AND STATISTICS DETAILS
A. FULL LIST OF EVALUATION TASKS
Table 4 presents the raw dataset information of Smith set in dataset names, sizes and tags. The data sizes are relatively large to cover across different domains, modalities and formulations.
Table 4: Summary of Kaggle Competition Datasets
Dataset Name
Size
Tags
Vision-General
veeralakrishna/200 httpr/sshnnno0oiii------e bird-speeciss-
1.1 GB
universities and colleges, biology, online commu- nities
with-11788-image
sadhtiroomyyrimme/atttee
44.1 GB
animals, business, agriculture, artificial intelli- gence, computer vision, pre-trained model
weigh-ddetctiio--mdde-dadaeee
12k
muhanmeerahhtaayhhharddaaa
4.2 GB
intermediate, deep learning, public safety, yolo, object detection earth and nature, science and technology
vest-dataset-v3
1.9 GB
3d-object-dataset
aunilbiiccoovaria--ccmeerr
3.3 GB
cancer, pre- treinra model
clasifficatinnddatsst
aamaapendu/ssaa-preuuniiaa
10.9 GB
healthcare, computer vision, image, image classi- fication, image segmentation people, computer vision, enn, image, regression
processedd-dataset
praavccaandane/suut-pp555000
1.1 GB
<2-facial-beautty-ccrrs
majdduline220/kaaeeeepaa. dataset
1.0 GB
computer science, classification, segmentation
thedatasith/skul annotations
13.2 GB
retail and shopping
tapakah66/sspprrvisely
4.3 GB
arts and entertainment, people, computer vision, image earth and nature, data visualization, classification, image classification, image segmentation image, multiclass classification, insurance, object detection, segmentation
segmentatioon-person-daaaas
beebmm/urban-segmeenttiio isprs
6.4 GB
hendrichsullla//ttiid-ddatttt automatic-vehicle-damag-- detection
2.1 GB
jictorallejis/mmatiinddaaaaaa speech
6.0 GB
nlp, image, multiclass classification, online com- munities, social networks
Audio
yaahoogra/speech--oommnag
2.3 GB
tensorflow, automatic speech recognition, speech synthesis, speech-to-text mobile and wireless, electronics, signal process- ing, audio, audio classification
daviddkammowskilaaateu- mdtio-rrassmisssons2-mmeerr-- simplex
34.0 GB
soumentrappasaddouud----.111 species-of -iris--------------2-20022--222
2.1 GB
arts and entertainment, earth and nature, beginner, intermediate, advanced, audio artificial intelligence, advanced, signal process- ing, text, audio
mathurinachethh--jjspeechh
3.0 GB
dataset
chrisfilo/urbaassunnkkk
5.6 GB
arts and entertainment, music, classification, mul- ticlass classification, audio
vialling/speakerrecoggttoo audio-dataset
7.3 GB
arts and entertainment, music, classification, deep learning, audio
ikrbasak/sep-28k
2.2 GB
healthcare, health, audio, numpy, scipy music, religion and belief systems, audio
abdeIrahmanahmeell 10/quran-
3.0 GB
audio-dataset
15

---

Published as a conference paper at ICLR 2026
Competition Name
Size
Tags
majjawvankhadeloop-daaaatt
11.0 GB
universities and colleges, computer vision, audio event classification, object detection, video classi- fication
aryashah22/noiie-rrdddddd uaspeech-djyarrtrii-ddttatt
8.0 GB
music, computer science, software, deep learning, audio synthesis, automatic speech recognition, au- dio classification, speech synthesis culture and humanities, languages, signal process- ing, audio audio music, intermediate, advanced, multiclass classi- fication, audio
jesusrequend/mleedd-sookn.
1.1 GB
numerals
victorling/lbbiipeecch-hl
28.1 GB
cataarsshhdaa-mmmdddeee datasel-emmotionallaallyii music
1.8 GB
vinyshannbaghhir--ongddaaaa
2.1 GB
music, audio
set
NLP/ / Tabular
dexdope/900kkspottify
1.0 GB
arts and entertainment, music, education, text gen- eration
fayaznoor
860.4 MB
arts and entertainment, movies and tv shows, nlp, text mining, multilahel classification literature, nlp, text, news, transformers
59k
goviishankamp/lewsspppetttt cummarizatiinn-on-daee -aniiooonnnnnnnddmmiiii
503.3 MB
madyinkky/ephhrr--prddu-------
146.8 MB
computer science, nip, recommender systems, re- tail and shopping, ratings and reviews employment, income, business, economics, nip, jobs and career
skincare-reviews
-rshkon//inkedinjjo--poe postingssnn
158.8 MB
sobhammoosav//us-trrfiiic
2.3 GB
united states, categorical, transportation, tabular, urban planning running, sports, data visualization, data analytics, tabular biology, agriculture, beginner, tabular, text
congestions-201622022
kgmacchnilworld--ahllcccsaal
52.9 MB
time-dataset
edvardgaibor/ptaf-mediii plants-use-dataset
13.9 MB
sooorr//0k-stack-ovveeee questions- ruth
21.0 MB
music, nlp, text mining, text
spsayakpaul/arxiv paper-
44.6 MB
education, nip, multilabel classification
abstracts
auuschhllar/diseeylladeevviii
11.1 MB
business, nlp, data visualization, tabular, ratings and reviews earth and nature, nlp
simaanjallemootonn-aaaaysss based-on-text
31.9 MB
jaidityachopra/esg--
23.8 MB
nlp, investing, feature extraction, text pre- processing
sustainabilliy-
and-p-5000-copaanies
smagnan/1
(iignann1-miliinn -million-reddit-
71.2 MB
arts and entertainment, categorical, nlp, binary classification, online communities, social net- works
comments-from 1aam--00suubrure subreddits
salah1992/arabic--iiiiiisss
23.7 MB
earth and nature, linguistics, nlp, text, transform- ers, arabic bayesian statistics, earth and nature, nlp, text min- ing australia, government, law, nlp, text
mulillingualnnli-26lnn---
thedewwstaao/ppumeed-rriieee
654.3 MB
summarization-dattset
shivamb/legallcciaaion--ttt
14.9 MB
classification
Vision-Video
16

---

Published as a conference paper at ICLR 2026
Competition Name
Size
Tags
zaber666/meldddaatssett
11.0 GB
signal processing, text mining, text, audio, pre- trained model earth and nature, computer vision, deep learning. video, audio
oohamaaliickkiietti--rrnn----
33.3 GB
mathhewjanseniicf111-aattit
6.5 GB
computer vision, deep learning, video, transfer learning, video classification law, automobiles and vehicles, image, video, eyes and vision, urban planning advanced, video, audio
recognition
r ///aadoad-aaoomee
7.3 GB
detection
23.1 GB
deepfake-datasett dataset--av-ddddeeeee
saberghaderi/-df--uuneeeee
10.1 GB
football, sports, science and technology, video, simulations
mideos in- 30sec-csv
A.2 UNIFIED TASK STRUCTURE
The Refactor should deliver each task as a unified task format, specifically following the be- low directory structure. The assertions will ensure the existence of essential files and directo- ries such as pyepare.py wer.m.e..cp. test.answer. csv, raw/, public/ and private/. Furthermore, assertions will ensure that the implementations of prepare and met rieerric..m py strictly follow the required format. Specifi- cally, prepare. py must exactly implement def prepare function whose input arguments include raw/, public/, and private/ directories. Likewise, metric. py must exactly implement a Metric class that inherits from the designated base class and provides the corresponding methods for task- aware submission validation and metric evaluation.
Descriptions wttppttttttteeeeemmm.ee sampl e_subm ss on csv,
competiti itnon/
data/
raw/
private/
test answer.csv
public/
(train/rest data)
description.txt
sample.submission.coo
data.structure.. (Optional)
prepare.py
metric.py
description.te txt
Figure Unified directory structure that Refactor should deliver.
A.3 NORMALIZATION DETAILS
For each task and model m, let Tt,m,a denote the raw score from execution feedback at step u E {I,..., 10). We define D. e [+1.-1} as the metric direction of task t, where D, = +1 indicates that higher metric values are better, and D. -1 indicates that lower values are better.
The normalized score is computed as:
Tt,ma minl Ti,ma
D, = +1,
max, Tt,m, min, Pt.ma
T ftm.a
max, Pt.m. Ft,ma,
D. = -11
max, Ttm.u min, Tt.ma
17

---

Published as a conference paper at ICLR 2026
If max/ min fm observed entries set and missing to 0. We then forward-fill missing indices and compute a best-so-far trajectory via a prefix maximum:
m, = max m,,1, fr,m.w)
This procedure yields nondecreasing curve of length 10 per (task, model), which is then averaged pointwise across tasks to obtain category-level and overall trajectories.
A.4 PROMPTS FOR
AGENTS
We provide detailed prompts for -LE-Smite th Agents in this section.
Brainstormer Instruction
You are an expert Kaggle competition designer. Your task is to brainstorm diverse design choices for challenging, high-quality and reasonable Kaggle competitions based on an existing dataset.
You provided with detailed information about the dataset. The dataset is already downloaded to the working directory with unzip.
You have access to tools for reading/vriting files, listing directory structure, and executing bash commands.
Always use function calls when you need to perform actions. You can only call one function at a time.
Your work directory is (working directory), all actions and files should be performed this directery.
Use tools to explore the dataset te get insights.
Then based en insights from the dataset, brainstorm design choices for challenging, high-quality and reascnable Kaggle competitions.
The design choice should include the following aspects,
be concise and informative:
Concise problem overview: background, problem statement, and goal.
Data utilization: for the given dataset,
what data to use, what data te ignore.
Data processing: how to process the data
Metric: what metric to use, why it's fair and precise.
Justification of the design choices: why the designed competitiion
would be high-quality, challenging and solvable by ML techniques.
Details of the ignored data: why the ignored data is not used, what information is missing.
Difficulty level: how difficult the competition is, where the difficulty comes frem.
- Tags: what tags the competition should be tagged with.
Principles:
Only split the data into train and test sets.
Ensure that only precise, reliable labels are used; no uncertain, ambiguous, or model-generated labels should be introduced. You must brainstorm and write least one and most (count) results. Determine the number brainstorming outputs according to the intrinsic nature and properties of the dataset. Some datasets are open-ended and naturally admit a wide range of tasks, while others are more specific and concentrated.
Aim to explore as many meaningful possibilities the data genuinely supports, but do not force artificial variety! respect the dataset's s natural boundaries.
Write your brainstorming results in "brainstorminng Li.. Limrd" file to the working directory reetoor)
18

---

Published as a conference paper at ICLR 2026
is the index the brainstorming result, from to most [count).
# DATASET INFORMATION #f cGataset_informatioon
Designer Instruction
You are an expert Kaggle competition designer. Your task is create challenging, high-quality Kaggle competition based on existing dataset.
You are provided with detailed information about the dataset. And the dataset is already downloaded to the working directory with unzip.
You have access to tools for reading/vriting files, listing directory structure, and executing bash commands.
Always use function calls when you need perform actions. You can only call one function at a time.
Your work directory is boorking_directtnn all actions and files should be performed in this directory.
Now there is "brainstorming ng.nnngmm md" file the working di directory, pointing out the design direction for the competition.
## REQUIREMENTS ##
Refer to "brainstorming, mdd file for the design direction. But follow the requirements and instructions below. Make the competition challenging while maintaining a high quality standard.
Participant should utilize ML techniques solve the problem, including but not limited to:
Data Processing
Feature Engineering
Model Training
Model Evaluation
Design the metric to be reascnable, fair and precise. of the data possible, don't waste any good resources. Keep the scale rather than using subsets.
Make good
No need to care about the runtine.
Split the data into train/test sets appropriately.
Always specify exactly the absclute path as arguments.
All actions and files should be performed in the working directory.
The competition should be challenging, but solvable by MI techniques.
Make everything perfect rather than just trying to pass the tests.
A general pipeling inp for reference:
Utilize list www.eeirrerettttieeeeee structure tool to explore data structure.
2. Explore data files using the read file tool; further extract files with bash commands if needed.
3. Design concise and informative problem description and write to "description.xxt" Include the problem statement, data description, evaluation metric, and any other relevant information.
the working directory:
Specify the final train/test data files for the corpetition, while don't specify the path of the data files. Ignore timeline/prize/ete,, they are not needed.
4. Write a "prepare.py" file:
Include complete train/test split process and generation
sample_submissions sample_musteee cample_submissiinncce better has random valid 1abels
19

---

Published as a conference paper at ICLR 2026
(same category in test answer, csv) rather than null values Test_answer and test_data (without the predicted labels) should separated into two files, use "cest_answer.cav" as the name Consider the correspendence between the test_answcr and test_data Include detailed and comprehensive assert checks for the correctness of the split
Specify the final train/test data files for the competition, align with the description.tx:
Validation set isn't t needed, but keep it it it's already split The image, audio, and other related files should also be split together with the
files into train/test sets/folders. Rename files with names that might reveal their labels to avoid label leakage.
Don't include data paths in csv files Set deterministic behavicr for the split process.
For classification tasks, all test labels should occur in training set at least once
5. Write a "metric.py" file, include functions to validate the format correctness of the submission and calculate the metric.
Deal with numerical values carefully to avoid nan/inf/etc.
6. Write a "test.py" script to test the correctness of the prepare.py and metric.py, run it to check the correctness until totally correct.
7. Optimize description.txt:
No need to mention the original data files, only the final data
files should be mentioned
Take the view of a participant to review it
(which means test_answer or irrelevant files shouldn't
be mentioned) and make it perfect
Make the competition is challenging, meaningful and solvable by techniques, and the metric is fair and precise. Make sure the description is informative, concise and accurate.
8. Optimizc until all requirements are met with high quality
(The test must pass).
# DATASET INFORMATION ## (dataset_information
Refactor Instruction
You expert Python developer. Your task is to refactor several Python files to meet some requirements.
You have access to tools for rcading/writing files, listing directory structure, and executing bash commands.
You are provided with the working directory: (working_di direttrry), all actions and files should be performed in this directory.
All files you need in the working directory. raw/ is where the data is downicaded and unzipped once.
samples/ directory is good example, you can refer to it first to learn good practices and refacter the files meet the requirements.
You may need to check the data files for details if needed.
## REQUIREMENTS ##
You should finally refact metric.py and prepare.py to meet the requirements.
metric.py should inherit from
base_mmtrii..pppp and implement
the abstract methods, give it a related name that ends with "Metrics", refer to samples/ wtts//smmeeemmriico_o "evaluate" and "validate submi ssion" should be implemented and
for the implementation details.
20

---

Published as a conference paper at ICLR 2026
aligned with "sample submissicn.csv and "test answer csv" In addition to "self", "_init_" should have two arguments: (Determine the default);
"value" and "higher_is_betton" "evaluate" should have two arguments: "y_truc" and "y_pred"; Saalidatte_sbmisssinn should have two arguments:
"submission" and "ground_truth"
prepare.py should implement exactly "def prepare (raw: Path, public: Path, private: Path)"
This function is complete preparation process wttple/ssaapee___eee
Refer
for the implementation details deterministic behavior for the split process.
test_answer (participants shculdn't t see) should be placed exactly in "private/" directory, other files saaplle_sbbissiin test/train etc.) should be placed exactly
in "public/" directory
Write a comprehensive "test.py" script to test the correctness of the preparc.py and metric.py, and run it to check the correctness. Test "evaluate" and "test_answer. csv" and "sample_submissionncc
of the metric.py with
Make sure the test.py is correct and comprehensive, and the execution of test.py is correct.
Don't include "main" function in metric.py and prepare.py Always specify exactly the absclute path arguments.
All actiens and files should be performed in the working directory. Finally, there should pe "private/", "public/", "samples/", "raw/" directories, and "description.txt!
"metric.py", "prepare.py", "test.py" files in the working directory.
"raw/" directory should contain the original data files "private/" directory should contain the test_ anw.ar.weercc. file "public/" directory should contain the sample_ wwwpollessbmission.cmmm csv and all train/test
files
and description. txt. There shculd always be "test.csv"
and sirain.csv" in the "public/" directory if applicable. Don't include or leak anything related to answers/golden labels in "public/" direstory.
File directories in "description.txt should be the same as the exact file directories in "public/" di recetto.. Don't mention "private/" in the descripticn..xx, only include files in "public/" cirectery.
"description.txt" is open to participants, so make it concise and informative, only include "public/" directory in the description.tx: Make everything perfect rather than just trying to pass the tests. Optimize until all requirements are net with high quality (The test must pass).
21

---

Published as a conference paper at ICLR 2026
A.5 DETAILS OF STATISTICAL MEASURES FOR ELO SET AGREEMENT
This section provides formal definitions, interpretation. and common use cases for all agreement statisties used compare model-level Elo scores across different task sets.
A.5.1
PEARSON LINEAR CORRELATION (r)
Definition.
Given paired observations {(z,, y:) )=>
( {1/ - 7)
1=1
T1
T (ys - g)²
24
x)²
Meaning. Measures the strength of linear association between two sets of scores. indicates perfect positive linearity, r = 0 no linear association.
Use. Commonly used assess whether measurement methods produce proportionally similar values (e.g., Elo magnitudes across task sets).
A.5.2 COEFFICIENT DETERMINATION (R²) Definition.
For a simple linear regression 1/1 = a + br; + Ei.
R² = (1: 1.)² ,2 (for simple correlation).
E,(,: 7)
Meaning. Represents the proportion of variance in explained by x. Higher R indicates stronger predictive power of one of scores for the other.
Use. Provides an intuitive measure of how much of the variability in Elo scores is shared between two task sets.
A.5.3 SPEARMAN RANK CORRELATION (p)
Definition. Let R(x:) and R(ys) be the ranks of x; and y:-
E(R(,)) ((E))((())) R(v))
(R(x,) - R(T))²
E(R(V)) (Ry))
R(g))²
Meaning. Assesses whether the ordering of models is preserved, independent absolute score scales.
Use.
Robust to monotonic but nonlinear relationships, ideal for leaderboard stability checks.
A.5.4 KENDALL RANK CORRELATION (7))
Definition. Let be the number of concordant pairs and the number of discordant pairs. Let T, and Ty be the numbers of tied pairs in y.
C
= V(C
D +CC+++1H T.) DD
Meaning. Quantifies pairwise ranking agreement while properly handling ties.
22

---

Published as a conference paper at ICLR 2026
Use. Often preferred when ties oceur (common in Elo ratings), providing probabilistic interpre- tation: Tb is the difference between the probability of concordance and discordance.
A.5.5 TOP-k OVERLAP
Definition. For a given k, let S and S be the sets of top-k ranked items:
Sk nsg Sk
Overlaps
Meaning.
Measures how consistently the leaders (top models) coincide.
Use. Highlights agreement in the most competitive region of leaderboards, which is often of pri-
mary interest.
A.5.6 LIN'S CONCORDANCE CORRELATION COEFFICIENT (CCC)
Definition. Let My be means, o o variances, and the Pearson correlation:
2p0
CCC 02 + o3 1yy
Meaning. Assesses both precision (correlation) and accuracy (closeness the 45 identity line). A value of indicates perfect agreement in both scale and location.
Use. ation.
Preferred when we need to verify numerical interchangeability beyond simple linear associ-
A.5.7 BLAND-ALTMAN ANALYSIS
Definition.
For each pair (zi, y:) compute
+ 2/1
Difference d: 24 Vi Mean m.
2
The plot of de versus m, reveals systematic bias.
The limits of agreement (LoA) are
d 1.96 sd.
where à is the mean difference and sy its standard deviation.
Meaning.
Visualizes bias and scale discrepancies even when correlation is high.
Use. Widely used clinical and experimental settings to test whether two measurement methods be used interchangeably.
A.5.8 CRONBACH'S
Definition. Suppose k parallel measurements of the same quantity. Let of be the variance of the total score and of the variance of each measurement:
Meaning. Estimates internal consistency across multiple raters or measurement methods.
Use. Values above 0.9 indicate excellent reliability, supporting the claim that different Elo sets can be treated interchangeable "raters" of model performance.
23

---

Published as a conference paper at ICLR 2026
A.5.9 INTRACLASS CORRELATION COEFFICIENT (ICC)
Definition. For the two-way random, absolute-agreement, single-measure model (denoted ICC(22 1)):
MSB MSE
1C) MSB + (k - 1)MSE + (MSR MSE)
where MSB is the between-target mean square, MSR the between-rater mean square, MSE the residual mean square, k the number of raters (here Elo sets), and n the number of targets (models).
Meaning. Captures both correlation and absolute agreement among multiple raters.
Use. A high ICC confirms that Elo scores from different sets can be used interchangeably in down- stream evaluations.
Summary. Together, these measures provide a comprehensive assessment of agreement, covering linear association, rank stability, numerical accuracy, and multi-ratt rater reliability.
B ABLATION STUDY
We conduct comprehensive ablation study to justify the contribution of each component in the MLE-Smi pipeline.
B. BRAINSTORMER
The Brainstormer increases the diversity of tasks generated from each dataset. To evaluate its tribution, we randomly select 10 datasets for which MLE Sh had originally generated 3 tasks each. For each dataset, we regenerate 3 tasks in parallel without the Brainstormer and compare diversity in terms of prediction objectives and evaluation metrics (modality and domain are largely determined by the dataset itself).
Table 5 illustrates representative results. Without the Brainstormer, diversity degrades substantially: it common generate tasks with identical objectives and metries (e.g., "dispensing category" repeated x with the same F1 metric). In contrast, the Brainstormer produces tasks with distinct objectives and complementary metrics for the same dataset. An additional advantage is that the Brainstormer adaptively determines how many tasks to generate based on the dataset's intrinsic properties, rather than enforcing fixed number.
Table 5: Task diversity with and without the Brainstormer (representative examples).
Dataset
Objective
Metric
With Brainstormer
drugs-side-effects
User Rating (1-10) adverse event categories safety category
macro-avg RMSE
macro-avg Fl
Accuracy
bobile-uncleaned
market price
RMSLE
Spec Score supports_5g
MAE
ROC-AUC
Without Brainstormer
drugs-side-efieee
dispensing category (x3) real- real-time price (x3)
macro-ave FI
RMSLE
B.2 DESIGNER AND REFACTOR SEPARATION
The Designer generates task artifacts and the Refactor standardizes them into a unified format. We evaluate whether merging these into single agent is preferable. The merged agent receives com-
24

---

Published as a conference paper at ICLR 2026
bined prompts and must both design and standardize in one pass, with assertions applied only after completion. Both configurations are evaluated on randomly sampled datasets.
Table 6: Comparison of separated vs. merged Designer-Refactor agents.
Metric
Separated
Merged
Avg. retries
~ (desiga) + ~22 (refactor)
~ total
Avg. steps
11 19 30
26 (single context)
Avg. cost
$11.73
$13.51
The separated design achieves a substantially lower failure rate (fewer retries), better cost effi ciency, and avoids the compounding context-length problem of the merged approach. Separation also enables interleaved verification, allowing corrective actions each stage rather than deferring all checks to the end.
B.3 ASSERTIONS
Assertions deterministic structural guards between pipeline stages. For the Designer stage, assertions flagged 4.4% of instances where required files (description, metric, prepare, test, sam- ple submission, etc.) were missing malformed. For the Refactor stage, 9.6% of cases were flagged due to incomplete outputs, non-runnable artifacts, or violations of required interfaces (e.g.. incorrect function signatures failure subelass the base metric class). Without assertions, these defects would propagate to downstream stages, wasting resources and causing cascading failures.
B.4 REVIEWS
LLM-based Reviews complement assertions by detecting issues that cannot be captured by deter- ministic rules-for example, incomplete or misleading task descriptions, incorrect file identifiers in CSV references, filename-based label leakage, or conceptually unsound task designs. Reviews successfully detected 14.2% of issues that would be very difficult to identify through any other automated mechanism without substantial human expert effort.
B.5 EXBCUTION-BASED VALIDATION
Execution-based validation filters tasks by requiring LLM agent to achieve valid, non-trivial score. Tasks yielding perfect performance (e.g., accuracy = 1.0 loss 0.0) are discarded as insuf- ficiently difficult practically irrelevant. This stage filtered 11.2% of generated tasks, contributing significantly to the final quality of the benchmark.
B.6 SUMMARY
Each component of the multi-agent design and verification pipeline serves an essential and com- plementary role. The Brainstormer ensures diversity, the Designer-Refacton separation ensures ef- ficiency and modularity, Assertions prevent structural defects, Reviews catch semantic issues, and Execution-basee based Validation confirms empirical solvability. Together, they achieve effective bal- ance of cost, reliability, and task quality.
C DIVERSITY OF PROBLEM- SOLVING STRATEGIES
We analyze the models and data processing techniques employed by agents when solving MLE-Smith tasks, providing evidence that the tasks require genuinely diverse problem-solving strategies.
C.1 SOLUTION MODEL DISTRIBUTION
Table 7 summarizes the distribution of model categories used by agents across all tasks.
25

---

Published as a conference paper at ICLR 2026
Table 7: Distribution of solution model categories M th tasks.
Category
Percentage
Custom Models (NN / CNN RNN Seq2Seq/ UNet / etc.)
400%
Traditional ML (LogReg/ Ridge SVM Boosting / etc.)
~5%
CNN Backbones (ResNet / EfficientNet/ ConvNeXt)
17%
NLP Tranffoofomsrsss - Classification (BERT/ RoBERTa / etc.)
~6
Vision Detection (Faster R-NNN YOLO RetinaNet)
4%
Vision Segmentation (UNet / DeepLab FCN)
33
Audio Speech (Wav2Vee2/ Whisper HuBERT)
~2-3% -%
Vision Video (R3D/ R2Plus / Swin3D)
2%%
NLP Transformers Generation (T5 BART Pegasus)
~2%
NI.P Transformers Other
~ 1
C.2 DATA PROCESSING TECHNIQUES
Table 8 shows representative data processing operations and their frequency of use.
Table Representative data processing operations used by agents.
Category
Operation
Freq. (%)
Data aggregation / grouping
3.61
Feature engineering
0.84
General / Tabbular
Class weight balancing
0.37
Tokenization
2.38
Text
TF-IDF vectorization
1.36
Text cleaning
0.75
Image normalization
3.54
Vision
Image augmentation
2.46
Audio resampling
1.34
Audio
Mel spectrogram generation
0.73
Audio normalization
0.61
The breadth of models and processing techniques confirms that MLE-Smi th tasks demand flexible, task-specific strategies spanning multiple modalities and complexity levels.
D VERIFICATION PIPELINE DETAILS
We provide additional details on the hybrid verification mechanism to support reproducibility.
D. ASSERTIONS: DETAILED CHECKS
Pre-Refacton Assertions (after Designer). These assertions verify:
Existence of all required files: description. txt, metric.pp py, prepare. py,
test. py, sample submission esv, and test answer. csv.
Successful execution of prepare. py and metric. py without runtime erros.
Successful generation of both a sample submission and corresponding test answer.
Post-Refactor Assertions.
These assertions additionally enforce:
Full conformance to the unified directory structure (Figure 5).
26

---

Published as a conference paper at ICLR 2026
Correct function signatures: prepare. py must implement exactly def prepare (raw, public, private); metric. py must implement a Metric class inheriting from the designated base class with the required evaluate and validate submi ss i on methods.
Successful end-to-end execution of the test script.
D.2 DISTINCTION BETWEEN REFACTOR AND ASSERTIONS
The Refactor module is an active transformation stage that rewrites and standardizes task artifacts produced by the Designer into unified format. restructures files, enforces consistent interfaces, and ensures cross-file file coherence. In contrast, Assertions are purely verification checks that never modify files. They serve as deterministic gatekeepers, confirming that the produced artifacts are structurally valid and automation-ready. In short: Refactor builds the standardized structure; Asser- tions verify that the structure meets all rules.
D.3 REVIEWS: COMMON ERROR TYPES
LLM-based Reviews have detected the following categories of issues:
Incomplete descriptions: task descriptions omitting necessary information about data for- mat, evaluation criteria, or submission requirements.
File reference errors: incorrect file identifiers listed in CSV files or description text.
Label leakage: filenames directory structures that inadvertently reveal ground-truth la- bels.
Conceptual design flaws: task setups that are technically executable but conceptually un- sound (e.g., predicting feature that is trivially derivable from other columns).
D.4 EXECUTION-BASED VALIDATION: FILTERING CRITERIA
The validation agent is instructed to solve each task using meaningful ML methods and is prohibited from submitting trivial (sample) solutions. A task is retained only if:
The full pipeline (data preparation, model training, evaluation, scoring) executes success- fully without human assistance.
The agent achieves a valid, non-trivial score.
The score does not indicate degenerate task (e.g., accuracy 1.0 or loss 0.0 are discarded as indicating insufficient difficulty or data leakage).
APPLICABILITY TO Raw AND UNSTRUCTURED DATASETS
To demonstrate that MLE Smith h generalizes beyond well-structured competition datasets, eval- uate on three categories of notably raw data sources.
E UNPROCESSED TABULAR DATA: META KAGGLE
Meta Kaggle' consists of unprocessed CSV files (44.28 GB in total), containing comprehensive metadata on Kaggle datasets, competitions, and notebooks. There are no predefined features or pre- diction targets, making this challenging and genuinely raw tabular dataset. MLE-Smi generates 3 diverse, validated tasks (as determined by the Brainstormer):
E.2 RAw SERVER LOGS: LOGHUB LINUX
The Loghub Linux dataset² contains raw syslog messages collected from /var/lod / messages on a Linux server over a period of 260+ days. MLE-Smite generates 1 validated task:
'https:/ ///www kaggle, com/ datasets httpa/eeteeekaggeeeeeeeeeee-eeee weeeeeeeeeaaaaaeeeeeeeeee kaggl e/data chttps://gt thub. com/1 com/logpa logaa i oohub
27

---

Published as a conference paper at ICLR 2026
Table 9: Tasks generated from the Meta Kaggle dataset.
Objective
Metric
Description
Total downloads
RMSLE
Predict the eventual total number of downloads dataset will receive given metadata creation time. Predict each notebook's eventual community popularity measured by its Votel
Totel
RMSLE
Number of competitors
sMAPE
Predict the final number of competitors in Kaggle com- petition using only information available at announcement time.
Table 10: Task generated from the Loghub Linux dataset.
Objective
Metric
Description
Event template IDs
Macro-ave avg F1
Classify raw Linux syslog messages into fine- event template IDs given log metadata and free-text content. challenging multi-class text classification problem with hundreds of distinct templates and strong class imbalance.
E.3
Raw SCIENTIFIC DATA: WEARABLE DEVICE DATASET
The Wearable Device Dataset contains physiological recordings from induced stress and structured exercise sessions captured by research-grade wearable device. MLE- Stitmmitt generates 1 validated task:
Table 11: Task generated from the Wearable Device dataset.
Objective
Metric
Description
Physiological condition
Macro-avg F1
Classify the physiological condition of recording session into three classes based on wearable sensor data.
E.4 DISCUSSION
These results confirm that MLE-Smi SmttSmith can autonomously organize raw datasets and define appropriate features and labels, even when the data are complex and lack predefined struc- ture. Representative examples of similarly raw datasets already present in the main bench- mark include 900k-spotify, novie-transcriptse 2-222, which similarly lack clearly defined predic- tion targets. We estimate that such raw datasets account for approximately 10-20% of the final benchmark.
and cerffficconngstiinn
/0.1//
28


Reviews:
[{"review_id": "boxTiSF4T0", "reviewer": "Reviewer_5zJ5", "summary": "The paper proposes MLE-Smith, a fully automated multi-agent system that converts raw datasets into competition-style machine learning engineering (MLE) tasks using a generate–verify–execute paradigm. Three specialized agents—Brainstormer, Designer, and Refactor—collaborate to create, standardize, and validate MLE tasks. Performance rankings on MLE-Smith tasks strongly correlate with those on human-curated benchmarks (MLE-Dojo), Pearson r ≈ 0.98, Spearman ρ ≈ 0.95, demonstrating high realism. Overall, MLE-Smith achieves scalable, diverse, and verifiable generation of MLE challenges for agent training and benchmarking.", "questions": "1. Could MLE-Smith handle domains outside Kaggle-like structured datasets?\n2. Are there known failure patterns or bottlenecks in verification throughput? How many failed at the Assertions, Reviews, and Execution-based Validation stages, respectively?\n3. What empirical benefit does the multi-agent separation provide versus a monolithic prompting approach?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Thorough evaluation – Quantitative correlation between synthetic and real tasks across multiple LLMs; diverse modalities and metrics.\n2. Strong empirical realism – Demonstrated high rank and score correlation with human benchmarks, suggesting the tasks are faithful surrogates for real-world ones.\n3. Reproducibility – Detailed description of environment, budgets, and execution setup; appendix lists datasets and code schema.", "weaknesses": "1. Ablation missing: The pipeline is explicitly broken into three agent roles (Brainstormer, Designer, Refactor) and a three-layer verification mechanism (Assertions, Reviews, Execution-based validation). However, it does not provide an empirical ablation study to justify the necessity of each individual component.\n2. The pipeline was tested on 300 datasets from Kaggle. These datasets are typically well-structured and pre-cleaned for competition. It is unclear how MLE-Smith would handle \"rawer\" datasets (e.g., unstructured server logs, complex scientific data) that lack clear, pre-identified features or labels, and which may require significant domain expertise to formulate a task. In practice, those rawer data are even more useful since it requires good feature engineering strategies. \n3. Lack of in-depth discussion: The paper states 807 tasks were generated and 606 were \"fully verified\". This implies a failure rate of ~25% (201 tasks). The paper does not provide a breakdown of why these tasks failed. clearer framing of scientific insight (why this works) would strengthen the whole paper as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OkgrOT25el", "reviewer": "Reviewer_pQFv", "summary": "This paper addresses the critical bottleneck in scaling the creation of high-quality Machine Learning Engineering (MLE) benchmarks, which are essential for developing and evaluating sophisticated AI agents. The key of this work is MLE-Smith, a fully automated, multi-agent pipeline designed to transform raw datasets into competition-style MLE challenges at scale. The authors demonstrate the efficacy of their system by applying MLE-Smith to 224 real-world datasets, successfully generating 606 fully verified tasks spanning a wide range of modalities, objectives, and domains.", "questions": "1.\tCould you consider supplementing the benchmark correlation with qualitative feedback from human MLE experts to provide a more holistic validation of task quality?\n\n2.\tCould you provide further analysis on the diversity of skills or problem-solving strategies required by your tasks, beyond the performance ranking correlation?\n\n3.\tHow sensitive is the system's performance to changes in key components, such as the agent's prompt or the removal of a specific reasoning step?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1.\tThe manuscript is well-motivated and well-written. The paper tackles a problem of significant importance. The ability to automatically generate diverse, high-quality MLE benchmarks at scale would be a major catalyst for research in autonomous MLE agents. Moreover, the proposed generate-verify-execute pipeline is well-presented and can be easily understood.\n\n2.\tThe authors successfully demonstrate the system's capability at scale. They generate 606 tasks from 224 diverse datasets. The subsequent evaluation is extensive, involving eight different LLMs on a 100-task benchmark.", "weaknesses": "1.\tThe paper's main justification for task quality is that its performance rankings correlate with the MLE-Dojo benchmark. While this is a useful check, it feels like a narrow definition of \"quality.\" Relying only on this metric makes it hard to judge other important aspects, like whether the tasks are truly novel, realistic, or test a wide range of skills. The paper's claims would be much more convincing if backed by more evidence, such as qualitative feedback from human MLE experts or an analysis showing that the tasks require diverse problem-solving strategies.\n\n2.\tThe paper proposes a sophisticated system with multiple components, but there are no ablation studies to show what each part is actually contributing. For example, how much does performance change if you alter the agent's prompt or remove a specific reasoning step? Without this analysis, it's hard for the reader to know which components are essential to the system's success and which might be less important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wzaaQxZ4Ml", "reviewer": "Reviewer_9fGm", "summary": "Automating machine learning engineering (MLE) is an important task in LLM. However, the acquisition of high-quality MLE training data is difficult. In this work, the authors introduce MLE-smith, a fully automated multi-agent pipeline, to transform raw datasets into competition-style MLE challenges through an efficient generate–verify–execute paradigm for scaling MLE tasks with verifiable quality, real-world usability and rich diversity and a hybrid verification mechanism that enforces strict structural rules and high-level semantic soundness. The proposed benchmark utilizes 224 datasets to derive 606 tasks spanning multiple categories, objectives, and modalities.", "questions": "1. The difference of Refactor and Assertions seems ambiguous to me. It looks like that they are both executing formal checks. I would appreciate it if the authors could further clarify their differences, e.g. Refractor is for execution while Assertions is for check only?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The writing is clear and easy to understand.\n2. The design of the pipeline is reasonable, detailed and economical. The pipeline incorporates a hybrid verification stack: deterministic assertions (format/structure-checks), semantic reviews (via agent), and execution-based validation (empirical solvability).  This provides multiple levels of guarantee that the generated tasks are structurally correct, semantically meaningful, and actually executable by agents while the cost is mere $2.11 per dataset. Given the cost, the scalability of this method is ensured.\n3. They conduct comprehensive experiments on 8 different language models to testify to the quality of their dataset and their strong correlation with human-designed tasks.", "weaknesses": "1. I think the authors should include more details about the hybrid verification check in the Appendix, to ensure the reproducibility of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5QLTkByDMg", "reviewer": "Reviewer_Z7gA", "summary": "This paper addresses the scarcity of high-quality, scalable training data for Machine Learning Engineering (MLE) tasks. The authors propose MLE-Smith, a fully automated multi-agent pipeline designed to convert raw datasets into competition-style MLE challenges. The pipeline operates on a \"generate-verify-execute\" paradigm, utilizing specialized agents to create tasks. A hybrid verification mechanism is employed to ensure structural integrity, semantic soundness, and empirical solvability. The authors apply this pipeline to 224 real-world datasets, generating 606 new MLE tasks.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper tackles an important problem. The bottleneck in creating high-quality, large-scale benchmarks for MLE agents is a real obstacle to progress in the field.", "weaknesses": "1. The contribution of the paper is limited. The proposed \"generate-verify-execute\" pipeline is a relatively common and established strategy for automated data synthesis and benchmark generation. Many prior works have employed similar paradigms, and the paper does not sufficiently articulate how this multi-agent application fundamentally differs from or improves upon those, beyond its application to the MLE domain. \n2. While the verification mechanism ensures that tasks are executable and solvable (i.e., an agent can run code and get a non-trivial score), it does not appear to guarantee the correctness of their reference solutions. It is possible for a generated task to pass all verification checks while containing subtle logical flaws, or for the intended solution path to be suboptimal. The validation relies on \"non-trivial predictive performance\" rather than a strong guarantee of ground-truth correctness, which could impact the quality of the benchmark for training.\n3. The experimental evaluation is lacking in-depth analysis. The results are largely aggregated, focusing on high-level Elo rankings and correlations. While this shows that the benchmark can rank models, it fails to provide insight into why models perform as they do or what specific challenges the new benchmark presents. The paper lacks a qualitative or fine-grained analysis of model failures. For example, what kinds of tasks are most difficult? What specific errors do top-performing agents make? Without this analysis, it is difficult to confirm the \"challenging\" nature of the benchmark, as it's unclear if it introduces new, harder problems or simply more of the same problems found in existing benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)