CoMind: Towards Community-Driven Agents for Machine Learning Engineering
Authors: Sijie Li
Weiwei Sun
Shanda Li
Ameet Talwalkar
Yiming Yang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agent
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce MLE-Live, a live framework for evaluating ML agents in community-driven settings, and propose CoMind, a state-of-the-art agent that collaborates and competes like a real Kaggle participant.

Abstract: Large language model (LLM) agents show promise in automating machine learning (ML) engineering. However, existing agents typically operate in isolation on a given research problem, without engaging with the broader research community, where human researchers often gain insights and contribute by sharing knowledge. To bridge this gap, we introduce MLE-Live, a live evaluation framework designed to assess an agent's ability to communicate with and leverage collective knowledge from a simulated Kaggle research community. Building on this framework, we propose CoMind, an multi-agent system designed to actively integrate external knowledge. 
CoMind employs an iterative parallel exploration mechanism, developing multiple solutions simultaneously to balance exploratory breadth with implementation depth. 
On 75 past Kaggle competitions within our MLE-Live framework, CoMind achieves a 36% medal rate, establishing a new state of the art. Critically, when deployed in eight live, ongoing competitions, CoMind outperforms 92.6% of human competitors on average, placing in the top 5% on three official leaderboards and the top 1\% on one.

Published as a conference paper at ICLR 2026
COMIND: TOWARDS COMMUNITY-DRIVEN AGENTS FOR MACHINE LEARNING ENGINEERING
Sijie Li
Weiwei Sun 2* Shanda Li
Ameet Talwalkar Yiming Yang²
Peekng University Carnegie Mellon University planargestu, pku. edu. on
$Datadog
ABSTRACT
Large language model (LLM) agents show promise in automating machine learn- ing (MI.) engineering. However, existing agents typically operate in isolation on a given research problem, without engaging with the broader research community, where human researchers often gain insights and contribute by sharing knowl- edge. To bridge this gap, we introduce MLE-Live, a live evaluation framework designed to assess an agent's ability to communicate with and leverage collec- tive knowledge from a simulated Kaggle research community. Building on this framework, propose CoMind, a multi-agent system designed to systematically leverage external knowledge. CoMind employs an iterative parallel exploration mechanism, developing multiple solutions simultancously to balance exploratory breadth with implementation depth. On 75 past Kaggle competitions within our MLE-Live framework, CoMind achieves 36% medal rate, establishing new state of the art. Critically, when deployed in eight live, ongoing competitions, Co- Mind outperforms 92.6% of human competitors on average, placing in the top 5% on three official leaderboards and the top 1% on one.
INTRODUCTION
The capabilities of large language model (LLM)-based agents are rapidly advancing, showing signif- icant promise in automating complex tasks across domains including software engineering (Jimenez al., 2024; Merrill et al., 2026), mathematical
(Ren al., 2025; Wu et al., 2025; Collins et al., 2025; Li al., 2026), and scientific discovery (Romera- Parends et al., 2024; Yamada et al., 2025; Sun et al., 2025; Feng et al., 2026). A particularly challenging and impactful frontier for these agents is machine learning engineering (MLE). Automating the multifaceted MLE pipeline, which spans the design, implementation, and rigorous evaluation of high- performance models, re- mains critical test of an agent's 's autonomous reasoning and decision-making abilities.
Recent advances have introduced LLM agents capable of autonomously developing machine learn- ing pipelines for Kaggle-style competitions (Chan et al., 2025). Current approaches have demon- strated range of techniques, from the ReAct-style reasoning in MLABB (Huang et al., 2024) and the tree-based exploration of AIDE (Jiang et al., 2025), to the skill- skill-specialized multi-agent system of AutoKaggle (Li et al., 2024). Although these systems represent important steps toward automating MLE, they are fundamentally designed to operate in isolation, exploring the solution space individ- ually.
This isolated approach stands in stark contrast to how human experts operate. In real-world data science competitions and research, participants thrive on community knowledge sharing: learning from public discussions, shared code. and collective insights to enhance solution quality and drive innovation (Wuchty et al., 2007). By failing to engage with this dynamic external context, current agents are prone to converging on repetitive strategies and ultimately plateauing in performance. This critical gap motivates our central research question:
How can we evaluate and design research agents that utilize collective knowledge?
"qqual contribution.

---

Published as a conference paper at ICLR 2026
Performance on Ongoing Keggle Competitons
Performance E-LEchccch Benchmark
Golind
Nro
ACE
Arg
92.6
36
RDMM-tneu yom
95.6
302
Imposte-fart
53.
DIG1800 Ramon
High
2b2
AREL 2025
357
MAde dchavkor
Diamond- frice
Heal-stttte
67.3
902
51.5
Hoppourd 1985
$2.0
M1
75.0
80.
85.0
00.0
25.0
100 0.0
10.0
230
30.0
40.0
10.0
00.0
750
Figure 1: Left: CoMind's win rates on eight ongoing Kaggle competitions compared with the public best entry. Right: Any Medal results on 75 MLE-Bench competitions grouped by task difficulty levels. CoMind achieves state-of- ofte-of-thee-rr performance on MLE-Bende compared to strong baselines.
To address this question, introduce MLE-Live, a controllable evaluation framework that simu- lates realistic Kaggle-styl research communities with time-stamped public discussions and shared code artifacts that public before competition deadline. This ensure the information access same as human participant. MLE-Live enables rigorous evaluation of agents' ability to leverage community knowledge temporally grounded settings, supporting both offline evaluation on past competitions and online evaluation ongoing competitions.
Building upon this framework, propose CoMind, multi-agent system designed to systemati- cally incorporate external knowledge and iteratively refine solutions. CoMind's architecture consists of five specialized agent role operating in concert. central Coordinator manages the overall work- flow and community interactions. To process external knowledge, an Analyzer first summarizes and suggests on improvements and weaknesses for curated group of solutions, while an Idea Proposer brainstorms a diverse pool of ideas and synthesizes novel strategies. These strategies are then passed to multiple parallel Coding Agents for implementation and report generation. Finally, a dedicated Evaluator, which creates robust scripts for solution assessment and selection. This collaborative process allows CoMind to effectively utilize external community knowledge and construct novel solution for the targeted research problem.
We conducted comprehensive, two-pronged evaluation assess CoMind's performance in both static and live environments, First, on a static benchmark comprising 75 past Kaggle competitions from MLE-Bench (Chan al., 2025). CoMind achieved an overall medal rate of 0.36, establishing a new state of the art by significantly outperforming prior leading agents such Neo and ML- Master (Liu et al., 2025). Second, to validate its real-world practicality, we deployed CoMind in eight ongoing Kaggle competitions (detailed in Figure 1). In this challenging live setting, CoMind proved highly effective, achieving an average rank better than 92.6% of human competitors while placing in the top 5% three official leaderboards and the top I % on one. These results demonstrate CoMind's robust effectivvness against contemporary challengers.
In summary, our contributions are:
A live evaluation framework simulating community-driven machine learning re- search with realistic shared discussions and code.
MLE-Live:
CoMind: A novel agent excelling collective knowledge utilization and iterative exploration, achieving medal-level performance in real competitions.
Community-Driven Multiagent Collaboration: An iterative parallel exploration mechanism enabling continuous knowledge accumulation.
RELATED WORK
The rise of large language models (LLMs) has sparked a new wave of research into LLM-driven agents, systems that leverage L.MMs' reasoning and language capabilities to autonomously perceive,
2

---

Published as a conference paper at ICLR 2026
plan, and act within digital physical environments. Early works such as ReAct (Yao et al., 2023; Schick et al., 2023; Shen et al., 2023; Hong et al., 2023; Boiko et al., 2023) introduced frameworks that transform LLMs into programmable reasoning engines by interleaving natural language rea- soning with tool-use actions. Subsequent studies have extended these agents to various domains, including computer usage (Xie al., 2024; Zhou al., 2024) and software development (Wang et al., 2025; Jimenez et al., 2024).
In parallel, the field of automated machine learning (AutoML) aims to reduce human involvement in building ML pipelines by automating tasks such as model selection, hyperparameter tuning, and architecture search. Early systems like Auto-WEKA (Thornton et al., 2013), HyperBand (Li et al., 2018) and Auto-sklearn (Feurer et al., 2022) used early stopping and Bayesian optimization to search over pipeline configurations, while methods like DARTS (Liu al., 2019) expanded automation to neural architectures. More recent frameworks such as AutoGluon (Erickson et al., 2020) and FLAMI. (Wang al., 2021) emphasize efficiency and ease of use.
Building on these developments, recent efforts have applied LLM-basee agents machine learn- ing engineering (MLE) tasks (Hollmann et al., 2023; Guo et al., 2024; Li et al., 2024; Grosnit et al., 2024; Hong et al., 2024; Chi al., 2024; Trirat et al., 2024; Huang et al., 2024). However, most eval- uations remain constrained to closed- world settings with predefined search spaces, offering limited insight into how these agents perform in open-ended or collaborative ML environments. While some agents (Guo et al., 2024; Al-Researcher, 2025) incorporate basic retrieval tools, these are typically based on simple semantic matching, and robust evaluation methodologies remain underdeveloped.
Meanwhile, several benchmarks have been proposed to evaluate machine learning (MI.) engineering capabilities. MI.Perf (Mattson al., 2020) assesses system-level performance, including training speed and energy efficiency. To evaluate end-to-end ML workflows, MLAB (Huang et al., 2024) tests the capabilities of LLM-based agents across 13 ML tasks. MLE-Bench (Chan et al., 2025) and DSBench (Jing et al., 2025) further extends to about 75 Kaggle competitions covering tasks such as preprocessing. modeling, and evaluation. However, these benchmarks typically evaluate agents in isolation, overlooking the collaborative dynamics of real-world ML development. In contrast, our work introduces a framework that simulates community-driven settings, enabling evaluation of agents' ability to engage with and benefit from shared knowledge, while ensuring that resource access remains fair and realistic.
3 MLE-LIVE
Existing machine learning benchmarks typically evaluate agents in static, isolated environments. This approach fails to capture the dynamic and collaborative nature of real-world platforms like Kaggle, where progress is driven by community knowledge sharing. Participants constantly learn from shared code, public discussions, and the iterative work of others, making these community interactions a decisive factor in developing top-tier solutions.
To bridge this gap, we introduce MLE-Live, a live evaluation framework that extends the widely- used MLE-Bench (Chan et al., 2025). The core innovation of MLE-Live its simulation of com- munity interactions, providing agents with a time-stamped stream of discussions and code artifacts that mirrors the natural flow of public knowledge during competition.
Each competition environment in MLE-Live includes the following components: (i) Task descrip- tion: The background, specifications, evaluation metries, and data structure, seraped directly from the original Kaggle competition. (ii) Competition dataset: A cleaned train-test split of the official data. When necessary, this includes reconstructed test sets account for data that is longer public. (iii) Submission grader: An evaluation script that precisely mimics Kaggle's official scoring mechanism. (iv) Leaderboard: A snapshot of the final public leaderboard. (v) Community arti- facts: A curated set of discussions and code notebooks that were published before the competition deadline. These artifacts are enriched with valuable metadata (c.g., vote counts, public scores, au- thor tiers) to signal quality and are accompanied by any public datasets or models they reference, creating self-contained and realistic research environment.
MLE-Live aggregates a substantial dataset of 12,951 discussions and 15,733 kernels from 75 Kag- gle competitions. To ensure faimness and eliminate post-hoc data leakage it strictly includes only resources available prior to competition deadlines, forcing agents to operate under the same informa- tion constraints as human participants. This approach offers numerous benefits for robust evaluation:
3

---

Published as a conference paper at ICLR 2026
(1) CoMind Simulated Community
Coordinator
(3) Analyzer
(4) Proposer
Sabreate
Possille CIN apposst
Manage Ideas
Rebort
Brainstorm
Memory
Nevsage
T Filter & Refine
Til comepttonn?
Oherding A lighowight Uo
Dapeott
Ratieade
Kernels
Datasets
Discussions
Collect Resalts
oepaseeng
Refined Meas
werlend 311
Solution Dvafl
Runnable Solution
Sccenscti Run w/ Suttmieroo
Spir
Traising Valilazos Set
Genensie
suberarin oer 'alidasoch
Kernel
pyter
Seceeschi Rin Suteriiiion
/
Diraaet
Valittion Labels
exchare
Referesedd Dunasets
Training Validation Set
Erre traccback
high an maalco0
X
Labeb
by
Vallatoe Menie
M
(6) Coding Agent
Evaluator
Figure 2: Overview of CoMind. Specialized agents (Coordinator, Analyzer, Idea Proposer, Coding Agent, Evaluator) interact with simulated Kaggle community of kernels, datasets, and discussions.
it grounds agents in diverse, objectively-graded ML problems from Kaggle, while the controlled in- formation scope allows for fair assessment of their retrieval and reasoning abilities. These features enhance reproducibility and enable consistent, longitudinal comparisons between different agents.
COMIND
We propose CoMind, community- augmentd large language model (LLM) agent designed to automate machine learning (ML) engineering an iterative, collaborative setting. Figure 2 is an overview of CoMind workflows.
4.1 COMMUNITY SIMULATION
CoMind's effectiveness stems from simulating the collaborative dynamics that drive breakthrough performance in competitive ML environments. Unlike isolated automated ML systems, CoMind replicates how Kaggle participants leverage community knowledge: drawing insights from discus- sions, adapting public notebooks and datasets, and contributing discoveries back to the collective knowledge pool.
The simulated community is represented (Ke, De, Tr) iteration where K., contains all ker- nels with evaluation metrics, DD includes published datasets and model checkpoints, and T, cap- tures the dependency relationships between resources. CoMind initializes a high-quality community (Ko, Do. To) by fetching koemel top-performing kernels and Kdiscussion most popular discussions from Kaggle, along with all referenced datasets and models. The system constructs a dependency graph To = (V, E) where vertices represent kernels or datasets and edges capture resource dependencies.
This dependency structure enables CoMind to systematically trace solution construction, identify influential artifacts, and prioritize resources that drive performance improvements. The graph fa- cilitates intelligent ensemble strategies by combining complementary approaches while avoiding redundant components.
CoMind operates an active community participant, iteratively analyzing promising kernels, gener- ating novel solutions, conducting experiments, and contributing successful results back to the com- munity. Fach iteration produces new artifacts: enhanced kernels, augmented datasets, or ensemble checkpoints, that expand the community knowledge base with associated performance metrics.
Through this continuous cycle of exploration and contribution, CoMind simulates the collaborative dynamics of competitive ML development, where collective intelligence progressively advances performance frontiers at automated scale and speed.
4.2 MULTI-AGENT SYSTEM
CoMind orchestrates machine learning experimentation through a coordinated multi multi-agent system. Specialized agents collaborate in distinct roles, mirroring the division of expertise in human research
4

---

Published as a conference paper at ICLR 2026
teams across ideation, implementation, and evaluation. The workflow is iterative loop managed by the Coordinator, which delegates tasks to the other agents.
The Coordinator serves CoMind's central orchestration hub. Its primary respon- sibilities are managing the workflow, interfacing with the community environment, and allocating resources. At the start of each iteration I, the Coordinator initiates the process by strategically sam- pling promising code notebooks (kernels) K2 and relevant datasets Dj from the community. This focused sampling directs the system's 's attention toward high-potential areas. After receiving re- fined ideas from the Idea Proposer, the Coordinator translates them into concrete solution drafts Si, which are comprehensive blueprints detailing model architecture, feature engineering, and training procedures. It then instantiates multiple Coding Agents in parallel, assigning each a distinct draft and all referenced resources. Upon completion, the Coordinator aggregates the results and publishes successful solutions back to the community, advancing the environment state for the next iteration.
Coordinator
Analyzer The Analyzer is responsible for distilling raw community artifacts into structured, ac- tionable intelligence. It receives the sampled kernels and discussions from the Coordinator and performs a deep analysis across four key dimensions: novelty, feasibility, effectiveness, and effi- ciency. For each artifact, it generates 0-10 score these metrics, accompanied by qualitative explanations of successful patterns, emerging trends, or potential pitfalls. The output is a set of structured analytical reports R,, which serve as the primary input for the Idea Proposer.
Idea Proposer The Idea Proposer functions as CoMind's 's creative engine, tasked with generating novel solution concepts. It uses the analytical reports R, from the Analyzer and its own persistent memory of historical ideas Iū to ensure that new concepts are both innovative and informed by past results. The ideation process follows three phases: (1) Brainstorming: Generating a wide array of diverse ideas, prioritizing creativity and exploration. (2) Filtering: Ranking these ideas based on feasibility, potential for improvement, and alignment with the analytical reports. Only the most promising subset of ideas I is selected. (3) Memory Integration: Updating its knowledge base with the newly generated ideas (I = Ie U allowing for increasingly sophisticated strategies over time. The final output, filtered set of high-potential ideas I, is sent back the Coordinator to be developed into full solution drafts.
Coding Agent The Coding Agent is the implementation workhorse, responsible for converting the abstract solution drafts from the Coordinator into executable code. Following an iterative, ReAct- style approach, it conducts trial-and-erros experiments using the training and validation data pro- vided by the Evaluator. To maximize efficiency, the agent maintains persistent Jupyter Notebook session to eliminate data reloading overhead and employs a monitor LLM to track execution and terminate failed runs immediately. This iterative process of coding, debugging, and optimization continues until a viable solution is produced or a time budget is exhausted.
The Evaluator ensures objective, standardized, and reproducible assessment across all experiments, mirroring official Kaggle protocols. It first partitions the public dataset D into a train- ing set D* and validation set with inputs V, and ground-truth labels Vy- Crucially, only D and Vz are accessible to the Coding Agents, preserving the integrity of the validation process. When Coding Agent submits predictions Vy, the Evaluator computes the performance score using the official competition metric (V₁,, Vy)) It maintains a global leaderboard of all experimental runs, enabling CoMind to reliably track progress and make informed decisions about which solutions to prioritize and publish.
Evaluator
5
BENCHMARK EVALUATION
5.1 SETUP
Task Selection. Based on MLE-Live evaluation framework, evaluate our agent on 75 Kaggle competitions on -eeeBBenc. Using the MLE-Live framework, CoMind has access to shared dis- cussions and public kernels published the competition websites before the competition deadline. Since the MLE-bench test set may be constructed from Kaggle's official public training set, and publicly available datasets model checkpoints may have been trained on this portion of the data,
5

---

Published as a conference paper at ICLR 2026
Table I: Any Medal (%) scores on 75 MLE-Bench competitions. CoMind achieves state-o-ttheaar
results across difficulty levels. Best results in each column are bolded. Baseline numbers are taken from the official MLE-Bench leaderboard.
Agent
Low (%)
Medium (%)
High (%)
All (%)
CoMind 04-mini
59.09
23.68
33.33
36.00
Neo multi-agent
48.48
29.82
24.44
34.22
R&D-Agent o3 GPT-4.1
51.52
19.30
26.67
30.22
ML-Master deepseek-rl
48.50
20.20
24,40
29.30
R&D-Agent op-revieww
48.18
8.95
18.67
22.40
AIDE ol -preview
34.30
8,80
10.00
16.90
AIDE gpt-4o
19.00
3.20
5.60
8.60
AIDE claude-3-5-sonnet
19.40
2.60
2.30
7.50
OpenHands gpt-4o
11.50
2.20
1.90
5.10
AIDE
8.30
1.20
0.00
3.10
MLAB gpt-4o
4.20
0.00
0.00
1.30
restricted CoMind's access public datasets minimize potential data contamination. It can only view code published by other contestants.
To validate CoMind under realistic conditions, we further evaluate CoMind on eight ongoing Kaggle competitions. These competitions span diverse domains, including tabular learning. text regression, image classification and video recognition. Rather than approximating the official scoring locally, we directly submit CoMind's generated submi ssi inion... csv files to the Kaggle platform, so that all reported ranks reflect genuine, live leaderboard positions.
Implementation Details. CoMind employs o4 -mi ni -ari--2000 2025-00-2221 (OpenAI, 2025) as its backend LLM. We limit the hardware constraint each run 32 vCPUs and single A6000 GPU. Each competition is evaluated in separate containers with maximum of 24 hours to produce the final submission file. Every single code execution session is limited to hour. Each Coder limited to maximum of 30 steps. The number of parallel agents is set to 4.
During code generation, agents provided with the test inputs (without labels) and prompted to generate a submi sssssnoio csv file. The submission is then evaluated by a grader that compares the predicted labels with the ground truth. Following the setting of MLE-Bench, to avoid potential overfitting, test set labels and the competition leaderboard are strictly withheld from the agent's accessible environment. Instead, each agent must rely solely self-constructed "runtime test set", a held-out split from the original training data, for code evaluation and performance estimation.
Metrics. Following the evaluation metrics in MLE-Bench, we measure the performance of Co- Mind by Any Medal, the percentage of competitions where the agent earns gold, silver, bronze medal.
Baselines. We compare CoMind against the MLE-Bench leaderboard including open-sourced systems like R&D-Agent (Yang et al., 2025), a dual-agent framework (Researcher/Devellppe) that explores multiple solution branches and merges promising ideas into improved pipelines; ML- Master (Liu et al., 2025), which integrates exploration and reasoning via selectively scoped memory that aggregates insights from parallel trajectories; AIDE (Jiang et al., 2025), a purpose- built tree-search scaffold that iteratively drafts, debugs, and benchmarks code for Kaggle-style tasks; OpenHands (Wang al., 2025), a general-purpose CodeAct-based scaffold that executes code and calls tools in sandboxed environment; MLAB (Huang et al., 2024), referring to ResearchAgent scaffold from MLAgemtBench, a general tool-callingppla-nae baseline; and Neo pestps://heynae so/), close-sourced multi-agent system for autonomous ML engi- neering.
httpps:/ggitee whw.. cttpbb/.co//omemii///// /ml bench

---

Published as a conference paper at ICLR 2026
playgrannd seriee-
chisa wew.eedattteeeeeeooooooooo penpaat-ddddaeeeedddddii
Competition
Rank
Teams
Top %
Playground S5E9
1966
0.2%
China Real Estate
43
437
9.8%
Diamond Price
8
67
11.9%
MABe Behavior
3
51
5.9%
ARIEL 2025
90
827
10.9%
DIGABIO Raman TL
22
167
13.2%
nna-intracrmnn ensurrue---netttttinnnn
Impostor Hunt
26
1037
2.5%
RSNA Aneurysm
35
788
4.4%
Figure 3: Left: Score distributions across participants in eight ongoing Kaggle competitions. Each curve shows the relationship between leaderboard rank (X-axis, inverted) and competition score (y- axis). Vertical lines indicate CoMind's position (red) and public best performance (yellow). Right: Results eight ongoing Kaggle competitions. Reported are leaderboard rank, total teams, and percentile rank (Top %, where lower means better standing).
5.2 RESULTS
Table 1 compares CoMind with baseline methods on 75 MLE-Bench competitions. CoMind achieves state-of-the-art performance with an Any Medal rate of 36.00%, significantly outper- forming open-source competitors such R&D-Agent (submitted on 2025-08-15) and surpass- ing the closed-source multi-agent system Neo. Appendix provides a detailed case study on deroissiggalllyyy-oooeeeett
On the eight evaluated ongoing competitions, CoMind ranked top 7.35% average and improved the best public kernel on competitions. Details including authentic scores and win rates per task are provided in Figure These authentic results demonstrate CoMind's capability to tackle a variety of problem domains and achieve competitive performance in live, evolving ML workflows.
ABLATION STUDY
6.1 SETUP
Task Selection. To evaluate the impact of introducing public resources, we conducted an ablation study on 20 competitions from MLE-Bench-Liie based on MLE-Live. These tasks span across various categories, including image classification/geerrttoo text clasiffcattio//geeratii image regression, audio classification, and tabular analysis.
Baselines.
We compared CoMind against the following baselines. For consistency, all baselines use the same backend model CoMind:
AlDe+Code. To enable the use of publicly available code (c.g., Kaggle kernels), we extend AIDE with access to public kernel per draft node, which is selected by highest community votes. AIDE+Code augments the prompt with both the task description and the selected kernel alongside the tree summarization.
AIDE+RAG. We further equip AIDE with retrieval-augmenntr generation (RAG) mecha- nism. Before generating code, the agent retrieves the titles of the top 10 voted discussions and kernels. The LLM selects the most relevant ones, receives a summarization, and then pro- poses its plan and implementation. For debugging or refinement, can optionally re-query documents. Retrieval is based on cosine similarity between query and candidate document embeddings, using Multilingual
Text Embeddings (Wang et al., 2024).
CoMind w/o R. R denotes all public resources. In this variant, CoMind operates without access to any external community resources. It starts with an empty community and relies solely on its own generation history to propose candidate ideas and assemble solution drafts.
Following the evaluation metrics prior research (Chan et al., 2025), the relative capa- bility of generating high-quality solution compared with human is measured by:
Metrics.

---

Published as a conference paper at ICLR 2026
Win Rate
Gold
Any Medal
Above Median
0.668
6.10
0.400
0.650
9.65
0.200
0.200
6.35
0.350
0.00
0.00
0.150
0.300
0.55
0.55
0.345
0.35
0.50
0.512
0.510
0.45
0.50
0.400
0.45
0.05
0.050
0.150
3.40
0.00
6.10
0.50
AIDE
AIDE AIDE+Codee Code
AIDE++RAG
CoMind w/o R
CoMind it
AIDE
AIDE RIDE+CCode Code
AIDE+RAG
CoMind wio R2
CoMind R
AIDE
AIDE Mode
AIDE+RAG
CoMind wio
CoMind w/ i
AIDE
AIDE Code
AIDE-+RAG
CaMind w/o ft
CoMind wi
Figure 4: Performance of CoMind and other baselines 20 competitions from MLE-Bench- Lite. Volid Submission is the ratio of submissions meeting format requirements and validation crite- ria. Win Rate is the percentage of human competitors outperformed by the agent. Any Medal, the proportion of competitions where the agent earned Gold, Silver or Bronze medals. Above Median is the fraction of competitions where the agent's score strietly exceeded the median human competitor.
Table Average win rate of CoMind and other baselines across task categories on 20 competi- tions from MLE-Bench-Lite, # Tasks refers to the number of competitions in the corresponding category. CoMind consistently outperforms baselines across most domains.
Category
# of Tasks
CoMind
AIDE+Code
AIDE+RAG
AIDE
Image Classification
0.597
0.459
0.434
0.525
Text Classification
0.740
0.157
0.338
0.61
Audio Classification
0.901
0,272
0.259
0,271
Seq2Seq
0,408
0,503
0,550
0.228
Tabular
0.664
0.673
0.688
0.483
Image To Image
0.988
0.932
0.617
0.568
Image Regression
0.992
0.342
0.992
0.992
All
20
0.668
0.469
0.510
0.512
Above Median:
Indicates whether the submission outperforms at least 50% of competitors on
the leaderboard.
Win Rate: The percentage of competitors whose final scores are lower than the agent's 's score. If the agent fails to produce valid submission, the Win Rate is 0.
Medals: Medals are assigned based on the agent's 's score relative to Kaggle leaderboard thresh-
olds for gold, silver, and bronze medals.
Any Medal: The percentage of competitions in which the agent earns any medal.
Implementation Setup. All agents use 04 settings of our main experiment, the hardware constraint is further limited to 4 vCPUs and 5 hours per competition. Each execution session is limited hour. Access to public datasets are restricted. In accordaace with baselines, CoMind has access to 10 top-voted discusions and kernels.
as their backend. Based on the
6.2 RESULTS
Figure 4 shows the results. Our key findings are as follows: (i) CoMind consistently outper- forms all baselines across every metric. (ii) Among the AIDE variants, AIDE+RAG outperforms AIDE+Code, and both surpass the original AIDE on most metrics, demonstrating the benefits of integrating community knowledge. CoMind further exceeds these approaches, highlighting the ef- fectiveness of its deeper and more strategic community- resource access causes significant drop in valid submission rates and other metrics, showing that strategic access to public resources helps CoMind balance extending established methods for relia- bility with exploring novel approaches.
exploration. (iii) Removing CoMind's
8

---

Published as a conference paper at ICLR 2026
MLE-Live Win Rate over Time
MLE-Live Code Length over Time
0.7
0.6
Average Code Length
8000
Aerrage Win Rate
85
6000
0.4
0.9
AIDE
AIDE
82
AIDE+Code
AIDE+Code
CuMind
CoMind
AIDE+RAG
AIDE+RAG
Time (hours)
Time (hours)
Figure 5: Win rate over time. CoMind sustains improvement while baselines plateau.
Figure 6: Code complexity over time. CoMind generates longer, richer solutions than base- lines.
7 ANALYTICAL EXPERIMENTS
For analytical experiments, we adopt the same setup as the ablation study and evaluate model per- formance across multiple dimensions, including task categories, win rate over time, and code com- plexity.
Task Categories Table reports the average ranks across seven task categories. CoMind outper- forms all baselines Image Classification, Text Classification, Audio Classification, and Image- to-Image tasks, highlighting its strong adaptability. We manually inspect the tasks where CoMind underperformed and find that the issues are often related to the use of large models or datasets. For example, in Seq2Seq tasks, CoMind explores complex fine-tuning strategies for large language models which often fail to complete within the one-hour runtime constraint.
Win Rate Over Time Figure 5 shows the evolution of average win rate over time. AIDE quickly produces concise, functional solutions, leading to a rapid rise in performance during the first hour. In contrast, CoMind spends more time debugging and exploration early resulting in slower ini- tial improvement. However, after the first two hours, AIDE's performance plateaus, while CoMind continues to improve through iterative refinement and deeper exploration, ultimately surpassing AIDE and achieving higher-qualij solutions.
Code Complexity Regarding code complexity, Figure 6 illustrates the average code length during the entire competition. CoMind consistently generates significantly longer and more complex code, while other baselines begin with simpler implementations and introduce only incremental modifica- tions. Appendix B offers a comparative analysis across code complexity metrics and task categories. Notably, CoMind's s solutions for Image Regression and Audio Classification nearly twice long as those of other baselines. Additionally, solutions from CoMind are, on average, 55.4% longer than those produced by AIDE.
8 CONCLUSION
We introduced MLE-Live, the first framework to evaluate ML agents in community-drive settings, simulating the collaborative dynamics that are essential to real-world progress in Kaggle competi- tions and beyond. Building upon this benchmark, we proposed CoMind, a community-augment LLM agent that iteratively selects and synthesizes ideas, implements solutions, and shares reports within a simulated ecosystem. Our results demonstrate that CoMind not only achieves state-of-the art performance on retrospective MLEE-enchh tasks but also attains medal-level standings in live Kaggle competitions.
Limitations and Future Work. While current experiments focus Kaggle-style ML tasks, the MLE-Live framework can be extended to broader domains, such as scientific discovery, open- ended coding, robotics, enabling research agents to contribute meaningfully across diverse fields.

---

Published as a conference paper at ICLR 2026
REFERENCES
Al-Rescarcher. Ai-researcher: Fully-automated scientific discovery with 11m agents, 2025, URL https://github: wwt.w//.o////... Ree--reeeeere Accessed: 2025-05-15.
Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes, Autonomous chemical re- search with large language models. Nature, 624:570 578, 2023. URL https://api.
semanticscholar: htt.:/aa......eooooooooua1222323
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. MLE- bench: Evaluating machine learning agents on machine learning enginecring, In The Thirteenth International Conference Learning Representations, 2025. URL https: /openreviee. net/forunt i id= 5uXNGGIh.
Yizhou Chi, Yizhang Lin, Sirui Hong, Duyi Pan, Yaying Fei, Guanghao Mei, Bangbang Liu, Tianqi Pang, Jacky Kwok, Ceyao Zhang, Bangbang Liu, and Chenglin Wu. Sela: Tree-search enhanced IIm agents for automated machine learning- ArXiv, 11s/2410.17238, 2024. URL https:// api semant cwchoettcscchcaac. www.eealrrrrrrrraaooooeoeo www.org//ooo. wwt.:o/orss.::2233550000
Katherine M. Collins, Simon Frieder, Jonas Bayer, Jacob Loader, Jeck Lim, Peiyang Song, Fabian Zaiser, Lexin Zhou, Shanda I.i, Shi-Zhuo Looi, Jose Hernandez-Orallo Joshua B. Tenenbaum, Cameron Freer, Umang Bhatt, Adrian Weller, Valerie Chen, and Ilia Sucholutsky. Al impact on human proof formalization workflows. In The 5th Workshop on Mathematical Reasoning and Al at NeurlPS 2025, 2025. URL https:t////.mm. https://
openreview. /tt.tttsss//opeeeeeee...e..
net forum? ?id= EVKMVs.
Nick Brickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexan- der Smola. Autogluon-tabuil tabular:: Robust and accurate automl for structured data. arXiv preprint vrXiiv2003306505. 2020.
Shengyu Feng, Weiwei Sun, Shanda Li, Ameet Talwalkar, and Yiming Yang. FrontierCO: Real- world and large-scale evaluation of machine learning solvers for combinatorial optimization. In The Fourteenth International Conference on Learning Representations, 2026. URL https:
openzevie.. net. fortt/form?iii wttp:/orrnnnd-BBBPaaaaaaa.. di-riddkawVypkwW
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto- skleam 2.0: Hands-free automl via meta-learming. Journal of Machine Learning Research, 23 (261):1-61, 2022. URL http: / jmlr. org pppppppprsss/a//eaa/eee v233 2 0992.. html.
Antoine Grosnit, Alexandre Max Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refi- nath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khyati Khandelwal, Ignacio lacobacci. Abdelhakim Benechehab, Hamza Cherkaoui, Youssef Attia El Hili, Kun Shao, Jianye Hao, Jun Yao, Balázs Kégl, Haitham Bou-Ammar, and Jun Wang. Large language models orchestrat- ing structured reasoning achieve kaggle grandmaster level. ArXiv, abs/2411.03362 2024. URL https: //api. semant icscholar. org. Corg/Corpussi/i : 277850235.
Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. DS-agent: Au- tomated data science by empowering large language models with case-based reasoning. In Pro- ceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 16813-16848. PMLR, 2024.
Noah Hollmann, Samuel G. Müller, and Frank Hutter. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. In Neural Information Processing Systems, 2023. URL https:/ //as:///pp..../ www.nticschollee........ .org/
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zi Hen Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework. ArXiv, 0bs/2308.00352. 2023. URL https:tt////.......t //api///ppii.. semant ineemnticcccolllo.o.. cachol a ort.
10

---

Published as a conference paper at ICLR 2026
Sirui Hong, Yizhang Lin, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang. Jinlin Wang, Lingyao Zhang, Mingchen Zhuge. Taicheng Guo. Tuo Zhou, Wei Tao, Wenyi Wang, Xi- angru Tang, Xiangtao Lu, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zhibin Gou, Zongze Xu, Chenglin Wu, Li Zhang, Min Yang, and Xiawu Zheng. Data interpreter: An Ilm agent for data science. ArXiv, abs/2402.1800 18679, 2024. URL https: /api semanti www.eniiiccchhcc.ccommmo C 2680632222
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MI.ggentench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, 2024. URL https: /tttps::/oppnnevieew...
opwpreveeeeeeeeeeeeeeeeeeereeeee net/ http://ww.ee//oouumddeeeeee wow.mmidddlsslsss.
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code, 2025. URL httph: org/abs 2502. 13138.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL ht /oppr/////peeeeee...
8yNQM66.
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents from becoming data science experts? In The Thirteenth International Conference on Learning Representations, 2025. URL /ttettts:///peeeeeviiii net. wtt.riiiwwweeeerunneee forum? idd:DssSPPPPP Dr
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(18))11552, 2018. URL http:// /mm wwwp///p/prrr../eeeeee /vlless/ll B 11/1166-5666 html..
Shanda Li, Tanya Marwah, Junhong Shen, Weiwei Sun, Andrej Risteski. Yiming Yang, and Ameet Talwalkar. CodePDE: An inference framework for LLMM-driven PDE solver genera- tion. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL https: J2C Certification.
Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: multi-agent framework for autonomous data science competitions. arXiv preprint 11X.i:2410.20422. 2024.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. https:/ /ttp:////eeeeeeeeeeeeeeee
net/ / S1eYHooCSS
Zexi Liu, Yuzhu Cai, Xinyu Zhu, Yujie Zheng, Runkun Chen, Ying Wen, Yanfeng Wang, E Weinan, and Siheng Chen. MI-master: Towards ai-for-ai via integration of exploration and reasoning. ArXiv, abs/2506. 16499, 2025. URL /tt.o///tt////pp....aaaaaaoooooo CorpusID: 944426.
Peter Mattson, Christine Cheng, Gregory Diamos, Cody Coleman, Paulius Micikevicius, David Pat- terson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, al. Mlperf training benchmark. Proceedings of Machine Learning and Systems, 2:336-349, 2020.
Mike A Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi. Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Jenia Jitsev, Marianna Nezhurina, Di Lu, Orfeas Me- Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Kumar Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Kwesi Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Ag- garwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou. David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Man- ish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin,
11

---

Published as a conference paper at ICLR 2026
Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjöm Kolbeinsson, Christo- pher Michael Rytting, Ryan Marten, Yixin Wang. Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-bench Benchmarking agents on hard, realistic tasks in command line inter- faces. In The Fourteenth International Conference on Learning Representations, 2026. URL attpss//ooge /ttp:////peeeeeee.e/mmm net aoporummid-aa00 CCHak..
OpenAl. Introducing openai o3 and 04-mini, 2025. URL introducing-onn and-o4-mini.
Z. Z. Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang. Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, Z. F. Wu, Zhibin Gou, Shirong Ma, Hongxuan Tang, Yuxuan Liu, Wenjun Gao, Daya Guo, and Chong Ruan. beepseek-prover-ee Advancing for- mal mathematical reasoning via reinforcement learning for subgoal decomposition, 2025. URL
bettps///ariii www.aawwwww....raaaaaaeaeeeeeeee wwt./osgg//aoooooooooooooooess
21801.
Bernardino Conera--Paeedes Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models.
Nature, 625(7995):468 475, 2024.
Timo Schick, Jane Dwivedi-Yu. Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL https: / arxiv. org/ abs / /2 202. 04761.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- ginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. URL https: wlaariiv..rg/abs//220
17580.
Weiwei Sun, Shengyu Feng, Shanda Li, and Yiming Yang. Co-bench: Benchmarking language model agents in algorithm search for combinatorial optimization. ArXiv, abs/2504.04310, 2025.
URL https::p////m /ttr///axxi /wwpia.vvvvo////////aaa orgg abs 2504. 04310.
Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 847-855, 2013.
Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent agentt multi-agent Ilm frame- work for full-pipeline automl. ArXiv, abs/2410.02958, 2024. URL https://api.
semanticscholat orwee/rr////ooo ICC.rrpuSI::2311233310
Chi Wang, Qingyun Wu, Markus Weimer, and Erkang Zhu. Flaml: fast and lightweight automl library. Proceedings of Machine Learning and Systems, 3:434-447, 2021.
Liang Wang. Nan Yang, Xiaolong Huang. Linjun Yang, Rangan Majumder, and Furu Wei. Multi- lingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. 2024.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma. Mingzhang Zheng. Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for Al soft- ware developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https: : //oprreeeiiieeee net./ www.rmmii=odddee....
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ h/tppsss///e/eeeiie... openreview.
Stefan Wuchty, Benjamin F. Jones, and Brian Uzzi The increasing dominance of teams in production of knowledge. Science, 316:1036 1039, 2007. URL https: / api. semant cschol arr wtt.oorr/ooopuattn220000000
12

---

Published as a conference paper at ICLR 2026
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Stheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese. Caiming Xiong, Victor Zhong, and Tao Yu. OSWorkd: agents for open-ended tasks in real computer environments. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: wogpe//ww.ew..etttoooooeeeeeeeee
Benchmarking multimodal
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.088 08066, 2025.
Xu Yang, Xiao Yang, Shikai Fang, Bowen Xian, Yuante Li, Jian Wang, Minrui Xu, Haoran Pan, Xin- peng Hong, Weiqing Liu, et al. R&d-agent: Automating data-driven ai solution building through llm-powered automated research, development, and evolution. arXiv preprint arXiv:2505. 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. The Eleventh International Conference on Learning Representations, 2023. httptt / /openeeview net. forum?
id=WE_vluYUl-
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uni Alon, and Graham Neubig. Webarena: A real- istic web environment for building autonomous agents. In The Twelfth International Confer- Learning Representations, 2024. URL https: /top//peeeeiieeeeeeeeeeeeeeeeeee net forum?id= oKn9c6ytIa
13

---

Published as a conference paper at ICLR 2026
A USE OF LLMS
We employ large language models (LLMs) exclusively for the purpose assisting in the drafting and refinement of our manuscripts, with the objective of enhancing clarity and coherence.
B ADDITIONAL ANALYSIS ON CODE COMPLEXITY
In this section, provide comprehensive analysis of the generated code using broad of soft- ware complexity and quality metrics, beyond mere line counts. Specifically, we report the following indicators: Cyclomatic Complexity (CC), Pylint score, Halstead Metries: Volume, Difficulty, Effort, Source Lines of Code (SLOC), Number of Comment Lines and Code Length. We prior- itized these over human annotation ensure reproducibility and avoid subjective bias.
Table 3: Code complexity and quality metrics (Cyclomatic Complexity, Pylint score, Halstead metrics, SLOC, etc.) across task categories. CoMind produces more complex solutions compared to baselines.
Category
Metric
CoMind
AIDE
AIDE+RAG
AIDE+Code
CC
1.68
1.59
1.93
1.29
Pylint Score
7.43
9.06
8.90
8.92
Volume
330.88
143.26
84.20
175.88
Difficulty
4.95
2.90
2.32
2.59
Image Classification
Effort
1960.22
507.06
286.31
725.59
SLOC
198. 25
133.50
120.88
115.71
Comment Lines
15.62
12.88
13.75
14.43
Code Length
7638.40 40
4624.30
4701.30
5192.10
CC
3.58
4.28
2.00
0.00
Pylint Score
8.82
9.09
8.89
9.26
Volume
286.38
384.07
47.68
29.25
Difficulty
3.76
3.94
1.25
1.31
Text Classification
Effort
1183.11
2332.22
61.56
35.16
SLOC
181.67
133.00
141.00
69.50
Comment Lines
14.67
15.33
14.00
13.50
Code Length
6974.70
3094.50
5920.50
5629.30
CC
2.00
0.00
0.00
0.00
Pylint Score
7.92
9.11
9.49
8.86
Volume
718.63
244.20
115.95
227.48
Difficulty
7.39
6.46
3.19
6.38
Audio Classification
Effort
5308.07
1577.11
369.58
1451.30
SLOC
256.00
82.00
92.00
72.00
Comment Lines
20.00
11.00
16.00
16.00
Code Length
9449.00
3508.00
4151.00
3352.00
CC
4.38
2.25
22.33
15.75
Pylint Score
8.58
9.04
9.14
8.51
Volume
492.55
52.33
390.46
324.00
Difficulty
3.87
2.14
5.26
3.68
Seq2Seq
Efforrt
1935.02
140.58
2083.84
1686.74
SLOC
184.50
63.50
222.50
147.50
Comment Lines
22.50
13.00
23.00
19.50
Code Length
6925.50
5649.50
8357.50
2728.50
CC
2.78
1.62
2.38
0.25
Pylint Score
8.65
8.96
8.87
9.31
Volume
1264.61
856.12
815.29
435.46
Difficulty
7.37
4.83
6.05
3.69
Tabular
Effort
10
93
6163.62
5564.22
2001.06
SLOC
218.77
139.75
147.50
93.50
Comment Lines
18.25
14.75
15.25
10.50
Code Length
8570.00
3534.00
6064.00
5759.80
14

---

Published as a conference paper at ICLR 2026
Category
Metric
CoMind
AIDE
AIDE+RAG
AIDE+Code
CC
1.72
2.00
3.00
1.88
Pylint Score
8.43
6.25
6.64
7.74
Volume
1298.
1481.62
414.59
431.08
Difficulty
9.68
6.73
3.94
3.79
Image Image
Effort
12 565.66
9967.24
1633.22
1631.93
SLOC
228.00
175.00
121.00
128.00
Comment Lines
26.00
8.00
23.00
13.00
Code Length
8800.00
5231.00
4815.00
6671.00
CC
1.68
2.00
2.40
2.00
Pylint Score
8.62
8.75
8.80
8.89
Volume
1310.92
241.08
70.32
72.00
Difficulty
8.75
3.88
2.18
2.73
Image Regression
Effort
466.58
934.17
153.43
196.36
SL.OC
267.00
145.00
116.00
133.00
Comment I. Lines
36.00
15.00
12.00
12.00
Code Length
109.9000
4841.00
4655.00
5614.00
C TOKEN USAGE ON MLE-BENCH COMPETITIONS
Agent
Uncached Tokens
Cached Tokens
Completion Tokens
Cost
CoMind 18.23 M 13.05
1
1.96 M = 1.04M $32.25 + 19.43
Table 4: Average token usage (mean + standard deviation) and monetary cost for CoMind across the 75 Kaggle competitions in MLEE-Bench. The cost calculated based on the API prices of o4-mini published by OpenAI.
D LOCAL EVALUATION DYNAMICS OF COMIND ON ONGOING COMPETITIONS
Figure plot the agent's locally estimated validation metrics its execution time, computed by the Evaluator module using held-out validation splits constructed from the public training data. These trajectories capture how CoMind iteratively explores, implements, debugs, and refines so- lution pipelines through its multi-agent workflow. It illustrates several consistent patterns: (i) rapid early improvements as baseline solutions are assembled; (ii) mid-phas refinements driven by sallyzer-guided idea selection and iterative debugging; and (iii) late-stage stabilization once the agent converges to high-perforning configurations.
14401
1442
Naar-moste belteiiondeeeectin
131
Figure 7: Evolution of CoMind's Locally Estimated Competition Metrics Over Runtime Across Eight Ongoing Kaggle Tasks.
15

---

Published as a conference paper at ICLR 2026
E CATEGORIES AND DIFFICULTIES IN MLE-BENCH
ML.--Bene (Chan et al., 2025) curates 75 MI. wwwineerinnn competitions from Kaggle, creat- ing a diverse set of challenging tasks that test real-world MI engineering skills. These competitions span 15 diverse problem categories. Each competition has an associated description, dataset, and grading code. MLE-Bench categorizes competitions based on human evaluation results: Low (29%) an experienced ML engineer can produce a sensible solution in 2 hours (excluding training time), Medium (51%) for 2-10 hours, and High (20%) for more than 10 hours.
Video Clessiiiatioon
Complexity
Image Text
Text (Other)
Medium
Signal Processing
Low
Sequence Sequence
Category
Image to Image
Forecesting
Image (Otner)
Training LLMs
Audio Clessifiation
Segmentation
Tabular
Text Cosssiication
Image
25
Number Competitions
Figure 8: (From MLE-Bench) Competitions in MLE-Bench spans 15 cafegories. MLE-Bench categorizes competition difficulties in Low, Medium and High.
F COMIND ON COBENCH
To directly assess CoMind's ' extensibility beyond Kaggle-style environments, we conducted an ad- ditional experiment on CoBench (Sun et al., 2025), dataset of combinatorial optimization prob- lems that is structurally different from Kaggle competitions and does not involve notebooks, dis- cussions, or leaderboard dynamics. Using gpt-5-mini as shared backbone, we compared CoMind (without access to public resources) against AIDE (liang et al., 2025), FunSearch (Romera-Pareee et al., 2024), and Greedy Refinement (iteratively prompting the LLM to refine the current best solu- tion). Following the settings of CoBench, each agent operates through 64 research steps, receiving feedbacks on a test set at each step and has a 10-second execution time limitation. Across rep- resentative CoBench tasks, CoMind's normalized score achieves competitive performance and, average, outperforms the baselines, as shown Table
G MLE-LIVE ON MLE- BLE-BENCH COMPETITIONS
We collected all public kernels and discussions posted before each competition's deadline, preserv- ing the temporal information to simulate realistic research environments. Table 6 presents detailed statistics for each competition in MLE-Live. The dataset encompasses 15,733 kernels and 12,951 discussions across competitions varying difficulty levels (Low, Medium, High).
Figure shows the distribution of kernel votes across all competitions. The distribution is heavily skewed. with over half the kernels receiving fewer than 10 votes. This long-tai tail pattern reflects real-world community dynamics where small fraction of high-qualite contributions attracts sub- stantial attention, while most submissions receive minimal engagement.
16

---

Published as a conference paper at ICLR 2026
Task
CoMind
AIDE
FunSearch
Greedy Refinement
Crew scheduling
0.915
0.448
0.546
0.602
Graph colouring
0.879
0.850
0.893
0.968
Constrained guillotine cutting
0.975
0.911
0.993
0.989
MIS
0.909
0.932
0,860
0.874
Aircraft landing
0.865
0,863
0.760
0,378
Bin packing (1D)
0.903
0.925
0.975
0.821
Euclidean Steiner problem
0,690
0,636
0,701
0.760
Set covering
0.922
0.887
0,918
0,916
TSP
0.923
0.606
0.860
0.832
Avg
0.887
0.784
0.834
0.793
Table 5: Evaluation on 9 representative task of CoBench.
Table Statisties of MLE-Live on MLE-Bench competitions. Kern. is the number of public kernels. Disc. is the number of discussions. Dsety is the number of external datasets referenced by public kernels, Lines refers to the average number lines all kernels, Comms the average number of comments in all discussions.
Competition
Kern.
Disc.
Dsets
Lines
Comms
Low
aeriallcactus-ddennffcacton
275
27
12
177.40
3.52
186
503
70
357.18
8.33
Denoising-dirty dirty-documents
59
19
70.36
4.53
penetingiinsultsinn-- somiel-commenttrry
4.26
dog-breed iinteiiientiiiittoonn
33
161.75
5.15
dogs-vs vs-cats- kemnls-ccditin
0
31
4.68
118
51
12
228.19
9.53
128
394
56
80.88
7.19
leaf-classification
143.37
1.61
mlsp-2013-bal 2--biddss
0
35
4.94
new www-york-iqu city-taxi--fee seri-fare--pue sneepprdiitioo
143
53
256.69
5.75
nomad2018 wet.-rraranttcooniiee
53
40
57.14
5.30
hant-pathology--
207
91
281.84
5.14
offfom-acts-of-piizaa
8
35.25
3.74
323
289
156
326.17
6.99
276
707
135
345.59
10.87
ppooky-autho--denniffaiiion
153
68
13
103.45
3.75
134
233.22
6.84
adbular-payygoon--eerie--may
247
64
23
273.53
5.02
57
51
3
153,93
4.45
oett-oommiiizatoorcclllegggnnnii language
12
20
2
37.67
2.85
nee-icml-2213-wwale-hhhalll 20111-1----aal---hllllog-rrrghh rucu
8
1.62
Medium
A14Code
159
178
93
395.78
5.76
alaska2-imagee saskaa2-mmag--segaaaayisss
109
154
317.38
6.35
billion- word-imputation
23
3.43
Lassava-eeaf-disasee claasasiisssssiiatiiinn
411
724
209
358.37
6.77
ediscount- image-classifeee chhallennee
91
109
3
$7.10
9.25
pond-tamillquustionn
269
147
137
341.48
6.74
ttamps-ssalar-couue
290
340
307.36
8.54
facebook-res ceapnoo--rrruitin--iii
tii--eyyord-eetracee
0
72
5.32
freesound-audio- -ww-a--iiiiiin-221199999
109
128
18
349.46
7.23
poogle-qust-ccaallens
225
258
128
442.04
9,04
--and-mm-personalizedd
419
223
63
240.76
6.30
Continued on next page
17

---

Published as a conference paper at ICLR 2026
Competition
Kem.
Disc.
Dsets
Lines
Comms
herbarium herbarium-22020-fgee 2020-fgve7
19
15
205.83
3.67
herbarium crrbarium-222--11 2021-fgvc8
26
266.86
2.46
herbarium- perbarium-2022-fiee 2022-fgvc9
37
274.52
3.84
hotel-id-2021-ffww
30
30
15
196.00
2.07
hubmaap-kidney--eemmeniiiioo
321
340
178
325,47
6.70
iccoube-neurrrnn--nddde-----ee
187
156
319.70
5.58
imet-2020-fgve7
21
298.37
3.67
inaturalixt20019-ffvc
12
11
3
244.17
4.27
Soildcam-2020-fgee
21
22
256.86
3.55
jigaawuninneended-biass
410
413
118
301.86
8.16
kuzushijj-eecognition
226.57
5.00
477
226
301
349.24
7.78
imsys-chatbot-arge
305
260
123
299.33
8.51
multi-modal
-esttur--ccooniition
0
39
3.36
513
499
80
386.23
5,92
153
240.50
5.86
plantppathology-222
325
112
135
273.29
4.20
seti-bree breakthrough-lise listen
230
222
72
333.81
7.82
aattoll-cceerg-ccasifeeeer challenge
classifier-challe
160
178
15
80.98
6,09
55.94
6.83
tensorfoo2--uussiinnnnveerrr
102
193
420.29
6.71
213
336
24
329.40
12.16
wweet--entiment-extaaee
536
387
117
324.96
9.86
us- us-patent-Po patent-phrase-tee www.ttoo-hhh-aaeeeeeee
456
233
352
284.25
7.15
uw uw-madison -i--gii-re gi-tract-ine wee-----g-eegeeeeeee
124
445.30
6.32
ventilator
centilator-pressuure pressure- prediction pressure-prediction
407
268
70
246.07
9.05
whale-categorizatio wat.ggorriaaiioo-playrood
32
22
4
61.93
5.00
High
116
565.09
6.19
167
218
57
463.33
10.39
google-researc--dentii
143
136
115
322,43
6.24
hms hanshhrmful- brain activity-classiinee
631
390
258
573.88
6.64
iwildcam-2019 2019--gvv6
196.06
3.77
94
81
32
342.78
4.17
102
55
21
271.21
3.55
209
152
106
302.68
4.34
mma-bbreast-caneerddttttttio
538
433
291
342.78
7.11
414
334
131
321.36
5.72
586
419
382
337.11
5.59
49
56
16
244.82
3.79
stanford-cone stanford-covid-vacciee
201
194
342.38
8.55
vesuvius-chale challlnnge-in--dtteette
190
387.63
6.19
chest chase/rrry-bbbooomaltttddedddee
317
187
115
379.55
6.70
H ERROR ANALYSIS
In this section, we analyze why CoMind failed to surpass the strongest public kernel in the take-or-real--a www.rreeeeeeieessoo---e------ -mpostoo--uut Kaggle competition. The task involves identifying the fake within each pair of samples. Although the dataset originates from The Messenger journal, both the "real" and "fake" texts have been heavily modified by LLMs, making the distinction more subtle and challenging. Below, we summarize the main obstacles encountered during CoMind's execution:
Noised external resources. Public voting did not reliably indicate kernel quality; many highly upvoted kernels were merely ensembles reused outputs from stronger solutions, offering little actionable insight. With a large volume of heterogeneous public contributions, identifying genuinely informative resources remained difficult.
Sparse evaluation signal. CoMind depends on the Evaluator's feedback to guide iteration, but the task's extremely small dataset allowed validation on only 10 examples. This produced highly unstable feedback and limited the system's ability differentiate between small performance variations. As shown in Figure 7,
18

---

Published as a conference paper at ICLR 2026
10a
10a
Percentage (%)
10a
10
-0
100
300
400
Number Kernel Votes
Figure 9: Distribution of kernel votes across all competitions in MLE-Bench. Over half of the kernels received fewer than 10 votes, demonstrating the long-tail nature of community contributions where most submissions receive minimal engagement while small fraction attracts substantial attention.
Evaluator accuracy saturated at 100% within the first 7 hours, leaving no meaningful gradient for further im- provement.
Insufficient ablation analysis. While CoMind explored multiple strategics, its assessment of individual module effectiveness was inconsistent. For example, early iterations attempted to fine-tune local LLMs via LoRA and train binary classifiers on their outputs, but due to poor base model selection and insufficient hy- perparameter exploration, these approaches underperformed simple classical signals such TF-IDF cosine similarity and token-level similarity.
19

---

Published as a conference paper at ICLR 2026
I PROMPTS AND RESPONSES FOR COMIND
This section provides some examples prompts and responses in CoMind, including Coordinator, Analyzer, Idea Proposer. Coding Agent and Evaluator.
1.1 COORDINATOR
Prompt Solution Draft Synthesis
Introduction
You are an expert machine learning researcher preparing for the Kaggle competition described below.
Task Description {description of the specified task)
Ideas {entries in the idea pool}
Reports (entries the report pool}
Public Pipelines fall public pipelines extracted before)
Goals
1. Carefully read the reports provided above.
Based ideas and reports, propose [num,pipes) promising self-contained pipelines that are likely to perform well.
3. The Public pipelines section contains top-ranked public pipelines during the competition. Use them as reference polish your pipelines.
4. Each pipeline should not overlap with others. Your proposed pipelines should include one baseline pipeline that well-known methods but is robust and relatively easy to implement. You should reinforce public pipelines and previous pipelines based their reports (if provided).
5. Ensure that each pipeline can be trained within hours single A6000 with 48GB mem- ory.
6. Read the submission format requirements in the task description carcfully. The format requirement is possible to be different from the training datasct. THIS IS EXTREMELY IMPORTANT. Mention the pipeline descriptions and be sure include the code that handles input output.
7. DO NOT USE tensorflow, use pytorch instead
Response Template for Solution Draft Synthesis
Submit Pipelines Descriptions and codes pipelines, separated each pipeline by ==EPAAA TOR=== mark. For each pipeline, attach code that captures its essential. You must include the code in public pipelines that handles input and output, and if there are parts of the public pipelines that similar the current pipeline, you should include them well.
I.2 ANALYZER
20

---

Published as a conference paper at ICLR 2026
Prompt for Strategy Distilation
Introduction
You are expert machine learning researcher preparing for the Kaggle competition
described below.
Task Description {description of the specified task)
Goals These are top-ranked public scripts during the competition. Your job is to:
1. Carefully read the following scripts.
2. For each script, if it's self- self-contained, i.e. including model architecture (if there's a model), training strategies, evaluation, etc., then summarize its pipeline.
If the pipeline contains technical details, such extensive feature engineering, hyperparam- eter tuning, etc., then list them in full detail.
4. Select representative code segment for each pipeline. You must include dataset reading / submission generation parts. If tas-specciiic details such feature engineering included, the code segment should contain them as well.
Public Kernels
{contents of public kernels)
Response Template Strategy Distillation of Public Kernels
Pipelines Description of each strategy, separated by =F mark. For each strategy,
follow this format:
Pipeline: A full detailed description of the pipeline. All input/output format. hyperparameters, training settings, model architectures, feature engineering, validation metric, and any other relevant information should be included. Do omit any feature engineering details.
Code abstract: A representative code segments that captures the essence (including input/output) and novelty of the pipeline. You MUST go through all the publicly available code and include the parts that generate the submission file. Contain task spec-spe engineering details. Mark the remainder ellipses.
Prompt for Strategy Distillation of Public Discussions
Introduction
You are expert machine learning researcher preparing for the Kaggle competition
described below.
Task Description (description of specified task)
Goals These are top-voted public discussions during the competition. Your job to: Public Discussions [contents of public discussions}
1. Carefully read the following discussions.
2. For each discussion, you should decompose it into critical, novel and inspiring ideas that have potential to win this competition.
Response Template of Strategy Distillation of Public Discussions
Ideas required format: python list of strings, each element description of an idea extracted from the discussions. e.g. ['idea I', "idea 2'].
1.3
IDEA PROPOSER
Prompt Brainstorm
Introduction
You are an expert machine learning researcher preparing for the Kaggle competition
described below.
Task Description {description of the specified task}
Goals I already have list of ideas that partially explore how to approach this competition. Your job is to:
Think creatively and construct least alternative and highly novel solution paths that are likely to perform well, especially if combined with careful experimentation.
2. Each solution path be strategy, pipeline, method that combines multiple techniques. Try to make them different as possible from existing "ideas" list.
21

---

Published as a conference paper at ICLR 2026
3. After describing each full solution path, break down into individual minimal ideas- these should be the smallest units of implementation (c.g.. "use LightGBM for bascline", "normalize input features", "apply stratified K-fold CV")
4. Ensure these ideas do not substantially duplicate items already in "ideas".
Refer the "Reports" section the latest updates and suggestions on ideas and previ- ous pipelines.
Ideas
{entries in the idea pool}
Reports {entries the report pool}
Public Pipelines {all public pipelines extracted before}
Instructions Format your output this (one line, one idea): (your understanding of the task and explanation of your approaches}
SSOLUTION
descrripiion of this approach)
Iminimal idea 1}
minimal idea 2}
minimal idea 3}
SOLUTION PATH 2
SOLUTION PATH 3===
Be ambitious but realistic many ideas later be tested small subset of the data. Focus novelty, diversity, and decomposability. Ready? Start.
Prompt for Idea Filtering and Reconstruction
Introduction You are machine learning expert. After carefully searching relevant literature,
you have come up with a list of ideas to implement. However, this idea list has some issues:
Some ideas are too similar and should be merged into one.
Some ideas overlapping, you should rephrase and decouple them.
You should discard ideas that are irrelevant to the final performance, such as error visualization, etc.
You should refer to the Reports section and Public Pipelines section for previous implemented pipelines. Please decompose, merge, and reconstruct the ideas listed below.
Ideas (entries of the idea pool}
Reports {entries of the report pool}
Public Pipelines
{all public pipelines extracted before}
Response Template of Idea Filtering and Reconstruction
Ideas required format: Python list of strings, each element description of an idea. e.g. ['idea I', 'idea 2'}.
Prompt Coding Agent Report Compilation
Please summarize the results and submit comprehensive report.
Response Template for Coding Agent Report Compilation
pipeline detailed description of the pipeline that generated the best results. All hyperparameters, training settings, model architectures, feature engineering. validation metric, and any other relevant information should be included. Describe potential improvements and future work. summary A comprehensive evaluation of each individual component of the pipeline. For each component, summarize in the following format:
name of the component)
22

---

Published as a conference paper at ICLR 2026
Novelty: 0-10 (0: trivial, 10: clearly novel major differences from existing well-known methods) (your rationale)
Feasibility; 0-10 (0: almost impossible to implement and require extensive engineering, 10: Easy to implement)
[your rationale)
Effectiveness: (0:0 minimal performance improvement, 10: very strong performance, significantly outperform most baselines) fyour rationale)
Efficiency: 0-10 (0: very slow, over-dependent on CPU and hard to produce meaningful results within the time limit, 10: high utilization of GPU) (your rationale)
Confidence: 0-10 (0: no emprical results, not sure whether evaluation correct, fully verified on large scale with ahundant results)
1.4 EVALUATOR
Prompts Dataset Splitting and evaluate.py
You experienced machine learning engineer. Please generate self-contained Python code
for local evaluation Kaggle agent. Your code should robust, reusable, accept command- arguments and print necessary information.
Background
Kaggle competitions usually provide labels only for the training set. To evaluate agent locally, need split the training set training and validation split.
The validation set must hide its labels from agent. The agent only sees the training set (with labels)
and the validation inputs (without labels).
The hidden validation labels will be stored separately and used only for offline evaluation.
Importantly: /ppblic must never contain validation labels. Validation labels are saved only //private
Kaggle Competition Description {description of the specified task)
Data Preview (schema of the input file structure)
Deliverables Please generate two scripts (both in Python 3, runnable from the command line):
1) pptit_dataset.py
Goal: Split the original training data into 90% training and 10% validation. Store validation inputs (without labels) in /public, and validation labels in .private. The training set (with labels) and orig- inal test set must remain in j/pulic, preserving the original structure as closely as possible. The structure of validation imputs should also match the test set. Generate sample validate submission validate sample submission.os under /public. All original data (training and test) visible in (path the input directory..
Example: If the original data is structured as:
kaggle_evaluataae (official evaluation tool provided by Kaggle)
init__
train.cav
train/
test/
sample_submission.c
You should split the dataset into:
t./public/)
keggle Legglseeevlls lofficial evalustion tool provided by Kaggle) (unchenged, soft
links)
--init_
train.cov (this consains 904 of the troining data)
train/ (this contains 904 the training data, keep unchanged data as sott links) Lest.csv funchanged, soft link)
test/ lunchenged, soft link) (unchanged, soft link)
sampleesubmiccionncco
wa validate.csv (chia containg 10% of the training data with labela withheldi validate/ (sef:
wal validate_aamph cample__uume gubmi www.uuummssooonnmmoom.o.... asvon.cav sample acbmtaston file for va oal4 validation on set)
23

---

Published as a conference paper at ICLR 2026
(./private/1 selidate.ccv (labela of validation set)
the training data contains zip files, you should extract them to the public directory before splitting the dataset. You should always print the directory structure after the split. Do not extract files to the original directory and keep it unchanged.
the training data contains multiple classes, you should use stratified sampling. You should strictly task description and ensure validation set repre-
follow evaluation metric mentioned sentative of the overall class distribution. Never write validation labels into ./public.
Your code will be executed by command line as follows:
"baah
fatii_ddassetttt -y --input_df <poth the input di rectooyy> --public_átt ./publ ie -private_dir //privaee
DO NOT store the training and files other folders such as /public.,TIMESTAMME. the //pbblic folder will be exposed to later code agent. Make sure the s/public directory has similar structure with the original data folder.
2) evaluate.py
Goal: Evaluate the agent's predictions validation against the hidden ground truth (//riivate/....
Output evaluation results (json format) to console and write //prvateeeval report.json.
will be executed by command line as follows:
bash
python eva py --publict_étt -/pua' to --private\_dts /pprivate --pred spath to the validation submission file>
We will pass the path the sample validation submission file as the argument to your evaluate.py script. typically produces low scores. The script should generate in following json format at ./private/eval report.json:
json
*score": flost number sepresenta the evalustion acoze on the validat ion set. Do evaluation is unsuccessful
not cmit field. involid, this field should ba set to null,
predictions are successful. otherwise provide necessary details why failed.
ava evalusston was successtul rot, "meseage": & string provides adcitional in.oorattion about the evaluation zesult. Leave it an empty string if the predictions are velid and evalustion
success": boolean value cates whether
Do not raise any eтror or exception. If the evaluation is unsuccessful, you should set the score to null and provide detailed explanation in the message field.
Now, let's write these scripts step by step. Your should first generate split dataset.py. We will execute the code by command line as mentioned above. You should correct the code in case of any issues. You should always generate full, self-contained code. No part the code should be omitted. Respond in the following format:
"Mouurent_ rile
This should be sither pelis_dataset. -y evaluste.py Leave this None if both generated and functioned. this indicates the current file you are editing.
Mexplanation
You explanation the workflow of code.
''yython
full content
t! le.
1f both generated
foned.
24

---

Published as a conference paper at ICLR 2026
1.5
CODING AGENT
Prompts for Coding Agent Iterative Implementation
Introduction You're an expert Kaggle competitor tasked with implementing a pipeline into Python code. You can modify the details (training parameters, feature engineering, model selection, etc. ) but do not change overall architecture of this pipeline. The goal is to obtain best score on this competition. Task Description {description of the specified task]
Pipeline {description of the solution draft to implement)
Data Overview {schema of the input file structure] Follow the pipeline description and the code abstract implement it. All the input files are visible in .//input folder, this folder typically contains the competition data and external resouces, including public datasets, models and outputs of other kernels. DO NOT USE Kkaggle/ipput paths in your code. USE .//nput instead.
file structure:
input/ t../input)
competition
the official competition dazaget
a1loe/caraset11 other public dataseta elice/kernell/ refereaced kernels
working/
sgent.ipysb the notebook you will working on (./agent.
other files
You will develop the pipeline hased on this codebase. Any output files of the codebase, such as csvs, checkpoints, etc., are visihle in ., which is also your current working directory. (Description Selected Codebase)
You should note that checkpoints generated by this codebase is store in // other than ../input. You must load the checkpoint file under the / directory for ensemble prediction. Your code must produce submission at //submissionncs this is EXTREMELY IMPORTANT. Before generating the submission, you must print the value of the evaluation metric computed on hold-out validation set. You can use custom evaluation functions during training, but the final metric MUST FOLLOW THE EVALUATION SECTION IN THE TASK DESCRIPTION on validation set. If other kernels with submission.csv are provided in the input folder, you can ensemble them before generating your own submission. This is important because we will pick your best code based on this metric. You allowed to load the checkpoints of other models. Do contain any absolute paths in your code. Time limit per run 2 hours. Your code will killed if timeout.
Your code will be executed on a single A6000 GPU. Use large batchsizes to maximize the gpu utiliza- tion. If the code segment is provided in this prompt, you should follow the input/output structure. You are allowed to install any packages you need inspect the workspace (e.g.. print file contents, check folder structure). Always use gpu for acceleration. DO NOT USE ABSOLUTE PATHS IN YOUR CODE.
The workspace will be maintained across iterations. That is, your first iteration code produces checkpoint, you can load it in the second iteration. You can ensemble submissions generated by yourself and other kernels. You should generate model checkpoints for future loading. If you load the external submissions successfully but failed merge them with your own predictions, you should print the headers of the external submission and your own predictions and check if the ids are aligned. All the externa submissions are valid. Your predictions should be in the same format as them.
To evaluate your submission locally. You should also generate submission file the validation set. All the validation data are typically structured similarly the test data. An external grader will be used evaluate your validation submission. That is to say, you should generate TWO submission files: one is for the validation set and the other is for the test set. Generate two submission files in the same code cell.
You allowed to install any packages by running 'pip install ppackage_namey" in your script. Your installation will take effect in the NEXT cell. persistent Jupyter Notebook session is maintained. Your proposed code cell will be directly ap- pended to the notebook and executed. You should separate data loading, training and evaluation in different cells. Now, please propose THE FIRST CELL of your code (not your full code!) using the following format:
<goc1>
The explanation fon of your first cel.. You should describe the dest red execution. time and output of this cell. Explain how to interpret the ion output.
</goal>
<oode>
conteat of this cell. Do not weap the code markdown block. Your code be appended to the notebook, which is stored at ./agent. ppyna. Your code must print necessary information lon after each milestone.
25

---

Published as a conference paper at ICLR 2026
</code>
evalidation seal:datioonssumeesiionn submission>
The name of the aubmission file for the validation set, e.g, validate', validate'_subuiee wubmiesionn.commmmm cev. your current code doea not produce two submi saton riles, leave this as None.
e/validatioon_uumiissii
caubmiasions
name of aubmission
set. e.g. submiseion.cev This submission should be ready for Xaggle submission. If your current code cel: does not produce twe submissicn files, leave None.
</submission>
The validation submission tag and the submission tag should must be both empty or both non-empty.
Prompt for Execution Monitor
You Al assistant monitoring code execution. Your task is to analyze the current execution output decide whether code should continue running.
Code being exccuted:
{code to analyze)
Goal: {execution target of this code}
Runtime Information:
Current runtime: {code execution time elapsed)
Maximum runtime: {maxinum execution time }
Remaining time: {remaining execution time)
Current Output: {current output of this code cell}
Consider these factors:
1. Is the loss exploding (becoming very large NaNy?
Is decreasing normally over time?
3. Are there error messages indicating failure?
Does output suggest normal training/execution progress?
5. Based current progress and remaining time. is it possible to complete within the time limit?
Respond in the following format:
<action>
CUNTINUE/SII
</action>>
<explanatiion>>
Your Lonale action. Deacrine progreas, your eatimated remaining time, and explain why you think the execution should continue or stop. DO NOT sIVE SUGSOSTIONS FIXES.
</explanation>
Prompt Consequent Code Revisions
The exccution takes (execution time) seconds and ends with the following output: (Iruncated output) Execution completed successfully. You should keep updating your code (e.g., try different hyperpa- rameters, augmentations, model architectures) after you have made successful submission. Your best submission will be recorded. Now, respond the following format:
evalidation. wtw.idddiionssiiiiiissiiseee suimission>
The name of the submission file for the validation set. e.g. validate_ssteee ssiin.siinn esv. sode cell does not produce submisaion file the validation leave this aa Mone.
your
saubmise Lon>>
26

---

Published as a conference paper at ICLR 2026
The of the aubmission file for the test e.g. submissien.cav. This submission should be ready for Kaggle submissionn If your current code coll doeg not produce
aubmiasion file the set, </submiesion>
<gosl>Describe the gual and how to inspect the output of next code cell</goal>
<eode>
The content of your next code cel Fo lowing the pravicus format, do not wrap your code
should keep updating your (e.g., try different Syperparsmeters, augmeatations, model architectures) evea after you have successful submission. Always evaluate your submission print metric validation set.
withis
</code>
The validation
submission salidation_ubbmissinn tag and the submission tag should must both empty or both non-empty.
27

---

Published as a conference paper at ICLR 2026
J CASE STUDY: DENOISING DIRTY DOCUMENTS
J.1 DATASET PREPARATION
Besides the task description and datasets prepared in MLE- LLE-Bench, MLE-Live collects public kernels and 19 discussions which are available on Kaggle and are posted before the competition ends.
J.1.1 EXAMPLE OF PUBLIC KERNEL
A
simple feed-forward neural network that denoises one pixel at time
import numpy np
import theanc
6
import theano.tensor as T
import cv2 import
import itertools
11
theano.confit floatX
float - (float32'
def Load image (path): return cv2. anaeaddpatt, cv2. IMREAD GRAYSCALE)
15
16
def feature_matrix (img):
"**Converts grayscale image feature matrix
19
The output value has shape (<number of pixels>, <number of features>)
20
$ select all the pixels in square around the target pixel as features
22
window (5, 5)
23
nbrs Cev2.getRecte SubP ix (ing, window, (y, x)) ravelll () for x, y in petertolss.prodiee (range ong.shappe (0]), range (ing. shape[1]))]
34
26
add some more possibly relevant numbers as features
27
median5 cv2 medianBluu (img, 5) r).vell ()
median25 = mww.median (img, .bbell ()
grad np abs (cv2. Sobel (img, cv2.cV_16C, 1, ksize=3) ravel() rraeel ())
div np. abs (cv2. Sobel (ing, cv2.cV_16s, 2, ksize=3) ravel ())
31
32
(omitted)
for fname os. www.iiittrrr............... (sstttstiiit...//pp////ess/
35
for fname in ('1.png'] ]:
36
test_image = load imade (cs. (ath. join w.t.nntt/tesst.....e.e.... fname)
37
test_x = tttruuee_aariixtttttttttt
Y_pred, predict (test_x)
40
output reshaae
reshape frest_imaaee stset_iiaageesaeee) +255.0
41
cv2. www.imertte// original'' fnane, test image)
cv2. iv2.imwrite cleaned fname, output)
44
45
46
if
name ==
main
main ()
J.1.2 EXAMPLE OF DISCUSSION
Edge Diffraction ecean_eleaned
(Sance <TIER: N/A3) <pai'm studying the pixels in train_cleaned Anbssp I attached colorized blow-up varsion part of the mage train_oleanent w...ppg..nnnae.. 6nbap: The
28

---

Published as a conference paper at ICLR 2026
yellow pixels are any pixels that were not pure white [ Im OxF:' gray scale) in image 45, prg, the green wad pure white (DXFF) <p>80 you see what locks Like an edge diffraction line lining the outer edge of all the
le:ters.</p>
<p>ukey, maybe got something wrong in my Anbsp; can anyone conficm this edge diffraction thing in the train_claaned dsta, so for example the first word in Srain_cleaned/45,
(There))&rbspp You need to make the non-white (byte 1- OXFF) mare contrasting color or you may not see it,</p> <p>rim guesaing that the clean
pixels
files were at some point acanned in using some kind of op. ical scanaing machine which added these edge diffraction lines when the Light diffracts off the edge of the black ink character.</9
(omitted)
(Rangel Dakav <TIER: MASTER>] <p>res, there is
noise, which doaan't look like it should be there in the clean aet... I ran teat aetting everything whiter that oxPs to OxFF and the RMSE was 6.305, which should he an upper bound the affects from the halos. This will Likely be large enougn to make the top of the leaderboard game of luck, but since this is just a playground competitien I'm not Lerribly worzled about it.</p>
J.2 EXAMPLE AGENT WORKFLOW
In our experiment settings, CoMind only accesses lop-10 voted discussions and kernels and ignores the rest. The community is initialized with these artifacts. Upon completion of this process, 7 ideas and 10 pipelines are generated. Below is excerpt of the idcas and reports generated by the Analyzer.
(0) Use behavi our-based clustering of neura' networks: cluater modela by thair error patterrs and ensemble them for document entancement
(11 Implement sliding-wind window patch- based models that take an input window and output multiple cleaned pixels simultaneously for both deneising and resolution erhancement
(2) Apply a Wi Rsifu2x-insg red deep iovalutional neural network with gradually increas'ng filter counts (e.g., ->>
-> 256
-> 1) and Leakyseal activations
for affective dencising
(3) Carefully initialize convolutional weights te.g., stdv sgrt 12/ ((NeKBrnourgutPPaan))) and LeakyBeI0 to impreve model convergence and performance
(4) Ensemble multiple models with different input preprocessing: cembine outputs from pure CNN, background-zemovad images, edge maps, and thresholded inputs capture diverse noise characteristios
(5) Augment training data to simulate real-world deformations shadows on text, not just 2n nolse, to hetter match test-time artifacts
(61 Account for systematic artifacts in 'clean' training data (e.g., angle-pixes halos) by :reating then. noise adjusting tazgets accordingly during training
Public pipeline (0): Pipeline: A simple Ieed forward neural network that denoises one pixel
at sime (Theano)
seature engineezing: for each pixel extract 5x5 window of gray values (neighbors), 5x5 median blur, 25425 redian hlur, dobel gradient and second-arder derivative magnitudes, stack Into feature vector, Normalize features to [0,1].
Modal architecture: two-layer MLF; hidden layer size N_H13DEN=10, tanh astivation, output layer with ouston activation 01
Training: MSE cost, stochastic gredleat descen: with learning sate 0.1, batch size 20, one image (3.png) each epoch by RMS2.
epechs 100. Velidation
Prediction: apply same sesurreaariix to test imagas, predict pixel valuas, rashopa to full image, write out cleaned PNGa.
Code abstract:
def feature_mat reature_marixx(igg))
window
nbrs= (ete:c/regeeReettubeeee ix (ing, window, (y,x)) .ravel ()
for x,y in itertsels. produssss (range (ing. shape [01), range shmpe [1])
aodians=ev2,mmdiaaee punrriigg Himg, ravel ()
median25=ov2 medianBlur (img, 251 raavel
grad-np. abs 1ev2.Sobel (ing, www.cv_162e,eee..e 2, k,izeik) ravee1
divenp.abs Sobe I (1m., ww2.cv_166...... 2, 2e- rravell
discenp.vstace meediat5,, grad, div))
festuresemp.e hatack mibrrs
return (featurea/255.. asstye (1 float32")
class Model (ob
def _intt_
self, laye (.. a_i outt -LLLDCEN__ att.tatt.....iva...oo....oo
self..Lyys:2Layeerl...... HIDDEN, n_out-n_out,
activaticn=lamen Lambde x: r.clip (x+0.50 5,,0.5 def cost return T.mean soupf..oututty))
PIPELINE SEPARATOR
Publis pipeline (1): Pipeline: Katching image backgrounds in ML model).
Reada test PNGS in batches of 12 imagen.
Flattens each into vectors of 258+540, stacks colunas.
for each pixel lacation, takes the maximum value across images an estimate of background
29

---

Published as a conference paper at ICLR 2026
cut background images ENG.
Code abstract:
for in 1:4)
matches=seq 21,201,,y=12 (1 1)
sambstarmatrix 250, (matches)
for (1 in sag_along (matcheas)
pet.teleeeeaapeeeee.eee ttetpaattttttttrraaeaeeeee imatches [1),' .png' ))
rawtatal, (, =es. ve..or (wmgy 1:540)
background-matzlan wawpanaaaaaaalllllllaoaaa (zawDaLa, 1, max)) 258,
writePhG (background, pestel (' backkroundd" matches [j],' .pgg'))
PIPELINE SEPARATOR
pipeline ( Pipeline: Pixel-wise Random Forest regreasicn (Python, chunk size=1e6).
Feature engineezing: pad image mean value (padeing=1); extract 3+3 neighborhood pixel, flatten. as features.
Training data: load all train noisy images, compute features via jeblib parallel (n_jobs
=128), load targats as flattened clean pixel intancitics/25.. R...adnooreettegrraaaae
Model:
(warm_atart-Trua, n_1oog=-1), Tnorementalll
add one estimator at Lime: split Lraining rows into CHUNKSIZE-1e6 slices, in each fit Prediction: extract teat Teatures similarly, generate index strings "image_rov_coo", predict pixel values, write submission CSV.
increaze a_estimators by
slice.
Code abstract:
def get get__added (ing, padding=1):
padval=int (rog.d ()n
return padded
fet__eettues__tt foreee__for__eee for_image paddingli)
paddedinee__pedded iigg:eeedddddddddddddeeee padding)
return np.vstack [[p wtw.hppee...eaapppeeeeee. (1,-1) for i
sange (rows) for in rango (cols)]
get_nedel y)
start, end slices:
model..et__aree (n_est iestimmsssssseeee.... otw.
model. [X[start y (ataatteedd y
return modal
PIPELINE
(onitted)
Public pipeline (9): Pipeline: XcBoost with basic features in 8.
Features: row pixel value, kmeansThrasheld hinary, Background__amone feature.
Sarple 25k pixels, but MMtrix, 5-fold early atop, select rounds, train xgb. Predict on Lest, clip, build id atrings, wzite submission.cav. abstract:
dat=cbind ravmimg2ves
threshol i b backgroundRemovall wimg//ve (hwckggooudd_Reeeeee Remoooaan_eeeaalllaaaaaa (IngX))
rows=sample (nrow (dat), 25000)
deroinsxgb.nne DMtttiitttixi (dat (rows, -1], label labelllllde [rowa, 1])
xgo, tab-xgb, (data-dt rain, nfold=a,eate eerlyearly.ator wtt..tto..oouoobb...........
xgb.mode mod-xgboogb idata-atrain, neounds-min men.eer............mmm.......... eer.e (dx)
yHat"predict (xgb (xgb.mod, mod, newdatamss mew.atanas.matrii(ee. mat.iii
Then, Idea Proposer begins brainstorming to propose more original strategies.
sSSSOLUTION, BAATTTOON_ATT_____ lss=
Multi-Scale Wevelet-U-Net with Attention and Multi-Term Loas
Description: decompose each image via discrete wavelet transforn IDWT) into Low-/high
frequency bancs, process eact band in dual-branch U-Net enhanced with channel- and wavelet reconstruct.
spatial aliention, weighzed sum of pixel-wise RMSE, wavelet-band consistency, and edge preservation losses. apply to neiay input abtain Ld, LE, HI, ни sub-banda
end
Feed Mr. into "ecarse" enooder branch, feed concatenated 18/AT//R into "detail" ercoder bzench
U-Net decodes upsample each branch back petch size, Euse via lesrned 1+: convolutions
iinsert Convolutional Block Atcention Moduies (CBAN)
encoder decodez
define loss = siihhasiiillss
output] betaa-baddxss (wavelez (clean) Navelez (autput)) gamma (Sobel (clean), Sabel (output!)
-rainn full Images with Adanf and costne-anrealing &R schedule
Stroke-Awara Conditional CAN with oCR-Guided Perceptual Laas
Description: build conditionsl (generator deep residual encoder-decočes, discriminator = PazchcAN) that not cnly minimizes pixel less but also preserves taxt strokes-enforce strake-lave)
pre-traired amall CNN elassifier that predicts presencs/width
30

---

Published as a conference paper at ICLR 2026
strokes. an UCR-based perceptual loss: feed predictions through Erozen UCR engine embedding and minimize ze distarce clean ambedding.
implement generator as Reslet blocks + akip connectiona 164-2128-225-2122---6)
implement discriminator include Li pixel loss adverseriel less stroke consistency loss (11 between stroke-CMB features clean va, restored)
70x70 PatchSAN focus local texture
freeze a ama text-structure CNK (trained on binary masks) to extract atroke features run resseract lor lightweight OIR CaN) on restored vs. clean, extract ponultimate-layer act activaaioonl add perceptual loss tern
Lzain with R1 gradient penally and spectzal normalization on discriminstor
Joint Dictionary Tearning Non-Lacal Patch Aggregstion
Descript iptions learn paired dictionaries (D_noisy, D_clean) for small patches (e.g, 8+81 via coupled K-3VD, At tast time, extract overlapping patches, compate aparse codes alpha under D_noisy via OMe, reconstruct clean patchea D_olean<aappa. Then, perforn nen-local means on the recenstructed patches to exploit self similarity and average aggregates. sample Large beak noisy/clean patch pairs, initialize R_noisy, D_clean with DCC basis run couplad K-SVD to minimize I ph y_nnisyll + | IDD_clean-slpha y_clear|| w,r.
7. P_nalsy, D_clese,,
at Leat Lime, [or each Image patch y_noisy, compute alpha via Orthogonal Katching Pursuit I sparsity k)
reconatruct y_clean_est D_cleanselpha for each patch
perform matccing to find K nearest patches per reference patch (Buclid dist), stack them aggregate reconstructed patches with non-local weights (e.g. Gaussian on reconatruation residual)
Self Supervized Bliad Denoising via Noise2Void Test Time adaptation
Description: exploit purely neisy daza-train a small U-llet with masked pixel prediction
koiss2void) each
image inference (test-time training). The network learns to predict a pixel from its context, gradually adapting to local haise statiatics, then you cun Iorward pass to oblain the cleaned Image. Ne clean Larget needed.
define blind-spot or random masking scheme: mask 1t pixels per batch, replace with neighbors build lightweight CNN (e.g. down/up blocks with skip connections) that predicts full
image
fine-tune this CNN on each test image for k_iter (e.g. 500 steps) using only masked 42 loss data augmentetion (ratations, flips) the single tast image diversify contexts after adaptation, perform a clean forward paas without masking to get the denoised output opt ionally ensemble outputs from multiple random initielizations
reduce variance
To remove similar idcas and decompose overlapped idcas, reconstruction is performed subsequently, 9 idcas are preserved after the filtering and reconstruction. These ideas are then merged with the idea memory.
(0) Use diverse ensembles by clustering models based on their ercor patterns and combining outpuss from differeatly preprocessed inputs (e.g., raw, bakkground-removee shresholdad) to capture varied noise characteristica.
edge maps,
(1) Develop adiding-enddo, patch-based madal that takes input region and predicts multiple dencised and resov-re pixela simultaneously for efficient document erhancement.
Implement Kaituzx-inaptred deep convolutional network with progressively increasing filter counts and LeakyReLU activations, initialized using lle aormalization for rebust convergence denoising tasks.
(31 Augment the trzining set with simulated 30 daformations, lighting variations, and shadows on text to better match real-world scanning artifacts.
(4) Identify and compensate fer systematic actifacts the *clean" iraining Lazgeta lingle-pieel halog) by modaling them as noise or adjusting grosnd-truth labals during sraining.
(5) Build nevelet-domele dual-branch C-Net: decompaze inputs with 20 DWT into approximation and detail bands, encode seperately, decode integrate CRAM attentior madyles, and train end-to-and with composi loss combining pixe1 RESE, wavelet-bant RMSE, and edge loas, optimized using Adamii with a cosine- annesling LR schedule.
fuse learned 1+1 convalutions,
(6) Apply a GAN-based restoration fromework: use a bessee- generater and 70+70 PatchGAN discriminator, train with a combination consistency loss (via a frozen stroke-feature (NN), and perceptual OCR-feature loss, incorporating R1 gradient penalty and spectral normalization.
pixel loss, adversarial loss, stroke
(7) Teverage coupled diotionary learning and non-loca) aggregation: learn noisy and d'ean diet ionacies simult tanecssly via coupled K SVE, zeconatruct patches using OMP, enhance self self-similes similasstty with beighting.
and fuse reconstructed patches with non-locel
Implement blind spot test-time adaptations zandomly mask small percentage of pixels per batch, train a lightweight CaN with skip connecticns on the single test image using masked loas and augmentationss fine-tune per image for a few hundred steps, and optionaly ensemble multiple initiallzations for variance reduction.
Based on the ideas received from the Idea Proposer and previous reports, the Coordinator then compiles 2 solution drafts.
31

---

Published as a conference paper at ICLR 2026
sipeline (Saseline UaN U-Rez
simple augmentations)
Overview
À lightweight U Net trained 256x256 patches of noisy/clean document pairs. single-brance encoder-dacoder with skip connections, ReLU activations, batch norm.
Loas: L1 pixel-sise loas (less sensitive outliers than optional total variation regularieacion.
Optimizer: Adam;
A6300 within
Data Prepazation
1. Read sll noisy (input) and clean (target) train images, cormalize intensities [0,1].
2. Extract randon 256+256 patches (stride = 128) with matching noisy/clesn pairs.
Dota augmentation. random horizental /vertioal flipa, +-90 dagree rotations.
4, Create PyTorch Datatoader wita batch_size=16 (fits 48 GB) for "100 k patchea.
Network Architecture (PyTorch paeucocode)
class Uset (nn.kedsle);
def init init_ (self):
super (nitt_
Eacoder
DeubleConv
self.enc2 (64, 128)
self.enc3 Dewn (128,
self. ence Dcxn
Retckk
belt.bootteeeck
5122,
Decoder
self.ups .uf. up (1024, 256)
self.up3 (512,,
self.up2 Up (256, 641
self.upl (128,
self.tinal =_n. mnn.Convdd14, kernel_atzew
[or,ard(selI,x
elmseif.encl (x)
#2mself.enc2 [el1
e3=seif.enc33ee. (e21
ed=seif.enc44e31
dd-self. eb,
(upselE..p2(d)... di=self.ug) (1),
Ha selper moduless
DoubleConv (Conv2d >> Datchhoormdd >> RelC) +2
(MaxPool2d DeubleConv)
= (Conviranspose2d for upsampling -> concatenste skip -> boubletonv)
Training
Loas 11toss (output, target) lamada- (lambdoste-3 for amcothneas).
Opt inizer Adam[1r=1e-3, welgnt_deeas=le--).
LR schedule: Reduce12GnPiateau (nenitor=vsi__oss factor=0.5, patience=5).
up to 50 epocha; early-stop if loss stagnates.
validation: hold eut 10t patches to nonitor RMSE.
inference
For each test image (e.g., 540+258), sl ide 256-256 window with stride=128, predict, and average oveslapping outputs.
Threshold nothing; output zaw [0,1] floats per pixel.
Compute Budget
"100 patches, batch 16, x steps per epoch, On A6000: "2-3 min/epoch -> 50 epochs hr with early stepping hr.
Pipeline (Advanced
Composite
Rual Trall-baane U-Net operating in wavelet donain (Baar (NT) expl feptil denot se tona" and
:extural componente.
CBAи (Cenvelutional Block Attention Kedules) edaptively weigh spstial/chaanel features.
Loss - alpha+ti__ipe beta+12 beta+I2__avelee wavelet + gamma
Optinizer
CosinelnnealingIR.
Data Preparation
Same as Pipeline 1. (256+256 patchea augmentations!.
On the fly DWT: for each noisy patch, compute one level llarr DoT yields approximation (A) and details ()),,
Network Architesture
32

---

Published as a conference paper at ICLR 2026
(implemented PyTerch)
class NaveletUNet (nn. Module)
def _iiit_(see1
supe: (tne=))__ii.... init_
Sharee CBAM-RoS blocks for Approx and Detail branches
self.encA1
CRAMResBlock (1, 61,
CBAMResBlock (3,
self.pool
nn, MaxPoo (2)
self.encAZ
CBAMResBlock (64,
self. encD2
CBMResBBock 128)
Battleneck
self. l.f.btttlemeck CRAMResBlock (56)
Decoder
self.up2 Upsea (256, 128)
self.upl UpRea (128, (128, 64)
self.final Conv2d11,, kernel_siee=)
def (sela x)
$ x shape: [8, [,,1,
Pyllavelets
d0 Lorch.cat( ([aaell.ennaaa (A), self.encDl (Lorch.cat (12, 0, dim=l)) dim-1)
p1 self.pool
d1 Lorch.cat ([eelI.encA2 :128,), self.encD2(p1/ (pl 128: 1111),dddmm))
b self.bottleneck (self.
u2 self.up4 (1,
u1 self.upl (u2,
self.finall
[,
Reconstzuct decoised vis inverse DRT
A_hat, H_hat, V_hat, ou 1;2]; 2ut(;, (u;3::3:4 return ihaar_dwt hat, (Laht,,
building blocks:
CRAMResRlock residual block with Conv-RV-Ret3-<Conn-- plus CRAM (channel apatial allen-ion
UpRes: ConvTranapese2d upsample-<concatenat skip Skips
Loss
Li_pixel u1 (output, target).
10_wovalet 12 (DKT (output) DNT (target))
EdgeCosa t. (Sobel (output), Sobel (target1).
TotalLoss sppa+Ll__iiell beta=12_waveiet.
gemma-BdgeLoss (alpha-1, beta-0.9, gamma-0.2)
Training
Opt imizer: Admm/[1==le-3, weigit_decay=le--) Schedulers Cost neAnnes Momannallnggii inngt
max=500 (T_xax5__
Batch size=8 (higher memozy due to dual branches). Epocha=50, easly-stop by composite
set.
Inference
Same sliding-winder approach Pipeline
Compute Budget
CDAM and wavelet layers add "1.5x compute but within A6000 48 GB: patch batch-8 yields 7250 steps/epocis, 50 epochs he. early-stop T_max=30 finish
Raferences
CRAM: Woo et al.,, "CRAM: Convolut fonal lonal Block Attention Modula* |FCOV 20181.
PyNaveleta
filters.
Edge loss: Scbel via fixed cony kernels.
Expected Renefits
Explicit noise auppression in aigh- Irequency bands. Attention modules focus text strokes and edge restorstion. Composite loss lances global structure and fine detaila.
In our experiment, 2 parallel Coding Agents are invoked to implement the solution drafts respectively. Each instance has quota of 20 steps, 3 hours total and hour per execution. An LLM is instructed to monitor the execution process and terminate failed runs.
The best code snippet generated in the first iteration is:
import os, glob, random, math
from PIL import Image
import torch
import torch. nn as nn
import torch. nn. functional as
from torch. is. data import Datasett litade
33

---

Published as a conference paper at ICLR 2026
from torchvision import transforms
1) Haar DRT / inverse DNT class HaarDWT (nn. (on.Muuue)
def _iiii____ss (
13
super (). init_ ()
14
wt = forch.zeros (4, 1, 2, 2)
wt10, 0, i, = 0.5
wtll, 0, 0, 01, wt wtll, 0, 11, wt 01, wt 11, O, 0.5,
18
-0.5,
0.5,
-0.5,
22
wt [2, O, 0, 01, wt 12, 0, 0, 11, wt 12, 0, 1, 01, wt 12, 0, 1, 11
23
0.5,
24
0.5,
-0.5,
-0.5,
27
28
wt [3, 0, 0, 0], wt 13, 0, 0, 11, wt. [3, 0, 1, 01, wt [3, 0, 1, 11 (
0.5,
-0.5,
31
-0.5,
32
0.5,
33
self.registes www.sterriiifefeeeeeeeeeeeeeeeee ("weight", wt)
36
def fsew,, x):
37
return E.conv2d self.weight, stride=2)
38
40
class HaarIDWT (nn. Modulel)))
41
def _ini _init__(ee11_ (self):
42
super () 10.._init inin__tt
- Hаarоит wwe.ghh.co.eeee.......o.oo
self.cons mn.cooniiransppe22.44
stride-2, bias-Pal
45
welff.cow..wiigt.cco copy.
46
self. (Faeeeessrr__aaaaeeeaaeeee (False)
grad_
def forward coeffs):
49
return self. conv (coeffs)
50
51
-- 2) Sobel edge for EdgeLoss
class Sobel (nn. Module):
54
def __nit_(self)
55
super init init ()
kx = (
[[1, 0, -1], 12, 0, -21, [1, 0, -1]], dtype-torch.
(iview 1, 3, 3)
ky - torch.
60
[11, 2, 11, 10, 0, 01, [-1, -2, -111, dtypestorch.
].view(1, 1, 3, 3)
self.register bufferuufffree ("wx", kx) oolf.roggeter__bfffeee
("wy", ("wy", ky)
64
65
def forward(self, x):
gx = F.conv2d self.wx, padding-1) gy c.conv2d (x, self.wy, padding=1)
68
return torch. sorch.sqrttge gx + gy gy 1e-6)
69
3) CBAM, ResBlock, UpRes, RaveletUNet
34

---

Published as a conference paper at ICLR 2026
class (nAMunn.ouulee)
73
def inn_itt_ssee C, r-16, k=7);
74
super () in_itt___
75
sclf nn. Sequentiia
nn. Li.eare // r, bias=False),
78
mn.Linea // f, c, bias-False),
79
self.spatial Conoo22,,,, 1, sernel_size=k padding=k // 2, blas= False)
82
def (sel,, x):
83
b, h, shape
= adaaptive _ee___ae____ (xg_pool 1). vbi c)
adaatiive pool2d (b,
86
ca torch. (ir
+ self.mip vvieww,) view e, 1, 1)
87
x2 = * ca
88
avgc x2. (1, Truc)
maxc, m2x (1, Truc)
tirch, (sel.. (torch. cat ([avge, maxc], 11))
91
return x2 sa
92
class CEAMResBlock (nn (nn.Medlle)
95
def
init _nniiiitiee_t___________________ (self, inp, outp)
96
super ()__t))t_...._... initt_(()
97
self.convl - nn. Convon (inp, cutp, 3, padding-l, bias-False)
self.bnl
www.Bachhormdddocc (cutp)
self.conv2 nn. Co. (outp, cutp, 3, padding=1, bias=False)
100
self.bn2 nn dd..Bacchorm2d (cutp) self.cbam = (BAM()))))
101
102
sclf.skip =
nn ConC2nd (inp, outp, 1, bias-False) if inp != outp else Identity
104
105
self.relu - nn. Rell (inplace
106
def forward x):
108
ict self.skip
109
out = self, relf.relu(sel...nn) (onef. (x) )
out = self.bn2 (cut)) (cut))
+ idt return relfu
112
113
114
class UpRes (nn, Module)
def iiiiiit___i (self, ic, sc, oc):
117
super () init_()
118
self up = nn. wm.Convrransppee2llc,, oc, 2, stride=2)
119
selk.block = CBAMResBLock (cc + sc, oc)
120
def forward (self, x, skip):
122
x self.up(x)
123
if x.shapel-2211 != skip. s -2:]:
124
x = Fiiterpooltee
x, size=skip shape [-2]], mode="bilinear align_cormers-Palle
126
127
return self.block (torch. cat ((x, skip], 1))
128
class NaveletUNet (onuMod
131
def init (self):
132
super () init_ ()
self dwt HaarDWT ()
self, idwt BaarIDWT
35

---

Published as a conference paper at ICLR 2026
self.eAl CBAMResBlock (1, 64)
136
self.eD1 CBAMResBlock (3, 64)
137
self.pocl nn.MaxPoo12d((2)
138
sclf.cA2 CBAMResBlock (64, 128)
self.e02 CBAMResBlock (64, 128)
self.b CBAMResBlock (256, 256)
141
self.u2 UpRes (256, 256, 128)
142
self.ul UpRes (128, 128, 64)
143
self.final = Co. (64, 4, 1)
def forward (self, x):
146
A, H, v, self.dwt (x) chunk (4, 1) al - self.eAl (A)
147
al = self.eD1 (torch. cat (IH, v, DI, 111
torch. ([al, dl], 1)
150
p1 self.pool
151
PA, PD = pl[:, :64), pll:, 64:]
152
a2 = sclf.cA2
d2 sclf.eD2
dlb tarch,cat (Ia2, d2], b self.b (d1b))
155
156
u2 - self.u2 (b, d1b)
ul = self.ul (u2, d0)
self.final (ul)
159
return self.idwt (out)
160
161
4) OCRDataset for full training
class OCRDataset (Dataset):
164
def
init __iii_inisll____________________ (self, noisy, clean, pP, ps, mode)
105
self.ncisy = noisy
166
sclf.clean = clean
self.pp pp
168
self.ps ps
169
self.mode mode
170
self.teT Tewafoormm.oooeencc...
def ________s (self):
173
return len (enls....nngg self.pp
174
def
(self, idx):
11 idx // self.pp
177
(og.:eeoooonslll.ooolosololl//le [ii]) convert ("D")
178
C [hange.opn(selff.cccmiiiii (oo))
convert ("L")
179
tn, tc (e), self.tcT(c)
H, W tn. shapee sn.
y wandom,randiag random.randint(00 W self.ps)
H self.ps)
182
183
self.ps, tn = tnl;, y + x x self.ps] tc = tel:, -- "train":
selt.ps,
self.ps]
if random.random. () 0.5:
186
if tn, tc torch. flip [-1]), flrch.flip(te, [-1]) if randon. random < 0.5:
187
188
tn, te = flichhflip [-2]), torch.flip(te. [-21)
190
k Sanndom.randine (0, 3)
191
if
192
tn, tc torch. k, [1, 2)), torch.rot90 (tc, k, [1, 2])
return tn, tc
194
195
196
-- 5) Prepare train/val split -- ("//inuu//rrain/ all cl- sorted (glob.gooo (". (t.ipp/tttn /ttp://ttttaaillleeeeeeeeeee/ee/ cleaned/=.pang
all no
36

---

Published as a conference paper at ICLR 2026
idxs list (range (len (all (all_no))
200
random. seedd (42)
201
shufflc (idxs)
202
split = int (0.9 * len (idxs))
tr_idxs, va_idxs = idxs[:split] idxs [split:]
noisy_tr [all_no[i] for in tr_idxs]
205
clean_tr [all_cl[i] for in tr_idxs]
206
noisy_va = [all_noli] for in va_idxs]
va = [ali_cili] for in va idxs]
batalioaders
210
train_ás OCRDataset (noisy_tr, clean_tr, pp-25, ps-256, mode="train")
val_ds OCRDataset OCHIataset (noisy_va, clean_va, pp-l, ps-256, mode="val") 211
train_lcaai loader = D
train ds, batch size=8, shuffle-True, num womkork,sss pin memory=True
214
215
val loader D val_ds, batch_size-l, shufflc=False, num_workers=2, pin_momory-Irue
216
219
7) Model, optimizer, scheduler, loss w/ tuned weights device ce Cerche ("cuda" if cerch.cuda. s_aavillblle else "cpu")
230
model = WaveletUNet () to (device)
dwt HaarDWT () .to (device)
223
sobel Sobel () to (device)
204
opt torch. opt Adamw (model, paw.oeeersssamtesssseeeee (), 1r-le-3, weight_decay-le-4
225
sched terch.opt httpeeiimm.scchhheeeeeeeo CcsineAnnea
te lingLR (opt, T_max=30)
Tuned weights
228
alpha 0. 769634
229
beta = 0.430968
230
gamma = 0.250750
232
233
def composite_lasee (o, t):
234
11 F.11_loss (o, t)
12 F.mse_ loss (dwt. (o), dwt (t))
el F.11_ooss (sobel (o), sobel (t))
237
return alpha
beta 12 ganma el
238
- 8) Train + validate best_rmse 1e9
242
patience =
243
wait =
for epoch in (1, 31):
model.traing
246
tloss 0.0
247
for xb, yb in Loade_
248
xb, = xb. (device), yb.to (device)
pred model (xb)
250
loss composite_loss (pred, yb) opt. (ot.zero_grad()
251
252
loss. loss.backward
tloss += Loss.itemt
sched.step()
256
257 f sliding-winde validation model .vall ()
rmse_sum 0.0
260
stride, ps 128, 256 with torch g): torca.no_grrad)::
261
262
for nł, ot in zip (noisy_vva clean_ va): imn Image.open (nf) convert ("I")
37

---

Published as a conference paper at ICLR 2026
imc Image. open (cf) cenvert ("I")
tn transforms ToTensor (imn) unsqueeze (0) to (device)
266
tc Cransforms.Tore
() (imc) unsqueeze (0) to (device)
267
- Н, | = tn. shape (tn)
torch. 1ir.
270
xS list (range (0, W ps + 1, stride)) [W ps]
271
ys 1ist (range (0, H ps + 1, stride)) [a ps]
272
for x in xs:
for y in ys:
p tn[:, :, y y ps, x x ps]
275
out model (p)
276
acci:,
+ ps, x ps] += out
ent [:, ent
ps, x
+=
279
mse mms_____ loss (rec, tc) ()tm())
280
rmsc_sum += math. matt (msc)
281
val_rmse = rmsc_sum len (ncisy_va)
print
f"Epoch (epoch): train_loss-ttleeee [[val_rmse:.0e
(traaa/innnentt_naanaaaaa (rann :.4f), val. RMSS
284
285
if val
best
287
best rmse val rmse
288
wait 0
289
torch. save (model sicte_ "o. w.//orkkigg/bestt.ptte
else:
wait
292
wait patience:
293
print ("Early stopping.")
294
break
296
print ("Best validation RMSE:", best_rmse)
207
208
-- 9) inference & submission --
model. [odel_1oad___e [oad_state_diaa (torch. load (" htt///rkkngg//bbsttt
model.evall
301
test files fttleesssacreeegooooooooppo////d
302
with hoan"
"w") as fout:
fout ("d.
for tf in test_files:
Image.open convert ("L")
306
t transforms. () (im) unsqueeze (0) to (device)
307
_ H, = t.shape
korch.zeros_iie
torch. like (t)
310
xs 1ist (range (0, ps 1, stride)) [ ps]
311
ys = list (range H ps + 1, stride)) [ ps]
312
with Berch.no_grad/l for x in xs:
for y ys:
315
p - tl;, ;, y y ps, x : x ps]
316
out = model (p)
acel:, 1,
ps,
ps] += out
entl:, :, y y ps, x x ps]
319
rec (ace / ent) cllamp (0, 1) .nuumyy () [0, 0]
320
id = 0s os. pa.path.pplitox. img_id
was.ppath.aasnne (tf)) [0]
321
for in (E)
322
for in range (W):
fout.write (f" iin i=]_]d][
cr) An")
334
325
print ("Submisste ssion written to htt.:tt........ttt........ ./werkingl wwweekingg/summsssinn.ccm
38

---

Published as a conference paper at ICLR 2026
The Coordinator submits the scripts generated by Coding Agents and their workspace to the community. Then, proceeds to the second iteration. The report generated by the Analyzer for the second Coding Agent
Pipeling:
our rinal pipeline is an Advanced Wavelet U-Ket with CRAM attent ton modules, Composite Losa, sliding window infereace. Specifically:
1. Data Frepazation Read all 112 neisy/clean train pairs, split 90/10 train/vel file indices (zandom seed 42)
On-the-fly extract 25 random 256x256 patchea per train image each epech, with random porisontallvertion toal :lipa and 90 degree rotations.
Validation uses single image sliding window pess petches with everlap) the 10% hold-out.
Model Architecture
Hoar DWT and inverse DMr implemented as fixed convalutional filters (Li, L8, HI, as bands)
Dual branch encoder: sepazale CDAMResBlocke for the approximation (1 channel) and detail channels) coefficients.
Two-level downsampling with MaxPool, interleaved CBAMResBlocee, and 256 chanceln bottleneck.
L'ecader with Conviranspase upsempling and CeAMesssleck exip-cennecticns lUpRes), recombining wavelet banda.
1x1 convolution coefficient channels inverse Image domain.
Composite
Ll (pixelst weighted s1pha-0. 769634
(wavelet coefficients) weighted by beta=0.
Scbel) weighted by gamma=0. 250750
Training
Optimizes: Adanli (1:=1e-3, weight_
schedules: tossreamealinguu (1__axx300)
Ratch size 8, 30 opochs, early stopping |patience=5] based on full-inage RMSR on the val set.
5. Inference
sliding window image pixels. Cenerate submission.cov with id=imago_row rov_col float valuas.
2564256 patches, stride=128, averaging overlapped
6. kyparparameter tuning
Optina over (ph, epochs and 10 patches/image Lmage.
gamms) 20-train/5-va) mini-aplit for trials, with 2
found: 1.pha=0.76963 beta-0.130958 gamma-0.250750.
valtionion RMSE
hold-out: "0.312685.
Raferences: CRAM (Noo et al., ECCV 2018), Haar DNT.
Summacy:
Haaz DWI:
Novelty: 2
Rationale: Usea atandard 1-level Basr wavelet decorpoaition
Feasibillty: 10
Raticnale: straightforraar f' xed sonv impl mmllmmntation in FyTorch.
ECCecLiveness:
Raticnale: Explicit frequency separation aids denoising of Sugh-frequence noise.
afficiency:
Rationale: Iiihhweiigtt fixed kernals on GPD.
Confidence:
Rationale: Verified on and full runs.
CBAMResDlock:
Novelty: 5
Ratienale: Incerporates
spatial actention context-sware features.
Feasibi 1iy: 8
Ratienale: Moderate complexity but supported by existing PyToreh modules.
Effectiveness:
Raticnale: Improves focus on text atrokes, empirically lowe:s RMSE,
Efficiency:
Raticnale: Adds overhead ("1.5+), but fits within ueu budget.
Confidence:
Rationale: Confirmea acrosa toy and full training.
Upkes (Conviranspese свл)):
Novelty: 3
Raticnale: Standard upsampling with skip-conaections actention.
Feasibility: 9
Racionale: Simple extenaton ReaRlook.
39

---

Published as a conference paper at ICLR 2026
affectiveness:
Rarionale: Kaintains detail in reconstruation,
Erficiency:
Ratienale: Ressonable GPC use.
confidence:
Roticnale: Working end-to-end script.
overall:
Novelty: 7
Ratienale: Dial bzanch Net in wavelet demain combining DWI, CBAX, skip connections. Feasibility: 7
Raticnale: Noderate engineering but fully implemented.
iffectiveneasy
Raticnale: Achieved RMSE "0,0127, strong denoiaing.
Efficiency:
Rationale: 25 patches*8 batch; 30 epecha in "20min.
Confidence:
Raticnale: seprodueible A6000.
Composite Toas:
Novelly:
Raticnale:
pixel, frequency, edge
Peasibillly:
Ratienale: Simple weighted sum.
sffectiveness:
Raticnale: Lower RMSE than tt or 12 alone.
Erciciency:
Raticnale: Small overhead for extra transforms.
confidence:
Ratienale: Verified in Luning and full
lata Augmentation:
Novelty:
Rationale: Pllps
Feasibility: 10
Raticnale: Trivial implement.
Effectiveness:
Raticrale: Standard but helpful for invariance.
afficiency:
Raticnale: C3U-1evel only.
Confideace:
Raticnale: Common practice.
CosineannealingLR SarlyStepping:
Novelty:
Rationales Standard learning-rate decay wal-based stoop.
Feasibility: 10
Raticnale: Built-in PyToreh.
sffectiveneas:
Rationale: Relps convergence prevents overfit.
afficiency:
Raticnale: Minimal overhead.
Confidence:
Raticnale: Observed smoothed training curves.
sliding-window Inference:
Novelty: 3
Raticnale: Ensures full-image coverage.
Feastbi l1ty: 10
Ratienale: Straighttorwaad loops.
Effectiveness:
Raticnale: Accurate measurement, outputs.
afficiency:
Ratienale: Reasonable inference time ("10 min).
Confidence:
Rationale: Verified end-to-end.
Hyperparaneter Tuning (Optuna): Novelty:
Rationale: Automated search of loas weights.
Feasibility:
Raticnale: Low-cost mini-training.
ELCectiveness:
Ratienale: Found
weights defaults.
afficiency: 5
Rotienale: trials epochs
Confidence: 7
Ratienale: Verified chosen paremeters.
Weaknesses and Suggestions:
40

---

Published as a conference paper at ICLR 2026
EMA TIA Bug: The EMA 4-way TTA implementation caused RMSE blow-ups, likely due to incorrect accumilation. Before reintroducing,, validate TTA on toy aplita or isolste EXA without TIA,
2. Multi Level Wavelete: Extend to 3 levele of DWT to capture coarser scales, with separate brenches per level.
3. Advanced Loss Torma: Incorporate perceptual (VGG) structural (8SJM) losses encousage
visual ridelity.
Dynamic Less Keights: Schedule (alpha, beta, during training (0.g., start with high 11 gradually ly emphasize wavelet).
5. Mixed Precision & Gradient Checkpointing: Purther reduce nemory and speed up with AMP and checkpointing deep layers.
6. pheper//biquitonn Astention: explore transfomer-style attention (Swin, Vir) in encoder for global context.
7. Ensemble Modela: Train multiple randon seeda or acchitectures IReatNet, UNett-) and average outputs.
8. Multi-Scale Inference: Use patchea at multiple scales or whole-image passes via down/ upscaling for context.
9. Adaptive Overlap: Use weights edges of patches smoothly blend eveslaps (Gaussien windows). Distriouted Hyperparamster Search: Soale optura across more trials BORR for acchilecture search.
Implementing these will likely push RMSE even lower and improve robustness across document noise patterns.
Pull code: (..iitted))
Following the brainstorming and idea refinement phases, the current idea memory is presented below. Notably, entries (3), (4). (5), and (7) are newly proposed by the agent, while the remaining ideas have been refined from those in the previous iteration.
(0) Patch-based convolutional super-reaolution deroiser: a sliding-window, patch-based model that predicts multiple denoised and super-resolved pixels per patch. Architecture inspired by Kai with progreasively increasing filter counts, LeakyRetu sativations, and initialization for robust convergence.
(1) Advanced wavelet domain dual branch U Net with CBAM attention and cemposite loss:
deccapose inputs via fixed 1-level DWT (Li, L8, HL, bands), encode appreximaticn and detail separately with CBAX ResHlocks, decode and fuse via 1x1 convolutions, and train ard-to-end vaing weighted sum pixal 14, wavelet-band 12, and edge 1.1 losses. Uptimized with Adami and cosine-arnealing L8 scheduling.
(2) CAN-based reatoration ion framework: a ResVat-based garerator and 70+30 PatchGAN discriminator trained with combined losses-tl pixel losa, adversarial loas, stroke- consisscnsi loas via frozen stroke- feature (KN), and perceptual OCR--CCCCte festure loss. Includes RI gradient penalcy and spectral nermalization for stability.
(3) Masked autoencoder with vision transformer for denoising: patchify each image into non- overlapping aquare tokens, randomly mask a high percentage, pretrain a Vi7 encoder (12 layere, hidden 768, 12 heads) plus light transformer decoder on L2 zeconstruction of dirty images, then apperd an MtP head and fine-tune end-to-end on notay- psire with pixel differentiable jitter during fine-tuning; at inference use full-image encoding er averaged mask schedules.
conffiennc loss, Erploy zandom bleck dropout and color
(4) conditions diffuaton-based restoration: de forward dauastan-noise diffusion schedule, train a 5 5-1evel U-Nett condit on the dirty image via channel concatenation and SLLM/cross-attentee loss with classitier
of sinuroidal tizestep enbeddings. Use the standard DDPM MSE
guidance, and sample with a deterministic DDTM sampler ("30 steps!. Opticnally post process with bilateral or median filtering to zemove speckles. (5) Learnable spectral gating in the fourier domain: compute the 20 8rT of the dirty image, split its spectrum into low/mid/high radial bands, apply learnable complex masks per hand and moculate each by gate scalars predicted by a Lightweight CNN on the dirty image. Recombine via Lnverse EPT and train end to end with 12 pixel loss plus a spectrel smoothness zegularize:
the masks.
(61 Hypesnetwork modulated 0 Net: extract per image noise staliatics (mean, std, skew, kurtosis, histogram bins), feed into an MLE hypeznetwork that outputs PILM scale (gamma) and shift (beta) parameters for selected convolutional feature naps of base U-shaped CMN. Randomly augrent noise levals during training; train end-ta-end on noisy->elean with Li loss and a amall seqularizer pushing gamma->l, beta->U, At Inference compute atats per image, generste 7I1M parems, and denoise via the modulsted U-Net.
(7) elind-spet test-time adaptation: for each
image, randomly mask subset of pixels and fine tune lightweight CNN with skip consections on the single image using maeked L2 loss and augnentations for a few hundred gredient steps. Optionally ensemble multiple random snicializations to reduca variance.
(8) Mult t-moeel ensemble with diverse preprocess!ngs cluster trained models by their error patterns and combine their outpuis, Apply different preprocessing pipelines (zaw, beckground- removed, edge maps, thresholded) to the input, denoise with clustered sub- ensembles, and fuse predictions for robustness
noise charaeteriaticss.
(9) Bahanced sugmentation and target refinement: simulate realistic scanning artifacts by applying text deformations, lighting variations, and sindoos to clean images. identify and compensate for systematic fatti in the provided "elean' targeta (e.g., single- pixel haloa) by either modeling them as noise or adjusting groand-tzuth labels during :reining.
And solution drafts generated in this iteration are:
41

---

Published as a conference paper at ICLR 2026
Pipeline ResNet-34
U-Net Multi-Scale kdge Ictal-variation Loss
Overview:
& robust baseline using pretrained Reslet 34 backbone
Net encoder fused with light weight decoder. Cembines 4i loss, sobel edge loss at multiple scales, and total- variotion regularizer to preserve strokes while smoothing background noisa. Mixed prectaton training and s sliding-window Inference ensure the entire pipeline runs In -45 46000.
1. Data Preparation
Read all train noisy/clean 2NGs, nermalize [0,1]. Extract on-the-fly 256+256 patches; random. crop + zandom horizontsl/pertice flips + 90 degree rotations.
90/10 split by file indices laeed=62), Use batch aise 8-16.
2. Model Architecture
Encoder: worchvision. wtt.:/sooooooddsseeeeeeeeeeeeeee (gretzained=Iruee) first conv modified to >61 channels.
Dacoder:
four upsampling stages (SonvTranspase2d + Convid+RN<Rold) mirrering Restect blocks, with belp-connecttors from layers.
Final cony 64 >1 Sigmold.
3. Loss Punction
Let y_hat and y be predictions and targets.
Liloss LLioss (y.hat,))
záge loss: 11 between Sabel (y_hat) and sobel (y) as both full resolution and half resolution
(downsample by
TV: IV where TV mean (Iansll xy xy_bat Anabla yy_hatl).
aipha+il betavsdge_full ganmarsdge_hale deltaxIV, e.g. sipha=1.0, beta -0.5, gamma-0,25, deltamle 5.
optimization Opt imizers Adart[] decght_decay----).
Schedulers 2osinekmeelling (
Mixed precision vis torch. cuda.amp.
garly scopping validation RMSE (patience=5).
5. Inference Submission
Parform sliding-window inference
image with 255+256 patches, stride=128.
Average overlapping patches.
Clamp outputs [0,1], submission.ce id=imege_cow_col.
Compute budget:
inference.
ine 2: Leplacian-Eyramia Multi-Scole Residual U-Niet with Pyramid less
Overview:
dimmain dencises each hand via shared-waight residuel blocks, and merges then back. Multi-level both coarse structures
pyzamid- domain network that decomposes images into multi scaii Laplacian bands,
losses focus fully end-zo-end in zyTorch, training finishes in "50 min A6020.
text details. Efficient
Data Augmentation
Same da:a split and patch sampliag (256x256, flips, cotations).
2. taplacian Pyramid Transform (LPT)
the fly, for each patch: create 2-level Savastan pyranid vaing avg-pool ing (scale 1-x0.5-20.25), then compute Laplasian bands Lb= (1-opsample LI= (1/2-upasmplee)//44) and a reaidua" low band 12=T/4.
[1,
input channels.
#ncoder-dacader U-Net with 3-channel input and 3-channel output, down/spsanpl ing stages, channel wiaths.
Each: block: 3+3 Coav-BNiReLA and residual skip connectien.
Reconstruction
Output is 3 residual corrections 80, to each band. Raconstruct clean patch: ao T. + R0; al T. R1; a2 + R2; I a0 (ala)) uu
5. Loss Function
losses: WOxLl (a0, HC) (21, (2xL1 are ground truth bands and w0=1.0, wi=0.5, 22=0.25.
lambda_pixet.ll 17, Igh) lambda_pixe0.1.
6. Training
Adamn (1r=5e-4, deccy_leec)))--55) one-cycle scheduler Ir=še-,, epcchs=30), (patience=6).
Batch sice 8, mixed precision.
Early szopping
42

---

w1, w2, w3, wt 1.0, 0.5, 0.25, le-5 1ambda_aux Published as a conference paper at ICLR 2026
7. Inference
full test Images, run eliding-wirdow with 256+256 patchea, stride=128.
For each patch, apply pyramid model reconstruct > accunclate and average. Clamp to [0, write csV.
Compute budget: "25 min training
"8 nin inference.
The best code proposed in the second iteration is:
import os, random, csv import numpy as np from glob import glob from PIL import Inage
import torch
7
import torch.nn nn as nn import torch. nn. functional as from torch. wtw.s.iilld............ data import Dataset, DataLoader from torchvision import transforms, models tv_models from corch.cuda. amp import autocast, GradScaler
11
12
1) Roproducibility
SEED = 42
random. (SEED)
16
np. random. seed (SEED)
Barch.manual_seee (SKED)
torch. manual seed all (SEED)
20
DEVICE terch. device ("cuda" terch.cuda cuda. available () else "apu")
21
2) Paths 22
TRISN " ///input/train"
TRAIN_CLEAS CLEAN "./Amptt/trriin_cleaeddd
25
TEST TEST_DIR DIR "//input/testt"
36
WORK_DD DIRK_DIR_S_ = "/vorking0"
27
os us.makecirs (WORK DIR, exist_ok-True)
29
3) Hyperparameters
30
PATCH_S1K 256
31
STRIDE 128
BATCH_SIZE 12
LR, WD 1e-3,
le-4
34
MAX EPOCHS 50
35
PATIENCE = 10
36
T_MAX 50 # for LR scheduler
4) Loss-won Loss-weight constants (from tuning)
39
40
1.4394633936788146
1ambda_mse 0.1312037280884877 lambda ssim 0.0319999406/240032 lambda ssim2
43
Lambda ssim
44
45
5) Cataset augmentations
47
class OCRDataset (Dataset):
48
def init__ noisy_list, clean_list, ps, train):
49
self.ncisy. self.clean = ncisy_list, clean_list self.ps, self. ps, train
self.to_tensor transforms. TcTensor self.aug transforms.Comm Compose
52
53
54
transforms Lransoomms.Acmmmhhccieee
transforms. wandemHorizontaal
43

---

Published as a conference paper at ICLR 2026
transforms Landenneeetiisallippppppp (1.0)
transforms Randomootation (90),
39
transforms RandonRotation (180),
60
transforms, Ranoommntooannnnonotttttiio (270),
63
transforms. Rarommallly((taassssooscccssaaas (3, (0.1, 2.0)
} p=0.3), transforms Saucaaghhoyltatttttoooooaoaaaaao Randdmdd (2.0)
64
p=0.3),
66
67
def _een___
return (em(seff.noisy)
70
71
def cetiteen(ssll i):
72
Image.open seeef.nnisyy.. [i]) convert ("D")
(scl.. convert ("D")
8, h n.size
75
# pad
76
if w self or h < self.ps:
pad = (0, 0, max (0, selt.ps w), max (0, self.ps - h))
transforms. www.tiunan............ pad pad, fil1=255)
с transforms. functicnal. pad(c, pad, fill=255)
80
8, h n.size
81
8 crop if self.train:
random. randint (0, self.ps)
y random. randint (0, h self.ps)
85
else:
x (v self.ps) // 2
y (h self.ps) //
n n. crop (x, y, x self.ps, y + self.ps))
89
с c. crop (x, y, x self.ps, y + self.ps))
90
if self.train and fandom.randem. < 0.5:
self aul
с self.aug(a) return (e)
93
94
6) Prepare train/val split
97
noisy_files (glob (f" (YRAAINNOISS)/4.pps.
98
clean_files [E" (TRAIN os. cs.path. basename (x) for x in
molsy_files]
N = len (en
idx list (range(N)
101
random..hufflelida ntr = int (0.9 N)
102
tr_idx, va idx idx]:ntr), idx(atr:]
train_noisy [noisy_files(i] for in tr_idx] train_clean [clean_filessi] for in tr_idx] val_noisy [ncisy_files for i in va_idx] val_clean = (clean_files/f for i in va idx]
106
107
train OCRDataset (train_noissy train train_clean, PATCH SIZE, train=True) val_ds ocRDataset (val_noisy, val_clean, PATCHLSIZZ, train=False) train_loadcr = Dataloader
110
111
112
train_ds, batch_sizo-BATCILB5 shuffle=True, num_workers=4,
pin_memory=True
val loader D 114
115
val_as, batch_ www.-ssieeeeettt-eeeeee shuffle=False, num workerse,,,,,
pin-menory=
44

---

Published as a conference paper at ICLR 2026
118
sobel, IV, SSIM helpers
119
sob_x
120
torch.tensor o, -1l, 12, 0, -21, [1, 0, -111, dtype=torch. 32
vveew (1, 1, 3, 3)
122
.to (DEVICE))
123
124
sob_y = sob x. transstra 3)
136
127
def sobel (x)
128
- F. convdd (x, sob_x, padding-1)
= cxnv2d sob padding=1)
return torch. sqrt (gx gx gy gy + 1e-6)
131
132
133
def
dh (xli, 1i,
xli, i, :-1, :1) abs () mean
(x[:, i :, 1:] x(:, : :, :-1]) abs () mean
136
return + du
137
138
def ssim map
c1=0.01++2, C2=0.03++2);
140
mu a F. avavg_pool2dia 3, 1, 1)
141
mu_b F.avg_poo12dd (b, 3, 1, 1)
142
sa = F.avg_pool2d a, 3, 1, 1) - mm_a * mu_a
sb (b b, 3, 1, 1) -mu mu
F.avg_pool2d(s 0, 3,
1) mu mu
145
num (2 mu_a mu b C1) * (2 sab + C2)
146
den (mu_a m mu * mu C1) * (sa + sb C2)
147
return num / (den + 1c-8)
149
150
def ssim loss (a, b): ssim_lcss return 1.0 ssim_map (a, mean ()
151
152
154
8) loss terms
155
11 loss nn nn LlLoss()
mse loss = nn. MSELoss
158
159
def loss_tcrms ppred, target):
160
Liv 11_ 11__oos (pred, target)
MSEv = mse mas_loss (pred, target)
Ef (s_]e
scbel (target))
163
p2, t2 F..vv (pavg 2), a. (targee (target, 2)
164
Eh = (sabell)))) sobel
165
TVv lettalvariaaton (pred)
166
return 1.1,, MSKV, Ef, kh, TVV
167
168
9) Model w/ deep supervision 109
class ResUNetDS (nn. (on.Modlle):
def init (sel___sef))
172
()per()._iit___
173
r34 = wwwmodelss.resette (prpteeeeeepettttteeeeeeeeeee
174
sclf.cnc0 = nn. (1, 64, 7, 2, 3, bias-Falsc)
self.enco..eight.dda = convl c34.conll..eiight........ wwt.:nwwwwwt...att............ True)
waw.aaaaaaa..... mean (dim=l, keepdin-
176
be0,,bn0, relu, self.poc10 r34.bn1, r34.relu, r34.ma 334.moxpool. encl, self.enc2 layer1, r34.1ayer2 self.enc3, self.enc4 = 13yer,, rayera
177
178
45

---

Published as a conference paper at ICLR 2026
def up o): return um.ConvTrangppse2dii o, 2, 2)
181
182
183
def cb o):
184
return Sequentnt
Cn. 0, 1, bias-False),
186
nn, mm.Batchhorm2dd)). nn.ReLU (inplace-reeeee)
187
188
nn. Conv2d 3, 1, 1, bias=False), un.PBatchNorm22(e)). (n.eeluuinplllcc-rrre))
100
191
192
self.up4, self.dec4 up (512, 256), cb56 256, 256) self. self.dec3 up (256, 128), cb28 128, 128)
195
self.up2, self.dec2 up (128, 64), cb + 64, 64) sclf.aux_up, self.aux_out up4 64), Cn.oonvd(64,, 1, 1) sclf.upl, self.dec1 up (64, 64), cb + 64, 64) self.up0, self.oute up (64, 64), Cn.oon2d 1, 1)
196
197
self.sig (ngiidg
200
201
def (selff, x):
x0 = self. reluo (self.bn) .n0 (self. enc0 (x) self.poo10 (x0)
204
self.encl (x1)
205
x3 self.enc2 (x2)
206
x4 - self.enc3 (x3)
x5 self. (x4)
209
self. decd (torch.
x4], dim=1)) )
210
d3 = self belf.co dec3 s (torch. ([self. x3], dim-l))
211
d2 = sclf. dclf. (torch.cat ([sl1f..plddf) x2], dim=1))
self aal.:::igeeeII.....oooooooo.....
213
self tf main (rt.:siiigge....octtttaoff..... return main, aux
x0], dim-1))
214
215
216
218
model
ResUNetDS () (DEVICE)
219
10) Optimizer, scheduler, scaler
aptimizer torch, optim Pottim.AamM
1r=LR, weight_decay=ND)
222
scheduler torch. torch.optim. opt wt_.imm.rscchhulerr.cmmm iagaaarrocccccccaaaaaaaaaaaaaaa (optinai,,,,,,,,,, T_ T MAX)
223
scaler GradScaler()
234
11) Training snapshot saving
226
best_rmes float ("inf")
227
patience =
228
snap_epochs = set (110, 20, 30, 40, 501)
for epoch range(1, MAX_ EAXCSS 1): 230
231
model.traing)
232
train_loss = 0.0
for noisy img, clean img in train_rlasaalt loader:
poisy_img, clean img noisy_img (DEVICE), clean_img to (DEVICE)
Detimizer.zero_gee
236
with autocast
237
main_pred, aux_pred = medel
Llv, MSEv, Ef, Eh, TVv = doss_terms (main_predd, clean_img) ssim_lost (main_pred, clean_img)
340
p2, t2 a_avv_pool (ma (main_pred, 2), F.avg_pool2d (clean__ccnnn 2) s2 = ssim loss (p2, t2)
241
242
main loss = w1 Llv
46

---

Published as a conference paper at ICLR 2026
Lambda MSEV
* Ef
246
+ w3 * Eh
247
+ w4 * TVV
+ lambda_ssim s1
lambda_ssim2
250
251
aux_up = = i.interpolate
252
aue_pred,
size=clean sha.pepee-.hhp_111 mode="bilinear"
255
align_corners=talse,
256
La, Ma, Ea, Eh2, TVa loss sasss_trrs__eeee (aux_up, clean clean_img) ssin loss (aux up, clean clean_img)
259
pa, ca avg poopppool2dlll taux_up, 2), F.avg_pool22 12d (cleen_ img, 2)
260
sa2 = ssim_loss (pa, ca)
261
aux_loss =
w1 La
lambda_mse Ma
264
+ w2 * Ea
265
+ w3 * Eh2
266
+ w4 * TVa
Lambda_ ssim
208
Lambda ssim2 sa2
269
270
lcss main_loss lambda_aux * aux_loss
scaler (loss) backward scaler. (oppeersteppoptimizzee
273
scaler.update()
274
train_loss += loss.item. ()
235
scheculer.stap()
277
validation
278
model. .evel)
279
se, count - 0.0, 0
with terch.no_grad
for neisy_img, clean_ing in in val_loader:
282
noisy ing, clean ing noisy to (DEVICE), clean img, (DEVICE
with autocast ():
pred, model (noisy_ing) se ((pred clean_img) ** 2) sum .item ()
286
count += pred. ()
257
val_rmse rmse = np. (se / count)
print(
f"Epoch (epoch): Traintoss-(train_loos///// (train_loaden :.4f),
VaIRMSE= (val rmse:.6ff
290
best snapshot
val_rmse best_rmse:
204
best_rmse val_rmse
205
torch, state__ttteeddiit___ os.path. join [NOR "best_full. pth"))
patience
297
else:
298
patience +=
299
if epoch in snap_epochs:
torch.save (model. sttat_dicct (),, os.path. join (WORK_DIR, f"snap_iepoch }.pth"))
301
patience >= PATTENCE:
302
print ("Early stopping. break
47

---

Published as a conference paper at ICLR 2026
print ("Best validation RMSE:", best_rmse)
306
307
12) Ensemble load
308
akpts = ("best_full.ppt pth"] sorted
[f for f in os listdir (WORK_DII) DIR) if f.startswith ("aaathh"""nnaa ("snap_"))
key=lambda x: int (p.sp ("_") [1] sp).st [0]),
511
) [-2:]
312
ensemble_nets 11
for ck in ckpts:
net ResUNetDS to (DEVICE)
eet.load_state_de (torch. lead (cs, pas., join (WORK_DIR, ck)))
516
net eva mnsemble_nets.. append (net)
317
320
13) wiiding-window ensemble inference
321
def ensemblc_infer (img_aerrrimaarrrl
322
h, w = img_arr. shape
inp forch..rom_nunge (img_arr 255.0) 0) unsqueeze (0) unsqueeze (0) .o DEVICE)
324
ph (PATCH_STZF
& STRIDE) STRIDE
325
(PATCH_SI2E - w & STRIDE) STRIDE
326
inp F.pad (0, pw, 0, ph), mode="reflect""
W Lnp.shape out torch. like(inp)
328
309
wt torch.zeros_like (inp)
330
for in range(0) (0, H PATCI_SIZE + 1, STRIDE) for in range (0, W PATCHLSIZE + 1, STRIDE);
patch inp [:, :, y y PATCH_SIZA SIZE, PAICH_SI2E] preds []
333
334
with torch. corch.no_gradll (), autocast ():
335
for net in ensemble_nets:
P, p net (patch)
337
preds.
338
avg_p corch.stace (preds, 0) mean (0)
339
out [:, ;,
PATCH_81ZH, x x PATCH_8TZR] +- avg_P
wt wt[:; ;,
PATCH_SIXF,
PATCH_81Z8] +- 1.0
out
342
out out 1 1, th, :w]
343
return out cut.detach (). cppu () numpyy () squeeze ()
346
14) Write submission. csv
347
submission sabmissioon_pth os. oas.pah..jo (WORK_ Ssubisssion.cav
348
with open (wwbmission_ppsteetteeeeeeooeeee path, "x", newline="") as f:
writer = cav. www.trr(er()
writer. pritter.writerow[""M1 for tf sorted
"value"])
351
352
glob (2" gl" IESTI_DRR//..pgggg key=lambda x: int (os. pat.pa.b.bseeaaee (x) split(".")
img_id os os. path. basename (tf) split [0]
355
img arrarray (Image. open (t.) convert ("L"), dtype-np. float32)
356
den = ensemble_ infer (img)
H, W = den.
for in range(H):
359
for j range :
360
writer.writeron ([f" ((ngL)))0 f" [1, j] :1:.f166.0. :16f
361
print ("Submission saved to", submission_path,
48


Reviews:
[{"review_id": "X546MX6JRZ", "reviewer": "Reviewer_d6GG", "summary": "This work tackles the problem of automating machine learning engineering. Differently from most existing work, the authors take a community-driven approach and setup where AI agents can perceive real-time information and knowledge from external (human) communities. To this end, they first present their own framework, named MLE-Live. Building on top of MLE-Bench, MLE-Live simulates a live stream of publicly available information, including source codes and discussions (as for Kaggle competitions), namely \"community artifacts,\" which are based on the pieces of information that were uploaded before the competition deadlines. In addition, the authors propose CoMind, a multi-agent framework that leverages such human inputs for machine learning engineering tasks and is made of five sub-agents with different roles (Coordinator, Analyzer, Idea Proposer, Coding Agents, and Evaluator). Evaluating CoMind on MLE-Bench (or MLE-Live) and ongoing Kaggle competitions, the authors present that it outperforms the baselines in most settings.", "questions": "1. How did CoMind's scores improve as it iterated during the \"ongoing\" Kaggle competitions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Novelty of the main idea  \nThe idea of incorporating real-time information from external communities for machine learning engineering agents is somewhat novel and interesting. In addition to machine learning competitions, CoMind has the potential to be leveraged for performing collaborative engineering tasks where human inputs are provided as real-time guidance, and this could be a more important potential application of CoMind.\n\n2. Presentation  \nOverall, the manuscript is easy to follow and understand, primarily due to the fact that it contains intuitive figures as well as clear tables. Also, the authors show fair details of the experimental results, including statistics, prompts used, and examples of task completion.", "weaknesses": "1. Comparison across MLE-Live and MLE-Bench  \nWhile it is an interesting idea to take inputs from external communities, the community artifacts may contain too many hints. Although I think it makes sense to utilize the information under fair settings (such as comparisons with other Kaggle participants), comparing CoMind with other baselines on MLE-Bench (Table 1) is unlikely to be an apples-to-apples comparison. I'm aware that there are the results with CoMind but without the resources, but presenting the CoMind results as the primary finding could be misleading.\n\n2. Reliance on and potential vulnerability to information from external communities  \nOn the other hand, it may also pose a concern that such information from external communities might harm the AI agents and their progress. One of such cases would be inaccurate information; even though the information is from humans, it is still possible that some information is inaccurate or noisy. Another possibility is getting attacks with specific intentions to hinder the participating automated research agents' progress, where such attacks may come from either humans or other AI models. Having defense against them may be important for the proposed approach to be properly launched for real-world competitions or tasks.\n\n3. Minor issues\n- I believe the other entities in the MLE-Bench leaderboard have the error range in addition to the scores, whereas the error range is not presented by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PBX1IzRIny", "reviewer": "Reviewer_41Ga", "summary": "The paper proposes MLE-Live, a live evaluation framework that can evaluate AI agents to gather knowledge from a simulated Kaggle research community. The paper then proposes CoMind, a multi-agent system that can combine external knowledge. The CoMind simulates a Kaggle community, including a coordinator, an analyzer, an idea proposer, and an evaluator. The paper compares against multiple SOTA agent frameworks. The paper also includes an ablation study, qualitative and quantitative analysis.", "questions": "See weakness", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides a live evaluation framework, which contains not only competition, but also discussions and kernels. The paper cut the data before the competition deadline to mitigate data leakage.\n2. The CoMind has an analyzer to distill knowledge from community artifacts. The paper also evaluates the eight Live competitions. Apart from quantitive analysis, the paper has some qualitative analysis on task category, winrate, and code complexity. \n3. The paper provides the code and its model. In the appendix, the paper shows training details, additional evaluation details, and detailed case study.", "weaknesses": "1. The evaluation metrics follow the standard MLE-Bench for the main table. The ablation study only reports the win rate. I did not see any metrics directly related to the collaborative nature of agents. The novelty of MLE-Live seems to be limited.\n2. Some parts of the paper are not clear. For example, what are the tasks included in the MLE-Live? The paper did not list any statistics or any details of those competitions. The paper also seems to categorize the tasks into three levels. However, it is unclear how those levels are determined, whether through the complexity of tasks or datasets. The external knowledge shared in the MLE-Live is also unclear. What is the average length of a discussion? What are their topics? The main architecture is very high-level and purely prompt-based. The paper also fail to provide cost analysis.\n3. Compared to other benchmark papers, the evaluation is pretty limited. For example, in MLE0Dojo https://openreview.net/pdf?id=5W5mFU4oMO, they evaluate eight different backbones and provide task difficulty analysis for different domains. Using code length to evaluate code complexity instead of human annotation seems to be too coarse. The paper also fails to include any error analysis for the failure case.\n4. The paper fails to include a use of LLMs section. The evaluated closed source LLMs are also not SOTA>", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wCUVkm1oNG", "reviewer": "Reviewer_vka8", "summary": "The paper CoMind introduces MLE-Live, a benchmark designed to evaluate agents’ ability to leverage shared knowledge in realistic, community-driven Kaggle-style settings, and proposes a multi-agent system that simulates and contributes to such a research community. CoMind comprises five specialized agents—Coordinator, Analyzer, Idea Proposer, Coding Agent, and Evaluator—that iteratively synthesize ideas, implement pipelines, and refine solutions by integrating public discussions and code. Experiments on 75 Kaggle competitions from MLE-Bench show that CoMind achieves a 36% medal rate, surpassing prior systems like Neo and R&D-Agent, and in eight live competitions, it ranks within the top 7.35% on average, outperforming 92.6% of human competitors. The study demonstrates that community-augmented collaboration significantly enhances autonomous ML engineering performance.", "questions": "- Is the benchmark going to be public for people to use soon?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper introduces the novel concept of community-driven evaluation and agent collaboration in machine learning engineering. MLE-Live and CoMind simulate social learning and information exchange, a key feature of real-world research. I think this is quite meaningful.\n- The paper is well-organized and readable.\n- Its strong empirical gains and reproducible open-world setup could influence future benchmarks and frameworks in agentic AI, AutoML, and collaborative reasoning.\n- I like the ablation. One downside of the main table is that the base model is all over the place. The ablation helps to put things in perpsective.\n\nOverall I think the paper has its merit. A MLE benchmark that includes discussion, notebooks can be a useful asset for future agentic research.", "weaknesses": "```Gain came from the community data```\n\nFirst I want to make the point that I believe a MLE-Bench with public resource is good. I have nothing against that. This weakness is mainly concern about the scaffolds proposed. In figure 4 there is the ablation where CoMind w R is much better than CoMind w/o R. CoMind w/o R is roughly equivalent to AIDE with the same resource and backend model. The improvement from scaffold itself is constrained. It would be great to see more evidences that CoMind is a better scaffold than others. \n\nOne evidence I think would be helpful is to actually use the same model (o4-mini) for the main table or at least for a couple top baselines in the main table. This is to eliminate the factors of backend llm. Another thing is to let other agents to have some kind of overall view on the public accessible resources. The current setup in the ablation study has a problem that top voted notebooks may not be the most helpful/effective solution. Having that ablation would greatly help the scaffold's case. \n\n```More evidences of collaborative reasoning```\n\nWhat reasoning modes do they workflow commonly show? Does it take the strongest notebook and improve upon it? Making those clear is crucial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7wazzgdlUU", "reviewer": "Reviewer_4mfQ", "summary": "The paper proposes CoMind, a multi-agent LLM system for ML engineering that integrates external community knowledge. It also proposes MLE-Live, a framework simulating Kaggle-style community environments with public code and discussions. CoMind uses five specialized agents (Coordinator, Analyzer, Idea Proposer, Coding Agent, Evaluator) to iteratively generate, refine, and evaluate ML solutions. The authors evaluate it on 75 past Kaggle competitions (MLE-Bench) and 8 live competitions. It achieves a 36% medal rate, better than some other open-source baselines (R&D-Agent, ML-Master, AIDE). In live tests, CoMind ranks in the top 10% on average, outperforming the majority of human competitors. Ablation shows strong dependence on public resources, removal causes major drops in submission success and performance. It generates longer, more complex code than baselines, which could mean a focus on exploration but potentially lower efficiency.", "questions": "Can you provide more information on the runtime and computational costs per competition (tokens, wall-clock, GPU hours)?\n\nDoes CoMind's higher code complexity correlate with actual performance gains or just verbosity?\n\nHow would the framework generalize beyond Kaggle-style settings (e.g., real ML research or open-ended tasks)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper's strenghts include the MLE-Live setup with realistic access to public resources. They show strong empirical results on both retrospective and live tasks, with a clear system design, their multi-agent division of roles aligns with human research workflows. The results are there, demonstrating practical success in real Kaggle leaderboards.\n\nThe results overall on Kaggle are strong enough that I am leaning towards accept, but I still feel like their approach could be overfitting to some Kaggle property and urgently needs more validation to show its generalisability on another, similar type of challenge (see weaknesses)", "weaknesses": "The weaknesses include the efficiency tradeoff with CoMind producing longer and more complex code, with higher compute/time cost.\nI would have liked to see a more thorough analysis with limited tokens, and of runtime efficiency. As such, the scaleability is not really clear to me. It may be too heavily dependent of Kaggle-specific properties, it would have been good to include (even if in a limited manner) some other, similar use case as well, simply to avoid overfitting on some weird Kaggle idiosyncracy. Some of the improvements over baselines are rather modest given the additional architectural complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)