InnoGym: Benchmarking the Innovation Potential of AI Agents
Authors: Jintian Zhang
Kewei Xu
Jingsheng Zheng
Zhuoyun Yu
Yuqi Zhu
Yujie Luo
Lanning Wei
Shuofei Qiao
Lun Du
Da Zheng
Shumin Deng
Huajun Chen
Ningyu Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: innovation
agent
benchmark
Primary area: datasets and benchmarks

Abstract: LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present \textbf{InnoGym}, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide \textbf{iGym}, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both.

Published as a conference paper at ICLR 2026
INNOGYM: BENCHMARKING THE INNOVATION POTENTIAL OF AI AGENTS
Jintian Zhang Kewei Xu JJinsheng Zheng Zhuoyyn Yu
Yuqi Zhu Yujie Luo .Lanning Wei ,Shuofei Qiao Lun Du
Da Zheng Shhmiin Deng Huajun Chen " Ningyu Zhang 9f
Zhejiang University & Ant Group National University of Singapore
Zhejiang University Ant Group Joint Laboratory of Knowledge Graph
echangjinte ian, zhangningyuj@z edu. cr. zhengda. zhenggantgroup.ccmm com
https: / /ete:://gitttie //gtttttt..///////// com/ commzjunnp/igym
ABSTRACT
LLMs and Agents have achieved impressive progress in code generation, math- ematical reasoning, and scientific discovery. However, existing benchmarks pri- marily measure correctness, overlooking the diversity of methods behind solu- tions. True innovation depends not only on producing correct answers but also the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of Al agents. InnoGym introduces two complementary metries: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark in- cludes 18 carefully curated tasks from real-world engineering and scientific do- mains, each standardized through resource filtering, evaluator validation, and so- lution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits per- formance gains. These results highlight a key gap between creativity and effec- tiveness, underscoring the need for benchmarks that evaluate both.
INTRODUCTION
In recent years, LLMs (Jaech et al., 2024; DeepSeek- et al., 2025a) and Agents (Wang et al., 2024; Guo et al., 2024) have made rapid progress in areas such code generation (Jain et al., 2025; Chen et al., 2021), mathematical reasoning (Hendrycks et al., 2021; Cobbe et al., 2021), and scientific discovery (Majumder et al., 2025; Jing al., 2025). However, most existing benchmarks focus solely on whether an answer is correct. Under this paradigm, any output that passes test cases or matches the reference answer deemed successful. Yet intelligence and innovation lie only in the results, but also in the methods: two agents may arrive at the same correct answer while following entirely different approaches. Such methodological differences are often overlooked in current evaluation frameworks.
To address this gap. we propose framework for evaluating innovation that formalizes each task as quadruple (P. S, V. D). Here, P denotes the problem instance, S the solution space, V the perfor- mance measure, and D the measure of dissimilarity between solutions. On top of this formulation, introduce two key metrics: Performance gain G and Novelty N. Performance gain G quantifies the improvement of a solution relative to the best-known baseline, while Novelty N captures the methodological difference between new solution and prior ones. Together, these metrics enable us to assess both performance breakthroughs and methodological innovation.
Building this framework, present InnoGym, which consists of two complementary com- ponents: iBench and iGym. iBench the first benchmark specifically designed to evaluate the innovation potential of Al agents. It includes 18 carefully curated Improvable Tasks, selected from
Corresponding author.

---

Published as a conference paper at ICLR 2026
real-world engineering (e.g., ROADEF Challenge') and scientific problems (e.g.. 2D-BPP (Chung et al., 1982)) where there remains clear room for improvement both performance and method- ology. These tasks stand in contrast to solved problems (with remaining improvement margin) and exploratory problems (lacking human baselines or reliable validation). To ensure fairness and reproducibility, we standardize each task through multi-stage filtering and augmentation, including resource availability checks, evaluator validation, solution collection, validator construction, and dataset partitioning. Complementing this, iGym provides unified agent execution environment that supports robust tool use and long- nog-hhrizon problem solving, ensuring consistent comparisons across diverse systems.
We conduct extensive experiments on InnoGym with several existing agent frameworks. Our find- ings reveal that current agents still perform significantly below the human state of the art on complex tasks. While some methods demonstrate high novelty, their lack of robustness prevents these inno- vations from translating into meaningful performance gains. This highlights key bottleneck in today's s agents: in real-world scientific and engineering problems, novelty alone is insufficient-trrue innovation must combine originality with correctness and effectiveness.
1) We propose principled framework for defin- ing and measuring innovation in Al agents, combining performance gain and novelty as two com- plementary evaluation dimensions. 2) We introduce InnoGym, the first benchmark specifically tar- geting innovation potential, consisting of 18 standardized Improvable Tasks curated from real-world engineering and scientific domains. 3) We provide iGym, unified agent execution environment that supports reproducible, long-horizon evaluations across diverse systems. 4) We conduct systematic experiments state-of-the-art agents, uncovering key limitations robustness and highlighting the gap between novelty and effective performance.
Our contributions can be summarized as follows:
In summary, InnoGym establishes both a principled framework and standardized benchmark for measuring innovation in Al agents, offering a reproducible and cross-domain platform to support future research on systematically evaluating Al's creative and innovative capabilities.
DEFINING AND MEASURING INNOVATION
Different
Feasible Solutions
C(s)
C(s)
C(s)
C(s)
C(s)
1
Solutons
Sknown
82
83
Skown
$2
Skown
Sknown
$6
85
Performance Novelty
83
S unkknown
$8
unknoww
Sunknown
Sunknown
810
Vieul Plea)
412
$11
813
812
84
(a) Innovation
(b) Partition of Solution Space
(c) Closed-form tasks.
(d) Improvement
(e) Exploration. (Math Conjecture)
Evaluation Overview
(GSMBK/SWEBnnch)
tasks. (ML)
Candidate Solutions
Infeasible Solution
Sub-optimal Solution
Optimal Solution
An illustration of our definition framework. (a) Core evaluation metrics. Innovation is evaluated along two dimensions: Performance (V) and Novelty (N). The colored shapes represent different candidate solutions, while the radius of the background concentric circles corresponds to the magnitude of performance (ss) (larger radius indicates higher performance). (b) The solution space is partitioned by feasibility (C(s)) and prior knowledge. Feasible solutions (i.e., C(s) = 1) are candidates for evaluation. (c-e) Categorization of three innovative tasks based on the spatial distribution of solutions relative to the knowledge boundary.
Figure 1:
Most existing benchmarks judge agents by answer correctness, overlooking the solution that yields the answer. Existing benchmarks for intelligent agents primarily focus on the correctness of the final answer, neglecting the underlying solution used to obtain it. Yet intelligence and innovation lie not only in what achieved, but in how. Two agents may output the same answer for a problem while
httpps: www.. roadef orgg chal lenge/
2

---

Published as a conference paper at ICLR 2026
employs fundamentally novel solution. This section introduces framework for quantifying innovation in terms its performance and novelty.
2.1
TASK AND NOTATION
We define a task as quadruple T = (P,S, D), where:
Problem instance P contains the task description, constraints, objectives, and evaluation artifacts (e.g.. ground-truth answer or test cases). Agents observe Pvisible C P. Phidden = P Piisible is for evaluation only.
Solution space S is the set of executable solutions S that can be submitted to solve P (e.g., code, proof/derivation, an algorithmic strategy).
Performance V S + I quantifies the quality of a solution. We define it as V($) C(s) R(s), where C(s) {0, }} checks feasibility or legality (format, execution, constraint satisfaction) and R(s) measures the degree to which a feasible solution satisfies the problem's 's objective (e.g., accuracy, pass rate).
Distanee D S x S -> R2 measures dissimilarity between two solutions. larger value implies greater dissimilarity in the underlying solution. Conceptually, D can be any task-appropriate dissimilarity function (e.g.. an embedding- distance). our implementation, we instantiate D as Agent-as-judge score that compares two solutions. See Appendix F for details.
We denote prior-known (human or literature) solutions by Sknown C S and the unknown region by Suuknown = S Skuwwn For brevity, we omit the task subscript on (P,S, V, D). Then we can define the optimal solution set S* that achieves the maximal performance score V*:
S
V
max V(s). #ES, C(a)=1
( V(s) = V
However, the optimum S* often unknown, intractable, or may not exist for many challenging tasks. Our framework therefore grounds its evaluation in the empirical and dynamic set Sknowns encompassing the best known solutions ranging from fixed theoretical optima to the evolving SOTA.
2.2 DEFINITION AND EVALUATION OF INNOVATION
What constitutes innovation? The management theorist Peter Drucker famously defined innovation "change that creates a new dimension of performance." Inspired by this insight, we formalize innovation within our task framework. We define a candidate solution as innovative if, subject to
satisfying feasibility constraints (i.e., C(s) = 1), demonstrates a meaningful differentiation from the set of known solutions Sknown- This differentiation is not one-iinensional: it implies creating value either through superior results or through distinct methodologies. To systematically quantify this, we introduce two complementary metrics: Performance Gain (G) and Novelty (N).
Performance Gain (G) measures the performance improvement of new solution relative to the frontier of known solutions. We define it as:
maxXE S V(h),
Sknown + 0,
G(s) Vs))
V
known!
Vinown
Vo.
Skoww = 0,
where Vo is a task- task-dependent constant baseline for the no no-prior case. A positive value of G signifies super-human performance breakthrough that pushes the fate-of-the-art
Novelty (N) quantifies dissimilarity prior solutions and is awarded only to feasible solutions:
minkes D(s h), Sknown * 0.
N(ss) C(s)
too,
Sknown 0.
We only compute novelty for feasible solutions by multiplying by C(s). For a problem with no prior known solutions, any feasible solution is considered maximally novel.
3

---

Published as a conference paper at ICLR 2026
What kind of solutions considered innovative? Given feasible solution with perfor- mance gain G(s) and novelty (s), we treat innovation as occupying specific regimes in the (G. .) space. In particular, and high N, which substantially improve task value while remaining methodologically distinct from all known baselines. We refer to solutions with high G but relatively low N as 2) performance in- novation: they push the state of the art primarily along the performance axis, often as sophisticated refinements of existing methods. Conversely, solutions with G(s) 0 but high N constitute 3) conceptual innovation: they achieve comparable performance to the best known baseline while introducing a markedly different, feasible paradigm. All other regimes, solutions that are neither better nor different (low G, low N), or those that are highly novel yet substantially worse than be- fore (large negative G with high N), are treated as unsuccessful exploration rather than innovation.
regard as 1) breakthrough innovation those solutions with both high G
2.3 DISCUSSION: A TAXONOMY OF INNOVATIVE TASKS
We categorize task instances according to the spatial distribution of known solutions relative to the feasible region and the knowledge boundary, as illustrated in Fig. 1(c-e). These categories are defined from cemanneenneeed perspective, rather than that of an agent. A formalized definition of each category is given in Appendix D.I.
Solved Problems: As shown in Fig. I(c), tasks with known, optimal solutions, such as problems in MATH (Hendrycks et al., 2021) or SWE-Bench (Jimenez et al., 2024). For these tasks, the per- formance ceiling is fixed (Vinowa the optimal score). Innovation primarily measured by N(s), rewarding new and potentially more efficient methods to reach the known optimal performance.
As shown in Fig. I(c), tasks with existing solutions but no known optimum, common in machine learning and optimization challenges. Sknown is non-empty but suboptimal. Innovation can be demonstrated either by achieving a new state-of-the-ar performance (G(s) > 0) or by discovering fundamentally different method to match current performance (high (s)).
Improvable Problems:
Exploratory Problems: As shown in Fig. I(C), open-ended challenges with no known feasible solutions, such proving mathematical conjectures tackling unsolved scientific problems. Here, Skuown = 0. The first feasible solution (C(s) = 1) found by an agent constitutes a monumental innovation, yielding both positive performance gain (G(s) V(s) 0) and maximal novelty (N(s) = oc). The focus is on the O-to-1 breakthrough.
INNOGYM: BENCHMARK AND SYSTEM (IBENCH & IGYM)
InnoGym consists of two complementary components: iBench, benchmark designed evaluate innovation capability, and iGym, a unified development and execution environment. iBench covers 18 carefully curated tasks drawn from real-world engincering and theoretical problems. We focus only on Improvable Tasks, which leave clear room for improvement in both solution quality and methodology. In contrast, Solved Problems (with known optimal solutions) and Exploratory Prob- lems (without human baselines or reliable validation) are excluded from the core benchmark, as they either provide no measurable improvement margin or cannot be reliably evaluated.
3.1
TASK SOURCES AND TWO-STAGE FILTERING
across top academic and industrial competitions and workshops (NeurIPS Competitions, KDD Cup, ROADEF, GMCM3, MLArchSys'), ), as well as from classic NP-hard problems in science and engineering. This produces initial pool of 197 items, as shown in Fig. 2(a). These tasks span diverse domains and are rooted in real problems that often require multi-disciplinary expertise and sustained collaborative effort, typically ranging from one week to one year. All selected tasks are public, peer-reviewed, and allow the use of a wide range of tools (c.g., CPLEX).
Task Sources. We collect tasks from
Stage One: Resource Availability and Affordability. As shown in Fig. 2(b), we first filter tasks by whether key resources accessible: datasets, validators evaluators, leaderboards, and at least
"Official website: wcwp://ce.ppccc....p.../// wcg..or.................... en/.
'Oficiil website: https: httpps://sitess googl. com/ wwwee//wwmmmmmmm.mmm httm////w//lrrccsssss... archsys
4

---

Published as a conference paper at ICLR 2026
Besource Avaiabte
Collected Solutions
Partition
NourIS
FODDD
Solutions X Evaluator/ Validator
Leederboard [2] Dev Evall Data
Evolustor
eecutabiliss
Environment Setup
Specifiction
GMCM
Rescurce Affordable
Comrciness
Valldator Construction
ROADEF MLArchSys
GPU Memory (d] Disk Memory
Solution Collaction
Runtime Cost Pi] Rundmee Memory
(R) Task Source (Collact 197 Itoms]
Resource Filtering
(c) EomainEEalluttrr F Fittring;
Data Augmentation
[Remaining teeks)
Augment (Remaining
Hard
Tosk
Data
Validator Example Submission
Task Description Development
Reference Solutions
Evaluster
Dependency
Leaderboard Evalustion
GRU
CPU
Butimiiation
Visilble for Agents
Invisible for Agents
x18
Data
(c)
Figure 2: Dataset curation overview. We collect 197 tasks from public competitions, filter by resource and evaluator availability, and standardize scoring (executability, correctness, absolute metrics). After augmentation with validators, task specifications, solutions, and environments, the benchmark yields 18 balanced and diverse tasks across domains and hardware.
one reference solution. We also examine computational cost, ensuring that GPU/CPU memory, disk usage, and runtime demands remain feasible. Tasks that pass this stage, and that be decomposed into multiple sub-tasks, are expanded into individual entries. This stage yields 72 tasks in total.
Stage Two: Evaluator Quality and Domain Balance. Next, we validate the correctness and executability of each evaluator. Tasks with unfixable evaluators removed. To maintain diversity, we further balance across domains, prioritizing newer and more representative tasks. After this process, obtain 18 high-quality Improvable Tasks tasks, shown in Fig. (ec)-(e).
3.2 ENHANCEMENT AND STANDARDIZATION
To ensure reproducibility and fairness, augment each task with six types of steps (Fig. 2(d)).
Task Specification & Environment Setup. We rewrite descriptions in Markdown, specifying task goals, input/output formats, and submission requirements with clear examples and figures. We package dependencies into reproducible environments (e.g., containerized builds).
Validator Construction (C). We build or refine validators to check submissions for format, fea- sibility, and constraints. For example, submitting code tasks validates function signatures, and sub- mitting answer tasks validates fields, ranges, and constraints. See Appx. G.3 for details.
Solution Collection. We collect leaderboard solutions and papers. For each solution, prompt Codex (OpenAI, 2025b) with an extraction prompt (see Appx. I.l) to distill its core strategy into structured representation for novelty analysis. See Appx. G.1 for details.
Evaluator Normalization (R)). 1) Absoluteness. We convert relative participant-dependent scores (e.g., ROADEF) into instance-level absolute scores, verifying consistency with original rank- ings (Pearson > 0.9, Kendall-7 0.8). 2) Executability. We ensure evaluators run correctly across languages via standardized command-ling or container entry points. 3) Correctness. We cross-check with public solutions and random baselines, adjusting until leaderboard consistency is achieved. Tasks failing this check are discarded. See Appx. G.2 for details.
Data Partition. We split datasets into development (visible) and evaluation (hidden) sets, aligned with leaderboard conventions. All collected resources explicitly divided into agent-visible and agent-invisible parts, as shown in Fig. 2(c).
3.3 TASK FORMALIZATION
Each task instance is formalized as a quadruple = (P,S.V, consistent with the definitions in Section 2. P (Pvisble, Phideen), where visible parts include descriptions, examples, development
5

---

Published as a conference paper at ICLR 2026
data, and dependencies, while hidden parts include evaluation data, reference solutions Skowwn and leaderboards. (s)) ((s) R(s). where C' is the validator (feasibility check) and is the evaluator (performance measure). D is distance function used to compute novelty with respect to Skowwn- Agents are given access only to Pisibbe and C, while hideen Rt, and Sknown remain hidden (Fig. 2(e)).
3.4 EVALUATION PIPELINE
VisibleData 4- Validator
Novelty
The evaluation process proceeds in three stages, as shown in (Fig. 3). 1) Submission. The agent sys- tem produces a solution artifact using only visible data and tools. 2) Performance Evaluation. The evalu- ator R computes a score if C(s) = (valid submis- sion); otherwise the attempt is rejected. 3) Novelty Evaluation. The submission is feature-extracted (via Codex prompts, as shown in Appx. H.2) and com- pared against known solutions Sknowa using the dis- tance function D. yielding a novelty score.
Agent System
AgentTools i Environment
D
1
Submission
Codex
Feature
P Performance
Evaluator
Skmow
Performance Evel Pipeine
Novelly Eval Pipeline,
Agant Infor Pipeline
Invisible Data Paidem
Figure 3: Overview of evaluation pipeline.
Together, these steps provide two complementary measures:
performance gain (G(s)) and novelty (N()). Both are required for task considered an innovative success. The key differences between iBench and prior benchmarks are summarized in Table 1.
3.5 IGYM: A UNIFIED AGENT EXECUTION ENVIRONMENT
In addition to iBench, our framework also introduces iGym, unified SDK designed to support diverse agent systems and long-horizon problem solving. While existing SDKs such as Open- Hands (Wang et al., 2025), AutoGen (Wu et al., 2023), and LangGraph (LangChain A1, 2024) simplify orchestration, they lack several crucial features needed for our setting, including robust recovery for long- long-running tasks, native concurrency, and consistent tool management. iGym ad- dresses these limitations by providing a common abstraction layer where agents can interact with environments, tools, and resources under both workflow-style and agent-style paradigms.
1. Seqence
2. KeyValue Information
3. Tree Plan
4. G.Grph Relationship
Memory Used
Chat History
Storage Retriever
Service
Function Sisteince Tool
Session Steend Tool
VLLLM
Search
Jupyter
Workflow
Autonomous
T Database
Calculatoor
Terminal
Observation Result Resource Info
AgentSystem
Agent
Agent
Agent
RAY
Memory
Memory
Memory
Action Call
Manage
Global Memory
CPU
Trace Trajectory
Tool
Checkpoint
Apply
Hardware Resource
Manager
Resume
Environment
Plugin
Apply
Figure The architecture of iGym.
show the overview of igym Fig. 4. Due to space constraints, we defer the full system design, detailed architecture, and illustrative figures to Appx C. There, we present the complete description of iGym, including its asynchronous Tool Dispatcher, recovery mechanisms, and examples con- current tool usage. This ensures that readers focus the benchmark construction in the main paper, while still having access to the complete implementation details of the runtime environment in the appendix.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of existing agent benchmarks and our proposed benchmark. "Ref. Sol." marks whether collected reference solutions are available. "Eval Perf."/"Eval Novelty" denote whether the benchmark explicitly evaluates performance and novelty.
Compute Profile
Eval Perf.
Eval Novelty,
Benchmark
Source
Data Domain
Difficulty
Sol.
MLAgennBench (Huang 2024)
Kuggle
Machine Learning
/
Easy
GPU
/
x
DSBench
Machine Learning
Easy Hard
(Jing et al., 2025)
Kaggle
Science
GPU
/
x
MLEBench
Machine Learning
Easy
(Chan et al., 2025)
Kuggle
(Cross-domain)
Hard
GPU
/
ScienceAgentennch (Chen et al., 2025)
Machine Leaming
Easy
Publication
Science
Hard
CPU/GPU
/
MI. Gym
(Nathani et ul., 2025)
Kaggle
Machine Leariing
Fasy
GPU
/
x
MLRCBench
NIPS: ECCV:
Machine Learning
Easy
(Zhang 2025b)
KDD Cup
(Cross-domain)
Hard
GPU
/
x
InnovatorBench (Wa 2025)
NIPS, ICLR; COLM
Easy
EMNLP; ACI
Muchine Learning
Hard
GPU
/
x
NIPS: GMCM;
ML (Cross-domain) Science; OR:
Easy
Ours
Classical; KIDI) Cups
Hard
CPU/GPU
/
/
ROADEI; MLArchSys;
Systems; Math
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Models and Agent Scaffolds. Following MLE-BENCH (Chan et al., 2025), we select three rep- resentative agent frameworks, MI.AB (Huang et al., 2024), CODEACT (Wang et al., 2025), and AIDE (Jiang et al., 2025) iGym environment, so that differences outcomes primarily reflect the agent design rather than infrastructure. In the main experiments, backbone language model. Sec. 4.3 further investigates the performance of GPT-5 (OpenAI, 2025a) and Ceemini-2.5-Pro (Gemini Team, 2024) as alternative base models. See Appx. E.1 for details.
main evaluation targets. All agents are executed in the unified
DeepSeek-v3.. (DeepSeek-Al et al., 2025b) the
Metrics and Evaluation Protocol. Our overall evaluation protocol follows Section 3.4. For each solution submitted by an agent, we measure innovation along the two dimensions defined Sec- tion 2.2: Performance Gain G(s) and Novelty (s). We now describe the concrete instantiation of N(s). Novelty is defined as the minimum dissimilarity between and the known solution space Sknown: where dissimilarity is measured by distance function D. Conceptually, D can be any tapo-approe dissimilarity measure. Here, we instantiate D via an Agent-as-judge procedure implemented with Codex. For each solution, we first apply extraction prompt (Appx. I.1) to Codex (OpenAI, 2025b) to obtain a structured representation of its core strategy. Given the ex- tracted profiles of agent solution and reference solution in Sknown- novelty prompt (Appx. H.2) asks GPT-5 to rate their methodological dissimilarity along six rubric dimensions, each scored a 0~44 scale. We average the scores across dimensions, aggregate over all h Skmown via the minimum distance, and then rescale the resulting value to [0, 100J to obtain (s). See Appx. F.1 for more details. To facilitate comparison across tasks, we further report a normalized ratio Rstio(s) ciple in our evaluation is that novelty is only meaningful when it effective: high novelty scores are considered important only when accompanied by substantial performance gains. We provide a more detailed analysis of the behavior and reliability /) in Appx. F.
(s)) where larger values indicate larger relative improvement. A key prin-
Implementation Details and Runtime. Among the 18 tasks in our benchmark, we select 10 tasks as our main evaluation subset. These tasks are relatively more tractable under our computing and engineering constraints (e.g., smaller resource footprint and fewer environment dependencies). For each task-agent- lakk-aann-modee configuration, we allow up to 12 hours of wall-clock time, or terminate earlier once submission is completed, whichever comes first. Following MIEE-BENH (Chan

---

Published as a conference paper at ICLR 2026
Table 2:
Comparison of three agent frameworks with leaderboard upper and lower bounds the 10 main iBench tasks. "Highest" and "Lowest" are the best and worst known leaderboard scores.
LeaderBoard
MLAB
COdEAcт
AIDE
Competition
Highest
Lowest
Gain
Ratio
Novelty
Gain
Ratio
Novelty
Gain
Ratio
Novelty
BEETL(M))
76.33
31.47
33.66
-0.47
66.67
BEETL(SIcep)
69.23
27.91
14.64
-0.21
62.50
/
/
-53.62
-0.77
54.17
Belka
30.62
1.26
-9.00
-0.62
45.83
-28.14
-0.92
45.83
-30.01
-0.98
20.83
CirelePacking
0.96
-0.43
-0.16
50.00
-0.008 /
-0.003
-0.25
-0.09
33.33
CDML
69.90
26.50
/
NPR
41.21
29.53
-17.10
-0.42
66.67
-41.16
-0.99
58.33
OAG
-28.59
-0.34
-30.38
-0.36
62.50 /
-29.87
70.83
PTTALC
48.59
14.50
RCIC
99.76
49.15
-99.67
-0,99
83.33
-99.67
-0,99
54.17
TrojanDetection
57.70
6.70
-54.80
-0.95
33.33
-50.10
-0.87
54.17
Average
57.94
23.79
-24.32
-0.45
56.55
-41.58
-0.69
54.86
42.68
-0.64
46.67
et al., 2025), we use the same decoding hyperparameters for all agents. Due to computational cost, each configuration run three times the main experiments. We report the best score over these three runs, restricted to runs that yield a valid submission. If all three runs for given configuration fail to produce a valid submission, corresponding entry is reported as "/" Table 2.
4.2
MAIN RESULTS
Our analysis the experimental results, presented in Table 2, suggests three primary takeaways regarding the current state of Al agents for innovation. See Appendix E for more details.
Substantial Performance Gaps on Complex Tasks. Our primary finding is that existing agents exhibit significant limitations on complex, open-ended problems. Across all evaluated tasks, no agent managed to surpass the tfate-of-the-ar human solutions. On tasks with intricate data for- mats complex requirements, such Aattion-Localisation-Chhaal
and Perception- Test- all tested agents failed to generate valid and executable solutions. These results highlight substantial performance gap between current agent capabilities and the robustness required for real-world scientific and engineering problems.
Differentiation in Existing Frameworks. Agent frameworks show distinct profiles. MI.ab leads both Performance Gain and Novelty, indicating a rare blend of innovation and execution. Code- Act and AIDE lag on both, likely due weaker handling of complex file structures and tool use. Notably, CodeAct nears the state of the art on CirclePacking, suggesting strength on well-specified mathematical optimization that does not generalize to broader tasks.
The Primacy Robustness over Novelty. Finally, our findings illuminate the intricate relation- ship between performance and novelty across different frameworks. While the three evaluated frameworks exhibited comparable levels innovation, their performance diverged significantly. This underscores the dominant role of solution correctness and robustness in the context of complex tasks. For example, RCIC and Trojan still returned some of the lowest performance scores. This disparity suggests that the primary bot- tleneck for agents on complex tasks is not deficit of novel ideas, but rather the inability to translate them into correct and robust implementations. Consequently, ensuring reliable execution quality is the foremost challenge and a critical prerequisite for their real-world applicability.
tasks, frameworks achieving mid-to-high novelty
4.3 EXPERIMENTAL ANALYSIS
To further dissect the agent's 's behavior and the utility of our metrics, we conduct a series controlled experiments the challenging Circle Packing problem.
8

---

Published as a conference paper at ICLR 2026
Null Solution
Rool
I: 1.84, 0.58
II: 2.59, 0.46
III: 2.60, 0.54
IV: 2.60, 0.63
V: bigg, 0.00
Rc
ITT E1I VI
X
Solution
VIII: 2.60, 0.04
VI: 0.00
2.61, 0.04
IX: 2.60, 0.04
VI
IV
IX
Best Solution
VII: 2.56, 0.18
(b) Vector-Space Representation of Solution Development Process
Solution Space Development
An illustration of the solution development process. (a) Solution Space Tree for Devel- opment: each node represents candidate solution, where the Roman numeral denotes the iteration order, the first value indicates performance, and the underlined value denotes novelty. (b) Vector- Space Representation of the Solution Development Process: complex-plane mapping that jointly encodes performance gain (magnitude) and normalized novelty (angle), providing richer interpretation of the development trajectory.
Figure 5:
Performance Gain
Novelty
2.23
0.52
2.29
2.43
2.43
2.49
0.66
2.44
0.58
2.35
Paromanceisumni
Novelty
1.50
0.19
1.00
0.00
18r
2Hr
3Hr
4Hr
Despseek V3.1
Gemini- 2.5
Got 2025 08-07
0.25
0.5
0.75
1.0
(a) Execution Time Analysis
(b) Base Model Comparison
(c) Effect of Sampling Temperature
Figure 6: Analysis experiments. (a) Execution Time Analysis: the effect of varying execution time budgets performance gain and novelty, running 3 times. (b) Base Model Comparison: the impact of different backbone LLMs, running times. (c) Effect of Sampling Temperature: the trade-off different temperature settings.
The Impact of Prior Knowledge on Innovation. We first investigate if AIDE can iteratively refine a strong, pre-existing solution. Starting with a solution generated by Gemini-2.5-PP (sum_rade rauu_- ratio=0.98), discover superior outcomes. As illustrated in Fig. 5(a), the agent follows an effective trajectory (e.g., 'Null III X'), where Performance Gain steadily increases. Concurrently, Novelty ini- tially peaks-reflectin a significant departure from the starting point-and then gradually decreases as the solution converges toward a local optimum. To better visualize this process, we propose a complex-plen representation. We normalize Performance Gain (G) to represent the vector's s magnitude and the normalized Novelty score (Nsta) define angle (2x (Fig. 5(b)), this mapping reveals directional information obscured by the scalar novelty score; solu- tions with similar N values can represent distinct methodological shifts. This demonstrates that our G and N metrics can be synergistically combined to form a richer, multidimensional representation of the innovation process.
observe that AIDE successfully navigates the solution space to
shown in
Temporal Dynamics of Innovation. Next, analyze the evolution of G and N over extended period, where each is measured relative to the previous time step. As shown in Fig. 6(a), G tends to improve over time, while N decreases. This reflects the principle of diminishing returns: as

---

Published as a conference paper at ICLR 2026
the solution improves, finding substantial further gains (lower G) becomes harder, and the agent's methodology naturally converges (lower N). Importantly, G remains non-negative throughout, indi- cating stable, monotonically improving search process, validating metrics' ability to capture the typical dynamics of iterative refinement.
Impact of Foundation Models Performance. To isolate the impact of the underlying LLM, we ablate the foundation model while keeping the agent framework constant. The results in Fig. 6(b) show that performance is heavily dependent on the base model's strength. More powerful models like Gemini-2.5-Prro and a hypothetical GPT-5 achieve high scores of 2.49 and 2.44, respectively, closely approaching the AlphaEvolve (Novikov et al., 2025) of 2.65. In contrast, DeepSeek-v3.1 achieves a score of 2.40. This aligns with general community perceptions of these models' capabil- ities and underscores that agent frameworks act as powerful amplifiers of the base model's intrinsic reasoning and coding abilities, rather than being a substitute for them.
citiorattioo-xplitaattiiin Trade-offs at Different Sampling Temperatures. Finally, we inves- tigate the effect of sampling temperature on agent performance and novelty. Fig. 6(c) reveals a classic exploration-exploiatiio trade-off. Performance Gain is highest at low temperatures, where the agent exploits known good strategies. Conversely, Novelty increases with temperature as the agent is encouraged to explore more diverse, less probable solutions. Our analysis identifies "sweet spot" in the mid-temperature range 0.75), where the agent achieves near-optimal performance while significantly boosting methodological novelty.
RELATED WORK
Evaluation for ML Engineering and Scientific Discovery. Initial evaluation efforts for LLMs (Jaech et al., 2024; Yang et al., 2025) centered on foundational capabilities in domains like mathematical reasoning (Hendrycks et al., 2021; Cobbe et al., 2021) and code generation (Chen al., 2021; Jain et al., 2025; Jimenez al., 2024). The primary metric in these benchmarks is typically correctness, verified through unit tests or exact-match answers. A subsequent line of work evaluates LLMs on open- open-ended, improvable tasks where the goal is to discover high- performefoninng solutions rather than single correct one. For instance, MLE-Bench (Chan et al., 2025) challenges agents to develop ML pipelines for Kaggle competitions, directly measuring the solution's value via leaderboard rankings. This focus on performance-driven evaluation is echoed in other MI engineer- ing benchmarks (Huang et al., 2024; Zhang et al., 2025b; Chen et al., 2025; Nathani et al., 2025), as well as in scientific discovery (Majumder et al., 2025) and data science (Zhang et al., 2025a; Jing et al., 2025). The key limitation of this approach is its conflation of solution value with methodolog- ical novelty. It fails to distinguish between a genuinely novel method and the effective tuning of conventional one, as long as both achieve similar performance.
LLM Agents of Innovation. Beyond evaluating performance on well-defined tasks, a more am- bitious direction assesses the capacity of LLM agents to drive innovation on open-ended scientific problems. Initial efforts in this direction focus on the agent's role as an "idea generator". Exist- ing benchmarks (Ruan et al., 2025; Qiu et al., 2025) and agent systems (Lu et al., 2024; Su et al., 2025; Gottweis et al., 2025) are pivotal in formalizing the assessment of research ideation, but the downstream value of the generated ideas often remains speculative, they are not executed to solve concrete problems. Building on this demonstrated creative potential, a subsequent line of work has leveraged LLM agents "problem solvers". These systems translate abstract creativity into con- crete breakthroughs, achieving landmark, high-value results on long-standing scientific challenges. AlphaEvolve (Novikov et al., 2025), for example, provided superior solutions for matrix multiplica- tion and the problem of sphere packing.
CONCLUSION
We introduce InnoGym, a benchmark and framework for evaluating the innovation potential of AI agents. By combining performance gain and methodological novelty, InnoGym moves beyond correctness-only evaluation and provides a principled way to measure both effectiveness and cre- ativity. With 18 standardized tasks and unified execution environment, it enables reproducible, cross-domain comparisons. Experiments reveal that current agents often achieve novelty without robustness, highlighting a persistent gap between creativity and reliable performance.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This study was conducted in full compliance with established ethical standards and research best practices. All data employed are derived and synthesized exclusively from publicly available sources; proprietary confidential information was used. Every reference to these data sources is accurately and appropriately cited throughout the paper. We strongly encourage all users of our training dataset to uphold the highest ethical standards, ensuring faimess, transparency, and respon- sibility in their research. Any use of the dataset that could cause harm or negatively impact society strictly prohibited.
REPRODUCIBILITY STATEMENT
Due to OpenReview's file size limit, we open-source the benchmark, including the dataset and framework on GitHub at https: pttps://gtt //gittutl com/ 2 junlp/ /igm. We also detail the experi- ment settings and prompt in Section 4.1, Appendix E.I, H.2, and I.I.
ACKNOWLEDGEMENT
We would like to express sincere gratitude to the reviewers for their thoughtful and constructive feedback. This work supported by the National Natural Science Foundation of China (No. 62576307, No. NSFCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities (226-2023-00138)) the 2025 Zhejiang Provincial Center for Disease Control and Prevention Science and Technology Talent Incubation Project (No. 2025-A-04), the 2025 Zhejiang Health Informatics Association Scientific Research Program (Key Project, No. 2025XHZN titled "Research on Monitoring and Early Warning Methods of AI Large Model and Infectious Dis- ease Epidemic Data Fusion", undertaken by the Zhejiang Provincial Center for Disease Control and Prevention, Yongjiang Talent Introduction Programme (2021 A-166GG), and Information Technol- ogy Center and State Key Lab CAD&CG, Zhejiang University. This work was supported by Ant Group and Zhejiang University Ant Group Joint Laboratory of Knowledge Graph.
REFERENCES
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. Mle- bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: http///eereeeeeeeeeeeeeeeee net/ forum//ruummmm wtt.mmmmdddsssssmeemm
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder. Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code. CoRR, abs/2107.03374, 2021. URL https:tp/////..... /ttp:////iiii org/ abs/ /210 03374.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020. URL http: WRL:ttt://ppooooto /ttpp//ecccee...cc chen20j html.
ml wilr.press/vll h1tp///1//1//11111

---

Published as a conference paper at ICLR 2026
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. Scienceagent- bench: Toward rigorous assessment of language agents for data-driven scientific discovery. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https:://ppeeeeeeeee /wwp//eveeeeeeeeeeeeeeeeee net forum?id 624YKr0GK6.
Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, et al. Glyph: Scaling context windows via visual-text compression. arXiv preprint 200.iv2550..1.... 78000 2025.
Fan RK Chung, Michael R Garey, and David S Johnson. On packing two-dimensional bins. SIAM Journal on Algebraic Discrete Methods, 3(1):66-76, 1982.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL
https: //ar/////xxivv
org/ alss 22100111111.1 14168.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang. Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, 2. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025a. URL https: http///axivvv.////i///////..a../ org/ abs//551 12948.
DeepSeck-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu. Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang L.,, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li. Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao. Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J.
12

---

Published as a conference paper at ICLR 2026
Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Ximnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang. Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao. Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou. Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang. Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025b. URL https: / aaaxiv.v.v..////vv wwt.//ggg////oeoeeee/o//eeeeeeee 2412.4122222 19437.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang. and Mao Yang. Longrope: Extending LLM context window beyond million tokens. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:/ /ttp:///ppp/eeeeeeeeeeeeeee net forumiid- ONQtpXLqqw.
Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
Juraj Gottweis, Wei-Hung Weng, Alexander N. Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan, Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago R. D. Costa, José R. Penadés, Gary Peltz, Yunhan Xu, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan. Towards an AI co-scientist. CORR, abs/2502. 18864, 2025. doi: 10.48550/ARXIV25022 188644 URL ht https: //tp:///ooioo//oo
http:///oo.oggg//////// www. 48550/ arxiv 2502 18864.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Process- ing Systems 33: Anmual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL ht / httpp/ppoceennnnneeeeeeppee meurips.com 00 42b1 Aa .hmll.
paper/2020 hash/ #3
Abstract
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.00 01680, 2024.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9726 9735. Computer Vision Foundation / IEEE, 2020. doi: 10. N0.1099CP22220022020000 2020. 00975.
URL
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sii-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
13

---

Published as a conference paper at ICLR 2026
htmml.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents machine learning experimentation. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
httts://cpperevie
net forum?id=1Fs
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, An- drey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lu- garesi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang. Ed- mund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O' O'Comnell, Ian Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya. Openai ol system card. CoRR, abs/2412.16720 2024. doi: 10445500/ARXI.24122.000 https: /tw..diioo/// wtog///// 10. 48550/ arxiv 2ttp://.................. 16720.
URL
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
URL hwt.h:ttp///mmm http//ppnnevviiieeee net/ ittpr//reeeeeeeettoooooeeeed--ei
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration the space of code. 2025. URL https: //aarv/aa..iv.ai org/abs 2502. 13138.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httpp:://openree VIF8yNQM66.
net ffoummidd
Ligiang Jing, Zhehui Huang. Xiaoyang Wang. Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents from becoming data science experts? In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL / net/forunt 1d=DSsSP
LangChain AI. Langgraph. https: thub. wtt.//oo/o//a/ao GitHub repository.
2024.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Ticr Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: httpptt//pocceeeeiiiiiie http://pooeeeennn.eeee/ b4b758964 f1 808 8888888776668888 6e -abbfaabbbb-bbaattttee html.
www.eeeeddinn....eeeeeeeeeeeeeee nwwriiui.p.... ctc pecp/aperr220 saaper/2020/haihh
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery. CoRR, abs/2k0..02299999 066992 2024.
14

---

Published as a conference paper at ICLR 2026
doi: 10.48550/ARXI.2248800000 URL 06292.
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeets- ingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Dis- coverybench: Towards data-driven discovery with large language models. In The Thirteenth In- ternational Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: http///ppeeeeeeeeeeeeeeeeeeeeeee net/ corrmmiddvyy gpwffw.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part I, vol- ume 12346 Lecture Notes in Computer Science, pp. 405 Springer, 2020. doi: 10.1007/ 978- 3--030-5555--8- 24. URL https: /oii.iiooooooo org/ 10. 1007 97 030 58452-852222 24.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Trans. Graph., 4104): 2022. doi: 10. 10.1145732823333330222 URL https: /ooi.ii../////// 1t..11...11..... 1ttp///...11133122333333333
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob N. Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. Mlgym: A new framework and benchmark for advancing Al rescarch agents. CoRR, (tl//2502.14499, 2025. doi: 1048500/RXX.:552.44499
URL https:
/ /ww.///oo/o//o//////////oo// /11g//1000
Alexander Novikov, Ngån Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mchrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. Alphaevolve: A coding agent for scientific and algorithmic discovery. CoRR, 1a//22506611131,, 2025. doi: 10.48550ARRIV22506.1111 URL https://ddn /ttpa://ddoi.coo oog.looroooooo 48550/ arxiv 2506.13131
OpenAl. GPT-5 System Card. Technical report, OpenAI, August 2025a. URL https://
openai cwm/ indein///iiii -pp----ysstetteeeeeeeeeeepeeeeee
Last accessed: 2025-09-25.
OpenAl. Openai codex. ht http:::/ggihhuu..cmm/pppaaa/
2025b. GitHub repository.
Yansheng Qiu, Haoquan Zhang, Zhaopan Xu, Ming Li, Diping Song, Zheng Wang, and Kaipeng Zhang. Al idea bench 2025: Al research idea generation benchmark. CORR, abs/2504.14191, 2025. doi: 10:4855/ARXII.2354.111D URL https:/ httpe/oro//....///aaaa//. 2504. 14191.
Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu, and Hao Sun. Liveideabench: Eval- uating Ilms' divergent thinking for scientific idea generation with minimal context, 2025. URL https:/ fettss://axiii /aaaiiivv. org/ abs / 2. 17596.
Haoyang Su, Renqi Chen, Shixiang Tang, Zhenfei Yin. Xinzhe Zheng, Jinzhe Li, Biqing Qi, Qi Wu, Hui Li, Wanli Ouyang, Philip Torr Bowen Zhou, and Nanqing Dong. Many heads are better than one: Improved scientific idea generation by A llm-based multi-agent system. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 28201-28240, Association for Computa- tional Linguistics, 2025. URL https: /tc/aaclann laclannhoooooyyoooooooooooooolla org/ org/2025..0.00 2025 acl -ong. 1368/.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
15

---

Published as a conference paper at ICLR 2026
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, and et al. Openhands: An open platform for Al software developers as generalist agents. International Conference Learning Representations, ICIR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https::ppp.. wwtp//penneeeeeeeeee/ee/ee net www.eeeettoneeeeee forum? www.mmidddooooo wormmmdd--ddaayDD...
agentss In The Thirteenth
Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang. Yuan Liu, Thi- ago SFX Teixeira, Diyi Yang. Ke Wang. et Equibench: Benchmarking large language models' reasoning about program semantics via equivalence checking. In Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, pp. 33856-33869, 2025a.
Haoran Wei, Yaofeng Sun, and Yukun Li. Deepseek-oer: Contexts optical compression. arXiv preprint arXiv:2510. 18234, 2025b.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework, CoRR, abs/2308.08 08155, 2023. doi: (0.48550/ARXIV.2300. 08155. URL https::t/////..t. //dp:::///... 2rgg/1..0. 48550/ a 2308. 08155.
Yunze Wu, Dayuan Fu, Weiye Si, Zhen Huang, Mohan Jiang, Keyu Li, Shijie Xia, Jie Sun, Tianze Xu, Xiangkun Hu, et al. Innovatorbench: Evaluating agents' ability to conduct innovative 1lm research. arXiv preprint arXiv:2510.2753 2025.
An Yang. Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abs/2505509938, 2025. doi: 10445550/RRXII.5555..o 088.888 URL https:///of
wrrg////000 0ttgg//..85000000 48550/arXX5. 2505. 093888
Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, and Yisong Yue. Datascibench: An LLM agent benchmark for data science. CoRR, abs/2502. 13897, 2025a. doi: 10.48550/ARXIV.5502.2 138977 URL https::/t/// /ww.i/ooii.o//////// /10. 48550/arXis 25.2..5222200 13897.
Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan, Grant D. Murphy, Lajanugen Lo- geswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Mire-bench: Can language
agents solve machine learning research challenges? CoRR, abs/2504.09702, 2025b. doi: 10.
0tt.44555//.t11..50..999000. 48550/ARXIV.25000000 URL https: http://oi..oooo //t.://tp/////oooooo//0//44/////
A USAGE OF LLMS
Throughout the preparation of this manuscript, we used L.LMs to assist with improving grammar, clarity, and wording in parts of this work. The use of LLMS was limited to language refinement, with all ideas, analyses, and conclusions solely developed by the authors.
B LIMITATIONS
While InnoGym provides a principled framework for evaluating innovation AI agents, it has several limitations. First, the benchmark currently focuses only Improvable Tasks with clear evaluation pipelines; solved problems and open-ended exploratory problems are excluded, which narrows the scope of applicability. Second, metrics for performance gain and novelty, though principled, may not capture all dimensions of innovation such as efficiency, interpretability. or long- term impact. Third, novelty is estimated relative to a finite set of known solutions, which may bias
16

---

Published as a conference paper at ICLR 2026
evaluations when prior coverage is limited. Finally, iGym emphasizes reproducibility and robustness but is constrained by computational resources, preventing the inclusion of extremely large-scale tasks. These limitations suggest important directions for extending InnoGym in future work.
C IGYM
Existing SDKs (OpenHands (Wang al., 2025), AutoGen (Wu et al., 2023). Lang- Graph (LangChain AI, 2024)) simplify agent orchestration and tool use, but cannot rewrite different agent systems under unified framework, nor do they support long-horizon recovery, resource man- agement, or professional tool integration. iGym addresses these needs with new SDK that supports diverse system designs, recovery, and concurrency.
Motivation.
Architecture (Fig. 4). iGym consists of two parts:
Environment: A set of tools and resources accessible via a redesigned asynchronous Tool Dispatcher, supporting thread-pool process-pool execution. Agents launch long- running tasks in parallel with others, monitor progress, and receive real-time results.
Agent System: collection of agents and memory, interacting with the environment via Action + Observation loop. We support both (I) workflow mode (LLM as function), and (2) agent mode (multiple agents with a scheduler, analogous to CPU clock scheduling).
Key Features.
Recovery: Workflow mode replays recorded I.LM/tool calls; agent mode resumes directly from a saved state.
Concurrency: Native support for parallel tool calls and dependency scheduling.
System Compatibility: A unified abstraction layer allows fair comparison across different agent system designs.
D TASK
The information for the 18 tasks is shown Table 3. For each task T, we collect a set known
solutions Skmowa Am1...., To characterize how diverse these reference solutions on each task, we report a diversity statistic Div. Let DAGENT(-,-) denote the method-level distance between solutions introduced in Section 2 and implemented in Appendix F.I. We define
hm
Div(T) = m(m 1)
Σ Isicism
DAGENT (h4, hy),
ie., the average pairwise distance among all reference solutions for task T (Rao's quadratic diversity with uniform weights). Because DACENT E [o, 100), the diversity score Div(T) also lies in [0. 100]. Larger values indicate that the known solutions for that task occupy wider region of method space, making it comparatively harder for new solutions to achieve both high performance and high novelty. When only single reference solution is available (m 1), Div(T) is undefined, and therefore leave the Div. column blank in Table 3.
D.1 DETAIL TAXONOMY OF INNOVATIVE TASKS
This appendix gives a more explicit account of the task taxonomy introduced in Section 2, and em- phasizes that all categories are defined per concrete task instance and relative specific, human- defined goal.
Consider task formalized as T (P,S, V, with problem specification P, solution space S, value function V, and distance function D. Let C 5 + {0, 13 be the feasibility validator, and let Sknown C S denote the set of known reference solutions (typically human desannde previously published methods) for this particular instance. With this notation, the three categories in Section can be restated succinctly.
17

---

Published as a conference paper at ICLR 2026
Number Ref. Sol.
ID
Task
Title
Source
DNv.
BEETLAM))
BEETL 2021 (Motor imagery)
NeurIPS 2021 Compesition
66.11
+2
BEFTT (Steep)
BEFTT. 2021 (Sleep Staging]
NewriPS Compesition
71.67
+3
Belka
Predict New Medicines with BELKA
NearIPS 2024 Compesition
67.08
F4
CirclePacking
Cirele Packing in Unit Square
Math / Computationa Goomeity
4
69.10
85
CDML
Demain Meta- Learnnn Chalkengo
NearIP5 2022 Compettton
F6
NPR
Maitiliagual Recommendation Task Next Product
KDD Cup 2023
3
59.72
+7
OAG
OAGi-Chlleeng
KDD Cup 2024
3
70.14
+8
PTTALC
Penception- Temporal Action Localisation (Task
ICCV 2025
2
25.00
RCIC
Rocussion Collular Inagee Classilication
NearlPS 2019 Compesttio
63.20
F10
TrojanDeuction
The Trojan Detection Challenge (LLM Edition)
NewrPP 2023 Competition
3 54,52
Readef2018
ROADEFEURR 2018
112
Roadef2020
ROADEMEURR 2020
+13
Rondef2022
ROADEFFELRC 2022
r14
GMCM 2022 (B Q1)
China Graduate: Maihh Modeling 2022
2
53.21
F15
GMCM
Q2)
Gradnate Math Modeling 2022
+16
CompilerCiym
CompilerCiym Benchmark /SCC 2022 related
ISCA 2022 Open-source
59.63
F17
2D Bin Packing
2D Bin Packing Problem
Classic Combinatorial Opsimization
6
43,32
Gmmph Coloring
Gimph Coloring Problem
Classic Graph Theory
57.77
Table 3: Tasks in InnoGym, with source benchmark, number of reference solutions |Sknown]: and diversity score Div. (average pairwise method-level distance under D Dagent in ([0,100]))
Solved Problem. A task instance is called solved if there exists at least one known solu- tion that is both feasible and optimal with respect to the specified validator and objective. Formally, this means that there exists E Sknown such that C(s) 1 and V(s) V*. Intuitively, the explicit goal encoded by C' and V has already been fully achieved by some known solution.
Improvable Problem. A task instance is called improvable if at least one known solution passes the validator but none of the known solutions attain the optimal value. Equivalently, there exists an Sknown with C(s) 1 and (ss) V*. In this case, feasibility has been demonstrated, but there is remaining headroom in the explicit objective.
Exploratory Problem. A task instance is called exploratory if none of the known solutions pass the validator, that is, if C(s) = O holds for all Sknown- Here even feasibility has not yet been established; the immediate challenge is to discover any valid solution at all.
It important that these categories relative the chosen validator C and value function V rather than to some universal notion of difficulty or optimality. The same underlying real-world problem can fall into different categories if the goal specification changes.
concrete example is SWE-Bench (Jimenez et al., 2024). Under the standard formulation, the objective is produce patch that passes the provided test suite, and the validator checks that the patch applies cleanly and the tests run successfully. Let Skown E SSoown denote the human reference patch. Since Sknown passes all tests, satisfies C' (((knoo))) defined by the test suite. Under this specification, typical SWE- -WE-Bench instance is therefore a solved task in the sense above: the stated goal has already been fully met by a known solution.
1 and achieves the maximal value V*
This does not mean that no better implementation exists in an absolute sense. For example, one could change the objective to "fix the bug with minimal edits while still passing all tests" or to "maximize robustness across an extended test suite". Such change would alter V and V*, and the same instance might then become improvable, since the existing patch could be feasible but suboptimal under the new criterion.
The taxonomy in Section 2 thus deliberately operational: each task instance is classified as solved, improvable, exploratory with respect to the explicit, human-defined goal encoded by (P,C, V) and the current set Skoown, rather than with respect to an abstract, task aask-agnostic notion of optimality.
E ADDITIONAL EXPERIMENT RESULTS
E. SETUP
Agent scaffolds. These three scaffolds span complementary design choices for ML engineering agents. MLAB is the ResearchAgent from MIAggenBBnnh (Huang al., 2024), ReAct-ttyle
18

---

Published as a conference paper at ICLR 2026
Metric
MLAB
CODEACT
AIDE
R (higher is better)
-0.62
0.82,
00.2
0.81
0.98,
0.59
0.82
0.99, -0.60]
N (higher better)
39.58
[22.08, 56.25
32.92
14.58, 51.25
23.33
(7.50, 40.83)
Table 4: Task-level maero-averages and 95% bootstrap confidence intervals for normalized ratio R and novelty N under pessimistic imputation (R 11, N for failure cases). Means are computed over the 10 main iBench tasks, treating tasks as independent observations.
workflow agent that plans in natural language, issues high-level actions such reading and edit- ing files, executing training scripts, and inspecting logs, and iteratively refines an ML pipeline in a Kaggle-like workspace. CODEACT (Wang et al., 2025) instead unifies all agent actions into executable Python code: the agent generates short programs that directly call libraries, run shell commands, and perform self-debugging through repeated code execution, which has been shown to significantly improve success rates across tool-using benchmarks and underlies the Code gent implementation used in ML.E-Bench (Chan et al., 2025). Finally, AIDE (Jiang et al., 2025) is tree-search-basee ML engineering agent that views the task code optimization: starting from an initial solution, it repeatedly proposes code edits, runs training and evaluation, and branches on promising variants, reusing and refining strong configurations trade compute for performance and achieve state-of- satte-of-th--asd theeharrt results on Kaggle competitions. Together, MLAB, CODEACT, and AIDE cover workflow-styl planning, code-centric action spaces, and search-based exploration, providing a diverse set of agent scaffolds for evaluating innovation on InnoGym.
E.2 STATISTICAL ANALYSIS FOR TABLE 2
In Table 2, "Y" entry indicates that all runs for given task. agent failed to produce valid submis- sion within the budget. The main analysis computes macro-averages over tasks on which each agent has at least one valid solution. To explicitly incorporate failure cases ("f" entries) and to perform hypothesis tests on our key metries, we adopt the following pessimistic imputation scheme.
For any task, an agent with no valid submission, we assign the minimum possible normalized ratio R = -1 and a novelty score N = O. Intuitively, agent that never returns a valid solution to a task contributes neither usable performance nor innovative methodology to that task. Under this encoding, each agent framework obtains ratio and novelty score every one of the 10 main tasks, and we treat tasks as independent units for statistical analysis.
Macro-averages and confidence intervals. For each framework f and metric M (R, N). compute the lesel macro- aaerreeverage
T
Mf
and estimate uncertainty via non paramarratticc bootstrap over tasks (B = 10,000 resamples). The 95% confidence intervals are obtained from the empirical 2.5th and 97.5th percentiles of the bootstrap distribution of M.M. The resulting averages and confidence intervals are reported in Table 4.
Even under this pessimistic treatment of failure cases, all three agent frameworks remain far from the best-known solutions average (ratios close -1), with MLAB attaining the best normalized ratio and the highest average novelty.
Paired tests across frameworks. To compare frameworks on R and N, we treat tasks as paired observations. For each metric M E (R, N) and framework pair (J, we define the task-wise difference dM M, Mp and estimate the mean difference
1
M f
d
T
together with 95% confidence interval and an approximate two-sided bootstrap p-value. Specif- ically, perform non-parametric bootstrap resampling over tasks (B 10,000 resamples) and
19

---

Published as a conference paper at ICLR 2026
Metric
Pair
A
95% CI
Pooot
R
MLAB & CODEACT
+0.20
0.011 0.40
0.035
MLAB & AIDE
+0.20
0.05, 0.37
0.007
CODEACT & AIDE
+0.01
1-0.06, 0.05)
0.785
MLAB & CODEACT
+6.67
18.75, 30.42]
0.575
MLAB & AIDE
+16.25
-5.00, 37.08
0.133
N
CODEACT & AIDE
+9.58
100.00, 29.17]
0.333
Table 5: Paired bootstrap tests over tasks under pessimistic imputation (R -1, N for failures). A denotes the macro-ane averree difference between the first and second framework (f f)). We report 95% bootstrap confidence intervals and approximate two-sided bootstrap p-values over the 10 main tasks.
compute A f.f on each resample; the 2.5th/97.5tt percentiles of this bootstrap distribution give the confidence interval, and the bootstrap p-value is obtained as twice the smaller of the fractions of bootstrap means above and below zero. Results are summarized in Table 5.
Under this pessimistic encoding, MLAB consistently achieves higher macro- avcrrea ratio R than both CODEACT and AIDE; the improvement statistically significant in both comparisons accord- ing to the bootstrap test. For novelty N, MLAB also has the highest mean, but differences between frameworks are not statistically significant at the 0.05 level given the small number of tasks. We therefore interpret the N results descriptive trends, and use R. as the primary performance metric for formal cross-framework comparisons.
E.3 OTHER ANALYSIS
Task
MLAB
CODEACT
AIDE
BEETL (MI)
100%
0.00%
0.00%
BEETL (Sleep)
100%
0.00%
100%
Belka
100%
100%
100%
CirclePacking
100%
100%
100%
CDML
0.00%
0.00%
0.00%
NPR
33.33%
33.33%
0.00%
OAG
100%
100%
66.66%
PTTALC
0.00%
0.00%
0.00%
RCIC
0.00%
33.33%
33.33%
TrojanDeeecction
33.33%
33.33%
0.00%
Table 6: Submission success rates of different agents across 10 tasks over runs.
Submission success rates. We report the submission success rates across three runs in Table 6. Notably, in 2 out of the 10 sampled tasks (CDML. and PTTALC), no agent was able to produce a valid submission. Combined with the performance gaps shown in Table these results confirm that our benchmark is distinctly future-oriented. Unlike prior tasks that are often solved in minutes, these high-value challenges- derived from real-world scientific and engineering competitions-requine continuous iteration and runtimes spanning tens of hours. We believe these are precisely the com- plex, long-horizon problems that the next generation LLMs and agents must master, and we invite the community to tackle these rigorous standards to drive the next leap in machine intelligence.
Encourage Novelty Explicitly. We further investigated the effect of explicitly prompting (see Fig. 7) the AIDE agent to prioritize innovation on three tasks, with results reported Table While this strategy successfully improved the Novelty scores, significantly in the case of CirclePacking, it consistently resulted in a decline Performance Gain. This demonstrates that exploratory behavior imposes cost agent performance. Consequently, conclude that the pursuit of methodological novelty must not come at the expense of solution correctness, and future agents must learn to balance creativity with effectiveness.
20

---

Published as a conference paper at ICLR 2026
Gain (Ratio)
Novelty
Task
Baseline
Innovative
Baseline
Innovative
BETTL(sleep)
-53.62 (-0.77)
-54.56 (-0.79)
54.17
58.33
OAG
-29.87 (-0.36)
-31.76 (-0.38)
70.83
50.00
CirclePacking
-0.25 (-0.09)
-0.37 (-0.14)
35.33
58.33
Table 7: Performance comparison of agent AIDE (Jiang et al., 2025) behavior with and without innovation prompts across three tasks. Baseline refers to standard prompting, while Innovative indicates prompts explicitly encouraging creative solutions.
F VALIDATING THE DISTANCE FUNCTION D AND NOVELTY METRIC N
In the main text, we formalize each task as T (P,S, S, V, D), where D(s1, 82) measures the distance between solutions 81 and 52. The novelty of candidate solution is then defined in terms of its distance to the set of known solutions Skmowa (Eq. 3). Intuitively, solution is more novel if, in terms of "how solves the problem," it is far from previously observed solutions.
Here, we instantiate D with an Agent- based pipeline, which we denote by DAGENT. The pipeline has two stages: an extraction step that summarizes each solution into standard representation, and (ii) a comparison step that scores method-level differences along a small set of dimensions. Both stages are implemented by prompting an agent named Codex (OpenAI, 2025b). In this section, first recap the pipeline, then describe triplet-based protocol to validate D DGGETT against human judgments, and finally present two experiments: one on code-level equivalents (EquiBench (Wei et al., 2025a)), and one human-collected method triplets across different AI subfields.
F.1 RECAP: NOVELTY EVALUATION PIPELINE
Extraction. For each solution s (including both historical leaderboard entries and new agent sub- missions), first extraction prompt (Appx. L.1 over the entire solution repository. The codex agent produces two standardized artifacts:
summary.. md: A structured Markdown file clearly describing the solution's core ideas, data processing pipeline, and model architecture in natural language. pseudocode tex: A LTT forpa-fomattedd pseudocode file, outlining the solution's 's logic and key steps in an algorithmic format.
These two files serve as a normalized, human-readable representation of the solution, stripping away incidental details such as file layout or naming conventions.
Comparison. Given two solutions and $2, along with their corresponding s summary md and pseudocode tex, we then run second comparison prompt (Appx. H.2). The prompt asks the GPT-5 to act as reviewer and assess how different the two solutions are along fixed set of method dimensions For each dimension E K,, the agent assigns discrete score
da(s1,82) E 1,, 4),
where 0 means "essentially the same" and means "completely different paradigm" for that particu- lar aspect. We then aggregate these per-dimension scores into single distance. First, normalize each d₂ to the range (0, 1] by dividing by 4, then we average over all dimensions, and finally rescale 0-100 scale:
d4 s2) 100.
DACENT 52)
kE
In all experiments in the main paper, we use DAGENT as the concrete instantiation of D when com- puting novelty scores.
F.2 TRIPLET- BASED VALIDATION PROTOCOL
Having defined D DAGENT, we now ask whether it agrees with human intuition about method similarity and novelty. To answer this, design simple triplet-based protocol.
21

---

Published as a conference paper at ICLR 2026
DAGENT B)
DAGENT C)
Mean over 50 triplets
1.00
9.75
Table 8: EquiBench results over 50 triplets. Mean DAGENT scores on 0-100 scale.
ID
D DAGENT DAGENT(A,C)
DAGENT [A, B)
Human(A,,C)
Human(A,B)
Case
29.17
4.17
33.33
0.00
Case
29.17
0.00
33.33
0.00
Case
0.00
4.17
12.50
0.00
Case
54.17
0.00
29.16
0.00
Case
0.00
0.00
0.00
0.00
Case
0.00
0.00
0.00
0,00
Case 0.00
0.00
8.33
0.00
Case 8
12.50
0.00
16.66
0.00
Average 15.63
1.04
16.66
0.00
Table Comparison of DAGENT scores against human judgments on the sub-sampled EquiBench triplets. Both agent and humans the distance from base solution (A) algorithmic variant (C) and a superficial variant (B). Scores are 0-100. See Table 10 for the correlation scores between human and agent judgments.
Each evaluation instance is triplet (A, B, C), where A is a base solution (reference method for given task); and C' are two alternative solutions to the same task. In our settings, one can think of A as an existing known solution, and B and C' as two When we construct triplets for validation, we explicitly choose and C' so that, within each triplet, have clear expectation B is relatively closer to A in terms of method; C' is relatively farther from A, and therefore more novel. This gives us ground-truth notion of "who should be more novel relative to A" for each triplet.
solutions proposed by different agents.
For each triplet, we compute D DAGENT A, B) and DAGENT (A,C). C). We also collect human judgments: 0-100 scale based standard shown in
annotators how different B and C are from Appx. H.2, producing Human(AA A, B) and Human C). Higher scores mean that the solution is viewed as more methodologically distant and therefore more novel.
We compare the agent against humans in two ways. First, we examine score-level correlation, computing Pearson and Spearman correlations between agent and human scores, separately for the (A, B) and (A, C') pairs. Second, we look at triplet-level agreement: for a given triplet, we the agent and the human agree they both judge B be more novel, both judge C' to more novel, or both judge them to be roughly tied. The agreement rate is the fraction of triplets where this holds.
We apply this protocol to two datasets: code-level equivalents from EquiBench (Wei et al., 2025a), and human-construuted method triplets from three Al subfields.
F.3 EXPERIMENT 1: EQUIBENCH CODE-LEVEL SANITY CHECK
Data Collection. EquiBench (Wei et al., 2025a) groups functionally equivalent programs into sev- eral categories. We focus on two:
OJ A: functionally equivalent solutions that use different algorithms or implementations:
OJ purely superficial variants with identical logic, such variable renaming.
This gives straightforward sanity check: a reasonable method distance should assign trivial distance to OJ_A pairs, but a near-zero distance to OJ_V pairs.
We randomly sample 50 triplets of the form A base solution, B OJ.V variant, C' OJ A variant, and compute D DAGENT(A, (A, B) and (AG for each. Table reports the average agent scores all 50 triplets.
22

---

Published as a conference paper at ICLR 2026
Dataset
Triplets
Peurson y (A,B)
Pearson (A,C)
Spearmon p (A,B)
Spearman p (A,C)
Triplet agreement
EquiBench (ammot.)
Human triplets
1.00
0.99
1.00
1.00
3/3 100%
Table 10: Correlation between AGGNT and human judgments. For EquiBench A pairs, all human scores zero, correlations are not defined.
Domain
Reference
Within-paradigm B
Cross-paradigm C
SimCLR
MoCo
BYOL (Grill et al., 2020)
Self-supervised vision
(Chen et al., 2020)
(Ie et al., 2020)
NeRF (Mildenhall al., 2020)
NSVF (Liu al., 2020)
Instant-NGP (Müller al., 2022)
Neural radiance fields
Glyph
DeepSeck-OCR
LongRoPE (Ding al., 2024)
Long-context modeling
(Cheng et al., 2025)
(Wei et al., 2025b)
Table 11: Human-collected method triplets across three domains. In each triplet, A and B are con- current methods within the same paradigm, while C7 addresses the same problem with a qualitatively different modeling approach.
For human evaluation, we sub-sample 8 of the 50 triplets and ask three Computer Science graduate students with strong programming backgrounds to score each (A, B) and (A, C') pair on the 0-100 scale, based on standard at Appx. H.2. We average their scores per pair. Table 9 shows the detailed numbers. The alignment between human and agent judgments, quantified by correlation, is detailed in Table 10.
Results and takeaway. Over 50 triplets, OJ variants are on average substantially farther from the base solution than OJ_V variants (9.75 vs. 1.00), which is exactly what we want. On the 8 human- annotated triplets, the agent and humans give very similar mean scores; around 16 for (A, C*), and essentially 0 for (A, B). Correlations are high for the A, C' pairs, and the agent matches human pref- erences (which of B or more novel) on 6 out of triplets. In short, DAGENT ignores superficial edits and reacts to real algorithmic changes in much the same way human programmers.
F.4 EXPERIMENT 2: HUMAN-COLLECTED METHOD TRIPLETS
Data Collection. We next look higher-level methodological differences. We construct three triplets from three Al subfields. In each triplet, A is a reference method, B is a within-paradigm method, and C a cross-paradigm method that is widely viewed as more novel. We summarize the collected triplets in Table 11.
For each domain, we recruit one PhD student working that subfield (and with reviewing expe- rience) to rate (A, B) and (A, C*) on the same 0-100 scale as before based on Appx. H.2. We also compute DAGENT(A (A, B) and DACENT A, C') using the same pipeline. Table 12 summarizes the scores. The alignment between human and agent judgments, quantified by correlation, is detailed in Table 10.
Results and takeaway. Across the three domains, both the agent and the human experts consis- tently judge C' to be more novel than B relative to A, and the average scores are close in magnitude. Despite the tiny sample size, the score-level correlations are essentially perfect as shown in Ta- ble 10, and the triplet-level agreement rate is 3/3. This suggests that DAGENT is sensitive not only to code-level changes, but also to the kind of paradigm shifts that researchers care about.
F.5 OVERALL SUMMARY
Putting the two experiments together, we see consistent picture. On EquiBench, DAGENT treats purely cosmetic variants essentially pero-distance while assigning noticeably larger distances to algorithmically different solutions, and its preferences align well with those of human programmers. On cross-domain method triplets, it agrees with domain experts which methods are more novel
23

---

Published as a conference paper at ICLR 2026
Triplet (Domain)
D DAGENT(A, B)
DAGENT (A, C)
Human(A,B
Human (A,C)
Self-supervised vision
29.16
45.80
33.33
54.17
Neural radiance fields
50.00
54.17
41.67
66.67
Long-conteer modeling
66.00
95.83
50.00
100.00
Average
48.39
65.27
41.67
73.61
Table 12: Human-collected method triplets. Each row corresponds to one subfield; scores are on 0-100 scale. See Table for the correlation scores between human and agent judgments.
and produces scores on roughly the same scale. In practice, this means that DAGENT is doing what we want: measures method-level differences rather than surface edits, and its notion of novelty tracks human intuition both at the code level and at the level of high-level method design.
G DETAILS OF BENCHMARK CONSTRUCTION
G.1 SOLUTION COLLECTION
To construct relatively comprehensive and high-quality set of known solutions (Sknwwn) for the 18 tasks in iBench, we executed systematic, multi-stage collection, filtering, and post-processing pipeline that ensures the solutions used to calculate Novelty are robust and representative.
Sources of Candidate Solutions. Our collection strategy was primarily divided into two cate- gories based task type:
Classical NP-hard Problems: For classical combinatorial optimization or mathematical problems (e.g., #17 2D Bin Packing and #18 Graph Coloring in Table 3), we first con- sulted authoritative operations research and algorithm design textbooks, as well aca- demic surveys. This allowed to identify standard methods, classical heuristics (e.g.. greedy algorithms, simulated annealing), and exact algorithms in the field.
Subsequently, we searched for prominent open-source implementations of these classical methods, particularly those widely cited in academia or implemented in standard libraries, and added them to our candidate set.
For tasks originating from academic or industry competitions such NeurIPS, KDD Cup, and ROADEF, we collected solutions through three main channels:
Specific Competition Tasks:
(i) Academic Literature: For tasks from academic venues (e.g., NeurIPS Competitions), used tools like Google Scholar to find methodology papers that cited the orig- paper. We prioritized papers that detailed their methodol- ogy and provided public source code.
inal competition-orranize
(ii) Public Leaderboards and Code Repositories: We meticulously reviewed the of- ficial leaderboards for each competition. We focused collecting high-ranking entries where the authors publicly shared their full solutions (e.g., via GitHub repositories Kaggle Notebooks).
(iii) -As-iisitted Search: During the search process, we utilized ChatGPT to assist in generating diverse search keywords (e.g., alternative task names, related algorithm families) and to help quickly summarize technical blogs and forum posts to discover additional potential candidates.
Validation and Filtering Process. To ensure the comprehensiveness and quality of the Sknowa set, we employed a rigorous validation process:
Independent Search and Merging: For each task, we assigned three team members to independently conduct the collection process described above. This cross-validation ap- proach was designed to maximize coverage of solutions from different sources and ensure the comprehensiveness of the candidate set.
24

---

Published as a conference paper at ICLR 2026
2. Reproducibility Filtering:
All collected candidate solutions tested within the stan- dardized environment provided by iGym. We executed all relevant solutions and retained only those that met both of the following conditions:
(a) Executability: The code had to compile and run successfully without substantial modification.
(b) Performance Consistency: The reproduced performance score had to meet or closely approach the level reported in the original paper on the leaderboard.
3. Final Set: Through this strict filtering process, we removed entries that were inoperable or whose performance was reproducible. The final number of available, validated solu- tions for each task is presented in Table 3.
Structural Extraction of Solutions. To perform systematic novelty analysis the filtered Skoown set, we needed to abstract each solution from its code implementation to its core methodology.
As described in Section 3.2 of the main paper, for each retained solution, used Codex with carefully designed Extraction Prompt (see Appendix I,1) to automatically "distill" its core strategy.
This process generated two standardized output files for each solution in Skmown
summary.. md: A structured Markdown file clearly describing the solution's core ideas, data processing pipeline, and model architecture in natural language.
pseudocode tex: A L'TpX-ffrmatted pseudocode file, outlining the solution's logic and key steps in an algorithmic format.
These two structured representations collectively form the baseline database used for Novelty Eval- uation.
G.2 EVALUATOR NORMALIZATION
To ensure fair, consistent, and reliable benchmarking across 18 diverse tasks, implemented rigorous three-part normalization process. This process guarantees that every evaluator adheres to our standards of Absoluteness, Executability, and Correctness.
G.2.1 A ABSOLUTENESS
Rationale:
To meaningfully quantify Performance Gain (G, see equation 2), framework requires an absolute scoring metric. Relative scores, such as rankings, are insufficient as they cannot measure the magnitude of agent's improvement over the state-of-the-art.
Problem: Several of our collected tasks, most notably the three ROADEF challenges (Roadef2018. Roadef2020, Roadef2022), used rank-based scoring their official leaderboards.
Solution: We developed procedure to convert these rank-based systems into absolute scales.
1. We first collected all available scores from the public leaderboards to identify the best- known (highest) and worst-known (lowest) performance scores.
2. These maximum and minimum values were then fixed as static hyperparameters for our new evaluation function.
3. We applied a logarithmic normalization function to transform the raw scores onto a con- sistent, absolute scale. This allows any new, valid solution to receive an absolute score, making it directly comparable to existing solutions.
To ensure new absolute metric preserved the qualitative integrity of the original leaderboards, we validated its consistency with the original rankings. We calculated the correlation between normalized scores and the original ranks using Spearman's 's rank correlation (p) and Kendall's rank correlation (T,). As shown in Table 13, the high correlation values confirm that our absolute scores strongly maintain the original relative ordering of solutions.
Validation:
25

---

Published as a conference paper at ICLR 2026
Table 13: Correlation between normalized absolute scores and the original ROADEF leader- board rankings.
Task
Spearman's
Kendall's Ta
Roadef2018
0.960
0.877
Roadef2020
0.859 0.960
Roadef2022
0.982
0.924
G.2.2 EXECUTABILITY
Rationale:
Our iGym environment requires unified interface to trigger any task's evaluation from a wython-based workflow, regardless of the evaluator's original implementation.
The evaluators we collected were highly heterogeneous. Some were only described in documentation (requiring us to implement them), while others were provided as binaries or source code different languages (Java, C, C++). Many of these had strict and often conflicting environ- ment dependencies.
Problem:
Solution: We standardized all evaluators by containerizing the non-Python components.
1. For every evaluator that was not simple Python script, built a Docker container. This container encapsulated all necessary dependencies, such as specific Java Runtime Envi- ronment C compiler.
2. We then created lightweight Python wrappers that use a subprocess to call the executable within the container.
This abstraction allows the iGym framework treat every evaluator identically: simple Python function call that takes a submission file and configuration as input, and returns a score.
G.2.3 CORRECTNESS
The evaluator's correctness is paramount, A faulty evaluator could reward invalid solu- tions penalize valid ones, rendering the benchmark useless.
Rationale:
Problem:
We needed to verify the correctness of all 18 evaluators, especially those we implemented ourselves from descriptions.
Solution: We employed a multi-pronged validation strategy:
Known Solution Verification: For tasks with known solutions (i.e., Skoown and reported scores, we executed these solutions in our environment. We verified that our evaluator produced a score that was identical or (in the case of stochastic algorithms) statistically very close to the one reported on the official leaderboard.
Baseline Sanity Checks: For non-code submission tasks (e.g.. classification), we gener- ated trivial random submissions. For example, we would create submission file that predicted the same label for every instance. We then verified that this submission produced valid and appropriately lower score.
Monotonicity Check: We compared the scores from the baseline submissions against the scores from known high-performing solutions. This was a simple but critical check to ensure that our evaluators correctly scored better solutions higher than trivial ones.
If an evaluator failed any of these checks, it was flagged for review. We iteratively debugged and refined the evaluator's logic (or its containerized environment) until it successfully passed all cor- rectness tests.
G.3 VALIDATOR CONSTRUCTION
For each task, we implement a dedicated validator to process the agent's submission before it is passed to the evaluator. The primary purpose of this validator is filter out ill-formed invalid
26

---

Published as a conference paper at ICLR 2026
submissions failure mode frequently encountered in practice. We support two types of submis- sions:
Code submissions. In this mode, the agent submits a code file that is required implement prescribed interface. The validator first checks that the expected entrypoint function is present and that its inputs and outputs conform the task specification (for example, the argument list and return type). It then executes the function on small, fixed test input to verify that the code runs without errors and produces a result of the correct type. If any of these checks fail, such missing entry point, runtime exception, return type, the submission is rejected and the validator returns an error instead of a score.
mismatched
2. Answer-file file submissions. In this mode, the agent submits a structured output file, typically in CSV or JSON format. The validator enforces that the file type matches the required format and that its schema (for example, column names and field structure) agrees with the task specification. It also checks basic constraints individual fields, such required presence, allowed value ranges, or discrete label sets. Submissions that violate any of these structural value-level constraints are rejected.
All validators are implemented purely procedural code, invoking no language models or other stochastic components. This design renders the validation process fully deterministic: a given sub- mission will always produce the same validation outcome. This simplifies debugging and ensures that evaluation results are precisely reproducible.
H PROMPT
We designed two structured prompts to systematically analyze and compare solutions: one for methodology extraction and one for solution comparison. Their purposes briefly described be- low.
H.1 EXTRACTION PROMPT
This prompt instructs the model to act as a senior ML engineer and technical writer, reading entire code repository (and any accompanying paper) to reconstruct the high-level solution method- ology. focuses on problem context, modeling choices, data flow, training strategy, and evaluation protocol, while omitting low-level implementation details. The output is structured markdown summary (summary.md) and a LaTee-stylle pseudocode (pseudocode.tex) providing a consistent and human-readable description of the solution for downstream analysis.
H.2 COMPARISON PROMPT
This prompt evaluates the similarity between two solutions-ad Agent Solution and Base- line tine-across six dimensions (e.g., problem framing, methodology, architecture, experiment design). For each dimension, the model assigns score from 0 (completely similar) to 4 (completely differ- ent) with brief justification. The resulting JSON object enables quantitative, reproducible compar- ison of solution approaches.
I DISCUSSION
I.1 PROPERTIES OF INNOVATION
We posit that innovation exhibits Contextual Relativity and Temporal Dynamicity.
Contextual Relativity. Innovation is not an absolute scalar but a relative metric contingent on the specific task. First, the innovation of solution is strictly defined with respect to the task formu- lation T. A solution may be innovative for task Ta yet trivial for Tg- This distinction is particularly relevant when tasks share the same problem space P but differ in their value functions V. For in- stance, a solution that maintains parity in accuracy but significantly reduces resource consumption becomes innovative only if the task definition Tn explicitly incorporates computational cost into
27

---

Published as a conference paper at ICLR 2026
V, whereas it might be deemed trivial under formulation Ta that values accuracy alone. Second, innovation is measured relative to the reference set known solutions, Sxnown Since different tasks have distinct baselines, comprchensive Sknwwn is essential for precisely estimating the novelty score (s)) and performance gain G()), thereby quantifying how significantly a candidate deviates from the status quo. Finally, the threshold for what constitutes "non-trivial" improvement is also rel- ative. In mature Nomains--uch as the Circle Packing problem, where optimization has converged near the theoretical limit, even marginal performance gains minor methodological refinements sufficient to constitute valid innovation.
Temporal Dynamicity. Innovation is inherently time-variant; method considered standard prac- tice today was likely novel at its inception. We model this evolution by explicitly updating the Sknown over time: s(f) ((++1) Once solution is deemed innovative time and accepted, it is assimilated into the baseline at + 1 (i.e., known (t)
known U {s))). This monotonic update implies that the measured novelty of future solutions similar to s immediately diminishes. Consequently, framework captures the natural lifecycle of task transitioning from an exploratory phase (no feasible solutions), to an improvable stage (room for optimization), and eventually to a solved state, modeling innovation as the continuous movement of the frontier rather than static classification.
28

---

Published as a conference paper at ICLR 2026
Extraction Prompt
Act as a senior MI. engineer and technical writer. You are given two input paths: background description of the research problem competi-
1. Essearch_problent
tion task (context only).
Path: reseaach..rrobeem. txt
code project - the actual implementation that aims to solve this problem (may also include associated paper. if provided).
Path: so volute ut i on condode/+
Your task to read and understand the entire code repository. file by file, in order to reconstruct the solution methodology comprehensively. Use research research.problem.. only as context, but focus primarily on the code project's solution design (and paper, if present).
You must produce two outputs:
1. A summary.md file (structured textual summary).
2. A pseudocode.tex file (LaTeX pseudocode).
Requirements:
Do not include low-level implementation details (file paths, function names, variable names).
Do not add subjective evaluation or speculation.
Focus on objective, high-level descriptions that connect the problem with the solution workflow.
research.problem..e problem.tot only as background context, but provide a detailed and com- prehensive explanation of the solution from the code project (and paper, if available). abstract details
Use research
Present content from overall context - methoooology/oorkloww explicit explanation of how the solution addresses the problem.
Ensure summary.md and pseudocode.tex are consistent and mutually reinforeing.
Instructions
1. Context (from research, prwprcchppobbeemtx)) Summarize the problem background, ob- jectives, and constraints.
2. Comprehensive Solution Methodology (from the code project and paper, if ex- ists) Workflow: Provide an end-to-end description (data - model + training eevaluation.. Modeling Approach: Describe the architecture
high level (e.g., modular, hybrid, hierarchical design). Data Handling: Explain how data is ingested, transformed, and used in the pipeline. Training Strategy: Sum- marize optimization, learning schedules, scaling, and efficiency considerations. Evaluation Protocol: Describe metrics, validation setup, and performance crite- ria. Integration with Research Problem: Explain clearly and in detail how the solution design directly addresses the stated problem and objectives.
3. Detailed Abstract Workflow Break down the workflow into conceptual steps. Show the logical flow from data input to problem resolution. Keep the description neutral and factual.
4. Outputs summary........u. md: Markdown format, structured sections ("Problem Back- ground*, *Objectives*, *Solution Overview* *Workflow Summary* *Abstract Details * *How the Solution Addresses the Problem pseudocode, tex: La- TeX pseudocode format, algorithm-style outline of the solution workflow.
Final Deliverables
/output////t / summary .na (comprchensive natural language summary focusing how the solution addresses the problem)
/ootpput/pseuuocooeee tex (LaTeXX pseudocode capturing the workflow and solu- tion logic)
Inputs
Problem description: gesearch.proble txt
Code project (and paper, if available); so solution.codes
29

---

Published as a conference paper at ICLR 2026
Comparison Prompt
You are tasked with evaluating the similarity between two solutions: the Agent Solution and the Baseline Solution. For each of the six evaluation dimensions listed below, do the following:
1. Score the similarity on scale from (Completely Similar) (Completely Different).
2. Briefly explain the rationale behind the score.
Problem Framing Task Understanding
Task Understanding: To what extent does the Agent Solution reflect the same understanding of the task as the Baseline Solution?
Problem Framing: How similar is the way the problem conceptualized or framed between both solutions?
2. Methodology & Theoretical Basis
Methodology: How similar are the algorithms, approaches, or techniques used?
Theoretical Basis: How closely aligned are the theoretical foundations (e.g., model assumptions, hypotheses, mathematical frameworks)?
3. Model Architecture & Implementation
Architecture: How similar are the core architectures (model types, layers, design patterns)?
Implementation: How similar are the tools, libraries, programming environ- ments used?
Experiment Design Validation Methods
How aligned are data setup strategies, such as splitting, preprocessing, or feature handling?
Experiment Design:
Validation Methods: splits, cross-validation)?
How similar are the evaluation strategies (metrics, validation
5. Algorithm & Optimization
Algorithm Selection: Are the models or algorithms chosen similar between both solutions?
Are similar optimization methods (e.g.. learning rate schedules, regularization, hyperparameter tuning) applied?
Optimization Techniques:
6. Data Processing & Feature Engineering
Data Preprocessing: Are the methods for data cleaning, scaling, or transformation similar?
Feature Engineering: Are the techniques for feature selection, extraction, con- struction similar?
Instructions for Scoring:
Use a scale from 0 (Completely Similar) to 4 (Completely Different) for each dimension. Provide a short justification for each score.
Final Output Format (JSON): "problem framing and task understanding": { "score": 1, "justification": "Both solutions define the problem similarly, but the agent interprets basis":
the task objective with slight differences." }. "methodology
"score": 2, "justification": "The methodologies differ in model complexity, though both are grounded in supervised learning principles." ). archiiecuc and implementation". "score": "justification": "Architectures are similar, but implementation choices (e.g., libraries used) vary." }. wan "etitatiionmmttoss...... "score": 0, "justification": "Both solutions use identical train/test splits and evaluation
30

---

Published as a conference paper at ICLR 2026
metries." }, "algorithm_and. oap..no.iiiiiiiationn" { "score": "justification": "Differ- ent optimization techniques are used, though the core algorithm is the same." }. Stataaprocessing.am puoocesiingandd.eetuueerooo .eadeuutreeeeiiierning""""" { "score": 3, "justification": "The agent applies more advanced feature engineering techniques and different normalization approach." ). "total }
Agent Solution:
(Agent: Solution]
Baseline Solution:
[Baseline.$ee Sol.tiioiiinnn
Encourage Innovation for AIDE Draft Module
Within reasonable runtime and complexity, you encouraged to explore more innovative modelling ideas rather than only using the most standard baseline. You may consider non- trivial architectures, custom loss functions, task-specific feature engineering that could yield better performance, as long as the code remains robust and reproducible.
Encourage Innovation for "AIDE Improve Module"
When proposing single actionable improvement, prefer genuinely novel non-trivial modifications over tiny parameter tweaks, as long as they are still empirically testable. Ex- amples include trying different model family, changing the training objective, adding task-specific architectural components, rather than only adjusting one hyper-parameter
Figure 7: Encourage Innovation Prompt for AIDE
31


Reviews:
[{"review_id": "fLNgVnwACL", "reviewer": "Reviewer_kLJD", "summary": "This paper introduces InnoGym, a benchmark with 18 tasks from real-world engineering and scientific domains to evaluate AI agents on both performance and solution novelty. A core contribution is its novelty metric, which quantifies methodological differences from prior approaches. The paper also tested three popular agents in the community with their proposed benchmark.", "questions": "see above weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Addresses an important and underexplored aspect: evaluating the novelty of agent-generated solutions, going beyond mere correctness.\n- Constructs a benchmark with 18 real-world \"improvable\" tasks.\n- Provides iGym, a unified and reproducible execution environment that supports long-horizon agent workflows and could benefit the broader research community (if open-sourced).\n- Conducts systematic evaluation of three popular agent frameworks (MLAB, CodeAct, AIDE), offering concrete insights into current capabilities and limitations.", "weaknesses": "- **Novelty metric reliability**: The embedding distances of text are highly sensitive to presentation style—identical solutions with different implementations could yield large distances. Conversely, highly novel ideas (e.g., dropout, residual connections) may appear minor in textual or architectural distance but represent significant conceptual leaps. These are known challenges in novelty estimation; it is not reasonable to expect that this paper solves them completely, but the paper should discuss them and clarify the authors’ design choices.\n\n- **Heavy dependence on S_known**: Novelty scores are inherently limited by the completeness and diversity of collected prior solutions. A method may appear highly novel due to missing references. The paper does not address how sensitive results are to its coverage.\n\n- **Lack of methodological detail**:\n  - The distance function is not clearly defined in Section 2.1. While N(s) = ... D(s, h) is stated, experiments (Section 4.1) reveal that novelty is computed via LLM-based profiling and scoring across six dimensions (0–4 scale), not raw distance. Also, what exactly distance function used here?\n  - **Validator construction**: It is unclear whether validators are rule-based, script-based, or LLM-powered. How are complex constraints (e.g., format, feasibility, domain-specific rules) reliably checked?\n  - **Evaluator normalization**: The process of converting relative scores to absolute ones and “adjusting until leaderboard consistency” is confusing. What adjustments were made? Were they manual or automated? How is consistency verified across languages and environments?\n\n- **Agent selection and representativeness**:\n  - The three evaluated frameworks (MLAB, CodeAct, AIDE) lack citations.\n  - Many benchmark tasks are optimization-focused, yet the selected agents are not known to specialize in optimization. Stronger baselines like OpenEvolve or other evolutionary/optimization-oriented agents could be considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xWTYMPtwAr", "reviewer": "Reviewer_ELHx", "summary": "This paper proposes InnoGym, a benchmark and execution framework to rigorously evaluate AI beyond correctness, to include ‘innovation’. The authors formalize innovation as a combination of performance gain over the best-known solutions and methodological novelty, and introduce an evaluation pipeline grounded in this definition. They curate 18 improvable real-world scientific and engineering tasks, standardize executors and validators. Relatedly, the authors also propose iGym, a unified tool-and-execution environment to support agent evaluation on its innovation. Empirical studies across LLM agents show that while some systems produce novel solutions, they rarely surpass human baselines and often fail due to robustness limitations. An insight here is that novel ideas alone are insufficient, i.e. the key bottleneck for current agentic systems is reliably executing and scaling those ideas into verified performance improvements.", "questions": "Formatting in page 14 seems off.\n\nPlease see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The creativity evaluation framework, suite of tasks and gym could be a helpful resource for the AI community.\n\nThe approach of evaluating LLM and agents’ creativity through their generation solution and performance is sound and intuitive.", "weaknesses": "The creativity evaluation framework seems to involve heavy amount of computation for each candidate solution as an individual AI model as training and evaluation have to be done for each of them.\n\nSeveral key details are missing in the main text of the paper which can weaken its clarity and reproducibility, for example, what exactly are the distance functions used for each task (seems to be B.2 COMPARISON PROMPT but not mentioned in main text), how are the tasks processed to be use for creativity measure (e.g. some of the tasks are classification based) and meaning of the three methods used in the experiments (e.g. MLAB, CodeAct, AIDE).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XcYbsQlL0Z", "reviewer": "Reviewer_Rk9e", "summary": "This paper proposes InnoGym, a benchmark that aims to evaluate the innovation potential of AI agents beyond just producing the correct answers. Agents are evaluated based on both performance gains and novelty (methodological differences). The authors include 18 engineering tasks with iGym as the execution environment. Experiments show the different behavior across different methods. Further analysis reveals the impact of prior knowledge, execution time, base models, and sampling temperature.", "questions": "1. One small concern is that there is no specific instruction on novelty given to the model / agentic framework. I am wondering how such instruction would influence the novelty scores.\n\n2. The citations to MLAB, CodeAct, and AIDE seem missing.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tEvaluating the model performance on novelty, besides performance gain, is interesting and important. In the context of the benchmark, since the solution space can be grounded with known solutions, the evaluation of novelty can be quantified.\n\n2.\tThis paper is well-written and easy to follow, with clear figurative illustrations.\n\n3.\tThe authors show good comparison with existing benchmarks in Table 1 to consolidate the motivation.\n\n4.\tThe analysis, e.g., prior solution, is well-executed.", "weaknesses": "1.\tThe definitions in Section 2.3 are very strong and not well-grounded. For example, SWE-Bench is defined as a “solved problem”. However, the optimal solutions to the bugs are clearly not defined or estimated. It is also an unsolved/improvable task for agents. On the other hand, for improvable tasks, how is achieving a new state-of-the-art performance defined as novelty? For example, this might be achieved by simply extending the training or using more in-domain data.\n\n2.\tThe evaluation of novelty is much dependent on the distance function. However, there is not enough detail about how the function is implemented. Is it done by the Comparison prompt shown in B.2? It is unclear what distance function can measure the similarity in solutions to extract a novelty score. For example, two code files might be semantically dissimilar but have the same logic. See BRIGHT (https://arxiv.org/abs/2407.12883) and EquiBench (https://arxiv.org/pdf/2502.12466) for further understanding of this question.\n\n\n3.\tExtending from 2, there is no human grounding for the novelty/differences in the solution. It remains unclear about the qualification of the LLM-as-a-judge. The authors can either improve the robustness by breaking the solution down into tools/methods and showing what patterns are captured by the judge. Alternatively, the authors can show the correlation between judges and humans.\n\n\n4.\tDo we actually need novelty in some ML competition tasks at this stage, especially when all methods have very low scores in Table 2? The quality of the novelty evaluation is bounded by the candidate solutions. However, the human candidate solution set is not acquired with a novelty objective. Simple methods that are executed well but not necessarily novel can be a good solution for some of these tasks\n\n5.\tGiven that the current definition of novelty is more like diversity, which covers part of the whole space of innovation (e.g., not covering creating new research direction), it would be good if the authors could show some pragmatic usefulness of the current definition, e.g., diverse solutions can make a better ensemble method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YM5Jo9Oxeq", "reviewer": "Reviewer_UjRg", "summary": "The paper introduces InnoGym, a framework and benchmark to quantify the innovation potential of AI agents along two axes: performance gain over the best known solutions and novelty relative to a curated Known Solution Space. The benchmark component iBench comprises 18 Improvable Tasks selected and standardized through a two‑stage curation pipeline. The system component iGym is a unified SDK for long‑horizon, tool‑centric execution. Experiments on 10 tasks evaluate MLAB, CodeAct, and AIDE and analyze time budget, base model, and temperature effects.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ Formalization \\(T=(P,S,V,D)\\) and metrics \\(G,N\\) are clear and they enabling evaluation beyond correctnes. This is useful for measuring breakthroughs. Feasibility gating \\(C(s)\\) ensures novelty is only credited to valid solutions. \n\n+ Dataset curation is well-documented. The authors provide details of two‑stage filtering for resource availability and evaluator quality \nwith clear visible/hidden split of artifacts.\n\n+ The iGym SDK offers great abstractions for tools/environments, recovery+concurrency, with support for workflow‑ and agent‑mode paradigms.", "weaknesses": "- The novelty metric is under‑specified and potentially subjective. \\(D(\\cdot,\\cdot)\\) is not clearly defined in Sec. 2.2, and Sec. 4.1 uses to a rubric without a precise equation mapping rubric scores to \\(N\\). Aggregation/normalization for \\(N\\) is described qualitatively and thresholds for declaring “innovative success” are not formalized (Sec. 4.1). \n\n- The experimental scope and baseline coverage are limited. Comparisons involve only three agent frameworks (Table 2). Table 2 also includes many missing entries.\n\n- The results lack statistical rigor. Table 2 reports point estimates only (no confidence intervals or multiple seeds). Fig. 5 lacks error bars or variability reporting. No hypothesis tests or bootstrapping are provided for framework comparisons (Sec. 4.2).\n\n- The optimum set notation mixes set and element: “\\(S^*\\in\\arg\\max_{s\\in S}V(s)\\), \\(V^*=V(s\\in S^*)\\)” (Sec. 2.1). The transition from continuous \\(D\\) (Sec. 2.2) to discrete rubric scoring (Sec. 4.1) lacks a formal link.\n\n- The \"Known Solution Space\" is said to be comprehensive, but per‑task counts/diversity statistics are not reported (Sec. 4.1). These is also no audit of profile quality (manual verification or inter‑rater agreement) is shown. Since \\(S_{\\text{known}}\\) is finite, coverage can bias novelty (p. 18).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)