InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research
Authors: Yunze Wu
Dayuan Fu
Weiye Si
Zhen Huang
Mohan Jiang
Keyu Li
Shijie Xia
Jie Sun
Tianze Xu
Yang Xiao
Pengrui Lu
Xiaojie Cai
Lyumanshan Ye
Wenhong Zhu
Xiangkun Hu
Pengfei Liu
Venue: ICLR 2026 Poster (2026)
Keywords: InnovatorBench
ResearchGym
End-to-End Evaluation
Primary area: datasets and benchmarks
TL;DR: We introduce InnovatorBench, a benchmark for evaluating LLM-based agents on realistic, end-to-end LLM research tasks. To support testing, we build ResearchGym, an environment for long-horizon, distributed agent execution.

Abstract: AI agents could accelerate scientific discovery by automating hypothesis formation, experiment design, coding, execution, and analysis, yet existing benchmarks probe narrow skills in simplified settings. To address this gap, we introduce InnovatorBench, a benchmark-platform pair for realistic, end-to-end assessment of agents performing Large Language Model (LLM) research. It comprises 20 tasks spanning Data Construction, Filtering, Augmentation, Loss Design, Reward Design, and Scaffold Construction, which require runnable artifacts and assessment of correctness, performance, output quality, and uncertainty. To support agent operation, we develop ResearchGym, a research environment offering rich action spaces, distributed and long-horizon execution, asynchronous monitoring, and snapshot saving. We also implement a lightweight agent that couples explicit reasoning with executable planning using frontier models such as Claude-4, GPT-5, GLM-4.5, and Kimi-K2. Our experiments demonstrate that while frontier models show promise in code-driven research tasks, they struggle with fragile algorithm-related tasks and long-horizon decision making, such as impatience, poor resource management, and overreliance on template-based reasoning. Furthermore, agents require over 11 hours to achieve their best performance on InnovatorBench, underscoring the benchmark's difficulty and showing the potential of InnovatorBench to be the next generation of code-based research benchmark.

Published as a conference paper at ICLR 2026
INNOVATORBENCH: EVALUATING AGENTS' ABILITY TO CONDUCT INNOVATIVE LLM RESEARCH
Yunze Wu 33 Dayuan 2 Weiye Si Zhen Huang Mohan Jiang Keyu Li
Shijie Xia 12233 Jie Sun Tianze Xu Yang Xiao², Pengrui Lu" 1,333 Lyumanshan Ye Wenhong Zhu Xiangkun Hu2.3, Pengfei Liu Shanghai Jiao Tong University, "Shanghai Innovation Institute, "GAIR
Xi.3.2.3,
Lu',2.3, Xiaojie Cail.3
ABSTRACT
Al agents could accelerate scientific discovery by automating hypothesis forma- tion, experiment design, coding, execution, and analysis, yet existing benchmarks probe narrow skills simplified settings. To address this gap, we introduce In- novatorBench, benchmark-plattee
pair for realistic, end-to-end assessment of agents performing Large Language Model (LLM) research. comprises 20 tasks spanning Data Construction, Filtering, Augmentation, Loss Design, Reward De- sign, and Scaffold Construction, which require runnable artifacts and assessment of correctness, performance, output quality, and uncertainty. To support agent op- eration, we develop ResearchGym, a research environment offering rich action spaces, distributed and long-horizon execution, asynchronous monitoring, and snapshot saving. We also implement a lightweight ReAct agent that couples ex- plicit reasoning with executable planning using frontier models such as Claude-4, GPT-5, GLM-4.5, and Kimi-K2 K2. Our experiments demonstrate that while fron- tier models show promise in code-driven research tasks, they struggle with fragile algorithm-related tasks and long-horizon decision making, such as impatience, poor resource management, and overreliance on template-based reasoning. Fur- thermore, agents require over 11 hours to achieve their best performance Inno- vatorBench, underscoring the benchmark's 's difficulty and showing the potential InnovatorBench be the next generation of code-based research benchmark.
Dataset
Agent Working
Research Domains
Time Horizon
Conda Environment
Scatfold Construction
2-36h
Data Augmentation
+
Related Repository
InnovatorBent Bench
Data Construction
Task Description
Reward Design
Evaluation
Data Filtering
Hint
Loss Design
Infrastructure Support
Available Actions
Multi-Computer Control
Operate Files
ResearchGym Gym
Online Browse
Asynchronous Command Execution
Operate Terminal
Snapshots Saving & Loading
Web Search
Files
Figure 1: Overview of InnovatorBench and ResearchGym. InnovatorBench consists of 20 LLM research tasks from 6 research domains. Each task requires 2-36 hours complete. ResearchGym provides the infrastructure support for the agent to work on InnovatorBench.
The order of co-first authors alternates between InnovatorBench and Apollo hhttps://arxis> htt////www..or/pf//5/110 2.26300, Emails: yyunze, wue wtu.sjjt..ed..oom Fdyēbupt.edu.
'Equal contribution.
on
Github Page: /ttpps:/ //tp////gttbb..////// com/ BAIR-NLE S.L.I-NW//Innoatorrttt InnovattBevtennc Corresponding authors. Email: stwfannenggeii

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Artificial intelligence (AI) is becoming central to scientific discovery (Chen et al. 2024; Starace jt al. 2025). Traditional workflows require humans to hypothesize, design experiments, implement and debug code, process data, manage resources, and analyze results. As Al rapidly advances, the potential to automate entire research workflows is on the horizon (Liu et al. 2025). We refer to these systems as "Al researchers": agents that integrate multiple stages of research and target human-level behaviors, including insight generation and implementation (Team et al. 2025b). Since Large Lan- guage Models (LLMs) act the "brains" of such agents (Xi et al. 2025). they can finish auxiliary tasks such as data cleaning, augmentation, loss design, reward design, or architecture design as LLM capabilities improve in planning, code generation, and debugging. Moreover, better LLMSs agents can propose hypotheses and execute experiments more reliably, which accelerates discovery and feeds back into improving themselves (Liu et al. (025). As result, transferring improvements in LLM capabilities into genuine progress for AI research agents requires more than isolated skills. The key question is whether these abilities can be orchestrated into coherent, end-to-end research workflows (Chen et al. 2024, Edwards et al. 2025). This motivates systematic and realistic evalua- tion of AI research agents and has prompted recent efforts to benchmark them.
Recent efforts to Al research benchmark have provided valuable insights and represent important first steps toward formalizing this emerging area (Starace et al.) 2025 Chen et al., 2024] Edwards et al. 2025 Xu et al. 2025 Team 2025). These studies show that current agents can already achieve non-trivial performance on experiment design, implementation correctness, and even limited replication of advanced research results, establishing clear baselines for progress. At the same time, these benchmarks highlight several structural limitations. Many existing tasks concentrate narrowly on single performance dimension, such as code implementation accuracy, parameter tuning (Hua et al.] 2025). , rather than evaluating the entire research process end to end. Success is often framed as reproducing existing results (Starace al.) 2025), ) which measures fidelity but not the capacity for innovation, new objective design, or architectural creativity. Moreover, the research environments where agents are evaluated are simplified and resource-constraaied, so large-scale and long-horizor training or inference are typically unsupported, and asynchronous monitoring of processes that span multiple hours is rare (Kon et al. 2025). Action spaces are also constrained, preventing agents from engaging in realistic research behaviors such as managing files, executing commands, or browsing literature (Chen al.] 2024). These limitations collectively restrict the
conclusions that can be drawn about an agent's 's potential as a true research collaborator.
In this paper, we try to address these challenges by introducing Innovator and a new experi- mental platform ResearchGym that evaluates Al research agents in real scientific practice.
InnovatorBench systematically evaluates core subproblems in LLM research, encompassing data construction (DC), data filtering (DF), and data augmentation (DA), loss design (LD), reward design (RD), and scaffold construction (SC). It consists of 20 tasks, each task isolates distinct research domain, requiring agents to propose creative methods, implement their own ideas, refine ideas & implementation based on the results, produce concrete outputs, and submit their outputs for sev- eral times. To establish baselines and ensure reproducibility, we construct reference solutions and relative evaluation scripts. The reference solutions remain hidden during evaluation so that agents must rely on their own reasoning and design choices. The evaluation scripts quantify metrics like correctness, quality, and/or even the output uncertainty like the entropy of predictions after Rein- forcement Learning (RL) (Yu al. 2025), thereby providing a multifaceted view of agent capabil- ities. This setup emphasizes both diversity and openness because the tasks span different types of challenges, allow multiple solution strategies, and reward innovation rather than simple replication. Consequently, InnovatorBench moves beyond narrow tests of implementation fidelity and provides a rigorous framework for assessing whether agents can execute end-to-end research workflows that mirror the demands of real LLM development.
In parallel, ResearchGym offers a scalable and realistic environment that addresses limitations of existing platforms (Nathani et al. 2025 Wang et al. 2024a). It provides rich action space that covers terminal commands, file operations, web search, and web browsing. Building on this foun- dation, ResearchGym supports large-scale experiments that may run for many hours even days, with facilities for launching, monitoring, and adapting long-rumning processes, as well as distributed training across multiple machines and GPUs. It also provides snapshot saving and loading for paus-
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of AI benchmarks across key evaluation dimensions. Time Horizon refers to the time the ReAct-based Agent takes to reach its best score. ML-Bench doesn't report this result.
Multi-GPU
Save and Restore
Time
Benchmark
Task Resource
Max Eval times
MuuttiNNode
Creativity
Horizon
SWE-bench Jimenez al
2024
GitHub Issues
30m-2h
SeienoeAgentencch Chen al.
2024
Scientific Papers
10m
RExBench (Fdwards et al.
2025
NeurIPS, ACL*, etc. Paper
3
x
6h-12h 12h
RE-Bench
Wijk et
Design Manually
12m-2h
EXP-Bench
Kon ct al
2025
NeurIPS, ICLR Papers
X
X
x
35m
PaperBench
Starace
2025
ICML 2024 Papers
1h-3h
MI--Bench
Tang
Kaggle Competitions
Unknown
MLE-bench
Chan et al.
2024
Kaggle ML Tasks
0C
V
10m
InnovatorBench
NeurIPS, ICLR, etc. Papers
2h-36h
ing and resuming experiments without loss of progress. Importantly, ResearchGym is not tied to single benchmark; is general and extensible platform to which the community can contribute new tasks, datasets, and evaluation protocols, similar how models and datasets shared in the Hug- gingFace (Wolf et al.) 2019). This openness allows ResearchGyym to evolve with research needs, serving the foundation for InnovatorBench and ideas, building baselines, and comparing agents across diverse experimental settings.
independent environment for testing new
To demonstrate the utility of framework, deploy ReAcct-based agent on InnovatorBench with several frontier LLMs, including Claude Sonnet 4 (Anthropic 2025), GPT-5 (OpenAT 2025). and GLM-4.5 (Zeng et al. 2025), Kimi-K2 (Team al. (2025a). These experiments provide sys- tematic basis to analyze how different foundation models perform across diverse subproblems in LLM research, revealing that these models have the potential to handle code-based research tasks longer than 10 hours. However, they struggle with fragile algorithm design and long-horizon decision making, often exhibiting impatience, resource mismanagement, poor library choice, and reliance on template-based reasoning. Such comparative analysis offers new insights into the align- ment between model capabilities and the requirements of end-to-end agentic research.
Our contributions can be summarized as follows:
We introduce InnovatorBench, the first benchmark systematically evaluate Al research agents on end-to-end LLM research tasks, spanning data construction, filtering, and augmentation, loss design, reward design, and scaffold construction under multiple dimensions.
We develop ResearchGym, a general and extensible research environment supporting long- duration and distributed experiments, asynchronous execution, snapshot saving and loading, and broad action space for realistic research workflows.
We perform an empirical analysis of InnovatorBench across multiple leading LLMs, demonstrating its potential and weaknesses in handling real LLM research tasks.
RELATED WORK
Recent years have seen growing efforts in developing code agents, which generally fall into two categories:
repository-leve code benchmarks for assessing specific technical competencies, and agent frameworks that offer execution environments and scaffolding for interactive mng-oorizon tasks. Table I presents a comparison of several related benchmarks.
Repository-level code benchmarks. Several benchmarks focus on assessing whether agents can solve software engineering or machine learning tasks within realistic repositories. SWE-bench (Jmenez et al., 2024 Yang et al.) (025b]c) and its variants evaluate an agent's ability to resolve GitHub issues by generating executable patches that pass unit tests (Yang al. 2024 Yao et al. 2023). Science/Agettennc (Chen et al. 2024) extends this paradigm to scientific domains, requir- ing agents to write programs that replicate or analyze results derived from real papers. RExBench (Edwards et al. 2025) and EXP-Bent -XP-Bench (Kon et al. 2025) target reproducibility and experiment ex- ecution, testing whether agents can reconstruct pipelines reproduce known results. PaperBench
3

---

Published as a conference paper at ICLR 2026
(4) Datasets
(b) Evaluations
Tark Deseription
Agent's Workspace
Rratuaitin Script
Not tostion
workspaoe/
faskeenden del ran 3iolff Dacifatt, Holl
Lange Loogagge mootss aften suffers froe aneetreyy
conday
dulia!
deg_enitwyy, esc watertiwwyannn www.pan..seooooolll.ooo....
beconss ever dcter sinistat carly treining
cleskkooints/
ceeept Exccstion es
ane ror stratips for gveg Your bodel reteforomest Teamning augrect acturacy and present entrapy collapse....
Tosk
dataset/
failest, ...
"erront: "Ehrers cates setestaneeenn cateelanseettrrpp wwtrrsy.inttta.ooooo anctarryy
repositories/
tatt
Dev
Test Cutts
scripts/
acc_store selfaaccoracy scoretacc)
retore
"acoreft estropy suure ascserr 100,
score selfiestrayy scoropys
/Hist for the Agost
"Loss_ao (ressl Parrent Nave,
Worning LINO Lunte hoursive
coplly
Si Sclant dau Lopsa bon, Sepocton VaclLals
Era Lives
Rrvaliaitonn Dinetrey
Reference Satution
Entrspy doalysts
Cidh/
/tackrrapos Lverl) aloo..y
datal
Ne houe catip the conds enxtreemen for you roned Dworksppce/coods
reference/
"earropy" 18.21
provided ustil deceah
task//
scriats/nccel ecrger upy serge the sceel weights into Megpinghaut for
You
the
con' by
scripte/
The sgent cin only receive de some descripiio
will dedaredd
evallattion.py
An illustrative LLM research task from DAPO (Yu et al., 2025). (a) Datasets. The agent receives task description and a starter workspace; optional hint is only revealed upon the agent's explicit request via the view hint tool at a final score penalty. (b) Evaluations. An evaluation directory includes evaluation scripts and reference data. Evaluation is performed externally using hidden scripts and reference data. The agent submits its output via the eval tool and receives a score with feedback, preventing answer hacking. The full example is in Appendix D
Figure 2:
Starace et al. 2025) collects machine learning tasks from papers to evaluate large-scale replica- bility. DatasetResearch (Li et al. 2025) emphasizes dataset discovery and reasoning about data usage. Whereas existing benchmarks focus on narrow aspects of research (e.g,, code modification, experiment reproduction), InnovatorBench targets a broader set of LLM--cntric research abilities, evaluating agents' proficiency across the entire research lifecycle.
Agent scaffold and environments. A complementary line of work focuses on platforms for deploy- ing code-capable agents in interactive environments. OpenHands (Wang et al. 2024a) allows agents to interact with sandboxed environment via coding, command- operations, and web browsing. Commercial systems such as Claude Code demonstrate practical coding assistance but prioritize short-term tasks over long-running research-orientee orrented workflows. Other research systems, includ- ing WorldCoder (Tang al. 2024) and multimodal variants such as OpenHands- Versa (Soni al.) 2023). highlight the potential of tool-augmented agents for general problem solving. Correspond- ingly, environments like MI.Gym (Nathani et al.) 2025) provide structured contexts for ML-rellated tasks but often constrain the experiment duration, scale, or action space. A common limitation across these frameworks is the lack of support for extended scientific research: they rarely provide distributed training, asynchronous monitoring of multi-hour jobs, snapshot saving, and integration of open-endes research goals. Our ResearchGym directly addresses these gaps by exposing a rich and extensible action space, enabling Long-horizon and distributed experiments, and offering a foun- dation where new tasks and evaluation protocols can be shared and extended by the community.
INNOVATORBENCE
InnovatorBench evaluates Al agents' ability to complete end-to-end, imnovation-oriented AI research tasks. Each task is derived from an influential AI research paper and its open-source source codebase. This coupling captures the full scientific workflow by linking high-level research questions to concrete implementations. As shown in Figure 2 each task entry comprises a task description, an initial starter workspace, hint for the agent, evaluation scripts, and reference solution derived from the original research artifacts. The agent's 's objective is to extensively explore this task in our environ- ment and aim to achieve performance that surpasses the ground-truth solution.
Benchmark Overview and Statistics. InnovatorBench currently comprises 20 research tasks drawn from 14 influential papers, detailed in Appendix[A] These tasks span diverse LLM research areas, including data construction, filtering, and augmentation, loss design, reward design, and scaffold construction. They sourced from top-tier venues, namely NeurIPS, ICLR, COLM, EMNLP, and
4

---

Published as a conference paper at ICLR 2026
ACL, and the latest publications. This breadth ensures coverage of diverse experimental paradigms, coding practices, and research challenges prevalent in LLM research. Together, these features make InnovatorBench comprchensive testbed for assessing the capabilities of Al agents.
Task Description. Each task description provides the agent with the following components:
(1) Motivation: The research motivation and provenance of the question
(2) Task: A high-level description of the objective for the agent. To encourage exploration and avoid overfitting to prescribed procedures, we do nor specify step-by-step instructions: instead, the agent is expected to aim surpasses the reference solution no matter what method it selects.
(3) Data: Details of the relevant datasets and checkpoints, including content description, storage paths, file formats, and illustrative examples.
(4) Constraints: The operational constraints under which the agent must complete the task, like working time limits, GPU quotas, and output file format.
(5) Evaluations: The evaluation metrics like accuracy, FI, and BLEU, and introduction to the scoring function this task.
Information about the execution environment, including the conda environment and the workspace directory layout.
(6) Environment:
(7) Seripts: The description of several supplementary unified scripts and repositories that the agent can use in their process.
Workspace. The workspace is writable directory containing essential task artifacts, over which the agent has complete control. The workspace comprises three major components:
(1) Conda environment: We pre-build minimal conda environment that replicates the original paper's setup to run baseline experiments. We recommend not modifying this base environment; however, to preserve the agent's autonomy, do not prohibit modifying packages when necessary.
(2) Data: For the agent to validate whether its proposed methods enhance model performance, we supply complete datasets (training, validation, and a test set without ground-truth for agents' sub- missions) and model checkpoints suitable for fine-tuning, The agent may also search for, download, and reformat additional data to meet the repository's requirements, or even synthesize new datasets using the provided models or by generating chain-of- thteoooof-ooout-ttttyoloe data for augmentation.
(3) Task: This directory contains the code repository and set of helper scripts. The repository is adapted from the original paper's 's codebase: we remove the implementation of the paper's key novelty and git commit history while keeping the project runnable. In most tasks, the repository is LlamaFactory (Zheng et al. 2024) or Verl (Sheng et al., 2024). The scripts folder offers scripts for data construction, training, inference, and evaluation; the agent may add its own scripts and files.
Hint for the Agent. To assist with these challenging tasks, we provide optional hint for each task. Hints are not included the workspace; an agent may query their contents via the view_hint tool, choosing whether to adopt them. Our main evaluation disabled this tool, while the ablation study provides a hint immediately after the task description.
Evaluations. Our evaluation follows Kaggle-styld procedure with multiple submission oppor- tunities and immediate score feedback on the test set. First, a submission is checked for format validity, with failures receiving score of 0 and error message. Subsequently, valid submissions are scored based on a function calibrated between a baseline (anchored near 0) and a reference so- lution (anchored near 80). The entire evaluation runs externally to the workspace. As shown in Figure 3, in real-time operation, the agent can use the eval action to evaluate its results and obtain evaluation details (c.g., results) from the judge system. If the agent evaluates for max eval times, the system will automatically use the finish action and stop this process.
InnovatorBench Construction We first design 20 raw tasks based on the following principle:
(1) The task can be reapplied; the result is aligned with the original paper.
(2) The task result can gain sigmificant improvement in days
hhtps///www.kgglle.om
5

---

Published as a conference paper at ICLR 2026
Snapshots
Description
Agent's Workspace
InnovatorBench Dataset
workupaccr
Hint & Evaluations Scripts
tasi is reaent new strategy for language
candal
Agent's s Contest
detal
Mint for the Agent Apply
class TaskBencheork Basprrrkrraaeeeenaaaaao rk2s
chockpolints/
def run, (ssll -o Diet:
Worksgsee
dataset/
SooxtfclupNr..ss.c2a Vepsilon + "delltals
return Msconelt
reinforcemen
Remaining
learning in orowr
cospute_ Licy.
"test_ set___st___sttteeeeeeee resuters ...)
Time
scriptsz
Initialize
eval view_hint
Evaluation results for agent
ResearchGym
score: 4.5,
Multi-Computee
System
Actions Observations
Observation
Ageat
CPU Machine
Command
task_Besc.
Reasuning &
GPUCPP
File
Machines
Pame
Action
Context Summarize
hint.nd
Internet Access
Interal
Web Search
Connection
Web Drowse
finish
Figure 3: InnovatorBench evaluates AI agents on research tasks extracted from AI papers. Re- searchGim is initialized with the InnovatorBench dataset; the agent receives a task description and workspace, reasons over observations, and issues tool calls that are translated into actions executed on target computer, with results returned as structured, agent-readable observations. The agent iterates this process, optionally using view hint for hints and eval for submitting answers, until calling f finish. ResearchGym then performs a final evaluation and saves a state snapshot.
(3) The tasks can evaluate the different abilities of LLM Agents in LLM Research.
(4) The task uses common models like 1lama3.1 (Grattafiori al. 2024), Qwen2.5
TTeam 2024
Guo et al. 2025 Qwen et al., 2025), or Qwen2.5-Vi VL (Bai et al. 2025). ) etc.
More details about the annotation process can be seen in Appendix
4 RESEARCHGYM
Prior agent systems such as OpenHands (Wang et al. 2024a) and IterativeAgent Starace et al, 2025) operate within single Docker container. They execute commands synchronously, so the next action cannot be chosen until the previous one finishes. This design constrains the scale of experi- ments and reduces action throughput. To overcome these limitations, introduce ResearchGym, an environment designed to approximate real-world LLM research. Inspired by OpenHands's 's de- sign, ResearchGym provides 42 primitive actions that agents freely compose, supports control of multiple machines and asynchronous command execution, and allows users to save and restore environment snapshots. It should also be emphasized that the ResearchGym is not solely used for InnovatorBench, we believe it can be used in other research tasks.
Actions and Observations. Actions of ResearchGym are grouped into five families: Command, File. Parse, Web Search, and Web Browse. Command actions can manage execution sessions, run commands within a session, and retrieve outputs. File actions can perform file operations (e.g., create, edit, delete, read, and search), and query file metadata. Purse actions can extract and pre- view content from multi-modal sources (e.g., images, audio, and video) for text-only models. Web Search and Web Browse grant networked retrieval and browsing for accessing up-to-date methods and datasets. Each action family is paired with an observation that normalizes raw outputs into a structured, agent-readable return. Details can be referred to Appendix
Multi-Computer Control. ResearchGym allows agents to control multiple machines (or Docker containers) concurrently via HTTP. Each computer runs HTTP server to receive and execute terminal commands, allowing an agent initialized on a single machine to orchestrate long- horizoo distributed experiments across cluster.

---

Published as a conference paper at ICLR 2026
Asynchronous Command Execution. ResearchGym decouples action execution from selection to prevent decision blocking. Agents can bind commands to specific sessions, or let ResearchGym cre- ate new ones. This ensures ongoing jobs continue uninterrupted and enable immediate subsequent planning. Agents can later retrieve the result via get. sess on output asynchronously. To avoid nonsensical actions during model training, ResearchGym provides dedicated sleep action.
Snapshots Saving and Loading. A snapshot records the task specification, the agent's context, the final state of the workspace, and the remaining time budget. ResearchGym can periodically save the full state as snapshots, and it can restore the system from any snapshot. Snapshots support branching. Experiments can resume from different points or proceed along multiple branches.
Pipeline. Figure 3 depicts the end-to-end interaction loop. The process begins when ResearchGym loads a task from InnovatorBench, providing the agent with a task description as its initial observa- tion and a starter workspace. Given an observation, the agent reasons and issues a tool call. If it's not a command action, the ResearchGym will produce locally; otherwise, ResearchGym converts this call into action, wraps it in an HTTP request, and dispatches it to target machine. The target machine executes the action or launches it as a background process. ResearchGym packages the outcome as a new observation in an agent-readable format. Synchronous actions immediately update the workspace, whereas asynchronous actions return a session ID and status for the agent to poll with subsequent commands. The agent repeats this loop, optionally submitting answers for evaluation and consulting hints when needed, When the agent deems the task complete, it invokes finish. ResearchGym performs final evaluation, saves snapshot, and finalizes the task.
5 EXPERIMENTS AND RESULTS
5.1
EXPERIMENTAL. SETUP
We evaluate Ieading LLMs commonly used in related benchmarks on InnovatorBench. Specifically,
consider Claude Sonnet 4 (Anthropic 2025), GPT-5 (OpenAl 2025), and GLM-4.5 (Zeg al, 2025). Kimi-K2 K2 (Team et al. 2025a). Qwen3-32b (Yang et al. 2025a) using a ReAct-style agent Yao et al. 2023). The agent has the fundamental thought and action capabilities, augmented by a summarization capability. When the context length nears the model's maximum, the agent will summarize the earlier half of the context. All models wrapped as agents and executed inside Docker container on Ubuntu 22.04 with 800 GB of memory. The agent can also, via a cluster HTTP service, dispatch additional compute to server(s) with 8x 80 GB GPUs and 1600 GB of memory each, with the number of servers allocated varying by task. We also provide a clean working directory containing the relevant data, a starter code repository, and the task description for each task. Data Construction and Data Augmentation can connect the internet. We disable the web search and browse tools in other tasks.
Table 2: Performance comparison on various LLMs when tested against various research do- mains. Final Score: last submission score; Best Score: highest achieved score. Details of all research tasks can be referred to Appendix C
Claude Sonnet
GPT-S
GLM-4.5
Kimi-K2
Qwen3-32h
Research Dumain
Final Score
Best Score Final Scure
Best Score
Final Score
Best Scure
Final Score
Best Score
Final Score
Best Score
Data Construction
25.47
26,88
8.41
8.41
15:29
22.65
14.01
14.08
0.00
0.00
Data Filiering
30.89
31.47
8:97
9.48
5.16
5.36
7.39
7.97
0.00
0.00
Data Augmentation
0.00
25.49
2.47
Loss Design
12.98
12.98
0.04
2.74
7.63
7.63
0.00
0.00
0.00
0.00
Reward Design
11.56
11.56
0.00
0.00
0.00
0.00
3,23
3.23
0.00
0.00
Scaffold Constraction
36.63
37.74
60.07
60.07
3.33
3.33
3.33
0.00
0.00
Weighted Average
24.01
24.54
12.04
12,52
11.85
13:35
5.35
5:45
0.00
0,00
5.2
MAIN RESULTS AND FINDINGS
As demonstrated in Table we compare three agents across six research-oriented tasks and report both the final and best scores achieved. Overall, all the agents get non-zero scores, which show they have the potential to handle code-based research tasks. Claude Sonner 4 demonstrates the most superior performance among its counterparts, attaining the highest average final score and best
7

---

Published as a conference paper at ICLR 2026
score, and leading four of six tasks. GPT-5 and GLM-4.5 yield middling results final score and best score, respectively. Qwen3-32b gains O, because its context window too small and often omits important information when summarizing. Besides, also obtain the following findings:
All LLMs have relatively higher scores on data-related tasks than on algorithm-related tasks. This difference arises from the nature of these tasks, tasks such data construction, filtering, and augmentation are inherently more robust: is relatively tolerant minor noise. For example, the agent can gain a relatively high score in data construction as long as find the data with the same topic. In contrast, algorithmic design tends to be more brittle; imperfect reward or loss functions can lead to catastrophic failures like gradient explosion or systematically flawed policies.
It is hard for models to use appropriate tools in algorithm-related tasks. We discover that Claude Sonnet 4 performs relatively better than other LLMs on loss/reward design, primarily due to its reliable tool use. Trace inspection reveals that GPT-5 enters a
loop once training begins, causing early termination, while GLM-4.5 wrongly specifies critical tool parameters some- times and stalls before training starts. Kimi-K2 cannot generate correct code in most cases. How- ever, Claude Sonnet consistently produces executable code and correctly suspends activity during training without intervention. These findings suggest that reliability in tool-grounded execution is the key determinant of success in loss/reward design tasks.
GPT-5's code is more robust in Scaffold Construction. GPT-5 excels notably in scaffold con- struction, achieving score 60.07, which raises its overall average to 12.04. Analysis shows its generated scaffolds most robust, attributable three key design choices: explicitly restating the options provided the prompt to prevent invalid selections, allowing up to three retries instead of immediately resorting fallback answer upon timeout, and enforcing strict output format to reduce evaluation failures caused by formatting issues.
5.3 PERFORMANCE OF MODEL. WITH GROUND TRUTH HINT
Table 3: Effect of hint provision on agent performance across research domains. Comparison between Claude Sonnet 4 with and without hints. Final Score: last submission score; Best Score: highest achieved score; Execution Time: agent runtime (hours); Cost: monetary expenditure (USD).
Claude Sonnet w/ Hint
Claude Sonnet
Research Domain
Final Score
Best Score Execution Time Cost
Final Score
Best Score
Execution Time Cost
Data Construction
15.21
19.80
1.78
25.56
25.47
26.87
3.24
33.09
Data Filtering
16.87
20.02
6.97
32.15
30.89
31.47
4.80
32.57
Augmentation
1.00
1.00
3.71
26.03
22.73
22.73
4.48
30.70
Loss Design
22.65
25.32
9.05
45.11
12.98
12.98
6.32
34.78
Reward Design
15.06
15.06
6.22
41.69
11.56
11.56
9,23
46.14
Scaffold Construction
21.04
27.71
3.73
23.22
36.63
37.74
6.43
28.65
Weighted Average
13.88
16.67
4.87
30.86
24.01
24.54
5.13
32.92
Table 3 compares model performance with and without ground-truth hints. Hints substantially im- prove performance in Loss Design and Reward Design. These domains are inherently more ex- ploratory nature, requiring the agent to devise novel solutions based on test data and algorithmic understanding, With the solution provided, the agent shifts from exploration to implementationn fo- cusing on replicating a known approach rather than inventing one, thereby increasing success rates.
In contrast, performance declines in Data Construction, Filtering, and Augmentation when hints are supplied. Although the ground truth is provided, the model's coding proficiency becomes a bot- tleneck mentioned in $5.4 The model's tendency to rigidly replicate the hint introduces imple- mentation errors-particullle particularly in script generation- -wheee minor mismatches critically undermine functionality. Consequently, the presence of the hint actually degrades performance compared the symbolic method used by the agent when no hint provided.
These results underscore that strong performance in research-oriented tasks demands both creativity and code implementation ability of the agent. A deficiency either high-level reasoning or low- level coding significantly limits overall effectiveness.
8

---

Published as a conference paper at ICLR 2026
5.4
CASE STUDY
Impatience
(b) Resource Mismanagement
The tmaining costs hours The agent still has 21 hours.
Observation:
1601
(83e02357004) Berning step 2: 17%
1.40//.00
Action 174:
"coepstor_ip "Agpus rerrs Lipe, "consoos"" "cd /verispacer task/scripp ipts
("nase"; conmest" "ergamente"s ("reectton Littt "textn",
proto 45%
9.00/20.0
(hffersereral terton-panal
181:12-58:40, 3.78//38
Think:
Action 229;
The second trainang is progressing very Loo, Siver taat have Limited
("asselt conmuvst, "angymentsfs ("rasstion lett Teaal", "cooputer loft sages 1he peat tooh, "ener andtt hed pwn borbocace/tadt wtw.ee//ttt//iiie torpost tet//////llle 2ltt- fuctary de (Dzzh Ave sapace
time and only 3 eval vite resiane, sheal Tor tryling aiffeneer
and consider tryling thire, stapler app noxch that night train raster.
Observation 231 (obscrvation Gereesonnuuputrrci
Action:
"guu_ computer 1p 1wppe
S"nasehh PRa1 acosion precosses") "arguacncs"s t"s"
torch. OutoMearryttrr:: cuDA cet menory. Trasd to allocate 192.00 capacity
(c) Selection of Suboptimal Library
(d) Template-uused Reasoning
Action (create ffle::
Action file)::
from trarsforme wers seport AutoTakentzer, AutoMode thottaus LLE class OecressistesCleree
def create exnnnced trainiag ditalorigiast dato, outeat aathil
reaso Ling temo labes ('Lt ane this step by staps Srlan. Parss, ELLL soentity the kay information gaven in the problen. vaz. Roxt, 211L
protrainenl 1nlT pathy Tords Toreh Toat
ypely-153. Then, analasia, prtectples fanowor), deras
dovico_sap= "usto", trust_ www.oee cods--rrel
irputs - self, toicniaeripreeot return, tonserse"pp" "pety truncatiionTrue, Nox longth-18241
for 1, late in cruecratetrossoning templaateet emhanced Litn
with
terch-ma_gracll:
"instriction") m'satve this scientific reasening protlee stop by steps traetsttooeess
ourpacs selr anterrrerrtte
inpets. 1.mporatured:.1.
*Arput": "outpaa"i training wop.aa.oooddddiiiooooo ontorcooiin
Figure 4: Four representative cases of agents' actual failures.
Impatience. As shown in Figure a). the training run takes about 10 hours; at that point, the agent knows it still has roughly 21 hours of budget. It is sufficient to wait for completion rather than terminating the process. However, the agent wants to find a more efficient way train the model and kill the training process, which causes sub-optimal result. The objective mis-specification and shortsighted piccision-makingg reflect the agent's impatience.
Resource Mismanagement. Figure 4b) demonstrates that the agent first launches an inference script with one GPU; 55 steps later, on the same computer, it launches a training script that requires all GPUs, causing resource contention. The agent no longer finds that an inference job was already active after more than 50 steps, shows the LLM's weakness in degraded memory and attention
Selection of Suboptimal Library. Figure 4c) depicts that the agent systematically opts for scale- mismatched implementations: it continues to run inference with Transformers in high-throughput settings instead of adopting the more efficient vLLM. This is because the time-budget constraint does not provide a direct, learnable feedback signal that rewards efficiency, so fails to shape the agent's 'ss decisions. It may also be attributed to the lack of training data for the optimal library, like vLLM, since the optimal library is relatively new.
Template-based Reasoning. Figure 4 d) shows that when synthesizing chain-of-thouuht (CoT) rationales for QA data augmentation, the agent often instantiates a highly templated, semantically vacuous reasoning pattern and concccnnnctts the question and answer, rather than reasoning from the problem's 's actual semantics. We find that this pattern often appears after the agent fails to generate a correct CoT via VLLM. The agents can't figure out why it needs to synthesize CoT and just do it mechanically. Although this case shows the agentic ability, it also reflects the agent's lack of understanding of high-level intent.
5.5 TEST-TIME SCALING PERFORMANCE
Figure 5 compares test-time scaling between InnovatorBench and PaperBench. The score represents the average score up to that time point. For example, if the agent eval its result at 1,5, hours in task I and 1,,2 hours in task 2. The score of hour 16 will be the average score of hour 13 in task I and hour 10 in task 2. As shown in the figure, agents achieve their best performance on PaperBench in approximately 1.75 hours. but require over 11 hours InnovatorBench. indicating its greater complexity. This disparity arises because complex tasks like Data Augmentation and Reward Design involve extended training phases, making them more time-intensive than tasks like
9

---

Published as a conference paper at ICLR 2026
PaperBench Saturation Point
6.5x longer
InnovatorBench Saturation Point
(a) InnevatorBench
Averae Seare
30-
Replication Soore
0.3
(b) Paperbench
0.2
of Allie
0.1
0.0
16
32
Working Time (Hoorrs)
Figure 5: Test-time scaling: InnovatorBench vs. PaperBench (Starace et al., 2025). Paper-
Bench's 's result comes from the original paper. Agents require about 6.5x longer test-time to reach the saturation point InnovatorBench, highlighting that benchmark's difficulty stems from the need for extended runtime before performance plateaus.
Data Construction. This trend shows that task complexity increases, environment interaction costs dominate the overall working time. Since the time costs can reflect the difficulty of the tasks, we believe that InnovatorBench more difficult than PaperBench and will the next generation of the code-based research benchmark.
CONCLUSION
In conclusion, this work introduces two key contributions to the development of Al research agents: Innovatorßencch comprehensive benchmark for evaluating end-to-end LLM research tasks, and ResearchGym, an extensible platform that supports large-scale, long-horizon experiments and real- istic research workflows. InnovatorBench goes beyond basic task reimplementation, offering rig- orous framework that evaluates agents' ability to address complex LLM research challenges across multiple dimensions. This emphasis innovation, adaptability, and creative problem-solving sures a more comprehensive assessment AI research agents. Empirical results using leading LLMs reveal promising capabilities in code-based tasks, but also expose weaknesses in reward de- sign, resource management, and long-horizon planning. Together, these contributions provide foundation for rigorous, real-world evaluation of AI agents, supporting their development effec- tive tools for scientific discovery.
ETHICS STATEMENT
This work adheres the ICLR Code of Ethics. this study, no human subjects or animal ex- perimentation were involved. All datasets used, including the InnovatorBench we proposed, were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care to avoid any biases or discriminatory outcomes in our research process. No personally identifiable information was used, and no experiments were conducted that could raise privacy or security concerns. We committed maintaining transparency and integrity throughout the search process.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. The ex- perimental setup, including training steps, model configurations, and hardware details, is described in detail in the paper. We have also provided full description of our InnovatorBench and Research- Gym to assist others in reproducing our experiments. We believe these measures will enable other researchers to reproduce our work and further advance the field.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Anthropic. Introducing claude 4, 2025.
URL.
claude-4 Accessed: 2025-09-22.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering. arXiv preprint arXiv:2410.07095. 2024.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu. Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu- Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery, 2024. URL
hitts://aarivv www.xiiv.og/aaao. abs 24 410 05080
Xinrun Du, Yifan Yao, Kaijing Ma. Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling Ilm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14733 2025.
Nicholas Edwards, Yukyung Lee, Yujun (Audrey) Mao, Yulu Qin, Sebastian Schuster, and Najoung Kim. Rexbench: Can coding agents autonomously implement ai research extensions? arXiv preprint, 2025.
Dayuan Fu, Jianzhao Huang, Siyuan Lu, Guanting Dong, Yejie Wang, Keqing He, and Weiran Xu. Preact: Prediction enhances agent's planning ability. arXiv preprint 24Xiv:2402.11552 2024a.
Dayuan Fu, Biqing Qi, Yihuai Gao, Che Jiang, Guanting Dong, and Bowen Zhou. Msi-agent: In- corporating multi-scale insight into embodied agents for superior planning and decision-making. arXiv preprint ar arXiv:2409, 16686, 2024b.
Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng, Wei Wang, Jingang Wang. Xunliang Cai, and Weiran Xu. Agentrefine: Enhancing agent generalization through refinement tuning. arXiv preprint 2:Xiv2251..01000 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. arXiv preprint arXi::407.221783, 2024
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah Smith, and Ranjay Krishna. Visual sketchpad: Sketching visual chain of thought for multimodal lan- guage models. Advances in Neural Information Processing Systems, 11111339333333317 2024.
Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T Truong, Weixin Liang, Fan- Yun Sun, and Nick Haber. Researchcoddbench: Benchmarking llms on implementing novel machine learning research code. arXiv preprint arXiv:2506.02314. 2025.
Mohan Jiang, Jin Gao, Jiahao Zhan, and Dequan Wang. Mac: live benchmark for multimodal large language models in scientific understanding. arXiv preprint arXiv:2508. /5802, 2025.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-hench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL htthttts///ppeeeeiitii
net/ Sett/Eorum?id-y30 6yNQM66
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095 2025.

---

Published as a conference paper at ICLR 2026
Patrick Tser Jemn Kon, Jiachen Liu, Xinyi Zhu, Qiuyi Ding, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungji Lee, et al. Exp-bench: Can conduct ai research experiments? arXiv preprint arXtv:2505.24785, 2025.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Keyu Li, Mohan Jiang, Dayuan Fu, Yunze Wu, Xiangkun Hu, Dequan Wang, and Pengfei Liu. Datasetresearch: Benchmarking agent systems for demand-ding -drien dataset discovery, 2025. URL
https://aas //axxiv org/ abs / 2508 06960
Yixiu Liu, Yang Nan, Weixian Xu, Xiangkun Hu, Lyumanshan Ye, Zhen Qin, and Pengfei Liu. Alphago moment for model architecture discovery. arXiv preprint arXiv:2507. 2025.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist r1-style vision- vision-language action model for gui agents. arXiv preprint arXiv:2504. 10458, 2025.
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vin- cent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. Mlgym: A new framework and benchmark for ad- vancing ai research agents, 2025. URL WRLhhtps:// http///xxwvoooggr///////
14499
OpenAl. Gpt-5: Language model, 2025. URL 2025-09-22.
gpt-
Accessed:
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL hattps://arxivg org/ abs/ 2412 15115
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024.
Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen, and Graham Neubig. Coding agents with multimodal browsing are generalist problem solvers. arXiv preprint arXiv:2506.03011 2025.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504.011 01848, 2025.
Jie Sun, Junkang Wu, Jiancan Wu, Zhibo Zhu, Xingyu Lu, Jun Zhou, Lintao Ma, and Xiang Wang- Robust preference optimization via dynamic target margins. arXiv preprint arXiv:2506.033: 03690, 2025.
Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based Imm agent: Building world models by writing code and interacting with the environment, Advances in Neural Information Processing Systems, 37:70148-70212. 2024.
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, et MI-bench: Evaluating large language models and agents for machine learning tasks repository-level code. arXiv preprint arXiv:231 1.0983, 2023.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint a arXiv:2507.20534 2025a.
12

---

Published as a conference paper at ICLR 2026
NovelSeek Team, Bo Zhang, Shiyang Feng, Xiangchao Yan, Jiakang Yuan, Zhiyin Yu, Xiaohan He, Songtao Huang, Shaowei Hou, Zheng Nie, et al. Novelseek: When agent becomes the scientist- building closed-loop system from hypothesis to verification. arXiv preprint arXiv22505.16935 2025b.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv: 2, 2024.
The Terminal-Bench Team. Terminal-bench: A benchmark for ai agents in terminal environments, com autdd-inntttttt//eeeriiiiiin -bbench
Apr 2025.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, al. Openhands: An open platform for ai software developers generalist agents. arXiv preprint mrXiv24407.66441 2024a.
Yejie Wang, Keqing Hc, Dayuan Fu, Zhuoma Gongque, Heyang Xu, Yanxu Chen, Zhexu Wang, Yujia Fu, Guanting Dong, Muxi Diao, et al. How do your code lims perform? empowering code instruction tuning with high-quality data, arXiv preprint (.Xi::249.008//11 2024b.
Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts. arXiv preprint arXiv:241 151144 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-arr natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et The rise and potential large language model based agents: A survey. Science China Information Sciences, 68(2): 2011,, 2025.
Yang Xiao, Jiashuo Wang, Qiancheng Xu, Changhe Song, Chunpu Xu, Yi Cheng, Wenjie Li, and Pengfei Liu. Towards dynamic theory of mind: Evaluating Ilm adaptation to temporal evolution of human states. arXiv preprint 2rXiv:2505.1700 2025.
Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry. arXiv preprint arXiv:2507. 16280, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 17:50528-50665 2024.
John Yang. Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang. Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang. and Ofir Press. SWE-bench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, 2025b. URL https:
//openreveew. http://foorrmm...rr........00.
John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig. Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025c. URL htt.hhttss//// /tte:////xxiii a wtgg:bb//g/b///a//////////// .11988
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICIR), 2023.
Lyumanshan Ye, Xiaojie Cai, Xinkai Wang, Junfei Wang, Xiangkun Hu, Jiadi Su, Yang Nan, Sihan Wang, Bohan Zhang, Xiaoze Fan, et al. Interaction as intelligence: Deep research with human-ai partnership. arXiv preprint 1555 2025a.
13

---

Published as a conference paper at ICLR 2026
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025b.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.0647/, 2025.
Yaowei Zheng. Richong Zhang, Junhao Zhang. Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguis- abs 2403. 13372
tics. /ttp::::///rxx
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in weal-world environ- ments. arXiv preprint arXiv:2504.0316) 2025.
Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, and Pengfei Liu. Programming every example: Lifting pre-training data quality like experts at scale. arXiv preprint 1rX1v:2409.77115, 2024.
Wenhong Zhu, Ruobing Xie, Weinan Zhang, and Rui Wang. Flexible realignment of language models. arXiv preprint 12000 2025.
14

---

Published as a conference paper at ICLR 2026
4
Task Task Count
3
3-
DA
DF
DC
LD
SC
RD
Figure 6: InnovatorBench' 's dataset comprises tasks from a diverse set of research categories. DA denotes Data Augmentation, DC stands for Data Construction, DF represents Data Filtering, LD is the Loss Design, SC denotes Scaffold Construction, and RD means Reward Design.
A EXTENDED DETAILS OF THE INNOVATORBENCE DATASET
Figure 6 shows the task composition of InnovatorBench, and Table presents the details of each task InnovatorBench.
Table 4: The introduction of the InnovatorBench
Paper
Key Description
Research Domains
DesssetRescarhh:: Bench-
Collect synthesize politics domtics-domain news summarization dataset consisting of English articles with sponding one-sentence human-written summaries fine-tuning.
Llama-3.1 L
Data Construction
marking Agent Systems
dataset discovery synthesis
for
Demand-Drivenn
48h, 8 GPUs
Dataset Discovery (Li
DataserResearch Bench-
Build a medical English-Tamil paral- lel dataset and fine-tune Llama-3,1-88 Instruct for translation.
Llama-3. 3.1mn-3..8888 I-Stt-nnsttt
marking Agent Systems
dataset discovery synthesis 48h, 8 SIGB GPUs
Construction
for
Demand- Driven
Datasel Discovery
(La
eet al. 2025)
DatasetResearch: Bench-
Build a 5K-10K woul-world document summarization Llama-3. I-SB-Instreet generate con- cise, accurate summarics.
Llama-3 1--8----nstern Instract datasct discovery / synthesis 48h, 8 80GB GPUs
Data Construction
marking Agent Systems
for
Demand-Driver
Dataset Discovery
(L]
ce 2025)
Desasctkescarch: Bench-
Fine-tune Llama-3.1-88
medical Q&A data to improve USMLE-style multiple-choice accuracy baseline to 95% target.
Llama-3, I-ststtttt
Data Construction
marking Agent Systems
dataset synthesis SDGB
Demand- Driven
Dataset Discovery
(Li
0t | 2025
Programmiing
Every
Design and implement a systematic cleaning pipeline for 100K web texts to produce high-quality data for LLM pre training.
5h, 8 80GBB GPUs high efficiency
Data Filtering
Example:
Lifting Pre-
training Data Quality
Experts Scale
(Zhou et 2024)
LTMO: Less More for
Select 800 quality math problems from 10K a training set for reasoning optmizaaiion
fixed training hyperparameter select 800 problems 48h, 16 80GB GPUs
Data Filtering
Reasoning Yo et al.
20255
Your Code LLMs Perform! Empow-
How
Filter code instruction datascts by moving contaminated samples and se- lecting top 160k highest difliculty proh. lems for clean model training.
8h.
Filtering
ering
Code
Instruction
High- Tuning with
Quality Dala
[Wang]
2024h
Supergpqa: Scaling L.M
Enhance and fine-tune Qwen2.5-7B ing curiched multidisciplinary seientific reasoning maximize set
48h, 8 80GB GPUs
Data Augmentation
Evaluation Across 285
linal model
Disciplines
Qwen2.5-7B
ot 2025
curncy.
Continued on next page
15

---

Published as a conference paper at ICLR 2026
Paper
Key Description
Reseurch Domains
MurninaMath::
The
Fine Qwen2.5-7B B-BB Instruct mathematical reasoning prohlems using dataset enhancement and auxiliary mod- cls to maximize test accuracy beyond
48h, 8XBGGB GPUs
Duta Augmentation
Largest Public Dataset A14Maths with 860k
final model trained from Qwen2.5-78
Pairs of Competition
Problems
Solutions (Lt al. 2024 2024
DeepReseurcher:
Scal-
Synthesize scarch-calsanced seasoning
24h. 8 80GB GPUs fixed inference script final model trained from Qwen2.5-7B
Data Augmenttation
ing Deep Research via
data with Qwen-2.5-72B and line-tune Qwen-2.5.77 2.em-2.5777 7B maximize per-
Reinforsement Learning in Real-world Environ-
lormance.
Zheng
2025
Towards Dynamic The- ary of Mind: Evaluating
Synthesize dynamic Theory-of-Mind training data and fine tune Qwen2- 7B- Instruct predict evolving mental in multi-step social sccuarios.
Data Augmentation
48h, 8 80GB GPUs Qwen2-7B-Instruet SFT only
LLM Adoptation Tem-
poral Evolution of Hu-
2025
MAC:
A Live Bench-
Augment scientific image-text datasets and fine-tume Qwen2.5-VL..77 I--srrcttt improve maltimodal reasoning journal cover visual understanding.
24h, 8 80GB GPUs Iinal model trained from
Data Augmentation
mark For Multimodal
Large Language Modelss
Dov-2.5-VI--7B-Ime
Scientific Under-
standing Jiang
2025
13
Flexiole Realignment of (Zhu
Implement a DualAlign algorithm in LLaMA-Pactory efficiently realign DeepSeck RI Direek-----iiiilee------- with DeepScaleR Preview- 1.5B improved reasoning efficiency. Implement GRPO variant Verl framework to prevent entropy collapse during RL training while maximizing curacy mathematical reasoning lusks.
fixed training hyperparameter 48h. 8x GPUs
Loss Design
Language Models
2025
DAPO: Open- Source
max per training
LLM
Reinforcement
48h, 80GB GPUs
Design
Learning System Scale
Special ourput format
YYu et al. 2025
Robust Preference
Develop GammaPO algorithm that adeptively adjusts reward margins based outperform SimPO on AlpucaEval2 benchmarks.
Loss
cimization via Dynamic
24h, 80GB Modity training script only
Design
Tarper Margins
preference
0E 2025
16
Training LLMs to Reason and
Search-RIF
Design and implement reward function in the framework to train Qwen-2.5- for search-augmente reasoning tasks with exact match evaluation.
48h, 8 80GB GPUs
Reward
final model trained from Qwen2.5.30
Dosign
Leverage Search Engines
with Reinforsement Jin
Learning
2025
17
GUERI:
A General-
Implement unified reward function in Verl framework to train GUT grounding models for multi platform action predic- tion exceeding SpreenSpot baseline ac- curacies.
24h, 8 8BGB GPUs final model trained from
Reward
RI-Style Vision-
Design
Language Action Model
Qwen2.5-VLL VL-7BB
For GUT Agents
(Lan
2025
DeepRescachher.
Scal-
prompt-based deep research agent using GPT-4, and weh tools to handle complex multi-stop research questions with accurate source-attributed unswers.
24b, GPU
Scaffold Construction
ing Deep Research vin
GPT.4.1 research
Reinforsement Learning Real-world Enviroa- ments [Tu et 2024
GPT-4.1 -mii browsing
19
Visual SKETCHPAD:
Develop efficicnt multimodal mathemat- reasoning workflow using GPT-lo to solve geometry. graph connectivity. maxflow, convexity problems with structured ISON ourpots.
12h, 0 GPU GP 1-44 via API
Scaffold Construction
Sketching as Chain of Thought for
Multimodal Language Models [Hu 2024
Visual SKETCHIAD::
Develop efficient visual reasoning sys- tem using GPT-4o and visual tools to biick.viscorr jigsk, and blink depth tasks with structured outpuls.
14 GPU GPT-4o via API
ScafTold Construction
Sketching as Visual
Chain Thought for
Multimodal Language
Models 2024
B DATASET CURATION AND BENCHMARK CONSTRUCTION DETAILS
InnovatorBench Construction We first design 20 raw tasks based on the following principle:
(1) The task can be reapplied; the result is aligned with the original paper.
(2) The task result can gain significant improvement in days
(3) The tasks evaluate the different abilities of LLM Agents in LLM Research.
16

---

Published as a conference paper at ICLR 2026
(4) The task uses common models like Ilama3.1 (Grattafioni al.) 2024), Qwen2.5 (Team 2024
Guo et al. 2025 Owen et al. 2025), or Qwen2.5-VL 5LL [ai et al.) 2025), etc.
There are 13 annotators to annotate InnovatorBench. Each task costs from days to 2 weeks for the annotators to construct the workspace and evaluation code. After collecting 20 tasks, 2 authors further organize these tasks, workspaces, and evaluations into ResearchGym. Each annotators were asked to reapply the original paper and gain the reference score, and the baseline score often comes from the base model's result. After obtaining these two scores, the annotators were asked to design the score function based on these scores, usually a linear interpolation. The score function has principles: (1) the baseline score should result in final score of O, while if agents gain score higher than the baseline score, their final score should not be O, and (2) the reference score should be about 80.
After testing the first version of InnovatorBench, we found that even the most advanced model can't 't generate and save the SFT data correctly, as mentioned in Figure + so we just changed the task little bit to reduce the difficulty in the Data Argument by adding some relevant scripts.
C DETAILED EXPERIMENTAL RESULTS
C.I MAIN RESULTS
Table 5: Performance comparison of each task on various models when tested against various evaluation metries. FS denotes Final Score, BS represents Best Score, ET stands for Execution Time in hours, and Cost is the monetary spend USD.
Claude Sonnet 4
GPT-5
GLM-4.5
Kimi-K2
Task
FS
BS
ET
Cost
FS
BS
ET
Cost
FS
BS
ET
Cost
F$
BS
ET
Cost
1
19.05
19.05
1.53
27.02
0.00
2.41
16.09
20.51
20.51
0.84
1.63
0.00
1.95
4.92
2
39.35
39.35
4.26
47.41
0.00
0.00
0.79
4.16
13.26
13.26
0.83
1.91
0.20
0.20
3.57
5.97
3
17.41
18.23
4.98
30.42
0.00
0.00
1.44
7.49
18.85
18.85
2.06
4.31
14.70
14.98
4.25
4.40
4
26.09
30.87
2.21
27.51
33.62
33.62
3,98
19.42
8.55
37.97
1.37
5.70
41.16
41.16
2.53
3.40
5
5.00
5.00
0.54
6.86
13.36
14.88
0.60
2.37
5.00
5.00
0.16
0.65
12.76
13.12
0.64
1,40
6
81.37
81.74
11.45
65.09
0.00
0.00
1.58
4.28
0.00
0.00
5.32 11.11
5.00
5.00
3.33
6.14
7
6.29
7.66
2.41
25.75
13.55
13.55
1.28
3.94
10.48
11.08
0.56
3.77
4.40
5.80
1.57
2.03
8
27.33
27.33
4.83
21.57
0.00
0.00
6.32
36.15
37.30
37.30
1.54
4.87
7.33
7.33
3.38
3.98
9
0.00
0.00
7.08
36.01
0.00
0.00
2.19
4.75
0.00
0.00
0.64
3.95
0.00
0.00
11.57
7.36
10
0.00
0.00
2.34
41.15
0.00
0.00
1.06
3.86
0.00
0.00
0.56
3.95
0.00
0.00
0.80
0.46
11
86.34
86.34
5.49
32.58
0.00
0.00
3.99
4.77
5.00
5.00
2.93 7.53
5.00
5.00
6.62
6.24
12
0.00
0.00
2.65
22.19
0.00
0.00
0.86
2.99
85.15
85.15
8.45
8.05
0.00
0.00
1.29
4.46
13
0.00
0.00
4.97
58.67
0.00
0.00
1.59
12.12
0.00
0.00
3.31
10.74
0.00
0.00
7.28
10.68
14
4.50
4.50
10.40
26.29
0.12
8.21
16.23
80.50
0.00
0.00
0.68 5.47
0.00
0.00
4.70
12.20
15
34.44
34.44
3.58
19.38
0.00
0.00
2.27
6.60
22.90
22.90
4.71
18.78
0.00
0.00
3.48
8.49
16
0.00
0.00
8.91
51.67
0.00
0.00
1.74
8.67
0.00
0.00
0.63
3.31
0.00
0.00
1.86
6.60
17
23.11
23.11
5.66
32.81
0.00
0.00
2.04
9.63
0.00
0.00
0.13 1.20
6.47
6.47
3.79
6.33
18
16.67
20.00
14.01
44.25 0.00
0.00
1.26
6.56
0.00
0.00
1.34
5.47
0.00
0.00
2.42
1.96
19
63.95
63.95
2.18
23.30
92.45
92.45
1.77
3.70
10.00
10.00
0.64
3.78
10.00
10.00
5.66
2.87
20
29.26
29.26
3.08
18.41
87.76
87.76
5.05
36.38
0.00
0.00
0.99
3.04
0.00
0.00
0.68
3.47
Avg.
24.01
24.54 5.13 32.92
12.04 12.52 2.92 13.72
11.85
13.35
1.92 5.68
5.35
5.45
3.57
5.17
The benchmark evaluated the performance of three large language models - Claude Sonnet 4, GPT-5, GLM-4.5, and Kimi-K2 multiple tasks with varying priority levels. For each task, the metrics recorded include the final score, the highest score, and the runtime (in hours). This analysis focuses comparing model effectiveness (scores) and efficiency (time cost).
17

---

Published as a conference paper at ICLR 2026
C.2 PERFORMANCE MODEL WITH GROUND TRUTH HINT
Table 6 presents the performance, execution time, and cost results between Claude Sonnet 4 with the hint and Claude Sonnet 4 without the hint.
Table 6: Performance comparison between Claude4-hint and Claude4 on evaluation metries, runtime, and cost.
Claude4-hint
Claude4
Task
Final Score
Best Score
Run Time
Cost
Final Score
Best Score
Run Time
Cost
6.52
18.52
0.68
17.97
19.05
19.05
1.53
27.02
8.68
8.68
0.66
29.87
39.35
39.35
4.26
47,41
8.53
12.88
3.85
27.62
17.41
18.23
4.97
30.42
37.10
39.13
1.93
26.78
26.09
30.87
2.21
27.51
5
12.80
13.24
1.75
12.74
5.00
5.00
0.54
6.86
29.49
34.49
14.13
61.68
81.37
81.74
11.45
65.09
7
8.32
12.32
5.04
22.04
6.29
7.66
2.41
25.75
8
0.00
0.00
6.99
43.19
27.33
27.33
4.83
21.57
0.00
0.00
6.68
31.10
0.00
0.00
7.08
36.01
10
0.00
0.00
2.78
24.63
0.00
0.00
2.34
41.15
5.00
5.00
0.67
15.42
86.34
86.34
5.49
32.58
0.00
0.00
1.44
15,80
0.00
0.00
2.65
22.19
13
0.00
0.00
3.33
65.13
0.00
0.00
4.97
58.67
14
29.37
37.39
18.34
34.88
4.50
4.50
10.40
26.29
15
38.57
38.57
5.49
35.32
34.44
34.44
3.58
19.38
16
0.00
0.00
4.71
43.57
0.00
0.00
8.91
51.67
17
30.13
30.13
7.73
39.81
23.11
23.11
5.66
32.81
18
0.00
20.00
7.48
35.77
16.67
20.00
14.01
44.25
53.12
53.12
0.54
13.65
63.95
63.95
2.18
23.30
20
10.00
10.00
3.19
20.26
29.26
29.26
3.08
18.41
Avg.
13.88
16.67
4.87
30.86
24.01
24.54
5.13
33.92
D EXTENDED INNOVATOR BENCH BENCH EXAMPLES
Example of task 14's description
## Motivation
Reinforcement Learning (RL) training for Large Language Models often suffers from *eettoopy collapse**, where the model's s output distribution becomes overly deterministic early in training. This severely limits exploration and prevents the model from discovering diverse reasoning paths. Understanding and mitigating entropy collapse crucial successful long-form reasoning tasks where exploration of different solution strategies is essential.
## Task
** Your task is to implement new strategy for GRPO in language model reinforcement learning in order get the highest accuracy and prevent entropy collapse.es
18

---

Published as a conference paper at ICLR 2026
We provide a GRPO algorithm for you as background knowledge. For a specific question-ansee answer pair S(q. a)S, the behavior policy $ pi-Attheta") mathht(oodd)} samples group of $G$ individual responses 5 [--i) the group-level rowards $ (iiRi))-i
Then, the advantage of the i-th response is calculated by normalizing
S$
(nabla_) \thela J. (.(GROO)( (ma_hbbEE)-((. a) /si D. \ooii)-(==1"" ssim \pi-(),heta.
(heta
(olo\))))))))))) /leftf Adfrac (1) (G) \sum_(i=1}"G {dfrac (I) (o \summ(t=
1 [1] (atthallll (eett \iin Aleft( ((heta)) (ha((A)(ii.))
((e 1
/epsilon, (epsilon) har(A] (epsilo))hat((AA--- A-- t) (right) \right) (night]
S$ where Sr {i.}} (drrac (p (hhtaa(o (i. t)/q.0 (i. (tp))) \theta ( (mmathitoold) Yo {i. HH/q.
{i. <t))sS is the importance sampling ratio.
#*Now, you need to implement variant of GRPO to get the highest accuracy and prevent entropy collapse.**
You should work under the '/oorkspace/task and 'woorkspac/ddtaa" directories.
You need the original GRPO rollout. Only modify the advantage/loss compute part in // algos.py". , do not modify other files in verl.
After you implement the algorithm in wvortspace/askkrrpssittreee/// task/scripts/train.sh train the model based your code; will also show the dev results in the training log. (You can train multiple times, just select the best one to evaluate.)
you can use '/workspace/
- Do not modify the parameter in //vakkspacctaasssrriistttaaiti except test freq
Finally, you need to select the best checkpoints in your training, using //wakkspacchaak/seripss//// inference.sh <model pptth>> to generate wwwckkaaccdda//uttuuuaaaaaaraaae workspaccedaaa/nuput///aaaeccec parquet"
and '/
### Output Format Requirements:
**File Location+:: /vookkspacodaaalluuppussmmaa wwwptatthhssssssssseeeeeessseese pearuts..p.ruuuttt
**Filc Formate4: Parquet format containing model responses
**Required Fieldsææ;
The parquet file should contain:
"dd (int): Unique identifier for each test sample (must follow order in test set) ""output"" (string): The model's complete response. including reasoning and final answer
Reequire Output Format for each response:**
<rasoning process>>
Answer: boxed Your Answer)
#+Logits Filese: V/workspaceddaaaiouuuuusst results logits.pt*
Torch tensor file containing logits entropy calculation
Should he list of tensors, for each test sample
Each tensor should have shape "[sequence length, vocab size]' where:
"sequence length:: length""elength' Number of generated tokens for that sample
"recabsizee" Size the model's 's vocabulary
IMPORTANT Notes:es
Use wwwokkxpccccsssssrpps//eeeeenenn for inference #edo not modify this scripte*
19

---

Published as a conference paper at ICLR 2026
The reasoning process should demonstrate clear mathematical thinking
Final answer must be enclosed in " format
Your task to implement a new strategy for GRPO to prevent entropy collapse and achieve the highest accuracy
The evaluation will check these two files give the final score based both accuracy and entropy metrics.
## Data
Train sete#: Stored in /Noorkspacc/dnn///atstttttaaaha http://tttttttttttt//aataaaaaaae htt//rain.paquueee
data source: 'mathök"
prompt: the math question
reward model: the ground truth answer of the math question
Dev set*#: Stored in /vadkppacedauudaaasss//aaat Sk/dev.parquet
data source: 'GSM8k' & 'AMC23'
prompt: the math question
reward model: the ground truth answer of the math question
Test set*#: Stored in hatpa//acccctaaaaaaaece///aaaaas
#hhcckpointt::: Stored in "vporpppaccdaaaccccppisssseeeeE0
## Constraint
Training time constraint: 24 hours.
Evaluation time constraint: 10 minutes.
You 80GB GPUs.
You only have 3 chances to evaluate your submission.
Total time limit: hours.
## Evaluation
**Evaluation metrics:**
1. Accuracy: Measure the accuracy of the variant policy the test set the end of the training. Entropy Analysis: Track entropy the variant policy along the training process.
**Success criteria:**
The accuracy of variant policy on MATH500 should be least higher than the baseline policy. Try your best to the highest accuracy! The average entropy of the variant policy should reasonable range.
3. The final score will be the accuracy score entropy score 100
## Environment
have setup conda environment for you named /workspace/condat; and we have activated the environment.
## Scripts
You can generate scripts the 'wwrrkspaceftask/crrisss directory. #sshould not modify scripts ** that are originally the /woorkppccctaskksrriisss directory.
20

---

Published as a conference paper at ICLR 2026
The following scripts provided you; do modify them:
Given model path of verl checkpoint
which a directory containing multiple 'model.world... world_size.8 rank (rane..mbeenbbrr pt' files, you can use this script to merge the model weights into HuggingFace format.
-Input:
---local.dir" The path of the verl checkpoint.
Output:
The checkpoint in HuggingFace format.
E LIMITATIONS AND FUTURE WORKS
Despite the advancements brought by InnovatorBench and ResearchGym, there are several areas for improvement in future work:
Task Diversity InnovatorBench currently covers a limited of research tasks. Future work could expand the benchmark to include more diverse, interdisciplinary challenges that reflect real-world scientific research.
Generalization of Agents Al agents still show performance variation depending on the model. Fur- ther research is needed improve their generalization across different research tasks and improve transfer learning for broader applicability (Fu et al. 2025 2024a b).
Human-Al Collaboration The current framework largely focuses on autonomous Al agents. Future work could explore hybrid human-Al workflows, incorporating real-time feedback and collaboration for more realistic research
YYe al. 2025a
F SUPPORTED ACTIONS OF RESEARCHGYM
We referred to the design of OpenHands (Wang et al. 2024a) and adapted it to the multi-GPU, asynchronous, and other environments required by ResearchGym.
In our settings, the agent is not allowed to run nohup or kill commands, since we found the agent would like to kill the script support the script).
control (because it seems like a "unrelated"
COMMAND ACTIONS
The command actions manage terminal session lifecycle and interaction, including session creation, listing, command execution, input/output handling, status inspection, and session termination. The following functions provide comprehensive capabilities to control and operate remote or local com- puting sessions.
def create session action (computer_Aps str 'Localhost', session str
None, attp_port: int = None, use proxy: bool True) -> Dict [str,
Any]:
"**Create new terminal sessicn on the computer speci fied by computer_ip'.
This function initializes connectivity via 'http_port"' and "use_proxy 1. Use 'use huse_proxy=Faise for http//ooolooss_oooooooooooo machines and use_proxy=Irue* for 'gpu' machines.
Args:
21

---

Published as a conference paper at ICLR 2026
computer.ip(stz]: The IP address of the computer. Default is localhost',
session.id(stz]: Unique identifier of the target session. If absent, a new
session is created and 'computer_ip'. Default is None.
'session id is assigned the host
http--por [int]: The HTTP port
connect to the session.
use proxy [bool]: Whether to use a proxy for connecting to the session. Set
"use_proxy=False" for 'cpu' and "localhost_cpa' computers, and
Luse_proxy=Trrr for 'gpu" computers. Must align with your network topology
or the connection will fail, Default is True.
Returns:
Dict Any]: Dictionary containing session creation status and information.
def list sesstsssssios.ee action (computor_ip: str = None) -> [ict, Any]: : ***List all existing sessions.
Key <ccmmuteer "ccoppter_..rrcessssiin ip>:<session idv' the output refers to the session <session ids on <computer_ip>,
Args:
computer : The IP address of the computer. If None, 1ists sessions
all machines. Default is None.
Returns:
Dict [atr,
Any] Dictionary containing information about all active sessions.
nmm
def run.comman..cttion
command: str,
computer_ip: str = 'Localhost', session_id: str None,
http_port: int None,
valt_for_completion bool False,
usc_proxy: bool = Truc
-> [ict Anyl:
""*Execute a single bash command in the session identified by id''.
If the session does not exist, it will be created and bound to the target host (determined by 'computer_ip") and will be connected via 'http_port"" and 'use_proxy', Only command may run concurrently per session.
Args:
command[stx]] Shell (bash) command to execute in the target session's s
working directory and envirenment. computer ip[str]: The IP address of the computer. Default is hocahhost'.
22

---

Published as a conference paper at ICLR 2026
session.id(str]: Unique identifier of the target session. IE absent, a new
session is created the host determined by 'computer_ ip Default is
None.
port [int]; The HTTP port to use to connect to the session. Default is
None.
weit.for.completion [bool]: (bool] Whether block until the command finishes:
True: block up to 10 seconds; timeout the command process is killed.
False: return immediately and let the command in the background.
use proxy [bool] Whether to use proxy for connecting to the session. Set
"use_proy==False" for 'cpu" and "localhsst_ppu' computers, and set
'use_proxy=True" for 'gpu' computers. Default is True.
Returns:
Dict [str,
Anyl: Dictionary containing command execution results and status.
def seapitii..essioo..cttion (computer ip: str 'Localbost', sessicn id: str = None, input_text: str (r) -> Dict ssrr, Anyl: """Kavigate to webpage based on URL and display its content.
The environment will cache the webpage content for another action to use until perform next web_browa action.
Args:
[sl[str]: The URL to navigate to. line number [in]er The line number to start viewing from. The envi ronment
will perform nine_number to 1ine_number+100 lines of content.
Default is
Returns:
Dict [str,
Anyl: Dictionary containing page content and status information.
def get.session.co patassiio.outuu..ccci. www.ouutttciioon....oo
computer_is ip: str = 'localhost', session id: str None, start start_lines:
lines: int = 50,
end lines: int None, since_timestan float = None -> Dict[str, Any]:
""*Reriieee the output buffer of the terminal session identified by session_id'.
If Siince_timestang is provided, incremental output since that time is zeturned; otherwise, output is sliced by 1ine window "'start_line" ' required, 'ine__ optional).
Args:
23

---

Published as a conference paper at ICLR 2026
computer.ip(stz]: The IP address of the computer. Default is localhost',
session.id(stx]: Unique identifier of the target session. The session must
exist and be active. Default is None.
start lines [int]: start offset counted from the end of output
(>=2).
Effective only when "since_timestan" is not set. Usage: 'start_lines=y" only: returns the last lines.
Rith end returns the slice between 'start_ lines" and end lines'.
end lines [int]: End offset counted from the end of output (>=1). If not
specified) this tool will return content from the 'start_1ine lines the end
of the output. If specified, the slice is [start_lines, end 1od1110 lines
inclusive of 'start_linee lines', exclusive of eend lines', Default is
None
tince.timestampliicc [float]; Optional. Fetch output since this Unix epoch
timestamp (seconds, float). when set, it overrides Start_liness
and
'end_iines', Default is None.
Returns:
Dict [str, Any]: Dictionary containing session output and status information.
def session.status action (computer_ip: str = 'localhost', session_ia id: str None) Dict (str, Anyl:
***Get the status of a specific terminal session.
Args:
computer ip[str]: The IP address of the computer. Default is lca hest'
session_id[str]: Unique identi fier of the target session. If
absent, the status of the default sessien is returned. Default is None.
Returns:
Dict [str, Any]: Dictionary containing session status information.
n
def session.idle.actin cerrittioonoooopiio (computer_ig ip: str 'localhost', session id: str None) -> Dict|stx: Anyl:
"**Check it a specific terminal session is idle.
Args:
computer ip[str]] The IP address of the conputer. Default is localhost'.
dession.ddsstan The ID of the session to check whether it is running some
command whether it is idle. Default is None.
Returns:
Dict [str,
Any]: Dictionary containing session idle status information.
24

---

Published as a conference paper at ICLR 2026
def clear session buffer action (computer ip: str 'localhost', session id str Nene) -> Dict str, Anyl:
"""Clear the output outfer of specific terminal session.
The output buffer is queue of output lines, it will automatically clean if the total lines exceed 10000 lines, regardless of using this action not.
Args:
computer. [strr] The IP address of the computer. Default is localhost', session id[str]: The ID of the session to clear the output buffer.
Returns:
Dict [str,
Anyl: Dictionary containing operation status Information.
def plose.sossion.attiooo wtt.ssoon.aiiiooocooottttt..... str, session_ id: str) -> Dict
Anyl: "**Ce specific terminal session and kill all sub-processes in the session.
Args:
computer : The IP address of the computer. session id[str]; The ID of the session to cLose.
Returns:
Dict [str,
Any]: Dictionary containing operation status information.
def close all.sessionn sessions..ationn (computer_ip: str = None) -> Dict Anyl:
"**cl all sessions on specific machine all machines.
you want to close all sessicns on speci fic machine, you should set the 'computer_ip'.
Args:
computer.iplstz]: The address of the computer. None, closes
sessions
on all machines. Default None.
Returns:
Dict [str,
Any]: Dictionary containing operation status information.
def kill session processes action action(computer (computer ip: str = 'localhost', session_id: str None, force: bool False) Dict Dict[str, Any]: "*Mkill all processes on specific session.
Args:
immuterrip(stta. The IP address of the computer. Default is localhost'.
session [s[st The ID of the session to kill all processes. force [bool]: Whether to force to kill all processes. Default is False.
25

---

Published as a conference paper at ICLR 2026
Returns:
Dict [str,
Anyl: Dictionary containing operation status information.
F.2
BROWSE ACTIONS
The browse actions enable webpage navigation, viewing, scrolling, in-page keyword search, iterative result traversal, and hyperlink extraction from cached web content. Specifically, web-page woo.paggegoooo goto. action, web. webb-pagegoooe -oteeooooo_ii. wetee-oooooneeeeeeeeee web-page wereesearchhattte tively provide unified interface for interacting with and extracting information from web pages.
web.page poo--pagesccrve sogge-scroll.de sowoll.dow..actonn web.page waage-scroll.u.aee actaction, Seb--paaeseardee seaacc_next_acteee aecton and webb-pagegge..lne action
lactsss..aniio collec-
def web wwwbppageggooo www.goooooiiiooo (url: str, tine_number int 1) Dict (str, Any
1:
"" "Naaiggtte to webpage based en the given URL and display its content.
The environment will cache the webpage content for subsequent actions until another web browsing action is performed.
Args:
url[str]] The URL to navigate to.
line.number [int]: The line number start viewing from (1-indexed
).
The environment will provide content from Lime_number to 1ine_numerr+000.
Returns:
Dict [str,
Any]: Dictionary containing page contert and status information. ==
def wab.page.goto.line.cc. (line number: int) Dict Anyl: Jump directly to a specific line in the currently cached webpage.
Args:
number [int]: The 1ine number to jump to (1-indexed).
Returns:
Dict [str,
Anyl: Dictionary containing page content and status information.
def web page scroll down action -> Diet |str, Anyl : ""*Scroll down the currently cached webpage by a fixed number of lines.
This displays the subsequent 100 lines of content.
Returns:
Dict [str,
Any]: : Dictionary containing page content and status information.
def web page_ageaaa.... scroll up action -> Dict [str, Any]: "**scrll up the currently cached webpage by fixed number lines.
This displays the previous 100 lines content.
26

---

Published as a conference paper at ICLR 2026
Returns:
Dict [str,
Anyl: : Dictionary containing page content and status information.
nea
def web page search action (keyword: str, context_ ines: int +> Dict str, Anyl:
"**Search for keyword in the currently cached webpage and return surrounding context.
The search returns the first eccurrence of the keyword along with the specified
number of context lines.
Args:
keyword The keyword to search for.
context _ines [int]: Number of context 1ines to display around each match.
Returns:
Dict [str, Anyl: Dictionary containing search results and status information.
def web page wwwpppaaaa........e..eppppp
www.sh next action (criiiotcconttt___nneee lines: int 5, search_inded index: int None) -> Dict (str, Anyl:
"""Advance to the next (or specified) search result in the cached webpage.
If search index exceeds the number of matches, it wraps using modulo ari thmetic..
Args:
context. lines [int]: Number of context lines to display around the match.
search index [int]: Index of the search result to jump to. If None,
advances
to the next result.
Returns:
Dict [str, Anyl: Dictionary containing search results and status information.
def geb.page.get..inns..ctee (page_size: size: int 10, number: int 1) Dict|str, Anyl:
"*"Extract hyperlinks from the currently cached webpage.
Args:
page.size [int]] Number of links to return per page. Default is 10. page.number [int]: [ The page number results to display. Default is 1.
Returns:
Dict [str,
Anyl: Dictionary containing link 1ist and status information.
mmw
27

---

Published as a conference paper at ICLR 2026
F.3 FILES ACTIONS
The file manipulation module provides capabilities to navigate, inspect, create, modify, and search files directories. It includes editing file.edit_action, opening and navigating within files open_file file-action, goto line actton delesscrolldown.cce down.action, file silesscrol.up-. acctllion creating new files create file action, searching directories or files search dir action, search file action, find file action, listing directory contents list files action, and retrieving metadata about the current file get file gel. info_actions Together these operations provide complete toolkit for programmatic file system interaction.
def sile.edit_action (path: str, start_lines int, end_ line: int, content:
str) -> Dict [str, Anyl:
"**Edit a file given path.
The rile's Istart,end lines will be edited to the content. Remenber this edit
will change the file's Line-Linenumerr index, so do not edit
consecutively
until you use filee tools to read the now file version.
Args:
path[stz] The path to the file to edit.
start.line [int]: The starting line to be edited (including).
end line [int] The ending line to be edited (including).
content [str]: The content to be written or edited in the rile. It will
replace the content between 'start' and ennd lines.
Returns:
Dict[str, Anyl: Dictionary containing edit operation status and information.
def open file file action (path: str, iine_number: int = 1, context_linea. Optional [int] None) -> Dict [str, Any]:
"**Open a file and display its content around a specific line.
The envirenment will cache the file content for another file action to until perform next open file action.
Args:
path[str]] The path to the file to open.
line number [int]: The line number focus on (1-indexed). Default
is 1.
context_linee [Optional [int]]: Number of lines show context. Default is None (uses default window size).
Returns:
Dict [str,
Any]: Dictionary containing file content and status information.
def goto line action [line_nnmber int) Dict [str, Any]: "=mJump to specific line in the currently open file and show the content around the line.
Args: line number [int]: The line number to jump to (1-indexed).
28

---

Published as a conference paper at ICLR 2026
Returns:
Dict [str,
Anyl: Dictionary containing file content and status information.
def file scroll down action () => Dict [str, Anyl: ""rocrll down 100 lines in the currently open file.
Returns:
Dict [str,
Anyl: Dictionary containing file content and status information. ***
def file www.rooww.p...cionn.oomm -> Dict [str, Anyl:
scroll 100 1ines in the currently open file.
Returns:
Dict fict[str,
Anyl: Dictionary containing file content and status information.
=*.
def create. fiw..actionnnncmmmo (filenamo: str, content: str "") Dict [str, Any
1:
"**Create new file with the specified content.
It will also replace the original file if it already exists.
Args:
filename [str]: The name/path of the file to create.
content [str]: The content to write to the new file. Default empty string.
Returns:
Dict [str,
Anyl: Dictionary containing file creation status and information.
def search dir action (search_temn str, dir_path: (.7%) Dict
Anyl:
"**Search for text pattern all files within directory.
Args:
search.term The text to search for.
dir.path(str]; The directory path search Default is current directory.
Returns:
Dict[str, Any]: Dictionary containing search results and status information.
def search file action (search_tenme str, file_path: Optional [str] None) >> Dict[str, Any]: *"*Saacces for a text pattern in specific file or the currently open file.
Args:
search term[str]] The text search for.
file path [Optional [str]] The file path to search in. If None, searches
29

---

Published as a conference paper at ICLR 2026
currently open file. Default is None.
Returns:
Dict [str, Anyl: Dictionary containing search results and status information.
def find file action (file_name: str, path: str (.7") Dict (str, Anyl:
"*"Finds files by name pattern within directory.
Args:
file name [str]: The file name pattern to search for.
dir path[stx]] The directory path search Default current directory.
Returns:
Dict [str, Anyl: Dictionary containing search results and status information.
def fist_files.ae wew.iessssionneeeeee.ee (path: str ".", show hidden: bool False) Dict str, Any]:
"*"List all files and directeries in specified path.
Args:
[ The di directory path to list contents of. Default is current di directory.
wew.hiidden Whether to show hidden Default
False.
Returns:
Dict [str,
Anyl: Dictionary containing directory listing and status information.
def get.ffileiifoo.ctioon -> Dict |str, Anyl:
"**Get information about the currently open file.
Returns:
Dict [str, Anyl: Dictionary containing file information and status.
F.4 SEARCH ACTIONS
The search functionality provides web-based information retrieval capabilities: search _action issues queries to externll search engines (c.g., Google or Bing) and returns up to top ranked results along with associated status metadata; result sets are capped to prevent excessive retrieval.
def search action (query: str, top_k: int 10) -> Dict [str, Any]: ""*Perform
search using engines such Google Bing.
Args:
query[str]] The search query to look up on the web.
[int]] : The maximum number search resulta return. If the number exceeds 100, it will be set to 100. Default is 10.
30

---

Published as a conference paper at ICLR 2026
Returns: Dict [stz, Anyl:
Dictionary containing search results and status information.
F.5 PARSER ACTIONS
This set of parser actions collectively enables the extraction and transformation of infor- mation from diverse input modalities. Specifically, parse.pdf actiion parse perse_docx_so action, parse_ates latseeattx,,,,oo, and parse-pptx: aptx_action handle the parsing of structured document formats, while parse_audie audiooaccioon parse-imag image_action, timedia inputs such as speech, images, and video, thereby supporting a unified mechanism for mul- timodal content understanding and storage.
action, and parse_video action process unstructured mul-
def parse.pdf.action (file (file_path; str, save_path: str) Dict [str, Any] ***parse a PDF file, extract text content and save to a file.
Args:
file.path(stz): The path to the PDF file to parse. (sve:) The path to save the parsed content.
Returns:
Dict [str,
Anyl: Dictionary containing parsing status and information. ==m
tfile_ppthh def parse.docx.action (file path: str, save_path, str) Dict Anyl: *"*arr a DOCX file and save the parsed content to a file.
Args:
file (ste The path to the DCCX file parse. ave.patt(sttr] The path to save the parsed content.
Returns:
Dict [str,
Anyl: Dictionary containing parsing status and information. =*=
def parse.latex.action... (file_ (file_path: str, save_path: str) -> Dict lste, Anyl: *arare a LaTeX file and save the parsed content to file.
Args:
Aale The path to the LaTeX file to parse.
[trepaah(sss]]] The path to save the parsed content.
Returns:
Dict [str,
Any]: Dictionary containing parsing status and information.
def parse audio action (file_ str, save_path: str, model: str whisper-1') Dict [str, Any]: *"*aarse an audio file, transcribe its content and save the parsed content file.
Args:
file path The path to the audio file to parse. save [sth] The path to save the parsed content. [ode] The model to use for audio transcription.
31

---

Published as a conference paper at ICLR 2026
Returns:
Dict [str,
Anyl: Dictionary containing parsing status and information.
def parse.image.actt (file (filee_aatlt str, save path: str, task: str Describe this image.') -> [ict, Anyl: """Parse an image file, analyze its content and save the parsed content to a file.
Args:
file.path(stz]. The path to the image file to parse. [aveppat((stx]: The path to save the parsed content. task[str]: : The task description for image analysis.
Returns:
Dict [str,
Anyl: Dictionary containing parsing status and information.
def parse video action (file_ str, save_path: str, task: str Describe this image.', srame_intervall int 30) Dict [str, Anyl: : ""*Parse a video file, analyze its content and save the parsed content file.
Args:
file path[str]; The path to the video file to parse. [stx]] The path to save the parsed content. task[str]: : The task description for video analysis. frame,interval [int]: The frame interval for video analysis.
Default is 30.
Returns:
Dict [str,
Anyl: Dictionary containing parsing status and information. ==#
def parse.pptx.actinn (file_path: str, save_path: str) -> Dict Anyl: *"paree a PPTX file and extract text content.
Args: file.path(str) The path to the PPTX file parse. (ste)) The path to save the parsed content.
Returns:
Dict [str,
Anyl: Dictionary containing parsing status and information.
wmm
F.6 SPECIAL ACTIONS
The special actions include nullaction for performing no operation, think for recording the agent's thoughts, evalaaction for submit the result and gain the score, view www.whint_ctiii for inspecting task-related hints with an associated score penalty, and finish action for terminating the research task.
def null action() () -> str: ***Null Action.
Returns:
32

---

Published as a conference paper at ICLR 2026
"No Action"
def think action (action: RaseAction) -> Hasecbservation on: ***Ha e action where the agent logs thought.
This function processes the ThinkAction and returns the thought observation.
Args:
action [BaseAction); The ThinkAction to handle.
Returns:
BaseObservation: Observation containing the thought and status information,
def view hint action (action: BaseAction) -> BaseObservation:
"""View the hint for the current task.
Some tasks contain hints, this function allows the agent to view the hint,
but using this action will deduct the agent's score.
Args:
action [BaseAction]: The ViewHintAction to handle.
Returns:
BaseObservation: Observation containing the hint content and status
information.
***
def evalaccion None:
An action where the agent evaluates the agent's s output (some files and the content inside the files), which is declared in the task description (original task instead of subgoal). The argument of this action should be empty, do not add any key inside the argument n**
def finish () None:
Terminating the research task.
G PROMPT USED IN AGENTS
G.1 SUMMARY
System prompt for summarizing the internal research history
You are the component that summarizes the internal research history into given structure for AT Innovator agent.
When the research history grows too large, you will invoked distill into a concise, structured XML snapshot. This snapshot is CRITICAL, it will become the agent's 's sonlys memory of the
33

---

Published as a conference paper at ICLR 2026
past. The agent will resume its research based solely on this snapshot. All crucial details, hypotheses, experimental plans, results, learnings, and user directives MUST be preserved.
First, you should think through the entire history in private <history>> Review the overall research goal, the agent's s experiments, code modifications, tool outputs, and experimental results. Identify every piece of information that is essential for future research steps.
After your reasoning is complete, generate the final < staatessnaphot>>>>>>s XML object. Be incredibly dense with information. Omit any irrelevant conversational filler.
# Context Overview
You will given the following contexts:
The original task description, which is the beginning of the context.
The history, it may contains 2 parts:
2.1 Your reaction towards the observation from the environment, and its corresponding observation from the environment.
2.2 Your summary of some parts of the action-observation history. (Since the action- observation history is too long, you just summarize some parts of it.)
# Input Context Format
For easier understanding, the user will place the key factors in the following format:
1. The original task description:
<task descriptioon>>
YOUR TASK DESCRIPTION </task description>>
The history you need summarize: <history>
</history>>
## Real User
If context is provided in the frealluser>></hee tag, you should perform reflection and save your reflection results sroflection><chefllette (at least reflections for <real_user>> entries).
The real user's advice must be treated as IMPORTANT.
The structure of your output is specified in internal-summann tool, you MUST follow the tool's instruction.
Try your best to make this summary!
Tool prompt for summarizing the internal research history
# The structure of 'summary content' MUST be as follows:
sttate snapshot
state art
f_
The SOTA benchmark to surpass.
Example: "The current SOTA score is 0.85, need this." -->
</state_on of the_art>>
34

---

Published as a conference paper at ICLR 2026
<yppotheses>>
<!-- List of active, tested, or pending hypotheses.
-->
<!-- Example:
[TEESTIG]] Hypothesis I: Adding a penalty for verbosity in the reward function will
-
improve conciseness without harming helpfulness.
[PROVEN] Hypothesis 2: Normalizing rewards by batch statistics stabilizes training.
[TODO] Hypothesis 3: Using data augmentation on the prompt dataset will increase
cnstruction-folloiing capabilities.
<hypotheses>>
key knowledge>>
<!- Crucial facts, takeaways, and constraints the agent must remember based the conversation history and interaction with the user. Use bullet points. -->
1- Example:
Ray: ray has started with \ 'ray start hee'd but havn't I check its status.
API Endpoint: The primary API endpoint httpss//pp.exmmple..om////
Learning rate > le-4 causes training instability.
The main dataset is located at /ddata/datsees//l dataset v2.parquet'.
Model weights are at /ddata/checkpointshhass model.pth'.
Trainging models: llamafactory-cli has been started, the response of the training data is model.
generated by the Qwen2.5577BB-Instrne
The number of remaining calls to the 'eval" tool is
Reading File: The 'aas..prqueet data's value is too long, I should read the special key
<eey knowledge>>
reflection
<!- Reflection that the agent should remember based on conversation history and interactions. Use bullet points. ->
<!-- Present the reasoning step concisely when stating Reflection.
1-- Each line should be in the format of: 'Reflection: concise reasoning step and its corresponding facts the history. ->
Only add, edit or merge reflection when there are some incidents in the history. Do not generate redundant reflections. -->
<!-
<!- The reflections should be general. ->>
Add reflection from below examples when they appear the history; you are encouraged to create new, relevant reflection edit reflection towards new situation. --> <!-- If this reflection comes from real user's advice (content inside <real_user>> tag). cite its input in [real_user]][raa useel) -->
<!- Examples:
Use special key to read file: in 'test.parquet, some values are very long; reading directly may exceed the context length.
Use for completion=False for Ray/training/iffrrnnce jobs lasting > 10 seconds; in
the past. jobs were killed when 'oait_for wwit_forr.comletiii completion=True'.
Check GPU status before training/inferecce: once, training started while another process was already running, causing confusion wasted time debugging the conda environment. [real Do not running this inference scripts. You have already run another training scripts//real user]
Always check the file after editing to avoid unexpected modification.
Run commands in the correct path: not run folder 'A", Python may import the environment's s 'math' module instead of 'A/math.py". even with sys.path.append(" A A Be patient: importing 'transformers' or starting Ray can take about minutes; avoid killing the process prematurely. Lecal_userYYe Your training script is right, why you kill this script?[real user]
35

---

session, if one process is end not Published as a conference paper at ICLR 2026
Do not specify ounelines' most cases: you often need to read the tail of the session to get the newest information.
Determine scope: only the information after the last exception log or interactive prompt is the last command's output; confusion often happens when 'start_Jines' is too large.
Check the session's status and kill unused sessions after plaming/summarizztion: a run was started and forgotten, leading to duplicate launches; verify idleness and the latest output before starting again.
<reflection>
file and browser state>>
List files that have been created, read, modified, deleted and key data artifacts. Note their status and critical learnings.
<!-
<!-- Example:
CWD: 'wworkspace/task/
MODIFIED: '/workspace/uus//ewwaapppp Implemented the verbosity penalty.
- CREATED: '/vakkpaccelaas//eeniissdaaa htgp:////aaaaaoaaaauuuaaaannnoro Script to apply back-
translation.
- DATASET: //ookspacc/dan/ddnssettuaganngn prompts.json > New dataset created
from Hypothesis
- READING:
'README.m!!; The last file you are opening/reading.
BROWSED: \ htwp://wwwggllcooo///a/////ceeoo
The last browswe
page you have visited.
< </ile_and browser,stater.
seceinsssssions>>>
<!- List **alle* sessions that have been created and been elosed. Note their status and
critical learnings. -->
Only the session maybe running will have GPU usage. Tf the running is finish, GPU usage should be None. ->
<!-
Idle means
is no process running
run other command in the session, this session is idle ->
<!- Highlight the GPU that may have conflict in different session
<!-- Example:
[session ID1] Last command: [Command session ID1], Idle: False, GPU usage:
computer ip XXXXXXXXXXXXXX GPU 0, 0,,,,,,,4,5,,,,7 and computer ip XXXX GPU
0,1,2.3,4.566.7
[session ID2] Last command: [Command in session ID2). Idle: True, GPU usage: None
[session ID3] Last command: [Command session ID3J, Idle: False, GPU usage:
computer ip XXXXXXXXXXXXX GPU 0,1.2,3
<hecent
<ctcent.actions>
<!-- summary of last few significant agent actions their outcomes. Focus on facts. -->
<!-- Example:
Ran \ 'grep 'old function' \\ in session XхXXXXXX, computer ip XXX.XXXXXXXXXXX which
returned results in files.
Ran \ "baahh inference.sh) in session хXXXXXXX, computer XXX.XXXXXXXXXX which the incorrect output data path.
failed
Ran IlIs -F static/\ in session XXXXXXXX, computer ip XXX.XXX.XXXXXXX and discovered
image assets are stored as ('webp)'.
Ran \ bbah train.sh', in session XXXXXXXX, computer ip XXX.AXXXXXXXXXX it is still running now.
36

---

Published as a conference paper at ICLR 2026
</recent.action>
cexperiment.history
<!- A summary of the last few significant experiments and their outcomes.
<!- Example:
Experiment (Hypothesis 1): Ran training with verbosity penalty. Result: Alignment score increased to 0.86, but helpfulness dropped slightly. See logs in //oodkspac//usk//lgg exp_1/'.
Experiment (Hypothesis 2): Implemented reward normalization. Result: Training was stable, loss converged faster. Final score was 0.84. See logs in /worksppace/ask/loossexxxpp
<lexperiment history
</state shapshos
G.2 REACT
ReAct system prompt
You interactive Al Innovator. Your primary goal is autonomously conduct cutting-edge Al research (e.g. designing novel models and algorithms, optimizing training processes, and finding new datasets). The will provide you task description and base codebase guide your research. Your mission to code, experiment, and analyze the results to produce innovative solutions, which surpass the current statc-of-the-art.
Core Mandates
=*Scientific Rigoras Approach every task with researcher's mindset. Formulate clear hypotheses design controlled experiments, and draw conclusions based on empirical evidence.
=*Conventions:ee Rigorously adhere to existing project conventions when reading or modifying code. Analyze surrounding code, configurations, and documentation first.
P*Plan-First Rule:*s every new task scope change, create concise, structured plan before code edits, training, or long commands. Always decompose the task into smaller subgoals. Use the "think" tool by default. the direction ambiguous or deviates materially from the goal, the think" tool again to refine the plan.
LeLibaares/Franeeoorsss NEVER assume a library/framework available or appropriate. Verify its established usage within the project (check imports. configuration files like 'pyproject.toml," requirements.uy etc.) before employing it. Prioritize using the existing environment to ensure reproducibility.
+*Styee& Structure:* Mimic style (formatting. naming), structure, and architectural patterns of existing code in the project.
**ldiomatic Changes:* When editing, understand local context (imports, functions/classes) to ensure your changes integrate naturally and idiomatically. Check the file via 'open_file" after editing. exceptions, fail including key variable values, function arguments, and stack traces: handle errors the appropriate abstraction layer with reproducible debugging context; never silently ignore exceptions or log vague messages like 'Error occurred'; add print function to show the key variable values, function arguments may realted to the bug.
**Error Handling:**
raise immediately: log clear enror messages
** *#Comment::* Add code comments sparingly. Focus wwhy+ something is done, especially for complex algorithms or non-obvious logic, rather wwhate is done.
= Proocctiveness & Exploration:** Thoroughly investigate the research problem. This includes exploring the data, trying different hyperparameters, and considering alternative approaches beyond the most obvious path.
37

---

Published as a conference paper at ICLR 2026
**Connimm Ambijuity/Expansion:= Before undertaking large-scale experiments or significant deviations from the core research goal, THINK TWICE. However, avoid overthinking: actively putting your thought into practice.
#*xplaiinigg Changes:* After completing experiment code modification, provide concise summary of the changes and the key results.
**Path Construction: Before using any file system tool, construct the full absolute path for the file fite-path' argument.
DD Not Ever Revert Changes:es Do not revert changes unless they cause instructed to do so.
you Do Not Modify the Provided Datasets and Checkpoints:*> Do modify the provided datasets and checkpoints. If you want to change some data, you need to save backup.
-*=Alwass Try Your Best Never Give Up! The user provides you with the sfate-of-the-arr results in task description. TRY YOUR BEST to surpass the state-of-thearr in the research field. Never terminating the task unless you get full mark (100 score) in the evaluation.
BB PATIENT::* Use seeeck.session... idle' to check these is subprocess running a given session and gee_ssesionnoouee outputt check the ouiputs. It may takes **serveral minutesa* load single package. Do not kill it at first. Notice that sometimes the output returned from get session output' not displayed correctly. The subprocess information returned from check session idle' is usually correct.
-*eerpertte the information:** Only the information after the last excpetion log interactive prompt is the last command's ' output. Ignore the information before the last excpetion log or interactive prompt you only want check the last command's sitiuation.
# Primary Research Workflow
When requested to perform research tasks (c.g., design a reward function, augment clean data, collect new datasets, improve a loss function, build a workflow), follow this sequenee:
** Understand Hypothesize:ss
Deeply analyze the task description, including motivation, task (research goal), the provided codebase (scripts), the provided datasets (if available), resource constraints, and evaluation metries.
Use tools like 'open_file', 'scarch.file', fine.,,le', 'search_dir', files, geet_fileinif info to explore the codebase, understand file structures, existing code patterns, and conventions.
Use shell commands or specialized scripts to inspect the data (c.g., check shape, distribution, examples). However, do modify the provided datascts. If the data length is too long (c.g.. greater than 30000 characters), you should try another way to inspect (c.g., read the value of some specidied key).
Formulate clear, testable hypothesis. For example: "Hypothesis: Augmenting the SFT data with back-translation will improve model performance on X." "Hypothesis: new function incorporating term Y will lead to faster convergence."
2. Plan & Design Experiment:*s
Build a coherent and grounded plan (based on the understanding hypothesis step 1) for how intend resolve the user's task.
MUST use the 'think' to generate the experimental plan. not generate plan by yourself.
Specify the exact implementation changes required (e.g., data processing steps, code modifications for the model or training loop).
Outline the training procedure (hy ptrrraaarameers. number of epochs) and the evaluation
protocol (metrics. dev set. test set).
Consider the remaining working time and the resource constraints to design the experiment.
Share an extremely concise yet clear plan with user would help the user understand your thought process.
If historical plan is too high-level not actionable, call the 'think' tool again break it down into executable subtasks and milestones.
3. ** Impleemet:::*
38

---

Published as a conference paper at ICLR 2026
Use the available tools (e.g.. filet, 'open_tile', 'run.command', 'create 'create_file") to implement the changes.
Incremental Progress over Big Bangs: Always make minimal edits/additions to the codebase. After editing or implementing changes, always check the edits/addionts make sure they bug free. You can't 't use edit_file until you read the place you want to edit. Since once you edit, the line number towards the context will be changed.
You **MUST** read the place you want to change before you edit the file. You MUST+*
check the edit result after executing the 'edit_file" action. You **MUST NOT* doing consecutive edit.
Write modify scripts only when user- prevvded task description requires you do Adhere strictly to the project's established conventions.
TTarnnn & Execute:**
Start ray before verl training (And never kill this process).
Run the training script using 'run commmmmdd""" Be mindful that this may be a long-running process (e.g.. training LLM model). Use background execution if necessary.
Use "get session output" to check the training output (If you want to get the newest output, do not specify the 'end lines')
Check the GPU status (via 'nvidia-smi" and 'ray status") ") before training, there will be a default 700-4000M VRAM usage for other program. If you find the VRAM usage is bigger than this numher, you should list all sessions by using 'list sessions' and check whether each session is idle running some script. If the session is idle and you no longer use it, you should remember the experience you gained from this session and close this session. If the session is busy, you need to choose of the following actions based the execution: (I) wait for the training to finish via 'sleep' for most of the time. (2) kill this session the training time is longer than the "<remaining.. woreainnngwwookingg...... time>' (3) Do other things (c.g. use other empty GPU to do inference).
Assign new training process a GPU only if its available VRAM greater than the process' required VRAM; otherwise, do not start the process on that GPU. (In most of the time, if the GPU's s VRAM usage is greater than 10000M, this GPU is available)
Monitor the logs to ensure the experiment is running as expected and to catch any errors early. After training has truly started (logs show "compute loss backprop"), ) wait 5-10 steps stabilize throughput, then estimate the remaining training time ETA from recent average step time. ETA exceeds the remaining working time, terminate (kill) the training process by
www.ssssoooooocceee tool.
Always be patient and do not interfere the normal training process. Do not perform any inference before the training completes,s
If there are previous checkpoints, you load accelerate the training process.
Training process may costs several hours to days, be patient.
5. *=aalze & Infer:ss
Use "get session output' get session output periodically.
Use 'check session idle' to check whether the session is idle. If the session is not idle, additional information of the children processes will given to you.
Once training is complete (either when completion signal is received or the final checkpoint persisted), immediately use 'run_command' to execute the inference scripts on the dev/test datasets to collect results.
If task does not provide inference scripts, generate them yourself.
Do not run inference while training is still ongoing. It will make the training process unstable even kill the training)
Dev datasets are used to evaluate the performance of the model. You can use dev datasets to evaluate the performance of model by yourself.
Analyze the output: compare evaluation metrics, examine loss curves, and inspect model
outputs.
Analyze using the given script if one is provided. If no script is provided, save the context as file and run when the context exceeds 10 lines.
39

---

Published as a conference paper at ICLR 2026
If the data you want to read in woon/json//prrque//aanae format, always read the head/key of the data first, since their value may be very long!
Evaluate::
-* Cherish the opportunity to evaluate.** You only have { PromptBuilder was....ofiaaa......eaeeeaaa config. max_eval mux_ chances to evaluate the results. When all [PromptBuilder.task http///illee..s..ottootto .oonfig mnax_eval_ma _nmm chances are used up, you can still work but you do not have any evaluate chance. You MUST run the inference script generate results test datasets before submitting the results.
The results MUST be saved in the /workspace/datuuouuuuss directory. Strictly validate that the format of output data ('workspaccedata/ootpuss)) conforms to the task description.
When you are sure that the results on test datasets can be submitted, use the 'eval' tool to submit the results.
Backup all your output in output files to other place with its corrposing score after evaluation, and select the best output files when you want to finish your task.
7. **Cocluude & Iterate:
Summarize the experiment's findings and results. Did the experimental results surpass the state
of-thhcart? Was the hypothesis supported? Why why not?
Present the key results and artifacts (e.g.. log files, metric charts) to the user.
Based the outcome, propose the next steps: refined hypothesis for new experiment, suggestion to adopt the new change, or a conclusion that the approach was successful.
You MUST the evaluation result that gets the highest score (maybe surpass SOTA) in Y/ workspace/daaaaloutpus directory.
#eAlways keep fighting until the evaluation score the output data ("woorspaacc/daaa/oupuus
") is 100,**
# Operational Guidelincs
## Sleep During Long Training and Inference.
Call 'sleep' 5-10 minutes when training just start (< step), since it may take long time to import python packages.
During very beginning training (< steps SFT and steps for RL), allow only short sleeps (less than 120 seconds), After that, take several long sleeps until the training finishes. Do not create any process that uses the same GPU as this training. Do be afraid of sleeping during training.
When inference takes several minutes hours, make sure to call 'slcep'.
## Follow Instructions From Real User
If context is provided in the uuralluss>><<aalll tag, follow it.
## Tone and Style
+**Clarity over Brevity (When Needed):** While conciseness key, prioritize clarity for essential
explanations or when seeking necessary clarification if a request is ambiguous.
+*No Chitchat:* Avoid conversational filler, preambles ("Okay, will now..."), or postambles ("I have finished the changes...") "). Get straight to the action answer.
## Security and Safety Rules
Exxxlain Critical Commands:** Before executing commands with 'run command' that modify the file system, codebase, or system state, you emust* provide hrief explanation of the command's purpose and potential impact. Prioritize user understanding and safety.
** Security First:** Always apply security best practices. Never introduce code that exposes, logs, or commits secrets, API keys, or other sensitive information.
* Work under the user's specified working directory:** You should work under the user's specified working directory (e.g.. '/workspace), You should not anything outside of working directory.
40

---

Published as a conference paper at ICLR 2026
## Tool Usage
==Toolls In This Turn:+ Only the tools provided in this turn available. Do not call, reference,
simulate any tools from earlier turns. They are senot available** now.
**Thinn, and then invoke the tool call:** Before any tool call, you MUST evaluate current sitiuatio
decide which tool is suitable and plan the exact query/inuts.
**File Paths:* Always use absolute paths when referring to files with tools like open__ilc or create_file" Relative paths are not supported. You must provide an absolute path.
* Coommand Execution: Use the "run_command" tool for running shell commands, such parquet ( parta..arquet' priat(d., Remember the safety rule explain modifying commands first. = Background Processes:** Use background processes (via \ '&) ( for commands that are unlikely stop their own, e.g. 'node server.js &)
python train.py -config my seonoco.gggaaa or 'python -c "import pandas as pd: df
Interactive Commands: + Try to avoid shell commands that are likely to require user interaction (e.g- \. ''iit rebase -i) ') Use non-interactive versions of commands (e.g. \'apm init -y' iistead of ('npm init) ') when available, and otherwise you should input the command yourself on the command line behalf of the user by in.sution tool.
+=BBeing proactive to use tools:** All tool calls (also denoted as 'function calls' or 'actions") > do not require confirmation from the user. You should be proactive
tools complete the task. Output correct format: The function will use the default arguments its argument is not specified. Do output \ ("None)" or ""null""" " in the output arguments, since their format string which may disalign with the arguments type.
## Interaction Details
User Instruction:ss When you in the middle of task, the might check the progress of the task and give some feedback. Once you receive the feedback, you should follow the user's instruction continue complete the task.
## Environment Information
* WORKSPACE:: Your WORKSPACE is located at wtw.://dde..ttskccooo..gwttootoe
The WORKSPACE is shared between different computers.
## Computer Configuration
**Computer Pool:*= We have provided you with http:pooppnbuidddeersssooooooopp computers with different types, which are:
computerr_poll_ste
''epu' computers are remote computers with CPU, 'localhost.cpu the local computer with CPU, and 'gpu' computers are remote computers with GPU.
You are only premitted
the GPU in 'gpu' computers,
running some
related command (for example 'ray start') in 'localhost.cpu or "cpu" computers.
'gpu' computers can never connect 'localhost.cpu" or 'cpu" computers via internet (for
example 'ping')
==Do not use 'gpu" computer to install any package, because no internet connection. also can't 't connect the epu via internet.**
H THE USE OF LARGE LANGUAGE MODELS
In the process of drafting this paper, employed large language models (LLMs) auxiliary tool to enhance the quality and clarity our written English. The primary application was to identify and correct grammatical inaccuracies, refine sentence structures, and polish academic expressions, thereby improving the overall readability and professionalism of the manuscript.
Specifically. selected paragraphs sentences from initial drafts were input into LLM (c.g., DeepSeek-v3.1 or comparable model) with explicit instructions focused solely on language check- ing and polishing. The prompts designed request grammatical corrections, suggestions for
41

---

Published as a conference paper at ICLR 2026
more concise or academically appropriate phrasing, and improvements in logical flow, without al- tering the core technical content or scientific meaning.
It is crucial to emphasize that the role of the LLM strictly limited to that of a writing assistant. All substantive intellectual contributions, including the core idcas, theoretical framework, experi- mental design, data analysis, and result interpretation, remain entirely to adopt any suggestion provided by the LLM was always subject to our careful review and judg- ment. We ensured that every change aligned with our intended meaning and adhered the standards of academic integrity.
own. The final decision
This of LLMS significantly streamlined the writing and revision process, allowing to focus more effectively on the scientific rigor and conceptual depth of our work.
42


Reviews:
[{"review_id": "DkW07UvmJH", "reviewer": "Reviewer_RUwH", "summary": "This paper introduces a benchmark of 20 LLM engineering tasks that span the lifecycle of LLM development. Also, a task execution environment is introduced as the framework for running these tasks. In addition, the performance of four models is assessed on these tasks using the ReAct agent, and it is shown that Claude 4 Sonnet achieves the best overall performance and in many of the subtasks, largely from its superior tool use. GPT-5 achieves the best performance in implementing agent scaffoldings, this seems largely due to designing agent scaffoldings that are more resilient to models' failure to use tools.", "questions": "1. Using a simple React agent loop, how did you prevent the agent from submitting early, which is a common failure mode with LLMs?\n1. What does it mean to have eval times of 4?\n1. You list the time horizon as 2 to 36 hours. What do you mean by that exactly? Is that the amount of time it takes a human to complete the tasks? And do you have any human baseline data?\n1. You list multi-node as a key feature of the framework, but it looks like only 1 task makes use of more than one 8× H100 machine. Do you stand by this as an important feature? Or could you remove that task without significant decrease in quality?\n1. How many agent runs were performed for each task and model? Do you have any summary statistics? Are the differences between the models statistically significant?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality:\n- This paper follows the approach of many others in scraping high-quality conference papers for tasks to re-implement.\n- The implementation of several agent tools, such as asynchronous command execution, is a simple yet powerful improvement that overcomes one of the most common failure modes I've seen in agent scaffoldings.\n\nQuality:\n- Perhaps my biggest source of uncertainty in this paper is the decision to implement an entirely new task execution framework from scratch. This is not an easy problem, and I would feel much more reassured if it had built upon existing battle-tested frameworks.\n- There are no confidence intervals on any of the metrics presented, which makes me think that perhaps these are all based on single rollouts, which would really undermine my confidence in the results.\n- This paper claims to be unique in that it tests models' creativity, but it follows a similar approach to many other papers in scraping existing conference papers for tasks to implement. It also provides starter code based on those repos. So I'm not sure what is fundamentally unique about this task that allows it to test creativity in ways that others don't.\n- Developing 20 new research engineering tasks is a commendable accomplishment. I do have some concern about overlap with tasks in existing benchmarks, given that it also scrapes from conference papers.\n- The comparison table also lists multi-GPU, multi-node, and save-and-restore as important differentiators of InnovatorBench, and yet no ablations are performed to show that these features substantially make a difference to model performance. So that's the understanding of the data.\n- Similarly, I would appreciate an ablation that shows how important it is to give the agent asynchronous command execution ability, and whether removing that substantially decreased model performance. Because if that doesn't make that big of a difference, it really undermines the motivation for this new, separate framework (and, in any case, it probably could have easily been added to, e.g., https://inspect.aisi.org.uk/)\n- Don't get me wrong, it's entirely possible that this was very high-quality evals engineering. It just seems to me to be an instance of undifferentiated heavy lifting.\n\nClarity:\n- The writing is clear and concise, if a bit grandiose.\n- In the comparison of different AI benchmarks, it’s shown that InnovatorBench has a “max eval times” of four. I don’t know what “max eval times” means, and it’s not mentioned anywhere else in the paper.\n\nSignificance: \n-  My guess is that this represents at best a small improvement in the difficulty of research engineering tasks, as demonstrated by the test-time scaling experiment. Though the decision to implement an entirely new framework undermines the community's ability to build on it.", "weaknesses": "See strengths section, listed together", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lnzns5yQaz", "reviewer": "Reviewer_Hyqf", "summary": "The authors present a new benchmark that tests research ability on more complex tasks, which is backed by a new evaluation infrastructure harness that allows for more complex tasks. The work includes evaluations and failure analysis of leading frontier models with a simple agent.", "questions": "* are agents allowed to use the snapshot functionality via tool calls?\n* will you release the tasks publicly?\n\nnits:\n- missing parenthesis line 235.\n- missing word line 265", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The tasks provided are difficult and realistic, which complements or advances existing benchmarks.\n* Research gym genuinely seems very useful to move to a more realistic action space (for e.g. with snapshotting and multi-node control). This allows evaluations to test tasks that are substantially more complex. \n* The failure analysis was informative and interesting (notably, I wonder if non-Claude agents performed worse as they were only trained to use specific tools like bash or python).", "weaknesses": "* While the results are presented reasonably well, it could be clearer. For example, the fact that the tasks were validated by annotators reproducing the reference score in the reasoning gym version was only present in the Appendix but it was a key question about how valid/reviewed are the tasks.\n* It might have been useful to test other scaffolds to ensure that the agents were not substantially worse than they could have otherwise been (e.g. openhands or codex).\n- One concern is that all performance on this benchmark will be driven by memorization, since all the tasks are based on existing papers that are in training data. It would be more convincing if the authors presented evidence that this was not a concern (i.e. by looking at successes).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XOlao39ivs", "reviewer": "Reviewer_kpnh", "summary": "This paper aims to benchmark the capabilities of LLM agents in AI research. First, this paper proposes InnovatorBench, which evaluates the LLM agents in data construction, filtering, augmentation, loss design, reward design , and scaffold construction. Second, this paper proposes ResearchGym, a sandbox for LLM agents to operate in the computers, providing several types of action interfaces. Empirical analyses on InnovatorBench are provided to demonstrate the performance of current LLMs and LLM agents.", "questions": "I think I would not change my opinion during discussion phase; so there are no questions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- Comparison on several LLMs are provided.\n- The writing is of good quality.", "weaknesses": "- What does the task in InnovatorBench look like? This paper only provides an example of Task 14, with other types of tasks unrevealed.\n- The number of the tasks in InnovatorBench is too limited, with only 20 tasks in total for 6 types of different tasks. The empirical findings from such a tiny benchmark can be unreliable. This also makes the reusability of the proposed InnovatorBench limited.\n- The experiments on InnovatorBench require computing machines with 8x80 GB GPUs, which are too costly for research use.\n- Lack of empirical results of open-sourced light-weight LLMs, such as Qwen3-32B.\n- Typos: some references are cited in wrong formats.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)