Towards Self-Evolving Agent Benchmarks : Validatable Agent Trajectory via Test-Time Exploration
Authors: Dadi Guo
Tianyi Zhou
Dongrui Liu
Chen Qian
Qihan Ren
Shuai Shao
Zhiyuan Fan
Yi R. Fung
Kun Wang
Linfeng Zhang
Jing Shao
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark Evolution
Agent Evaluation
Test-Time Exploration
Multi-Agent Systems
Large Language Models
Dynamic Task Generation
Primary area: datasets and benchmarks

Abstract: Recent advances in large language models (LLMs) and agent system designs have empowered agents with unprecedented levels of capability. However, existing agent benchmarks are showing a trend of rapid ceiling-hitting by newly developed agents, making it increasingly difficult to meet the demands of evaluating agent abilities. To address this problem, we propose the Trajectory-based Validated-by-Reproducing Agent-benchmark Complexity Evolution (TRACE) framework. This framework takes an original task from an existing benchmark and encourages agents to freely explore and evolve it into a new task with higher difficulty while recording the corresponding execution trajectories. The framework proceeds in three stages: (1) evolutionary proposal mining, which generates task evolution proposals through preliminary exploration and divergent thinking; (2) problem construction via free exploration, where proposals are instantiated into concrete problem instances through agent exploration, with execution trajectories recorded along the process; and (3) multi-level validation, which ensures that the evolved tasks are accompanied by reproducible and logically coherent trajectories. Experiments on the GAIA benchmark demonstrate that the TRACE framework consistently enhances task complexity while improving correctness reliability through trajectory-level validation. In addition, our framework can successfully adapt to and improve reasoning benchmarks such as AIME-2024. This work marks a paradigm shift from static, manually curated benchmarks to dynamic, self-evolving evaluation systems, providing a sustainable and challenging foundation for agent development. Code and data can be found at https://github.com/titanwings/trace-benchmark-evolving.

Published as a conference paper at ICLR 2026
TOWARDS SELF-EVOLVING AGENT BENCHMARKS: VALIDATABLE AGENT TRAJECTORY VIA TEST-TIMM EXPLORATION
Dadi Guo Tianyi Zhou Dongrui Liu'r, Chen Qian', Qihan Ren', Shuai Shao Zhiyuan Fan", Yi R. Fung", Kun Wang", Linfeng Zhang", Jing Shao
Shanghai Artificial Intelligence Laboratory HHong Kong University of Science and Technology Uniivesiity Michigan *Renmin University of China SShanhaii Jiao Tong University
"Nanyang Technological University
dgucae@connectt ust uk zht iattiayy88 sumictt ch. edd [11 Luuongruu,, shaojing: 8pjiab. hta.jaab.rrg..o org.en
ABSTRACT
Recent advances in large language models (LLMs) and agent system designs have empowered agents with unprecedented levels of capability. However, existing agent benchmarks are showing a trend of rapid ceiling-hitting by newly developed agents, making it increasingly difficult to meet the demands of evaluating agent abilities. To address this problem, we propose the Trajectory-ban Validated- by-Reproducing ggent-benchmark Complexity Evolution (TRACE) framework. This framework takes an original task from an existing benchmark and encour- ages agents to freely explore and evolve into while recording the corresponding execution trajectories. The framework pro- ceeds in three stages: (1) evolutionary proposal mining, which generates task evolution proposals through preliminary exploration and divergent thinking; (2) problem construction via free exploration, where proposals are instantiated into concrete problem instances through agent exploration, with execution trajecto- ries recorded along the process; and (3) multi-level validation, which ensures that the evolved tasks are accompanied by reproducible and logically coherent tra- jectories. Experiments on the GAIA benchmark demonstrate that the TRACE framework consistently enhances task complexity while improving correctness reliability through trajectory-level validation. In addition, our framework can suc- cessfully adapt and improve reasoning benchmarks such as AIME-2024. This work marks a paradigm shift from static, manually curated benchmarks to dy- namic, self-evolving evaluation systems, providing sustainable and challeng- ing foundation for agent development. Code and data can be found at [https:
task with higher difficulty
benchmark benchmark--vvoviing
INTRODUCTION
The paradigm artificial intelligence is rapidly shifting towards autonomous agents capable of complex reasoning (Comanici et al. 2025 Huang & Yang 2025). planning (Huang et al. 2024]
and tool utilization (Qu et al. 2025 Wang et al. 2024a). This progress is starkly evident in the
performance challenging agent benchmarks (Mialon al. 2023; Timenez et al. 2023), which
were once considered formidable. For instance, on the GAIA benchmark which is designed to test real-world assistant capabilities, top-performing agents have achieved scores exceeding 90% (GAIA Benchmark Team 2025), rapidly closing the gap with the human baseline. This rapid pace signals an urgent challenge: existing benchmarks are becoming saturated, diminishing their ability differ-
"Equal contribution. Work done during an internship at Shanghai Artificial Intelligence
Laboratory, supervised by Dongrui Liu "Corresponding author

---

Published as a conference paper at ICLR 2026
Pass@1 Score Across Different Levels and Rounds
Original
Pass@1
0.4
0.1
12 Total
Total
Deepseek-va.1
Gemini-2.5-flash
KIMI-K2
GPT-S-Mini
Figure 1: Model performance comparison the Pass@1 metric across four distinct difficulty levels and evolution rounds under the TRACE framework. As the number of evolution rounds increases, the performance of models shows downward trend, demonstrating that our framework successfully evolves more challenging tasks.
entiate state-of arrt agents and risking progress being driven by overfitting to static test sets rather than generalizable intelligence. However, the cost of manually creating novel, complex, and reliable tasks is labor-intensive, time-consuming, and expensive process, which highlights an urgent need for an automated and scalable approach to agent benchmark evolution.
However, evolving agent tasks presents unique challenges not found in conventional domains like mathematical reasoning (Hendrycks et al., 20211 Guo et al., (025) or knowledge-based question an- swering (Ynng et al. 2018). Agent tasks are defined by two key characteristies: (1) their procedural nature, which emphasizes complex, multi-step interactions with dynamic, scalable real-world envi-
ronments (e.g., websites, APIs) (Huang al. 2025 Liu al. 2025a Gao et al., (025a), and (2) their immense diversity, spanning from web navigation to software operation. These characteristics render traditional task evolution methods, such rule-based parameter mutation scaling largely ineffective. For instance, rule-based changes (Wu & Liu| (2025] Wang et al., (2024b) (e.g., altering specific keyword search query) often insufficiently robust. In dynamic web environment, such a minor change could break the task's 's solvability entirely rather than increasing its reasoning complexity. Similarly, merely scaling up task (Liu al.) (0255) (e.g., asking to book three flights instead of one) often increases repetition, rather than the core cognitive or planning challenges. Thus, new paradigm required that moves beyond superficial modifications to fundamentally enhance the procedural, logical and semantic complexity of agent tasks.
To bridge this gap, we introduce TRACE (Trajectory- benchmark Complexity Evolution). Departing from rigid, rule-based heuristics, TRACE leverages agentic capabilities of LLMs drive task evolution through three specialized components. The Evolutionary Proposer identifies diverse evolution directions via preliminary exploration; the Ex- ploration Executor navigates roal-world environments to materialize these proposals into recorded execution trajectories; and the Trajectory Validator enforces logical coherence and reproducibility. This ensures that evolved tasks are grounded in verifiable solution paths rather than relying solely on final-answer correctness.
Validated-by halidated-hy-Reproduuenn Agent-
The coordination between these agents enables a structured workflow for task evolution. This pro- cess begins with the Evolutionary Proposer, which analyzes task bottlenecks and explores its mantic space to identify evolutionary forks. Consider the question: "What is the name of Taylor Swift's debut album?" Rather than stopping at the direct answer, it examines related artifacts, such as the music video for the single Teardrops on My Guitar, and generates multiple evolution pro- posals. These proposals introduce additional reasoning dependencies, such as linking the video's actor to their filmography or tracing their cross-dos cross-domain career trajectory. Subsequently, the Explo- ration Executor materializes these proposals through proposal-guided trajectory construction. traverses the original solution path and integrates compatible proposals at appropriate intermediate states. For example, it may first inject a proposal to identify the actor in the video (Tyler Hilton) and the TV series he starred in (One Tree Hill), then introduce second proposal that pivots to his music career. This process transforms a single-hop lookup into a multi-step reasoning task with cross- domain dependencies, culminating in more complex final question (Trinh al. 2024 Fang al.)
2

---

Published as a conference paper at ICLR 2026
Sun et al. 2024 Gao et al. 2025b). such "The male lead in the music video for song from Taylor Swift's s debut album is multi talented individual who also starred in long- cong-running TV series that premiered in 2003. What is the title of this individual's 's own debut studio album?" More broadly, this bottleneck-aware proposal and injection process extends beyond information-seeking tasks to sessoning-intensive mathematical, and coding problems, demonstrating the generality of the workflow. Importantly, each exploration step is recorded as an execution trajectory that serves as the basis for validation, ensuring that the introduced complexity remains transparent and verifiable.
2023
This architecture balances flexibility and rigor in task evolution. Flexibility arises from the separa- tion of the Evolutionary Proposer and the Exploration Executor, enabling evolution along multiple dimensions, such as deeper tool integration, increased logical dependency, or combinatorial compo- sition, without reliance on hard-coded templates. To maintain rigor, the Executor outputs the evolved task together with its execution trajectory. including reasoning steps, tool invocations, and observed results. The Trajectory Validator then audits logical coherence and re-executes each step to confirm reproducibility. This validation loop ensures that generated benchmarks are complex, solvable, and grounded in executable evidence.
In essence, TRACE models the workflow of human benchmark designer while replacing manual curation with an autonomous evolution process. Unlike static rule-based approaches, it enables agents to explore real-world environments (e.g., the live internet) structured task substrates. This exploration is guided by bottleneck analysis: by diagnosing limitations of a seed task such as shallow tool usage limited reasoning depth, the framework steers evolution toward targeted structural augmentation. As a result, TRACE evolves tasks into tool-integrated, multi-step challenges that better reflect real-world problem-solving dynamics, repositioning the agent from passive solver to an active benchmark constructor.
Our contributions are threefold:
We propose TRACE (Trajectory-basee Validated-by-Repee Reproduciin Agent-benchmark Complexity Evolution), benchmark self-evolving framework that encourages agent ploration and records execution trajectories as first-class artifacts, ensuring transparency and reproducibility in task evolution.
We empirically validate TRACE on general-purpose agent benchmarks such as GAIA as well as reasoning benchmarks such as AIME-2024, demonstrating that it consistently produces tasks of higher difficulty, on which prominent agent systems exhibit significant performance degradation shown Figure thereby validating the effectiveness of our approach.
Beyond mor-mmre-hop" edits, our experiments reveal a From Seed to Spark pattern: under our TRACE framework, the model autonomously explores and evolves problems that can shift into entirely different capability domains (e.g.. from retrieval to math and coding), thereby substantially increasing the diversity of evolved tasks. The evolved tasks exhibit greater task diversity and require deeper reasoning depth, providing a robust methodology for rigorously evaluating and advancing future agents.
2 RELATED WORK
Agent Benchmark A range of agent benchmarks have been proposed to evaluate LLM-based agents under realistic problem-solving settings. GAIA (Mialon et al., 2023) focuses on human- centric questions that require reasoning, multimodal understanding, web browsing, and tool use. USACO (Shi al., 2024) adapts competitive programming problems with unit tests and reference solutions to assess algorithmic reasoning. ML.E-bench (Chan et al.) 2024) evaluates end-to-end MI. engineering by grounding tasks in Kaggle-style competitions spanning data preparation, model training. and experiment management. SWE-bench (Jimenez et al. (023) measures software engi- necring ability through real GitHub issues that require generating patches under authentic repository contexts.
Beyond answer-verified benchmarks, several works emphasize process-level evaluation by analyz- ing agent trajectories. WebArena (Zhou et al.) 2023) provides sestehosted websites with executable scoring and trajectory replay for reproducible step- and lask- analyses. Mind2Web (Deng et al.) 2023) contributes human demonstrations real websites with fine-grained metrics (e.g., element
3

---

Published as a conference paper at ICLR 2026
accuracy and step success), and its multimodal extension (Pahuja et al. 2025) augments trajectories with multimodal signals and step-wise evaluation.
Despite their breadth, these benchmarks largely static: tasks and environments pre-defined and updated infrequently. As agent capability improves, static benchmarks can saturate and be- come costly to refresh at cadence aligned with model iteration, motivating scalable approaches to benchmark evolution.
Benchmark Evolving Benchmark evolution has been explored mitigate saturation of static datasets. Benchmark Self-Exolving (Wang et al., 2024b) and AutoEvoEval (Wu & Liu 2025) apply predefined atomic operations to reformulate reasoning and close-ended QA tasks via struc- tural or semantic perturbations, primarily targeting robustness rather than increasing procedural task complexity. AdamMeme (Chen et al. 2025b) adopts multi-agent refinement to iteratively up- date meme datasets for probing harmfulness-reiatee reasoning, but is specialized to this domain. EvoCodeBench (Li et al. 2024) updates code benchmarks by periodically ingesting new reposito- ries to reduce leakage, relying curated data streams and fixed schedules rather than model-driven exploration.
In contrast, TRACE evolves agent tasks via test-time exploration in real-world environments. In- stead of predefined edits or scheduled refreshes, agents autonomously construct harder tasks and output their execution trajectories, which are subsequently validated for logical coherence and repro- ducibility. This trajectory-aware evolution supports scalable benchmark updating while preserving solvability and verifiability.
PRELIMINARY
3.1 AGENTIC WORKFLOW AS DAG
In this section, we represent an agentic workflow W as directed acyclic graph (DAG) G = (S, ES, where the node set invoking steps, and the edge set & C S x S encodes data dependencies and control-flow constraints. Specifically, an edge (S,, S,) E E signifies that the output of step S, is required as input for step Sj. Each node can be abstracted as the quadruple below:
(Si, S2, .,,......., SSy} corresponds to an ordered sequence of discrete L.L
S = (G-1, rs. a4 o.),
where
Context q-11: the interaction history up to step i-1 (previous actions and observations together with fixed task/context).
Reasoning in test-time ry: the agent's latent reasoning state at step (internal scratchpad/- planning variables and intermediate choices that are not executed by the environment).
Action a;: the external action/message emitted step conditioned a tool/API call with arguments, code to run, a retrieval query, or a user-visible response), which is the only component actually executed outside the agent. This mtae-transit process could modeled as p(a: G-1,rr)) = x G-1,7:).
C-1 and (c.g.,
Observation o,: the feedback returned by the environment after executing a; (e.g., tool outputs, retrieved documents, execution logs, state deltas, optionally numeric score).
After execution, the context is updated as = 1-110 oz), and edges (S,, Sj) arise when artifacts from S, typically o; the updated context c; are consumed by Sy; tool-free steps are a special case where is user-facing message and may be empty.
3.2 EXPLORATION TRAJECTORIES FOR BENCHMARK EVOLUTION
Given the DAG formulation of an agentic workflow, we next define how exploration trajectories serve the foundation for benchmark evolution. An execution trajectory is a path through the workflow DAG.
== (S1, S2, S2....,Sr).
4

---

Published as a conference paper at ICLR 2026
where each St is realized by the tuple (c,, 0,). Unlike static benchmarks that only verify the final output or, we treat the entire trajectory 7 as a first-class artifact: captures the agent's reasoning, tool-use decisions, and environment feedback across all intermediate steps.
Formally, we denote the trajectory distribution under an agent policy x as
T
Pr (r( - 11 (a TT) 04s Ci-1),
==1
where Ta governs the action selection and the environment dynamics determine p(o: at ct-1).
Trajectory as Evolutionary Material. Benchmark evolution proceeds by exploring alternative trajectories r' that diverge from the original trajectory т. A proposal identifies a modification point Ss and suggests a new branch (c.g., adding a constraint, substituting a tool, transferring to another capability domain). The exploration process then unfolds the agent executes along this modified branch, yielding an evolved trajectory r'. The final benchmark task is reconstructed from r', and its complexity grows with the depth, diversity, and interdependence of such exploratory branches.
3.3 PROBLEM STATEMENT
We now formalize the benchmark evolution problem. Given seed benchmark Bo 2o} comprising tasks paired with their original trajectories т, the objective construct an evolved benchmark B such that
{(g, r) g
Diffieulty(B") > Difficulty(B))) (Bo,, and x'g B': E Tvalidatable-
Here, Tvalidatable denotes the set of trajectories satisfying the validity conditions defined above. In practice, operationalize Difficulty(-) via a trajectory-infommed bottleneck assessment inspired by theory-of-mine (Chen et al., 2025a): given the solution trajectories of the seed and evolved tasks, the difficulty judge estimates which task would impose larger intrinsic bottleneck for solver with comparable planning, reasoning, and tool-use capabilities (including access to a similar tool set), potentially requiring additional attempts non-trivial conceptual insight to resolve. The TRACE framework addresses this objective through a three-stage pipeline: proposal mining, proposal-guided exploration, and multi-level validation.
THE DESIGN DETAIL OF TRACE
TRACE is multi-agent framework that not only generates tasks but also encourages free explo- ration and records a complete, validatable trajectory. The system comprises complementary roles that collaborate end-to-end: Evolution Proposer suggests evolutions from seed task; Exploration Executor does not solve problem but defines one by turming proposal into an actionable explo- ration setup and conducting test-time exploration to produce an execution trajectory; and Trajectory Validator verifies and replays the trace to ensure determinism, safety, and correctness. Crucially, the product of evolution is not the problem alone, but the pair (evolved problem, validatable trajectory). This pairing both grants the model an automatic route to discover harder variants and preserves an auditable record of its own decision process, enabling reproducible, povcess-wware evaluation. For the core concept of this section, we provide example explanation in Figure 2
4.1 STAGE 1: EVOLUTIONARY ALOPOSAL. MINING
The Evolution Proposer takes the description of an original task from an existing dataset, optionally along with potential solution paths and answers, as input. Rather than directly generating modifi- cations, it first performs a bottleneck-aware pre-exploration stage. In this stage, the agent analyzes the original task and its solution trajectory to identify which agent capabilities the task primarily evaluates (e.g. planning, reasoning, trinsic difficulty. Based on this bottleneck analysis, the agent then probes the task's 's semantic space to uncover dimensions that can be further strengthened such extending evidence chains, increas- ing tool interaction complexity, or deepening reasoning requirements, thereby formulating targeted evolution proposals.
tool use) and where solver is most likely to encounter in-
5

---

Published as a conference paper at ICLR 2026
PROPISALS
Original
Bottleneck Analysis:
New Task Design:
Lightweight check:
Wher sau velunee lo of Ann catts seee
The bettonec of the took th wob
The prspecal intandi so wass nine about moder weight oft mealth roh..
Ansace Borpart cleck plsted, far conn ded nesdedin 17t tep....
WANTED
search Rocemem unddrr
lelendy dfseceon "Cue
Hheupp Supey gl to
Pre- Exploration:
Evalved Task
Reproducbility Check:
could saturry ous ansile ared
Solution Steps:
paaned toull is caroiten with the invervation
d'aneetii to coterttrr co mecherr
Clec"
Seav Cre Creitingg Sapely
toundd endt
cuadle informaiin about tas, This yell
crnaat of
"nea, paak Winul "adlatox_repp _rops"
Step poosed.
sorll chevst piee ther
From of
heil iapataaaeee/
Doore poer
of
info
"troush" seeaddhthee saper..
Solvability & Logicality Check
Auh aoos prrzu and "row
an litiis feoi ievernser
Proposals:
sonde www.uetrttteeeer:c ownet "ebernation" That paper
Hucapp Supaly Chough Parh
volu me of fuam bag.
Answer:
wooilsennt single problems maandoond in paper into acamplex
chocket the ertine proc whode lagiza chalas car secs sirable...
scee datain
"enswer" "700.""
Original Tock
Exploration Exiccator
Validator
Task Evolved
Figure 2: TRACE evolution pipeline. Starting from GAIA Original Task, the Evolution Proposer conducts bottleneck analysis and pre-exploration, drafting a concrete proposal to increase difficulty. Crucially, the Evolution Executor constructs the evolved problem from its own trajectory: as runs ReAct Thhough--Actioon-Obbeevaa etc.) and uses this trajectory to parameterize and scaffold the new task, while simultaneously pro- ducing a complete solution trace. A Multi-Level Vulidator then applies lightweight schema checks, dynamic replay for reproducibility, and solvability/logic audits to ensure trace validity. The result an Evolved Task that preserves origina benchmark's ' interface yet requires deeper reasoning (math coding), achieving a systematic benchmark- difficulty increase.
it collects evidence (numbers, constraints, citations,
Based on this preliminary diagnosis, the proposer then synthesizes multiple evolutionary proposals. It is prompted with comprehensive set of evolutionary strategies that serve as directional guidance rather than rigid transformation rules. These strategies encourage the agent to lengthen evidence chains, complicate tool interactions, target specialized domains, or escalate reasoning demands, while preserving high degrees of creative freedom.
To ensure that exploration remains principled rather than arbitrary, the proposer is governed by set of core guiding principles. These principles grant the autonomy to think divergently and even pivoting to new but semantically grounded scenarios when appropriate, while enforeing that all proposed modifications must yield deterministic and validatable solutions. Ultimately, the proposer consolidates its reasoning into a diverse set of actionable evolutionary proposals, each formulated as clear and imperative instruction for how the seed task should be transformed.
4.2 STAGE 2: EXPLORATION AND TRAJECTORY RECORDING
The Exploration Executor operationalizes the proposer's ideas by turning a high-level proposal into feasible problem and by conducting test-time exploration that yields validatable execution trace. Starting from the seed task, the Executor follows the current solution path and performs step-wise proposal injection:
an opportune step, it concretizes one evolutionary idea (e.g., adding a con- straint, substituting a tool, transferring to another capability domain), creating a controlled "fork in the road" that increases difficulty. The agent then explores along this branch with full tool access, producing a trajectory that records reasoning, action, and observation. This process serves a dual role: provides the model with a validatable trace to discover harder variants of the seed problem and captures the model's 's own execution path for subsequent auditing and analysis. Following the principle of inverse problem creation, the Executor's 's primary creative act is not solving a problem, but defining one. With the new, complex solution trace in hand, its final task is to formulate new problem description that fits this solution. Finally, the executor ensures the evolved task adheres to strict principles of authenticity, logical integrity, and solvability. It meticulously crafts the problem to guarantee the final answer is a single, deterministic, and validatable value, free from ambiguity, thereby producing high-quality and challenging new task.

---

Published as a conference paper at ICLR 2026
Algorithm TRACE Pseudo-code
Require: Proposer P. Executor &, Validators V, Tools T, Seeds 2o. Ensure: Evolved set Q = ((d'T')
retries R
º 0; C 2o
2: while C * 9 do
3:
q - SClect(C)
4:
for + to R do
5:
A < ProposeMining(? (P4))
6:
T' TaskEvolve(E, 9, A, 7)
test-time exploration yields trajectory b define the problem post hoc from T
7:
q' f QuestionFormulation (T)
8:
if Validate(), 9. then
9:
Q" + ('), U?: C+ C {q}: break
10:
return Q'
4.3
STAGE 3: MULTI-LEVEL VALIDATION OF TRAJECTORIES
The final stage of our framework is the Trajectory Validator, an autonomous agent that audits evolved tasks through multi-level validation. At the first level, it performs step-by-step replay to ensure reproducibility: for each step in the trajectory. the validator re-executes the tool call and ver- ifies that the resulting output aligns with the recorded observation. At the second level, it conducts global trace audit to assess the overall logical coherence of the solution. Under the assumption that all intermediate observations are reproducible and the reasoning chain is internally consistent, the task is deemed solvable and well-posed.
Beyond trace validation, the framework evaluates whether the evolved task reflects a genuine in- crease in difficulty, using the theory-of-mind aof--mnnd-ispieed bottleneck assessment described earlier. Ad- ditional integrity checks are also applied, including answer determinism (to ensure reliable eval- uation) and accessibility of referenced resources (e.g.. URLs). Only tasks satisfying all criteria including reproducibility, logical validity, increased intrinsic difficulty, and evaluation integrity are accepted into the final dataset.
To further guard against superficially modified but still easy items, we introduce a trajectory solver an auxiliary validator. This solver does not see the generation trajectory and operates purely under a ReAct (Yao et al. 2023) paradigm and enjoys tool-use parity with the main executor (same multi-modal access, web browsing, and coding). We the solver under budgeted attempts; it reliably solves the evolved task within resource limits, the item is flagged as insufficiently challenging and rejected sent back for re-evolution. Only tasks that resist this blind, tool- equipped solver proceed, providing an empirical difficulty floor independent of the authored trace.
4.4 TEST-TIME EXPLORATION
Given the inherent stochasticity of large language model sampling and the open-ended nature of trajectory-level evolution, the generation of evolved tasks is not single-shot procedure. but struc- tured form of test-time exploration. Within each run, the model engages in multi-turn interactions with real tools and code execution in an open environment, where intermediate states, tool calls, and observations collectively define a candidate solution trajectory from which new task is constructed.
Crucially, exploration unfolds at two levels. At the intra-run level, the Evolution Executor explores alternative reasoning paths and tool invocations to materialize stance. the inter-run level, multiple such trajectories are generated and subjected validation. Rather than treating failed attempts mere sampling noise, interpret them natural components of this broader exploration process. Ill-formed, logically inconsistent, or nonrreprod trajecto- ries are filtered through the multi-level validator, which performs step-level replay and global logical auditing. Only trajectories that withstand this scrutiny are admitted into the evolved benchmark.
-rouuddee problem in-
In this sense. TRACE extends the notion of test-time scaling beyond answer refinement (Meen- nighoff et al., 2025): additional computation is allocated not to produce more reliable answers, but explore and validate harder trajectory- grouuded tasks. Thus, test-time scaling becomes driver
7

---

Published as a conference paper at ICLR 2026
of benchmark evolution rather than merely improved problem solving. Our pipeline pseudocode is summarized in Algorithm
5 EXPERIMENTS
5.1 EXPERIMENTS SETUP
Benchmark. We evaluate TRACE on the GAIA benchmark (Mialon et al. 2023). a diverse suite of human-centric tasks requiring reasoning, multimodal understanding, web browsing, and tool use. Our experimental setup includes three variants: the original GAIA dataset (ROUND O), the first round of evolved tasks generated by TRACE (RoUND 1), and the second round of evolution (ROUND 2). A key property of our design is that the evolved tasks inherit GAIA's evaluation format without modification, enabling a seamless transition from the original benchmark to progressively harder variants.
Baseline. As the baseline, we report model performance on the original GAIA benchmark across the established difficulty levels (LEVEL LEVEL 2, LEVEL 3). These results serve both as refer- ence for the starting point of task complexity and as a comparison against performance degradation observed the evolved datasets.
Metric. We adopt GAIA's official evaluation, measuring accuracy by Pass @ I through its original answer-verification evaluation. By reusing GAIA's metrics directly, we ensure that improvements degradations observed across rounds can be attributed purely to the increase in task complexity, rather than confounding changes evaluation methodology.
Beyond accuracy, further track the average token length per task proxy for test-time com- putation and reasoning effort. Concretely, for each model and each round, we fix decoding hyperpa- rameters and compute the mean number of generated tokens over all questions. Intuitively, the more tokens a model needs to produce before committing to an answer, the longer it must stay in the loop perform additional reasoning steps, tool calls, and self-corrections.
Implementation Details. For all reported scores from baseline ROUND 0 evolved sets ROUND 1-2, we use a unified solver instantiated with inspect_eval ReAct Agent. The agent is capped 100 interaction turns per task, operates with the same tool-use affordances other agents, and has no access to any generation trajectories or validator outputs. Importantly, this solver is used only for evaluation and is independent of the validation pipeline.
To guarantee the effect of benchmark evolution isolating from model/back-end variance, all evo- (Yang al. (025). with
lution stages in TRACE use single back-end, Biven3-Coder4480--A1 matched capabilities across agents, Concretely, the Proposer, the Exploration Executor, and the Multi-level Validator are instantiated on the same back-end, receive the same tool-use capability. The auxiliary validator model is 1wen3-2355-A222 A-2B-Instrucc (Yang et al. 2025).
For fair comparison, the only moving part is the LLM back-end; all prompts, decoding configs, tool access, and budgets remain the same. Concretely, we evaluate four I.LMs: Kimi-K2 (Team) al., 2025), DeepSeek-Ve / (DeepSeek-Al et al.) 2025). Gemini-2.55-Fassh (Google Deep 2025), , and GPT-5-Mini (OpenAI) 2025). We report per-level and per-round accuracies together with relative deltas from ROUND 0.
5.2
EXPERIMENTAL RESULTS AND ANALYSIS
5.2.1 LONGER REASONING, LOWER PASS I ON EVOLVED TASKS
To assess the generality of our evolutionary framework, we compare model performance before and after evolution on two benchmarks: GAIA and AIME-2024. In both cases, models exhibit clear performance degradation on the evolved tasks.
GAIA Table I iillustrates the performance changes of different models on tasks from tasks from the original GATA dataset and two evolutionary stages. "Level" refers to the difficulty labels of the
8

---

Published as a conference paper at ICLR 2026
Table 1: Model Evaluation Results Pass@1
GATA benchmark
Level
Level
Level
Total
Models
Fvi.t-Orig.
Fvo. Orig.
Evo.+-Orig. A
Evo.t-Orig.
ROUND
Deepseck-v3.1
0.200--0.509
(-0.309)
0.279+-0.419
(-0.140)
0.200-0.2231
(-0.031)
0.247-0.418
(-0.171)
Gemini-2.5-faas
0.2004-0.471
(-0.271)
0.163+-0.322
(-0.069)
0.000-0.115
(-0.115)
0.151-0.291
(-0.140)
KIMI-K2
0.250+-0.377 0.377
(-0.127)
1.186--.2333
(-0.047)
0.100-0.007
(+0.023)
0.192-0.255
(-0.063)
GPT-5-Mini
0.250 0.566
(-0.316)
0.279 0.465
(-0.186)
0.200 0.192
(+0.008)
0.260 0.455
(-0.213)
ROUND 2
Deepseek-v3.1
0.200;-0.509
(-0.309)
0.222:-0.419
(-0.197)
0.000-0.231
(-0.231)
0.188 0.418
(-0.229)
Gemini-2.5-ffast
0.040;-0.471
(-0.431)
0.194+-0.232
(-0.038)
0.125+0.115
(+0.010)
0.291 (-0.161)
KIMI-K2
0.080+-0377
(-0.297)
0.250+-0.233
(+0.017)
0.125-0.077
(+0.048)
0.174-0.255
(-0.081)
GPT-5-Mini
0.160:0.566
(-0.406)
0.139-00:65
(-0.326)
0.375-0.192
(+0.183)
0.455-0.455
(-0.281)
MIXED
Deepseek-v3.1
(-0.309)
0.253 0.419
(-0.166)
0.117:-0.31
(-0.231)
0.218
0.418
(-0.200)
Gemini-2.5-faast
0.111: 0.471 (40.360)
0.177(
0.232
(-0.055)
0.056: 0.115
(-0.059)
0.141 0.291
(-0.150)
KIMI-K2
0.156: 0.377
(-0.221)
0.215
0.233
(-0.018)
0.111
0.077
(+0.034)
0.183
0.255
(-0.072)
GPT-5-Mini
0.200 0.566 (-0.366)
0.215 0.465
(-0.250)
0.278:
0.192
(+0.086)
0.218: 0.455
(-0.237)
Table 2: Token length results on the GAIA-TRACE benchmark.
Metric
GPT-5-Mini
KIMI-K2
Evo. <- Orig.
Evo. t- Orig.
Avg. Length (Round 1)
4898.7 2 <- 2864.5
(+2033.77 7)
6609.0 <- 3389.6
(+3219.4)
Avg. Length (Round 2)
6275.7 + 2864.5 (+3411.2)
8454.7 < 3389.6 (+5065.1)
original Gaia dataset, while 'Mixed' denotes the overall pass@1 we measured after combining data from two rounds of evolution. We did this because the tasks before and after evolution often differ significantly, to the extent that they even be treated independent problems. Experimental results indicate that, in most cases, different models experience a significant performance degrada- tion the new data, demonstrating the effectiveness of our evolution framework. For instance, the Remini-2.5-flash model experienced a 43.1 % drop in pass@1 accuracy for Level problems following the second evolutionary round.
As shown in Table average answer length increases for GPT-5-mini and KIMI-K2 substantially as the benchmark evolves, while Pass @ concurrently degrades. This joint trend indicates that TRACE is not merely introducing noise, but is making tasks genuinely harder in a way that forces models into longer, more demanding reasoning trajectories.
AIME-2024 Table 3 presents the performance of three reasoning models(DeepSeek--R RIiittR-l- Qwen3-235B-A228) different scales on the AIME- 2024 benchmark, well as its second and fourth evolutionary benchmarks. Average Acc. refers to the average accuracy over 10 test runs on the benchmark, while Average Length denotes the average number of tokens in the chain-of-thouu tf-ttuugttt generated by the model. We can observe that after four rounds of evolution, the model's performance on the benchmark has significantly declined, and the token count has noticeably increased. For instance, the Average Acc. of the Qven3-235B- A22B model decreased by 22.33%, and the average token count increased by over 8000. These experimental data demonstrate the significant change in the difficulty of the benchmark questions.
Qwen-7B, -l-Seee------sstt----eeerr
5.2.2 FROM SEED TO SPARK.
In some instances, the evolved items produced by TRACE remain within the original capability domain (c.g., web browsing with multimodal reading), where difficulty increases primarily through denser evidence requirements, longer tool chains, tighter formatting constraints. By contrast, the exemplar showcased here manifests an inspired emergence: beginning with a seed that is essentially single-hop retrieval question, the evolution pipeline kindles spark that reframes the problem into

---

Published as a conference paper at ICLR 2026
Table 3: Model Evaluation Results AIME-2024 benchmark
DeepScek-RI Di-a--RR--DistI--eee
DeepSeck H--Disiill-ween--ee
Qwen3-235B-A22B
Models
Evo. Orig.
A
Evo.e-Orig Orig.
Evo.t-Orig. Orig.
ROUND 2
Average Ace.
0.4933 0.5667
(-0.0734)
0.6233, 0.7300
(-0.1067) (+643.2)
0.9033
0,9400
(-0.0367)
Average Length
14853.4 16890.5
(-2037.1)
10556.7c 9912.8
21808.9: 19265.0
(+2543.9)
ROUND 4
Average Acc.
0.3933-00.5667
(-0.1734)
0.5333 0.7300
(-0.1967)
0.7167=00.940
(-0.2233)
Average Length
21092.7 16890.5
(+4202.22)
15140.4 9912.8
(+5227.6)
27283.2 19265.0
(+8018.2)
Table 4: Inspiration Emerges: Original vs. Evolved Task by TRACE.
Aspect
GAIA (Original, Round
Evolved by TRACE (Round exemplar)
Source Prompt
"What was the volume m² of the fish bag that was calcu- lated in the University of Le- icester paper "Can Hiccup Sup- ply Enough Fish to Maintain Dragon's 's Diet?""
"What is the total weight in kilograms of fish that Hiccup can transport in single oper- ation, utilizing full set of identical, open- topped cylindrical containers optimally de- signed and manufactured from a total of m² of metal sheet, given that each filled con- tainer must adhere to an 80 kg lifting limit and all necessary fish data (mass and vol- une) is be sourced from the paper "Can Hiccup Supply Enough Fish to Maintain a Dragon's Diet?""
Capability Domain
Web browsing factual retrieval
Mathematical modeling + coding web browsing
Trajectory Abstract
Single-hop lookup cite and extract a scalar value
formalize geomet- ric constraints + derive objective (ra((r) + apply calculus to optimize (find criti- cal points) - solve for (r* ,h") under con- straints
Multi-step derivation:
Difficulty Change
Low-moderate
High
quantitative modeling task requiring math coding + calculus, is shown in Table 4 This is not a superficial "one more hop" edit; it is a capability transposition from locating a cited scalar to constructing variables and constraints, deriving Vina(r), optimizing via calculus. Mechanistically, this shift arises because the proposal and executor grant the agent substantial freedom, encouraging exploration beyond the original capability domain. The final Trajectory Validator, with its multi- level replay-and-chede design, enforces format and tool-call consistency and ensures reproducibility and correctness, admitting only well-formed, validatable items.
Candidly, this seed-to-spark transformation exceeded our expectations. It demonstrates that TRACE can induce benchmark-level difficulty gains not only by deepening existing reasoning chains, but by rearticulating the task's 's background and assessed capability domain through trajectory-groundl proposal drivendrinnnn autonomous exploration and valf-vaiiitionn. More cases are shown in Figure I6 in our Appendix.
CONCLUSION
We introduce TRACE, framework that simulates human expert workflows to autonomously evolve benchmarks, replacing labor-intensive manual curation. Through bottleneck analysis that guides ex- ploration in real-world environments, TRACE transforms saturated tasks into structurally harder, trajectory-validdteed challenges. Empirically, it raises difficulty barriers on benchmarks such as GAIA. exposing model limitations often hidden by static evaluation. TRACE promotes shift toward sustainable, self-evolving benchmarks that remain rigorous, reproducible, and aligned with rapidly advancing agent capabilities.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
We gratefully acknowledge the support from Shanghai Artificial Intelligence Laboratory.
REFERENCES
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering. arXiv preprint arXiv:2410.07099, 2024.
Ruirui Chen, Weifeng Jiang. Chengwei Qin, and Cheston Tan. Theory of mind in large language models: Assessment and enhancement. In Proceedings of the 63rd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume I: Long Papers), pp. 31539-31558, 2025a.
Zixin Chen, Hongzhan Lin, Kaixin Li, Ziyang Luo, Zhen Ye, Guang Chen, Zhiyong Huang, and Jing Ma. Adammeme: Adaptively probe the reasoning capacity of multimodal large language models on harmfulness. arXiv preprint arXiv:2507.01702 2025b.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprin! arXiv:2507.0626/ 2025.
AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo. Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye. Shirong Ma. Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An. Wen Liu, Wenfeng Liang, Wenjun Gao. Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL www.ooggg/bb.ooo.oo /a0os/241..10 19437
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web, 2023.
Tianqing Fang, Zhisong Zhang, Xiaoyang Wang. Rui Wang, Can Qin. Yuxuan Wan, Jun Yu Ma, Ce Zhang, Jiaqi Chen, Xiyun Li, et al. Cognitive kemee--pro A framework for deep research agents and agent foundation models training. arXiv preprint arXiv:2508.004 00414, 2025.
11

---

Published as a conference paper at ICLR 2026
GAIA Benchmark Team. GAIA Leaderboard.
https://hugghagfaccccco///sss
gaia- wataaabnnccaaakeeeeeerooooo
2025.
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence. arXiv preprint arXiv:2507.21044 2025a.
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint ar WXX1::258.00770000 2025b.
Google DeepMind. Gemini 2.5 models.
/esooi//mmiinn 2025.
Dadi Guo, Jiayu Liu, Zhiyuan Fan. Zhitao He, Haoran Li, Yumeng Wang, and Yi R Fung. Mathe- matical proof as litmus test: Revealing failure modes of advanced large reasoning models. arXiv preprint arXiv:2506.177 /71 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang. Hao Wang. Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Ilm agents: A survey. arXiv preprint arXiv:2402.02716. 2024.
Yichen Huang and Lin F Yang. Gemini 2.5 pro capable of winning gold at imo 2025. arXiv preprint arXiv:2507.15855 2025.
Yuchen Huang, Sijia Li, Minghao Liu, Wei Liu, Shijue Huang. Zhiyuan Fan, Hou Pong Chan, and Yi R Fung. Environment scaling for interactive agentic experience collection: A survey, arXiv preprint #xXiv:2511.09580 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770. 2023.
Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodehench: An evolving code gen- cration benchmark aligned with real-world code repositories. arXiv preprint arXiv:2404.0055 2024.
Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang. Bingxiang He, and Yi R Fung- Costbench: Evaluating multi-turn cost-optimal planning and adaptation in dynamic environments 2025a.
for Ilm tool-use agents. arXiv preprint (iXx:251110025::
Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, et al. Synlogie: Synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond. arXiv preprint arXiv:2505. 19641, 2025b.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Seialom. Gaia: a henchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20286-20332 2025.
OpenAl. Introducing gpt-5. 2025.
12

---

Published as a conference paper at ICLR 2026
Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Hassan Awadallah. Explorer: Scaling exploration-driven web trajectory synthesis for multimodal web agents. In Findings of the Association for Computational Lin- guistics: ACI. 2025, pp. 6300-6323, Vienna, Austria, July 2025. Association for Computa- tional Linguistics. ISBN 979-8-89176-256-5. htthhtts:://at http://acaaanncooo.oooooooo anthelogyy.org2222.. Findings acl. 326/
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen, Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343, 2025.
Quan Shi, Michael Tang. Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? arXiv preprint arXiv:2404.1090 2024.
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.1972 197233 2024.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao. Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong. Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang. Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma. Xinyu Ma, Yingwei Ma. Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng. Ruoyu Qin, Bowen Qu, Zeyu Shang. Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung. Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang, Feng Wang, Haiming Wang, Jianzhou Wang, Jiaxing Wang. Jinhong Wang, Shengjie Wang, Shuyi Wang, Yao Wang. Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang. Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang. Ying Yang. Zhen Yang, Zhilin Yang. Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi k2: Open agentic intelligence, 2025. URL httpss//::///ett arxi org. / abs 2507 20534
Trieu Trinh, Yuhuai Tony Wu, Quoe Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625:476 482,
482, 2024. doi: 10. 10.1038//41586 00118-8800200-7--55555
Jize Wang, Ma Zerun, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta: benchmark for general tool agents. Advances Neural Information Processing Systems, 37: 75749-75790, 2024a.
Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei, and Xuanjing Huang. Bench- mark self-evolving: A multi-agent framework for dynamic I1m evaluation. arXiv preprint arXiv:2402.11443, 2024b.
JiaRu Wu and Mingwei Liu. Autoevoeval: An automated framework for evolving close-ended Ilm evaluation data. arXiv preprint arXiv:2506.23735, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv. Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang.
13

---

Published as a conference paper at ICLR 2026
Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. httpttss:: /ttp///axivvva org/ ats/ /205 09388
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arxiv preprint arXiv: 0tti::8809.00600 009.09600, 2018
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https://arxi>:
org/abs/2210 03629
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307 13854, 2023, /tattts://wwbbee dev
14

---

Published as a conference paper at ICLR 2026
LIMITATIONS
While TRACE provides a scalable framework for self-evolving benchmark construction, several limitations remain. First, we do not provide detailed analysis of computational cost. The multi- agent architecture, combined with trajectory-level exploration and validation, can incur substantial overhead due to repeated tool execution, code evaluation, and multi-round search. Although this design enables reliable evolution, a systematic study of cost- cost-difficulty trade-offs is left for future work. Second, while the validator plays a central role in filtering and assessing evolved tasks, its judgments are not fed back to refine the evolution proposer or executor. Incorporating validator signals as feedhack for iterative refinement could form a tighter closed-loop system and represents promising extension of test-time scaling beyond exploration alone.Third, although the validation pipeline enforces reproducibility, logical coherence, and determinism, we do not conduct compre- hensive human review all generated tasks. As result, subtle biases unintended artifacts may persist in the evolved benchmark. Incorporating human auditing or hybrid verification strategies would further strengthen reliability. We leave these directions future work.
REPRODUCIBILITY STATEMENT
We have provided detailed description of the TRACE framework in the main text, including the agentic workflow formulation, the three-stage evolution pipeline, validation mechanisms, and ex- perimental setup. Implementation details such model back-ends, solver configurations, and eval- uation protocols are explicitly specified to facilitate independent replication. In the Appendix, we further include the system prompts used for different agent roles, along with representative examples of evolved tasks. We will release the implementation of TRACE.
THE USE OF LARGE LANGUAGE MODELS
In this work, we use large language models (LLMs) solely for language refinement during manuscript preparation, including grammar correction, clarity improvement, and identifying ex- pressions that may cause misunderstanding.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
In this section, we present extended experimental results and visualizations. We also provide the prompts we set for different agents, as well as examples of tasks before and after evolution.
EXTENDED EXPERIMENTAL RESULTS AND VISUALIZATIONS
shows how TRACE evolution reshapes the capability mix of GAIA. Compared to the orig- inal dataset, the evolved benchmark slightly down- wwwn-weights pure web-browsing and multi-modalite questions, while increasing the share of coding and diverse filetype reading tasks, which typically require more structured reasoning and robust tool use. The higher "Integrative Capabilities" mass further reflects the emergence of more complex, mixed-capability items that do not fall cleanly into single category, indicating that TRACE places greater emphasis on multi- multi-faceted agentic behavior rather than simple lookup-style queries.
Table 5: Comparison of capability distribution ratios before and after TRACE evolution GAIA.
Capability Category
GAIA Original ( (%)
Evolve Round 1 (%) Capability Totals (%)
Web browsing
43.9
42.2
27.1
Coding
19.1
20.5
24.7
Multi-modalit
17.1
4.8
8.2
Diverse filetype reading
16.0
20.5
28.2
Integrative Capabilities
4.0
12.0
11.8
For both Round 1 and Round 2, further applied trajectory-agnostic solver auxiliary validator to eliminate items that remained solvable under a unified ReAct-style solver without seeing trajectories. This pruning removed 13/23/6 items from Levels 1/2/3, respectively, totaling 42 items per round. The effect is most pronounced at LEVEL 2, reflecting that mid-tier items are more likely be filtered under this trajectory-agnont Round and 73 (from 115) for Round 2. These details are shown in Table
check. The final post-filter sizes are 123 (from 165) for
Table 6: Item counts before/after applying the trajectory-agnostic solver as an auxiliary validator. Each cell shows Before >> After (-Removed).
Round
Level 1
Level 2
Level 3
Total
Round 1
40 + 20 (-20)
64 + 43 (-21)
15 + 10 (-5)
119 + 73 (-46)
Round 2
45 + 25
(-20)
57 + 36 (-23)
11 + 8 (-5)
115 69
(-42)
PROMPTS IN TRACE FRAMEWORK
THE SYSTEM PROMPT OF EVOLUTION PROPOSER AGENT
4 shows evolution proposal agent's system prompt. This prompt clearly defines the agent's tasks and workflow, and provides detailed, categorized hints regarding the agent's 's bottlenecks. Guided by this prompt, the agent first analyzes the original task identify its bottlenecks. Subsequently, it conducts preliminary exploration of the problem, seeking opportunities to increase its difficulty. Finally, it proposes several feasible proposals. We provided some bottleneck demonstrations for the agent's 's reference, as shown in Figure
THE SYSTEM PROMPT OF EXPLORATION EXECUTOR AGENT
The system prompt of our Exploration Executor agent is shown in Figure 6 It outlines highly structured and prescriptive framework designed to guide an advanced Al agent in its mission of task evolution. The agent's primary objective is to take existing intelligent agent task, and evolve it
16

---

Published as a conference paper at ICLR 2026
DeepSeek-V3.1 (Round
Nemini-2.55-llas (Round
GPT-5 (Round 1)
Kimi-K2-0905 -rreeie- (Round 1)
25%
25%
95%
25%
50%
75%
75%
75%
75%
DeepSeek-V3.I (Round
Gemini-2.5 2.5-Flssh (Round
GPT-5 (Round
10ti-K2-0905-peeee
(Round
50%
50%
50%
50%
75%
75%
100%
100%
100%.
100%
Metrics
task amount
Accuracy Capabilities Coding
Error rate
Web browsing
Multi-modallty
Diverse filetype reading
N/A
Figure Capability-wise profiles explaining ranking shifts on TRACE-evolved GAIA. For each model, radar plots show accuracy across capability categories, together with the distribution of tasks in each category (outer pink region), on Round (top) and Round 2 (bottom). TRACE evo- lution re-weights the benchmark distribution, while simultaneously demanding longer, tool-heavy trajectories, which in turn exposes different capability "fingerprints" across models and accounts for changes in their relative ranking.
17

---

Published as a conference paper at ICLR 2026
into more complex and difficult version based improvement proposals provided by the Evolu- tion Proposer Agent. This evolution is underpinned by two core principles: the first one divergent evolution, which encourages exploring intermediate states identify opportunities for increased difficulty. Another one is inverse problem creation, where the agent first executes and verifies a complex solution path, then defines the problem that fits this proven trajectory. The agent must deliver its output in meticulously specified Python dictionary format, with the recorded solution trace being critical historical record compiled directly from the successful code and observation pairs of its exploration, ensuring the problem's 's verifiable solvability. This comprehensive prompt thus serves as a meta-programming guide, enabling the AI to systematically generate challenging, well-defined, and fully verifiable tasks.
THE PROMPT OF TRACE VALIDATORS
Our pipeline employs multi-level validators. Here, we introduce the prompts for two types of val- idators. The first is a fine-grained validator, which individually checks whether the actual output of each code or tool call within the solution trace provided by the evolution executor is consistent with its claimed observation; its prompt is shown in Figure The second type an overall val- idator, whose purpose is to check the logic of the entire problem problem-solving process to determine the problem's result is easily verifiable (e.g., shouldn't problem's solvability (e.g., whether the problem context is complete, the answer is unique, etc.), and the correctness of the solution process. The prompt of overall validator shown in Figure
an open-ended question), as well as
TASK EVOLUTION DEMONSTRATIONSS
In this subsection, we present comparison between pre-evolution and post-evolution problems demonstrate the effectiveness of our proposed framework. Figure 9 16 show 8 real-world cases of evolution. Specifically, Figures 9
and 14 demonstrate tasks themed around tool invocation and 16 present tasks focused on logical reasoning and program writing. For each case, we provide an analysis within the figures, detailing the differences between the two tasks and explaining why the evolved task is more difficult.
10
I3
and searching, while Figures II
12
L
18

---

Published as a conference paper at ICLR 2026
Evolution Proposer
Part Your Mission & Role
You are a powerful intelligent agent task designer. Your job is to design modifications that meaningfully increase the cognitive and operational difficulty of existing agent tasks by ex- ploiting real bottlenecks in agent capabilities. The ideas you provide will be used to increase the difficulty of a real-world agent task. These agents possess strong reasoning capabilities, are able write and execute code, and can exactly the same tools you do. Therefore, you must come up with creative and diverse ideas to increase the difficulty of the task. Your proposals must:
Target verifiable, real-world data sources (no fabricated data; abstract games/logic-onl tasks the only exception, and must programmatically verifiable). Introduce multiple, distinct bottlenecks that measurably raise difficulty, not just extra steps. ** Produce tasks with unique, deterministic solutions that can be independently verified. To do so, you have been given access to list of tools: these tools are basically Python func- tions which you can call with code. To solve the task, you must plan forward proceed in a series of steps, in cycle of Thought, Code, and Observation phases.
The Broought-Code-OObeevatteee Your entire process is continuous, step-by-step cycle of #*Thought, Code, and Observa- tion. You MUST follow this structure for every action you take...
Part 2: Your Working Process:
Cycle
Part 3: What Can Be Bottlenecks And How to Create Them [Bottleneck Demonstrations)
Part 4: Guiding Principles
1. Embrace Creativity and Complexity...
2. Innovate Beyond the Scenario...
3. Develop Synergistic and Diverse Ideas...
4. Design for Verifiable Solutions...
5. Leverage World, Complex Data Sources...
6. Freedom to Traverse the Open Web...
7. Evidence Handling and Reproducibility..
Here are a few simple examples using notional tools, and your task should be more complex:
(Workflow Demonstrations)
Above example were using notional tools that might not exist for you. On top of performing computations the Python code snippets that you create, you only have access to these tools, behaving like regular python functions:
[%- for tool in tools, values() %]
|l tool.to_ tool calllaalllggppe calling__poopp(() }
%6 endfor %)
Figure 4: The system prompt of our Evolution Proposer agent.
19

---

Published as a conference paper at ICLR 2026
Bottleneck Demonstrations
A. Multiple-Source Conflict and Reconciliation (Breadth)
Positive (Conflicts): Require collecting and comparing information from at least three inde- pendent, credible sources that naturally disagree in numbers, definitions, time ranges.
Positive (Convergent Corroboration): Require gathering multiple sources that each provide partial, non-identical evidence so that only one candidate remains after intersecting constraints. Negative: Merely collecting multiple sources that restate the same fact or number without adding constraints exposing disagreements. Avoid prompts where additional sources only echo one uncontested answer and do not help narrow the candidate set or force reconcil- iation.
B. Long Evidence Chains with Structure (Depth)
Positive: Require chain of at least six steps where later steps depend earlier findings. Negative: Tasks solvable via straightforward, procedural retrieval like "search - open top result copy value."
C. Multi-Modal, Complex Media Comprehension
PPositiv: Combine heterogeneous media that require different extraction strategies-scanned scanned PDFs with non-selectable text, tables with merged headers, figures with tiny tick labels, map- s/plots requiring numeric reading, and videos where a chart must be captured at specific timestamps. Mix machine-readable files (CSV/SOO//PP) with non-machine-readaae facts (images, scanned documents) so that visual decoding or OCR is unavoidable.
arti-
single clean HTML page simple, fully searchable PDF with neatly structured text that can be solved via copy-paste without visual parsing or layout rea- soning.
Negative: Relying
D. Domain Transfer to Specialized Contexts
Positive: Migrate familiar capability into niche, expert-only setting where surface skills no longer suffice. The goal is to preserve the underlying task type while drastically increasing domain complexity and data difficulty.
Negative: Superficial re-skinning that keeps the data simple (e.g., switching to another modern font or different but equally clean dataset). Tasks that can still solved with generic OCR or shallow keyword matching without domain-specific adaptation do not con- stitute meaningful domain transfer.
E. Toolchain Planning and Dependency
Positive: At least three distinet tools a dependent chain (e.g., web search - file down- loader/OCR - parser/vision model - code interpreter), where the output of one is the input of the next, including at least one aalidation/backtracking step.
Negative: One-shot of single tool or parallel, non-dependent calls whose results don't t constrain each other.
F. Abstract Logic, Board/Game, and Modeling Tasks
Positive: Begin by diagnosing the primary difficulty nucleus of the task (e.g., search branch- ing factor, constraint density, horizon length, state observability, symmetry/degeneracy, or proof depth).
Negative: Superficial changes that do not touch the core difficulty (c.g., renaming pieces, small cosmetic rule tweaks, or adding single extra variable/constraint that leaves search trivial). Avoid expansions that only increase input length without increasing constraint inter- actions, that introduce ambiguity making solutions non-unique unverifiable.
Figure 5: The bottleneck demonstrations in the system prompt of our Evolution Proposer agent.
20

---

Published as a conference paper at ICLR 2026
Exploration Executor
Part Your Mission & Role
You are an advanced Al agent specializing in "Problem Evolution". Your mission is to take an existing intelligent agent task and evolve it into more complex and difficult version.
You will be provided with the specific description of this task. Additionally, you may also be given materials such as the general solution steps for the task, the final answer, and the tools that need be used. More importantly, you will also receive some improvement ideas, and you are required to increase the difficulty of the problem based on the guidance of these ideas. ** Your Core Principles of "Problem Evolution".**
1. ** *Divergent Evolution:**
At each step. if solution steps are provided, you may simulate or refer them to explore the intermediate states in the original problem-solving process. If no solution steps are provided, you may conduct exploration according to your own ideas. If you believe that the difficulty of the problem can be increased based on one of the ideas from the "improvement ideas" during the exploration at this step, then carry out specific exploration to determine the detailed implementation of this idea.
2. **Ineerse Problem Creation:**
After you have implemented some of the improvement ideas (it is not necessary to implement all of them, as some ideas may not be feasible to put into practice), you will create a more difficult task in reverse based on the new information you have acquired. Cycle Your entire process is a continuous, step-by-step cycle of **Thought, Code, and Observa- tion. You MUST follow this structure for every action you take...
Part 2: Your Working Process: The Mhought-Code-Obseee
Part 3: Guiding Principles of Evolution
#*1. The Golden Rule of Evolution: The Burden of Discovery (CRITICAL PRINCIPLE)** Your primary goal is maximize difficulty by forcing the solver to perform two distinct, non-negotiable actions: #*1) Discovering all necessary information** and #*2) Deducing the entire solution path.* ** Complexity must arise from the solver's own exploration and problem decomposition, not from following recipe you provide.
**2. Principle of WoaldWorld Grounding & Authenticity (CRITICAL PRINCIPLE)** The new task must be meaningful evolution of the original, grounded in verifiable reality.
**3. Principle of Logical Integrity and Solvability (CRITICAL PRINCIPLE)** You must ensure the evolved task is #*well-defined, solvable puzzle** It must be complete, clear (unambiguous), and consistent (no contradictions). Your goal is to create a challenging but fair and solvable task.
#*4. Uniqueness, Determinism, and Verifiability of the Answer (CRITICAL PRINCIPLE)**
Your primary directive is create problems where the final answer is * **singular, determinis- tic, and verifiable via simple string match ('=')**
Here are few simple examples using notional tools, and your task should be more complex: (Workflow Demonstrations
Above example were using notional tools that might not exist for you. On top of performing computations in the Python code snippets that you create, you only have access to these tools, behaving like regular python functions:
1%- for tool in tools, values() %
[( tool.to_toool xool_calling_promppeee }}
%% endfor %)
Figure 6: The system prompt of our Exploration Executor agent.
21

---

Published as a conference paper at ICLR 2026
Fine-grained Validator
You meticulous AI Output Verifier. Your mission is determine if an agent's claimed "Observation" is informationally consistent with the "Actual Output" from its code execution. This is NOT core question is: **oDo the Actual Output contain sufficient evidence to justify the Obser- vation?"**
for exact string equality. is test for **informational fidelity* The
Part 1: The Core Principle
* **Juugement TRUE (Consistent):*** ** The Observation is correct and logical conclusion that can be derived from the Actual Output. The Actual Output factually supports every piece of information claimed the Observation.
**Judgement FALSE (Inconsistent): ** The Observation makes claim that is contradicted by, or cannot be verified from, the Actual Output.
Part 2: Rules of Semantic Equivalence
A. PERMITTED VARIATIONS (Judgement: TRUE)***
1. ** Whitespace and Formatting: ** Differences irrelevant.
2. **Data Structure Order:** Order of keys in JSON or items in unordered lists does not
matter.
3. **Floating-Poin Precision:**
Minor differences are acceptable (e.g., *0.333* vs.
*0.333333').
4. **Extraneous Information:* The Actual Output can contain more details; the Observation can be a subset or summary.
5. **Natural Language Equivalence: Different phrasing with the same meaning is accept-
able.
**B. FATAL FLAWS (Judgement: FALSE):**
1. Factual Contradiction: Conflicting facts (e.g.. Actual: '"42", Observation: '"43"').
2. ** Missing Information: Observation claims something present in the Actual Output.
3. *Tpppe Mismatch: Actual Output is an error, but Observation claims success.
Part 3: Output Format
You MUST provide your response in JSON object format with exactly two keys: "Final Judgement" and "Reason".
**Template
{ "Final Judgement": "TRUE/FALSE"" "Reason": "Your reasoning here.") "
Part 4: Your Task
Now, analyze the following pair of outputs and provide your response the specified JSON format.
Figure 7: The prompt of our fine- fine-grained validator.
22

---

Published as a conference paper at ICLR 2026
Overall Validator
Part 1:
Your Mission & Role
You strict agent task reviewer. Your mission is judge two things: the validity of a new task adapted from the original task, and whether the new task has increased in difficulty compared to the original one.
First, you will be provided with a description of the original task. Additionally, you may also given reference information such as the original task's solution steps, answer, required tools, and attached file materials.
Subsequently, you will receive the new task adapted from the original task. along with all rele- vant information including its description, solution steps, answer, required tools, and attached file materials.
Cycle Your entire process is a continuous, step-by-step cycle of **Thought, Code, and Observa- tion. ** You MUST follow this structure for every action you take...
Part 2: Your Working Process:
The Thouugh-CCdd--Ohsrraaa
Part 3: The Core Verification Conditions
The conditions for the new task pass verification are as described in (1) (5) below. task must satisfy ALL five conditions to pass.
**(1) Check the Verifiability and Format of the New Task's Answer** e ** Answer Verifiabil-
ity:** The question should have verifiable answer. It must not rely on subjective qualifiers or ask for open-ended explanations.
**(2) Verify the Solution Steps of the New Task**
### 1. Step-by-Stad Verification For each step in the 'solution trace", you must perform the following checks:
** Logical Continuity (Thought vs. Previous Observation):**
* Thhought-Code Implementation:
**Critical Rule of Evidence: Prohibit Factual Simulation:
### 2. Holistic Final Review After verifying all individual steps, you must perform final,
holistic review of the entire logical chain: Problem Solved?:** Does the final conclusion or output actually and accurately solve the new task?
**(3) Check the Completeness of the Question and the Uniqueness of Answer When you have finished reviewing this new task and its solution trace, you need to make the following
judgments: **Completeness of the Question:
# +* Unnqueness of the Answer: #*
** **(4) Verify the Task's Complexity Improvement** The new task must be demonstrably and significantly more complex than the original in at least one of the following, with explicit evidence in the solution trace that the main bottleneck has been made harder:
**oolttion Path Complexity (Depth & Coupling):***
* Problemm Formulation Complexity (Discovery Burden):**...
DDomain Transfer Hardening (Same core skill, harder domain):***.
* TTollchai Planning & Dependency:***
Abstract Logic/ / Board Modeling Hardness:**..
Pass Criteria for (4): You must identify (a) the original bottleneck, (b) the new bottleneck, and (c) concrete evidence from the solution trace (e.g., added modalities, reconciliation steps, hashes/metadata, -omaii--peecffic references) showing a net increase difficulty along at least axis above. If these not clearly demonstrated, mark as FAIL. Here are a few simple examples using notional tools, and your task should be more complex: (Workflow Demonstrations)
Above example were using notional tools that might not exist for you. On top of performing computations the Python code snippets that you create, you only have access to these tools, behaving like regular python functions:
( 1%- for tool in tools, values() %
|l tool.to tool calling calling_prompt(( H
{ %6 endfor %)
Figure 8: The system prompt of our Overall Validator agent.
23

---

Published as a conference paper at ICLR 2026
Case
Original Task:
If Eliud Kipchoge could maintain his record-making marathon pace indefinitely, how many thousand hours would take him run the distance between the Earth and the Moon its closest approach? Please use the minimum perigee value on the Wikipedia page for the Moon when carrying out your calculation. Round your result to the nearest 1000 hours and do not use any comma separators if necessary.
Evolved Task:
Calculate the time it would take to the distance of the Moon's minimum perigee, using the average pace from Eliud Kipchoge's 2018 marathon world record, and again using the average pace from his 2022 marathon world record. The minimum perigee distance and both world record times should be sourced from Wikipedia. What is the absolute difference between these caleulated travel times? Provide your answer in hours, rounded to the nearest integer. The answer should be single numerical string. Analysis:
The evolved task is harder because it requires retrieving and cross-checking more Wikipedia data (the Moon's 's minimum perigee plus two separate marathon records from 2018 and 2022), deriving two average paces, performing two long-distance time computations with consistent unit conversions, and then taking an absolute difference, all while adhering to stricter out- put formatting and rounding to the nearest hour. This multiplies the number of chained steps, increases opportunities for parsing and calculation errors, and demands tighter procedural con- trol than the original task, which needs only single pace, single computation, and coarser rounding instruction.
Figure 9: Task evolution case 1.
Case
Original Task:
How many studio albums were published by Mercedes Sosa between 2000 and 2009 (in- cluded)? You
the latest 2022 version of english wikipedia.
Evolved Task:
Calculate the number of full years that passed between Mercedes Sosa's return Argentina after her exile and the year she posthumously won her last Latin Grammy Award for 'Best Folk Album', Your answer must be single integer. Analysis:
The evolved task is harder because requires integrating biographical chronology and awards data from multiple Wikipedia pages, disambiguating the precise date of Mercedes Sosa's 's return from exile (which can be described variably sources) and identifying the exact year her posthumous "Best Folk Album" Latin Grammy, then interpreting "full years that passed" to handle date boundaries correctly before performing the subtraction. By contrast, the original task is constrained filtering/counting problem over discography. It locate studio albums and count those with years between 2000 and 2009 inclusive, which entailing fewer sources, less ambiguity (once "studio" is applied), and single straightforward tally rather than multi-step temporal reasoning.
Figure 10: Task evolution case
24

---

Published as a conference paper at ICLR 2026
Case 3
Original Task:
Here's a fun riddle that think you'll enjoy.
You have been selected to play the final round of the hit new game show "Pick That Ping- Pong". In this round, you will be competing for large cash prize. Your job will be to pick one several different numbered ping-pong balls, and then the game will commence. The host describes how the game works.
A device consisting of a winding clear ramp and a series of pistons controls the outcome of the game. The ramp feeds balls onto platform. The platform has room for three ping-pong balls at a time. The three balls on the platform are each aligned with one of three pistons. At each stage of the game, one of the three pistons will randomly fire, ejecting the ball it strikes. If the piston ejects the ball in the first position on the platform the balls in the second and third position on the platform each advance one space, and the next ball on the ramp advances to the third position. If the piston ejects the ball in the second position, the ball in the first position is released and rolls away, the ball in the third position advances two spaces to occupy the first position, and the next two balls on the ramp advance to occupy the second and third positions on the platform. If the piston ejects the ball the third position, the ball in the first position released and rolls away, the ball in the second position advances one space to occupy the first position, and the next two balls on the ramp advance to occupy the second and third positions the platform.
The ramp begins with 100 numbered ping-pong balls, arranged in ascending order from to 100. The host activates the machine and the first three balls, numbered 1, 2, and 3, advance the platform. Before the random firing of the pistons begins, you are asked which of the 100 balls you would like pick. If your pick is ejected by one of the pistons, you win the grand prize, $10,000.
Which ball should you choose maximize your odds of winning the big prize? Please provide your answer as the number of the ball selected.
Evolved Task:
You have been selected for the final round of 'Pick That Ping-Pong'. The game's rules are as follows:
A machine controls the game. A ramp feeds numbered ping-pong balls onto platform that holds three balls at a time, in positions 1, 2, and 3. Three pistons are aligned with these positions.
the piston at position fires, the ball is ejected. The ball from position 2 moves to 1, position 3 moves to 2, and new ball from the ramp takes position 3. If the piston position 2 fires, the ball is ejected. The ball at position 1 is discarded. The ball from position 3 moves 1, and two new balls from the ramp take positions 2 and 3. the piston at position 3 fires, the ball is ejected. The ball at position is discarded. The ball from position 2 moves to I, and two new balls from the ramp take positions and
Each piston has equal 1/3 probability of firing at each stage.
This time, the ramp is loaded with 100 balls numbered according to the Fibonacci sequence, starting with F(1)=1, F(2)=1, F(3)=2, and so on. The first three balls (1, 1, 2) are already on the platform.
Your goal is to maximize your expected score. The you receive is equal to the number on the ball if it's ejected. You must choose one of the first three balls on the platform. Which ball number should you choose? Provide your answer single integer. Analysis:
The evolved task harder than the original because it replaces a pure hit-probability maxi- mization with an expected-value optimization under heterogeneous rewards. In the evolved version, starting from the specific state [1, 2], you must compute, for each of the three can- didate balls, its probability of being ejected (not merely discarded) and then weight that by its Fibonacei value, which breaks the symmetry that simplified the first problem. The dupli- cate "I"s introduce labeling subtlety, the "2" offers higher payoff but different positional risk, and the loss uniformity means expected scores depend sensitively on the detailed transition structure. This added value- cooeee-probbbttyy trade-off and the need for finer conditional expecta- tions make the evolved task more intricate.
Figure 11: Task evolution case 3.
25

---

Published as a conference paper at ICLR 2026
Case 4
Original Task:
My family reunion is this week, and I was assigned the mashed potatoes bring. The atten- dees include my married mother and father, my twin brother and his family, my aunt and her family, my grandma and her brother, her brother's daughter, and his daughter's family. All the adults but me have been married, and no one is divorced or remarried, but my grandpa and my grandma's sister-in-law passed away last year. All living spouses are attending. My brother has two children that are still kids, my aunt has one six-year-old, and my grandma's 's brother's s daughter has three kids under 12. figure each adult will eat about 1.5 potatoes of mashed potatoes and each kid will eat about 1/2 a potato of mashed potatoes, except my sec- ond cousins don't carbs. The average potato is about half pound, and potatoes are sold 5-pound bags. How many whole bags of potatoes do I need? Just give the number. Evolved Task:
I'm making mashed potatoes for family reunion. The recipe requires stick of butter for every 8 potatoes. My family includes my married mother and father, my twin brother and his family (wife, two kids), my aunt and her family (husband, brother, his daughter, and his daughter's family (husband, three kids). All living spouses are attending. However, my aunt's family decided not to come since the outdoor picnic forecast calls for rain. Additionally, my great-uncle's 's daughter and her entire family are strict keto diet and will not be eating any potatoes.
child), my grandma, her
For those eating, standard adult portion is 1.5 potatoes and standard kid portion is 0.5 potatoes. However, my twin brother is bulking and will eat a double portion, while my mother is watching her carbs and will only eat half portion.
An average potato weighs half pound. Potatoes are sold 5-pound bags costing $3.99 each. Butter is sold in individual sticks costing $1.25 each. You must buy whole bags of potatoes and whole sticks butter. What is the total cost for the potatoes and butter need to buy? Provide the answer string in the format "XX.XXX. Analysis:
The evolved task is harder because it requires filtering who actually attends and who eats, accounting for dietary exclusions, handling variable portions (a double portion for your twin brother and half portion for your mother), and managing two separate ietts-potatoes and butter-with different pricing and integer rounding constraints, all while converting from por- tions to potato counts to weight to bags and then calculating butter sticks and total cost; the original task only involves counting adults and kids with uniform portions, converting weight, and rounding up whole potato bags, making it much simpler scope and steps.
Figure 12: Task evolution case
26

---

Published as a conference paper at ICLR 2026
Case
Original Task:
the year 2022, and before December, what does "R" stand for in the three core policies of the type of content that was violated in the public logs on the Legume Wikipedia page? Evolved Task:
Find the user who performed the original page move for the Wikipedia article 'Legume'. What is the title the article associated with that user's first-ever edit on Wikipedia? The answer should be the article title, correctly capitalized. Analysis:
Both tasks require digging through Wikipedia, but they emphasize different skills and sources: the original task hinges on interpreting the Legume page's public logs to identify the specific content violation and then mapping that violation to Wikipedia's "three core policies" as they were defined before December 2022, isolating what the "R" stands for; this involves policy literacy, time-bounded interpretation, and resolving potential ambiguity in log descriptions and policy acronyms. The evolved task is a chain of archival lookups: locate the original page move for "Legume." " identify the user who performed it, navigate that user's contribution history, and extract the title of the article from their first-ever edit with correct capitalization; this stresses precise provenance tracing, familiarity with page histories and user logs, and attention to naming conventions rather than policy interpretation.
Figure 13: Task evolution case 5.
Case
Original Task:
What writer is quoted by Merriam-Webesn for the Word of the Day from June 27, 2022?
Evolved Task:
Visit the Merriam-Webster 'Word of the Day' page for June 27, 2022. According to the ymology provided in the 'Did You Know?" section, the word 'jingoism' originated in the Identify the full name of the primary treaty that offi- cially and finally concluded this war. Your answer should be the name of the treaty. Analysis:
context of a specific 19th-century
Both tasks start from the same Merriam- Webster Word of the Day page for June 27, 2022, but they emphasize different skills: the original task is a direct fact lookup identify which writer is quoted on that page-requiring accurate navigation and citation capture from sin- gle source. The evolved task chains that page to external historical research: use the "Did You Know?" etymology to identify the specific 19th-century war linked to "jingoism," then determine which treaty officially and finally concluded that war, taking care to distinguish pre- liminary accords from the definitive settlement and to provide the treaty's 's full formal name. This adds steps of context extraction, cross-referencing, and precision in historical nomencla- ture beyond the initial page.
Figure 14: Task evolution case 6.
27

---

Published as a conference paper at ICLR 2026
Case 7
Original Task:
Given this table defining on the set S b, c, d.
|*lalblcddle
lalalble/bldI
lbbblclalelcl
lclclalblblal
ldible/bleld!
leldiblaldicl
provide the subset of involved in any possible counter-examples that prove is not commu- tative. Provide your answer as comma separated list of the elements in the set in alphabetical order.
Evolved Task:
Consider binary operation *** defined on the set $ = 0, 1, 2, 5, The operation defined by the formula: x y= (3*x 5*y) mod 8. Determine the total number of ordered triplets (a, b, c), where a, b, and are elements of S, for which the associative property fails, (b c). Provide the final answer string containing
i.e., (a b) is not equal
single integer.
Analysis:
Both tasks concern algebraic properties of binary operations but require different approaches: the original task table-based analysis commutativity and then report the subset of elements involved table, detecting asymmetries zy # yæ, and aggregating the implicated symbols in sorted order. The evolved task shifts to an eight-element modular operation defined by zy == (3x + 5y) mod and asks for a global count of associativity failures over all ordered triplets, which entails formulating and checking the associativity condition either by full enumeration of g3 cases or by deriving algebraic criteria for when (ab)c ff a (bx c) holds; emphasizes modular arithmetic, structural properties of linear-combination operations on Zs. and programmatic verification to ensure complete and accurate counting.
find any pair witnessing non- five-element this is about reading a Cayley
Figure Task evolution case
28

---

Published as a conference paper at ICLR 2026
Case 8
Original Task:
You are telecommunications engineer who wants build cell phone towers stretch of road. In the reference file a layout of the road and nearby houses. Each dash, "-", a marker indicating mile. Each capital H indicates house located next to mile marker, appearing above or below the stretch of road. Each cell phone tower can cover houses located next to the road within 4-mile radius. Find the minimum number of cell phone towers needed cover all houses next to the road. Your answer should positive numerical integer value. Evolved Task:
You are telecommunications engineer tasked with deploying cell towers on a 2D grid defined in the provided Excel file, eeepoyment_grid.xxe deployment cost to provide coverage to all houses.
Your goal is find the minimum total
Grid Rules: The grid defined in the DeploymentGrid' sheet of the Excel file. Cell values represent the content of the grid: 'H': A house that needs coverage. Towers cannot be built on these cells. 'X': no- no-build zone. Towers cannot be built on these cells. 'G': Green terrain (Terrain Cost 1) "Y': Yellow terrain (Terrain Cost 3) 'B': Blue terrain (Terrain
Cost 5)
Tower Types & Costs: You have two types of towers available: 1. 'Pico Tower': Base Cost
= 10, Coverage Radius = 2 2. 'Macro Tower': Base Cost = 30, Coverage Radius = 4 The "Total Deployment Cost' for single tower is its 'Base Cost' + the 'Terrain Cost' of the cell it is placed on. The total cost for the project is the sum of costs for all deployed towers.
Coverage Rules: A tower '(r1, cl)' covers house at '(r2, c2) if the Manhattan distance between them is less than or equal to the tower's 's radius. Manhattan Distance = 'Irl - r21 + Icl -c2
Your task is to determine the absolute minimum total deployment cost to ensure every house on the grid is covered by least one tower. The final answer should be a single integer representing this minimum cost. Analysis:
The evolved task is substantially harder than the original: the original is one-dimensional placement problem along a road with identical towers and a uniform 4-mile radius, reducible classic interval covering problem solvable in linear time by simple greedy strategy that repeatedly places a tower as far right as possible within miles of the leftmost uncovered house; by contrast, the evolved task operates forbidden cells (H, X), buildable terrains (G/Y/B) that impart different terrain costs, and two tower types (Pico radius with base cost 10, Macro radius 4 with base cost 30), where each tower's 's deployment cost is its base cost plus the terrain cost and coverage uses Manhattan distance, making the objective of minimizing total cost while covering all houses weighted set cover/facility location problem: each buildable cell paired with a tower type is a candidate facility with a specific cost and coverage set, and we must select a minimum-cost subset cover- ing every H, which is NP-hard in general and thus best modeled via an integer linear program with binary decision variables and coverage constraints or approximated with heuristics such cost-effictiveenes greedy or Lagrangian methods, all preceded by parsing the spreadsheet and precomputing coverage sets.
two-dimensional Excel-defined grid with
Figure 16: Task evolution case
29

---

Published as a conference paper at ICLR 2026
Case
Original Task:
Jen enters a lottery by picking 4 distinct numbers from S {1, 2,, 9. 10). numbers are randomly chosen from S. She wins a prize at least two of her numbers were of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers. The probability of her winning the grand prize given that she won a prize is 122 where and are relatively prime positive integers. Find + Evolved Task:
Jen picks set J of 4 distinet numbers from S {1, 2, ..... 10]. Tom then picks set 7 of 4 distinct numbers from S, chosen uniformly at random from all such sets that have exactly one number in common with (i.e., [J
A lottery then randomly draws L of 4
distinct numbers from S.
'J-only Prize' is awarded the lottery set intersects Jen's J but is completely disjoint from Tom's 's set T. The probability of this prize being awarded can be expressed as a fraction =, where m and are relatively prime positive integers. Find m n..
Analysis:
The adapted problem is significantly more difficult and conceptually deeper than the original. The original problem is standard exercise in combinations and conditional probability. The solution path straightforward: one simply needs to calculate the number of outcomes for the winning event ("at least two matches") and for the grand prize event ("four matches"), then find their ratio. It primarily tests a student's fluency with combinatorial formulas. The adapted problem, however, introduces a higher level of complexity by adding a third set, T, with specific relationship to (JJ nT = 1). This transforms the problem from simple counting exercise into one requiring sophisticated logical and set-theoretical reasoning. The key solving it is not direct calculation, but the conceptual leap of partitioning the universal set S into four distinct subsets based on and Once this structure understood, the calculation becomes surprisingly simple. In essence, the difficulty shifts from mechanical computation to conceptual abstraction. The adapted problem is harder because demands deeper insight into the underlying structure of the sets involved, making it more elegant and challenging mathematical puzzle.
Figure
Task evolution case
30

---

Published as a conference paper at ICLR 2026
Case 10
Original Task:
Alice and Bob play the following game. A stack of tokens lies before them. The players take turns with Alice going first. On each turn, the player removes either token or tokens from the stack. Whoever removes the last token wins. Find the number of positive integers less than or equal to 2024 for which there exists a strategy for Bob that guarantees that Bob will win the game regardless of Alice's play. Evolved Task:
An impartial game is played with two stacks of tokens, with sizes n and m. Two players take turns making moves. A move consists of removing k tokens from $each stack. The set of allowed values for depends on the current minimum stack size, p min (n, m): [oo]:-Ifpp If < 20, the allowed moves are {1,2, 4}. If 20, the allowed moves are k E {1,4].
A move is only possible both stacks have at least tokens. The player who makes the last possible move wins. A starting position (n, m) is 'losing position" if the second player has guaranteed winning strategy.
Find the total number of losing positions (n, m) such that Analysis:
100 and S 100.
The leap in difficulty from the original the adapted problem is enormous. The original a standard one- miedennnsional game theory exercise where the solution hinges on discovering a simple, repeating pattern in the losing positions (P-positions). primarily tests pattern recog- nition.
The adapted problem, while appearing to be a more complex two-dimensional game, is a sophisticated test of abstract reasoning. The critical insight, which is far from obvious, is that the game's two-dimensional state (n, m) collapses. The winning or losing status of any position is determined solely by the minimum of the two stacks, p min(n, m), and which set rules applies at that value p. This reduces the problem to one-dimensional analysis again, but that is masked by misleading setup and complicated by conditional rules. The difficulty is therefore elevated from simple nattern-findig followed by significantly more complex counting phase. It's a shift from solving problem figuring what the problem really is.
much higher level of model simplification and abstraction,
Figure 18: Task evolution case 10.
31


Reviews:
[{"review_id": "Vm74CXrx13", "reviewer": "Reviewer_H9KX", "summary": "This paper explores building self-evolving AI agents that can continuously improve their capabilities through experience accumulation. The proposed framework includes iterative cycles of experience collection, reflective learning, and capability updating. Agents learn not only from successes but also extract lessons from failures. Through meta-learning mechanisms, the system gradually optimizes decision strategies and behavioral patterns. Experiments demonstrate that self-evolved agents show sustained performance improvement on long-horizon tasks.", "questions": "1. How do you distinguish valuable experiences from noise? Is there an experience filtering or quality assessment mechanism?\n2. When agents evolve undesirable behaviors, how can you rollback or correct them?\n3. In multi-task scenarios, how do you balance learning across different tasks? How to avoid specializing in one task while regressing on others?\n4. How can self-evolution integrate with human feedback? Can users guide evolutionary direction?\n5. What accounts for the performance fluctuations observed in long-term evolution experiments? How stable is the process?\n6. Do different agent types (rule-based vs. learning-based) differ in evolvability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Long-term perspective**: Focuses on continuous agent evolution rather than single-task performance, aligning with real-world needs\n2. **Learning from failure**: Values not just successes but also reflection on failures, more closely resembling human learning\n3. **Well-designed experiments**: Long-term tracking studies observe capability improvements over sufficient evaluation periods\n4. **Good interpretability**: Experience repository and reflection logs enable tracing the agent's learning process\n5. **General framework**: Method isn't limited to specific task types, showing reasonable generalizability", "weaknesses": "1. **Catastrophic forgetting**: The paper insufficiently addresses this classic continual learning problem - how to ensure learning new capabilities doesn't erase old ones?\n2. **Evolution direction control**: Self-evolution may proceed in undesired directions, like over-optimizing certain metrics. How is evolutionary direction guided and constrained?\n3. **Convergence concerns**: No theoretical guarantee that evolution converges to better strategies - may get stuck in local optima. Experiments also show some performance fluctuations\n4. **Experience quality dependency**: Relies on automatically labeled experience quality, but labels may be inaccurate. Erroneous experiences can mislead subsequent learning\n5. **Computational resources**: Continuous self-optimization requires sustained computation, potentially infeasible in resource-constrained environments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z3Q8k3JVYz", "reviewer": "Reviewer_aYop", "summary": "This paper proposes a method for the self-evolution of benchmarks, aiming to address the issue of benchmark performance saturation. Through multi-agent collaboration, the method further extends the difficulty and diversity of questions in existing benchmarks.\n\nThe approach consists of three stages, each carried out by a different agent:\n* The first agent proposes potential directions for question evolution.\n* The second agent generates evolved questions according to the proposed directions.\n* The third agent verifies the feasibility and validity of the evolved questions.\n\nThis framework provides an automated mechanism for question expansion within benchmarks, and experimental results show that the evolved questions are significantly more challenging.", "questions": "* Could you include quantitative analyses of model capabilities after benchmark evolution? For instance, comparing the distribution of capability dimensions between the original and evolved benchmarks, or analyzing differences in token lengths of correct responses, could help clarify what specific aspects of model performance have degraded. Such statistics would make it clearer what the observed performance drop actually indicates about model weaknesses.\n\n* Moreover, how to interpret changes in the ranking order of models across the original and evolved benchmarks? If the relative ranking of models shifts significantly, does it suggest that the evolved benchmark measures distinct or more nuanced abilities that were underrepresented before?\n\n* Finally, the authenticity and validity of the evolved questions should be further examined. It remains unclear how the framework ensures that increased difficulty does not lead to artificial or ill-posed questions that merely make tasks harder without providing meaningful evaluation signals.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This framework provides an automated mechanism for question expansion within benchmarks. The expansion process is fully automated, requiring no additional human annotation.  \n* Evaluation results show that the evolved tasks are significantly more challenging for models to complete, while the validation stage ensures that all evolved tasks remain solvable and valid.", "weaknesses": "A benchmark is not designed merely to make models score lower; rather, its purpose is to ensure that the scores reflect meaningful aspects of model capability. Therefore, this work appears to lack some necessary analyses to demonstrate that the benchmark evolution preserves this interpretability of scores.\n\n* This work lacks a detailed statistical analysis of the evolved tasks — for example, in the style of GAIA, which provides statistics on the types of skills or capabilities required to complete each task.  \n\n* In addition, it does not provide a systematic analysis of the observed score degradation after benchmark expansion, which would help to clarify which aspects of task difficulty contribute most to the model's performance drop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n7XUpNbIFQ", "reviewer": "Reviewer_dJpA", "summary": "The paper proposes TRACE, a self-evolving benchmark framework generating harder tasks via 3 stages (proposal mining, exploration, validation) with validatable trajectories. It enhances complexity/diversity, outperforms static benchmarks on GAIA. Paradigm shift to dynamic, self-evolving benchmarks enabling sustainable, adaptive, and continual agent evaluation progress.", "questions": "none.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The framework is carefully engineered and empirically validated, featuring a well-defined three-stage pipeline supported by detailed algorithmic design.\n\n2. The paper is clear, written making it easy to follow the motivation, methodology, and outcomes.", "weaknesses": "1. While the paper presents an interesting multi-agent system for benchmark evolution, its distinct contribution relative to existing “agentic” methods is not sharply delineated. \n\n2. Experiments are mostly centered on GAIA, with limited exploration across other benchmarks or domains. \n\n3. The multi-agent pipeline (Proposer–Executor–Validator) introduces substantial systemic and computational overhead, which may limit practical deployment. The paper does not discuss resource consumption, time efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mmcgaEgl7E", "reviewer": "Reviewer_s49g", "summary": "This paper proposes **TRACE**, an automated pipeline for generating new and harder tasks from existing benchmarks.  The system leverages a combination of LLM-based reasoning and validation to transform existing agent tasks into more complex variants, with the stated goal of alleviating benchmark saturation and stimulating progress in reasoning-heavy domains.  TRACE operates by (1) analyzing existing benchmarks such as GAIA, (2) proposing modifications that make the tasks more challenging, and (3) validating the modified tasks through an auxiliary solver before including them in the new dataset.  Empirically, the authors demonstrate that models perform worse on these generated tasks, suggesting that TRACE successfully creates more difficult evaluations.  The idea of using LLMs to automatically expand and stress-test benchmarks is promising and timely.\n\nThis is an exciting and well-motivated step toward automated benchmark evolution, but the methodology currently conflates task difficulty with model failure and lacks grounding in construct validity.   With better controls, clearer interpretability, and a modular release for community use, this work could become a genuinely valuable contribution to the field.", "questions": "1. **Auxiliary validator model.**  \n   What exact model serves as the validator? How sensitive are the results to its capacity (e.g., Qwen3-32B vs Qwen3-480B)?  \n\n2. **Independent validation of difficulty.**  \n   Could human annotators or external scoring metrics (e.g., model confidence, entropy, or step length) be used to verify that TRACE-generated tasks are indeed harder for legitimate reasons?  \n\n3. **Construct validity assessment.**  \n   How do the authors define the “meaningfulness” of a task?  Have they considered measuring correlations between TRACE scores and other established benchmarks of reasoning or problem-solving ability?  \n\n4. **Integration and reproducibility.**  \n   Will the authors release TRACE as an open-source toolkit that can plug into existing agent benchmarks (e.g., via a simple adaptor API)?  This would significantly enhance its impact and adoption.  I also hope that we can see if the new tasks can be turned into an RL environment.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Originality and relevance.**  \n   The notion of an *automatic benchmark generator* is highly relevant. Existing benchmarks saturate quickly, and manually curating new tasks is costly. TRACE offers a compelling automation framework that can rejuvenate established benchmarks by introducing more challenging variants.  \n\n2. **Methodological soundness (conceptually).**  \n   The pipeline’s modular design—comprising generation, validation, and filtering—provides a flexible structure that could be extended to multiple domains. In principle, this could evolve into a general-purpose benchmark-expansion tool.  \n\n3. **Potential for community impact.**  \n   If released as an open-source package, TRACE could become a widely adopted toolkit. It could serve as a plug-and-play module attached to popular agent benchmarks (e.g., GAIA, SWE-Bench, Terminal-Bench, WebArena).  I encourage the authors to release TRACE as a standalone, modular Python package or API so that the community can easily apply it to existing datasets.  Since some teams (e.g., Terminal Bench) have been building adapters to existing agent benchmarks, this is probably not too much work.  Doing so would make TRACE a genuine contribution of infrastructure and reproducibility.\n\n4. **Empirical value.**  \n   Demonstrating that LLMs perform significantly worse on TRACE-generated tasks suggests that the system indeed increases difficulty and task complexity.  The paper shows that the generated tasks are longer, require more steps, and involve deeper reasoning than the originals.\n\n\n\nLastly, the command of English is excellent.  Good job.", "weaknesses": "Despite its creativity and potential, the methodology has a few central problems.  Problems 1 is the most severe since it undermines all data reported in the paper.  Problem 2 should also be fixed.  I don't mind if problem 3 is left unsolved.\n\n---\n\n### 1. Selection bias — selecting on what you are reporting\n\nThe main methodological flaw is **selection bias** in the task-filtering procedure.   The pipeline uses an “auxiliary validator” (the specific model is unclear to me—possibly Qwen3-Coder-480B?) and retains only tasks that this validator cannot solve.   Subsequently, the paper reports that LLMs perform worse on these filtered “new tasks” than on the original dataset.  This comparison is inherently **unfair** because the filtering criterion (unsolved by validator) directly ensures lower model performance.  \n\nTo test whether the decreased performance is meaningful rather than tautological, I encourage the authors to perform a **control experiment**:  run the same filtering procedure on the original GAIA dataset, selecting only the tasks the validator cannot solve, and then compare model accuracy on this filtered subset.   I predict the same “performance drop” would appear even without TRACE-generated tasks, simply because of selection bias.\n\nBy the way, this problem mirrors the issue in *Humanity’s Last Exam (HLE)*: if one only includes tasks that all frontier models initially fail, one will trivially obtain a benchmark where all frontier models score poorly.  Such difficulty is not an intrinsic property of the tasks—it merely reflects transient weaknesses of current models.  As newer models emerge, the scores rise sharply, revealing that the selection mechanism was biased toward the temporary state of frontier capabilities.  This is what happened with HLE.  A few months later, when the newer models are released, there was an instead jump in scores.\n\nTo make the “decreased performance” claim meaningful, the authors should either:\n- provide an **independent definition of difficulty**, not one derived from model failure; or  \n- report **orthogonal measures of benchmark quality**, such as separability with confidence (see the Agent-Hard paper) or correlation with human difficulty ratings.\n\n---\n\n### 2. Construct validity — realism and meaningfulness of new tasks\n\nConstruct validity is the core problem here. See the literature on measurement in general.  Goes back to Cronbach, L. J., & Meehl, P. E. (1955).  While the new tasks may indeed require more steps and longer trajectories, it is unclear whether they measure anything meaningful or realistic.  In benchmark design, *construct validity*—the degree to which a test measures what it purports to measure—is paramount.  \nA benchmark should not only resist saturation but also preserve *semantic relevance* and *economic relevant*.\n\nEven thouhg SWE-Bench Verified has a lot of bugs, it has a clear external meaning: solving GitHub issues corresponds to resolving real software bugs.  If all models achieve $100%$ on GSM8K, we can still interpret that success as mastery of grade-school math reasoning.  If all models do well in telling dogs and cats apart, we can still understand the state of progress in Computer Vision.  But what exactly does it mean, conceptually or practically, for deepseek-v3 to achieve $24.7%$ on GAIA-TRACE versus $41.8%$ on GAIA?  If these scores cannot be linked to a real-world applicable capabilities, then the numbers risk becoming meaningless.  \n\nThis issue parallels the design philosophy behind overly artificial exams (such as the Chinese Gao Kao): one can always construct harder and more complicated problems, but hardness alone does not confer value.  TRACE currently optimizes only for *difficulty* and *saturation resistance*, rather than for *meaningfulness* or *construct validity*.  Without grounding in a real-world construct, the benchmark may end up measuring the ability to chain API calls or manipulate evidence density—interesting but not inherently valuable.  How should you explain to my grandma, an outsider to the field, what this 24.7% mean here, if she wants to make an investment decision in AI agents?\n\nTo address this, I recommend that the authors:\n- provide human evaluations of *task realism* and *relevance*; \n- discuss construct validity of the benchmark in light of the measurement-theory literature;\n- help readers understand why the new tasks are not just messier and more complicated to solve but also meaningful and realistic to expect an agent to solve.\n\nThis will encourage community adoption. \n\n---\n\n### 3. Missing comparisons and broader context\n\nThe paper would benefit from situating TRACE in relation to similar ideas.  For example, Jason Weston’s *Self-Challenge Agents* also propose generating new tasks automatically to train and evaluate agents.  How do TRACE’s generated tasks differ in structure, diversity, or usefulness?  Do these novel tasks help for training?  Benchmarks nowadays are not just used for measurements but also serve as RL training environments.  It'd be wonderful if TRACE can make the dataset of RL environments for agents larger, since we are all running out of high quality data nowadays.  Clarifying this comparison would strengthen the conceptual framing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)