Gaia2: Benchmarking LLM Agents on Dynamic and  Asynchronous Environments
Authors: Romain Froger
Pierre Andrews
Matteo Bettini
Amar Budhiraja
Ricardo Silveira Cabral
Virginie Do
Emilien Garreau
Jean-Baptiste Gaya
Hugo Laurençon
Maxime Lecanu
Kunal Malkan
Dheeraj Mekala
Pierre Menard
Gerard Moreno-Torres Bertran
Ulyana Piterbarg
Mikhail Plekhanov
Mathieu Rita
Andrey Rusakov
Vladislav Vorotilov
Mengjue Wang
Ian Yu
Amine Benhalloum
Grégoire Mialon
Thomas Scialom
Venue: ICLR 2026 Oral (2026)
Keywords: benchmark
agents
rlvr
multi-agent systems
reasoning
large language models
evaluation
framework
Primary area: datasets and benchmarks
TL;DR: Gaia2 evaluates LLM agents in asynchronous, dynamic environments with action-level verification, revealing fundamental trade-offs between reasoning, speed, and robustness.

Abstract: We introduce **Gaia2**, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action-level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-art proprietary and open-source models shows that no model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open-source models with 21% pass@1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the “sim2real” gap. Gaia2 is built on a consumer environment with the open-source **Agents Research Environments** platform and designed to be easy to extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with a flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.

Published as a conference paper at ICLR 2026
GAIA2: BENCHMARKING LLM AGENTS ON DYNAMIC
AND ASYNCHRONOUS ENVIRONMENTS
Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, lan Yu, Amine Benhalloum, Grégoire Mialon, Thomas Scialom
Meta SuperIntelligence Labs
(rfroger, amineben, gmialon, tscialom ameta. com
ABSTRACT
We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evalu- ations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action- level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-an proprietary and open-source models shows that model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open- source models with 21% % pass 1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the "sim2real" gap. Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform and designed to be easy extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.
Gaia2 Budget Scaling Curves
50igh
0.4
Chrute-4-Sorme
Thisking
GP (PT-
Gal2 pass@1
0.3
Gumini 2.5 Pro
0.2
Kimi-K2
Grai44
Qwend 2050 Thinking
0.1
0.0
$0.01
$0.1
Max Budget per Scanario (5)
Figure Gaia2 budget scaling curve: for each max budget, we plot Σ (scenario result True A scenario_cose cost max budget. Equipped with simple ReAct-like scaffold (see Section no model evaluated here dominates across the intelligence spectrum each trades off capability, efficiency, and budget. At equal cost, that standard scaffolds and/or models miss ingredients for sustained progress. Cost estimates from Artificial Analysis model pricing data (accessed September 10, 2025).
models fare better, yet all curves plateau, suggesting

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Reinforcement learning from verifiable rewards (RLVR) has emerged as a promising path for im- proving large language model (LLM) agents at scale in domains such as reasoning, coding, and tool- use, offering a more reliable alternative to preference-based methods (OpenAl 2024b) DeepSeek-A] eet al., 2025, Mistral-Al et al. 2025] MoonshotAl et al.) (025). At the same time, the use-cases of modern agents increasingly involve sustained monghorizon interaction with dynamic environments, where time, uncertainty, and collaboration play a central role. This has motivated the creation of LLM agent benchmarks (Mialon al. 2023] Jimenez et al., 2024 Yao et al. 2024, [Bakklud & Petersson 2025), yet most such benchmarks are static or synchronous: environments only change when the agents act, and evaluation typically ignores intermediate steps actions. As result, many of the challenges agents face in real seployments-sueee as handling asynchronous events, operating under temporal constraints, or adapting to noise and uncertainty- remain untested.
We introduce Gaia2, benchmark designed to address these limitations by evaluating agents in asynchronous environments with verifiable tasks that, like GAIA (Mialon al. 2023), are simple for humans but challenging for today's Al models. Gaia2 scenarios are motivated by real deployed use cases: it generalizes information seeking to environments instead of web-only, Gaia2-Time reflects the requirements of scheduled task products (e.g., calendar and reminders), and Aaiaa-Agemm2Agoon mirrors the recently proposed Agent2Agent protocol for interoperable multi-agent systems (Google Developers] 2025). Gaia2 consists of 1,120 human-annotate scenarios set in smartphone like environment with realistic apps (email, messaging. calendar, contacts, etc.), similar to AppWorld and ToolSandbox (Trivedi et al. 2024 Lu et al. 2024). Each scenario requires capabilities be- yond search and execution, including adaptability to new events, robustness to noise, resolution of ambiguity, temporal awareness, and collaboration with other agents. To enable reproducible and fine-grained evaluation, Gaia2 introduces write action verifier that checks every state-changing action against oracle annotations, making the benchmark directly applicable to RLVR. Built on the Agents Research Environments (ARE) platform, Gaia2 provides abstractions for creating asyn- chronous environments and supports continuous extension of benchmarks. The core concepts of ARE, illustrated in |Figure 2] allow generalization beyond Gaia2 to the definition of other bench- marks. practice, this design reveals new failure modes: while frontier models achieve overall success rates around 42%, no system dominates across all capabilities, with strong reasoning often traded off against speed, robustness, or cost.
Contributions
This paper makes three main contributions to advance the evaluation of LLM agents and to chart open directions for the next generation of practical systems:
We release Agents Research Environments, a pprrppse platform for building asynchronous, event-driven benchmarks that support scalable evaluation and data gen- eration for RL.
ARE framework:
We introduce Gaia2, the first benchmark unifying asynchronous execution, temporal reasoning, noise robustness, ambiguity resolution, and multi-agent collaboration under a verifiable evaluation framework directly usable for RLVR.
Gaia2 benchmark:
Empirical study:
We evaluate leading proprietary and open-source models on Gaia2, exposing fundamental trade-offs between reasoning strength, efficiency, robustness, and cost.
2 RELATED WORK
Benchmarking LLM agents A wide range of benchmarks have been proposed measure agent capabilities. Embodied and web-based environments such as ALFWorld (Shridhar et al. 2021).
WebShop (Yao et al. 2023a), WebArena (Zhou et al., 2024), and Aren (Drouin et al. 2024
emphasize grounded execution. Synthetic environments such as AppWorld (Trivedi et al. 2024
and ToolSandbox (Lu et al. 2025) introduce app-like tasks with state verification or milestone- based evaluation, while BFCL (Patil et al., 2025) targets large-scale function calling. Other efforts incorporate temporal dynamics and ageti interaction, including VendingBench (Backlund &
Petersson 2025). 7 T-Bench and 7² -BBnch (Yao et al.] 2024 Barres et al., 2025). MultiAgemtBench
(Zhu et al.
2025). and MCP-based benchmarks (Wang et al. [025]] Team 2025 Gao et al., 2025]
Anthropic 2024
Finally, static setups such GATA (Mialon et al.
2023). SWE-hench (Jimenez
2

---

Published as a conference paper at ICLR 2026
Scenario
Intbut State
Everts
Ver tnaation
Scenorio intraltree the state the anvirommentt
User
Environment
RestStenttttd AppuMMCP
State
based agent
Toots
APl
Event Loop
Agent
ARE environments are event-based, time-driven simulations, that run asynchronously from the agent and the user. ARE environments allows playing scenarios, which typically contain tasks for the agent and verification logie. Whether initiated by agent or user, interactions happen through the same interfaces and can be either tool calls, or tool output/notification observations. Extensive simulation control and logging allow precise study of agents behavior.
Figure 2:
al. 2024). and BrowseComp (Wei al. 2025) evaluate only final outcomes. While these bench- marks each capture valuable aspects of agent reasoning. tool use, or collaboration, they remain synchronous and agent-driven: environments only change when the agent acts, and evaluation typi- cally ignores intermediate steps actions. Gaia2 differs by introducing asynchronous, event-driven environments that stress temporal constraints, robustness, ambiguity resolution, and multi-agent co- ordination under unified, verifiable evaluation.
Verification in agentic benchmarks
Verification strategies vary across benchmarks. GAIA (i--
alon et al., 2023) evaluates correctness at the final output level via exact match. This suits search- style tasks but lacks flexibility in format and content, especially in web-based, evolving domains. ToolSandbox (Lu et al., (025)) introduces milestones and minefields that constrain the agent's trajec- tory, enabling early checks of both outcomes and intermediate behavior. Beyond strictly verifiable domains, the Rubrics as Rewards framework (Gunjal et al., 2025 Starace et al. 2025 Lin et al.) 2025) shows how checklist-style rubrics tasks, highlighting the broader potential of rubric-based evaluation. Gaia2 extends this with the ARE Verifier, which evaluates every state-changing write action against oracle annotations. It combines exact argument checks, rubric-guided judgments for flexible cases, and causal and temporal con- straints. Importantly, the verifier is standalone contribution: general mechanism for fine- graigndd reproducible credit assignment reusable beyond Gaia2. While today's models underperform, we expect future RLVR-trained systems close the gap and eventually solve Gaia2.
as interpretable reward signals for subjective
3 ARE: SCALING UP AGENT ENVIRONMENTS AND EVALUATIONS
ARE is a research platform for creating simulated environments, running agents in them, and analyzing their behavior. ARE envi- ronments evolve continuously and are decoupled from the agent. Time advances in the simulation as the environment introduces events. Agents run asynchronously and interact with the user and environment through dedicated interfaces.
App Usage Distribution Gala2
Systemiipp
Core concepts At its foundation, ARE introduces a of ab- stractions, illustrated in Figure 2] that make it possible to design rich. dynamic environments. More precisely: (i) apps are stateful APIs with associated content, analogous to applications such as messaging or email, each exposing tools that can be typed as read- only write, enabling fine-gme fine-grained control and verification; (ii) a collection of apps together with time manager and governing rules forms an environment, which can host one several agents; (iii) within these environments, events represent everything that happens, from tool calls and state changes to scheduled updates, and are fully logged, scheduled either at absolute timestamps or relative others, and organized into dependency graphs; (iv) to surface relevant dynamics, notifi-
21%
Figure App usage distribu- tion across the 12 Mobile libilee apps in Gaia2 for Llama Maverick.
3

---

Published as a conference paper at ICLR 2026
cations provide configurable observability layer: a policy selects which events are pushed to the agent's 's context, enabling the study of proactive and reactive behavior under varying observability; and (v) scenarios extend static tasks into dynamic trajectories by specifying an initial state and DAG of events, including the user's request, intermediate events, and a verification method. Verifi- cation
offline at the end of the run online via scheduled validation events, and focuses on write operations to avoid over-constraining exploration strategies. To demonstrate the generality of these abstractions, we validated that ARE can faithfully reimplement existing agentic benchmarks such as r-bench, 7 b2-bench GAIA, and BFCL-v3, VendingBench( Yao et al., 2024, Barres et al. 20255 Mialon et al.) [023] Patil al.) 2025 Backlund & Petersson, 2025), confirming that the platform both subsumes current benchmarks and provides a foundation for the next generation of agentic evaluations. More details about ARE concepts are provided in Appendix A.T
Asynchronicity and time Because environments run asynchronously, model generations directly consume simulated time: if an agent takes longer to respond, the environment clock still advances, and external events may happen during its reasoning process. This design unlocks evaluations of temporal awareness and responsiveness, which are impossible capture in synchronous settings.
Mobile environment To demonstrate the versatility of the ARE abstractions, we release Mobile as an instantiation of a consumer mobile environment. It features twelve apps (Messages, Chats, Emails, Calendar, Contacts, Shopping, Cabs, Files, etc.) and 101 associated tools, similar in spirit to AppWorld (Trivedi et al., 2024) and ToolSandbox (Lu et al. 2024). Each "universe" represents a complete instance of this environment- full state of all apps centered around a specific user. Applications are populated with synthetic but coherent data, seeded with personas sampled from Per- sonaHub (Ge et al. 2024) and propagated across apps via a dependency graph to ensure cross-app consistency (e.g., contacts align across messaging and email, events match calendar availability). Universes contain between 400K and 800K tokens of structured and unstructured content (exclud- ing filesystem contents), making them suited for long-context and long-horizon tasks. Mobi 1 is governed by clear rules: each turn starts with a user message or a notified event and ends when the agent replies the user. During the turn, simulated time advances continuously, and scenar- ios terminate either on task completion, when constraints on time or steps are exceeded, or when verification fails. While Mobile focuses on the consumer domain to leverage unified app con- cept, the underlying ARE platform is cetrroommett-ggnsttt across domains-for example, the interface for a Chats tool is identical whether in a mobile or desk- top setting. Consequently, the architecture presented here extends naturally to other domains such as desktop automation, customer support, and web browsing, where creating a new environment requires only defining the relevant tool interfaces.
The API definitions remain invariant
Agent orchestration Running agents in ARE requires orchestration compatible with its ab- stractions. For fair evaluation, we use a model-agnostic scaffold based on a ReAct loop (Yao et al.) 2023b), where the agent outputs one tool call per step in structured JSON. The orchestration is aug- mented with pre-step sp and post- -oost hooks: before each LLM call, notifications queued in the environment are injected into the agent's context; after the tool call, the agent termination condition is checked. This minimal extension preserves the simplicity of ReAct while making it compatible with asynchronous and multi-turn environments. Alternative orchestrations can be easily plugged in. To ensure that this sequential scaffolding does not artificially bottleneck performance, we compared it against a Parallel Tool Calling (PTC) orchestration in Appendix B.3.2] Results show that PTC can improve efficiency (wall clock time and token usage) but not performance (see Table 6). confirming that the observed limitations are intrinsic to model capabilities rather than the scaffold.
4 GAIA2: EXPANDING GENERAL AGENT EVALUATION
Building on the abstractions of ARE, we introduce Gaia2, consisting of 800 unique verifiable sce- narios, carefully annotated by humans across 10 distinct universes in the Mobile be environment, with 101 tools each. The scenarios are organized into splits, each targeting one agent capability defined below. To support rapid and cost-effective evaluations, we also curate 160-scenario subset, Gaia2-mine mini The benchmark includes two augmentation setups derived from Gaia2-mini, mini, adding 320 scenarios to the original 800 for total of 1,120 scenarios.
4

---

Published as a conference paper at ICLR 2026
Capability
Example Task
Explanation
Execution
Update all my contacts aged 24 or younger to be one year older than they currently
Evaluates the ability to chain long seq. write actions in the right order
Search
city most my friends live case of a tic, roturn the first city alphabotically
Evaluates the ability chain long seq, read actions in the right order
Ambiguity
Schedule a Th Yoga event each day at 6:00 PM October 2024 to October 21, 2024
Tests whether agents ask for clarification on impossible, contredictory, ambiguous tasks
Adaptability
have to meet my friend Kaida view a property [..] if she roplies suggest another property time, update the calendar event
Requires agents adapt dynamically to environmental changes
Time
Send messages each supposed to meet today, asking who is supposed to order the cab. after 3 minutes there no response, order a from [...
colleagues am
Evaluates whether agents complete tasks maintain temporal awareness
Agent2Agent
*Same Search task as above but the Contacts and Chats apps replaced by app sub-agents"
Tests whether agents can collaborate with other agents to use tools & complate tasks
Noise
"Same Adaptability task as above but with random tool execution errors and random environment occurring during execution"
Evaluates whether agents are robust to environment noise & distractors
Figure 4: The seven core agent capabilities evaluated by the splits of Gaia2.
4.1 CAPABILITIES EVALUATED
Gaia2 evaluates agents across 1,120 scenarios. To provide a clear laxonomy, we distinguish between Core Capabilities (Execution, Search, Ambiguity, Adaptability, Time) and Augmentations (Noise, A2A). The five core splits comprise 800 unique, human- authord scenarios, each instantiated with a unique event DAG and initial environment state. We treat these core categories as dominant "flavors" rather than strictly orthogonal dimensions. In practice, any natural task inherently compositional (e.g., Time task often requires Search and Execution). Consequently, we explicitly chose not to in- troduce a separate "compositional"" split; our early experiments with scenarios artificially combining three more distinct capabilities resulted in unnatural tasks that lacked clear evaluation signal. Instead, we rely on the organic compositionality present in the core splits to ensure tasks remain realistic while still allowing for clear mailure-mode attribution.
Environment augmentations The Noise and Agent-to-Agent (A2A) splits are environment-level modifiers applied to base scenarios to stress-test robustness and collaboration. Because our veri- fier checks state changes rather than specific tool traces, these augmentations do not require new annotations. In the Noise split, we inject controlled perturbations, including tool anomalies (c.g., random exccution failures, signature changes) and irrelevant environment events (c.g., incoming spam emails). In the A2A split, apps are replaced by "app-agents". The main agent loses direct access to these apps' tools and must instead coordinate with the app-agents via messaging solve the task. App-agents subtasks and return a report. This setting explicitly evaluates the main agent's 's ability to decompose goals and coordinate under partial observability. In evaluation setting, main- and app-agents use the same underlying model.
not fully autonomous: they are invoked on-demand to execute specific
4.2 SCENARIO DESIGN AND ANNOTATION PROTOCOL
We construct Gaia2 scenarios using the ARE annotation interface (see Appendix A.4 for details), which lets annotators explore generated Mobi le universe. Their task is to create DAGs of write actions and environment events as ground truth. Starting from the generated environment, annota- tors design scenarios that isolate and stress single capability at time (c.g., Adaptability, Time), ensuring a clear signal model strengths and weaknesses.
Each scenario undergoes multiple rounds of validation by independent annotators, followed by consistency check. We complement this process with automated guardrails (e.g., structural con- straints on event graphs) and post-hoc difficulty calibration using baseline agent. This combination yields a diverse, challenging, and verifiable set of scenarios while reducing annotation errors. We provide more details our annotation process and guidelines in Appendix B.1
5

---

Published as a conference paper at ICLR 2026
Table ARE Verifier and In-context Verifier on 450 hand-labeled validation trajectories.
Verifier
Agreement
Precision
Recall
In-context Verifier (LLM judge only)
0.72
0.53
0.83
ARE Verifier
0.98
0.99
0.95
4.3 VERIFIER
The ARE Verifier evaluates agent trajectories against minimal oracle sequence. Unlike final- answer judges, it is goal-oriented rather than path-optimal. We distinguish read from wri te actions: agents may execute unlimited read steps (e.g., browsing) for exploration without penalty, as only write steps modify the environment and count toward goals. The verifier is also order- agnostic regarding independent goals.
Evaluation covers four dimensions: (i) Consistency tool names/counts must match; arguments exact matching for rigid fields (IDs) and LLM rubrics for flexible ones (text), with anti- anti-hacking checks; (ii) Causality- actions must respect dependency DAGs (parents before children); (iii) Tim- Suceess ing enf-ree via tolerance windows; and (iv) Completeness success requires matching all oracle write actions.
On 450 labeled trajectories (Table the verifier achieves 0.98 agreement and 0.99 precision, outper- forming LLM-only baselines. Beyond Gaia2, it serves as a reusable component for faithful bench- marking and RLVR training any ARE-based environment. Further details are in Appendix B.2
5 EXPERIMENTS
In our core experiments, we evaluate state-of-the-art models each Gaia2 capability split (Moon- shotAl et al. 22255 Gemini Team 2025 Yang et al. [025 Llama Team [024 OpenAl 2024a). We also test the sensitivity of models various evaluation configurations for Time and Agent2Agent.
Experimental setup We use the same ReAct-style baseline scaffold (Section 3 for all evaluations in order to ensure consistent comparisons across models and providers. All L.MSs are evaluated full context length (2 (2128K tokens), temperature 0.5, and 16K token generation limits per turn. Scenarios are run three times account for potential variance, and are terminated when one of the following conditions is met:
(i) 200 steps, (ii) context overflow, i.e., the agent exceeds the available context window (failure), (iii) verification completion, i.e., the verifier determines the trajectory outcome eitcome-ethher by failing some turn or by successfully passing verification at every turn, or (iv) timeout. The environment provides tools and notifications via system prompts, with notification verbosity set to med i um by default: agents receive systematic alerts for high-priority events while filtering out lower-priority background notifications.
We handle deployment issues like outages and rate limits using a simulateen generation time-pausing during responses and resuming with matching time offset-to preserve realistic timing while enabling robust evaluation. The ARE Verifier uses Llama 3. - Inst at temperature 0. For more details experimental procedure, please see Appendix B.4
5.1
CORR RESULTS
Our core experimental results are presented in Table 2 Figure 5) and Figure 6 Among Gaia2 splits, Execution and Search emerge as the easiest, consistent with prior benchmark saturation (Trivedi et al. 2024 Lu et al. 2024). Ambiguity and Adaptability remain challenging, with only Claude-4- Sonnet and GPT-5 (high) achieving robust performance. The Time split further differentiates frontier models: only Gemini 2.5 Pro and Sonnet achieve meaningful scores, reflecting their efficiency- latency advantages (Figure 6). Noise robustness also lags, with most models scoring below 20 de- spite GPT-5 (high) reaching 35.4%. Agent2Agent collaboration benefits weaker models more than frontier systems (see Figure 10p. Overall, GPT-5 (high) leads with 42.1% pass@ 1, maintaining an 8-point margin over Sonnet across all categories. Kimi-K2 distinguishes itself among open mod-

---

Published as a conference paper at ICLR 2026
Table 2: Pass scores on Gaia2 scenarios per model and capability split. All models are evaluated with the same baseline ReAct scaffolding described Section 3 and with three runs account for potential variance. The overall score is the average across splits.
Execution
Scurch
Ambiguity
Adapiability
Time
Noise
Agent2Agent
Overall
3.3 70B Instruct
11:
11.3
19.6
1.9 10.
1033
3.8 10.9
210
Llama 4 Maverick GPT:4o
13.8 sL6
14.4 #1.
2.1
5.0 21.0
1.2 10.5
6.2 $1.1
9.2 #13
7.4
8,3 21.3
17.5 11.7
4.4 10.
6.2 21.1
5.8 11.1
4.6 11.0
5.2 11.0
7.4
GPT-OSS 120B (high)
17.9
8.3
#1.
10.6 #14
13.7
Qwen 2-35B
22.7 11.
22.3 11.9
6.5 11.1
8.1 11.2
1.2 10.5
10.8 11.4
9.4 21.3
11.6
Qwen3-235B Thinking
28.1 #2.1
36.2 =0.5
10.0 #2.4
#2.9
0.0 #0.0
6.9 #2.0
12.5 #2.6
15.7
Grok-4
8.8
$7.5
9.4 12.3
4,4
0.0
15.6
14.4
15.7
Kimi-K2
34.2 34.2 23.3
36.0 27.
8.3 11.3
24.0 21.9
0.8 20.4
18.8 11.8
18.3 #1.8
20.1
Gemini S.mmini-2..5PPrr 2.5-Pro
39,2
22.3
#1.
20.4 11.
20.4 11.8
25.8
Claude-4-55 4-Soneet
57.9 12.3
59.8 10.
24.2 1200
38.1 12.2
8.1 11.2
27.7 12.0
27.9 1200
34.8
Claude-4-S5 S-onnet Thinking
62.1 :2.2
60.6 #4.
27.3 $2.0
42.1 #2.
8.5 #1.
31,2 #2.1
32.5 #2.1
37.8
GPT-5 (minimal)
31.9
26.2 10.0
20.6
19.2
11.0
13.1
11.5
18.2
GPT-5 (low)
52.7 =2.3
64.2 #2.
39.6 #2.2
30.2 #2.1
2.3 +0.7
28.3 #2.1
24.6 #2.0
34.6
GPT:5 (high)
21.8
12.2
10.0
12.2
17.9
Execution
Search
Adeptasiltty
Lieme deanni comm
cheame Keven
on
Time
No00
Agent2Agent
Hira
Google
Moonshot
Archoook
rar dan mnn
Figure 5: Gaia2 scores per capability split. Models reranked independently for each capability, highlighting where they excel or struggle.
els, particularly Adaptability. While instruction-followw and search tasks largely solved, robustness, ambiguity resolution, and collaboration remain open challenges.
In Figures and 7 extend our analysis beyond raw scores to identify the finer-grained factors that drive performance differences between models on Gaia2. In addition, since agents are ultimately intended for deployment in production settings, we evaluate their performance in relation to their computational cost and execution time.
post-performance trade-offs Figure reveals clear cost-performance-iine trade-offs. GPT-5's reasoning models demonstrate direct scaling: higher test-time compute yields better performance but longer solution times. Claude 4 Sonnet costs roughly 3x more than GPT-5 (low) for comparable accuracy but operates much faster. Outliers include the inefficient Grok-4 and cost-effective Kimi- K2. While an average human annotator can solve every task, they are slower than all models, partly due using ARE's GUT rather than native OS. These findings highlight the need for cost- normalized evaluation metrics. Comparing model parameters or FLOPs alone inadequately reflects real-world deployment conditions. Success rate per dollar better captures how agents will be judged in practice- by reliable, efficient task completion under resource constraints.
We examine behavioral factors correlating with Gaia2 performance. Two hypotheses guide our analysis: (1) exploration drives success through increased tool use and sys- tematic information gathering before write operations, and (2) comprehensive reasoning via token generation improves performance. Figure confirms both relationships: performance correlates
Performance drivers
Cost estimates from Artificial Analysis model pricing data (accessed September 2025)

---

Published as a conference paper at ICLR 2026
Overal Gaix2 Score vs Avarage Costpor Sconario (JSD)
Thmee per Sohved Gala2 Sconario
50
det Hight
4.0
35-71
30
Cladde
Lana
Opandi
071-1km))
019:055
Ove ct
Hoorshozal
$0.0
$02
$0.6
$0.6
50.6
51.6
S16
10
salved sccraan
Figure 6: Left: Gaia2 score vs average scenario cost in USD. Right: Time taken per model to successfully solve Gaia2 scenarios compared to Humans.
positively with tool calls (left) and output tokens (right). However, Claude-4 Sonnet and Kimi-K2 stand out as notable outliers, achieving high performance (35% and 21% respectively) while pro- ducing relatively few lokens-suggesting exceptional efficiency, perhaps due larger parameter counts or specialized architectures. Within families, we observe a striking contrast between the base and "Thinking" variants of Claude and Qwen: the latter generate more tokens per step but take fewer steps overall, leading to higher pass@) I and lower cost per solved scenario, effectively trad- ing verbosity for efficiency (e.g.. Qwen-23en 235B Thinking vs. Qwen-235B). App usage patterns were nearly identical across models (Figure 3), indicating that performance differences stem primarily from general reasoning capabilities rather than app-specific preferences.
5.2 TIME REVEALS THE IMPACT OF INFERENCE SPEED AND SYSTEM RELIABILITY
We evaluate Gaia2-Time in two modes. As shown in Figure 8 (left), removing generation latency ("instant" mode) improves all models. with the largest gains for reasoning models: Sonnet rises from 8.1% to 26.7%, and GPT-5 (high) from 0.0% to 34.4%. Weaker models improve modestly due to the difficulty of the tasks, while Gemini 2.5 Pro combines strong performance with low latency and therefore best supports timing requirements. In the default mode, we observe inverse scaling in the Time capability: models trade Time performance for Execution performance due to longer thinking, see[Figure right. This underscores the need for adaptive compute-using shallow models and performing deeper reasoning only when necessary. Besides inference speed, the Time split also underlines the need for reliable infrastructure to serve responsive models without rate limits and server downtime in order to handle senstsen tasks. Finally, some Time scenarios require concurrent actions within narrow windows, which our single-threaded scaffold cannot fully express. Parallel orchestration is promising direction to solve this type of scenarios.
5.3
CLOSER LOOK AT AGENI COLLABORATION ON GAIA2 WITH AGENT
Inspired by recent work pushing beyond single-LLM agent tool-use and towards agent teams that message, coordinate, and divide labor (oogle Developers 2025), we study multi-agent collabora-
Ovarell
Amirgge
Pyriato
Pyrrcu (i
12
10k
120K
LLMt
Figure 7: Left: Gaia2 pass@1 versus average model calls per scenario. The performance of models highly correlated to the number of tool calls, emphasizing the importance of exploration. Right: Gaia2 pass@1 score versus average output tokens per scenario (log scale). Claude 4 Sonnet, while costing a lot exists beyond the Pareto frontier.
8

---

Published as a conference paper at ICLR 2026
Gala TTime parsget: instent Defautt Time Mode
Co x2 TTime Execut on passga (GPT mode
6P744
pode Dei
GPT-5
Oetaul mocert ad Sorr
50
GPT-5 (oo)
CPT-3 tighn
10
25
sasagg
20
23
30
Execution peratn Instruction lowing Tostnuee)
Figure 8: Left: Pass@1 Gaia2-Time in default vs. instant. Time remne-rrasning--heay models are slower and miss deadlines.
Right:
Inverse scaling on
tion on Gaia2 scenarios. We focus two models at different points in the cost-quality curve: Llama 4 Maverick, a lighter-weight model, and Claude 4 Sonnet, the strongest overall LLM on standard Agent2Agent (Table 2)
For the weaker Llama Maverick, centralized collaboration on Gaia tasks improves both perfor- mance with @ass@k and operational stability. As the agent-to-agent ratio increases, observe more favorable scaling with repeated sampling and a lower incidence of tool-call errors per step (Figure(9rightt Figure IOp. However, the trends observed for Llama 4 are not universal. For Claude 4 Sonnet, increasing the collaborator ratio r and thus the degree of task decomposition does not improve nostnnommllize performance under best-of-k sampling: score per token plateaus with or without multi-agen collaboration. Similarly, collaboration ratio with Agent2Agent has a weak negative effect on tool call error frequency.
One explanation for these findings may lie in the fact that Agent2Agent induces hierarchical decom- position into decision makiin.. As shown in Figure 9 left, sub-goals issued by main-agent to an app-agent instantiate temporally extended actions akin to options (Sutton et al. 1999). Under this lens, gains performance may materialize only when the benefits of decomposition outweigh the costs. For example, Agent2Agen may increase task score only when sub-goals set by main-agents are well-scoped and both app- and main-agents are capable of reliably exchanging state & intent during presssge-passingg Likewise, the addition of hierarchy can result in cascading errors and/or saturating gains if post-training has fit models to long-formi single-agent planning and tool-use; in this regime, coordination may introduce overhead that offsets accuracy and efficiency gains.
Heterogeneous teams open a new compute scaling axis for task automation, for example, by keeping strong main agent to plan/decompose tasks while swapping in cheaper app-agents to execute sub- goal Empirically, replacing Llama 4 Maverick app-agents with Claude app-agents boosts pass @ for both main-agent settings (16.2 with Llama-main, 29.3 with Claude-main). while the fully light
Galad Multi-Ag Auuti-Agent Collaboretion AgentZAgurtt
aactont "Smal agaen"
Mann-gertLLM
"ectioouiguut rnguth ( "Taak" "Deete ROSOE
tsiggle agenti (portal agent2agentt
Planse lool
1.0 thul ageniZageno
oOUTRUTOR or STEP Coservation
The rest socent crea Ast Nakomarr been your INSOX.
AppagettLM Reczones
Figure 9: Agent2Agent tests whether LLM agents can collaborate through message passing in or- der to solve Gaia2 tasks via sub-task decomposition. For lighter-weight LLMs, collaboration in Agent2Agent results in lower incidence of tool call errors. Left: Sample exchange between Llama 4 Maverick main app agent in an Agent2Agent scenario. Right: Frequency of errors per tool call (lower is better) on Gaia-2 mini for Llama Maverick and Claude 4 Sonnet.
2 ARE natively supports controlled evaluation of heterogeneous teams, making team composition primary experimental factor alongside standard inference hyperparameters.

---

Published as a conference paper at ICLR 2026
Fasagi: Scaling Laws on Galaz Mini with Multi-Agent Ctt.--ggeenclllbbottteenoo Agent2igent
Liema 4 Maverick
Clouds 4 Sornet
100K
1M
12M T00K
1M
total wokkookesppe socmer
tetal wow.ooperrrroooooo scenero Dog s0.r0
Figure 10: Increasing the number of multii-geent collaborators in Gaia2 scenarios by increasing the Agent2Agent ratio "r" improves pass@k @k scaling laws for Llama Maverick, but does not improve token cost vs score tradeoffs with repeated sampling for Claude 4 Sonnet.
Main-Agent LLM
Llama 4 Maverick
Claude 4 Sonnet
Llama 4 Maverick
8.5 =1.7
16.2 +0.7
App-Agent LLM
Claude 4 Sonnet
18.3 10.7
29.3 ::299
Table 3: Probing cross-model collaboration in Gaia2-mini Agent2Agent scenarios: we evaluate pass @ I across main- app-agent pairings with Llama 4 Maverick and Claude 4 Sonnet in the fully collaborative Agent2Agent setting (r = 1). The results are averaged over three runs and presented with the standard error.
contiguration is weakest (8.5). This suggests that for existing LLMs, Gaia2 task completion re- mains sensitive execution fidelity at the app-agent level: stronger executors improve outcomes even when the main agent is light. Similarly, pairing a strong main agent with light executors still outperforms the all-light team (18.3 with Claude-main Llama-app), indicating that higher-quality sub-goal specification and critique from the main-agent contribute independent gains. These find- ings are consistent with prior work suggesting heterogeneous multi-agent systems can trade planning capacity against execution fidelity to manage compute-quality trade-offs.
6 CONCLUSION & DISCUSSION
ARE introduces asynchronous, event-driven evaluation framework with action-level verification, enabling reproducible benchmarking directly applicable to RLVR. Its abstractions- events, notifications, and scenarios-alon with the MOBILE environment provide extensible foundation for community-din driven evaluations and RL data generation. Gaia2 demonstrates that no model domi- nates across all capabilities: GPT-5 (high) achieves the best overall accuracy (42% pass @ 1), Claude- 4 Sonnet offers competitive performance with lower latency, and Kimi-K2 leads among open-source systems (20%). Scaling curves reveal fundamental cost-time-accuracy trade-offs, highlighting the need for cost-normalized reporting.
Verification the action level scales more effectively than end-state comparisons and supports fine- grained credit assignment. The ARE Verifier matches human annotations with high fidelity (0.99 precision, 0.95 recall), while uncovering issues such "judge-hacking" Robust verifier design is thus critical for both evaluation and RL training; hybrid approaches combining scalar rewards with preference signals remain an open direction.
Finally, Gaia2's Time split and A2A experiments underscore the critical role of orchestration. The inverse scaling observed Time-sensitive tasks suggests that future agents require adaptive com- pute strategies: deploying fast, lightweight reasoning for routine tasks while reserving deeper delib- eration for complex ones. Simultancously, the A2A results demonstrate that orchestration extends to collaboration, where heterogeneous teams can outperform monolithic models through effective delegation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Nikolay Bashlykov, Radhika Bhargava, Misha Bilenko, Carly Burton, Adrien Carreira, Onur Çelebi, Neha Choudhari, Mike Clark, Levi Corallo, Paul Deveau, Jenny Fant, Clémentine Fourrier, Thibaut Frere, Avijit Ghosh, Christian Keller, Pascal Kesseli, Abhishek Kumawat, Flo- rian Laplantif, Baohao Liao, Alexandre Linares, Chaya Nayak, Rohit Patel, Daryl Rodrigo, Marija Sakota, Antoine Saliou, Tatiana Shavrina, Matt Staats, and Mik Vyatskov for their support for the development of ARE and Gaia2.
We also thank Pillar Al and Surge AI.
REFERENCES
Anthropic. Introducing the model context protocol. https://www/ httta://ww..aatirppic.cooo anthropic..omm/eewe/
mode context- - 2024. Accessed: February 12, 2026.
Axel Backlund and Lukas Petersson. Vending-bench: A benchmark for long-term coherence of autonomous agents, 2025. URL
15840
Victor Barres, Honghua Dong. Soham Ray, Xujie Si, and Karthik Narasimhan. 72 benbhhh: Evaluating conversational agents in dual-control enviromment, 2025.
/abs/
2506. 07982]
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An. Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren. Zehui Ren, Zhangli Sha. Zhe Fu. Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning. 2025. URI cinttps://arxiii org/ abs/ 2501 12948
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024. URL/atts:://aae
org. abs/ 2403.4403.. 07718
Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang (Eric) Zhu, and Saleema Amershi. Interactive debugging and steering of multi-agent ai systems. In Proceed- ings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI '25, pp. 1-15.
11

---

Published as a conference paper at ICLR 2026
ACM, April 2025. doi: 10.1 10.1145/37065983713381 UtL.http://dk.....or///100 $706598.371337
Xuanqi Gao, Siyi Xie, Juan Zhai, Shqing Ma, and Chao Shen. Mcp-radar: A multi-dimensional
benchmark for evaluating tool use capabilities in large language models, 2025. URL https:
//arsiv.oro//as//5555.
Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Sealing synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:246..0000 2024.
Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning. multimodality, long context, and next generation agentic capabilities, 2025. URL. http::://arii..or///assss
2507. 06261
Google Developers.
Announcing the agent2agent protocol (a2a). opers Blog, April 2025. URL h goog..ooollloooooo lelglgebloogggoo www.ellloo...o..o.eoeoeeee.eeeee cwm/em////////o
Google Devel-
Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as
rewards: Reinforcement learning beyond verifiable domains, 2025. URL https://arxi::
orgg/bba aaaaaba/50000 17746
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL httpttts///opeeeeeeee
net/ fort/Eormm3 Faroooummiiiii id=VTF
ByNOM66
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking Iims with challenging tasks from real users in the wild. In Proceedings of the International Conference on Learning Representations (ICLR), Online, May 2025. URL htttps://pperrriee..cr
forum SataummdddMKHCx22555
Llama Team. The llama 3 herd of models, 2024. URL 21783
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin. Zirui Wang, and Ruoming Pang. Toolsandbox: A state- ful, conversational, interactive evaluation benchmark for Ilm tool use capabilities, 2024. URL
https: abtp//b////4 04682
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. Toolsandbox: A state- ful, conversational, interactive evaluation benchmark for Ilm tool use capabilities, 2025. URL
ht.tps://arxiv> http://aaxxvv..raaaaaaaaa aaas//s55500 .06882
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Mistral-AI, :, Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lam- ple, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, Léonard Blier, Lucile Saulnier, Matthieu Dinot, Maxime Darrin, Neha Gupta, Roman Soletskyi, Sagar Vaze, Teven Le Scao. Yihan Wang. Adam Yang. Alexander H. Liu, Alexandre Sablay- rolles, Amélie Héliou, Amélie Martin, Andy Ehrenberg, Anmol Agarwal, Antoine Roux, Arthur Darcet, Arthur Mensch, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Chris Bam- ford, Christian Wallenwein, Christophe Renaudin, Clémence Lanfranchi, Darius Dabert, Devon Mizelle, Diego de las Casas, Elliot Chane-Sane, Emilien Fugier, Emma Bou Hanna, Gauthier Delerce, Gauthier Guinet, Georgii Novikov, Guillaume Martin, Himanshu Jaju, Jan Ludziejewski, Hran-Hadrien Chabran, Jean-Malo Delignon, Joachim Studnia, Jonas Amar, Josselin Somerville Roberts, Julien Denize, Karan Saxena, Kush Jain, Lingxiao Zhao Louis Martin, Luyu Gao, Lélio Renard Lavaud, Marie Pellat, Mathilde Guillaumin, Mathis Felardos, Maximilian Au- gustin, Mickašl Seznec, Nikhil Raghuraman, Olivier Duchenne, Patricia Wang, Patrick von
12

---

Published as a conference paper at ICLR 2026
Platen, Patryk Saffer, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Pavankumar Reddy Mud- direddy, Philomène Chagmiot, Pierre Stock, Pravesh Agrawal, Romain Sauvestre, Rémi Dela- court, Sanchit Gandhi, Sandeep Subramanian, Shashwat Dalal, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Thibault Schueller, Thibaut Lavril, Thomas Robert, Thomas Wang. Timothée Lacroix, Valeriia Nemychnikova, Victor Paltz, Virgile Richard, Wen-Ding Li, William Marshall, Xuanyu Zhang, and Yunhao Tang. Magistral, 2025. URL
https:/ /arss//aaxxxn /arxivv. org/ abs 2506 10910
MoonshotAI, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arKiv:2507.22 20534, 2025.
OpenAl. Gpt-4o system card, 2024a, URL UKChttpps//arrityorr/a////////1 16720
OpenAl. Openai ol system card, 2024b. URL
Rock Yuren Pang, K. J. Kevin Feng, Shangbin Feng, Chu Li, Weijia Shi, Yulia Tsvetkov, Jef- frey Heer, and Katharina Reinecke. Interactive reasoning: Visualizing and controlling chain-of- thought reasoning in large language models, 2025. URL httttpp://aaiii..ooottttt www./ogg///....... 23678
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agen- tic evaluation of large language models. In Forty-second International Conference on Machine Learning. 2025.
Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, and Jarek Szlichta. Ladybug: an 1lm agent debugger for data-driven applications. In Proceedings of the 28th International Conference on Extending Database Technology (EDBT), pp. 1082-1085, 2025. ISBN 978-3-89318-0994
Demo paper.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, 2021.
URL httphhttts////t/tleeeee org/ abs/2010 03768
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504.01888, 2025.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
The MCPMark Team. Mcpmark: Stress-testing comprehensive mep use. https://g
com/ eevelvalllse ewwallsss/mppparrr...aaa 2025.
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In ACL, 2024.
Zhenting Wang, Qi Chang, Hemani Patel, Shashank Biju, Cheng-En Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, and Eugene Siow. Mcp-bench: Benchmarking tool- using llm agents with complex real-world tasks via mep servers, 2025. URL Atlphhtss:
arg/abs/2000 2508 20453
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025.
12516
13

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang. Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan abw/ 09388
Qiu. Qwen3 technical report, 2025. URL
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real- world web interaction with grounded language agents, 2023a. URL https://axxiv..rrr
abs/2207. 01206
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023b.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL 2406. 12045
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. 2307 13854
/bbg//////
Kunlun Zhu, Hongyi Du. Zhaochen Hong. Xiaocheng Yang. Shuyi Guo, Zhe Wang. Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, and Jiaxuan You. Multiagentbench:: Evaluating the collaboration and competition of Ilm agents, 2025.
01:355
14

---

Published as a conference paper at ICLR 2026
A ARE APPENDIX
A.1 ARE FOUNDATIONS
ARE is time-driven and built the principle that "everything is an event". Specifically, five core concepts work together:
Apps are stateful API interfaces that typically interact with data source.
2. Environments are collections of Apps, their data, and governing rules that define system behavior.
3. Events are anything that happens in the Environment. All Events are logged.
4. Notifications messages from the Environment that inform the agent about Events. They are configurable and enable selective observability of the Environment.
5. Scenarios are sets of initial state and scheduled Events that take place in Environment, and can include a verification mechanism.
A.1.1 APPS
Apps are collections of tools that interact with data source. For instance, an Emails app contains tools like sendLemail 1 and delete_email that all operate on the same email database. Similar approaches have been explored in AppWorld (Trivedi 2024).
2024) and ToolSandbox (Lu et al.
Apps maintain their state Each app starts in the simulation with initial state and keeps track of changes as agents use its tools or as events occur the environment. Apps store their data internally rather than relying on external databases. This design makes it convenient study agent tasks that require to modify the state of the environment, and ensures that experiments can be reproduced consistently.
Tool creation and taxonomy Apps are implemented by adding Python methods within an App class. When the simulation runs, these methods are automatically converted into properly formatted tool descriptions that agents can understand and use. ARE classifies tools into two types via deco- rators: read, which only read app states (e.g., search ema i s), and write, which modify app states (e.g., send_emaill 1). This distinction is helpful egg. for verification, see Appendix B.2 Tools are role-scoped-agentt user, or env.
Extensibility Beyond ad hoc app creation, ARE can also connect with external APIs through MCP compatibility (Anthropic) 2024). The framework also offers flexible options for data storage. While our current implementation stores data in memory, users can easily connect SQL databases or other storage systems without changing the core framework.
Core apps Developers can choose which apps include in their environment or create new ones. However, every ARE environment includes two core apps that handle the basic interaction between agents and their environment:
Anent face the communication channel between users and agents: mes- sages tool calls, and user messages generate notifications (Appendix A.1.4) that agents can process asynchronously. This enables asynchronous interactions during task execu- tion. The interface supports modes: blocking (the agent waits for user reply) and non-blocking (the agent continues loop regardless of reply).
System provides core simulation controls like get.current.time (query time), wait (pause for duration), and wait_for wwt.teteetttiiaaaaaoeoaooa
(pause until an event). When any wait tool is invoked, the simulation accelerates: it switches from real time to queue-base based, event-to-event loop. Scenarios that would take hours in the real world can thus in minutes, enabling practical long-horizon testing.
15

---

Published as a conference paper at ICLR 2026
A.1.2 ENVIRONMENT
An environment is Markov Decision Process with states, observations, actions, and transition rules. The environment state includes the states of all apps, the time manager, and the notification system. Apps define the action space by exposing their tools. The environment runs deterministically given a fixed starting state and seed, ensuring reproducible evaluations. It can host one multiple agents simultaneously. supporting both single-agent and multi-agent setups. The environment's rules de- fine time progression, action permissions, reward computation, and how agent actions affect the environment state.
A.1.3 EVENTS
In ARE, an event is any agent action or app-state change. Each event is timestamped, logged. Events can be scheduled, e.g., friend's message minute after simulation start. This design yields (i) deterministic execution- events run in scheduled order; (ii) complete auditability all actions can be replayed and analyzed; and (iii) flexible eventts events can be set at absolute times or relative to others.
Event lifecycle Events flow through four stages described in Figure 2
(i) creation events
created from tool calls or scheduled by the simulation; (ii) scheduling events enter a time-ordered Event Queue with dependency management using directed acyclic graphs, supporting both abso- lute timing (at specific timestamps) and relative timing (relative to other events or conditions); (iii) execution the Event processes events and captures results, state changes, and exceptions; and (iv) logging executed events are stored in an EventLog with detailed metadata for analysis, debugging, and validation of agent behavior.
Event types There are different types of events. While most events track interactions within the environment, other special events are needed enable dynamic scenarios and verification strategies:
Agent/User/Env events are generated by tool calls. Agent Events are initiated by the agent (c.g.. sending message). User Events by the user (e.g.. replying to the agent), and En- vironment Events by the simulation itself to introduce external changes (c.g., a scheduled message from friend).
Conditional events periodically check predefined conditions and complete when criteria are met (c.g., cancel ride only if one was booked).
Validation events check milestone achievement or constraint violations for verification, and fail the simulation if not completed on timeout (e.g., stop if no ride is booked within 30 seconds of the user request).
Oracle events pre-scheduled "ground truth" actions used by verifier for comparison.
Dependencies and scheduling Events are modeled as Directed Acyclic Graphs (DAGs) as illus- trated in Figuue IT An event can only be triggered upon successful completion of all its predecessors (e.g., e1 processes immediately at simulation start, e4 needs both e2 and e3 to be completed). This data structure also supports multiple branches running simultaneously to model independent events. Conditional and Validation events can be used in the DAG to trigger other events and make the environment more dynamic.
A.1.4 NOTIFICATION SYSTEM
At each environment step. processed events can trigger notifications according notification pol- icy (see Figure 2). similar to mobile device notifications. Apart from tool outputs, notifications are the only signals agents receive from the environment. Notifications are queued by timestamp and exposed to agents through a notification queue, enabling asynchronous interactions. In our orches- tration (see Appendix B.3). notifications are injected into the agent's context the beginning of each agent step.
Notification policy The notification system follows configurable policy-i.. i..., a whitelist of events authorized to emit notifications. ARE pre-defines three verbosity levels: low (only user
16

---

Published as a conference paper at ICLR 2026
E2
E4
E3
start
True
E5
Cond1
Val
False
Figure 11: Event dependency graph illustrating ARE scheduling patterns. Events E1 and E5 exe- cute in parallel after simulation start, and E2/E3 executing parallel after their prerequisites, both need to be executed for E4 to execute. Conditional execution is shown through Condl leading to validation (Val) with true/false outcomes.
messages are notified), med um (emails, messages and calendar events are notified), and high (everything is notified), creating graduated spectrum of environmental observability.
Notifications and agent proactivity Notifications are not the only way for agents to observe envi- ronment changes. For example, even if the notification policy doesn't alert the agent when messages arrive from contacts, the agent can still proactively check for new messages by browsing the user's inbox. Notifications add realism and complexity environments, potentially creating different agent behaviors based on whether the environment is notification-rich or notification-poor This system enables researchers tackle new capabilities such proactivity.
A.1.5 SCENARIOS
ARE shifts from static, single-turn tasks to dynamic scenarios. Scenarios attempt to capture real- world complexity through temporal dynamies, events, and multi-turn interactions. This enables eval- uation of agent capabilities that cannot be assessed through traditional request-response paradigms. In practice, scenarios are implemented in scenario. py containing the apps. scheduled events, and arbitrary verification logic.
Scenario runtime Scenarios typically start with environment instance and www.meesaaeet..e.
tool call, waking the agent up. The environment operates on discrete time steps, executing scheduled events and managing state transitions until the agent reaches exit condition, see Figure II All interactions with the user are through the AgentUse: Int wnw.efaattnaee facc,, with verification triggered upon task completion.
Scenario example Consider this two-turn scenario (see Figure 2 and Figure 12): a user asks the "Can you ask my mom to send me our family streaming pass- word?". The agent is initialized from this first notification, starts checking messages, and requests the password in the Chats app: the tool calls modify the Chats app state and are recorded in the EventLog. The agent confirms to user that the request was sent, after which the environment pauses execution and applies first-turn validation.
agent via AgentUserInters
At turn two, the user asks follow up question: "As soon as receive the password from my mother, transfer it to my father". The agent resumes upon the send_ gernd_msssaeeeto_geeee notification, and looks for the mother's reply in the Chats app (where previously requested it). the meantime, scheduled environment event is triggered and Email from the mother containing the code is received. The agent reacts to this email notification by stopping searching the Chats app. processes the Email, extracts the code, forward it to the father, and report success to the user. Final verification reviews the complete interaction in the Event Log, and the environment issues a termination signal end execution.
17

---

Published as a conference paper at ICLR 2026
A.2 NOTIFICATION POLICIES ARE
The notification system in ARE follows configurable policy where researchers can choose which Env events are notified to the Agent. The Mobile environment pre-defines three notification poli- cies with different levels of verbosity, which we describe in detail in Table [ Note that messages sent by the user via send of the verbosity level.
systematically notified to the agent, regardless
Table 4: Pre-set notification policies in Mobi le (Compressed).
Verbosity Notified Environment Tools
Description
low
None
No environment events are notified.
medium
Email:
Notifies events that are consequences of agent ac- tions, analogous to mobile notifications. Default in Gaia2.
send_emmal..to.userr.n......
replyy.oo.emal..froo.uee..
Chats/Messages: create and add message Shopping: cancel order,
updatee.oreersstauus
Cabs: cancel.ride, wtw.:eeeeee......e.ii..... end_ride
Calendar: add add. del.ate.ccceeddrr.eeeete..aeeeee
high
All medium tools plus:
Notifies environment events, including those in- dependent of agent actions (e.g., products).
Shopping: add.product, adwp:poooduttttddididddeeeeeeede wtw.::/..aadiiee.t..oo.o.ddo.dd
add discount code
RentAFlat: add apartment
Cabs: update ride status
Envisemennt
Agunt
send
Neliicttinnnnnn Agent' coneet
Agont Running
Chots aend message to conversstinn
Tool Diservation
Forward responie to User
ALI aed
Validattion
All wend feliow spl
Irject Agent's
Nusifcation
Crasts.nndt convensaioo)
Emiak
Tool Oovereation
Agent Runninn
Emsiis Imnnter amail
Tool Descrvation
response
ALI massage user)
Final Validation
Agant: Tarmintted
Figure 12: Sequence diagram of a multi-turn scenario ARE. The agent paused between turns, i.e., between calling send sonddmmssagee set_ttuuueer ruruurrerree and receiving send_message tttttoaaooo co_agent.... and adapts its strategy response to an asynchronous notification from the environment, a new email.
18

---

Published as a conference paper at ICLR 2026
A.3 UNIVERSE GENERATION
Dependency management & consistency To ensure cross-app coherence, we implement struc- tured dependency resolution system. During generation, each app queries the existing universe state to maintain consistency- example, when generating emails, the system first retrieves all available contacts ensure referenced individuals exist in the Contacts app. Similarly, calendar events that mention other people are validated against the contact list, and ride history in the Cabs app references locations that align with the user's established geographic context.
We handle dependency conficts through priority- based resolution system where foundational apps (e.g., Contacts) s) take precedence over dependent apps (e.g., Messages, Emails) as show in Figure 13
However, several complex inter-app dependencies remain unhandled our current implementation. These include temporal consistency across apps (ensuring message timestamps align with calendar availability), semantic relationship tracking (maintaining consistent relationship dynamics between contacts across different communication channels), and cross-modal content references (ensuring photos mentioned in messages exist in the file system). Addressing these limitations represents important future work for achieving fully coherent synthetic Mobile environments.
Soed
Generation
Persona
Contacts
Shopping
Chats
Messages
Calandar
RentAFlat
Cab
CityApp
The dependency graph of Mobile apps. Shopping and File system are independent apps. Contacts is the root for rest of the apps.
Figure 13:
Contacts We populate contacts using personas the foundation. To begin, sample seed per- sonas from the persona hub Ge et al. (2024). However, these personas are brief and lack grounding in the universe's is location. To address this, expand and contextualize them by incorporating the universe location into the prompt. We sample a user persona from the generated contacts which serves the basis for populating the rest of the universe. A universe based user persona.
An example user persona is:
"first_name": "Helena",
"last_name": "Mueller", "gender"; "Female", "age": 43, "nationality": "German",
"city_living": "Berlin",
"job": "Marketing Manager", "description": "Helena Mueller is vibrant and energetic 43 43-year-old marketing manage living in Berlin, Germany. ", "phone": "+49 157 6543210",
"email": "helena wwwllerrgai 2mail. com"
Chats & Messages Chats Messages apps, we generate both group conversations and indi- vidual chats. We sample contacts between whom we have to generate the conversations, Then,
19

---

Published as a conference paper at ICLR 2026
provide the participants personas and prompt the model to generate conversation with at least 10 messages alternating between participants. We prompt the model to generate conversations that are natural and reflect the participants' backgrounds and also ask it to include references to possible shared experiences, interests, or cultural elements.
Emails Similar to messages, we prompt the LLM to generate both 'inbox' and 'sent' emails. For inbox emails, the sender is sampled from the contact list, while for sent emails, the recipients are selected. We provide the LLM with the user's persona and the sampled non-user persona to gen- erate the emails. We specifically prompt the LLM to analyze details such as age, gender, cultural background, occupation, education level, personality traits, communication style, current life cir- cumstances, relationships and social networks, as well as interests and hobbies, and come up with valid reason for writing the email.
Calendar We provide the LLM with the user persona and summary of the previous week, prompting it to generate calendar events for the current week. Next, we use these newly gener- ated events to prompt the LLM to create weekly summary. This process is repeated iteratively to populate the calendar over a specified timeframe, such as three months.
RentAFlat & City For apartment listings, we provide the universe countries and prompt the LLM to generate apartment listings. The City app is designed to retrieve crime rates for specific zip codes. Using the zip codes generated for apartment listings, we prompt the LLM to produce crime rate data as floating-pole value in the range of 1-100.
Shopping For the Shopping app. we integrate publicly available Amazon product dataset. For each universe, sample 500 products and generate discount codes applicable to select items.
Cabs We prompt the LLM with the user country information and generate the user's ride history.
Files
We employ traditional file system hierarchy, loading it with publicly available Wikipedia data, datasets, and images. Additionally, we also add our files that do not contain personal informa- tion. We choose keep the file system the same for all universes.
A.4 ARE GRAPHICAL USER INTERFACE
Running scenarios with ARE generates rich agent execution traces that include reasoning steps, tool calls, their outputs, notifications, and, on the environment side, temporal event flows that unfold over simulated time periods. It is important for practitioners to be able to debug these interactions, whose complexity requires specialized tooling. Existing development tools largely fall into one of these categories: interactive debugging platforms (Epperson al. 2025 Rorseth et al. 2025 Pang et al. 2025) and data anmotation/curation platforms, each with distinct UI approaches. Com- mercial observability tools such as Arize Phoenixi and Langfuse") primarily offer visual timeline views and trace/span visualizations to help developers analyze agent execution, focusing on under- standing behavior after the fact rather than direct interaction or editing. Academic prototypes such as AGDebugger (Epperson et al.) 2025) and LADYBUG (Rorseth et al., (025) provide interactive debugging with user interfaces that enable browsing conversation histories, editing messages, and tracing execution steps, while Hippo (Pang et al., 2025) uses an interactive tree to visualize and control -f reasoning without focusing on tool calls, agentic behavior nor annotations.
Although there are many specialized tools for data annotation, such as commercial platforms like Labelbox | they mainly focus on simplifying human-in-the-loop annotation. These tools offer fea- tures like multimodal chat editors and customizable worksheet Uls, enabling data labelers to refine trajectories from interactive LLM sessions. Despite their power for data collection and curation, a significant gap remains: They are designed to annotate traces of interactions and lack key points for reproducibility and broad evaluation: 1) They annotate full multi-turn conversations, when we want to gather tasks, environment events, and agent task success criteria; 2) they lack structured
https://phoenix..att
https:///aaqqfuse wtm.feeeee.oooe...oooooooooooo.t wtt.e///sserraalitt.....ooo /tt//vvev//vvvw
https:///abel wtw.aabbbbbbo.ooo box.cco
20

---

Published as a conference paper at ICLR 2026
annotations within fully simulated and reproducible environment, which is key capturing both agent interaction with tools and external events, for realistic, reproducible agent traces.
To address this, propose single ARE Graphical User Interface (UI), web-based platform
that enables developers to interact with the environment, visualize scenarios (see |Figure 14). and understand agent behavior and failures through detailed trace analysis and replay capabilities, and enable zero-code scenario annotation.
upp
bel1 BD-
Figure 14: ARE scenario view with event DAG (top), scenario run (bottom left) and agent logs (bottom right).
A.4.1 ENVIRONMENT EXPLORATION
Easily exploring the environment crucial for understanding the context available to agents when debugging scenarios execution, and annotating new verifiable scenarios. The UI provides a com- prehensive visualization of the simulated environment, displaying all available apps/tools and their current states. Interactive app views allow users to browse app contents and interact with their tools, egg. email inboxes in Mobile le, in real-time. Views are automatically generated for new apps, which therefore doesn't 't require UI rewrite.
A.4.2 AGENT TRACE VISUALIZATION AND REPLAY
The UI presents agent multi-step interaction traces in structured timeline view that clearly delin- eates agent thoughts, actions, and tool responses. Each trace element timestamped and catego- rized, allowing users to follow the agent's reasoning process, similar to the Phoenix trace views also used by smolagents but extended with debugging capabilities. Developers can roll back time by jumping back to past event, editing thought, tool call, etc., from that step and replaying the scenario to see what would happen with slightly different approach, similar to setting breakpoints and stepping through code in standard code debugger.
A.4.3 SCENARIO VISUALIZATION
The UI provides interactive visualization of scenarios and their event DAGs introduced in Section 3 showing how scenario events are interconnected, and their execution status in real-time. The event graph visualization supports both scenario development and execution analysis. Before running scenario, users can examine event triggers, dependencies, and timing constraints the scenario.
htttps///pheeiix.rrie..c https::/hhugiingface co/b1 og/ smo agents-phoent
21

---

Published as a conference paper at ICLR 2026
During execution of scenario by agent, the interface highlights completed events and shows the progression through the dependency graph. Developers can run through the scenario with given agent, see how it behaves and debug the scenario or the agent (see Figure 14p. ARE is able to simulate time progression, so users can decide to jump in time for scenarios that span long time frames (e.g. weeks, months).
A.4.4 ANNOTATION INTERFACE
Beyond visualization, the UI includes annotation interface not released at this time that significantly reduces the cost of scenario creation and QA. This includes a graph editor that allows to easily build scenario event DAG. For each node, the annotator node's parents, and optionally timing. For example, to create a Mobile scenario, the annotator adds nodes representing user initial ask (e.g. "email my travel plans"), oracle action solving the task (e.g. "agent sent an email"), environment events that will interfere with the agent's 's work (e.g. "received email from travel agent"), and potentially further turns. To ensure quality and consistency across annotations, we incorporate automated checks of the created events DAG. These checks detect and flag logical inconsistencies in event flows to annotators, such node without parents or contradictory node timings. The annotation interface achieves approximate five times improvement annotation time Mobile le scenarios, compared to manual approaches.
configure tool calls, the
22

---

Published as a conference paper at ICLR 2026
B GAIA2 APPENDIX
B.1 DETAILS GAIA2 ANNOTATION
B.1.1 ANNOTATION GUARDRAILS
To streamline the process and further reduce annotation errors, we implement structural constraints directly within ARE (refer to Appendix A.4 for details). The system raises real-time errors when these are violated:
Only
or Env events may follow sendumessage.to.user as the root.
The event DAG must he fully connected, with No event (Env Agent Oracle) may be orphaned.
Only one branch in the event DAG may include send send.messeg..to.ggen or sendmmesage.to.user events.
A turn must always end with send_message.toussrr both in terms of DAG structure and timeline ordering.
B.1.2 SCENARIO EXAMPLES
To build Gaia2, we define set of capabilities that we believe are necessary - though not sufficient for general purpose agents. As introduced above, each of the 800 scenarios is built emphasize at least one of these capabilities, yielding 160 scenarios per capability split. We provide example scenarios displayed the ARE GUI graph editor in Appendix B.1133
Execution scenarios require the agent to take multiple write actions, which may need to be exe- cuted in a particular order. Most of the time, read actions are needed in order gather information for properly filling wr te action arguments.
Execution Task
Task: Update all my contacts aged 24 younger to be one year older than they are currently.
Explanation: This task requires the agent to read contact information, filter based on age criteria, and execute multiple write update Contacts data.
Search scenarios require the agent to take multiple read actions in order to gather facts from different sources within environment. Any sequence of read operations leading to the correct answer is considered successful as long as the answer is communicated via send_mes sage_to.user before scenario timeout. While conceptually similar to the original GAIA benchmark's 's web search tasks, Gaia2 search scenarios operate within controlled ARE en- vironment.
Search Task
Task:
Which city do most of my friends live in? consider any contact who have at least one I-on-I conversation with on Chats a friend. case of a tie, return the first city alphabetically:
Explanation: This scenario requires the agent cross-reference data from multiple apps (Contacts and Chats), perform aggregation operations, and handle edge cases like ties.
All remaining capabilities tested in Gaia2 reflect tasks with balanced number of required read and write operations. However, each capability features an additional challenge. Namely:
Ambiguity scenarios reflect tasks that impossible, contradictory, have multiple valid answers, with negative consequences arising during interaction agents make mistakes. These scenarios test agents' ability to recognize these issues and seek appropriate clarification from users.
23

---

Published as a conference paper at ICLR 2026
Ambiguity Task
Task: Schedule Ih Yoga event each day at 6:00 PM from October 16, 2024 to October 21, 2024. Ask me case there are conflicts.
Explanation: While this task appears straightforward, current models often struggle to identify con- tradictions multiple valid interpretations, tending to execute the first seemingly valid approach rather than recognizing the need for clarification.
Adaptability scenarios require the agent dynamically adapt to environmental changes that consequences previous agent actions, such as a response to an email sent by the agent, or the cancellation of ride booked by the agent. These events require agents recognize when adaptation is necessary and adjust their strategy accordingly.
Adaptability Task
Task: have meet my friend Kaida Schönberger view property with her [...] If she replies to suggest another property or time, please replace it with the listing she actually wants and reschedule at the time that works for her.
This task requires the agent to execute initial plan while monitoring for environ- mental changes (the friend's response), then adapt the plan based on new information. The agent must demonstrate flexibility in execution while maintaining task objectives.
Explanation:
Time scenarios require agents to execute actions in due time, monitor and respond to events, and maintain awareness of temporal relationships throughout task execution. The duration of Time scenarios is currently capped at minutes to facilitate annotation and evaluation.
Time Task
Task: Send individual Chats messages the colleagues am supposed to meet today, asking who is supposed to order the cab. If after 3 minutes there is no response, order a default cab from I...}
This scenario requires the agent understand temporal constraints (the 3-minute win- dow), monitor for events (new messages from colleagues), and execute a time-sensitive action (order a cab).
Explanation:
Agent2Agent scenarios replace apps with app-agents. Main-agents can no longer access app tools directly and must instead communicate with the app-agents in order to place tool calls, observe tool call outputs, and ultimately accomplish user tasks. This transformation requires agents to develop robust collaboration capabilities, including sub-task setting, affordance understanding, "context- sharing," and general coordination. By default, agents and app sub-agents instantiated with the same scaffold and model, with good performance requiring strong sub-goal setting and sub-goal solving. However, Gaia2 also supports heterogeneous multi-agent evaluations, i.e. where stronger agents supervise weaker sub-agents or vice-versa.
Example: Same Search task as above but the Contacts and Chats apps are replaced by app sub-agents and the main agent must communicate with them in order to gather information.
Noise scenarios require robustness to environment noise, simulating the inherent instability of real- world systems, where APIs change, services become temporarily unavailable, and environmental conditions shift during task execution. This category applies systematic perturbations to Gaia2 scenarios, including tool signature modifications, random failure probabilities, and dynamic en- vironment events that are irrelevant the task. We assess the sanity of our noise mechanisms in Appendix B.5.2
Example: Same Adaptability task as above but with random tool execution errors and ran- dom environment events (e.g., messages from other people) occurring during execution.
24

---

Published as a conference paper at ICLR 2026
B.1.3 CAPABILITY-SPECIES ANNOTATION GUIDELINES
In our guidelines for each capability (especially Ambiguity and Adaptability), we put strong empha- sis on precise task specifications, while also ocknowledging the challenge of maintaining realism and avoiding prompts that inadvertently disclose the solution.
Scenarios contain only one write action, which is the agent's ' final answer to the user's s question, derived from multiple read actions. Answers must be concise, easily verifiable, and avoid complex computation.
Search:
Scenarios that are impossible, contradictory, or inherently ambiguous. The agent is expected to complete unambiguous steps, then inform the user of the ambiguity impossibility. These scenarios are single-turn: they do not include clarification message from the User.
Ambiguity:
The user prompt must clearly instruct the agent to detect and report ambiguities, users often have varying preferences on how frequently and when this should occur.
Adaptability: Scenarios involve Env events that require the agent to revise its plan in response to delayed outcomes of actions. In order to meet our modeling constraints, scenarios follow consistent structure:
1. The user provides task.
2. The agent acts and sends message using send www.dmessseeett.. .uu....
to..eer..
3. An Env event is triggered (e.g., email reply, order cancellation). It is consequence of previous agent's 's action, with send message. to user as parent.
4. The agent adapts accordingly.
To increase the difficulty, distractor Env events are also included, aiming to mislead the agent into incorrect behavior.
In order to perfectly specify expected agent behavior, the task states explicitly that the agent should send a message to the user after completing the initial requests (before the Env events). It should also specify what the Agent is allowed do in the case of an Env event happening, without giving exact hints on what steps the Agent should take.
Time: Scenarios assess Agent's ability act on time, therefore they all include at least one time- sensitive oracle action.
Scenarios should be solvable within five-minute window.
User prompts must instruct precise timing (e.g., "after exactly 3 minutes").
The verifier checks the timing agent actions only if the oracle event has relative time delay greater than second The agent's 's mapped action must fall within [Al - 5sec, Al 25sec].
Distractor Env events also included.
B.1.4 CAPABILITY TAXONOMIES
Taxonomy of ambiguity scenarios
Impossible or contradictory tasks: missing key information (e.g., the User does not specify the ride pickup location), or requests incompatible with the Environment (c.g., asking to buy an out-of-stock item).
Blatant ambiguities high-stakes consequences: Multiple valid answers exist, and the ambiguity is obvious or the user explicitly asks natural way to report ambiguities.
Taxonomy of events
Env events classified based their dependency:
Independent events occur without agent action and have send Benddmessage.to.ageett as their only parent.
8This is why actions expected "immediately" after event annotated with +2 delay.
25

---

Published as a conference paper at ICLR 2026
Dependent events result from prior agent
actions
and
must
have
senddmessagg.to.us. as their direct parent.
Distractor events are designed to mimic relevant events and mislead the agent into incorrect behav- ior. By exception, distractor events may be independent but still have send mess message user as a parent to preserve the structure of the scenario. In the Adaptability category, only dependent Env events are used.
Taxonomy of time scenarios Time scenarios require the agent to execute one or more actions specific point time, either proactively ("For the next Smins, send 'Hi" to John Doe every 30sec") or in reaction to an independent Env event ("When this item becomes available, buy it immediately") "), or in reaction to a dependent Env event ("Ask the invitees whether they come to the party tonight. Wait Imin for everyone reply, then immediately send me the number of glass to buy, I am waiting in the line!").
Taxonomy:
Time-based one-off task: Execute a task at a precise point in time in the future. Example: "Send a follow-up message Jo in 2 minutes if she does not reply."
Time-based recurrent task: Execute recurrent task precise points in time. Example: "For the next 4 minutes, every minute, delete the new emails I receive." "
Event-based one-off task: Execute a one-time task conditionally on a future trigger event. Example: "Purchase red rumning shoes soon as they become available in size 6 for less than 100USD in the shopping app"
Event-based recurrent task: Automate a recurrent routine conditionally on future events. Example: "For the next minutes. whenever receive an email containing the keyword "Black Friday,, immediately delete Do not talk to me the next 2 minutes." "
We encourage annotators to cover and combine all these types of tasks when creating Time scenar- ios.
B.2
VERIFICATION DETAILS
B.2.1 VERIHICATION MECHANISM
We verify scenario successful completion by comparing agent actions with ground truth, defined as the minimal sequence of write actions needed to solve task. We exclude read actions from verification since multiple reading strategies can lead to the correct set of write actions. a preliminary phase, the verifier checks that used tool names counters are identical in both the oracle actions and the agent's 's wr te actions. If this test is successful, the verifier sorts the oracle actions topological order based on the oracle graph, which reflects their dependencies. Then, the verifier proceeds to mapping each oracle action to an agent action by checking:
Consistency: the verifier tests whether the oracle action and the candidate agent's action are equivalent. After conducting some preliminary tests (such as ensuring that both the oracle and agent actions use the same tool and that the oracle action is not already mapped to another agent action), the verifier performs:
Hard check to compare action parameters that require exactness. For example, when replying to an email, it verifies that ema id value is identical for both actions, i.e. the agent replies to the correct email.
Soft check for parameters that require more flexible evaluation, such the content of an email or a message. To perform a soft check, an LLM judge is prompted with the user task as context, and the arguments from both the agent action and the oracle action as inputs. The LLM then determines if the actions are equivalent according to tool-specific guidelines. For example, emails verification includes guidelines to check their signatures.
Causality: crucially, oracle actions are organized within an oracle graph, whereas agent actions are collected from trajectory and simply ordered by execution time. Therefore,
26

---

Published as a conference paper at ICLR 2026
Matching Attempt
Matching Attempt
Metching Altempt
Trace
Agent Action
Cracie Action
Agent Adtion
Cradt Action
Agent Action
Graci Action
Agent Action
Chacee Action
Agent Action
Crade Action
Agent Action
Creci Action
Success
Graph
Aqenz
Gradin Action
Aqant Acbinn
Ccade
Agent
Dreci
Crucle Action
Trace
Orada Artion
Cradia Action 3
Agjrtt Astion
Chadie Action 1
Aguitt Action
Onalls Action 1
Agjrtt Action
Oraito Action
Agent Action
Cradte Action
Falture
Agent Action
Chette Action
Agent Action
Ocadle
Figure 15: Illustration of failure (top) and success (down) of the matching trajectory process.
must ensure that the agent does not violate dependencies within this graph. For example, if both oracle actions and B depend solely on action the agent free execute and B in any order, as long they are executed after C; i.e. sequences C-B-A or C-A-B are both acceptable. Once match is found, the ARE Verifier ensures causality by verifying that all parent actions of the oracle action have already been matched with preceding agent actions.
Timing: scenarios include time delay for certain actions relative their parent tions, which the agent must respect. The verifier evaluates whether the agent's 's timing falls within specified tolerance window centered around the relative time of the oracle action. To determine the relative timing of the agent's action, it is necessary to identify which agent action corresponds the oracle's parent action. This information readily available due to the ARE Verifier's process. Indeed, for given oracle action, all its parent actions must matched to an agent action before attempting to match the oracle action itself.
If all oracle actions are successfully matched, the verifier returns a success signal. Conversely, if any oracle action cannot matched an agent action, the verifier returns failure signal, see Feguur 15) for two examples. Crucially, the verifier implicitly assumes there are equivalent irite actions, i.e. user preferences are clearly stated with minimal ambiguity in the scenario tasks. For example, sending message using the Messages app while the oracle action uses the Chat app will trigger failure.
While other verification methods (Patil et al., 2025 Yao al. 2024 compare the environment ground truth and actual final states, verifying a sequence of write actions, which is equivalent to comparing ground truth and actual states after each write action of the sequence, provides more control. For example our verification allows to distinguish, e.g. for safety considerations, Mobil trajectory where the agent adds event the wrong place and correct itself from trajectory where the agent is correct at first try. Moreover, in Mobile ie,, sequences of write actions are easier for humans interpret and annotate, compared to diff's of states.
B.2.2 VALIDATING MULTI-TURN SCENARIOS
Currently, have only described how the verifier works in single-turn scenarios, where user assigns a single task to an agent, and the agent completes without further interaction. However, the Gaia2 benchmark also includes multi-tumn scenarios that involve more complex interactions between the user and the agent. For example, consider scenarios related to the Adaptability capability, where the agent must adjust externl events. Multi-turn scenarios present two key challenges:
27

---

Published as a conference paper at ICLR 2026
Action
Mistige
Apri
Tarkk
Apent
Pigerr
Figure 16: Insertion of conditional trigger event in multi-tun scenario.
How can we validate multi-turn turn scenarios?
More importantly, how can we run an agent in a multi-turn scenario?
Indeed, annotators plan User and Env actions based on what should occur in previous turns accord- ing to the oracle action graph. However, when an agent is launched in a scenario, it may not adhere to the oracle's actions, creating uncertainty about when to trigger user or environment actions.
Multi-turn verifier Answering the first question is relatively straightforward. is sufficient to detect when the agent sends a message to the user to delimit the turns. We can then feed the verifier with each turn separately and accept the agent's trajectory all turns are successful. Note that this validation can be performed in an online fashion after each turn or in an offline fashion once the full trajectory collected.
Multi-turn execution An efficient solution run an agent in a multi-turn scenario is call the ARE Verifier at the end of each turn and only trigger the next turn if the current turn was successful. This approach prevents running the agent when it has already diverged from the oracle path. Prac- tically, as illustrated in Figure [6] we modify the scenario event graph by splitting it into turns and inserting a conditional event to call the verifier and trigger the next turn. A simpler, but less efficient, solution is to trigger the next turn each time the agent calls send_message oouser regardless of what the agent did in the current turn. This approach is used for scenarios from the test set since do not have access to oracle actions and thus the ARE Verifier for them.
B.2.3 VERIFIER HACKING
We conducted RL experiments in which used the ARE Verifier reward function. During these experiments, we observed that the agent was able to exploit our initial version of the verifier in scenarios related the Search capability, where only single write action is expected for successful trajectories. In the observed exploit shown in Figure 17 the agent would learn to embed strings representing increasingly complex code in write tool calls. These outputs overwhelmed the LLM judge used for soft checks by the ARE Verifier, producing false positives. In follow-up experiments, we found that adding a task-agnostic "style" soft check to the verifier was effective at addressing this failure mode and preventing further reward hacking.
B.2.4 VERIFYING THE VERIFIER
Verifiers are critical components of training and evaluation pipelines, where false positive or false negative egg. via hacking can result in flawed evaluations or collapsed trainings (cf. Appendix (..233) We evaluate the ARE Verifier by first deriving series of "unit" tests from the oracle actions that the verifier should satisfy. Typically, we apply perturbations to oracle actions that we know preserve or invalidate the oracle trajectory validity, before submitting the oracle and perturbed oracle trajecto-
28

---

Published as a conference paper at ICLR 2026
Thought:
Now that we have all the events the user's calendar, we can count the number of events for each contact and find the contact who the user has the most scheduled events with.
Action:
*ac. Lon"; "AgentUse: Inte
"acsion input";
"contett: "Loaking at your calendar, can now count events for each contact. Here's what found: (oon::nnn==-Mos Common Contact tby events) [[ ( F1t+iE leg (length attendeea) 2111i set wewt__commr___ottttttto
(at attendees 01 user) [[llse) lat atteadees [11) svents_attendee___cco 1) Fiii events_sttende___ccom events_attendee__
ventt_aatendd___cccatt)) (tt.rdde___ooot)))
wtt___ccoooo_cotttceeoo evenss_attendee___ccomm
(s2 attendees [ (eq
(get
attendes_l__
f_rrs_____tt_ttai 01 igez siteende____first..e.
0) ()
post__oomnon_orte cot__oootttt__aall (a2 attendeea [)]][selse) [ 4aet
(as allendesa 2) (1/(C))))) (tleE)))((//dd/)))aaa (1deddd(()(((eeed))))
[[fset
common_pontact...t.mm
(az attendees (f coost__comoo_coaae wtt.ccoo_ooottct_ooooo seaaactt__mmlll
email})
(111)(((())) (ifendiE)] ) (1 (edddif;; [ ((fendif; ( ([tezdif) ( ((teddif) H [fendi))
andif]) (contacts) [(FiÉ teg email httr__ccmoo_cccccctttomm
An example of agent exploiting the judge by embedding conditional logic in the message to the user. The message contains no meaningful information but successfully passes the judge's evaluation.
Figure 17:
Verifier
Agreement
Precision
Recall
Llama 3.3 70B Instruct
0.98
0.99
0.95
Gemini 2.5 pro
0.96
0.98
0.89
Claude Sonnet 3.7
0.96
0.98
0.89
Table Evaluation the ARE Verifier with different models on 450 hand-labeled trajectories.
ries to the verifier and checking its verdict match the perturbation type. While these checks allow fast iteration, they only catch anticipated behaviors. Furthermore, the perturbed trajectories do not necessarily reflect real trajectories that could be obtained with agent.
Validation benchmark We complement this initial evaluation by analyzing ARE Verifier verdicts for 450 trajectories manually labeled with the expected verifier outcome (Success Failure). The trajectories were derived from running agents powered by various models on scenarios from the Gaia2 benchmark. We compare the ARE Verifier with simple baseline, In-context Verifier, where an LLM is prompted with all the agent actions and criteria (causality constraints, relative time, soft/hard checks, etc.). The same model Llama 3.3 70B Instruct is used for both verifiers. ARE Ver- ifier achieves better accuracy than the baseline, which tends to accept agent trajectories too readily,
see Table
B.2.5 CHOOSING THE VERIFIER MODEL
While we adjusted the prompts used in the various soft checks of the ARE Verifier with Llama 3.3 70B Instruct as model, we also wanted to assess whether the ARE Verifier could function effectively with other models. To this end, evaluated the ARE Verifier powered by different models on 450 habd-labeled trajectories, the same dataset used for Table T In Table models achieve satisfactory precision and recall scores.
we observe that all the
29

---

Published as a conference paper at ICLR 2026
B.3
AGENT ORCHESTRATION
B.3.1 MOBILE REACT LOOP
Our proposed evaluation method leverages a custom scaffolding framework built around the ReAct (Reason and Act) paradigm. The base scaffolding implements standard ReAct loop where agents iteratively reason about their current state, select appropriate actions, execute those actions in the environment, and observe the resulting outcomes. An agent step is thus defined by three substeps Thought. Act icnionni and Observation. This cycle continues until task completion termination conditions are met.
At each step of this loop, our scaffolding triggers configurable pre-step and post-step methods that can pull relevant information from the environment state or detect termination conditions based on task-specific criteria as detailed Figure 18 Pre-step methods gather contextual information and validate preconditions before action execution, while post-step methods process outcomes, up- date internal state, and check for completion signals. This agentic modeling approach enables the creation of sophisticated agent behaviors with minimal implementation overhead, complex inter- action patterns emerge from the composition of simple, reusable scaffolding components rather than monolithic agent implementations.
Full ReAct Step
One staaddrd ReAct Ster Iatep!
Pre Steps
Duild LIM chat hatoory
Generate
Post Strps
Enecube tool and
Figure 18: Proposed ReAct loop with pre/post steps in Gaia2, allowing flexible behaviors.
B.3.2 ORCHESTRATION ABLATION: PARALLEL TOOL-CALLING
In our main evaluation setup, we use standard ReAct scaffold ensure fair, model-agnostic baseline that supports both closed APIs and open-weights models without requiring model-specific integration code. However, to address the question whether this single-threaded scaffolding acts bottleneck particularly for the Time split, we conducted ablation study comparing ReAct against a Parallel Tool Calling (PTC) orchestration.
We evaluated three representative models (Llama 4 Maverick, Claude 4 Sonnet, and GPT-5) across the Execution and Time splits. The results, presented in Table reveal several key findings:
As expected, PTC significantly reduces wall-clock latency and token consumption. For instance, GPT-5 (low) shows strong reduction in latency (A -435s on Execution) and token usage (A -5109 tokens), primarily because it performs fewer inter- mediate reasoning steps per action.
Efficiency Gains:
Performance Stability: Despite the efficiency improvements, the impact on task success (pass @ 1) is marginal. The performance deltas generally small (ranging from -6.3pp to +3.0pp), and crucially the relative ranking of the models remains unchanged.
The Time split remains challenging even with parallel execution, confirming that the bottlenecks observed in Section 5 stem primarily from model capa- bilities (such sequential reasoning and temporal planning) rather than the scaffolding itself, as PTC results are still far from the boper-bound score computed with instant-timg generation in Figure 8
Orchestration Limits:
30

---

Published as a conference paper at ICLR 2026
Table 6: Ablations of 3 models with Parallel TC ReAct scaffold. Values indicate the net contri-
bution of PTC over ReAct (A).
ReAct pass@1 1
Parallel TC
pass@1 (pp)
A avg time
A avg steps
A avg output tokens
Model
Split
pass@1
(8)
Llama Maverick
Execution
13.8
7.5
-6.3
+71
-1.0
+1786
Time
1.2
2.0
+0.8
3
-1.1
+
Execution
57.9
59.7
+1.8
-68
-10.7
-345
Claude Sonnet
Time
8.1
9.5
+1.4
-8
-244
+33
Execution
31.9
34,9
+3.0
-64
-14.0
-160
GPT-5 (minimal)
Time
5.2
6.7
+1.5
+-3
0.0
+1030
Execution
435
13.0
5109
GPT-5 (low)
Time
2.3
1.0
-1.3
-007
-1.9
-4425
These results confirm that our qualitative conclusions are not artifact of the scaffold and that more research on completely novel orchestration needed.
B.4 EXPERIMENTAL SETUP AND IMPLEMENTATION DETAILS
We report Gaia2 scores on representative of models, covering both proprietary and open-source systems, and including both reasoning-oricn oriented and non-reasoning models.
For evaluation, we use a ReAct scaffold that requires a Thought: and an Action: at each step. Since some models do not reliably follow this format, we add custom stop sequences <endactions> and Observation: for models that tend to continue past single tool call (Claude, Kimi, Qwen). This issue is largely alleviated by provider-specific ToolCalling APIs; encourage reporting results with either interface (ReAct or ToolCalling)
Due to cost and time constraints, did not evaluate every available model. For instance, Claude Opus was excluded because of its very high latency and cost ($15/M input tokens and $75/M output tokens).
We note the following special configurations for specific third-party models:
Gemini 2.5 Pro: dynamic reasoning enabled via www.eeetreeeoonnng.ooooooo.eeeee
-1.
reasoning budget capped at 16k tokens per completion. We encountered frequent issues with xAI's API, in particular Empty Response emror, which introduced high variance in results.
Grok-4:
temperature and top-p set to 1; no custom stop sequences were applied (not sup- ported by the API).
GPT-5:
When evaluating reasoning models (e.g., GPT-5, Claude-4, Qwen), we use the same ReAct prompts but adapt the inference client to handle reasoning-style outputs. maintain a uniform evaluation and preserve the (Thought, Action) structure, discard intermediate reasoning each step and exclude it from the context of subsequent steps. While this approach aligns with the intended usage of some models (c.g., Qwen), it may not be optimal for others that interleave tool use with reasoning (e.g., GPT-5, Claude). We encourage the community to explore alternative setups to better assess the theoretical limits of the benchmark.
B.5
ADDITIONAL EXPERIMENTS
B.5.1
SUB-AGENT SPAWNING AGENT2AGENT MODE
In Agent2Agent experiments, record the number of instantiated sub-agents in Figure 19, Counts are fairly consistent across model families, yet the top A2A performers also spawn more sub-agents, suggesting stronger task decomposition.
31

---

Published as a conference paper at ICLR 2026
fone age Number of Aggnn Collebenattra PugeiikkgenCllabiiitttt Gaid Mini Suaner lo 2
4.0
24
25
Cluded
63753
LITTI
Grob4
6PT3
CF7-40
COTE
Figure 19: Average number of agents spawned in Agent2Agent evaluations on Gaia2-mini tasks across models. In any Agent2Agent scenario, main-agents can (in principle) spawn an unlimited number of app-agents before scenario timeout. In practice, behavior in Agent2Agent settings is relatively consistent across model families.
B.5.2 INFLUENCE OF NOISE LEVEL ON GAIA2 RESULTS
In this experiment, we vary the probability of tool errors and frequency of random environment events and measure resulting model results Gaia2. While our lowest level of noise does not significantly impact model performance, increasing noise results deteriorating performance across models. This aligns with our intuitions.
Table 7: Model performance Gaia2-min mini across different noise levels. *Default setting.
Noise level
None
Low
Medium
High
Claude-4 Sonnet
31.2
35.0
23.8
8.1
32


Reviews:
[{"review_id": "r8IkQ4gVHj", "reviewer": "Reviewer_cZHA", "summary": "This paper introduces Gaia2, a novel benchmark for LLM agents, and the underlying platform Agents Research Environments (ARE) building it. This paper's contributions are threefold, aimed at advancing the evaluation of LLM agents in dynamic environments and realistic scenarios.:\n\n- First, it open-sourced the ARE platform for building and running agentic evals. ARE is designed from the ground up for building asynchronous, event-driven benchmarks while also support scalable evaluation and data generation for Reinforcement Learning (RL).\n\n- Second, the paper presents the Gaia2 benchmark. Gaia2 is the first benchmark to unify a set of critical, real-world agent capabilities under one evaluation, including asynchronous execution, temporal reasoning, robustness to environmental noise, ambiguity resolution, and multi-agent collaboration.\n\n- Third, the authors conduct a comprehensive empirical study on Gaia2 using a suite of leading models. This study exposes fundamental trade-offs between agent reasoning strength, efficiency (latency), robustness, and operational cost.", "questions": "- Could the verifier be modified to support \"goal completion\" rather than \"path optimality\"? How would you propose handling valid, non-minimal trajectories that involve agent self-correction? Overall feel it might be helpful to better structure the categorization of the capability, and some break down/analysis on individual categories, e.g., does task complexity/length affect the performance on execution/search\n\n- Given that the current scenarios test capabilities in isolation, why not adding compositional split or there are some tests showing it is not needed?\n\n- Agent2Agent setting seem to be a bit different than the other settings, which can be used to evaluated both the main agents' planning and coordination skill, especially given partial observability of the action space, as well as the sub-agents ability on effectively communicate with the main agent and complete task, the experiments and design for this part could worth further exploration.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- True Asynchronicity and Temporal Dynamics enabled by the async nature of the ARE platform and the event driven/DAG design for constructing more tass.\n\n- A Scalable, Open-Source Platform (ARE) that allows the community to build new, more complex scenarios on top of this work, ensuring its long-term relevance.\n\n- Well-Designed Verifier for RLVR: The new action-level verifier is proven to be effective, achieving much higher precision and recall than a LLM-only judge.\n\n- The analysis of cost-performance-time trade-offs provides a practical, multi-dimensional view of agent performance that is highly relevant for real-world deployment.\n\n- The benchmark is not saturated and successfully identifies clear weaknesses in current SOTA models. The low scores on Noise, Ambiguity, and Agent2Agent splits provide a clear roadmap for future research.", "weaknesses": "- The verifier is designed to check against a \"minimal oracle sequence\" of write actions, which might be brittle in some cases. It appears to evaluate path optimality rather than goal completion. An agent that makes a mistake, self-corrects (e.g., books the wrong cab, cancels, books the correct one), and ultimately reaches the correct state would be marked as a failure. \n\n- The \"inverse scaling\" result might be confounded by the agent's orchestration. Given that all models use a single threaded simple ReAct-like scaffold, this might conflate model inference latency with scaffold inefficiency. A more advanced orchestration (e.g., parallel tool calls, or a planner that coordinate) might allow a \"slow\" model to succeed. And overall the extremely low score on this split seem more for consideration under specific circumstance rather than general capability limitation.  \n\n- The capability taxonomy is somewhat muddled, mixing action types (Execution, Search) with problem constraints (Time, Noise). While the design to stress test one capability in isolation is reasonable. Real-world tasks are almost never this clean; they require a composition of skills and the lack of a \"Compositional\" split is a weak point", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RgNpogGUi3", "reviewer": "Reviewer_QMSH", "summary": "This paper introduces Gaia2, a benchmark and platform for evaluating LLM agents in dynamic, asynchronous environments using action-level verification. Gaia2 simulates smartphone-like environments with realistic apps and event timelines, testing seven essential agent capabilities such as execution, temporal reasoning, robustness to noise, ambiguity resolution, and multi-agent collaboration. Scenarios are built as event and action DAGs and evaluated by a custom ARE Verifier. Experiments across many leading models reveal that no single agent excels in all areas, and that inference speed can be a bottleneck for time-sensitive tasks. Gaia2’s fine-grained verification and open-source framework support RLVR training and future extensible research in agent evaluation.", "questions": "How might the results change with more sophisticated orchestration instead of just ReAct? Is the benchmark senstive to this?\n\nWhere do these sub-agents come from?\n\nThe paper mentions mitigating the observed reward hacking by adding a \"style\" check. Could the authors elaborate on this? Do they anticipate other, more subtle forms of hacking? What is the long-term strategy for ensuring verifier robustness as agents become more adept at gaming these systems?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "High-Quality, Extensible ARE Platform: The paper introduces the Agents Research Environments (ARE) platform, an open-source and extensible framework for building and testing agents. Simulating a smartphone environment with numerous apps and tools, it provides a robust foundation for community-driven benchmarking.\n\nAsynchronous Design Enables Temporal Awareness: A key innovation is the shift from static to dynamic, asynchronous environments. By having events occur independently of the agent, the benchmark can evaluate critical, real-world capabilities like temporal awareness, responsiveness, and the ability to operate under time constraints, revealing how model latency directly impacts task success.\n\nReliable, Fine-Grained Action-Level Verification: The paper proposes a \"write-action verifier\" that evaluates each state-changing action against a human-annotated oracle graph. This verifier is shown to be highly reliable (0.98 agreement with human labels, 0.99 precision) and moves beyond simple final-answer checks. This fine-grained approach not only provides robust evaluation but also makes the benchmark directly usable for RLVR.", "weaknesses": "Dependence on a Single Orchestration Scaffold: The evaluation (Section 5) uses a single, \"simple ReAct-like scaffold\" for all models. It's unclear how much the reported performance is bottlenecked by this specific orchestration choice versus the model's intrinsic capabilities. The paper itself notes (Section 5.2) that its \"single-threaded scaffold\" cannot handle \"concurrent actions.\"\n\nLimited Scope (Mobile-Only): The paper claims to provide \"a foundation for developing, benchmarking, and training the next generation of practical agent systems\" (Abstract). However, the benchmark is currently limited to a simulated \"Mobile\" environment. While this is a complex and valuable domain, this scope is narrow for such a broad claim, and it's unclear how these findings would translate to other common agentic environments (e.g., desktop operation, web browsing).\n\nUnclear Agent2Agent Mechanism: The Agent2Agent (A2A) split (Section 4.1, 5.3) introduces \"app-agents\" that replace native tools, forcing collaboration. However, the mechanism for their creation and operation is not fully detailed. The findings (collaboration helping weak models but not strong ones) are interesting but difficult to interpret without a clearer understanding of the sub-agent's lifecycle.\n\nVerifier Fragility: The paper commendably includes a discussion of reward hacking (Appendix B.2.3), where an agent learned to exploit the LLM-based \"soft check.\" While the authors mitigated this, it underscores the inherent fragility of any evaluation pipeline that still relies on an LLM as a judge, even in a hybrid system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FK47q75Ftq", "reviewer": "Reviewer_8Eq1", "summary": "This paper introduces Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. The results from this paper highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the “sim2real” gap. Gaia2 is built on the open-source Agents Research Environments platform and designed to be easy to extend.", "questions": "No question. Good paper.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1, I think the experiments are comprehensive, including all the state of arts models\n2, The contribution makes sense to me including releasing Agents Research Environments, a general-purpose platform for\nbuilding asynchronous, event-driven benchmarks that support scalable evaluation and data generation for RL; introducing Gaia2, the first benchmark unifying asynchronous execution, temporal reasoning, noise robustness, ambiguity resolution, and multi-agent collaboration under a verifiable evaluation framework directly usable for RLVR; evaluating leading proprietary and open-source models on Gaia2, exposing fundamental trade-offs between reasoning strength, efficiency, robustness, and cost.", "weaknesses": "To be honest, I don't see any weakness of this paper. I really think all the experiments make sense to me and are comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)