Scaling Agent Learning via Experience Synthesis
Authors: Zhaorun Chen
Zhuokai Zhao
Kai Zhang
Bo Liu
Qi Qi
Yifan Wu
Tarun Kalluri
Xuefei Cao
Yuanhao Xiong
Haibo Tong
Huaxiu Yao
Hengduo Li
Jiacheng Zhu
Xian Li
Dawn Song
Bo Li
Jason E Weston
Dat Huynh
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agent
Reinforcement Learning
Data Synthesis
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce DreamGym, a unified framework that synthesizes diverse agent experiences to enable scalable RL for LLM agents, outperforming baselines in synthetic and sim-to-real settings with minimal real interactions.

Abstract: While reinforcement learning (RL) can empower autonomous agents by enabling self-improvement through interaction, its practical adoption remains challenging due to costly rollouts, limited task diversity, unreliable reward signals, and infrastructure complexity, all of which obstruct the collection of scalable experience data. To address these challenges, we introduce DreamGym, the first unified framework designed to synthesize diverse experiences with scalability in mind to enable effective online RL training for autonomous agents. Rather than relying on expensive real-environment rollouts, DreamGym distills environment dynamics into a reasoning-based experience model that derives consistent state transitions and feedback signals through step-by-step reasoning, enabling scalable agent rollout collection for RL. To improve the stability and quality of transitions, DreamGym leverages an experience replay buffer initialized with offline real-world data and continuously enriched with fresh interactions to actively support agent training. To improve knowledge acquisition, DreamGym adaptively generates new tasks that challenge the current agent policy, enabling more effective online curriculum learning. Experiments across diverse environments and agent backbones demonstrate that DreamGym substantially improves RL training, both in fully synthetic settings and in sim-to-real transfer scenarios. On non-RL-ready tasks like WebArena, DreamGym outperforms all baselines by over 30%. And in RL-ready but costly settings, it matches GRPO and PPO performance using only synthetic interactions. When transferring a policy trained purely on synthetic experiences to real-environment RL, DreamGym yields significant additional performance gains while requiring far fewer real-world interactions, providing a scalable warm-start strategy for general-purpose RL.

Published as a conference paper at ICLR 2026
SCALING AGENT LEARNING VIA EXPERIENCE SYNTHESIS
Zhaorun Chen!.3. Zhuokai Zhao', Kai Zhang' Bo Liu", Qi Qi', Yifan Wu', Tarun Kalluri', Sara Cao', Yuanhao Xiong', Haibo Tong', Huaxiu Yao', Hengduo Li', Jiacheng Zhu', Xian Li", Dawn Song", Bo Li Jason Weston Dat Huynh Meta Superintelligence Labs, "FAIR at Meta, University of Chicago, SUNC, UCC Berkeley
ABSTRACT
While reinforcement learning (RL) can empower autonomous agents by enabling seff-improvement through interaction, its practical adoption remains challenging due to costly rollouts, limited task diversity, unreliable reward signals, and infras- tructure complexity, all of which obstruct the collection of scalable experience data. To address these challenges, we introduce DREAMGYM, the first unified framework designed synthesize diverse experiences with scalability in mind enable effective online RI. training for autonomous agents. Rather than relying on expensive real-environment rollouts, DREAMGYM distills environment dynamics into reasoning based experience model that derives consistent state transitions and feedback signals through step-by-step reasoning, enabling scalable agent rollout collection for RL. To improve the stability and quality of transitions, DREAMGY Y leverages an experience replay buffer initialized with offline real-world data and continuously enriched with fresh interactions to actively support agent training. To improve knowledge acquisition, DREAMGYM adaptively generates new tasks that challenge the current agent policy, enabling more effective online curriculum learning. Experiments across diverse environments and agent backbones demon- strate that DREAMGYM substantially improves RL training, both fully synthetic settings and in sim-to-real transfer scenarios. On non-RL- so--RL-ready tasks like We- bArena, DREAMGYM outperforms all baselines by over 30%. And in RL-ready but costly settings, it matches GRPO and PPO performance using only synthetic interactions. When transferring policy trained purely on synthetic experiences to real-environment RL, DREAMGYM yields significant additional performance gains while requiring far fewer real- world interactions, providing a scalable warm-start strategy for general-purpos. RL.
INTRODUCTION
(A) Agant Learning Paracigm
Autonomous agents based large language models (LLMs) are being widely adopted across broad range of tasks given their comprehensive pre-trained semantic knowl- edge. These agents have already shown promise in applications such web naviga- tion (Zhou et al.), embodied control (Shrid) har and multi-turn tool use (ao et al. 2024). However, while these agents can leverage strong language priors to reason and plan, their performance in downstream interactive settings remains limited (Wang et al. 2024). As we step into the era of expe- rience (Silver & Sutton 2025), promising direction for building more robust and adap-
Real Environment
Taskss
Agent
scarce & costly
not
reward signal
sparse & unstable
(b) Scalable Agent Learning via Experience Synthesis
actions
Tasks
Agent
Expariance Model
challenging tack variations
raward signals
vectorized unified
useful cheap
abundant
sdaptable
Figure 1: Compared to the traditional agent learning paradigm, DREAMGYM provides the first scalable and effective RL framework with unified infrastructure.
Work done at Meta. Joint last author. dathuynh) @meta.com.
Correspondence to zhaorun @ @uchicago.cdu, (jase,

---

Published as a conference paper at ICLR 2026
tive language agents is reinforcement learning (RL), where agents improve by interacting with environments and bootstrapping from their own experiences (Schulman et al. 2017), as illustrated in figure (a).
Despite its potential, training LLM agents via RL remains highly challenging in practice. The most fundamental barrier is the high cost and low sample efficiency of collecting large-scale, diverse, and informative online interaction data (Wei et al. 2025 Jiang et al. 2025). Real environments often involve long interaction sequences, high computational cost per step, and sparse reward feedback, making it prohibitively expensive to gather sufficient amount of data for modern RL algorithms (Patil al.; Shao et al.) 2024). Beyond computational cost, there is also a lack of diverse, scalable tasks, where most existing environments provide only a limited, static set of instructions. while RL training requires broad range of tasks for effective exploration (Eysenbach et al., 2018). However, scaling task instructions is inherently difficult, as validating their feasibility often demands costly human expertise (Xue et al.) 2025), leaving current environments insufficient for goal-conditioned RI.. The third major barrier is the instability of reward signals. Many interactive settings, such as web pages and GUIs, are highly dynamic and lack consistent behaviors, resulting in noisy, sparse, or even false feedback that hinders stable learning (Deng al. 2023). Safety concerns further compound these challenges, as certain actions are irreversible (e.g., deleting an item on a real website), and most environments lack reliable reset mechanisms (Zhou al.). Finally, the infrastructure difficulty of constructing RI RI--redyy environments also remain challenging. Existing systems are heterogeneous and often rely on heavyweight backends like Docker (Jimenez et al.) or virtual machines (Xie et al., 2024b), making large-batch rollout sampling ingineeringiinenne intensive and costly. These limitations make building general-purpose purpose and scalable systems for training agents with RL open and pressing challenge.
To address these challenges, we propose DREAMGYM, a unified and scalable RL framework that synthesizes diverse experience data in an online manner to enable efficient and effective training of LLM agents. At the core of DREAMGYM lies a scalable reasoning-basee experience model that abstracts environment dynamics into a discrete textual space. By interacting with the agent over multiple turns, it produces consistent transitions and feedback that reflect the consequences of the agent's actions through explicit reasoning. Unlike prior approaches that attempt to reproduce external systems (Chen et al. 2025| Assran et al. 2025), the design of the experience model is grounded in a key insight that agent training does not require perfectly realistic environments, but rather interaction data that is sufficiently diverse, informative, and causally grounded to acquire knowledge for the target task. Therefore, powered by strong reasoning, the experience model overcomes the key limitations outlined above and delivers useful experience data for RL training.
To ensure that synthetic experiences are diverse and informative, DREAMGYM equips the experience model with an experience replay buffer, from which it retrieves similar yet diverse trajectories to guide its current state prediction. This buffer is seeded with offline knowledge for essential context and is continuously enriched with trajectories generated on-the-fly, co-evolving the experience model with the agent to ensure the produced rollouts aligned with the agent's 's updated policy for stable training. In parallel, the experience model serves as a task generator, identifying valuable tasks with high reward entropy and producing progressively more challenging variations. This design yields effective curriculum, where agents are consistently exposed to harder problems as their capability improves. By unifying interaction, memory, and adaptive online task generation, DREAMGYM addresses the persistent challenges that have limited RL. for LLM agents training: prohibitive cost, scarcity of diverse tasks, unstable reward signals, and heavy infrastructure demands. It reframes training around an environment purpose-built for RL, enabling efficient synthetic training and effective sim-to-real transfer, improving generalization while minimizing reliance on costly real-world interactions.
Comprehensive experiments conducted to evaluate DREAMGYM across diverse environments and LLM agent backbones. For use cases lacking RL training support (e.g., WebArena (Zhou et al.) ), DREAMGYM provides the only viable approach for RL-based agent training, delivering over 30% improvement over all baselines and SOTA methods. In settings where RI. is supported but costly, DREAMGYM achieves performance on par with GRPO (Shao al.) 2024) and PPO (Schulman ct al. 2017), while training entirely within DREAMGYM without external interactions. Moreover, we introduce PREAMGYM-SSA (sim-to-real), which first trains agents in DREAMGYM using diverse, curriculum- driven synthetic experiences before transferring them to external environments. This approach yields over 40% performance improvement compared to training from scratch in real-world
2

---

Published as a conference paper at ICLR 2026
environments while using less than 10% of the external data, providing scalable warm-star strategy for purpose RL.
RELATED WORK
2.1 LLM AGENTS REINFORCEMENT LEARNING
RL offers a path to transform LLM agents from static generators into adaptive decision makers.
Classical RL algorithms such as policy gradients and actor critic methods (Williams] 1992] SSchulmnn
jt al. 2017) have achieved strong results in robotics, games, and control (Silver et al. | 2016). More
recently, RI. has been applied for post-training to improve their alignment (Bai et al. 2022) and
general reasoning ability (Liu et al. 2025). as well as to enhance performance on math and coding
problem-solving tasks (Shao et al. 2024).
However, extending RL from single-turn language modeling to interactive, multi-step agent tasks introduces substantial challenges (Zhang et al. 2025; Wang et al. 2025a). Tasks such as web navigation (Yao et al. 2022 Zhou et a..). operating-system control (Xie et al. 2024b). and multi-tool orchestration (Yao et al. 2024 Xie et al. 2024a) require long- long-horizon interaction yet provide sparse and delayed rewards (Qi et al. 2024), making policy improvement challenging and costly, and often result in training collapse. In addition, scaling RI for L.MM agents is constrained by the lack of diverse, verifiable task suites, which typically require substantial human effort to design and annotate reward logic (Zhou al. (0255). These challenges are further compounded by heterogeneous environment dynamics and engineering constraints in the real world. For example, open environments such as web pages are highly dynamic and produce noisy reward signals (Xue et al.] 2025). Meanwhile, they often lack reliable reset mechanisms and introduce safety risks during large-scale exploration (Zhou) et al.). Together, these challenges make real-environment RL impractical for general-purpose agents, motivating the need for a scalable solution that can supply adaptive, high-quality interaction data.
2.2 TRAINING AGENTS WITH SYNTHETIC DATA
Synthetic data has long been used to address the scarcity of expert demonstrations Zhang et al., (025). Early approaches relied on scripting oracle trajectories or generating them from stronger teacher models, training agents primarily through imitation learning (Yao et al. 2022 Pahuja et al. 2025
Deng et al. 2023). While effective for distillation, these static trajectories require substantial human labeling and lack diversity and adaptivity. To reduce manual effort, subsequent work (Xu et al.) Ou et al. 2024) leveraged indirect sources of expertise (e.g., online tutorials) to guide trajectory synthesis, while another line of work, including AgentSynth (Xie et al., 2025) and SCA (Zhou al., (025).
focuses on synthesizing diverse tasks to expand the space of RL exploration. However, these methods still depend on data collection in real environments, which inherently suffer from the scalability issue and are subject to the limitations outlined earlier.
Later research shifted toward building synthetic environments, such as AlphaGo (Silver al. 2016)
and Dreamer (Hafner et al. 2020 Ha & Schmidhuber] 2018). which interact with agents to generate unlimited on-policy experiences. More recently, WebDreamer (Gu et al. 2024) and WebEvolver (Fang et al.) 2025) constructed world models to produce environment feedback for agent planning and training. Closest to our setting, UI-Simulator (Wang et al.) (2025)) also leverages LLMs as a step-wise simulator, but it requires substantial expert engineering to adapt to different environments and is limited to generating only trajectory variations for supervised fine-tuning of the policy. contrast, DREAMGYM provides a complete toolkit for general-purpose RL, supporting diverse task generation and consistent rollout synthesis with rich reward signals, offering the first scalable solution for training generic LLM agents in arbitrary environments.
PRELIMINARIES
3.1 NOTATIONS
We formalize the agent learning problem as Markoy Decision Process (MDP) (Bellman 1957). defined by the tuple M = (S, R, 7, Po), where S denotes the state space and A denotes the action space. The transition function T: S x A + A(S) governs the environment dynamies, where ASS) denotes the probability simplex over S. The reward function R: S x A - IR provides feedback signals for the agent's 's actions. 7 E [0, 1] is the discount factor, and po E A(S)) specifies the initial state distribution that includes the task instruction 70-
3

---

Published as a conference paper at ICLR 2026
Task Instructions
Scalable LLM Serving Infra
actions
retrieve
Exper-
"Find out how much spant on food shopping from Jan to the end Jan 2023."
Agent
ience
Reasoning Exp- erience Model
Replay
update
Buffer
challenging task variations
CoT reasoning
Reward Entropy
ID
Curriculum Task Generator
T3
link:MMy Account" link
Done:
[1238] menuitom: Grocary
agent loop
[1474] Orders
the...
True
Policy-Aligned
115
terminated
[1606] link Pagination link to page 2...
[1670) laginaaiinnonn link page 3...
Success:
False
Task Generation
Task Value Estimation
Informative States
Reward Signal
Figure 2: Overview of the proposed DREAMGYM agent training framework. Given set of seed tasks, a reasoning-based experience model interacts with the agent to generate informative, diverse tasks and trajectories for RL training. At each step, the agent takes actions based on its current state and receives next states and reward signals derived by the experience model through CoT reasoning based on both interaction history and top-k similar experiences from active replay buffer. To expose the agent to increasingly informative scenarios, tasks with high reward entropy are proposed by the curriculum task generator for future training. With this unified design, DREAMGYM addresses both task and reward sparsity while enabling scalable RL with diverse and curriculum-dree driven environments.
In LLM agent environments, 7n is usually desired task specified by the user in natural language, and states & E S encode the environment configuration visible to the agent, such as webpage content, tool outputs, or textual environment descriptions. Actions a E AA represent discrete operations, including clicking UI elements, invoking external tools, or generating textual responses. The agent maintains policy o S ((A), parameterized by 0, which maps states distributions over actions.
3.2 AGENT LEARNING FROM EXPERIENCE
Given of online experiences where each experience {To 80, Go consists of a task To and state-action rollout {so, oo..... ax J. the goal of RL is to train an agent policy тe to maximize the expected cumulative reward, which typically optimized 0 via policy gradient as follows:
((()) (sscoe)~
V log (a; st) A(s):a))
(1)
where Á(se, az) is the advantage function, estimating how favorable an action is compared to others.
Proximal Policy Optimization (PPO). PPO (Schulman et al. 2017 popular policy gradient method that improves stability by computing Á with Generalized Advantage Estimation (GAE);
K-1
APPO
Σ (7A) [rett (V(+4++1) V(s++)))
where V() is value function approximated by LLM, and À controls the bias-variance tradeoff.
Group Relative Policy Optimization (GRPO). GRPO (Shao et al. 2024) extends PPO by discarding the value function and normalizing advantages within each group of responses & sampled for the same task instruction. Instead of GAE, the group-relative advantage is defined as:
AGRPO (r: meannsc((:))/stdeesss
where r, is the reward for output or, mean and stdieg (r:) are mean and standard deviation of rewards from group G. GRPO discards the value function and approximates the advantage using relative normalized rewards, making policy updates more scalable but potentially less sample-efficient Notably, our proposed DREAMGYM is orthogonal specific RL algorithms and focuses on scaling the synthesis of diverse, informative experiences, thereby amplifying the effectiveness of RL training.
4 SCALING AGENT LEARNING VIA EXPERIENCE SYNTHESIS
To synthesize diverse agent experiences for RL training, DREAMGYM is built around three key components: (1) scalable reasoning experience model that encodes the meta-dynamics of the
4

---

Published as a conference paper at ICLR 2026
target domain to efficiently generate informative trajectories; (2) an experience replay buffer that integrates offline enviromment knowledge with online synthetie transitions, co-evolving with the agent to stay aligned with its updated policy: (3) curriculum task generator that produces progressively challenging variations of high-value tasks selected via a reward-entropy heuristic. We elaborate each component in the following sections.
4.1 BUILDING REASONING EXPERIENCE MODELS FOR AGENT LEARNING
For effective RL training, instead of relying heterogencous external enviromments that are costly to interact with and difficult to control, DREAMGYM adopts a more adaptive and controllable approach by building L.L experience model that can efficiently interact with the agent over multiple turns to generate diverse experiences with consistent outcomes and rich feedback signals for learning.
Unlike prior data-hungry and costly approaches that build world models to replicate the real world in raw pixel spaces, we design an efficient reasoning experience model, denoted as Mexp. that operates in an abstract, meta-representational textual space The key insight is that synthesizing transitions in this abstract state space can reduce irrelevant dimensions and produce trajectories that are more informative and token-efficient than those derived from raw observations. For example, in a web shopping task, instead of processing raw HTML code, the experience model directly synthesizes clean element listings while discarding irrelevant structural artifacts such as headers and tags. This state-space design makes training the experience model highly sample-efficient, requiring only small pubic trajectory datasets in our experiments, while also enhancing the effectiveness of agent learning.
4.1.1 INFERENCE FOR EXPERIENCE ROLLOUT COLLECTION
Notably, we find that beyond the current state-action pair, three additional contexts are important for improving state quality: (1) interaction history {(s), a })] which incorporates the past trajectory in the context window to help maintain state consistency across multiple turns; (2) task instruction т, which conditions the experience model on the current goal, enabling it to interpret actions w.r.t. task objectives and thereby predict both state transitions and rewards more accurately; (3) past experiences, which are top-k demonstrations (d, 1 retrieved from the replay buffer based on semantic similarity with the state-action pair, i.c., 1 (opp (cos(((st de), ((s;, a;))), where Φ(-) denotes an arbitrary semantic encoder. Leveraging knowledge this way reduces hallucinations and improves factuality for wwwwllege-intensiiie state predictions. Therefore, given these inputs, the experience model predicts the next state 8 and reward T via chain-of-thought (CoT) (Wei al. 2022):
($1+1,7(+1) = M ((s, an) (d, (d)))=.:77
where RG is an explicit reasoning trace produced by the experience model that guides the state transition. With such reasoning, predicts the most consistent and informative transition and feedback that reflects the consequence of the agent action. For example, if the action is invalid, it transitions to failure state and assigns a zero reward to signal the error, and vice versa. In our experiments, following (Feng et al., 2025). we adopt outcome-based reward scheme, assigning r = 1 only at the final step when the task is successfully completed and r = 0 in all other cases.
4.1.2 TRAINING EXPERIENCE MODELS TO REASON
Benefiting from the abstract state-space design, training the experience model is highly sample- efficient and requires only limited data from the real environment. In practice, abundant offline trajectory datasets from public benchmarks such as the WebArena Leaderboard are sufficient for training. Our experience model distills such offline knowledge and then serves bridge to interact with the agent online for RL training.
Concretely, given a trajectory dataset D = [((,, des 8t41, 1)) each transition is annotated with an explicit reasoning trace R by LLM (prompt shown in Appendix D.II. which explains why the action a, taken in state 8, consequently leads to the next state 8t+1 and reward Tr+1 given the available contexts. To distill this knowledge, we train Mesp via SET with a joint objective over reasoning generation and next-state prediction:
LSFT = (ien -log log Po(Ri St, Oz, He, Dz) log Po(se41 St, R2 He. Di)). (5)
5

---

Published as a conference paper at ICLR 2026
where H, denotes the interaction history, Dj denotes the retrieved top-k demonstrations, and A denotes the parameters of Mexp- This objective ensures that the model (i) learns to generate faithful reasoning traces that explain the causal effect of an action, and (ii) leverages these traces to predict consistent and informative next states. By doing so, the experience model not only imitates expert trajectories but also acquires the ability generalize reasoning for novel rollouts during RI. training.
4.2 CURRICULUM-BASER TASK GENERATION
Diverse, curriculum-aligna task instructions important for RL agents to acquire knowledge (Zhou et al. 2025). However, scaling task collections is costly, as it requires significant human effort to verify the feasibility of each task in the target environment. DREAMGYM inherently alleviates this burden by adapting to arbitrary new tasks within the target domain through synthetic multi-turn transitions. Building on this capability, we propose puriicllumbbased task generation, where the same experience model actively generates new tasks as variations of a set of m. seed tasks:
Ti = Musk (f (fT(-1)=)))
mm
(6)
where Mink shares parameters with Mesp: Specifically, the seed tasks are chosen based on two criteria: (1) they are sufficiently challenging for the current agent policy, thereby maximizing information gain; (2) they are well-defined, such that unrealistic or malformed tasks can be discarded.
To satisfy both conditions, we introduce group baup-based reward entropy as a criteria for selecting high-quality and challenging tasks. Formally, for a task т, we define its value
V
- =) where F
where r¹ are the outcome rewards from n rollouts of task T within the group G. For GRPO, 9 can simply be the training group, while for PPO, tasks can be first clustered using a semantic embedder, and each cluster essentially forms a group G from which task variations can be generated. Notably, non-zero variance in G indicates that the agent observes both successes and failures on the task, signaling that the task is feasible yet challenging. A task reaches maximum entropy when successes and failures are evenly balanced in G. providing the greatest information gain for credit assignment. This observation is consistent with recent findings that LLMs learn most effectively from tasks of intermediate difficulty (Gao et al. 2025). Thus, by feeding such high-entropy tasks into Mrask, we generate progressively more challenging variations to enhance agent exploration and knowledge acquisition.
To stabilize training, we introduce a hyperparameter that bounds the proportion of synthetic tasks sampled per iteration. This preserves sufficient coverage of the original task distribution while directing exploration toward the current policy's 's weakness regions for curriculum- based improvement.
4.3 LEARNING FROM SYNTHETIC EXPERIENCES
Policy training in synthetic environments. As shown in figure 2 DREAMGYM begins with a seed task set and generates multi-tum rollouts for each task by alternating between the agent policy, which selects actions from states, and the experience model, which predicts next states conditioned on the agent action, history, and task context (as in section 4.1.I). The collected rollouts are used with standard RL algorithms (as in section 3.2p to update the policy. After each iteration, the experience model augments the task set by generating variations of challenging tasks with high reward entropy (as in section 4.2). This cycle of interaction, training, and curriculum expansion continues until convergence or predefined training budget is reached. Furthermore, we provide an analytical lower bound of the policy improvement in real environments when training with purely synthetic experiences from DREAMGYM under trust-region assumptions, as detailed in Appendix C.T
Sim-to-real policy transfer. We further extend DREAMGYM to a rialto-real (S2R) setting, where the agent policy is first trained with synthetic experiences and then transferred to RL in real environments. Pretraining in synthetic environments expands exploration coverage across diverse tasks and allows the agent to acquire broad knowledge at low cost, providing a strong initialization that makes subsequent enalronniion learning more sample-efficient (Da et al., (202).. To enable seamless transfer, we ensure consistency of the state space between synthetic and real environments by applying the same rule-based mapping function lightweight fine-tuned model (Lee et al.).

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of DREAMGYM with various agent training algorithms. We evaluate four groups: (i) offline imitation learning algorithms: SFT, DPO; (ii) online RL algorithms in real-world environments: GRPO, PPO: and (ii) DREAMGYM, where agents are trained via the same RL algorithms but with purely synthetic experiences; (iv) DREAMGYM-STR where agents are first trained with synthetic experiences and then transfer to RL in real environments. Real data indicates the number of individual transitions (a trajectory often has ~0 steps). Best performance is bolded.
Algorithm
Real
Data
WebShop
ALFWorld
Web.Area
L3.2-3B
L3.1-8
Q2,5-BB
L3.23B
L3.1-B
Q2.5-BB
L3.23B
L3.1-8
Q2.5-BB
Offine Initation Leorinng
35.1
DPO
40K
35.9
31.0
34.8
63.3
63.9
61.1
5.
4.8
4.8
GRPO
Traditional
80K
62.1
65.0
66.1
65.3
70.9
79.8
7.3
6.1
6.1
DREAMGYM
59.3
63.9
68.3
62.1
66.3
71.0
13.3
9.1
12.7
DREAMGYM-S7R
72.1
65.0
PPO
Traditional
59.9
68.1
47.0
72.9
DREAMGYM
0
60.5
$8.1
65.0
40.5
70.8
72.7
14.5
10.9
10.0
DREAMGYM-SIR
5K
66.0
63.9
73,7
49.1
73.3
79.9
13.3
10.9
13,9
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
We evaluate DREAMGYM diverse suite of agentic benchmarks and LLM backbones of varying sizes and model families to assess its generalizability and effectiveness in supporting RL for generic agent tasks while reducing costly interactions. Evaluation environments. We consider three challenging agent benchmarks that span diverse
domains, complexities, and levels of RL readiness: (1) WebShop Yao et al. (022). which requires reasoning to refine search queries and accurately identify products to complete e-commerce tasks; (2) ALFWorld (Shridhar al.). which involves multi-ture turn tool-based embodied control to navigate 3D environments; (3) WebArena-Lin (Zhou et al.), which offers realistic web interaction interfaces but is not RL-ready, as it inherently lacks scalable data collection and environment reset mechanisms while incurring high computational costs. This mixture of environments allows us to evaluate DREAMGYM both in settings where RI is feasible but computational expensive, and where RL training is not yet tractable.
Agent backbones. We instantiate agents from different model families and sizes: Llama-3.2-33
Instruct, 1---a--.--B--Instturee (Grattafiori et al. 2024), and Qwen-2.5-7B-Instrue (Team 2024.
Baselines. We consider two types of traditional traiiing strategies for agents. (1) Offline inttation learning: supervised fine-tuning (SFT), direct preference optimization (DPO) (Rafailov et al. 2023
(2) Online RL in real environments (traditional): GRPO (Shao et al.) 2024 PPO (Schulman et al.
2017
Implementation details. All main results are reported with the experience model trained from (see section 4.1. To demonstrate that DREAMGYM be applied different RL algorithms, we first evaluate both GRPO and PPO entirely within DREAMGYM, without any real interactions. We further evaluate hybrid scenario. DREAMGYM-S22 followed by a small-scale RL phase that requires only a limited number of rollouts in the original environments, demonstrating the effectiveness of using DREAMGYM as mid-training stage to improve both sample efficiency and the attainable performance after transfer. Detailed parameter settings for each scenario are provided in Appendix
where synthetic training is
5.2 MAIN RESULTS
Non-RT environment. DREAMGYM demonstrates the most significant advantage in en- vironments where large-scale RL infrastructure is not available such as WebArena (Zhou et al.). Unlike existing attempts that fail to make RL effective due to environment limitations, agents trained purely in DREAMGYM achieve success rates exceeding 30% across all backbones (Table[), whereas zero-shot RL baselines suffer from limited exploration diversity and sparse reward signals in the original environments. These results demonstrate that DREAMGYM is not merely surrogate for costly rollouts, but a mechanism that makes RL training feasible domains that were previously intractable due inherent task and engineering constraints.
7

---

Published as a conference paper at ICLR 2026
Success Paresstaraaan (5)
DraanEyma
Succee DaP 18
DywamPym-57
40
C4
20
15
DeansGym-500
SFT
WebShop
RL w/ WebShop
Webarere
Alwoorid RL w/ WebArena
27
63 130 120
Thening
10
Figure 3: (1) Left: Comparing the agent performance (success rate %) on WebArena (Zhou et al.) w.r.t. total training time across different training strategies and backbones. (2) Middle: Evaluating the cross-domain transferability of the agent policy trained via DREAMGYM with seed tasks from a different environment. (3) Right: Comparing the agent performance on WebShop (Yao al. 2022
w.r.t number of training steps across different training strategies.
RL-ready environments. On WebShop Yao et al. 2022 and ALFWorld (Shridhar et al.), DREAM- GyM-trained agents perform on par with GRPO and PPO agents trained on 80K real interactions, despite using only synthetic rollouts. The ability match strong RL baselines in RL-ready en- vironments without external interactions underscores that DREAMGYM produces transitions and rewards that are not only coherent and meaningful, but also sufficient for stable policy improvement. Furthermore, when a small-scale RL phase with an affordable number of real rollouts (5k) is applied consistently outperforms
the policy diddneraeed in the synthetic environment, DREAMGYM-SS both GRPO and PPO baselines trained from scratch in the real environments. This validates the hypothesis that synthetic training can serve an efficient warm-start strategy that establishes a strong foundation for more sample-efficient RL in the real environment.
Sample efficiency and training cost. Training efficiency further illustrated figure Left, where DREAMGYM M achieves substantial performance gains on WebArena (Zhou et al.) while reducing training effort (including both rollouts sampling time and GPU hours) to roughly one-third or even one-fifth of RL baselines in the real environment. The efficiency gain arises from both the dense feedback offered by curriculum- based rollout synthesis and the lightweight abstract state transitions produced by unified experience models hosted by scalable LLM services, which substantially reduce sampling cost and avoid heterogeneous environment bottlenecks, suggesting that DREAMGYM is not only a practical solution for RL training in complex and expensive environments, but also scalable way to generate low-cost data for stable policy improvement.
Generalization and learning transferability. The results in figure 3 Middle highlight the strong generalization and cross-domain transferability of the policy trained in DREAMGYM. Notably, (1) when trained WebShop (Yao al. 2022), the policy generalizes WehArena (Zhou et al.) and surpasses SFT models directly trained there, and (2) when trained on WebArena, it similarly transfers back to WebShop with superior performance than the SFT as well. This generalization suggests that DREAMGYM learns within an abstract meta-representation space, enabling the agent learn cammain-ggnstic behavioral priors rather than memorizing task sesciiic patterns. However, (3) when the domain gap becomes too large, such transferring from web-based environments (WebShop/WebArena)) to ALFWorld (Shridhar et al.), the performance drops significantly, indicating the limits of current meta-repreesnntaiinss
PERFORMANCE ANALYSIS AND ABLATION STUDIES
6.1 TRAINING CURVE ANALYSIS
figure Right compares training curves across Webshop (aoo et al. 2022) under different setups. Specifically, the success rate improves much more rapidly within the first 40 steps, showing that synthesized trajectories offer more informative gradients than sparse real rollouts. This further highlights the role of synthetic experiences in shaping a strong initialization that real rollouts cannot. Another observation is the reduced variance in learning dynamics. Baseline curves exhibit larger oscillations caused by sparse or unstable rewards, while DREAMGYM curves remain smoother across runs, which suggests that synthesized trajectories provide not only denser but also more consistent feedback, mitigating the training instabilities commonly reported in WebShop (Yao et al. 2022) and ALFWorld (Shridhar a
6.2 ABLATION ON TASK GENERATOR
8

---

Published as a conference paper at ICLR 2026
The curriculum-based task generator plays im- portant role in learning progress. As shown in fig- ure Right, removing this component causes agents to make some initial progress but then plateau more quickly in the WebShop (Yao et al. 2022) scenario. Similarly, Table Table[]shows that removing the task gen- erator leads 6.6% and 6.0% drop in success rate compared with the full DREAMGYM configuration
Table Average success rates (%) on the
different components of DREAMGYM.
Method
WebShop
WebArena
DREAMGYM
63.9
13.3
w/o Exp. Replay
59.2
9.7
Exp. Reasoning
55.8
7.3
w/o Task Generation
57.3
7.3
in WebShop Yao et al. 2022) and WebArena (Zhou et a respectively.
These findings support our discussion in section 4.2] without adaptive task generation, the replay buffer may saturate with low-entropy, repetitive trajectories, which limits the diversity experiences and stalls exploration. In contrast, the task generator continually produces progressively challenging, high-value tasks that push the agent beyond its current capability. This curriculum keeps the replay buffer informative and encourages exploration, yiclding higher success rates and sample efficiency.
6.3 ABLATION ON EXPERIENCE MODEL
figure demonstrates a detailed comparison of experiences generated by four variants of the experience models: traditional real environ- ment model, DREAMGYM, DREAMGYM with- access past trajectory history (w/o His- tory), and DREAMGYM without reasoning (w/o Reasoming). We evaluate each variant along four criteria: consistency, diversity, informativeness. and hallucination, using GPT-4o (Hurst al. 2024) as a judge. As detailed in Appendix|D... the judge assigns discrete scores in {0, 2 for each criteria, where higher values indicate better performance. For the first three metrics, larger scores mean more consistent, diverse, and informative; for hallucination, score of 2 means hallucination, while indicates more factual errors.
Judge Score
0.5
Consistency
Diversity
Informativeness Hallucination
Traditional
DreamGym
w/a History Reasoning
Figure 4: Evaluation of the experience model across key criteria using GPT-4o as the judge. We randomly sample 100 trajectories and prompt the model to assign discrete scores in {0. 1, 2 across four criteria, as detailed in Appendix D.4
The results highlight the role of each component. Removing trajectory history (w/o History) signifi- cantly reduces consistency: without awareness of prior turns, the model often drifts off-topic and breaks causal coherence in multi-step interactions. Removing reasoning (w/o Reasoning) mainly hurts informativeness of the states and increases hallucination: without reasoning capabilities, the generated experiences tend to become shallow and less factually grounded. Notably, removing experience reasoning also leads to a substantial drop in overall performance, as shown in Table 2 In contrast, the full DREAMGYM achieves the best near-best performance across all metrics, confirming that history and reasoning provide complementary benefits. More specifically, history preserves temporal and causal structure, while reasoning enhances depth and factual reliability. This validates that the experience model must operate in a structured, reasoning-drin driiven manner to maintain both diversity and fidelity of trajectories.
6.4 ABLATION ON EXPERIENCE MODEL BACKBONES AND OFFLINE TRAINING DATA
(a) WebShop
(b) WebArena
Notably, figure 5 investigates how the success rate of the experience model varies with both the amount of offline training data and the choice of model back- bone, evaluated on (a) WebShop and (b) WebArena.
60
Success Rate (96)
12
50
Success Rate
10
We first observe that the experience model is highly data-efficient. Even with a very limited number of of- fline samples (2k- 1kk, it already reaches competitive performance. On WebShop, for example, the Llama- 3. 1-18B exceeds 50% success rate with only 10k sam- ples, indicating that large-scale offline datasets are not strictly necessary for effective experience syn- thesis. Next, we find that smaller backbones remain viable. Although Llama-3. 3.B-BB underperforms the 8B model, it improves steadily as more data becomes
40
20
10
2k 10k 20k
40k
2k 10k 20k
40k
Dffline Samples (Steps)
# Offline Samples (Stops)
Llama-3.1-88 - Llama-3-2-38 - WebDreamer
Figure 5: Evaluation of the experience model across different number of offline training data size (transition step) and backbone.

---

Published as a conference paper at ICLR 2026
Task
Reasoning Experience Model
Agent
What is message Jack provided
in the first commit of Apr 2023?
clics TTatas action tho prsnot surowshul stote that denlaays fer commits groppd
the
which the interdled
dietat's should frossh tress commit-detal itate that rxposee withor timesamp, and commit massage the agont can extract the anower sithings
generate initial state
doee, includin one las from April 2003, which
H4.5) stsiodgee Torel Commit" 1,980
4422 card
wit let the agend further irppect 5nd... <othinks
448) pilinae Tals (Cveevien, Change ingi.
(512) buttonn/Opp "Open Change Log"
Click( 463)
(867) grouping
Clickl
(1500) pane Commi detais Thompean (1500) Apr 2023 11:30pm
Apr
(539) Lttess comnit by Jack
Thompvon ("Apr 02, 2023 11.3ppm UTC
(7143) listoon Conmint Aor
2023,
1144)
"And migration nates" 11504 meta aic2eet branch r
(163)) lystorm Commint
2023,
[1241] lartimm Comm't Apr 2023 by
changea (52 insertions 2 deletioon) [1505) new Buttur (Book fot. Copy SHA,
(1259) Vattim Commit Apr 8023, by
State
State 2
Figure 6: A case study of a trajectory sampled with DREAMGYM in WebArena. Starting from synthetic instruction, the experience model reasons over the agent's action to produce future states.
available, reaching about 55% success on WebShop with 20k samples, which suggests that lightweight models can still serve as practical experience generators when computational resources are constrained. Finally, in the extreme low-data regime, pretrained world knowledge becomes particularly valuable. On WebArena, WebDreamer (Gu et al. 2024) (a fine-tuned web world model) achieves around 13% success, initially outperforming the Llama-3. 1-8B model at smaller data scales but eventually con- verging as the number of offline samples increases. This suggests that while web-specific pretraining offers an early advantage, it is not a strict prerequisite, larger-scale experience learning can enable pre-trained models to bridge the performance gap.
6.5
CASE STUDY OF SYNTHETIC EXPERIENCES FROM DREAMGYM
figure 6 iillustrates how the reasoning experience model generates a synthetic task and progressively predicts states based on the agent's 's actions. Specifically, it predicts each state through explicit chain- of-thought reasoning that incorporates the agent's action, task instruction, and interaction history, producing next states that consistently ground the action and accurately reflect its consequences.
CONCLUSION
We introduced DREAMGYM, framework that reduces the high cost of real-envinee mell-enviromment rollouts in RL for language agents by generating scalable, sessoning-rrived synthetic experiences. DREAMGYM compresses environment dynamics into reasoning- -bsed experience model that produces state transitions and adaptive curricula, creating challenging yet solvable tasks tailored to the agent's evolving policy. Experiments across diverse environments and model backbones show consistent gains in both synthetic and sim-to-real settings. driven by the synergy of reasoning cassoning-based modeling, ruflay-buffer grounding, and curriculum generation. More broadly, our results suggest that the key bottleneck in RL for L.M agents lies in the quality and structure of interaction data. By treating environments as generators of structured, reasoning-rich experiences rather than mere simulators, DREAMGYM enables more scalable, sample-effcient, and generalizable RL. for agents.
ACKNOWLEDGEMENTS
We thank Sergey Levine, Xiaohan Fu, Canyu Chen for their valuable feedback on methodological conceptualizations, and Licheng Yu, Lizhu Zhang, Ravi Agrawal, Zhihan Liu, and Xiyao Wang for insightful discussions and project support.
This work partially supported by the National Science Foundation under grant No. 1910100, No. 2046726, NSF AI Institute ACTION No. IIS-2229876, DARPA TIAMAT No. 80321, the National Acronautics and Space Administration (NASA) under grant No. 8ONSSC20M0229, ARL Grant W91 2NF22322-111 Alfred P. Sloan Fellowship. the research grant from eBay, Al Safety Fund, Virtue AI, and Schmidt Science.
10

---

Published as a conference paper at ICLR 2026
USAGE OF LARGE LANGUAGE MODELS
The language in this paper was at times polished with the assistance of an LLM. The model was not used for research ideation, experimental design, or data analysis.
REFERENCES
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint 25f.::25000098833 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
Richard Bellman. markovian decision process. Journal of mathematics and mechanics, pp. 679-684, 1957.
Hyungjoo Chae, Sunghwan Kim, Junhee Cho, Seungone Kim, Seungjun Moon, Gyeom Hwangbo, Dongha Lim, Minjin Kim, Yconjun Hwang, Minju Gwak, et al. Web-shepherd: Advancing prms for reinforcing web agents. arXiv preprint arXiv:2505. /5277, 2025.
Delong Chen, Theo Moutakanni, Willy Chung, Yejin Bang, Ziwei Ji, Allen Bolourchi, and Pascale Fung. Planning with reasoning using vision language world model. arXiv preprint arXiv:2509.02722 2025.
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla Asri, Mahmoud Adada, et al. Textworld: learning environment for text-based games. In Workshop on Computer Games, pp. 41-75. Springer, 2015 8.
Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakar- ian, and Hua Wei. A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models. arXiv preprint arXiv:2502.131877 2025.
Xiang Deng, Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-281144 2023.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06077 2018
Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu. Webevolver: Enhancing web agent self-improvement with coevolving world model. arXiv preprint arXiv:2504.21024, 2025.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for I1m agent training. arXiv preprint rxXi:22505.1097, 2025.
Zhaolin Gao, Joongwon Kim, Wen Sun, Thorsten Joachims, Sid Wang, Richard Yuanzhe Pang, and Liang Tan. Prompt curriculum learning for efficient Ilm post-training. arXiv preprint arXiv:2510.01 135, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Yu Gu, Kai Zhang, Yuting Ning. Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. Is your Ilm secretly a world model of the internet? model- based planning for web agents. arXiv preprint arXiv:2411.06533 2024.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:/803 /0/22, 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
11

---

Published as a conference paper at ICLR 2026
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, et al. Verltool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint urXiv:2509.01055 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
Dongjun Lee, Juyong Lee, Kyuyoung Kim, Jihoon Tack, Jinwoo Shin, Yee Whye Teh, and Kimin Lee. Learning to contextualize web pages for enhanced decision making by Ilm agents. In The Thirteenth International Conference on Learning Representations.
Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. Spice: Self-play in corpus environments improves reasoning. arXiv preprint arXiv:2510224684, 2025.
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long, Yifan Xu, Xixuan Song. Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint (XXXv:2448.06632277 2024.
Sami Marreed, Alon Oved, Avi Yaeli, Segev Shlomov, Ido Levy, Offer Akrabi, Aviad Sela, Asaf Adi, and Nir Mashkif. Towards enterprise-ready computer using generalist agent. arXiv preprint arXiv:2503.01 06111 2025.
Tianyue Ou, Frank Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale. Advances in Neural Information Processing Systems, 37: 91618 91652, 2024.
Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Awadallah. Explorer: Scaling exploration-driven web trajectory synthesis for multi- modal web agents. In Findings of ACL,, 2025.
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph Gonzalez. The berkeley function calling leaderboard (bfel): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning.
Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training Ilm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-83 53741,, 2023.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.066777 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033.2 2024.
Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. Scribeagent: Towards specialized web agents using production-scat workflow data. arXiv preprint 2rXiv:2411./5004 2024.
12

---

Published as a conference paper at ICLR 2026
Mohit Shridhar, Xingdi Yuan, Alex-nle Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference Learning Representations.
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 1, 2025.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dicleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap. Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö Ank. Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501.10893, 2025.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https::/tp/ett github.. htt.bu...o//lo///eeeaeaae 2otpbl//ll///ee.5//
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186355, 2024.
Xiyao Wang, Chunyuan Li, Jianwei Yang, Kai Zhang, Bo Liu, Tianyi Xiong, and Furong Huang. Llava-critie-rl: Your critic model is secretly strong policy model. arXiv preprint arXiv:2509. 200.22509.00666, 2025a.
Yiming Wang, Da Yin, Yuedong Cui, Ruichen Zheng, Zhiqian Li, Zongyu Lin, Di Wu, Xueqing Wu, Chenchen Ye, Yu Zhou, et Lims scalable, general- purpose simulators for evolving digital agent training. arXiv preprint arXiv:2510. 14969, 2025b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, et al. Webagent-rl: Training web agents via end-to-end multi-turn reinforcement learning. arXiv preprint 2xxv:22555.1642 2025.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 256, 1992.
Jian Xie, Kai Zhang, Jiangjic Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: a benchmark for real-world planning with language agents. Proceedings of the 41st International Conference Machine Learning, pp. 54590-54613, 2024a.
Jingxu Xie, Dylan Xu, Xuandong Zhao, and Dawn Song. Agentsynth: Scalable task generation for generalist computer-use agents. arXiv preprint arXiv:2506. /4205, 2025.
Tianbao Xic. Danyang Zhang, Jixuan Chen, Xiaochuan Li. Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-52094, 2024b.
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong. and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. In The Thirteenth International Conference on Learning Representations.
Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion progress? assessing the current state of web agents. arXiv preprint Xiv:2504.01.882 2025.
13

---

Published as a conference paper at ICLR 2026
Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentoccam: A simple yet strong baseline for llm-baser web agents. arXiv preprint arXiv:2410.13825, 2024.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains, arXiv preprint arXiv:2406. 12045, 2024.
Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang. Yuting Ning. Zhaorun Chen, Xiaohan Fu, et al. Agent learning via early experience. arXiv preprint arXiv:2510.08558, 2025.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. The Twelfth International Conference on Learning Representations.
Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar. Self-challenging language model agents. arXiv preprint www.ii:2006001166 2025.
14

---

Published as a conference paper at ICLR 2026
A
APPENDIX
B DETAILED EXPERIMENT SETTINGS
In this section, we provide implementation details for each environment.
B. WFBSHOP
WebShop (ao et al., 2022) is large-scale agent benchmark designed to study language grounding in interactive environments. It simulates realistic e-commerce website with 1.18M real-world products and 12,087 crowd-sourced natural language instructions, where agents must search, customize, and purchase items. The environment poses challenges such as interpreting compositional product requirements, reformulating queries, handling noisy webpage text, and strategically exploring diverse page types.
We follow the standard setup and hyperparameter settings from Verl-Agent (Feng et al. 2025) and perform full full-parameter fine-tuning for all three agent backbones in our experiments.
RL Baseline Setup.
DREAMGYM Settings. To train the reasoning experience model, construct dataset by combining 1,600 human demonstration trajectories from the official WebShop repository with an additional 2,000 trajectories collected using an oracle agent and random exploration. We fix the test evaluation stability and collect training trajectories only from the remaining tasks to avoid test-set contamination. Each transition is further augmented with reasoning trace generated by strong teacher LLM, resulting in the dataset used for fine-tuning.
ensure
Computation Resources. All experiments, including both baselines and ours, are conducted on nodes with A100 GPUs and 4 nodes with H100 GPUs.
B.2 ALFWORLD
ALFWorld (Shridhar et al.) is a -ox--and-emoddee benchmark with hand-crafted task instructions designed for studying language grounding and cross-modal transfer. It pairs abstract text interactions from Tex(World with photo-realistic, physics-based execution in ALFRED/AI2-THON six household task families (e.g.. Pick & Place, Clean & Place, Heat/Cool Place) with 3,553 training tasks and seen/unseen splits across 120 rooms. Agents issue high-level textual actions (goto, open, take, clean/heat/cool, put) that must be realized as low-level visuomotor controllers, facing challenges such as partial observability, object search and manipulation, mapping language action preconditions and affordances, and bridging the gap between abstract plans and physical feasibility.
spanning
RL Baseline Setup. We adopt the standard setup and hyperparameter settings from Verl-Agent (Feng ct al. 2025) and perform full-parameter fine-tuning for all three agent backbones.
DREAMGYM Settings. We follow the default ALFWorld split (Shridhar et al.) with the TextWorld setup (Côté et al., 2018) under the Verl-Agent framework. From the training split, we extract 3,200 expert demonstration trajectories paired with task instructions, and additionally sample 2,000 offline trajectories using both oracle and random policies. These datasets form the basis for training the reasoning experience model. Each transition is further augmented with reasoning trace generated by a powerful L.M,, which used for fine-tuning.
Computation Resources. All experiments, including both baselines and ours, conducted on nodes with A100 GPUs and 4 nodes with H100 GPUs.
B.3 WEBARENA
WebArena Zhou et al.) is a self-hosted, realistic web environment for training and evaluating autonomous agents across fully functional sites such e-commerce, social forums, collaborative software development (GitLab), and content management, which are augmented with tools (map, calculator, scratchpad) and knowledge bases (e.g., offline Wikipedia, manuals). It provides 812 long-horizon tasks expressed as high-level natural language intents and evaluates agents by functional correctness rather than matching action traces, supporting multi-tab browsing and rich action space (click, type, navigate, tab operations).
Training Set Split. Since the full evaluation set WebArena is large and contains many similar tasks, we follow prior work (Qi et al. 2024; Wei et al.] 2025) and evaluate agents on WebArena- Lite (Liu et al. 2024), a more balanced subset of 165 high-quality, challenging tasks
15

---

Published as a conference paper at ICLR 2026
selected from the original 812. The remaining 647 tasks, excluding those in the evaluation set, used for training.
RL Baseline Setup. Since reliable open-source RL infrastructure exists for WebArena (Qi et al. 2024), we follow the standard workflow in Verl-Agent (Feng et al. 2025) and implement a vanilla RL pipeline for WebArena. The baseline collects action trajectories via browse r (Zhou al.) while hosting the WebArena websites on AWS servers, supporting both PPO and GRPO. Despite extensive engineering effort, we able to operate only four AWS servers, enabling most four parallel interaction sessions, which in turn imposes significant bottleneck on training throughput.
During RL sampling, we sequentially sweep through the task set and manually restart the servers and reset the environments once all tasks have been visited once, in order to avoid cross-task interference. Nevertheless, we observe that some trajectories fail execute properly and that certain tasks are incorrectly judged by WebArena's original evaluation function, a known issue also reported in prior work (Qi et al. 2024 Chae al. 2025). For fairness, we retain all collected trajectories and use them as-is for RL training.
DREAM MGYM Settings. obtain offline trajectories for training the reasoning experience model, we extract successful demonstrations from the highest-performing agents on the public WebArena leaderboard. Specifically, we use agents that incorporate accessibility tree information in their observations, including IBM CUGA (Marreed et al. 2025). SeribeAgent (Shen al. 2024). Learn- by-Interact (Su et al. 2025). and AgentOccam (Yang et al. 2024p. To mitigate distribution imbalance, additionally collect trajectories generated by high-performing agent and by random policy, In total, we obtain 4,800 offline trajectories for training. Each transition then augmented with reasoning trace generated by a powerful LLM, forming the dataset for fine-tuning the reasoning experience model.
Computation Resources. All experiments, including both baselines and ours, conducted nodes with 1.00 GPUs and 4 nodes with H100 GPUs.
16

---

Published as a conference paper at ICLR 2026
C THEORETICAL ANALYSIS
In this section, we analyze how policies trained in the synthetic environments of DREAMGYM can provably improve performance in real environments. We show that, under mild assumptions, performance guarantees can be established by optimizing learning-centrr signals of the experience model, such as reward accuracy and domain consistency, rather than strict fidelity metrics like state reconstruction error.
C.1 PROVABLY POLICY IMPROVEMENT IN REAL ENVIRONMENTS TRAINED WITH SYNTHETIC EXPERIENCES
DREAMGYM trains LLM agents using reasoning- baased experience model Mexp, which interacts with the agent and induces a synthetic MDP M. For brevity. we use M to denote any such synthetic environment, including Mexp, which is defined in the abstract textual state space, as stated in section 4.T] The learned policy is then evaluated the real environment M. projected into the same abstract space for comparison. We show that, under standard trust-region policy update assumptions, policy optimized in M guaranteed also achieve policy improvement the real environment M.
Theorem 1 (Policy Improvement J in Real Environment via Synthetic Experiences). Let the real MDP be M (S,A, P. R, the synthetic MDP induced by Mexp be M = (S, P, R 7), discount be E (0, 1), and let rewards be bounded R. R E [0, Rmax] with Vmax == Rmaxx (1 - 7). Assume one-step experience- aoperelnce-model ervors
ER = sup R(s, a) - a)). s,a
EP = sup aV) P(1ss,)), s,a
(8)
and a trust-region update π - nt obtained by optimizing in M with per-state KL radius sup, (T (-|s) (+1s)) s 5, as enforced by the soft KL penalty in PPO and GRPO. Hence
47 (1 7)- Vinas 8 2 (1 1-7
27 Buuxx
JM((') JM(T)= 2 Bandi ond I
+ (1 7)3
synthetic surengate gain in Maxp
inust srustregiion penalty
experience model
In particular, if the synthetic surrogate gain exceeds the two penalties, then
(T).
Specifically, (1) the synthetic surrogate gain denotes the agent's 's performance improvement when trained and evaluated within the synthetic environment provided by the experience model Mexp- (2) The trust-region penalty corresponds to the KI. radius ō constraint, which is softly enforced by PPO or GRPO. (3) The experience-mode error measures how well MMxp preserves learning-relae relevvnnt signals of the original environment for agent knowledge acquisition including two key components: (a) the faithfulness of feedback (Ex), i.e., how accurately reward signals reflect real outcomes, and (b) the domain consistency of state transitions (EP), i.e., how well state space distributions align with the dynamics from the original environment.
Notably, these two error terms align with our design insights in section 4.1 the synthetic envi- ronment need only provide domain-consistent transitions and correct, retrospective learning signals, without having to clone the original environment at the raw state level. In practice, both ER and & can be made very small even when Mexp is trained with minimal trajectory data annotated with explicit reasoning traces.
Proof of Theorem 7 We first decompose the policy improvement in the real environment through the synthetic environment:
JM(=') JM(()= (Jm(n') (x)()) + (JM(=') - J (m))) - (10)
By Lemma each of the two policy discrepancy terms JM(() - (z((-) is at most A model. hence
JM((T) JM((T) 2 -
-24 model.
(11)
remains to lower bound improvement inside the synthetic environment. Using the standard trust- region bound (Schulman et al. 2015), which is enforced in practice by PPO and GRPO via a per-state KL radius 5, we have
1
47 (1 71- Vmax 8.
J (M('') (T)
>
1 Bxndy Se: and (11)
(A 1
(3, [s,4]]]
(12)
17

---

Published as a conference paper at ICLR 2026
Combining these two terms yields the inequality in Theorem 1 which completes the proof.
Lemma (Multi-turn experience synthesis error bound). For any policy n. if
En = sup R(s, a) - R(ss,))l
Ep sup ()) P(110,,))))
(13)
8,0
3,4
then
ER + 2 RyRax
- 1
model
=
1 1-7 (1- Ep.
(14)
Proof. We first compare the Bellman operators of the real and synthetic environments. For any bounded value function V,
(T. Vs)) = BE 11
(15)
and let TH be the same expression with (R, replaced by (R, P). Thus for any bounded value function V, the operator difference is bounded as
TVV - T-V I sup R(s. a) R(s, a)) + + E. P() - B V(s')
8,0
8,a
(16)
SR + VIL
(17)
which is derived by simply using the definitions of en, En and the variational characterization of TV..
Now apply this bound to V Vý and add subtract:
VV
1 | π V
V Ix
(18)
$ V - T [T, T VL
lo
(19)
< R + 2> mmxx vllV Vo
(20)
By rearranging the contraction term into the left side, we have
(1 - 1) Vo SER + m
(21)
Hence
| M I ( + 2 2yVmmxx
(22)
Finally, since JE(T) Eag~a [V (80)], we obtain
[JM(T) - - [m)]
= Eso~yl (M) (80)
(23)
< |
V
(24)
M
ER + 27
Ep
(25)
1- 7
(1 7)2
A model.
(26)
This indicates that the gap of agent performance between real and synthetic environments depends only reward accuracy and domain consistency errors, rather than on strict fidelity metrics such as state reconstruction error, etc.
18

---

Published as a conference paper at ICLR 2026
D
DREAMGYM PROMPTS
D.1 WEBSHOP
Experience model reasoning step annotation
WebShop
System Prompt:
You are an expert web navigation and e-commerce environments, special- izing in providing actionable guidance for state transitions of an experience model that simulates the environment dynamics.
User Prompt:
You are synthesizing environment state transition plans for training world models in webshopping tasks. You provided with task instruction, a flag indicating whether the trajectory is successful, and a tra jratecr {(s, a,)),= of the environment state and the corresponding agent action at each step.
Task Context:
Task: (instruction) Success: (flag)
Trajectory Steps:
".ioin (eiin([Sttii ["Step: (i), Environment State: (ssh,
Action:
Your Task:
Task Tutorial: A high-level guidance of how the environment should transit step-by-step interact with the agent under the given task instruction. It should highlight the critical steps that the agent should perform in order for the environment transit the final successful state. State Transition Plans: For each step, first analyze whether the agent's 's action likely to success or fail based on the task tutorial (e.g. the search query is too vague or too specific, or the agent clicks the wrong product), and then provide a concise reasoning trace describing how the environment should transition given the current state and action.
CRITICAL: You MUST generate exactly one transition plan for each environment step provided and your state_trans ons array must contain exactly len (env_stepidd) entries, one for each step_id For product listing pages, the state transition plan should mention some actionable details such as the number of products shown this page, whether this page should contain the target product given the agent's 's action. Focus actionable guidance for training the experience model. Keep responses concise and practical.
Response Format:
json { "task tutorial":
("Overall Plan": "A one-sentence high-level guidance of how the environ- ment should transit step-by-step to interact with the agent under the given task instruction.".
"Success Mode": "Describe the critical steps that the agent should perform to succeed in the task, where the environment should correspondingly tran- sit to the successful state. Summarize in one sentence.",
"Failure Mode": "Describe the typical failure mode the agent should avoid, where the environment should correspondingly transit to the failed state once the agent performs the action. Summarize in one sentence." ) "state_ transitions": [[
"step_id": O,
"transition_plan" "Analyze whether the agent's action is good bad based on the next state and overall task tutorial, and corresponding plan for how environment should respond this action.")
]]
19

---

Published as a conference paper at ICLR 2026
Task variation dataset construction
WebShop
System Prompt:
You are an expert e-commerce task design and Al training data curation.
User prompt:
You are an expert in e-commerce task design. will give you an orig- inal web shopping task inst and several candidate variat ions of this task. Your job to select the most challenging yet feasible variation that would be good to train an Al agent to acquire the skills of shopping for the given product. Original Task: (task instruction) Product Information:
Category: (product_infee category') }
2. Product Name: (product_img) [' name" ]]
Available Attributes:
(','- jojnn (product_infol attributess]
Candidate Cariations: variations)
Criteria for selection:
Challenging but Feasible: The task should be specific complex than the original, but still achievable, so as to strengthen the agent's capabilities for shopping for the given product.
High Quality: The instruction should be clear, grammatically correct, and realistic.
Meaningful Variation: The changes should make the task meaningfully different (not just trivial changes). Realistic: The combination of attributes, options, and price should make sense for the product category.
Please respond with:
1. The number of your selected variation 1- (len (candidate var at ))ns))..
2. A brief explanation (1-2 sentences) of why this variation the most challenging and high-quality.
Format your response as:
SELECTION: [number]
REASONING: [explanation]
Agent Prompt Template
WebShop
You are an expert autonomous agent operating in the WebShop e-commerce environment. Your task is to: [task_deee descriptionn)
Prior to this step, you have already taken (step_count) step(s). Below are the most observations and the corresponding actions you took:
recent
(action (action_histons history)
You are now at step (current_step) and your current observation is: courrent_observatt Your admissible actions of the current situation are: davailable_actions
Now it's your turn to take one action for the current step. You should first reason step-by step-by-step about the current situation. then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags.
Once you't 'e finished your reasoning, you should choose admissible action for current step and present within <action> </action> tags.
20

---

Published as a conference paper at ICLR 2026
D.2 ALFWORLD
Task variation dataset construction ALFWorld
System Prompt:
You are an expert in embodied task design and AI training data curation for interactive embodied environments.
User prompt:
You are an expert in embodied task design. will give you feasible task instruct inn for an embodied agent and several candi e variat ions of this task. Your job to select the most challenging yet feasible variation that would be good to train an Al agent to acquire generalizable embodied reasoning skills. Original Task: (task instruction) Environment Context:
Room Type: fenv_info
info [' room' ]]
2. Objects Present: 1', join _nnf [' ob jects') ]))
Containers/Surfaes::
,' (env_infol locations') ]
Candidate Variations: (candidate variations]
Criteria for selection:
Challenging but Feasible:
The variation should add complexity
(e.g.. more objects, constraints, or multi-step actions) without being impossible.
High Quality: Clear, grammatical, and realistic in the ALFWorld
context.
Should involve non-trivial differences in action type, target object, or target location.
Meaningful Variation:
Realistic: The variation must be consistent with ALFWorld's
embodied environment dynamics (e.g., no placing a fridge on lamp).
Please respond with:
1. The number of your selected variation 1-(len (candidate var at (ons)).
2. A brief explanation (1-2 sentences) of why this variation the most challenging and high-quality.
Format your response as:
SELECTION: [number]
REASONING: [explanation]
Agent Prompt Template ALFWorld
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: (task_descriptioon)
Prior to this step. you have already taken [step_count] step(s). Below are the most observations and the corresponding actions you took:
recent (history_lengeee (action ho..oo_tsstoy...... You are at step (current_step) and your current observation is: wwtreen__bsseraatte
Your admissible actions of the current situation are: [[admissible_dd www.ssssceleeiiossssseeeeeesssse
Now it's your turn to take action. You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think tags. Once you'te 've finished your reasoning, you should choose an admissible action for current step and present within <act ion </act inn> tags.
21

---

Published as a conference paper at ICLR 2026
D.3 WEBARENA
Task variation dataset construction WebArena
System Prompt:
You are an expert in designing realistic, diverse, and challenging web interaction tasks for AI agents.
User prompt:
will provide you with several seed WebArena task instructions
Your job is to generate new task variations from each seed. The variations should keep the same general action type (e.g., search, filter, upvote, navigate, purchase, delete) but differ in target, constraints, or context, making them realistic, challenging, and meaningfully different.
Seed Instructions: [list of seed instructions)
Requirements for variations:
Action Consistency: Preserve the same type of action as the seed
task.
Change the entities, filters, domains, time ranges, or constraints so the new task is distinct but natural.
Meaningful Differences:
Challenging but Feasible: The variation should slightly increase reasoning or constraint complexity, but remain solvable.
High Quality: Grammatically correct, clear, and realistic web tasks.
Please respond with:
For each seed instruction, generate [K] new task variations. Format your response as:
SEED: [original instruction]
VARIATIONS:
1. [variation 1]
2. [variation 2]
Example:
SEED: List products from living room furniture category by descending
price.
VARIATIONS:
1. List products from bedroom furniture category by ascending price.
2. Show me the most expensive three dining tables available online.
3. Find discounted sofas under $500 the living room furniture category.
22

---

Published as a conference paper at ICLR 2026
High-quality task variation selection WebArena
System Prompt:
You are an expert in interactive web task design and Al training data curation.
User prompt:
You expert in web environment task design. will give you an original WebArena task instruction and several candidate variations of this task. Your job is to select the most challenging yet feasible variation that would be good to train an Al agent to acquire generalizable skills in web interaction.
Original Task: (task instruction)
Candidate Variations: fcandidate variations]
Criteria for selection:
The variation should require slightly more reasoning, precision, constraints than the original, but still be solvable by a web agent.
Challenging but Feasible:
High Quality: Clear, grammatical, and realistic within the web environment.
Keep the same action type (e.g., search, navigate, sort, submit, upvote, purchase) as the original, but change the context, target, or condition.
Meaningful Variation:
Realistic: The task should reflect plausible web interactions a user might request.
Please respond with:
1. The number of your selected variation 1-(len ( (candidate vari at i ooss) ).
2. A brief explanation (1-2 sentences) of why this variation is the most challenging and high-quality.
Format your response as:
SELECTION: [number]
REASONING: [explanation]
23

---

Published as a conference paper at ICLR 2026
AX-tree state mapping prompt WebArena
System Prompt:
You are an agent tasked with extracting and refine a subset of the webpage's observations based on the content of the page and user instructions. Perform the following tasks based on the provided [Information souree], including user instructions, interaction history, and the Arer observation at the current time step. First, provide high-level reasoning for the next action by analyzing the provided information. Second, extract relevant wehpage elements based on your high-level reasoning.
User prompt:
[General instructions]
You are currently the [domain info) website. Your task is to generate Reasoning and Refined observation based the provided inputs.
First, review the User instruction and History of interactions and, then, generate the Reasoning, Analyze the progress made so far, and provide a rationale for the next steps needed to efficiently accomplish the user instruction on the [doma Second, refine the Webpage observation at the current time step into Refined observation. Extract subset of the webpage observation (e.g.. chart, table, that contains necessary information for completing user instruction, and explain the extracted elements. Ensure that the information on the elements (e.g.. numeric element ID) is correctly included. Please follow the format in the [Reasoning & Refinement example] carefully.
website.
items)
[Information source]
User instruction: /user instruction]
History of interactions: interaction histcry) Webpage observation at the current time (AXTree observation)
|Reasoning & Retinement example]
Abstract example
Here is abstract version of the answer, describing the content of each tag. Make sure you follow this structure and format strictly, but replace the content with your own <reasoni ing>
Think step by step. Based on the User instruction, History of interaction, and AXTree observation at the current time step:
Provide a high-level description of the AXTreee observation at the current time step.
Based the User instruction and History of interaction track your progress and provide your
reasoning on the next action needed to accomplish the User instruction
Ensure that: Structure your reasoning concisely follow following format strictly: scontent_descrile ion> High-level description of current page state (max 2 sen- (ences)</conte cont wwwpeeeeettttttteeeeeeeeeeeepeee ion> pevent_progresss What has been accom- plished so far (max sentence) /agent _tcrrett_poresss_____e What should happen and why (max sentence)< http://eettttttttnnnnee lysis </reasonin ng>>
ract ion>
Based on your reasoning. identify the elements (e.g.. buttons, text fields, static text, table row, chart) to focus Then, explain the semantics and functionalities of each extracted element. Ensure that: You not alter the structure the AXTree observation. You extract the element ID (id in accurately without any errors. When extracting chart or table, you must extract the entire chart or table to avoid any confusion or loss of information. Unless necessary, try not extract url non-semantic identifiers which is not informative for agent actions. the elements you extract should actionable and discard irrelevant elements. Please follow the following format and do not provide any other text besides the element list.
[ELEMENT ID] TYPEEDESCRIPTION
[ELEMENT ID) DYPEDDESCRIPTION
[ELEMENT IDJ DYSERDESCNNPTION
(Extract 3-10 most relevant actionable elements only) </extraction>
24

---

Published as a conference paper at ICLR 2026
Agent Prompt Template WebArena
System Prompt:
You are an agent trying to solve a web task based on the content the page anda user instructions. You can interact with the page and explore. Each time you submit an action it will be sent to the browser and you will receive
page.
User prompt:
Instructions
Review the current state the page and all other information to find the best possible next action accomplish your goal. Your answer will be interpreted and executed by a program, make sure to follow the formatting instructions.
User instruction: (user instruction)
History interactions: (int:nnnttiiitneniiiiiiio history) Refined observation of current step: Reasoning (plan] Focused AXTree observation:
Action space: 13 different types of actions are available.
(wait ms: float= 1000)
1. Description: Do nothing, and optionally wait for the given time (in milliseconds).
2. Examples: noopO.noopp5500)
To save space, please refer to D.3.1 for the full list of actions.
Remark: Only a single action can be provided at once. Exam- fle:fill a12', 'example with "quotes">) actions are meant to be executed sequentially without any feedback from the page. Don't execute multiple actions if you need feedback from the page.
Multiple
Abstract Example
Here is an abstract version of the answer with description of the content of each tag. Make sure you follow this structure, but replace the content with your answer:
<think>
Think step by step. you need to make calculations such as coordinates, write them here. Describe the effect that your previous action had the current content of the page.
</think>
<act ion>
One single action be executed. You only action time.
</action ion
Concrete Example
Here is concrete example how format your answer. Make sure follow the template with proper tags:
<think>
My memory says that I filled the first name and last name, but I can't 't see any content in the form. I need to explore different ways to fill the form. Perhaps the form is not visible yet or some fields are disabled. I need to replan. </think>
<action>
fill (' ('al2', a12', eeammle with "quotes"
</action>
25

---

Published as a conference paper at ICLR 2026
D.3.1 ACTION SPACE OF WEBARENA
noop (wait_ms: ms: float 1000)
Description: Do nothing, and optionally wait given time milliseconds). 2. Examples: noop ;) noop (500)
send msg_to, user (text: str)
1. Description: Send a message to the user. You should send short answer as message and do not
ask questions through message.
2. Examples: send msg to user ('the city was built in 1751.');
send msg to user (" Yes') ; send msg_ to us
send_msg_to_user (" 311122); send_msg_to_user (' Yoshua Bengio') scroll (delta_x: float, delta_y: float)
Description: Scroll horizontally and vertically. Amounts pixels, positive for right or down scrolling, negative for left or up scrolling. Dispatches wheel event.
2. Examples: scroll (C, 200); scroll (-50.2, -100.5)
fill (bid: str, value: str)
1. Description: Fill out form field. focuses the element and triggers input with the
entered text. works <input?, <textarea> econtenteditable elements.
2. Examples: fill (" 237', example value'); fil1(" 'multi-line example') ) fi11 (2a12', 'exampie with "quotes"))
select_option (bid: str, options: str list[str]]
multiple options in <select> element. You can specify option value or label to select. Multiple options can be selected.
1. Description: Select
2. Examples: select_option 48', 'blue'); select_cption] 4'48', [' r'red',
green' "blue'1)
click (bid:
button: Literal['left' 'middle', 'right'] 'left', 'Control', 'Meta', 'Shift']]
modifiers:
(1)
1. Description: Click an element.
2. Examples: click ('51'); click ('b22', button='right") click ('488, ("
button=' ddle', midee',,
dbiclick (bid: str, button: Literall' left', 'middle', 'right'
Left', modifiers: list heteltyyin..ittrall Literal''laa,, 'Control', "Meta',
Shift' (1)
1. Description: Double click element.
2. Examples: dblclick (' 12 dblclick ( ca42', button='right) dblclick[ (' 178', button middle', modifiers- shift'])
hover (bid: str)
1. Description: Hover over element.
2. Examples: hover ("b"')
press (bid: str, key_ _ey_c str)
1. Description: Focus the matching clement and press a combination of keys. It accepts the logical
key names that are emitted in the keyboardEvent. key property of the keyboard events:
Backquote, Minus, Equal, Backslash, Backspace, Tab, Delete, Escape, ArrowDown, End, Enter, Home, Insert, Pagelown, PageUp, ArrowRi ght, Arrowup, P1 F12, DigitO Digit9, KeyA Key%, etc. can alternatively specify a single character you'd like to produce such as "a" or "#" Following modification shortcuts are also supported: Shift, Control, Alt, Meta.
2. Examples: press 8'', "Backspace") press a26", 'Controlta')
press ('161', 'Meta+Shifttt") focus (bid: str)
1. Description: Focus the matching clement.
2. Examples: focus (' (4b5')"
clear (bid: str)
1. Description: Clear the input field.
2. Examples:clear 996')
drag and and_drop (from_bid: str, to_bid:
str)
1. Description: Perform drag drop. Hover the element that will be dragged. Press left mouse
button. Move mouse to element that will receive the drop. Release left mouse button.
2. Examples: drag_and_drom
56', '498')
upload_fit file (bid: str, file: str list [str])
1. Description: Click element and wait for "filechooser" event, then select multiple input
files for upload. Relative file paths resolved relative the current working directory. An empty list
clears selected files.
peceipt.pcie upload file 63', /home/bool [' /hooo////t /hobl//ob// / Documents/ / image. ipg' '26' hhome/bob/Document Document s/ fwl..ll.22. zip'])
2. Examples: upload file 572',

---

Published as a conference paper at ICLR 2026
D.4 EXPERIENCE MODEL JUDGE
Judge Prompt for Evaluating Experience Models
You are an expert judge for scoring the quality of predicted state transition WehShop environment simulator.
You are given:
Current state (before the action)
-The agent action
Predicted next state (after the action)
Your task:
1) Evaluate the predicted next state four rubrics, each scored 1, 2.
2) Provide brief step-by-step reasoning each rubric.
3) Output valid JSON object with the rubric scores and the total (sum of the four rubrics). Do not include extra fields.
General rules:
Base your judgment only on provided inputs; do not assume hidden context.
Use integers only (0/1/2) for rubric scores.
If an action is invalid or should not change the page, correct behavior may include no-op with explicit failure/empty-reete resullt signal.
Be concise specific your reasoning (1-3 sentences per rubric).
# # Rubrics (0/1 (0/1/) with anchors:
1) Causal State Consistency Question: Is the predicted next state both logically consistent with the prior state and causally grounded in the agent's action semantics (c.g., click + detail page, pagination new results, scarch updated listings, back prior view)?
2: Coherent and action-approoriate: all expected updates appear with no contradictions.
-1: Mostly consistent, but has minor logical semantic gaps.
Inconsistent
causally linked the action.
2) Diversity State Variation Question: Is there a meaningful, non-degenerate change from the prior state (when change is expected)?
Substantive, coherent differences (new results, updated filters, changed details).
1: Minimal or superficial change.
O: No meaningful change, or incoherent jump.
3) Informativeness Question: Is the predicted state rich, relevant, internally coherent (e.g.. listings with meaningful attributes; filters aligned with content)?
2: Detailed, relevant, and coherent information.
Some useful details, but sparse partially incoherent.
0: Uninformative, irrelevant, or incoherent.
4) Hallucination & Failure Feedback Question:
When the action is invalid or yields results, does the state reflect appropriate failure/empty-result signal instead of hallucinating success?
2: Correctly signals failure or success as appropriate, no hallucination.
1: Partial/ambiguuus handling of failure.
-0: Hallucinates success ignores failure.
### st-tep Evaluation (use this structure):
1. Causal State Consistency: <your reasoning> Score: 0/1/2
Diversity State Variation: <your reasoning> Score: 3/1/2
3. Informativeness: <your reasoning> Score: 0/1/2
4. Hallucination & Failure Feedhack: <your reasonings Score: 0/1 /2
# # # Final ISON Output:
Output single valid JSON object. Replace angle brackets with integers only.
1 "rubric__r ""rubric_scores" scores":
"causal_cooe "oauuaal_consssteccyy" <0 2>, "diversity": <0/112>,
" Ainformativeness" <01112>, "hallucination"; <01112> 1]
27


Reviews:
[{"review_id": "wEfQIW5eZi", "reviewer": "Reviewer_D1t6", "summary": "This paper tackles the significant challenges of applying reinforcement learning RL to LLM agents, namely the high cost of environment interaction, unreliable reward signals, and the difficulty of scaling task diversity . The authors propose DreamGYM, a framework that synthesizes agent experiences to enable effective and scalable online RL. Instead of interacting with a costly real environment, the agent interacts with a \"reasoning-based experience model\". This model, an LLM trained on offline trajectories annotated with CoT reasoning , learns to predict state transitions and feedback signals in an abstract textual space. The framework has two other key components: an experience replay buffer (seeded with offline data and enriched with new interactions) to stabilize training, and an adaptive curriculum task generator that uses a \"reward entropy\" heuristic to create progressively more challenging tasks for the agent. The experiment show that the purely synthetic approach is competitive with baselines trained on 80K real interactions. On the \"non-RL-ready\" WebArena benchmark, DreamGYM enables RL training and achieves a relative improvement of over 150% compared to baselines. The sim-to-real approach demonstrates significant performance gains and sample efficiency.", "questions": "Please refer to the weakness part above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper targets an important bottleneck in the field of autonomous agents. The prohibitive cost and low sample efficiency of online RL in complex, real-world environments (like web browsers) is a major barrier to progress. The paper's goal of creating a scalable, synthetic training environment is highly relevant and impactful. \n\nThe framework's ability to enable effective RL on WebArena, an environment considered \"not RL-ready\", is the most compelling result. Achieving success rates over 30-45% where previous methods (including traditional RL) struggle to get past 10-19%  is a significant practical achievement.", "weaknesses": "The central idea is to use an LLM as a learned world model for model-based RL. The paper cites related work like Dreamer and other LLM-based environment models but claims to be different by focusing on \"policy improvement\" rather than \"fidelity-first\". This is a weak distinction, as policy improvement is the goal of all MBRL systems, including Dreamer. The main novelties are the (effective) use of CoT reasoning in the model's SFT training and the entropy-based curriculum generator. These are good contributions, but the overall framework is a (well-executed) application of established MBRL concepts to the LLM agent domain, not a fundamental new paradigm.\n\nBesides, the paper emphasizes that it avoids costly real-world rollouts. However, it includes two important, undiscussed costs:\n- Annotation: The experience model is trained on thousands of trajectories (e.g., 3.6K for WebShop, 5.2K for ALFWorld ), each annotated with reasoning traces by a \"powerful LLM\". This is a substantial one-time computational cost, relying on a (presumably) proprietary, high-capacity model (like GPT-4) that is not part of the open-source setup.\n- Inference: The \"fast\" synthetic environment is itself a Llama-3.1-8B model. Generating each step of a trajectory requires a full inference pass from this 8B model . While this avoids infrastructure issues like Docker, it is still computationally intensive. Figure 3 (Left)  shows a reduction in total training time, but a clearer analysis of the wall-clock time per step (Real Env vs. DreamGYM) is needed to fully assess the *efficiency* claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MRppFCp6Wp", "reviewer": "Reviewer_JoEU", "summary": "This paper proposes DREAMGYM, a unified and scalable RL training framework for LLM agents. The core idea is to compress environment dynamics into a reasoning-based experience model that operates in an abstract textual state space, producing consistent state transitions and feedback. Coupled with an experience replay buffer (seeded with offline data and continually enriched with online trajectories) and reward-entropy–based curriculum task generation, DREAMGYM enables low-cost, efficient RL training entirely in a “synthetic environment,” while supporting sim-to-real transfer. Experiments show that on the non-RL-ready WebArena, DREAMGYM surpasses baselines by >150%; in RL-ready settings (WebShop/ALFWorld), purely synthetic interactions can match GRPO/PPO; and with sim-to-real, using ≤10% real interactions yields a further 64.5% gain. The paper attributes practical difficulties to costly real-world interactions, insufficient task diversity, unstable reward signals, and heavy infrastructure burdens; DREAMGYM addresses these via a unified abstract state space, vectorized/unified rewards, and an expandable task set.", "questions": "Please refer to the “Weakness” section for related questions.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. By training in synthetic environments and combining a model capable of distilling experience with experience replay and an automatic problem-generation (curriculum) mechanism, the authors reduce dependence on the real system and improve training stability and efficiency. On benchmarks such as WebArena, WebShop, and ALFWorld, the method achieves strong results, which further improve after incorporating a small amount of real data.\n\n2. The exposition is clear: the paper’s narrative is well structured and, overall, easy to read.", "weaknesses": "1. The work lacks experimental comparisons with representative methods in the “LLM Agents Reinforcement Learning” line of research; existing comparisons focus mainly on SFT/DPO and PPO/GRPO, making it difficult to accurately define this method’s relative advantages and applicability boundaries among peer approaches. Meanwhile, the related-work survey is neither systematic nor sufficiently comprehensive, with inadequate coverage of recent developments.\n\n2. Although the paper claims reduced real-world interaction and engineering cost, it does not report verifiable metrics (e.g., wall-clock time, GPU hours, throughput/token cost). This prevents a reliable assessment of the return on investment across different compute budgets and scales.\n\n3. Missing pseudocode; key details—such as data filtering/cleaning procedures, prompt templates, random seeds, critical hyperparameter tables and stopping criteria, and the order/frequency of alternating updates between the experience model and the policy—are insufficiently specified, and no accompanying scripts are provided, resulting in a high overall barrier to reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GrhK5P8L6e", "reviewer": "Reviewer_SnpT", "summary": "This paper introduces DREAMGYM, a scalable framework for training LLM-based agents without heavy environment interaction. Instead of collecting real rollouts, the framework employs a reasoning-based experience model that generates synthetic trajectories through reasoning, simulating next states and rewards. It further incorporates curriculum-based task generation, selecting tasks with high reward entropy to maximize information gain and progressively increase difficulty. Experiments on WebShop, ALFWorld, and WebArena show that DREAMGYM achieves comparable or superior performance to PPO and GRPO while using less than 10% of real environment interactions.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Targets a practical yet challenging problem in LLM agent RL — reducing training cost, wall-clock time, and instability arising from expensive environment interactions.\n- The paper is well-structured and clearly written: the problem is precisely defined, and each subproblem is addressed through three clearly delineated components — experience model inference, experience model training, and curriculum-based task generation. It makes the overall paper easy to follow, even for non-expert readers.\n- The experimental design is extensive and well-organized, effectively addressing questions that naturally arise during reading.", "weaknesses": "- Potential error accumulation in self-training loop: Since the experience model continuously generates synthetic rollouts and refines itself using those same transitions, any early bias or inaccuracy in its learned dynamics may compound over iterations. While this may not be severe in relatively simple domains (as Figure 4 suggests small gaps between real and synthetic transitions), it could become problematic in more complex or long-horizon environments where early modeling errors propagate and amplify.\n\n- Unclear buffer update strategy: Are all synthetic transitions stored, or only feasible, high-quality ones are retained? Without a clear filtering or prioritization mechanism, low-quality or hallucinated transitions could accumulate in the buffer, potentially degrading both policy learning and experience model stability over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kv9W6iNhP1", "reviewer": "Reviewer_PzPS", "summary": "This paper introduces DreamGym, a framework designed to synthesize diverse experiences with scalability in mind to enable effective online RL training for autonomous agents. This method embeds transition knowledge into the a experience model and leverage it to generate new challenging task. To verify their method, the author conduct some experiments on several agentic tasks like Webshop.", "questions": "See weaknesses. If none of the concerns is addressed，I will consider to downgrade the score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The idea of experience model is novel. \n2. The paper is well written.\n3. The idea of embedding transition dynamic is good.", "weaknesses": "1. The baselines are too simple. There are many intrinsic rewards designed in reinforcement learning to encourage exploration, we recommend you include at least one.\n2. Moreover, depending success label to select task group is not feasible for most situations. There can be a lot of circumstances in RL, The agent performs correctly in the first half of the task, but it is still making mistakes in the second half, in which case the task will not selected because the variation is zero. \n3. No task generation example is provided. I expect to see some instances of the selected task groups and the generated new task from task model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)