Scaling Agent Learning via Experience Synthesis
Authors: name: Zhaorun Chen; name: Zhuokai Zhao; name: Kai Zhang; name: Bo Liu; name: Qi Qi; name: Yifan Wu; name: Tarun Kalluri; name: Sara Cao; name: Yuanhao Xiong; name: Haibo Tong; name: Huaxiu Yao; name: Hengduo Li; name: Jiacheng Zhu; name: Xian Li; name: Dawn Song; name: Bo Li; name: Jason Weston; name: Dat Huynh
Published: 2025-11-05 18:58:48

Abstract: While reinforcement learning (RL) can empower large language model (LLM)
agents by enabling self-improvement through interaction, its practical adoption
remains challenging due to costly rollouts, limited task diversity, unreliable
reward signals, and infrastructure complexity, all of which obstruct the
collection of scalable experience data. To address these challenges, we
introduce DreamGym, the first unified framework designed to synthesize diverse
experiences with scalability in mind to enable effective online RL training for
autonomous agents. Rather than relying on expensive real-environment rollouts,
DreamGym distills environment dynamics into a reasoning-based experience model
that derives consistent state transitions and feedback signals through
step-by-step reasoning, enabling scalable agent rollout collection for RL. To
improve the stability and quality of transitions, DreamGym leverages an
experience replay buffer initialized with offline real-world data and
continuously enriched with fresh interactions to actively support agent
training. To improve knowledge acquisition, DreamGym adaptively generates new
tasks that challenge the current agent policy, enabling more effective online
curriculum learning. Experiments across diverse environments and agent
backbones demonstrate that DreamGym substantially improves RL training, both in
fully synthetic settings and in sim-to-real transfer scenarios. On non-RL-ready
tasks like WebArena, DreamGym outperforms all baselines by over 30%. And in
RL-ready but costly settings, it matches GRPO and PPO performance using only
synthetic interactions. When transferring a policy trained purely on synthetic
experiences to real-environment RL, DreamGym yields significant additional
performance gains while requiring far fewer real-world interactions, providing
a scalable warm-start strategy for general-purpose RL.