Agent Learning via Early Experience
Authors: name: Kai Zhang; name: Xiangchao Chen; name: Bo Liu; name: Tianci Xue; name: Zeyi Liao; name: Zhihan Liu; name: Xiyao Wang; name: Yuting Ning; name: Zhaorun Chen; name: Xiaohan Fu; name: Jian Xie; name: Yuxuan Sun; name: Boyu Gou; name: Qi Qi; name: Zihang Meng; name: Jianwei Yang; name: Ning Zhang; name: Xian Li; name: Ashish Shah; name: Dat Huynh; name: Hengduo Li; name: Zi Yang; name: Sara Cao; name: Lawrence Jang; name: Shuyan Zhou; name: Jiacheng Zhu; name: Huan Sun; name: Jason Weston; name: Yu Su; name: Yifan Wu
Published: 2025-10-09 17:59:17

Abstract: A long-term goal of language agents is to learn and improve through their own
experience, ultimately outperforming humans in complex, real-world tasks.
However, training agents from experience data with reinforcement learning
remains difficult in many environments, which either lack verifiable rewards
(e.g., websites) or require inefficient long-horizon rollouts (e.g., multi-turn
tool use). As a result, most current agents rely on supervised fine-tuning on
expert data, which is challenging to scale and generalizes poorly. This
limitation stems from the nature of expert demonstrations: they capture only a
narrow range of scenarios and expose the agent to limited environment
diversity. We address this limitation with a middle-ground paradigm we call
early experience: interaction data generated by the agent's own actions, where
the resulting future states serve as supervision without reward signals. Within
this paradigm we study two strategies of using such data: (1) Implicit world
modeling, which uses collected states to ground the policy in environment
dynamics; and (2) Self-reflection, where the agent learns from its suboptimal
actions to improve reasoning and decision-making. We evaluate across eight
diverse environments and multiple model families. Our approaches consistently
improve effectiveness and out-of-domain generalization, highlighting the value
of early experience. Moreover, in environments with verifiable rewards, our
results provide promising signals that early experience offers a strong
foundation for subsequent reinforcement learning, positioning it as a practical
bridge between imitation learning and fully experience-driven agents.