Agent Learning via Early Experience
Authors: name: Kai Zhang, name: Xiangchao Chen, name: Bo Liu, name: Tianci Xue, name: Zeyi Liao, name: Zhihan Liu, name: Xiyao Wang, name: Yuting Ning, name: Zhaorun Chen, name: Xiaohan Fu, name: Jian Xie, name: Yuxuan Sun, name: Boyu Gou, name: Qi Qi, name: Zihang Meng, name: Jianwei Yang, name: Ning Zhang, name: Xian Li, name: Ashish Shah, name: Dat Huynh, name: Hengduo Li, name: Zi Yang, name: Sara Cao, name: Lawrence Jang, name: Shuyan Zhou, name: Jiacheng Zhu, name: Huan Sun, name: Jason Weston, name: Yu Su, name: Yifan Wu
Organization: Meta Research

A long-term goal of language agents is to learn and improve through their own
experience, ultimately outperforming humans in complex, real-world tasks.
However, training agents from experience data with reinforcement learning
remains difficult in many environments, which either lack verifiable rewards
(e.g., websites) or require inefficient long-horizon rollouts (e.g., multi-turn
tool use). As a result, most current agents rely on supervised fine-tuning on
expert data, which is challenging to scale and generalizes poorly. This
limitation stems from the nature of expert demonstrations: they capture only a
narrow range of scenarios and expose the agent to limited environment
diversity. We address this limitation with a middle-ground paradigm we call
early experience: interaction data generated by the agent's own actions, where
the resulting future states serve as supervision without reward signals. Within
this paradigm we study two strategies of using such data: (1) Implicit world
modeling, which uses collected states to ground the policy in environment
dynamics; and (2) Self-reflection, where the agent learns from its suboptimal
actions to improve reasoning and decision-making. We evaluate across eight
diverse environments and multiple model families. Our approaches consistently
improve effectiveness and out-of-domain generalization, highlighting the value
of early experience. Moreover, in environments with verifiable rewards, our
results provide promising signals that early experience offers a strong
foundation for subsequent reinforcement learning, positioning it as a practical
bridge between imitation learning and fully experience-driven agents.