RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn
  Reinforcement Learning
Authors: name: Zihan Wang; name: Kangrui Wang; name: Qineng Wang; name: Pingyue Zhang; name: Linjie Li; name: Zhengyuan Yang; name: Kefan Yu; name: Minh Nhat Nguyen; name: Licheng Liu; name: Eli Gottlieb; name: Monica Lam; name: Yiping Lu; name: Kyunghyun Cho; name: Jiajun Wu; name: Li Fei-Fei; name: Lijuan Wang; name: Yejin Choi; name: Manling Li
Published: 2025-04-24 17:57:08

Abstract: Training large language models (LLMs) as interactive agents presents unique
challenges including long-horizon decision making and interacting with
stochastic environment feedback. While reinforcement learning (RL) has enabled
progress in static tasks, multi-turn agent RL training remains underexplored.
We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a
general framework for trajectory-level agent RL, and introduce RAGEN, a modular
system for training and evaluating LLM agents. Our study on three stylized
environments reveals three core findings. First, our agent RL training shows a
recurring mode of Echo Trap where reward variance cliffs and gradient spikes;
we address this with StarPO-S, a stabilized variant with trajectory filtering,
critic incorporation, and decoupled clipping. Second, we find the shaping of RL
rollouts would benefit from diverse initial states, medium interaction
granularity and more frequent sampling. Third, we show that without
fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge
through multi-turn RL and they may show shallow strategies or hallucinated
thoughts. Code and environments are available at
https://github.com/RAGEN-AI/RAGEN.