UserRL: Training Interactive User-Centric Agent via Reinforcement
  Learning
Authors: name: Cheng Qian; name: Zuxin Liu; name: Akshara Prabhakar; name: Jielin Qiu; name: Zhiwei Liu; name: Haolin Chen; name: Shirley Kokane; name: Heng Ji; name: Weiran Yao; name: Shelby Heinecke; name: Silvio Savarese; name: Caiming Xiong; name: Huan Wang
Published: 2025-09-24 03:33:20

Abstract: Reinforcement learning (RL) has shown promise in training agentic models that
move beyond static benchmarks to engage in dynamic, multi-turn interactions.
Yet, the ultimate value of such agents lies in their ability to assist users, a
setting where diversity and dynamics of user interaction pose challenges. In
this work, we propose UserRL, a unified framework for training and evaluating
user-centric abilities through standardized gym environments paired with
simulated users. We systematically vary turn-level reward assignment and
trajectory-level score calculation to analyze how different formulations affect
learning under the GRPO algorithm. Our experiments across Qwen3 models reveal
three key findings: (i) SFT cold start is critical for unlocking initial
interaction ability and enabling sustained RL improvements; (ii) deliberate
trajectory scoring yields more efficient and effective multi-turn interactions;
and (iii) while stronger simulated users (e.g., GPT-4o) facilitates training,
open-source simulators (e.g., Qwen3-32B) remain a cost-effective and
transferable option. Together, these results highlight that careful design of
reward shaping and user simulation choice is as crucial as model scale, and
establish UserRL as a practical pathway for developing robust user-centric
agentic models. All codes and data are public for future research.