UserRL: Training Interactive User-Centric Agent via Reinforcement
  Learning
Authors: name: Cheng Qian, name: Zuxin Liu, name: Akshara Prabhakar, name: Jielin Qiu, name: Zhiwei Liu, name: Haolin Chen, name: Shirley Kokane, name: Heng Ji, name: Weiran Yao, name: Shelby Heinecke, name: Silvio Savarese, name: Caiming Xiong, name: Huan Wang

Reinforcement learning (RL) has shown promise in training agentic models that
move beyond static benchmarks to engage in dynamic, multi-turn interactions.
Yet, the ultimate value of such agents lies in their ability to assist users, a
setting where diversity and dynamics of user interaction pose challenges. In
this work, we propose UserRL, a unified framework for training and evaluating
user-centric abilities through standardized gym environments paired with
simulated users. We systematically vary turn-level reward assignment and
trajectory-level score calculation to analyze how different formulations affect
learning under the GRPO algorithm. Our experiments across Qwen3 models reveal
three key findings: (i) SFT cold start is critical for unlocking initial
interaction ability and enabling sustained RL improvements; (ii) deliberate
trajectory scoring yields more efficient and effective multi-turn interactions;
and (iii) while stronger simulated users (e.g., GPT-4o) facilitates training,
open-source simulators (e.g., Qwen3-32B) remain a cost-effective and
transferable option. Together, these results highlight that careful design of
reward shaping and user simulation choice is as crucial as model scale, and
establish UserRL as a practical pathway for developing robust user-centric
agentic models. All codes and data are public for future research.