The Era of Real-World Human Interaction: RL from User Conversations
Authors: name: Chuanyang Jin; name: Jing Xu; name: Bo Liu; name: Leitian Tao; name: Olga Golovneva; name: Tianmin Shu; name: Wenting Zhao; name: Xian Li; name: Jason Weston
Published: 2025-09-29 17:50:31

Abstract: We posit that to achieve continual model improvement and multifaceted
alignment, future models must learn from natural human interaction. Current
conversational models are aligned using pre-annotated, expert-generated human
feedback. In this work, we introduce Reinforcement Learning from Human
Interaction (RLHI), a paradigm that learns directly from in-the-wild user
conversations. We develop two complementary methods: (1) RLHI with User-Guided
Rewrites, which revises unsatisfactory model outputs based on users'
natural-language follow-up responses, (2) RLHI with User-Based Rewards, which
learns via a reward model conditioned on knowledge of the user's long-term
interaction history (termed persona). Together, these methods link long-term
user personas to turn-level preferences via persona-conditioned preference
optimization. Trained on conversations derived from WildChat, both RLHI
variants outperform strong baselines in personalization and
instruction-following, and similar feedback enhances performance on reasoning
benchmarks. These results suggest organic human interaction offers scalable,
effective supervision for personalized alignment.