Rank-GRPO: Training LLM-based Conversational Recommender Systems with
  Reinforcement Learning
Authors: name: Yaochen Zhu; name: Harald Steck; name: Dawen Liang; name: Yinhan He; name: Jundong Li; name: Nathan Kallus
Published: 2025-10-23 02:56:00

Abstract: Large language models (LLMs) are reshaping the recommender system paradigm by
enabling users to express preferences and receive recommendations through
conversations. Yet, aligning LLMs to the recommendation task remains
challenging: pretrained LLMs often generate out-of-catalog items, violate
required output formats, and their ranking quality degrades sharply toward the
end of the generated list. To this end, we propose ConvRec-R1, a two-stage
framework for end-to-end training of LLM-based conversational recommender
systems. In Stage 1, we construct a behavioral-cloning dataset with a
Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded
demonstrations from powerful blackbox LLMs to warm-start the RL training. In
Stage 2, we propose Rank-GRPO, a principled extension of group relative policy
optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats
each rank in the recommendation list as the unit instead of token (too
fine-grained) or sequence (too coarse), redefining rewards to remove non-causal
credit assignment and introducing a rank-level importance ratio based on the
geometric mean of rank-wise token probabilities to stabilize policy updates.
Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges
faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and
datasets are released at https://github.com/yaochenzhu/Rank-GRPO.