Rank-GRPO: Training LLM-based Conversational Recommender Systems with
  Reinforcement Learning
Authors: name: Yaochen Zhu, name: Harald Steck, name: Dawen Liang, name: Yinhan He, name: Jundong Li, name: Nathan Kallus
Organization: Netflix

Large language models (LLMs) are reshaping the recommender system paradigm by
enabling users to express preferences and receive recommendations through
conversations. Yet, aligning LLMs to the recommendation task remains
challenging: pretrained LLMs often generate out-of-catalog items, violate
required output formats, and their ranking quality degrades sharply toward the
end of the generated list. To this end, we propose ConvRec-R1, a two-stage
framework for end-to-end training of LLM-based conversational recommender
systems. In Stage 1, we construct a behavioral-cloning dataset with a
Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded
demonstrations from powerful blackbox LLMs to warm-start the RL training. In
Stage 2, we propose Rank-GRPO, a principled extension of group relative policy
optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats
each rank in the recommendation list as the unit instead of token (too
fine-grained) or sequence (too coarse), redefining rewards to remove non-causal
credit assignment and introducing a rank-level importance ratio based on the
geometric mean of rank-wise token probabilities to stabilize policy updates.
Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges
faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and
datasets are released at https://github.com/yaochenzhu/Rank-GRPO.