Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning
Authors: Yaochen Zhu
Harald Steck
Dawen Liang
Yinhan He
Vito Claudio Ostuni
Jundong Li
Nathan Kallus
Venue: ICLR 2026 Poster (2026)
Keywords: Conversational recommender system
Reinforcement learning with Verifiable Reward
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) are reshaping the recommender system paradigm by enabling users to express preferences and receive recommendations through conversations. Yet, aligning LLMs to the recommendation task remains challenging: pretrained LLMs often generate out-of-catalog items, violate required output formats, and their ranking quality degrades sharply toward the end of the generated list. To this end, we propose ConvRec-R1, a two-stage framework for end-to-end training of LLM-based conversational recommender systems. In Stage 1, we construct a behavioral-cloning dataset with a Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded demonstrations from powerful blackbox LLMs to warm-start the RL training. In Stage 2, we propose Rank-GRPO, a principled extension of group relative policy optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats each rank in the recommendation list as the unit instead of token (too fine-grained) or sequence (too coarse), redefining rewards to remove non-causal credit assignment and introducing a rank-level importance ratio based on the geometric mean of rank-wise token probabilities to stabilize policy updates. Experiments on the on the Reddit-v2 and Redial datasets show that ConvRec-R1 converges faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and datasets are released at https://github.com/yaochenzhu/Rank-GRPO.

Published as a conference paper at ICLR 2026
RANK-GRPO: TRAINING LLM-BASED CONVERSA- TIONAL RECOMMENDER SYSTEMS WITH REINFORCE- MENT LEARNING
Yaochen Zhu Harald Steck", Dawen Liang", Yinhan Hc², Vito Ostuni", Jundong Li", Nathan Kallus2.3
University of Virginia, Netffix Research, *Cornell University. fuqp4qh, nee7ne, jundong/@virginn edu {hsteck, dliang, vostuni, mkallus)enettlix.com
ABSTRACT
Large language models (LLLMs) reshaping the recommender system paradigm by enabling users to express preferences and receive recommendations through conversations. Yet, aligning LLMs the recommendation task remains challeng- ing: pretrained LLMs often generate out-of-catalog items, violate required output formats, and their ranking quality degrades sharply toward the end of the generated list. To this end, we propose ConvRec-R1, two-stage framework for end-to-end training of LLM--based conversational recommender systems. Stage 1, we con- struct a behavioral-cloning dataset with a Reflect-Reeee-AAllii pipeline, which produces high-quality,
demonstrations from powerful blackbox LLMs warm-start the RL training. In Stage 2, we propose Rank-GRPO, principled extension of group relative policy optimization (GRPO) (Shao et al.) 2024 tailored to tasks with rank-style outputs. Rank-GRPO treats each rank in the recommendation list as the unit instead token (too fine-grained) or sequence (too coarse), redefining rewards to remove non-causal credit assignment and in- troducing a rank-level importance ratio based on the geometric mean of lank-wise token probabilities stabilize policy updates. Experiments and REDIAL datasets show that ConvRec-Ro RI converges faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and datasets are released https: / thub. ctw www.///wyy../y..ccccocoooo Rank-Rnnk--aaaa GRPO
the REDDIT-V2
INTRODUCTION
Recommender systems (RS) are undergoing a fundamental shift in the era of generative AL. Instead of passively predicting user behaviors, the emerging conversational RSs (CRS) paradigm position themselves as proactive agents that allow users to articulate nuanced preferences and receive tailored recommendations through interactive dialogue (Zhang et al. 2024). Large language models (LLMs) are at the center of this transformation: Their broad knowledge and reasoning capabilities enable both preference understanding and recommendation generation directly from user queries. Unlike earlier LLLM-base RS that embed item II) tokens into the L.M (Hua et al.] (023). recent studies emphasize the benefits of representing items in natural language (He et al. 2023] Zhu et al. 2025). This representation not only preserves the LLM's core language capability but also provides flexi- bility to verbally guide CRS toward different recommendation objectives, such as novelty, diversity, mnnotion-oriente relevance, while seamlessly integrating recommendation functionality.
Despite this promise, aligning LLMs with real-world recommendation tasks remains challenging, primarily due to the following three key limitations. First, trained on general web datasets, LLMs inherently lack awareness of item catalog specific to a given platform (e.g., the full set of TV shows available streaming service platforms). Without explicit catalog grounding, they frequently gen- erate out-of-catalog or even non-existent items in a zero-shot manner, severely limiting their prac- tical utility. Second, LLMs often fail conform to predefined output formats for the items (e.g..
Work done when Yaochen machine learning research intern Netflix Research.

---

Published as a conference paper at ICLR 2026
including both the title and release year for movies), which complicates item matching and inte- gration with downstream systems. Third, the quality of generated recommendation lists deteriorates sharply toward the end, due to the scarcity of high-quality ranking-style data in the pretraining stage. These challenges are particularly severe for smaller L.Mss, which are less powerful but commonly adopted in practice meet industrial-level efficiency and latency constraints. Collectively, the above limitations present significant barriers to the deployment of LLM-based CRS in real-world settings.
Recently, reinforcement learning from verifiable reward (RLVR) (Lambert et al. 2024) offers promising direction for alignment, as it introduces structured rewards that can explicitly guide LLMs toward generating catalog- grounded. high-quality recommendation lists. However, applying RL to LLM-based CRS presents two fundamental challenges. First, since RLVR relies on self-exploration for policy update, a warm-start behavior cloning stage with high-quality human demonstrations is generally required beforehand to ensure efficiency and stability (Shao et al.) 2024 Zhu et 2024a Wei et al. 2025). Yet it is impractical to expect human annotators to produce large-scale catalog- grounded item lists with good rankings, both due to the difficulty of recalling a large and dynamic catalog and the influence of subjective biases. Furthermore, prevalent RLVR algorithms such as
group relative policy optimization (GRPO) (Shao et al., 2024 Yu al. 2025 Liu et al.) 2025) are fundamentally misaligned with tasks with rank-style outputs. These methods typically conduct token-wise policy updates based on sequence-level rewards. However, sequence-level rewards (e.g.. the overall NDCG of a recommendation list) are too coarse to capture the contribution of individual items in each rank, while loken-level updates are overly fine-graine mnne-grined as each item can be represented by multiple tokens. This mismatch leads to non-causal credit assignment, misaligned importance weighting, unstable policy optimization, and ultimately poor quality in generated recommendations.
To address these challenges, we propose ConvRec-RL, a novel two-stage framework for end-to-end training of LLM-based CRS. Specifically, in Stage we design lightweight distillation pipeline, Reflect-Adjust which leverages zero-shot recommendations from powerful teacher LLM on training conversations to construct
demonstrations for supervised fine-tuning (SFT). Speciicley,,,,,,yy, teacher-generated recommendation lists are first projected into the target catalog space (Remap). then refined with contextual judgments to improve ranking quality (Reflect), and finally corrected for residual popularity biases relative to training distributions (Adjust). This process establishes exemplar demonstrations of high-quality ranked items for training conversations, which warm-starts the LLM with catalog awareness, proper item formatting, and initial ranking ability before subsequent RL post-training Furthermore, in Stage 2, we develop Rank-GRPO, a principled extension of GRPO designed for tasks with rank-style outputs, where user feedback can be treated as verifiable reward. Rank-GRPO considers each rank as the unit for both advantage estimation and importance reweighting, introducing reward masking to prevent non-causal credit assignment and defining a rank-level importance ratio via the geometric mean of item token probabilities to stabilize policy updates. Experiments on the public REDDIT-V2 v2 dataset demonstrate that ConvRec- RI substantially improves the recommendation quality over zero-shot LLMs, converges faster and achieves superior Recall and NDCG in recommendations than GRPO-style baselines.
2 METHODOLOGY
2.1 PROBLEM FORMULATION
In this paper, we formulate conversational recommendation sequential decision problem. Given a dialogue between a user and the system x = (#1...... .r.) composed of Tr tokens, the goal is to leam policy To (y) (r), parameterized by target LLM with learnalee parameters 6, that generates ranked list of N items y (,()) (2) ......... .......()) (N)) as recommendations. Here, we note that each ...... (k) is itself a sequence of tokens represented in natural language (c.g., movie title with release year), rather than numerical identifier in traditional RS. Ideally, each selected item should belong to predefined item catalog, i.e., (a) E C. The generation process of y can be decomposed into a sequence of rank-wise decisions. At each rank k. the policy generates the next item (k) conditioned on the context x and the previously generated items y <<))
item (a) = (0) (8) ..... Vin
(y( y ()k During generation, we instruct the LLM-based policy To to separate items in the recommendation list with special delimiter (c.g., a newline token (n) as follows:
Heeu (1) \n
+ B (N) <eos>],
2

---

Published as a conference paper at ICLR 2026
where denotes concatenation and <eos> marks the end of the generation process. To simplify the notation, we absorb |n with the previous item (and <eos> for the last item) and use y and yext interchangeably. The LLM-based To generates this sequence y auto-regressively at the token level.
2.2 SUPERVISED FINE-TUNING
The Stage of the ConvRec-R R1 framework supervised fine-tuning (SFT), which aims to warm- start the LLLM-bsedd CRS policy ,o(y)æ) with the foundational knowledge of the item catalog, required output formats, and the ability to generate and rank lists of items for recommendations, To address the critical challenge of lacking high-quality in-catalog item ranking data, we construct behavior cloning dataset, DSFT = {(x;"T S...... ) by distilling exemplar catalog-groundl ommendation demonstrations n for training conversations æदaH from a powerful teacher LLM (e.g., GPT-4o), which is often infeasible for direct deployment due high costs and latency.
rec-
2.2.1
PRELIMINARY DEMONSTRATIONS
The distillation process begins by generating zero-shot list of N candidate items for each dialogue SPS in the SFT training set. This is achieved by prompting powerful teacher L.M to produce
a preliminary item list yj
(y) following the same structure as Eq. ( The items and
ranks in SFT benefit from the teacher's advanced knowledge and reasoning ability, which represents the state of the in open-item recommendations (He al.) 2023 Zhu et al. (0225). However, these recommendations lie within the teacher's s own recommendation space Ce. which is not necessarily aligned with the target catalog C. Consequently, UIS may still include out-of-catalog (OOC) items, minor formatting inconsistencies, or rankings with the teacher's 's internal biases.
paw
2.2.2 THE REMAP- SOMAP-RREFEETTADIICE PIPELINE
To correct these misalignments, we refine the teacher's 's raw recommendations SFT through three- step Remap-ReflectAAdjaay pipeline. This process calculates a score 8final the catalog which then ranked to produce the demonstration yiat (see Appendix for details).
IR|S| over all items in
(i) Remap. This step grounds the teacher's recommendations by mapping them from the teacher's recommendation space Ce into the target catalog space C. Specifically, we compute an initial score Sremup
Scomv-item- Here, p R|Col is a sparse vector where non-zero entries represent the positional scores of items in SFT
RCCl using the aggregation: Sremap p Stem-ite + Ie)
heuristically calculated 1/Vk for the item rank k. The item-item similarity matrix Stem- R|Ce|x[C| maps each item from Ce the target catalog C based semantic similarity, where I,c {0, 1) 11 is a sparse indicator mapping matrix with Lio/u, u| if the u-th item in Ce equals the v-th catalog item, which directly transfers scores for in-catalog items. Finally, Sconv-ite E R [C] encodes the content similarity between the dialogue SPT and in-catalog items.
(ii) Reflect. While the remapping step grounds y scores in the catalog space, the ranking quality based on semantic similarity can be further improved. In the reflect step, we use LLM-as-a- judge to enhance the contextual relevance by asking the same teacher LLM to rate candidates with top-N, > N remapped scores 8remap a numerical scale from - (worst) to +L (best) based on their suitability recommendations for the context 2.SET These form sparse vector Trettect E [CI,, which is scaled by weight y and added to the remapped scores to produce a context-enhancee refined score vector as: Speftett = Stemap + Tredlect-
ratings
then normalized to
(iii) Adjust. The reflected scores already provide strong basis for ranking, but the final adjust step can further correct for residual popularity biases inherited from the teacher model. Specifically, align the scores with the empirical distribution of the groundtruth items in the training data by learning item-specific multiplicative bias
[R| and an additive bias b RIC|, where the final score is computed as spnal = w @ Stedlett + b, where @ denotes the Hadamard (element-we) wise) product. The final demonstration list SFT can then be constructed by selecting the top-N N items from the catalog according to their scores in Srinal and formatting them as the structured textual string shown in Eq. (1). With the established SFT dataset DSFT, the objective of behavior cloning can be
3

---

Published as a conference paper at ICLR 2026
formalized minimizing the negative log- likelihhod of the demonstrations follows:
LSFr (.. ( Dor llog To (v; SFT |x: ).
(2)
2.3 REINFORCHMENT LEARNING- BARDED ALIGNMENT
After the initial SFT stage, we further align the LLM-based CRS policy Teaer ()) (hereafter, sub- script SFT will be omitted for brevity) using reinforcement learning (RL), optimizing it directly against the structured reward derived from user feedback. We use DRL {(2 Mi
et )i=1 to de-
note the training data in the RL phase, where z. is training conversation and yf S C is the set of in-catalog items that the user provided positive feedback after the conversation.
2.3.1 VANILLA GRPO
In this paper, we mainly focused on the prevalent RL alignment method, group relative policy op- timization (GRPO) (Shao et al.] 2024), which leverages the relative rewards among a group of re- sponses to estimate advantage and eliminate the need for a learned value model (Schulman et al.) 2017). For each dialogue x.RLL DRL., GRPO first generates group of G responses {yr from a sampling policy ae (r). where lota is typically the model update-steps before 0. Under the setting of CRS, each yi is ranked list of N items, i.e., yi (W; ..... y/ (7)) which is evaluated against the groundtruth yf to calculate reward score reflecting overall quality. Based on the relative rewards across the G generations in the group, GRPO estimates a low-variance advantage for each response Å, which can be used to update the policy by maximizing the following objective:
lud
(f) = -Bn
1
Σ
min (ws,((0)A,,, clip(ws,,
e)A..))
(1z)
G
MA
(3)
where the importance ratio (f) and advantage As, for each token Vi.t are:
wi,+(0) = Te To(W,,clr.Bh.ck c1,
A = A r(z, v.) mean( (r(a, w))
Town (v, Vi,<,)"
(td(( w:)
We omit the KL-divergence with base policy (i.c., the SFT model 0SFT) in Eq. B for readability. Here, the importance sampling term (0) allows unbiased ad advantage estimate with samples gener- ated from the old policy T To facilitate stable updates, the objective function clips the importance ratio, limiting how far the new policy can diverge from Te.u The sequence-level reward r(æ, y:) is
typically ranking-based metric such DCG@ N, defined ΣN
log, The relevance score
nel,
(k) rel, is the item yi at rank is in the groundtruth set yf and has not appeared at an earlier rank; the relevance is 0 for all other cases, including repeated, non-relevant, and out-of-catalog items.
Gradient Analysis and Limitations. Although GRPO has shown effectiveness in many other do- mains with verifiable rewards, such as math problem solving (Guo al., 2025)) and code generation (Zhu et al. 2024a), the gradient analysis of its unclipped objective (for simplicity) reveals two fun- damental misalignments when applied to ranking tasks (see Appendix for derivation):
G Iual
V (CRFPO( (0) x E
(
Vo log (yoee Vi
(4)
UV
token (d1-level importance ratio
sequence (O-eevel advantage
token (t)-level gradient of Ing-likelitood
Intuitively, Eq. (4p shows that maximizing Eq. 33) increases or decreases the log- log-likelihood of each token in the generated response yi according to the sign of the advantage (scaled by the importance weight). However. (i) for the ranking task, the sequence-lever reward r(x. y.) (c.g.. DCG@ N of the whole list) is assigned uniformly every token, leading to non-causal credit assignment: tokens in the later items inherit reward from earlier ranks, but the auto-regressive generation goes the other way around. Take the DCG reward as an example, we note that can be naturally decomposed as sum of temporally discounted rank-level rewards (i.e., the relevance rel; here) as follows:
DCG@N= Σ
N
rely
reward,
reward,
N
reward,
(5)
loga
discount,
logo +1
logg +1
j=k
non-causal for ötem at rank
causal for item at rank
4

---

Published as a conference paper at ICLR 2026
Here, tokens of item rank & not only take credit for itself and items recommended afterwards, but also from previous recommendations. This becomes especially problematic toward the end of the list (i.e., when & is close to N), the recommendation quality deteriorates significantly. but they accumulate credit from strong recommendations at front ranks. Moreover, (ūi) in the off-policy step where y O, the token-level importance ratio is too fine-grained and mismatched with sequence- level advantages. Recent methods such as group sequence policy optimization (GSPO) (Zheng et al.) 2025) mitigate the mismatch with sequence-level importance weights, yet they remain misaligned for the ranking task, each rank should be the natural action unit instead of the whole sequence.
2.3.2 RANK-GRPO
To address the misalignment of vanilla GRPO and GSPO, we propose Rank-GRPO, a principled RL alignment algorithm for tasks with rank-structured outputs (e.g., recommendations)) The core insight is to treat each rank as a distinct action unit instead of a token (too fine-grained) or a se- quence (too coarse), which enables weighting for policy updating. Here, another major challenge is that computing the true advantage for item at rank k would, in principle, require enumerating trajectories that share the same pre- fix items y(<a)) which grows combinatorially. To address this, Rank-GRPO adopts rank-wise advantage estimation based on the same G trajectories used in GRPO, reducing complexity from exponential linear while retaining effectiveness. The objective of Rank-GRPO defined as:
precise and rank-aware credit assignment and importance
JRank-GRPO (u (12)
min( min(w,, (0) (.i 14,,,,,,,,)),..
GN
(6)
where the rank-level importance ratio wi,k (0) and advantage ÅLi are defined as:
wi, (0) = #u(y; (x)
A r(x, ( mean, |r(2,, ( where
Ta (1i (x0)
std, [r u. )(
(a)
1/10/*1
+ log To (Vi,.,,,, (.....))))))
(y (z)
To To(yi, (,,. vt.k,c()
=exp
1
We again omit the KL-divergence with the SFT base policy in Eq. for readability (like in Eq. Here, (v, (*) denotes the effective probability for rank-k, which is defined the geometric mean of the token probabilities for the item at the k-th rank. This normalization ensures stable importance
weights across items with varying token lengths. The rank-wise advantage Á is derived from
rank-level return r(x, (k)) that evaluates both the immediate reward (quality) of the item at rank and its downstream influence subsequent recommendations from rank & + through N.
Gradient Analysis and Insights. Before delving into the specific form of the rank-level return r(x, y/ (/()), ). we first analyze the gradient of Rank-GRPP and demonstrate that it effectively addresses the two fundamental misalignments of vanilla GRPO on mtr-strre outputs. The policy gra-
dient for GRRO (Here we also consider the unclipped objective like Eq. (4) for simplicity)
highlights its rank-centric policy update mechanism (see Appendix for the derivation):
Ve TRank-GGROOO( x IE
Wi, (0) Vo log क (
(R) (x)
1
E
Wi.k
Ack
Vo log To(Vi VaRee)
(k) Mi
rank (kivlevel importance ratio
rank (R))-evel advantage
rank (iihleeee gradient of loge likeliiel
The formulation in Eq. shows that Rank-GRPO fundamentally resolves the issues of vanilla GRPO: First, instead of applying uniform sequence-level reward to every token, assign rank (k)-specific advantage A for all the tokens of items at the k-th rank, ensuring that all tokens of
5

---

Published as a conference paper at ICLR 2026
the k-th item are updated consistently based on its contribution. Second, the importance weight and the gradient also operate at the rank level, where the effective probability #o <>( (2) aggregates token-level information for each rank, thereby improving the stability for policy updates.
2.3.3 REWARD SHAPING
Following the analysis of the sequence-level DCG@N reward as Eq. O in Rank-GRPO, we define the rank (k)-level return r(z, yi only the "causal part", which we denote as D as follows:
(k)) by masking out the "non-causal part" in DCG@ N and keeping
r(z, vt) - DCG@N E
rely
r(x, (L ) A DCG@KNN- E
relj loga( + +)
(8)
lo (
DCG@k:N ensures each item y ( receives credit only for its own contribution, respecting the sequential generation from higher rank to lower rank where earlier decisions influence the later ones. Although DCG@ N: minimally adapts the metric of interest (NDCG@ @ N), it necessarily the best reward for generative retrieval tasks, as the most influence of each item generation should come from the user query x as context compared the already-generated items in the higher ranks y/ (<R) Therefore, we introduce an exponential decay variant of reward for Rank-GRPO as follows:
N
Texpr (2,0i
(k))
rely ( j=k
= rel: + " Texpy (z. yi ((+11))
where r controls the discount rate. In the limit I = co, only the current item's relevance is credited. In practice, we find that I = provides a simple, stable and effective learning signal. We distinguish the two Rank-GRPO variants with the rank- wise return defined in Eqs. (8) and as Rank-GRPO (log) and Rank-GRPO (exp.x), respectively.
To further stabilize RL training, we add two penalties for instruction- featuuction-oolowinng failures:
(i) Incomplete Lists. If the model generates fewer than N items before emitting an <eos> token (N, < N), the missing ranks are treated irrelevant (zero return), and the premature stop token this rollout receives penalty 6, < 0 to discourage under-generation.
(ii Overflow Items. If the model generates more than N items, each overflow item beyond rank N receives a penalty 6, < discourage vee-geenerttion despite explicit instructions.
Although more sophisticated reward shaping methods are possible (e.g., using sliding-window con- straints on the horizon of the return), as the first rank-aware RL alignment framework, we find that Eqs. and are effective in practice and leave these extensions to future work.
3 EXPERIMENTS
3.1 DATASETS
In the main paper, focus the REDDIT-V2 dataset, the largest publicly available benchmark for CRS (~ 400k) that could support LLM post-training (He et al.) 2023 Zhu et al. (025). REDDIT- v2 consists of multi-turn dialogue sessions which users mention or request movies, paired with groundtruth items that received positive feedback during the conversation. Following the protocol of et (2023) and et al. (205).. the dataset is split into training, validation, and test sets with 383,013, 9,421, and 10,972 conversations. One notable difference with prior work is that we also require the LLM to output the release year together with the title for each recommendation. which is more difficult but important for disambiguation and catalog matching. To construct our supervised fine-tuning (SFT) dataset, we apply the Remap-Rea Ceeitapp-eeeet---e Ceffect-Adjust pipeline to 25% of the training set and the entire validation set (for monitoring). This produces catalog grounded behavior cloning dataset DSFT that serves to warm-start the RL phase and is released along with trained SFT checkpoints to facilitate future research. The full experiments on the REDIAL dataset are reported in Appendix D.7

---

Published as a conference paper at ICLR 2026
Val. NDCG@E and NDCG@20
In-Catalog Recommendation
1.2
Trait
100
1.0
Val deeion
0.00
5% In-Cattalo
80
0.5
0.00
80
0.04
0.02
N
20
Meon
N2C0g200
Sid
0.00
100
300
100
700
800
1100
1300
1500
200
400
000
BCD
1000 1200
1400 1600
202
400
600
800
1002
1200
1600
1600
Step
Step
Step
OwerrOwen2 5-.-------...
mageelarra//llmma3 2-18-nnatnug -
Figure 1: Training dynamics on loss, validation metrics, validation in-catalg recommendation ratio
with different backbone models during the SFT stage the REDDIT-V2 dataset.
Rank-GRPO (log-decay)
Rank-GRPO (exp.)
Validation NDCG
Owen2 1-.. Si-insturr
ren:1
0.14
Rarsotio sopoeat))
Raward
Reward
0.12
0.10
9.1
0.05
0.00
0.8
0.00
0.00
Rank-GRPO (log-decay)
Rank-GRPO (esp.)
Validetion NDCG
Ruward
Reeard
0.2
0.10
NDC0
0.10
0.05
0.08
Fan CBPC 1002
0000
Training dynamics of the reward acquired by ConvRec-RI with Rank- GRPO different rank during the RL stage. Right: Comparison of validation NDCG between GRPO and Rank-GRPO (both on-policy) on the REDDIT-V2 dataset.
Figure 2: Left Middle:
3.2 IMPLEMENTATION
We evaluate ConvRec-RI with three open- open-source instruction-tunee LLMs as the backbone models: Qwen2.5-0.5B -..52.5-0.5BInsttrucc. Llama-3.2- 1..2--1BInnstui IB-nnstuc,, and Alama-3.2-3B-Instteee
models larger than 3B may introduce higher latency and computational overhead, which pose significant challenges for their practical deployment. More details the implementation and hyperparameters are provided in Appendix C We also provide the detailed establishment process of the SFT dataset, the exact prompt templates used for the reflect step and recommendation, additional experimental studies, and qualitative analyses generations in Appendix A and respectively.
3.3 EVALUATION OF THE SFT STAGE
In Sections 3.3 and 3.4 we analyze training dynamics on the train/validation sets, while test set evaluation is deferred to Section 3.5 During the SFT stage, we observe three key phenomena in Fig. First, while training loss decreases steadily, validation loss plateaus around 800 steps"] re- flecting the difficulty of learning long, structured recommendation lists purely through behavior cloning. Second, the high-quality SFT data provides a strong initialization by grounding the model in the catalog and enforcing output format, where the proportion of in-catalog recommendations rapidly surpasses 99% across backbones. Third, the ranked list generation ability improves drasti- cally over the initial zero-shot model, with NDCG@ @ 20 increasing over 30x, 3x, and 1.5x for the 0.5B, 1B, and 3B model, respectively. Overall, SFT is crucial for warming up the models, i.c., con- straining them to the catalog, ensuring proper formatting, and improving long list generation ability before RL, which improves both sampling efficiency (for self-exploration) and training stability.
3.4 EVALUATION OF THE RL STAGE
3 We then analyze the training dynamics of ConvRec-RI I the RL stage. Digures[aandy show the re- wards obtained at different ranks in the generated recommendation list on the training set. From the
Nevertheless, adopt the 1,500-step SFT checkpoint for RL for stronger catalog memory. Starting with the InstructGPT (Ouyang al. 2022 it is common to initialize RL. from slightly overfitted SFT model.
7

---

Published as a conference paper at ICLR 2026
Rank-GRPO (log cog-decay)
Rank-GRPO (exp..
Va Viliatioon NDCG
Qwen2.5550 5-0.5500----10 nart--nsttrrtt
20
NDCG
0:10
HEcogs
6.0
6:0
BRR
20R
4000
000
1E90 IIKI 2300 1450
2000
6000
500
1000
0.04
2030
+000
BEE3
198 EER 7200 WEB
Step
Step
Stop
Rank-GRPO (log-decay)
Rank-GRPO (sxp.)
Validation NDCG
0.16
Lamma 2.BB-eetrur Roward
1O00830
0.14
Nov
0.10
Nccogo
0:00
0.00
San.0RR0 (ogaina))
0:00
201
1930 1230 1400e
2000
3000
1000
1.00
590
E9R 1000
Step
Figure 3: Training dynamic of reward and validation NDCG between GRPO, GSPO and Rank-
GRPO (off-policy) on REDDIT-V2. See Appendix Fig. 5 for results with Llama3.2 2-1 Insttrcct..
figures we observe that during training with Rank-GRPO, rewards increase monotonically across ranks in both on-policy and off-policy settings. Figures 2 and (right) further report the dynamics of NDCG on the validation set. Compared to GRPO and GSPO (in the off-policy setting, as both are the same in the on-policy setting), Bank-GRPO converges faster and achieves higher NDCG, with particularly larger improvements with large k. This suggests that the generated recommendation se- quences with Rank- GRPO-trained LLM not only improve in overall quality but also maintain good quality toward the tail of the list, where standard methods typically degrade. Theoretically, differ- ences in reward between GRPO and GRRNOGRPO&aa at the front ranks are relatively small (c.g.. for both sequence-level DCG@ N and rank-level DCG@ 1:N, rewards on the first item are identical. see Eq. (B). while improvements accumulate more substantially at later positions. In contrast, Rank-GRPO (expxx) significantly improves the performance at higher ranks, demonstrating the major influence of context on recommendations compared to the items generated in the higher ranks. Another inter- esting phenomenon we observed from the training dynamic of Rank-GRPO (expoe) (see the middle plots in Figures Jad B) is that the relevance (reward) of recommendations generated at lower ranks (e.g., 15, 20, represented by the darker lines) increases and then decreases as the training proceeds, while the relevance of recommendations generated at higher ranks (e.g.. 1, 2, 5 represented by the lighter lines) increases monotonically in trend. This is interesting as it shows the RI, alignment stage of Rank-GRPO (exp..) resembles a retrieval and rerank strategy where relevant items are first included in the generated list and then moved to higher ranks motivated by the reward.
2
In addition, the improvement is also evident in the off-policy setting where we set the per-sampling policy update step μ = 2 (Figure 3. which demonstrates the advantage of Rank-GRPO over GRPO and GSPO by aligning rank-wise importance weights with rank-wiss rewards. We also note that off- policy performance at the same step lags behind on-policy performance due to the higher variance introduced by importance weighting, though the ability to reuse trajectories sampled by previous policies offers a favorable trade-off in efficiency. The full experiments with the remaining LLM backbones are reported in Appendix D.4
3.5 COMPARISON WITH BASELINES
Finally, we present the one-time evaluation on the test set, which is collected one month after the validation set (He et al., 2023) Zhu et al. 2025). We compare different backbone LLMS trained with ConvRec-RI with state-of the- CRSs, including training- methods and prompting-base black-box LLMs. The detailed description of baselines is provided in Appendix [.22 From Table we observe that both stages of ConvRec-RI contribute substantially to its superior performance over the baselines. The SFT stage provides strong boost over the zero-shot baseline by grounding the recommendations in the catalog with the correct format, which improves the efficiency and stability of the RL training. The RL stage further enhances performance, especially at higher ranks, by aligning generation with rank-wise rewards. With these improvements, ConvRec-RI with Owen2.5-0.5B-IInstte Llama3.2- 1B Instrrctt backbone performs on par with GPT-4o, and ConvRec-RI with Llama3.2- 3B--Inttrutt backbone even surpasses GPT-4o and CRAG (GPT-4o) Deall/NDCG@20 Notably,
backbone substantially outperforms GPT-4o-mini, ConvRec-RI with
8

---

Published as a conference paper at ICLR 2026
Table I: Comparison between ConvRec-RI and various baselines on the REDDIT-V2 dataset. Here, R k and @@ & stand for Recall@A k and NDCG@ k. respectively.
Method
R@S
R
R R@15
R 20
N@5
N6
N@
No
Traditional and Transformer-Be based CRS
Redial cl
2018
0,0103
0.0228
0.0274
0.0322
0.0073
0.0113
0.0128
0.0143
KBRD
ben ct al
2019
0,0444
0.0813
0.1058
0.1223
0.0305
0.0418
0.0490
0.0545
KGSF
hon
2020
0.0579
0.0921
0.1246
0. 11343
0,0405
0.0503
0.0899
0.0662
UniCRS
Wang gl.
2022
0.0722
0.1053
0,1344
0. 0.1494
0,0548
0.0640
0.0726
0.0778
NBCRS
Xie al. 2024
0.0801
0.1290
0.1655
0,2019
0.0661
0.0833
0.0954
0.1048
L.M Prompting-Basee
Methods
GPT-to-mini (zero-shol)
0,0949
0.1348
0.1600
0.1687
0.0747
0.0877
0.0950
0.0973
GPT-4o 4o shor)
Ho et al.
2023
0.1106
0.1625
0.1992
0.2147
0.0861
0.1028
0.1136
0.1197
CRAG Zhu et al.
0.1146
0.1715
0,2030
0.2212
0.0885
0.1065
0.1164
0.1227
L.M RL
Ood-Training-Baa Based Methods (off policy results see Appendix Table
Most rosults are signifiiant is standard error are hetwen 0.002000.0555 for R and 0.000-00000000 for N885-20
Qwen2.5-0.58 (zem-shot)
0.0021
0.002.
0.0021
0.0021
0.0023
0.0022
0.0021
0.0021
+ SFT
0.0642
0.1027
0.1212
0.1308
0.0502
0.0625
0.0678
0.0704
SFT+ Vanilla GRPO
0.0834
0.1298
0.1617
0.1803
0.0651
0.0801
0.0895
0.0945
STTT+Rakk GRPO (log)
0,0892
0.1353
0.1720
0.1946
0.0701
0.0849
0.0957
0.1017
SSPT-Rank.ce TTT--RRkk-RRPPOO (e
0,0946
0.1468
0.1813
0.2047
0.0744
0.0915
0.1016
0.1079
Llama-3.2-IB (2cro-shot)
0.0187
0.0240
0.0257
0.0263
0.0157
0.0170
0.0175
0.0176
+ SPT
0.0754
0.1148
0.1354
0.1498
0.0595
0.0723
0.0782
0.0819
+ SFT+Vanilla GRPO
0.0979
0.1474
0.1806
0.2037
0.0762
0.0920
0.1026
0. 1097
+ SFT-RRak-GRPO (log)
0.1014
0.1527
0.1907
0.2159
0.0792
0.0956
0.1067
0.1134
-SITTRaakk GRPO (e
0.1039
0.1569
0.1937
0.2165
0.0813
0.0985
0.1092
0.1153
Llama--32-33 (2cro-shot)
0,0574
0.0805
0.0912
0.0961
0.0463
0.0535
0.0566
0.0580
+ SFT
0,0788
0.1191
0.1410
0.1541
0.0615
0.0744
0.0807
0.0842
+ SFT+Vamilla GRPO
0.1034
0.1568
0.1946
0.2186
0.0824
0.0996
0.1106
0.1170
SFT-Raak-GRPO (log)
0.1103
0. 0.1638
0.2033
0.2302
0.0862
0. 10355
0.1169
0.39
+ SFT-Rank-GRPO (eXP
0.1178
0.1756
0.2150
0.2368
0.0919
0.1107
0.1223
0.1283
CRAG requires 5-7 GPT-4o API calls per recommendation (for mentioned item extraction, reflec- tion, and re-ranking), incurring far higher cost and latency, whereas ConvRec-RI enables smaller open-source LLMs directly produce high-quality recommendations. These results highlight that properly aligned small LLMs can match or even exceed much larger proprietary models in CRS.
3.6 ABLATION STUDY
In this section, we present an ablation study where we remove the remap and reflect steps in the behavior cloning dataset construction, as well as the entire SFT stage. The results summarized in Table[2] We observe that SFT plays a crucial role in warming up RL with strong catalog awareness and preliminary ability to generate coherent ranked item lists. In particular, removing the remap step significantly harms performance by weakening catalog grounding, while removing the reflect step degrades the quality of the learned ranking, leading to lower RecalI/NDCG at larger k. The full Remaa--Releeee Reflect-Adjust pipeline followed by SFT and Rank-GRU thus yields the best overall performance, confirming that all components contribute meaningfully the final CRS quality.
RELATED WORK
4.1 LLM-BASED CONVERSATIONAL RECOMMENDER SYSTEMS
LLM--ased conversational recommender systems (CRS) leverage the language understanding and generation capabilities of LLMS to capture user preferences and deliver recommendations through dialogue (Zhao et al. 2023 Zhang et al. 2024 Lambert et al. 2024). Early approaches often en- coded item IDs as special tokens in the model vocabulary, allowing the LLM to generate IDs directly recommendations (Hua et al. 2023 Zhu et al. 2024b Zhang et al.) 2025b). More recent work, however, has highlighted the benefits of representing items in natural language (e.g., titles with at- tributes descriptions) and prompting the model to output recommendation lists (He et al. 2023

---

Published as a conference paper at ICLR 2026
Table 2: Comparison between ConvRec-RI and various ablation models the REDDIT-V2 dataset.
Method
R@5
R@
R@15
R@
N@5
N@10
N@15
N@:0
Qwen2.5-0.5R (SFT)
0.0642
0.1027
0.1212
0.1308
0.0502
0.0625
0.0678
0.0704
remove remup-reflect step
0,0579
0.0548
0.0614
0.0637
remove reflect step
0,0623
0.1307
0.0496
0.0600
0.0670
Qwen2.5.0.5B (SIT Rank (exp,)))
0.0946
0.0915
0.1016
0.1079
remove SFT stage (R) (RI-2))))
0.0440
0.0545
0.0584
0,0618
0.0373
0.0405
0.0419
0.0431
1. 1.2m-3.2-18 (SFT)
0.0754
0.1148
0.1354
0.1498
0.0595
0.0723
0.0782
0.0819
remove remap-reflect step
0.0713
0.1085
0.1312
0.1421
0,0574
0.0090
0.0757
0.0786
reflect step
0.0714
0.1102
0.1339
0.1460
0.0576
0.0098
0.0767
0.0800
B.2-1BB + Rank-GRPO (exp,..)
0.1039
0.1937
0.2165
0,0813
0.0985
0.1092
0.1153
remove stage (R1-zero)
0.0769
0.1175
0.0651
0.0771
0.0854
0.0884
Llama-3.2-3B (SFT)
0.0788
0.1191
0,1410
0.1541
0.0615
0.0744
0,0807
0,0842
remove remap-reflect step
0.0724
0.1187
0.1369
0.1382
0.0611
0,0740
0.0801
0,0809
reflect step
0.0748
0.1236
0.1371
0.1441
0.0619
0.0757
0.0801
0.0818
Llama-3.2-B8 3-3- (SFT Rank-GRPO (e)
0,1178
0.1756
0,2150
0.2368
0.0919
0.1107
0.1223
0.1283
romove SFT stage (RI-zaro)
0.0802
0.1295
0.1639
0.1771
0.0737
0.0898
0.1001
0.1039
Zhang et al., 2025a Zhu et al. 2025) or rankings (Hou et al. 2024). This natural-language for- mulation of the recommendation task maximally preserves the LLM's s core language generalization and reasoning abilities and enables flexible verbal guidance of the CRS by conditioning conversa- tional context, attribute constraints, or other objectives such as novelty and diversity (He et al. [023] Zhang al. 2024). However, zero-shot generation suffers from catalog unawareness, formatting er- rors, and a sharp decline in quality toward the end of the recommendation list. While tine-tuning approaches (Luo et al.] (025) or aatreval-augmenner generation (RAG) (Zhu et al. 2025 can mit- igate these issues, they require large amounts of high-quality data. In contrast, large-scale RI. for LLLM-based CRS with inexpensive, verifiable rewards remains largely underexplored.
4.2 REINFORCHMENT LEARNING WITH VERIFIABLE REWARD
Reinforeement learning from verifiable rewards (RLVR) has emerged central paradigm for aligning LLMs. A core algorithm in this domain is group relative policy optimization (GRPO) (Shao)
al. 2024). which estimates advantages by comparing relative rewards across groups of responses. Several extensions, such as DAPO Yuu et al. 2025). Dr. GRPO (Zheng et al.) (2025), and GSPO (Liu et al. 2025), have enhanced GRPO through improved length normalization, sample efficiency, and sequence-level stability. Yet, GRPO-style methods remain underexplored for tasks with rank- structured outputs such as recommendation. To our knowledge, only two works, Rec-RI (Lin al., 2025) and ReeLLM-RI (Xie et al. 2025), have applied GRPO to recommender systems. Rec-Rl generates item descriptions that are later matched our goal of enabling L.LMs to directly generate ranked lists of in-catalog items in natural language. ReeLLM-RI generates lists directly but still applies a sequence-level reward for token-level updates, which misaligns with the structure of ranking tasks and limits performance as reflected in the vanilla GRPO baseline. In contrast, we propose Rank-GRPO, which introduces rank-level advantages for precise credit assignment and rank-level importance ratio based the geometric mean of item- token probabilities, providing a principled framework for aligning LLMs on uank-structured outputs.
catalog to compute rewards, differing from
5 CONCLUSION
We introduced ConvRec-R1, a two-stage framework for aligning LLM-based conversational rec- ommender systems with reinforcement learning. ConvRec-RI combines a lightweight distillation pipeline, Refep-Reflett-Adjjsst, with Rank-GRPO, principled extension of GRPO tailored to rank-style outputs, addressing its fundamental misalignment of sequence-level rewards and token- level updates. Empirically, ConvRec-RI significantly improves the recommendation quality of the base model on the REDDIT-V2 dataset, surpassing GRPO-style baselines and even large zero-shot black-box models in Recall and NDCG. Beyond recommendation, our method shows promise for broad range of tasks with aun-structured outputs, suggesting new opportunities to adapt RL-based alignment techniques for ranking, retrieval, and other sequential pacision-making problems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
Yaochen Zhu and Jundong Li are supported in part by the Office of Naval Research (ONR) under grant N000142412636 and the gift funding from Netflix.
REFERENCES
Kegin Bao, Jizhi Zhang, Wenjie Wang, Yang Zhang, Zhengyi Yang, Yanchen Luo, Chong Chen, Fuli Feng, and Qi Tian. A bi-step grounding paradigm for large language models in recommendation systems. ACM Transactions on Recommender Systems, 3(4):1-27, 2025.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 394344)3 1952.
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. Towards knowledge-basse based recommender dialog system. In EMNLP. 2019.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 16X/v:4604.06/74, 2016.
Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. On softmax direct preference optimization for recommendation. NeurIPS, 37: 27463-27489, 2024.
Tri Dao. FlashAttentionn-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08669, 2023.
Chongming Gao, Ruijun Chen, Shuai Yuan, Kexin Huang, Yuanqing Yu, and Xiangnan He. SPRec: Self-play debias LLM-based recommendation. WWW, pp. 5075-5084, 2025.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arxiv:2501. 2025.
Shirley Anugrah Hayati, Dongyeop Kang, Qingxiaoyang Zhu, Weiyan Shi, and Zhou Yu. Inspired: Toward sociable recommendation dialog systems. In EMNLP, pp. 8142-8152, 2020.
Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng. Bodhisattwa Prasad Majumder. Nathan Kallus, and Julian McAuley. Large language models as zero-shot conversa- tional recommenders. In CIKM, pp. 720-730, 2023.
Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In ECIR, pp. 364-381. Springer, 2024.
Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. How to index item IDs for recom- mendation foundation models. In SIGIR, pp. 195-204, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with PagedAttenttinn. In SOSP, pp. 611-626, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.
1. 1.15124, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented gener- ation for knowledge-intensive NLP tasks. NewrIPS, 33:9459-9474, 2020.
Raymond Li. Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. Towards deep conversational recommendations. In NeurlPS, 2018.

---

Published as a conference paper at ICLR 2026
Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In WWW, pp. 689-698, 2018
Jiacheng Lin, Tian Wang, and Kun Qian. Rec-R1: Bridging generative large language mod- els and user-centric recommendation systems via reinforcement learning. arXiv preprint 0rXiv:2503.24289 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 2025.
Sichun Luo, Bowei He, Haohan Zhao, Wei Shao, Yanlin Qi. Yinya Huang, Aojun Zhou, Yuxuan Yao, Zongpeng Li, Yuanzhang Xiao, et al. RecRanker: Instruction tuning large language model as ranker for top-k recommendation. ACM TOIS, 43(5):1-31, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 35:27730-27744 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. NeurIPS, 04533728-33744 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In Supercomputing Conference, pp. 1-16, 2020.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. In EMNLP, pp. 3982-3992, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06:: 06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03330, 2024.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. NeurIPS, 12, 1999.
Xiaolei Wang, Kun Zhou, Ji-Rong Wen, and Wayne Xin Zhao. Towards unified conversational recommender systems via knowledge-enhanced prompt learning. In KDD, pp. 1929-1937, 2022.
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, et al. Webagent-rl: Training web agents via end-to-end multi-turn rein- forcement learning. arXiv preprint arXiv:2505.16422 2025.
Junda Wu, Rohan Surana, Zhouhang Xie, Yiran Shen, Yu Xia, Tong Yu, Ryan A Rossi, Prithviraj Ammanabrolu, and Julian McAuley, In-context ranking preference optimization. COLM, 2025.
Yu Xic, Xingkai Ren, Ying Qi. Yao Hu, and Lianlei Shan. RecLLM-RI: A two-stage training paradigm with reinforcement learning and chain-of-thought v1. arXiv preprint 202,2 2025.
Zhouhang Xie, Junda Wu, Hyunsik Jeon, Zhankui He, Harald Steck, Rahul Jha, Dawen Liang, Nathan Kallus, and Julian McAuley. Neighborhood- based collaborative filtering for conversa- tional recommendation. In RecSys, pp. 1045-1050, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. DAPO: An open-source LLM reinforcement learning system scale. arXiv preprint 2ati:2553..44400 2025.
12

---

Published as a conference paper at ICLR 2026
An Zhang, Yuxin Chen, Leheng Sheng, Xiang Wang, and Tat-Seng Chua. On generative agents in recommendation. In SIGIR, pp. 1807-1817, 2024.
Junjie Zhang, Ruobing Xie, Yupeng Hou, Xin Zhao, Leyu Lin, and Ji-Rong Wen. Recommendation as instruction following: A large language model empowered recommendation approach. ACM TOIS, 43(5):1-37, 2025a.
Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. CoLLM: Integrat- ing collaborative embeddings into large language models recommendation, IEEE Transactions on Knowledge and Data Engineering, 2025b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, et al. survey of large language models. arXiv preprint ar arXiv:2303.1822 1(2), 2023.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint AxXi:2557..187/1 2025.
Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. Im- proving conversational recommender systems via knowledge graph based semantic fusion. In KDD, pp. 1006-1014, 2020.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 1100:::200..19911, 2024a.
Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong. and Jundong Li. Collaborative large language model for recommender systems. In WWW 2024, pp. 3162-3172, 2024b.
Yaochen Zhu, Chao Wan, Harald Steck, Dawen Liang, Yesu Feng. Nathan Kallus, and Jundong Collaborative retrieval for large language model-based conversational recommender systems. In WWW. pp. 3323-33344 2025.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A DETAILS FOR THE REMAP-RELEEC-ADIUS. PIPELINE
This section provides the detail for the Remap-Rellett-ddius pipeline used to generate the saaalog-grondder behavior cloning dataset for the supervised fine-tuning (SFT) stage of ConvRec- RI (see Section 2.2 We note that the following pipeline is specifically designed for the movie recommendation task, but the generalization other items generally follows similar procedure.
A.1 STEP 1. REMAP
The remap step aims to ground the raw recommendations SFT in the teacher LLM's recommenda- tion space Ce, into score vector Stemap R in the target catalog space C. The process involves three components: item metadata generation, similarity calculation, and score aggregation.
A.1.1 METADATA GENERATION
remapping, we first generate consistent and comprehensive metadata for every item in both Ce and C. Based whether the teacher LLM @ has the knowledge of the item and whether the item is in the catalog, we have the following three cases:
To facilitate semantic similarity-
(i) For in-catalog items known to teacher LLM θ we use zero-shot prompt generate summary including keywords and plot description. This ensures a consistent style and broad coverage of aspects that facilitate high-quality item-item similarity calculation.
For out-of-catalog (OOC) hallucinated items. Since these items are recommended by the LLM, it can still generate plausible metadata crucial for mapping them back to the catalog.
(iii)
For in-catalog items unknown to the LLM (e.g., niche or recent items), we employ retrieval augmented generation (RAG) (Lewis al.) (220)) and in-context learning (ICL). We retrieve external information about the item via Google Search API on Wikipedia IMDb] and Rotten Tomatoes and provide the LLM along with few-shot examples guide the generation.
The prompts we are using to generate the metadata for known/unknown items are as follows.
Prompt to Generate Metadata for Items Known to the Teacher LLM θ
Summarize the matadata for the movie movie_name (zelease year) based the information the format of: Movie Name (Release Year))
Keywords:
ccomma-separated keyxords>
Plots:
cone-paragraph plot summary>
You should separate the keywords with commas without. numbering. The plot should be one paragraph.
Prompt to Generate Metadata for Items Unknown to the Teacher LLM O with RAG and ICL
Summarize the meLadata fer Lhe movie movie.name (release year) based the information in the format of: Movie Name (Release_ Year))
Keywords:
sepparated keywords>
Plots:
pane-parag:aph plat summary>
You should separate the keywords with commas without numbering. The plot should be one paragraph.
Here are the information related to the target movie: (movie.info)
Here are some examples for the generated metadata for different movies: (examples)
14

---

Published as a conference paper at ICLR 2026
Here provide some examples of the generated metadata follows:
Examples for the Generated Metadata in the Remap Step
Case (4). In-catalog items known to teacher LLM
Stranger Things (2016)
Keywords: supernatural, mystery, 1980a, parallel universe, friendahip, government conspiracy, small town, telexinesis, missing child, menater In the small Lown of Hawkinæ, Indiana, the sudden disappearance of young boy named Will Byers sets off chain of events that unravels series of supernatural mysteries. As his friends and family search for him, they encounter mysterious girl with telekinetic powers who may hold the key to finding Kill. Meanwhile, secretive gevernment agency conducts experiments that open pertal parallel universe, unlaashing terrifying creatures into the
Plots:
Case (ii). out-of-catalog (0OC) or hallucinated items The Divergent Series (2014) (a series but not movie) Keywords: dystoplan, factions, rebellion, identity, society, future, action, advenLuze, science Lletien, young adul., survival, government, centrol, freedom Plots: In future society divided into five distinct factions based on human virtues, everyone must choose where they belong. Tris Prior discovers she is Divergent, meaning she doesn't fit into
group. As she uncovers a conspiracy eliminate nate all Divergents, Tris must rely her unique abilities survive and protect thosa she loves. With the help of the mysterious Four, she embarks journey self-discovery and rebellion aqainst the oppressive syslem, challenging
sLaLus quo Lighting Ioz Ireedom.
Case (111). In-catalog items UNKNOWN to teacher LIM La strada verso casa (2011)
Keywords: drama, pain, death, life, Michelangelo, Antonio, Giulia, Italy, aubtle pressures, intertwine, healing
Plots: "La strada verso casa" tells the poignant story of Michelangelo, Antonio, and Giulia, three individuals whose lives are abruptly di by pain and death. As they face their personal struggles, they find solace and strenęth in eact other, subtly influencing one another's paths. Through their intertwined journeys, they navigate the complexities of grief and healing, ultimately discovering renewed senss of life and purpose. Set agsinst the backdrop of Italy, the film exp' ores thamas of connection, resi lience, and the transformat ivv power of human relationships.
A.1.2 SIMILARITY AND SCORE CALCULATION
With the generated metadata for items in Ce and C in place, we compute two types of similarities:
(i) Item-Item Similarity (Snem-item): We calculate the content-based semantic similarity be- tween every item in the teacher's space Ce and every item in the target catalog C. The gen- erated metadata are embedded using pre-trained mantence-transfomme model (Reimers & Gurevych] 2019) (we use KoovaSearchsseella.n...00.
in this paper). The cosine similarity between these embeddings forms the item-item similarity matrix Site-ittmm R[C]|
(ii Conversation-Item Similarity (Scomtext) To further enhance the contextual relevance, we compute the similarity between the conversation x; SPT and each item the catalog C. We embed SFT with the Stella model and its cosine similarity with each catalog item's metadata embedding is calculated, resulting context vector Scontxt [C
The final remapped scores calculated by aggregating the signals from the teacher's raw recom- mendations y and Sitem-item- Scontext as follows:
8remap p (Stem-item 1e) + À Scontexts
(10)
where p E CCe a sparse vector representing the positional scores of items in the teacher's raw recommendation list. The score for an item at rank is heuristically set to 1/Vk. and for items not
present, I E {0, 1] |Cal [CI is sparse indicator mapping matrix with Ie [u, u] = 1 if the w-th item in Ce equals the v-th catalog item, which directly transfer the positional scores of any recommended items that are already in the catalog, À a hyperparameter that weights the importance of the conversation-iten similarity for the remapping. In the paper, we set X =
15

---

Published as a conference paper at ICLR 2026
A.2 STEP 2. REFLECT
The reflect step adjusts the ranking of the top candidates from the remap stage by leveraging the teacher LLM's judgment on the contextual relevancy the conversation .:S While the remap- ping step grounds the raw recommendations from the teacher V the catalog space 9remup. the ranking may not be perfectly aligned with the nuances of the user's conversational context as they conducted based embedding models, which are known to be relatively insufficient cap- ture multiple aspects and subtle preferences behind the lines that are crucial for recommendations. Therefore, we leverage LLM-as-a-judge to rate the top-N, N items (e.g., top 100) from 8remap based on their suitability for the conversation. We use the following prompt template:
Prompt for LLM-as-aaJJddee in the reflect step
Pretend you are a movie recommender system. will give you conversation between a user and you (a recommende: system) as well as some movie candidates from our movie database.
You need to rate each retriaved movie as recommendations into five levels based on the conversation:
2 (great), 1 (good), 0 (normal), -1 (no: good), -2 (bad).
Here is the conversation: (context)
Here are the movie candi sandidates: (rec_tizles).
You need reply with Lhe rating of each movie in line, In Lhe form of movie_name (releass_year ttftraaing
The numerical ratings from the LLM form sparse reflection vector Prefect RICI, This vector is then scaled by a hyperparameter and added to the remapped scores to produce the reflected scores:
(11) This step helps elevate contextually more relevant items higher ranks, and therefore further improves the quality of the exemplar ranking demonstration data. In this paper, we choose five-scale reflection scores from [-2, representing bad great and therefore 7 is set 0.5 to normalize the reflection scores to [-1, 1, 1]. which has similar scale as 8remap-
Scefeect = Syemap + y Tretlect-
A.3 ADJUST
Finally, the adjust step is a valuable final stage that corrects for residual popularity biases that may be inherited from the teacher model or the remapping process due to its unawareness of the current trend. Specifically, we align the score distribution with the empirical distribution observed in the groundtruth training data (i.e., items with positive user feedback). This is framed as residual learning problem. We learn an item-specific multiplicative bias vector RCC and an additive bias vector b E R [C] to fine-tune the scores. The final, adjusted scores are computed as:
(12) where denotes the dementt-wie product. The vectors w and b are optimized by maximizing the multinomial likeliill of observing the groundtruth positive items from the training data D [Liang et al. 2018). 8) If we denote the groundtruth items for conversation as multi-hot vector y!" the optimization objective can be formulated as:
Sinal = w ② Spelect + b.
exp ((w @ 8 + b)z)
Ludjust
Σ
Σ log
Aue //
11 + X Abll (13)
E (EDmu A Σtec exp ((w Syellect, b){)
where An and ÅE are hyperparameters for weight decay and are set to 0.01. After this final adjust- ment step, for each training conversation 2.SFT the top-N items from the catalog are selected based on their scores in Simal to form the final, high-quality demonstration list SPT used in the SFT stage.
A.4 EXAMPLE OF DEMONSTRATION GENERATED BY
PIPELINE
In this section, we present an example from the constructed behavior cloning dataset DsFT for ConvRec-R1, along with qualitative ablation study of the Remap-Reflect-Adjjuy pipeline. The
16

---

Published as a conference paper at ICLR 2026
example illustrates given conversation 2²PT how the the teacher's zero-shot recommendations and the intermediate results evolve across each stage of the pipeline. To facilitate future research on LLM-based CRS, also release the behavior cloning dataset, together with the accompanying code and trained models, at htttta///g.ubbb...cc//cccceeeeen
Exemplar Demonstrations from the Remap-ReflectAAjjsst Pipeline
Conversation
User: good ocean sailing movies like master and commancar?. would prefer more of the old days and not something with sabmarines/ww/ but did 1ike hont for red october let. Just looking for something like master and commander. It doesn't necessarily have be militery, can be pirstes tco. know of the obvious stuff like pirates of carribean mories, that netflix animated (Imaol,
movie with jared herris/karl urban, waterworld
think this copic and someone suggested horatio herublewer. liked the show
quality episodes watched think subtitles, which elmost must for me. Appreciate any
watch on youtube didn't like
auggesttonn.
(11) Groundtruth y!": 1492: Conquest of Paradise (1992)
(ii1) Outputs from the Remap-Reflect-Kee Pipeline:
Zero-Shot Ve,.ow (GOC titles red)
Step Remap
1 Mutiny on the Bounty (1935)
The Hounty (1984)
2 The Bounty (1984)
Captain Bload (1935)
3 Captoin Blood (1935)
The Sea Bowk (1940)
4 The See. llakk (1310)
Treasure Island (1950)
5 Treasure Island (1950)
The Crimson Pirate (1952)
6 Xoby Dick (1956)
The Spanish Main (1945)
7 The Buccaneer (1958)
Cutthroat Island (1995)
8 Billy Budd (1962)
The Adventures of Robia Bood (1938)
9 Swashbuckler (1376)
The Vikings (1958)
The Black Swan (1942)
Master and Commander (2003)
The Crimson Pirate (1952)
Silent
(1958)
Defians! (1962)
(1981)
Song Ships
Black Sails (2014)
(1926)
Captain Phillips (2013)
Cutthroat Adventures
(1995)
Moziny (1954)
Road (1938)
Grayhound (2020)
The Vikinga (1958)
Captaina Courageous (1337)
18 The See wals (1341)
20,000 Leagues Under the Sea (1954)
19 The Spanish Kain (1945)
in the Heart the Sea (2015)
20 Against All Flags (1952)
The Wrack the Mary Beare (1959)
Step Reflect (vs. Remap)
Step Adjust erz (with popularity)
Bounty (1984)
Bounty (1984) 56
2 Captain Blood (1935)
Captein Blood (1935) 31
3 The See. Havk (1340)
Master and Commander (2003) 262
4 Treasure Island (1950)
Das Boot (1981) | 523 A
5 Master and Commander (2003) 4
The Sea Fawk (1940) 17 r
6 The Crimson Pirate (1952) 11
Treasure Island (1950) 27 m2
7 Das Boot (1981) 4
Cutthroat Island (1995) 66
8 The Spanish Kain (1945) 2
The Adventures of Robia flood (1938) I 120
9 Ostthroat Tsland (1995) 2
The crimson Pirate (1952]
16 The Adventures of Robin Rood (1938)
The Vikings (1958) | 24
The Vikinga (1958) 2
Run Silent Run Deep (1958) 27
12 Silent Sun Ueep (1958) 11
The Spanish Main (1945)
13 Black Sails (2014)
slack Sails (2014) 103
Captain Phillips (2013)
Captain Phillips (2013) 183
The Caine Motiny (1954)
Cafne Muziny (1954) 59
Greyhound (2020)
Greyhound (2020)
Captains Courageous (1927)
captains Courageous (1937)
20,000 Leagues Under sea (1954)
20, Leagues
(1954) |
19 In the Deart of the sea (2015)
In the lleart the Sea (2015)
20 The Nreck of the Eary Deare (1953)
The Wzeck the Mary Deare (1959)
17

---

Published as a conference paper at ICLR 2026
B GRADIENT ANALYSIS FOR GRPO, GSPO AND RANK-GRPO
In this section, we provide the full policy gradient derivations for GRPO Shao et al. (Zeeng al. 2025), and our proposed Rank-GRPO. The clipping term from the Ptyle objective is omitted for brevity. All derivations leverage the log-derivative trick (Sutton et al. 1999). which states that for function f(0), its gradient can be expressed (of(@) f(JV 9 log f(8). This allows us to rewrite the gradient of the objective in a form suitable for sampling- based estimation.
2024 GSPO
(i) GRPO. The gradient of the GRPO objective is derived with the following steps. We start by bringing the gradient operator inside the expectation based on the interchangeability of integration and differentiation. Then we apply the gradient the terms dependent on 0 (which is only the importance ratio w (0)), and then use the log-derivative trick as follows:
G
1
VoJGRPO = EgEnS
wi (O)
= Enows
Σ
A V u
ludl
1
ETO
Arear (0)Vo log (s))
(Log-derivative trick)
G
134
iml
lod1
1
1
(O)AA) Vo log To (v vice))
EEroas
G
1/4
G
lull
α Eroas
(
AS Sequeeee advantage
Vo log (o vi,ee)
token (t)-level gradient log-Rikdlhhodd
importance ratio
(14)
From Eq. 14), we see that GRPO updates the token-level policy using sequence-level rewards. This introduces two sources of misalignment: (i) the importance ratio defined at the token level, while the reward is defined at the sequence level, conflicting with the principle of importance sampling (Zheng et al., 2025); and (ii) the reward signal is too coarse for ranking tasks, whereas token-level updates are overly fine-graine -me-grrined. Together, these mismatches make the vanilla GRPO algorithm fundamentally ill-suited for reinforcement learning on rank-structured outputs.
(ii) GSPO. Different from GRPO that combines sequence-level rewards with token-level updates, group sequence policy optimization (GSPO) defines both the reward and the importance ratio the sequence level. Its derivation largely follows Eq. (4) follows:
8 Taspo (0) V VORET
(()) A
Brrews
G
8: (0) A, Vo o log 82 (0)))
E Erra
(Log-dcrivative trick)
G
Ivil
1
Vo log To (Br,o Biec))
(15)
EEroy
8: (0)A,,
G
[
1=1
G
(0)
A
V. o log To vi,e))
yil
sequence (6)-eevel importance ratio
sequenee (53-eevel advantage
sequence (i--evve gradient of Iog-likelitood
1/ly:
is the length-normalized sequence importance ratio. From Eq. 15) we observe that GSPO aligns the importance ratio with sequence-level rewards, which has been shown to stabilize training in mixture-of-expeere models such as Qwen3 series (Yang et al.
where (0) =
[plgg Te (3r [x)
2025) where different experts may be activated between the roll-out and current policies. However, both the reward and importance ratio remain too coarse for stuu-struttured tasks, where each rank should naturally serve as the action unit rather than the entire sequence.
18

---

Published as a conference paper at ICLR 2026
(iii) Rank-GRPO. Finally, for the Lank-GRPO proposed in this paper, the objective, advantage. and importance ratio are all defined at the rank level, where the gradient can be derived as follows:
C
N
G
N
Vo J.Jaak--RFPF (0) ())= V VeEnse
Wss (8)x(0)AA..
= Exosu
A Nowso (0)
GN
GN
i=1
Jmm
i=1 Am
N
E Ex
Wi, (0) ( Volog
(Log-derivative trick)
GN
i= ==k==
N
1i [
GN
G
N
(O) A Vo log to (v: (k) 12)
Wii
N
G
N
Ereas
ΣΣ
WiA (0)
A
Vo log na (Vr Meckyre)
N Nl
rank (F:)-evol importance ratio
rank (k-)-lovel advantage
rank (ke --leell gradient Ing-Iheilhood
(16)
Here, (0) nb
denotes the rank-level importance ratio, and 7e the length-normalized
# [z]
item probability, computed as the geometric mcan its token probabilities. From Eq. (16), we see that fundamentally resolves the shortcomings of GRPO and GSPO for rank-structured outputs. By treating each rank as the action unit, it provides a more natural granularity for both credit assignment and importance weighting, leading more stable and task-aligned optimization.
C IMPLEMENTATION DETAILS
(i) Training Setup. All models are trained on a single node via full-parameter fine-tuning. The smaller models (Qwen2.5-0.5B-Instme
were trained on single node with four NVIDIA H100 GPUs, each with 80GB of memory. The larger (Jamm-3.2-33-Inns- model was trained on four NVIDIA H200 GPUs, each with 141GB of memory.
-lama-.2-II--nnsttee
(ii) Optimization and Efficiency. To optimize GPU utilization and training speed. we adopt several standard techniques. We use DeepSpeed (Rajbhandari al. (2220) with the ZeRO-3 offload strategy for distributed training. We also enable gradient checkpointing (Chen et al. 2016)) FlashAftention2 (Dao, 2023), and 6 mixed-precision training to improve computational efficiency. The effective
batch size for most experiments is set to 384, which is calculated as follows:
batch size #GPUs x Per GPU batch size x Gradient Accumulation Steps
(17)
(ii)) Hyperparameters for the SFT stage. For the SFT stage, with a cosine learning rate scheduler that includes a 5% warm-up period.
learning rate of 5 10-5
(iv) Hyperparameters for the RL stage. For the RL stage, first tune the learning rate and KL weights for the GRPO baseline and apply the same hyperparameters to Rank-GRPO to ensure fair comparison that slightly favors the baseline. For the on-policy experiments (Section 3.4). we use learning rate of x 10-6 for the 0.5B model, and x 10-6 with decay to 1 x 10-7 for the 1B and 3B models prevent collapse due larger size. For the off-policy experiments (Section 3.4 with = 2). we keep the same learning rate for the 0.5B model but halve it to 5 x 10 with a decay 10-8 for the 3B models to account for the instability of importance weighting. In addition, we found that training is comparatively less stable for the 1B model (which we attribute to the base model itself as training for both 0.5B and 3B backbones are stable), so we adopt a finer-grained schedule 5 x 10 + 2.5 x 10-7 + 10 5 10-8 to prevent divergence. The KL coefficient is set to 0.001. The clip range Clow and Chigh is set to 0.2 and 0.26 for GRPO and 0.06 and 0.08 for Bank-GRPO to ensure similar expected clipping range after applying the wise geometric mean and effective rank probabilities. The clip range for GSPO is set to 3e - and 4e recommended by the paper (Zheng al. 2025). For Rank-GRPO. both 6, and e, set to -
19

---

Published as a conference paper at ICLR 2026
(V) Inference and Generation. For efficient LLM rollouts during the RL stage, vLLM Kwon et al. 2023) with a tensor parallel size of 2 for the 0.5B and 1B models and for the 3B model. We set the GPU memory utilization ratio to 0.5, All rollouts are generated with both temperature and top-p (for nucleus sampling) set to 1.0, respectively.
D ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
D.1 PROMPT USED TO GENERATE RECOMMENDATIONS
Following He et al. (023) and Zhu et al. (025). we fix the number required recommendations to N 20 and use consistent prompt across all experiments elicit item recommendation lists from the LLMs. The prompt explicitly instructs the model to output standardized English movie titles with release years in line-by-line format, ensuring uniformity and facilitating evaluation.
Prompt to Generate Recommendations
Pretend are movie recommender system.
conversation between user and you recommender aystem). Based on the conversation, you need to reply with 20 recomeedattinns.
I will give
List the standardized English title of each movie in each line in the form of "movie name" (ralease_vear) with
words sentences.
conversation: (context)
D.2 BASELINES USED IN THE MAIN PAPER
Redial (Li al. 2018 employs denoising autoencoder model user-mentioned items and generate recommendations while simultancously using recurrent neural network (RNN)-based architecture model and produce conversational utterances.
KBRD (Chen et al., 2019) augments CRS with external knowledge graphs. applies rela- tional graph neural network (RGNN) over the DBpedia knowledge graph to encode entities and maximizes the similarity between co-occurring words and entities, thereby fusing semantic and knowledge-level signals for recommendation.
KGSF (Zhou et al., 2020) further extends nnowledge-enhancee CRS by integrating a word- KG (from ConceptNet) into conversations. It leverages mutual information maximization between conversational semantics and entity KG embeddings to fuse entity information at the word level.
UniCRS (Wang et al. 2022) introduces a unified pre-trained transformer backbone to capture dialogue context, with cross-attention mechanism over entity embeddings learned from RGNN. This enables the model to fuse knowledge graph information with conversational semantics.
Zero-shot LLM (He et al. 2023) applies large language models directly to CRS. Dialogues are input with task-specific prompts and formatting instructions elicit recommendation lists, but without grounding to catalog or augmenting with external retrieval.
NBCRS (Xie et al. 2024) uses Sentence-Bril BERT to retrieve training conversations similar to given test conversation and takes the majority vote the groundtruth items as recommendations. Neighbor size is selected based on the validation set.
CRAG (Zhu et al. 2025) retrieves collaborative knowledge based mentioned items and his- torical user-item interactions. Specifically, it introduces a two-step reflection mechanism with LLM-ae-a-judge refine the retrieved items and the final ranking.
D.3 ADDITIONAL COMPARISON WITH RL-FREE POST- TRAINING BASED METHODS
In this part, compare Rank-GRPO with several RL-free post-training methods for LLaeb recommender systems based on direct preference optimization (DPO) (Rafailov et al. 2023 DPO assumes access pairwise preferences over responses to the prompt and shows that, under Bradley-Terry model (Bradley & Terry [952) for the latent reward, a simple maximum likellhood objective can be derived to directly optimize the policy me from preference data. Due to the pairwise
20

---

Published as a conference paper at ICLR 2026
Table 3: Comparison between ConvRec-RI and various DPO-based RL-ffree preference alignment baselines with different backbone models on the REDDIT-v2 dataset.
R@10
R@15
R@:20
Nēs
N@
Qwen2.540.5B (zero-shot)
0.0021
0.0021
0.0021
0,0021
0,0023
0.0022
0.0021
0.0021
SPT (DPO) DPO
0,0716
0.1058
0.1300
0.1473
0.0570
0.0681
0.0752
0.0797
SFT (DPO)+ S-DPO
0.0875
0.1240
0.1488
0,1699
0.0657
0.0772
0.0845
0.0888
(DPO) SPRce
0.0841
0.1256
0.1524
0.1686
0.0657
0.0775
0.0898
Rack-GRPO (exp
0.0916
0.1468
0.1813
0.2047
0.0744
0.0915
0.1016
Lisma-332-44 (2cro-shot)
0,0187
0.0240
0,0257
0,0263
0.0157
0.0170
0.0175
0.0176
SFT (DPO) DPO
0.0823
0.1158
0.1437
0.1636
0.0646
0.0752
0.0834
0.0887
SIT (DPO) S-DPO
0.0904
0.1307
0,1588
0.1660
0.0718
0.0829
0.0926
0.0987
SPT (DPO) SPRec
0.0928
0.1375
0.1591
1889
0.0758
0.0887
0.0953
0.1050
Rank-GRPO (exp
0.1039
0.1569
0.1957
0.2165
0.0813
0.0985
0.1092
0.1153
I 1ama-332-38 (zero-shot)
0,0574
0.0808
0.0912
0.0961
0.0463
0.0538
0,0566
0.0580
SFT (DPO) DPO
0.0894
0.1279
0,1491
0.1697
0.0708
0.0808
0.0915
0.0896
SFT (DPO) S-DPO
0,1072
0,1419
0,1734
0.1943
0.0807
0.0933
0.1019
0,1079
SUT (DPO) SPRec
0,1053
0.1491
0,1809
0.1965
0.0828
0.0956
0.1043
0.1109
SPT CRPO (expoc)
0.1178
0.1756
0.2155
0.2368
0.0919
0.07
0.1223
0.1283
comparison nature of DPO (where it is difficult to define preference between two ranked lists), existing DPO variants for recommendation are primarily designed for candidate-selection scenarios, i.e., choosing positive items from a small candidate set given in the prompt (Wu et al., 2025). In contrast, Rank-GRPO targets a more generative setting, where a ranked list N items is generated directly by the L..M-based policy based on the conversational context, removing the need for an explicit retriever. In addition to vanilla DPO, the DPO-based baselines considered in this paper are:
S-DPO (Chen et al.
2024) extends DPO to multiple negatives in the candidate set and optimizes a softmax- based DPO loss over one preferred item and several dispreferred items.
2025 builds on S-DPO with self-play procedure that alternates SFT on pos- itive items and DPO steps where negatives are drawn from the model's own previous predictions.
SPRec (Gao et al.
During training, we first perform an SFT (DPO) phase, where we modify the prompt used in Rank- GRPO (see Section D.I to request only single recommendation, and use the ground-truth items as supervision. We find this setup substantially improves performance for DPO-based methods compared to directly using ranked-list data for SFT (as with ConvRec-R1), To adapt vanilla DPO and S-DPO to the generative ranking setting studied in this paper, we follow SPRec (Gao et al.) 2025) and adopt BIGRec (Bao al., 2025) score items and induce a full ranking over the catalog. Even though we cache shared prefix for the conversational context, this pipeline is still slower than Ron,Rec-RI, as it needs score large number of catalog items. Empirically, as shown in Table 3 both vanilla DPO and its extensions (S-DPO and SPRec) significantly improve upon the zero-shot model. However, there remains gap between these adapted DPO variants and Rank- GRPO. This highlights the advantage of Rank-GRPO for generative ranking: the model directly produces a ranked list by jointly considering the context and previously generated items, rather than relying on post-hoc ranking from per-item scores. At the same time, we emphasize that DPO and S- DPO are not originally designed for generative ranking tasks; they remain strong RL-free basclines when a high-quality retriever is available to propose compact candidate set in the prompt.
D.4 ADDITIONAL RESULTS FOR THE LLAMA3 2- 1B MODEL
In this section, we provide the training dynamics of ConvRec-R R1 with 10ama3.2-1B-Instruee model as the backbone under the on/off-policy setting, as illustrated in Figs. 5 The results mirror the observations from smaller and larger backbones in Section 3.4 (i) training rewards increase con- sistently across ranks, confirming that Rank-GRPO enables more stable credit assignment even in mid-sized models; (ii) validation NDCG improves more rapidly than with vanilla GRPO, indicating that the rank-level importance weighting contributes to faster convergence; and (ii) improvements accumulate particularly toward the tail positions of the list, where methods with sequence-level
21

---

Published as a conference paper at ICLR 2026
Rank-GRPO (log-decay)
Rank-GRPO (exp.)
Validation NDCG
0.54
0.28
T0NC2
0.12
0.22
0.18
0.10
0.10
0.04
0.02
0.08
-3.02
2000
4000
6003
8000 10060 12000 14000
-0.02
2000
4000
0000
0000
10000
12500
1400
2000
4000
6000
3000
10300
12200 14000
Step
Step
Step
Figure 4: Training dynamics of ConvRec-RI with Llama3.2- 1-- backbone (on-policy).
Ranl-GRPO (log decay)
Rank-ORPO
Validation NDCO
30
1.16
Tame-5hi0 Jeo
Varia Chtn
0.65-
0:00
Step
Step
Figure 5: Comparison of training dynamics of reward and validation NDCG between GRPO, GSPO and Rank-GRPO (off- (off-policy) with Gaama3.2-1BIInstree backbone on the REDDIT-V2 dataset.
ward typically degrade. These dynamics highlight that the advantages of Rank-GRPO generalize across backbone sizes, striking balance between training efficiency and model capacity.
D.5 TEST SET EVALUATION RESULTS FOR THE OFF-POLICY SETTING
Table presents the test set performance of ConvRec-RI and baselines under the off-policy setting (4 2). As expected, all methods perform worse than their on-policy counterparts in Table ??, reflecting the additional variance introduced by importance weighting. Nevertheless, Rank-GRPO consistently outperforms both vanilla GRPO and GSPO across nearly all metrics, delivering the most substantial improvements over the SFT baseline. These results highlight that, although off-policy training is inherently more challenging for RL-based post-training the use of rank-level impor- tance weighting enables more stable and effective updates than sequence- or token-level approaches. Consequently, ConvRec-RI maintains strong recommendation quality (compared with GPT-4o and GPT-40-mini) while benefiting from the efficiency of reusing previously sampled trajectories.
D.6 ADDITIONAL. ANALYSIS CATALOG MEMORIZATION DURING THE RL SETTING
To further evaluate the impact of RL on catalog grounding, we track the catalog hit ratio on the validation set throughout training (Figure (6) We find that the recommendations from both GRPO and Rank-GRPO gradually drift away from the catalog as training progresses, due to the absence of direct catalog grounding as in the SFT stage. However, the drift is noticeably slower for Rank-GRPO, since its rank-level return explicitly assigns zero reward in- in-oosition to out- of-catalog items, whereas GRPO struggles with proper credit assignment with the sequence-level reward. An additional artifact is observed when comparing backbone sizes: the 0.5B model exhibits stronger fluctuations in the in-catalog ratio, while the 3B model yields a smoother curve. This dif- ference stems from our stabilization strategy: on larger models (1B and 3B), we reduce the learning rate in the second epoch to prevent vanilla GRPO from collapsing (we use the same learning rate on for fairness of comparison), whereas the 0.5B model remains stable with 1 10 8
Oves 15terratt
ship
Figure 6: Dynamics of percentage of in-catalog rec- ommendations on the REDDIT-V2 validation set.
D.7 ADDITIONAL RESULTS ON THE REDIAL DATASET
In this section, we present experiments on the REDIAL dataset (Li et al. 2018). Compared to the REDDIT-V2 dataset reported in the main paper, the REDIAL dataset is smaller in scale, with fewer conversations and a smaller movie catalog. It consists of multi-turn dialogues between seeker and a recommender, where the seeker describes their preferences and the recommender suggests movies in natural language. We follow the standard split protocol: 80% for training (20,896 conversations),
22

---

Published as a conference paper at ICLR 2026
Table 4: Comparison between ConvRec-RI and various baselines the REDDIT-V2 dataset in the off- ouf-ppolcy setting. Here, @@k and @@ stand for Recall @ and NDCG@ respectively.
Method
R@S
R
R@15
R 20
N@s
N@
N@15
No: 20
Traditional and Transformer- CRS
Redial
2018
0,0103
0.0228
0.0274
0.0322
0.0073
0.0113
0.0128
0.0143
KBRD
ben et al
2019
0,0444
0.0813
0.1058
0.1223
0.0305
0.0418
0.0490
0.0545
KGSF
hon
2020
0.0579
0.0921
0.1246
0. 1438
0.0405
0.0503
0.0899
0.0662
UniCRS
Wang gl.
2022
0.0722
0.1053
0.1344
0. 0.1494
0.0548
0.0640
0.0726
0.0778
NBCRS
Xie al. 2024
0.0801
0.1290
0.1655
0,2019
0.0661
0.0833
0.0954
0.1048
L.M Prompting-Bosee
Methods
GPT-to-mini (aero-shol)
0,0949
0.1348
0.1600
0.1687
0.0747
0.0877
0.0950
0.0973
GPT-4o 4o shoro Ho et
2023
0.1106
0.1625
0.1992
0.2147
0.0861
0.1028
0.1136
0.1197
CRAG Zhu al.
0.1146
0.1715
0.2030
0.2212
0.0885
0.1065
0.1164
0.1227
L.M Pod- Truining-Baaar Bused Methods (off-policy,
Most results are signifficntt is standard error are hetwenn 0.002000.0055 for R 0.000 5-0000000000 for N885-20
Qwen2.5-0.5R (zem-shot)
0.0021
0.002.
0.0021
0.0021
0.0023
0.0022
0.0021
0.0021
+ SFT
0.0642
0.1027
0.1212
0.1308
0.0502
0.0625
0.0678
0.0704
SFT+VVnilla GRPO
0,0781
0.1176
0.1429
0.1538
0.0623
0.0748
0.0823
0.0852
SITT+ GSPO
0,0816
0.1243
0.1511
0.1641
0.0647
0.0784
0.0862
0.0896
STTT-Rank GRPO (log)
0,0818
0.1256
0.1604
0.1764
0.0644
0.0784
0.0886
0.0930
SFT-Rank-GRPO (exp.>)
0.0935
0.1412
0.1738
0.1999
0.0735
0.0887
0.0982
0.1025
1lana-3.2-IR (zero-shot)
0.0187
0.0240
0.0257
0.0263
0.0157
0.0170
0.0175
0.01
+ SFT
0.0754
0.1148
0.1354
0.1998
0.0595
0.0723
0.0782
0.0819
+ SFT+Vanilla GRPO
0.0937
0.1415
0.1726
0.1974
0.0728
0.0882
0.0975
0. 0.1042
SIT++GSPO
0.0901
0.1389
0.1693
0.1899
0.0700
0.0857
0.0947
0.1002
SFFT-Rank-GRR GRPO (log)
0.0937
0.1464
0.1832
0.2066
0.0731
0.0902
0.1011
0.1074
SSPT-Rank-GRO (exp...
0.1037
0.1555
0.1927
0.2166
0.0806
0.0973
0.1081
0.1146
Llama-3.2-3B (2cro-shot)
0.0574
0.0805
0.0912
0.0961
0.0463
0.0535
0.0566
0.0580
+ SFT
0.0788
0.1191
0.1410
0.1541
0.0615
0.0744
0.0807
0.0842
+ SFT+Vanilla GRPO
0.0952
0.1425
0.1726
0,1952
0.0766
0.0916
0.1004
0.1065
+ SFT+GSPO
0.0942
0.1436
0.1770
0,1943
0.0758
0.0914
0.1011
0.1058
SFFT-Rank-GRO (log)
0.1037
0.1572
0.1958
0.2220
0.0816
0.0989
0.1101
0.1169
J.FT-Rank-RRRO texp...
0.1092
0. 1656
0,2010
0.2272
0.0845
1028
0.1
0. 1203
NDCG@5 NDCG@20
In-Catalog Recommendation
Train
100
Validation
In-Catalog
Loxs
0.4
Score
0.10
0.2
40
0.05
NDOQ@S
20
Mean
NDCGg220
0.8
0.00
50
100
150
200
200
300
350
200
60
100
100
200
250
300
350
Step
Step
Owen//wen2255 5-0. In-Sruct
meta Liama 3.2 In-Instrucl
mela iamaa Inslrutt
Figure 7: Training dynamics on loss, validation metrics, validation in-catalog recommendation ratio with different backbone models during the SFT stage on the REDIAL dataset.
10% for validation (2,612), and 10% for testing (2,613) (He al. 2023). In addition, choose
a different teacher model, i.c., Claude-3.7 S.uude-37--Somaee 3.7-Snnnet, to demonstrate the generalization over different large teacher LLM generate the catalog-grounde behavior cloning dataset.
D.7.1 EVALUATION ON THE SET STAGE
As shown Fig. 7 as with the results on the REDDIT-V2 dataset (see Section 3.3). training loss decreases steadily, while validation loss quickly plateaus, reflecting the difficulty of direct imitation learning when the outputs from teachers are long, structured recommendation lists. At the same time, the in-catalog recommendation ratio rapidly approaches nearly 100% across all three backbones, indicating that the Reflect-Adjust pipeline plus SFT is effective at grounding the models in the Redial catalog. Finally, NDCG@ 20 the validation set improves substantially over the zero-
23

---

Published as a conference paper at ICLR 2026
Rank-GRPO (log)
Rank-ORPO (esp.)
Validation NDCG
Qwan2 5-0. 1-0.5B--m S-B-hstrruct Rewaard
0.2-
0.20
0.20
Dam 50F0 INc.1
0.15
Vania G4000
NDCG
0.20
0.10
0.05
0.00
0.10
40
20
120
1293
215
Rank-GRPO (log)
Rank-GRPO (exp.)
Validation NDCG
Liana- w.w.--2-----iae- Fa--nstruet
0.3
Reward
0.2
NDOG
0.20
0:10
211
0.07
0.15
0.0
0.00
0.10
050
809
1000
1200
200
600
973
1297
250
400
90
1089
12:0
Rank-GRPO (log)
Rank-GRPO (exp..)
Validation NDCG
Liana 3.2.BB..BBBB. 1.m.a--.----nnntnne instnce Reward
0.5
0.20
0.25
0.15
NDOG
0.00
0.10
Whis 0070
Figure 8: Left Middle: Training dynamics the reward acquired by ConvRec-RI with Rank- GRPO on different rank during the RL stage. Right: Comparison of validation NDCG between GRPO and Rank-GRPO (both on-policy) on the REDIAL dataset.
Table 5: Comparison between ConvRec-RI and various baselines on the REDIAL dataset in the on-policy setting. Here, @@ and N@ k stand for Recall @ k and NDCG@k respectively.
Method
R@5
R@10
15
R@20
N@S
N@
N5
N@
Traditional and Transformer-basee CRS
Redial
Li
2018
0.0178
0.0380
0.0203
0.0221
Chen
0.0763
0.0686
0.0778
0.0840
KGSF
Zbou et
2020
0.0995
0.1535
1.99
0.2235
0.0630
0.0826
0.0951
0,1020
UniCRS Wang ct
2022
0.1241
0.1756
0.2156
0,2330
0.0851
0.1050
0.1153
0,1199
NBCRS Xie ct al.
0.1848
0.2874
0.3687
0,4108
0.1228
0.1568
0.1788
0.1891
I.L Prompting- Rased Methods
Claude-3.7 (zero-shot)
2023
0.2088
0.3135
0.3717
0.1691
0.1849
0.1941
CRAG /hu 2025
0.2097
0.3810
0.1720
0.1976
LLM RL Post- Most-Training-Be Based Methods
Qwen2.5.0.5B (zero-shot)
0.0050
0.0050
0.0050
0.0050
0.0042
0.0042
0.0042
0.0042
.FT
0.1434
0,2570
0.3145
0.3512
0.1036
0.1345
0.1502
0,1590
VaTTaaanill GRPO
0.2114
0,3095
0.3633
0,3917
0.1431
0.1756
0.1902
0.1971
SSFT-Rmn-GRPO (log)
0.2251
0.3253
0.3885
0,4272
0.1531
0.1857
0.2029
0.2123
(exp,e)
0.2385
0.3542
0.1844
0.2252
Llama-3.2-1B (acro-sbot)
0.1036
0.1522
0.1672
0.1704
0.0657
0.0818
0.0859
0.0867
SFT
0.1714
0.2731
0.3345
0.3715
0.1078
0.1414
0,1580
0,1671
Vanillaillaa GRPO
0.2236
0,3296
0.3924
0.4196
0.1526
0.1876
0.2047
0,2114
+SFT+Rumk-GRPO (log)
0.2365
0,3392
0.4059
0.4472
0.1605
0.1942
0.2125
0.2226
+SFT-Rmi--GPOO (exP.2)
0.2516
0.3651
0.4255
0.4526
0.1714
0.2092
0.2252
0,2323
Llama- L.a.3-3.2.3B 3.2.3B
(sero-shot)
0.1473
0.2030
0.2284
0.2309
0.0972
0.1158
0.1228
0.1234
SFT
0.1756
0.2750
0.3392
0.3793
0.1114
0,1441
0.1615
0,1712
Vanilla GRPO
0.2282
0.3265
0.3936
0.4300
0.1574
0.1898
0.2080
0.2168
SET+Rank- SARUO--RPOOOOO (log)
0.2378
0.3391
0.4084
0.4471
0.1630
0.1964
0.2153
0.2247
S..T+Rakk GRPO (exp..)
0.2691
0.3740
0.4361
0.4648
0.1820
0.2167
0.2336
0.2405
shot models, confirming that SFT again provides a strong warm start for the subsequent RL stage by bestowing the backbone LLM with preliminary generative ranking abilities.
24

---

Published as a conference paper at ICLR 2026
Rank-GRPO (log)
Rank-GRPO (exp.)
Validation NDCG
Owent 1-4 14.088-0 ri--isttruu
2.0
0.2
NDOG
0:20
0.05
0:15
Ran--Giio (np
GRRO
0.5
0.00
0:10
124
190
1990
Stop
Step
Stop
Rank-GRPO Bog!
Rank-GRPO jaxp..
Validation NDCG
120
0.2
Saano-.21BB-miii Roware
189
Roward
MDCO
0:20
0.05
0.00
1248
100
120
104
Rank-GRPO (log)
Rank-GRPO jexp.]
Validation NDCG
020
0.25
Reward
2.15
NDC0
0:20
0:12
1190
0.00
0:10
Ram-GR00 Ing
Step
Stop
Stop
Figure 9: Left Middle: Training dynamics of the reward acquired by ConvRec-RI with Rank- GRPO on different rank during the RL stage. Right: Comparison of validation NDCG between GRPO and Rank-GRPO (off-policy) on the REDIAL dataset.
D.7.2 EVALUATION ON THE RL STAGE
On the RL stage, we also observe similar trends the results on REDDIT-V2 (Section 3.4 In
the on-policy setting (left and middle panels of Fig. (). the rank-wise rewards under Rank-GRPO increase across ranks, indicating the ability of ConvRec-R1 to improve the entire recommendation list. The right panel shows that Rank-GRPO consistently converges to higher validation NDCG than vanilla GRPO for all three backbones, with the gains being more pronounced at larger k, again suggesting that Rank-GRPO is particularly effective maintaining quality toward the tail of the list. In the off-policy setting with per-sampling update step μ 2 (see Fig. (9). Rank-GRPO also compare favorably over GRPO and GSPO in terms of validation NDCG. The off-po curves lag slightly behind their on-policy counterparts due to the additional variance introduced by importance weighting, but they still achieve clear improvements over the SFT and vanilla GRPO baselines.
show that on REDIAL., ConyRec-R again substantially improves over both zero-shot and SFT-only backbones, and consistently outperforms vanilla GRPO (and GSPO in the off-policy case) across backbones and metrics. The Llama-3.2-3B backbone with Rank-GRPO (exp.<) achieves the best overall performance, surpassing both zero-shot Claude-3.7 and a RAG- based LLM system composed of Claude-3.7 with substantially less inference cost.
Furthermore, Tables show
D.7.3 SENSITIVITY W. REWARD SHAPING HYPERPARAMETER
In this part, we examine the influence of the reward- shaping hyperparameter introduced in Section 2.3.3 Recall that we add per-token penalty 6 0 to enforce instruction following: premature <eos> to- kens that stop generation before N items are pro- duced receive a penalty, and any overflow items be- yond rank are also penalized. We sweep e, (0, (0.0 00.1, -0.5, - and train Rank-GRPO backbone:
0.45
Decall
0.15
0.00
5
200
25
20
Top-
6--1 -
(exp.e) with the Swve2.5-0.5B-IInstee the resulting Recall @ K curves shown in Fig. I0
Sensitivity of Recall@k for Rank- GRPO (expsc) to with Sivem2.5-0.5B-Instruet
Figure 10:
When no penalty is applied (c, = 0), performance is noticeably worse, reflecting the fact that fail- ures to respect the required list length not corrected by the reward signal. In contrast, once
25

---

Published as a conference paper at ICLR 2026
Table 6: Comparison between ConvRec-RI and various baselines in the off-policy setting on the REDIAL dataset. Here, @@ k and N@ k stand for Recall @K k and NDCG@ k. respectively.
Method
R@5
R 10
R@15
R 20
N@5
N@
N@15
N60
LLM Post- Meee-rraining-Bane Based Methods (off-policy, = 2)
Owen2.5-0.5B (zero-shot)
0.0050
0,0050
0.0050
0.0050
0.0042
0.0042
0.0012
0.0042
+ SFT
0.1634
0.2570
0.3145
0.3512
0.1036
0.1345
0.1502
0.1590
SFT+Vanilla GRPO
0.2095
0,3005
0.3506
0.3829
0,1422
0.1720
0.1857
0.1936
SFT+GSPO
0.2056
0.3005
0.3536
0,3841
0.1379
0,1692
0.1837
0.1911
SFT+Raak-GRRO (log)
0.2140
0.3143
0.3694
0,4038
0,1470
0.1801
0.1950
0.2034
SSTT+Rank-GRR (exp..)
0,2363
0.3399
0.3993
0.4350
0,1614
0.1956
0.2118
0.2205
Llama-3.2-1B (zero-shot)
0.1036
0.1522
0.1672
0.1704
0.0657
0,0818
0.0859
0.0867
+ SFT
0.1714
0.2731
0.3345
0.3715
0.1078
0.1414
0.1580
0.1671
+ SIT+VVnilla GRPO
0.2218
0.3255
0.3790
0.4143
0.1495
0,1835
0.1981
0.2066
SFT+GSPO
0.2156
0.3102
0.3705
0,4157
0,1438
0,1748
0.1911
0.2022
SFT+Raak-GRPO (log)
0.2249
0.3277
0.3910
0,4345
0,1535
0,1874
0.2046
0.2153
SFT+Rank-GRPO (exp
0.2479
0.3562
0.4225
0,4429
0.1659
0.2020
0.2200
0.2251
1Jama-3.2-38 (zero-shor)
0,1473
0.2030
0,2284
0.2309
0.0972
0.1158
0.1228
0.1234
+ SFT
0.1756
0.2750
0,3392
0.3793
0.1114
0.1441
0.1615
0.1712
SITT+ Vanillai GRPO
0.2237
0.3228
0.3895
0.4268
0.1546
0,1872
0.2053
0.2144
SIT+GSSPO
0,2242
0.3198
0.3781
0.4188
0,1535
0,1853
0.2012
0.2111
SFT+Rank-GRPO (log)
0.2327
0.3404
0.3997
0.4424
0.1620
0.1976
0.2137
0.2240
SFT+Rank-GRPO (exp
0.2648
0.3713
0.4532
0.4610
0.1792
0.2148
0.2332
0.2385
Table 7: Comparison between ConvRec-RI R1 and various baselines on the REDIAL dataset with
Qwen2.5-7B-Insstee backbone. R@ l and N@ stand for Recall@ k and NDCG@ k, respectively.
Method
R@5
R@:10
R@15
R@020
N@5
N@
N@15
N60
Qwen-2.5-7B (zcro-sbot)
0.1272
0.1913
0.2175
0.2217
0.0795
0.1006
0.1078
0.1088
SFT
0,1807
0.2723
0,3342
0,3703
0.1164
0,1465
0.1636
0.1723
SFT+Vanilla GRPO
0.2286
0.3288
0,3753
0.4207
0,1516
0,1879
0,2019
0.2095
SFT-Rank-GRPO (log)
0,2415
0.3491
0.4012
0.4495
0.1646
0.1986
0.2176
0.2245
S.TT+ (exp...)
0.2723
0.3764
0.4427
0.4745
0,1811
0.2158
0.2338
0.2416
any negative penalty is introduced, the performance first increases and then decreases when co de- creases from -0.01 to 1, but the fluctuation is not very significant. This indicates that performance of Rank-GRPP is quite robust to the exact magnitude of the penalty, with only mild fluctuations.
D.7.4 GENERALIZATION TO LARGER MODELS
Although in practice we expect models up to 3B parameters to be the most suitable for real-world deployment of CRS due to latency and memory constraints, also conduct an exploratory study with a larger backbone to assess the scalability of ConvRec-RI and Rank-GRPO. Specifically, Ta- ble reports additional results on the REDIAL dataset with the Qwen2.5-7B-Innstuec backbone. Training this 7B model on the full REDDIT-V2 dataset (~400k conversations) is beyond our cur- rent computational budget, observe trends that are consistent with those of smaller models: SFT provides a strong improve- ment over the zero-shot baseline, and Rank-GRPO further boosts the ranking ability of the LLM. In particular, the Qwen2.5-78 7---7-IIIntttt + Rank-GRPO (exp.e) model slightly surpasses the L1ama-3.2- 3B-Instruct counterpart on most metries (see Table These findings suggest that ConvRee-RI and Rank-GRPO generalize well to larger backbones when computational resources permit.
only experiment the smaller REDIAL benchmark. Notebly,
E DISCUSSION ON GENERALIZATION OF CONVREC-RI
There are three main datasets widely used in CRS research, i.e., REDDIT-V2 He et al. 2023 Zhu et al. 2025), REDIAL (Li et al. 2018), and INSPIRE (Hayati al. 2020). all focused movie recommendations. We exclude INSPIRE from experiments because it contains only 1,000 conver- sations, which is insufficient reliably fine-tune L.MSs with billions of parameters. An advantage
26

---

Published as a conference paper at ICLR 2026
of the movie recommendation setting is that each item (movie) naturally comes with short textual representation (c.g.. title with release year), which can be understood by the LLM and directly used as the item tokens. To generalize ConvRec-RI to other domains such e-commerce
video recommendation, we can either use pretrained textual tokens (Hou et al. 2024 Zhang et al. (025a) introduce new special tokens (Zhu et al. 2024b Hua et al. 2023)
codebook to represent
items. However, since not all arbitrary token combinations in the codebook correspond to valid items, the proposed SFT stage with reflect-rank-adius can still be applied teach the LLM cata- log awareness and to endow it with preliminary generative ranking ability over predefined item catalog. The subsequent RL stage with Kank-GRPO can then be applied to utilize the verifiable, rank-wise rewards defined over the generated items to align the generations with user preferences.
27

---

Published as a conference paper at ICLR 2026
F QUALITATIVE EXAMPLES
Comparison between Zero-Shot and SFT/RL stage of ConvRec-RI with Qwen2.5-0.5B
(3) Conversation "t?"
User:
Rest 36 movies ee to watch?. What are the best 36 movies to watch on my vr headser
texcept Avacar and mou movies)? ? wart the 3d effects to actually be good and not some cheap garbage 3d processing some studios spply to their novies. I also want the 3d to actually be innersive and cool. Novies I have already seen in 3d are: Avatar, avengers, godzilla, star ward the Force аиažena, pixels, terminator geneays and jurasase world.
Judgment Day (1991), Rugo (20'1), The Walk (2015), Titanie (199%), Beady Player (2318), Fredator (19871, Jurassic Park (1992), Max: Fuzy (2011), Gravity (2013),
(41) Groundtruth v!":
[2015], Kong:
(2012),
(201)),
(2013)
(111) Outputa Qwen2 5B:
Rk..
Zero-Shot Raw Output
Processed Rec. (NDOG820=0.000
Avengers
Avengers
Godzilla
Godzilla
Star Mara: Force Awakens
The Force Awakens
Pixala
Terninater: Genesys
Teroinators Genesya
Jurassic World
Jurasaic World
83.
(b) After 8PT (Checkpoint-1500) Raw Output
Processed Rec.
(NDCG)20=D, 0.249 A
Guardians Galaxy
Guardians Galaxy (2014)
Max: Fury Read (2315)
Mad Fury Road (2015)
Jungle (2016)
Jungle (2016)
Incerssellat (2014)
Interstellac (2011)
Gravity
Gravicy (2013)
Tooeptioon (2013)
Troept (2010)
The Matrix (1993)
The Matrix (1939)
8 Ready Player Cne (2018)
Ready Player One (2018)
9
Oblivion (2013)
Oblivion (2013)
10 The Dark Knight Rises (2012)
The Dark Knight Rises (2012)
the Adventures of Tintin (2011)
The Adventures of Tintin (2011)
22 Tron: Legacy (2013)
Tron: Legacy (2015)
23 Vaierian and city
Thousand Flanets (2011)
Va-erian and City et Thousand Flanets (201
The Firth ziement (1997)
The Firth Element (1997)
Jumanji: Welcome the Jungle (2017)
Junangi: Kelcome to the Jungla (2017)
16 The laat of Us (2013)
The Last of Ua (2313)
Hobbit: Unexpested Journey (2012)
The Bobbit: Unexpested
(2012)
(c) (Checkpoint-158800 Output
Processed Rec. (NDCC)20=D. 444,
Med Max: Zury Read (2015)
Mad Mox: Pury Road (2015)
2 Insers:ellor (2014)
Interstellar (2014)
3 The Matrix (1999)
The Matrix (1989)
4 Blade Runner 2049 (2017)
Blade Runner 2049 (2017)
5 Gravity (2013)
Gravity (2013)
5 Incepticn (2013)
Inception (2010)
Ready Player One (2018)
Ready player One (2018)
8 Buardians of the Galaxy (2014)
Guardians of the Galaxy (2014)
9 Alita: Battle Angel (2019)
Alita: Battle Angel (2019)
Tron: Legacy (2913)
Tron: Legacy (2012)
Firth Zlement (1997)
The Pirth Blament
Ready
Ready Not
Vaterian and
Thousand Planets
arian City of Thousand Planeta (2017)
Edge Tomorzok (2014)
Edge Toncrreи (2014)
(2013)
Oblivien (2013)
Dark Raight Rises (2012)
The Snigat (2012)
Dunkirk (2017)
Dunkirk (2017)
28

---

Published as a conference paper at ICLR 2026
Comparison between Zero-Shot and SFT/RL stage of ConvRec-R1 with Llama-3.2-1B
(4) Conversation =
User:
Best 36 movies to watch?. What are the best 3d mevier to watch on my vr headse:
texcept Avanar and nou movies)? wart the 3d efrects to actually be good and not some cheap ge rarbage 3d processing some studios apply to their novies. I also want the 3d to actually be Innersive and cool. Bovies I have sIresdy seen in 3d ares Avatar, avengers, godeilla, star wers the force awakens, pixels, terminator geneeys and jurassie world.
(41) Groundzruth v.:: "erminator Judgment bay (1991), Rugo (2011), The Walx (2015), Titanie (1997), Ready Player (2318), Predato" (19871, Juresa'e Park (1993), (2015), Kong: Skull Island (2017), Gravity 12013), Life of (2012), (2013)
Fory Road 12017),
(111) Outputa Llama-3.2-18
Rk. (a) Zero-Shot Raw Output
Processed Rec.
(NDC6220=0 (0000
Inception (2919)
Inception (2010)
2 Interatellar (2014)
Interatellar (2014)
3 The Natrix (1939)
The Natrix (1939)
4 The Prestige (2006)
The Prestige (2006)
Lord the Rings
The Lord the Rings
Rnight (2008)
The Rnight (2508)
Tdentity (2002)
Identity (2002)
8-20 (repeats Prestige)
(repeats Prestige)
(b)
(Checkpoint-1500) Output
Processed Rec.
(NDCGg20=D.
Blade Bunner 2049 (2017)
2049 (2017)
2 Inception (2010)
Inception (2010)
3 Gravizy (2013)
Gravity (2013)
4 The Natrix (1999)
The Hatrix (1999)
5 Interstellar (2014)
Interstellar (2014)
6 Dunkirk (2017)
Dunkick (2017)
7 Ready Player One (2018)
Ready Player One (2019)
8 Alita: Battle Angel (2019)
Alita: Battle Angel (2019)
9
Vaterian and City of Thousand Planets (2017)
Va erian and City of Thousand Planets 72017
10 Trons Legacy (2010)
Trons Legacy (2013)
Jupiter Ascending (2015)
Jupiter Ascending (2015)
Guardians of the Galaxy (2014)
Guardians of the Galaxy (2014)
Rogue
Star Wars Story (2016)
Rogue One:
Story
The Pinal (2034)
The Pinal the Tnferno
Green Tnferno (2013)
Thirteenta
tor tae (1999)
The Tairteenta Flooz (1999)
Apes (2017)
Mar tor tae Planet Apes (201')
the Rings (1978)
The the Ringa (1978)
(c) (Checkpoint-15800) Output
Rec.
(NDCOg20=D.11) 129,
Gravicy (2013)
Gravity (2013)
2 Blade Runner 2049 (2017)
Blade Runner 2049 (2017)
3 The Matrix (1999)
The Matrix (1999)
4 Allta: Battle Angel (2019)
Allta: Battle Angel (2019)
5 Ready Player One (2018)
Ready Player One (2018)
6 Tron: Legacy (2010)
Tron: Legacy (2010)
7 Vaierian and City of Thourand Planets (2017)
Valerian and City of Thousand Flanets (2017)
8 Rogue One: Star Wars Story (2016)
Rogue One: A Star Wars story (2016)
Trceptiion fon (2010)
Irception (2010)
Guardians of the Galaxy (2014)
Guardians of the Galaxy (2014)
Intersteliar (2014)
Interstellar
(1975)
(1973)
Pacific Rim for Planet Apes (2017)
Pacific Rig
Planet Apes (2017)
Nightnare on
(1987)
Nigazmare
(1987)
Blade Runner (1982)
Blade Runner (1982)
(2312)
(2312)
Edge of Tonorrow (2014)
Edge of Toncerow (2011)
29

---

Published as a conference paper at ICLR 2026
Comparison between Zero-Shot and SFT/RL stage of ConvRec-R1 with Llama-3.2-3B
(4) Conversation =!t;
Best 24 movies to watch?. What are the best Ja mevies to watch on my vr headse: texcept Avazar and meu movies)? want the 3d effects co actually be good and not some cheap garbage 3d processing some studios apply to their novies. I alse want the 3d to actually be inmersive and coel. Novies I have slresdy seen in 3d are: Avazat, avengers, godzills, star wars the force auakens, pixels, terminato" geneays and jurasa'o world.
User:
(iá) Groundtruth v.t Cerminazor Jadgment Day (1991), lugoo (2011), Halk (2015), Titanic (1997), Ready Player (2310), Fredatoz (19671, Jurassic (1993), Max: Fury Road (2017), Gravity (2013), life (2012),
(2015), Kong:
2049 (2017),
(2013)
(T11) Outputa L1ama-3.2-38:
Output
(NDCG920=0.
Tnserscellar (2014)
Tnteratellar (2014)
2. Nartian (2015)
The Martian (2315)
3 Nax: Tury Read (2015)
Kad Nax: Pury Road (2015)
4 The Jungle Book (2316)
The Jungle Book (2016)
5 Pantaacic Beasta and Where to Find Then [2026] Fantastic Beasta and Where to Fird then (2026)
6
The RPG (2016)
The RPG (2016)
Dunkir's (2017)
Dunkirk (2017)
8 Nonder Koman (2017)
Nonder Koman (2917)
Justice Ieague (2917)
Justice Ieague (2017)
20 Ready Player One (2018)
Ready Player One (2018)
Mission: Impossible Fallout (2016)
Kission: Impossible Pallout (2016)
Spider-Kan:
the Spides-Vezee (2018)
Spider-Kan: Inte Spider-Verze 12016)
Alita: Battle Aagel (2019)
Aiita: Battle Angel (2019)
Captain Karvel (2019)
Captain Karvel
Godzilla: King
Godsilla: King
(b) After 8PT (Checkpoint-1500) Raw Output
Processed Rec. (mDCO320=0. 385, 0.238
Blade Runner 2019 (2017)
Blade Runner 2019 (2017)
2 Med Max: Jury Read (2015)
Mad Max: Pury Road (2015)
3 Troeptioon (2013)
Treeption (2010)
4 The Revenant (2015)
The Revenant (2015)
5 Insers:ellar (2014)
Interstellar (2014)
6 Alita: Battle Angel (2019)
Alita: Battle Angel (2019)
7 Dunkirx (2017)
Dunkirx (2017)
8 Gravity (2013)
Gravity (2013)
The Pa11 (2006)
The Pa11 (2006)
Adventures
otis (1986)
The Adventures Milo and Otis (1986)
Legacy (2913)
Tron: Legacy (2012)
Ready Player One
Ready
Jungle
Jungle (2016)
Shang-Chi andLegend Apes 12013)
Rings
Shang-Chi Legend Apes (2013)
Ringe 12025)
Mar
ak. (c) After RL (Checkpoint-15800) Raw Output
Processed Rec. (NDCG920=0, 475, 0.237
Life of P1 (2012)
Life of P1 (2012)
2 Alita: Battle Angel (2019)
Alita: Battle Angel (2019)
3 Ready Player Cae (2010)
Ready Player One (2019)
4
The Maurix (1999)
The Matrix (1999)
5 Blade Runner 2349 (2017)
Blade Runner 2049 (2017)
6 Gravity (2013)
Gravity (2013)
Inserssellar (2014)
Interstellar (2014)
Legacy (2013)
Tron: Legacy
Max: Fury Road
Mad Fury Road (2015)
Dunkirz
Dunkirx (2017)
Valerian and
Planets
City Thousand Planeta (2017)
Inception (2013)
Inception (2019)
Fall (2006)
The (2006)
Labyrinta (2306)
Pan's Labyrinta (2306)
Blade Runner (1982)
Blade Runzer (1902)
30


Reviews:
[{"review_id": "yg3YAhTiRS", "reviewer": "Reviewer_o3UH", "summary": "This manuscript introduces ConvRec-R1, a two-stage framework for training LLM-based conversational recommender systems. The first stage uses a novel \"Remap-Reflect-Adjust\" data distillation pipeline to create a high-quality, catalog-aware dataset for SFT. The second stage introduces Rank-GRPO, a new RL algorithm tailored for ranking tasks. Rank-GRPO reframes the RL update unit to the \"rank position\" level, addressing fundamental flaws in standard GRPO like non-causal credit assignment. Experiments show the framework improves recommendation quality (Recall and NDCG) and allows smaller open-source LLMs to outperform larger, zero-shot models.", "questions": "1. Can you elaborate on measures taken to mitigate potential biases (e.g., position bias) from the LLM-as-a-judge used in the \"Reflect\" step?\n2. What is the final out-of-catalog recommendation rate on the test set, and what are potential strategies to better enforce catalog constraints during the RL phase?\n3. Why was the GSPO baseline omitted from the main on-policy results in Table 1, given it is a key point of comparison?\n4. How sensitive is the model's performance to the new penalty hyperparameters (ϵ_u, ϵ_o) introduced in Rank-GRPO?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1.\tRank-GRPO is a main contribution. The manuscript clearly identifies the core weaknesses of applying standard GRPO to ranking tasks and proposes an new solution by re-framing the problem at the rank level. The technical solutions are well-motivated and supported by theoretical analysis.\n2.\tThe \"Remap-Reflect-Adjust\" pipeline is a significant engineering contribution that provides a sophisticated solution to the critical data scarcity problem in this domain. It is quite practical for researchers and practitioners aiming to deploy LLM-based CRS.\n3.\tThe experimental design is robust. The results clearly show the value of each stage of the framework and validate the superiority of Rank-GRPO over its baselines. The finding that a well-aligned 3B model can outperform a much larger zero-shot model is certainly relevant.", "weaknesses": "1.\tThe manuscript utilizes an “LLM as a judge” in its data pipeline but does not adequately discuss or account for known biases of this paradigm, such as position or verbosity bias, which could affect the quality of the SFT dataset.\n2.\tAll experiments are conducted on a single dataset in the movie domain. The manuscript would be stronger with a discussion on the potential challenges of applying the framework to other domains like e-commerce or music.\n3.\tThe manuscript notes that the model's outputs tend to drift out-of-catalog during RL training. This practical limitation deserves more prominent discussion in the main text, including the final out-of-catalog rate and potential mitigation strategies.\n4.\tThe primary results table (Table 1) omits a direct on-policy comparison with GSPO, a relevant and stronger baseline discussed elsewhere in the manuscript. Including this would make the evaluation of Rank-GRPO's advantage more complete.\n5.\tThe multi-step SFT data pipeline involves a lot of components and hyperparameters. The manuscript lacks an analysis of how sensitive the final data quality is to these choices, which would be valuable for reproducibility and practical application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FvPpo5UOnu", "reviewer": "Reviewer_NBhB", "summary": "The paper proposes ConvRec-R1, a two-stage approach to prevent LLMs from generating out-of-catalog items and to avoid sharp drops in ranking quality. Stage 1 performs supervised fine-tuning (SFT) on high-quality synthetic data produced by GPT-4o; Stage 2 applies Rank-GRPO, an extension of GRPO tailored for ranking optimization. Experimental results demonstrate that ConvRec-R1 achieves strong performance on the evaluated benchmarks.", "questions": "see weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-motivated.\n\nThe paper is readable.", "weaknesses": "1. **Performance on larger models (e.g., 7B) is unclear.**\n    \n    Please provide experimental results or discussions on how the proposed method scales to larger backbones (e.g., 7B parameters). This will help verify whether the observed improvements generalize across model sizes.\n    \n2. **Baselines in Table 1 are insufficient.**\n    \n    Table 1 should include more **post-training baselines** specific to LLM-based recommender systems, rather than comparing only SFT or SFT + GRPO. Incorporating recent LLM-agent or ranking-enhanced recommender baselines would make the comparison more convincing.\n    \n3. **A more detailed ablation study is needed.**\n    \n    Please include ablation experiments isolating the effects of key components, such as **without remap**, **without reflect**, and **without SFT**. These results would clarify each module’s contribution to the overall performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PRHuL0dQ7f", "reviewer": "Reviewer_uEVq", "summary": "The paper presents ConvRec-R1, a two-stage framework for training LLM-based conversational recommender systems. It combines a supervised fine-tuning pipeline (Remap-Reflect-Adjust) with a new reinforcement learning method (Rank-GRPO) that optimizes recommendations at the rank level. The authors present experiments on REDDIT-V2 that demonstrate solid improvements over baselines, achieving near-GPT-4 performance with smaller open-source models.", "questions": "1) The paper only uses the REDDIT-V2 dataset without sufficient justification. For a research paper, relying on a single dataset is insufficient to demonstrate generality. Why weren't other conversational recommendation datasets considered?\n\n2) ConvRec-R1 performs worse than CRAG under off-policy evaluation, despite its more complex training. What causes this drop? reward misalignment, data shift, or instability in Rank-GRPO? A clearer analysis would strengthen the paper's claims.\n\n3) The paper reports results on models only up to 3B parameters. Have the authors tested or considered larger LLMs to see whether the proposed Rank-GRPO continues to scale effectively with model size?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) The paper addresses an important and emerging problem i.e aligning LLMs for conversational recommendation, which has practical utility for the industry and is a relevant problem for the community.\n2) The authors provide code, data, and detailed implementation notes, making the work easy to reproduce and build upon.\n3) The proposed rank-level GRPO is a generally useful method for ranking tasks. It's well motivated, includes gradient analysis, and performs reasonably well in practice.\n4) The experimental section is detailed and comprehensive, covering different model scales, baselines, and both SFT and RL stages.", "weaknesses": "1) The core ideas, namely supervised fine-tuning plus RL alignment, primarily extend existing GRPO and RLHF frameworks, without introducing a fundamentally new paradigm.\n2) The approach is tightly focused on conversational recommendation and may not generalize well to broader LLM alignment or other ranking tasks, which could limit the paper's impact.\n3) The performance improvement over strong prompting baselines (e.g., CRAG) is modest given the added training complexity, and under off-policy settings the method can perform even worse, suggesting limited robustness and stability.\n4) All experiments are conducted on the REDDIT-V2 dataset, leaving open the question of how well the method generalises to other domains or item catalogs.\n5) The work lacks online or human evaluation, making it unclear whether the improvements translate to better real-world user experience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Joqw464ckC", "reviewer": "Reviewer_1bEF", "summary": "This paper introduces R1 style reinforcement learning and self improvement techniques into conversational recommender systems. It first develops a remap reflect adjust pipeline to improve data quality, and then introduces a ranking GRPO reinforcement learning objective with reward shaping techniques for performance improvement. Experiments demonstrate that it can improve the performance of LLM-based conversational recommender system.", "questions": "Is the method senstive to the predefined prompt templates or is it robust to different templates?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation of the paper is clear.\n2. The proposed method shows performance improvement.\n3. Case studies show that after RL, the ranking output of the model is better, demonstrating that rank-GRPO is functioning properly.", "weaknesses": "1. It will be better if the method can be tested on more datasets. Current only one dataset is used.\n2. It will be better if other aspects of conversation quality of the framework such as helpfulness and informativeness, other than the ranking accuracy, can be evaluated.\n3. Some claims can be more rigorous. For example, authors claim GRPO is fundamentally misaligned with tasks with rank-style outputs, but the proposed method is still a simple extension of GRPO. If GRPO is fundamentally misaligned, the proposed framework might be something very different from GRPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)