Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning
Authors: Jiacheng Lin
Tian Wang
Kun Qian
Venue: Accepted by TMLR ()

Abstract: We propose Rec-R1, a general reinforcement learning framework that bridges large language models (LLMs) with recommendation systems through closed-loop optimization. Unlike prompting and supervised fine-tuning (SFT), Rec-R1 directly optimizes LLM generation using feedback from a fixed, black-box recommendation model—without relying on synthetic SFT data from proprietary models like GPT-4o. This avoids the substantial cost and effort required for data distillation. To verify the effectiveness of Rec-R1, we evaluate Rec-R1 on three representative tasks: product search, sequential recommendation, and product re-ranking. Experimental results demonstrate that Rec-R1 not only consistently outperforms prompting- and SFT-based methods, but also achieves remarkable gains over strong discriminative baselines, even when used with simple retrievers like BM25. More impressively, Rec-R1 preserves the general-purpose capabilities of the LLM, in contrast to SFT, which often impairs instruction-following and reasoning. These findings suggest Rec-R1 as a promising foundation for continual task-specific adaptation without catastrophic forgetting.

Published in Transactions on Machine Learning Research (10/2025)
Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning
Rec-R1:
Jiacheng Lin
1125, edu
University of Illinois Urbena- Channnng
Tian Wang
Stolmmoong@@mmiil.ccm
Independent Reseorcher
Kun Qian
kungiaan699@@mmaillccm
Independent Reseorcher
Reviewed on OpenReview: https: // openreview. net/ forum? id= YBRU9HV2VE
Abstract
We propose REC-R1, general reinforcement learning framework that bridges large language models (LLMs) with rocommendation systems through closed-loop optimization. Unlike prompting and supervised fine-tuning (SFT), REC-RI directly optimizes LLM generation using feedback from fixed, black-box recommendation model-without relying on syn- thetic SFT data from proprictary models like GPT-4o. This avoids the substantial cost and effort required for data distillation. To verify the effectiveness of REC-R1, we evaluate REC-R1 on three representative tasks: product search, sequential recommendation, and product re-ranking. Experimental results demonstrate that REC-R1 not only consistently outperforms prompting- and SFT-based methods, but also achieves remarkable gains over strong discriminative baselines, even when used with simple retrievers like BM25. More impressively, REC-R1 preserves the general-purpose capabilities of the LLM, contrast to SFT, which often impairs instruction-foolowingg and reasoning. These findings suggest REC- R.1 promising foundation for continual task-specific adaptation without catastrophic forgetting.
1 Introduction
Recommendation systems (RecSys) have become essential components in various real-world applications, from e-commerce (Schafer et al. 1999: Valencia-Arias et al. 2024) and video platforms (Lubos et al. 2023
Covington et al.) 2016) to news delivery (Raza & Ding] 2022 Wu et al. 2023) and social media (Campana & Delmastro 2017). Despite the remarkable progress of RecSys over the decades, modern systems still face fundamental limitations. Most notably, they lack open-domain world knowledge and struggle to understand users' underlying motivations and preferences (Lin al.] 2025). These shortcomings often lead to subop- timal recommendation performance, especially in complex scenarios such when user intent is implicit 2023
expressed in natural language (Hou et al. 2024a He et al.
Recent advances in generative large language models (LLMs) have opened new possibilitics for enhancing recommendation systems (Xi et al.) 2024 Bao el. al. 2023). Trained on massive web-scale corpora, LLMs
possess extensive open-world knowledge, alongside strong capabilities natural language understanding and reasoning (Achiam al. 2023 Grattafiori et al. 2024 Yang al.) 2024 Lin et al., 2024). These strengths make LLMs particularly suitable for user-centric recommendation scenarios, where the key challenge is understand language-driven inputs, capture implicit user intent, and align recommendations with user needs.

---

Published in Transactions on Machine Learning Research (10/2025)
LLM for Feature Engineering
User Info
EQ
BM25 Retriever
OR
OR
Discriminative Model
Large Language Model
Large Language Model
User Query
Rewritten Query
Info
Augmented Text
Query Rewriting
User/ttem-level Feature Augmentation
LLM as Scoring/Ranking Function
Info
Matching
User Info
Large Language Model
Candidate Items (efter filter)
Large
Language Model
Item Pool
LLM-ggeerated Items
Ranked Item List (Text format)
Closed-Set Item Generation
Open-Set Item Generation
Figure 1: Illustration how generative LLMs are applied recommender systems (LLM4Rec), following the taxonomy in Lin al. (2025). The upper row shows the use of LLMs for feature engineering, including (1) Query Rewriting, where the LLM reformulates the input query to improve retrieval, and (2) User /Item-level Feature Augmentation, where the LLM encodes user item information into richer textual representations input to a downstream model. The lower row demonstrates the use of LLMs as Scoring/ /Ranking Functions, including (3) Closed-Set Item Generation, where the LLM ranks a given candidate list, and (4) Open-Set Item Generation, where the LLM directly generate candidate items and matches them to a product pool. Note that this figure primarily reflects the inference- time setting thus all LLMs are frozen. Our proposed Rec-R1 is compatible with all paradigms shown here (sce Appendix A
As result, recent studies have explored using LLMs in various stages of the recommendation pipeline, including query rewriting (Peng et al. 2024 Li et al. 2023b), user intent sumnarization (Torbati et al. (223),, and so on, to improving the performance of downstream recommendation tasks such as retrieval and
ranking. These mothods typically employ either zcro- or fow-shot prompting (Xi ct al. 2024 Lyu ct al. 2023
Ren el al. 2024 et al. 2023b) or supervised fine-tuning (SFT) (Luo al, 2024b Li el al.) 2023a
Yang
et al. 2023 Liao et al. 2024 et al. 2024) adapt LLMs to recommendation tasks. Figure illustrates
the major paradigms where LLMs are applied in RecSys.
However, most existing approaches still treat LLMs and recommendation models as disjoint components, with no closed feedback loop between LLM generation and recommendation performance (Peng
et al. 2024 Zheng et al. [2223 Luo et al.) 2024b Li et al. 2023a] Yang et al. 2023 Liao et al. (0224). As
result, LLMs are typically optimized using proxy objectives rather than being directly trained using feedback from ReoSys, which is often inconsistent with the ultimate goal of improving recommendation quality. More- over, constructing high-quality supervision data for intermediate tasks-such as query swwriing-ttypccllly requires either human annotation, LLM APIs (e-g,, GPT-4), mining from historical interaction logs (Peng) et al. 2024 Hou et al.) 2024a). Nonetheless, these sources rarely produce data that is truly aligned with optimal recommendation performance. Worse still, generating such data at scale is both time-consuming and financially expensive, especially when relying on human annotation or commercial LLMs. Figure ( presents our proof-of-concept. comparison that highlights the limitations of SFT in terms of both effectiveness and overhead.
To address these challenges, we propose REC-R1, general framework that leverages reinforcement learning (RI.) bridge generative LLMs and downstream black-box RecSys through closed-loop optimization. Tn
2

---

Published in Transactions on Machine Learning Research (10/2025)
(a)
Porformance on ESCI datasets across different domains
(b)
35
Ganed
R1-3B
GPT4c
Gea 2.5 hailhhuudddd (SPT)
Time Comparison
Comparison
1he-tt1-35
$15.80
15
NDCG@ 100
6
25
Time (Hours)
Cost (USD)
10
4-
20
-210s
$0.40
10
Games
Baby
Office
Sports
SFT Rec-R1
SFT Rec-R1
Figure 2: Proce-of-concepp comparison under a small-scale setup, illustrating the limitations of SFT based on GPT-4o-gonerated data. (a) Performance the ESCI datasct. The SFT baseline fine- tunes using data generated by GPT-4o, and its performance is inherently upper-bounded by the performance of GPT-4o itself. (b) Comparison of training time and cost. The total time for SFT includes both the data generation phase using GPTT40 and subsequent model fine-tuning. REC-R1 requires no additional data generation, and we report the minimal training time and cost required to match the performance of the SFT and GPT-4o model. See Appendix [ for cost estimation details.
contrast to existing approaches that rely on static data and proxy supervision, REC-R1 enables LLMs to learn directly from recommendation feedback such retrieval ranking metries thus aligning the generation process with the ultimate goal of improving recommendation quality, Specifically, given any input, such as user query or behavioral history, an LLM generates a textual output that is consumed by a downstream recommendation model. This textual output varies in form depending on the task, such as rewritten query, a synthesized user profile, or an enriched textual description of an item. The recommendation system then evaluates the quality of the LLM-generated text using rule- based performance metrics (e.g., NDCG, Recall), which are transformed into reward signals for optimizing the LLM via. RL. Through repeated interaction with the recommendation system, the L.M gradually learns to generate inputs that are better aligned with the system's 's objectives, thereby improving recommendation performance without relying suboptinal intermediate supervision.
recommendttion-ellaaan
REC-R1 is model-agnostic and task-fiexible: it can be integrated with a wide range of recommenda- tion architecturos- -nniirsi-iicuiinnng sparse retricvers (c.g., BM25), dense discriminative models, and hybrid pipelines without requiring any modifications their internal structures. It also supports diverse gen- eration tasks long as the generated text can be consumed by the downstream recommendation system. Moreover, since REC-R1 relics solely on black-box feedback in the form of recommendation performance metrics, it does not require access to model gradients internal parameters, making it easy to deploy top of existing production systems. It also eliminates the need for constructing SFT data, allowing the generative model to be optimized directly through interactions.
We evaluate REC-R.1 in three representative recommendation scenarios-prost search, sequential recom- mendation, and product re-ranking demonstrate its effectiveness, though the framework itself broadly applicable to wider range of recommendation tasks (see Appendix A). In product search, we observe that applying the REC-R.1 framework significantly improves overall recommendation performance, achieving state-of-the-art results the evaluated benchmarks. In the sequential recommendation setting, REC-R1 leads to consistent gains. More importantly, it shows strong performance in cold-start scenarios, where user profile information is absent, oulperforming widely used sequential baselines. In product re-ranking, REC- R. also shows clear superiority over both strong cross-encoder and LLM- based reranker baselines. Beyond performance gains, we further investigate whether REC-R1 preserves the general capabilities of the under- lying language model. On the FFEval benchmark (Zhou al. (2023). which measures instruction following capabilities, REC-R1 maintains
improves performance, while SFT causes drop of over 27 points.
3

---

Published in Transactions on Machine Learning Research (10/2025)
(a)
(b)
(c)
RL Training
No Training
Closed- Source LLM
No feedback from RecSys
Reward Signal
Large
Large Language Model
Large
Language
Language Model
Model
Text
RecSys
SFT Data
Text
RecSys
RecSys
Inference
Training
Inference
Inference
Prompting
Supervised Fine-Tuning (SFT)
Rec-R1 (Ours)
Figure Comparison of three paradigms for using LLMs in recommendation systems.. (a) Prompting uses frozen LLM to generate textual inputs for the recommendation system, without any model updates. (b) SFT trains the LLM to imitate outputs generated by stronger model (e.g., GPT-4o). but the training process does not involve any RecSys feedback. (c) REC-R1 introduces closed-loop RL framework, where the LLM is optimized directly using reward signals from the recommendation system, without requiring external annotation or data distillation. Unlike SFT,, which relies on labeled intermediate outputs (e.g., rewritten queries) from closed-source models, REC-R1 operates directly on the same data and learns from the recommendation performance.
This suggests that REC-RI enables task-specific adaptation without compromising the general- ca- pabilitics of the initialized LLM. We summarize our main contributions as follows:
We propose REC-R1, a general reinforcement learning framework that bridges generative LLMs and recommendation systems through reward-driven optimization. Our approach is model-agnostic with respect to the recommendation system and supports diverse tasks.
We conduct extensive experiments on three representative recommendation tasks, i.e. product search, sequential recommendation and product re-ranking, demonstrating that REC-R1 signifi- cantly improves performance across different recommendation architectures. Notably, in product search, REC-RI improves the NDCG@100 score by up to 21.45 points for BM25-based retrievers, and by up 18.76 points for dense discriminative models, compared to their respective baselines.
REC-R1 preserves the general-purpose capabilities of the initialized L.M while achieving strong task- specific performance, outperforming supervised fine- seeetunnng in both recommendation effectiveness and instructionnfolloviin generalization.
2 Rec-R1
2.1
Problem Formulation
We begin by modeling how LLMs are integrated into RecSys. In this general setup, the LLM receives an input E S, which may represent a user query, behavioral history, or contextual information. The LLM then generates a textual output a E A, such as a rewritten query, an enriched item description, or a synthesized user profile. This output is consumed by a downstream recommendation model, which produces performance-basse evaluation f(ass) E R, such as NDCG, Recall, or any task-specific metric.
The behavior of the LIM is governed by conditional generation policy (a(a))) where 0 denotes the pa- rameters of the generative LLM. The objective is to find policy that maximizes expected recommendation performance:
max Essp ()ff([|s)]
(1)
Here, p(s) denotes the empirical distribution recommendation--eleeaatt inputs provided to the LLM.

---

Published in Transactions on Machine Learning Research (10/2025)
2.2 Theoretical Analysis of Existing Paradigms' Limitations
While the goal in recommendaation-oiientedd LLM usage is to maxinize downstream performance [[j(als)), existing paradigms fail to optimize this objective directly.
Prompting-ban based mothods, including zero-shot and few-shot prompting, treat the LLM as a frozen generator. These approaches rely on manually constructed prompts or lew examples elicit desirable outputs. However, since the model parameters 0 are not updated, the policy =o(a]s) remains fixed and cannot adapt task- specific feedback, resulting in suboptimal recommendation performance.
Supervised fine-tuning forces the LLM to imitate outputs generated by stronger model, such as GPT-4o. Formally, this corrosponds to maximizing the log-likelihood of actions sampled from the data-generating policy By under the learned policy me:
max Bepppppppsss...p.. tal [log Ta [a(s)]
(2)
This maximum likelihood estimation (MLE) objective encourages the learned policy to imitate Tg. but does not consider the downstream performance (als) during optimization. We now show that this training procedure imposes a fundamental performance ceiling, which we formalize in the following fact.
Fact 1 (SFT Converges Toward the Data-Genorating Policy). Let Ry(a)s) be a fixed dota-generating policy, and consider the supervised fine-tuning (SFT) objective:
TT- arg max
log (als))
(3)
Assume:
(i) (Sufficient Expressivity) The policy class {Te(}|s)} is expressive enough to closely approzimate the gell-yynnnrliing policy Tg(+1s), i.e,, inf, Ex-p(e) (1)] =0.
(ti) (Optimization Convergence) The optimization process converges to global marimum of the MLE objective.
(tti (Data Sufficiency) Data-generating policy generates sufficiently large amount of training sam- ples, the empirical distribution p(s, a) accurately approrimates the true distribution (glls), i.e.,
p(s, a) a ((s)) as - oc,
where N is the number of training samples generated.
Then the optimal policy ae- is the one that minimizes the KL divergence to the data-generating policy Tg.
Tg- = arg min ((s) ([sss((((ss]]]](
(4)
The proof can be found in Appendix C.1 Fact reveals fundamental limitation of supervised finc-tuning: the learned policy иy- is inherently constrained to imitate the dala-generating policy Tg. Consequently, the recommendation performance of an SFT-trained model can at best approach- but never exceed- -the per- formance of the policy used to generate the training data (e.g., GPT-4o). This fact is empirically supported by our experiments on the ESCI dataset, as illustrated in Figure 2 a), where the performance of the SFT- trained model closely matches but does not surpass GPT-4o. However, as GPT-4o itself is not explicitly optimized for the downstream recommendation task, its performance is typically suboptimal.
2.3 The Rec-R1 Framework
To overcome the limitations of prompting and SFT, we introduce REC-R1, a general framework that bridges generative LLMs and recommendation systems through reinforcement learning. Rather than imitating a static data-generating policy, REC-R1 directly optimizes the I.I.M policy no based on feedback from the
5

---

Published in Transactions on Machine Learning Research (10/2025)
downstream recommender- thereby aligning the generation process with the true objective: maximizing recommendation performance. See Figure for a comparison of these paradigms.
At its core, REC-Rl casts the LLM-RecSys interaction as a closed-loop optimization process, where the LLM produces text (e.g., rewritten queries, user profiles, or item descriptions), and the recommendation model evaluates the result using task-specific metrics. These evaluation scores are then transformed into scalar reward signals for policy optimization via reinforcement learning.
A key strength of REC-R1 lies in its ability to optimize the LLM using direct feedback from the recommen- dation system. This feedback is formalized as scalar reward r f(als) E IR, which quantifies how well the LLM-generated output a performs in the downstream task given imput s. The reward can be instantiated using any differentiable or non-differentiable metric that reflects recommendation qualily, such as NDCG&K and R Formally, the optimization objective is to find a generation policy (a(s)) that maximizes the expected reward:
max E Pamp(s) [f(a]s)]
Unlike SFT, this objective does not rely manually labeled supervision imitatiin of fixed policy, Instead, it allows the model to continuously adapt its behavior maximize performance on the downstream recommendation tasks. From the perspective of the optimization objective, Eq. 2 minimizes proxy loss that indirectly relates to recommendation quality, whereas Eq. 5 directly optimizes the true recommendation objective, ensuring alignment with the downstream evaluation metrics. Following DeepSeek-RI (Guo et al. 2025), we adopt Group Relative Policy Optimization (GRPO) (Shao al., 2024) to optimize the LLM policy. Compared traditional algorithms such PPO (Schulman et al. 2017). GRPO significantly reduces memory consumption during training while maintaining competitive performance. Further, use rule-based reward functions derived from standard evaluation metrics (e.g., NDCG, Recall) rather than training a separate reward model, which helps mitigate reward hacking and avoids introducing additional biases.
3 Experiments
To validate the effectiveness of REC-R1, we conduct experiments on three representative recommendation
scenarios: product search ($3.1) sequential recommendation ($3.2), and product re-ranking ($3.3). We also perform detailed analyses to examine the generalization ability after training ($3.4). More discussions and
analysis can be found in Appendix F and case study in Appendix G
3.1 Product Search
3.1.1 Experimental Setup
Task Definition. In the product search task, the provides natural language query E S, which expresses an information need (e.g., "a waterproof camera for hiking"). The goal of the recommendation system is to retrieve ranked list of items that best match this query. To improve retrieval quality, the LLM generates a textual transformation a such as a rewritten or clarified version of the query which is then fed into a downstream retriever.
The retriever returns a ranked list of candidate items based on the textual input a, and the system evaluates performance using a relevance dictionary D that maps each original input to its corresponding ground- truth item list. The reward score (als) E R is computed by comparing the retrieved list (from a) against the target set D(s) associated with the original query s. We use NDCG@100 as the evaluation metric, which captures both relevance and ranking position. This reward function serves as the feedback signal in R1 to optimize the LLM's generation policy (a(a))) While D derived from the original datasets in our experiments, in real-worid deployments can be constructed from various sources such as most recent user interaction logs or thick-through data, making REC-R1 promising to production recommendation systems.
6

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Performance comparison of different methods on conventional product search (ESCT) tasks. We report the NDCG@100 scores. The best performance score denoted in bold, with the second and third best underlined. The numbers in gray indicate the absolute improvement of REC-RI their corresponding base retrievers (BM25 or BLAIR).
Model
Video Games
Baby Products
Office Products
Sports and Outdoors
Sparse retrieval basclines
BM25
12.44
15.12
23.96
19.48
GPT-40,
26.06
23.05
27.98
27.38
19.63
16.03
19.96
21.36
Dense retrieval baselines
RoBERTapASE
0.16
0.00
0.00
0.17
SinASE
2.21
5.68
8.58
8.03
BLAIR BASE
9.75
15.20
17.19
17.08
ROBERTaLARGE
0.00
0.00
0.00
0.00
SinCSELAGGE
9.71
13.63
11.90
BLAIRAAGGE
15.88
15.96
21.17
18.30
GPT-4o+ GPT--O-BLAIR-BBAS
20.13
24.57
21.83
22.97
GPT-40+ - -ETRR
23.99
24.10
22.99
24.67
10.56
16.23
13.61
17.09
2.w.-2.5533-Insstrrcc BLAAS
17.34
16.10
17.29
17.74
Ours
MEC-R1-3B,RMa.
33.89
29.27
34.61
31.92
(+21.45)
(+14.11)
(+10.68)
(+122444)
REC-R 1----1133B,pp0 R.S.BBBAARBBASSR
28.51
29.24
33.98
30.71
(+18.77)
(4.04.0)
(4-16.70)
(+-13.63)
REc-RR1-B.. LLGR
31.41
28.76
34.12
32.49
(+15.53)
(+12.80)
(+-12.95)
(+:1..19)
Datasets. We consider two distinct settings for product search: (1) conventional product search, where the input query is a short phrase keyword-based expression (e.g., "noise-canceling headphones") and (2) complex product search, where the input. a rich and long natural language context, often involving implicit preferences or use-case scenarios. To evaluate these two settings, we adopt two datasets: the ESCT dataset for conventional product search (Reddy et al., 2022) and the Amazon-C4 dataset for complex product search (Hou al., (2024)). More details can be found in Appendix E.1.1 For all experiments in this paper, we report test performance based on the checkpoint that achieves the best validation score.
Baselines. We compare REC-R1 against range of baselines. For sparse retrieval, we use the BM25, well as prompting enhancee variants where frozen LLM (GPT-4o or Qwen-2.5-22 12.5-3B-Instrun (Yang et al. 2024h) rewrites the imput query before retrieval. For dense retrieval, we include discriminative models such as RoBERTa [Liu et al. 2019), SimCSE (Gao et al. 2021). and BLAIR (Hou et al. 2024a). with and without prompting-bese based query rewriting. In contrast, REC-R1 starts from the Qwen-2.5-2 Instrrcct model and is trained via RL to generate rewritten queries that directly optimize retrieval performance.
3.1.2 Results
Results on ESCI. Table reports NDCG@100 scores on the conventional product search benchmark ESCT. We observe that REC-R1 consistently improves retrieval performance across all four domains and retriever architectures. Notably, even when applied to the sparse BM25 retricver, REC-RI yields substantial gains up to +21.45 NDCG points the Video Games domain demonstrating its ability to enhance classic lexical systems. For dense retrievers such BLAIR, REC-R.1 brings improvements of up to +18.76 over the base models, and consistently outperforms prompting-basee rewriting with GPT-4o. Remarkably, REC-R1 achieves the best performance across all four product categories, underscoring its effectiveness and overall superiority.
Results on Amazon-C4. We further evaluate REC-R1 on the Amazon-C4 datasct, which contains complex product search queries expressed in natural language. This setting also enables us to assess the model's cross- domain generalivation ability: train queries from all categories ercept the four test domains (Video

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Performance comparison of different methods on complex product search (Amazon- C4) tasks. We report the NDCG@100 scores. The best performance score is denoted in bold, with the second and third best underlined. The numbers in gray indicate the improvement of REC-RI over their corresponding base retrievers (BM25 or BLAIR). Baby Products
Model
Video Games
Office Products
Sports and Outdoors
Sparse retrieval basclines
BM25
7.82
6.39
8.70
7.28
GPT-40,
13.02
12.45
12.64
11.59
10.33
8.62
9.61
9.05
Dense retrieval baselines
RoBERTapASE
0.22
0.12
0.33
0.13
SinASE
6.05
6.22
3.71
4.33
BLAIR BASE
19.14
19.53
17.43
20.02
ROBERTaLARGE
0.00
0.06
0.00
0.00
Sim SinCSELAGGE
7.43
8.12
BLAIRAAGGE
24.86
22.44
18.92
24.54
GPT-4o+ GPT--O--LAAR-BBAST
21.12
19.54
17.22
22.68
GPT-40+, - --T--O-BLAIR-LAGGEE
23.40
21.00
18.69
25.74
15.82
18.07
14.34
16.96
18:20
19.19
15.37
18.94
Ours
REC-R1-3B,RMe.
18.91
20.35
19.24
20.06
(+11.00)
(+14.16)
(+10.554
(+12.77)
REC- B.CC-RR--,,BATR--BRR BLAAI-RA-A
21.69
25.62
22.17
24.22
(+2.82)
(46.00)
(+44.74)
(+4.200)
REC-R113B R LAGG
26.51
27.04
23.10
27.40
(+1.65)
(+4.60)
(+4.11)
(422866)
Games, Baby Products, Office Products, and Sports and Outdoors), and evaluate performance on these held-out domains.
As shown in T REC-R1 achieves the best performance across all four domains and retriever architec- tures, demonstrating strong generalization beyond the training distribution. This cross-domain evaluation highlights REC-RI's ability to generalize from training on one set of product calegories to effectively han- dling unseen categories during testing. Moreover, prompting-ben based query rewriting using frozen LLMs (e.g., GPT-4o Qwen-unst either yiclds negligible improvement or even degrades dense retrievers. In contrast, both BM25 and dense models see notable gains when combined with REC-R1, indicating the value of interaction-based learning. By receiving direct feedback from the recom- mendation system during training, the LLM gradually learns how to rewrite queries in way that maximizes downstream task performance.
Note that we do not report SFT results in our tables, as prior analysis (Theorem shows that, under sufficient data and optimization, the learned policy from SFT converges toward the data-generating model (e.g., GPT-4o). Therefore, we use GPT-40 performance a practical reference point. Across both ESCI and Amazon-CAa C4, REC-R1 consistently outperformn GPT--4o-bsee based prompting methods demonstrating its potential to go beyond the limitations of the SFT paradigm in both performance and adaptability.
3.2 Sequential Recommendation
3.2.1
Experimental Setup
Task Definition. In this task, the model reccives user's historical interaction sequence E S (c.g., list of previously viewed or purchased items) and is expected lo recommend the most relevant next item. To support this process, the LLM generates a text a E A-a a query describing what the user probably will purchase next. This could take the form of key attributes, product type, or usage scenario, serving as a query-like signal input the downstream retriever. To evaluate performance, we define a relevance dictionary D that maps each historical sequence to the ground-truth next item. The reward score s(ass) R is computed by
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Performance comparison of different methods on the sequential recommendation task Amazon Beauty datasct. We report the Recall (R) and NDCG@k (N) scores. The best performance score is denoted in bold. The numbers in gray indicate the absolute improvement of REC-R1 over the initialized policy, Le., Oyen-2.5-33-Instruuee
Transductive Setting
Inductive Setting
Model
R@10
N@10
R@50
N@50
R@00
N@10
R@50
N@50
Text-aware SRec baselines
SASReC+BLARR-BAN BASE
3.72
1.55
7.81
2.44
0.20
0.90
1.40
0.35
SASRec DILIR-LARCE
3.90
2.12
8.74
3.17
0.40
0.15
1.50
0.40
UniSRec BLAIR-BASE
4.09
2.23
8.74
3.21
3.70
2.08
6.20
2.61
LniSReC-BLAIR-LL LARGE
4.09
2.17
8.55
3.17
3.60
2.08
6,00
2.60
Query rewriting baselines
GPT-4o, BM25
0.00
0.00
1.48
0.33
1.00
0.50
2.90
0.90
2rem-2.5-3B-Instruee DM25
1.30
0.75
2.41
0.98
1.80
0.84
3.60
1.25
Ours
REC- R1-3B-BM25
3.53
1.74
5.76
2.22
6.00
3.38
8.30
3.89
(+2229)
(+0.99)
(+3335)
(+1.24)
(1-4.20)
(+22554
(+44770)
(+22.64)
comparing the retrieved items (from a) against the target set D(s) using standard retrieval metrics such as Recall@K and NDCG@K. In our implementation, we use NDCG@K as the training reward for REC-R1. In real-world systems, the dictionary DD can be constructed from user interaction logs, purchase sequences, or other behavioral data sources.
Dataset. We conduct experiments the Amazon Beauty dataset following the split protocol from Hou ct al. (0224)), where data are partitioned into training, validation, and test sets by absolute timestamp. To eva evaluale different generalization capabilities, we define two lest-time settings: (1) Transductive setting: All candidate items in the test. sequence (both history and target) have appeared in the training set; (2) Inductive setting: None the test-time items are seen during training.
Baselines. We compare REC-R1 with two families of basclines: (1) Tezt-aware Sequential Recommendation (SRec) models, including SASRec (Kang & McAuley 2018) and UniSRec (Hou et al. BLAIR the item encoder (base and large variants). These methods use sequence modeling with features from textual encoders. (2) Prompting- comptiing-based based query reuriting, where frozen LLMs (GPT-4o or Qwen-2.5-3B- Instruct) generate rewritten inputs from user histories, which are then fed into a retriever (e.g., BM25).
2022 combined with
3.2.2 Results
Table 3 summarizes the performance under both transductive and inductive scttings. We observe that the prompting-based baselines using frozen LLMs (e.g., GPT-4o and Qwen-2.5-3B-Instteee perform poorly across the board- highhlhiigg the difficulty of this task for generic LLMs without adaptation. However, when trained under the REC-R1 framework, the finetuned Sowen-2.-3B-Innstree model exhibits significant performance gains, especially in the inductive setting. For instance, in Recall@10 and NDCG@50, REC-R.1 improves upon its initialized policy by +4.20 and +2.64 points respectively. Moreover, ReC-R1 outperforms strong sequential basclines like UniSRec the inductive sctting, demonstrating its superiority in cold-start or unseen-item scenarios.
In the transductive setting, REC-R1 remains competitive but lags behind specialized SRec models. This is not surprising, as traditional soquential recommendation methods are explicitly trained on large-scale user- item sequences and directly model sequential dependencies. In contrast, REC-R1 leverages LLMs to generate natural language queries about the user's next likely purchase relying on their strength in reasoning and gencralization. However, next-item prediction based solely on interaction history often not a task that lends itself explicit reasoning in language space. The relationship between past and future items may be weak non-causal, making it inherently difficult for LLMs to perform this task effectively. More analysis
9

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Performance comparison of different re-rankers on ESCI. We report the NDCG@10 scores across four domains. The best performance is denoted in bold. The numbers in gray indicate the improve- ment of REC-RI over its corresponding initialized model, Le., 10ven25--3-IInsrueee
Model
Video Games
Baby Products
Office Products
Sports and Outdoors
Cross-encoder re-rankers
Qwen3- Beens-Reranker-4B
44.29
46.41
48.06
47.42
Awen3-Reranker-ae
44.28
44.95
47.73
44.51
LLM-based based re-ranker baselines
Qwen2.5-28 InS-tuutttt
5.38
9.21
4.26
4.37
GPT-4o
61.57
56.18
53.62
53.62
Ours REC-RI-3B
53.04
57.08 (+:47.87)
60.91
62.91
(1:47.66)
(4-56.65)
(4 (+-58.54)
of why REC-R1 performs better the inductive setting than in the transductive setting can be found in Appendix F.2.4
3.3 Product Re-ranking
3.3.1 Experimental Setup
Task Definition. In the product re-ranking task, the input is natural language query and initial candidate list of m items, denoted as (g, C) E S where is the user query and C = [i,,..., is the candidate set. The goal is to reorder C so that items more relevant to are ranked higher. Given 8, the LLM produces a E A, a re-ranked list of the m candidates (i.e., a permutation of C), denoted by a a= ((x( Re(m)) E A, where и a permutation function over (I). To evaluate performance, we define a relevance dictionary D that maps each input to corresponding ground-truth item list. The reward score (als) E IR is computed by comparing the re-ranked list against the target. item list D(s). We use NDCG@10 as the ranking metric.
We conduct experiments on the ESCI dataset (Reddy et al. 2022). The candidate items C are retrieved using the Rec-RI-Retriever given the user query, and subsequently used for the re- ranking stage. To ensure fair comparison, all re-ranking models operate on the same candidate lists obtained from the Rec-RI-Retrriver. We compare REC-RI with two families of baselines:
Dataset and Baselines.
(1) traditional cross-
encoder re-rankers (Zhang et al.) 2025),, where we adopt the state-of-the-art open-source Qwen3-Reran Rerrnkkr family, including the 4B and 8B variants; and (2) LLM--based re-rankers, including GPT-4o and Qwen2.5- 3B-Instruct, which directly generate ranking orders conditioned on the query and candidate list. Further implementation details and hyperparameters are provided in Appendix E.3
3.3.2 Results
Table 4 shows the ro-ranking performance of different models the ESCI dataset. From the table, REC-R1 consistently achieves the best overall performance across all domains. Notably, compared with the initialized model, REC-R1 delivers very large improvements. Furthermore, against SoTA open- source embedding/reraat /reranker models (Qwen3-Reranker family), REC-R1 demonstrates substantial superiority. These results confirm that REC-R1 optimization framework can also be applied to the closed-set item generation paradigm within LLM4Rec, further supporting our claim that Rec-R1 provides a general way bridge LLMs and ReeSys.
2wwen2.-33B-Inttee
3.4 Does Rec-R1 Forget? A Generalization Analysis
While REC-R1 is explicitly designed enhance recommendation performance, an important question is whether can preserve the general-purpose capabilities of the underlying language model. To this end, compare three models: (1) Q (2) Qwwen-2.533-IInstmiee (SFT), fine-tuned on
10

---

Published in Transactions on Machine Learning Research (10/2025)
Qwwnn-2.5-B-innsrrucc
Qwwn--25--B-inssruuct (SFT)
Rec-R1-3B
60
Score
40
20
ESCI MMLU
IFEval
GSM8K
MBPP
HumanEval
Figure 4: Generalization analysis across six benchmarks. We compare the initialized model (Qwen-2.5- (BBlnntr, its SFT variant trained on GPT-4o generated SFT-data (ESCI), and our REC-R1-3B model trained via RL. Note that Rec-R1 is only trained the task-specific ESCI data, whose format drastically differs from the other benchmark datasets.
FSCT query rewriting data generated by GPT-4o; and (3) Rec-R1-3B, trained using our RL framework on the same ESCI task, but without access to GPT-4o outputs. We evaluate all models across six tasks spanning different axes of generalization: ESCI (recommendation), MMLU (factual knowledge), IFEval (instruction following), GSMSK (math reasoning); and two coding benchmarks: MBPP and HumanEval. Results are shown in Figure A On ESCI, SFT yields improvements over the base model, while REC-R1 achieves substantially higher gains without relying on GPT-4o-geeeerted data. On MMLU, all models including REC-R1 achieve comparable accuracy. This suggests that neither SFT nor REC-R1 compromises the model's general knowledge.
However, striking differences emerge on EFEvl. Here, SFT suffers dramatic performance drop losing over 27 points while Rec-R1 not only avoids degradation but actually improves slightly over the original model. This highlights a key advantage of our REC-RI: by optimizing directly for task- specific performance without overriding the model's generative distribution via next token prediction, Rec- R1 preserves instruetionn-ollowiii capabilities more effectively. We observe similar trend GSMSK, where REC-R1 improves upon the initialized model while SFT lags far behind. In contrast, both SFT and REC-R1 maintain strong performance on coding tasks like MBPP and HumanEval. This is likely because our SFT data involved JSON-style outputs to facilitate answer extraction, which do not interfere with the model's code generation ability. These results highlight the promise of Rec-R1 as a new paradigm for LLM adaptation one that enables strong task-specific improvements without compromising general capabilities.
4 Conclusion
In this work, we present REC-R1, a reinforcement learning framework that bridge LLMs with recommenda-- tion systems through direct feedback. Rec-R1 achieves strong performance across tasks and retrievers while preserving general-purpose capabilities, offering scalable alternative to prompting and SFT.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprind Stt::2333.0877, 2023.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arxiv preprint Xix::2208.. 07732, 2021.
Kegin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the I 17th ACM Conference on Recommender Systems, pp. 1007-1014, 2023.
Mattia G Campana and Franca Delmastro. Recommender systems for online and mobile social networks: A survey. Online Social Networks and Media, 3:75 97, 2017.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXit:2107.0399 03374, 2021.
Karl Cobbe, Vineet, Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arxiv preprint arXiv:2110. 14168, 2021.
Paul Covington, Jay Adams, and Emnre Sargin. Deep neural networks for youtube recommendations. Tn Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. M6-rec: Generative pretrained language models are open-ended recommender systems. arXiv preprint arXie:2205.0808. 2022.
Sunhao Dai, Weihao Liu, Yuqi Zhou, Liang Pang, Rongju Ruan, Gang Wang, Zhenhua Dong, Jun Xu, and Ji-Rong Wen. Cocktail: A comprchensive information retrieval benchmark with Imm-gnerated documents integration. In Findings of the Association for Computational Linguistics ACL 2024, pp. 7052 7074, 2024.
Dario Di Palma. Retrieval-angmented recommender system: Enhancing recommender systems with large language models. In Proceedings of the 17th ACM Conference on Recommender Systems, pp. 1369 1373, 2023.
Dario Di Palma, Giovanni Maria Biancofiore, Vito Walter Anelli, Fedelucio Narducci, Tommaso Di Noia, and Eugenio Di Sciascio. Evaluating chatgpt as a recommender system: A rigorous approach. arXiv preprint arXiv:2309.0S5 006113,33333...0 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Authony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoclkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thile, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL httpss://zenee wrrollooorrrreeeooeeeooooooooooe wtt.///o/ddd//066000
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1762 1777, 2023a.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Mrrni-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punto Cana, Dominican Republic, 7-11 November, 2021, pp. 6894 6910. Association for Computational Linguistics, 2021. doi: 10.18653/V1, 211/222..EMN.P... WWW.LENNN--MMIAAN... URL https:://doi. http:////oooor//1..// 18153/181 2021. emnlp-main 552
Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. Chat-ree: Towards interactive and explainable llms-augmented recommender system. arxiv preprint arXiv:2303.1455 14524, 2023b.
Shijic Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Rocommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM conference on recommender systems, pp. 299-29930 300-315,-00 2022.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llma 3 herd of models. arXiv preprint (077:22407,21788, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1mss via reinforcement learning. Xiv preprint arXiv:2501.1290 2025.
Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Ma- jumder, Nathan Kallus, and Julian McAuley. Large language models as zcro-shot conversational rec- ommenders. In Proceedings of the 32nd ACM international conference on information and knowledge management, pp- 720 730, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXic preprint arXiv:2009 20ix:2009003000, 2020.
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 585-50 593, 2022.
Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. Bridging language and items for retrieval and recommendation. arXiv preprint Xiv:24403 03952, 2024a.
Yupeng Hou, Junjic Zhang, Zihan Lin, Hongyu Lu, Ruobing Xic, Julian McAuley, and Wayne Xin Zhao Large language models are zero-shot rankers for recommender systems. In European Conference on Infor- mation Retrieval, pp. 364-381. Springer, 2024b.
Jihwan Jeong, Yinlam Chow, Guy Tennenholtz, Chih- Wei Hsu, Azamat Tulepbergenov, Mohammad Ghavamzadeh, and Craig Boutilier. Factual and personalized recommendations using language models and reinforcement learning. arXiv preprint arXiv:231000617% 2023.
Jianchao Ji, Zelong Li, Shuyuan Xu, Wenyue Hua, Yingqiang Ge, Juntao Tan, and Yongfeng Zhang. Genrec: Large language model for generative recommendation. In European Conference on Information Retrieval, pp. 494-502. Springer, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yuc, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprint arxi::2503. 095166, 2025.
Jeff Johnson, Matthijs Douze, and Hervé Jégon. Billion-scale similarity search with GPUs. IEEE Transac- lions on Big Data, 7(3):535 547, 2019.
Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE interna- tional conference on data mining (ICDM), pp. 197-206. IEEE, 2018
Omar Khattab and Matci Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 39-48, 2020.
Jinming Li, Wentao Zhang, Tian Wang, Guanglei Xiong, Alan Lu, and Gerard Medioni. Gpt4rec: A generative framework for personalized recommendation and user interests interprelation. arXie preprinl orXiv:2304.0388 2023a.
Xiaopeng Li, Lixin Su, Pengyue Jia, Xiangyu Zhao, Suqi Cheng, Junfeng Wang, and Dawei Yin. Agent-rankinn: Semantic robust ranking via personalized query rewriting using multi-agent 1mm. arXiv preprint xrXi:22312. 15450, 2023b.
Yangning Li, Shirong Ma, Xiaobin Wang, Shen Huang, Chengyue Jiang, Hai-Tao Zheng, Pengjun Xie, Fei Huang, and Yong Jiang. Econgpt: Instruction-tuning large language models with chain-of-task tasks for e- commerce. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18590, 2024.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, Xiang Wang, and Xiangnan He. Llara: Large langungg-rcommenattin assistant. In Proceedings of the 47th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, pp. 1785 1795, 2024.
Jiacheng Lin, Hanwen Xu, Zifeng Wang, Sheng Wang, and Jimeng Sun. Panacea: A foundation model for clinical trial search, summarization, design, and recruitment. medRriv, pp. 2024-06, 2024.
Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Hao Zhang, Yong Liu, Chuhan Wu, Xiangyang Li, Chenxu Zhu, et al. How can recommender systems benefit from large language models: A survey. ACM Transactions on Information Systems, 43(2):1-47 2025.
Jimmn Lin, Xueguang Ma, Sheng-Chiel Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense repre- sentations. In Proceedings of the 44th Armual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pp. 2356 2356-2362, 2021.
Fan Liu, Yaqi Liu, Huilin Chen, Zhiyong Cheng, Liqiang Nie, and Mohan Kankanhalli. Understanding before recommendation: Semantic aspect-aware review exploitation via large language models. ACM Transactions on Information Systems, 43(2):1 26, 2025.
Linqing Liu, Minghan Li, Jimmy Lin, Sebastian Riedel, and Pontus Stenetorp. Query expansion using contextual clue sampling with language models. arXiv preprinl arXiv:2210. 0.07093, 2022.
Shuchang Liu, Qingpeng Cai, Bowen Sun, Yuhao Wang, Ji Jiang, Dong Zheng, Peng Jiang, Kun Gai, Xiangyu Zhao, and Yongfeng Zhang. Exploration and regularization of the latent action space in recommendation. In Proceedings of the ACM Web Conference 2023, pp. 833 833-844, 844, 2023.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http://arxiii...a/ http:/orgg/abbb/...1eee.e/
Wensheng Lu, Jianxun Lian, Wei Zhang, Guanghua Li, Mingyang Zhou, Hao Liao, and Xing Xie. Aligning large language models for controllable recommendations. ar preprint arXiv:2403. 05066, 2024.
Sebastian Lubos, Alexander Felfernig, and Markus Tautschnig. An overview of video recommender systems: state-of-the-art and research issues. Frontiers in big Data, 6:1281614, 2023.
Sichun Luo, Bowei He, Haohan Zhao, Wei Shao, Yanlin Qi, Yinya Huang, Aojun Zhou, Yuxuan Yao, Zong- pong Li, Yuanzhang Xiao, et al. Recranker: Instruction tuning large language model as ranker for top-k recommendation. ACM Transactions on Information Systems, 2024a.
Sichun Luo, Yuxuan Yao, Bowei He, Yinya Huang, Aojun Zhou, Xinyi Zhang, Yuanzhang Xiao, Mingjie Zhan, and Linqi Song. Integrating large language models into recommendation via mutual augmentation and adaptive aggregation. arXiv preprint or 2rXiv:2/01.19700 2024b.
Hanjia Lyu, Song Jiang, Hanqing Zeng, Yinglong Xia, Qifan Wang, Zhang, Ren Chen, Christopher Leung, Jiajic Tang, and Jiebo Luo. Llm-rec: Personalized recommendation via prompting large language models. arXiv preprint ar (xxx::3307157660 2023.
Iain Mackie, Shubham Chatterjee, and Jeffrey Dalton. Generative relevance feedback with large language models. In Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, pp. 2026 2031, 2023.
Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, and Hongjin Qian. Large language models know your contextual scarch intent: A prompting framework for conversational search. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1211-1225, 2023.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. choice, 2640:660, 2016.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Rodrigo Nogucira, Jimmy Lin, and AI Epistemic. From doc2query to docttttiquery. Online preprint, 6(2), 2019.
Wenjun Peng, Guiyang Li, Yue Jiang, Zilong Wang, Dan Ou, Xiaoyi Zeng, Derong Xu, Tong Xu, and Enhong Chen. Large language model based long-tail query rewriting in taobao search. In Companion Proceedings of the ACM Web Conference 2024, pp. 20 28, 2024.
Shaina Raza and Chen Ding. News recommender system: review of recent progress, challenges, and opportunities. Artificial Intelligence Review, pp. 1-52, 2022.
Chandan K Reddy, Lluís Marquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. Shopping queries dataset: A large-scale esci benchmark for improving product search. arXiv preprint arXix:2206.000 06588, 2022.
Xubin Ren, Wei Wei, Lianghao Xia, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Representation learning with large language models for recommendation. In Proceedings of the ACM Web Conference 2024, pp. 3464-3475, 2024.
Zhaochun Ren, Na Huang, Yidan Wang, Pengjic Ren, Jun Ma, Jiahuan Lci, Xinlci Shi, Hengliang Luo, Joemon Jose, and Xin Xin. Contrastive state augmentations for reinforcement learning- based recommender systems. In Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, pp. 922 931, 2023.
Ben Schafer, Joseph Konstan, and John Riedl. Recommender systems in e-commerce. In Proceedings of the 1st ACM conference on Electronic commerce, pp. 158-166, 1999.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithmss. Xiv preprint arXiv: 1000170770000 06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402. 000.00000 2024.
Kaize Shi, Xueyao Sun, Dingxian Wang, Yinlin Fu, Guandong Xu, and Qing Li. Llama-e: Empowering c-commerce authoring with multi-aspect instruction following. CoRR, 2023.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Toannis Antonoglou, Veda Panneershelvam,, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree scarch. nature, 52977587):444-489, 2016.
Chao Sun, Yaobo Liang, Yaming Yang, Shilin Xu, Tianmeng Yang, and Yunhai Tong. Rirf4rec: Re- inforcement learning from recsys feedback for enhanced recommendation reranking. arXio preprint arXiv:2410.. 10.05999, 2024.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Tryna Gurevych. BEIR: A het- erogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Confer- ence on Neural Information Processing Systems Datascts and Benchmarks Track (Round 2), 2021. URL https://opennreviet httpo//w....oooooorroooo/o//oo/
Ghazaleh Haratinezhad Torbati, Anna Tigunova, Andrew Yates, and Gerhard Weikum. Recommendations by concise user profiles from review text. arXiv preprint arXiv:2311.01314, 2023.
Alejandro Valencia-Arias, Hernán Uribe-Bedoyaa Juan David González-Ruiz, Gustavo Sánchez Santos, and Edgard Chapoñan Ramírez. Artificial intelligence and recommender systems in e-commerce. trends and research agenda. Intelligent Systems with Applications, pp. 200435, 2024.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350 354, 2019.
Pengfei Wang, Yu Fan, Long Xia, Wayne Xin Zhao, ShaoZhang Niu, and Jimmy Huang. Kerl: A knowledge- guided reinforcement learning model for sequential rocommendation. In Proceedings of the 43rd Interna- lional ACM SIGIR conference research and development in Information Retrieval, pp. 209 218, 2020.
Chuhan Wu, Fangzhao Wu, Yongfeng Huang, and Xing Xie. Personalized news recommendation: Methods and challenges. ACM Transaclions on Information Systems, 41(1):1 -0, 2023.
Yunjia Xi, Weiwen Liu, Jianghao Lin, Xiaoling Cai, Hong Zhu, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, and Yong Yu. Towards open-world recommendation with knowledge augmentation from large language models. In Proceedings of the 18th ACM Conference on Recommender Systems, pp. 12-22, 2024.
Xin Xin, Alexandros Karatzoglou, loannis Arapakis, and Joemon M Jose. Self-supervised reinforcement learning for recommender systems. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 931 931-940, 2020.
Xin Xin, Alexandros Karatzoglou, Toannis Arapakis, and Joemon M Jose. Supervised advantage actor-critic for recommender systems. In Proceedings of the fifteenth ACM international conference on web search and data mining, pp. 1186-1196, 2022a.
Xin Xin, Tiago Pimentel, Alexandros Karatzoglou, Pengjic Ren, Konstantina Christakoponlon, and Zhaochun Ren. Rethinking reinforcement learning for recommendation: A prompl perspective. In Pro- ceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pp. 1347-1357, 2022b.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arvv 24.:2412215115, 2024.
Fan Yang, Zheng Chen, Ziyan Jiang, Eunah Cho, Xiaojiang Huang, and Yanbin Lu. Palr: Personalization aware Iims for recommendation. arXiv preprint arXiv:2305.07625 2023.
Fanghua Ye, Meng Fang, Shenghui Li, and Emine Yilmaz. Enhancing conversational search: Large lan- guage model-aided informative query rewriting. In Conference on Empirical Methods in Natural Language Processing (EMNLP), DEC 06-10, 2023, Singapore, SINGAPORE, pp. 1985-6006. Association for Com- putational Linguistics, 2023.
Chanwoong Yoon, Gangwoo Kim, Byeongguk Jeon, Sungdong Kim, Yohan Jo, and Jaewoo Kang. Ask optimal questions: Aligning large language models with retriever's preference in conversation. In Findings of the Association for Computational Linguisties: NAACL 2025, pp. 5899 59210 2025.
Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation. In Proceedings of the 1 17th ACM Conference on Recommender Systems, pp. 119-999,222 2023a.
Junjie Zhang, Ruobing Xic, Yupeng Hou, Xin Zhao, Leyu Lin, and Ji-Rong Wen. Recommendation instruction following: A large language model empowered recommendation approach. ACM Transactions on Information Systems, 2023b.
Tianhua Zhang, Kun Li, Hongyin Luo, Xixin Wu, James Glass, and Helen Meng. Adaptive query rewriting: Aligning rewriters through marginal probability of conversational answers. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 13444 13461, 2024.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xic, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. ar preprint orXiv:2506.00 05176, 2025.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Xiangyu Zhao, Liang Zhang, Zhnoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. Recommendations with negative leedback via pairwise deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1040-100000 2018
Zhi Zheng, Zhaopeng Qiu, Xiao Hu, Likang Wu, Hengshu Zhu, and Hui Xiong. Generative job recommen- dations with large language model. arXiv preprint arXiv:237.002157, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloinng evaluation for large language models. arXiv preprint 101.011111.07911, 2023.
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. Towards lopie-guided conversational recommender system. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4128-4139 2020.
Changtai Zhu, Siyin Wang, Ruijun Feng, Kai Song, and Xipeng Qiu. Convsearch-rl: Enhancing query reformulation for conversational search with reasoning via reinforcement learning. arXiv preprint wrXiv:2505.17776 2025.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Contents of Appendix
Applying Rec-R1 to Diverse Recommendation Paradigms
20
B Related Work
20
B. Generative LLMs for Recommendation Tasks
20
B.2 Reinforcement Learning for Recommendation Systems
21
B.3 LLMs for Query Rewriting
21
C Theorems and Proofs
22
Prools of Pacl
22
C.2 SFT Performance Bound in Terms of KL. Divergence
23
C.3 Dominance of RL Objective over SFT
24
D Estimated Cost of SFT Data Generation and Rec-R1 Model Training
25
E
Additional Experiment and Results Details
25
Product Search
25
F.I.1
Dataset Details
25
1.2
Implementation Details
25
E.1.3
Additional Analysis: The Role of Prompt Design and Exploration in RL.
26
E. 1.4
Comparison with Other Fine-TTuing Strategies: Rejection Sampling and DPO
27
E.5.5
Applicability to Alternative Backbone Models
28
E.1.6
Comparison with Modern Strong Query Rewriting Baselines
28
E.2
Sequential Recommendation
29
E22.1
Dataset Details
29
E.2.
LLM Input Construction for Sequential Recommendation
29
E.2.3
Implementation Details
30
E.2.4
Additional Analysis: Why REC-R1 Performs Better in the Inductive Setting?
30
E.3
Product Re-ranking
31
E.3.
Dataset Details
31
E.2
Implementation Details
32
E.4
Evaluation of Generalization and Forgetting
32
E.4.1
Implementation Details
32
E.4.2
Additional Analysis: Impact of Reasoning and JSON Format in SFT
................................
33
Evaluations on Standard IR Benchmarks
33
E.6
Comparison with RL-for-Search Approaches
34
F.7
Rec-R1 under Multi-Objective Reward Optimization
34
18

---

Published in Transactions on Machine Learning Research (10/2025)
F Discussion and Future Directions
35
G Case Study
36
Product Search: ESCI
38
Sequential Recommendation: Amazon Beauty
38
19

---

Published in Transactions on Machine Learning Research (10/2025)
A Applying Rec-R1 to Diverse Recommendation Paradigms
Figure provides a taxonomy of how generative LLMs are incorporated into rocommender systems, adapted from Lin et al. 2025 The laxonomy is divided into two main categories: LLM for Feature Engineering and LLM as Scoring/ Ranking Functions, each with two sub-pau para-paaadggss. In our experiments, we validate the applicability of REC-RI across all of the four major paradigms:
Query Rewriting: In the product search task, we use BM25 as the retriever, and the LLM generates rewritten user queries to improve retrieval. This corresponds to the top-left panel of Figure and demonstrates the of REC-R1 for textual query rewriting.
User-Level Feature Augmentation: When combined with dense discriminative models such as BLAIR, the LLM augments the input query with semantically richer expressions user intent. This mirrors the top-right panel of Figure where L.LSs act as textual feature generators for downstream ranking models.
Open-Set Item Generation: In the sequential recommendation task, the LLM provided only with user-side information (e.g,, purchase history), and required to generate a textual prediction of the next item. This text then matched against the item pool to select the final recommendation. This aligns with the bottom-right panel of Figure
Closed-Set Item Generation: In the product re-ranking task, the LLM trained to rank or score candidates conditioned on user query and initialized product candidates through reward signals from ranking quality. This aligns with the bottom-left panel in Figure 1 Notably, prior work in this direction has largely relied prompting (Zhang et al. 2023a Hou et al. 2024b) Di Palma et al. 2023 Di Palma 2023) or SFT (Yang et al. 2023 Luo et al. 2024b). However, as shown in our generalization experiments, such approaches may lead to catastrophic forgetting. In contrast, REC-R1 maintains strong domain-specific performance without sacrificing general capabilities ($3.4)
making it a more robust solution.
These results collectively highlight the versatility of REC-R1 -it is applicable across diverse paradigms in recommender systems, and agnostic to the architecture of the retriever or scoring model.
A Path Toward Unified Training. Given its compatibility across paradigms and its ability to retain general-purpose capabilities, we believe REC-RI provides strong foundation for continual, reinforcement- based alignment of LLMs with evolving recommendation goals. Future work will explore extending REC-R1 into more tasks, ultimately enabling lifelong recommendation agents that adapt flexibly to new tasks and domains without retraining from scratch.
B Related Work
B.1
Generative LLMs for Recommendation Tasks
Recently, large language models (LLMs) have significantly impacted recommendation systems by leveraging their strong generalization, reasoning, and semantic understanding abilities. These approaches can be broadly categorized into several main directions.
Feature Engineering and Augmentation. LLMs have been extensively used to enrich recommendation data. They generate auxiliary features that enhance user profiling and item understanding, thus addressing
data sparsity and improving the recommendation quality (Xi et al. 2024 Liu et al. 2025 Torbati et al.
2023
Shi et al. 2023 Li et al. 2024).
LLM as Scoring and Ranking Functions. Researchers have adapted LLMs as direct ranking or scor- ing components within recommendation pipelines. Methods such P5 (Geng al.) 2022). M6-Rec (Cui et al. 2022), and InstructRec (Zhang al. (2033)) explore LLMs' ability simultancously handle multi- ple recommendation subtasks, including scoring, generation, and re-ranking. Models like RecRanker [Loo
20

---

Published in Transactions on Machine Learning Research (10/2025)
et al. 2024a further leverage LLMs' natural language understanding to integrate multiple ranking strategies iveey.
Conversational and Interactive Recommendations LLMs facilitate more sophisticated interactions
between users and recommendation systems through conversational agents, significantly enhancing user
engagement and recommendation explainability (Luo et al. 2024a Zhou al.
2020 Gao et al. 2023b
For a comprehensive review of how recommender systems benefit from LLMs across different pipeline stages and application scenarios, readers can refer to the recent survey by Lin et al. (025). Notably, our proposed REC-R1 framework is broadly applicable across these paradigms- is not tailored to any specific application scenario or retriever architecture, but instead provides a general modeling and optimization approach for aligning LLMs with recommendation lasks via closed-loop learning.
B.2 Reinforcement Learning for Recommendation Systems
Before the era of large language models (LLMs), reinforcement learning (RL) had been explored in rec- ommendation systems for various objectives, such as optimizing peng-ermm user satisfaction and improving soquential pociiion-mmkinn. These works typically reformulate recommendations as a Markov Decision Pro-
cess (MDP), enabling agents to learn from user interaction sequences (Wang et al. 2020; Zhhoo el al. 2018). For example, Liu et al. (2023) extend DDPG session- based recommendation, while Xin al. (2020) pro- pose self-supervised RL with SQN and SAC. Additional efforts incorporate negative sampling, contrastive learning, and reward modeling to enhance learning signals (Ren et al. 2023 Xin et al.) 2022a b). Different from these approaches, which aim to improve the recommendation model itself, our method treats the rec- ommender as a black-box environment. We instead apply RL to optimize the LLM's generation policy, using feedback from the recommendation system to guide LLM training, which enables task-specific alignment without altering the underlying recommender.
In contrast, with the rise of LLMs, recent efforts begin to explore how integrate RL and LLM improve recommendation systems. For instance, [eoong et al. (2023) apply RLHF to align language model with factuality, personalization, and appeal in movie recommendations. However, their framework uses reward model trained offline, without interacting with the recommender system in the loop-mirgoring the RLHF setup of InttructGPT. This approach not only lacks real-time adaplation to system feedback but also risks reward hacking. Other efforts take alternative views: [Sun et al. (2024): Lu et al. (2024) attempt to bring in RecSys feedback but either restrict to DPO-style offline preference tuning (casily overfit on the static datasets) or narrow scenarios like sequential recommendation with fixed candidate sets.
Different from all of the above, method REC-R1 directly optimizes the LLM with real-time reward signals from the recommendation system. The recommender is treated as a black-box environment, and the LLM adapts its generation policy (e.g., query rewriting or user profile generation) to maximize actual downstream task performance. This closed-loop RT. training allows for general applicability across various recommendation tasks and retrievers, without relying on complex reward models or curated preference labels.
B.3
LLMs for Query Rewriting
A growing body of research has investigated using large language models for query rewriting in retrieval tasks. In particular, recent works have explored leveraging LLMs to reformulate queries either in a zero-shot manner or via supervised fine-tuning (Gao et al., 2023a Khattab & Zaharia 2020 Ye et al.] 2023 Mackie ct al. 2023 Liu et al. 2022). These methods demonstrate that LLMs can generate scmantically richer queries that better align with downstream retrieval models, thereby improving recall and ranking performance.
While these works provide important insights into query rewriting and optimization, there remain several key differences compared with our proposed Rec-R1 framework. First, closed-loop recommendation feedback: prior methods typically rely on synthetic or proxy reward signals, whereas we directly employ recommenda- tion evaluation metrics (e.g., NDCG, Recall) from downstream models as reward signals- thus establishing a true closed-loop interaction between the LLM and the recommender system. Sccond, model scale and capability: most existing approaches were developed before the LLM era and used relatively weak models without strong reasoning or generalization abilities. In contrast, our work explicitly adapts powerful open-
21

---

Published in Transactions on Machine Learning Research (10/2025)
domain LLMs recommendation scenarios, addressing the capability gap of traditional RecSys. Finally, task scope: prior studies primarily focused on improving search query refinement, while our framework is designed to support broader range of recommendation tasks.
C Theorems and Proofs
C.1 Proofs of Fact
Lemma 1 (MLE Minimizes KL Divergence). Let (g(a))) a fized target policy (e.g., the data-generoting policy), and let no(ass) be parameterized policy class. Consider the following marimum likelihood estimation (MLB) objective:
max (aa) [og (a(s))
Then mazimizing this objective with respect to 0 is equivalent to måndmizing the expected Kullback-Leible (KL. divergence between the target policy ag and the porameterized policy mo:
min B.-(()) (F((()())]])
Proof. We start by considering the expected negative log- likelikdd under the distribution induced by Tg.
ta)p log Te Te(a|s)] Essp(() Bans) log (a(ss)
(6)
E sp(()
Σ Ta(als){ loggllooaaa =o(ass))
(7)
By definition of the KL divergence, we have the identity:
Ty (a(a(s) (a(a)s)
DKL (T(())))][0)()
Σ Ty(als) (s)) log
Rearranging terms gives:
Panny(ale)) log [a(as)] = DRL. (-1))==]]== + TH mg (11))
where H((r((1ss) Ea Tg log (als) the entropy of the fixed distribution Tg(-1s), which indo- pendent of 0.
Taking expectation over p(s), we obtain:
Panple) log na(a]=]] = Egsp(e)
+ Essp(e) [H(T
Since the second term is independent of 0, minimizing the negative log- is equivalent to minimizing the KL divergence. Thus the lemma follows.
Below is the proof of Fact
Proof. By Lemma 1, the supervised fine-tuning (SFT) objective of maximizing the expected log- lioeiiiooood::
max (a na (a]s)]
is equivalent to minimizing the KT. divergence:
min E Es Da
22

---

Published in Transactions on Machine Learning Research (10/2025)
Under assumption (iii) (Data Sufficiency), the number of samples N + co, the empirical distribution P(s, a) almost surely converges to the true distribution p(s)= Hence, the empirical optimization ob- jective:
Σ
log (a(as)
N
(s,0)~((a.))
almost surely converges to the true expectation:
Eemp(s) e) [lg (a(a)))]
Thus, under assumptions (i) (Sufficient. Expressivity) and (ii) (Optimization Convergence), the optimiza- tion process applied to the empirical objective finds the global optimum that minimizes the expected KT divergence. Formally, we have:
Tg- = arg min Bs ( mm
This completes the proof.
C.2 SFT Performance Bound Terms of KL Divergence
In this subsoction, we invostigate the performance gap between the supervised fine-tuning (SFT) policy TsFT and the fixed target policy Fg. Our goal to establish that the difference expected return between any policy and Tg is controlled by their expected KL divergence. This provides theoretical justification that SFT, which minimizes the KL divergence to Tg. achieves performance within O(yk) of the teacher policy Wg.
Let the downstream metric f(a 8) be bounded in [0, Rmax], and define the performance of a policy " as J(T) == Essp(s) Eas s)]
Then, we have the following theorem, which is simplified version of the classic performance difference theorem [Schulman et al. 2015).
Theorem 1 (Performance Difference Upper Bound). Let be any policy and Tg be the data- duta-generuting (teacher) policy. Suppose the dounstream metric is bounded by 0 S f(a s) I Rmax- Then the performance difference salisfies
[J(T) - J(rg) $ Rmax / Eump(s) [DKL (T 8) ( [s)]]
Proof. Fix state Define
P(a)= ay s), Q(a) Taa s), ga(a) [lals]
E 10, 1]. Ruux
The reward difference at can be written as
(8) = Ea>qlf(a s)] Barrsf(e 8)] Rmax (Eelga) Er
By the dual representation of total variation,
[as)]| S Rmax TV( (8), T (-(()s)) 8)).
Since the performance of policy and my is defined
((T) Espp Ea (a |s]] [(TT) = Eamp Ban [(s(a) 8)]
Then we can write
((x) [(Tg)
Egnp A(x).
23

---

Published in Transactions on Machine Learning Research (10/2025)
By applying the inequality [[[]]| - ||| obtain
[m(m J(Tg) EBpp [()) (espl(A(()]] I Rmax Egpu [T[( 88, To(:1 [s)]]
Next, by Pinsker's incquality, for each s we have
TV( (() (-)1)))))))))))))))))))) s)) s V D ( (s)
Averaging over and applying Jensen's inequality for the concave square root, we conclude
E.[TV] S / B.IDKL(TT( 8) (-- s))]
Combining these inequalities gives the desired bound. Done.
Implication for TSFT- Since Theorem [ established that #SFT minimizes the expected KL divergence to Rg: the above result implies that the performance difference between TSFT and Tg is most O(vR), where
K E Espp(s) [ (s) TSFT(- s))]
Thus, TSFT's performance is limited by the teacher policy Tg-
C.3 Dominance of RL Objective over SFT
We now compare the performance of the REC-R1 solution Tal with that of the SFT solution TSFr- Recall that the performance of policy is defined as
J(m) (e-pte)) s)],
Then, we have the following thcorem.
Theorem 2 (Superiority of RL over SFT). Let Tg denote the teacher policy, and let TSFT E II be the supervised fene-tuning solution oblained by minimizing Rgpppe)) (DRI +)][z(- s))] over the policy class
II Let TRL E arg en (x) denote the RL solution, i.e., the policy in 11 that mazimizes the expected reward. Then
J(TRL. 2 [TSST))
Proof. From Theorem we know that the performance gap between TSFT and the teacher policy T, satisfies
(((SFTT) J(TT)) s ESFT,
where ESFT Rmaxx V x and (MRr (: () TSFT( s))].
Since TnL is by definition the maximizer of J(r) over II, and #SFT E II, we directly have
J(TRL) 2 J(TSFT).
Done.
Implication. This theorem shows that reinforcement learning (REC-R1), by explicitly optimizing the downstream objective J(T), guarantced achieve performance no worse than SFT. In particular, while TSFT constrained to imitate ng (with performance at most O(vk) close the teacher), the RL policy ARL can potentially surpass the teacher by exploiting exploration and reward optimization.
24

---

Published in Transactions on Machine Learning Research (10/2025)
D Estimated Cost of SFT Data Generation and Rec-R1 Model Training
To assess the cost-effectiveness of our approach, we compare it against supervised fine-tuning (SFT) baseline that relics on GPT-too-enerated instruction data. The SFT pipeline requires two stages: (1) generating 5,408 samples using GPT-4o, which costs approximately $10.82 based on OpenAT's pricing ($2.50 per million input tokens and $10.00 per million output tokens), and (2) training the model on two NVIDIA A100 GPUs for 35 minutes (2 epochs), which adds an additional $4.78 (at an on-demand rate of $4.10/ hour per A100 via AWS). The total cost for SFT amounts to approximately $15.60.
In contrast, REC-R1 requires no external data generation: it trains directly on synthetic data produced online by itself during learning. Remarkably, with only about 210 seconds of training on the same two A100 GPUs (costing just $0.48), REC-R1 already matches and even surpasess the performance of the SFT-trained model. Morcover, performance continues to improve with further training. This comparison highlights the substantial cost-cfficiency of REC-R1: achieves superior performance at less than 1/30 of the SFT pipeline cost.
While our cost comparison is conducted small-scale experiment, the implications become more pro- nounced in real-world deployments. Supervised fine-tuning methods typically require generating millions of training examples and running long training cycles, leading to substantial costs in both data creation and computation. In contrast, REC-R. eliminates the need for offline data generation and learns efficiently through online interaction, making significantly more cost-effective and practical solution for large-scale recommendation systems.
E
Additional Experiment and Results Details
E.1
Product Search
E.1.1 Dataset Details
For the conventional setting, we use the ESCT dataset processed by Hou et al. (2024a): a benchmark derived from Amazon product search logs. Following their preprocessing protocol, we focus on four representative product categories: Video Games, Baby Products, Office Products, and Sports and Ouldoors. We construct category-specifie splits with 4,510 training examples, 898 validation examples, and 798 test examples. For all ESCT experiments, we use the same item pool as Hou et al. (2024a), which contains 1,367,729 product listings. Each category uses the full item pool for retrieval.
For the complex setting, we use the Amazon-C4 dataset introduced in Hou et al. 2024a). which contains complex natural language product queries. Since the released dataset provides only category-labeled test queries, we treat the four domains used in ESCI as our test set, and use queries from all other domains the training and validation data. This results in 18,126 training examples, 2,722 validation examples, and 1,722 test examples. The corresponding item pool consists of 1,058,417 products, identical to that used by Hou et al. (2024a). As with ESCI, cach domain-spocific split uses the full item pool for retrieval. This cross-domain setup allows us to evaluate the generalization capability of REC-RI when applied to unseen product categories in a more realistic, open-ended retrieval scenario.
E.1.2 Implementation Details
We implement REC-R1 using the VeRL library]] and run all experiments on two NVIDIA A100 80GB GPUs.
Retriever Setup. We support both sparse and dense retrievers in our framework. For sparse retrieval, we use Pyserini (Lin et al. 2021) with Lucene's s BM25 implementation. Following standard practice, we set the BM25 hyperparameters as k = 1.2 and b 0.75. For dense retrieval, we build HNSW-based FAISS first I.2-normalized to enable cosine similarity
(Johnson et al. 2019) indices. The dense embeddings search. We use IndexHNNSWFlat with M 32 and e(Construction-200 to balance search accuracy and indexing speed.
https///gghhbb.cm///oeennnng
25

---

Published in Transactions on Machine Learning Research (10/2025)
Table 5: Prompt used in REC-R1 for product search tasks with BM25, where the LLM generates structured query terms based user query.
Prompt Template for Rec-R1 + BM25 (Product Search)
pen_ssaar/>ssstte
You are helpful AI assistant. You first think about reasoning process in the mind and then provide the user with the answer. <lim_end|>
< <lim_start>>use /Suser
You are an expert in query generation. Given a query, your task is to create query terms to retrieve the most relevant products, ensuring they best meet customer needs.
Below is the query:
{user query) (user_qqeryy)
Show your work in <think>\think> tags. Your final response must JSON format within <answer>\answed tags. The generated query should use Boolean operators (AND, (R) to structure your query logically. For example, <answer>
{ "query": xxx }
</answer>.
<
< <lim_stard /assisstssstt
Let me solve this step by step.
<think>
Training Configuration. We use Group Relative Policy Optimization (GRPO) our reinforcement learning algorithm, following the implementation in VeRL. The language model is initialized from Qwen-2.5- 3B-Instruct, and optimized with KL-regularized policy gradients. To control policy divergence, we apply a low-variance KL. loss with coefficient. 0.001.
Each prompt is used to generate 12 sampled responses using top-p sampling (p 0.95) and temperature 0.6. The rollout engine uses vLLM with memory budget capped at 30% GPU utilization. Training is run for 5 epochs with a learning rate of 1e-6, 6, global mini-batch size of 256, and micro-batch size of 2. We also enable gradient checkpointing and use Fully Sharded Data Parallelism (FSDP) with parameter and gradient offloading for memory efficiency.
We use NDCG 1000 as the reward during training (instead NDCG@ 100 at evaluation time) to reduce reward sparsity and stabilize learning. All other paramoters follow VeRL defaults unless otherwise specified.
The prompts for product search be found in Table for BM25 and Table f and for Dense retrievers.
E.1.3 Additional Analysis: The Role of Prompt Design and Exploration in RL
In this section, we investigate how different prompt strategies affect retrieval performance when paired with dense retrievers pantisularyyylyy BLAIR, which is pretrained using user reviews and item metadata through contrastive learning. Our hypothesis is that aligning the generation style of the query rewriting process with the pretraining distribution of the retriever could lead better synergy and downstream performance.
As shown in Table g using generic prompts (Table 6) to rewrite queries into natural language variations yields limited or even negative impact across all models, including GPT-4o and 2.om-225-3B--nnttree Moreover, initializing REC-R1 with such rewriting behavior also results in modest performance.
To address this, we experimented with a more targeted prompt strategy: instructing the model to convert the input query into a uscr-style review (Table 7 which better mirrors the training data format used by BLAIR. Interestingly, without reinforcement learning, this "review-style rewriting" strategy alone often hurts performance. However, when training REC-R1 under this revised prompting strategy, we observe that
26

---

Published in Transactions on Machine Learning Research (10/2025)
Table 6: Prompt used in REC-R1 for product search tasks with dense retrievers, where the LLM expands the original query with semantically relevant information to improve retrieval.
Prompt Template for Rec-R1 + Dense Retriever (Product Search)
<lim_start Pssystem
You are helpful AI assistant. You first think about reasoning process in the mind and then provide the user with the answer. <lim_end|>
< |lm_starr/>user
You are an expert in generating queries for dense retrieval. Given a customer
query, your task is to retain the original query while expanding it with additional semantically relevant information, retrieve the most relevant products, ensuring they best meet customer needs. If no useful expansion needed, return the original query as is.
Below is the query:
... fusser query)
Show your work in <think>\think> tags. Your final response must be in JSON format within <answer>\answoe tags. For example,
<answer>
{ "query": xxx }
</answer>.
<
< <lim_start| staar\/aassisantt
Let solve this step by step.
<think>
the model gradually learns to generate review-style queries that significantly enhance retrieval performance. REC-R1 not only recovers from the initially degraded performance but also surpasses all baselines across all four domains, achieving now state-of-the-art results with both BLAIR-BASE and BLAIR-LARGE.
This experiment also sheds light on the importance guided exploration reinforcement learning, especially in language generation tasks with extremely large action spaces. In our setting, the LLM selects an action at each Loken position, and the final output is long sequence meaning the search space is exponentially large. Without meaningful initial guidance, for those very hard problems, early exploration can easily fall into suboptimal regions, from which recovery is difficult due to sparse mislcading reward signals.
E.1.4 Comparison with Other Fine- Tuning Strategies: Rejection Sampling and DPO
To better understand the strengths of our method, compared REC-R1 against two widely used fine-tuning strategies Rejection Sampling Fine-Tuning and Direct Preference Optimization (DPO) within the ESCI + BM25 setting. Table summarizes the NDCG performance across four product domains.
Rejection Sampling Fine- TinenTaning involved filtering training examples where outputs from top-performing LLMs (GPT-4o, Claude-3.--Somnet, Claude-3-Haiku) achieved an NDCG greater than 0.5. These examples were then used to fine-tune the Qwen-2.5 Inssrruttt model.
DPO was trained by selecting, for each training instance, the best and worst candidate queries (by NDCG) from the same LLM pool. The model was fine-tuned for one epoch using these pairs. However, the results were substantially lower across all domains.
From the results, we can observe clear performance gap between REC-R1 and the two bascline fino-tuning methods. While DPO provides theoretically grounded approach to preference learning, its reliance on static preference pairs- without any feedback from the downstream task invironment-linee its generalization
27

---

Published in Transactions on Machine Learning Research (10/2025)
Prompt used in REC-R1 for Amazon- C4 dense retrieval with BLAIR, where the LLM rewrites queries into review-style texts aligned with BLAIR's pretraining objective. Prompt Template for Rec-R1 + Dense Retriever (Amazon-C4, Review-Style Rewriting)
Table 7:
You are helpful AI assistant. You first think about reasoning process in the mind and then provide the user with the answer. <lim_end|>
jim_start|>user
You are an expert in query rewriting for dense retrieval systems. Rewrite the following product search query as if you are a real customer writing a natural, authentic review after using the product. Maintain the meaning and details of the original query, but shift the tone to more casual, emotional, and based on personal experience. Include specific comments about product performance that match the query's intent.
Below is the product search query:
["serr _uuser_
Show your work in <think> </think> tags. Your final response must be in JSON format within <answer> </answer> tags. For example,
<answer>
{ "query": xxx
</answer>.
< <lim_end|>
< im__ttart Passistant
Let me solve this step by step. <think>
ability. The poor performance of DPO across all domains suggests that, in complex ranking scenarios, such offline preference supervision may be insufficient and prone to overfitting.
Rejection sampling achieves moderately better results. However, this approach still lacks mechanism to iteratively refine the policy based on task-specific feedback. In contrast, REC-R1 benefits from a closed-loop optimization process, where model updates are continually guided by performance within the recommenda- tion environment.
E.1.5 Applicability to Alternative Backbone Models
To evaluate the generalizability of REC-R1, we conducted additional experiments using a variely of backbone models beyond Qwen-2.5-3B. Specifically, we tested REC-R1 on smaller models such as Qwen-2.5-0.5B and different model family Laaily--LLM--3.2-3B All experiments were performed under the same ESCI + BM25 setting, with results presented in Table 10
Qwen-2.5-1.5B, well
The results show a consistent and significant performance boost when applying REc-R1 across all backbone sizes and architectures. For instance, REC-R1-0.55 achieves an average improvement of over 13 points in NDCG compared to the base Qwen-2.5-005B model, outperforming even some larger models. Similarly, REC-RI applied I.I.MA-3.2-3B yields results comparable to or better than its Qwen-basod counterpart, despite architectural differences.
E.1.6 Comparison with Modern Strong Query Rewriting Baselines
We further compared REC-R1 with several recent and strong LLM-assisted query rewriting and expansion baselines both the ESCT dataset and the Amazon-CA corpus with BM25 retrieval. The compared baselines
28

---

Published in Transactions on Machine Learning Research (10/2025)
Table 8: Performance comparison between general query rewriting and review-style query rewriting under BLAIR-Base and BLAIR-Large. We observe that while prompting alone offers marginal negative gains, REC-R1 achieves significant improvements especially when aligned with the inductive biases of dense re- trievers (e.g., BLAIR pre-trained on review-style text).
Model
Video Games
Baby Products
Office Products
Sports and Outdoors
Base retriever
BLAIRBASE
19.14
19.53
17.43
20.02
BLAIRLARGE
24.86
22.44
18.92
24.54
Using general query rewriting prompts
GPT--lo,BLARRBBA
21.12
19.54
17.22
22.68
23.40
21.00
18.69
25.74
Qwen-2.5-3B-Insst
-..--3--nnstruut--LLAee BASE
15.82
18.07
14.34
16.96
LARGE
18.20
19.19
15.37
18.94
R- BLAIR-BASS
19.65
20.85
18.91
22.29
REC-R1-3B, BLLIB-BABAARR
19.26
21.63
18.93
64
Models convert queries into reviews
GPT-4O.BIAARRBAAS
20.61
19.35
17.63
21.74
GPT-40+BLARR LARGE
16.59
15.14
15.05
16.76
20enn22553B-Instrucee BLLRRRCC
19.40
16.73
15.59
18.50
22.06
18.31
16.51
20.65
REc-R1-3B,BLAe BASE
21.69
25.62
22.17
24.22
RRC-R1-3B, LARGE
26.51
27.04
23.10
27.40
Table 9: Comparison of NDCG performance across product domains using different fine-tuning strategies under the ESCI + BM25 setup.
Model
Video Games
Baby Products
Office Products
Sports and Outdoors
Owen-2.5-3B-Rol Rej-Sample
22.92
20.67
27.01
24.89
Owen-255-BBDDPO
14.53
12.57
13.52
13.32
R1--R113B
33.89
29.27
34.61
31.92
include DocT5Query Nogueira et al. 2019), RetPO Yoon et al. 2025 LLM4CS (Mao et al.
2023 and
AdaQR (Zhang et al. 2024). Results are summarized in Tables I1 and 12
On ESCI, REC-R1 achieves substantial gains over all baselines across the four domains. On Amazon- C4, the performance gap is even more striking. REC-R1 surpasses competing approaches by large margins. Overall, these results clearly demonstrate that REC-R1 consistently outperforms modern strong LLM-assisted query rewriting basclines.
E.2 Sequential Recommendation
E.2.1
Dataset Details
We conduct our sequential recommendation experiments on the Amazon Beauty dataset curated by Hou ct al. (2024a). Following their protocol, we split the data chronologically based on absolute timestamps into training. validation, and test sets. The final splits include 96,778 training samples, 3,538 validation samples, and 1,538 test. samples-comprising comprising 1,000 inductive and 538 transductive test cases. All experiments use the Amazon Beauty item pool containing 43,982 unique products, consistent with the setting in Hou et al. 2024a).
E.2.2 LLM Input Construction for Sequential Recommendation
To adapt the sequential recommendation task for use with generative LLMs, we convert each user's interac- tion history into natural language format. Specifically, for each historical item, concatenate the titles
29

---

Published in Transactions on Machine Learning Research (10/2025)
Table 10: NDCG performance of REC-RI across different backbone models under the ESCI + BM25 setting.
Model
Video Games
Baby Products
Office Products
Sports and Outdoors
Qwen-2.5-0.5B
10.88
12.93
20.21
17.09
REc-R1-0.5B
27.18
25.14
34.80
30.70
Qwen-2.5-1.5B
15.14
14.09
21.77
15.84
REC-R1-1.5B
31.68
26.46
32.92
30.87
LLaMA-3.2-3B
19.16
16.87
19.35
16.90
REC-R1-3B (LLaMA)
32.41
29.40
34.26
31.39
Table 11: Comparison with modern query rewriting baselines on the ESCI + BM25 setting, We report NDCG@100 across four domains. The best performance is denoted in bold.
Model
Video Games
Baby Products
Office Products Sports and Outdoors
DocT5Query 15.02
14.37
17.19
16.83
RetPO
23.43
17.88
19.00
20.80
LLM4CS
23.29
21.02
28.06
28.00
AdaQR 23.53
20.59
29.02
26.36
REC-R1 33.89
29.27
34.61
31.92
using newline characters (An) as separators. To ensure compatibility with the LLM's input length limit (set to 512 tokens in our experiments), we retain only the latest 10 items from the history list. The resulting text sequence serves the context for generation.
This processed history text is then formatted into prompt for the LLM to generate guess of the next item the user might want. An example of this prompt. format is shown in Table 13
E.2.3 Implementation Details
The training sctup for the sequential rocommendation task largely mirrors that of product search, including the use of the VeRL library and two NVIDIA A100 80GB GPUs. One notable difference lies in the input length configuration: we set max prompt length to 512 (instead of 256), since each input includes full user history list composed of multiple product titles, which tends to be significantly longer than single-turn queries in product search. All other hyperparameters remain unchanged.
E.2.4 Additional Analysis: Why Rec-R1 Performs Better in the Inductive Setting?
Notably, find that REC-R1 achieves stronger performance in the inductive setting compared to the trans- ductive dee-des the latter having more item overlap with training data. This may seem counterintuitive at first, but we believe the reason lies in the nature of our framework and the task formulation.
In the transductive setting, many test items have already appeared in the training data. Traditional models can exploit this overlap through direct memorization or overfitting to item co-occurrence patterns. However, in REC-R1, the LLM is trained infer the next item via natural language generation, which requires capturing underlying intent or semantics. When the task itself lacks strong logical mapping from history to future items is often the case in sequential recommendation language- based reasoning becomes less effective and may even introduce noise.
In contrast, the inductive setting removes such memorization shortcuts, as the target items are completely unseen during training. This forces the model to rely on more transferable semantic patterns, which better aligns with REC-R1 's learning mechanism. The LLM is incentivized to produce generalized, meaningful descriptions that reflect what kind of item could come next-rather than relying on item identity. As a result, the inductive setting provides a clearer signal for reward-driven optimization, enabling REC-R1 shine where conventional models struggle.
30

---

Published in Transactions on Machine Learning Research (10/2025)
Table 12: Comparison with modern query rewriting baselines on Amazon -4 BM25. NDCG@100 across four domains. The best performance is denoted in bold.
We report
Model
Video Games
Baby Products
Office Products Sports and Outdoors
DocTryuery 3.90
5.81
4.93
6.43
RetPO
5.20
1.40
1.64
2.54
LLM4CS
13.05
9.97
11.51
11.05
AdaQR
12.98
9.15
10.47
10.41
REC-R1
26.51
27.04
23.10
27.40
Table 13: Prompt format used for LLM-based generation in the sequential recommendation task. The input
includes the user's purchase history and instructs the model to output structured query terms for the next likely purchase.
Prompt Template Used for LLM Input in Sequential Recommendation
< <|im_stee Siim_star|/aa |>system
You are a helpful AI assistant. You first think about the reasoning process in the mind and then provide the user with the answer. <lim_end|>
< <lim_star//uuse
You intelligent shopping assistant that helps predict what users may want purchase next. Below is list of items user has purchased recently. Your task is to infer one or multiple kinds of products they may want to buy next, and generate relevant query terms that can be used to search for these potential products.
Below is the purchase history: fpprrchas__haae history)
Show your work in <think>\thinke tags. Your final response must be in JSON format within <ansver>\answer. tags. The generated query should use Boolean operators (AND, OR) structure your query logically. For example, <answer>
{ "query": xxx }
</answer>.
< im im__nnd|>
< im__tarr Passistant
Let me solve this step by step.
<think>
E.3 Product Re-ranking
E.3.1
Dataset Details
We build the re-ranking dataset starting from the ESCI corpus curated by Hou al. (024a)). For each split (train/validation/tees
we first obtain initial candidate lists using the RectRi-Retrieve trained in the Product Search task (Section 3.18, combined with BM25 retrieval. We select the top-16 items as the candidates. To ensure data quality, we filter out samples with zero retrieval performance, ie., those for which the upstream retriever achieves NDCG@20 0. After filtering, cach data point consists of the user query, the candidate list (input to the re-ranker), and the corresponding ground-truth relevant items. We finally obtain 2, 166 521, and 465 samples for the train, validation, and test splits, respectively.
31

---

Published in Transactions on Machine Learning Research (10/2025)
Table 14: Prompt used for the product re-ranking task.
Prompt Template for Rec-R1 Product Re-ranking
<lim_start Psystem
You are helpful AI assistant. You first think about reasoning process in the mind and then provide the user with answer. 

<lim_start /usee
You are an expert in product reranking. Given customer query and a list of candidate products, your task is to rerank the products so that the most relevant items to the query are placed at the top. Consider semantic meaning, product attributes, and customer intent.
Below are the candidate products (item id and metadata): # [candidates)
Below is the query: # "[user_query]
Show your work in <think> </think> tags. Your final response must be in JSON format within <answer> </answer> tags. The output JSON should contain the reranked list stem_ids in order of relevance. The length of the reranked list should match number input candidates. For example, <answer>
{ "reranked_iten" ["Item_4", 4", "Item_8", VItem_x", ... all candidate item ids in order]
</answer>
< im__ndd>>
< im_start Passissant
Let me solve this step by step. <think>
E.3.2 Implementation Details
The overall training and evaluation setup follows the same configuration as in the product search and sequen- tial recommendation tasks. We highlight the key differences below: (1) the LLM prompts are specifically designed for the re-ranking scenario, as shown in Table 14 and (2) the input/ length limits are adjusted, with max prompt length set to 3000 and max response length set 1024.
E.4
Evaluation of Generalization and Forgetting
E.4.1
Implementation Details
To assess whether REC-R1 preserves the general-purpose capabilities of the underlying LLM while achiev- ing strong lask-specific performance, evaluate all models across suite of generalization benchmarks. Specifically, we consider six datasets spanning different task types and reasoning skills:
ESCI (NDCG@100) - Product search recommendation, serving as the target task of optimization.
MMLU (Aceuracy) - A factual knowledge benchmark covering multiple-choice questions across various domains (Hendrycks et al. 2020
IFEval (Strict Accuracy) A benchmark designed to evaluate instruction-followingg and alignment
with user intent (Zhou et al. 2023
GSM8K (5-shot, EM) Math reasoning with elementary school word problems a few-shot
setting, measured by exact. match (Cobbe al. 2021
32

---

Published in Transactions on Machine Learning Research (10/2025)
MBPP (3-shot, pass@1) A coding benchmark consisting of short Python problems, evaluated using pass@1 (Austin et al. 2021
HvmanEval (0-shot, pass@1) A high-quality Python programming test measuring zero-shot code generation performance CChnn et al., 2021
All evaluations are conducted using the im-evalluation--arnes library (Gao el. al. 2024 ensure consis- tency and reproducibility. For ESCI, we directly compute NDCG@ 100 based medel-generated rewritings. For all other datascts, we use the official protocols defined in lmeval.
E.4.2 Additional Analysis: Impact of Reasoning and JSON Format in SFT
To better understand the effects of prompt format on supervised finc-tuning (SFT), we explore four SFT variants that differ in whether the training data includes intermediate reasoning steps and whether the answers are wrapped in structured JSON format. We use -P----o-generated data on the ESCI product scarch task for training all four SFT models. Table 15 shows results ESCI, and Table 16 evaluates generalization to broader benchmarks.
On the task-specific ESCI dataset, all SFT variants outperform the base model (Qwen-2.55) (.n-ru--nst demonstrating the effectiveness of supervised fino-tuning on task-specific data. However, all variants fall short compared to REC-RI, which uses the same data but trains via reinforcement learning. This highlights the advantage of reward-driven learning aligning with downstream task metrics.
We then assess the general-purpose capabilities of these models. On MMLU, knowledge-intensive bench- mark, all SFT variants retain performance close to the original model (within 2 points), suggesting factual knowledge is preserved. In contrast, IFEval results reveal catastrophic forgetting across the board-aa accuracy, regardless of format. This under-
SFT variants suffer 20 30 point drops in instruction-follwwiii scores the risk of overfitting in SFT, where tuning on narrow task data compromises broader generalization.
An interesting observation arises on GSMSK: the variant with JSON formatting but no reasoning shows improved performance over the base model (+4.7). We hypothesize that the strict output format (JSON) acts "shield," isolating the fine-tuning effects from interfering with the model's native reasoning process. In contrast, the reasoning-heave variants modify the generative behavior more substantially, harming out- of-domain reasoning.
On the coding benchmarks (MBPP, HumanFval), all four SFT variants exhibit comparable or slightly improved performance relative to the original model regardless of whether the training outputs used JSON format. This suggests that coding ability is relatively robust task-specific SFT, and may even benefit from it. One possible explanation is that the ESCI task, although unrelated to coding, implicitly encourages structured generation and logical formatting (e.g., conditionally constructed queries Boolean expressions). which aligus well with the formal nature of code generation.
In contrast, REC-RI R1 avoids these trade-offs altogether. Rec-R1 improves task-specific performance while maintaining general capabilitics across reasoning, knowledge, and code generation. These results provide further evidence that REC-R1 a more stable and generalizable learning framework than conventional SFT.
E.5
Evaluations on Standard IR Benchmarks
To further examine the generality of our framework, we conducted evaluations on several widely used infor- mation retrieval (IR) benchmarks, including MS MARCO (Nguyen el al.) 2016), two representative datasets from BEIR (hakur et al. 2021) (NFCorpus and FEVER), as well as the TREC Deep Learning tracks DL'19 and DL'20 (Dai et al. 2024). Rosults are summarized in Table 17 Across all benchmarks, REC-R1 consistently outperforms the baselines. These results further validate the effectiveness and robustness of our framework under standard TR benchmarks, even though retrieval is not the primary focus of this paper.
33

---

Published in Transactions on Machine Learning Research (10/2025)
Table 15: Performance on ESCI datasets across different domains. Note: The REC-R1 results here may differ slightly from Table due to using the checkpoint at step 1400 instead of step 1390. This minor difference has negligible impact performance. We report the performance of models under different training steps because evaluation on validation and test set is done every 10 steps and models are saved every 100 steps. We use the 1400-step checkpoint for consistency in follow-up experiments.
Model
Reasoning
JSON
ESCI (Games)
ESC1 (Baby)
ESCI (Office)
ESCI (Sports)
Qwen- 20----2------stteii
19.63
16.03
19.96
21.36
Qwen-2.5-3D3 1ett-...-3B-nsstuuee (SFT)
V
25.70
19.66
27.34
24.82
Qwen-2.5-33 1-5-3B-Instruct (SFT)
X
26.87
22.83
26.10
26.42
1ett-..5-3--nstrrrce (SFT)
X
X
25.08
21.50
28.75
25.18
Qwen-2.5-3B-Instruue (SFT)
23.31
20.77
24.34
24.78
RE--R1-3B (1400 steps)
32.92
29.62
35.05
31.21
Table 16: Performance general-purpose reasoning and coding benchmarks. Color-coded deltas show change from baseline Qwwn-2.553B--nnttree
Model
Reasoning JSON
MMLU
IFEval
GSM8K
MBPP
HumanEval
Meen-2.5-3B-Instrree
65.4
58.2
63.4
53.6
46.3
Qwen-2.5-3B-Instruuc (SFT)
v
63.7 (-1.7)
31.4 (-26.8)
35.7 (-27.7)
54.8 (+1.2)
53.6 (+7.3)
Doven22.533B-nnstrre (SFT)
x
64.1 (-1.3)
30.5 (-27.7)
68.1 (+4.7)
50.4 (-3.2)
57.3 (+11.0)
Swen-225-3B-Innsee (SFT)
64.3 (-1.1)
34.4 (-23.8)
37.3 (-26.1)
52.2 (-1.4)
53.6 (+7.3)
Bren--25-3B-Insttree (SFT)
63.6 (-1.8)
35.0 (-23.2)
50.2 (-13.2)
55.4 (+1.8)
52.4 (46.1)
R1---R1-3B (1400 steps)
65.3 (-0.1)
60.1 (+1.9)
69.1 (+5.7)
54.4 (+0.8)
46.3 (40.0)
E.6 Comparison with LL-for-Search Approaches
Recent studies have also applied reinforcement learning with verifiable reward (RLVR) for search. We high- light two representative approaches-Searreee compare them against our proposed Rec-R1.
(Jin et al. 2025 and ConvScarch-R1 (Zhu et al. 2025 and
Search-R1. As described in Section 3.4 of their paper, Search-RI -R1 relies solely on single exact-matching reward. In contrast, Rec-R1 directly optimizes downstream recommendation metrics (o.g., Rocall, NDCG) the reward function. This distinction is crucial: due to the nature of RLVR optimization, Search-R1 cannot explicitly constrain the quality of the intermediate retrieval process, while Rec-R1 is directly aligned with the final RecSys evaluation objectives. As shown in Tables 18 and 19 Rec-R1 consistently and significantly outperforms Scarch-RI across all domains in both ESCI and Amazon- C4 datasets.
ConvSearch-RI. ConvSearch-R1 differs its reward design: instead of directly optimizing evaluation metrics, it introduces a piecewise-defined reward function (Eq. (1) in their paper). Their motivation is to mitigate reward sparsity, arguing that optimizing NDCG@3 directly would result in too few positive signals. However, argue that such sparsity is only problem if one insists on optimizing at very small cutoff (e.g., k == 3). In fact, optimizing at larger & implicitly improves smaller-k NDCG as well. Moreover, ConvScarch- RI's reward formulation does not directly align with evaluation metrics. Empirically, Rec-R1 consistently outperforms ConvSearch-RL on both ESCI and Amazon-C CC (Tables and 19
In summary, Roc-R1 distinguishes itself by (i) reward design that is directly tied to ReeSys evaluation metrics, and (ii) consistently stronger empirical performance on LLM4Rec tasks compared recent RL-for-search baselines.
E.7 Rec-R1 under Multi- Obiti-vveeetivee Reward Optimization
The reward structure in REC-R1 is designed be general and flexible. Formally, the reward defined function f(a(s), where denotes the input state and the model out.ut. This formulation naturally
34

---

Published in Transactions on Machine Learning Research (10/2025)
Table 17: Performance on standard IR benchmarks. We report MRR@10 for MS MARCO and NDCG@10 for BEIR and TREC DL datasots. The best performance is denoted in bold.
Model
MS MARCO NFCorpus
FEVER
DL'19
DL.20
BM25
44.80
14.67
44.25
42.31
47.80
D-ren2.--3-Insstructt Instruct
23.00
29.23
46.68
43.56
52.24
GPTT40
39.43
30.67
53.94
59.87
54.72
Rec-R1
53.13
34.00
66.38
63.25
59.71
Table 18: Comparison with RI-for-search approaches ESCI + BM25 (NDCG@100). Rec-R1
consistently outperforms both Search-R1 and ConvSearch-R1 across all domains.
Model
Video Games
Baby Products
Office Products Sports and Outdoors
Search-R1
18.03
17.38
22.67
19.62
ConvSearch-RI
23.67
22.35
27.89
25.64
Rec-R1
33.89
29.27
34.61
31.92
supports multi-objective optimization: f(als) can be extended to incorporate multiple reward signals via linear combination, such that the optimization encourages trajectories with higher aggregated rewards.
To further validate the flexibility of our framework, we conducted additional experiments to examine whether Rec-R1 can be stably trained under a multi-reward setting. Specifically, we combined two types of rewards: (i) downstream metric reward (e.g., Recall or NDCG), which measures recommendation quality; (ii) a format reward, which ensures well-structured and valid outputs; (fii) Calegory Consistency Reward, which measures the proportion of retrieved items belonging to the same category the ground-truth items given query. The results are shown in Table 20 Rec-Rl continues to perform strongly across all domains, substantially outperforming baseline LLMs such as Qwen2.5-388 10--5----IIsseee and GPT-4o, while maintaining training stability.
These findings indicate that Rec-RI is not limited to single-metric optimization but can be readily extended multi-objective reward structures, offering flexible framework for incorporating diverse recommendation signals.
F Discussion and Future Directions
In this section, we reflect on several insights and limitations emerging from our experiments, and highlight future directions for building stronger recommendaaionn-orenteed LLMs.
LLMs Can Learn to Recommend Without Access to the Item Space. In our product search experiments, REC-R1 operates without any access to the downstream item catalog it only receives the user query 8 and generates a rewritten query a, without knowing which products exist in the recommender's database. Despite this apparent limitation, REC-R1 consistently delivers strong performance across domains. This aligns surprisingly well with human behavior: when people search for products, they rarely know the exact contents of platform's inventory. Instead, they refine their queries iteratively based on vague goals and system feedback. REC-R1, trained in closed loop with the recommender, learns this refinement. process efficiently via reinforcement learning. This result highlights the potential of LLMs to simulate user-side reasoning, making them powerful agents for optimizing recommender interaction.
Toward Better Integration with Sequential Recommendation. Our sequential recommendation setup frames the LLM as a next-item predictor: receives a history of product titles and generates a guess of the next likely item in natural language, which is then fed to a retriever. While this approach works well in the inductive sctting- -here test items are unscon-it underperforms traditional models in the transductive case. We have previously explain in $E.2.4 A promising direction is use LLMs not as generation agents, but feature augmentation modules: the LLM could enrich each item in the user history with additional
35

---

Published in Transactions on Machine Learning Research (10/2025)
Table 19: Comparison with RI-for-search approaches on Amazon-C4 + BM25 (NDCG@100). Rec-R1 achieves significant improvements across domains.
Model
Video Games
Baby Products
Office Products
Sports and Outdoors
Search-R1
12.06
9.12
10.21
9.96
ConvSearch-RI 13.57
13.08
13.62
13.34
Rec-R1
26.51
27.04
23.10
27.40
Table 20: Performance under multi-objective reward setting ESCI + BM25 (NDCG@100). Rec-R1 combines format reward, NDCG, and category consistency reward.
Model
Video Games
Baby Products
Office Products
Sports and Outdoors
Svee2.5-3B-Instruce
19.63
16.03
19.96
21.36
GPT-4o
26.06
23.05
27.98
27.38
Rec-R1 (multi-reward)
33.06
29.39
34.22
32.27
descriptive cantextual information. These enriched sequences could then be encoded using text encoders like BLAIR and passed into standard SRec models. This hybrid approach could combine LLMs' semantic understanding with the modeling power of sequential architectures for the transductive setting.
Initialized LLM's capabilities Matter. Our findings underscore the importance of the base LLM's capabilities when applying reinforcement learning to complex decision-making tasks. Like traditional RT. pipelines that rely on imitation learning to bootstrap strong initial behaviors e.g., in high-stakes environ- ments like Go and MOBA games (Silver et al. 2016 Vinyals et al. 2019 -RE--RE also benefits significantly from a well-initialized model. In our experiments, Qwen-2.5-32B Ins-trutttttt provides a strong general-purpor foundation. However, we also observe that this general strength does not guarantee effectiveness on every domain-specifie task. For instance, in sequential recommendation, the base model lacks any prior expe- rience in predicting the next item from user histories-resuiee optimization. This raises compelling direction: could domain-specific pretraining (c.g., training LLMs to imitate existing sequential recommender outputs) better equip models to serve REC-R1 agents? Combining domain-aware LLMs with REC-R1 could unlock more powerful and seman- tically aligned generation strategies, especially for tasks like sequential recommendation. We leave it as a future work.
in a weak starting point for RL-based
instruction tuning
Leveraging RecSys Feedback: From Static Logs to Live Interaction. A core advantage of REC- R1 lies in its ability to leverage feedback signals from recommendation systems. In our experiments, these signals are derived from historical interaction logs that are commonly available in large-scale recommender platforms. While the current feodback log-based, this setup aligns well with real-world deployment, where user interactions stream of logs allows REC-R1 to stay aligned with evolving user preferences and content trends. Moreover, REC-R1 is fully compatible with real-time feedback: it can be trained via online interactions with a live recommendation engine, where the LLM receives immediate performance signals (e.g., engagement rates or conversions). This makes REC-R1 a flexible framework capable of serving as a foundation for LLM-based recommendation systems that evolve with real-world usage.
continuously collected in vast quantities. In practice, maintaining up-to-date
G
Case Study
To better understand the behavior and effectiveness of REC-RI, we present qualitative case studies from the product scarch task (ESCI dataset) and the sequential recommendation task (Amazon Beauty dataset). These cases offer insights into how REC-R1 generates more effective textual inputs than prompting- based methods.
36

---

Published in Transactions on Machine Learning Research (10/2025)
Target items
BCOAEXX180
BcCevoOeLL
80034UOHH
B02108238K
DECENGLIIO
Original user query
NDCG@100: 0.00
play station 3
B00OLDKE2A
FK
BOOGCBOSZY
B09L356C36
B07JB1PLIJ
BOOOHQ9RVA
B08VHSHVQV
B0025PYFT2
GPT-4o
NDCG@100: 7.42
("PlayStation 3" OR "P53") AND ("Slim" OR "Super Slim" OR "console" OR "gaming" OR "controller" OR 'games")
B018J1004S
B018JIEETY
B01N9EJ811
B003B01
B00701PFI0 B00701PL42
BOBBT TMBWJ
B07SKFR228
Rec-R1
NDCG@100: 78.53
"playstation 3" (playstation console playstation 3 gaaming system Sony PlayStation
Blu-ra PlayStation 3 Stayita System Architecture
PLAYSTATION Cansole 360 G8 PLAYSTATION 3 1TB PLAYSTATION 3 720p HD Playstation 3 SGB RAM PlayStation 500Ge Blu ray PlayStation 3 Gaming System
PLAYSTATION Console Blu my Disc PlayStation 3 32 GB PS3 GB RAM s00GB Hard Drive PlayStation Gamiing Cansale 360 SSD Original PlayStation Cansole 8GB RAM 720p Resolution PlayStation 3- Best Buy PlayStation Amazon PLAYSTATION Console 500GB Blu-ray Used 60GB PS3 32GB Solid State 8GB RAM 40GB PlayStation Player
Sony PlayStation Entertainment System 36 Trillion Operations Second PlaSStation
GB Station
Sony PlayStation Gaming System
Gaming PlayStation
Storage
RAM 500GB Blurry + PlayStation 3 Console with 2TB Hard Drive"
B000XGJH10
B018/1004S
B000SNLPXU
B0772ZYSVQ
BODAEX81SG
B009DL2TBA
B00E3695DM
B00PXP975S
Figure 5: Qualitative comparison of retrieval results on the ESCI Video Games domain. We visualize the top-8 items retrieved by BM25 using different query formulations: the original user query (play station 3). a rewritten query by GPTT-0, and the output of our REC-R1. Ground-truth relevant items are shown at the top. REC-R1 significantly improves NDCG@100 by generating highly detailed and semantically rich query, enabling precise matching with relevant items. Items correctly retrieved (i.e., appearing in the target set) are highlighted with red bounding boxes.
37

---

Published in Transactions on Machine Learning Research (10/2025)
G.1
Product Search: ESCI
Figure 5 illustrates query rewriting example from the ESCI Video Games domain. The user's original query is simply 'play station 3', which fails to retrieve any relevant items using BM25 (NDCG@10) 100 = 0.00). When using GPT-4o for rewriting, the generated Boolcan query covers many relevant keywords (c.g., "Slim" "controller", "games") but lacks grounding product-specific terminology. leading to limited improvement (NDCG@100 7.42).
In contrast, REC-RI generates highly detailed and context-rich query that resembles human-level product search behavior. It incorporates specific configurations (e.g., storage size, "Blu-ray", "used" vs. "new"), product features, and variant types, all while maintaining semantic coherence. This significantly improves retrieval accuracy, correctly retrieving most of the target items (NDCG&100 78.53). Items that hit the target list are marked with red boxes in Figure []] clearly demonstrating REC-R1 superiority in aligning generation with downstream relevance signals.
G.2 Sequential Recommendation: Amazon Beauty
Figure t shows a case from the Amazon Beauty dataset. The user previously purchased a set of products. The goal is to predict the next item the user is likely to purchase. GPT-4o generates a broad and somewhat vague Boolean query with high-level categories such as "makeup brushes" or "serum", failing capture the continuity in user intent. As result, none of the retrieved items match the target (NDCG@10 100 0.00).
REC-R1, on the other hand, generates natural-sounding pseudo-review that maintains semantic consistency with the user's history while narrowing in on likely next items such as winged eyeliner and related cosmetic tools. This leads to successful retrieval of the ground-truth target item (NDCG@10 = 43.06), as highlighted in rod.
38

---

Published in Transactions on Machine Learning Research (10/2025)
History Items
Target
2 Pack
1 Pack
OI
ABSAN OIL
B01GS8J18E
B085ZXRQ1D
B0852TR42P
GPT-40 ("hair mask" OR "styling gel" OR "hair oil") OR ("mascara" OR "eyeshadow" OR "eyebrow kit") OR ("facial cleanser" OR "moisturizer" OR "serum") OR ("makeup brushes" OR "hair NDCG@10: 0.00 styling tools")
B085MRMX9K
B087NMGGRV
B07N8VSYHN
B07H9823TH
B01GZV7MG2
Rec-R1
Winged eyeliner stamp black liner Perfect Cat Eye Makeup for Moroccan Argan Oil Hair Treatment Gift Set from Moroccan Argan Oil Shampoo and conditioner damaged hair repair Winged liner tips from Leila using natural products Wingliner from Perfect 2 in 1 dual end eyeliner brushes
NDCG@10: 43.06
Pack
2 Pack
B08DKG57TP
B08DKL47BR
B085ZXRQ1D
B085ZTR42P
BO7JQGD781
Figure 6: Case study on the Amazon Beauty dataset for the sequential recommendation task. Given a user's historical purchase sequence (top-left), we compare the top-5 items retrieved by BM25 when queries are generated by GPT-4o (top) and REC-R1 (bottom). Ground-truth target items are shown at the top- right. REC-R1 produces a more contextually relevant and descriptive query, enabling accurate retrieval of both target items, significantly oulperforming GPT-4o. Items correctly retrieved are highlighted with red bounding boxes.
39


Reviews:
[{"review_id": "CSleA9v2gC", "reviewer": "Reviewer_Q5qx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Rec-R1 is a closed-loop reinforcement learning framework that connects LLMs with recommender systems, directly optimizing the generation policy using recommendation metrics. Experiments on product search and sequential recommendation show that Rec-R1 outperforms prompting and SFT approaches while preserving general knowledge and reasoning abilities. The authors demonstrate the broad applicability of different recommender architectures and tasks.\n\nI list key strengths and weaknesses below:\n\nStrengths:\n- The authors effectively use Figure 2 to establish a clear position within existing LLM-based recommendation system methods, effectively communicating their approach to readers.\n- The authors attempt to theoretically support that Rec-R1 can overcome SFT's limitations through Theorem 1. By showing SFT's 'performance ceiling' via Theorem 1, they tried to demonstrate why reinforcement learning is necessary, and empirically show the limitations of SFT through Figure 1(a).\n- The method performs well in cold-start scenarios where user profile information is absent.\n\nWeaknesses:\n- While Theorem 1 attempts to reveal the research motivation and SFT's limitations, the three assumptions required for this may not be realistic.\n- Although the title mentions \"user-centric\" recommendation, the proposed method and experiments are closer to rule-based RL that uses static offline metrics as rewards.\n- Rec-R1 only uses a single reward based on final recommendation quality metrics (e.g., NDCG, Recall). This may lead to insufficient feedback regarding diversity or format consistency.\n- Experiments are limited to small-scale datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "- The title and introduction prominently feature \"user-centric\" and \"closed-loop,\" but the rewards consist of ranking metrics (e.g., NDCG and Recall) from static benchmarks rather than signals approximating actual user utility or long-term satisfaction. This may come across to readers as offline quality optimization rather than learning with user-centric signals.\n\n- The authors use Figure 1 (a) to empirically support SFT's fundamental limitations. From this perspective, their claim is somewhat convincing.\n\n- The proof in Appendix C.1 focuses only on SFT without directly comparing REC-R1's theoretical superiority. This may claim that RL \"overcomes\" SFT's limitations, but it appears to rely solely on the empirical results in Figure 1 for the readers.", "audience_yn": "Yes", "audience_explanation": "TMLR's audience would likely be interested in this paper's approach of using reinforcement learning to connect LLMs and recommendation systems via feedback optimization. In addition, Rec-R1 has attracted considerable attention since its release as a preprint.", "requested_changes": "- I believe it would be beneficial if the authors could additionally address the limitations of the three assumptions in Theorem 1. For example, 3B-sized or smaller LLMs may struggle to mimic the complex distribution of larger models like GPT-4o perfectly. If the learning model has insufficient capacity, the theorem's conclusion may not hold. Additionally, regarding the second assumption, a global optimum is rarely achieved in deep learning. For the third assumption, while the paper mentions \"sufficient data,\" actual recommendation system data is noisy and sparse. Therefore, it seems unlikely to ideally reach the theorem's conclusion. A section or discussion addressing these limitations would be helpful.\n\n- When introducing Equation (5), adding explicit differences from Equation (4) could clarify REC-R1's theoretical superiority. For example, adding a statement like \"Unlike Equation (4), Equation (5) optimizes the true objective rather than a proxy...\" would help readers understand better.\n\n- The results in Table 3's transductive setting, where Rec-R1 lags behind some baselines like UniSRec, need further analysis and explanation.\n\n- From the perspective of Rec-R1's reward structure, I'm curious whether it is less stable compared to the multi-reward structures proposed in Stream-Rec [1] or Search-R1 [2].\n\n- Can the authors report learning and stability results under multi-objective rewards (accuracy, diversity, fairness, efficiency) rather than relying solely on recall and NDCG rewards?\n\n> [1] Zhang, Junjie, et al. \"Slow Thinking for Sequential Recommendation.\" arXiv preprint arXiv:2504.09627 (2025).\n> \n> [2] Jin, Bowen, et al. \"Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\" arXiv preprint arXiv:2503.09516 (2025).", "broader_impact_concerns": "N/A", "additional_comments": "- As mentioned in the introduction, since the proposed framework assumes RecSys as a \"fixed black box,\" wouldn't it be difficult for the LLM to access RecSys's internal logic (e.g., lack of understanding user preferences, difficulty processing implicit feedback)?\n- While the early parts of the paper use terms like \"model-agnostic, task-flexible,\" I'm uncertain whether these terms apply throughout the entire paper. Given the prerequisite that rule-based reward computation must be possible, readers holding onto these terms might become confused as they read through the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "feL2wN5Op1", "reviewer": "Reviewer_MD4W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies ways LLMs have been used in recommendation systems. They focus specifically on the setting where there is an existing recommendation system that turns natural language input into a user recommendation.\n\nPrior methods have tried using LLMs to transform initial natural language input into queries with more information, either using prompted versions of the LLM or applying SFT to the LLMs. This paper instead applies RL, using the reward from the recommendation system as the target.\n\nThere is nothing particularly wrong with the paper's experiments or problem setups, showing that a GRPO based RL training method can optimize the recommendation system better than an SFT setup can, and that RL can outperform the data in SFT generated by a fixed expert (in this case, GPT-4o). I mostly find myself skeptical that this is of value, given that\n\n1) It has been shown outside of recommendation systems that RL can be more effective than SFT at satisfying user preferences, across a wide swath of LLM work, including the very first InstructGPT papers.\n\n2) It has been shown that RL can be an effective means of optimizing reward within recommendation systems, for different deep learning architectures.\n\nSo it's not clear to me where the value of the paper is. I also think that it's a bit silly to include a proof that the MLE objective minimizes the KL divergence between the learned policy and a fixed target policy - this is just a well-known result.\n\nWithin Appendix B, the authors make the case for related work, arguing that prior applications of LLMs to recommendation systems either used an offline reward model rather than the true environment, or used DPO, or used more narrow forms of a recommendation system environment. I'm not sure these are meaningful enough differences.\n\nAs for the estimated cost breakdown, where the authors argue the training of the model runs at 1/30th of the cost of the SFT pipeline, I agree with the numbers as stated, but it is worth mentioning that the Rec-R1 proposal would also involve regret from the currently learning model (the $ lost from exploring different recommendations than the baseline model, from the live recommendation system setup).", "claims_and_evidence_yn": "Yes", "claims_explanation": "Experimental setups seem reasonable, with ablations done across model architectures, different recommendation system settings, etc.", "audience_yn": "No", "audience_explanation": "As mentioned earlier, it is unclear to me that a paper showing RL outperforms SFT for LLMs is of interest, when this has been shown for LLMs in wider settings, and when it has also been shown in recommendation systems without LLMs. That it works in the intersection seems clear to me.", "requested_changes": "Most of my reservations about the work are at the structural level, rather than content level, so I'm unsure there's any adjustment that would affect my recommendation. I would need to see a more convincing argument that this paper says something of interest to the wider ML community.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2pqSqfYAY6", "reviewer": "Reviewer_DH2W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors hook-up a tunable LLM into some tasks the require text (query-rewriting, mostly) and the train it with RL using rewards computed from the end-task (e.g. NDCG). Seems to work well!", "claims_and_evidence_yn": "No", "claims_explanation": "The paper looks good in terms of results on the two recommendation tasks—no concerns there. But the overall framing needs adjustment. Your method essentially tackles a \"query rewriting/expansion\" problem, making it applicable to retrieval systems in general. Given this, you should be comparing your approach against established query rewriting baselines like Doc2Query, DocT5Query (and their modern SOTA brethren), and evaluating it on standard IR benchmarks like TREC-DL, MS MARCO, and BEIR.\n\nRight now, the two recommendation datasets you're using function essentially as retrieval tasks, which means you're doing query reformulation/generation/rewriting + retrieval. But you're claiming this as a general user-centric recommender system framework, which feels overstated. If you truly want to position this as a broad RecSys method, you should show that it also works outside of retrieval-only scenarios. For instance, how would this method integrate with classic collaborative filtering tasks (Netflix Prize, Movielens), where inputs are user/item IDs instead of textual queries? Or do you think this doesn't count as \"user centric\"? If so, what DOES and how is it different from just retrieval? \n\nWhat about the literature? Has anyone else used RL to teach LMs to rewrite queries? This is a pretty interesting and timely topic...", "audience_yn": "Yes", "audience_explanation": "LLMs + RL + retrieval -> what's not to love?", "requested_changes": "Strongly recommended: reframe the paper as being about RL-learned query rewrting. \nIF you choose not to reframe the paper, then (required for acceptance): provide a complete discussion on which kinds of recsys datasets/problems your approach applies to and which ones it does not.\n\nRequired for acceptance: compare to SOTA query rewriting methods on DL19, DL20 and BEIR.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)