Title: Think before Recommendation: Autonomous Reasoning-enhanced Recommender
Venue: NeurIPS 2025
Accepted: yes
Keywords: Recommendation

Large Language Models

Reinforcement Learning

Rating Prediction
Abstract: The core task of recommender systems is to learn user preferences from historical user-item interactions. With the rapid development of large language models (LLMs), recent research has explored leveraging the reasoning capabilities of LLMs to enhance rating prediction tasks. However, existing distillation-based methods suffer from limitations such as the teacher model's insufficient recommendation capability, costly and static supervision, and superficial transfer of reasoning ability. To address these issues, this paper proposes RecZero, a reinforcement learning (RL)-based recommendation paradigm that abandons the traditional multi-model and multi-stage distillation approach. Instead, RecZero trains a single LLM through pure RL to autonomously develop reasoning capabilities for rating prediction. RecZero consists of two key components: (1) "Think-before-Recommendation" prompt construction, which employs a structured reasoning template to guide the model in step-wise analysis of user interests, item features, and user-item compatibility; and (2) rule-based reward modeling, which adopts group relative policy optimization (GRPO) to compute rewards for reasoning trajectories and optimize the LLM. Additionally, the paper explores a hybrid paradigm, RecOne, which combines supervised fine-tuning with RL, initializing the model with cold-start reasoning samples and further optimizing it with RL. Experimental results demonstrate that RecZero and RecOne significantly outperform existing baseline methods on multiple benchmark datasets, validating the superiority of the RL paradigm in achieving autonomous reasoning-enhanced recommender systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes two novel LLM-based rating prediction models, i.e., RecZero with pure RL and RecOne combining SFT and RL. The RecZero pipeline adopts a CoT-based reasoning process, which consists of <analyze user>, <analyze item>, <match>, and <rate> steps. Moreover, the RecOne pipeline incorporates SFT with RecZero pipeline, which first generates reasoning process based on LLM teacher model and then adopts the reasoning process and supervision. The authors succeed in transferring the achievements gained by DeepSeek and GRPO into the rating prediction task, with the excellent performance on four public datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The motivation of RecZero is well-elaborated. The limited recommendation capability and the costly supervision are both the key weaknesses of existing distillation-based methods.\n- The whole manuscript is well-organized, with the problem statement and the proposed methodology are clearly introduced.\n- Inspired by DeepSeek, the combination of SFT and RL is spontaneous, enlightening the recommender system community.\n\n**Weakness**\n- **Model Scale.** The scale of baseline models are not mentioned, which may given rise to unfair performance comparison. It is unclear whether the improvement originates from the proposed method or just the LLM backbone.\n- **Method Scalability.** Similar to the first concern, I wonder whether the proposed method can still be effective with backbones of different scale. Further investigation towards the impact of backbone is recommended.\n- **Inference Efficiency.** This manuscript lacks report of the inference time and further comparison with baselines.\n- **Availability.** The proposed method is closed-source.\n\nQUESTIONS:\n**Problem**\n- In my regard, the terminology **Cold-Start** in Section 3.2.1 is different from that mentioned in *cold-start recommendation* and the following *cold-start samples* with less interactions. If yes, renaming the title of Section 3.2.1 and related content is helpful to avoiding ambiguity.\n\n**Discussion**\n- As mentioned in the caption of Figure 1, I wonder the reason and necessity of multiple models.\n> Conventional paradigms train multiple student LLMs using diverse query-based datasets with reasoning traces...\n\n- Whether the ground-truth of rating prediction task is the actual user rating or not, as we did on the online shopping website?\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the task of rating prediction in recommendation using large language models (LLMs). It first proposes a system prompt designed to generate reasoning trajectories that analyze the input user, analyze the item, evaluate user-item compatibility based on these analyses, and finally produce a reasoning process that leads to the predicted rating. Using data generated in this way, the authors train LLMs with GPRO. In addition, they propose RecOne, a model that first performs supervised fine-tuning (SFT) on a subset of high-quality generated data before further training the LLM using GPRO. The paper demonstrates the superiority of the proposed approach over both conventional recommendation systems and other LLM-based recommendation systems across four datasets.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper clearly identifies challenges in applying LLMs to rating prediction and presents an interesting study that explicitly generates a reasoning process for recommendations.\n\nHowever, the following weaknesses are as follows:\n- The paper should provide clearer evidence on whether the proposed reasoning process is more robust than other chain-of-thought (CoT) processes, and whether the designed prompt is indeed superior. The ablation study currently only contrasts the proposed method with a simple <think> generation. It would strengthen the work to experimentally compare different reasoning step structures, such as variations that omit item analysis or modify the reasoning process itself.\n\n- Beyond generating data and performing GPRO and SFT on selected data, the methodological novelty appears limited. The approach does not introduce fundamentally new techniques beyond this pipeline.\n\n- Additional experiments on robustness are needed. For example, whether the proposed method generalizes well across different LLMs beyond the ones tested.\n\n- The paper should clearly state whether the same LLMs were used for the baselines. In particular, for models like LLMRec or Reason4Rec, it is unclear whether they were evaluated using Qwen2.5-7B, as was the proposed method. While this is briefly mentioned in the appendix, such critical details should be explicitly stated in the main text. Furthermore, since the proposed method likely incurs higher training costs compared to existing approaches, the paper should also clarify whether the same data sizes were used for training across all models, and provide a detailed analysis of training cost and inference time.\n\n- Since the proposed method fine-tunes Qwen2.5-7B, it is important to also compare against results where Qwen2.5-7B is used for reasoning with the proposed template, but without reinforcement learning, in order to better isolate the contribution of the reinforcement learning step.\n\nQUESTIONS:\n- How robust and effective is the proposed prompt template in comparison to other prompt designs?\n\n- What LLMs were used in the baseline methods in the experiments?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper fine-tunes LLMs to solve the rating prediction problem. LLMs would use user-item interaction history in text as the input to match ground-truth ratings from the data. Previous work usually uses a teacher model to generate reasoning traces from interaction history in multiple steps and then distill those traces by fine-tuning a smaller student model. This paper first proposes RecZero by following the methodology of DeepSeek-R1-Zero to optimize a MAE-like reward (plus a format reward) with GRPO instead of doing supervised fine-tuning. As a result, RecZero can automatically develop reasoning ability from a relatively simple prompt template. It also proposes RecOne which is analogous to DeepSeek-R1 in terms of using supervised fine-tuning and cold-start reasoning samples. The results of this paper show that both RecZero and RecOne outperform traditional matrix factorization or deep learning models on on multiple rating prediction domains. More importantly, both proposed methods outperform previous multi-stage distillation methods without explicitly transferring it from a teacher LLM. There are ablation studies on different components of RecZero/RecOne.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+ It seems novel to solve the rating prediction problem by fine-tuning an LLM with RL (GRPO) using an MAE-like reward instead of pure maximum likelihood. Before I am not aware of any work optimizing MSE/MAE rewards using policy gradient on a regression problem.\n+ RecOne seems not a direct application of DeepSeek-R1 technology on the rating prediction problem by having their own reasoning samples.\n+ Tables 1 and 2 show promising results of RecZero / RecOne though I have some concerns on the results.\n\nWeaknesses\n- In Tables 1 and 2, I assume that the authors reuse results from EXP3RT and Reason4Rec papers without references as I am seeing identical numbers. In particular, results of baselines on Amazon-book and IMDB datasets are from the EXP3RT paper if available even if the Reason4Rec paper also presents results on Amazon-book. Results of baselines on Amazon-music and Yelp are from the Reason4Rec paper.\n- More importantly, EXP3RT and Reason4Rec are using different data processing/filtering procedures so they present different MAE/RMSE of EXP3RT on Amazon-book. This paper does not specify their data processing/filtering procedure for generating results of RecZero/RecOne and other baselines.\n- Both Reason4Rec and EXP3RT utilize QLoRA but not sure if RecZero/RecOne fine-tune everything to get more performance gains. Also, Reason4Rec and EXP3RT use GPT-3.5 as the teacher and Llama3-8B-Instruct as the student while RecOne uses Deepseek-R1 as the teacher and fine-tunes Qwen2.5-7B-Instruct-1M. Performance gains can be attributed to the use of different models.\n- This paper does not evaluate the quality of reasoning traces either quantitatively or qualitatively. Average length per response of Deepseek-R1-Zero increases with more training. This paper does not evaluate inference time.\n\nQUESTIONS:\n* Eq. (1) which defines predicted ratings which is not an ML objective. Eq. (4) which rewards small errors is more like an objective.\n\n* Both Reason4Rec and EXP3RT use logit-weighted decoding resulting in a different definition of predicted ratings. Would be interesting to evaluate the choice.\n\n* Could the authors explain the discrepancy between Table 2 and Figure 3 as MAE is higher in Figure 3?\n\n* Could the authors also either plot other baseline(s) in Figure 4 or present raw numbers?\n\n* This paper uses LLMRec as a baseline without citing it.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RecZero, an innovative reinforcement learning (RL)-based approach for recommender systems that leverages large language models (LLMs) to autonomously develop reasoning capabilities for rating prediction tasks. It abandons traditional multi-stage distillation methods, adopting instead a structured, reasoning-guided prompt and rule-based reward modeling via group relative policy optimization (GRPO). A hybrid variant, RecOne, combining supervised fine-tuning (SFT) and RL is also proposed. Extensive experiments validate the performance improvements of both RecZero and RecOne across several datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength：\n1. Technically sound and well-designed experiments across four diverse and challenging benchmark datasets (Amazon-book, Amazon-music, Yelp, IMDb). Rigorous comparisons with traditional and recent LLM-based recommendation models.\n2. Clearly structured and logically coherent. Methods, assumptions, and experimental setups are thoroughly explained.\n3. Demonstrates clear improvements over existing approaches, addressing key limitations like static supervision and superficial reasoning capability transfers. The proposed frameworks significantly advance recommendation model capabilities, especially in cold-start scenarios.\n4. The use of RL (GRPO) integrated with structured reasoning prompts and hybrid initialization strategies is novel and represents a meaningful departure from standard distillation approaches.\n\nWeeknesses：\n1. The motivation is somewhat unclear. In the recommendation search domain, using large models to distill smaller models for recommendation is a reasonable approach, as the smaller model is used for online inference, reducing latency. The motivation raised in the paper are:\n    - The teacher model has limited capabilities. However, I think the teacher model is large and powerful, and the student model has not yet fully learned from the teacher model. So \"limited capabilities\" is not convincing.\n    - Calling teacher models are costly. But I think the primary cost is during training, and inference with the smaller model is fast.\n    - Superficial Transfer of Reasoning Ability. This point is interesting but lacks experimental evidence to demonstrate that there is a shortcut to transferring reasoning ability.\n2. The proposed method does not address the motivation, as it simply shifts the training paradigm from distillation to reinforcement learning (RL), which I believe is not ideal. Firstly, RL does not reduce training time; on the contrary, RL exploration is more complex. Secondly, the paper does not provide experiments proving that RL training is faster than the original distillation approach. Moreover, RL seems like an overcomplicated approach, and in the recommendation domain, there are not many RL-related techniques because they have been less effective over time.\n3. The experiments are insufficient. The baseline models are limited, and classic traditional algorithms such as DeepFM, DIN, and SIM are not included, nor are LLM-based recommendation algorithms like TallRec, ReLLa, CoLLM, and RecLoRA. Additionally, important recommendation metrics such as AUC, LogLoss, and NDCG are missing from the evaluation.\n4. The writing lacks detail. Algorithms like WDL and RGCL in the baseline are not properly cited, and there is no subsection explaining the baseline models, resulting in a lack of necessary details.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-09"}