Title: ShiQ: Bringing back Bellman to LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning with Human Feedbacks

Reinforcement Learning

Q-Learning
Abstract: The fine-tuning of pre-trained large language models (LLMs) using reinforcement learning (RL) is generally formulated as direct policy optimization. This approach was naturally favored as it efficiently improves a pretrained LLM with simple gradient updates. Another RL paradigm, Q-learning methods, has received far less attention in the LLM community while demonstrating major success in various non-LLM RL tasks. In particular, Q-learning effectiveness stems from its sample efficiency and ability to learn offline, which is particularly valuable given the high computational cost of sampling with LLM. However, naively applying a Q-learning–style update to the model’s logits is ineffective due to the specificity of LLMs. Our contribution is to derive theoretically grounded loss functions from Bellman equations to adapt Q-learning methods to LLMs. To do so, we interpret LLM logits as Q-values and carefully adapt insights from the RL literature to account for LLM-specific characteristics. It thereby ensures that the logits become reliable Q-value estimates. We then use this loss to build a practical algorithm, ShiQ for Shifted-Q, that supports off-policy, token-wise learning while remaining simple to implement. Finally,  ShiQ is evaluated on both synthetic data and real-world benchmarks, e.g., UltraFeedback, BFCL-V3, demonstrating its effectiveness in both single-turn and multi-turn LLM settings.


Peer reviews:
comments: SUMMARY:
The authors consider the backward recursion for computation of KL-regularized state-action function (Theorem 1). They then consider the minimization of KL-regularized Bellman error and provide a more tractable shifted-Q formulation of Bellman error (Theorem 2).  Finally, they propose an algorithm that supports off-policy, token-based KL regularized Q learning for LLM training and evaluate performance in various benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths

The paper is well-written, and the results seem correct. Developing a backward recursion scheme for KL-regularized Q learning is not necessarily a significant contribution. Instead, I see the main contribution of the paper in the adaptation of such framework to LLM training.

Weaknesses

- The  authors do not expand upon how a reliable token-based reward model can be assumed when in practice this token-based feedback is generally not available. 
- It appears the computational effort of the shifted- Q algorithm grows exponentially in T_max. This may restrict its practical application to short sequences.

QUESTIONS:
1.  I would appreciate the  authors comment on the merits of the proposed approach given the lack of token-based rewards. Perhaps the proposed approach provides a principled approach for credit assignment. If that is the case the right benchmarking comparison should be other approaches credit assignment rather than sequence-based LLM training methods. 

2. Can the authors elaborate on the computational burden imposed by the length limit T_max ? Could there be ways to mitigate this ?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an offline reinforcement learning algorithm called ShiQ (Shifted-Q) for fine-tuning LLMs. The core contribution is deriving theoretically grounded loss functions from Bellman equations to adapt Q-learning methods to LLMs, addressing challenges like sample inefficiency and the high computational cost of sampling with LLMs. The authors claim that ShiQ supports off-policy, token-wise learning and is simple to implement. They evaluate ShiQ on synthetic data and real-world benchmarks, including UltraFeedback and BFCL-V3, in both single-turn and multi-turn LLM settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper tackles a crucial problem in LLM fine-tuning, specifically the sample complexity associated with reinforcement learning. Efficiently fine-tuning LLMs with RL is a significant challenge.\n2. The paper provides a comprehensive review of existing literature, covering various RL fine-tuning approaches for LLMs.\n3. The motivation behind adapting Q-learning to LLMs, highlighting its potential for sample efficiency and offline learning, is well-articulated and reasonable.\n4. The approach of deriving theoretically grounded loss functions from Bellman equations to adapt Q-learning to LLMs, especially with the proposed transformations (easing sampling, improved initialization, multi-step extension), shows a degree of novelty.\n\n**Weaknesses:**\n\n1. While the method appears theoretically sound, the derivation involves multiple transformations and notations that make it overly complex. It's challenging to grasp the full intuition behind each step.\n2. The empirical evaluations are a major weakness.\n    * Most experiments are on toy datasets, which doesn't fully demonstrate the method's effectiveness in real-world, large-scale LLM scenarios.\n    * Crucially, the experimental figures lack error bars, making it difficult to assess the statistical significance and robustness of the results. \n3. The significance of the results is questionable. In Figures 3 and 4, ShiQ's performance is comparable to, or sometimes even slightly worse than, the baseline CoPG. The benefits claimed in the introduction (e.g., sample efficiency) are not clearly demonstrated by the empirical results. \n4. The paper's clarity suffers from inconsistent notation. It mixes RL and LLM notations throughout, which is confusing. A unified and simplified notation would greatly improve readability.\n\nQUESTIONS:\n1. From the three sub-methods proposed (Easing sampling, Improved initialization, Multi-step extension), which one do you believe contributes most significantly to ShiQ's performance, and why?\n2. Beyond empirical performance, what are the primary theoretical advantages of your Q-learning-based approach compared to traditional policy-gradient methods in the context of LLM fine-tuning?\n3. How would you address the scaling challenges if ShiQ were to be applied to significantly larger LLMs and datasets? What potential issues might arise in a large-scale deployment, and how would you mitigate them?\n4. Is ShiQ more suitable for preference learning tasks (like aligning with human preferences) or for fine-tuning LLMs on structured reasoning tasks, similar to chain-of-thought (CoT) training in DeepSeek-R1?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author starts from the Bellman equation of soft Q-learning and introduces three improvements: reducing the size of the final policy, ensuring zero loss at initialization when the target is a reference policy, and extending the optimization objective to multi-turn scenarios, ultimately deriving the ShiQ optimization formulation. Experiments were conducted on toy examples, single-turn tasks (HH and UltraFeedback), and multi-turn tasks (BFCL-V3). The results show that ShiQ performs comparably to alignment methods like CoPG in single-turn tasks and outperforms baseline methods in BFCL.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The formulation of ShiQ is theoretically grounded, starting from soft Q-learning.  \n2. Ablation studies are provided for the initialization trick and multi-turn extension, demonstrating their effectiveness.  \n\nWeaknesses:\n\n1. ShiQ's final formulation closely resembles DRO [1], without significant improvements.  \n\nThe author claims two main improvements over DRO: \n\n(1) Token-level and multi-step rewards: ShiQ handles these, but DRO could also be trivially extended by replacing single-step rewards with multi-step returns. \n\n(2) Memory-friendly value modeling: ShiQ replaces DRO's separate value network with the proposed value difference method, but no experiments compare the performance impact of value difference versus an additional value network.\n\n[1] Richemond, Pierre Harvey, et al. \"Offline regularised reinforcement learning for large language models alignment.\" arXiv preprint arXiv:2405.19107 (2024).\n\nQUESTIONS:\n1. For the implementation details, during optimization, are $v_{\\ell}$ and $\\pi_{\\ell}$ in Eq.10 updated simultaneously? Is stop-gradient applied? If the gradient of ShiQ with respect to $\\pi$ can be provided in the paper, it may be more convenient to understand the optimization objective.\n\n2. In the multi-turn experiments, does DRO use single-step rewards or multi-step returns? If DRO is modified to use multi-step returns, would it be equivalent to ShiQ (ignoring value modeling differences)? What would the empirical results show?  \n\n3. For smaller models (e.g., 7B/4B), where memory constraints are less severe, would using an additional value network, a tiny shared-parameter network, or the proposed value difference yield different results? Is an extra value network still necessary for optimal performance?  \n\nMinor Issue:  \nThe subscripts in Equation 9 are incorrect？\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}