Title: Greedy Sampling Is Provably Efficient For RLHF
Venue: NeurIPS 2025
Accepted: yes
Keywords: RLHF

Greedy Sampling

General Preference Model
Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique for post‑training large language models. Despite its empirical success, the theoretical understanding of RLHF is still limited, as learning the KL-regularized target with only preference feedback poses additional challenges compared with canonical RL.  Existing works mostly study the reward-based Bradley-Terry (BT) preference model, and extend classical designs utilizing optimism or pessimism. This work, instead, considers the general preference model (whose practical relevance has been observed recently) and obtains performance guarantees with major, order-wise improvements over existing ones. Surprisingly, these results are derived from algorithms that directly use empirical estimates (i.e., greedy sampling), as opposed to constructing optimistic or pessimistic estimates in previous works. This insight has a deep root in the unique structural property of the optimal policy class under the KL-regularized target, and we further specialize it to the BT model, highlighting the surprising sufficiency of greedy sampling in RLHF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretically grounded analysis of learning from human preferences using the Bradley-Terry model under a greedy sampling regime. Contrary to prior work emphasizing the need for explicit exploration or reward regularization, the authors prove that simply greedy sampling with maximum likelihood reward estimation (MLE) achieves sublinear regret under realizability and finite eluder dimension.\n\nThe paper analyzes both online and offline learning settings and provides regret bounds in both, leveraging a notion of eluder dimension adapted to preference learning. Empirically, the method shows strong performance compared to RLHF-style fine-tuning methods like PPO and DPO.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\t1.\tStrong Theoretical Results:\n\t•\tThe regret bounds are tight and provide new understanding into how greedy methods can still ensure learning in RLHF scenarios, despite the lack of explicit exploration.\n\t2.\tNovel Use of Eluder Dimension:\n\t•\tThe adaptation of eluder dimension to pairwise preference settings is nontrivial and elegant. The clean separation of statistical and algorithmic complexity is a plus.\n\t3.\tSimple Algorithm with Guarantees:\n\t•\tThe fact that MLE + greedy policy selection suffices under reasonable assumptions is appealing from both practical and theoretical perspectives.\n\t4.\tOffline Guarantees:\n\t•\tThe analysis extends to the offline setting, which is especially relevant for industry RLHF pipelines where preference data is pre-collected.\n\n⸻\n\n Weaknesses and Concerns:\n\t1.\tRealizability Assumption:\n\t•\tThe assumption that the true reward function R^* \\in \\mathcal{R} is strong and often unrealistic in practical LLM alignment settings. The paper would benefit from a discussion or robustness analysis under mild misspecification.\n\t2.\tFinite Function Class:\n\t•\tThe assumption that |\\mathcal{R}| < \\infty is unrealistic when using neural networks. The theory doesn’t currently extend to infinite hypothesis classes, which would typically require Rademacher complexity or covering numbers.\n\t3.\tNoisy Preferences and Human Errors:\n\t•\tThe analysis assumes well-behaved (BT-consistent) preference labels, but human data is often noisy, biased, or adversarial. How robust is the regret to these deviations?\n\t4.\tLack of Exploration in Practice:\n\t•\tWhile the theory shows greedy sampling suffices, in high-dimensional or sparse reward regimes, greedy policies may still perform poorly. Empirical evidence supporting exploration sufficiency is somewhat limited in scale.\n\t5.\tLimited Scope of Experiments:\n\t•\tThe experiments are promising but largely small-scale or on synthetic/hard-coded preferences. A large-scale RLHF-style experiment with real human preference data would dramatically strengthen the case.\n\n⸻\n\nQUESTIONS:\nDiscussion:\n\t•\tAdd an empirical misspecification experiment, e.g., training with an incomplete reward class.\n\t•\tDiscuss (or attempt to bound) regret under function approximation error.\n\t•\tConsider extensions to continuous function classes or neural function approximators.\n\t•\tProvide more experimental evidence where greedy sampling fails, to better delineate the scope of the theory.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical analysis of Reinforcement Learning from Human Feedback (RLHF), arguing that simple \"greedy sampling\" is a provably efficient method. This contrasts with previous theoretical work, which relied on more complex principles of optimism or pessimism to guarantee performance. The authors focus on the KL-regularized contextual bandits framework, which is a common theoretical model for the post-training of large language models (LLMs).\n\nThe key insight is that the KL-regularization term fundamentally changes the structure of the problem by ensuring that any optimal policy has a bounded likelihood ratio with respect to the reference policy. This structural property makes it possible to prove the efficiency of a much simpler algorithm that directly uses empirical estimates (greedy sampling) rather than constructing complex confidence bounds.\n\nSTRENGTHS AND WEAKNESSES:\nStrehgths: This work achieves optimal bound for RLHF under KL regularization without optimism or pessimism. Besides, they comprehensively consider both online and offline cases and unify the general preference and BT model. They provide rigurous proof and more importantly, their greedy sampling is more efficient.\n\nWeaknesses:\n1. Their greedy sampling is more computationally efficient, but it comes with the sacrifice of an additional exp(\\eta) term. It would be clearer if they added some discussion to admit this trade-off. What are the cases when the order exp(\\eta) can be neglect. For example, when the sft policy is good enough and \\eta is in constant order, the greedy sampling bound totally matches the optimal bound.\n\nQUESTIONS:\n1. In the simulation, their metrics are the suboptimal gap and regret. Do those metrics have KL regularization with them? It would be better to also plot the accuracy and compare the performance.\n\n2. Could the authors add more experiments about different values of \\eta?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper establishes the theoretical efficiency of greedy sampling in Reinforcement Learning from Human Feedback (RLHF) under KL-regularized contextual bandits. For both **general preference (GP)** and **Bradley-Terry (BT) models**, it proves that directly using empirical estimates (without optimism/pessimism) achieves:  \n\n- **Online regret**: \\(O(\\log T)\\)  \n\n- **Offline sample complexity**: \\(O(\\epsilon^{-1})\\)  \n\nThe key insight lies in exploiting the **bounded likelihood ratio** property induced by KL regularization (Lemma 1), which confines optimal policies within a neighborhood of the reference policy \\(\\pi_0\\). This structural constraint eliminates the need for confidence-bound constructions, reducing computational overhead while matching state-of-the-art bounds derived via optimism/pessimism.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:  \n\n**Theoretical novelty**: First to achieve \\(O(\\log T)\\) regret for GP models, improving \\(O(\\sqrt{T})\\) bound.  \n\n**Algorithmic simplicity**: Greedy sampling avoids costly confidence-bound optimizations.  \n\n**Unified analysis**: Extends guarantees to both GP and BT models via KL regularization’s policy-ratio boundedness.  \n\n**Empirical validation**: Experiments show greedy sampling matches optimism’s convergence rate.  \n\n**Weaknesses**:  \n\n**Insufficient comparison**: Fails to explicitly contrast proof techniques with Zhao et al. (2025a,b)—critical for assessing contribution originality.  \n\n**Weaknesses from an Experimental Perspective**: The experiments focus mainly on theoretical validation with limited diversity in the experimental settings.\n\nScalability and offline sample complexity could be further empirically demonstrated.\n\nThe experiments assume a relatively idealized feedback model, where preferences are clear and consistent. However, in real-world applications, human feedback is often noisy, inconsistent, or even subjective. The paper does not investigate how greedy sampling performs under these conditions or its robustness to errors in human-provided feedback.\n\nThe paper does not provide a sensitivity analysis to examine how sensitive the greedy sampling algorithm is to hyperparameters, such as the learning rate or the degree of KL-regularization. Testing the algorithm's robustness across a range of hyperparameter values would help assess the method’s stability and provide practical guidance for tuning the algorithm in real applications.\n\nQUESTIONS:\n1. **Proof technique transferability**:  \n   > Zhao et al. (2025a,b) use confidence-bound constructions for BT model regret bounds. Can their approach directly yield the bounds for the GP model? If not, what modifications are essential?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the theoretical underpinnings of RLHF, focusing on the KL-regularized contextual bandit setting. The authors present a surprisingly strong result: a simple greedy sampling algorithm, which directly uses empirical estimates from maximum likelihood estimation without optimism or pessimism, is provably efficient. For the general preference model, the paper establishes, for the first time, a logarithmic regret bound of $O(\\log(T))$ in the online setting and a sample complexity of $O(\\epsilon^{-1})$ in the offline setting. The central theoretical insight is that the KL-regularization term restricts the optimal policy class, ensuring that any candidate optimal policy has a bounded likelihood ratio with respect to the reference policy $\\pi_0$. This structural property mitigates the need for explicit exploration (optimism) or conservatism (pessimism).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper's core claim—that a simple greedy algorithm is provably efficient for RLHF—is both counter-intuitive and highly impactful. It challenges the prevailing wisdom that optimism or pessimism is necessary for achieving strong performance guarantees in learning from feedback. This finding could significantly simplify the design and analysis of future RLHF algorithms.\n- The paper provides tight, instance-agnostic performance bounds. The $O(\\log(T))$ regret and $O(\\epsilon^{-1})$ sample complexity are the best possible in many learning settings. Achieving these for the general preference model for the first time is a major theoretical leap forward.\n- The work is thorough in its scope, addressing both the general and BT preference models, as well as both online and offline learning settings.\n\n### Weaknesses\n\n- While the experiments corroborate the theoretical findings, they are conducted in a relatively simple linear setting with a small, fixed action space. The paper is motivated by LLMs, but it does not provide evidence of how this approach scales to the massive, high-dimensional, and non-linear nature of LLM fine-tuning.\n- The big-O notation in the theoretical results hides constants. The authors acknowledge that the greedy sampling algorithm may have a \"slightly worse constant\", which is also suggested by the cumulative regret plots in Figure 1. A more direct discussion of the trade-offs between computational simplicity and these performance constants in the main body would be beneficial.\n- The theoretical analysis relies on the standard but strong assumption of realizability, *i.e*., that the true preference or reward model lies within the considered function class (Assumptions 1 and 2). In practice, human preferences are incredibly complex, and model misspecification is a real concern. The paper does not discuss the robustness of the greedy approach to violations of this assumption.\n\nQUESTIONS:\n## Questions\n\n1. Could the authors elaborate on the practical implementation of the MLE and policy update steps (*e.g.*, Step 6 and 7 in Algorithm 1) in a real-world LLM setting?\n2. The analysis hinges on the optimal policy being close to the reference policy $\\pi_0$. How does the quality of the initial SFT model (used as $\\pi_0$) affect the final performance of the greedy algorithm? If $\\pi_0$ is a poor-quality model, does that place a more severe restriction on the achievable performance for greedy sampling compared to an optimism-based approach?\n3. The empirical results suggest optimism-based methods might have a slight edge in cumulative regret, likely due to better constants. In what practical scenarios would the computational simplicity of greedy sampling outweigh the potentially better constants of optimism/pessimism, and vice-versa?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}