Title: Avoiding exp(R) scaling in RLHF through Preference-based Exploration
Venue: NeurIPS 2025
Accepted: yes
Keywords: RL theory

RLHF

sample complexity

llm
Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for large language model (LLM) alignment.
This paper studies the setting of online RLHF and focuses on improving its sample efficiency.
All existing algorithms for online RLHF, whether doing passive exploration or active exploration, suffer from a sample complexity that scales exponentially with the range of the reward function.
This statistical inefficiency hinders their effectiveness in scenarios with heavily skewed preferences, e.g. questions with objectively correct answers.
To address this, we introduce Self-Exploring Preference-Incentive Online Preference Optimization (SE-POPO), an online RLHF algorithm that for the first time achieves a sample complexity that scales polynomially with the reward range, answering an open problem raised by Xie et al. [2024].
Theoretically, we demonstrate that the sample complexity of SE-POPO dominates that of existing exploration algorithms. Empirically, our systematic evaluation confirms that SE-POPO is more sample-efficient than both exploratory and non-exploratory baselines, in two primary application scenarios of RLHF as well as on public benchmarks, marking a significant step forward in RLHF algorithm design.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a solution to a known problem in RL in the context of modern LLM post-training: the exponential scaling of reward uncertainty as a function of the max possible reward difference between a preferred and dispreferred responses (eg. $\\mathcal{O}(exp(R_{max}))$). Their algorithm operates by iteratively improving the policy used to sample the dispreferred responses, thereby closing the reward gap, and they argue that one can do so in $\\mathcal{O}(R_{max})$ iterations of the algorithm's subroutine for improving the sampling policy. They theoretically derive how their algorithm implies their improved bound. Finally, they conduct standard preference tuning experiments to show that in practice, their method outperforms two existing baselines over three rounds of policy optimization over the same prompt dataset.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. Presentation of the preliminaries are clear (but they use too much space given the lack of details in the actual contributions sections that follow.)\n\n2. Presentation of the key problem statement is clear, but I implicitly read the bolded L141 as \"How should we sample the responses y and y' in online RLHF? ..._to minimize reward uncertainty?_\" as the section sets this up quite well.\n\n(Authors don't need to imply that this is the only and central unanswered question in online RLHF as that would be an overly broad scope. Rather, assuming this posed bound is indeed an issue that significantly hampers overall learning performance, they have developed a theoretically grounded solution.)\n\n\n### Weaknesses\n1. Core elements of the theoretical setup and the claims made are not explained in clear enough detail in the main body, which is a problem, since this is a theory paper.\n    1. An explanation for how Eq. 9 can be interpreted as encouraging exploration is missing as well as even a statement or sketch of how optimizing Eq. 9 minimizes Eq. 8. Forgive if this is obvious, but again as a theory paper, the core material seems to be lacking some key details that the reader needs to understand in a particular order to buy the contribution during a read-through.\n    2.  Immediately dropping the actual bonus term from Eq.11 to Eq.12 seems premature. There exist frameworks to do this type of training efficiently (or even inefficiently). It should at least be shown that this is not a critical change. The concern stems how only increasing the second term of the original expression (just the second line of Eq 12) is somehow equivalent to incentivizing exploration in the same way as in the original objective. Maxing the neg log of the difference between the current policy and the reference policy (making them closer) on a _dispreferred_ sample has unclear utility. It is in fact the missing term where we max the difference between the current policy and ref policy on newly sampled positive completions that constitutes useful exploration of pushing the current policy away from the reference in the direction of potentially better responses. Imagine the pathological case of ending up with a policy such that for negative samples y' both the current policy and the reference policy assign equivalent likelihoods. Eg. solutions to the proposed optimization include identical policies or two non-identical, but otherwise uninteresting polices that both assign similar likelihoods to a given low reward response. Note that the proof in Appendix B is intended to show that Thm 3.5 still holds approximately when substituting with Eq. 12. However, the concern raised here is a separate point regardless of whether the bound holds or not and connected to further questions about what the alternate training algorithm is actually doing in practice (see below).\n\n2. The empirical results are not particularly strong, but more importantly do not _directly_ support the theoretical claims of the paper; there is no demonstrated link between the overall performance improvement on the benchmarks and the key focus of the theory which centers to reward gaps and $R_{max}$.\n    1. While the chosen experimental setup is standard practice (Table 1) the improvement of SE-POPO over other methods is relatively small. \n    2. Since the argument is for sample complexity improvement (not necessarily absolute improvement at any specific budget) what we expect to see is evidence that as a function of available samples the rate of improvement for the proposed method is better. To be sure that the lead shown is reliable, it is better if experiments are taken to convergence, or peak WR and AvgR is achieved for the method given the starting policy and dataset. Does SE-POPO achieve 92.4 or 40.12 WR faster than DPO and XPO _initially_ but when  all methods are provided one or two more iterations DPO and XPO catch up? Does SE-POPO degrade faster at higher iteration counts? Whether SE-POPO maintain a consistent and durable lead at all budgets is not clear.\n    3. If there is no way to connect the actual goal of the proposed theoretical improvement and how the method performs in the evaluation, then it is generally unclear whether the _reason_ for why SE-POPO is performing better is indeed because of how the scaling of the regret bound is modified. SE-POPO could be better, but not necessarily having anything to do with the $R_{max}$ problem. (see related question)\n\nQUESTIONS:\n### Questions\n1. What is the reward-based exploration bonus $J(r, \\pi)$? This is presented without definition and is important for later contrasting with the preference based regret defined in terms of $G(r, \\pi)$ and $I(r, D)$. (If necessary, just present one concrete expression from a single prior work that the authors prefer.) \n2. Is the preference based regret posed in Eq. 8 a novel definition in this work? Regardless, please style it as  definition and if drawn from literature, cite it.\n3. Can the authors measure and report the empirical worst case reward gaps between $y$ and $y'$ for the datasets under all three methods? \n\nWhy is Q3 important for the paper? If the worst case gap between the preferred and dispreferred sample rewards under the same reward model is in fact reduced in practice this is an interesting observation to help connect the proposed theoretical improvement to the empirical improvement. It will not prove causality, but currently there is no evidence presented that the theory is related to the experiments. the algorithm could just be better for some other correlated reason.\n\n\n### Minor/Writing Suggestions\n- DPO is not just a way to avoid the explicit reward model, but it was also first posed in the offline setting where y and y' will be from a reference set. The notation around Eq 5 should note that canonically, y and y' come from D_train but that online variations of DPO allow one to continue to sample y,y' \\sim theta(\\cdot|x). Maybe the authors intend to immediately identify a fixed dataset of positive completions with generations from an optimal model? and negative completions with responses from a weak model (which need not necessarily be the current model though)? In any case, this distinction about offline setting of canonical DPO ([Rafailov) should be clarified.\n- L131,L159 \"optimism\" is noun, so instead use the adjective \"optimistic RLHF\" to describe the concept of RLHF algorithms that factor in the principle of optimism. \n- Please format Lemma 3.1 on its own line as it's difficult to parse as written. Also, the proof is central to the contribution of the work; the proposed algorithm relies on the surrogate minimization actually improving the exponential scaling problem posed. A sketch, poentially only in prose if necessary, should be presented in the main body.\n\nLIMITATIONS:\n- It is not clear that the empirical results support the theory, and without this demonstrated mechanistic connection, it is difficult to predict the generalizability of the method. Whereas, if the connection were clear, then any scenario where the exponential bound was relevant would be expected to also benefit from the application of the new algorithm.\n- Generally, experiments on at least one other model family for the main policy, one other dataset, and one other reward model would help prove the robustness of the results, but I understand this is primarily a theory paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors theoretically analyze the issue that existing online RLHF algorithms suffer from sample complexity that grows exponentially with the reward range, leading to inefficiency and poor exploration in scenarios with extreme preferences (e.g., tasks with objectively correct answers). To address this, they propose an algorithm similar to *PO, named SE-POPO, which resolves the aforementioned exploration problem. The authors further provide a theoretical proof that the sample complexity of their method scales polynomially with the reward range, overcoming the exponential limitation of existing algorithms.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper presents its arguments clearly, analyzing and addressing the problem of exponential sample complexity.\n- The proposed method is engineering-friendly, simple to implement, and can be directly integrated into existing DPO frameworks.\n- Experimental results demonstrate the effectiveness of the proposed approach.\n\n### Weaknesses\n\n- The experiments appear to have been conducted only on Llama-3.1-8B. Could the authors evaluate their method on additional models, such as the Qwen series, to further validate its effectiveness?\n- Could the authors provide additional statistics on computational overhead? Iteratively updating the sampler may increase the computational resources required during training.\n- The experiments use Llama-3.1-8B-SFT and compare the results with Llama-3.1-8B-Instruct to demonstrate the method’s effectiveness. Could the authors also conduct experiments directly on Llama-3.1-8B-Instruct to further substantiate their claims?\n\n### Additional Question\n\n- Could the authors further explain the meaning of Equation (6)? My understanding is that the training data for the reward model in the next stage should be sampled from responses that the latest policy receives high scores from the previous round’s reward model, in order to prevent reward hacking. I would appreciate a more detailed explanation to clarify this point.\n- Could you please explain how Llama-3.1-SFT was trained? Was the RLHFlow-Ultrafeedback dataset used as the training set? However, it seems that this dataset is pairwise in nature.\n\nQUESTIONS:\nPlease address all the issues mentioned in the Weaknesses section as thoroughly as possible.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of sample inefficiency in online Reinforcement Learning from Human Feedback (RLHF) in theory. The authors identify a key limitation in existing theoretical online RLHF algorithms: their sample complexity scales exponentially with the range of the underlying reward function ($R_{\\max}$). To overcome this, the paper introduces a new algorithm called SE-POPO. Instead of using a reward-based exploration bonus like prior work, SE-POPO uses a preference-based bonus. This directly incentivizes the policy to find responses that are preferred over a given sampler policy, which is shown to be more efficient when the reward scale is large. Besides, SE-POPO iteratively calls a subroutine POPO that uses the preference-based exploration. The output policy from each iteration is then used as the sampler for the subsequent iteration. Theoretically, the paper demonstrates that SE-POPO is the first algorithm to achieve a sample complexity that scales polynomially with the reward range. Empirically, the authors conduct extensive experiments on LLM alignment tasks, comparing SE-POPO against state-of-the-art passive (DPO) and active (XPO) exploration baselines. The results show that SE-POPO consistently outperforms these baselines across various settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Significance: The paper tackles a fundamental problem in the theory of online RLHF. The exponential dependency on the $R_{\\max}$ is a major drawback of the prior theoretical works. By being the first to propose an algorithm with polynomial scaling, this work answers an open question from [Xie et al., 2024] and significantly advances the state of the art. The ideas presented are likely to be highly influential for future research and practice in LLM alignment.\n2. Originality: The technical contributions are highly original. This paper proposes the Preference-based Exploration, which shifts from a standard reward-based optimism bonus to a preference-based one. It directly targets the source of information (preferences) rather than an inferred, and potentially poorly scaled, proxy (rewards). This is a clever and effective idea. Besides, the mechanism of iteratively updating the sampler policy to \"bootstrap\" performance is an elegant solution to avoid comparing against a weak or fixed reference policy. This gradual improvement strategy is what ultimately breaks the exponential dependency, as shown in Theorem 3.5.\n\nWeaknesses:\n\n1. Significance: The entire theoretical framework are built upon the bandit formulation, which deviates from the token-level MDP used in practice. Besides, this paper assumes the BT preference model, which may not capture the complex human preferences. The above two gaps may restrict the impact of this paper.\n2. Quality: As discussed in experiments, the proposed method SE-POPO introduces a bias towards generating longer responses. As such, a natural question is whether the better empirical performance of SE-POPO is mainly due to this length bias. This paper may benefits from more empirical investigations on the length bias.\n\nQUESTIONS:\n1. In the POPO objective (Equation 9), the paper introduces an indicator function I(r, D_t) to truncate the exploration bonus. However, the paper mentions in Section 3.3 (lines 202-203) that this truncation is \"rarely active and can therefore be omitted\" in the practical implementation. Is this truncation term primarily a theoretical tool needed to secure the proofs, or are there practical scenarios where it would be crucial?\n2. SE-POPO iteratively updates the sampler $\\pi_{sam}$ by running the POPO subroutine for T iterations. In experiments (Section 4 and Appendix K.1), the paper uses a 3-iteration implementation. Could the authors clarify the relationship between the number of \"intervals\" K in Algorithm 1 and the \"iterations\" in the experiments?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SE-POPO, an online Reinforcement Learning from Human Feedback (RLHF) algorithm that avoids the exponential dependence on the reward scale ($R_\\max$) that appears in various previous theoretical works. Existing RLHF approaches, even with active exploration, suffer from sample complexity that scales exponentially with $R_\\max$ due to the nonlinear nature of human preference modeling via the Bradley–Terry model. SE-POPO addresses this by introducing a self-updated sampler strategy that is key to achieving polynomial sample complexity in the leading term. Theoretical analysis establishes regret bounds without exponential scaling, and empirical results show SE-POPO outperforms previous baselines (DPO, XPO) on alignment benchmarks and public datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The theoretical results are significant and interesting. The analysis tailored for the sampler updating mechanism could be of independent interest for future research.\n2. Experimental results demonstrate the empirical effectiveness of the theoretical algorithm to some extent.\n\n**Weaknesses:**\n1. The POPO subroutine is relatively direct given some prior works on provably sample-efficient online RLHF with reward bonus to incentivize exploration.\n2. Some unclarity issues, please see the Question part.\n\nQUESTIONS:\n1. In Line 224, $\\phi(x,y,y’)$ should be $\\phi(x,y)$ for all $x,y$. The same problem also exists throughout the proof of Theorem 3.4 (Appendix D). This casts concerns to the overall correctness of the proof. \n2. In the last equation in Page 17, how is the second to last inequality derived? Does it make sense to directly bound the derivative of sigmoid by constant here? since here the goal is to bound the preference difference by reward difference, the direction that does not incur exponential factors.\n3. Typos in Equation (14).\n4. From my understanding of the proof, the exponential dependency still exists for the logarithm term, despite that it is a non-leading term (in terms of $N$). However, this means that the problem still suffers from a burn-in cost that is exponential in the scaling of the reward. I suggest presenting this explicitly in the main results, as opposed to hiding all the non-leading terms in terms of $N$.\n5. Is it possible to avoid such exponential dependency for offline RLHF setups? where only fixed preference data are provided.\n\nLIMITATIONS:\nPlease see the Questions part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}