PILAF: Optimal Human Preference Sampling for Reward Modeling
Authors: Yunzhen Feng
Ariel Kwiatkowski
Kunhao Zheng
Julia Kempe
Yaqi Duan
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning from Human Feedback
RLHF
Sampling Scheme
Preference Labeling
Optimization
Primary area: deep_learning->large_language_models
TL;DR: We propose a novel sampling scheme for preference labeling that leads to better RLHF.

Abstract: As large language models increasingly drive real-world applications, aligning them with human values becomes paramount. Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique, translating preference data into reward models when oracle human values remain inaccessible. In practice, RLHF mostly relies on approximate reward models, which may not consistently guide the policy toward maximizing the underlying human values. We propose Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response sampling strategy for preference labeling that explicitly aligns preference learning with maximizing the underlying oracle reward. PILAF is theoretically grounded, demonstrating optimality from  both an optimization and a statistical perspective. The method is straightforward to implement and demonstrates strong performance in iterative and online RLHF settings where feedback curation is critical.

PILAF: Optimal Human Preference Sampling for Reward Modeling
Yunzhen Feng' Ariel Kwiatkowski2*
Kunhao Zheng?*
Julia Kempe 21 Yaqi Duan
1
Abstract
(Dong et al., 2023; Liu et al., 2024b) or leveraging pre- existing preference datasets (Ethayarajh et al., 2024; Azar et al., 2024; Xu et al., 2024), comparatively less attention has been paid to the critical challenge of collecting effective data for human-labeling in the RLHF pipeline, to maximize its utility. This is an important problem, as the quality of preference data directly impacts the effectiveness of reward modeling and, consequently, the overall success of fine- tuning. This challenge is further compounded by the high cost of expert preference labeling (Lightman et al., 2023).
As large language models increasingly drive real- world applications, aligning them with human val- ues becomes paramount. Reinforcement Learn- ing from Human Feedback (RLHF) has emerged as a key technique, translating preference data into reward models when oracle human values remain inaccessible. In practice, RLHF mostly relies on approximate reward models, which may not consistently guide the policy toward maximiz- ing the underlying human values. We propose Policy-Interpolated Learning for Aligned Feed- back (PILAF), a novel response sampling strategy for preference labeling that explicitly aligns pref- erence learning with maximizing the underlying oracle reward. PILAF is theoretically grounded, demonstrating optimality from both optimiza- tion and a statistical perspective. The method is straightforward to implement and demonstrates strong performance in iterative and online RLHF settings where feedback curation is critical.
Preference data is usually generated by sampling response pairs (g) f to prompt 2; from a policy, and present- ing them to human labelers for preference annotation. It is commonly assumed that the annotation follows the Bradley- Terry (BT) model, under an oracle reward. Next, we use maximum likelihood estimation (MLE) on these preference data train reward model, which then serves as mea- surable objective to optimize the policy (i.e. LLM) while staying close to reference policy. In Direct Preference Optimization (DPO) (Rafailov et al., 2023), this pipeline is simplified by optimizing the policy with implicit reward modeling. However, all these pipelines give rise to a mis- aligument of objectives: RLHF (or DPO) should, in princi- ple, train its policy to maximize the (inherently inaccessible) oracle objective which combines the oracle reward from the BT model with reference regularization. In practice, RLHF relies on preference data through the MLE objective in re- ward modeling or through methods like DPO, which are not designed to guide policy optimization towards maximizing oracle rewards. Thus, reward optimization (either directly or implicitly via DPO) and (optimal) policy optimization are not inherently aligned, potentially leading to inefficiencies (see Section 2).
1. Introduction
Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) has revolutionized large language mod- els (LLMs) by incorporating human preferences, enabling significant progress in applications such as conversational AI (Achiam al., 2023), personalized tutoring (Limo et al., 2023), and content curation (Yue et al., 2024). At the core of RLHF is reward modeling. critical process that translates human feedback-such as pairwise comparisons or rank- ings-into a measurable objective for model training. By formalizing human preferences, reward models then guide LLMs towards alignment through policy optimization.
In this work, we study this misalignment by examining the sampling scheme that generates response pairs (F), f.) for preference labeling, which is especially important when additional preference data is collected mid-RLHF training to mitigate the off-policy distributional shift, as is empirically standard (Touvron et al., 2023; Bai al., 2022). We show that uniform sampling from the current policy, as is common, leads to misaligned gradients of the two objectives (reward model loss and true oracle objective).
While numerous studies have focused on improving lan- guage models (LMs) by optimizing fixed reward functions
NNww York University MMeaa FAIR *Joint second authors Joint senior authors. Correspondence to: Yunzhen Feng (512231@nyu.cde
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To tackle this issue, we present Theoretically Grounded

---

Optimal Sampling for Reward Modeling
Gradient G
Optimization
Sampling
Human value
(Steepest)
MLE
PILL
J (
(MLE gradicnt)
Preference labeling
Reward model
7 0
Vanilla
T
"Online" " Alignment
Policy optimization
8
Statistics
Sensitivity o (mL))
Tef
LM policy
Vanilla
How to sample y. g"
7
PILAF
Space of 0 (c)
(b)
Figure 1. Overview of our approach. (a) We consider full RLHF training setup, where a language model (LM) policy is iteratively refined through active data collection. Our goal is to develop an optimal response sampling method for preference labeling. (b) We introduce PILAF, which generates responses by interpolating between the current policy and reference policy, balancing exploration and exploitation. (c) Our theoretical analysis shows that T-PILAF aligns the parameter gradient with the steepest direction for maximizing human values and achieves more favorable convergence in regions of high sensitivity.
Policy-Interpolated Learning for Aligned Feedbuck (T- PILAF), novel sampling method that aligns reward model- ing with value optimization. Specfically, T-PILAF generates responses by interpolating the policy model and the refer- ence model for a balanced exploration and exploitation. We provide rigorous theoretical analysis to show that for pref- erence data generated with T-PILAF, the gradient of the MLE loss with respect to the policy network's ' parameters is aligned with the policy gradient the oracle objective in a first-order sense. This alignment enables the policy opti- mize directly for the oracle value, achieving both alignment and efficiency. Furthermore, we separately show from sta- tistical perspective that T-PILAF aligns optimization with the steepest directions of the oracle objective. It thus makes the sampled preference pairs more informative, reducing variance and improving training stability.
We then align the Llama-3. 1-8B base model (Dubey et al., 2024) using these proxy-labeled preference data in two set- tings: iterative DPO (Xiong al., 2024) and online DPO (Guo et al., 2024). In both scenarios, preference data is collected on-the-fly, either after each full training epoch in the iterative setting or after every training step in the online setting. Across all configurations, PILAF outperforms all the baselines, producing a policy with higher reward (as measured by the proxy) and lower KL divergence from the reference model, reducing annotation and computation costs by over 40% in iterative DPO.
Our key contributions are as follows:
(Practical sampling algorithm) We propose PILAF (Section 5), an efficient sampling algorithm for gener- ating response pairs in the RLHF pipeline for improved sample efficiency and performance, derived from its theoretically grounded variant T-PILAF (Section 3).
We then present PILAF, simple modification of our the- oretical sampling scheme T-PILAF, which naturally lends itself to practical implementation. For clarity of exposition, we present our method in the context of DPO; however, PILAF can be adapted to a wide class of preference opti- mization methods.' See Figure for an illustration our setup, method, and the optimization and statistical principles underlying PILAF.
(Theoretical optimality) We provide theoretical guar- antees for the efficiency of our approach from both optimization and statistical perspectives (Section 4). (Empirical validation) We validate PILAF in both iter- ative and online DPO settings (Section 6) and observe that it consistently outperforms baselines by achieving higher reward and lower KL divergence from the ref- erence model. Moreover, PILAF achieves comparable performance at significantly reduced annotation and computational costs.
We conduct extensive experiments to validate PILAF's effec- tiveness and robustness. As a stand-in for expensive human annotators, we use well-trained reward Sodel-Skyworrk Llama-3.1-8B (Liu et al., 2024a)-244- as a proxy for the oracle reward. Throughout training, we query this model exclu- sively for preference labels, simulating human feedback.
1.1. Related Work
Existing Sampling Schemes. In academic papers, uni- form vanilla sampling the most commonly used approach,
See Appendix G for extension to PPO.
2

---

Optimal Sampling for Reward Modeling
while methods such as best-of-N and worst-of-N have also been explored (Dong et al., 2024). Xie et al. (2024) propose sampling one response from the current policy model and another from a reference model, modifying the loss function to encourage optimistic behavior. Similarly, Zhang et al. (2024) sample one response from the current model but rank it alongside two offline responses from the reference model. Shi et al. (2024) present a formula similar to ours based on intuition, introducing several hyperparameters and analyzing convergence speed with DPO in a tabular setting. Liu et al. (2024c) train an ensemble of reward models to approximate a posterior distribution over possible rewards and use Thompson sampling to generate responses with exploration. In contrast to these works, we theoretically es- tablish the principles of response generation for preference labeling, making minimal assumptions and simplifications while demonstrating the optimality of our approach. Our approach eliminates the need for hyperparameter tuning.
preferences,
D D= ( Di: v.) i=1
where each sample contains: (i) prompt 24, independently drawn from a distribution p. and (ii) a pair of responses [ Ji), where yf is preferred over y in human label- ing. The response pair (1),", V is first generated from a joint distribution μu- x) and then presented human la- belers for preference annotation. Human preferences commonly modeled using the Bradley-Terry (BT) model, which assumes:
P(ù" > 1²) = orr((z,, "") - r (z, g"). (1)
where (", Tj) represents the (unknown) oracle reward of response given a prompt, and a(2) = [1 + exp(-2)) is the sigmoid function, mapping differences in rewards to probabilities. We adopt the BT model throughout this paper.
Reward Modeling. The preference data, encoding human judgment, is then used to train reward model, rg, which serves as a measurable objective for training the policy model. ro trained by solving a MLE objective:
Policy Gradient. Our theoretical principle is closely related to the family of policy gradient methods (Williams, 1992: Sutton et al., 1999) in reinforcement learning, which opti- mize a policy ny by estimating and ascending the gradient of the expected return (O) Significant advancements have been made to improve the efficiency of these meth- ods, including variance reduction techniques (Greensmith et al., 2004), off-policy gradient estimation (Degris et al., 2012), interpolating on-policy and off-policy updates (Gu et al., 2017), deterministic policy gradients (Silver et al., 2014), and three-wa three-way robust estimation approaches (Kallus & Uehara, 2020). Our study extends these principles pref- erence learning for LMs, aligning the MLE gradient with the oracle objective gradient by controlling the response sam- pling distribution, thereby improving learning efficiency.
min 2(0) = 1 Σ log a (ro T.T) ro (o o.)). 0
This empirical loss approximates the expected negative log- likelihood [(0)= :=
E Z-p ( D (-nn(-a))
log or (re(x, - ro(x,, ro(x,"')) }) (3)
Policy Optimization. To align a language model ø with human preferences, we optimize it to maximize the learned rewards ro while staying close to a reference policy Tr The objective is
A review of other RLHF literature, particularly on data se- lection for the preference dataset, is deferred to Appendix A.
max Ez-p.g m) [re(e, 0) -3DKL Tree)- (4)
2. Problem Setup and Motivation
It consists of two parts:
(i) The reward term Ez<p, g-n(()))))))))) [re(z, Ø)] encourages
2.1. Aligning LMs with Human Preferences
the policy to generate high-quality responses.
Language Model (LM). At the core of our framework is a language model that processes prompts E X' and gen- erates responses ğ E y. Each response is represented as a sequence of tokens ğ = (y1 M2..... .. The primary goal of RLHF to guide the model to generate responses that align with human preferences. This translates to design- LM) that maps
(ii) The regularization term DxL(x || TTea) penalizes devi- ations from the reference policy That and is defined as Ex<p DKL ( x) (- x))]
Here, B is regularization parameter that balances the trade- off between reward maximization and adherence to the refer- ence policy. We assume 3 is fixed and practitioner-speccffeee.
ing decision policy 7 (parameterized prompts to responses, maximizing reward that reflects human preferences (with a KL regularization).
2.2. Direct Preference Optimization
The above-described RLHF pipeline typically leverages the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) to perform policy optimization. This approach requires loading the policy network, reward model, refer- ence model, and value model onto the GPU during training.
Preference Data. The oracle reward for human values inherently inaccessible. Instead, the alignment process approximates the reward using a dataset of haben-labeled
3

---

Optimal Sampling for Reward Modeling
making it highly resource-intensive. To improve computa- tional efficiency and practicality, Direct Preference Opti- mization (DPO) (Rafailov et al., 2023) has been proposed, enabling direct alignment without the need for a reward model or a value model.
We observe that these two gradients share a similar struc- ture. The key difference is Ar* Are for Vo (TTo) and o ((Arr)- for Ve (()).
In this work. we design a sampling distribution μ to correct this mismatch. Our new sampling method is optimal in the sense that DPO, when using our sampling, will maximize the oracle objective J((r),, even without direct access to it. The sampling strategy improves the quality of the preference dataset, maximizes the utility of limited data, and enhances both performance and efficiency.
A key insight of DPO is that any policy Teg can be viewed as the optimal solution to problem (4) if the reward ry is
Tta y x) Trof 1 x))
(a ÿ)- = 3 log
(5)
Thus, DPO can directly optimize the policy Tie using Σ(() in Equation (2), where re is replaced by πh as defined in Equation (5). This reformulation makes the objective depen- dent solely on 0. with the reward being implicitly learned through the policy itself. As result, the optimization pro- cess becomes significantly more efficient.
This focus is particularly crucial in scenarios where addi- tional data is collected during mid-training- a key phase in the iterative fine-tuning of LMs (Touvron et al., 2023; Bai et al., 2022; Xiong et al.. 2024; Guo et al., 2024). At this stage, preliminary policy The (distinct from Tret) already in place, but its performance may fall short of expectations. It is thus necessary to gather additional preference data, ide- ally on-policy data that target areas where the current policy shows room for improvement. An effective sampling design can significantly enhance the efficiency of leveraging human feedback in this process.
2.3. Motivation: Realigning Oracle Reward Maximization
To fully align with human values, RLHF should, in principle, train the policy to maximize the oracle reward, r", as defined in the BT model. The corresponding oracle objective is then:
3. T-PILAF: Theoretical Sampling Scheme
( : = Ez-p. (-m()11)) [r" Ti]
We now present T-PILAF- - theovetically grounded policy interpolation for aligned feedback - our sampling scheme for generating responses in data collection". The scheme is shown (in Section 4) to be optimal from both optimization and statistical perspectives.
- 3 DKI. (TT || Teet).
(6)
Since direct access to r* is unavailable, RLHF instead relies on preference data, either through MLE-based reward mod- eling or methods like DPO. However, these processes are not inherently designed to train the policy to directly maxi- mize the oracle objective, (xr). The following comparison of the gradient will highlight the differences.
Consider we have an initial policy ny and aim to collect pref- erence data to further refine its performance. We propose two complementary variants of policy ne: one that encour- ages exploration in regions more preferred by Tg, reflecting an optimistic perspective, and another that focuses on areas less favored by TT, reflecting a conservative adjustment.
To make the notation concise, we introduce the follow- ing shorthands: Ar+ : r"(x,, 3j)) rx, T²). Are : ro(2, 8²t) re g²) and g = Vero(z,
Vo (1x J¹). The lemmas give the following expressions for the gradients of the scalar value (TTO) (the optimal gra- dient towards the oracle objective) and the training loss (()) (the gradient actually used by DPO):
Specifically, we define policies 7े and ng around ni as
T x) = 7 (x)
Tal x) exp (ro 9)}, (9a)
7g y x) =
To (
x)) exp { r0 Ø)} (9b)
Lemma 2.1 (Gradient of value J(To)). For any no, the gradient of the expected value J (TT0 satisfies
Zg (x)
where the reward function re is defined in equation (5). The partition function 2 (x)) (or 2g (x)) is given by z (x) = Iy To( x)) cxp(ro( 7)} dÿ.
Vo = 23 Expp: "0
1 [o [[Ar* Aro] g]
For any prompt x E X, our sampling procedure involves the following steps:
Lemma 2.2 (Gradient of the loss function [()) for vanilla sampling). For any ne and the vanilla response sampling scheme, the gradient of the negative log-likeldoood function ((0) is given by
(i) Draw a random variable & from Bernoulli(po (1)), where po(x) := 2 (x) 2n (x)/ + 2 (z) 2g (x))
Vo (()) = - -Ez=p: "P.'"~mo(-11
Thh T in T-PILAF serves distinguish the thenretical scheme from the derived, simplified, efficiently implementable PILAF.
[(((Ar)) (a(drr)-((A0)) (A(rr))
4

---

Optimal Sampling for Reward Modeling
(ii) If & 1, independently draw responses "" "." y according ~ T (- u) and ÿo To (. (-1x). g O, draw responses as y"," y Tol. xx).
term T2 represents second-order error.
The detailed proof of Theorem 4.1 is deferred to Ap- pendix C.I. Recall from Lemma 2.1 and Lemma 2.2 that the difference between two gradients is the sigmoid func- tion: the most notable technical contribution here is show- ing how to leverage our sampling scheme to approximate the derivative o' of the sigmoid function. By using T- PILAF sampling, we can transform the difference term of the form a (Ar*) - (Aro) in Vo L(0) into a linear dif- ference Ar* - Are in Ve J ). This bridges the gap between the non-linear sigmoid differences and the linear reward differences.
In the next section, we will theoretically analyze T-PILAF. To account for the changes in sampling, we adopt a slightly modified loss function in the theoretical framework:
[(0) = 1 n n
Σ (m:) log a (z; v.) - ro (1. v!)
The newly introduced weight function is defined as w(x)= = {1 2 (x) Zg [ Zo,
(10)
where the normalization constant Zo > 0 is given by Zn = 1 Jx 2 (z) Z (z) p(z) dr. We also modify the population loss L in Equation (3) with the weight function.
Theorem 4.1 establishes the gradient alignment property, demonstrating that minimizing the likelihood- based loss function & closely aligns with maximizing the oracle ob- jective function J, with only a minor second-order error. It highlights how the proposed sampling scheme enables the DPO framework to effectively guide the policy toward opti- mizing the expected reward. Beyond DPO, in Appendix G, show how the same principle can be applied to PPO- based RLHF algorithms to help improve the sampling.
4. Theoretical Analysis
This section provides the theoretical grounding and analysis of our proposed sampling scheme from two perspectives. In the optimization analysis (Section 4. we show that T-- PILAF aligns two objectives (gradient aligument property): maximizing the likelihood function (Equation (3)) becomes equivalent to gradient ascent on the value function J(TT)) (Equation (6)). Consequently, policy updates on To move the parameters in the direction of steepest increase of J. T- PILAF thus provides the potential to accelerate training and improve generalization, compared to vanilla (uniform) sam- pling. In the statistical analysis (Section 4.2) we focus on statistical error and show that the asymptotic covariance of the estimated parameter 0 (inversely) aligns with the Hessian of the objective function J when sampling with T-PILAF. As a result, T-PILAF makes the sampled comparisons more informative, as they align with directions where J is most sensitive. The net outcome is reduced statistical variance of our method through tighter concentration of estimates in directions that matter most for performance.
4.2. Statistical Considerations
From statistical standpoint, we first examine the asymp- totic distribution of the estimated parameter @ when it (ap- proximately) solves the optimization problem (2). In Theo- rem 4.2, we formally characterize the randomness or statisti- cal error inherent in 0 under this idealized scenario. The de- tailed proof of Theorem 4.2 is provided in Appendix C.2.2.
Theorem 4.2. Assume the reward model r* the BT model (1) satisfies r" = Tg- for some parameter 6*. Under mild regularity conditions, the estimate 0 asymptotically follows Gaussian distribution
Vn (0 (0) - (0, as n2 + 00.
We have an estimate of the covariance matrix Ω:
4.1. Optimization Considerations
Ω C1 E
We begin by analyzing the DPO algorithm from an opti- mization perspective.
where C1 > 0 is universal constant. When using T-PILAF. the matrix Σ, is given by
Theorem 4.1 below formally illustrates how T-PILAF en- sures alignment between the MLE gradient, Vo L(0), and the oracle objective gradient, Vo (TTo).
Σ, : = Expo [oV (11)) V r (z. ②) x
(11)
Next we analyze the performance of the output pol- icy Ωे = Tg from Theorem 4.2 in terms of the expected value J(T).. In Theorem 4.3, we show that our proposed sampling method guarantees that the covariance of the sta- tistical error in 0 aligns inversely with the Hessian of J at the optimal policy πτ' This alignment prioritizes conver- gence efficiency along directions where the Hessian has large eigenvalues, adapting to the geometry of the optimiza-
Theorem 4.1 (Gradient structure in DPO training). Using data collected from our proposed response sampling scheme T-PILAF. the gradient of ((0) satisfies
3 Zo
Vo
Vo ((To) + T2,
where the constant 7.0 is defined in equation (10), and the
5

---

Optimal Sampling for Reward Modeling
tion landscape. It highlights the efficiency of our sampling scheme in reducing statistical error. For the detailed proof of Theorem 4.3, see Appendix C.2.3.
where Tÿ (ye I x. v1:e-1)
1
To (3le
2r (1.31ee11 1)
Z (z, V1:-11) To To Ir V1-t1))
Tree (3t
22, (1:-11)))
Theorem 4.3. The value function J(r)) we define in equa- tion (6) satisfies Vo J(x') )= 0 and
with (z. 91-1-1) being a partition function. The substi- tution of re uses the correspondence between the reward model ro and the policy Th in Equation (5), under the assumption that this correspondence holds for all trunca- tions 91.0-1- It gives direct per-token prediction rule:
V J(T()) = 3 Σ,
(12)
for matrix Σ. defined in equation (11). As a corollary, suppose E. is nonsingular, then there exists a constant C2 > such that for any E > 0,
T (- x. V1:-1)
=sofmmax {(1 ((1+3) ho heee J (1, Bra-1))
a( d([+e)]
sum P. J(Rt < J(T') C2
(13)
Here hg and hre are the logits of the policies Ti and Tyefs respectively. 8 is the regularization coefficient from the objective function J(x) in Equation (6). Responses are then generated using standard decoding techniques, such as greedy decoding nucleus sampling. Similarly, the generation for xg follows
1-000
n
I P p2xa > =1+
S <exp { 2( (s- log(1 + }))
d
2
Our proposed sampling distribution ensures that the output policy áी performs predictably and reliably. The value gap ((x*) - (ft) asymptotically follows a chi-square distribu- tion, irrespective of the problem instance details, such as the underlying reward model r* This structure -iivarrann statis- tical efficiency allows the method to achieve asymptotically efficient estimates without requiring explicit knowledge of the model structure.
Tg (. xr 11=-1)
softmax ((1-3) 33 ho + B href } (z, m))-
For a detailed, step-by-step proof, see Appendix D.4.
We formalize our final algorithm in Algorithm 1. Vanilla DPO (Rafailov et al., 2023; Guo et al., 2024) employs basic generation approach, sampling 5 T. Thp at Step 3. In contrast, instead of only sampling from Teg our sampling scheme interpolates and extrapolates the logits ho and h with coefficient 3, enabling exploration of a wider response space to align learning from human preference with value optimization. The /3 here is the same parameter that controls the KL regularization in Equation (4), as set by the problem.
We further derive general lemma describing how u affect the covariance in Appendix B. This result provides broader insights into what constitutes good preference data in RLHF
5. PILAF Algorithm
We now demonstrate that the T-PILAF sampling scheme defined in Equation (9a) and (9h) can be naturally extended into an efficient empirical algorithm (PILAF).
Cost analysis. We summarize sampling and annotation costs per preference pair for PILAF and related sampling schemes in Table 1. In Vanilla sampling (from πT), two generations and two annotations are required for human preference labeling, same to PILAF when the pair is sam- pled from Tg. which happens half the time. With 50% probability, PILAF uses πg. and Ti to generate, requiring two forward passes with Ti and Tref to generate one sample. Thus, on average, a preference pair sampled with PILAF requires a sampling cost of 3 forward passes (1.5 time the cost of Vanilla) with the same annotation cost. To compare, Xiong et al. (2024); Dong et al. (2024) perform Best-of-N sampling with N = which generates and annotates all 8 responses, selecting the best and worst of them. Xie et al. (2024) use a Hybrid method that generates with Th and Tie thus matching the sampling and annotation costs of the Vanilla method. We empirically compare PIL.AF with these methods in the next section.
The first challenge in implementing these definitions lies in calculating the normalizing factors % (xr) and Zz (x), which can be computationally expensive for LLMs. To ad- dress this, we simplify the process by omitting these factors and replacing them with 1.5 Consequently, the sampling pro- cess becomes straightforward: with probability 1/2, we sam- ple using nıg and otherwise, we sample using zτ and Th-
The second challenge lies in sampling a response ÿ from noo Ty .r) exp + (a } } in an autoregressive way for next-token generation. We argue that the policy TiJ (and Tt ) can be approximated in a token-wise manner:
n (ū ()) n (y1 x)) nt (y2 xr y1)
T (yel xr, 11) ... nे (UT x., ar 1))
'When the regularization coefficient 9 is sufficiently small, the term exp(ro(z, ;)} in equation (9a) stays close to and has only a minor effect. Consequently, the partition function Z (x) is approximately 1. A similar reasoning applics to 7g (z).
6

---

Optimal Sampling for Reward Modeling
Table A cost summary of PILAF and sampling methods from related works. Best-of-N method Xiong et al. (2024) uses the oracle reward to score all candidate responses, then selects the highest- and lowesng ones ones-instead of providing preference label for only two responses. We restrict the oracle to providing only preference labels. Thus, we create a Best-of-N variant that uses the DPO internal reward for selection and then applies preference labeling, with an annotation cost of 2. We compare with this variant in the experiment.
METHOD
y
SAMPLING COST
ANNOTATION COST
Vanilla (RAFAILOV ET AL., 2023) Best-of-N (XIONG ET AL., 2024), N=8 Best-of-N (WITH DPO REWARD), N=8 Hybrid (XIE ET AL., 2024)
T
75
2
2
BEST OF To BEST Tgg T
WORST OF Te
8
8
WORST OF TIp
8
2
Tlyef
2
PILAF (OURS)
72g /To
Tg TTo
2
Algorithm 1 DPO with PILAF (ours).
SFT dataset and perform full-parameter tuning.
input Prompt Dataset D,, preference oracle O, Te, Tyef-
Evaluation. We present our results using the reward-KK -LL curve, following Gao et al. (2023), with the reward evaluated by the oracle reward model O. To monitor the impact of our sampling scheme on the optimization trajectory, we evaluate the model every 50 gradient steps during training. We use the entire testset of HH-RLHF (8.55K samples) to evaluate.
1: for step t= ..., T' do
2: Sample n,, prompts { from Dp--
3: With probability 1/2, sample y y. ~ nι; with prob-
ability 1/2, sample o T and J Tg-
4: Query O to label (zi, D ( into (zi, D. y
5: Update Te, with DPO loss using {Tx, U. F. He
6:
end for
Baselines. We compare our sampling method against exist- ing methods in Table 1, and with VPO (Cen et al., 2024), which uses the vanilla sampling but incorporates an explicit exploration term in the loss. Since we treat the oracle O as proxy for human labelers that can only provide pairwise preferences, all baselines are constrained to query the oracle with exactly two samples at a time. We thus adapt Best- of-N variant that deploys the internal DPO reward to select the top and bottom candidates, which are then presented to the oracle for preference labeling. as listed in Table 1. We compare PILAF against the baselines: Vanilla Sampling. Best-of-N Sampling (with DPO reward), Hybrid Sampling combined with modified DPO loss (Xie et al., 2024), and VPO (Cen et al., 2024).
6. Experiments
In this section, empirically evaluate PILAF in both iterative DPO setting (Section 6.1, following Xiong et al. (2024); Dong et al. (2024)) and an online DPO setting (Sec- tion 6.2. following Guo et al. (2024)) where the model undergoes multiple rounds of refinement through active data collection. Our findings indicate that, without requiring any hyper-parameter tuning, our sampling scheme stabilizes training, achieves higher reward scores, and maintains lower KL divergence from the reference model.
General Setup. We align the Llama- 3-BB base model (Dubey et al., 2024) in terms of helpfulness and harmless- ness using the HH-RLHF dataset (Bai et al., 2022), a widely- used benchmark dataset for alignment. It consists of 161k prompts in the training set. For response preference labeling, use well-trained reward model simulate human pref- erences by assigning preference to pairs of responses under the BT assumption in Equation (1). Specifically, we em- ploy the Skywork-Rewee Reward-8B model (Liu et al., 2024a), a pop-performing 8B model on RewardBench (Lambert et al., 2024), as our oracle O. During training, interaction with this reward model is limited to providing two responses for comparison. We set B = 0.1 in all the experiments.
Full experimental details can be found in Appendix F.
6.1. Iterative DPO
Implementation. We first consider the iterative DPO frame- work (Xiong et al., 2024; Dong et al., 2024), in which prefer- ence data is collected in successive iterations rather than as single fixed dataset. At the start of each iteration, a large dataset of responses is sampled using the current model, annotated for preferences, and then used to train the current model. Concretely, we set n, [Dpl in Algorithm 1, mean- ing that all prompts are used to generate new responses at each iteration. During the first iteration, when Tyet and n& are identical, PILAF reduces to Vanilla Sampling. Hence, we choose to focus our comparison on the second iteration. For consistency, we initialize all runs with the same policy model obtained the end of the first iteration via Vanilla Sampling.
(SFT). To initialize training, fol- lowing Rafailov et al. (2023), we first fine-tune the base model to obtain the SFT model as Tyefs which we fix as the reference model in all the experiments. We use the origi- nally preferred responses from the HH-RLHF dataset as the
Supervised
7

---

Optimal Sampling for Reward Modeling
Iterative DPO
the same Th and focus on comparing the subsequent opti- mization. Further details are in Appendix F.
-9.
Online DPO
-.50
Varilla Hybrid
Reward
-.75
-10.5
Best-offN
PILAF
Vanilla
-11.0
Reward
-626
Best-of-N
6.50
Hybrid
PILAF
-.75
VPO
15 20
25
35
-600
(LT (T
Figure 2. Reward-KL curve for Iterative DPO, All training rums start from the same model obtained at the end of the first iteration via Vanilla Sampling. Each dot represents an evaluation performed every 50 training steps.
(LT mTm))
Figure 3. Reward-KT. curve for Online DPO. Each dot represents an evaluation performed every 50 training steps.
Table 2. Results of Iterative DPO. We report the average reward, KL divergence from the reference model, and objective J on the testset. Higher reward and J are better, while lower KL divergence is better. We use boldface to indicate the best result and underline denote the second-best result.
Results. Figure 3 demonstrates the effectiveness of PILAF in the pure online setting, and we summarize the final per- formance in Table 3. Compared with Vanilla, VPO, and Hybrid Sampling, PILAF achieves significantly better Reward-KL trade-off curve, attaining higher reward with lower KI.. Although Vanilla and VPO eventually achieve roughly the same reward value as PILAF, it comes at the cost of a substantially higher KL. When compared with Best-of-N, PILAAF traces a similar Reward-KL. trajectory but ends with a higher reward and a better final objective after the same number of iterations, translating to lower sample complexity and reduced annotation and computational cost.
METHOD
REWARD (1)
KL (4)
J (t)
Vanilla
-10.16
35.20
-13.68
Best-of-N
-10.13
32.38
-13.37
Hybrid
-10.51
22.86
-12.80
PILAF (OURS)
-9.80
25.01
-12.30
Results. Figure 2 presents the reward-KL. curve for iter- ative DPO. PII.AF significantly outperforms all the other methods: it achieves the end-point rewards of the baselines already around halfway through training, with around 40% less training time. This reduction directly translates to sav- ings in both annotation and computational costs. We sum- marize the final performance in Table 2. PILAF produces final policy with a high reward value and a modestly small KL divergence from the reference model, thereby achieving the highest overall objective J.
Table 3. Results of Online DPO. We report the average reward, KL divergence from the reference model, and objective J on the testset.
METHOD
REWARD (1)
KL (L)
(t)
Vanilla
-4.96
21.50
-7.11
Best-of-N
-5.54
12.35
-6.77
Hybrid
-6.42
16.46
-8.96
VPO
-4.91
22.31
-7.09
PILAF (OURS)
-4.88
15.42
-6.42
6.2. Online DPO
Robustness Analysis. Having established the effective- ness of PILAF, we further evaluate its robustness by testing whether improves optimization and statistical convergence under challenging conditions, as predicted from our statisti- cal theory Section 4.2. Specifically, we replace the initial model with one that has overfit fixed off-policy dataset. This setup allows us to examine how different methods handle optimization starting from poor initial point.
Implementation. We further evaluate our sampling method in the online DPO setting (Guo et al., 2024), where new re- sponses are generated and labeled at every training step, and these preference data are immediately used to update Te. This setting corresponds to the case where n, (in Algo- rithm 1) is set to the training batch size, resulting in the most annotation-intensive and most actively on-policy align- ment. By collecting and utilizing preference data on the fly for each batch, the policy is continuously refined using on-policy feedback throughout the entire training process. Similar to Iterative DPO, we initialize all training runs with
In Figure 4, we compare the performance of PILAF and Vanilla Sampling when both are initialized from an over- fitted policy. We observe that Vanilla Sampling rapidly
8

---

Optimal Sampling for Reward Modeling
increases its KL divergence from the reference model while its reward improvement diminishes over time. In contrast, PILAF undergoes an early training phase with fluctuating KL values but ultimately attains a policy with higher reward and substantially lower KL. divergence. We hypothesize that PILAF's interpolation-batt based exploration design enables it to escape the suboptimal region of the loss landscape in which Vanilla remains. These results underscore PILAF's effectiveness in more robustly optimizing overfitted (or even adversarially initialized) policies.
Online DPO
-..00
-5.00
5.25
Vanilla PILAF-extrapollt
Reward
PILAF-intopplate
PILAF
-800
-625
Online DPO with an Overfitted Initial Model
KL( |I meae)
Figure 5. Reward-KL curve for Online DPO with ablations. Each dot represents evaluation performed every 50 steps.
5
Roward
our theoretical prediction that the full PILAF algorithm is the best performing approach.
-7
-0
Vanilla PILAF
7. Conclusion
In this paper, we introduced Policy-Interpolant Learning for Aligned Feedback (PILAF), a novel sampling method de- signed to enhance response sampling for preference labeling. Theoretical analysis highlights PILAF's superiority from both optimization and statistical perspectives, demonstrat- ing its ability to stabilize training, accelerate convergence, and reduce variance. The method is straightforward to im- plement and requires no additional hyperparameter tuning. We empirically validated its performance in both iterative DPO and online DPO settings, where it consistently outper- formed existing approaches. To achieve the same level of performance, PILAF consistently requires lower annotation costs, which can be substantial when annotations require experts in snowledge-intenste domains.
KL(TT,
Figure 4. Online DPO with an overfitted initial policy. Each dot represents an evaluation performed every 50 training steps. Color saturation indicates the training step, with darker colors representing later steps.
6.3. Ablations
We further conduct two ablation studies to isolate the contributions of PILAF's interpolation and extrapolation components. Each component was replaced individually with vanilla sampling, yielding two baselines: one with (Ø"; .²)) = (Tz. : To) (ablation of the interpolation compo- nent) and one with (j", Thh = (To, कह (ablation of the extrapolation component). We denote these ablation vari- and interpllttee where one response is obtained via vanilla sampling and the other via extrapolation or interpolation, respectively.
In future work, we hope to extend PILAF to other paradigms, such as KTO (Ethayarajh et al., 2024) and IPO (Azar et al., 2024). Due to resource constraints, our evaluations were conducted using SB models and a reward model to simulate human feedback. Future studies involving larger-scal ex- periments and real human labeling would further generalize our findings
ants as PILAF-extrapon
The results are presented in Figure 5. Our theory suggests that the two sampling responses should come from different distributions in order to yield a controlled difference that the model can effectively learn from. Both ablation vari- ants introduce such differences and outperform vanilla sam- pling. However, the variant with only interpolation (com- bined with vanilla sampling for the other response) performs much worse than full PILAF, highlighting the importance of the extrapolation response. The eltaplextaa variant achieves slightly worse final results, and its convergence is much slower (each dot in our figure represents one evalua- tion after 50 steps). Overall, these ablation results confirm
Overall, this work takes an important step toward improv- ing preference data curation in RLHF pipelines, laying the groundwork for more effective methods in alignment.
9

---

Optimal Sampling for Reward Modeling
Acknowledgment
Degris, T., White, M., and Sutton, R. S. Off-policy actor- critic. arXiv preprint arXiv: /20:: 1205.4899 2012.
YF and JK acknowledge support through NSF NRT train- ing grant award 1922658. YD acknowledges support from NSF grant DMS-2413812. The authors would like to thank Gabriel Synnaeve, He He, Wei Xiong, Pu Yang, Angelica Chen for helpful discussions. This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W.
Pan, R., Diao, S., Zhang, J., KaShun, S., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rihf, 2024.
Impact Statement
Our work aims advance the state of Machine Learn- ing ung-paticularrly in aligning large language models with active preference feedback. We believe that our work could contribute to more reliable and user-aligned language mod- els, potentially improving downstream applications. By reducing the amount of annotation needed, the approach may lower the barriers to producing specialized models in resource-constrained environments. However, our exper- iments currently limited to using a reward model to simulate human preferences. Additional studies with real human annotation are necessary to understand the risk of misalignment.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21786 2024.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866, 2023.
While our work has implications for various societal as- pects, we do not identify any specific consequences to be specifically highlighted here.
Greensmith, E., Bartlett, P. L., and Baxter, J. Variance reduc- tion techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
References
Gu, S. S., Lillicrap, T., Turner, R. E., Ghahramani, Z., Schölkopf, B., and Levine, S. Interpolated policy gradi- ent: Merging on on-policy and off-policy gradient estimation for deep reinforcement learning. Advances in neural in- formation processing systems, 30, 2017.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprinl arXiv:2303.08774, 2023.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4452 PMLR, 2024.
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792. 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli. D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05882. 2022.
Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y. Openrlhf: An easy-to-use, scalable and high nigh-performance rlhf framework. arXiv preprint arXiv:2405.1 143, 2024.
Ji, K., He, J., and Gu, Q. Reinforcement learning from human feedback with active queries. arXiv preprint arXiv:2402.09401, 2024.
Cen, S., Mei, J., Goshvadi, K., Dai, H., Yang, T., Yang. S., Schuurmans, D., Chi, Y., and Dai, B. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arxiv preprint arXiv:2405. 19320, 2024.
Kallus, N. and Uehara, M. Statistically efficient off-policy policy gradients. In International Conference on Machine Learning, pp. 5089-5100. PMLR, 2020.
Das, N., Chakraborty, S., Pacchiano, A., and Chowdhury, S. R. Active preference optimization for sample efficient rlhf. In JCML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024.
Kosorok, M. R. Introduction to empirical processes and semiparametric inference, volume 61. Springer, 2008
10

---

Optimal Sampling for Reward Modeling
Lambert, N., Pyarkin, V., Morrison, J., Miranda, L., Lin, B. Y. Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Re- wardbench: Evaluating reward models for language modeling. https: //ugg igg.ugginggaaan sagffce.coom spaces/ allenai/rewarde bench, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms.
arXiv preprint 1rXXiv:707.06388 2017.
Shi, R., Zhou, R., and Du, S. The crucial role of samplers in online direct preference optimization. arXiv preprint arXiv:2409, 0et.:24099.96000 19605, 2024.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J.. Sutskever, I., and Cobbe, K. Let's verify step by step, 2023. URL https: //arxiv. org/ abs/ / 2205. 20050.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387- 395. Pmlr, 2014.
Limo, F. A. F., Tiza, D. R. H., Roque, M. M., Herrera, E. E., Murillo, J. P. M., Huallpa, J. Flores, V. A. A., Castillo, A. G. R., Peña, P. F. P., Carranza, C. M., et al. Personalized tutoring: Chatgpt as a virtual tutor for personalized learning experiences. Przestrzen Spoteczna (Social Space), 23(1):293-31) -122, 2023.
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Liu, C. Y., Zeng, L.., Liu, J., Yan, R., He, J., Wang, C.,, Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451 2024a.
Williams, R. J. Simple statistical tradient-following algo- rithms for connectionist reinforcement learning. Machine learning. 8:229-256, 1992.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024b.
Xie, T., Foster, J. Krishnamurthy, A., Rosset, C., Awadal- lah, A., and Rakhlin, A. Exploratory preference optimiza- tion: Harnessing implicit q qm-aporoxination for sample- efficient rlhf. arXiv preprint arXiv:2405.22 2046, 2024.
Liu, Z. Chen, C., Du, C., Lee, W. S., and Lin, M. Sample-efficient alignment for llms. arXiv preprint arXiv:2411.014 2024c.
Xiong, W., Dong. H., Ye, C., Wang. Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Con- ference on Machine Learning, 2024.
Mchta, V., Das, V., Neopane. O., Dai, Y.. Bogunovic, I., Schneider, J., and Neiswanger, W. Sample efficient inforcement learning from human feedback via active exploration. arXiv preprint arXiv:23 22..22222.00262 2023.
Muldrew, W., Hayes, P., Zhang, M., and Barber, D. Ac- tive preference learning for large language models. In Forty-first International Conference on Machine Learn- ing, 2024.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L.,, Van Durme, B., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. In Forty-first International Con- ference on Machine Learning, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 11-27730-27744 2022.
Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Preference opti- mization with the pairwise cringe loss. arXiv preprint arXiv:2312. 2iti:2312..666 16682, 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2023.
Yue, Z.,, Zhuang, H., Bai, A., Hui, K., Jagerman, R., Zeng, H., Qin, Z., Wang, D., Wang. X., and Bendersky, M. Inference scaling for long-context retrieval augmented generation. arXiv preprint arXiv:24400003343 2024.
Zhang, S., Yu, D.,, Sharma, H., Zhong, H., Liu, Z., Yang, 2. Wang. S., Hassan, H.,, and Wang, Z. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405. 19332, 2024.
Scheid, A., Boursier, E.,, Durmus, A., Jordan, M. I., Ménard, P., Moulines, E., and Valko, M. Optimal design for reward modeling rihf. arXiv preprint arXiv:2410. 11101007.5.. 77055, 2024.

---

Optimal Sampling for Reward Modeling
Contents
A. Additional Literature Review
12
B. Additional Statistical Results
13
C. Proof of Main Results
13
C.1. Optimization Considerations: Proof of Theorem 4.1
14
C.1.1. Building Blocks
14
C.1.2. Derivation of Theorem 4.1
15
C.1.3. Proof of Claim (18)
16
C.2. Statistical Considerations
17
C.2.1. Proof of Lemma B.1 (Theorem C.4)
17
C.2.2. Proof of Theorem 4.2
18
C.2.3. Proof of Theorem 4.3
19
D. Proof of Auxiliary Results
20
D.1. Proof of Auxiliary Results for Theorem 4.1
20
D.1.1. Gradients of Policy ny and Reward re
20
D.1.2. Proof of Lemma C.2, Explicit Form of Gradient Vo J(To)
21
D.1.3. Proof of Lemma C.3, Explicit Form of Gradient V C())
23
D.2. Proof of Auxiliary Results for Theorem 4.2
24
D.2.1. Proof of Condition (27)
24
D.2.2. Proof of Lemma C.5, Explicit Form of Hessian V C()*)
26
D.2.3. Proof of Lemma C.6, Asymptotic Distribution of Graident Vo [(0*)
26
D.3. Proof of Auxiliary Results for Theorem 4.3
27
D.3.1. Proof of Equation (12) from Theorem 4.3, Explicit Form of Hessian V J(T')
27
D.3.2. Proof of the Asymptotic Distribution in Equation (34)
28
D.3.3. Proof of the Tail Bound in Equation (13)
29
E. Supporting Theorem: Master Theorem for Z Z-Estimators
30
F. Experimental Details
31
G. Extension to Proximal Policy Optimization (PPO)
A. Additional Literature Review
RLHF. RLHF has emerged cornerstone methodology for aligning large language models with human values and preferences (Achiam et al., 2023). Early systems (Ouyang al., 2022) turn human preference data into reward modeling to optimize model behavior accordingly. DPO has been proposed more efficient approach that directly trains LLMs on preference data. As LLMs evolve during training, continuing training on pre-generated preference data becomes suboptimal due to the distribution shift. Empirically, RLHF is applied iteratively- geiilyyiiggrniingggg on-policy data at successive stages to enhance alignment and performance (Touvron et al., 2023; Bai et al., 2022). Similarly, researchers have introduced iterative DPO (Xiong et al., 2024: Xu et al., 2023) and online DPO (Guo et al.. 2024) fully leverage online preference labeling, Ultimately, the quality of preference data play critical role in determining the effectiveness of the alignment.
Sampling in Frontier LLMs. Technical reports of Frontier LLMs briefly mention sampling techniques. For instance,
12

---

Optimal Sampling for Reward Modeling
Claude (Bai et al., 2022) utilizes models from different training steps to generate responses, while Llama-2 (Touvron et al., 2023) further use different temperatures for sampling. However, further details are provided, leaving the development of principled method open challenge.
Data Selection. There is a line of research aimed at improving sample efficiency for preference labeling by selecting question and response pairs. Scheid et al. (2024) conceptualize this as a regret minimization problem, leveraging methods from linear dueling bandits. Das et al. (2024); Mehta et al. (2023); Muldrew et al. (2024); Ji et al. (2024) draw insights from active learning, using various uncertainty estimators to guide selection by prioritizing sample pairs with maximum uncertainty. These approaches focus directly on dataset of questions and responses and are orthogonal to our work.
Other Changes in Response Sampling. Several works also modify the sampling design directly (Liu et al., 2024b; Dong et al., 2023), but with the goal of improving policy network optimization based on reward model, rather than enhancing the reward modeling itself. Liu et al. (2024b) employ rejection sampling to approximate the response distribution induced by the reward model, thereby improving optimization. However, this approach requires access to the reward model and incurs higher computational and labeling costs. Similarly, Dong al. (2023) use best-of-N sampling with the reward model to generate high-quality data for supervised fine-tuning (SFT). We consider these approaches orthogonal to our work.
Additionally, Cen et al. (2024) introduce a bonus term in the policy learning phase of online RLHF to promote exploration in response sampling, which aligns with the optimism principle.
B. Additional Statistical Results
In addition to our analysis of T-PILAF in Section 3, here we present a generalized version of Theorem 4.2 that applies to any response sampling distribution μ. While not directly tied to the main focus of this work, this broader result may be of independent interest readers. The proof of Lemma B. is provided in Appendix C.2.1.
Lemma B.1. For general sampling distribution H. redefined as
the statement in Theorem 4.2 remains valid with the matrix Σ.
w(x) Var(1 = - x. T" F") 99
(14)
where the expectation taken over the distribution
a(U": T6 [x) = _ (H(T" 36d |x) u |z)}
(15a)
2
The variance term is specified as and the gradient difference g is defined as
(1 J x. T") = F - r -"r" (z,,
F -x,
(15b)
V Vor In) Ver
(15c)
The general form of the matrix Σ. offers valuable insights for designing a sampling scheme. To ensure Σ, is well- conditioned (less singular), we must balance two key factors when selecting responses ÿ" and T":
Large variance: The variance in definition (15b) should be maximized. This occurs when r*(x,g*) & r (2, g²). Intuitively, preference feedback is most informative when annotators compare responses of similar quality.
Large gradient difference: The gradient difference g from definition (15c) should also be large. This requires responses with significantly different gradients. Only then can the comparison provide a clear and meaningful direction for model training.
C. Proof of Main Results
This section provides the proofs of the main results from Section 4, covering both optimization and statistical aspects. In Appendix C.I, we prove Theorem 4.1, which establishes the gradient alignment property. For the statistical results, Appendix C.2 begins with the proofs of Lemma B.1 and Theorem 4.2, which derive the asymptotic distribution of the estimated parameter 0, and concludes with the proof of Theorem 4.3, analyzing the asymptotic behavior of the value gap J(x') - J(t)..
13

---

Optimal Sampling for Reward Modeling
C.1. Optimization Considerations: Proof of Theorem 4.1
We begin by presenting rigorous restatement of Theorem 4.1, formally detailed in Theorem C.1 below.
Theorem C.1 (Gradient structure DPO training). Consider the expected loss function [ during the DPO training phase. Using dala collected from our poposed response sampling scheme μs, the gradient of C(0) satisfies
B VoJ((T)) T2
Vo C()) = Zo
where the constant Zo is defined in equation (10), and the term T2 represents second-order error. To control term T2, assume the following uniform bounds:
(i) ||r" 11x0 R.
(ii) For any policy Th E II, the induced reward rg satisfies Irole I R and supa. Vore(z, Wll I SG. G.
Under these conditions, T2 is bounded as
Øo) - r r2 (x) - (re(x, ro }²))}
1Telle S C' Ez 8 3 mm)(11)))))
where the constant C is given by C = 0.1 (1 (1+ )
The proof of Theorem C.I is structured into three sections. In Appendix C.I.I, lay the foundation by presenting the key components, including the explicit expressions for the gradients Vp (TT)) and V [(O), well as for the sampling density μ. Then Appendix C.1.2 establishes the connection between V, (TTO and Vo [(0) by leveraging these results, completing the proof of Theorem 4.1. Finally, in Appendix C.1.3, we provide a detailed derivation of the form of density function μ.
C.I.I. BUILDING BLOCKS
To establish Theorem 4.1, which uncovers the relationship between the gradients of the expected value J(Tg) and the negative log-likelihood function [()), the first step derive explicit expressions for the gradients of both functions. The results presented in Lemmas C.2 and C.3, with detailed proofs provided in Appendices D.1.2 and D.1.3, respectively.
Lemma C.2 (Gradient of value J(To)) For any Th in the parameterized policy class II, the gradient of the expected value (To) satisfies
Vo J(To)
Fu 6- gh (((x)))))r((1) - (ro(s, 77) ro(o, r'))
23
(Voro(z. Voroot))) (16)
Lemma C.3 (Gradient of the loss function C(O)). For any Ti in the parameterized policy class II and any sampling distribution μ of the responses, the gradient of the negative log- log-likelihood function L(0) is given by
Ve (()) = Exppp (7) 1'M0 w(x) (o(r"(2, - -()))
- (ro(z,, 3r) - ro(x, 8d1)}
Vereezz )") (ero(z,8)) (17a)
where the average density μt is defined as
1 x)) = z {u(j 101 + pl h" xx)
(17b)
as previously introduced in Equation (15a).
14

---

Optimal Sampling for Reward Modeling
In Lemma C.3, observe that the gradient V [()) is expressed as expectation the probability distribution μ.. By applying the sampling scheme outlined in Section 3, we can derive more detailed representation of Vo [(0). This refined form will reveal its close relationship to the gradient Va J (TTe) given in expression (16).
Before moving forward, it is crucial for us to first derive the explicit form of μ. Specifically, we claim that the distribution satisfies the following property
(18)
TT n
2
Tog
x)
2 Zz (x) Zσ (x)} d'(ro(z, vo(xz
where o' denotes the derivative of the sigmoid function given by
d' (z) = (1 + cxp(-2))(1 + exp(2))
o(z)
for any 2 E R.
(19)
With these key components in place, we are prepared to prove Theorem 4.1.
C.1.2. DERIVATION OF THEOREM 4.
With the tools provided by Lemmas C.2 and C.3 and the sampling density expression in (18), we are now ready to prove Theorem 4.1.
We begin by applying Lemma C.3 and reformulating equation (17a) as
( 5" x)
Vo (()) - -Eesp 8e (-)(((112))
w(xx (0 x)) ná ( x)
(r F*) - r (x, (") a aroo(, ro(o, -r)(,x
(or Verala, [')
(20)
Substituting the density ratio from equation (18) into expression (20) and incorporating the weight function w(x) defined in equation (10), we obtain
(o("(z gn) r (z, FD)) (ro (0) - ro(x, rr(xz
Ve (())
E Ex-p: p: 8" 8 (1)(()
2
o' (ro(x, ja) ro( Fo))
(Voro ge) - 8a1}
(21)
Using the intuition that the first-order Taylor expansion
ax**) - o(2) -2=- - 2)²)
o'(2)
valid when z > 21, with 2" = rx, (0) r* (1r, T7d)
and : = ro( TJa) ro J²), we find that
a (r+ (x. To) (2, 84)) σ (ro( T²) r0(2, 8²1)
o'(ro 77) (1 57))
(r (x, - (z. v¹) - (ro (2, - ro (2 8²1)} + second-order term.
Reformulating equation (21) in this context, we rewrite it as
V ((o)
E gnp: 2Z.
(x* (x))))r",x)))) (re (z, (2, re (0())"")) b²1)}
0" ())(())))))))
Voro "²) Vore(x, T2.
(22)
15

---

Optimal Sampling for Reward Modeling
where T3 represents the second-order residual term related the estimation enror rø r-, By applying Lemma C.2, observe that the primary term in equation (22) aligns with the direction of Vo J (TT)) resulting in
B Z
V (())
Vo ((To) + T2.
(23)
Next, we proceed to control the second-order term Tz. The conditions
||r oo 11rolloc S R and
Supossieerso///ooottel 0112 S G,
lead to the bound
- o(2) -(2 (2 -2) 0.1 (z* =))2 (2)
which in turn implies
0.1 (1 + 22 G
Ezrp; gh m -(((0)))))))((())))) -x,,,")));,";)))))))))))) (ro(e, -o)(,, /
[22ll <
(24)
Zo
Finally, combining equation (24) with equation (23), conclude the proof of Theorem 4.1.
1.3.3. PROOF OF CLAIM (18)
The remaining step in the proof of Theorem 4.1 is to verify the expression for the density ratio in equation (18).
Based on the sampling scheme described in Section 3, we find that the sampling distribution for the response satisfies
T Jd {1 po(x)) To ( 2)) (8" (x) + Po(x) T (3
x)) Thp (i" xx),
(25)
where the probability (o() is defined as
po(x) 2z (x) 2p (1) Zz) (x))
and the policies nT and ri are specified in equations (9a) and (9b), respectively. This allows to simplify equation (25) to
x)) 1)
no 7
x)) 20 (x)
{ + exp (re(2, ;.) - زリ)}
" Jd x)
(x)
Similarly, derive an expression for u(j", go we obtain
ax). By averaging the expressions, for u(go J6 x)) and u(T", ga Ix),
To T To J xx)
{2 exp (ro(0, 0") ro(x, - exp fro(z, D²) ro(a,, 0ー}}}
To (1
2 2
Zz (x) Z (x)}
Rewriting this expression using the formula for o' in equation (19), we arrive at
n( ÿ x))
{1+ (x) (x))
TT
x)) Tea
(zj x))
2 1 exp - H + exp (ra(2,00) ra( 8²1}
ro
20' (ro
Finally, rearranging terms, we recover equation (18), completing this part of the proof.
16

---

Optimal Sampling for Reward Modeling
C.2. Statistical Considerations
In this section, we present the proofs for Theorems 4.2 and 4.3 and Lemma B.1 from Section 4.2. We start with the proof of Lemma B.1 in Appendix C.2.1, with rigorous restatement provided in Theorem C.4 below.
Theorem C.4. Assume the reward model r' in the BT model (1) satisfies r² Te- for some parameter 0*, Assume that i minimizes the loss function [(0) in the sense that vn V [(0) 4 and that 0 4 O the sample size + oc. Additionally, suppose the reward function ro(x, G), its gradient Vo (2. ū) and its Hessian Voro(z, V) uniformly
bounded and Lipchitz continuous with respect to 0, for all (x, T) E x y
Under these conditions, the estimate 0 asymptoticallyy follooss a Gaussian distribution
Vn (0 0*) - (0,
-
as 71 - 00.
We have an estimate of the covariance matrix S2:
Ω w00oo E.!
For a general sampling scheme chosen, the matrix E, is given by
:: Ezpp, (1" )))-
g²) w(xx Var(1 "" x. J") .99
where the expectation is taken over the distribution
u("" T (x) = 1/2 (H(T"" 5" [x)++ + m(j" ,"
5" |
The variance term is specified as and the gradient difference g is defined as
Var(1 // 1 T²) = (() (""
= Vor Vo (x,
Theorem C.4 establishes the asymptotic distribution of the estimated parameter 0, which serves as the foundation for the subsequent results. Next, we show that Theorem 4.2 directly follows as a corollary of Theorem C.4, with the detailed derivation provided in Appendix C.2.2. Finally, in Appendix C.2.3, we prove Theorem 4.3, which describes the asymptotic behavior of the value gap J(T") J (
C.2.1. PROOF OF LEMMA B. (THEOREM C.4)
In this section, analyze the asymptotic distribution of the estimated parameter @ for general sampling distribution μ.. The parameter 0 is obtained by solving the optimization problem
_ m 1 Σ w(xx) log l (To 57t ro (xi: 575)
minimizes 2(0)
We assume the optimization performed to sufficient accuracy such that V 2(0) = or (n Under this condition, ² qualifies 7-estimator. To study its asymptotic behavior, the formal statement of which is provided in Theorem E. in Appendix E.
the master theorem for Z-estimators (Kosorok, 2008),
To apply the master theorem, we set V = Ve C and v = Ve E and verify the conditions. In particular, the smoothness condition (64) in Theorem E. translates to the following equation in our context:
Vn (v 20) - V C vn (.. ((0)) (()))) V ((0)) o, (1+ vn 0 11))
(27)
This condition follows from the second-order smoothness of the reward function ro with respect to 0. A rigorous proof is provided in Appendix D.2.
We now provide the explicit form of the derivative ig- = V2 L(8*). as captured in the following lemma. The proof of this result can be found in Appendix D.2.2.
17

---

Optimal Sampling for Reward Modeling
Lemma C.S. The Hessian matrix of the population loss (()) at 0 0+ is
((0())) = Σ,
(28)
where the matrix Σ. is defined in equation (14).
Next, we analyze the asymptotic behavior of the gradient V [(0*). The proof deferred Appendix D.2.3.
Lemma C.6. The gradient of the empirical loss [(0) fr satisfies
Vnl (V) - V (()))) ب (O)
as n2 + 0o,
(29a)
where the covariance matrix 52 E Rdxdip is bounded as follows:
5 3 |ooloo E.
(29b)
with E, defined in equation (14).
Combining these results, and assuming Σ, is nonsingular, the master theorem (Theorem E.1) yields the asymptotic distribution of 0:
(0-00 & N(0,E,'0E,"
Furthermore, from the bound (29b), the covariance matrix 12; Σ, ii satisfies
Ω = 3 || ET
Therefore, we have established the asymptotic distribution of 0, completing the proof of Lemma B.
C.2.2. PROOF OF THEOREM 4.2
Theorem 4.2 is a direct corollary of Lemma B.I, using our specific choice of sampling distribution μ. To establish this, demonstrate how the general covariance matrix Σ, in equation (14) simplifies to the form in equation (11) under proposed sampling scheme.
To establish the result in this section, we impose the following regularity condition:
There exists a constant C' > satisfying
Var (1 - ""
x. "r g C. Var, (1 = -" Ir, "","""" g")
(30)
for any prompt X and responses J" 3" V. Here Var,, (1(J j x. J" Iz) denotes the conditional variance under the BT model (1), when the implicit reward function r* replaced by ro- The term Var,- (1 ( )) x., 77 77)
Var (1 ( ~ x, g", g²) represents the conditional variance under the ground-truth BT model, where the reward function is given by
We begin by leveraging the property of the sampling distribution μ from equation (18) and the derivative of of the sigmoid function a, given in equation (19). Specifically, we find that
Jo
7i "
Tig
x)
2 2 + 2 (x) 2g (x))
(ro(I, To) - ra( gD)) a (ra(z, (x, gh) ro(a, ge))
2 2 + 2/ (x) Z (x))
Var,, (1fza = J Ta gD)
We then apply condition (30) and derive
I 7
C-1
1 Var,- (1(1² -
(31)
Tig
x
74g
x)
2 (1
Z (x) 2 (z))
". 77
18

---

Optimal Sampling for Reward Modeling
Next, substituting this result (31) into equation (14), alongside the weight function w(w)) from equation (10), we reform Σ,
x (y
Σ, Exopp Ur, g0 ())
w(x)
Var Var(1(3* -J" 2.,". 99
Too (
I
TTp
x)
Fr Fxpp: 8. (>)(=())))) [g9]
(32)
Zc
The conditional expectation gg simplifies as
Eg - (-1x) 9g
x|
Eijo je m (11z)
Ver (z. 3")
Vo Ver (z, (" 0²) (xr }²)} x
2 Fij (11z)
Vor (z. Vor 5)
]] 2 Fo (1) Vor ÿ) Fnj Vor (2.
2- Covg~no(11x)
(s. 0) x]
Substituting this result into equation (32), arrive at the conclusion that
E.
C Eznp (ovg-TT(())) Vorr) (x, 21]
which matches the simplified form in equation (11) as stated in Theorem 4.2.
C.2.3. PROOF OF THEOREM 4.3
Gradient Vo J(T*) and Hessian V J(r'): The equality J(T') follows directly from the gradient expres- sion (41) for Ve (TTe), evaluated at 8 with rg =r"
The proof of the Hessian result, Vg (rT"
(1/3) E., involves straightforward but technical differentiation of equation (41). For brevity, we defer this proof to Appendix D.3.1.
Asymptotic Distribution of Value Gap J(T²) J(+): To understand the behavior of the value gap J(rt") J(t),, we start by applying Taylor expansion J(mT)) around 0* This gives
J(TT) ( = Jo (0- 0) -000 J(T') -0) + 01
2
By substituting Vo J(T*) = (a direct result of the optimality PT*), the linear term vanishes. Introducing the shorthand H - J(TT) (1/3)- E. the expression simplifies
J( - J( = 2 T 8*) + - 8" 12]
(33)
When the sample size is sufficiently large, 0 approaches 0*, making the higher-order term negligible. Therefore, the value gap is dominated by the quadratic form.
From Theorem 4.2, we know the parameter estimate satisfies
Vn 6*) 4 S).
Substituting this result into the quadratic approximation of the value gap, find that the scaled value gap has the asymptotic distribution
n (J(TT))-- - ( a 1 T H =: X
where (O,
(34)
This approximation provides a clear intuition: the value gap is asymptotically driven by weighted chi-squared-like term involving the covariance structure Ω and the Hessian- like matrix H.
To rigorously establish this result, we will apply Slutsky's 's theorem. The full proof is presented in Appendix D.3.2.
19

---

Optimal Sampling for Reward Modeling
Bounding the Chi-Square Distribution: matrix S provided by Theorrm 4.2:
To bound the random variable X, we first leverage the estimate of the covariance
22 3 Zz 110lo0 Σ,
where the constant C comes from condition (30). It follows that the matrix 92 HΩ2 appearing in equation (34) can be bounded
Ω HU I
C Cl
E, HΣ. C
Zo |
I
C
1+
2g Zg 100
8 I.
B
Here the last equality uses the definition of the weight function w from equation (10). Substituting this bound into the quadratic form, we derive
x = }- 2 2
T HΩ S C
1
2 Zz 10
2 2,
233
where 2 ~ (O, I). Since =Tz follows a chi-square distribution with degrees of freedom, X is stochastically dominated by a rescaled chi-square random variable
+ 1 Zp
1xc.
C'
28
Equivalently, we can express this dominance as
lim sup sup P (J(T)) - J(R)) > C 1+
1 2p
1000
1 s P(xa>11
for any / > 0.
(35)
2,
+20
This incquality, given in equation (35), corresponds to the first bound equation (13).
The second inequality in equation (13) provides precise tail bound for x2- As its proof involves more technical details, we defer it to Appendix D.3.3.
D. Proof of Auxiliary Results
This section provides proofs of auxiliary results supporting the main theorems and lemmas. In Appendix D.I, we present the auxiliary results required for Theorem 4.1. Appendix D.2 details the proofs supporting results for Theorem 4.2. Finally, Appendix D.3, we establish the auxiliary results necessary for Theorem 4.3.
D.1. Proof of Auxiliary Results for Theorem 4.1
In this section, we provide the proofs of several auxiliary results that support the proof of Theorem 4.1. Specifically, Appendix D. presents the forms of the gradients the policy ni and the reward ræ, which serve as fundamental building blocks for deriving the lemmas. Appendix D.1.2 analyzes the gradient of the return function (TT)), as defined in equation (6).
Appendix D.1.3 focuses deriving expressions for the gradient of the negative nigllikl function L(0).
D.1.1. GRADIENTS OF POLICY Tg AND REWARD
In this part, we introduce results for the gradients of policy no and reward ro with respsect to parameter 6, which lay the foundation of our calculations.
Lemma D.1 (Gradients of policy and reward function rg). The gradients of the policy Ta and the reward function can be expressed in terms of each other as follows
V To (d x) (dğ
x)
1
Vere(z.ū) Eg En Vora(z, 8')]
(36a)
3
Ve Vo V
x)
Vero(z,
B
(36b)
no
x))
20

---

Optimal Sampling for Reward Modeling
We now proceed to prove Lemma D.1.
To begin, recall our definition of the reward function Fg as given in equation (5). It directly follows that
Voro(a, 3) = B Toll x)
Th y x)
This result confirms equation (36b) as stated in Lemma D.1.
Next, we express the policy nto (dý
x) in terms of the reward function ro (2 Ø). By reformulating equation (5), we obtain
Zo (x) Tret (dy x)) exp { { ro
ro( 3)
To (dÿ
x))
(37a)
where (z) is the partition function defined
Za(#)) = /, Trrs (dý x)) exp { ra(a,, 5)}
(37b)
We then compute the gradient of TTa (T(dd | x)) with respect to 0. Applying the chain rule, we get
V Ao x)) = Z [dy [x) exp { ro( 5)} 3 Voro(z,
(z)x) The x)) exp ro(a, 3) Vo Zo(x).
(38)
We need the gradient of the partition function (z())
I,, (rer(d exp
exp { ro(s,
( })} 1 Vora(z,
V Zo(x)
= -Zolz) / Toldÿ x) V Vere(æ.8) Ø)
1
Zo(x) (igono(-la)) ra 3)
(39)
3
Substituting equation (39) back into equation (38), simplify the expression for the gradient of noo (dğ x:::
Vo (dğ x) = (x) The (dă 2)) exp { (a ;) 3 (Vo Vor Ø) - Eg [oro(z,
This matches equation (36a) from Lemma D.I, thereby completing the proof.
D.1.2. PROOF OF LEMMA C.2
Equality (16) in Lemma C.2 can be derived as consequence of a more detailed result. We state it in Lemma D.2. Lemma D.2. For policy ng, the gradients with respect to the parameter 0 of its expected return Ez-p, Fama ( (x) [r (x. 8)] and its KL divergence from a reference policy (TL |l Tret) are given by
Ee Vo (~aa(-12)) (r" (x, Ø7)
= Exwp, (-1)())
r" (x, σ) Voro(z, ÿ) Egg Voro(z,
(40a)
1
Ve DKL T(at) = Exmp, j~mo (1b))
re( 87) Vo (z 3) - -Eg (orrat
(40b)
B2
Recall that the scalar value J (TT) of the policy is defined
(TT) Exup, grono (-lr) [r (2, Ø)] (KL ll TTor).
21

---

Optimal Sampling for Reward Modeling
Using Lemma D.2, we derive the gradient of J (TTO) as
Vo J(Te) Vo Ex-p, ( [r* (z, H)] Vo (TL Treet)
1 Hewp g~mo((111)
(I) ro(a, .)} Vere(z, 3)) (or
(41)
We rewrite the expression in equation (41) in two equivalent forms by exchanging the roles of y and g²:
((To) = B Ezmp, (0)(1())
(x" (3) - ro(z, 77) ) Voro(a, - Eggnao (1x) Vara
(42a)
( Ezm (1
re(a, ( きー)} re,,z""""rr""""""
} Vara(z, Eg (+12) V
(42b)
By taking the average of the two equivalent formulations above, we obtain equality (16) and complete the proof of Lemma C.2.
We now proceed to prove Lemma D.2, tackling equalities (40a) and (40b) by one.
Proof of Equality (40a) from Lemma D.2: We begin by expressing the expected return
Egep, (><)(1))) [r (r. 7]] = Expp 1/8 r" (x, ()) no (o x)
Taking the gradient of both sides with respect to 6, we have
Egpp [/ r" ( Vo [+)
Ve Ezpp, (11) [r" (z. T)
(43)
Using the expression for the policy gradient V Tig provided in Lemma D.1, the right-hand side of (43) simplifies to
RHS of (43) Fumu I (", () no (oyy x))
1 B
(oro(a,)) E (112) Vo F0
1 Exxp.gmma(le)
(" Ø) V Vara(z, Ø) E (ggraa())))) V Voro(a,
B
This completes the verification of equation (40a) from Lemma C.2.
Proof of Equality (40b) from Lemma D.2: Recall the definition of the KL divergence
(xL TTet) = Ezup / no no(dj xx) log
Tha y x)
Tye
xT)
Applying the chain rule, we obtain
Vo DKL. (TTo
TTat)
Empop I/ Vo Nip (dā x:) log
T& y
x))
)
Bamp [/8
Vo Too(d 2x)
(44)
Tref y
x:)
Since the policy integrates to i.e., Jy To (dğ x)) 1, it always holds that
J V no (dī
Ve /
x))
(d x) 0,
(45)
i.e., the second term on the right-hand side of (44) is zero.
Using the expression (37a), we take the logarithm
Tog
x))
log
ro( Φ) log (z)x)
(46)
Tee
x)
22

---

Optimal Sampling for Reward Modeling
Combining equations (45) and (46), get
/
Th (ÿj x) The ạ (x)
V (o
2) log
S ro(ar, () Vo VorTo (d ()) log (o( /, Vo roo x)
I ra 3)) To x).
(47)
Now, similar to the proof of equation (40a), derive
1
[/s ro(xx; Vo To |x)
RHS of (44)
Fizpp
B
1
Ezpp, gmm061))) ro(a, 30) Voro Egg (o Voro(s, (oroott
32
which verifies equality (40b) from Lemma D.2.
D.1.3. PROOF OF LEMMA C.3
In this section, we prove full version of Lemma C.3 stated in Lemma D.3 below. Equation (17a) from Lemma C.3 follows directly as straightforward corollary.
In Lemma D.3, we consider a general class of distributions parameterized by that models the binary preference
Po > 7 x). The negative log-likellnn log-likelihood function is defined as
(()) - Ez (5*. )~)(((())))))) w(x) log Pe(y" > z7 x
The Meadley-Terry (BT) model described in equation (1) and the corresponding loss function L(0) in equation (49) represent special case of this general framework.
Lemma D.3 (Gradient of the loss function C(0), full version). For a general distribution class (Pa). the gradient of L()) with respect to is given by
Vo L()) =
IF Expop (1 T ((((a)))))
w(x) (
(P(T'' > 56 12) -
x)
V Pa
x)
(48a)
Po /
x)
Po
> ÿ
x))
where I is the average distribution defined in equation (17b). Specifically, for the Bradley- Bradley-Terry (BT) model where
Po > J"h x) = (rool, Ø²) re(xz = { (
(TTg/Tree) ) (3)
x))
(TTO/ Ty (zja xx)
the gradient of L()) becomes
Vo C(0)
Expop (""."")p((l))) w(x) (()((,,)))~(<=1"p"" oro(a, re(o,
(Vrro(z V (oro(e,0")) (48b)
For notational simplicity, we focus on the proof for the case where the weight function w(x) 1. The results for general weight function w(z) > can be derived similar manner.
Recall that the negative log-likelihood function L(0) is defined
((0)
E
log Po > 5/ x))
23

---

Optimal Sampling for Reward Modeling
Based the data generation mechanism, can expand the expectation in [())
(()) Ezup: (8 (-)))))
P P
y" 2²) { log Po Po(J [J > g² [x)]
P(j n x) { - log Pn /" x
(49)
Notice that we can exchange the roles of 3J and J in the expectation above. This means that can equivalently express the expectation using the pair (j', T²) ~ u(- z). This symmetry allows us to replace in equation (49) with the average distribution şi as defined in equation (17b).
Next, we take the gradient of the loss function [(0) with respect the parameter and obtain
P PØ > 3" x)
((O) Exzp, (8" (.)11)))))))))))))
Vg Po(
> 8" [)}}
P Po > g" x)
IP P > 7 x)
{ Po 9" |x)}
Po > 7 x)
Note that PT(" > q" x) = PP " x)) and Po Po > 7
x)) Po /" x). Using this, we can
rewrite the gradient
P( 7 x.)
P > J" 12)
v C(0 = Ez (0' {~(())
Po(ij T7 ())
Po > 3" (x)
We simplify the expression further to obtain
Ve L())
px (5" (")0"))((--))
1P"0""~'' {Po x)) 1
P(5 - 10)
V8 Po 9" x))
Polg > 1" x Po
y
x))
This establishes equation (48a) from Lemma C.3.
As for the Bradley-Terry (BT) model, use the equality
o' d'(2) (1 + exp(((==2)) + exp(22) =(x))(--2)
for any 2 E R
to derive the following expression
Voro(e, - ())
(50)
Po(j J x Po
By substituting this gradient expression from equation (50) into equation (48a), we directly obtain equation (48b), thereby completing the proof of Lemma C.3.
D.2. Proof of Auxiliary Results for Theorem 4.2
In this section, we present the detailed proofs of the supporting lemmas used in the proof of Theorem 4.2. We begin in Appendix D.2.1 by establishing condition (27), which is crucial for the valid application of the master theorem for Z-cstimators. Following this, in Appendix D.2.2, we compute the Hessian matrix V C(0*) explicitly. Finally, in Appendix D.2.3, we derive the asymptotic distribution of the gradient Va Z(0*).
1.2.1. PROOF OF CONDITION (27)
We begin by rewriting the left-hand side of equation (27) as follows:
A = Vn V. 200 - VoL(O)} (n Z(0') - V ((('))
(V v. [(00)
-V, -.
(51)
24

---

Optimal Sampling for Reward Modeling
We then leverage the smoothness properties of the function ra, which guarantee the following approximations:
Ve Z Ve v ((0)) ( 6*) op 0* 0=12))
(52a)
V ((0) Vo [())) V ((0') (0- 8*) op (0((2)
(52b)
Assuming these equalities (52a) and (52b) hold, we substitute them into equation (51), leading to
A v ( ((0) @ 07) + o [1=)) vn [()) (0 0*) - 0*1)))
v - ((0)) - 0*) o0 (1 +1+ v 12))
(53)
Using the law large numbers, we know that V Z(0r) & vg ( which implies
V V= ((0))) 8*) (v 12))
Therefore, we conclude that as claimed in equation (27).
A - +1+ 0
The only remaining task to establish the validity of equalities (52a) and (52b).
Proof of Equalities (52a) and (52b): We express the loss function Σ(0) in the form
[(0) = 1 Σ w(x;) lo lo 0." 0.)
where the function Eo is defined
la 91, .2) (rg ro T2))
We then calculate the gradient V lg and Vi lo follows:
V Vololx, 12)
(ro(z, 52) ra 01)) (Voro(z, 02) Vorolz, 01) and
Vol,, d'(oo(z, T2) re(a, (Vero(z, (2) Vere(a, (orezze()) (e)
+ (2) ro(o, 0.) ( 02) (1))
When the reward function ro, F), along with its gradient V rs 7) and Hessian re( F), is uniformly bounded and Lipschitz continuous with respect to 0 for all (x, E X x 3: it guarantees that the Hessian of the loss function, V la. is also Lipschitz continuous. This holds with some constant L > 0 across all (z, 3) E X' x y. as demonstrated below:
Co T., F2) - V3 lo (z. T. 12) - L. 110 0 2
From this Lipschitz property, we deduce
L
|| la D1. T2) - lo - V (z, F1. (z, F2) (0 8-112 S
10 0r
2
and further derive
|||w|lo
Vo 2(0) - VoΣ(00) Vz (0 - 0²) <
||0
11
2
||l||
V. C()) - Vo vg (())) (0 00)12 <
|10
0* 11.
2
Finally, under the condition that Ø 2 8*, these results simplify to the expressions given in equations (52a) and (52b), as previously claimed.
25

---

Optimal Sampling for Reward Modeling
D.2.2. PROOF OF LEMMA C.5, EXPLICIT FORM OF HESSIAN C(0))
From equation (17a) in Lemma C.3, we recall the explicit formula for the gradient Va L()). Taking the derivative of both sides of equation (17a), obtain
V2 (()) = Exps (8. (p)))))((( w(x) o' (ra(æ, 0r) ro (2,
(orra<z V Vere( [" - Vera(z, =""r ']
E Ba (7", (P)T(())))))))
w(x) {r (z, 3²) - (r, 1) 3")
(54)
v0r,(zz0") - Viro زリ
When set 0 0", it follows that ro r*. This simplification eliminates the second term in expression (54), reducing the Hessian matrix to
2(0)) = Egwps (8 (-)(a)
w(x) d'
8²))
Vxr 5²) - Va r (z, vor
Substituting the derivative of with its explicit form, dz) a(2) 2-) for any E R, we refine the expression ((000)) E.
where the covariance matrix Σ, defined in equation (14). This completes the proof of expression (28) from Lemma C.5.
D.2.3. PROOF OF LEMMA C.6, ASYMPTOTIC DISTRIBUTION OF GRAIDENT V Σ(0*)
In this section, we analyze the asymptotic distribution of the gradient Va (()) at 0 = 0", where the loss function Z() is defined as
re (z., 81)) E(0) _ Σ w(x) log o o (ro D4")
Using the definition of the sigmoid function o, we calculate that
(log (z)) d' d((2//(22) = ==))) = =0(2) for any real number E R.
This allows us to reformulate V _[( as the average of n iid. vectors {u, Vimi
Vo 2(0)
1 Σ
wi
(55)
n 1=1
Here each vector u; E Rd is defined as
u, w(x)= σ (ro ; ro(o,, Ø.") { ror Vo (0, -oro 7(
At 0 = 0", we denote u and g, g. Notably, vector u, can be rewritten
u, = w(x) {0 (x, Ø.) - ro( xt)) ( = ". Th = f.1 9i:
(56)
where g, is given by
9: Vere( v7) Vore(x;.77)
From the structure of the BT model, it holds that
E [1
y. J.
J( xi] ((r v.) r" (x,, 1.))
26

---

Ω 1 ||w|loo E. Optimal Sampling for Reward Modeling
which implies E|u;]] O.
To analyze the asymptotic distribution of V ((0)), we apply the central limit theorem (CLT) to its empirical form given in equation (55). By the CLT, we have
Vn (n - Vo - 4 N(.
n --+000 00,
(57)
where the covariance matrix st Raxd is given by
n = Cov(ut) = E (ut) "]
Here have used the property E[(t]] 0 in the second equality.
now compute the explicit form of the covariance matrix Ω. Using the definition of u, from expression (56), we find that
f E (uF)
Ee (g+ ()(()(
(z) (o ( 0i)) --
-
(58)
Taking the conditional expectation over the outcomes winners and losers, and using the relation
E [o("" (21, 0i) r" (21, Øi)) - 1 = 5. J = mi}} V9. Ø4
Var Var(1(68 [ = 5". 5 fí}
x., x1. vi. 5イ)
(rr (T) r (c., かه)) a (xr F.)
(x 0.),
we reduce equation (58) to
Ω Ezmp: (0" (-))(()) wx(x). Var(1 Dr デी ブ 01, 07, 0') 9i(
Bounding the weight function w(x by its uniform bound ||w|ooo we simplify further:
Ω <
||0 bo E w((x)
Var(1 DT. si vi 21 vi) oi(oi)
This ultimately reduces to
(59)
where E, is defined in equation (14).
Finally, by combining equations (57) and (59), we establish the asymptotic normality of Vo Z(0*) and complete the proof of Lemma C.6.
D.3. Proof of Auxiliary Results for Theorem 4.3
This section contains the proofs of the auxiliary results supporting Theorem 4.3. In Appendix D.3.1, we derive the explicit form of the Hessian V (7T".. Appendix D.3.2 rigorously establishes the asymptotic distribution of the value gap (equation (34)). Finally, Appendix D.3.3 proves the tail bound (13) on the chi-square distribution x²
D. PROOF OF EQUATION (12) FROM THEOREM 4.3, EXPLICIT FORM OF HESSIAN V (TT)
We begin by differentiating expression (41) for the gradient V (TTe) to obtain the Hessian matrix V J(TT). The resulting expression can be written as
ViJ(TO) Ti T2 Ta.
27

---

Optimal Sampling for Reward Modeling
where the terms defined as follows:
Exmp S/ (r" ū) rooo 8)} v) voroz, - Eg Pii Voro(a, (io Vo To x)
Tı =
1
{ (Voro(o, Egg-To(112) (or Voro(e, 87T
T2 Exnp, smmoo(-1))
B
L3
Lemp, (1 (1))
ro(xz))) { (zra(z. Vollano (- Vore(zz
At the point 0 = 0", we know that =r. This simplifies the expression significantly:
F1 = 0 and r3 = 0.
Therefore, only term 12 contributes to the Hessian, and it further reduces to
22
Exwp, graa (-1x)
Vo (or 5) Vo (a(
B
+ Exwp [gg
Vo Vor 8¹)] (g)) [oro(z,
Enwp (ov Voro Vorg
B
From this simplification, we deduce
J(T')) =
3 Famp (oo) Vor [=]
which establishes equation (12) stated in Theorem 4.3.
D.3.2. PROOF OF THE ASYMPTOTIC DISTRIBUTION IN EQUATION (34)
The goal of this part is to establish the asymptotic distribution of n{J(x*)) - J( as stated in equation (34) from Appendix C.2.3. To achieve this, we first recast the value gap into the product of two terms and then invoke Slutsky's theorem.
We start by writing
J(TT) (
n --(( = n n. 8r) T 8*)
(60)
(0
H (Ø
Un
V.
By isolating U, and Vn in this way, we can handle their limiting behaviors separately:
Un 4 ETT Hn22
with = I),
(6la)
V2 P
(61b)
If these two results are established, the desired asymptotic distribution of the value gap, given in equation (34), follows directly from Slutsky's theorem.
To complete the proof, we proceed to verify equations (61a) and (61b). It is worth noting that equation (61a) is straightforward corollary of Theorem 4.2, so the main task is to establish the convergence result in equation (61b).
Proof of Equation (61b): Since Σ. is nonsingular, the matrix H = (Ze/3) Σ. is also nonsingular. From equation (33), we know that for any e E (0, 1), there exists threshold v/(e) > such that whenever ||0 0*1/2 S vi(e), the following inequality holds:
11 e)
(0 )-(0000
0 (0- (0)) s J(TT) J(TT) 5 g +5) (0 (0-0") 10-))TTT H (0 0*)
28

---

Optimal Sampling for Reward Modeling
This can be reformulated Next, under the condition that 0 4 0*, for any 6 > O, there exists an integer (ee,)) E Zz such that for any n 2 N(e, 6).
Vn 2 S E.
P[I: - 12 > v(e)) 66.
Therefore, for any 2 N(e,)), we can conclude
P Vn
ef S 6.
In simpler terms, Va 2 J. 2. which establishes equation (61b).
D.3.3. PROOF OF THE TAIL BOUND IN EQUATION (13)
We now establish the tail bound
P(x2 > (1+ s exp - d
(
1o-lo(1 -=olog(11++ -l)og(1++))} }
(62)
2
as stated in equation (13).
We first note that the moment-generating function (MGF) of distribution xă is given by
Maa (4) = (1 - 2() for any 1 < 1
Using Markov's inequality, for any f >>> 0, we have
P( > (1 s expf-th + c]d} Ma²(t) exp(-t(( + +)dd (1-2t) 2t)- for any t < 2.
(63)
We optimize the bound by choosing to minimize the exponent -((1 + 8)d 1 log(1 - 21).. Solving for the optimal I, we obtain
2(1 +
Substituting back into inequality (63), the bound simplifies to the desired inequality (62).
D.4. Proof of the next-token version of PILAF sampling
Proof of the Explicit Forms of TT (- | x, V1a-1) and np (- x. V1:-1): To begin, we express the policies Ti and Tye
in terms of their logits. Specifically, each policy can be written the exponential family form, normalized by a partition function:
TV x. 11:0-11 Co exp (yo x. v1--1)}
where Co: = Σ exp (oo x. v1a-1));
NEV
(y (1,1)=111
C.rl exp (fret (ye x. 11:01)1) where Cret
Σ
exp (hrer(y) x, v)=-)))
DEV
Now, let us consider the modified policy no (Ve F. V1. Substituting the expressions for Ti and Thet into its definition, we get
1
Tp(ye Jr, 11= 9
TT( (vel) xr, v1:t-1)
(v Ix, v1a-1))
(x, V1:-1)
Tye x., 3/1:1-1)
C exp( 8, mia-1)) ) C.1 chpe (8), x, M1a-1)}
Cp exp (ho(ve x. 1))
Z(x, V1:111)
29

---

Optimal Sampling for Reward Modeling
At this point, we observe that the terms (x, V1--11,, Co. and Czet do not depend on token yt, given the prompt æ and previous tokens 11=--1- Therefore, we treat them constants when focusing the structure of Th probabilities over token yr. This allows us to simplify the expression:
exp 8, 11a-113 1]] = exp ((1++))
Th (ye n. 11=-1)
(3 ho heef Hyt x. 1))+
exp (Ve 2; (1a-1)p
It shows that the modified policy xg is proportional to the exponential of linear combination of logits from Th and Tet To convert this into proper probability distribution, we normalize over the token space:
cxp (((1 B)hB S (yve 2, V1 V12-11)
Ti (y/e z. v1--11
Eyen exp (f(1 + 8)ho - (y 2, V1:-1))
In other words, the new policy simply the softmax over the combined logit:
TH (- x. V1:1-1) = softmax [(11 +11+ B) he - 3 heet ) (x. V1n-1))
as claimed.
The expression for the negatively weighted policy Ti. can be derived in a similar manner to sg..
E. Supporting Theorem: Master Theorem for Z-Estimators
In this section, we provide a brief introduction to the master theorem for Z-estimators for the convenience of the readers.
Let the parameter space be θ, and consider data-dependent function Vp θ - L, where L is a metric space with norm || I|.. Assume that the parameter estimate 0, E O satisfies ,,, ll 2 O, making O,, Z-estimator. The function v is estimator of fixed function V O + IL, where Vr(0)) 0 for some parameter of interest Og E O.
Theorem E.1 (Theorem 2.11 in Kosorok (2008), master theorem for Z-estimators). Suppose the following conditions hold:
1. (0o) = 0, where Og lies in the interior of
2. vn M (ō.) 2 0 and 00n - 0oll 2/4 for the sequence of estimators (ū,) C O
3. Vn(V,- - ()))) 3 Z, where Z is a tight random variable.
4. The following smoothness condition is satisfied:
(VT(V.(O.) - V((r)) vn (0,(0)) (((6)) & 0.
(64)
1 Vn 110n - -boll
Additionally assume that 0 + (0) is Fréchet differentiable' at Og with derivative oo and that Ve, is continuously invertible Then
Vinno, @.. -b0)+ + Vn(V, (B)))l &
and therefore
Va (0) $
J 00 Z.
'A random variable 7 is tight if, for any >> 0, there exists compact set K C IR such that P(Z $ K) <.
Fréccett differentiability: A map Φ D > T, Fréchet differentiable at if there exists a continuous, linear map % : - such that (o6(8 + hn,) (00) (h,))))ll 1/z |m + for all sequences (hx) C D with ||,, || -+ 0 and 0 + h2 E e for all m 2 1.
"Continuous invertibility: A map A - is continuously invertible if A invertible, and there exists constant
such that
|| [(())) - A(l2)lla 2 c| 01 02 for all 0, 02 E e.
30

---

Optimal Sampling for Reward Modeling
F. Experimental Details
We implement our code based on the open-sourced OpenRLHF framework Hu et al. (2024). We will open-source our code in the camera-ready version.
We use both the helpful and the harmless (HH) sets from HH-RLHF (Bai al., 2022) without additional data selection. We adopt the chat template from the Skywork-Rewer wewark-Reward--BB model (Liu et al., 2024a) to align with the reward template. This reward model, fine-tuned from Llama-3.1188 is used to simulate human preference labeling and matches our network trained for alignment.
For SFT, we apply full palaamreettrr tuning with Adam for one epoch, using a cosine learning rate schedule, a 3% warmup phase, a learning rate of 5 x 10-7, and a batch size of 256. These hyperparameters are adopted from Hu et al. (2024).
For all the DPO training in both iterative and online settings, we use full-parameter tuning with Adam but with two epochs. The learning rate, warmup schedules, and batch size are all the same.
During generation, we limit the maximum number of new tokens to 896 and employ top-p decoding with p = 0.95 for all experiments. For Online DPO, we use a sampling temperature of 1.0, following Guo et al. (2024), while in Iterative DPO, we set the temperature to 0.7 to account for the off off-policy nature of the data, following Dong et al. (2024); Shi et al. (2024).
Prompts are truncated to maximum length of 512 tokens (truncated from the left if the length exceeds this limit) for SFT, DPO, and generation tasks. For SFT data, the maximum length is further restricted to 1024 tokens. When the combined length of the response and the (truncated) prompt exceeds 1024 tokens, the response is truncated from the right. These truncation practices align with the standard methodology described by Rafailov et al. (2023). In contrast, for DPO, responses are not further truncated, as we are already limiting the maximum tokens generated during the generation process.
When reproducing the Hybrid Sampling baseline (Exploration Preference Optimization, XPO) from Xie et al. (2024), we use a = 5 10 as suggested in the paper.
We do not include a comparison with Shi al. (2024) and Liu et al. (2024c) in our experiments. While Shi et al. (2024) employs a sampling method similar to ours, their approach requires significantly more hyperparameters tune, whereas our method involves no hyperparameter tuning. On the other hand, Liu et al. (2024c) relies training ensemble of 20 reward models to approximate the posterior. Their sampling method requires solving the argmax of these rewards, which is computationally intractable. As a workaround, they generate 20 samples and select the best one using best-of-N with N = 20. This approach demands at least six times the computational resources compared to our method.
F.1. Additional Results
We present the full results for Online DPO with the overfitted initial policy, including a scatter plot in Figure 6 and a summary of the objective values in Table
We observe that Vanilla Sampling rapidly increases its KL divergence from the reference model while its reward improvement diminishes over time. In contrast, PILAF undergoes an early phase of training with fluctuating KL values but ultimately achieves a policy with higher reward and substantially lower KL divergence. We hypothesize that PILAF's interpolation- based exploration enables it to escape the suboptimal region of the loss landscape where Vanilla Sampling remains trapped.
Conversely, Hybrid Sampling. despite its explicit exploration design, remains biased by the policy model and continues to exhibit high KL values. While KL. divergence decreases over training, the reward improvement remains limited. Meanwhile, Best-of-N Sampling introduces an implicit exploration mechanism through internal DPO, which selects the best and worst responses, leading to wider coverage than Vanilla Sampling. However, despite achieving KL divergence similar PILAF, results in lower reward. These findings highlight the superiority of PILAF sampling, demonstrating its effectiveness in robustly optimizing an overfitted policy.
G. Extension to Proximal Policy Optimization (PPO)
In this section, we briefly explore how the core principles of our PILAF sampling approach can be extended to PPO-based RLHF methods.
31

---

in Optimal Sampling for Reward Modeling
Online DPO with an Overfitted Initial Model
Table 4. Results of Online DPO with an overfitted initial policy. We report the average reward, KL divergence from the reference model, and objective J on the testset.
x
x
Vanilla Hybrid
METHOD
REwARD (t)
KL (4)
(t)
Rewarr
Vanilla
-3.95
39.85
-7.93
Best-of-N
Best-of-N
-4.49
27.90
-7.28
PILAF
Hybrid
-6.00
18.20
-7.82
PILAF (OURS)
-3.54
26.45
-6.19
(L( II mar)
Figure 6. Online DPO with an overfitted initial policy. Full results of Figure Each dot represents an evaluation performed every 50 training steps. Color saturation indicates the training step, with darker colors representing later steps.
Integrating Response Sampling in InstructGPT: The PPO-based RLHF pipeline used in InstructGPT (Ouyang et al., 2022) consists of three key steps:
ii) Supervised Fine-Tuning (SFT) that produces the reference model Tzef-
(ii) Reward Modeling (RM) by solving the optimization problem (2), yielding estimated reward function Tø-
(iii) Reinforcement Learning Fine-Tumine where the policy nT is optimized against the reward model re using the Proximal Policy Optimization (PPO) algorithm, following the optimization scheme (4).
The key distinction between the PPO and DPO approaches lies in how the reward model ra is explliiteedeeplicittyy explicitly in PPO and implicitly in DPO. In response sampling for data collection, it is crucial to consider the iterative nature of the InstruciGPT pipeline. During each iteration, additional human-labeled data is collected for reward modeling (step (ii)), and steps (ii) and (iii) are repeatedly applied to refine the model. Our proposed PILAF algorithm naturally integrates into this pipeline by improving the data collection process in step (ii), thereby enhancing reward model training and, in turn, policy optimization.
Extending response sampling methods, PILAF and T-PILAF, the PPO setup with explicit is both natural and straightforward.
Extensions of T-PILAF and PILAF:
Within the theoretical framework of T-PILAF, as introduced Section 3, the only required modification replacing Ty with the language model πT and redefining the interpolated and extrapolated policies, Λ5 and πz following the same formulation as in equations (9a) and (9h). Specifically, we define
x))
n&e y
x)) exp {ro(z,
(65a)
Z
(x)
x))
nse x) exp { (2(2,
(65b)
(x)
where rg is now explicitly produced by reward network, rather than being implicitly derived from ne. equation (5).
To extend empirical PILAF algorithm, described in Section 5, propose applying the same interpolation and extrapolation techniques directly to the logits of the language models πτ. and Tea.. In particular, we take
x ( xr 11:2-) softmax ((1 + 3) hs 3 hrot (It v1)-
n. (: x, V14-1) softmax {(1 - 3) he + 3 hret ) (a, (1,.yo-))).
32

---

Optimal Sampling for Reward Modeling
where hi and het represent the logits of the language models T and T4ef respectively.
Adaption of Theoretical Analysis: Our theoretical analyses can be extended to the PPO framework, assuming that the
optimization process (4) in step (iii) of InstructGPT is solved exactly. In this case, the policy satisfies Th = Tra. where
1 Za(x)
The (x) exp { re re 8))
Tre
Under this assumption, the output language model TT is implicitly function of the parameter 0. Building on this, we can adapt our optimization and statistical analyses as follows:
Optimization Consideration: Using the same argument as in Theorem 4.1, we can prove that
V ((0) = C² Vo J(Te) + T2,
where C" > O is a universal constant, and 72 represents a second-order approximation error.
In other words, if the policy optimization step is sufficiently accurate for the reward model ra, then performing gradient descent on the MLE loss with respect to 0 is equivalent to applying gradient ascent on the oracle objective J, following the steepest direction in the parameter space of 0.
Statistical Consideration:
Even with the new parameterization, the asymptotic distribution of 9 from Theorem 4.2 remains unchanged. Moreover, the gradient and Hessian of J with respect to 0 retain the same form in Theorem 4.1. As result, the statistical analysis extends naturally to PPO, allowing us to conclude that PILAF also maintains structure-invarian statistical efficiency for PPO methods.
33


Reviews:
[{"review_id": "Uxm9TUxAef", "reviewer": "Reviewer_qMsM", "summary": "This paper introduces a sampling strategy for collecting human preference data in RLHF (specifically, DPO) setting. It aims to align preference-based reward modeling with the true (oracle) objective by interpolating between the current and reference policies during response generation. Theoretical analysis shows that the proposed aligns gradients of the true (oracle) objective in the first order, making training more consistent and efficient. The authors validate this in both iterative and online DPO settings, demonstrating the effectives regard the DPO loss.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful comments, which have helped improve the presentation of the misalignment problem. We are also glad that the reviewer enjoyed our theoretical analysis and empirical validation.\n\n1. > uniform sampling - misaligned gradients\n\nWe thank the reviewer for raising this point — a clear, formal statement helps improve our presentation. The discrepancy arises from the difference in gradient formulations between Eqns 14 and 15a.\nTo make notations concise, we introduce the following shorthands: $\\Delta r^* := r^*(x, y^a) - r^*(x, y^b)$, $\\Delta r_{\\theta} := r_{\\theta}(x, y^a) - r_{\\theta}(x, y^b)$ and $g := \\nabla_{\\theta} r_{\\theta}(x, y^a) - \\nabla_{\\theta} r_{\\theta}(x, y^b)$.\n\n**Lemma C.2.** \n$\\nabla_{\\theta} J(\\pi_{\\theta}) = \\frac{1}{2 \\beta} E_{y^a, y^b \\sim \\pi_{\\theta}(\\cdot | x)} [ ( \\Delta r^* - \\Delta r_{\\theta} ) g]$.\n\n**(Corollary of) Lemma C.3.** For vanilla response sampling scheme, $\\nabla_{\\theta} L(\\theta) = -E_{y^a, y^b \\sim \\pi_{\\theta}(\\cdot | x)} [(\\sigma(\\Delta r^*)-\\sigma(\\Delta r_{\\theta}))g]$.\n\nThese two gradients share a similar structure. The key difference is $\\Delta r^* - \\Delta r_{\\theta}$ for $\\nabla_{\\theta} J(\\pi_\\theta)$ and $\\sigma(\\Delta r^*) - \\sigma(\\Delta r_{\\theta})$ for $\\nabla_{\\theta} L(\\theta)$.\nTo correct for this mismatch, T-PILAF adjusts the response sampling distribution: It reweights the pairwise response sampling so that the density ratio between the vanilla scheme and T-PILAF approximates the derivative $\\sigma'(\\Delta r_\\theta)$. This bridges the gap between the non-linear sigmoid differences and the linear reward differences, leading to better gradient alignment during training.\n\n2. > Q1: Section 2.3\n\nLet us clarify the root cause of the misalignment. The true objective for both PPO and DPO is to optimize the expected return under the true reward function $r^*$ in (6). In PPO, a reward model $r_\\theta$ is trained from human preferences and then used to guide policy optimization. In DPO, $r_\\theta$ is learned via $r_\\theta(x,y) = \\beta \\log\\frac{\\pi_\\theta(y | x)}{\\pi_{\\mathrm{ref}}(y | x)}$. In both frameworks, the final policy’s performance depends on how well $r_\\theta$ approximates $r^*$, as you noted.\n\nPrior work assumes that on‑policy samples from $\\pi_\\theta$ suffice for improving both the policy and its reward model. We show this is incorrect: gradients from standard on‑policy sampling do not align with the true policy gradient under $r^*$, limiting  $r_\\theta$’s ability to close the approximation gap. As Reviewer PDrJ perfectly summarizes: \"it may not generalize well to reflect true preferences because the sampled comparisons do not represent the broader preference landscape.\" This applies to both PPO and DPO; we present the result in the DPO setting for clarity.\n\nIn contrast, our T‑PILAF sampling scheme is designed to align the empirical gradient with the true policy gradient under $r^*$. This ensures each update of $r_\\theta$ approximates $r^*$ in the optimal first-order direction. Our statistical results also shows T-PILAF minimizes variance. All of our results focus on comparing between the empirical objective and the true objective in (6), showing precisely how vanilla sampling produces misaligned gradients when updating the reward model.\n\n3. > How to turn $y^a$ and $y^b$ into $y^w$ and $y^l$.\n\nBefore Eqn (1), we noted that $y^w$ and $y^l$ were human-annotated as the preferred and unpreferred responses, respectively. Eqn. (1) introduced the commonly used Bradley-Terry (BT) model, and we stated explicitly that the BT assumption was adopted throughout this paper.\n\n4. > Definition of $\\pi^*$ in eqn 9\n\nThe ground-truth reward function in the BT model (Eqn 1) is denoted by $r^*$. Then, the notation $\\pi^*$ refers to the optimal policy that maximizes the value function $J(\\pi)$, as in Eqn (6):\n\\begin{align*}\n \\pi^*(y | x) = \\frac{1}{Z(x)} \\pi_{ref}(y | x) \\exp(\\frac{1}{\\beta} r^*(x,y)).\n\\end{align*}where $Z(x)$ is a partition function that ensures $\\pi^*(\\cdot | x)$ sums to 1.\n\n5. > line 294 and 295, proofs.\n\nFollowing the suggestion, we add the derivation:\n\n**Proof:**\n\nStarting from\n$\\pi_\\theta^+(y_t | x, y_{1:t-1}) = \\frac{1}{Z(x, y_{1:t-1})} \\pi_\\theta(y_t | x, y_{1:t-1}) \\big(\\frac{\\pi_\\theta(y_t | x, y_{1:t-1})}{\\pi_{ref}(y_t | x, y_{1:t-1})}\\big)^\\beta$, \n\nwe rewrite it as:\n$\\pi_\\theta^+(y_t | x, y_{1:t-1}) \\propto \\exp\\big((1+\\beta)\\log \\pi_\\theta(y_t | x, y_{1:t-1}) - \\beta \\log \\pi_{ref}(y_t | x, y_{1:t-1})\\big)$.\n\nDefine the logits:\n$h_\\theta(y_t | x, y_{1:t-1}) = \\log \\pi_\\theta(y_t | x, y_{1:t-1})$,\n$h_{ref}(y_t | x, y_{1:t-1}) = \\log \\pi_{ref}(y_t | x, y_{1:t-1})$.\n\nThen\n$\\pi_\\theta^+(y_t | x, y_{1:t-1}) \\propto \\exp((1+\\beta) h_\\theta(y_t | x, y_{1:t-1}) - \\beta h_{ref}(y_t | x, y_{1:t-1})).$\n\nNormalizing over all $y_t$ leads to the softmax form: \n$\\pi_{\\theta,\\beta}(\\cdot | x, y_{1:t-1}) = \\mathrm{softmax}((1+\\beta) h_\\theta - \\beta h_{ref})$.\n\nPlease let us know if our responses address your concerns.", "claims_and_evidence": "In general, the motivation of this paper is clear and the claims and analysis make sense.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes I have checked Theorem 4.1 and 4.2, which look fine to me.", "experimental_designs_or_analyses": "Yes, although the experiments on DPO is not comprehensive compared to other papers, it supports the claim that the algorithm better aligns with the DPO loss regarding the true objective.", "relation_to_broader_scientific_literature": "Related to general alignment of LLMs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength is that this problem study a important aspect of RLHF, that is how to better sample data in the online or iterative DPO setting.\n\nOne of the weaknesses is that, it is mentioned that \"We show that uniform sampling from the current policy, as is common, leads to misaligned gradients of the two objectives\". However, in the theoretical development, I only saw theorem 4.1 showing the result for alignment gradients, but there is no formal results on the claim that uniform sampling results in misaligned gradients. It is not hard to show a theoretical results like this.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have the following questions, and am willing to increase my score if properly addressed:\n\n1. I have a question regarding section 2.3. It is mentioned that the true goal for DPO should be (6). However, the common DPO setting is not very different from (6). It basically replaces $r*$ in (6) by the learned $r_{\\theta}$. In other words, the common DPO setting approximates the goal (6) with some errors. I think there should be some results or statement on the analysis between the previous objective and the current objective (6).\n\n2. It was not described how to turn $y^a$ and $y^b$ into $y^w$ and $y^l$. This is problematic when looking at eqn (3).\n\n3. how is the optimal policy $\\pi^*$ defined in equation 9?\n\n4. in line 294 and 295, I think it is bettwe to provide a formal statement with proofs or derivation in the appendix.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CecS58VQwR", "reviewer": "Reviewer_PDrJ", "summary": "This paper investigates strategies to leverage interpolated response sampling for improving human preference data collection and reward modeling in RLHF. The authors propose a Policy-Interpolated Learning for Aligned Feedback PILAF method that generates response pairs by interpolating between a reference policy and the current policy to better align reward model training with the true preference objective; then, they develop a practical version of PILAF and evaluate it in iterative and online DPO training setups. They find significant gains in reward model performance, alignment quality, and sample efficiency compared to Vanilla sampling and Best-of-N sampling methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our work, especially its development from theory to algorithm design. We are sincerely pleased that the reviewer aacknowledged the misalignment problem we identified, found the combination of mathematical proofs and experiments to provide a reasonable justification, and considered our experimental results to be extensive.\n\n1. > The authors may consider adding an ablation study to analyze whether the gains from PILAF are truly due to its interpolated sampling mechanism and gradient alignment rather than other confounding factors. For example, comparing PILAF with versions that remove or vary the interpolation component would help validate this core contribution.\n\nThank you for the suggestion. Following it, we added two ablation studies to isolate the contributions of PILAF’s interpolation and extrapolation components. Each component was replaced individually with vanilla sampling, yielding two baselines: one with $({\\vec y}^a, {\\vec y}^b)=(\\pi_\\theta^+, \\pi_\\theta)$ (ablation of the interpolation component) and one with $({\\vec y}^a, {\\vec y}^b)=(\\pi_\\theta, \\pi_\\theta^-)$ (ablation of the extrapolation component). We denote these ablation variants as PILAF-extrapolate and PILAF-interpolate, where one response is obtained via vanilla sampling and the other via extrapolation or interpolation, respectively. Due to time constraints, we completed these ablations only for the online setup; we plan to extend this to the iterative setting in the camera-ready version.\n\nWe include the figures at [anonymous URL](https://drive.google.com/file/d/11wfcQT07dxy0IpXuIcdRUrtRBoPPuwXD). Our theory suggests that the two sampling responses should come from different distributions in order to yield a controlled difference that the model can effectively learn from. Both ablation variants introduce such differences and outperform vanilla sampling. However, the variant with only interpolation (combined with vanilla sampling for the other response) performs much worse than full PILAF, highlighting the importance of the extrapolation response. The PILAF-extrapolate variant achieves slightly worse final results, and its convergence is much slower (each dot in our figure represents one evaluation after 50 steps). Overall, these ablation results confirm our theoretical prediction that the full PILAF algorithm is the best performing approach.\n\nWe thank the reviewer again for their time and constructive feedback.", "claims_and_evidence": "I think the mathematical proof and experiments together provide a reasonable justification for the effectiveness of the proposed method. Although more analysis could strengthen the connection, the current results are generally convincing.", "methods_and_evaluation_criteria": "The idea of interpolated response sampling for improving reward modeling in RLHF makes sense for the problem. Most existing work focuses on sampling responses directly from the current policy or using simple heuristics, but these approaches cannot effectively align reward model learning with the true human preference objective. This could limit the efficiency and quality of preference data. This paper introduces a method that interpolates between a reference policy and the current policy to generate more informative and aligned comparisons, as well as significantly improves reward model quality and sample efficiency.\n\nThe workflow is well-structured, as it builds on theoretical insights T-PILAF and adapts them into a practical algorithm PILAF to make the method applicable to real-world RLHF pipelines. This approach balances exploration and exploitation during response sampling and further enhances the alignment between reward model training and human preferences.", "theoretical_claims": "The authors provide two main parts of theoretical claims.\n\n (1) The authors formalize the oracle objective that an ideal reward model should be optimized to reflect true human preferences. They analyze how standard response sampling strategies, e.g., sampling only from the current policy, cause a gradient misalignment between the reward model's learning objective and the oracle objective. This misalignment means that even if a reward model fits the data it sees, it may not generalize well to reflect true preferences because the sampled comparisons do not represent the broader preference landscape.\n\n(2) The authors propose T-PILAF. This framework generates response pairs by interpolating between a reference policy and the current policy. They prove that this interpolation mechanism aligns the reward model's gradient with the oracle gradient, thus correcting the bias from single-policy sampling. The proof shows how interpolated sampling balances exploration and exploitation. In this way, the reward model training is more statistically efficient and better aligned with the underlying preference function. Overall these theoretical results motivate the practical design of PILAF.", "experimental_designs_or_analyses": "The experiments are extensive, with detailed analysis of the results. These experiments validate the effectiveness of PILAF in improving reward model quality, sample efficiency, and alignment performance and demonstrate the robustness and scalability of the method across different training setups, iterative and online DPO, and model sizes. The evaluation includes comparisons to strong baselines including Vanilla, Best-of-N, and Hybrid sampling, and covers both quantitative metrics, reward model performance, KL divergence, and training dynamics.", "relation_to_broader_scientific_literature": "This paper is related to preference-based RL, reward modeling in RLHF, and sampling-based data efficiency methods. The main differences are that (1) it introduces interpolated sampling to address gradient misalignment (against standard preference data collection), (2) it provides a formal analysis of gradient alignment (against heuristic sampling approaches), and (3) it focuses on optimizing sample efficiency through theory-grounded methods rather than relying solely on empirical strategies.", "essential_references_not_discussed": "Not found.", "other_strengths_and_weaknesses": "Thank you for taking the time to write and submit this work. The key strengths that I have observed are as follows:\n1. I really appreciated the effort that the authors put into the related works section. They clearly did their research into the relevant domains and introduced me to new papers as well.\n2. Their preliminary and motivation works section was very clear. I especially appreciated how they took the time to lay out the theoretical formulation and explain the connection between reward modeling gradients and preference sampling in detail.\n3. Figure 1 is very clear. it illustrates the core idea of this method.\n\nOverall, the authors bring up an interesting problem of aligning reward model training with true human preference gradients. In reward modeling, this would be an interesting setting to see how we can optimize preference data collection strategies to improve alignment given the constraints of human annotation cost and model sample efficiency.", "other_comments_or_suggestions": "The authors may consider adding an ablation study to analyze whether the gains from PILAF are truly due to its interpolated sampling mechanism and gradient alignment rather than other confounding factors. For example, comparing PILAF with versions that remove or vary the interpolation component would help validate this core contribution.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EwowyUvxmb", "reviewer": "Reviewer_mZPC", "summary": "The paper \"PILAF: Optimal Human Preference Sampling for Reward Modeling\" introduces Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel sampling strategy designed to improve reinforcement learning from human feedback (RLHF), particularly in reward modeling for aligning large language models (LLMs) with human values.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time in providing the review.\n\n1. > HH-RLHF not enough. More benchmarks.\n\nPlease allow us to put our work into more context. Our contributions extend beyond simply empirically validating a new sampling algorithm. Rather, first, we identify and rigorously characterize a previously overlooked misalignment problem in RLHF. Second, we develop a general theoretical framework—free of restrictive assumptions—that directly informs a principled solution for addressing the problem with guarantees. Third, we translate these insights into T‑PILAF, a practical algorithm which we validate at scale using 8B LLMs and the HH‑RLHF dataset. Together, we believe the novel theoretical results and large‑scale experiments convincingly demonstrate (1) the existence of misalignment in standard RLHF, (2) how it can be addressed in a first‑principles manner, and (3) the effectiveness of our method in practice. For further discussion of our contributions, please refer to response #3 to Reviewer kgC8.\n\nWe believe the strength of our proposed algorithm lies in its theoretical grounding, provably solving the alignment problem that we expose. As such, we feel that validation on a prominent benchmark, HH-RLHF, with a large LLM summoning a significant amount of compute at our disposal, is convincing evidence.\n\n2. > The approach is online iterative DPO, but they simply compare with the vanilla DPO, which is unfair, because online DPO outperforms the vanilla one. The authors are suggested to compare their algorithm with more online methods, such as the general online DPO [1,2,3]\n\nFollowing the definitions in [1,3], we distinguish between iterative and online data collection as follows: iterative sampling generates all preference data from the policy network at the beginning of each iteration, whereas online sampling produces preference data continuously, with each new batch drawn from the current policy network. \n\nImportantly, we are not comparing vanilla DPO against these modes; instead, we hold the underlying RLHF iterative/online framework fixed and vary only the sampling strategy. Concretely, in both the iterative and online setups, every method collects preference data from the policy network—iteratively in the iterative setup (Sections 6.1) and batch‑by‑batch in the online setup (Section 6.2)—with the sole difference being how those samples are generated. This distinction is detailed in the implementation paragraphs and via different values of $n_t$ in Algorithm 1. The term *Vanilla* only denotes the sampling method as vanilla, as shown in Table 1. Thus, we are ensuring a fair comparison exactly as the reviewer suggested.\n\n3. > Besides, the works [1,2] are actually online DPO instead of simply iterative DPO since they collect samples generated from the trained policy and get them labeled by a preference oracle.\n\nWe adopt the definitions of “online” and “iterative” sampling exactly as presented in the referenced papers. In particular, collecting samples from a fully trained policy at each iteration is referred to as iterative DPO in [1, 2].\n\n4. > The theoretical analysis is a little confusing and it is hard to get the intuitions why they use such sampling strategy \\pi- and \\pi+. The authors are suggested to clarify the insights more clearly instead of just listing theorems and equations that seems distinct from the algorithms. Now, it's hard to find connections between the theorem and experiments.\n\nTo provide an intuition for the misalignment problem, note that DPO implicitly defines the reward as $r_\\theta(x,\\vec y) = \\beta \\cdot \\log\\left(\\frac{\\pi_\\theta(\\vec y \\mid x)}{\\pi_{\\mathrm{ref}}(\\vec y \\mid x)}\\right)$ which is trained via maximum‑likelihood estimation. When training with preference data generated by $\\pi_\\theta$, the optimization is biased. Reviewer PDrJ also summarizes this aptly as \"it may not generalize well to reflect true preferences because the sampled comparisons do not represent the broader preference landscape.\" Nonetheless, this intuition does not prescribe a concrete sampling strategy for correcting the misalignment.\n\nInstead, we use theoretical analysis to reveal the root cause and derive a principled solution. By comparing Equations 14 and 15a, we demonstrate how the gradient produced by standard sampling diverges from the true alignment gradient. Following Reviewer qMsM’s suggestion, we have added a lemma in the main text that explicitly shows this discrepancy. These insights directly inform T‑PILAF, a sampling algorithm that leverages $\\pi^-$ and $\\pi^+$, to realign the empirical gradient with its theoretical counterpart (as explained in line 245 left). PILAF is then the practical instantiation of T‑PILAF, and our experiments validate its effectiveness. This seamless integration of theory and practice is precisely what Reviewer PDrJ praised: “these theoretical results motivate the practical design of PILAF.”\n\nPlease let us know if our responses address your concerns.", "claims_and_evidence": "See weaknesses.", "methods_and_evaluation_criteria": "1. The authors simply use HH-RLHF dataset, which is not enough to validate their conclusion. It would be better to execute on more benchmarks.\n\n2. The approach is online iterative DPO, but they simply compare with the vanilla DPO, which is unfair, because online DPO outperforms the vanilla one. The authors are suggested to compare their algorithm with more online methods, such as the general online DPO [1,2,3]\n\nBesides, the works [1,2] are actually online DPO instead of simply iterative DPO since they collect samples generated from the trained policy and get them labeled by a preference oracle.\n\n\n[1] Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Con- ference on Machine Learning, 2024.\n[2] Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf, 2024.\n[3] Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024.", "theoretical_claims": "The theoretical analysis is a little confusing and it is hard to get the intuitions why they use such sampling strategy \\pi- and \\pi+. The authors are suggested to clarify the insights more clearly instead of just listing theorems and equations that seems distinct from the algorithms. Now, it's hard to find connections between the theorem and experiments.", "experimental_designs_or_analyses": "See Methods And Evaluation Criteria.", "relation_to_broader_scientific_literature": "They provide realistic ways for online exploration for RLHF.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "See comments above.", "questions_for_authors": "Please answer the questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xfOMjJXEmj", "reviewer": "Reviewer_kgC8", "summary": "This paper introduces PILAF (Policy-Interpolated Learning for Aligned Feedback), a novel sampling strategy for iterative/online DPO. The authors show that with this new sampling algorithm, the gradient of the loss function matches the KL-regularized objective function, and they further provide asymptotic analysis of DPO with PILAF. For the experiments, they implement PILAF with some empirical approximation and evaluate it both with iterative and online DPO, where it consistently outperforms baseline methods (XPO and Best-of-N).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, particularly for recognizing the strength of both the theoretical and experimental parts, and for checking the proofs of all theoretical results.\n\n1. > \"Value-Incentivized Preference Optimization\". The authors should also add this work into discussion.\n\nWe thank the reviewer for bringing this up. Please note that we did cite Cen et al. around line 621. As we discussed there, their method does not modify the sampling scheme; rather, it adds a regularization term to the DPO objective to encourage departure from the calibration samples. Therefore, we keep this reference in the appendix and focus on those papers - more directly comparable to our work -  that modify the sampling scheme in the main body of our paper. We can relocate this distinction in related works to the main body of the paper if you would consider that it adds clarity to our exposition.\n\n2. > Add VPO to the baselines.\n\nAt the reviewer’s suggestion, we have added VPO as a baseline in the online setting. Due to the need to generate responses in all experiments, completing the full VPO comparison is infeasible within the rebuttal period. Therefore, we are adding this baseline for the online setting and plan to complete the comparison for the iterative setting in the camera-ready version. We report the figure at [anonymous URL](https://drive.google.com/file/d/11wfcQT07dxy0IpXuIcdRUrtRBoPPuwXD). The results show that PILAF outperforms VPO, even after a small hyperparameter search for VPO (details can be found in the provided link). \n\n3. > However, this contribution is limited because it is specifically designed for DPO, whose performance is not SOTA on most benchmarks. The authors can try whether this sampling strategy is also useful in other algorithms like IPO, KPO and PPO. \n\nWe would like to clarify that our motivation—and the scope of our theoretical and experimental contributions—is not limited to DPO, but rather to reveal and address a fundamental misalignment that pervades the two‑phase RLHF framework. A prevailing assumption in recent RLHF work is that on‑policy data generated during training constitutes “good” alignment data. However, we demonstrate that—even when using on‑policy samples—the alignment process remains suboptimal.\n\nSpecifically, RLHF consists of two sequential phases: first, preference data are collected and used to extract human values and train a reward model— via maximum‑likelihood estimation either explicitly (as in PPO) or implicitly (as in DPO, IPO, and KPO)—and second, the learned reward model guides policy optimization. Our main theorem shows that this two‑phase design and the MLE optimization of the reward creates the misalignment between the update gradient and policy gradient maximizing the true human values. Although this issue affects both PPO and DPO equally, we present our theorem in the context of DPO for clarity. We articulate this motivation in the Introduction and provide a detailed discussion of its implications for PPO in Appendix G. By the same reasoning, our theoretical principle extends to IPO; however, because IPO modifies the optimization objective (and thus the gradient), its optimal sampling scheme differs from that derived for DPO. For clarity and focus, we leave a detailed analysis of IPO and its corresponding optimal sampling scheme to future work.\n\nThe contribution of our work is multifaceted. First, we rigorously identify and characterize the misalignment problem inherent to the two‑phase RLHF framework—an issue largely overlooked by prior work. Second, we develop a comprehensive, assumption‑light theoretical analysis that directly yields an optimal sampling strategy; unlike Cen et al. (VPO), our approach makes no restrictive assumptions (e.g., reward‑model linearity) and therefore holds under very general conditions. This requires substantial theoretical innovation. Third, we empirically validate PILAF's effectiveness on modern large language models, demonstrating significant and consistent improvements over existing baselines. Consequently, our contribution extends far beyond proposing a new algorithm: it offers a universal, theory‑driven perspective on addressing misalignment for RLHF.\n\nFinally, we respectfully disagree that DPO is not state‑of‑the‑art. Recent studies confirm that the principal driver of performance in RLHF is whether an algorithm incorporates online data generation. When implemented online, DPO matches PPO’s performance as reported by [Noukhovitch et al. 2025, Tang et al. 2024]. Accordingly, we evaluate our method in both iterative and online settings.\n\nNoukhovitch, Michael, et al. \"Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models.\" ICLR 2025.\n\nTang, Yunhao, et al. \"Understanding the performance gap between online and offline alignment algorithms.\" arXiv preprint arXiv:2405.08448 (2024).\n\nPlease let us know if our responses address your concerns.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I checked Theorem 4.1~4.3. They look good to me.", "experimental_designs_or_analyses": "The experiments look good to me. However, I suggest that the authors add VPO in \"Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF\" by Cen et al. to the baselines because it can also be considered as a variant of DPO.", "relation_to_broader_scientific_literature": "The key contribution is the proposed new sampling algorithm for DPO, which has better empirical performance than XPO and best-of-N on the HH-RLHF dataset. However, this contribution is limited because it is specifically designed for DPO, whose performance is not SOTA on most benchmarks. The authors can try whether this sampling strategy is also useful in other algorithms like IPO, KPO and PPO. If this sampling strategy is shown to be useful universally, this work would have a greater impact.", "essential_references_not_discussed": "\"Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF\" by Cen et al. also proposes a variant of DPO, which has similar structures to XPO. The authors should also add this work into discussion.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces PILAF (Policy-Interpolated Learning for Aligned Feedback), a sampling strategy for collecting human preference data in DPO. The idea is to interpolate between the current and reference policies during response generation to better align reward model training with the true preference objective. The reviewers recognized the novel sampling strategy, the contribution of theoretical analysis, and the positive experimental results in iterative and online DPO settings. The reviewers also raised concerns and suggestions about additional experiments and comparisons to literature and baselines, which are mostly addressed in rebuttal. All reviewers recommend accepting the paper during the discussion phase, and the AC agrees with the recommendation.
Decision: Accept (poster)