Policy-labeled Preference Learning: Is Preference Enough for RLHF?
Authors: Taehyun Cho
Seokhun Ju
Seungyub Han
Dohyeong Kim
Kyungjae Lee
Jungwoo Lee
Venue: ICML 2025 spotlightposter (2025)
Keywords: Reinforcement Learning from Human Feedback
Direct Preference Optimization
Regret Minimization
Primary area: reinforcement_learning->deep_rl

Abstract: To design reward that align with human goals, Reinforcement Learning from Human Feedback (RLHF) has emerged as a prominent technique for learning reward functions from human preferences and optimizing models using reinforcement learning algorithms. 
However, existing RLHF methods often misinterpret trajectories as being generated by an optimal policy, causing inaccurate likelihood estimation and suboptimal learning. To address this, we propose Policy-labeled Preference Learning (PPL) within the Direct Preference Optimization (DPO) framework, which resolves these likelihood mismatch problems by modeling human preferences with regret, reflecting the efficiency of executed policies. Additionally, we introduce a contrastive KL regularization term derived from regret-based principles to enhance sequential contrastive learning. Experiments in high-dimensional continuous control environments demonstrate PPL's significant improvements in offline RLHF performance and its effectiveness in online settings.

Policy-labeled Preference Learning: Is Preference Enough for RLHF?
Tachyun Cho
Seokhun Ju
Seungyub Han
Dohyeong Kim
Kyungjae Lee² Jungwoo Lee
Abstract
robotic manipulation (Christiano et al., 2017; Hwang et al., 2023), aligning agents with human intentions has become increasingly important.
To design rewards that align with human goals. Reinforcement Learning from Human Feedback (RLHF) has emerged for learning reward functions from human prefer- ences and optimizing policies via reinforcement learning algorithms. However, existing RLHF methods often misinterpret trajectories as being generated by an optimal policy, causing inaccurate likelihood estimation and suboptimal learning. In- spired by Direct Preference Optimization frame- work which directly learns optimal policy with- out explicit reward, we propose policy-labeled preference learning (PPL), to resolve likelihood mismatch issues by modeling human preferences with regret, which reflects behavior policy infor- mation. We also provide a contrastive KL regular- ization, derived from regret-based principles, enhance RLHF in sequential decision making. Ex- periments in high-dimensional continuous control tasks demonstrate PPL's significant improvements in offline RLHF performance and its effective- ness in online settings. For more information, visit our project page: https: / / jush. gi thub. io /P/PL//.
prominent technique
Early RLHF research (Lee et al., 2021; Park et al., 2022)
assumed humans prefer trajectories with higher cumulative rewards, leading to a two-step learning process: (I) training a reward model to align with human preferences and (2) applying a RL algorithm to optimize the policy using the learned reward. Recently, Rafailov et al. (2024b) introduced Direct Preference Optimization (DPO), which bypasses the need for an explicit reward function and directly optimizes the policy based on preferences. This simplification reduces computational complexity and dependency on potentially imperfect reward models, improving training stability. DPO has demonstrated superior performance over reward-based RLHF methods, particularly in fine-tuning LLMs (Bai et al., 2022; Stiennon et al., 2020; Ziegler et al., 2019) and offline RL benchmarks (Hejna & Sadigh, 2024).
While DPO has shown strong performance in LLM fine- tuning and offline RI. benchmarks, its assumptions are largely shaped by the structure of LLM training. Many RLHF studies assume contextual bandits or deterministic Markov decision processes (MDPs), where the next state is determined solely by the previous state (prompt) and ac- tion (response), leading to a simplified transition model. However, in standard RL settings, state transitions involve environmental stochasticity, introducing additional uncer- tainty that complicates both policy optimization and MDP estimation ( (Yang et al., 2022). Since outcomes depend on external stochasticity beyond the agent's 's control, observed transitions may not always accurately provide the agent's s policy performance. Consequently, inferring optimal behav- ior from observed sequences becomes more challenging. For example, an agent executing a suboptimal policy might transition to a preferred state with low probability, whereas an optimal policy could occasionally lead to an undesirable state due environmental randomness. This contrast with the deterministic nature of LLM training, where responses are directly mapped from prompts, highlights key limita- tion when applying DPO to general RL problems.
1. Introduction
Preference-basee reinforcement learning (PbRL), a branch of RLHF, focuses on learning optimal policies directly from human preferences, avoiding the needs for explicit, hand- crafted rewards. Unlike traditional RLHF, which infers nu- merical rewards, PbRL derives reward signals from prefer- ence comparisons between trajectory pairs. As RLHF appli- cations expand into complex domains such as large language models (LLMs) (Achiam et al., 2023; Touvron et al., 2023; Ye et al., 2024; Meng et al., 2024; Xiao et al., 2024) and
'Equal contribution 'Seoul National University. Seoul, South Korea *Korea University, Seoul, South Korea 'HodooAI Labs, Scoul, South Korea. Correspondence to: Kyungjae Lee <kyung- jee_ korca.ac.kr>,, Jungwoo Lee <junglee@ kr..
This challenge becomes more critical offline RL, where learning relies on pre-collected datasets rather than direct interaction with environment. When data comes from di-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada, PMLR 267, 2025. Copyright 2025 by the author(s).

---

Policy-labeled Preference Learning: Is Preference Enough for RLHF?
verse policies but lacks explicit information about the behav- ior policies, distinguishing whether outcome quality stems from policy suboptimality or environmental stochasticity be- comes difficult. Given this ambiguity, a key question arises:
Table I: Comparison for different preference models under PbRL framework.
Direct Preference Likelihood Optimization
Algorithm
Score Function
Matching
Can preference data generated by diverse policies suffi- ciently guide sequential decision- makiing or is additional information required?
(Lee et al., 2021)
PEBBLE
rp(or,a))
x
x
DPO
log (ogngloll//ssetlll
/
(Rafailov et al., 2024b)
DPPO (An et al., 2023)
-Panntt mulla
To address this question, we propose novel RLHF frame- work, Policy-labeled Preference Learning (PPL), which leverages regret-basen based preference modeling while explic- itly labeling the behavior policy. Unlike conventional ap- proaches that rely solely on preferences between trajectory pairs, PPL incorporates policy information directly into the learning process, disentangling the effects of environmental stochasticity and the suboptimality of behavior policies.
CPL (Hejna et al., 2023)
(*((e))t)) (*(()))
/
PPL.
-(VTo(a)) - (o())))))
/
[Ours]
For clarity, we say MaxFal as a-optimal. In addition, soft Q-function Q" (s, a) is defined as the expected cumulative return augmented by an entropy terms, expressed as;
To provide theoretical insights, we define reward equiva- lence class-a a set of reward functions that induce the same optimal policy-and derive a bijective mapping that allows regret to be expressed as a function of the optimal policy. We show that, unlike the partial sum of rewards, regret is uniquely defined with respect to given optimal policy, mak- ing a well-structured metric that mitigates issues related to reward sparsity and enhances the stability of learning. Furthermore, we introduce contrastive KL regularization, which sequentially aligns policies with preferred trajecto- ries while explicitly contrasting them against less preferred ones. Empirically, to consider the fact that real-world of- fline data often consists of rollouts from diverse policies, we construct homogeneous and heterogeneous datasets in the MetaWorld environment and evaluate performance across various offline datasets.
((,))=rss00) r(s, a) E
Σ (r(ss,), att (++))) 150
Analogously, we can derive soft value function (T(s) and soft Bellman equation as follows:
V (*) = Ea 1Q (s, a) - a log (a]s)] a= = + BB'e V (s')
for all state-action pairs (s E S x A. Note that the inter- pretation of the value function is modified by involving the entropy term in the MaxEmtRI, i.e., V (s) + Ex( (s. a)]. For an a-optimal policy ~², Ziebart (2010) derived the re- lationship between the optimal policy and optimal soft Q- function Q²
2. Preliminaries
" (als) = exp
V"
Maximum Entropy Framework. We define the MDP as M (S,A,F, characterized by state space S, action space A, transition kernels IP which represents the prob- ability of the next state s' given the current state 8 E S and action a E A, underlying reward r E ["min, m and discount factor y. For notational simplicity, we denote the expectation over trajectories T = (sp. ag, $1, a1, ".,..)) gener- ated by a policy ı and the transition kernel as Ey--r []]
vx (8) = a log / exp (a-tQ" (s. a)da)
aaEA
2.1. Preference-based Reinforcement Learning
Designing reward function that accurately aligns with human behaviors is inherently challenging. To address this, PbRL focuses on learning the optimal policy directly from human preferences rather than relying on predefined rewards. In this context, we adopt reward-free MDP M within the MaxEnt framework. We define seg- ment < = (so. ao...., Sk; ax) as sequence sampled from dataset D. Specifically, human annotators or Al systems are tasked with comparing pairs of trajectory segments (5+,5-), where ç+ is preferred over < (i.c., ç+ > )))
The MaxEnt framework provides an optimal policy which not only maximizes the expected cumulative return, but also the entropy for each visited state:
MaxEm =
arg max ETAEE
Σ
(r((s aH + o7H (1s))) 120
where (-1s) = EE, T (-1s)] is the entropy of policy ा at state s. Here, a is a temperature hyperparameter that determine the relative importance of entropy and reward.
Score-based Preference Model. Score-based preference model is a natural generalization of RLHF for modeling hu- man preferences through score functions, instead of partial
2

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI?
sum of rewards (Lee et al., 2021). This approach extends the Bradley-Terr model (Bradley & Terry, 1952), where pair- wise comparisons are used to infer relative preferences, by introducing score function that evaluates all observed state- action pairs within a segment. The preference model then assigns probabilities proportional to the sum of these scores, aligning with the Bradley-Terry framework. To implement the preference model using a neural network, the score func- tion is parametrized as Sg, and the model is trained by minimizing the cross-entropy loss between its predictions and the preference labels derived from the dataset D, as follows:
timesteo
Timestep
Figure 1: Visualization of 5000 samples in Jin-Picking-yy environment. While the ground- truth reward (left) is sparse and mainly provided upon task completion, regret (right) is more evenly distributed across all timesteps, making a more informative score function for partial trajectory evaluation.
Ps.lst >5]= Σ Su at) - -5 oo)) 120
be reformulated into a policy-based expression, enabling the optimal policy to be learned directly without relying on reward:
D) (c
. S-D log psy 1st + ]5
where (1:) = 11((1 + +) and each (st a and (s7 at) is the t-th state and action of preferred segment ç+ and less preferred segment ( respectively. For notational simplic- ity, we abbreviate B<c- >~D as EED
(TP: D)
= -alE logo
Σ 120
log Tt (af - The (ar (s
Although it is unclear how humans evaluate their prefer- ences, preference models can be improved to better align with human judgment by refining them based on intuitive ex- amples. If the score function does not align with human pref- erence evaluation, the model may produce counterintuitive outcomes. For example, Knox et al. (2022) demonstrated that using the partial sum of rewards as a score function over- looks a critical issue in sparse reward MDPs: all segments that fail to reach the goal are treated as equally preferable, regardless of their contributions.
However, the standard score-based preference loss is con- vex but not strictly convex, leading to the existence of multiple optimal solutions. Hejna et al. (2023) identified that the shift-invariance property of the loss function (i.e., Ps J+C = P)(a.)) causes aat-of-distribution actions to be overly weighted, deteriorating learning performance. To mitigate this issue, they introduced asymmetric regular- izer X, which reduces the gradient weight on less preferred actions, breaking the inherent symmetry and stabilizing the learning process.
As shown in Figure 1, sparse reward MDPs provide little feedback for the states that do not reach the terminal goal, leading to meaningless comparison of the preferences in the early- and mid-stage segments based solely on return sums. In contrast, regret is more evenly distributed across timesteps, making it more effective score for comparing segment preferences regardless of their position in the trajec- tory. This highlights the importance of modeling preference with the score function that aligns with human intuition. Various approaches to designing such score functions have been proposed, as summarized in Table 1.
3. Policy-labeled Preference Learning
This section introduces the regret-based preference model and its distinctions from prior work, with a focus the issue of likelihood mismatch, where sampled segments are misin- terpreted as optimal, leading to suboptimal learning. To ad- dress this, we propose Policy-labeled Preference Learning (PPL), which employs a regret-based model to accurately estimate segment likelihoods. Finally, we present theoretical results derived from the PPL framework.
Optimal Advantage-based Preference Model. Hejna et al. (2023) proposed Contrastive Preference Learning (CPL), which is based on an optimal advantage-baser pref- erence model (Knox al., 2022), treating as a regret-based preference model. The CPL score is defined as the differ- ence between the value of the action taken and the aver- age value under the optimal policy, (i.e., Ag- (st, a.) == Q=" (s,, a,) - V*" (s,) = a log TT" (a,ls,)) Leveraging the relationship between the optimal advantage and the optimal policy within the MaxEnt framework, their objective can
3.1. Is Preference Enough for RLHF?
Negative Regret vs Optimal Advantage. In prior work, Hejna et al. (2023) utilized the optimal advantage as the score in CPL to introduce regret-based preference model. While they presented these two concepts as equivalent, they differ significantly in their precise definitions and impli- cations. Optimal advantage refers to the relative benefit of taking a specific action a under the optimal policy x" (i.e., Q (8, V* (s)). In contrast, negative regret cap-
3

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
Policy-labeled Preference Learning
Σ
log (a:ll)) - (p-FF(
Px (+(1a)) =*(bs))
Policy-lbblldd
Dataree
Likelihood matched
Contrastive Regularization
Figure 2:
Unlike existing DPO algorithms, PPL aligns segment likelihoods by incorporating behavior policies. It reweights gradients based on closeness to the optimal policy, forming contrastive learning framework.
tures the performance difference between the behavior pol- icy π and the optimal policy n² (i.e., Q" (s, a) - V (s)). The key difference between these concepts lies in whether the behavior policy is incorporated into the score.
higher score (+10) to $2, making it appear more prefer- able than $1 In contrast, the black trajectory, generated by the optimal policy π², assigns a higher score (+20) to si, leading to the opposite preference. These conflicting results can be properly distinguished when policy labels are avail- able, allowing the model to infer the suboptimality of by evaluating preferences separately for each policy.
From a perspective of regret, the optimal advantage disre- gards the source of the trajectories and evaluates the actions taken solely based on Q Consequently, it implicitly treats all trajectories as if they were generated by the optimal pol- icy. This raises an important question: what impact does this assumption treating all behavior polices as optimal - have on the regret-basse learning process?
Now consider the right-side Figure 3, where the same data is used but without policy labels. Since all data is incor- rectly assumed to originate from π², the model observes contradictory outcomes- being preferred in one case and 8i in another another-despite despite assuming single policy. Lacking policy labels, the model misinterprets this discrepancy as environmental stochasticity rather than differences in poli- cies, distorting the learned MDP and leading to incorrect likelihood estimates for trajectories. To mitigate this issue, it is crucial to explicitly track and incorporate the behavior policy и for each segment, ensuring accurate interpretation and proper differentiation feedback. Thus, replacing opti- mal advantage with regret, which reflects the suboptimality of the behavior policy, provides a principled solution.
Estimated MDP
Ground-Truth MDP
without policy label
+20
20
90
+-15
+-1
+10
+10
Figure 3: Illustration of the likelihood mismatch problem. Although the behavior policy differs from the optimal policy π², the learning process incorrectly assumes all data is generated by n². As result, while n prefers this misinterpretation leads to the incorrect conclusion that 82 is preferred, causing suboptimal learning outcomes.
Regret-based Model Requires the Behavior Policy. In essence, regret quantifies how much better we could have done if we had followed the optimal policy instead of the behavior policy. A larger regret indicates that the behavior policy significantly less efficient compared to the optimal policy. We remark that the regret is the difference between the expected return under optimal policy and the achieved return under behavior policy. Based on the conventional definition of regret, we reformulate negative regret in policy-based form within the MaxEnt framework:
Likelihood Mismatch. Likelihood mismatch occurs when outcome differences between two segments, which ac- tually stem from behavior policy differences, are mistakenly attributed to environmental stochasticity. This misinterpre- tation leads to incorrect likelihood assignments. Figure 3 illustrates this issue in an offline setting where offline data from both a suboptimal policy π and an optimal policy n² lacks explicit policy labels. In this scenario, all data is mis- takenly assumed to be generated by the optimal policy n², leading to misinterpretations during learning.
Reg". (Se: V (st)
Q" (5 at) achioved return under
(2)
expected return under x='
(Thm 3.4)
a log "t" (a, (a,ls))
DRL. ( (n||mT"; | St,
To understand how preference labels are assigned in this setting, let us first consider the left-side figure. The red trajectory, generated by the suboptimal policy æ, assigns
likelibood
decrease sequential forward KI.

---

Policy-labeled Preference Learning: Is Preference Enough for RLHF?
In summary, the regret for the preferred segment can be decomposed into two components: First, it increases the likelihood of actions taken in preferred segments, aligning the behavior policy with human preferences. Second, it re- duces the sequential forward KI. divergence, correcting for likelihood mismatch by considering long-term differences between the behavior policy and the optimal policy. Analo- gously, for the less preferred segment, the regret exhibits the opposite tendencies. Based on Equation (3), our objective can be formulated as follows:
Ra is defined the (a, "t "e.""""eeevveeence class of Q" function, denoted by La.
Definition 3.1 indicates that a reward function class R or a -f-ftunttinn class Q" can be partitioned based on the a-optimal policy x². For notational simplicity, we denote the ground truth reward function corresponding to the a-- optimal policy π as r. and the Q" ff-ffuctionn induced by r. as Q7, simplifying the subscript to *.
Lemma 3.2 (Structural Condition for a-optimality). A re- ward function and a soft optimal Q-function where T- (-1s) is a a-optimal have a one-to-one correspondence with state- dependent functionn 3 S + R. defined as follows:
(To;:)
- -Ep log 0
(-
Σ Reg (st (st,at) - Reg", (5T o0)
$20
Ra, (r.(s, a log x (a|s) + B(s) - -Ep 2 (Q" (s, a) = a aog -t (a|s) + +{s}}
where the policy label for the preferred and less preferred segments are denoted as x- and x respectively. The de- tailed derivation of this formulation will be introduced in the next section and Appendix B.1.
for all E and E A.
Lemma 3.2 demonstrates that the (a, x >""-quuvvale class of soft optimal Q-functions can be uniquely expressed as the sum of log-probabillty term, a log x" (als). and a state- dependent function, (s).. This result improves upon the prior lemma of Rafailov et al. (2024b), which established only a surjection from reward functions to optimal policies. By contrast, we ensure a bijection, rigorously defining the equivalence class of reward functions. Furthermore, Lemma 3.2 refines the concept of policy invariance introduced by Ng et al. (1999); Gleave et al. (2020) by specifying that the action-dependent term must be a log x (als) to guarantee "r" is the a-optimal policy.
3.2. Theoretical Analysis
Consider a triplet (T" ments Ç+ and <- are generated by policies t and n-, respectively. The (unknown) optimal policy x" serves as the basis for determining the underlying reward and ensuring consistent preferences. During the learning process, we as- sume that each segment is labeled by its behavior policy. Under this setup, the policy-labeled preference model is expressed as:
where the seg-
p(# n"
= 0
Σ
4" (s - V= (s+))
Lemma 3.3 (Unique Fixed Point of Soft Bellman n-operator). Let x be a-optimal. For a given policy x and Q-function Q7 E Q for any (s) E S x A, define the Bellman n-operator T, Q" - Q" where
120
Ac(
[x=* (st aaa)) Q² (5, a,)
+
Be(a"aat,aa)
T,Q7(3,0) = Q5' (8,a) ~1Ep a (aH HH" (+11)- - 77 (-16))
This expression is decomposed into two components: (i) Ar(x*), ), which depends solely on Q"' (note that V"" (s) Easx 1Q" (s, a)] + aH" (-|s)), and (ii) B, x ;T)) which involves Q** and Q"
+E, (Q= a') - Ex (x (d'')))
Then, T.¹h has a unique fixed point QT.
The main theoretical challenge in performing a direct policy update is expressing the soft optimal Q-function and soft Q- function of given policy π closed-form with respect to the optimal policy x² Before proceeding, we introduce the concept of equivalence classes within the MaxEnt frame- work to analyze the reward structures that make a given policy optimal.
Lemma 3.3 describes an operator that links the soft Q- function of given policy π to the optimal soft Q-function Q identifying Q as its unique fixed point. Notably, this relationship is established without requiring explicit knowl- edge of the reward function r.. From the novel design of the soft Bellman n-operator, we now derive the following important theorem.
Definition 3.1. The set of reward functions where n" is a-optimal is defined as (a, x²) -equivveleee class of reward function, denoted by Ra,x". For every policy ग, the set of fu-ftncto generated by any reward function Ta E
Theorem 3.4 (Policy Deviation Theorem). If a policy n² is a-optimal, then for any policy n.
(s,aa) - Q% a) aDKi(=lla";000)
5

---

Policy-labeled Preference Learning: Is Preference Enough for RLHF?
where the sequential KL divergence defined as
Our setting follows the classical DPO, but with the differ- ence that we manage preference queries by labeling the behavior policy for each trajectory in the dataset. Due to page limitations, see Appendix C for the pseudocode.
(8, = ETTPy 2,0
Σ Lis0
( (1s))
Pseudo-labels. In general RL settings, the behavior policy that generated trajectory is typically known or accessible, making policy labeling relatively inexpensive. However, in offline datasets, the behavior policy is often unknown. To address this, we assign pseudo- assuming each segment was generated by a deterministic policy that executed the observed actions.
Here, P5,a is the distribution of trajectories T (so, 1s0,00.-.. ao.-.., a.....)) generated by policy ι and the tran- sition P, starting at (su, ao) (s)
as an alternative,
Theorem 3.4 establishes that the difference between the soft Q-function of any policy π and the optimal soft Q-function is constant and can be expressed as the sequential forward KL divergence. Intuitively. Dx DxL(=||"": s, a) represents the discounted sum of the forward KI, divergence between π and π over the states visited during a rollout starting from (s, a). This property is particularly valuable, as it quantifies the performance gap using only π and n²"
Contrastive KL Regularization. As previously dis- cussed, the regret decomposed into two components. In particular, the sequential KL divergence plays pivotal role in aligning the learned policy with the preferred policy while diverging from the less preferred policy:
While related results were proposed by Shaikh et al. (2024) and Zeng et al. (2024), their proofs were restricted to contex- tual bandits and token-level MDPs with deterministic transi- tions, respectively. Moreover, their formulation depends on a KL-regularized objective that explicitly incorporates ref- erence policy. In contrast, Theorem 3.4, formulated within the MaxEnt framework, does not require a reference policy to be well-defined, making it more broadly applicable.
Regny (s" ., - Regg ($,,a) 444
+200
log Th Ta (stt)
(10g a
Σ 120
TTp (a, |s
-DKL(T (ipii at) + DKL(T |ipiis .a)
contrastive KL. regularization R(Foi x xt))))
Corollary 3.5. For given (a,7') and a policy n, Reg.. (1)) is uniquely determined regardless of ((s).
We call this term contrastive KL regalarization, which requires performing rollouts for each (st, a,) with respect to π + or ... This regularization term ensures that the learned policy ग.. aligns more closely with the preferred policy n while pushing away from the less preferred policy x =-
Since regret is invariant to transformations of (()), does not require additional variance reduction techniques (Schul- man et al., 2015) to ensure stable learning. For a detailed explanation, refer to Appendix B.2.
In practice, implementing contrastive KL regularization can result in a computational overhead, as it requires multiple rollouts with each state-action pair as the initial point every timestep until the terminal is reached. This approach can also increase memory usage as it requires additional timesteps outside of the sampled segment. To address these technical challenges, we replace the discounted sum with an L-horizon undiscounted sum. We normalize the contrastive KL regularization to balance their scale, and the process further simplified by reusing segments c+.C- as a single rollout of policy x.; , """""""ree,t"iii respectively.
Corollary 3.6. Maximizing the MaxEnt objective with neg- ative regret as the reward is equivalent to minimizing the sequential forward KL divergence between the learned pol- icy and the behavior policy for each preferred state-action pair in the dataset, i.e.,
(EctaDi-Reghh Reg (st,at) a alog Mlggz((at1e (st(allt)))
arg max xx
= arg min (E<t~D(DkL(T** (Ippst aat))) (4)
Theorem 3.6 implies that regret- based RLHF operates by ag- gregating behavior policies from preferred segments, align- ing the learned policy toward preferred actions. Notably, if all preferred segments are assumed be generated by the optimal policy, the formulation reduces to the standard CPL objective, highlighting its connection to prior methods. For more detailed analysis, see Appendix B.3.
RTTT ,,,
1 T,
- log ग (arinlstte)) + log क ( Th aa (841) T (a (4t))
RJ
Here, L corresponds to the step of look-ahead during roll- outs. When 1, = 0, the framework fully reduces to CPL, which does not account rollout for sequential planning, An- other interesting observation is that we assume the seg- ments in the offline dataset were generated by the refer- ence policy (i.e., n π = Ted) the framework recovers
3.3. Practical Algorithm and Implementation Details
In this section, we present PPL, a practical algorithm that leverages the policy label to solve the likelihood mismatch.
6

---

Policy-labeled Preference Learning: Is Preference Enough for RLHF?
Table 2: Success rates of all methods across six tasks on the MetaWorld benchmark on different datasets. Each score is reported with the maximum average performance across four seeds over 200 episode evaluation window.
Bin Picking
Button Press
Door Open
Drawer Open
Plate Slide
Sweep Into
SFT
39.7 + 19.2
71.5 + 3.3
48.0 15.6
56.2 1.8
64,8 + 0.8
70.0 + 6.5
Homogeneous Dense
P-IQL
62.0 4,4
72.3 1.0
47.7 =
- 5.7
70.5 +
65.8 1.3
CPL
22.7 5.5
64.3 1.4
29.0 4.3
54.0 4.3
65.5 1.1
69.8 3.3
PPL
83,5 1 4,4
79.8 4: 4.8
39.3 2.0
69.2 - 5.5
64.7 2.0
72.8 -: 4.8
SFT
33.5 5.4
67.4
31.3
54.9 2.7
6..1 3.7
78.3 2.5
Homogenous Spurse
P-IQL
1.
0.0
58.5 1,
51.4 4.6
76,3 1.6
79.0 +: 2.6
CPL
1.0
1.3
28.5 5.8
50.1
2.8
72.9 6.1
PPL
3.5
2.
49.3 - 6.5
68.5
64.0
3.5
SFT
18.5 23.8
63.7 12.2
26.0 12.5
3.70
6
53.0
Heterogeneous Dense
P-IQL
5.3
32.0 4.9
4.8
67.0
59.3
CPL
1.2
49.7 3.0
17.3 2.5
26.0 2.2
59.2
51.2 4.0
PPL
59.7 = 18.6
73.8
25.8 2.0
58.5 = 3.8
69.8 + 2.3
57.3 + 8.6
SFT
12.2 1.0
4.7
17.8 = 0.8
38.7 3.0
70.7 3.8
60.7 2.5
Heterogeneous
P-IQL.
5.6
71.0
44.1 3,2
47.5 3.0
72.0 + 4.0
64.3 + 1.0
Sparse
CPL
18.0 6.1
50.8 40.8 0.8
18.5 3.0
32.1 1.6
67.3 + 5.5
55.5 3.3
83.8 3.8
83.5 1.8
34.3 7.6
60.8 7.3
71.2 + 1.9
63.3 4.2
the original DPO formulation, der.ford KL-constrained
ditional baselines, we include supervised fine-tuning (SFT) and Preference-baseee Implicit Q-Learning (P-IQL). Specif- ically, SFT first trains policy via behavior cloning on all preferred segments in the preference dataset. P-IQL (Hejna & Sadigh, 2024) is a reward-based RLHF algorithm that first learns a reward function from preference data and then derives an optimal policy using Implicit Q-Learning (IQL) algorithm (Kostrikov al., 2021). Notably, P-IQL is expected to achieve higher performance, it not only learns a policy but also simultaneously optimizes a reward function, Q-function, and value function.
RLHF implicitly minimizes regret.
4. Experiments
In our experiments, we aim to answer the following ques- tions: (1) Can effectivlyy learn in offline settings com- posed of heterogeneous data generated by diverse policies? (2) Does incorporating policy labels improve learning perfor- mance? (3) Can PPL be effectively applied to online RLHF algorithm? A full report for each question is provided in the Appendix F, G and H.
4.1. Experimental Setup
Implementation Details. To generate preference queries without human supervision, we pretrain an SAC model as an oracle that achieves a 100% success rate. Using this pre- trained model as a critic, we uniformly sampled segments of length 64 and assigned labels based on estimated regret. evaluate performance heterogeneous datasets, we fur- ther construct an additional offline dataset by rolling suboptimal policies with 20% and 50% success rates and combining them. For preference datasets, we conduct exper- iments under two settings: Dense, where comparisons are made between all segment pairs, and Sparse, where only one comparison is made per segment.
For fair comparison, we first evaluate the performance of PPL six robotic manipulation tasks in MetaWorla (Yu al., 2020), using the same rollout data provided by Hejna et al. (2023). Results from the reproducibility check are in- cluded in Appendix E.3. To evaluate performance on offline datasets generated from diverse policies, we aimed to follow CPL's preference dataset generation procedure. However, there are two key differences in our implementation of the critic. First, utilize rollout data without any trajec- tory truncation. Second, whereas CPL applies specific technique to reduce TD-error by re-training the critic with all rollout data added to the replay buffer, we generated pref- erence labels without such retraining. As a result, our labels may be noisier than those in CPL. Nevertheless, to ensure a fair comparison, all algorithms were trained using the same set of labels. For further details, please see Appendix E.4.
4.2. Can PPL be effectively trained on both homogeneeous/heeergggmmn offline dataset?
In the previous works, the evaluation of offline datasets has been conducted under homogencous conditions. However, in practice, offline datasets are more commonly generated by a multiple different policies. Thus, we investigate the following question:
Baselines. We consider CPL as our primary baseline, where the key distinction between PPL and CPL lies in whether the label of the behavior policy is utilized. For ad-

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI?
Homogenewws
Neterogenetus
20% 50%(ours)
FRL
1250
50%/ours)
dioem
50%/CPL)
Count
1000
750
300
250
0
11200001
Rocciion
Return
Figure 4: Distribution returns in homogeneous vs hetero- geneous offline dataset in Button Suttonn-reess-- Press-v2.
Figure 5: Ablation deterministic pseudo-labeling. We compare the average performance of PPI. and PPL.- deterministic across six environments in MetaWorld. The dashed line indicates the point where BC pretraining stops.
How would PPL and the baselines perform if the offline dataset were heterogeneous?
To investigate this, we examine the distribution of segment returns for both types of datasets, shown in Figure 4. Com- pared to the homogeneous dataset, the heterogeneous dataset includes rollout data from a policy with 20% success rate, leading to a higher density of lower-return segments.
is assumed to be fully deterministic (See Lines 4-5 of Algo- rithm 1). We then compare its performance with PPL.
As shown in Figure 5, comparing PPL with CPL reveals that when behavior policy information is not incorporated into learning, distinguishing environmental stochasticity from behavior policy suboptimality becomes more difficult, re- sulting in significant performance gap. As an alternative, using deterministic pseudo-labels for training on offline data without policy labels proves to be viable approach in homogeneous datasets, causing only a slight performance drop. However, in heterogeneous datasets, their effective- ness decreases, leading to a substantial performance gap. This result suggests that the dataset becomes more di- verse in behavior policies, incorporating policy labels into learning becomes increasingly important.
In Table 2, we report the impact of diverse behavior poli- cies on performance. PPL consistently outperforms other methods across various dataset conditions in the MetaWorld benchmark, particularly in challenging scenarios with pref- erence sparsity and policy diversity. Interestingly, unlike baseline algorithms, PPL achieves higher performance in Spare rse settings compared to Dense settings. This im- plies that PPL benefits more from datasets with broader state-action coverage rather than relying on dense pairwise comparisons across all segments. Furthermore, PPI. exhibits greater robustness in heterogeneous datasets, outperforming or matching P-IQL despite utilizing only about 6.3% of its parameters. This highlights PPL as an efficient algorithm that maintains strong performance while incurring lower computational costs.
4.4. Can PPL be effectively applied to an online RLHF algorithm?
One possible explanation for CPL's lower performance on dataset is the absence of the retraining technique to reduce TD-error-a method uniquely applied within CPL and not commonly adopted in standard practice. However, since all algorithms were trained using the same labels, we attribute this performance gap primarily to CPL's sensitivity to label noise. This sensitivity appears to arise from an implicit assumption within CPL that all training trajectories are generated by an optimal policy.
Method
Pomms
PPL
Flonwele)
PEBBLE
85%
10
Figure 6: Online learning curves across five MetaWorld tasks, comparing PPL and PEBBLE.
In the online setting, rollouts are directly executed, provid- ing explicit access to policy labels. Leveraging this advan- tage, we conducted experiments can effectively serve as an online DPO algorithm. The ex- periments were conducted from scratch, without any pre- training. Unlike the offline setting, we did not apply the asymmetric regularizer in Eq. 1, as out-of-distribution is- sues were mitigated by the iterative data collection process. We used PEBBLE (Lee et al., 2021) as an oracle because it employs a learned policy trained with unsupervised pre-
4.3. Does incorporating policy labels improve learning performance?
evaluate whether PPL
In this experiment, we examine how the presence and accuracy of policy labels affect performance. Since the offline dataset are fixed and behavior policies are typi- cally unknown, we ablate a pseudo-label setting, assum- ing each segment was executed deterministically based on the observed actions. Specifically, we introduce PPL- deterministic, where the behavior policy for each segment
8

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
training, which accelerates learning. Further implementation details provided in Appendix E.5.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Figure 6 illustrates the average success rates across five MetaWorld tasks. Notably, despite learning from scratch, the online version of PPL. achieves performance compa- rable to PEBBLE, which leverages unsupervised pretrain- ing. Furthermore, since PPL does not require learning reward model or critic, it uses only 8.8% of the parameters compared to PEBBLE, yet still achieves comparable perfor- mance. This demonstrates that PPL can serve as a highly efficient online RLHF algorithm.
5. Conclusions
In this work, we introduced PPL, a novel DPO frame- work that incorporates information from the behavior policy through regret-based modeling. We highlighted the issue of likelihood mismatch and addressed it by proposing con- trastive KL regularization. Furthermore, we theoretically es- tablished that minimizing regret is fundamentally equivalent to optimizing the forward KL-constrained RLHF problem. Empirically, PPL demonstrated strong performance across offline datasets containing rollouts from diverse policies, showcasing its robustness to dataset variations. In online setting, policy labels can be obtained more easily than in the offline case, and PPL effectively learned as an online DPO algorithm. However, we observed that online RL.FF method quite sensitive to the sampling of queries from preference data, suggesting that a more refined analysis is needed for future research.
Acknowledgements
We would like to thank LG AI Research (Youngsoo Jang, Geonhyeong Kim, Yujin Kim, and Moontae Lee) for their valuable feedback and for providing GPU resources that supported parts of this research. We are grateful to Joey Hejna for sharing implementation details regarding CPL.
This work is in part supported by the National Research Foundation of Korea (NRF, RS--2024-00451433(55555 RS- (024-00411957(155))) RSS-2023-002113557109))) Insti-
tute of Information & communications Technology Plan-
ning & Evaluation (ITTP, RS-20221-11212068(110))) RS-
2025-02305453(159)) RS-22025-02233557155)) 2021-0- 00180(10%), Artificial Intelligence Graduate School Pro- gram (Scoul National University, RS-2021-112) 1343)(10% grant funded by the Ministry of Science and ICT (MSIT), Institute of New Media and Communications(INMACC)) and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Scoul National University in 2025.

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
References
Knox, W. B., Hatgis- S., Adalgeirsson, S. O., Booth, S., Dragan, A., Stone, P., and Niekum, S. Learning op- timal advantage from preferences and mistaking it for reward. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 38, pp. 10066-10073, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
An, G., Lee, J., Zuo, X., Kosaka, N., Kim, K.-M., and Song, H. O. Direct preference- policy optimization with- out reward modeling. Advances in Neural Information Processing Systems, 36770247-70266, 2023.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprini arXiv:2110.06169, 2021.
Lee, K., Smith, L., and Abbeel, P. Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.0509), 2021.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint 22Xw::204400562, 2022.
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405. 2024.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-345) 1952.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference- free reward. arXiv preprint 2rXiv:2405.14755 2024.
Ng, A. Y., Harada, D., and Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pp. 278-287, 1999.
Gleave, A., Dennis, M., Legg, S., Russell, S., and Leike, Quantifying differences reward functions. arXiv preprint arXiv:2006. 13900, 2020.
Nika, A., Mandal, D., Kamalaruban, P., Tzannetos, G., Radanović, G., and Singla, A. Reward model learning vs. direct policy optimization: comparative analysis learning from human preferences. arXiv preprint arXiv:2403.01857 2024.
Greensmith, E.,, Bartlett, P. L.,, and Baxter, J. Variance reduc- tion techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- off-policy maximum entropy deep reinforce- ment learning with stochastic actor. International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Noukhovitch, M., Huang, S., Xhonneux, S., Hosseini, A., Agarwal, R., and Courville, A. Asynchronous rlhf: Faster and more efficient off-policy rl for language models. arXiv preprint arXiv:2410. 18252, 2024.
Hejna, J. and Sadigh, D. Inverse preference learning: Preference-based rl without reward function. Advances in Neural Information Processing Systems, 36, 2024.
Park, J., Sco, Y., Shin, Lee, H., Abbeel, P., and Lee, K. Surf: Semi-supervised reward learning with data augmen- tation for feedback-efficient preference- based reinforce- ment learning. arXiv preprint arXiv:2203. 10050, 2022.
Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Nickum, S., Knox, W. B., and Sadigh, D. Contrastive prefence learn- ing: Learning from human feedback without arXiv preprint arXiv:2310. 13639, 2023.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From to y²: Your language model is secretly a q-function. arXiv preprint arXiv:2404.12352 2024a.
Hwang, M., Lee, G., Kee, H.. Kim, C. W., Lee, K., and Oh, S. Sequential preference ranking for efficient reinforce- ment learning from human feedback. Advances in Neural Information Processing Systems,
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances Neural Information Processing Systems, 36, 2024b.
2023. Knox, W. B., HatgissKessell S., Booth, S., Nickum, S., Stone, P., and Allievi, A. Models of human prefer- ence for learning reward functions. arXiv preprint arXiv:2206. 020000,0.222...2 2022.
Schulman, J. Trust region policy optimization. arXiv preprint / 2015.
10

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI?
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.0244 02430,28,, 2015.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 1xXiv:1707.06441 2017.
Zeng, Y., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization. arXiv preprint arXiv:240..11999 2024.
Shaikh, O., Lam, M., Hejna, J., Shao, Y., Bernstein, M., and Yang, D. Show, don't tell: Aligning language models with demonstrated feedback. arXiv preprint 008802 2024.
Zhong, H., Feng, G., Xiong, W., Cheng, X., Zhao, L., He, D., Bian, J., and Wang, L. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922 2024.
Shan, Z., Fan, C., Qiu, S., Shi, J., and Bai, C. Forward kl regularized preference optimization for aligning diffusion 2024.
Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mel- lon University, 2010.
policies. arXiv preprini
Stiennon, N., Ouyang, L., Wu. J.. Ziegler, D., Lowe, Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021, 2020.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, arXiv preprint arXiv: (tt9:/9990089999 2019.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Be- yond reverse kl: Generalizing direct preference optimiza- tion with diverse divergence constraints. arXiv preprint arXiv:2309. 16240, 2023.
Xiao, T., Yuan, Y. Zhu, H., Li, M. and Honavar, V. G. Cal-dpo: Calibrated direct preference optimiza- tion for language model alignment. arXiv preprint arXiv:2412.14516 2024.
Xiong, W., Dong, H., Ye, C., Wang, 7., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. Forty-first International Con- ference Machine Learning, 2024.
Yang, M., Schuurmans, D., Abbeel, P., and Nachum, O. Dichotomy of control: Separating what you control from what you cannot. arXiv preprint ar 230.:32210. 2022.
Ye, C., Xiong, W. Zhang, Y., Dong, H., Jiang, N., and Zhang, T. Online iterative reinforcement learning from human feedback with general preference model. In The eight Annual Conference on Neural Information Processing Systems, 2024.

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
A. Main Proof
Lemma (3.2). (Structural Condition for a-optimality) A reward function and a soft optimal Q-function where T' (-1s)
a-optimal have a one-to-one correspondence with state-dependent function 3 + P. as follows.
Ra (r+ a) x" (als) B(s) - > As S, A, log
2
{0" (s,a) a) = lo (als) + [()), vs E S E Al 2 0, 8
Proof. (π² a-optimal - º. (s, a) a log x (als) + (s)) for some 3 S R.)) Remark that the policy x² is a-optimal, if and only if there exists the optimal soft Q-function satisfies the following relation:
म.' (als) = exp ( (0 (8, (s, -
V- (s)). V (8) = a log [ exp
Q (8, a)da)
(8,
JaEA a
Since V* is merely partition function, letting ((ss a) = exp Q (s, (s. we can derive
X(s,a) SaEA X X(s, a)da
x² (als)
= x(s,a) a) = d( (al for some d S + R
- Q" (s,a) a log - (als) + B(s) for some B : S + R,
where 33 is defined as B(x) = log d(s).
Using the soft Bellman equation, consider a reward function for any state-dependent function 3 IR and substitute the expression of Q= (s, a). Then, we have:
r(s,a) a) = Q" (s, a) - VV (s')] a log (als) + ((s) - (sEs
where x² and are given. By the definition of optimal soft Q-function, recursively substitute the soft Bellman equation and sum over timesteps:
Q (s,a) r a) + + E.
Σ
(r(ss, + + ( ao =
1>0
= a log x (als) + B(s) TBe (B(x))
+ E,war
Σ (B(ss) yB (8(
80 0, =
aa (aass) + B(s) 32 [8(52)]
+ E E,
(B((())) yB 80 = 8, 0,
t>1
= a log "aggT"(aap (als) + B(s).
Lemma (3.3). (Unique Fixed Point of Soft Bellman n-operator) Let n² is a-optimal. For a given policy π and Q-function Q E Q" for any (s, a) E S x A. define the Bellman π n-operator T Q - Q" where
TTQ%(s, = Q. (s. 7Ep [a (H (-18) - H-(-11²))
+E,+IQu (5', a') E, (s' a
Then, T has unique fixed point Q
12

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
Proof.
Consider Qa: Qh Q".
Then
sup 8,0
S sup 7BP
B (s -E,1Q
8,0
sup Q7 -2(s',a') Qb''''a')
s'
Hence, T." is y-contraction for any 07:07 Q". Since Q is complete metric space, by using Banach fixed point theorem, T." has unique fixed point.
Notice that Q and Q satisfies soft Bellman equation respectively, i.e.,
(. a) Ep Tx (x a) -.. (0" (s', + aH" (1s))] (s, Er
+ ((s.a)
Then,
(s,))
= Q (s, a) -BBp a H1" (116)) - ((6')s(('1)) (-155) + E. E.. (0'" [s)
= O (8, - 7Ep a7"" (16') + E.. 10. (d, - (-18))))) + B. (7(,
= Ep + 7B 10, a') + a+H))
= (s,ss) ((s E E S A.
Hence, Q a unique fixed point of T.".
Theorem (3.4), If a policy x² is a-optimal, then for any policy n,
Q" (s,a) -
where the forward
forward KL divergence is defined as
DKL(T||T:: ;s EynPT
Y DKL (T)(
120
Here, IPT is the distribution of trajectories (so, ao: at (so, an) as.
8t, ...) generated by policy x and the transition P, starting
Proof. Let a)) QT (s, a Eiso ( Then
a. = a) for all (s, E S A.
T.7 (s.
= QT (s,u) - -PE a HT (118) -H=((1s') Ex- (5', 1 E (6T(('''))))
(H*((18))
= - (s, -BB (a ((<T" (11') (16')- -H (a " (s', + 3(6')
- Ex Q [E (s',a') a "ETNE [
81 01.1=d)
1>0
= Q. (s, -17 B(s')
aH." (-s) - E.
(s',a'))
+ alE Σ
Er ((
81 = 8 = a
1>
13

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI?
Q as ayEp
Σ
E/ET~? (x(( so = s, 00 =
a
1>
Q (s,a) a Σ
-e ( (-1s)) 8,00 a]
1
= (8, a)
which implies that Q7 is a unique fixed point of T." In Lemma 3.3, we observe that 7," has a unique fixed point Q Hence,
(s. Q. (s. Σ
E ENEPY [ (-Ise)) so s. ao
B. Further Theoretical Analysis & Discussion
B.1. Mathematical derivation of PPL framework
We recall the PPL model and objective:
p(r 1st 5-
Rega .0) - Reg (8, a.7 a))
120
Reg (st, at) Reg (87 (8,, ]
CPPl (Tp:(TT;)))
-Bict <- w.
log a
120
where
- RegT (st, a4) = -(V." (st) - (sr,,,,
Here, a negative regret (st, at) can decomposed into two components:
- Reg" (8), = a log x xa (s,)
Ey-ry tt
Σ
Y DEL (-1s))
increase likelihoed
1>0
decrense sequential forward KL. divergence
Proof. By the definition of regret,
-Regz. (st; ax) - -VV (st) - (Te,
Exr 1Qth (st, a) a log (-1s:)] Q (st, -DDk((rll
B(o7 a loog " (arl0)) + ((5T a aDKi(T|l (1,00))
= =a( (log x (a.ls))) (
(5)
B.2. Regret is invariant under policy-invand invariat transformations (Corollary 3.5)
As noted in Lemma 3.2, any policy-invariant transformation can be expressed as combination of a state-dependent function 3(s) and scaled log-likelihood term log (a)s),, where represents the temperature parameter in the MaxEnt framework. Specifically, for any transformation of the reward function that preserves the optimal policy, we can rewrite the modified reward as:
r(s, a) = a T (a)s) + Bs))
14

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI?
This formulation extends the classical reward shaping result of Ng al. (1999) by explicitly incorporating the policy- dependent term log x (a]s), which accounts for transformations in the likelihood space. This insight allows generalize policy-invariant transformations and directly integrate them into preference-based learning objectives.
Using this representation, we can reformulate the sequential DPO objective with a p policy-invariant transformation as follows:
{{ log (a 11 + B(st) - - P at))((
(TP; D) = -Ep log
Teet a1) 188
120
{{ log
Tep (a, |,
+ B(s
- - gppPlt a 1oriia,,,66660 [8(6/11))]
(6)
Trer(a, (s
The existence of multiple objectives that preserve the optimal policy through reward shaping has been explored in previous work, particularly in the variance reduction schemes of policy gradient methods. Schulman et al. (2015) introduced the generalized advantage estimate (GAE) as a method to reduce the variance of policy gradient estimates, effectively selecting an appropriate (()) to improve stability and efficiency in learning. Similarly, in Equation the standard DPO framework assumes ((s) but optimizing (s) to minimize the variance of gradient estimates could lead more stable training.
In contrast, shown in Equation 5, regret-based formulations naturally eliminate ((s) by definition, avoiding the challenges transformations. This property ensures that regret serves unique and well-defined objective function, making it inherently robust without requiring explicit variance reduction techniques.
associated with
B.3. Reformulating the MaxEnt objective with negative regret as the reward (Theorem 3.6)
Corollary (3.6). Maximizing the MaxEnt objective with negative regret as the reward is equivalent to minimizing the sequential forward KL divergence between the learned policy and the behavior policy for each preferred state-action pair in the dataset, iee.
(Ect~ol-Ree Reg (s* atttaat)== - a log (x(aa(s*)))
arg max R5
arg min FBstNDDDKLT |ipist at))).
Ty
Proof. Consider a dataset D and a set of sampled preferred segments {c N which are generated by behavior policy प.. respectively. To avoid notation ambiguity, we emphasize that the subscript in this proof denotes the index of each individual samples. When defining the reward function the negative regret, the optimal policy of Maxent objective Tieg can be reformulated as:
N
Regad (57 . a log n (atls))))
Treg == arg max
Q (stt att VoLy (x+) log ( lo,t)]
arg max
N
log कe (a (xt) DKr tt a a log (o (ot)))
arg max By
N
arg min T
(Ki(T (mpist,at)
Notably, the minimum is achieved and only if (a = x(a+(s$) for each E [N]. This formulation demonstrates that maximizing the MaxEnt objective with regret- reward is fundamentally equivalent to minimizing the sequential forward KI. divergence for each segment.
15

---

Policy-labeled Preference Learning: Is Preference Enough for RLHF?
Discussion. The regret-based DPO framework be reinterpreted process that aggregates the behavior policies underlying the given dataset, aligning the learned policy to preferred actions by reducing the sequential forward KI. divergence. If, as assumed in CPL, the behavior policies of all preferred segments in dataset D correspond to the optimal policy x² (or can be constructed as such), then PPL is guaranteed to converge to the optimal policy.
However, in practical RLHF settings, such an assumption rarely holds. Unlike standard reinforcement learning, where an agent maximizes predefined reward function, RLHF optimizes for policy alignment rather than absolute optimality. the DPO framework, the reward function is implicitly constructed make the aligned policy the optimal one within the given preference dataset. As result, the optimal policy under the learned reward function is already the policy obtained through alignment, making it unnecessary to perform additional RL algorithm to reach the optimal policy.
To achieve further improvements, it is crucial to expand the dataset by rolling out new policies and incorporating additional preference data. This process enhances dataset coverage while enabling the learned reward function to extrapolate more effectively. Without such iterative expansion, RLHF remains constrained by the limitations of the static dataset, preventing meaningful policy improvements beyond the scope the initially collected preferences.
C. Pseudocode
Algorithm Policy-labeled Preference Learning (PPL.
Input: number of queries N, trajectory dataset &, minibatch size
1: Initialize policy parameters yb
2: for n = 1,.... do
3: Sample ç. ~
4: if policy label TT ( 57 s/)) unknown then
5: x(((se)= f- 6a, Nt (-15))) ddg
6: end if
7: Label the behavior policy (T,, n²)
8: Instruct the preference label (v(0), (())
9: Store preference D + D U {(S, S' H.P))
I/ Create Policy-labeled Preference Queries
10: end for
for to 7 do
Sample minibatch {(S, ,',,,',,,,, 2)
13: vb <- arg min& (T D)
|/ Policy Learning
14: end for
16

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
D. Variants of PPL and Baselines
BC: BC (Behavior Cloning) is the initial stage in RLHE, where the policy is trained to maximize the likelihood of
demonstrated actions given the corresponding states:
Luc(T D) = -BQND
Σ
log ग. (a)ls)))
t20
SFT: SFT (Supervised Fine Tuning) is trained to maximize the likelihood of the demonstrated actions given the corre-
sponding states in preferred segments:
(TFT(TT; -IEstND
Σ log Te (a (st [st]
CPL:
CPL (Hejna al., 2023) is our primary baseline, where the optimal advantage is defined as the function:
(a (s7
SCEL (TTiS)))) SceL(TTi
Σ log
To ( 186
120
The objective is to minimize the following loss function:
LCP C D) = -F(st, >-> log a (Tp: - Scei (T (T)
A key issue raised in CPL is assigning high weights OOD actions while still maintaining the optimal policy. This leads to extrapolation too much into unseen states, ultimately degrading performance. To mitigate this, asymmetric
regularizer is introduced:
> log Tel (a (a,t (a, |s )a
(CPL (co: = ScpL(TTi(t)
- ASCCL(TO:
+20
PPL: Based on policy deviation lemma Theorem 3.4, PPL extends CPL by incorporating entropy regularization and KL divergence-bart based constraints, making preference learning more structured. The score function includes multiple terms:
SpPL(TTi "t,,xx) x) (PPL(TTi ,xx
log (o + 1 Σ (a 1s;
DKz (ett(ar(sstttt (lst + DKz(= Dxz(==((s5,)) (155)))l
and the objective function is:
CPPL(TT; (E(sttc-)-a log a (Tvi - SppL (TWi (T)
The score function of PPL with the same asymmetric regularizer as CPL given by:
SppL(Xy(TTp) st,xtt (A)ppi "T Spr St, m+) ASppL(TTi (Toi xx)
= =Σ log Tola,
Σ
DKL (or (1sttttt)) + ADxe(T (165
( )²
1200
17

---

Policy-labeled Preference Learning: Is Preference Enough for RLHF?
PPL-deterministi:: If policy-label is unknown, apply deterministic pseudo-labels by assuming that each segment was generated by deterministic policy that executed the observed action.
( s Tel (a, 15
log Te T T ( (541))
(T)) (T))
Σ 120
log
SppLice(TTiS (T) SPpLIct(T(i()) Σ log
T 18,
log Ty(a |sp ) [eer(a; (8
Trefl (1 (s+t)
+ *H*x(1187 7H* Σ
( (-1 (1 +
DKL(T (1
1=1
- (105))))))) 71t T (117) + Σ PTI ( (1 Σ D (mer
18

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
E. Implementation Details
E.1. Hyperparameter Setting
Table 3: Hyperparameter settings for offlin implementation.
Hyperparameter
State
Total Training Steps
500k
Pre-training Steps (except P-IQL)
200k
Batch Size
96
Segment Size
64
Fixed log std
-1.5
Actor Dropout
0.0 (0.25 for CPL reproduce)
Architecture
[256, 256] MLP Gaussian
Table 4: Hyperparameters for online implementation
Hyperparameter
State
Total Environment Steps
lm
Segment Size
32
Fixed log std
-1.0
Query Frequency(steps)
1000
Policy update Frequency(steps)
1000
Episode Length
250
Learning rates
3e-4
Temperature
0.1
Asymmetric regularizer
1.0
BC weights
0
?
1
Actor Dropout
0.0
Architecture
[256, 256] MLP Gaussian
Table 5: Hyperparameters for PPL, CPL, SFT, and P-IQL
Hyperparameter
PPL
CPL
SFT
P-IQL
Learning rates
le-4
le-4
le-4
le4
Temperature a
0.1
0.1
0.1
0.1
Asymmetric regularizer À
0.5
0.5
BC weights
0
0
0
7
Number of Parameters
76k
76k
76k
859k
19

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
E.2. Meta World Benchmark
Our experiments were conducted six MetaWorld environments: Bin -acc-iickng, Button Setton-Presse Press, Door Door-Open, Drawer-Open. Plate-Slide, and Sweep-Into. Each task requires precise control of a robotic interact with objects in structured environment. The diverse task set includes object relocation, pushing, pulling, and fine-grained manipulation, making it suitable testbed for reinforcement learning from preference-based feedback.
Bin Picking
Button Press
Door Open
Drawer Open
Plate Slide
Sweep Into
Figure 7: Visualization of the MetaWorld Benchmark Tasks.
Each environment is designed with handcrafted reward function tailored to its objective. Instead of human annotations, we trained a critic using SAC to assign labels. During our experiments, we observed that return did not always align well with success rates. case of Dwor--peeeteeeooo despite achieving the highest return, PPL exhibited a relatively low success rate. This implies that the environment allows reward exploitation due the imprecise design of the reward function.
E.3. Reproducibility Check
For a fair comparison, we first verified the reproducibility of CPL using the Metaworld State Dense and State Spa rse datasets provided by Hejna et al. (2023) and evaluated the performance of PPL. on these datasets. We used the official petttps:/ggithn
com/ sheejna/cpp) without modifications and ensured reproducibility by fixing the random seed ([123, 312, 321]). The figure below presents the PPL performance alongside the reproduced CPL results. The horizontal dashed line represents the scores reported in CPL, confirming the reproducibility of the algorithm. The vertical dashed line indicates the point where behavior cloning (BC) training stops.
CPL implementation
In all environments except P Plate-S1 iideiddvi v2, the reproduced CPL, performance closely matches the reported values, with deviations attributed to seed variability. Across the provided datasets, PPL exhibits comparable overall performance to CPL.
Table 6: Success rates of all methods on six tasks from the MetaWorld across different datasets from Hejna et al. (2023).
Each score is reported as the highest average performance across four seeds over a 200-episode evaluation window.
Bin Picking
Button Press
Door Open
Drawer Open
Plate Slide
Sweep Into
CPL(Reported)
80.0 + 2.5
24.5 2.1
80.0 6.8
83.6 = 1.6
61.1 3.0
70.4 3.0
State
2.5k Dense
(Reprouudd)))))
76.0 4.1
24.9 4.7
75.5
87.6 +
45.3 10.4
74.5 3.4
PPL
77.7 + 2.6
30.2 +: 7.8
76.7 = 7.1
84.2 = 2.4
41.7 + 3.2
79.2 +: 5.5
(Pe.ooep))
83.2++3.5
29.8 1.8
77.9 9.3
79.1 +
56.4 +.4 3.9
81.2 1.6
State 20k Sparse
(Pe(Reprol
69.1 10.1::20 21.4
25.5
74.4 = 3.5
80,9 - 4.5
41.1 4.9
80.5 + 2.8
PPL
3.7
25.4 + 2.8
72.2 1.7
79.0 - 4.0
42.9 +: 1.6
76.0 16 2.0
20

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
Picking
Open
Successs rate:
Success rale
0.8
PPL
300000
200539
300099
400300
500390
300000
200000
300039
480000
500390
300000
200000
300539
480000
500290
Iteration
Iteration
Iteration
Drawer Open
Plate Slide
Sweep Into
0.0
Succes rote
0.7
0.0
0.7
Itoration
Itoration
Itoration
Figure 8: Reproducibility check on State Dense dataset
Bin Picking
Button Press
Door Open
0.2
0.275
Succeto rate
Seccess rate
0.000
2000
0.500
300330
300030
500300
300030
Drawer Open
Plate Slide
Sweep Into
3.75
Su0cess rat2
Succoas patt
rate
360
06
35000
206530
300038
SEC300
106330
300000
206030
300030
680300
105330
300000
200000
300530
483300
105390
Figure 9: Reproducibility check on State Sparse dataset
21

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
E.4. Offline dataset generation and its distribution
We construct a heterogeneous dataset by incorporating various policies, following the dataset generation method used in Hejna et al. (2023). Specifically, load suboptimal SAC checkpoints with success rates of 20% and 50% using the same approach. During rollouts, introduce Gaussian noise with standard deviation of 0.3 and rolling out 20,000 episodes, each lasting 250 steps, using their suboptimal soft actor-critic (SAC) (Haarnoja et al., 2018) checkpoints, which achieved an approximate 50% success rate.
While following this data generation procedure, we found a step in the reference code where transitions following success signal were explicitly truncated. This truncation was intended prevent segments from being overly dominated by successful transitions. However, opted retain the data without truncation. As result, the distribution of our 50% success rate dataset differs from that of Hejna et al. (2023). To highlight this difference, provide visualization of the data distribution across environments.
Bin Picking
Button Press
Door Open
20%- 50%)oos()rrss
1500
50%soo0%s
20%-50%(ours)
8000
50%(ours)
50%[oour) 50%(CCPL)
50%(ours)
Count
6000
50%(CPL)
Count
1000
1250
50%(CPL)
4000
750
750
300
500
2000
250
250
10
20
50
40
50
60
10
20
30
40
50
0
30
40
50
Return
Drawer Open
Plate Slide
Sweep Into
20%-50%% (0%sss)(orrrs
5000
20%- 50%%o50%lorr)
20%-50%(tors)
50%(ours)
50%soour)
50%(ours)
2500
50%(CPL)
4000
50%)(CPL)
50%(CPL)
Count
2000
Count
Count
2000
1000
500
1000
20
40
50
60
Return
Figure 10: Comparison return distributions across environments different dataset configurations. The histograms illustrate the distribution of the partial returns for segments with 20% and 50% success rates generated using our method (red and blue) and the 50% success rate dataset from Hejna et al. (2023) (gray).
For experiments, generated the following four datasets: Homogeneous Dense, Homogeneous Sparse, Hetereogeneoous Dense, and Heterogeneaue Sparse. In the additional experimental setting, we kept all as- pects-such as the hyperparameters of all algorithms, the SAC critic, and the label generation method-identical to the original setup, modifying only the dataset. Interestingly, CPL exhibited significant performance variations depending on the dataset, whereas PPI. demonstrated robust performance across diverse datasets. The robustness PPL's performance can be attributed its ability to adjust the magnitude of feedback for diverse policies and accurately reflect the likelihood of each segment.
22

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
E.5. Online Implementation
In the online setting, we use Gaussian actor with fixed standard deviation maintain consistency with the offline setting, The model is trained from scratch without any pretraining. The online learning process consists of three phases. First, rollouts are conducted in the environment for fixed number of steps to generate trajectory data. Next, preference queries and labels are constructed from segments of these trajectories. Finally, the policy is updated using the generated preference query data.
During the rollout phase, actions are sampled from a stochastic policy without additional exploration strategies. the query generation phase, two policies are selected for comparison, with one always being the most recent and the other randomly chosen from the last 25 policies. Segments from the most recent policy are first over-sampled three times the required number, then ranked based their regret scores relative the current policy. The top-ranked segments retained, while segments from the other policy sampled uniformly at random. Preference labels are assigned according to the method described in Appendix D.2 of Hejna et al. (2023).
In the policy update phase, stochastic gradient updates are applied over a fixed number of epochs using all preference query data collected up to that point. Unlike reward-based preference learning methods, which predominantly generate preference queries early in training and subsequently optimize policies using learned reward function and an RL algorithm, the online PPI. algorithm continuously collects preference queries throughout the entire training process. This ensures sustained policy improvement over time.
To reproduce the online baseline PEBBLE algorithm, we utilized the official B-Pref implementation (https: //githu. com/ (eallrreeearch/BPee
and adhered to the hyperparameter settings and random seeds reported in the original paper. Our online experiments were performed on five tasks from the MetaWorld benchmark: Button Press, Door Open, Drawer Open, Plate Slide, and Sweep Into. All hyperparameters were kept consistent across tasks, except for the total number of preference queries, which was set to match the values specified for each environment in the PEBBLE paper.
23

---

Policy-labeled Preference Learning: Is Preference Enough for RLHF?
F. Experimental Results on Homogeneous/ Heterogeneous Datasets (Section 4.2)
F.1. Homogeneous Dense Offline Dataset
Open
PPL
PIZL
Succrso rale
0.4
2,00
ass
0.1
30000
200030
200020
4BC000
506230
200000
200000
200020
400000
500230
200000
200000
200039
482000
500200
Iteration
Iteration
Iteration
Drawer Open
Plate
Sweep Into
Success
0.7
Iteration
Itoration
Itoration
Bin Picking
Button Press
Door Open
Beturm
Return
TPL CPL 3/O
Iterattio
Iteraition
Iteration
Drower Open
Plate Silid
Sweep Into
100
Requr
200
Iteration
Iteretion
Iteretion
Figure 11: Performance comparison of different methods the Homogeneous Dense dataset across six MetaWorld
tasks. The top shows the success rate training iterations, while the bottom presents the corresponding return values.
24

---

Policy-labeled Preference Learning: Is Preference Enough for RLHIF?
F.2. Homogeneous Sparse Offiin Dataset
Bin Picking
Button Press
Door
0.8
m
RPL
Succett rate
00.00
Success
PIQL
M
8.
SFT
2.0
0.2
01.
000
300000
200030
300039
48C090
500290
300000
200000
300039
480000
500290
300000
200000
300039
480000
500300
Iterationn
Iterationn
Iteration
Plate Slide
0.7
0.75
0.5
Succeso rale
Success rale:
1.00
300000
200030
300039
4BC300
500090
300000
200030
300039
680000
500390
300006
280000
300039
482000
500390
Itoration
Itoration
Itoratioon
Picking
Open
100
Return
M
Return
210
3PL
20O
300000
200000
200699
4B0000
500230
200000
200000
200039
4B0000
500390
300000
280000
300039
480000
500300
Iteration
Iteratton
Iteratton
Drawer Open
Plate
Sweep Into
300
192
Retumm
1:0
Retum
4.0
Itoration
Itoration
Itoration
Figure 12: Performance comparison of different methods the Homogeneous Sparse dataset across six Meta MetaWorld World tasks.
25

---

Policy-labeled Preference Learning: Is Preference Enough for RLHIF?
F.3. Heterogeneous Dense Offline Dataset
Bin Picking
Button Press
Door
PPL
PIOL SFT
Succiss rate
Success
0.10
300000
200030
300039
4BC090
500290
300000
200000
300039
480000
500290
300000
200000
300039
480000
500300
Iteration
Iterationn
Iteration
Plate Slide
0.5
0.7
0.5
Success rafe
300000
200039
300039
4BC300
500290
300000
200030
300039
680000
500390
300006
2800000
300039
482000
500390
Itaration
Itoration
Itoratioon
Picking
Open
210
Return
Return
3PL
40
20L
22
300000
200000
200099
4B0000
500230
200000
200000
200539
480000
500230
200000
280000
500039
480000
500300
Iteratton
Iteratton
Iterattoo
Drawer Open
Plate
Sweep Into
753
100
Retum
31.0
Retum
20
200
200
Itoration
Itoration
Itoration
Figure 13: Performance comparison of different methods the Heterogeneous Dense dataset across six Meta World tasks.
26

---

Policy-labeled Preference Learning: Is Preference Enough for RLHIF?
F.4. Heterogeneous Sparse Offline Dataset
Bin Picking
Button Press
Door
PPL
PIQL
rate
SFT
rate
Success
0.2
01
300000
200000
300639
4BC090
500290
300000
200000
300039
480000
500290
300000
200000
300039
480000
500300
Iteration
Iterationn
Iteration
Plate Slide
0.00
Successo rake:
0.3
30000
200030
300039
6BC300
500090
300000
200030
300039
680000
500390
300066
280000
300039
480000
500390
Itaration
Itoration
Itoratioon
Picking
Open
340
300
Return
Return
3PL
20O
300000
200000
200699
480000
500290
300000
200000
200039
4B0000
506230
200000
280000
200039
480000
500300
Iteration
Iteratton
Iterattio
Drawer Open
Sweep Into
200
Man
120
Recum
Returm
31.0
400
Itoration
Itoration
Itoration
Figure 14: Performance comparison of different methods the Heterogeneous Sparse dataset across six MetaWorld tasks.
27

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI?
G. Comparison with Deterministic Pseudo-labels (Section 4.3)
G.I. Homogeneous Dense Offline Dataset
Open
63
Succeso rale
4.7
Success rate
0.30
2,30
0.4
0.2
PPL
PPI
230
30000
200000
200030
400000
506230
200000
200000
200030
400000
500230
200006
200000
200039
482000
500300
Iteration
Iteration
Iteration
Drawer Open
Plate
Sweep Into
0.700
0.8
Successo rate
0.00
Success refte
2.00
0445
Iteration
Itoration
Itoration
Figure 15: Comparison of PPL and PPL-deterministic the Homogeneous Dense Offline Dataset.
G.2. Heterogeneous Dense Offline Dataset
Bin Picking
Button Press
Door Open
0.7
3.75
0.5
0.5
Succett rate
Success rate
0.4
2.00
0,20
0.3
87
0.1
PPL
a18
donorm
0.2
0.30
Iterattonn
Iterattion
Iterattoo
Drawer Open
Plate Slide
Sweep Into
3.70
Success rate:
Success rate
0.4
3,52
2.5
6.00
Itoration
Itoration
Itoration
Figure 16: Comparison of PPL and PPL-deterministic on the Heterogeneous Dense Offline Dataset.
28

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI?
H. Experimental Results on Online Implementation (Section 4.4)
H.1. Online Learning Curves
We evaluated the performance of PPL in an online setting across five MetaWorld tasks. The number of preference queries (#Pref) varied for each environment based on the quantities used in PEBBLE, and these differences are illustrated in each plot.
Button Press
Door Open
Drower Open
Sucocsa 1200
Sucocss rato
Sucecss rate
0.4
04
Ph 10000)
POL 10003)
MU 200001
PEBBLE Pra 10000)
PEBBLE oracie 10000)
PEBBLE FracciutFred 20000)
Enstronment steas
Endrronmen
34
Envtronment
Sweep
Pr 20000)
PEBBLE, praciete Pret 20000)
Success EOS0
19000)
PEBBLE Lww..EEeeeeeeeeeee 10000)
Environment stops
Environment Ste55
Figure 17: PPL and PEBBLE learning curves in online learning.
H.2. Ablation Preference Query Count
We evaluate the performance of PPL over iterations with different numbers of preference queries (#Pref). Overall, increasing the number of preference queries leads to improved performance, demonstrating the benefit of richer preference feedback in online learning.
Doar Open
Open
3.8
PFu/ePret:: 5000)
Pru 5000)
PPre 10000)
PFUe#Prot: 10300)
PFL 10000)
PP_[ePret 20000)
Success rote
Success robe
0-
Envtronment stacs
Environment staps
Endrronmen stops
Plate Slide
Sweep Into
PP-(aProf 5000)
PP_ 10000)
Success rate
Pioc 10000)
(aPrl (0000)
36
dtps
Envrrement
Figure 18: Effect of preference query count in online learning.
29

---

Policy-labeled Preference Learning: Is Preference Enough for RLHI??
H.3. Ablation Rollout Length
We analyze the impact of different rollout lengths on the performance of PPL in an online RLHF setting across five Meta World tasks. Each plot compares the success rate over training iterations for three rollout lengths: L {5, 10, 20}.
Button Press
Door Open
Droier Open
PPLL 53
PPuL 5:
PPUL 10)
PPUL
Succesa rote
Succesa
PRUL
PALL 10)
08
68
PEUL = 20)
06
Environment steas
Environment steps
Environment stops
Plate Slide
Sweep
PP-IL- 101
PP.IL 201
Succers roto
PILL 5]
10)
20)
36 stops
15
Figure 19: Effect of rollout length online learning.
30


Reviews:
[{"review_id": "a6U6JEVpXQ", "reviewer": "Reviewer_Gpnc", "summary": "This paper proposes an algorithm called policy-labeled preference learning where the preferences are assumed to be formed by a regret-based metric and the preference data are labeled by the policy that generated the trajectories. The paper did a very good job explaining why the regret-based preference model may be preferable and how it is different from the advantage-based model in the literature. The results are very strong against the state-of-the-art baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and effort in reviewing our paper. We have organized our responses to your comments below. If any of our reconstructed responses miss the intent of your questions or if there are remaining concerns, please let us know so we can address them.\n\n**1. Difference between PPL and cited literature**\n\nThe regret-based preference model used in the CPL algorithm, which defines regret in terms of the optimal advantage function, was first introduced in [Knox et al. 2022]. Similarly, [Knox et al. 2024] interprets preferences using the same optimal advantage-based regret model, arguing that the Bradley-Terry model yields an approximation of the optimal advantage function rather than a true reward function. The regret preference models used in CPL, [Knox et al. 2022] and [Knox et al. 2024] are both essentially based on optimal advantage, which is a different design from the regret model proposed in PPL.\n\n**2. Response to [Other Comments and Suggestions]**\n\nThank you for pointing out the typo, and I will incorporate it into the revised version for better clarity.\nRegarding Figure 4, it was designed as a single plot to enable an at-a-glance comparison of the distributions across the three datasets. However, we recognize that overlapping colors make it difficult to distinguish the overlapping regions. Given the space constraints in the main text, we will either separate the plots or incorporate an envelope to improve boundary delineation.\n\n**References**\n* [Knox et al 2022] : Knox, W. Bradley, et al. \"Models of human preference for learning reward functions.\" arXiv preprint arXiv:2206.02231 (2022).\n* [Knox et al 2024] : Knox, W. Bradley, et al. \"Learning optimal advantage from preferences and mistaking it for reward.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 9. 2024.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. Overall, I find the paper to be difficult to argue against. Kudos to the authors.", "methods_and_evaluation_criteria": "The methods and the evaluation criteria make sense. While reading the paper, I always had in mind that the proposed method will not work for the cases where there already exists an offline dataset without policy labels. However, the paper considered this cases too and proposed a heuristic method to get around that issue. It also conducted experiments under that setting.", "theoretical_claims": "I did not check the proofs as they are in the Appendix, but the claims intuitively made sense to me.", "experimental_designs_or_analyses": "The experimental designs and analyses make sense. \n\nWhile there are a few cases where baselines perform better than PPL, in most cases PPL outperforms the others. Such consistent improvement is not very common in the RLHF literature, so I would say the results are very strong.", "relation_to_broader_scientific_literature": "The paper builds upon the CPL algorithm. One thing that I am not sure about is: the paper cites \"Learning Optimal Advantage from Preferences and Mistaking It for Reward\" by Knox et al. which also considers a regret-based method. Does their method also suffer from the same problem as the CPL paper, i.e., do they also present the regret-based model as equivalent to the advantage-based model? If so, perhaps that should also be mentioned when this limitation of CPL is discussed in the paper. If this is not the case, a clarification of why the regret-based model of this paper is different/better is needed, as otherwise the novelty becomes thinner (though the method and the theoretical analyses are still novel and I would still argue for the acceptance of the paper).", "essential_references_not_discussed": "N/A\n\n(though, just like many other recent RLHF papers, this one also mostly neglects the pre-InstructGPT preference-based reward learning literature other than the Christiano et al. 2017 paper)", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "There are a few typo-like issues:\n\n1) In the equation before Section 3.2, the term inside the summation must be inside a parenthesis as otherwise the second term is outside of the summation.\n2) In line 313 (right column), there seems to be an extra word that should be deleted.\n3) In line 379 (left column), \"implementation\" is misspelled.\n\nFinally, it might be a good idea to divide Figure 4 into three separate figures so that the histograms are separately visible -- currently, their overlapping regions are difficult to distinguish, if not impossible.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GJ7ddFr9cP", "reviewer": "Reviewer_MyYU", "summary": "The paper proposes a novel approach to preference-based RL, grounded in regret-based modeling of human preferences. Unlike prior work that also uses regret-based modeling, the paper explicitly labels the behavior policy. This, as argued by the authors, is important in order to  resolve the likelihood mismatch issues when utilizing regret-based modeling of human preferences. The paper provides a formal analysis of their novel framework, which yields a new algorithmic approach to preference-based RL. Finally, extensive experiments are conducted to showcase the benefits of the approach.\n\n---\n## update after rebuttal\n\nI thank the authors for their response. I will keep my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewers for their time and thorough evaluation of our paper. We have organized our responses to your comments below. If any of our responses fail to address the intent of your questions or if you have remaining concerns, please let us know.\n\n**1. Generalization to other domains**\n\nSince PPL is particularly beneficial for sequential and stochastic scenarios, we determined that a robotic manipulation task is more appropriate than a language domain with deterministic transitions, and thus conducted our experiments accordingly. If extended to the language domain, exploring the multi-turn LLM scenario you suggested would be an excellent direction for future work.\n\n**2.LLM literature reference**\n\nRecent research on RLHF has been particularly active in the LLM domain, and our PPL algorithm which directly learns policies without explicitly learning a reward function shares similarities with the Direct Preference Optimization framework. We briefly mention this connection in the main text. However, since our experiments focus on sequential RL tasks such as robotic manipulation, we decided not to deeply discuss related work specific to LLM tasks to avoid reader confusion. \n\n**3. Response to [Other Comments and Suggestions]**\n\nIn Equation (2), we noticed that the derivation of Equation (3) is referenced as being in Appendix B.1, but that reference appears about one paragraph after the equations are presented. To reduce any potential confusion for readers, we will adjust the sentence placement.\nAdditionally, the explanation for the subscript * is provided later than its initial usage. Since removing the subscript * does not change the original meaning of Equation (2), we will remove it from that equation. We will also correct the typo in Lemma 3.3 as you pointed out.", "claims_and_evidence": "In my opinion, he set of results in this paper is quite convincing and it clearly demonstrates the utility of the proposed approach. The paper provides an in-depths analysis (both theoretical and empirical) of the considered problem setting. I also appreciate that the intuitive examples in Section 3 which motivate the approach.", "methods_and_evaluation_criteria": "The paper provides both a theoretical investigation of the problem setting and an empirical evaluation of the proposed approach. The test-bed is primarily based on simulation-based robotics tasks. It would be interesting to see whether the results generalize to other domains, but I don't think this is necessary given that the contributions are substantial.", "theoretical_claims": "I've taken a look at the theoretical claims, including the proofs in the appendix. The claims and the proofs seem to be correct.", "experimental_designs_or_analyses": "The experimental test-bed is rigorous and follows closely related prior work on preference-based RL. The experiments are extensive and test various aspect of the proposed approach. I believe that the experiments are well-designed and I didn't spot major issues with the experimental analysis.", "relation_to_broader_scientific_literature": "This work is broadly related to the literature on AI alignment, focusing more specifically on preference-based RL. The proposed approach could potentially be more broadly applicable, e.g., for training LLMs in multi-turn dialogues. However, the current results primarily focus on RL and manipulation policies in robotics.", "essential_references_not_discussed": "The background and the main reference that this work extends is explained well. However, it would be great if the authors could include a separate related work section and provide an overview of the literature on RLHF and DPO. I can see several important references cited in the related work, including those that discuss MDP extensions of DPO (e.g., Rafailov et al. 2024a, Nika et al. 2024, etc.). However, these don't seem to be extensively discussed in the main paper.", "other_strengths_and_weaknesses": "The paper is well-written, easy to follow and enjoyable to read. The contributions are non-trivial in my opinion; while the main idea behind the proposed approach is quite intuitive, to my knowledge it is conceptually novel. The theoretical analysis is also novel, although the proof techniques seem to be based on standard arguments. The experimental analysis is also thorough. Overall, I found the results quite interesting and convincing.", "other_comments_or_suggestions": "Suggestions:\n- I didn't initially understand how Eq. (3) was obtained from (2). I noticed the proof in B.1, but it seems that $Q_*^{\\pi}$ is only introduced afterwards. Might be good to explain what  $Q_*^{\\pi}$ denotes beforehand. \n- Minor typo in Lemma 3.3: 'Let $\\pi^*$ is' -> 'Let $\\pi^*$ be'", "questions_for_authors": "Please see my comments above. I don't have specific clarification questions at the moment.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v5896FADg5", "reviewer": "Reviewer_xoy5", "summary": "The paper proposes Policy-labeled Preference Learning (PPL) to mitigate what it calls “likelihood mismatch” in RLHF. The authors illustrate how policy labels (i.e., knowledge of which behavior policy generated which trajectory) can help disentangle suboptimal policy actions from stochasticity in the environment and thereby improve preference modeling. They further introduce contrastive KL regularization as a tool to align the learned policy with the policies associated with higher-preference trajectories. \n\n\n**Update after Rebuttal:** I'm keeping my positive score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewers for their time and thorough evaluation of our paper. We have organized our responses to your comments below. If any of our responses fail to address the intent of your questions or if you have remaining concerns, please let us know.\n\n**1. Paper position disconnection with introduction**\n\nTo emphasize that the PPL algorithm learns policies without explicitly learning a reward function, we mentioned in the Introduction that we are adopting the Direct Preference Optimization framework. Although the DPO paper, which introduced this framework, has been studied in the LLM domain, we noted in line 25 (right column) that its deterministic transitions make it an unsuitable setting for demonstrating the advantages of PPL.\n\n**2. Label usage in experiments**\n\nIn the experiments in Section 4.2, PPL assumed that policy labels are accessible. However, since extracting reliable policy labels is nontrivial, in Section 4.3 we designed and experimented with PPL-deterministic, which utilizes pseudo labels.\n\n**3. Analysis under noisy label**\n\nThank you for your comment regarding noisy pseudo-labels. We are currently conducting experiments in which we add Gaussian noise, $\\mathcal{N}(0, \\sigma^2)$to the ground-truth labels. In the button-press environment, we observed that performance degrades as the noise level $\\sigma$ increases. Specifically, at $\\sigma$=0.1, the performance was lower than that of PPL but higher than that of PPL-deterministic. However, at $\\sigma$=0.3, the performance dropped below that of PPL-deterministic. This indicates that if the pseudo-label is not sufficiently close to the policy label, its performance will be inferior to the proposed PPL-deterministic, thereby confirming that the pseudo-label used in PPL-deterministic is a reasonable alternative. \n\n**4. Likelihood mismatch and role of coverage**\n\nAs described in Figure 3 and in the section on likelihood misalignment, the likelihood mismatch issue does not primarily arise from dataset coverage. Instead, it occurs when the algorithm assumes that trajectories are always generated by the optimal policy. As shown in Figure 3, let us consider the black trajectory is generated by the optimal policy, while the red trajectory is produced by a suboptimal policy. Let's consider the case where all four trajectories—which ensure complete data coverage—are included in the dataset. Without the policy label, the estimated MDP appears as shown in the right panel of Figure 3, and it is evident that it still misinterprets the ground-truth MDP depicted in the left panel. \n\n**5. Misleading title**\n\nThank you for your detailed feedback. However, we believe that the scenario considered in our work—where data is generated by heterogeneous/diverse policies—is more justified and encompasses a broader range of realistic situations than merely assuming a homogeneous dataset. Previous studies using offline datasets have not taken into account the policy that generated the dataset. In contrast, our paper highlights this point and proposes an algorithm that leverages policy information, thereby emphasizing that in RLHF settings, both preference and policy information should be considered. We consider the current title appropriate, but if you feel that this context is not sufficiently conveyed, we would greatly appreciate any further suggestions. \n\n**6. Comparing with DPO and DPPO**\n\nSince the [Rafailov et al 2024] presents an algorithm designed for bandit settings, it could not be directly applied to sequential tasks such as robotic manipulation, and therefore was not compared separately. In the case of the DPPO, since CPL assumes a fixed standard deviation Gaussian for the policy, its loss simplifies to an L2 loss between actions—identical to the implementation used in DPPO [An et al. 2023]. We confirmed this similarity based on the official CPL GitHub repository, so no separate comparison was performed.\n\n**7. Other offline likelihood (policy mismatch) literature**\n\nTo the best of our knowledge, we have not found any offline RL algorithm that considers likelihood mismatch or policy mismatch. \n\n**References**\n* [Rafailov et al 2024] : Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization:Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.\n* [An et al 2023] : An, Gaon, et al. \"Direct preference-based policy optimization without reward modeling.\" Advances in Neural Information Processing Systems 36 (2023): 70247-70266.", "claims_and_evidence": "In my opinion, the main claims of the submission are well-supported.", "methods_and_evaluation_criteria": "The authors perform experiments on six MetaWorld environments and test both homogenous and heterogenous offline dataset (homogenous/heterogenous here refers to the generating behavior policies). MetaWorld is a robotics benchmark, which is a reasonable choice in the context of RLHF. However, the authors begin the paper with a discussion around LLM fine-tuning so that there is a slight disconnect between how the paper positions itself in the introduction and the type of experiments it conducts.", "theoretical_claims": "I read through the proofs of Lemma 3.2, 3.3 and Theorem 3.4 and found no issues.", "experimental_designs_or_analyses": "I read through the experimental details and superficially skimmed through the additional details in the appendix. Question to the authors: Could you please clarify whether in the experiments of Section 4.2 you assume access to the policy labels? This is not fully clear to me from the text.", "relation_to_broader_scientific_literature": "The paper can be viewed as a refinement of contrastive preference learning (CPL) by explicitly taking into account the behavior policy and the \"likelihood mismatch\" that can arise.", "essential_references_not_discussed": "As far as I am aware related work has been appropriately discussed.", "other_strengths_and_weaknesses": "### Strengths\n1. The authors highlight the issue of likelihood mismatch due to heterogenous behavior policies. \n2. Using the regret w.r.t. the behavior policy in contrats to the optimal advantage as in CPL appears like a natural approach to address the challenge of diverse generating policies. \n3. The experiments support the claims that incorporating policy labels can improve substantially in certain cases. \n\n### Weaknesses \n1. One key assumption is that either we know the precise policy that generated each trajectory or we can assign a \"pseudo-label\". In many real-world applications (especially large, unstructured offline data), extracting reliable policy labels is nontrivial. The paper briefly uses deterministic pseudo-labels but does not deeply analyze the accuracy needed or possible mismatch when pseudo-labels are very noisy as far as I am aware. \n2. A minor weakness is that that the role of coverage is not sufficiently discussed in the paper in my opinion. Likelihood mismatch seems to only be an issue when coverage is incomplete (or biased). Discussing this in more detail especially seems important around Figure 3.", "other_comments_or_suggestions": "In my opinion, \"Is Preference Enough for RLHF?\" is a slightly misleading title, as you are explicitly working under the premise that examples are generated using a heterogenous/diverse policies. You are not discussing the limitations of RLHF w.r.t. pairwise preferences more broadly. This part of the title could maybe be made less broad and more accurate.", "questions_for_authors": "The experiments focus on comparing the performance of PPL only with CPL, P-IQL and SFT. For the case of heterogenous data, PPL is only compared to CPL in the main text and additional plots shown in the appendix. Is there a reason why you are not comparing with DPO and DPPO? Are there other offline RL methods that explicitly address similar issue to the likelihood mismatch (i.e., policy mismatch) that you could compare to?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "96yaCIqtaL", "reviewer": "Reviewer_4oay", "summary": "This paper introduces PPL, a new method for learning policies from human preferences based on the regret based model of preferences. The authors note a key distinction between the regret model used in prior work (CPL by hejna et al.) and propose an improvement upon it to consider the current policy. The authors evaluate the method on a number of tasks with heterogeneous and homogenous policy data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewers for their time and thorough evaluation of our paper. We have organized our responses to your comments below. If any of our responses fail to address the intent of your questions or if you have remaining concerns, please let us know.\n\n**1. Final practical implementation**\n\nWe agree that it would be beneficial for readers to have a clearer view of the final implementation in the main text, and we will revise to address this.\n\n**2. Regret-based preference model attribution**\n\nThank you for clearly outlining the relationships with the literature. We will revise and add a reference to [Knox et al. 2022] to clearly attribute the origin of the regret-based preference model as referenced in CPL.\n\n**3. Including the meaning of theoretical results**\n\nWe appreciate your insight that providing intuitive explanations for our theoretical results can further enhance the reader's understanding. We have endeavored to include interpretive statements following each theoretical result. Could you please kindly indicate if there are specific parts of the theoretical results that you believe would benefit from additional clarification?\n\n**4. Theoretical justification**\n\n- **(Lemma 3.2):** \nWe have confirmed that [Rafailov et al. 2024a] derived the same theoretical results as [Rafailov et al. 2024b], and we will cite this in the paragraph explaining Lemma 3.2.\n\n- **(Theorem 3.4):** \nWhile [Shaikh et al. 2024] presented their results in a contextual bandit setting, our Theorem 3.4 extends this to MDPs. This extension is non-trivial due to the presence of stochastic transitions. Though [Zeng et al 2024] achieved results more similar to ours than [Shaikh et al. 2024], their proof applies only to deterministic transitions. Thus, Theorem 3.4 makes a distinct theoretical contribution by addressing environmental stochasticity.\n\n**5. Theorem 3.6**\n\nFirst, Corollary 1 of [Hejna et al. 2024], which motivated Theorem 3.6, showed that maximizing the optimal advantage is equivalent to learning the same optimal policy as maximizing the reward. In this context, Theorem 3.6 was written to make it intuitively easier to understand what kind of policy is learned from a reinforcement learning perspective when maximizing negative regret. The reason for considering only the preferred segment here is to interpret this intuitive meaning more clearly. For less preferred segments, the PPL objective can be expressed as:\n\n$\\arg \\min\\_{\\pi\\_{\\psi}}\\Big(\\mathbb{E}\\_{\\zeta^- \\sim \\mathcal{D}}[-Reg^{\\pi^-}\\_{\\pi\\_{\\psi}}(s^- ,a^-)-\\alpha \\log \\pi\\_{\\psi}(a^- |s^-)]\\Big) \\equiv \\arg\\max\\_{\\pi\\_{\\psi}}\\Big( \\mathbb{E}\\_{\\zeta^- \\sim \\mathcal{D}}[\\bar{D}\\_{KL}(\\pi^- || \\pi\\_{\\psi} ; s^-, a^-)]\\Big)$\n\nHowever, since the left-hand side represents cost minimization rather than reward maximization in traditional reinforcement learning, it is difficult to interpret intuitively. To avoid this ambiguity, we only dealt with the preferred segment perspective.\n\n**6. Dataset Generation and Evaluation Procedure**\n\n- **Dataset Generation:** During SAC training, we saved checkpoints of models showing 20%, 50%, and 100% success rate. We loaded policy model to generate 20K segments, and for dense cases, randomly sampled 2.5K segments. For labeling procedure, we used $V^{\\pi^*}(s_k)-V^{\\pi^*}(s_0) + \\sum_{t=0}^{k-1} r(s_t,a_t)$ from CPL, and calculated $V^{\\pi^*}$ using 64 MCMC samples with a 100% success rate critic $Q^{\\pi^*}$. For fair comparison, all algorithms were trained with the same labels, and when policy labels were needed in PPL, logits were calculated using policy information stored during rollout.\n\n- **Evaluation Procedure:** We conducted evaluations following the same method as CPL: (1) run 25 evaluation episodes, (2) calculate averages across 8 neighboring checkpoints, (3) average across 4 seeds, and (4) report the maximum values in the table.\n\nFor clarity of experimental design, we will add the provided response to the appendix.\n\n**7.Retrain the SAC oracle for accurate advantage estimation**\n\nIn the CPL dataset creation process, data segments are added to the replay buffer of the oracle SAC and then re-trained, which helps lower the TD error and improve the accuracy of advantage estimation. While we do not adopt the same procedure, we used only a single SAC oracle and trained all algorithms with the same labels. \nTherefore, it is difficult to conclude that the inaccuracy of advantage estimates caused performance degradation only for CPL. \n\nOne likely reason for CPL's struggle with noisy labels appears to be similar to the likelihood mismatch issue discussed earlier. Since CPL does not consider policy labels, it assumes all training data segments originate from an optimal policy. We suspect that when noisy labels occur, suboptimal data segments might be incorrectly treated as if they were generated by the optimal policy, potentially making the model more sensitive to noise and degrading its learning performance.", "claims_and_evidence": "The claims of the paper are generally well supported. The authors do a good job:\n* Explaining the differences with prior work\n* Deriving a new update rule\n* evaluation on a nubmer of tasks.\n\nThere are few missing citations that I believe should be added, and adjustments to be made which I left to the essential references section.", "methods_and_evaluation_criteria": "The authors present a nice small-scale example of where the previous preference model might be incorrect in Fig 3.\n\nThe method's derivation is intuitive and makes sense.\n\nOne weakness I would like to point out is that though the authors spend time deriving their method, the final practical implementation is not easily grasp-able from the main body of the paper, and is instead most easily found in the appendix. I would encourage the authors to include a derivation of the final objective within the text of the main paper, I think this would make the work much more understandable and stronger.", "theoretical_claims": "* The theoretical claims around preference models are supported by apt examples. Note that I did not carefully check all proofs provided in Section 3.2, but I did scan the derivations in the Appendix. I believe some of the results need proper attribution to prior works (see later section).\n\n* While the theoretical results are interesting, it was less clear form an intuitive perspective why we should care about the theoretical results. If the authors could provide more support for what insights these are. \n\n* Some of the results could be explained a bit more. Why is the expectation in Theorem 3.6 taken over only preferred segments? This does not seem to be explained.\n\n* I do not believe that all claims made desrve the level of distinction provided. For example, I'm not sure that Theorem 3.4 constitutes as theorem. A similar result was considered a Proposition in \"Show, Don’t Tell: Aligning Language Models with Demonstrated Feedback\" by Shaikh et al.", "experimental_designs_or_analyses": "**Reproduction** \n\nThe experimental design is good -- it considers two cases of preference data from heterogeneous or homogeneous policies. The authors went the extra mile reproducing the original results from CPL.\n\nHowever, the performance on the authors own datasets for CPL is far worse than on the original CPL datasets, in fact so much so that SFT often performed better than CPL. In particular, performance doesn't go up at all for CPL in Fig 5 even in the homogenous case. One question I have is if the authors re-trained the oracle policy with the behavior segments used for data added to the replay buffer as done in the original CPL paper to ensure more accurate advantage function estimates.\n\n\n**Performance**\nThe authors method performs favorably in comparison to prior methods.", "relation_to_broader_scientific_literature": "The authors sufficiently cover related work, aside from the points brough up in the next section of the review.", "essential_references_not_discussed": "**Regret-based Preference Model** \n\nThe regret based preference model was not first introduce by Hejna et al. It was instead introduce by Knox & Hatgis-Kessell et al. in \"Models of human preference for learning reward functions\" published in 2023. This was further explored in \"Learning optimal advantage from preferences and mistaking it for reward\" by Knox et al. It would be nice if the authors could attribute the regret-based preference model to these works, rather than CPL which used it to derive their algorithm. \n\n**Theoretical Justification**\n\n* The derivation of Lemma 3.2 bears high similarity to \"From  to : Your Language Model is Secretly a Q-Function\" by Rafailov & Hejna et al. published in COLM 2024, and should probably be attributed there.\n\n* Theorem 3.4 seems highly related to Proposition A.1. from \"Show, Don't Tell: Aligning Language Models with Demonstrated Feedback\" from Shaikh et al., published in ICLR 2024. The only distinction between this and Theorem 3.4 is the use of the full discounted MDP. However, up to notation and changing to the discounted KL, they are the same.", "other_strengths_and_weaknesses": "Strengths: The paper identifies a theoretical weakness, and develops a method to address it and demonstrates improved performance.\n\nWeaknesses: some questions remain around the dataset generation and evaluation procedure. The presentation could be improved, particulary in providing the final practical method and explaining the theory. Finally, some critical citations are missing.\n\nShould the authors address the weaknesses listed (which I believe should be very do-able, I am willing to raise my score.)", "other_comments_or_suggestions": "* Include a discussion of why the presented theory is valueble. If space is a limitation, I would rather have this in the main text and delegate some of the intermediary results to the appendix.\n\n* Include the final practical objective in the main text. I believe this is currently in a later section of the appendix.", "questions_for_authors": "* Why is CPL performance on the re-created datasets so low if the authors used the same procedure?\n* Could the authors clarify if they trained their label-generating oracle policy with the trajectories used to sample preferences in the replay buffer of the SAC policy? This would lower the TD-error when estimating the advantage.\n* Why are only positive segments used for Theorem 3.6?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes an interesting regret based metric for modeling human preferences. All the reviewers agreed that the theoretical work is sound and the experimental results highlight the importance of the work. The proposed model is a viable alternative to existing advantage-based preference model, and can also be useful beyond deep RL settings.
Decision: Accept (spotlight poster)