Offline Preference-Based Value Optimization
Authors: Hyungkyu Kang
Min-hwan Oh
Venue: ICLR 2026 Poster (2026)
Keywords: offline reinforcement learning
preference-based reinforcement learning
Primary area: reinforcement learning

Abstract: We study the problem of offline preference-based reinforcement learning (PbRL), where the agent learns from pre-collected preference data by comparing trajectory pairs. 
  While prior work has established theoretical foundations for offline PbRL, existing algorithms face significant practical limitations: some rely on computationally intractable optimization procedures, while others suffer from unstable training and high performance variance.
  To address these challenges, we propose Preference-based Value Optimization (PVO), a simple and practical algorithm that achieves both strong empirical performance and theoretical guarantees.
  PVO directly optimizes the value function consistent with preference feedback by minimizing a novel \emph{value alignment loss}.
  We prove that PVO attains a rate-optimal sample complexity of $\mathcal{O}(\varepsilon^{-2})$, and further show that the value alignment loss is applicable not only to value-based methods but also to actor–critic algorithms.
  Empirically, PVO achieves robust and stable performance across diverse continuous control benchmarks. 
  It consistently outperforms strong baselines, including methods without theoretical guarantees, while requiring no additional hyperparameters for preference learning.
  Moreover, our ablation study demonstrates that substituting the standard TD loss with the value alignment loss substantially improves learning from preference data, confirming its effectiveness for PbRL.

Published as a conference paper at ICLR 2026
OFFLINE PREFERENCE-BASER VALUE OPTIMIZATION
Hyungkyu Kang
Min-hwan Oh Seoul National University Scoul, South Korea minoh@snu.ac. ac. kr
Scoul National University Seoul, South Korea
hyungkyu6 matpkyyu@19@ssu.cac ac.
ABSTRACT
We study the problem of offline preference-basee reinforcement learning (PbRL). where the agent learns from pre-collected preference data by comparing trajectory pairs. While prior work has established theoretical foundations for offline PbRL, existing algorithms face significant practical limitations: some rely on compu- tationally intractable optimization procedures, while others suffer from unstable training and high performance variance. To address these challenges, we propose Preference-based Value Optimization (PVO), simple and practical algorithm that achieves both strong empirical performance and theoretical guarantees. PVO di- rectly optimizes the value function consistent with preference feedback by min- imizing a novel value alignment loss. We prove that PVO attains a rate-optimal sample complexity of () and further show that the value alignment loss is applicable not only to value-based methods but also to actor crit-ccitcc algorithmss Empirically, PVO achieves robust and stable performance across diverse continu- ous control benchmarks. It consistently outperforms strong baselines, including methods without theoretical guarantees, while requiring no additional hyperpa- rameters for preference learning. Morcover, our ablation study demonstrates that substituting the standard TD loss with the value alignment loss substantially im- proves learning from preference data, confirming its effectiveness for PbRL.
INTRODUCTION
One the major challenges in reinforcement learning (RL) is designing suitable reward func- tions for real-world tasks. Reward design often requires costly instrumentation such motion capture (Akkaya et al. 2019; Peng et al. 2020), and poorly designed reward functions can signif- icantly degrade training performance. Preference-base compelling alternative by inferring the underlying reward signal from preference feedback, such as human comparisons between trajectories (Christiano al. 2017). This framework has demonstrated its effectiveness in domains where direct reward specification is difficult, including robotics (Brown)
reinforcement learning (PbRL) provides
al. 2019 Shin al. 2023), games (MacGlashan al. 2017 Warnell al. 2018), and language
models (Ziegler et al. 2019 Stiennon et al.) 2020 Ouyang et al. 2022).
In this work, we study offline PbRL, where learning is conducted using pre-collected datasets of trajectories and preference feedback (Kim et al.] [0023] An et al.) [023] Hejna & Sadigh [2024). Offline RI. (Levine et al. (020) is advantageous in scenarios where real-time online interaction may be costly or unsafe. This consideration is especially relevant for PbRL., collecting preference
feedback interactively can be prohibitively expensive or impractical.
Prior work on offline PbRL has introduced several algorithms with sample complexity guaran- tees (Zhu et al. 2023 Zhan et al. [2024a] Pace et al. 2025 Kang & Oh 2025). However, these meth- ods face significant practical limitations. Some are restricted to linear function approximation (Zhu et al. 2023), some require solving computationally intractable optimization problems (Zhan et al. 2024a), and others exhibit unstable performance in practice (Kang & Oh 2025).
In particular, Zhan et al. (202a)) formulate PbRL as a distributionally robust optimization problem:
f E arg max
min PER(D), PEP(D)
V VI, V/a

---

Published as a conference paper at ICLR 2026
where R(D) and P(D) are confidence sets for the reward and transition models. This formulation is computationally infeasible, primarily because the inner minimization over the confidence sets R(D) and ((D) requires searching over complex function classes. The overall joint optimization over policy, reward, and transition models further compounds the computational burden.
More recently. Kang & Oh (2025) proposed actor-critic-style PhRL algorithm, APPO. By formulating the distributionally robust optimization into a regularized optimization, their approach enables practical implementation. However, APPO achieves a suboptimal sample complexity bound of (e which is weaker than the (ss bound of Zhan et al. (2024a). This inefficiency arises from its reliance on standard actor-critic analysis in offline RL (Xie al. 2021| Zanette al., 2021 Cheng et al., 2022] Nguyenn Tguuen-TTang & Arora (2023), which requires bounding the cumulative con- servatism bias across 7' iterations. More importantly, APPO often exhibit from high performance variance and unstable training. Even with hyperparameter tuning, it can fail to learn effective poli- cies (see Section (5). These issues further motivate the need for more stable and efficient alternative for offline PbRL.
To address these challenges, we propose Preference-based Value Optimization (PVO), an offline PbRL algorithm that achieves both strong empirical performance and theoretical guarantees. PVO directly optimizes the value function by minimizing a novel value alignment loss, in conjunction with the concept of the induced reward function. Leveraging this formulation, we establish that PVO achieves a rate-optimal sample complexity bound of (s--)).
Beyond introducing PVO, revisit APPO (Kang & Oh 2025) and demonstrate that its variant incorporating the value alignment loss also admits a sample complexity guarantee. This finding highlights that the value alignment loss serves as unifying principle for provably efficient PbRL, applicable to both value-based and actor-critic algorithms.
We evaluate PVO high-dimensional continuous control benchmarks. Surprisingly, PVO consis- tently outperforms state-of-the-aad baselines, including empirical methods that lack theoretical guar- antees. It is notable that PVO exhibits robust and stable performance across various datasets, without introducing additional hyperparameters for preference learning. Furthermore, our ablation study demonstrates that replacing the standard TD loss with the value alignment loss improves the perfor- mance of RL algorithms applied to preference datasets, thereby validating the advantage of value alignment loss in PbRL. Our contributions are summarized as follows:
Algorithm. We propose PVO, a simple and practical offline PbRL. algorithm that achieves both strong empirical performance and theoretical guarantees. It directly optimizes the value function consistent with preference feedback by minimizing the novel value alignment loss.
Theoretical Guarantee. We prove that PVO attains a rate-optimal O sample complexity bound (Theorem 4.1.. We further show that APPO (Kang & Ohl 2025) can be modified to incorporate the value alignment loss, demonstrating that this loss is applicable to both value- based and actor-critic algorithms for provably efficient PbRL.
Empirical Performance. We show that PVO outperforms ttate-oftthe-s baselines on contin- uous control benchmarks. Notably, it maintains robust and stable performance across datasets where existing methods exhibit high performance variance.
Advantage of Value Alignment Loss. Our ablation study reveals that replacing the standard TD loss with the value alignment loss significantly improves the performance of RI. algorithms on preference datasets. This confirms that the value alignment loss provides more reliable learning signals in PbRL, where reward estimation errors are often unavoidable.
1.1 RELATED WORK
Offline PbRL Theory. In offline RL, ensuring a proper amount of conservatism in value or model estimates is essential for theoretical guarantees. The principle of conservatism still holds for offline PbRL, yet we also estimate the reward from preference feedback.
To deal with this challenge, Zhu et al. 2003) utilize pessimistic maximum likelihood estimation un- der linear function approximation r(s, a) 0 "0,(s, a)). Their algorithm first constructs a confidence set for the model parameter and then performs distributionally robust policy optimization to obtain conservative value function and corresponding policy. /Zhan al. (2024a) extended the idea to
2

---

Published as a conference paper at ICLR 2026
general function classes with bounded bracketing number. They showed that bounded trajectory- level concentrability is essential for offline PbRL by establishing a lower bound. While they provide sample complexity bound, their algorithms computationally intractable. Recently, Kang & Oh (225) developed a computationally efficient approach based on actor-critic style policy opti- mization. By framing PbRL as two-player game between policy and reward model, they replaced the distributionally robust optimization with tractable regularized optimization. A different yet re- lated problem setting was studied by Pace al. (2025). They developed a preference elicitation method for offline PbRL. By choosing trajectory pairs for preference queries, they eliminated the dependence on the reward concentrability coefficient.
Empirical Studies in PbRL Several works have explored applying deep learning techniques to preference-basee reinforcement learning. A simple yet effective approach involves training a re- ward model on preference dataset, then applying a standard RI. algorithm using the reward signal predicted by the learned model (Christiano et al. 2017, Ibarz al., 2018: Lee et al.] (02T). The reward model is typically assumed to produce Markovian reward in conventional RL, although some works have considered non-Markovian rewards that depend on the entire trajectory (Kim et al. 2023
Zhang et al. 2024 Swamy et al. 2024).
A separate line of research seeks to learn value functions or policies directly from preference data, without explicitly modeling rewards. Hejna & Sadigh (2024] Hejna al. (2024) derive direct relationship between the preference distribution and value (or policy). and optimize the likelihood of observed preferences accordingly. [n et al. (2023) use scoring function that evaluates policy based on preference, while Kang et al. (2023) propose hindsight information matching to directly optimize the policy. Zhang al. (2024) introduce generative model that learns from positive/negative trajectory pairs and applies behavior cloning to the model-generated positive trajectories.
Another active area of research focuses on improving the efficiency of preference data collection. Lee et al. (2021) demonstrate that increasing trajectory diversity through unsupervised pretraining improves performance, and Liang al. (2022) achieve similar goal via uncertainty-baat based explo- ration. Park et al. (2022) propose data augmentation techniques tailored for PbRI., while Hejna III & Sadigh (2023) introduce meta- meta-llen framework for few-shot preference learning. [Choi et al. (2024) extend the standard pairwise comparison setting to listwise comparisons, showing that this richer feedback provides more informative supervision.
PRELIMINARIES
Markov Decision Processes. We consider an episodic MDP (S,,, with state space S, action space A, and horizon H. p* [Pr M== ) are the transition probabilities. and r* = {r T== are the reward functions. For each episode, the agent starts at the initial state s and then interacts with the environment for H steps. At step h E [H], the agent takes action as based on the current state S/. The environment assigns reward (sh, ax)) and generates next state 8h+1 following the transition probability (- Sh, an). preference-basee learning, the agent does not observe rewards at each step, but preference feedback comparing pair of trajectories, as we will discuss.
The agent's strategy for taking actions represented by policy x {TT, Jne(r) where Tt(: ()) is probability distribution over A. We define the state value function and the action value function of policy и as the expected sum of rewards over an episode, following the policy æ. Formally,
H Σ
VV, (3) = Ex
Σ =
rsw aNd sh Q7, == Eg
rh(sw, aw) sh 8, ab
We write V as VTT and VIT (s1) = Vi" for convenience. For any policy क and reward r, the Bellman equation states the relation between state and action value functions as
2h, = (x + P*V (s, a), (s) = QT, T), (i)+(() = 0. where we write Pg(s, a) = E. and f(s x)) : E [f),
Offline Preference-based Reinforcement Learning. In Preference-based RL, the agent cannot observe the true reward y* but only binary preference feedback over trajectory pairs. For mono-
Our analysis naturally extends with initial distribution.
3

---

Published as a conference paper at ICLR 2026
tonically increasing link function + :::. + [0, 1] with bounded 1/(inf Ramd ('(x)),
assume the preference feedback ym {0, } is generated by the following model:
P(y = 1 7 = P( is preferred over -0) ofr (T') -r)
where ("(r) - 5 (sh, oh) for given trajectory T = ($1 a1s.... SH ai)). The widely used
Bradely-Terry-Luce model (Bradley & Terry 1952) is special of this model where is set to be the sigmoid function o(x) 1/(1 exp -x))-
We have offline datasets: preference dataset DpF {[rm., y. " y yy) m= and trajectory dataset DT = {(T".0 (or""" ) where every trajectories are sampled i..d. by executing the ref- erence policy μ.. The distinction between Dpp and DT for notational convenience. Generally, the two datasets may have common samples, e.g., we have large Drs and get labels for some trajectory pairs to create Dpy. Our goal find an c-optimal policy क with performance gap VT,,- (s1) Vi..- (s1) S for an optimal policy n².
Function Approximation. We define function classes that we use to approximate models and value functions. We have reward function class R R1 X
Rh C (8 A (- --Rmax: Rmax 1) transition function class P = P1 x .... x Pa C (S x A + A(S)) and function class F F x..... FH (S x X - [-V Vmax: Vmmx. 1)a We denote Ty for the greedy policy corresponding to f. i.e., (s) arg max,, ['(x, a), and define Vi {Vr. he(m) as Vy(s) [a(s, (s)) for all E S. As we do not make any assumption on structure of the function classes, the function classes can approximate complex structures such as neural networks. We define I as the set of greedy policies corresponding to F.
Additional Notations.
We denote [n] == {1,2.... n) for N. For given dataset D, use
(feD[f(z)) to denote
- ExED f(z). For reward function r E R and trajectories T², we
write ((r; ,0,7, ,
r((7) r(rt). For f A + R. use the notation PM fs,
E (s))))) [f [a)]_ The notation f(x) S g(x) means that f(x) S Cg(x),, Ve for some
absolute constant C. >
ALGORITHM
In this section, we discuss how learn value function that consistent with preference feedback. The key idea lies in the concept of induced reward function and our novel value alignment loss. Building on this idea, we propose Preference-based Value Optimization (PVO), simple offline PbRL algorithm with a sample complexity guarantee. We also show that the APPO algorithm introduced by Kang Ohl (2025) can be interpreted as an actor-critic algorithm using the value alignment loss, which implies a unified framework for PbRL.
3.1 ALIGNING VALUE FUNCTION WITH PREFERENCE
In preference--aare based reinforcement learning, feedback is provided for each trajectory pair (rmh.0, T.. To enable credit assignment, must estimate reward function, which can be per- formed by maximum likelihood estimation (MLE) over the dataset Dpp. Specifically, we train a reward model f E arg mingge LRW(r) where
Law (r)
Σ log (22 ()(r(7m.1) r()m)-)))
is the negative log likelihood loss. The standard MLE concentration bound (e.g., Lemma 2 in Zhan al. (024a)) guarantees the following:
x² [lg(([R(8-))
Epo -()
M
A key point is that this concentration bound is valid with respect to trajectory pairs (7", T)) ~ rather than each state transition. Consequently, the squared Bellman error used in standard RL not compatible with PbRL. Then how can we leam value functions consistent with preference feedback? The concept of induced reward function plays crucial role.
4

---

Published as a conference paper at ICLR 2026
Algorithm PVO: Preference-based Value Optimization
1: Input: Datasets Dpp {(TTH.) mm, 1 -15""))m= DTs {(T",), -
2: Estimate i argmin l. (R)(r) ①. Ph E arg
( (P) for forall h [M]
3: Optimize f arg n=1 (fy(T (y((" -(T")0) + f(("))))²
4: Return greedy policy = = such that n g - arg max, (1, a) for 8 E
Definition 1 (Induced Reward Function, Value Type). For f E F, we define the induced reward H function r { Th=1 (S x - IR)" satisfying Sh.5 = fn. - -P. PP.11.. Similarly, we define f as Fh.f = fh. Ph. 1 where P some transition model.
Our value type induced reward is different from the policy type induced reward used for the anal- ysis of actor-critic algorithms (Zanette al., 2021: Xie al. 2021 Nguyen-Tang Arora 2023 Kang & Oh 2025) (Formal definition is presented in Definition 3). While the policy type def- inition is rooted in the Bellman equation O = r + PT with respect to some policy n, our value type definition is inspired by the Bellman optimality equation QT, (s,a) = r (s,a) + Ex'Py |maxx 1 (s', (s', The value type induced reward enables directly optimize the value function without actor-critic iterations as in KKng & Oh (2025). leading to more stable and efficient learning,
will see in Section 5
Equipped with the definition of induced reward function, we introduce the value alignment loss, a simple loss function for consistent value learning:
()
R)
Σ - -(r")))) + )()")))) n=1
At first glance, Lva can be viewed as the squared trajectory-level error between the induced reward function Ty and the reward model f. While this view valid, more structured interpretation is obtained by Definition I
N
H
2
(v Σ
E(n Fr 7 ( 00 E(h An PE P of
This expression reveals that Lva represents the difference in the cumulative Bellman errors of f between a pair of trajectories. Minimizing Lu encourages f to be Bellman-consistent with respect to f, thus aligning it with the preference data. We therefore refer to Lva as value alignment loss.
3.2 PREFERENCE-BAR BASED VALUE OPTIMIZATION
We present PVO, a direct application of the value alignment loss with Definition] 1 The podudo-code is presented in Algorithm
Model Learning. Our algorithm consists of two phases: model learning (Line and value optimization (Line 3). In the model learning phase, we train reward and transition models via maximum likelihood estimation. Formally, compute arg min Lxw(r) arg min min, LIRR(()) where
I and Ph.
irR = Σ
Σ
log P(s (8) 3.7...7)) (...).
(3)
n=1je(0.1)
Value Optimization. the value optimization phase, we minimize the value alignment loss:
E arg min FEF
Σ (f ff( f(7".0) 2
(4)
value alignment loss Lov (P.)
5

---

Published as a conference paper at ICLR 2026
As we discussed, this facilitates the consistency of the value function of with respect the reward estimation, Since 4 is unconstrained, we can easily implement using neural networks and off- the-shelf gradient-based optimizers. We present practical deep RL implementation in Section
3.3 REVISITING APPO WITH VALUE ALIGNMENT LOSS
The APPO algorithm (Kang Oh 2025) alternates between policy updates of the form Th (a
8) (a 8) exp(nf[(s a)) and value function optimization:
I' E arg min fEF
Σ
Σ
Σ
[n - - fn (0% 0 + εUS)
where 8(f) = = (7 (- FT' (T"1) f(T=')) + f(T is the l loss between the policy
type induced reward fh.y and the reward model P. This can be viewed an & variant of the value alignment loss Lva, leading natural question:
Question: Can we still guarantee sample complexity bound if ((}) is replaced by Lva in APPO?
We answer this affirmatively: if APPO is modified to use value alignment loss as in Algorithm 2 it enjoys sample complexity guarantee (Theorem B.T). The detailed discussion and analysis IS presented in Appendix [ This suggests that the combination the value alignment loss and the induced reward function provides unified framework applicable to both value optimization and actor-critic methods.
3.4 PRACTICAL IMPLEMENTATION
PVO can be practically implemented with neural networks empowered by off-the-shelf deep learning methods. We adapt PVO to the standard discounted MDP setting for deep PbRL (Christiano et al. 2017). where we have preference feedback for trajectory segment pairs of length L.
Reward Learning. Since the value optimization objective uses a reward model A, we train reward model based the preference dataset Dpp. We train f by maximizing log likelihood Lgw(DD (pp)). ). yet it is possible employ other advanced techniques for preference learning (Park et al. 2022 Shin et al. 2023 Hwang et al.] 2024) Choi et al.) (024).. We note that reward model training requires minimal computational cost: In experiments, training reward model with 1000 preference samples takes less than a minute, while value and policy learning take more than hours for all algorithms.
Value Optimization. To implement the value optimization (Line 3 in Algorithm with deep neural networks, we parameterize Q and V functions separately. The V function is trained via expectile regression (Kostrikov et al. 2022):
Lv (Dr) = E,,))D [L](Q(s, a) - (s))
(5)
where Lī(u) = - {u < 0} 1u22 The optimization objective for the Q function is
(D(D = ))r [rqq (((()))rr((())) (() +(())))())) ((
(6)
where TQ.v(7) ΣL_ (Q())))) -V and T (s) is an action sampled from x(- 8). We use ((8+11) instead of (V a;). eliminating the need for training a transition model. This approximation leads to good empirical performance, as shown in the experimental results.
Finally, the policy is extracted via advantage weighted regression (Peng et al. 2019):
(Pp) = E,,,) (exp(3(Q(s, - V(s)) log T(a {))
THEORETICAL ANALYSIS
This section presents the sample complexity analysis of proposed algorithms, PVO. Our analysis
based on some standard assumptions in the PbRL literature. First, we assume that the function
classes are realizable (Chen et al. 2023 Zhan al. 2024a Pace et al. 2025 Kang & Oh 2025)

---

Published as a conference paper at ICLR 2026
Assumption (Reward Function Class). The reward function class is realizable, i.e., 1" E R. For every r E R and trajectory т, it holds that (r(T) S Rmax-
Assumption (Transition Function Class). The transition function class is realizable, i.e,, P E E.P. Assumption (Value Function Class), For any policy n, we have Q" E FF. .I addition, [fn(s, - Vmax for all f E F, h [ and (s, a) E E x A.
We define PbRL version of uniform concentrability coefficient Munos & Szepesvári (2008); Chen & Jiang 2019). Note that is defined with respect to trajectory density instead of itate-cction density. Definition (PbRL. Uniform Concentrability).
B ( (())) +r
CF(
sup sup +ETlz FEF v E V (r)(T)) - - -())
Now we present the sample complexity bounds. The proof' are presented in Section C Note that our analysis naturally extends to infinite function classes using the standard covering number argument.
Theorem 4.1 (Sample complexity of PVO). Suppose Assumptions
and and3han hold. With probability
at least - 8, Algorithm Jachieess an e-optimal policy with
M = o C/ (2)FF k² log (R|8
) .N -O (216-))) (Cu(F)PVi - H² (log (log( (P H0 log( log([FF8 [18
2²
8²
Compared to the sample complexity bounds of FREEHAND--ransitton (Zhan et al. 2024a), the bound in Theorem is looser since the uniform concentrability C, is stronger condition than the single-policy concentrability (Zhan et al. 2024a Kang & Oh 2025). However, the sample complex- ity bound of PVO still achieves rate-optimal O (E for both M and N. Moreover, the sharp bounds SEGEHHADD-raanstionn is computational intractable due its reliance on a distributionally robust optimization oracle. Compared with APPO (Kang & Oh 2025), PVO's bound on N is sharper than O (e-1) bound of APPO, but APPO relies on the weaker single-policy concentrability assumption. We note that APPO also has empirical limitations such unstable performance and additional hyperparameters as explained in Section 5 Therefore, the sample complexity bound of PVO reveals a trade-off: despite having a weaker bound, PVO offers significant advantages in terms of practical implementation and empirical performance.
of
comes at the cost:
5 EXPERIMENTS
In this section, we evaluate PVO in continuous control benchmarks with elaborate ablation study.
5.1 EXPERIMENTAL SETUP
We evaluate PVO the Meta-World Yu al. 2020) and DMControl (Tssa et al., 2018) datasets from Choi et al. 2024). They are widely used continuous control benchmarks with sighdimmessiona state spaces. We mainly use Meta-World datasets for evaluation, and the results with the DMControl dataset is presented in Appendix F We follow the experimental setup Choi et al. (2024) and Kang & Oh 2025). The preference dataset consists of pairs of randomly sampled trajectory segments of length 25, and the preference labels are generated based on the ground truth return of segments. We measure algorithm performance using the success rate for Meta- World tasks and the episodic return for DMControl tasks.
We consider the following baselines: (1) IQL. (Kostrikov et al. 2022 with a learned reward model is simple yet strong baseline that has been widely adopted in previous studies (Kim et al.) 2023 An et al. 2023 Hejna & Sadigh 2024 Hejna et al., 22024] Choi et al. 2024); (2) APPO (Kang &
Oh 2025) is provably efficient algorithm based on adversarial training; (3) Preference Transformer
(PT) (Kim et al. 2023) utilizes the Transformer (Vaswani (017)) architecture for sequential reward
modeling: (4) DPPO An et al.) 2023) directly optimizes policy with preference score metric; (5)
IPL (Hejna & Sadigh 2024) optimizes policy by maximizing the likelihood of observed preference
data. Further details the setup presented in Appendix G
7

---

Published as a conference paper at ICLR 2026
dial-turn
drawer-opan
lever-pull
RateeQ
8
Sweep-into
button-press-loooee
button-press-looe prs.bo--oss-todddwwwalll
Bervedo
80
PT
0990
PV0
DAPO IPL
PVD
D990
PVD
IPL
box-clase
dial-tum
drawer-ape
lever
108
100
fart
16
PVD
AP10
PVD
10L
AFPO
PVD
AMPO
PVD
APPO
sweep-into
sweep
cuiton-press-topdeee
button-press wwwpssstoddowwwweeeee
1:0
100
PVD 10L
AP70 PVD
101
APPO
PVD
I04
IPPD
PVD
10.
APPO
500 prefernnce teecbark
1030 pretereace feedbork
datasets and (Bottom two rows) medium-exper" datasets, measured by success rate. For medium-replay datasets, we in- clude the results of PT, DPPO, and IPL from Choi et al. (2024). For the medium- expere datasets, we evaluate the top three algorithms from the medi um medium-replay replay datasets. Each plot shows the mean and standard deviation over five random seeds.
Figure 1: (Top two rows) Performance Meta- Meta-World World
Meta World medium repl
Mcta World medium
(%)
(%))
80
PVO
101.
APPO
PT
DPPO
IPL
PVO
IGL.
AIPO
Figure 2: Average performance on SeradWorld medium wwwuumlrrrplyyeeeeeee ay and medi experp datasets.
5.2 EVALUATION ON PREFERENCE DATASETS
Figure presents the performance of algorithms on Sorad datasets, and Figure summarizes the overall results. PVO consistently outperforms baseline methods across diverse environments. In particular, while baselines often exhibit high variance across datasets, PVO maintains robust per- formance. For example, IQL performs comparably to PVO on the medium-reples sweep dataset, but fails to learn on the medi reppla button-press press-topdown dataset. This instability arises from the nature of preference feedback, where agents rely potentially misspecified reward estimation. The stable performance of PVO indicates greater robustness to such reward model errors, which is significant advantage in the PbRL setting.
8

---

Published as a conference paper at ICLR 2026
Advantage Value Alignment Loss
Effect Feedback Dataset Size
Effect of Data Quality
VQL+VA (PVO)
IQL
Success Rate/%
Ratecch)
+D3+BC+VA
TD3+BC
40
XQL+VA
PVD
IQL
IOL
XQL
APPO
APPO
520
0.3
Success Rate (%)
of feedback
Ratio Random Data
Figure 3: (Left) Overall performance of IQL, XQL, and TD3+BC with and without value alignment loss V+ indicates the modified version using the value alignment loss), aggregated by interquartile mean (Agarwal et al.] (022]. (Middle) Performance of algorithms on Meta- Meta-World medi um renlla dial-turn dataset, with varying number of preference feedback. (Right) Performance of algorithms on mixture datasets of random and expert trajectories, with varying proportions of random trajectories.
Moreover, PVO introduces no additional hyperparameters for preference learning, requiring exactly the same set of hyperparameters as IQL. This stands in contrast to existing PbRL methods that depend on extra hyperparameters, such as the conservatism parameter in APPO, the smoothness and conservatism regularizers DPPO, and the regularization parameter in IPL..
5.3 ABLATION STUDY
Advantage of Value Alignment Loss.
The improvement PVO over baselines stems from the value alignment loss (6). To isolate its effect, implemented additional baselines: TD3+BC (Fujimoto & Gul 2021 (actor-critic) and XQL (Garg et al. 2023) (value-based). We compared their original versions, which use the standard TD loss, with variants that employ the value alignment loss.
The left plot of Figure 2 reports results on the Seta-World medium-replay datasets (8 tasks) with 1000 preference feedback. For both TD3+BC and XQL, replacing the TD loss with the value align- ment loss yields substantial improvements. similar pattern appears in the gap between PVO and IQL,, which share identical network architectures and expectile regression but differ in the use of the value alignment loss. These results confirm that the value alignment loss provides a more informa- tive learning signal than the standard TD loss.
From a theoretical perspective, we hypothesize that its advantage lies in mitigating the propaga- tion of reward model errors. Unlike the TD loss, which can amplify such errors through Bellman backups, the value alignment loss distributes errors at the trajectory level, smoothing their impact on value estimation. This property may explain its empirical effectiveness PbRL, where reward model misspecification is often inevitable.
Effect of Preference Dataset Size and Data Quality.
We next study sensitivity to the amount of preference feedback. The middle plot of Figure 3 shows that PVO achieves effective learning with as few about 100 preference samples, exhibiting minimal performance degradation.
We also examine the effect of data quality. On the Meta- Woold dial-turn task, we constructed mixture datasets by combining expert and random trajectories with varying proportions r {0, 0.25,, 0.5, 0.75, 1}. Here, = 0 corresponds to an expert dataset, while = denotes fully random dataset. The result is presented in the right plot of Figure 3 As expected, performance declines for all algorithms as r increases, since a larger proportion of random trajectories reduces the amount of high-quality data available for learning. Importantly, PVO consistently maintains periority across all mixture settings, demonstrating robustness even when the dataset substantially diverges from the optimal policy distribution.

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
The experimental details are described in Section 5 and Section including hyperparameters, neu- ral network architecture, and dataset information. The code used run experiments can be found in the supplementary material where the README file explains how to configure training environ- ment and execute scripts. The Meta-Wood World medi um cam-aally datasets and DMControl datasets are available in the official repository of Choi et al. (2024). The Meta- World medium: expeert datasets are generated using the script provided in the official repository of Hejna & Sadigh (2024).
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant and the Institute of Information & communications Technology Planning & Evaluation (ITTP) grant both funded by the Korea government (MSTT) (No. RS-2022-NR071853 RS-2023-00222663, RS-2025- 25463302) and by Al-Bio Research Grant through Scoul National University.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 34:29304-299333 2021.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint 1rXiv:1910.07113. 2019.
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, and Hyun Oh Song. Direct preference-based policy optimization without reward modeling, Advances in Neural Infor- mation Processing Systems, 36:70247-70266, 2023.
Philippe Artzner. Thinking coherently. Risk, 10:68-71, 1997.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Jinglin Chen and Nan Jiang. information-theeredl considerations in batch reinforcement learning.
In International conference machine learning, pp. (042-1055) PMLR, 2019.
Xiaoyu Chen, Han Zhong, Zhuoran Yang. Zhaoran Wang, and Liwei Wang. Human-in momnaa--i-thh-looe loepp: reinforcement learning with general function approximation.
Provably efficient preference-ber In International Conference on Machine Learning, pp. 3773-3793. PMLR, 2022.
Yu Chen, Yihan Du, Pihe Hu, Siwei Wang, Desheng Wu, and Longbo Huang. Provably efficient iterated evar reinforcement learning with function approximation and human feedback. In The Twelfth International Conference on Learning Representations, 2023.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference Machine Learning, pp. 3852- 3878. PMLR, 2022.
Heewoong Choi, Sangwon Jung, Hongjoon Ahn, and Taesup Moon. Listwise reward estimation for offline preference-based reinforcement learning. In Forty-first International Conference on Machine Learning. 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- fems, 30, 2017.
10

---

Published as a conference paper at ICLR 2026
Christoph Dann, Mehryar Mohri, Tong Zhang, and Julian Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in Neural Information Processing Systems, 34:12040-12051 2021.
Yihan Du, Anna Winnicki, Gal Dalal, Shie Mannor, and R. Srikant, Exploration policy op- timization in RLHF: Theoretical insights on efficient data utilization. In Forty-first International Conference on Machine Learning, 2024.
Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:21 13487, 2021.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20155, 2021.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL
without entropy. In The Eleventh International Conference on Learning Representations, 2023.
URL httptt/////peeeeeiiee
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, 1861-1870. Pmir, 2018.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 2024.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without rein- forcement learning. In The Twelfth International Conference on Learning Representations, 2024.
Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for -uman-in-the-loop rl. In Conference on Robot Learning, pp. 2014-2025. PMLR, 2023.
Minyoung Hwang, Gunmin Lee, Hogun Kee, Chan Woo Kim, Kyungjae Lee, and Songhwai Oh. Se- quential preference ranking for efficient reinforcement learning from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018
Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34:13406-13418, 2021.
Hyungkyu Kang and Min-hwan Oh. Adversarial policy optimization for offline preference-based reinforcement learning. In The Thirteenth International Conference Learning Representations, 2025. URL http::///p/eeeeee ew net (tee//formmm1--SS9TT...222
Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference-guided policy optimization. International Conference on Machine Learning, pp- 15753-15768. PMLR, 2023.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Pref- erence transformer: Modeling human preferences using transformers for RL. In The Eleventh International Conference on Learning Representations, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations (ICLR), 2015. https::/://a org/ abs/ 1412. 6980 arXiv preprint arXiv: 141iv:1412.69800
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference Learning Representations, 2022.

---

Published as a conference paper at ICLR 2026
Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-effcieet interactive reinforcement learning via relabeling experience and unsupervised pre-training In International Conference on Machine Learming, pp. 6152-6163. PMLR, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprini arXiv:2005.016653 2020.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representa- lions, 2022. URL htttttt:///ppereeeeeee wwwp:nt//ooommmido//00000
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng. Guan Wang, David L Roberts, Matthew Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International conference on machine learning, pp. 1285-2294 PMLR, 2017.
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(), 2008.
Thanh Nguyen- Tang and Raman Arora. On sample-efficient offline reinforcement learning: Data diversity, posterior sampling and beyond. Advances in neural information processing systems, 36: 61 61115-61157, 2023.
Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sam- pling for preference-based reinforcement learning. In Conference on Uncertainty Artificial Intelligence, pp. 1029-1038, PMLR, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744 2022.
Alizée Pace, Bernhard Schölkopf, Gunnar Ratsch, and Giorgia Ramponi. Preference elicitation for offline reinforcement learning. The Thirteenth International Conference on Learning Repre- sentations, 2025.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL
https://openrrevie. .nt
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage--eigghted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 1rXiv:19100000/7,,200
2019. 2019.
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Lee, Jie Tan, and Sergey Levine. Learn- ing agile robotic locomotion skills by imitating animals. arXiv preprint wrXiv:2004.00784, 2020.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404. 037115, 2024.
Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013.
Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Dueling rl: Reinforcement learning with trajec- tory preferences. In International Conference on Artificial Intelligence and Statistics, pp. 6263- 6289. PMLR, 2023.
Daniel Shin, Anca Dragan, and Daniel S. Brown. Benchmarks and algorithms for offline preference- based reward learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
12

---

Published as a conference paper at ICLR 2026
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. minimaximalist approach to reinforcement learning from human feedback. In Forty-first International Conference on Machine Learning, 2024.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv: 100.:09800000000, 2018.
Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226, 2021.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rI? a theoretical perspective. Advances in Neural Information Processing Systems, 36:76006-76032, 2023.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interac- tive agent shaping high-dimensional state spaces. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Runzhe Wu and Wen Sun. Making RI. with preference-based feedback efficient via randomization. In The Twelfth International Conference on Learning Representations, 2024. URL https:// wtw.eentet//ooomdddee22ll
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683-6694, 2021.
Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Hassan Awadal- lah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit qa- approximation for sample-efficient RLHF. In The Thiiteenthh International Conference on Learn- ing Representations, 2025. httt:::::////eenteee http://eeweeee.e.e.ee/ooooonnana wwweee/ooouuimidiiigggeoeeeee
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rihf under kl-constraint. In International Conference on Machine Learning, pp. 547154447555 PMLR, 2024.
Yichong Xu, Ruosong Wang, Lin Yang. Aarti Singh, and Artur Dubrawski. Preference-basee based inforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 33:18784-18799 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 1835):1538-1555 2012.
Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic meth- ods for offline reinforcement learning. Advances in neural information processing systems, 34: 13626-13640, 2021.
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D. Lee, and Wen Sun. Provable offline preference-based reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024a.
Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D. Lee. Provable reward-agnostic preference- based reinforcement learning. In The Twelfth International Conference on Learning Representa- fions, 2024b.
13

---

Published as a conference paper at ICLR 2026
Zhilong Zhang, Yihao Sun, Junyin Ye, Tian-Shuo Liu, Jiaji Zhang, and Yang Yu. Flow to better: Of- fline preference- based reinforcement learning via preferred trajectory generation. In The Twelfth International Conference Learning Representations, 2024. URL https://openrevier
Yujie Zhao, Jose Aguilar Escamilla, Weyl Lu, and Huazheng Wang. Ra-pbrl: Provably efficient risk-aware preference-baseeeeeee based reinforcement learning. Advances in Neural Information Processing Systems, 37:60835-6088 2024.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feed- back from pairwise or k-wise comparisons. In International Conference on Machine Learning. pp. 43037-43067. PMLR, 2023.
Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. Advances in Neural Information Processing Systems, 35:35379-35391, 2022.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1wx.v::9999085999 2019.
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
Problem Settings in Preference-based RL. Preference-basee RL, sometimes called reinforce- ment learning from human feedback (RLHF), involves learning from preference feedback rather than explicit reward signals. Broadly, existing work PbRL can categorized into two lines: The first-which we focus on in this paper-studies learning from preference feedback in general stochastic MDPs (c.g. (Novoseller et al., 2020] Christiano et al., 2017)); The second line concen- trates on deterministic MDPs bandit problems, typically in the context of large language models (e.g. (Rafailov et al. 2024 Xiong et al.) 2024 Rosset et al., 2024 Xie et al. (025 The latter setting often considers fine-tuning pretrained policy using preference feedback, and incorporates regularization toward a pre-trained policy. In this work, we focus on the former setting and review related work accordingly.
Online PbRL Theory. The theoretical analysis on online PbRL has emerged from the dueling ban- dit problem (Yue et al.) 2012), where the agent makes sequential decisions based on preferences between selected actions. One of the earliest approaches was made by Novoseller et al. (022), who establish an asymptotic Bayesian regret bound for posterior sampling algorithm in tabular MDP. Xu et al. (2020) combine a reward-free exploration strategy and dueling bandit subroutines, offering a finite-tmme sample complexity bound. Several works have studied PbRL with linear models. [Saha et al. (2023) propose bandit-like algorithm that treats policy as action, and Zhan et al. (2024b) develop a reward-agnostic experimental design algorithm. [Wu Sun (2024) utilize posterior sam- pling techniques to prove worst-case regret bound for the linear setting, and a Bayesian regret for general function classes. Beyond linear settings, preference learning with general function approxi- mation has gained attention. For instance, Chen et al. (2022) design algorithm using exploration bonus that achieves a regret bound dependent on Eluder dimension (Russo & Van Roy 2013). Wang al. 2023) propose reward learning framework that solves PbRL when augmented with standard RL algorithms. Duu et al. (2024) analyze policy optimization algorithms for PbRL, under linear and neural function approximation. Chen al. (2023) and Zhao al. (2024) study risk-aware PbRL using the conditional value-at-risk (CVaR) objective (Artzner 1997). Another angle was explored by Swamy al.] (2024). who formulated the PbRL problem as a two-player zero-sum game over policies, thereby generalizing to arbitrary reward representations.
B APPLYING VALUE ALIGNMENT Loss TO APPO
Algorithm PAC: Preference-based Actor-Critic (A variant of APPO (Kang & Ohl (025))
1: Input: Datasets DpF, DTI, constants 77 A, Initial policy nh = Unif(AA for all h E [
2: Estimate é arg min [Rw(r) [. Pa. arg min PEP, i. for all h [H]
3: for t = 1,.....,. T'' do
4: f E Bargmin (I(1(s),=*) 7") - En - my (7)0)) + ( .t)) FEF
5: Update policy Th t (al 8) < s) (xp((ff((s for h [H]
6: end for
7: Return मे + EE=
In this section, we discuss how value alignment loss can be applied to APPO. We begin by formally defining the policy type induced reward function.
Definition 3 (Induced Reward Function, Policy Type (Zanette et al. 2021)). For f E F and π E II, the induced reward function r (r
defined as rh,5 = fh Th-P,, Ph fh+1. Similarly, define fT,5 as fT. fn. fT+ where P is some transition model.
) 1 E
We can naturally modify APPO to utilize value alignment loss with policy type induced reward. The pseudo-code is presented in Algorithm 2 In the modified algorithm PAC, the e, loss between the policy type induced reward ,g" and the reward model e is replaced with value alignment loss E (P f)-
15

---

Published as a conference paper at ICLR 2026
Now we theoretically analyze PAC. Following Zhan et al. (2024a); Kang & Oh (2025),
define the
PbRL version of single-policy concentrability (Xie et al. 2021
Uchara & Sun 2021
Definition 4 (PbRL Single Policy Concentrability Zhan et al.
2024a Kang Oh 2025
- - (+)))
(i,((F)= sup
/EF, V B (ry(79) - (-()))
where 11 = { {T= {TA) ThS & exp(n Ei-, fi) vh E [M,, F, [T 0 the set of softmax policies.
d
The single policy concentrability is bounded by the trajectory density ratio CTR sup, d= (r) and it is known that the sample complexity offline PbRL is lower bounded by CTR (Zhan et al. 2024a). We have the following sample complexity bound for PAC:
Theorem B.1 (Sample complexity of PAC). Suppose Assumptions 7 2 and 3 hold. For properly
set 1) and X, with probability at least 1 6, Algorithm 2
achieves an E-optimal policy with T
v2 H² log [A]
M ( C (Tr(()) log( R100 O (O Togg ()))
22
22
The sample complexity bound for M matches that of APPO, while the bound for N incurs an additional dependence (F). This dependency arises from the use of the quadratic value alignment loss, which requires variant of the decoupling argument (e.g. (Foster al., [021] Dann et al. 2021 Jin et al. (021)) rather than the direct suboptimality decomposition in Kang & Oh (2025). Theorem B. significant in that it demonstrates the value alignment loss can used in actor-critic algorithms. Together with the analysis of PVO, this suggests that the value alignment loss provides a unifying framework for provably efficient PbRL, applicable to both value-based and actor-critic methods.
C DETAILED PROOFS
We present the proofs omitted in Section
C.1 PROOF OF THEOREM 4.1
First, we prove Theorem 4.1 As we discussed, our PVO a new type of algorithm that directly optimizes the value function without actor-critic iteration policy optimization oracle. Therefore, our proof relies on a novel suboptimality decomposition utilizing the greedy property of = Tg.
Proof of Theorem 4.1
By Lemma C.I
we have the following regret decomposition:
VT (81) - (81)
VVT (81) Vi.r, (81) + VI.,, (s1) - (f
=(VT' (81) VT,,, (s1) + Vi.,, (s1) - v1((s) + (1rr((()) - (i)( - Vi,,, (s1) (s1)
Ey , (A 70 )t) A(rgit), +1) Exons, [A 7-) A (
The terms (I) and (II) represent the error of the induced reward function ry, under joint distributions (T" 11) and (ft p),, respectively. By Definition 2 each term is bounded by
(I) S (,,(F) . (((rt; 7-) - -(r 71)))))
We present the definition that APPO (Kang & Oh 2025) implicitly relies on although it is not explicitly stated in the which is slightly stronger than the one used in /Zhan al. (2024))
16

---

Published as a conference paper at ICLR 2026
Therefore, it is left to bound the term By (A(r";=0,7) ( which can be interpreted as the population version of our value alignment loss Lva. Lemma C.2 provides the bound, thus we finally have
VT' ( <22 ( log((R]8-1) (
Ha max [log( ( (1--) log(
M
N
This implies the sample complexity presented Theorem
Lemma C.1. For any f F and any policy n, it holds that Virry (51) S V1.s($1) Viry (81).
Proof. For h E [ 1], we have (x, x²) S fu(8, = (o,r()) for all 8 E S by definition (we
set fH+1 = 0). Therefore, we have
11
VT,,, E
Σ (fn
Pa (s
H
Ex
H
E
fT( - Ponn
< E,
Σ (sh, E Pann
[s1, m) (+1)
where we used telescoping sum in the second last step. Now applying similar argument in reverse order, we further have
Vi.s(11) = fi(s1, TT)
Σ Ev, [fn(sh, an) (o
H
= Ex, h=1
Σ E(n-
PV P2Va+1,)) an) -Vi,,
Lemma C.2. With probubility at least 8, we have
E, [( 70 T2) A(r;: "
2² log( (
H2V2 [log( (8-11)1)))))) + log((F(8-1))
M
N
Proof. Lemma C.3 and Lemma D.5 implies that
NE wu [A(r - (f; 7 +
< 22iva(r,, f) + v2 max 1og([F18-1)
<4iva(f 8HV2
ΣΣ
P P PA]] (o a.d ) + 16 log([Fl
max
max
A= jE (0.1)
s s4tva(ff.)) 882 log((P))))))) 11-1) + 16H2v2 max log F
17

---

Published as a conference paper at ICLR 2026
and similarly,
N (Arrr A(f; (
16
L (ra ()
22
log (og((=(0=1)
3
max
2
1 tv (f((fer
4
HV
ΣΣ
Ph - Ph. 11 (sh) m- v.d)
16 ,0
= max
log(1F]8-11
3
3
max
3
== h= [E(1,)
2 3 Llalforr
-2 2 max log( (P|
4
16
112 Vinax log( (18-11
3
3
On the other hand, the optimality of if (Line 3 in Algorithm
implies that
(f f) S Lva(font,
Combining the results, we have
-1) (r,,,,
A(;; -1))]
< 4fva (f f) 882 2 max log(
H6 11 1) 16// V2
max log([F18-1)
I 4iva(f of .)) + 8og 2 max log( (P| + 2 mux log( log( (F((11)
< 12N -l (( -0,
(f;;+
+ 24c² 2 max log([P|H8 (1 + 80 max 1og((F(5-1)
where we used the fact that ror r Therefore, it holds that
A ,r,;
2 (A 7') Aff:7 , ()]) + , Ar 7'))
< 26 ; , - (F;:7, 1)) H ma 24c2 (og(1P|H55 + 80 log (FF1
N
Now Lemma D.4 concludes the proof.
Lemma C.3. For any f F, the following inequalities hold:
H Σ
=va(t,,))2 < 2Lva(r,, + 4HV2
Ph - Ph 2 [|
(8h (6m.,aph) aa..a)))). )
n= jE(0,1)
-Lva(f, S - - ()
1
ΣΣΣ
PM -
n= jEf0.11
Proof. By definition, we have that
H (y(7 - -
> (P: Pa) )
h=1
Therefore, it holds that
Lva (F f) = Σ ( - -((7 + )()
n=1 N
2 Σ
(ry(7".0) ()) -( + )))
n=1
(Ph - Ph) n.0 ,a n.0) - (Pr - P.)Vntz. al (a. .a.1)) 2
h
2
n=1
h=1
<2Lva + 4HV2 2
ΣΣ
1 Pal1
PP--Pal? (5)
JE(0.1}
18

---

Published as a conference paper at ICLR 2026
where we use the Schwarz Similarly, using - -(x + y) W -22² v R. we also have
H
1 -Iv [va(rs;)) 2HV
ΣΣ Σ
Pj Ph (6% ,
C.2
PROOF OF THEOREM B.1
Now we prove Theorem B.1 Recall that the soft policy class 11 is defined
1 { exp(n fih) wh E [m],
(EFItE
for constant whose value is specified by Lemma E.3 It clear that log |1 S T log [F].
Throughout the proof, we write rf = rr and ft Py for convenience.
Proof of Theorem B.1
By Lemma E.1 and the fact क = + Σt n', we have that
VT' (s1)
(s1)
1
VT" (s1) V1 (s1)
T
t=1
1
( (') (s1) (s1) VT
T
B (+))
(7') (r()))
T
(f((1, - (81)
(81) + Branlr"(T) - -r(()))
T
(
1
Vi - (s1)
T
(II)
The term (II) is bounded by Lemma C.4 and (III) is bounded by Lemma E.3 For the term (I), note that Definition limpiie
E (r (7") ((())) +r+()*)]
E A - -((;7)))))
/ (E
+1) A (r+;
Er (( A(rt=797+)))
E - +1))2]
(F) 2
- A(r:; ());)))
(F) a
+
E, E (A(() - A(r":7",-')
2
2a
2 (F)
2aN (ra
1
1
Lval(r f) + - (vx
2
aN
19

---

Published as a conference paper at ICLR 2026
where the third inequality uses Definition and use the AM-GM inequality with constant in the second last step.
Combining the results, have
VT (s1) (1 a
(ZT(F)² + ( (22N 41 (x)()))))+++ 2
f)
1
+ 2
3 [va((f)")) + Vmnx H V
log |AA|
2aN
aN
2T
log log((=] (1 1) + o2 [og([P)HH01)
4V H.
N
+ X (c2H² max log(1P|H881) + H max log(1 [ 5 18-1 + (cn((NMM)) R|
2 (F)
+ 2aN 4x T + (2) + 2) NN (
1
log LA
Vmaa HV
2
aN
M
27
[ 18-1) + 02 log (P)((1
4V H.
N
16
+ A (c2H2V2 max
log( + H2v2 max log (
1 + log
c2H3V2
mu log(|P|H5 1)+ H2V2, log(1>) |11 (og))(( [og(1/2)5
Setting = o:N/2 and a
No. 2,
and using the fact log 1
<
T log Fl, we have
V VV
50
(Winllts (
(Vin_1(((ogg((11110)) + +oo(((FF1)))) x² (log((R((-1)) )) + Vmax H
log AA]
N
2T
This concludes the proof.
Lemma C.4. With probability at least 46, we have
[1(11=0 - VI (81) Ernwur"" (7) - ( - (3lua(r; [ua(t"))))
(1 saft
8 1-1) + 02 (og((())))1)
< 4Vmar
N
16
02 V2 m
log(l (o + 112V2
11 (og
waft
X
for all t E [T], where C1, are some absolute constants.
Proof. The optimality of ft (Line 4 in Algorithm 2 implies that
1 N X [1()), N Σ ((()))) ) + Lva(F' 5 + QT' ($L,
1 N
Σ pr²
Qu (tm)
+ L'v (roie.
N
n=
>
1 (81)
Σ
o (T"))
+ Eva(At
N n=1
20

---

Published as a conference paper at ICLR 2026
Combining this with Lemma C.3 and Lemma D.5
we have
fi(s1,
x-)
(81)
N
1
(T
Σ
prl
Q (7n.0)
1 i (Lralt (me,) - Exa(("
N
N
N
N
(7R.0) + 1 (2ivalra (r.,)) - 2
N ==1
Q+l
n=1
Vmax
N
H
Ph - Pall,
(5% 12,
n.0) +
8HV2
mux
ΣΣ m=1
N
H
- Ps (8h o2
N
n=
15(0011
N
N
(7
Σ
Qui (Tm.0) +
1 1 x (iivaraa ())
(ra(('))))
N
N
2
==1
n=1
882 H e2 log( (P] (11-11111
max log (og((P)118-1)
Vmnux H
N
Note that r
r* by definition. Using the concentration inequalities in Lemma D.2 and
Q
Lemma D.4 we further have
f1(81, n
VT (81)
N
Σ
1 Que (7 )
1 2ivv ( (ic; 2
N
N
n=1
02 log( [P] 1
802/12 log( log(1P((18-1) max
+ VmaxH Hy
N
A
1 X (3In((")
4 [vaarr .)) + Vana Hy
/
c2 log( (P1((P11881)
$ (r)
N
1 7 18c2 max log((P|H881) + 16H/2 max log [ [ F + 2HH max
2 2log(
I 18
Fir
N
This is the desired result.
Lemma C.5. For any f E F and π E 1 the following inequalities hold:
H
( 2lva(17,)) AHV
Σ
Ph Ps 11?
(5h aa an
n.j.
h=
(0,1)
N
H
Lv((F S [nl(r7,)) f)+ + 2HV2
ΣΣ
Σ
Ph Pr
( ,ap,)))
max
h= k
and
N N
N
N h
1
Σ m)
<
V Vmax
ΣΣ
P
P
(5 n0
N
N
R= h=1
Proof. The proofs for the first and the second inequalities are almost identical to that of Lamma C.3 The third inequality is obtained by
N
N N
N
H
Σ n=1
(r
=
Σ
M
(Pr
- P.O (s% To a.
N
N
==
h=1
N
Vmnx
ΣΣ PM
-
Pall, (5% n.O a,p ar.
N
where we use the notation f%
S +
|- Vmaxx Vinxl to denote the function satisfying (s(
Eg [f a)) for all E S.
21

---

Published as a conference paper at ICLR 2026
D CONCENTRATION LEMMAS
Lemma D.1. Given i, with probability at least 8, for all F and π E F. it holds that
N
2 ( sot 17-11)11)))
B [r--ur < 22H
N
N
==1
Proof. Fix ʃ E F. Azuma-Hoeffding inequality implies, with probability at least
(T()) E (r S 2HV max
V
2 1og((ō-1)
N
N
The union bound over all f F and E 1 concludes the proof.
Lemma D.2. Given f, with probability at least for all f E F, it holds that
Lva(r,,F) s 2Lm f) 16H mar log((F(8-1)
and
3 Lulr),)) S ()) 8/ max log( F 1)
Proof. Fix f E F, then define filtration 3n = o , 7 n.0 =m,l) (we will use |,| to denote E-| (3.l) and
Xn(U) = ". ,, A(r; 70, 0 - )
- -(D(r): pn.l)- 7 7".1
so that X,, E On. With this random process, we have E, (x,()))) = and
E, [x](
) ((,,)) ,7.1) 7f;:710, (.2))),
< V/2 En (A(rr: 7 mil)) (A; 7".0,==11))
Freedman's inequality (Lemma
E.5 implies that, with probability at least
N
N
log(5-1
Xa(U) <
E. [X(0))
1=1
n=1
N
A(f; -(f 1og(6-1)
< 4HH max
En [((([() n00 (.)) (,,
for any < E [0, 10,1/4H22 max
Setting 1/8H/2 max: we obtain
N
E, (A(ri (m)) (m))))))))))))))))))))))))))))) A ;
2
Σ (A(riiTn, (nl)) (mll)) AfF: )n.l)) 2 16H2 max log(8
which is equivalent Lva (ry, < 22 16H22 V2 log(5-1). We prove the first result by taking a union bound over all f E F. To prove the second result, consider (f) and follow the same logic with 1/8H21 v22
max.
Lemma D.3. Given f, with probability at least 25, all f E F and क E oof F.T it holds that
Lva(F7,F) f) < 2Lva () P) V2 max log [
18
and
(r) () Lva(rJ, V2 log (
aff taft
F.T 18-1 (8-1)
mar
22

---

Published as a conference paper at ICLR 2026
Proof. 18
The proof is almost identical to that of Lemma D.2 We further consider union bound over
Lemma D.4 (Lemma in Zhan al. 2024a)). With probability least 6, we have
1 N (r f) - ((((")))))))))))))) ;,,)))
A(f;r", ,()))),>))) C
(|66
M
for some absolute constant C1-
all Lemma D.5. With probability for all k E [K], h E [H], and {0, 13. it holds that
Σ
Σ
Pr pil
(5h nd an an)) 02 (og((P(H881)
n=1 JE(O.1)
where 0g is some absolute constant.
Proof. The standard MLE guarantee (e.g. Lemma 3 in Zhan et (2024a)) states that, with proba- bility at least 8,
E Ee ||Ah px]] (sh, an) S log((P|HH81))
(8)
for all h E [H).
For fixed Ph
Ph. define (P.h)) = || Pil (5% a and 3n
a-110 1...,, n,. 741). Applying Lemma E. (i) we have
N
3
Σ
Pr Pxll (6% a
E B. 1/112 (6% a a. log(8-1)
2
3 NE (B Ph PF (sh, an) log(6=
2
Taking a union bound over all Ph E Ph. and E [H], and repeating the same argument for (P, = Pr Pilli (s n.l ), with probability at least - 8, it holds that
Σ |P - PMIl (3 a < (e. - PE (sn, and + 8 81og(21P(H66))
m= jE(0,1)
for all h E [H] and P P. Combining this with 8 we conclude the proof.
E SUPPORTING LEMMAS
Lemma E.1 (Sub-optimality Decomposition (Lemmaa B.4 in Nguyen- Tang & Arora (2023))). For
any f E F and any policies π. n. we have
= Σ E. (EIE an) (511 V2 (s1) + V2,,, Vi.rg
where E fa(s,a) (s, Pr fn+i(s,a).
Lemma E.2 (Performance Difference Lemma). Let n,, pi be any policies. For any reward r, have that
H (s1) V (s, = Σ
Ex 1Q ((s)) C #(s)))
hm1
23

---

Published as a conference paper at ICLR 2026
Proof.
Using the identity th Qi. PK V 1,,, we have
(1,) - (s1) = En,l((1,, + P/ (s,
[er (s)) ( V, (81, ai)
- -= - (22)
E.
Ex [Q (sh, x(sh)) Qh, [ss)))l
Lemma E.3 (Online Regret Bound (e.g. Lemma D.3 in Kang & Oh (025 )) For any sequence of
functiins (f E Ft,, the policy update (Line 5) Algorithm that
with r
log A
guaraniees
log |A
Σ VT VI. virr( S VmarHy
2T
Lemma E.4 (Azuma-Hooue Hoeffd-g inequality). Let (Xi) be a sequence of random variables adapted to filtration (3t).T. If [xil S B for some B 0 almost surely, with probability at least - 8, we have
T Σ EX- 8r) S Bv 27 log(8-1).
Lemma E.5 (Freedman's 's incquality). Let (Xt)esT sequence of random variables adapted to filtration (8t) Assume [X,] S. B for some B and E[X, 3e] 0. With probability at least 6, for any 7 E [0, 1/B), it holds that
T
+ log(8-1)
Xx
S1
E[X?
"
Lemma E.6 (Lemma 2 in Zhu & Nowak
2022
Let (Xt)sST be sequence of positive random almost surely for all t, then with probability at
variables adapted to filtration (Bt)tTT
Xx
least 1 - 8, the following holds:
3
Σ
X.
E[X,
3e) 44 (og(()))1)
t=1
B
3c] 2 Σ
X, 8B log(5-1)
24

---

Published as a conference paper at ICLR 2026
F ADDITIONAL EXPERIMENTS
F.1 ON THE TRANSITION MODEL
Dataset (# of feedback)
-edlum-replay (500)
medium-replayy (1000)
medium-expert (500)
medium pedium-expert (1000)
PVO
66.20
69.14
68,06
73,00
PVO with transition model
64.08
71.18
68.46
71.16
JQL
53.86
58.76
57.78
67.84
Table Success rates on Meta- World datasets, averaged over five random seeds. PVO and its variant with learned transition model achieve similar performance.
The transition model in PVO plays a theoretical role in defining the value alignment loss and deriving the cample-complexity analysis, where the expectation Egr (s')) is taken over the estimated transition model. However, in practice, this term can be efficiently approximated using transition samples from the dataset. Our practical implementation therefore follows this sample-based approx- imation, a design choice originating from APPO Kang & Oh (2055), which employs a variant of our value alignment loss (as discussed in Section 3.3 and similarly replaces the model expectation with dataset samples. This simplification reduces computational cost while maintaining performance.
To verify whether this simplification affects performance, we implemented variant that explicitly trains a transition model and computes the expectation term via sampling from Table compares this variant with the original implementation 500 and 1000 preference feedbacks. The results show that both implementations achieve highly comparable performance, with neither consistently outperforming the other. These findings indicate that while the transition model is conceptually important for theoretical formulation, it is not prac- tically necessary for effective learning. Our sample-based implementation thus offers simpler and more computationally efficient realization of PVO.
Meta-Wa Meta-World datasets used in Section 5 trained with
F.2 EMPIRICAL COMPARISON WITH FLOW TO BETTER ZHANG ET AL. (2044)
Dataset
box-close
dial-turn
drawer-open
lever-pull
sweep-into
sweep
Dutton-press-topee
button butioo-press- topdoss-t
PV0
58.72 2
84.32
100 00 am
96.64
27.84
94,08 2:
43 36
48 16 ius:
2000
0.00 =
97.60
3m0
0.00 2.a
Table 2: Success rates on Meta- World medium-replay datasets with 1000 preference feedback. For FTB, we report the results averaged over three random seeds.
We also evaluated Flow to Better (FTB) Zhang al. (2024) the Meta-World redium-replay datasets with 1000 preference feedback. We used the official implementation of FTB and default hyperparameters, only adapting the episode length be consistent with our datasets.
Interestingly, FTB performs very well on drawer-open and even outperforms PYO on sweep-into, but collapses 0% success on all other tasks. We note that 0% success rate does not necessarily mean that the method failed to leam at all we do observe nontrivial improvements in episodic returns during training but rather that the learned policies rarely satisfy the success criteria on these tasks.
Overall, these results suggest that FTB can be effective on certain tasks, yet exhibits substantial sen- sitivity to the dynamics and data distribution. One possible explanation is that FTB ultimately relies on imitation of generated trajectories: when the dataset contains sufficiently good trajectories, the procedure may succeed, whereas in tasks where high-quality demonstrations are sparse, the model may end up cloning suboptimal behaviors. In contrast, PVO achieves consis- tently strong performance across all tasks, including those where FTB collapses. which aligns with our main goal: providing a simple, value-based offline PbRL algorithm that is both theoretically grounded and practically robust.
25

---

Published as a conference paper at ICLR 2026
cheetah-run
walker-walk
quadruped walk
250
Epo Raturn
Epis Return
300
600
150
200
100
100
PVO
IQL
APPO
PVD
IQL
APPO
PVO
IQL
APPD
500 preference fondback
1000 preference feedback
Figure 4: Performance DMControl datasets measured by episode return. Each plot displays the mean and standard deviation over five random seeds.
F.3 EVALUATION ON DMCONTROL DATASETS
We present additional experimental results that were omitted from Section 5 due to space constraints. Figure - shows the performance evaluation on DMControl Tassa et al. (2018) dataset. Overall, PVO outperforms APPO by large margin, which is consistent with the results in Section Com- pared to IQL, PVO performs better in the chectah-run and quadruped -wakk datasets, while showing comparable performance in the walker-walk dataset.
F.4 COMPLETE NUMERICAL. RESULTS
We provide the complete numerical results in the tables below. The results better than 95% of the best performance are highlighted.
Dataset and of feedback
DPPO
IPL
IQL
APPO
(ours)
box-close-50k
0.33
10.20
5.93 581
27.84
18.24
31.04
dial-turn-500
+ 68.67
26.67 223
31.53
74.56
80.96
82.72
drawer-open-500
88.73 + 11.64
35.93 11.88 19.00 = 1363
98.24
= 3.52
87.68 = 10.04
99.52 : 0.96
lever-pull-500
82.40 + 22
10.13
= 12.19
31.20 = 1576
85.28
3.18
75.76 7.17
93.92 4.92
sweep-into-500
20.53 + 8.26
23.07
4:700
32.20
7:35
20.96
1.1
24.08
5.91
37.92
11.00
sweep-500
43.07 24.77
10.47
27.20
90.88
26.80
87.68
nutton--pees-ttoodww--000
22.87 + 9.06
4:4
34.73
1392
18.88
12.14
53.52
13.86
56.80
13.28
button-press-toodoon--lllll
0.87 + 1.43
0.80 + 1.51
8.93 + 9.44
14.24
= 5.27
64.32
:: 20.99
39.36 + 14.52
box-close-1000
2.27 + 2.86
+ 9.00
6.73
+ 841
40,96
22.04
34.24 + 18.49
58.72 + 16.61
dial-turn-1000
68.60 5.50
36.40 + 21.95
43.93
+ 13.37
77.28
10.64
81.44
6.73
84.32 + 4.71
Staweropeen1100
95.40 7.27
36.47
1:7
28.53
1837
99.52
0.61
98.56
2.68
100.00
1: 0.00
lever-pull-1000
72.93
8.53
40.40
87.04
76.96
96.64
sweep-into-1000
20.27 1 7.84
23.33 7.50
30.40
1.7
24.00
5.97 18.16
11.14
27.84
sweep-1000
29.13 + 14:55
8.73 = 16.37
38.33
2487
98.80
= 1.01
17.36
12.44
94.08 = 6.31
hutton-ppeess-ooodooo---000
18.27 + 10.62
3.20 + 3.04
36.67
17.0
20.00
544 59.04
1899
43.36 + 19.03
nutton-pess-ttppoon--valll--1110
2.13 #
0.27 0.85
14.07
11.47
22.48
5.28 62.96
18:38
48.16
14.54
Average
39.78
15.47
26.86
56.31
55.01
67.63
Table 3: Success rates on Meta- Woold medium medium-replay dataset with 500 and 1000 preference feed- back, averaged over five random seeds. The results of PT, DPPO, and IPL are from Choi et al. 2024).
26

---

Published as a conference paper at ICLR 2026
Dataset and of feedback
IQL
APPO
PVO (ours)
box-close-500
19.68 11.96
1.92
49.44
12.19
dial-tum-500
13.60
42.56
73.44 22.66
drawer-open-500
100.00 0.00
88.48
9.65
100.00 + 0.00
lever-pull-5000
71.28 3.30
11.52
9.89
66.56 + 18.12
sweep-into-500
89.44 6.02
21.87 49.60
85.12 = 14.86
sweep-500
99.12 0.78
3.52
96.80 +: 2.48
butron-prees-topddown-000
59.84 18.13
3.20 6,40
70.40 = 14.20
butlor-pees--oploovwwall---0-
9.28 + 3.02
3.36 6.72
2.72 4: 2.41
box-close-10ke
57.28 7.72
1,92 1.09
63.36 :: 23.67
dial-tum-
26.72 24.13
42.56
12.81
76.64 +: 12.72
drawer-open-1000
100.00 + 0.00
80.16
5.29
96.00 + 8.00
lever-pull-1000
77.68
10,00 10.18
69.28 + 5.00
sweep-into-1000
95.04
32.96
93.28 + 98.88
sweep-1000
99.28 0.73
0.40 0.80
4: 1.39
button-press pettoo-pres--topoow---oo
78.24
11.15
14.40 #: 7.90
81.12
4: 5.02
button-press pwwprrppeees--oooowweeoeeeee 1000
8.48
3.94
0.16 0.32
5,44
+ 7.76
Average
62.81
24.17
70.53
Table 4: Success rates Meta- Woold medium-aa -mpexeert dataset with 500 and 1000 preference feed- back, averaged over five random seeds.
Datasct and # of feedback
IQL
APPO
PVO (ours)
cheetah-run run-5000
299.59
+ 47.6
91.48
+ 39.26
465.46 4 51.50
malker-walk-500
927.03
19.15
697.69
80.59
792.14
85.36
quadruped-walke walk-500
213.30
48.79
91.58
$2.99
210.90
41.72
chectah-run- run-1000
357.88
+: 1746
224.60
79.16
500.48
1: 41.60
walker-walk- 1000
931.48
+: 17.11
822.87
= 51.64
894.65
1: 28.04
quadruped-walk-1000
176.16
+ 78.54
91.92
32,83
188.42
+ 43.95
Average
484.24
336.69
508.67
Table 5: Episode returns on DMControl dataset with 500 and 1000 preference feedback, averaged over five random seeds.
G EXPERIMENTAL DETAILS
G.1 DATASETS
Dataset
box-close
dial-turm
sweep
sweep-into
drawer-open
lever-pull
button press topdown
button bution-press lopdown wall
mediun-reppi replay
2.4M
900k
2.1M
300k
300k
900k
300k
450k
expert
Table 6: The sizes of Meta- World medium replay datasets (Choi et al. 2024 and medium- exxpert datasets.
The DMControl medi and Meta-World medium-reples datasets are created by Choi (2024). The datasets are generated from the replay buffers of online SAC (Haarnoj et al.] 2018 agents. Following et al. (2024), we use different dataset sizes for each Meta- task, as shown in T All DMcontrol datasets contain 300k transition samples.
We created the Meta-World medium-expers expert data using the code provided by Hejna & Sadigh 20244.. For dial-turn, sweep-into, drawer-open, and lever-pull tasks, we collected trajectories with expert policy, 50 trajectories with expert policies for randomized variants and goals of the task, 100 trajectories with expert policies for different tasks, 200 trajectories with random policy, and 200 trajectories with E-greedy expert policy that takes greedy actions with 50% probabil- ity. In total, there are 600 trajectories (300k transitions) for each task dataset. Additionally, standard Gaussian noise was added the actions of each policy. The dataset sizes match those of the medium-
27

---

Published as a conference paper at ICLR 2026
replay dataset. For box-close and sweep tasks, collected 1800 trajectories, maintaining the ratio of sources.
To create the mixture datasets in the ablation study, collected 600 trajectories from expert policy trained on multiple tasks and goals, and another 600 trajectories from random policy. Then, for each mixture ratio E {0, 0.25, 0.5, 0.75, 1}, we formed dataset consisting of 600(1 r) expert trajectories and 600r random trajectories. For example, the r = 0.25 contains 450 expert trajectories and 150 random trajectories.
G.2 IMPLEMENTATION DETAILS.
We used the official implementation Ch et al. (2024) and Kang et al. (2023) for reward models, IQL agents, and APPO agents. The reward model an ensemble of three fully connected neural networks with three hidden layers of 128 neurons. The Q V, and policy are parameterized as fully connected neural networks with three hidden layers of 256 neurons. We set the hyperparameters of IQL and APPO suggested in Choi et al. (2024) and Kang & Oh (2025), except the advantage weight of IQL which we searched over 3 E (3.0, 10.03. The detailed hyperparameters are listed in Table 7
We run experiments on an Intel Xeon Gold 6226R CPU and an Nvidia GeForce RTX 3090 GPUs. For PVO, the use of trajectory does not significantly slow the training. All algorithms take approx- imately 2-3 hours to complete 250k gradient steps. The performance is measured by
28

---

Published as a conference paper at ICLR 2026
Algorithm
Component
Value
Neural networks
3-layers, hidden dimension 128
Activation
ReLU for hidden layers. Tanh for final output Adam (Kingma & Ba. 2015 with learning rate le-3 512
Optimizer
Reward model
Batch size
Epochs
300
Number of ensembles
Neural networks (Q. m)
3-layers, hidden dimension 256
Activaton
ReLU for hidden layers
Q.V. optimizer
Adam with learning rate 3e-4
PVO
Batch size
256
Target soft update
0.005
3 (TQI. advantage weight)
10.0
7 (TQI. expectile parameter)
0.7
discount factor
0.99
Neural networks (Q, T)
3-layers, hidden dimension 256
Activaton
ReLU for hidden layers
Q, V, x optimizer
Adam with learning rate 3c-4
Batch size
256
IQL
Target soft update
0.005
(IQL advantage weight)
10.0
7 (IQL expectile paramcter)
discount factor
0.99
Neural networks (Q. V. m)
3-laycrs, hidden dimension 256
Activaton
LeakyReLU for hidden layers
Q.V, optimizer
Adam with learning rate 3e-4
APPO
I optimizer
Adam with learning rate 3e-5
Batch size
256 transitions and 16 trajectory pairs 0,001
Target Q soft update
discount factor
0.99
Neural networks (Q. TT)
3-layers, hidden dimension 256
Activaton
ReLU for hidden layers
Q, V, x optimizer
Adam with learning rate 3e-4
Batch size
256
TD3+BC
Target Q soft update
0,005
a (BC weight)
2.5
Policy noise
0.2
Policy noise clip
0.5
discount factor
0.99
Neural networks (Q, V, m)
3-layers, hidden dimension 256
Activaton
ReLU for hidden layers
Q. V, x optimizer
Adam with learning rate 3e-4
Batch size
256
XQL
Target Q soft update
0,005
3 (Gumbel regression temperature)
1.0
discount factor
0.99
Table 7: Implementation details and hyperparameters.
29


Reviews:
[{"review_id": "BEblcKwX7D", "reviewer": "Reviewer_23sR", "summary": "This paper studies offline preference-based reinforcement learning and introduces Preference-based Value Optimization (PVO). The method directly aligns the learned value function with the reward inferred from human preferences through a novel value alignment loss, ensuring consistency between value estimation and preference supervision. The authors provide theoretical guarantees and strong empirical results, showing that PVO achieves stable and competitive performance across continuous-control benchmarks.", "questions": "- Line 3 of Algorithm 1 differs from Eq. (4); there appears to be a sign inconsistency (the “+” and “−” symbols might be reversed).\n- The proposed value alignment loss appears conceptually related to the value function inconsistency introduced in VIPO: Value-Inconsistency Penalized Offline Reinforcement Learning. A discussion clarifying the connection or distinction between these two ideas would be appreciated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed value alignment loss is conceptually sound, novel, and powerful enough to achieve strong results.\n\n- The empirical evaluation is comprehensive and consistent, demonstrating that PVO outperforms existing preference-based approaches across multiple continuous-control benchmarks.\n\n- The paper is clearly written and well-organized, offering valuable intuition on how human preference signals can be effectively integrated into value-based offline reinforcement learning.", "weaknesses": "- The reward learning module closely follows standard preference-based MLE approaches and thus contributes limited novelty in this part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cH16G5zkN6", "reviewer": "Reviewer_BMtm", "summary": "This work addresses a critical gap in offline preference-based reinforcement learning: the tradeoff between theoretical guarantees and practical usability in existing methods. By proposing Preference-based Value Optimization, which uses a value alignment loss unifying value-based and actor-critic PbRL with rate-optimal guarantees, it delivers a unified solution that excels both in theory and experiments.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The work targets the well-documented tradeoff in existing offline preference-based RL methods, where theoretically rigorous approaches are often computationally intractable, and practical implementations suffer from suboptimal sample complexity or training instability.\n2. One advantage of the work's insight is its shift from the standard offline PbRL paradigm (first infer a reward model, then train the value function using that reward directly) to a different viewpoint: it anchors value function learning to preference alignment via an \"induced reward\" derived from the value function itself, rather than treating the reward model as the sole driver of value training.", "weaknesses": "1. The first weakness is the paper’s inconsistent formatting of mathematical formulas, specifically, the absence of terminal punctuation in Line 318.\n2. The paper’s baseline set is limited in scope. Several existing works in PbRL use generative models, like trajectory generative adversarial networks and diffusion models for preference modeling, to infer preference-aligned behavior without relying on intermediate reward models. These should also be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NLw1lkvqQ6", "reviewer": "Reviewer_frBc", "summary": "To address the issues of high computational complexity and instability in PbRL, this paper proposes an optimization objective named the value alignment loss and validates the effectiveness of the proposed method both theoretically and experimentally.", "questions": "1) Motivation: Before Definition 1, the authors discuss learning a value function consistent with preference feedback. How is this reflected in the proposed value alignment loss?\n\n2) IQL Parameters: The framework is similar to IQL. For a fair comparison, what is the performance of IQL when its advantage weight parameter $\\beta$ is set to be consistent with PVO?\n\n3) Transition Model: In the practical deployment described in Section 3.4, the environment model does not seem to be used. What, then, is the purpose of training the transition model?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) Through experiments, the paper verifies that PVO outperforms other methods for most tasks.\n\n2) The paper provides sufficient theoretical derivation and analysis, including a derivation of the algorithm's computational complexity.", "weaknesses": "1) Algorithmic Aspect: The algorithm is essentially consistent with IQL [1], showing limited innovation. IQL minimizes $(R+\\gamma V-Q)^2$, while PVQ is equivalent to minimizing $(\\sum_l (Q_l-\\gamma V_l-R_l))^2$.\n\n2) Theoretical Aspect: The theoretical analysis is similar to that in paper [2], offering limited contribution.\n\n[1] Offline Reinforcement Learning with Implicit Q-Learning, ICLR, 2021.\n\n[2] Provable offline preference-based reinforcement learning, ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)