Toward Conservative Planning from Human-AI Preferences in Reinforcement Learning
Authors: Huazhong Wang
Wenzhuo Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
sample complexity
model-based planning
Primary area: reinforcement learning
TL;DR: We propose a novel model-based conservative planning algorithm with both sample and computational efficiency guarantees.

Abstract: We study reinforcement learning (RL) with trajectory preferences, where the RL agent does not receive explicit rewards at each step but instead receives human-AI preferences over pairs of trajectories. Despite growing interest in preference-based reinforcement learning (PbRL), contemporary works cannot robustly learn policies in offline settings with poor data coverage and often lack algorithmic tractability. We propose a novel **M**odel-based **C**onservative **P**lanning (MCP) algorithm for offline PbRL, which leverages a general function class and uses a tractable conservative learning framework to improve the policy upon an arbitrary reference policy. We prove that, MCP can compete with the best policy within data coverage when the reference policy is supported by the data. To the best of our knowledge, MCP is the first provably sample-efficient and computationally tractable offline PbRL algorithm under partial data coverage, without requiring known transition dynamics. We further demonstrate that, with certain structural properties in PbRL dynamics, our algorithm can effectively exploit these structures to relax the partial data coverage requirement and improve regret guarantees. We evaluate MCP on a comprehensive suite of human-in-the-loop benchmarks in Meta-World. Experimental results show that our algorithm achieves competitive performance compared to state-of-the-art offline PbRL algorithms. Our code is provided at https://github.com/Rshias/MCP.

Published as a conference paper at ICLR 2026
TOWARD CONSERVATIVE PLANNING FROM HUMAN-AI PREFERENCES IN REINFORCEMENT LEARNING
Huazhong Wang
Wenzhuo Zhou University of California, Irvine wenzhuzz@uci. edu
University of California, Irvine hua wchonwuucc.... edd
ABSTRACT
We study reinforcement learning (RL) with trajectory preferences, where the RL agent does not receive explicit rewards at each step but instead receives human-Al preferences over pairs of trajectories. Despite growing interest in preference- based reinforcement learning (PhRL), contemporary works cannot robustly learn policies in offline settings with poor data coverage and often lack algorithmic tractability. We propose novel Model-based Conservative Planning (MCP) algorithm for offline PbRL, which leverages a general function class and uses a tractable conservative learning framework to improve the policy upon arbitrary reference policy. We prove that, MCP can compete with the best policy within data coverage when the reference policy is supported by the data. To the best of our knowledge, MCP is the first provably sample-efficient and computationally tractable offinn PbRL. algorithm under partial data coverage, without requiring known transition dynamics. We further demonstrate that, with certain structural properties in PbRL dynamics, our algorithm can effectively exploit these structures to relax the partial data coverage requirement and improve regret guarantees. We evaluate MCP on a comprehensive suite of human- human-in-the-lowee in-the-loop benchmarks Meta- World. Experimental results show that our algorithm achieves competitive performance compared to state-of-the-ar offline PbRL algorithms. Our code provided at https: / /ii trrtthhbb.. com/ Rahisshi SahhassMCC /MCP.
INTRODUCTION
Reinforcement learning (RL) has become a prominent framework for addressing sequential decision- making problems. Most of the existing RI. algorithms typically assume access well-defined reward function that guides policy optimization. However, in many practical applications, the design of an appropriate reward function poses a significant challenge. This difficulty arises from the complexity of accurately capturing human intent, the susceptibility to reward hacking, and the risk of inducing unintended behaviors due to mis-specified objectives [Wirth et al., 2017]. To tackle the problems, the framework of the preference-based reinforcement learning (PbRL) has emerged as a possible solution. Rather than relying on numerical reward signals, PbRL leverages relative preferences obtained from human experts or large language models, thereby avoiding explicitly modeling the reward function [Christiano et al., 2017]. This framework has proven particularly effective in various domains, including games Glashann al., 2017; Christiano et al., 2017; Warnell et al., 2018], large language models [Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021; Nakano et al., 2021; Ouyang al., 2022: Glaese al., 2022; Bai et al., 2022; Ramamurthy et al., 2022; Liu et al., 2023], and robotics [Brown et al., 2019; Shin et al., 2023].
Many existing PbRL algorithms rely on online interaction with the environment, raising concerns regarding sample efficiency and safety [Christiano et al., 2017; Levine et al., 2020J. In contrast, offline PbRL operates on pre-collected datasets annotated with preference information, providing more practical solution in scenarios where environment interaction is limited or costly [Shin et al., 2023; An et al., 2023; Kim et al., 2023: Hejna & Sadigh, 2023; Choi al., 2024]. Despite its potential, much of the prior work in offline PbRL requires that the offline data be fully explored and often lacks cample-efficien guarantees [Zhan et al., 2023a]. The situation becomes even more challenging in
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
scenarios where the offline data distribution only partially covers the trajectory distributions induced by some (but not all) comparator policies- thatiess-thtttti is, under partial coverage [Jin et al., 2021; Cheng et al., 2022]. In such cases, PbRL algorithms generally fail to learn a good policy with near-optimal regret in a polynomial sample complexity.
Recent efforts have sought address the Obove-mentioned challenges under function approximation settings. Zhu et al. [2023] proposes a principled algorithm with sample complexity guarantees under partial coverage; however, their algorithm and theoretical guarantee are restricted to linear reward models. Zhan et al. [2023a]; Pace et al. [2024] extends this framework to the general function approximation setting by explicitly modeling the confidence sets and performing conservative learning. Nevertheless, constructing and optimizing over such confidence sets makes their algorithm computationally intractable in practice. Most recently, concurrent work [Kang & Oh, 2025] introduces a sample-efficient approach which is able to be implemented in practice. Their algorithms either assume known transition dynamics or require fitting an extra value function that depends on the learned transition model via Bellman recursion to perform conservatism. This allows the value function to locally smooth over gaps in data coverage-if the value function well-approximated. However, this smoothing based mitigation strategy intrinsically requires the realizability condition on the value function and does not guarantee near mear-optimal regret under partial coverage [Yu et al., 2020a]. We refer the readers to Table 1 for detailed comparisons prior works.
Table 1: Comparison of MCP and its two variants with existing methods in terms of data coverage assumptions, additional structural properties, probably approximately correct (PAC) guarantees with polynomial sample complexity, and computational tractability for practical implementation. MCP refers our algorithm developed for general function approximation. The superscript * indicates that the partial coverage condition is effectively refined by MCP through additional structural properties of the dynamics, resulting in improved regret bounds.
Mothods
Purtial Coverage
Additiona Structure
PAC Guaransee
Tractable Implementation
Principled-R1 HH (Tha
Linear function approximatioon
FREEHAND
2023a]
OPRL [Shin el al., 2023]
Sum- OPRL |Pace et al., 2024]
APPO (Kang& & Ob, 20251
Known transition dy dymamico/valoe function realizability
IPL (Rejin& & Sedigh, 2023]
MCP
MCP- FCCt-Factoeed
Factored model
MCP-KNR"
Kemelized nonlincar rogulatios
Motivated by the aforementioned challenges, we study the problem from model-based learning perspective and propose an implicit way for encoding conservatism that is compatible with PbRL under general function approximation, without requiring known transition dynamics. Specifically, we relax the stringent full coverage assumption and instead assume that the offline data only needs to cover the trajectory distribution induced by the (optimal) comparator policy. We introduce MCP: Model-based Conservative Planning, which learns a policy through a tractable conservative learning and model-based planning procedure. The resulting policy matches the performance of any comparator policy that is covered by the offline data. MCP addresses the intractability inherent in existing methods that rely explicitly constructing confidence sets to encode conservatism, and it avoids additional value function modeling by leveraging a purely model-based planning approach. To the best of our knowledge, MCP is the first offline PbRL algorithm that is both provably sample- efficient and computationally tractable under partial data coverage, without assuming access to the true transition dynamics. When instantiated within specialized PbRL structures, MCP further refines the concentrability coefficient in a tighter manner, leading to improved regret guarantees. In environments with factored dynamics, the regret bound of MCP scales with the number of factors and the size of their parent sets, thereby avoiding the exponential dependence on the state dimension present in non- con-factored models. Moreover, our analysis shows that the regret bound is adaptive to the offline data distribution and remains valid even when the state space is infinite when MCP is applied in kernelized nonlinear regulators (KNRs). Experimentally, we find that MCP achieves competitive performance compared to of-te- offline PbRL methods, using real human feedback across 8 different Meta- World tasks [Yu et al., 2020b]. In addition, the conducted ablation studies demonstrate the robustness and sample efficiency of the proposed algorithm.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES
Markov Decision Processes. We consider a fnite-horizon time-inhomogeneous Markov Decision Processes(MDPP), denoted as a tuple M = A, ( { |--1 H,)) Here represents the state space, and A denotes the action space. PA, S x A - A(S) the transition dynamics time step h. where (S)) denotes the set of probability distributions over states. r P + [0, Rmmax] is the reward function, where F (S A) represents the set of all trajectories of horizon length H. 80 is the initial state. We use r* to denote the ground-truth reward function, {P% W denote the ground- truth transition dynamics. The agent follows a history dependent policy π == { H-1 where each F (S x AA)^11 x S + ((A), specifies distribution over actions at step h E [0 H- 1]. conditioned on the entire past trajectory. Let II denote the set of all such history-dependent policies. Given generic reward function r and transition dynamics P = (Px) M the expected cumulative reward defined as J(x;r, ; P) == E. [r(r), where d% denotes the distribution over trajectories induced by executing policy æ under transition model P. We denote Ezy. [r())s>]] as the expected return of trajectories starting from initial state sº. The state-action visitation distribution at step is defined as: df (s, a) = P (8)- = 8, d₁ a), vh E [0 - -1], where Pp represents the probability distribution over trajectories generated by executing policy under transition model P. Additionally, we denote the trajectory-level distribution under the ground-truth model P* as (T).
Offline PhRL. Offline PbRL is variant of reinforcement learning that deals with situations where the reward function is not directly available and instead must be inferred from human pref- erences over trajectory pairs. Offline PbRL focuses on learning from dataset of trajectory pairs
n,0 p,,,H-1 H-1 D = [[7".) 7".1, yy" "")] which contains i..d trajectory pairs - {5% a Jh=0 - {s n. n.l H-1
0 Jh=0 sampled from reference policy μ and binary labels y". Given a pair of trajectories
human annotator provides binary preference label y (0, 1}, where y indicates that trajectory 7² is preferred over 7", and = 0 indicates the opposite.
To model the preference feedback between trajectories, we introduce link function + R - [0, 1]. which is a monotonically increasing function. Given a pair of trajectories 7"), the preference model assumes that the probability of preferring trajectory 7! over 70 is given by:
Ply = ; = ( preferred over 70) = (6r ( -r())))()7)))
One of the most commonly adopted link functions is the sigmoid function (x) = (1 exp((x))) This link function is associated with the Bradley-Terry-Luee (BTL) model [Bradley & Terry, 1952], which effectively models the relative preference between trajectories. And we define = (inf Rma. P Φ (z())) to measure the non-linearity of the link function T. The goal of offline PbRL is to learn a high-perforree perffornformin policy ALG e II which satisfies the following guarantee: , - J (TALG r-, P*) < e. Here, π denotes a comparator policy that the learned policy aims to match or surpass in performance. For the remainder of the paper, we abuse notation by referring to (TT) [(TALG as the regret.
General Function Approximation. offline PbRL. Specifically, we model the reward and transition dynamics using family of transition function classes (P and reward function class R. These classes are expressive enough to capture complex dynamics and reward structures through the use of linear approximators or neural networks. To quantify the complexity of the transition and reward model classes, we adopt the biaabbracktiinggg number metric, denoted as (1/N) and NR respectively [Geer, 2000].
this work, we consider general function approximation for
3 EEEERRENEE-GIDDEE CONSERVATIVE PLANNING
In this section, we present a novel sample-efficient and computationally tractable offline PbRL algorithm, Model-based Conservative Planning (MCP). The developed algorithm integrates model- based planning with implicitly encoded conservatism to guarantee the learned policy can compete with any (best) comparator polices within data coverage.
3

---

Published as a conference paper at ICLR 2026
3.1 ALGORITHM FORMULATION
We begin by presenting the motivation that underpins the design of our algorithm. The major challenge in offline PbRL is that directly performing policy learning based on the learned reward and transition models for agreement with preference feedback is inaccurate and may result in overestimation issues. To get rid of this problem, the existing works heavily rely on constructing explicit confidence sets to perform conservative learning, which is often not computationally tractable. This motivates to develop an algorithm that alternates between encoding the conservatism into the learned reward and transition models and learning the policy via model-based planning procedure upon the worst-case models that remain consistent with the observed preferences in offline data.
In MCP, we formulate the main objective to identify policy that performs favorably relative to reference distribution Mref. a common choice is the distribution to induce offline data. Specifically, aim to maximize the performance difference between the candidate policy and the reference distribution. This relative performance evaluation encourages policy improvement upon the reference policy while avoiding reliance potentially inaccurate absolute value estimates. Moreover, the evaluation can be easily performed through a model-based planning procedure, and avoids extra value function modeling.
max Jæ; v, (Pa) E [r(r)]-
MCP then takes two realizable hypothesis classes for the reward and transition kemnes-i.e., r* E R and P% Pr for all h [0: H -]--hich input, and computes the maximum likelihood models F and PA using the given offline dataset DD then formulates a minimax objective function.
which consist of potential data-consistent candidate models as
max min KEll (P
(T; (P (onmm.,/(((T) E2 +x
D), (3.1)
where the conservatism implicitly encoded via regularizing the empirical absolute discrepancy between the learning targets, i.e., r and Pr. and the data-consistent models F Fnd Pa:
N
(r; =
- (((7"))-(((()))
- ( -
N
n=1
N ΣΣΣ
E2(PPN)) D)
n.d Ph n. aar - (s 1s7 and
K= 1=0
This minimax formulation effectively searches for reward function and transition model Pa within the data-consistent model class by avoiding large discrepancy from maximum likelihood models, and then performs model-based planning using the searched conservative models. In (3.1), the parameters A and A2 are user-specified and control the degree of conservatism encoded in the learned models. Notably, MCP remains tractable-unlin conservatism through the unmeasurable width of confidence sets via constrained optimization, which often leads intractability.
existing approaches that encode
3.2 ALGORITHM IMPLEMENTATION
In this section, we present the details of the implementation of the MCP algorithm. building upon the above-formulated objective function. We will establish the rigorous theoretical guarantees for Algorithm later.
Model Estimation (Lines 2-3). The algorithm begins by estimating the reward model F and the transition model PA, through maximizing the log-likelihood function based on the offline dataset D.
Conservative Planning via Relative Performance (Line 5). this step, MCP enforces consistency with the offline data by regularizing the discrepancy between the learned reward and transition models and their maximum likelihood estimators. It then implicitly encodes conservatism by performing conservative evaluation--a plannninggnnggggnnnn under the worst-case model based the relative performance of the distributions induced by n, and the reference policy. This model- evaluation avoids learning an additional value function, improving computational efficiency.
4

---

Published as a conference paper at ICLR 2026
Policy Improvement (Line 6). MCP improves the policy without explicitly searching over policy function class II. Instead, it performs a mirror descent update, which is often utilized in online settings [Haamojj al., 2018; Geist et al., 2019], which bridges the gap between the policy space and the reward and transition model classes. As a result, the policy is no longer searched independently of R and (Pa)n=o H-1
Algorithm Model-based Conservative Planning (MCP)
Input: Offline data D, regularization parameters \1, X2, learning rate 1), reference distribution fref the uniform policy.
I: Initialize policy
2: Learn reward: F argmax,eR En log Po 0 ,,,0).
3: Learn transition kernel:
P = argmax PREPA ΣΣ
log (h . af wh E [0: 10:H-11
n=1 ==0
4: for t 1,2, ................................ T do
5: Obtain the conservative models: Te: (P
Tt: H-1
arg min (P.
Mmsr, - Bramre
ME(((: X222
6:
Update ne by: (als) x (r(ass) exp nB
(n((t))s,a)
7: end for
8:
Output ALG = Mixlter( (ix(ter((TT)TEa))
D mixing T1,..., over all iterations uniformly
4 THEORETICAL ANALYSIS
In this section, we establish the regret guarantee for the policy returned by MCP in Algorithm under the partial coverage condition and general function approximation. To start with, introduce the notation of the concentrability coefficient to characterize the condition for the partial data coverage. For comprehensive technical details regrading this section, please refer to Appendix B.
Definition 1 (Concentrability Coefficient for Reward), For comparator policy concentrability coefficient w.r.l. the reward function class R, and a reference policy pref:
define the
Er 1add77990a rree -(((())))((())))) -r - P B [Il(r(T') - (r"))
(
sup
Definition 2 (Concentrability Coefficient for Transition). For comparator policy define the concentrability coefficient w.r.f. the transition function class { (Pr) H-1 and a reference policy pref:
B(aaa=ay [Drv (Pa(-Is, a), (-|s, a))l
CP(T)
max sup hE(0:-11Ppp LE Essay)yp DDr (Pr(- a), P.((s,))))
We should note that the finite concentrability coefficients implies the single-policy concentrability that offline data covers a single good comparator policy (e.g., the optimal policy). The single-policy concentrability is the minimum condition for the offline data coverage [Chen & Jiang, 2022: Zhan et al., 2022] in existing literature. In addition, when the reference distribution fref is set to μ. the coefficients ( and ( is upper bounded by the vanilla density masid concentrability coefficients. i.c., CP(T) < suP(s,,,h) pa (s.a) and CR(x) I supre [((7) respectively. Before we present the main results, impose some regular assumptions on the function classes.
(s.a))
(r)
Assumption (Realizability). f" E R and P E Ph, vh E : -1. Assumption (Boundedness). 5 r(T) & Rmaxo Vr E R,
T T.
Theorem 4.1. Suppose Assumptions I and 2 hold. We set the learning rate /log (/22 and set the regularization coefficients X ((En(T)) and X2 O(Rmaxx Mr) Then,
5

---

Published as a conference paper at ICLR 2026
any comparator policy II with finite (T( and (p(x), with probability at least 8. the return policy N/ yielded by Algorithm I after iterations satisfies that
log
[A]
log(AR (1/N//5)
J(T) J
Rmmxx
+ 0
(eRmaxC((TT)
T
N
(P1/N(())))
+ Puax((p(T)) + Mp)H
N
where Mp is to measure the distributional shift between di and under the sequence of policies
yielded in the mirror-descent trajectory of Algorithm 1 for [] T and [0: H 1]:
E (s [Drv (PV
Mp
max max CEL:TEEUO-RR
((s,a)
Theorem 4.1 implies that the policy x ALG is the "best-effort" policy in single-policy concentrability. which can compete with any good comparator policy (including the optimal policy if it is covered by offline data). In the upper bound of Theorem 4.1, the regret can be split into three parts. The last two terms correspond to the statistical errors, which are amplified by the coverage of the offline dataset. In general, the small (P(T) and CR(TT) (T) results in better statistical error guarantee. In contrast, the large ones potentially pay high variance and statistical errors. Interestingly. we find that the distributional shift measurement Mp influences the regret, indicating that the miirer-descent trajectory indeed matters in Algorithm 1. Note that the first term, i.e., the optimization error O R /log [A|/T) can be reduced with the increase of the iterations T. Since the optimization error is well-controlled, this regret guarantee mainly determined by the last two statistical errors.
In many high-stakes applications, the safe policy improvement guarantees are of concern, i.e., the return policy ALG is no worse than the behavior policy for generating offline data [Levine al., 2020]. In the following, we show that Algorithm guarantees the safe policy improvement.
Corollary 4.2 (Safe policy improvement). Under the conditions of Theorem 4,1, we run Algorithm / many iterations, ie., T is sufficiently large, with probability at least 8, the regret ((x)) -J (TAAO)) is upper bounded by
o
( (Rmaack RmaazKy
log(NR (1/N)88) N
log(Np(1/N)) N
+0 Rmax HMp
Next, we study the sample complexity of MCP and demonstrate that our algorithm achieves improved performance in comparison to prior works.
Corollary 4.3 (Sample complexity). Under the conditions of Theorem 4.1, the ALG in Algorithm
satisfies (x) (TTAL)) < & with probability least 6, if the sample size attains
3 (1(
+ H²((P(TT)+ 2p) log (NP(1/N)/8))
2
In comparison to some prior works, Kang & Oh [2025] requires sample complexity:
o max R/
H log [A] log (FF)/)) R2 H² log log(H())1/5) H|P1/55)
C2 Cn Hog(([
e²
where | denotes the cardinality of the value function class, CTR = sup,es p(T) This implies that the results in Kang & Oh [2025] are restricted to finite classes, but we can handle the infinite classes. Moreover, the bound Kang Oh [2025] dependent on an extra model class F, and we avoid this additional complexity via leveraging a relative-performance model-based planning procedure. Also, the MCP is more sample efficient, i.e., O(e-?) vs Oce-A). In comparison to the work Zhan et al. [2023a], they attain a sample complexity of the same order ours, but it requires explicitly solving for the confidence set radius, which computationally intractable. In contrast, MCP achieves both statistical efficiency and computational efficiency by avoiding constructing explicit confidence
(r)

---

Published as a conference paper at ICLR 2026
5 SPECIALIZED STRUCTURES ON DYNAMICS
In the previous section, the results were established for general function approximation. In the following, we consider structured dynamics and show that MCP exploit additional structural properties to refine the model-based concentrability coefficients into more interpretable and natural quantities, leading to improved regret bounds. Specifically, we analyze two representative examples: (1) Kernelized nonlinear regulators (KNRs [Kakade et al., 2020)), which capture smooth nonlinear dynamics common in control and robotics, and (2) Factored model [Kearns Koller, 1999], particu- larly effective in dealing with high- nigh-dimensionall environments, e.g., via conditional independence. For comprehensive technical details regrading this section, please refer to Appendix and E.
5.1 KERNELIZED NONLINEAR REGULATORS
A kernelized nonlinear regulator is model that assumes the next state is a linear transformation of a nonlinear embedding of the current state and action, corrupted by Gaussian noise [Kakade et al., 2020]. Formally, the transition model is given by: s' W WV*((s,)) c,c ~ NO0,ç31) where Ç E RR, 8 E Rd" E Rd. and S X A + Rd is nonlinear feature mapping. The true model is parameterized by the unknown weight matrix W and the corresponding class of models is indexed by W, so that each candidate model is denoted by Pw. To establish the regret guarantee for MCP in the KNRs setting, we define a new concentrability coefficient that takes advantage of the structure of KNRs. Let En (e) (d(ss, a (s, a) "] Enp (e.a))=((((a a (8, a) and Σn = Ei= ((s, )g (st, a:) + AI. We define the relative condition number as:
I E.,..
c = ssup
xER
r E
We should note that (rv (Ps(s a), Pw. (+1s,a)) = - (os, [Devroye et al., 2018), where is universal constant. It is easy to show that C is upper-bounded by the relative condition number e (x). . We now tailor Algorithm 1 to the KNR setting. We need to modify the maximum likelihood estimator of the transition model to kernelized nonlinear regularized variant, i.e.
W = arg ED W ((, a) - (112) + W||
WeRd'xa
where [[W]| is the Frobenius norm of W. Now are prepared to state the regret bound for MCP in the KNR setting.
Theorem 5.1 (PAC Bound in KNRs). Under the conditions of Theorem 4.1, suppose [ a)ll2 S [| = O(1), 5² O(1), X = O(1,, and [W]] I cw hold. We set x1 = (T))(T)) and
d2 O(Rmax MP(T))MK) For any good comparator policy x, with probability at least 1 8, the yielded TAGG by the KNR-modified Algorithm I satisfies that
((T) ((TAL)) so
Rmuxx
log AA
+0
(xRmuxen(())
log(RR//)
T
N
log( log (HHHN///)
+ + [Rmax (TT) + MB (2
+ -F(N,))
N
where the coefficients
Ax-=
Armase AmmasxEEn MR = max sup LE[L-T
T(N,5) =
x T
/ //rank(Ep)) flog (exp (rank ( /8)]/N, and & = || + d' min(d, rank (In. 1og(1/5)) log(1 + N). The complexity of the function class in the KNR settings is characterized as Np = rank (.,)) d' log(1 - 2cw N), NR = d' log(1 + 2N).
In comparison to Theorem 4.1, the main contribution of the modified algorithm MCP-KNR is on exploiting the KNR structures and improving the regret bound. In particular, the appearance of rank(E,)) in the bound, instead of the feature dimension d, ensures that the result adapts to the complexity of the data distribution. Notably, the bound holds even when d is infinite, provided that the data concentrates
oiw-dimensional subspace.
7

---

Published as a conference paper at ICLR 2026
5.2 FACTORED MODELS
Factored models provide a compact representation for large-scale MDPs by exploiting structure in the state space [Osband & Van Roy, 2014]. Let N+ and B be small finite set. Rather than modeling the full transition probability over all state variables, they decompose the state E IRd into components and assume that each component s(i] of the next state depends only on small set of parent variables P S [L d]. This yields a transition function of the form P(s (s, a) = I (. [[]]s[ a). a), significantly reducing the number of parameters compared to the unfactored case. This factorization drastically reduces the model complexity: the number of parameters for the transition function becomes I, Σi= i=1 |A| |B I+ as opposed to the exponential size ((B)) in the fully connected case, enabling more sample-efficien learning from limited data. To adapt Algorithm to this structured setting and formulate new Seerittnn-MCP-Factton
modify the maximum likelihood estimation step by estimating each transition component P; separately via P: = arg maxp ED [log P
[i]|s [Pt), a)], and reconstruct the full transition model as P 1==1 P.. Before we present the regret guarantee for HCCP-Factored, let define new concentrability coefficient. Instead of using the density ratio over the full state space, we measure it locally for each factor of the transition model. Specifically, for any comparator policy TT, we define
d
(d ( ((s( ( 27
(f(())
rE[L:d] = max ((sa)r
We should note that this factored concentrability coefficient (F(T) single-policy concentrability coefficient, i.e., Cf(T)) S sup(s.a.h) pa(e.a) bound for the MCP-Factored algorithm.
() relaxes the density ratio-based (h,s,a)) We now present the regret
Theorem 5.2 (PAC Bound in Factored Models), Under the conditions of Theorem 4.1, we set
A = (T)) x2 = (Bmax / (f)) For any good comparator policy n, with probubility least 6, after sufficiently large number of iterations, the yielded TAGG by the Factored Factored-modifie modified Algorithm I satisfies that the regret J(TT) - (Tx is upper bounded by
o HC RT)(
log(rLr/6) N
+ (C5(T)+MM) HR
dLp log log(L,, (d
N
Here, Mē- = max max B
(s.0)=u
(113)))))
and L, = EE | where Ue
1E[1:T] 1:d]
denotes the effective dimension of the state variables on which the reward model depends.
The key observation in Theorem 5.2 is follows: Instead of depending the full state space as in the results for general function approximation, MCP-Factored leverages the structure of factored models and therefore the PAC bound scales with the number of factors and the sizes of their parent sets, summarized by the complexity term Lp and L,. This avoids the exponential dependence on the state dimension d seen the non-factored models.
EXPERIMENTS
Benchmarks and Evaluation. We evaluate the algorithmic performance of MCP on Meta-World benchmark datasets with preference feedback [Yu et al., 2020a; Kang & Oh, 2025]. We refer the readers to the Appendix F for details of benchmarks. In Table 2, we summarize the success rates of MCP and several competitive baselines, including Oracle (IQL [Kostrikov et al., 2021] trained on ground-trutt explicit rewards), MR [Kim al., 2023], PT [Kim al., 2023], DPPO [An et al., 2023], IPL [Hejna & Sadigh, 2023], and APPO [Kan & Oh, 2025). Most notably. MCP achieves the best mean performance for tasks with varying sizes of preference samples. Another noteworthy observation is that MCP overwhelmingly outperforms APPO for the majority of the tasks. Although MCP and APPO are all model-based algorithms, APPO leverages the extra value function modeling to locally smooth over gaps in data coverage. This smoothing based mitigation is still not sufficient to guarantee robustness in poor data coverage scenarios. Due to page limits, we provide the additional empirical results and implementation details in Appendix G.
8

---

Published as a conference paper at ICLR 2026
Effeet of Dataset Size. To evaluate the performance of MCP under varying amounts of preference data especiall in small-data regimes we train the model using different numbers of preferences, ranging from 100 to 2000, shown in Figure ((a). Notably, N 100, MCP maintains a relatively high success rate, highlighting its potential for applications with small preference datasets, such as those in medicine and finance.
with extremely limited data, i.c.,
Table 2: Below reports the success rates on the World medium-replay benchmark using 500 and 1000 preference based feedback samples, averaged across three random seeds. Baseline results are from the respective papers. Note that the top two algorithms in each experiment are highlighted bold, with the best-performing algorithm additionally shaded light gray.
Dataset & Methods
Oracle
MR
PT
DPPO
IPL
APPO
MCP
BPT-500
68.33
10.08=157
22.87=006
3.93=434
34.73-130
53.52=139
56.00
box closellseSSS
93.401..10
29.12
0.33
10.20_
5.93
18.24
7.20
sweep-500
98
86,9 96.9
43.07=046
10,47
27.20=135
26.8 36.80
11.47
BPT-wall-500
Le
0.3
8.93
1:0
dial-turn-500
75.40
61.44=605
68. 18.67=1124
26.67=223
31,5331125
80.96 40
38.67=7765
sweep-into-500
78.80 766
28 28.40=554
20.53
23.07=1.0
32.20=7.35
24
33,07
drawer drawer--pen--00
3
19.00_11s
87.68.
kow
lever-pull-500
98.47
79.28=2295
82,40=227
10.13=122
31.20=1855
75.76=7.1
80.27=
Average Rank Rank-500 -500
3.000
3,500
5,250
4.000
2,875
2,375
BPT-1000
18.27.2
3.20
36.67_1174
19.04:1
62.93
box-close-1000
93.40
27.04
2.27
9,33
6.73
34.24
10.53
sweep-1000 1000
87.52
14
8.73
38.33
17.36
20.80
BPT-wall-1000
56.27.4432
0.48
2.13
0.27
14.07=115
62.96
69.73=151
dial-turn-1000
75.40
69.44:
68.80 s.30
36.40
43.93
81.44 :o
48.40 :..440.aa2
sweep-into-1000
78. 76
26. 53
20.27:a7:4
23.33
30.40=774
18. 16
34.13
drawer-open- 1000
100.00g000p
98.40=224
95.32:2426
36.47=7.0
28.53
98.56
99.07:2038
lever-pull-1000 1000
98.47
88.96
72.93
8.53
40.40.:174
76.96
83.33
Average Rank -1000
2.750
4.125
5.375
3.875
2.750
2.125
Average Rank
2.875
3.812
5,312
3,938
2,813
2,250
button-prees-topdoo--allll
lever-pull
drawar-open-100
100
X0-
Succees PMA
68
1030
1.0 "aiining
feedback
feedback
(a)
(b)
Figure 1: (a) Model performance varying with dataset size, i.e., the number of preference feedhack, ranging from 100 to 2000. (b) Training curves for drawer-open with 1000 preferences. We evaluate the results over three seeds, and the horizontal lines indicate the baseline performance summarized in Table 2.
Training Dynamics. We study the training dynamics of MCP for drawer-open with 1000 preferences in Figure 1(b). We evaluate the training performance every 5000 steps. Figure (bb) demonstrates that MCP achieves comparable performance with state-of-the-ae baselines in the task. Also, shows that MCP maintains stability in the policy improvement process, especially after sufficient training steps.
A Ablation Study. We investigate the contribution of each design component of MCP on the lever- pull task with 1000 preferences in Figure 2(a). The figure compares the full MCP algorithm with several variants obtained by removing the reward regularization (Aą = 0). removing the transition regularization (Ag 0). dropping the relative performance term ("No Relative Performance"), "), and replacing the model-based planner with a value-based variant ("Value- based MCP"). We evaluate the success rate every 5000 training steps and report the mean and standard deviation over three

---

Published as a conference paper at ICLR 2026
seeds. Figure 2(a) shows that full MCP rapidly reaches high success rate and clearly outperforms all ablated variants. In particular, the variants without reward or transition regularization make little progress, indicating that the conservative modeling terms essential for avoiding over-optimistic and unstable policies. The "No Relative Performance" variant exhibits large variability and unstable learning, which is consistent with the lack of the safe policy improvement guarantee provided by the celatve-perroomande objective. The "Value-based MCP" variant improves in the early stage but then degrades, suggesting that relying solely on value-based updates introduces significant bias from model errors. Overall, these results confirm that all four model components are crucial for guaranteeing the consistent policy improvement.
lever.pul
drawen open
120
Success rete
0.7
0.2
2.4
0.
02
0.5
Trining Step
probablity hippiing
(6)
(b)
Figure 2: (a) Ablation of MCP design components lever-pull-1000, (b) Robustness to label noise on drawer-open-106 1000 (performance vs. tabel-flipping probability).
Robustness to Label Noise. We further examine the robustness of MCP to noisy preference labels on the drawer-open task in Figure 2(b). Starting from dataset with 1000 preferences, introduce synthetic noise by independently flipping each pairwise label with probability and vary p over several levels. The figure reports the resulting success rates of MCP and MR, averaged over three seeds. As the label-flipping probability increases, the performance of both methods degrades, but MCP consistently maintains higher success rates and exhibits a slower decay than MR. This result demonstrates that the conservative design of MCP provides improved robustness to label noise.
Table 3: Sensitivity of MCP to regularization hyperparameters on drawer-open-1000 and sweep-into- 1000.
A1
le-1
3e-1
lel
drawer-open- -000 1000
94.8 + 3.85
97.60 + 2.33
99.07 + 0.38
94.53 + 4.03
86.27 + 10.31 19.87 7.92
sweep-into-100l 1000
32.4 1: 5.85
34.13 1 3.27
30.67 1: 4.71
24.0 1: 9.27
A
3e-3
le-2
3e-2
le-1
drawer-open- 1000 1000
90.93 = 7.30
96.47 = 4.10
99.07 + 0.38
89.60 + 9.53
80.13 = 15.19
swcep-into-100m 1000
20.8 = 10.32
23.6 + 5.85
28.27 + 7.08
34.13 3.27
25.73 + 6.15
Hyperparameter sensitivity. conduct& hyperparameter sensitivity analysis for the regulariza- tion weights X1 A2 on the drawer-open- 1000 and sweep-into- tasks over three seeds. As shown in Table 3, across several orders of magnitude for each parameter, the success rate of MCP changes only mildly, indicating that our method is not highly sensitive to the exact choice of Y A2.
DISCUSSION
In this work, we present the first provably sample-efficient and computationally tractable offline PhRL. under partial data coverage without known dynamics. The developed algorithm implicitly encodes the conservatism into a relative performance model-based planning procedure, and guarantees the algorithmic traceability. The learned policy is able to compete with any policies within the data coverage, making it robust even in scenarios with poor data coverage. We further extend and refine the theoretical results with general function approximation under specialized dynamic structures. As a potential research direction, it is interesting to extend the Bradley-Tere Terry- framework for modeling human preferences by incorporating alternative preference models, e.g., Thurstone model.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Riad Akrour, Marc Schoenauer, and Michèle Sebag. April: Active preference learning-based reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pp. 116-131. Springer, 2012.
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, and Hyun Oh Song. Direct preference-based policy optimization without reward modeling. Advances in Neural Information Processing Systems, 36:70247-70266, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint S.XXi:2204.05866 2022.
Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. Advances in Neural Information Processing Systems, 36:66845-668599 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Nickum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference Machine Learning, pp. 1165-1177. PMLR, 2020.
Edoardo Cetin and Oya Celiktutan. Learning pessimism for reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 6971-69999 2023.
Jonathan Chang. Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34:965-979, 2021.
Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence, pp. 378-388. PMLR, 2022.
Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in corman-in-the-ae tean-----the--oppp Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning, pp. 3773-3793, PMLR, 2022.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pp. 3852-3878. PMLR, 2022.
Heewoong Choi, Sangwon Jung, Hongjoon Ahn, and Taesup Moon. Listwise reward estimation for offline preference- based reinforcement learning. arXiv preprint arXiv:2408.04199 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: ptttps::/pproceedingg. seerseeeeedinnnennnneepee.... wwtrriisssccpppppeeeeeeeee ffrf_iiliii willeepppe less/epprr//0177 F111:/62222aad000091112222000ddd Paapeedddde--PPee.pppppp
Luc Devroye, Abbas Mchrabian, and Tommy Reddad. The total variation distance between high- dimensional gaussians with the same mean. arXiv preprint arXiv:1810.0869) 2018.
Christopher Diehl, Timo Sebastian Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. Uncertainty-awar aware model-based offline reinforcement learning for automated driving. 1EEE Robotics and Automation Letters, 8(2): 167-11177117 2023.

---

Published as a conference paper at ICLR 2026
Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model: Future model- ing beyond step-by-step rollout for offline reinforcement learning. arXiv preprint arXiv:2402.03577, 2024.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Sara A Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International conference on machine learning, pp. 2160-2169. PMIR, 2019.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arViv preprint arXiv:2209. 14375, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. pp.
Pmlr, 2018.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 111188806187277 2023.
Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop rl. In Conference on Robot Learning, pp. 2014-2025. PMLR, 2023.
Pushkala Jayaraman, Jacob Desman, Moein Sabounchi, Girish N Nadkarni, and Ankit Sakhuja. A primer reinforcement learning in medicine for clinicians. NPJ digital medicine, 7(1):337, 2024.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the ninefeenth international conference machine learning, pp. 267-274, 2002.
Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Information theoretic regret bounds for online nonlinear control. Advances in Neural Information Processing Systems, 13:15122253225 2020.
Hyungkyu Kang and Min-hwan Oh. Adversarial policy optimization for offline preference-based reinforcement learning. arXiv preprint arXiv:2503. 25t.::5503003006, 2025.
Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In Proceedings of the 16th International Joint Conference on Artificial Intelligence Volume 2, IICAI'9,, pp. 740-747 San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Pref- erence transformer: Modeling human preferences using transformers for rl. arXiv preprint arXiv:2303.00957, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69888 2014.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint urXiv:2110.0616)) 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 13:1179-119, 2020.
Kimin Lee, Laura Smith, and Picter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106. 050911 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learming Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.016433 2020.
12

---

Published as a conference paper at ICLR 2026
Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics, 52(1))233-2661 2024.
Yuhan Li, Wenzhuo Zhou, and Ruoqing Zhu. Quasi-optimal reinforcement learning with continuous actions. arXiv preprint arXiv:2301.0m 08940, 2023.
Haohong Lin, Wenhao Ding, Zuxin Liu, Yaru Niu, Jiacheng Zhu, Yuming Niu, and Ding Zhao. Saaet causal representation for trustworthy offline reinforcement learning autonomous driving. IEEE Robotics and Automation Letters, 95):4633-4444 2024.
David Lindner, Matteo Turchetta, Sebastian Tschiatschek, Kamil Ciosek, and Andreas Krause. Information directed reward learning for reinforcement learning. Advances in Neural Information Processing Systems, 34:3850-3862, 2021.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback. arXiv preprint 2XXi:2302.00676. 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17X.0 2017.
Daniel J Luckett, Eric B Laber, Anna R Kahkoska, David M Maahs, Elizabeth Mayer-Davis, and Michael R Kosorok. Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the american statistical association, 2020.
Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. Action- quantized offline reinforcement learning for robotic skill learning. In Conference on Robot Learning, pp. 13611 PMLR, 2023.
James MacGlashan, Mark K. Ho, Robert Loftin, Bei Peng. Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael I.. Littman. Interactive learning from deeeedent human feedback. In Doina Precup and Yee Whye Teh (eds.), Proceedings the 34th International Conference Ma- chine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2285-2294 PMLR, 06-11 Aug 2017. URL https:://// http///occeeeeeeceecooooccoccccc ngs .mlr www...ppess/v/////. wtw.pp/ss////o//maeaaaaaaaaaaa/a html.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, al. Webgpt: Browser-assisted pnestion-answering with human feedback. arXiv preprint arXw:2111.009322.222 2021.
lan Osband and Benjamin Van Roy. Near-optimal reinforcement learning in factored mdps. Advances in Neural Information Processing Systems, 27, 2014.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: reinforcement learning with trajectory preferences. arXiv preprint arXiv:2 (tt:2///..04500. 1.04550, 2021.
Alizée Pace, Bernhard Schölkopf, Gunnar Rätsch, and Giorgia Ramponi. Preference elicitation for offline reinforcement learning. arXiv preprini urXiv:2406. 18450, 2024.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lec. Surf: Semi-supervised reward learning with data augmentation for feedback-efffcient preference-based reinforcement learning. arXiv preprint arXiv:2203. 10050, 2022.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE transactions on neural networks and learning systems, 358)) 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization, arXiv preprini 10.0022210.01241 2022.
13

---

Published as a conference paper at ICLR 2026
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning, Advances in neural information processing systems, 35:16082-16097, 2022.
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-based learning of reward functions. 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 150.0003000 2015.
Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 25(200): 1-91, 2024.
Daniel Shin, Anca D Dragan, and Daniel S Brown. Benchmarks and algorithms for offline preference- based reward learning. arXiv preprint arXiv:2301.01399 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Yihao Sun. Offlinerl-kit: An elegant pytorch offline reinforcement learning library. https: //github. cww.cooo//ooom haosunl 1eemnn124/444
2023.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4906-4913. IEEE, 2012.
Masatoshi Uchara and Wen Sun. Pessimistic model-based offline RL: PAC bounds and posterior sampling under partial coverage. CoRR, abs/2107 06626, 2021a. URL httpttpps//aaxx...r.rr. abs/2107. 02226.
Masatoshi Uchara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226, 2021b.
Martin J Wainwright. Hligh-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cam- bridge university press, 2019.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaii. v32in.....ii........ v32.1. 11455. URL /ttps::/ojs.. aaai. org/ sete/indxx... php/ AAAAAAAAI/II..
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136): 2017. URL http: / jmlr. org/ hatp//appprr//eeeeeeeeeeeeeeeeee
/ttesvvlB. vv/// 1 66 634. html.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.1088 2021.
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 14:6683-6694, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020a.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information 2020b.
Processing Systems, 33:14129--411
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline mhodel-based policy optimization. Advances in neural information processing systems, 34:28954-2899 2021.
14

---

Published as a conference paper at ICLR 2026
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 1158443-54466 2020.
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability, In Conference Learning Theory, pp. 2730-2775. PMLR, 2022.
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305. 14816, 2023a.
Wenhao Zhan, Masatoshi Uchara, Wen Sun, and Jason D. Lee. How to query human feedback efficiently in RL? In ICML 2023 Workshop The Many Facets of Preference-Basee Learning. 2023b.
BtRpstss: / /pwpreevee.eeeeeeeeeeeeeee net forumm
Wenzhuo Zhou, Bi-level offline policy optimization with limited exploration. Advances in Neural Information Processing Systems, 36:55022-550355 2023.
Wenzhuo Zhou, Yuhan Li, Ruoqing Zhu, and Annie Qu Distributional shift-aware off-policy interval estimation: A unified error quantification framework. arXiv preprini arXivv2309. 13278, 2023.
Wenzhuo Zhou, Ruoqing Zhu, and Annie Qu. Estimating optimal infinite horizon dynamic treatment regimes via pt-learingg Journal of the American Statistical Association, 119(545):225 -6886,2566666- 2024.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise k-wise comparisons. In International Conference on Machine Learning, pp. 43037-43067. PMLR, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint ar 1909:::99900000. 2019.
15

---

Appendix
Table of Contents
A
Additional Related Work
16
Technical Proofs for General Function Approximation
17
C
Optimization Error
................................
23
D
Technical Proofs for KNRs Technical Proofs for Factored Models Experimental Setup...
26
................................
30
33
Implementation Details H Visualization of Training Dynamies
................................
37
J
Additional Experimental Results
37
A RELATED WORK
Offline Preference-based Reinforcement Learning. Recent advancements in offline PbRL have investigated variety of methods for incorporating preference feedback into policy learning. Some of these approaches rely explicit reward modeling, while others aim to bypass it altogether. For instance, OPRL. [Shin et al., 2023] introduces an active querying mechanism over offline data infer reward model from preferences, which is subsequently used in standard offline RL pipelines. PT [Kim et al., 2023] proposes transformer- based reward model designed to capture non-Markoving dependencies in human feedback. In contrast, IPL [Hejna & Sadigh, 2023] avoids reward modeling entirely by directly optimizing a Q-function aligned with preferences through the inverse Bellman operator. Similarly, DPPO [An et al., 2023] formulates preference learning contrastive objective, enabling direct policy optimization based on preference data without estimating an intermediate reward function. While these methods demonstrate promising empirical performance, they generally lack sample complexity guarantees, which raises concerns about their theoretical robustness. To overcome these limitations, recent works have proposed offline PbRL algorithms with theoretical guarantees under partial data coverage. For instance, [Zhu et al., 2023] provides the first sample complexity results under linear reward models, while subsequent works [Zhan et al., 2023a; Pace et al., 2024] extend these to general function approximation using confidence-set-band set-baaed policy optimization. Yet, constructing and optimizing over such sets is often computationally intensive. the recent work [Kang& & Oh, 2025), their algorithms either assume known transition dynamics require fitting an extra value function that depends on the learned transition model via Bellman recursion to perform conservatism. This allows the value function to locally smooth over gaps in data coverage if the value function is well-apppoximmated However, this smoothing-basee mitigation strategy intrinsically requires the realizability condition the value function and does not guarantee near-optimal regret under partial coverage. In contrast, our work proposes a model-based offline PbRL framework that encodes conservatism implicitly via relative performance objectives, avoiding confidence set construction and value function estimation. By learning both the reward and transition models from offline data and using them for conservative planning, our approach achieves PAC guarantees under general function approximation, without assuming known dynamics. Moreover, when applied to structured settings (e.g., kernelized nonlinear regulators or factored models), yields refined generalization guarantees.
Reinforcement Learning from Human-Al Preference. Unlike traditional RL, which relies on explicit numerical rewards for each state-action pair, reinforcement learning from human-Al pref- erence infers reward function by collecting pairwise preferences over trajectories [Wirth al., 2017; Akrour et al., 2012]. Various strategies have been proposed for eliciting preferences, typically assuming access to either a known transition model or an environment that supports interaction or rollouts [Brown et al., 2020; Christiano al., 2017; Chen et al., 2022: Pacchiano et al., 2021; Sadigh et al., 2017: Zhan et al., 2023b; Lindner et al., 2021; Stiennon et al., 2020; Park et al., 2022; Hejna III & Sadigh. 2023]. While effective in interactive online settings, these assumptions limit the applicability many algorithms the offline setting, where the agent must learn solely from a fixed dataset without further interaction.
16

---

Published as a conference paper at ICLR 2026
Offline RL. Offline RL has gained attention because it allows learning policies without interacting with the environment, which is important in areas where safety concerns or data collection costs make interaction difficult, e.g., healthcare [Luckett al., 2020: Li et al., 2023; Zhou et al., 2024; Jayaraman et al., 2024), autonomous driving [Yurtsever et al., 2020; Diehl et al., 2023; Lin et al., 2024], robotics [Kumar et al., 2020; Yu et al., 2021; Zhou et al., 2023; Luo et al., 2023]. However, offline RL is also difficult because the available data may not cover all relevant state-action pairs. This lack of coverage can cause errors during policy learning. To address this issue, many algorithms have been proposed that introduce conservative learning strategies to ensure reliable performance under limited data coverage [Kumar al., 2020; Jin al., 2021; Xie al., 2021; Uehara Sun, 2021b; Zhan et al., 2022; Blanchet et al., 2023; Zhou, 2023; Cetin & Celiktutan, 2023; Ding et al., 2024; Shi & Chi, 2024; Li al., 2024]. For more discussions offline RI, we refer the readers to [Levine et al., 2020; Prudencio et al., 2023] for detailed reviews of this topic.
B TECHNICAL PROOFS FOR GENERAL FUNCTION APPROXIMATION
To begin with, we will introduce some technical notations used in this section to facilitate reading.
Summarization of Notations:
We use to represent the function class for the reward model, )/- represent the function class for transition dynamics. We use NR(1/N) to represent the 1/N-bracket number for reward model function class R. and Pp represent the 1/N-bracket number for transition dynamics function class (Pm)
We use y* to denote the ground-truti reward model, (P)) denote the ground-truth transition dynamics at time steps h [0 И 1]. We use F and M to denote their corresponding MI.E estimators, respectively.
Concentrability coefficient for reward:
E, Hree (1(((r))~((7)) (1-((())
CR(T) sup rER
E Byt (r(()) r(70))
Concentrability coefficient for transition:
(s (Dr (Pa(-Is, a), (+(ss,)))
Cp(T) max sup NE(D:H-1] Pa EPP E (ssa)aa [Dv a)
Empirical regularizer for reward model:
1 (r;r; D) N
-(((()))-(()))))) - (())
Empirical regularizer for transition dynamics:
N H-1 N
Eg((Ph)
Ph (a,)) - Phis 15 (
a
Lemma B.I. Let M ((P)) rM). M' ((Ph) rMr) be two MDPs defined over the
same state-action space and initial state. We define Jas (T)) == J(T; TaI, (Ph ( H-1), For any policy - A(A) E II. By Assumption 2, we have
(M (T)
H-1 R
Bn
Drv(PA Phr (1s,)))))) + Eroer
[Ta - ras (T)].
(nh 1
h=0
17

---

Published as a conference paper at ICLR 2026
Jas (xT, begin expressing it the sum of two terms: one corresponding to the difference in rewards under the same dynamics, and
Proof of Lemma B.I. To analyze the difference JA (T)
another capturing the difference induced by the change in transition dynamics. Specifically, we write
(M(T)) Jas (T)) = Erwd" (1 nn-
[T))(T]] - E2 Er
(TAs((T)
B Br
(TM(T) (TM
(Pl h-o
(A)
Eradt
[ras (T)] Br
Ir (T)]
(nh Minu
(B)
Term (A) remains unchanged. Now we aim to bound term (B). By Lemma 9 in Uehara & Sun [2021a]. we have
H-1
[(B)]
< R
Σ
Bs
(Dsv(PM((1s,,)) a), Pl (M
PM
h=0
This completes the proof.
Lemma B.2. Zhan et al. [2023a] For any reward model R, with probability at least 1 8, we huve
E. (l(r(()) -m(()))))r)))))))))))))))))))))) ((rn))))(((m1 ( CK2 log Log(NR(1//N)/))
N
where c is universal constant, K == infze|-roax:Fmal ('))) measures the nonlinearity of the link function.
Lemma B.3. Zhan et al. [2023a] With probability at least 1 5, for all h E [0: H - 11, it holds that
E (x,a)=u
clog (HNP(1/N)/))
N
where e > is a universal constant.
Lemma B.4. Let (Phh=u be the transition models selected line 5 of Algorithm corresponding for iterations E [1: T). Then with probability at least 1 8, have
Rmaxx
(L/N)/) + H.
ES(P)) <
X2
N
where is universal constant.
Proof of Lemma B.4. Recall that in line 5 in Algorithm 1, the transition model {PA) as minimizers of
H-1 is selected hh
HRgT, (P Eroppes [r(r)]
where ne is the policy at iteration t. So we have
J (TETe (P H-1) Erepoes [ne(T]] N
min
(J (PN (-1) Ereeeny [r(r)] & (eE()))))))
.ER.(P.EPP. M-11
< (TATe, (P E Erenyr (rr (T;) + +
18

---

Published as a conference paper at ICLR 2026
Rearrange the equation above, and we have
( (TATO (PT) - We also have
S S2281((P111100
(B.1)
J (m (PK) Brapeny [m)(T) (r (
min PER,PREPk
(J (TAi, ( Braweer (r(7) + (r; A
SJ (TATT (Pr Ex-yne (i + (c
By rearranging the equation above, we can obtain
J (TATE, (P)) M))) J 115T (PA)
(TATE (1%) (TiiT
1150 11 +
(TeTe ( (P
Teites (P
H-1
s : + Rmax
Drv(Ba
h=0
clog(HNNrL//N1/0)
S X2E2 (1 D) + H Rmmxx Mp
N
where Mp max max Eis.)ma Drv (Pr 1E[1:T The (0:- Eqa,ayeaa, Drv (7 the third step by equation (B.1)
and Lemma B.I. This completes the proof.
Lemma B.5. Let r* E R denote the ground -truth reward model, and let F denote the maximum likelihood estimator. Then with probability at least - 6, we have
N N
Er(r: D)
[r' (7m.2) - ()n.0))) -
log(NR (1/N)/8)
Dog(NR(1/N//))
scy
cRmax
where e is a universal constant.
Proof of Lemma B.5. By Assumption 2, both and are bounded by Rmax, for any pair of trajectories, we have
((()())--(((00))) - -(F(7()) -f+()) < 4ftmxx
Define f. f== = (r -
By Hoeffding's inequality and union bound, we obtain
1 N P
((
2N (T(ARmax))
Σ lffll - Ell 2 c) s 22Nx(1/N) exp
Now solve for c such that the RHS we get
log (2N (gg(2R(1//)//
4R 4m max
2NV
19

---

Published as a conference paper at ICLR 2026
Mcanwhile, from Lemma B.2, by Jensen's inequality, we also have
< ch² log Log(Nx<
ck2 (o22
E B||f+
E|ll -
N
N
Putting everything together, we obtain
x² log (NR(1/N)/0) + c
1og(NR(1/N))
; S
N
N
where с >> is a universal constant. This completes the proof.
Lemma B.6. Let (PRY denote the ground-truth transition dynamics at time steps E [0 H and let PP)]/= H-1 denote the maximum likelihood estimator. Then, with probubility at least 1 8, have
E2(1P))
ΣΣΣ
Ph .otk owt) Ph(shi a g..
log (NN((/))
ScHl
N
where e >> is a universal constant.
Proof of Lemma B.6. Since the N trajectory pairs sampled i.i.d, we first take expectation the data distribution 14h, and average over
Σ
Σ E Conanaup
(: 8h, an)
Pr 8h, an) |[]
h=0 t=
By Lemma B.3 and Jensen's inequality, summing over all and we obtain
([Es((P:) D)1 S cHl log (og(H/N))/)))))
N
where e > 0 is a universal constant. Applying Hoeffding's inequality and union bound over the transition dynamics function class
N
(-N2
Ne²
Σ
((P::) E(8,((P))) D)] 2c = 2HHN exp
(2(
N
2H2
n=1
Solve for such that the RHS is bounded by 5, we obtain
2 2og 2
e=Hl H1
N
By applying Lemma B.4, we have
Ee ((((P)) D)
cH Rp
log ((H (og(HNNr(1/N//5) (N/N//)
1og(HNp(1/N))
+ +cH
A
N
This completes the proof.
20

---

Published as a conference paper at ICLR 2026
Lemma B.7. Let ne denote the policy at iterations t [1 T). For the associated reward function Tt.
transition dynamics hhh 1-11 selected in line 5 of the Algorithm 1, we have
((x+) 2 J Tei (P%) - Br-eere [r((T]] Ero (T)(]]
- -r - - -
Proof of Lemma B.7. Please recall that J(m)) [ ( ( So we have
((T.) ((m.) - Er-erer (r)) &1& + (2 D) + Bramnes (r))
-11E1(+";)) -
min (J(TETT (Ph ) E B [r(T)] + (1;)(D;
((Ph)
+>gE2(( (P M- D)) + ( (T) (1))))-(1(()))))1))))) \rE((");))
-2 D)
J (mar: re (PP (=-)) ETippEy [r)) (1E((1 D) + 23 (
+ Ey (r* (T) - X1 (r"; D) X28 (PP) H-1, : D)
2 TATE (P/) (-1))-1
Exppees [T](T) + By [r (T)
where the third step by the optimality of the r and (PK) This completes the proof.
Now we want to make some modifications to the Lemma B.I to serve for proof of the main theorem later.
Lemma B.8. Let II be an arbitrary policy. Then, for the reward model ra selected in Line 5 of Algorithm I corresponding to n,, with probability at least 1 - 5. for all E [1 TT, we have
x² loog(N (1/N)/8)
Eyt 70 (
N
where 0 is universal constant.
Proof of Lemma B.8
Exi 7 (llre((') - rr((7) - -(-)))-((>)))]
< <R(TT) E, - r2(79) - - ((*()))
{c(n(T) R² N (1/N)/5)
N
where the second step is by Lemma B.2. This completes the proof.
Lemma B.9. Let II be arbitrary policy. Then, for the transition model (Pr) {Ph]
H-1 h=0 selected in
Line 5 of Algorithm corresponding to w, with probability at least 8, for all E [1 T], we have
#-1 Σ
Eag
D (T
Rmax-Mp A2
log(HNN(1/NN)/)
No
5 {((((T)
where is universal constant.
21

---

Published as a conference paper at ICLR 2026
Proof of Lemma B.9.
Σ h=0
Bgg
H
$ Cp(T) (T)
Eus [DiviPA((ss.))PP(((((s.))
h=0
H-1
S (p(r)
Ph(--s,a) Pn (-1s, a)
+ Bpn
(- a), Ph (ss,, a)
TV
(h)
The bound for term (b) can be obtained from Lemma B.3. Now we want to bound term (a) by Hoeffding's inequality and apply the union bound
NV Σ
H-1
Es((P))
E2(1
Σ
Eus
rv 26
==1
h=0
No²2
S (1/N exp (-2H²
can be obtained from Lemma B.6. Solving for e such that the where the bound for (2( - RHS is at most δ
H log (2HNN(1/NN)/))
N
Combining all the equations above completes the proof.
We are now ready to establish the proof of the main theorem. Before we start, we define JM, (T) H-1
J rı, (PA) where Ti and {PP k= are conservatively estimated models corresponding to T. and please recall that JTT) J [T;; (-)
Proof of Theorem 4.1.
J(T) ALG
T Σ (J(x)= -(x))
1
$ T Σ (J(z)= (Brwaver (T)] - E, Errane) - (m)))
1
+ +rE((";; + ( H-1
Ø-1
(T) + Rmax
Σ
Bl DTv (D
h=0
+Er-or (r* - -re(r)] (E (r* (T*(
E Eynepne (r((T))) -M, (mr))
(riEi(r";;)
1 T
#-1
(T) + +Rmax
E B(sra)aa (Dv
T [=1
1=0l
+
+ (r D) 1282(( +
1
S
- J. -ME - D))
T
[=1
22

---

Published as a conference paper at ICLR 2026
Mm
log HHHHN (1/NN))
log(HNp (1HHN)/)))))))
H RmaxCr(T)
A2
R²2 Log(NR(1/NN)/))
Cn(TT)
log AA
2 Rmaxx1
+ xrEI((";; x222((PP)))
T
Rmax
Iog(HNp(1/N)/0)
+ H Rmmax Cp(TT
A2
N
N
R2 log (og(NR(1/)))/)
CR(TT)
N
log LAI
122 log (NR (NR(1/N)) (1/N)/8)
log (2N (1 (1/N)/5)
2 Rmax
+
A
4Rmax
T
N
2N
log( (L/NN/())/)
2log (Lo
+ A2
HV
H
N
N
Mp
PA(HNP(L//N///
N
+ H Rmax C
Cp(T)
A2
N
N
R2 log Log(NR(1/N)//)
Cn(w))
N
where the second step by Lemma B.7, the third step is by Lemma B.1, the fifth step by Lemma B.9, Lemma B.8. the sixth step is by Lemma C.4. the seventh step is by Lemma B.5, Lemma B.6. completes the proof.
Substituting A1 = O (CR(T)), A = o (PTT)
C OPTIMIZATION ERROR
For simplicity, we assume that the transition model Pj is homogeneous across all steps h E [0 H-1). That is, write P; P for all h. This simplification is made only for notational clarity and does not affect the correctness of the result.
For each iteration E [1 TJ, the algorithm proceeds follows:
Step 1. Model Selection:
Let the reward model 7; and transition models P. be selected by solving the following penalized objective
Te, P. =min J (mir, P) Eron,e (rrm)),/((F]])) + (r; A2E)(P: r.P
Step 2. Policy Improvement:
Update the policy using an exponentiated update rule based on the estimated reward signal
(els(a)s) (als)) exp (n/Ea73 [ra(T)]s,al)
Let JM, (T) denote the expected return under reward model and transition dynamics P,. We define the cumulative regret over I iterations as
Σ
Ry = max xell
Additionally, define the entropy-like like regularization function
(T)
Σ
(aass) log (al))
17 aea
23

---

Published as a conference paper at ICLR 2026
Lemma C.1. Let π S + ((A) E II an arbitrary policy. Then for any state E S, the following
inequality holds
(et1(11)), R. ATA (f(T))s,))) (1)) 2 Σ (x((s). Bog (((r))s.))) (T).
Proof of Lemma C.I. We prove the results via mathematical induction over T by following [Xie al., 2021]. When T both sides of the inequality are zero. This holds because iterations executed, and we define (T1) to be the entropy of the initial uniform policy. Thus, the inequality trivially holds.
Assume the statement holds for T = '". That is,
T
T
meta(11s) Bayz (F((t))s.)) (1) 2 Σ
(-[ss) Bay (F(())ss.)) (T).
We want to prove it for 7 = T + 1. Consider: Et' (-+1)(13)) Ray (f(T))o..)) (m))))).
can decompose this as
T+4((1)), Eay [F((t]ss,]] - (T((T)) -
Te+1 (-1s),,,,, Bagz (F.(t))s.))) (T() + TT B. B dpps
(F)+s()))s.)))
tr, Ba [F((T) (3T+4))) + (mr+2(11s +), Bu (Fr+(()))..)))
Σ (n,, Rag [f((T)] - (xT++2)
(T, Bag [T) s, -)) - (TT)
where the second step is by the induction hypothesis, the fourth step is by the optimality of 42142(-1s).
Le, TT'22 = arg max, {
2'(1ss,, (1+s)),E,r+) xe pp1s
(F)+)())),.)))
vo((T))
The proof is completed.
Lemma C.2. Let E II be an arbitrary policy. For any state s E S, The following inequality holds
T
T
Σ ( - (.(-)8), Eazs (r((7))s, 1=1
S Σ (-16) Te(.1s), Eagz [r((T)(s, (.(T))
Proof of Lemma C.2. We begin by writing the LHS as
(-(1s)) Te((1)), Bagg (re(T))s.:1)
Σ ((T(116), Borg (r((T)ls.1) (Ta((Is), Bayg (r((t))s,:)))
24

---

Published as a conference paper at ICLR 2026
T Te+1(-18, Boy (r((t)ls,
(++1(118) (-(+))), Bay (rr(t)ls,))) +
+
(-11s)
t=1
(1)s) mT(-18 +), Bap (F((T)ls,.)) (TT),,
t=1
where the last inequality is by Lemma C.1. This completes the proof.
Lemma C.3. Let a E II be an arbitrary policy. Then for any state E S, if we set the learning rate
log 212 T
(x(+|s) - mc(-)s), [re(r)]s,-]> < 2mmax v log AT.
Proof of Lemma C.3. We define the surrogate objective accumulated over iterations as
Fa,(T) (r(-)s), [rx(T) v/. (T).
Let (-I.) denote the Bergman divergence with respect Fa... By the definition of Bergman divergence, we have
F. (m,) = FFa((T(1) + (Te(-)s)
TT41(18)) V. (Fea(T)) + (Te $ (T+(())11) + B5,, (Tx (T+11)
Far(Fe+1) - Bv, (Te((
= (o S (T41))))) - (Te)
(++)(11s) Te(-1s), (ra(t)ls.1).
where the second step is because Te is the maximizer of Fs,t, and the gradient term is non-positive, the third step is because both Fs,t and (m) are linear and convex, we have: By,, (Tel[
- BB, (++])
To convert the Bergman divergence into squared norm, we follow [Xie et al., 2021] by applying the second-order Taylor expansion
Bg, (Te = [|/T((18)) (-11(-)s)1 (x)) (t))
where ,/ am + (1 - a) for some a E [o, 1]. and H.. the hessian of vh...
Using Fe+1 Te, Ir(())s < || Tell |Ea (Tr( (x)
inequality
From the Hessian of wha is known that
B (T())s) mr!
Rmmax.
Combining everything
T++1 Te, Bazz [re (T]]ss -1) 5 2B., (Tell Rajmmax
Te41 Tx, Bag (r < V2 (Tett) Te, Bagy (rr((T) (a)))))))))))))))))) Vnlmmax
25

---

Published as a conference paper at ICLR 2026
T1+1 ne, Tr, Bag (n(())s 11) I 2
T++1 Te, Buz (x7((re(T)(s,))
< 21 muax? T.
By Lemma C.2, we have
T
T 2 TT-1 - Te, Bay (rx(T) s, 1.. (m))
n ne. E
Σ
1=1
log LA
< R max
+
where the second step is because is the uniform policy, Choosing proof.
log 212
concludes the
Lemma C.4. Let TAG argmax Et= (M. JM. (Te) and rj =
log AA
T we have
2
max
Ry 22mmxx v log [A].
Proof of Lemma C.4. Please recall that (x) J(T, rt, Pt). We apply the standard performance difference Lemma Kakade Langford [2002), which gives
T
RT
JMc (a ALG) - JM.
E () -E Ex (re |]
Pp
T
E
ALG
(-1s) m)-sss),, B ( -1)
2R
2T log
[A).
where the last step is by Lemma C.3. This completes the proof.
D TECHNICAL PROOFS FOR KNRS
In the Kernelized Nonlinear Regulator (KNR) setting, the structural constraint is imposed on the transition probability model, rather than the reward model. Consequently, we focus on modifying the related Lemmas the transition model accordingly. Before we begin, let's recall some definitions and notations for better readability.
Summarization of Notations:
We use d to represents the dimension of the feature mapping o(s,a), while denotes the dimension of the state space S.
Σn Ei= ((s )o (st, [xx.))// a)o(s, a) TI.
as) a:) + AI, Σx
B(s,a)=at (0( ()(s. and Σμ-
Relative condition number: eK (I)) = supwERd
x E
Lemma D.I. Let Φ S A - TRd be feature mapping with [|ō(s, a)lla 1, and define the KNR
transition model class
FRNR == (Sw(s, a) = W Wo(s.0l a) || LL), W e Bdd'
where d' is the dimension the state space.
26

---

Published as a conference paper at ICLR 2026
The following cover number bounds hold
H-1 log , 1/N) & (aan log (1 + 2L.N = Np.
No (R, & raan((p)))))) log + 2N)= NR.
Proof of Lemma D.1. We begin with the observation that for any W E Rd''dd the function [s a) W ((s, a) is linear in ((),a))
Let Hl, == (m,) C Rd be the image of E.. and let r = dim(H,,) = rank(_,)) By the definition of image, we have
H₂. Span {0(s, a) (s,a) a) E A(u)}.
Hence, every d(s, a) lies in Hn, and there exists an orthonormal matrix U E Rd whose columns form basis for Hys such that
((s,a) = U=(s, a), =(8, E R". Substituting into the function expression, we obtain
(s( a) = Wo(s,a) = wU=(8,
Define W = WU Rd'x xr, Then we can rewrite
fs = (2.,
Because U has orthonormal columns, we have
||]|F = =|WU & || |U1|a = [
where || Therefore, if ||W||= s L, then [[]] S L.
Thus, the function class FKNR is equivalent to the reduced class
f= $ (8, W=(s,a): W IR 1I s 43
By Example 5.8 in Section Wainwright [2019], we have that
logN((F S rd' log 1 2L
This completes the proof.
Lemma D.2. With probability at least 1- 8, we have
(((PM) - Σ
ni nn .op .ahit)
- Ph ni (s afit))
h=0
i=U
c1g S
log (2HNr/0) + HT((N.))
Hx
5
N
where &
a / d'rank(E,) (rank(EE)) + log(c2/0))
log(1 + N), I(N,0)
rank E, N Ci and 02 are universal constants.
Proof of Lemma D.2. By the definition of the regularization term, we have
N
H-1
Ee(PPi) D)
ΣΣΣ
Pt
(
oh (s n. ohh
n.
n=1 h=0 ==
Substitute the KNR matrix into the equation above
N カ-1
Es((P:) D) <
ΣΣΣ
IW*
W) .
NÇ
==1
R=0 1=0
27

---

Published as a conference paper at ICLR 2026
N H-1
Σ ΣΣΣ
Σ
( W) 1/
6(8 n.t -0
mi
NÇ
n=l h=0 1=0
H-1
1=
where the first step is by Lemma 13 in Uchara & Sun [2021b], the second step is by Sauchy-Schwarz inequality, and the last step is by Lemma in Uehara Sun [2021b], i.e.,
( ( W) (En 12 S V || 122 + d' rank(E,) (rank( (x,) + log(c2/0)) log(1 + N) = 5.
We assume that: Zo == En Ei 1666% n.d (.
so we have
E D)
Zz
N n=1
We also assume: Each ((s, a) E Re satisfies ||o(s, a)ll 1, have
(0(ss))lls I Amax(Ex))) [l a]](, = / Amax(En' = ASn
From theorem 21 in Chang al. [2021]. with probability at least - 8, we have
B(s.e)=plll(ss, l)l, s v rank p1 (rank | + In(c2/0)} (N)
N
By applying Hoeffding's inequality and union bound, we have
P
N
1
Σ
N
im) Φ
-
B[Zo]
2e < 2HN exp
2222222
Ne
n=1
2
log (2HNp/0)
H
N
By Lemma B.4, we have
E2((PA) H-1. D)
cH Paax M
ylog (HNp/8)
o
log (HNp/8) + HT(N,)))
+
X2
N
N
This completes the proof.
Now we want to modify Lemma B.1 to the KNR setting. Specifically, we only need to modify the first term because that the constraint of the KNR method, i.e, now we want bound the term
H-1
Σ B(s.ay=az (Drv ((.
Lemma D.3. Let n. be an arbitrary policy that belongs to IT. Then for the transition model {Pt (P))H- H-1 selected in Line of Algorithm I corresponding Tt, with probability at least - 5, the following holds
h=
Σ
Bissa)aay [Drv (P) PT(-ss,))
h=0
MI Xa
/
log (HNp/8)
log (HNp/8) + I(N..))
CC (T)
A
N
N
where O is universal constant.
28

---

Published as a conference paper at ICLR 2026
Proof of Lemma D.3. We can start by decomposing the term above
(oja))dd [Drv ()
4=0
eep
((aa))uu DDrv (-(s,,)))
e/ (T)
B B(xaa)aa
Patlsss)) lT.l)
a)
We already have the bound for the second term, so what we need to now is achieve bound for the first term. By Hoeffding's ss inequality and combined with Lemma D.2, we have
(s.a)-p || P1(-ls, a) Pa(-(s, ||v]
clmmaMp
log (HNPp/))
of <
log (HNP//) + P(N.0))
A2
N
N
where M
max sup E
En, En
This completes the proof.
Here is the proof of Theorem 5.1.
Proof of Corollary 5.1.
J ( (TALG
T Σ (J(z) - (((e))
< Σ (JM, (TO) + (1T)(T); >2Es((P)))
x² (1/N(1))
+En(T)y
N
MB
log (HNp/8) +
log (HNp/0 /8)
+ T(N.0))
+ H C CK (IT)
X2
N
N
log A
I Rmax V
R2 (08(Nx(1/N1/8)
+ CR(T))
N
Pemax MF
log (2HNp/0)
+ E A5.n1
log (2HNp/5)
+ +T(N.,0) s)
+ H eK (T)
Ag
N
N
log A
(v
R2 log (Nx/5)
log (2Nve/5)
< Rmaxx
d1
+ Rmaxx
T
N
2NV
log (HNp/5)
+ >2
Hd
+ HT(N,
N
29

---

Published as a conference paper at ICLR 2026
x² (1gNN/)(1/N))0)
C
N
R
/ log (HNp/8)
+ ( >
log (HNp/H / + T(NN ))
+ H Raxx e (T)
X2
N
N
where the second step is by Lemma B.7, Lemma B.I, the third step is by Lemma C.4, Lemma B.8, Lemma D.3, and The last step is by Lemma D.2, Lemma B.5. Substituting \ = (())(T))
A2 O(Rmax / e (x) completes the proof.
E TECHNICAL PROOFS FOR FACTORED MODELS
To begin with, we will introduce some notations that will be used in the proof' that follow. Summarization of Notations:
Number of parameters for the transition functions Lp = Ei |AI |B|
(1( 27 ((+l9%)..o)
Modified Concentrability Coefficient (T(TT) = max E ((saa>m
Lemma E.1. Let P7 be the ground-truth transition dynamics at time steps h E [0: H 11. with probability at least - 8, we have
Es(PP) H D)
ΣΣΣ
m.d T.d .ohht 13%
Pa (s 18h a.i)
BE
4=0 i=0
log HdL
dog LNd/5)
cH
cH
N
N
where is universal constant.
Proof of Lemma E.1. To start with, the following inequality holds
B(saa)-a
(Prr(
H B(aa)mm EDmrPPA(622220))PPPP(.
H-1
<
Σ V (B..)))
h=0
dL Loogoog(aal LNd/0)
ScH
N
where the second and third steps are by the definition of Factored models, and the last inequality is adapted from section C.5 in Uehara & Sun [2021b].
Define: Yn := Σ" - | P m. ah)))) Pa(--sh an.))
[0,4H]]
By Hoeffding's 's inequality and apply union bound, we have
1
N
-2Ne² (4H)²
log (og(2HL//0
Pr
Y
EYy) 26 <
2HdLexp
3 4H
N
2N
==1
Then with probability at least
log(HdL/0)
dL log LNd/0)
:((PP)) < cH
+ cH
30

---

Published as a conference paper at ICLR 2026
Then by Lemma B.4, we have
E2 ( :
H-1.
D)
H HMP Rmax
(dL(0))/0)
log /6)
dL (Llog(LNd/0)
+ H
+ Hy N
A2
N
N
This completes the proof.
Lemma E.2. Let be an arbitrary policy that belongs 1L. Then, for any transition dynamics H-1
(D selected in Line 5 of Algorithm corresponding The with probability at least 6, the following holds
H-1 Σ Eag ()) h=
[max
log (og(H))/0)
log (g
dL log (LN (LNd/o)
<cHef(TT)
A2
N
N
N
where is universal constant.
Proof of Lemma E.2. We start by decomposing the term above
#-
Eag
DIV(PP)(l,,,)
hm0
H-1
ef(T)
||| ()(()s,)) - a)
h=0
+ (s,)))p
-)) Pr [ (/lls
(.).)) -
d_ where (f(T) max ((,,a)=
By Hoeffding's inequality and combined with Lemma F.
E(s,))aa
cMF.
log ()
log( (HdT/)) + +av
dL dL (og((NN
+ov
A2
N
N
N
( (dT((o( {
where Mp max max ((,a))m
Combining all the terms above, we have
iel::
B(eea)=a (A(()s))))
PM [v]
MF
log( log(HdL/8)
log HdL/8)
dL log LNd/5)
S {
A2
N
N
N
This completes the proof.
Now we present the proof of the PAC bound for Factored Models.
Proof of Theorem 5.2.
((T) - ALCi
1 T
Σ (J(TT ( T
1=1
31

---

Published as a conference paper at ICLR 2026
T
(T) - (M.( + (1E1(r";)) + 2222(( D)
Rmax
log (og(H)L/0)
log log(HdL/8)
dL log (LN N
+ H Rmaxe C
d2
N
N
CK2 101(rL1/0)
+ CR(TT)
N
log AAI
< Rmax
+ +)EE)(r::)) + 22 D)
T
RmaxMM
log(HdL) (d)L)
log (dL/8))
L log (log(LNdd) LNN
H Rmxx (T(
A2
N
N
N
R2 log(rL1/6)
+ +
N
log
A
R2 log (rL/5)
log (r
$ Rmaxx
+ d1
+ Rmaxx
T
N
N
log(dL/0)
dL loogloodddi
+ A2
HV
+HV
N
N
log log(H (dL
log (dL/8)
dL (oN( N
H Rimax e
A2
N
N
x2 log(rL1//))
Cn(zr)
N
where the second step is by Lemma B.7, Lemma B.1, Lemma B.8, Lemma E.2, the third step is by Lemma C.4, and the last step by Lemma B.5, Lemma E.I. Substituting X ((Cn(T)).
d = O[max / (C((T))F) completes the proof.
32

---

Published as a conference paper at ICLR 2026
F EXPERIMENTAL SETUP
F.1 DATASETS
Our method is assessed using the Meta-World datasets introduced by Yu al. [2020a], specifically the medium-replay curated by Choi et al. [2024]. The primary set of experiments focuses on the medium-replay dataset. key advantage of these datasets is that policies cannot achieve good performance when trained with incorrect reward signals (e.g., random or constant), making them particularly well-suited for evaluating offline reinforcement learning approaches. This is important because some offline RL agents may still exhibit seemingly successful behavior even under faulty reward supervision. Further dataset-related information is available in Choi et al. [2024].
Note that the abbreviation BPT indicates button button-press-topee press-topoow.
The medium-replay dataset used in our study, originally introduced by Choi et al. [2024], comprises replay buffers created using SAC agents [Haarnoja et al., 2018], which exhibit an average success performance around 50%.
F.2 COMPUTATIONAL RESOURCES
Our experiments are conducted single Nvidia GeForce RTX 5090 GPU. Each training session consists of 100,000 gradient steps, taking approximately 50 minutes to complete (with evaluation).
33

---

Published as a conference paper at ICLR 2026
G IMPLEMENTATION DETAILS
G.1 EXPERIMENTAL DETAIL OF BENCHMARKS
The Table provides comprehensive summary of the reward model configuration used across benchmark implementations, along with the key experimental settings for algorithms including IQL (represented Oracle in the table), MR, PT, DPPO, IPL. and APPO. These settings cover essential implementation details such network architecture, optimizer type, learning rates, batch sizes, and number of training epochs, ensuring transparency and fairness in reproduction and comparison.
Table 4: Implementation details of the reward model and the baselines.
Algorithm
Component
Value
Optimizer
Adam Kingma & Ba [2014]
Learning rate
1c-3
Batch size
100
Hidden layer dim
128
Reward model
Hidden layers
Activation function
ReLU
Final activation
Tanh
Epochs
300
# of ensembles
Reward from ensemble models
Average
Optimizer
Adam Kingma & Ba (2014)
Actor, Value hidden
256
Critic, Actor, Value hidden layers
Critie, Actor, Value activation function
ReLU
IQL. Kostrikov al. [2021)
Critie, Actor, Value learning rale
0.5
Mini-batch
256
Discount factor
0.99
3.0
Neural networks (Q, V)
3-layers, hidden dimension 256
Activation
ReLU for hidden activations
V. optimizer
Adam with learning rate 3e-4
MR Lee et al. [2021)
Hatch size
256
Target network soft update
0.005
B (IQL advantage weight)
(IQI. expectile parameter)
0.7
Discount factor
0.99
Optimizer
AdamW Loshchilov & Hutter [2017]
# of layers
al. [2023]
# attention heads
Embedding dimension Dropout rate
256
0.1
Preference predictor
The same PT Kim et al. [2023]
DPPO An et al. [2023]
Smoothness regularization
1.0
Smoothness sigma m Regularization
20
Optimizer
Adam Kingma & Ba [2014]
Regularization
3c-4
IPL Hejma & Sudigh [2023]
Q. V. arch
3x256d
4.0
Subsample
16
Neural networks (Q, V,x)
3-layers, hidden dimension 256
Activation
LeakyReLU for hidden activations
Q, V. a optimizer
Adam with learning rate 3e-4
APPO Kang & Oh [2025]
3 optimizer
Adam with learning rate
Batch size
256 transitions and trajectory pairs
Target network soft update
0.001
Discount factor
0.99
34

---

Published as a conference paper at ICLR 2026
G.2 BASIC EXPERIMENTAL DESIGN
The preference dataset is constructed by randomly drawing pairs of trajectory fragments, each consisting of 25 time steps. Preference labels are assigned using the true reward: a label of (0,1) is used if the total rewards of the two trajectories differ by more than 12.5; otherwise, both segments receive an equal preference label of (0.5, 0.5). Performance is quantified by task-specific success rates, which reflect whether the agent manages to complete the intended task.
G.3 TRAINING DETAILS
In the following, we talk about the details of the model implementation. We firstly represent the dynamics model as an ensemble of neural networks that output Gaussian distribution over the next state given the current state and action:
P(s' 8,a) = ( ((s,a)
Following previous works Rigter et al. [2022]; Sun [2023], during the initial maximum likelihood model training (Line 3 of Algorithm 2) we train an ensemble of 7 such dynamics models and pick the best 5 models based on the validation error on a held-out test set of 10000 transitions from the offline dataset D. Each model in the ensemble is a 4-layer feedforward neural network with 200 hidden units per layer. We summarize the details of the algorithm in Algorithm
Algorithm 2 Model-based Conservative Planning (MCP)
Input: Offline dataset D; regularization parameters ÅT λ2, As: rollout length I; sub-trajectory length h:
1: Initialize the policy म as the uniform policy.
2: Learn reward: F F= arg max En= log (o
=0 P0=00"
)..). H 1}.
3:
Learn transition kernel: for all {0, (O.....H.
N
Pr = arg max PAE,,
Σ 1=0
log (n 88 mi)).
n=
4: for 1= 11,2..... do
(a) Model update:
(i) Initialize the reward model r and transition kernel (Pr) H-1 with the MLE models F and
(Ph) [Ph)
(ii) Collect a sub-trajectory bu from saub of length and compute r(7sub).
(iii) Roll गi from Saub under {Ph E Pr for steps to obtain yoval
(iv) Solve for r and {Pr) H-1,
+ min E ExTt [r(reval) Saub] r(7sub) + & 1 k.(P)) (PA
(b) Policy update:
Initialize +- TT.
Sample a batch of (8, a) from and draw a' ~ me(. 1).
iiii
Obtain T/ by
arg max ( 1s), E Ba (r((t) [. do (d' -a)2 (P
[ra(T)
(iv) Align T1o0(-) 8) with me(- 8) by minimizing the KI. divergence to obtain m
5: end for
6: Output: न ALG = -..
In (iii,,, we follow TD3-BC BC [Fujimoto & Gu, 2021] to include a regression staressiion-tylleee behavior cloning loss push the policy towards favoring actions contained in the dataset. In (b)(i), we leverage the subtle decremental training for KL-divergence trust region updating to avoid Ioc getting too close to ne. To ensure fair comparison with baseline methods, follow the official implementation
35

---

Published as a conference paper at ICLR 2026
provided by Choi et al. [2024] for training reward model. Specifically, the reward model is implemented as an ensemble three fully connected neural networks, each consisting of three hidden layers with 128 units per layer. ReLU is used the activation function for all hidden layers, and Tanh activation is applied to the final output. The ensemble prediction is computed as the average of the individual network outputs.
Table 5: Model architecture and hyperparameters.
Component
Value
Neural networks (r, P. TT)
3-layers, hidden dimension 256
Activation
ReLU for hidden activations
T optimizer
Adam with learning rate le-4
Batch size
256 transitions and 256 trajectory pairs
A1
0.01
A
Our work
Rollout length
2"
Sub-trajecton length h
5*
no. of model networks
7
no. of elites
5
model learning rate
3e-4
The data marked with asterisks are only approximate values and need to be adjusted according to different datasets.
36

---

Published as a conference paper at ICLR 2026
H VISUALIZATION OF TRAINING DYNAMICS
Figure 3 illustrates the training progress corresponding to the experiments summarized in Table Each algorithm undergoes training for a total of 100,000 gradient update steps, during which performance evaluations are performed at regular intervals of every 5,000 steps. To derive the final success rate reported the tables, we compute the average and standard deviation over the results from three random seeds, offering a more stable and representative performance estimate. This figure generated from the mean, standard derivation of the success rates.
563
89
Butter Sapdann
ICCS
1233
1000
1.0
04
57
Lutton
2.2
10C3
PCL DRu1 PMoceec--50
K0. Semo Wft.e www.teece-1000 1090
4000
Figure 3: Learning curves from the experiments in Table 2 of main text.
I THE USE OF LLM
The authors used a large language model (LLM) to edit and polish the writing (grammar, wording, and clarity). All ideas, analyses, and conclusions are the authors' own.
J ADDITIONAL EXPERIMENTAL RESULTS
Generalization to high-dimensional environments. To evaluate the performance of MCP in more complex environments with significantly larger state-action spaces, construct offline
37

---

Published as a conference paper at ICLR 2026
PbRL datasets based the MuJoCo tasks Ant [Schulman et al., 2015] and Humanoid [Tassa et al., 2012]. These two tasks have sigh-dimensional states and actions: Ant has a state space of dimension 105 (about 3 times larger than MetaWorld and action space of dimension 8 (about 2 times the action dimension of MetaWorld), while Humanoid has a state space dimension 348 (about 9 times larger than MetaWorld and action space of dimension 17 (about 4 times the action dimension of MetaWorld) Table reports the average total return over three seeds for each method. As shown in Table 6, MCP consistently outperforms MR and APPO on both high-ddmeesiooal tasks, demonstrating its potential in complex environments.
Table 6: Performance on high-dimensional MuJoCo tasks.
Algorithm
Oracle
MR
APPO
MCP
Humanoid
2672.48
1586.31
1365.34
1731.47
Ant
3007.22
1940.87
1638.42
2213.85
Runtime analysis. We measure the time required perform 100k training steps using a single NVIDIA RTX 5090 GPU. As reported in Table 7, MCP is slightly slower than MR but still faster than APPO, while achieving better performance than both. Table 8 further decomposes the MCP runtime into the conservative planning step (line 5 of Algorithm 1) and the policy update step (line 6).
Table 7: Running time for 100k training steps.
Method
MR
APPO
MCP
Total time (min)
22.7
36.8
33.5
Table 8: Decomposition of MCP runtime for 100k training steps.
Method
MCP-step5
MCP-step6
MCP-total
Total time (min)
24.2
9.3
33.5
Hyperparameter sensitivity. We conduct hyperparameter sensitivity analysis for the behavior cloning weights A on the drawer-open-1000 and sweep-into- 1000 tasks over three seeds.
Table 9: Sensitivity of MCP to regularization hyperparameters on draweer-ppen-1000 and sweep-inlo- 1000.
le-1
3e-1
3
lel
drawer-open-Iie
92.0 = 5.66
97.20
+ 0.38
96.40
89.87
9.92
sweep-into-1004 1000
22.53 7.39
26.8 4.25
29.2 1 6.76
34.13 3.27
31.6 5.02
38


Reviews:
[{"review_id": "vwkWdTXjML", "reviewer": "Reviewer_wXXH", "summary": "This paper tackles incomplete data coverage and high computational cost in offline preference-based RL by introducing Model-based Conservative Planning (MCP). MCP uses a model-based planning framework that implicitly enforces conservatism, enabling sample-efficient and computationally tractable policy learning with general function approximation. Theoretically, MCP competes with the best policy supported by the dataset under partial coverage, with regret bounds improved via dynamic structures (e.g., kernelized nonlinear regulators, factorized models). Empirically, across 8 Meta-World tasks, MCP outperforms baselines like APPO and IPL in average rank and shows strong robustness in low-data regimes.", "questions": "see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. simultaneously achieve sample efficiency and computational tractability in offline PbRL with unknown dynamics and partial coverage. It encodes conservatism via relative performance instead of explicit confidence sets or extra value modeling, simplifying the framework and overcoming the computational bottlenecks of methods like FREEHAND and Sim-OPRL.\n\n2. Supports general function approximation (linear models, neural networks, etc.), and derives adaptive concentration coefficients and regret bounds for structured dynamics (e.g., kernelized nonlinear regulators and factorized models), mitigating the curse of dimensionality and broadening theoretical applicability.", "weaknesses": "1. Compared with the APPO algorithm, this paper mainly introduces a model with conservative terms, but the experimental results do not show much improvement, while introducing additional model computational overhead.\n\n2. What is the robustness of MCP when there is noise in preference labels? Quantitative analysis of the impact of label noise on performance can be conducted to prove that the design of MCP can have better generalization.\n\n3. Each time a policy is updated, a new MDP model should be learned to challenge it, which is very wasteful in terms of compuational cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7P04r3PteC", "reviewer": "Reviewer_6XMk", "summary": "This work proposes MCP, a novel model-based conservative planning algorithm for offline preference-based RL that is both sample-efficient and computationally tractable. Under partial coverage, MCP does not require additional structural assumptions and provides a PAC guarantee with a tractable implementation in model-based RL. To the best of the reviewer’s knowledge, this is the first work in model-based RL that addresses the limitations of PbRL. In experiments, MCP demonstrates high performance compared to other baselines.", "questions": "1. The authors mention that MCP provides an implicit way of encoding conservatism, mainly through the minimax objective function. However, it is somewhat difficult to intuitively understand how this mechanism works in practice. Could the authors provide a simple or toy example to illustrate how the minimax structure implicitly enforces conservatism?\n\n2. In Table 5, $\\lambda_{1}$, $\\lambda_{2}$, and $\\lambda_{3}$ are important parameters for MCP. Therefore, it would be helpful to report their exact values for each dataset. Are these parameters highly sensitive?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors derive PAC bounds for the three variants of MCP (original, factored, and KNR). The paper is clearly structured, making it easy for readers to follow the overall flow and reasoning.\n\n2. This work is a natural extension to the model-based setting and effectively addresses the limitations of previous approaches.", "weaknesses": "1. In Figure 1-(a), MCP is compared only with MR and Oracle. It would strengthen the validity of the theoretical claims if the experiments also included a comparison with APPO, as was done in the theoretical analysis regarding sample efficiency (Line 310).\n\n2. The training curves are presented only for MCP. It would be beneficial to include those of other baselines as well, so that the results highlight not only the final performance but also the faster convergence of MCP, demonstrating the advantage of the model-based RL approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RkPuy6ARaS", "reviewer": "Reviewer_GUzu", "summary": "This paper proposes Model-based Conservative Planning (MCP), a framework for offline preference-based reinforcement learning (PbRL) that integrates model learning with conservative optimization to ensure reliable performance under partial data coverage. The authors provide generalization guarantees under general function approximation and extend the analysis to structured settings such as kernelized nonlinear regulators. Empirically, the paper evaluates MCP on the Meta-World medium-replay benchmark, showing improved success rates over several preference-based baselines (PT, IPL, DPPO, APPO).", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and logically structured.\n\n- The paper provides a solid theoretical framework for offline preference-based reinforcement learning under partial coverage. It establishes generalization guarantees for model-based conservative planning with general function approximation, which, to the best of my knowledge, has not been analyzed in prior PbRL literature.", "weaknesses": "The experimental evaluation exhibits certain limitations.\n\n- The experiments are conducted solely on the Meta-World medium-replay benchmark, which primarily contains deterministic, low-dimensional robotic tasks. This restricted evaluation makes it difficult to assess how well the proposed conservative planning approach generalizes to more diverse or high-dimensional offline preference-based RL settings. Including broader datasets such as D4RL or human-feedback benchmarks (e.g., Atari or MuJoCo preference datasets) would strengthen the empirical evidence.\n\n- The paper lacks ablations isolating the contribution of each design component, such as the relative performance regularization, model-based planning, and the choice of regularization weights.  Without these analyses, it is unclear how much improvement stems from the conservative objective itself versus other implementation factors. Including sensitivity studies or component removals would better validate the claimed effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hF6GxLNMDl", "reviewer": "Reviewer_Ec41", "summary": "In this paper, the authors study offline reinforcement learning (RL) with trajectory preferences, where the RL agent does not receive explicit rewards at each step but instead receives human-provided preferences over pairs of trajectories. They propose a novel Model-based Conservative Planning (MCP) algorithm for offline PbRL, which leverages a general function class and uses a tractable conservative learning framework to improve the policy upon an arbitrary reference policy. They prove that, MCP can compete with the best policy within data coverage when the reference policy is supported by the data. Lastly, the authors conduct some empirical evaluations of MCP.", "questions": "Please see the weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The problem of conservative planning in RLHF is an important and interesting problem.\n2. The technical part of this paper is solid, the proof looks correct to me.\n3. There are simulation results supporting the theoretical results.", "weaknesses": "1. My main concern is about the algorithm. The hyperparameter $\\lambda_1$ and $\\lambda_2$ depend on the concentrability coefficients, which further depends on the real reward or transition kernel. Would you please explain how to calculate (or estimate) such values efficiently?\n\n2. Furthermore, $\\lambda_2$ also depends on $M_P$, which is a max over all the intermediate steps during the training. How could the algorithm know this value before the algorithm starts?\n\n3. The idea behind line 5 and 6 of alg.1 is interesting. Would you please analyze the computational complexity of solving the optimization problem in line 5?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)