Preference Elicitation for Offline Reinforcement Learning
Authors: Alizée Pace
Bernhard Schölkopf
Gunnar Ratsch
Giorgia Ramponi
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Offline Reinforcement Learning
Preference-based Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We explore efficient methods for acquiring preference feedback for RL in a fully offline setup.

Abstract: Applying reinforcement learning (RL) to real-world problems is often made challenging by the inability to interact with the environment and the difficulty of designing reward functions. Offline RL addresses the first challenge by considering access to an offline dataset of environment interactions labeled by the reward function. In contrast, Preference-based RL does not assume access to the reward function and learns it from preferences, but typically requires an online interaction with the environment. We bridge the gap between these frameworks by exploring efficient methods for acquiring preference feedback in a fully offline setup. We propose Sim-OPRL, an offline preference-based reinforcement learning algorithm, which leverages a learned environment model to elicit preference feedback on simulated rollouts. Drawing on insights from both the offline RL and the preference-based RL literature, our algorithm employs a pessimistic approach for out-of-distribution data, and an optimistic approach for acquiring informative preferences about the optimal policy. We provide theoretical guarantees regarding the sample complexity of our approach, dependent on how well the offline data covers the optimal policy. Finally, we demonstrate the empirical performance of Sim-OPRL in various environments.

Published as a conference paper at ICLR 2025
PREFERENCE ELICITATION FOR OFFLINE REINFORCEMENT LEARNING
Alizée Pace
Bernhard Schölkopf
ETH AI Center, ETH Zürich
MPI for Intelligent Systems & ELLIS Institute Tübingen
MPI for Intelligent Systems, Tübingen
al zee. pacelaiccc.i... .et.eai..ehhz... ethz. ch
Gunnar Rätsch
Giorgia Ramponi University of Zürich
ETH Zürich
ABSTRACT
Applying reinforcement learning (RL) to real-world problems is often made chal- lenging by the inability interact with the environment and the difficulty of de- signing reward functions. Offline RL addresses the first challenge by consider- ing access to an offline dataset of environment interactions labeled by the reward function. In contrast, Preference-based RL does not assume access to the reward function and learns it from preferences, but typically requires an online interaction with the environment. We bridge the gap between these frameworks by explor- ing efficient methods for acquiring preference feedback in a fully offline setup. We propose Sim-OPRI, offline preference-based reinforcement learning al- gorithm, which leverages a learned environment model to elicit preference feed- back on simulated rollouts. Drawing on insights from both the offline RL and the preference-bar based RL literature, our algorithm employs pessimistic approach for out-of cuttof-ddssributtion data, and an optimistic approach for acquiring informative preferences about the optimal policy. We provide theoretical guarantees regarding the sample complexity of our approach, dependent on how well the offline data covers the optimal policy. Finally, we demonstrate the empirical performance of Sim-OPRL. various environments.
INTRODUCTION
While reinforcement learning (RL) (uttonn and Barto 2018) achieves excellent performance in var- ious decision-making tasks (Kendall et al. [019 Mirhoseini et al. 2020 Degrave et al.) (022), its practical deployment remains limited by the requirement of direct interaction with the environment. This can be impractical or unsafe in real-world scenarios. For example, patient management and treatment in intensive a reinforcement learning problem (Raghu et al.) [017]] Komorowski et al.) 2018). However, the timing, dosage, and combination of treatments required are critical to patient safety, and incorrect decisions can lead to severe complications or death, making the use of traditional RL algorithms unfeasible (Gottesman et al,, 2019 Tang and Wiens 2021). Offline RL emerges as promising solution, allowing policy learning from entirely observational data (Levine et al. 2020).
units involve complex decision-making that has often been framed as
Still, challenge with Offline RL is its requirement for an explicit reward function. Quantifying the numerical value of taking certain action in a given environment state is challenging in many applications (Yu et al. (02T). Preference-hased RI. offers a compelling alternative, relying on com- parisons between different actions or trajectories (Wirth et al.) (2017) and being often easier for hu- mans to provide (Christiano et al. 2017). In medical settings, for instance, clinicians may be queried for feedback which trajectories lead to favorable outcomes. Unfortunately, most algorithms for preference acquisition require environment interaction (Saha et al. 2023: Chen et al. 2022 Lindner et al. 2021) and are therefore not applicable to the offline setting.
Lack of environment interaction and reward learning are thus two critical challenges for real-world RI. deployment that are rarely tackled jointly. In this work, address the problem of prefer-

---

Published as a conference paper at ICLR 2025
ence elicitation for offline reinforcement learning by asking: What trajectories should sample to minimize the number of human queries required to learn the best offtine policy? This presents a challenging problem it combines learning from offline data and active feedback acquisition, two frameworks that require opposing inductive biases for conservatism and exploration, respectively.
To the best of our knowledge, the only strategy proposed in prior work is acquire feedback directly over samples within an offline dataset of trajectories (Shin et al. 2022 Offline Preference- based Re- ward Learning (OPRL)). We propose an alternative solution that queries feedback on simulated rollouts by leveraging a learned environment model. Our offline preference-
reinforcement learning algorithm, Sim-OPRL, strikes a balance between conservatism and exploration by combin- ing pessimism when handling states out-of-distribution from the observational data (Jin et al.) 202T] Zhan et al., 2023a), and optimism in acquiring informative preferences about the optimal policy (Saha et al. 2023] Chen et al.] (022). We study the efficiency of our approach through both theo- retical and empirical analysis, demonstrating the superior performance of Sim-OPRL. across various environments.
Our contributions are the following: (1) Section 3 we first formalize the new problem setting of preference elicitation for offlin reinforcement learning, which allows for complementing of- fline data with preference feedback. This framework is crucial for real-world RL applications where direct environment interaction is unsafe impractical and reward functions are challeng- ing to design manually, yet experts can be queried for their knowledge. (2) In Section 5
we provide theoretical guarantees eliciting preference feedback over samples from offline dataset, comple- menting work Shin et al. (2022). (3) Next, in Section O we propose our own efficient preference elicitation algorithm based on simulated rollouts in learned environment model, and establish its improved theoretical guarantees. (4) Finally, we develop a practical implementation of our al- gorithm and demonstrate its empirical efficiency and scalability across various recision-making environments.
2 RELATED WORK
Our problem setting shares similarities with Offline RL and Preference-based RL, which we sum- marize below. We position ourselves relative to our closest related works in Table and extend our discussion in Appendix B.
Offiine RL. Offline Reinforcement Learning has gained significant traction in recent years, as the practicality of training RL agents without environment interaction makes it relevant to real-world applications (Levine et al. 2020). However, learning from observational data only is a source of bias in the model, the data may not cover the entire state-action space. Offline RL algorithms therefore output pessimistic policies, which has been shown to minimize suboptimality (Jin et al. 2021 Model-based approaches show particular promise for their sample efficiency (Yu et al. 2020 Kidambi et al. 22200] Rigter et al.) 2022;) Zhai et al.) 2024 Uehara and Sun (021). In this work, we study the setting where reward signals are unavailable and must be estimated by actively querying preference feedback.
Preference-based RL. Rather than accessing numerical reward values for each state-action pair as in traditional online RL, preference-based based RL learns the reward model through collecting pairwise preferences over trajectories (Wirth al. 2017]. Different preference elicitation strategies have been proposed for this framework, generally based on knowing the transition model exactly or on having access to the environment for rollouts (Christiano et al., 2017: Saha et al. [2023] Chen al.)
2022 Lindner et al.) 2021 Zhan et al.) 2023b] (aaiihh et al. 2018 Brown et al. 2020). Prior theoretical and empirical work (Lindner et al., 2021; Chen al.) 2022) show that, in this setting, the most efficient preference elicitation strategy is to actively reduce the set of candidate optimal
Table 1: Comparison of related work preference elicitation.
Framework
Offline Policy-Based Sampling
Robustness Guarantees
Practicul Implementation
РЫОР Chen al.
2022
MoP-RL RL
Liu ct
ZUE3
REGIME Z Thin el al.
2023b
FREEHAND Zhan el
20233
X
OPRI. (Shin et 2022
X
/
Sim-OPRL (Ours)
/
2

---

Published as a conference paper at ICLR 2025
policies, rather maximize information gain the reward function our theoretical and empirical results reach the same conclusion for the offline setting.
Offline Preference-based RL. The development of preference-baset RL algorithms based on of- fline data only is critical to settings where environment interaction is not feasible for safety and efficiency reasons. Still, this framework remains largely unexplored in the literature. While (Zhu et al. 2023 Zhan et al.] (2023)) demonstrate the value of pessimism in offline preference- inforcement learning. they do not consider how query feedback actively. On the other hand, [Shin] et al. (202)) propose an empirical comparison of different preference sampling trajectories from an offline trajectories buffer. In Section [ we provide a theoretical analysis of their approach, then propose an alternative sampling strategy based on simulated trajectory rollouts in Section O which benefits from both theoretical motivation and superior empirical performance.
re-
3 PROBLEM FORMULATION
3.1 PRELIMINARIES
Markov Decision Process. We consider the episodic Markov Decision Process (MDP), defined by the tuple M H, R), where S is the state space, A is the action space, H is the episode length, T : x A + As is the transition function, R S x A - IR is the reward function. We assume an initial state 8o, but our analysis generalizes a fixed initial state distribution. At time the environment is at state E S and an agent selects an action E A. The agent then receives reward R(s,, at) and the environment transitions state St41 ~ T( |st. ae). We describe an agent's behavior through a policy function : S -, A such that [[ass) is the probability of taking action in state s. Let episode with the environment, With an abuse of notation, we also write R(T) = E, R(s,, az). Let d7 denote the distribution of trajectories (or state-action pairs, overloading notation) induced by rolling out policy π in transition model T. We denote the expected return of policy as V7,R dx [R] and π- argmax VT. denotes the optimal policy in M.
(so, ag .8 ag) denote the trajectory of state-action pairs of an interaction
Preference-based Reinforcement Learning. Rather than observing rewards for each state and action, we receive preference feedback over trajectories. For a pair of trajectories (T,, 72), we obtain binary feedback o E {0, }} about whether is preferred to 72. We define the preference function PR and assume that preference labels follow the Bradley-Terry model (Bradley and Terry 1952):
exp(R(TT))
PR(TI > T2) = P(o = 1/T1 exp(R(TT)) + exp((((2)) o(((T)) -(rz)) -K(()) (1)
where > denotes a preference relationship and a monotonous increasing link function. Within this framework, preference elicitation refers to the process of sampling preferences to obtain infor- mation about both the preference function and the system dynamics (Wirth et al. 2017).
3.2 OFFLINE PREFERENCE ELICITATION
We assume access to an observational dataset of trajectories Domline = [T: di where F. an unknown behavioural policy in M. As in Offline RL, we do not have access to the environment to observe transition dynamics or rewards under alternative action choices. We assume not to have access to the reward function, but we can query preference feedback from a human to generate dataset of preferences Dpret = ((TI, T2, o)}.
Optimality Criterion. Based only on our offline dataset Doftine our goal is to recover policy n that minimizes suboptimality in the true environment with as few human preference queries as possible. Let Toffin denote the optimal offline policy estimated based on the offline data, with access to the true reward function R, and let ej denote its suboptimality. Since preference elicitation only allows to estimate the reward function, do not aim to achieve a suboptimality less than er although this is not formally a lower bound for our problem, as shown in Appendix A.3 Our objective is then formalized follows.
Definition 3.1 (Optimality Criterion of Offline Preference Elicitation). Let x² be the optimal policy in M and R be the estimated optimal policy based on an offline dataset Domine and N, > preference queries. Let er be the inherent suboptimality assuming access to the true reward function.
3

---

Published as a conference paper at ICLR 2025
We say that sampling strategy is N,,),,orre if for every e 2 er, with probability at least (1 5), it holds that VT. T,R S 6.
Our work is the first to formalize this important problem, which faces the challenge of balancing exploration when actively acquiring feedback and bias mitigation in learning from offline data.
Function classes. We estimate the reward function and transition kernel with general function ap- proximation; let FR and Fr denote the classes of functions considered respectively. We also assume a policy class II. Our theoretical analysis also requires the following assumptions and definitions, which are standard in preference- based RL (Chen et al.) 2022 Zhan et al. 2023a).
Assumption 3.1 (Realizabilitty) The true reward function belongs to the reward class: E Fn- The true transition function belongs to the transition class: T E Fr. The optimal policy belongs to the policy class: n² E II.
Assumption 3.2 (Boundedness). The reward function is bounded: 0 S R(()) S Rmax for all R FR a and all trajectories T.
Definition 3.2 (e-bracketing number). Let F be a class of real functions f : X We say (l, u) is an e-bracket if (xx) I u(x) and [u(z)- 1(x)[1 S <for all x E X. The e-bracketing number of F, denoted (c), is the minimal number of c-brackets (1 u²t y== needed so that for any f E F, there is bracket i E [N] containing it, meaning (xx) S f()) S w (x( all x E X.
Let Nr and Ny, (c) denote the e-bracketing numbers of Fn and Fr respectively. This measures the complexity of the function classes (Geer) 2000). For instance, with linear rewards FR := {R:
T - 0t (()),. the e-bracket number is bounded by log Nog S O(d BC ). where |0]| S B
and [|((()) ll2 < GVT E T, and d is the dimension of the feature space (Zhan et al. 2023a).
Definition 3.3 (Transition concentrability coefficient, Zhan et al.] (2023a)). The concentrability co-
efficient w.E.t transition classes Fr and the optimal policy x is defined as:
((na)=d [[T()s,, - (((1ss)))aa) CT(FT,T))= = sup FEFT Lv ((aaa) (T((,,, (-18s
The concentrability coefficient measures the coverage of the optimal policy in the offline dataset. Note that Or is upper-bounded by the density-ratio coefficient: CT(Fr s (A di (s, a)/d?? ( (s,a), where TH is the behavioural policy underlying Dominee
4 OFFLINE PREFERENCE-BASED RL WITH PREFERENCE ELICITATION
In this section, we propose a general framework for offline preference-based reinforcement learning. The next two sections propose two different preference elicitation strategies. As learning must be carried out two distinct stages, with environment dynamics based on Domtin and reward learning on D prees adopt model-based approach which summarize in Algorithm
Model I Learning, We first leverage the offline data to learn a model of the environment dynamics, fitting a transition model î and an uncertainty function ur through maximum likelihood:
t argmax Fa(a,a.") log ( a))
a)) = max (i(-1s,a) - TT(.11, Raaxs
T.TTET
where T= ( Fr ((s..,.")~DD.mme [Tog (1''ss, a)/ m))] Br), defining a confidence
set over the maximum likelihood estimator (MLE), and B3 is a margin hyperparameter.
Iterative Preference Elicitation and Reward Learning. As with the transition model, our algo- rithm estimates the reward function R and its uncertainty function through maximum likelihood over iteratively collected preference data Dpret
N argmax REF E (T(T).2.0))> lolog PT(TT > 12) + (1- 0) log Pa( Ti),,
UR(T) = max [R((T) - -R2
ARE
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Offline Preference-basee Reinforcement Learning with Preference Elicitation
Input: Observational trajectories datasct Dottine- Significance 8 E (0, 1), preference budget Ng-
Output: #'
1: Estimate T and ur via maximum likelihood over the observational data Damine-
2: Dure <- 0.
3: for k 1= do
4: Generate trajectory pairs (r1, T2).
Preference Elicitation: Sections
and
5: Collect preference label o for (TI, 72).
6: Dreet &- Preer U [(TI, 72, o)).
7: Estimate R and ug via maximum likelihood over the preference data Dpret-
8:
end for
9: #" <- argmax (M(T) - uR(T) =r(r)]
where R. = (R E FR o)) [og (Pa(T) > p2)/Pp(T > r2))] < Bn] defines the confi- dence and BR hyperparameter. We also define preference uncertainty between trajecto- ries T1, T2:
Un (TI, T2)
max R2,
IP (11 T2) Pir (n 12)11-
(2)
The choice of trajectory sampling strategy for preference elicitation in line (Algorithm 1 critical to efficiently obtaining an e-optimal policy. We present two possible strategies in Sections and
Pessimistic Policy Optimization. Finally, our algorithm outputs policy 5* that is optimal while ensuring robustness to modeling error. This means optimizing for the worst-case value function over the remaining transition and reward uncertainties (Levine et al. 2020
argmax TEll min TETRRER
T.A
Based this objective, define the pessimistic transition and reward models follows: Tint, Piir argmin RER max V7 Our analysis provides worst-case robustness guar- antee when considering well-calibrated confidence intervals, as detailed in Sections 5.1 and 6.1 In other words, following prior work (Chen et al. 2022 Zhan al., (023a), theoretical analysis assumes that modeling elements can be identified with no optimization eтror. We then complement this algorithmic framework with flexible practical implementation in Section 6.3
5 PREFERENCE ELICITATION FROM OFFLINE TRAJECTORIES
A first strategy for preference elicitation without environment interaction is to sample trajectories directly from the offline dataset. Shin et al. (2022) propose this approach Offline Preference-basee Reward Learning (OPRL), and design a uniform and ancertainty-sapplle variant:
OPRL Uniform Sampling: OPRL Uncertainty Sampling:
TI, 72 Dotfine
T1, T2 = arggax D. up (T1, 72)
We provide a theoretical analysis of the performance of OPRL.
5.1 THEORETICAL GUARANTEES.
We obtain the following result, proven in Appendix A.4 The suboptimality of the estimated policy a* is bounded by the policy evaluation error for the optimal policy лτ. This error decomposes into a term depending on transition model estimation, and one on reward model estimation.
Theorem 5.1. For any E (0, 1]. let 3r = cyt log HHNr,, (1/No)/5)/N, and 3R. = where N. H[Domunel is the number of observed transitions in the M.FE
CR log(N (L//2)/////B/ observational dataset and M.F. cc are universal constants. The policy ** estimated by Algo- rithm 7 with preference elicitation based on offline trajectories, achieves the following suboptimal-
5

---

Published as a conference paper at ICLR 2025
ity with probability 8:
VF Va S Cr
CT No 8
log
NTr
No. + 2akCnlFF.e
eR Np
log
Npp
term
rewand term
where for uniform sampling or a S for uncertainty sampling, Cr is a concentrability measure for the reward function, K Ramue (r)) measures the degree of non-linearity of the link function, and cr. are universal constants.
In the special case where both the transition and reward functions learned on fixed initial pref- erence dataset (no preference elicitation; [Pottinel 2N,,). we recover Theorem from Zhan et al. (2023a). Importantly, the coefficient a allows to motivate the superior efficiency of uncertainty sampling over uniform sampling, observed empirically in SShin et al. (222)) and in our own ex- periments (Section 7. Uncertainty sampling learns accurate reward models with fewer preference queries when
I, but can perform like uniform sampling in harder problems (a = 1).
6 PREFERENCE ELICITATION FROM SIMULATED TRAJECTORIES
We now propose our alternative strategy for generating trajectories for offline preference elicitation: Simulated Offline Preference-based Reward Learning (Sim-OPRL). This method simulates tra- jectories (Ti, T2) by leveraging the learned environment model. This overcomes a limitation of OPRL, which is only designed to reduce uncertainty about the reward functions in R, by instead re- ducing uncertainty about which policies are plausibly optimal. Our approach is inspired by efficient online preference elicitation algorithms (Saha al. 2023 Chen et al., 2022), which we modify for practical implementation. We account for the offline nature of our problem by avoiding regions that out of the distribution of the data: the sampling strategy is optimistic with respect uncertainty in rewards, but pessimistic with respect to uncertainty in transitions.
We summarize our approach generating simulated trajectories for preference elicitation in Al- gorithm 2 First, we construct a set of candidate optimal policies Nommme containing policy Jaminee (optimal policy under the pessimistic model and the true reward function) with high probability as demonstrated in Appendix A.5.2 Next, within this set of candidate policies, we identify the two most exploratory policies IT T2, chosen to maximize preference uncertainty urg- Finally, we roll out these policies within our learned transition model to generate a trajectory pair (2),7) for preference feedback.
Algorithm 2 Preference Elicitation through Simulated Trajectory Sampling.
Input: Pessimistic transition model Tir. Reward confidence set R and preference uncertainty function uyn- Output: (7,72)
1: Estimate optimal offline policy set: Tamme = (m
ER R argmax,
sen En [A(r)]} Pur [uPA 72]]
2: Identify exploratory policies: TT, 22 argmax 7 E Bes
3: Rollouts in model: 71
dT Tial
Tia
We first provide a theoretical analysis of the performance of Sim-OPRL before proposing a practical implementation of our entire preference elicitation and policy optimization algorithm.
6.1 THFORETICAL GUARANTEES
We decompose suboptimality in a similar way to Section 5.11 but obtain a reward suboptimality term that depends on the learned dynamics model instead of the true one, and on π Tattine instead of π*;
V
V
(V
VF" R.
(4)
fransition term CT
reward term
Analysis of the suboptimality due to transition error identical to above, but the reward term is thus significantly different. By design, our sampling strategy ensures good coverage of preferences over Tottine within the learned environment model, which eliminates the concentrability term for the reward Cn- We refer the reader to Appendix A.5 for the proof of Theorem 6.1

---

Published as a conference paper at ICLR 2025
Theorem 6.1. For any 8 E (0, 1]. let 3r GF (1/N.)/8)/N and 3n MILE lo (1/Np)/0)/
where No = Dominel is the number of observed transitions in the observational dataset and oT c universal constants. The policy #' estimated by Al- gorithm [ with a preference sampling strategy based on rollouts in the learned transition model, achieves the following suboptimality with probability 6:
V Va I H Mmmix C
CT No
log
H NN
)
+ 2x
CR
log
No.
Np
Np
transition term
reward term
where supre[-Ree Ram Rand (r) measures the degree of non-linearity of the link function, and cr. CR are universal constants.
6.2 DISCUSSION
Our theoretical results demonstrate that the learned policy can achieve performance comparable to the optimal policy, and thus satisfy optimality criterion in Definition 3.1 provided it is covered by the offline data (Fr x²) Cn(Fn, T²) < co). Following our analysis, a suboptimal dataset requires more preferences to achieve certain policy performance, as the concentrability terms Cr or CR are large. Empirical results in Section 7 confirm that sample efficiency is worse when the behavioral policy more suboptimal.
Offline Trajectories vs. Simulated Rollouts. While both OPRL and Sim-OPRL depend on the offline dataset for estimating environment dynamics, they induce different suboptimality in model- ing preference feedback. Simulated rollouts are designed to achieve good coverage of the optimal offline policy Tilliee which avoids wasting preference budget on trajectories with low rewards or high transition uncertainty. In contrast, as shown in IZhan et al.) (2023a). due to the dependence of preferences on full trajectories, the reward concentrability term CR Theorem [5. can be very large. While sampling from the offline buffer is not sensitive to the quality of the transition model, good coverage of the optimal policy is needed from both transition and preference data to achieve low suboptimality.
Transition vs. Preference Model Quality. Our theoretical analysis also suggests an interesting trade-off in the sample efficiency of our approach, depending on the accuracy of the transition model. The width of the confidence interval reduces as significance parameter or dataset size increase, or function class complexity Nrr decreases. For a target suboptimality gap e, provided the optimal offine policy T Toffine has gap en < then the number of preferences required of the )7) A more accurate transition model should therefore require fewer
order of (log(1/0)) preference samples to achieve given suboptimality, which we again confirm empirically.
6.3 PRACTICAL IMPLEMENTATION
We complete the general algorithmic framework discussed above with a possible implementa- tion strategy, allowing for empirical validation. fact, with minor changes to the following frame- work. our paper also proposes feasible implementation of related theoretical algorithms (Chen)
et al. 2022
Zhan et al. 2023a). We refer the reader to Appendix for further detail.
Model Learning and Policy Optimization. Following prior work in offline reinforcement learn- ing (Yu al., (2020), we train ensembles of Ny and Na neural network models for the transition and reward functions different bootstraps of the data (Lakshminarayanan et al., 2017), denoted (f..... TN,) and (RI,... .......,,,, We estimate MLE and uncertainty functions as follows:
N₂
((11,a)
1
Σ
f((1s,,);; ur(s,0)= = max 1..[L.N [/((11,,) -15(11,,)) Rmux
NT
1=1
1
R((s.a) NR
ax, un(s,a)= max
[ -R (s,
[JE[LNN]]

---

Published as a conference paper at ICLR 2025
Each R, in the ensemble has associated preference function defined by the Bradley-Terry model, with a as the sigmoid function. We obtain preference uncertainty through variation over the en- semble as in Equation 2 Recall that transition and reward models trained on Dattine and D peef respectively; for computational efficiency, we sample preferences in batches of to reduce the number of reward model updates needed.
We approximate the pessimistic objective in Equation (3) by penalizing the reward function with the uncertainty, as in Lagrangian formulations of model-based offline RL (Yu al. 2020 Rigter al. 2022). We solve for the following objective with traditional reinforcement learning algorithm:
argmax (BB(s [A(s,,) a) - ARuR(s a) Arwr(s (s,a)],
(5)
where hyperparameters Xr, Aæ control the degree of conservatism. Note that in our theoretical anal- ysis, this was achieved through parameters 3r, n which affect the width of the confidence intervals wn and ur, but their exact value cannot be estimated. We show in Appendix A.2 that Equation (5) indeed lower bounds the true value function, under well-calibrated uncertainty estimates.
Near-Optimal Policy Set and Exploratory Policies. Sim-OPRL requires constructing Holline set of near-optimal policies within a pessimistic model of the environment. Following [indner al.) 2021), we obtain a policy model for each element R2 of the reward ensemble. Policy models are optimized to maximize returns under the transition model T and the reward function it; - ATUT, ensuring pessimism w.r.t transitions. Next, the most exploratory policies are identified by generating rollouts of each candidate policy within the learned model T. The trajectories (TI, 72) induced by different policies and maximizing the preference uncertainty function " T2) are used for preference feedback. We refer the reader to Appendix Cfor further detail.
7 EXPERIMENTAL RESULTS
In this section, we demonstrate the effectiveness of our preference elicitation strategy, Sim-OPRL, across range of offline reinforcement learning environments and datasets. We demonstrate its superior performance over OPRL, as expected from our theoretical analysis.
Since our closest related works do not propose any experimental validation (Chen al. 2022 Zhan et al. 2023a), we propose a practical implementation of sefference-baseer Optimistic Planning (PbOP) in Appendix [c this elicitation method queries feedback over trajectory rollouts in the true environment (Chen et al., 2022). We also compare against OPRI. (Shin et al., 2022) with uniform and uncertainty-sampling Finally, we report the performance of π ooftine and x as upper bounds for the performance of our algorithm: the former is trained in the learned transition model with access to the true reward, and the latter has full knowledge of both transition and reward functionn
We compare different preference elicitation strategies on range of environments detailed in Ap- pendix D Among others, we explore environments from the D4RL benchmark (Fu et al. 2020) identified as particularly challenging offline preference-basee reinforcement learning tasks (Shin) ct al. 2022), as well as a medical simulation designed to model the evolution of patients with sep- sis (Oberst and Sontag 2019). As detailed in Appendix [ these environments consist of high- dimensional state spaces with continuous or discrete action spaces, follow complex transition dy- namics, and have sparse or non-linear rewards and termination conditions. This makes them repre- sentative of the challenge of learning a reward function and learning offline in a real-world appli- cation. In particular, the sepsis simulation environment is commonly used in vedically-motivater offline RL work (Tang and Wiens 2021 Pace et al., 2023), and highlights another advantage of Sim- OPRL over OPRL: it does not require feedback on real trajectories from the observational dataset. In sensitive setting such as healthcare where data access is carefully controlled, it may be attractive to query experts about synthetic trajectories rather than real samples.
Performance against State-of-the-Arr. Performance and sample complexity results with different preference elicitation methods are given in Figure [ and Table 2 Within the offline approaches, Sim-OPRL consistently achieves better environment returns than OPRL with much fewer preference queries. In line with our theoretical analysis, our empirical results therefore demonstrate that policy- based sampling in Sim-OPRL is more efficient than maximizing information gain the reward function (uncertainty- based OPRL), which echoes similar conclusions reached in prior work on online preference elicitation (Lindner al. 2021 Chen et al. 2022).
8

---

Published as a conference paper at ICLR 2025
Table 2: Sample complexity .Vp under different preference elicitation strategies. reach suboptimality gap of 20 over normalized returns. Mean 95% confidence interval over 6 experiments. The best- performing offline method is highlighted in bold, marks when the target suboptimality could not achieved. Note that PhOP has an advantage by having access to direct interaction with the environment.
Environment
OPRL Uniform
OPRL Uncertainty
Sim-OPRL (Ours)
PbOP (Online)
Star MDP
32 +
30 + 4
4 : 2
4 + 2
Gridworld
105 + 11
66 +
49
32 1 4 1 36 1 3 75 +
MiniGrid-Fowe FourRoooms
+
1
41 1 5
HalfCheetah-Ramne
108 1
1
50 1 10
Sepsis Simulation
X
642 + 72
225 + 46
Envtrorment: returre
Envtronment:
40
40
OFRL Lncentsieyy
OPR Unitmm
5km (ITIL (Dung)
POOP (CXERN)
Number Preferences
Number Preterences
Number Preterences
(a) StarMDP.
(b) Gridworld.
(c)
D0
80
Number of Preferences
Number of Preferences
(d) LaaHaaaheetahRaaatiee
(e) Sepsis Simulation.
Figure I: Environment returns under different preference elicitation strategies. Mcan and 95% confidence interval over experiments. Environment returns are normalized between and 100. Only OPRL and Sim- OPRL are fully offline.
As an upper bound for the performance of our algorithm, we include baselines that have access to the environment: we report the performance of the optimal policy n², as well as that of an algorithm querying feedback over optimistic rollouts the real environment (Chen et al. 2022] PbOP). In Figure [ the PbOP method naturally reaches superior policy with fewer samples as allows environment interaction
can thus improve its estimate the transition model parallel to learning the preference function. As supported by our theoretical analysis, this result stresses the importance of having a high-quus nigh-quality transition model to make our method effective. We explore this in more detail in following ablations.
Algorithm Ablations. We conduct ablations for our al- gorithm simple tabular MDP, with results in Fig- ure 2 This example (transition and reward details de- ferred Appendix (D) illustrates the importance of pes- simism with respect to the transition model. Even with access to true rewards, Tamine is pessimistic avoid the nut-of-distribution state, as it is unclear how to reach it.
100
Environment returns
60
Sim-OPRI, (Ours)
Sim Hin-,PRL, pessimism in output policy
Thus, in Figure we see a drop in performance if pes- simism is not applied to the output policy (purple lines).
20
Sim OPRL, pessimism rollouts
Sim-OPRL, na aptimiom in rollouts
This confirms the theoretical insights from Zhu et al. 2023); Zhan al. (2023a), who demonstrate the impor- tance of pessimism in offline preference- based RL prob- lems. Pessimism is also crucial in simulated rollouts, to avoid wasting preference budget on regions of low confidence - as value estimates are inaccurate in any case. This is reflected in lower performance without pessimism w.r.t the transition model
Number of Preferences
Figure 2: Algorithm ablations (StarMDP).

---

Published as a conference paper at ICLR 2025
50
50
OPRI, Uncertaiity (Ours)
Number Proferences
Number Preferences
30
30
10
10
OPRL Uncertainty DPRL (Ours)
50
100
10
100
Number of Offline Trajectories
[d (a.a)/ (a,a)
(a) As function of offline dataset size.
(b) As function of dataset optimality.
Figure 3: Preference sample complexity N; as function the propertics of the observational data, reach a suboptimality gap of = 20 over normalized environment returns (Star MDP). Mean and 95% confidence intervals over 6 experiments. x marks when the target suboptimality could not be achieved.
in Figure 2 (brown line), and which could be seen as the naive adaptation of online preference elicitation methods to setting (Chen et al. 2022 Lindner al., 2022). We also note the im- portance of optimism with respect to the reward uncertainty, both in OPRL in Figure and in our own model-based rollouts in Figure
Transition vs. Preference Model Quality. Next, we empirically study the trade-off between transi- tion and preference model performance in our problem setting. Still in the Star MDP, in the low-data regime, the error er incurred in estimating the value function due to the misspecification of the tran- sition model is large. As dictated by our theoretical analysis and as visualized in Figure Bal this significantly increases the number of preference samples Np required to achieve good final perfor- mance. At the other end of the spectrum, if the offline dataset is large and allows modeling the tran- sition model accurately, then ér is small and the number of preference samples N, needed shrinks. We observe a similar trend for both Sim-OPRL and our OPRL mucertainty-sampliin baseline.
We also measure how the coverage of the optimality of the dataset affects performance in our set- ting. In Figure 3b we vary the behavioral policy Ty underlying the offline data, ranging from optimal (density ratio coefficient concentrability terms Cr and CR are challenging to measure as they require considering entire function classes, but we report the accuracy of the maximum likelihood estimate for both models in Appendix R We observe that preference elicitation methods perform best when the data is close to optimal (with the exception of fully optimal, non-diverse dataset making reward learning from preferences challenging). More preference samples are required if the observational dataset has poor coverage of the optimal policy (large Cr x²) )), as the transition and reward models become less accurate for the trajectory distribution of interest. We also validate this conclusion on HalfCheetah datasets of varying optimality in Appendix
1) to highly suboptimal (large density ratio coefficient). The
8 CONCLUSION
Our work shows the potential of integrating human feedback within the framework of offline RL. We address the challenges of preference elicitation in fully offline setup by exploring two key methods: sampling from the offline dataset (Shin al., 2022 OPRL) and generating model rollouts (Sim-OPRL). By employing a pessimistic approach to handle out-of-distribution data and an opti- mistic strategy to acquire informative preferences, Sim-OPRL balances the need for robustness and informativeness in learning an optimal policy.
We provide theoretical guarantees on the sample complexity of both approaches, demonstrating that performance depends on how well the offline data covers the optimal policy. Empirical evalua- tions on various environments confirm the practical effectiveness of our algorithm, as Sim-OPRI. consistently outperforms OPRL baselines in all settings.
Overall, our approach not only advances the state-of-the-arr in offline preference-based RL but also takes a significant step toward improving the practical utility of offline RL. This opens up new avenues for real-world applications of RL in healthcare, robotics, and manufacturing, where interac- tion with the environment is challenging but domain experts can be queried for feedback. Looking forward, a natural extension will be to explore alternative sources of information from experts, still without direct environment interaction.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 519-528. URL httpsttpprocceeeeiiiiiee http://......eeeea//a////laeeee
PMLR, 29-31 Oct 2018.
mlr wwwpppeessssse. v87
html
Erdem Brysk, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint arXiv:1910.04530 2019.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-444 1952.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe imitation learning via fast Bayesian reward inference from preferences. In Hal Daumé III and Aarti Singh, ed- itors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1165-1177. PMLR, 13-18 Jul 2020. URL https: btttps:/prroceeinngg
html
Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in sentun-ii-thh--ooye thop Provably efficient preference- reinforcement learning with general function approximation. In International Conference on Machine Learning, pages 3773-3793. PMLR, 2022.
Yifang Chen, Shuohang Wang, Ziyi Yang, Hiteshi Sharma, Nikos Karampatziakis, Donghan Yu, Kevin Jamieson, Simon Shaolei Du, and Yelong Shen. Cost-effective proxy reward model con- struction with on-policy and active learning. arXiv preprint arXiv:2407.021199 2024.
Maxime Chevalier-Boiswe
Bolun Dai, Mark Towers, Rodrigo Perez-Vicente Lucas Willems, Salem Lahlou, Suman Pal. Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Mod- ular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature 602(7897):414- 419, 2022.
Steven Diamond and Stephen Boyd. Cvxpy: A python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5, 2016.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Sara A Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Velez, and Leo Anthony Celi. Guidelines for reinforcement learning healthcare. Nature medicine, 25(1):16-15 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pages 1861-1870. PMLR, 2018.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without reward function. Advances in Neural Information Processing Systems, 36, 2024.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference Machine Learning, pages 5084-5096. PMLR, 2021.

---

Published as a conference paper at ICLR 2025
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 international conference on robotics and automation (ICRA), pages 8248-8254. IEEE, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Pref- erence transformer: Modeling human preferences using transformers for rl arXiv preprint arXiv:2303.00997. 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The arti- ficial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(1 ():11):1711116111 17:6 2018.
Balaji Lakshminarayanann Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive rein- forcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
David Lindner, Matteo Turchetta, Sebastian Tschiatschek, Kamil Ciosek, and Andreas Krause. In- formation directed reward learning for reinforcement learning. Advances in Neural Information Processing Systems, 34:3850-3862. 2021.
David Lindner, Andreas Krause, and Giorgia Ramponi. Active exploration for inverse reinforcement learning. Advances Neural Information Processing Systems, 35:5843-5853, 2022.
Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. When is partially observable reinforce- ment learning not scary? In Conference Learning Theory, pages 5175-5220. PMLR, 2022a.
Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-basee bassed reinforcement learning. Advances in Neural Information Processing Systems, 35:22270-22288 2022b.
Yi Liu, Gaurav Datta, Ellen Novoseller, and Daniel Brown. Efficient preference- meference-based rein- forcement learning using learned dynamics models. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 2921-2928. IEEE, 2023.
Luckeciano C Melo, Panagiotis Tigas, Alessandro Abate, and Yarin Gal. Deep bayesian active learning for preference modeling large language models. arXiv preprint arXiv:2406. 10023.. 2024.
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Sungmin Bae, et al. Chip placement with deep reinforcement learning. arXiv preprint 2xXv:20044110766 2020.
Michael Oberst and David Sontag. Counterfactual off off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning, pages 4881 1-8800. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 27730-27744. Curran Associates, Inc., 2022. URL httpps://ppoceeedngg..oot neurips cc paper foreer__flllee fiwes/ hitps/eaeeeeeeaee2222222
Alizée Pace, Hugo Yèche, Bernhard Schölkopf, Gunnar Ratsch, and Guy Tennenholtz. Delphic offline reinforcement learning under nonidentifiable hidden confounding. In The Twelfth Interna- fional Conference on Learning Representations, 2023.
Jongjin Park. Younggyo Sco, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lec. Surf: Semi-supervised reward learning with data augmentation for feedback-efficien preference-based reinforcement learning. arXiv preprint arXiv:2203. 10050, 2022.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http:: http://mllr..rg/aappp.ae wttppppppppp//...... 20 1364. html
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.000 2017.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems, 35:16082-16097, 2022.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learn- ing of reward functions. 2018
Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Dueling rl: Reinforcement learning with tra- jectory preferences. In International Conference Artificial Intelligence and Statistics, pages 6263-6289. PMLR, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06777 2017.
Daniel Shin, Anca Dragan, and Daniel S Brown. Benchmarks and algorithms for offline preference- based reward learning. Transactions on Machine Learning Research, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical con- siderations for healthcare settings. In Machine Learning for Healthcare Conference, pages 2-35.
PMLR, 2021.
Masatoshi Uchara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations, 2021.
Nils Wilde, Dana Kulić, and Stephen L Smith. Active preference learning using maximum regret. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10952-10959, IEFF, 2020.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fümkranz.. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46 2017.
13

---

Published as a conference paper at ICLR 2025
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offiin policy optimization. Advances in Neural Information Processing Systems, 33: 1124-11114,,, 2020.
Yuanzhao Zhai, Yiying Li, Zijian Gao, Xudong Gong, Kele Xu, Dawei Feng, Ding Bo, and Huaimin Wang. Optimistic model rollouts for pessimistic offline policy optimization. arXiv preprint arXiv:2401.05899 2024.
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline reinforcement learning with human feedback. In ICML 2023 Workshop The Many Facets of Preference-Based Learning, 2023a.
Wenhao Zhan, Masatoshi Uchara, Wen Sun, and Jason Lee. How query human feedback efficiently in rI? In ICML 2023 Workshop The Many Facets of Preference-Based Learning, 2023b.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feed- back from pairwise k-wise comparisons. In International Conference on Machine Learning, pages 43037-43067. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
A THEORETICAL DETAILS
This appendix provides proofs for the presented theorems and lemmas. In subsection A.1 we pro- vide details on how we define the maximum likelihood estimators and confidence intervals of the preference and transition models. In subsection A.2 we provide the proof that our uncertainty- penalized objective in Equation (5) lower bounds the true value function and thus forms valid pessimistic framework. In Appendix A.3] we show that the suboptimality of our offline preference elicitation framework is not lower-bounded by the performance of the optimal offline policy. In Appendix A..4] we provide our proof of theorem 5.1 analyzing the suboptimality of preferences sampled from an offline dataset. Finally, in Appendix A.55 we prove Theorem 6.1 the suboptimality of preference sampling over simulated rollouts.
which analyzes
A. MAXIMUM LIKELIHOOD AND CONFIDENCE INTERVALS
Let Fg denote a function class over X' + Ay. where X, are measurable sets, and g E Fg denotes a function to be estimated.
Let 9 denote the maximum likelihood estimator (MLE) of g based on dataset D {<"", y"" }") N argmax JEF, ((a.)) (z) We construct the confidence set around the MLE as follows:
log (vylx)
i E Fg (x,))
S 8}
g(y a:)
Lemma A.1 (MLE Guarantee, Lemma in Zhan al. (2023a)). Ler (0, 1] and define the event E that g E G.
CMLE
log (n
1
N
N
where CMLE > 0 is universal constant, then P(E) 2 1 5/2.
Proof. The proof follows that of Lemma in Zhan et al. (2023a) and uses Cramér-Chernofe method.
Let B be a L/N bracket Fr with |B] = Nrr (L/N). Denote the set of all right brackets in by
B (b: 3V.. (., bj E B). For E B, we have:
b(1"(z=")
b (=n))
E
exp
log
( (m))
E
exp
log
( (x
n=1
II I NN
g
(2.))
b(yl
< (1+1//N) Se.
samples in D as i.i.d. We the Tower property in the third step and the fact that b is 1/N- bracket for F, in the fourth: there exists g E F, such that [|g(-Iz) 6(-[x)ll S 1/N and thus [1((|=)]] < + 1/N, for all X.
Then by Markov's inequality, for any 5 E (0, 1). we have:
b(y"l"")
N
log b(y² (=")
P
log
g(y
> log(d)
< E
exp
log(1/))
ed.
By union bound, have for all E B.
b(y"
CALE log Nos, (N)) ))) < 8/2.
P
log
g(yn|/z))
15

---

Published as a conference paper at ICLR 2025
where CMLE is universal constant.
Finally, for all 9 E Fg. there exists B such that g(-1z) S 3(-1xr) for all x E X. As a result, for all E Fgg have:
g(y" (x") g0yn) (2)))
CMLE log N Nv) ))) S 8/2.
log
log
Under this event E, we have E 9 with probability 1 8/2. A confidence interval constructed via loglikelihood also incurs bound the total variation (TV) distance between and G: Lemma A.2 (TV-distance to MLE). Under the event &, we have, with probability 6, for all
9 E
< log NNF N ))
BynD [11)(1a)) 3(11))11
(6)
N
where e >> is a universal constant.
Proof. The proof follows that of al. (002a)). Proposition 14.
This guarantees that the true function is within interval around the MLE estimate with high probability.
We apply these lemmas to MLE estimates of transition and reward functions in Algorithm | to obtain the following guarantees.
Let ε denote the event R R and ET denote the event T T, R and T denote the respective confidence sets around the MLE. By Lemma A.1 events &n and Er have probability 6/2 choose AR MLL log(AFx (1/Np)/5)/N) and 3T caLE (1/N.))) where MLE MLE are universal constants. CR CT
A.2 MODEL-BASE PESSIMISM AND UNCERTAINTY PENALTIES
Lemma A.3 (Telescoping Lemma). For any reward model FR, and any two transition models T, FT:
T.R
V < Er RR
Σ ET
|[T(1);;,a)
(T-1s, a;)ll
Rmar
Proof. The proof follows that of Lemma 4.1 in Yu et al. (0200) or Lemma 4 in Zhan et al. (2023a).
Let W, be the expected return under policy with transition model 7 for the first ý steps, then transition model T for the rest of the episode. We have:
H-1
VT.
VT
Σ
W2 Wf+1.
Now,
W = R, + E 2y sy
E BBs 13. (1)
[V (7
W = R, + E e, a
(8)+1)]
where R3 is the expected return of the first j steps taken in T. Therefore.
W, W,+1
E E,, s, B, N
(es (85+ E (e
E sy ay ~ [[T((1s5,,,) T(- a,)ll Rmax
16

---

Published as a conference paper at ICLR 2025
under the boundedness assumption for R. Finally, have:
H-1
VRR
V
Σ
W
++1
==
H-1
Σ
Bey ay (ess t,
H-1
Σ
B Py
(1)) - a;)ll Rmax
==
Brady
Σ SARRET
(T{()));;g)) a()) Rmmxlli
Lemma A.4 (Pessimistic Transition Model). Under event &T, for all a E IT, R E Fr:
S V T,
f.
Proof.
V T&
-(V VT.R)
T.
Erady [[({T] - Brrady [UT(T)
Erod7 [B(T) - wr (T((T)
where we have used the telescoping lemma (Lemmal A.3 and where wr(T) = E(s,a)e- uT( >
E(sajer [T(-|s, a) - T(-|s, a),a)]11 Rmax under event Ex.
Lemma A.5 (Pessimistic Reward Model), Under event for all x E II, e Fr:
V T.R-ua
S V
T.R.
Proof.
T.R - (V R - )
T.R =
[R(T)] BTTNz (R(=)-((T)
ETTTY
[R((T) - uR((T)
2 Erndy
where we have used the fact that RR(T) R(T) 5 Em.er (R(s, R(s,a)) E(sneer uR(s, a) uR(T) under event &R.
Combining the above two lemmas gives the following result: Corollary A.1. Under events Er and ER, for all II:
V7. V-RR-u--u V.R.
This justifies the overall objective considered in our pessimistic policy optimization procedure in Section 6.3
A.3 SUROPTIMALITY LOWER BOUND: COUNTEREXAMPLE
Let Taffine = argmax, ETT minjer V F.R denote the optimal offline policy, which has access to the ground-truth reward function. In this section, we ask whether its suboptimality er =V= V.Tammn lower bound for the suboptimality of our learned policy ** after preference elicitation.
T.R
17

---

Published as a conference paper at ICLR 2025
81
R =
(00 0.8)
80
(a), 0.6)
R =
Figure 4: Tabular MDP. The environment starts in state so and has horizon H 1. Transition probabilities from state are given for the two binary actions di (which send the agent to the other state with comple- mentary probability).
Counterexample. Consider the MDP illustrated in Figure 4 The value function for the optimal policy n², which consists of taking action ag, is: V7,R = 0.8 1 + 0.2 0.8.
Now assume the following MLE estimate and uncertainty function for both the transition and reward models:
T(s an) 0.5;
UT ao) 0.4
((s So, 01) = 0.5;
ür (5 01) = 0.1
f(s1) F852) 0.5;
( (n())) 0.5
Assuming access to the learned transition model and the true reward function, we pessimistically estimate the value of both actions:
Vioo 0.1 1 0.1
Fun. R
v 0.6-0+ = 0.4
T
Thus, we have: IT mime (5o) argmax, a Va Tt A1. The offline policy picks the suboptimal action since the worst-case returns of this action are lower than those estimated for ag. Evaluating this policy in the real environment, we get V.Tam 0.6 + 0.4
0.4.
We now estimate the optimal policy in the learned transition and reward model. Applying pessimism with respect to both models, we get an equal estimated value of 0 for both actions ag and a1 If policy
optimization converges 7" ap, achieve optimal performance with TT 0.8 VT.
This example demonstrates that er VT.R V Tame 0.4 is lower bound for the suboptimality T.R
of R* as policy #* can achieve better performance (even optimal performance in this case) if errors in transition and reward model estimation compensate each other.
A.4 SUBOPTIMALITY OF OPRL: PROOF OF THEOREM 5.1
A.4.1 SUBOPTIMALITY DECOMPOSITION
Recall that T.r. Rm = argmin RE V denote the pessimistic transition and reward models, V
such that #* argmax,
We have:
V V*
= -F - V
(VF V Vin.Buu Rim
(V VT Rin (V R V Pran
< (V VA Prmd
S VŽ. VB"
Pra Rici
18

---

Published as a conference paper at ICLR 2025
where we have first used the optimality of ft (stating that V
The Rus 2 V
for all σ) and then
Tue Rudl
the pessimism principle (stating that V VV.R). Tur, Pur
Finally, we decompose the last term above as follows:
V
V S (V
R.
V
Puc Rust
+ (VT,
V Bad
(8)
transition
reward
We further analyze each term in the following sections.
A.4.2 ANALYSIS OF THE TRANSITION TERM
In this section, we now upper bound the transition term defined in Equation 8
Lemma A.6 (Lemma 4, Zhan et al. )2033)). Under the event Er, with probability - 6, we have for all T T.for all R E Gn. for all n:
H Nor
1
Bdg. [((T) -Ra (R(T) H Pime Or
Cr(FT,
log No
No.
where or is a constant.
Proof. From the telescoping lemma (Lemma A.3
we have:
V T,
VR S Rmaax By By
Σ
IT(+)s9,, (z) a-1os,
StayET
H Rmax E (s)add
---b,,,))
HRmm Cr xT) () Blaa) T(-(s,)) - T(-1s, a)
Under event Ex, by Lemma A.2 we have, with probability 1 - 8. for all T T:
1 Ees.a)~Domm (T a-)1) S N. or log
N5
H
1
No
This concludes our proof.
A.4.3 ANALYSIS THE REWARD TERM
Next, we upper bound the reward term defined in Equation
As in Zhan et al. (023a)), we consider the following value function: VF, Er [R(T)] Er-d []. where dpwet is fixed reference trajectory distribution. This baseline subtraction, which doesn't 't affect either the optimal policy or the analysis of the transition term, is needed as the approximated confidence set is based on the uncertainty in preference between two trajectories, not single one.
in the reward
Definition A.1 (Preference concentrability coefficient). The concentrability coefficient w.r.t. reward classes Fn. a target policy x² and reference trajectory distribution dpref is defined as:
En araeedee [u (TI.
CR(RR T²)
Ey ram-Dam [MPA (71, 72)]
Note that, for the purpose of our analysis, our definition differs from that Zhan et al. (2023a) who instead consider the max ratio of difference in rewards term: [R(T) - R( - R(T, + R( T2) the entire function class FR.
19

---

Published as a conference paper at ICLR 2025
Lemma A.7. Let trajectories for preference elicitation be sampled uniformly from the offline dataset. Under the event &R, with probability 1 - 6, we have for all T E Gr, for all R E R. for all n:
CR log Np
(NNTA (N))).
V T. S (2kkCn(Fu,
where > O is a constant and K = suprel- Rum, Raa] o'(r) measures the degree of non-linearity of the sigmoid function.
Proof.
V.R
V= P.Rut
=E Byddy" [R(r)] Er [R(T)] Tndd [Rim((T)] Byndyau (Rim [r]
En rdd ,7a sradd
[R(T)) R((T2) (Riar( (1) Rant(T2))]
S k k.Bii raddga P1((T > T2) Run (T1 x =22)
S k kBryaaf" modd
upp (TI,
KOR(FR, π ") )l [up (TI,
where Supre[-RPussc Baaree--aa--aae
Run.. Ruaa d')r measures the degree of non-linearity of the sigmoid function. In the first inequality, we have applied the mean value theorem, under Assumption 3.2 In the second inequality, we have used the definition of uncertainty function UP as we know Rint E R.
Now, under event &R, by Lemma A.2 we have, with probability
for all R
(T PR(T > T2) - -PA(T > T2) >
CR log Np
NN )
(10)
where en > 0 constant. This implies the following upper bound for the preference uncertainty function:
B (TI, S 2/
CR N,
log
(FNNTA
(11)
N₂
Under uniform sampling, the distribution of preferences in D Peet is that of the offline dataset:
E E(Trrs))Daamd [upa (TI, 12)) (T, [upx (T1, T2)]
Thus,
VT.
V
$2nCR(Fr,
CR my
log N,
N
1
1 N,
Lemma A.8. Let trajectories for preference elicitation be sampled through uncertainty sampling from the offline dataset. Under the event Eg, with probability we have for all T E frr all R. R, for all n:
CR log Np
(NNen
)
1 1
VT.
VT.... I (2nCO(FFr,T)
N₂
where CR is a constant and a S
Proof. The proof follows closely that of Lemma A.7 We introduce the preference concentrability coefficient defined for a general preference dataset:
E wdg woa [uPa (T1, T2)] (²(Fr,
En (ura (T1, T2)]
20

---

Published as a conference paper at ICLR 2025
We start from Equation
9
V T. Rui S KKB m so (T,,
T,R
=FCCeFFn a Pr (TI
CR log (/N5F < (2
1
No
N
where we have used Equation
11
Now consider the dataset of uncertainty-sammled preferences D pref. By definition, have:
Do [uPa (71, 72)]
E E m (uPM(TT. 72)]
Thus, we have: (R T²) I x)) In other words, we can write: CRFFR, x²) aCR(FRR x²), where a S This concludes our proof.
We now conclude the proof of Theorem 5.1 under events &R and Er.
From Lemma
A.6. we upper bound the transition term:
V T.R
V Rur
I
x /
CT No
log
H NFr
No
From Lemmas A.7 and A.8 upper bound the reward term:
VT.
V Run
S 2akCCR(FR TT"
log
N
N,
N
where a = 1 for uniform sampling < 1 for uncertainty sampling.
Combining with Equation
we obtain Theorem 5.1
A.5
SUBOPTIMALITY OF SIM-OPRL: PROOF OF THEOREM 6.1
A.5.1
SUROPTIMALITY DECOMPOSITION
We decompose the suboptimality slightly differently to Equation introducing the optimal offline policy (optimal in the pessimistic model under the true reward function): xffline argmax.en 1/2 1mt.R
V
V*
T.R
T.R Viaine
(VT
(V
Vaine Rus
Det Put.Bust Russ
S (V V Valine
(V
V Rund
Bun Rus
S VI.
Valine
(V
VT..R)
(VD"
Vddine
Tun.R
Prnt Puni Risk
(V
Va IT
+ (V Tua.R
(12)
Tat Ruad
transition
reward
where we have followed the same analysis as Appendix A.4.1 and used the optimality of Te in the last inequality.
The analysis the transition term identical to the above (Appendix) A.4.2 term next.
We analyze the reward
21

---

Published as a conference paper at ICLR 2025
A.5.2 ANALYSIS OF THE REWARD TERM
Lemma A.9 (Optimal Offline Policy In Set). Let Noffine denote the following set of near-optimal pessimistic policies, under the pessimitic transition model Twy and the reward confidence set R.
Hoffline
{ an Br [R(T)) VR R}
Under event &n, we have Tt offfine
TT Toftine.
Proof. Recall the definition of Tottine- Tottine argmax, a EnI V Itr.tt Note that there is no need to consider the preference baseline term in V when building Nommie since it is independent of the policy. Under event En, we have R Thus, Tomtin E HomMine.
Lemma A.10. Under event Eg, have, with probabillity 1 8:
V 1p0 Tag.
V Tagian Fag, Ruy
/ N₂
CR
log
( 1
N2p
Proof.
V Tun.
danne Rut
(V
ViT Rui.R) + (V.
Vx
Pui.
Pui.R
E
[R(T)] - Rereepp [R(T)]
E.
Jine (T) + Be ( (Riar(
E
[R(TT)) (((T))-[(T22] -
(Rma(TT)) - P
Ty modd Calise Fut
[Pn(T > T2) - P (ni T2)]
where x = SupreepPPP---- Russ Rand (r) measures the degree of non-linearity of the sigmoid function. We have applied the mean value theorem, under Assumption 3.2
As Ra E R, we have: PR(TI > T2) Pr (11 > T2) I upn (2)
Let dprer correspond to the distribution of the preference data, which consists of rollouts from ex- ploratory policies within the learned environment model: dpeef d% /2/2222 + / Recall that the near-optimal policy set Tomeine includes policy Toftine (Lemma A.9 and that 1 22 are the two more exploratory policies within this set:
rd boddiiiiin Time 72 mandd
(MPR (TI (TI, <
max #1l
E E
[uPA (T1, T2)] 2.ng
Now, under event ER, by Lemma A.2 we have, with probability 1 8 for all R E R:
B(T, PR(TI > 72) PE(TI + 12) <
CR N,
log N=
1
where en > O is constant. This implies the following upper bound for the preference uncertainty function:
CR Np
(FNOA (1)
lura (T,, r2)) < 2
log
(N
Thus, we obtain:
Tmme Rar
en log < 2
(nNtra
Np )
Bat
The resulting sample complexity of O O(kid) matches that of active preference learning within known environment (Saha et al. 2023 Chen et al. 2022).
22

---

Published as a conference paper at ICLR 2025
We now conclude the proof of Theorem 6.1 under events En and Ex. From Lemma
A.6
we upper bound the transition term:
VI.
V Tuc.R
S H Rmar Cr CT(Fr,
CT No
log JN
No.
From Lemma A.10
we upper bound the reward term:
ViTamine Tur.R
Vilamne Tur,Rur Rut
S 2
CR Np
log
L (.
NN
Combining with Equation
we obtain Theorem 6.1
B ADDITIONAL RELATED WORK
Table 3: Comparison of broader related work on preference elicitation and preference-based RL.
Pramework
Offline
Policy-hased Sampling
Robusiness Guarantees
Practical Implementation
PREFERENCE BASED DEEP RI.
D-REX Brown et al. 2020
PEBBLE
MRN Liu et al. 20226
X
X
SURI Park al al 2022
X
X
ala 2016
IPI.
Sadugh 2024
X
PREFERENCE ELICITATION
PbOP Chen et al. 2022
V
REGINE Zhin m. 2023b
FREEHAND ZZain
20233
34
OPRL (Shin c 2012
MoP-RL [L.u dl 2016
4
Max. Regret
2020
Biyik and Sudigh 2018
Biylk of al. 2019
Sim-OPRL (Ours)
In this section, we position this work against additional related literature. We summarize our analysis
T
First, we note that significant prior work on preference-based reinforcement learning is centered on improving preference modeling or policy optimization for fixed (uncertainty-based uniform) preference elicitation strategy (all methods in the top half of Table 3). This focus is orthogonal our work, and could certainly be combined with our preference elicitation method for optimal performance.
Next, comparing with other preference elicitation methods in Table (bottom half), we find that almost all works consider setting with online environment interaction, which makes them in- compatible with our problem setting. Online environment exploration eliminates need for pes- simism against transition dynamics (Levine et al., 2020), whereas we show that it is essential for both theoretical and empirical performance in the offline setting.
The uncertainty- based preference elicitation strategy adopted in most practical PbRL algorithms is to minimize uncertainty about preferences, or to maximize information gain about the reward function. From prior theoretical and empirical work (Chen et al. 2022 Lindner et al. 202T), know that this strategy is often suboptimal compared to actively reducing the set of candidate optimal policies, and both our theoretical and experimental results confirm this.
Finally, note that ours the first work to provide both a theoretical and empirical analysis of the sample complexity of preference elicitation in the fully offline setting.
23

---

Published as a conference paper at ICLR 2025
Beyond Offline Reinforcement Learning. With community's recent surge in interest in preference-basee reinforcement learning for its application to language model alignment (Stien- non et al. 2020 Ouyang et al.) 2022), we foresee many possible applications and extensions of our work beyond offline RI. Sample efficiency in preference collection is of paramount importance in Reinforcement Learning from Human Feedback (RLHF): recent works consider active learning pipelines based on uncertainty sampling to improve efficiency (Chen et al.) 2024] Melo et al.) 2024), similar to OPRL. The insights from our work suggest that a model-based elicitation strategy like Sim-OP OPRL,, designed to simulate responses from near-optimal policies, might perform even better. We believe this line of future work could prove critical to the language model alignment community.
C IMPLEMENTATION DETAILS
We trained all models on two 64-core AMD processors or a single NVIDIA RTX2080Ti GPU. The total wall-clock time for running all experiments presented in this paper amounted less than 72 hours.
Transition and Reward Function Training. For all baselines, transition and reward models were implemented linear classifiers (for the Star MDP), as two-layer perceptions with ReLU activation and hidden layer dimension 32 (Gridworld, Sepsis, MiniGrid environments), or as 5-layer MI.ss with ReLU activations and hidden sizes [512, 256, 128, 64,32) 32] for the HalfCheetah environments. Training was carried out for two or one epochs for the transition and reward models respectively, with the Adam optimizer KKngmaa and Ba 2014) and a learning rate of 10-3
We provide a more detailed practical algorithm for Sim-OPRL in Algorithm 3 For both our method and baselines relying on uncertainty sets (OPRL and PbOP), we estimated uncertainty sets by train- ing models initialized with different random seeds on different bootstraps of the data (sampling 90% of the data with replacement). We consider ensembles of size [T| | 5 for both transition and reward models. Hyperparameters Àr, An control the degree of pessimism in practice and could be considered equivalent adjusting margin parameters BR in our conceptual algorithm proposed in Section 4 Since the exact values prescribed by our theoretical analysis cannot be estimated, the user must set these parameters themselves. Hyperparameter optimization in offline RL is a chal- lenging problem (Levine et al. 2020);; for our experiments, we simply set Xy 0.5, AR 0.1 (StarMDP, Gridworld) and XT = ARR = 1 for the Sepsis environment.
Near-Optimal Policy Set and Exploratory Policies. Both Sim-OPRL and PbOP require con- structing set of near-optimal policies within learned model of the environment. Note that the PbOP algorithm in Chen et al. (2022) proposes to construct the near-optimal policy set by consider- ing all policies that have preference greater than 1/2 over all other policies in II, under transition and preference uncertainty bonus. This is infeasible to estimate in practice; we modified the algo- rithm allow for practical implementation. The motivation in building the set of plausibly optimal policies remains the same, but the theoretical guarantees may not hold.
We build Πomtine by maintaining a policy model for all E R, i.e., each element of the reward ensemble. Policy models are optimized to maximize returns under the transition model 7 and the reward function cies are identified by generating 10 rollouts of each of the candidate policies within the learned (SimOPRL) or true (PbOP) model. The trajectories (T1,72) maximizing the preference uncertainty function UP (TI, r2) are used for preference feedback. In PbOP, the trajectories are then added to the trajectories buffer and the transition model is retrained for 20 (Star MDP, Gridworld) or 200 steps (Sepsis).
dr (Sim-OPRL) or R + dy (PbOP). Next, the most exploratory poli-
Preference Feedback Collection. Preference labels provided through the ground-truth reward function associated with every environment. As stated in Section 4 for computational efficiency, we sample preferences in batches of (Star MDP, Gridworld) or 100 (Sepsis) reduce the number of model updates needed.
Policy Optimization. Policy optimization stages, both in estimating optimal policy sets in Sim- OPRL. and PbOP and in outputting final policies, are carried out exactly through linear programming
24

---

Published as a conference paper at ICLR 2025
Algorithm 3 Sim-OPRL: Practical Algorithm
Input: Observational trajectories dataset Doftine. Hyperparameters λT, ÅR.
Output: #*
1: Train an ensemble T of transition models via bootstrapping on the observational data Dottine:
T(-s,,)= = IT| Σ
T(-ls,), a); ur(s,a)= = max 7:,757
[Ti({s, -T2(-1a, a)lı Rmax
FET
2: Prrel f- 1.
3: for k == Np do
4: Estimate optimal offline policy set:
Hoffiine
{ π mrgmax=enB(.o)) [R(s, a) Arur(s, a)] vR RJ
5: Identify exploratory policies: T1, T2 = argmax mi me E nn sond [upa (T))
6: Rollouts in model: Ti di' 72 ~ dzp
7: Collect preference label o for (TI, 72).
8: Deret Preet J (T1, T2. Train an ensemble R of reward models via bootstrapping of the preference data Ppreti
9:
R(s,a)
Σ R(s,a); un(s,o)= RER
max Ri.
[1 a) R2(s,
[
10: end for
11: A +- argmax ren (s (Ā(s,a) Anun(s,a) Arur(s,a)
for the Star MDP and Gridworld using evxopt (Diamond and Boyd 2016). based on code from
Lindner et al. (202T). using Proximal Policy Optimization (Schulman et al. 2017) implemented stable- wetle--baeeiinee selsness (Raffin al. (02T) for the Sepsis and MiniGrid environments, and Soft Actor-Critic (Haarnoja et al.J 20188) for HalfCheetah. In the latter case, after every preference collection episode, reward and policy models were trained from the checkpoint of the previous iteration, for only 20 steps minimize computation.
Baselines and Ablations. We implement both OPRL. baselines within our model-based offline preference-based algorithm described in Section 4 Uncertainty sampling is taking the pair with maximum preference uncertainty over 45 pairs for every sample, to reduce the load of computing preference uncertainty over the entire trajectory buffer.
Our ablation study for Figure 2 is conducted as follows. For Sim-OPRL without pessimism in the output policy, we output the policy that maximizes the value function under the MLE estimate of the transition and reward function, π and R, after preference acquisition. For Sim-OPRL without pessimism in the simulated rollouts, we estimate the optimal policy set Hommine in the MLE estimate of the transition model instead of its pessimistic counterpart. Finally, for Sim-OPRL without optimism in the simulated rollouts, we generate rollouts from any two policies in Hoffine instead of the most explorative ones.
D ENVIRONMENT DETAILS
Star MDP. We illustrate the transition dynamics underlying the Star MDP Figure Transition probabilities 0.9 for all depicted solid arrows, and leave the state unchanged otherwise. Other actions also keep the state unchanged with probability 1. Episodes have length = 3 and start from Sg. Unless specified otherwise, the offline dataset Domine consists of 40 trajectories which only cover states (8p, $1, 8y) and (83, $1, 82).
Preferences collected over samples from the offline dataset learn slowly about the negative reward in the bottom state, as it is always included the sampled trajectories. Instead, simulated rollouts can query direct comparison between the optimal path and one that includes it. This example illustrates
25

---

Published as a conference paper at ICLR 2025
54
R == 10
09
4g
G0
$2
Saart
Q1
Doffiine
R
6
83
R
Figure 5: Star MDP. Transition probabilities 0.9 for all solid arrows. Omitted actions complementary transitions keep the state unchanged.
Start
-1
-1
20
10
Figure 6: Gridworld environment. Rewards every state are indicated non-zero. Transition probabilities are 0.9. Thick lines indicate an obstacle, through which state transitions have prohability zero.
clearly why querying feedback simulated rollouts achieves better environment returns than samples from the offline buffer.
Gridworld. We illustrate the gridworld environment Figure 6 The environment consists of a 4 x 4 grid with states associated with different rewards, including negative-reward region in the top-right corner, a high-reward but unreachable state, and moderate-reward state at the bottom right corner. Each episode starts in the top-left corner. Transition probabilities for each of the four actions (top, left, bot t om, right) are 0.9 for the intended direction, and 0.1 for the others; and action stay remains in the current state with probability through obstacles have probability zero, with the remainder of the probability mass for each action being distributed amongst other directions equally. The offline dataset contains 150 episodes and the behavioral policy is e-optimal with noise = 0.1. Episodes have length H 10.
Transitions beyond the grid limits or
MiniCrid--FurRoom: We also conduct experiments on the mimigrid--furrromm--- D4RL dataset (Pu et al., 2020), ignoring reward information the offline dataset. In this en- vironment, the agent must navigate maze consisting of four interconnected rooms and reach Bhevvlier-Boiivertt
green goal square (Chevalier- Boisvert et al. 2023). Agent and goal squares are randomly placed at the beginning of every episode.
SaafChhetaa-Randoon The www.heetth-randooor v2 dataset is also part of the D4RL benchmark (Fu et al., 2020). Our choice is motivated by Shin et al. (2022) who identify it as a particularly challenging offline preference-based reinforcement learning task. The dataset consists of I million transitions induced by a random policy in the MuJoCo environment Halfchectah-v2, which rewards agents if they move forward. The observation space 17-dimensional.
26

---

Published as a conference paper at ICLR 2025
Table 4: Transition dynamics of the sepsis simulator environment adapted from (Tang and Wiens 2021
(Oerrtt and Sontag 2019
Table
Step
Variable
Current
New
Change
Variable Affected
Effect
Heart Rate
high normal w:p. 0,5
Systolic Blood Pressure
high - normal w.p. 0.5
Antibiotics
Heart Rate
nornal - high w.p. 0.1
off
withdrawn
Systolic Blood Pressure
normal - high w.p. 0.5
Ventilation
on
Oxygen Saturation
- normal w.p.
off
withdrawn
Oxygen Saturation
normal low w.p. 0.1
low - normal w-p. 0.7 (mon diabetic)
normal high w.p. 0.7 (non-diahetic)
Vasodilators
Systolic Blood Pressure
low - normal w.p. 0.5 (diabetic)
on
high w.p. (diabetic)
nornal -+ high w.p. 0.9 (diabetic)
very low, - normal, normal high, high very high w.p. 0,5 (diabetic)
Blood Glucose
nornal - low w.p. 0.1 (mon diabetic)
high normal w.p. 0.1 (non-diahetie)
off
withdrawn Systolic Blood Pressure
normal - low w.p. 0.05 (diabetic)
high normal w.p. 0.05 (diahetic)
Heart Rate
Vitals spontaneously flactuate when not affected by treatmett (either enabled or withdrawn). the level fluctuates #1 w.p. 0.1, except: glucose fluctuates w.p. 0.3 (diabetic).
Systolic Blood Pressure
fluctuate
Oxygen Saturation
Blood Glocose
Sepsis Simulation. The sepsis simulator (Oberst and Sontag 2019) a commonly used en- vironment for medically-motive mottivtee RL work (Tang and Wiens, 202Ip. We use the original au- thors' publicly available code: hhtpps: / thub. com/ c i in Aaniialln htmmmm//mmmlllmmmmmmmmm mamubellmax--eee scm/ tree/ stt//iimnae v
(MIT license). The state space consists of 1,440 dis- crete states based on different observational variables (heart rate, blood pressure, oxygen concentra- tion. glucose, diabetes status). The action space consists of three binary treatment options (antibiotic administration, vasopressor administration, and mechanical ventilation). The complex transition dy- namics of the environment determine how each treatment affects the value of each vital sign, and are summarized in Table these were designed to reflect patients' physiology (Oberst and Sontag, 2019). The ground truth reward function is sparse and only assigns a positive reward of +1 sur- viving patients and negative reward of -1 if death occurs (3 or more abnormal vitals) during their stay. The offline trajectories dataset includes 10,000 episodes following an e-optimal policy with noise 0.1 and the episode length is H = 20.
E ADDITIONAL RESULTS
We include additional results in this section.
In Figure 7 we report the accuracy of the transition and preference model achieved for the Star MDP as we vary the size of optimality of the offline dataset. Accuracy is measured against all possible state transitions and over 100 pairs of random trajectories (random combinations of the 5 states and actions in a sequence of H = 3). This complements our analysis in Section 7 and fig. 3 We see a steady improvement in both transition and reward model quality as we increase the amount of observational data in Figure 7al which explains the observed dependence of N, on N, in Figure 3a
In Figure 7b we notice low model performance at both extremes of the x-axis. When the dataset is fully optimal, we find that all trajectories involve the same sequence of actions and states, so learning transition reward model from this data is challenging. We reach similar conclusion at the other end of the spectrum at high density ratios, where the coverage the optimal states reduces. We reach highest performance for both models at intermediate values, when diversity of the observational data is high.
Still, it is important to stress that the highest accuracy of both models does not necessarily translate to the best performmn policy: good performance on the distribution induced by the optimal policy is more important, as formalized by the concentrability coefficients.
27

---

Published as a conference paper at ICLR 2025
1.0
0.8
Accuracy
Accursiy
0.6
0.4
0.4
0.2
Transition
Transitonn
Rewaed (OPR)) Unowtaintyl
Reward (GPRL Uscurtaisty)
Rewand (Si--PRLL)
Reward OPRL)
0
20
40
60
80
100
10r
161
100
Namber of Offiine Trajectories
Donsity ratio confficient Id (s. 4t0/ al
(a) As function of offline dataset size.
(h) As function of optimal coverage.
Figure Transition and preference model accuracy as function of properties (Star MDP). Preference elicitation is carried out until 10 preferences are queried. Mean and 95% confidence intervals over 20 experiments. Note that the transition model is the same for the two methods, as they have access to the same dataset.
observational data
Table 5: Sample complexity N, with Sim-OPRL as function of dataset optimality, to reach suboptimal- ity gap 20 normalized returns for the D4RL HalfCheetah environment. Mean and 95% confidence interval over experiments.
Offline Dataset
OPRL. Uniform
OPRI. Uncertainty
Sim-OPRI.
SeliCheetah-Randdoe
108 = 9
71
50 + 10
HalfCheetah-Mediun
63 + 8
49 + 6
36 +
47
45
30
Table 6: Policy performance for fixed preference budget (N,, 30) for different D4RL HalfCheetah
datasets. Normalized environment returns, mean 95% confidence interval over experiments.
Offline Dataset
OPRL Uniform
OPRL Uncertainty
Sim-OPRL
HalfCheetah- Raniomm
17
47 10
49 +
Halfcheetah-Meet
43 6
63 +
69 +
Halfcheetah Seath--Medd---a-a Expert
57
72
80 +
To complement the analysis with a more complex environment, we also ran different preference elicitation algorithms on D4RL HalfCheetah datasets of varying optimality. In Table report the number of preference samples needed to achieve a target suboptimality with these different datasets. We reach the same conclusion above: fewer preferences are needed as the dataset becomes more optimal.
In Table 6 we report policy performance for a fixed preference budget. With fixed preference budget, policies learned from suboptimal observational datasets achieve lower returns, due to the worse quality of the transition model. [Shin et al.) (2022) reach similar conclusions when evaluating policy performance under different OPRL methods (in their Table 2).
F BROADER IMPACT
Better preference elicitation strategies for offline reinforcement learning have the potential to facili- tate and improve decision-making real-world safety-critical domains like healthcare or economics, by reducing reliance on direct environment interaction and reducing human effort in providing feed- back. Potential downsides could include the amplification of biases in the offline data, potentially leading to suboptimal or unfair policies. Thorough evaluation is therefore crucial to mitigate this before deploying models in such real-world applications. In addition, human preferences may not
28

---

Published as a conference paper at ICLR 2025
be fully captured by binary comparisons. As noted in our conclusion, we hope that future work will explore richer feedback mechanisms to better model complex becision-making objectives.
29


Reviews:
[{"review_id": "Fmtsa0bQ9G", "reviewer": "Reviewer_tF7g", "summary": "This paper studies preference-based reinforcement learning (PbRL) in offline setting, in which the agent utilizes a fixed trajectory dataset for policy learning and can query humans for preference feedback. In particular, the authors propose to sample preference queries by rolling out trajectory data using learned models of MDPs. The authors provides theoretical guarantees for the sample complexity of their proposed strategy and verify it on simple control tasks.", "questions": "1. Based on your theoretical analysis, could you discuss how you expect the performance will change on dataset of varying optimality?\n2. Could you present experiment results on other dataset for the Cheetah environment, such as medium, medium-expert and expert, to support your discussion?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of using simulated rollouts in preference queries is a natural but unexplored idea in the literature of PbRL. One strength of this paper is that, the authors show the effectiveness in terms of sample complexity both theoretically and empirically.", "weaknesses": "My concern is about the quality of learned policies. While I agree with the optimality criterion mentioned in 3.2, I think to ensure the practical value of the proposed strategy, it is important to include evaluations for offline dataset of varying optimality. This is because for high-dimensional tasks, under a fixed budget of offline trajectories, the coverage over state-action space and the optimality of the behavior policy, can be conflicting objectives. The state-action space is less covered by good behavior policies, yet this reduced coverage can raise concerns on learned transition model. See detailed question below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Az1dqDkJQ", "reviewer": "Reviewer_8W5G", "summary": "This paper uses the offline dataset to learn the environment model. They do not assume they have access to the reward in the offline data set. Such offline datasets contribute to the overall learning by providing an estimation of the transition probability. This paper provides a theoretical analysis of reinforcement learning with offline datasets to achieve preference elicitation. The experiments show their algorithms outperform other algorithms in several environments. They also conducted an ablation test to show the importance of pessimistic with respect to the transition model.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1. This paper provides a good theoretical analysis of preference elicitation with the offline datasets. It bounds the value difference between the optimal policy under the estimated transition model and the true optimal policy. Such bounds are achieved by decomposing the loss from the model estimation and the reward estimation.\n2. Experiments show the proposed methods outperform other algorithms in several environments.\n3. This paper conducted an ablation study to show the importance of pessimistic with respect to the transition model.", "weaknesses": "Weaknesses:\n\n1. The experiment environments are relatively simple. The grid world is quite small. It is interesting to try to extend this to more challenging reinforcement learning benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6tLmr2m7wT", "reviewer": "Reviewer_BKXk", "summary": "The paper presents Sim-OPRL, an offline preference-based reinforcement learning algorithm that addresses the challenge of acquiring preference feedback in a fully offline setup. It leverages a learned environment model to elicit preference feedback on simulated rollouts, balancing conservatism and exploration. The main idea is to employ a pessimistic estimation for the transition dynamics (based on the offline dataset) for the OOD issue, and use an optimistic estimation for the reward model (based on the preference elicitation data). The benefit of using simulated rollouts is to avoid wasting preference budget on trajectories with low rewards. The authors provide theoretical guarantees on sample complexity and demonstrate the empirical performance of a practical version of Sim-OPRL across various environments, showing its superiority over previous baseline methods (OPRL and PbOP).", "questions": "- I do not quite understand “An advantage of sampling from the offline buffer, however, is that it is not sensitive to the quality of the model” in L346. What does “the model” refer to?\n- Should $N_T$ in the second equation in L369 be $N_R$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper focuses on the preference elicitation problem on offline RL, which attracts wide attention recently from many fields (such as RLHF for LLMs).\n- This paper has theoretical results on the proposed algorithm with some high-level insights (e.g., pessimism for dynamics and optimism for reward modeling).\n- This paper has practical algorithm designs and good empirical results.", "weaknesses": "- **Complexity of Implementation:** The algorithm's reliance on learning several accurate dynamics model might be challenging in practice, especially if the model fails to capture the true dynamics. Moreover, Sim-OPRL requires the trajectory rollouts using the dynamics model and the error may accumulate, which poses higher requirements for the dynamics model. Do the authors have any idea on how to design practical algorithms with less computational overhead (e.g., estimating multiple models) and on more complex environments (e.g., when it is hard to learn an accurate dynamics model).\n- **Lack of study on the dependence on quality of offline data and feedback:** The performance of Sim-OPRL may be heavily dependent on the quality and coverage of the offline dataset. For the experiments in on the tasks listed in Table 2, how are the offline datasets are collected? Are they expert datasets (so the concentrability coefficients are small)? How the feedback is generated in the experiments? How would the algorithm perform when we vary the feedback quality?\n- Minor: What is ``\\hat{R}_\\text{inf}``? I can guess it is pessimistic reward, but ``\\hat{R}_\\text{inf}`` and ``\\hat{T}_\\text{inf}`` are not formally defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q6V1LkvUwj", "reviewer": "Reviewer_wreC", "summary": "The paper addresses the challenge of applying RL to real-world scenarios where direct interaction with the environment is impractical or unsafe. Traditional learning methods require environment interactions, which can be risky in certain fields (like healthcare applications). The paper proposes an algorithm called Sim-OPRL, an offline PBRL learning algorithm that learns from preferences without needing online interaction. This algorithm uses a learned environment model to simulate rollouts and gather preference feedback, balancing pessimism for out-of-distribution data and optimism for acquiring informative preferences. The paper formalizes the problem of preference elicitation in offline RL, proposes a novel algorithm, and provides theoretical guarantees on its sample complexity.\n\nThe paper also demonstrates the effectiveness of Sim-OPRL through empirical validation in various environments, including a gridworld and a sepsis simulation. The results show that Sim-OPRL outperforms an existing baseline algorithm (OPRL) in terms of sample efficiency and policy performance. The paper shows that by leveraging simulated rollouts, their algorithm efficiently learns the optimal policy while minimizing the number of human queries required.", "questions": "1. How does the complexity of the reward function impact the performance of Sim-OPRL? Have you (or do you plan to) test the algorithm with environments that are characterized by more complex, multi-objective, or non-linear reward functions? If the method is agnostic to the reward function (aside from sparsity) it would help to show that as well.\n2. Can you provide more details on the sensitivity of Sim-OPRL to its hyperparameters, such as the pessimism and optimism parameters? How do you recommend tuning these parameters in practice? It may be insightful to include ablation testing in the appendix that demonstrates the sensitivity (or robustness) to hyperparameter selection, especially as this could drastically affect the real-world viability of the algorithm. \n3. Are there any other algorithms that would serve as a effective and informative baseline for Sim-OPRL? If not, would it be possible to run experiments that demonstrate learning performance on naive methods?\n4. Could you please clarify the rationale behind limiting the experiments to the selected datasets and environments? Are there specific challenges that restrict the application of the method to a broader range of environments and dataset combinations? If there are no such constraints, additional experimental results would be valuable. Conversely, if limitations do exist, it would be beneficial to outline what they are, the reasons for their existence, and why they do not compromise the method's overall effectiveness and practical utility.\n5. Generally speaking could the authors please explain the motivations for the setting further? Specifically, would it be practical to compare the results of Sim-OPRL to running standard offline RL algorithms (CQL, IQL, TD3_BC etc.) on the offline dataset directly? If not, why not?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The authors provide strong theoretical guarantees on the sample complexity of their approach, ensuring that the algorithm is both efficient and reliable. Additionally, the empirical results across various environments demonstrate the practical effectiveness and scalability of Sim-OPRL, showing it outperforms existing methods in terms of sample efficiency and policy performance.\n2. Sim-OPRL incorporates a pessimistic approach to handle out-of-distribution data, ensuring robustness to model uncertainty. This is particularly important in offline settings where the data may not cover the entire state-action space. Being robust to OOD data makes the algorithm far more applicable to ‘real-world’ problems/settings.\n3. The paper makes a compelling case due to their incorporation of theoretical and empirical evidence. To back up their theoretical insights, they conduct extensive experiments across two different environments. This provided empirical data confirms the practical applicability and robustness of Sim-OPRL, illustrating its effectiveness in scenarios where direct environment interaction is not feasible.\n4. The attached code is well-written and largely self-documenting, with a clear and logical internal structure. This design not only facilitates ease of use for other users looking to implement the Sim-OPRL algorithm but also made the process of reviewing the practical implementation and validating the experiments straightforward and efficient. This made the review process much easier.", "weaknesses": "1. The paper’s empirical section does not properly consider different baseline algorithms to compare theirs with. The only algorithm that the authors use as a baseline is OPRL. This severely limits the ability to fully assess the relative performance and advantages of Sim-OPRL. To rectify this, The authors should consider including a wider array of offline PBRL algorithms/frameworks in their experiments.\n2. The paper demonstrates promising results in the demonstrated environments, but it lacks validation in more complex and realistic settings. To strengthen the evidence of the algorithm’s practical applicability, the authors should evaluate Sim-OPRL on several different datasets. One example could be MuJoCo style datasets. Other relevant papers in the field construct preference datasets from the D4RL offline benchmark. These datasets provide a more challenging and ‘closer to real world’ testbed. Evaluation on such environments (in conjunction with adding more baseline algorithms) could result in a better assessment of the algorithm’s robustness, scalability, and generalizability.\n3. The paper demonstrates the algorithm’s performance in relatively small-scale environments. Empirically, it does not seem to address scalability to larger, more complex environments. Due to the smaller scale test environments (Gridworld & Sepsis), the the actual scalability of the algorithm (particularly in real-world deployments outside of benchmarks) remains unclear. \n4. As the authors state, for the sepsis environment, the requisite number of preference samples is rather large, due to the sparse reward function. This seems like an inherent limitation, which they posit could be solved by warm-starting the reward model. It would be interesting to see this data and how it affects performance. If a sparse reward function is a true limitation of the Sim-OPRL method, the authors should show more experiments demonstrating that this can be 'worked around' by performing warm starts. This could also help to further justify the real world applicability of the algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0txlCLIAtL", "reviewer": "Reviewer_aVUF", "summary": "This paper delves into offline reinforcement learning from preference feedback and proposes an offline preference elicitation method to simulate trajectories from the learned environment model instead of sampling trajectories directly from the offline dataset. They provide theoretical justification for the previous RL with preference feedback method and show that their proposed method can effectively reduce the sample complexity upper bound. They also propose an empirical algorithm and show it can outperform prior methods and achieve SOTA on offline PbRL setups without access to the ground truth rewarded. They finally iid ablation studies to show the importance of incorporating the principle of pessimism.", "questions": "/", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. They delve into very interesting setups: offline RL with preference feedback.\n\n2. Their theoretical results are solid and show he advantage of their proposed preference elicitation algorithm over prior methods.\n\n3. They propose a practical algorithm for implementation and extensive experiments show that their method outperform prior methods in several environment.", "weaknesses": "I do not see any big issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)