Preference Elicitation for Offline Reinforcement Learning

## Abstract

Abstract Applying reinforcement learning (RL) to real-world problems is often made challenging by the inability to interact with the environment and the difficulty of designing reward functions. Offline RL addresses the first challenge by considering access to an offline dataset of environment interactions labeled by the reward function. In contrast, Preference-based RL does not assume access to the reward function and learns it from preferences, but typically requires an online interaction with the environment. We bridge the gap between these frameworks by exploring efficient methods for acquiring preference feedback in a fully offline setup. We propose Sim-OPRL, an offline preference-based reinforcement learning algorithm, which leverages a learned environment model to elicit preference feedback on simulated rollouts. Drawing on insights from both the offline RL and the preference-based RL literature, our algorithm employs a pessimistic approach for out-of-distribution data, and an optimistic approach for acquiring informative preferences about the optimal policy. We provide theoretical guarantees regarding the sample complexity of our approach, dependent on how well the offline data covers the optimal policy. Finally, we demonstrate the empirical performance of Sim-OPRL in different environments.

## 1 Introduction

While reinforcement learning (RL) [Sutton and Barto, ] achieves excellent performance in various decision-making tasks [Kendall et al., , Mirhoseini et al., , Degrave et al., ], its practical deployment remains limited by the requirement of direct interaction with the environment. This can be impractical or unsafe in real-world scenarios. For example, patient management and treatment in intensive care units involve complex decision-making that has often been framed as a reinforcement learning problem [Raghu et al., , Komorowski et al., ]. However, the timing, dosage, and combination of treatments required are critical to patient safety, and incorrect decisions can lead to severe complications or death, making the use of traditional RL algorithms unfeasible [Gottesman et al., , Tang and Wiens, ]. Offline RL emerges as a promising solution, allowing policy learning from entirely observational data [Levine et al., ].

Still, a challenge with Offline RL is its requirement for an explicit reward function. Quantifying the numerical value of taking a certain action in a given environment state is challenging in many applications [Yu et al., ]. Preference-based RL offers a compelling alternative, relying on comparisons between different actions or trajectories [Wirth et al., ] and being often easier for humans to provide [Christiano et al., ]. In medical settings, for instance, clinicians may be queried for feedback on which trajectories lead to favorable outcomes. Unfortunately, most algorithms for preference acquisition require environment interaction [Saha et al., , Chen et al., , Lindner et al., ] and are therefore not applicable to the offline setting.

Lack of environment interaction and reward learning are thus two critical challenges for real-world RL deployment that are rarely tackled jointly. In this work, we address the problem of preference elicitation for offline reinforcement learning by asking: What trajectories should we sample to minimize the number of human queries required to learn the best offline policy? This presents a challenging problem as it combines learning from offline data and active feedback acquisition, two frameworks that require opposing inductive biases for conservatism and exploration, respectively.

To the best of our knowledge, the only strategy proposed in prior work is to acquire feedback directly over samples within an offline dataset of trajectories [Shin et al., , Offline Preference-based Reward Learning (OPRL)]. We propose an alternative solution that queries feedback on simulated rollouts by leveraging a learned environment model. Our offline preference-based reinforcement learning algorithm, Sim-OPRL, strikes a balance between conservatism and exploration by combining pessimism when handling states out-of-distribution from the observational data [Jin et al., , Zhan et al., ], and optimism in acquiring informative preferences about the optimal policy [Saha et al., , Chen et al., ]. We study the efficiency of our approach through both theoretical and empirical analysis, demonstrating the superior performance of Sim-OPRL across various environments.

Our contributions are the following: (1) In Section 3, we first formalize the new problem setting of preference elicitation for offline reinforcement learning, which allows for complementing offline data with preference feedback. This framework is crucial for real-world RL applications where direct environment interaction is unsafe or impractical and reward functions are challenging to design manually, yet experts can be queried for their knowledge. (2) In Section 4, we propose a novel offline preference-based RL algorithm that is independent of the specific preference elicitation strategy and recovers a robust policy from an offline dataset and preference feedback. (3) Next, in Section 5, we provide theoretical guarantees on eliciting preferences over samples from the offline dataset, complementing work from Shin et al. []. (4) Then, in Section 6 we propose our own efficient preference elicitation algorithm based on simulated rollouts in a learned environment model. (5) Finally, we establish the theoretical guarantees of our algorithm and demonstrate its empirical efficiency and scalability in different decision-making environments.

## 2 Related Work

Our problem setting shares similarities with Offline RL and Preference-based RL, which we summarize below. We position ourselves relative to our closest related works in Table 1.

**Table 1: Comparison of related work on preference elicitation.**
| Framework | Offline | Efficient Sampling | Robustness Guarantees | Practical Implementation |
| --- | --- | --- | --- | --- |
| PbOP [Chen et al., ] | ✗ | ✓ | ✓ | ✗ |
| REGIME [Zhan et al., ] | ✗ | ✓ | ✓ | ✗ |
| FREEHAND [Zhan et al., ] | ✓ | ✗ | ✓ | ✗ |
| OPRL [Shin et al., ] | ✓ | ✓(^1^11We demonstrate this in the present work.) | ✗ | ✓ |
| Sim-OPRL (Ours) | ✓ | ✓ | ✓ | ✓ |

Offline RL.
Offline Reinforcement Learning has gained significant traction in recent years, as the practicality of training RL agents without environment interaction makes it relevant to real-world applications [Levine et al., ]. However, learning from observational data only is a source of bias in the model, as the data may not cover the entire state-action space. Offline RL algorithms therefore output pessimistic policies, which has been shown to minimize suboptimality Jin et al. []. Model-based approaches show particular promise for their sample efficiency [Yu et al., , Kidambi et al., , Rigter et al., , Zhai et al., , Uehara and Sun, ]. In this work, we study the setting where reward signals are unavailable and must be estimated by actively querying preference feedback.

Preference-based RL. Rather than accessing numerical reward values for each state-action pair as in traditional online RL, preference-based RL learns the reward model through collecting pairwise preferences over trajectories [Wirth et al., ]. Different preference elicitation strategies have been proposed for this framework, generally based on knowing the transition model exactly or on having access to the environment for rollouts [Christiano et al., , Saha et al., , Chen et al., , Lindner et al., , Zhan et al., , Sadigh et al., , Brown et al., ].

Offline Preference-based RL. The development of preference-based RL algorithms based on offline data only is critical to settings where environment interaction is not feasible for safety and efficiency reasons. Still, this framework remains largely unexplored in the literature. While Zhu et al. [], Zhan et al. [] demonstrate the value of pessimism in offline preference-based reinforcement learning, they do not consider how to query feedback actively. On the other hand, Shin et al. [] propose an empirical comparison of different preference sampling trajectories from an offline trajectories buffer. In Section 5, we provide a theoretical analysis of their approach, then propose an alternative sampling strategy based on simulated trajectory rollouts in Section 6, which enjoys both theoretical and empirical motivation.

## 3 Problem formulation

### 3.1 Preliminaries

Markov Decision Process. We consider the episodic Markov Decision Process (MDP), defined by the tuple ${\mathcal{M}=(\mathcal{S},\mathcal{A},H,T,{R})}$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $H$ is the episode length, $T:\mathcal{S}\times\mathcal{A}\rightarrow\Delta_{\mathcal{S}}$ is the transition function, ${R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function.
We assume an initial state $s_{0}$, but our analysis could be easily generalized to a fixed initial state distribution. At time $t$, the environment is at state $s_{t}\in\mathcal{S}$ and an agent selects an action $a_{t}\in\mathcal{A}$. The agent then receives a reward ${R}(s_{t},a_{t})$ and the environment transitions to state $s_{t+1}\sim T({}\cdot{}|s_{t},a_{t})$. We describe an agent’s behavior through a policy function $\pi:\mathcal{S}\rightarrow\Delta_{\mathcal{A}}$, such that $\pi(a|s)$ is the probability of taking action $a$ in state $s$. Let $\tau=(s_{0},a_{0},\ldots s_{H},a_{H})$ denote the trajectory of state-action pairs of an interaction episode with the environment. With an abuse of notation, we also write $R(\tau)=\sum_{t}R(s_{t},a_{t})$. Let $d^{\pi}_{T}$ denote the distribution of trajectories induced by rolling out policy $\pi$ in transition model $T$. We denote the expected return of policy $\pi$ as $V^{\pi}_{T,R}=\mathbb{E}_{\tau\sim d^{\pi}_{T}}[R(\tau)]$, and $\pi^{*}=\text{argmax}_{\pi}V^{\pi}_{T,R}$ denotes the optimal policy in $\mathcal{M}$.

Preference-based Reinforcement Learning. Rather than observing numerical rewards at each state and action, we receive preference feedback over trajectories. For a pair of trajectories $(\tau_{1},\tau_{2})$, we obtain binary feedback $o\in\{0,1\}$ about whether $\tau_{1}$ is preferred to $\tau_{2}$. We assume that preference labels follow the Bradley-Terry model [Bradley and Terry, ]:

$$ $P_{R}(\tau_{1}\succ\tau_{2})\vcentcolon=P(o=1|\tau_{1},\tau_{2})=\frac{\exp(R( \tau_{1}))}{\exp(R(\tau_{1}))+\exp(R(\tau_{2}))}=\sigma(R(\tau_{1})-R(\tau_{2} )),$ (1) $$

where $\succ$ denotes a preference relationship and $\sigma$ is the sigmoid function. Within this framework, preference elicitation refers to the process of sampling preferences to obtain information about both the preference function and the system dynamics [Wirth et al., ].

### 3.2 Offline Preference Elicitation

We assume access to an observational dataset of trajectories $\mathcal{D}_{offline}=\{\tau:\tau\sim d^{\pi_{\beta}}_{T}\}$, where $\pi_{\beta}$ is an unknown behavioural policy in $\mathcal{M}$. As in Offline RL, we do not have access to the decision-making environment to observe transition dynamics or rewards under alternative action choices. We assume not to have access to the reward function, but we can query preference feedback from a human to generate a dataset of preferences $\mathcal{D}_{pref}=\{(\tau_{1},\tau_{2},o)\}$.

Optimality Criterion.
Based only on our offline dataset $\mathcal{D}_{offline}$, our goal is to recover a policy $\hat{\pi}^{*}$ that minimizes suboptimality in the true environment with as few human preference queries as possible. Let $\pi^{*}_{offline}$ denote the optimal offline policy estimated based on the offline data, with access to the true reward function $R$, and let $\epsilon_{T}$ denote its suboptimality. Since preference elicitation only allows us to estimate the reward function, we do not aim to achieve a suboptimality less than $\epsilon_{T}$.(^2^22However, $\epsilon_{T}$ is not formally a lower bound for our problem, as shown in Section A.3.) Our objective is then formalized as follows.

###### Definition 3.1 (Optimality Criterion of Offline Preference Elicitation) .

Let $\pi^{*}$ be the optimal policy in $\mathcal{M}$ and $\hat{\pi}^{*}$ be the estimated optimal policy based on an offline dataset $\mathcal{D}_{offline}$ and $N_{p}>0$ preference queries. Let $\epsilon_{T}$ be the inherent suboptimality assuming access to the true reward function. We say that a sampling strategy is $(\epsilon,\delta,N_{p})$-correct if for every $\epsilon\geq\epsilon_{T}$, with probability at least $(1-\delta)$, it holds that $V^{\pi^{*}}_{T,R}-V^{\hat{\pi}^{*}}_{T,R}\leq\epsilon$.

Our work is the first to formalize this important problem, which faces the challenge of balancing exploration when actively acquiring feedback and bias mitigation in learning from offline data.

Function classes. We estimate the reward function and transition kernel with general function approximation; let $\mathcal{F}_{R}$ and $\mathcal{F}_{T}$ denote the classes of functions considered respectively. We also assume a policy class $\Pi$. Our theoretical analysis also requires the following assumptions and definitions, which are standard in preference-based RL [Chen et al., , Zhan et al., ].

###### Assumption 3.1 (Realizability) .

The true reward function belongs to the reward class: $R\in\mathcal{F}_{R}$. The true transition function belongs to the transition class: $T\in\mathcal{F}_{T}$. The optimal policy belongs to the policy class: $\pi^{*}\in\Pi$.

###### Assumption 3.2 (Boundedness) .

The reward function is bounded: $0\leq\tilde{R}(\tau)\leq R_{max}$ for all $\tilde{R}\in\mathcal{F}_{R}$ and all trajectories $\tau$.

###### Definition 3.2 ( $\epsilon$ -bracketing number) .

Let $\mathcal{F}$ be a class of real functions $f:\mathcal{X}\rightarrow\mathbb{R}$.
We say $(l,u)$ is an $\epsilon$-bracket if $l(x)\leq u(x)$ and $\|u(x)-l(x)\|_{1}\leq\epsilon$ for all $x\in\mathcal{X}$.
The $\epsilon$-bracketing number of $\mathcal{F}$, denoted $\mathcal{N}_{\mathcal{F}}(\epsilon)$, is the minimal number of $\epsilon$-brackets $(l^{n},u^{n})_{n=1}^{N}$ needed so that for any $f\in\mathcal{F}$, there is a bracket $i\in[N]$ containing it, meaning $l^{i}(x)\leq f(x)\leq u^{i}(x)$ for all $x\in\mathcal{X}$.

Let $\mathcal{N}_{\mathcal{F}_{R}}(\epsilon)$ and $\mathcal{N}_{\mathcal{F}_{T}}(\epsilon)$ denote the $\epsilon$-bracketing numbers of $\mathcal{F}_{R}$ and $\mathcal{F}_{T}$ respectively. This measures the complexity of the function classes [Geer, ]. For instance, with linear rewards $\mathcal{F}_{R}\vcentcolon=\{R:\tau\rightarrow\theta^{T}\phi(\tau)\}$, the $\epsilon$-bracket number is bounded by $\mathcal{N}_{\mathcal{F}_{R}}(\epsilon)\leq\mathcal{O}(d\log\frac{BR}{\epsilon})$, where $\|\theta\|_{2}\leq B$ and $\|\phi(\tau)\|_{2}\leq R\>\forall\tau\in\mathcal{T}$, and $d$ is the dimension of the feature space [Zhan et al., ].

###### Definition 3.3 (Transition concentrability coefficient, Zhan et al. [ 2023a ] ) .

The concentrability coefficient w.r.t. transition classes $\mathcal{F}_{T}$ and the optimal policy $\pi^{*}$ is defined as:

$$ $C_{T}(\mathcal{F}_{T},\pi^{*})=\sup_{\tilde{T}\in\mathcal{F}_{T}}\left[\frac{ \mathbb{E}_{(s,a)\sim d^{\pi^{*}}_{T}}[|T(\cdot|s,a)-\tilde{T}(\cdot|s,a)|]}{ \sqrt{\mathbb{E}_{(s,a)\sim\mathcal{D}_{offline}}[|T(\cdot|s,a)-\tilde{T}( \cdot|s,a)|^{2}]}}\right]$ $$

The concentrability coefficient measures the coverage of the optimal policy in the offline dataset. Note that $C_{T}$ is upper-bounded by the density-ratio coefficient: $C_{T}(\mathcal{F}_{T},\pi^{*})\leq\sup_{(s,a)\in\mathcal{S}\times\mathcal{A}}{
d^{\pi^{*}}_{T}(s,a)}/{d^{\pi_{\beta}}_{T}(s,a)}$, where $\pi_{\beta}$ is the behavioural policy underlying $\mathcal{D}_{offline}$.

###### Definition 3.1 (Optimality Criterion of Offline Preference Elicitation) .

###### Assumption 3.1 (Realizability) .

###### Assumption 3.2 (Boundedness) .

###### Definition 3.2 ( ϵ italic-ϵ \epsilon italic_ϵ -bracketing number) .

###### Definition 3.3 (Transition concentrability coefficient, Zhan et al. [ 2023a ] ) .

## 4 Offline Preference-based RL with Preference Elicitation

In this section, we propose a general framework for offline preference-based reinforcement learning. The next two sections propose two different preference elicitation strategies. As learning must be carried out in two stages, with environment dynamics based on $\mathcal{D}_{offline}$ and reward learning on $\mathcal{D}_{pref}$, we adopt a model-based approach which we summarize in Algorithm 1.

Model Learning. We first leverage the offline data to learn a model of the environment dynamics, fitting a transition model $\hat{T}$ and an uncertainty function $u_{T}$ through maximum likelihood:

$$ $\displaystyle\hat{T}\quad$ $\displaystyle=\text{argmax}_{\tilde{T}\in\mathcal{F}_{T}}\mathbb{E}_{(s,a,s^{ \prime})\sim\mathcal{D}_{offline}}\left[\log\tilde{T}(s^{\prime}|s,a)\right],$ $\displaystyle u_{T}(s,a)$ $\displaystyle=\max_{\tilde{T}_{1},\tilde{T}_{2}\in\mathcal{T}}|\tilde{T}_{1}( \cdot|s,a)-\tilde{T}_{2}(\cdot|s,a)|\cdot{R_{max}},$ $$

where $\mathcal{T}=\{\tilde{T}\in\mathcal{F}_{T}\>|\>\mathbb{E}_{(s,a,s^{\prime})\sim
\mathcal{D}_{offline}}\left[\log{\hat{T}(s^{\prime}|s,a)}/{\tilde{T}(s^{\prime
}|s,a)}\right]\leq\beta_{T}\}$, defining a confidence set over the MLE estimate, and $\beta_{T}$ is a hyperparameter. In a practical implementation, this can be achieved by training an ensemble of models on different data bootstraps [Lakshminarayanan et al., ].

Figure: Algorithm 1 Offline Preference-based Reinforcement Learning with Preference Elicitation

Iterative Preference Elicitation and Reward Learning. As with the transition model, our algorithm estimates the reward function $\hat{R}$ and its uncertainty function through maximum likelihood over iteratively collected preference data $\mathcal{D}_{pref}$:

$$ $\displaystyle\hat{R}\quad$ $\displaystyle=\text{argmax}_{\tilde{R}\in\mathcal{F}_{R}}\mathbb{E}_{(\tau_{1} ,\tau_{2},o)\sim\mathcal{D}_{pref}}\left[o\log P_{\tilde{R}}(\tau_{1}\succ\tau _{2})+(1-o)\log P_{\tilde{R}}(\tau_{2}\succ\tau_{1})\right],$ $\displaystyle u_{R}(\tau)$ $\displaystyle=\max_{\tilde{R}_{1},\tilde{R}_{2}\in\mathcal{R}}|\tilde{R}_{1}( \tau)-\tilde{R}_{2}(\tau)|,$ $$

where $\mathcal{R}=\{\tilde{R}\in\mathcal{F}_{R}\>|\>\mathbb{E}_{(\tau_{1},\tau_{2},o
)\sim\mathcal{D}_{pref}}\left[\log{P_{\hat{R}}(\tau_{1}\succ\tau_{2})}/{P_{
\tilde{R}}(\tau_{1}\succ\tau_{2})}\right]\leq\beta_{R}\}$ defines the confidence set and $\beta_{R}$ is a hyperparameter.
We also define preference uncertainty between two trajectories $\tau_{1},\tau_{2}$ as $u_{P_{R}}(\tau_{1},\tau_{2})=\max_{\tilde{R}_{1},\tilde{R}_{2}\in\mathcal{R}}|
P_{\tilde{R}_{1}}(\tau_{1}\succ\tau_{2})-P_{\tilde{R}_{2}}(\tau_{1}\succ\tau_{
2})|$.

The choice of trajectory sampling strategy for preference elicitation in line 4 is critical to efficiently obtaining an $\epsilon$-optimal policy. We present two possible strategies in Sections 5 and 6. Note that by focusing on sample efficiency as in prior work on preference elicitation [Chen et al., ], we do not necessarily optimize for computational efficiency; this could be improved by collecting preferences in batches to reduce the number of reward training loops.

Pessimistic Policy Optimization. Finally, our algorithm outputs a policy $\hat{\pi}^{*}$ that is optimal while ensuring robustness to modeling error. This means optimizing for the worst-case value function over the remaining transition and reward uncertainties [Levine et al., ]:

$$ $\hat{\pi}^{*}=\text{argmax}_{\pi\in\Pi}\min_{\tilde{T}\in\mathcal{T},\tilde{R} \in\mathcal{R}}V^{\pi}_{\tilde{T},\tilde{R}}.$ $$

This analysis provides a worst-case robustness guarantee when considering well-calibrated confidence intervals, as detailed in Sections 5.1 and 6.1. For a practical implementation of our algorithm, we penalize the reward function by the uncertainty as in model-based offline RL methods [Yu et al., , Chang et al., ]. Our optimal robust policy therefore maximizes the following objective:

$$ $\hat{\pi}^{*}=\text{argmax}_{\pi\in\Pi}\mathbb{E}_{\tau\sim d^{\pi}_{\hat{T}}} [\hat{R}(\tau)-u_{R}(\tau)-u_{T}(\tau)].$ (2) $$

We show in Section A.2 that this is indeed a lower bound of the true value function. This objective allows for controlling the degree of conservatism in practice through the width of the confidence intervals used to determine $u_{R}$ and $u_{T}$.

## 5 Preference Elicitation from Offline Trajectories

A first strategy for preference elicitation without environment interaction is to sample trajectories directly from the offline dataset. Shin et al. [] propose this approach as Offline Preference-based Reward Learning (OPRL), and design a uniform and uncertainty-sampling variant:

$$ OPRL Uniform Sampling : $\displaystyle\tau_{1},\tau_{2}\sim\mathcal{D}_{offline}$ OPRL Uncertainty Sampling : $\displaystyle\tau_{1},\tau_{2}=\text{argmax}_{\tau_{1},\tau_{2}\in\mathcal{D}_ {offline}}u_{P_{R}}(\tau_{1},\tau_{2})$ $$

We provide a theoretical analysis of the performance of OPRL.

### 5.1 Theoretical Guarantees.

We obtain the following result, demonstrated in Section A.4. The suboptimality of the estimated policy $\hat{\pi}^{*}$ is bounded by the policy evaluation error for the optimal policy $\pi^{*}$. This error decomposes into a term depending on transition model estimation, and one on reward model estimation.

###### Theorem 5.1 .

For any $\delta\in(0,1]$, let $\beta_{T}=c_{T}^{\prime}\log(H\mathcal{N}_{\mathcal{F}_{T}}(1/N_{o})/\delta)/N
_{o}$ and $\beta_{R}={c_{R}^{\prime}\log(\mathcal{N}_{\mathcal{F}_{R}}(1/N_{p})/\delta)}/
{N_{p}}$, where $N_{o}=H|\mathcal{D}_{offline}|$ is the number of observed transitions in the observational dataset and $c_{T}^{\prime},c_{R}^{\prime}$ are universal constants. The policy $\hat{\pi}^{*}$ estimated by Algorithm 1, with preference elicitation based on offline trajectories, achieves the following suboptimality with probability $1-\delta$:

$$ $V^{\pi^{*}}-V^{\hat{\pi}^{*}}\leq\underbrace{HR_{max}C_{T}(\mathcal{F}_{T},\pi ^{*})\sqrt{\frac{c_{T}\log(H\mathcal{N}_{\mathcal{F}_{T}})(1/N_{o})/\delta)}{N _{o}}}}_{\text{transition term }\epsilon_{T}}+\underbrace{2\alpha\kappa C_{R}( \mathcal{F}_{R},\pi^{*})\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{R}}(1/ N_{p})/\delta)}{N_{p}}}}_{\text{reward term}},$ $$

where $\alpha=1$ for uniform sampling or $\alpha\leq 1$ for uncertainty sampling, $C_{R}$ is a concentrability measure for the reward function, $\kappa=\sup_{r\in[-R_{max},R_{max}]}\frac{1}{\sigma^{\prime}(r)}$ measures the degree of non-linearity of the sigmoid function, and $c_{T},c_{R}$ are universal constants.

In the special case where both the transition and reward functions are learned on a fixed initial preference dataset (no preference elicitation; $|\mathcal{D}_{offline}|=2N_{p}$), we recover Theorem 1 from Zhan et al. []. Importantly, parameter $\alpha$ allows us to motivate the superior efficiency of uncertainty sampling over uniform sampling, observed empirically in Shin et al. [] and in our own experiments (Section 7). Uncertainty sampling learns accurate reward models with fewer preference queries when $\alpha<1$, but can perform like uniform sampling in harder problems ($\alpha=1$).

###### Theorem 5.1 .

## 6 Preference Elicitation from Simulated Trajectories

We now propose our alternative strategy for generating trajectories for offline preference elicitation: Simulated Offline Preference-based Reward Learning (Sim-OPRL). This method simulates trajectories $(\tau_{1},\tau_{2})$ by leveraging the learned environment model. This overcomes a limitation of OPRL, which is only designed to reduce uncertainty about the reward functions in $\mathcal{R}$, by instead reducing uncertainty about which policies are plausibly optimal. Our approach is inspired by efficient online preference elicitation algorithms [Saha et al., , Chen et al., ], which we modify for practical implementation. We account for the offline nature of our problem by avoiding regions out of the distribution of the data: the sampling strategy is optimistic with respect to uncertainty in rewards, but pessimistic with respect to uncertainty in transitions.

We summarize our approach to generating simulated trajectories for preference elicitation in Algorithm 2 and refer the reader to Appendix B for practical implementation details. First, we construct a set of candidate optimal policies $\Pi_{offline}$, containing policy $\pi^{*}_{offline}$ (optimal under the pessimistic model and the true reward function) with high probability – as demonstrated in Section A.5.2. Next, within this set of candidate policies, we identify the two most exploratory policies $\pi_{1},\pi_{2}$, chosen to maximize preference uncertainty $u_{P_{R}}$. Finally, we roll out these policies within our learned transition model to generate a trajectory pair $(\tau_{1},\tau_{2})$ for preference feedback.

Figure: Algorithm 2 Preference Elicitation through Simulated Trajectory Sampling.

### 6.1 Theoretical Guarantees

We decompose suboptimality in a similar way to Section 5.1, but obtain a reward suboptimality term that depends on the learned dynamics model instead of the true one, and on $\pi^{*}_{offline}$ instead of $\pi^{*}$:

$$ $V^{\pi^{*}}-V^{\hat{\pi}^{*}}\leq\underbrace{(V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}}_ {\hat{T}_{inf},R})}_{\text{transition term }\epsilon_{T}}+\underbrace{(V^{\pi^ {*}_{offline}}_{\hat{T}_{inf},R}-V^{\pi^{*}_{offline}}_{\hat{T}_{inf},\hat{R}_ {inf}})}_{\text{reward term}}.$ (3) $$

Analysis of the suboptimality due to transition error is identical to above, but the reward term is thus significantly different. By design, our sampling strategy ensures good coverage of preferences over $\pi^{*}_{offline}$ within the learned environment model, which eliminates the concentrability term for the reward $C_{R}$. We refer the reader to Section A.5 for the proof of Theorem 6.1.

###### Theorem 6.1 .

For any $\delta\in(0,1]$, let $\beta_{T}=c_{T}^{\prime}\log(H\mathcal{N}_{\mathcal{F}_{T}}(1/N_{o})/\delta)/N
_{o}$ and $\beta_{R}={c_{R}^{\prime}\log(\mathcal{N}_{\mathcal{F}_{R}}(1/N_{p})/\delta)}/
{N_{p}}$, where $N_{o}=H|\mathcal{D}_{offline}|$ is the number of observed transitions in the observational dataset and $c_{T}^{\prime},c_{R}^{\prime}$ are universal constants. The policy $\hat{\pi}^{*}$ estimated by Algorithm 1, with a preference sampling strategy based on rollouts in the learned transition model, achieves the following suboptimality with probability $1-\delta$:

$$ $\displaystyle V^{\pi^{*}}-V^{\hat{\pi}^{*}}\leq HR_{max}C_{T}(\mathcal{F}_{T}, \pi^{*})\sqrt{\frac{c_{T}\log(H\mathcal{N}_{\mathcal{F}_{T}})(1/N_{o})/\delta) }{N_{o}}}+2\kappa\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{R}}(1/N_{p})/ \delta)}{N_{p}}}.$ $$

where $\kappa=\sup_{r\in[-R_{max},R_{max}]}\frac{1}{\sigma^{\prime}(r)}$ measures the degree of non-linearity of the sigmoid function, and $c_{T},c_{R}$ are universal constants.

###### Theorem 6.1 .

### 6.2 Discussion

Our theoretical results demonstrate that the learned policy can achieve performance comparable to the optimal policy, and thus satisfy our optimality criterion in Definition 3.1, provided it is covered by the offline data ($C_{T}(\mathcal{F}_{T},\pi^{*}),C_{R}(\mathcal{F}_{R},\pi^{*})<\infty$). Empirical results in Section 7 confirm that performance is poor when the behavioral policy is suboptimal, inducing a large $C_{T}$ or $C_{R}$.

Offline Trajectories vs. Simulated Rollouts. While both OPRL and Sim-OPRL depend on the offline dataset for estimating environment dynamics, they induce different suboptimality in modeling preference feedback. Simulated rollouts are designed to achieve good coverage of the optimal offline policy ${\pi}^{*}_{offline}$, which avoids wasting preference budget on trajectories with low rewards or high transition uncertainty. In contrast, as shown in Zhan et al. [], due to the dependence of preferences on full trajectories, the reward concentrability term $C_{R}$ in Theorem 5.1 can be very large. An advantage of sampling from the offline buffer, however, is that it is not sensitive to the quality of the model.

Transition vs. Preference Model Quality. Our theoretical analysis also suggests an interesting trade-off in the sample efficiency of our approach, depending on the accuracy of the transition model. The width of the confidence interval reduces as significance parameter $\delta$ or dataset size increase, or as function class complexity $\mathcal{N}_{\mathcal{F}_{T}}$ decreases. For a target suboptimality gap $\epsilon$, provided the optimal offline policy $\pi^{*}_{offline}$ has a gap $\epsilon_{T}<\epsilon$, then the number of preferences required is of the order of $\mathcal{O}({\log(1/\delta)}/{(\epsilon-\epsilon_{T})^{2}})$. A more accurate transition model should therefore require fewer preference samples to achieve a given suboptimality, which we again confirm empirically.

## 7 Experimental results

Figure: Figure 1: Star MDP.
Refer to caption: x1.png

We demonstrate the effectiveness of preference elicitation for offline reinforcement learning in practice and compare the different sampling strategies introduced in Sections 5 and 6: OPRL with uniform and uncertainty-sampling, and Sim-OPRL. Our experiments explore both simple MDPs and more complex decision-making environments.

Baselines. For comparison, we also propose a practical implementation of Preference-based Optimistic Planning (PbOP), an uncertainty-based preference elicitation approach over trajectory rollouts in the true environment [Chen et al., ]. Finally, we report the performance of $\pi^{*}_{offline}$ and $\pi^{*}$ as upper bounds for the performance of our algorithm: the former is trained in the learned transition model with access to the true reward, and the latter has full knowledge of both transition and reward function. We refer the reader to Appendix B for implementation details.

Star MDP. First, consider the simple tabular MDP illustrated in Figure 1, consisting of four states branching from a central state. We defer transition and reward details to Appendix C. Preferences collected over samples from the offline dataset learn slowly about the negative reward in the bottom state, as it is always included in the sampled trajectories. Instead, simulated rollouts can query a direct comparison between the optimal path and one that includes it. We thus find in Figure 2 that our preference elicitation strategy based on simulated rollouts achieves better environment returns than OPRL approaches which sample from the offline buffer, with much fewer preference queries.

This example also illustrates the importance of pessimism with respect to the transition model. Even with access to true rewards, $\pi^{*}_{offline}$ is pessimistic to avoid the out-of-distribution state, as it is unclear how to reach it. Thus, in Figure 2(b), we see a drop in performance if pessimism is not applied to the output policy (purple lines). This confirms the theoretical insights from Zhu et al. [], Zhan et al. [], who demonstrate the importance of pessimism in offline preference-based RL problems. Pessimism is also crucial in simulated rollouts, to avoid wasting preference budget on regions of low confidence — as value estimates are inaccurate in any case. This is reflected in the lower sample efficiency of model rollouts which do not apply pessimism with respect to the transition model in Figure 2(b) (brown line), and which could be seen as the naive adaptation of online preference elicitation methods to our setting [Chen et al., , Lindner et al., ]. We also note the importance of optimism with respect to the reward uncertainty, both in OPRL in Figure 2(a) and in our own model-based rollouts in Figure 2(b).

Finally, as an upper bound for the performance of our algorithm, we include baselines that have access to the environment in Figure 2(a): we report the performance of the optimal policy $\pi^{*}$, as well as that of an algorithm querying feedback over optimistic rollouts in the real environment [Chen et al., , PbOP]. Final environment returns are higher than with our method, as they do not suffer from the limited coverage of the transition model. As supported by our theoretical analysis, this result stresses the importance of having a high-quality transition model to make our method effective. We explore this in more detail in the following experiment.

Figure: (a) Comparison against baselines.
Refer to caption: x2.png

Transition vs. Preference Model Quality. Next, we empirically study the trade-off between transition and preference model performance in our problem setting. In the low-data regime, the error $\epsilon_{T}$ incurred in estimating the value function due to the misspecification of the transition model is large. As dictated by our theoretical analysis and as visualized in Figure 3(a), this significantly increases the number of preference samples $N_{p}$ required to achieve good final performance. At the other end of the spectrum, if the offline dataset is large and allows modeling the transition model accurately, then $\epsilon_{T}$ is small and the number of preference samples $N_{p}$ needed shrinks. We observe a similar trend for both Sim-OPRL and our OPRL uncertainty-sampling baseline.

We also measure how the coverage of the optimal policy affects performance in our setting. In Figure 3(b), we vary the behavioral policy $\pi_{\beta}$ underlying the offline data, ranging from optimal (density ratio coefficient $=1$) to highly suboptimal (large density ratio coefficient). The concentrability terms $C_{T}$ and $C_{R}$ are challenging to measure as they require considering entire function classes, but we report the accuracy of the maximum likelihood estimate for both models in Appendix D. We observe that preference elicitation methods perform best when the data is close to optimal (with the exception of a fully optimal, non-diverse dataset making reward learning from preferences challenging). More preference samples are required if the observational dataset has poor coverage of the optimal policy (large $C_{T}(\mathcal{F}_{T},\pi^{*})$), as the transition and reward models become less accurate for the trajectory distribution of interest.

Figure: (a) As a function of offline dataset size.
Refer to caption: x4.png

Gridworld and Sepsis Simulation. Finally, we validate our empirical findings on more complex environments detailed in Appendix C. We explore a gridworld experiment as well as a simulation of patient evolution in the intensive care unit from Oberst and Sontag [], designed to model the evolution of patients with sepsis. This medically-motivated example highlights another interesting and potentially important advantage of Sim-OPRL over OPRL, as it does not require feedback on real trajectories from the observational dataset. In a sensitive setting such as healthcare where data access is carefully controlled, it may be attractive to query experts about synthetic trajectories rather than real data samples. Sample complexity results across the different environments considered are given in Table 2, with similar conclusions: Sim-OPRL affords a higher preference sampling efficiency than OPRL baselines. For the sepsis environment, we note the number of preference samples needed to achieve our target suboptimality is very large, likely due to the sparse nature of the reward function. In a real-world application, we could potentially warm-start the reward model by leveraging proxy rewards signals in the offline data [Yu et al., ].

**Table 2: Comparison of preference sample complexity $N_{p}$ with different sampling methods, to reach a suboptimality gap of $\epsilon=20$ over normalized returns. Mean and 95% confidence interval over 20 experiments. The best-performing offline method is highlighted in bold. Note that PbOP has an advantage by having access to direct interaction with the environment.**
| Environment | OPRL Uniform | OPRL Uncertainty | Sim-OPRL (Ours) | PbOP (Online) |
| --- | --- | --- | --- | --- |
| Star MDP (Figure 1) | 32 $\pm$ 4 | 30 $\pm$ 4 | 4 $\pm$ 2 | 4 $\pm$ 2 |
| Gridworld | 105 $\pm$ 11 | 66 $\pm$ 7 | 49 $\pm$ 7 | 32 $\pm$ 4 |
| Sepsis Simulation | 18,856 $\pm$ 427 | 2,246 $\pm$ 143 | 830 $\pm$ 88 | 261 $\pm$ 59 |

## 8 Conclusion

Our work shows the potential of integrating human feedback within the framework of offline RL. We address the challenges of preference elicitation in a fully offline setup by exploring two key methods: sampling from the offline dataset [Shin et al., , OPRL] and generating model rollouts (Sim-OPRL). By employing a pessimistic approach to handle out-of-distribution data and an optimistic strategy to acquire informative preferences, Sim-OPRL balances the need for robustness and informativeness in learning an optimal policy.

We provide theoretical guarantees on the sample complexity of both approaches, demonstrating that performance depends on how well the offline data covers the optimal policy. Empirical evaluations in various environments confirm the practical effectiveness of our algorithm, as Sim-OPRL consistently outperforms OPRL baselines across different environments.

Overall, our approach not only advances the state-of-the-art in offline preference-based RL but also takes a significant step toward improving the practical utility of offline RL. This opens up new avenues for real-world applications of RL in healthcare, robotics, and manufacturing, where interaction with the environment is challenging but domain experts can be queried for feedback. Looking forward, a natural extension will be to explore alternative sources of information from experts, still without direct environment interaction.

## Appendix A Theoretical Details

This appendix provides proofs for the presented theorems and lemmas. In subsection A.1, we provide details on how we define the maximum likelihood estimators and confidence intervals of the preference and transition models. In subsection A.2 we provide the proof that our uncertainty-penalized objective in Equation 2 lower bounds the true value function and thus forms a valid pessimistic framework. In Section A.3, we show that the suboptimality of our offline preference elicitation framework is not lower-bounded by the performance of the optimal offline policy. In Section A.4, we provide our proof of theorem 5.1, analyzing the suboptimality of preferences sampled from an offline dataset. Finally, in Section A.5, we prove Theorem 6.1, which analyzes the suboptimality of preference sampling over simulated rollouts.

### A.1 Maximum Likelihood and Confidence Intervals

Let $\mathcal{F}_{g}$ denote a function class over $\mathcal{X}\rightarrow\Delta_{\mathcal{Y}}$, where $\mathcal{X},\mathcal{Y}$ are measurable sets, and $g\in\mathcal{F}_{g}$ denotes a function to be estimated.

Let $\hat{g}$ denote the maximum likelihood estimator (MLE) of $g$ based on a dataset $\mathcal{D}=\{(x^{n},y^{n})\}_{n=1}^{N}$: $\hat{g}=\text{argmax}_{\tilde{g}\in\mathcal{F}_{g}}\mathbb{E}_{(x,y)\sim
\mathcal{D}}\log(\tilde{g}(y|x))$. We construct the confidence set around the MLE as follows:

$$ $\mathcal{G}=\{\tilde{g}\in\mathcal{F}_{g}\>|\>\mathbb{E}_{(x,y)\sim\mathcal{D} }\left[\log\frac{\hat{g}(y|x)}{\tilde{g}(y|x)}\right]\leq\beta\}$ $$

###### Lemma A.1 (MLE Guarantee, Lemma 1 in Zhan et al. [ 2023a ] ) .

Let $\delta\in(0,1]$ and define the event $\mathcal{E}$ that $g\in\mathcal{G}$. If

$$ $\beta=\frac{c_{MLE}\log(\mathcal{N}_{\mathcal{F}_{g}}(1/N)/\delta)}{N},$ $$

where $c_{MLE}>0$ is a universal constant, then $P(\mathcal{E})\geq 1-\delta/2$.

###### Proof.

The proof follows that of Lemma 1 in Zhan et al. [] and uses Cramér-Chernoff’s method.

Let $\bar{\mathcal{B}}$ be a $1/N$-bracket of $\mathcal{F}_{g}$ with $|\bar{\mathcal{B}}|=\mathcal{N}_{\mathcal{F}_{g}}(1/N)$. Denote the set of all right brackets in $\bar{\mathcal{B}}$ by $\tilde{\mathcal{B}}=\{b:\exists b^{\prime}s.t.[b^{\prime},b]\in\bar{\mathcal{B
}}\}$. For $b\in\tilde{\mathcal{B}}$, we have:

$$ $\displaystyle\mathbb{E}\left[\exp\left(\sum_{n=1}^{N}\log\frac{b(y^{n}|x^{n})} {g(y^{n}|x^{n})}\right)\right]$ $\displaystyle=\prod_{n=1}^{N}\mathbb{E}\left[\exp\left(\log\frac{b(y^{n}|x^{n} )}{g(y^{n}|x^{n})}\right)\right]$ $\displaystyle=\prod_{n=1}^{N}\mathbb{E}\left[\frac{b(y^{n}|x^{n})}{g(y^{n}|x^{ n})}\right]$ $\displaystyle=\prod_{n=1}^{N}\mathbb{E}\left[\sum_{y}b(y|x^{n})\right]$ $\displaystyle\leq\left(1+1/N\right)^{N}\leq e.$ $$

as samples in $\mathcal{D}$ as i.i.d. We use the Tower property in the third step and the fact that $b$ is a $1/N$-bracket for $\mathcal{F}_{g}$ in the fourth: there exists $g^{\prime}\in\mathcal{F}_{g}$ such that $\|g(\cdot|x)-b(\cdot|x)\|_{1}\leq 1/N$ and thus $\|b(\cdot|x)\|_{1}\leq 1+1/N$, for all $x\in\mathcal{X}$.

Then by Markov’s inequality, for any $\delta\in(0,1]$, we have:

$$ $\displaystyle P\left(\sum_{n=1}^{N}\log\frac{b(y^{n}|x^{n})}{g(y^{n}|x^{n})}> \log(1\delta)\right)$ $\displaystyle\leq\mathbb{E}\left[\exp\left(\sum_{n=1}^{N}\log\frac{b(y^{n}|x^{ n})}{g(y^{n}|x^{n})}\right)\right]\cdot\exp(-\log(1/\delta))$ $\displaystyle\leq e\delta.$ $$

By union bound, we have for all $b\in\tilde{\mathcal{B}}$,

$$ $\displaystyle P\left(\sum_{n=1}^{N}\log\frac{b(y^{n}|x^{n})}{g(y^{n}|x^{n})}>c _{MLE}\log(\mathcal{N}_{\mathcal{F}_{g}}(1/N)/\delta)\right)\leq\delta/2,$ $$

where $c_{MLE}>0$ is a universal constant.

Finally, for all $\tilde{g}\in\mathcal{F}_{g}$, there exists $b\in\tilde{\mathcal{B}}$ such that $g(\cdot|x)\leq\tilde{g}(\cdot|x)$ for all $x\in\mathcal{X}$. As a result, for all $\tilde{g}\in\mathcal{F}_{g}$, we have:

$$ $\displaystyle P\left(\sum_{n=1}^{N}\log\frac{\tilde{g}(y^{n}|x^{n})}{g(y^{n}|x ^{n})}>c_{MLE}\log(\mathcal{N}_{\mathcal{F}_{g}}(1/N)/\delta)\right)\leq\delta /2.$ $$

∎

Under this event $\mathcal{E}$, we have $g\in\mathcal{G}$ with probability $1-\delta/2$. A confidence interval constructed via loglikelihood also incurs a bound on the total variation (TV) distance between $g$ and $\tilde{g}\in\mathcal{G}$:

###### Lemma A.2 (TV-distance to MLE) .

Under the event $\mathcal{E}$, we have, with probability $1-\delta$, for all $\tilde{g}\in\mathcal{G}$:

$$ $\mathbb{E}_{x\sim\mathcal{D}}\left[\|g(\cdot|x)-\tilde{g}(\cdot|x)\|_{1}^{2} \right]\leq\frac{c\log(\mathcal{N}_{\mathcal{F}_{g}}(1/N)/\delta)}{N},$ (4) $$

where $c>0$ is a universal constant.

###### Proof.

The proof follows that of Liu et al. [], Proposition 14. ∎

This guarantees that the true reward function is within an interval around the MLE estimate with high probability.

We apply these lemmas to our MLE estimates of transition and reward functions in Algorithm 1 to obtain the following guarantees.

Let $\mathcal{E}_{R}$ denote the event $R\in\mathcal{R}$ and $\mathcal{E}_{T}$ denote the event $T\in\mathcal{T}$, $\mathcal{R}$ and $\mathcal{T}$ denote the respective confidence sets around the MLE. By Lemma A.1, events $\mathcal{E}_{R}$ and $\mathcal{E}_{T}$ have probability $1-\delta/2$ if we choose $\beta_{R}={c^{\prime}_{R}\log(\mathcal{N}_{\mathcal{F}_{R}}(1/N_{p})/\delta)}/
{N_{p}}$ and $\beta_{T}=c^{\prime}_{T}\log(H\mathcal{N}_{\mathcal{F}_{T}}(1/N_{o})/\delta)/N
_{o}$, where $c^{\prime}_{R},c^{\prime}_{T}$ are universal constants.

###### Lemma A.1 (MLE Guarantee, Lemma 1 in Zhan et al. [ 2023a ] ) .

###### Proof.

###### Lemma A.2 (TV-distance to MLE) .

###### Proof.

### A.2 Model-based Pessimism and Uncertainty Penalties

###### Lemma A.3 (Telescoping Lemma) .

For any reward model ${R}\in\mathcal{F}_{R}$, and any two transition models $T,\hat{T}\in\mathcal{F}_{T}$:

$$ $\displaystyle V_{T,{R}}^{\pi}-V_{\hat{T},{R}}^{\pi}$ $\displaystyle\leq\mathbb{E}_{\tau\sim d^{\pi}_{\hat{T}}}\left[\sum_{s_{j},a_{j }\in\tau}\|T(\cdot|s_{j},a_{j})-\hat{T}(\cdot|s_{j},a_{j})\|_{1}\right]\cdot R _{max}$ $$

###### Proof.

The proof follows that of Lemma 4.1 in Yu et al. [] or Lemma 4 in Zhan et al. [].

Let $W_{j}$ be the expected return under policy $\pi$, with transition model $\hat{T}$ for the first $j$ steps, then transition model $T$ for the rest of the episode. We have:

$$ $\displaystyle V_{T,{R}}^{\pi}-V_{\hat{T},{R}}^{\pi}=\sum_{j=0}^{H-1}W_{j}-W_{j +1}.$ $$

Now,

$$ $\displaystyle W_{j}$ $\displaystyle=R_{j}+\mathbb{E}_{s_{j},a_{j}\sim\pi,\hat{T}}\left[\mathbb{E}_{s _{j+1}\sim T(\cdot|s_{j},a_{j})}[V^{\pi}_{T,R}(s_{j+1})]\right]$ $\displaystyle W_{j+1}$ $\displaystyle=R_{j}+\mathbb{E}_{s_{j},a_{j}\sim\pi,\hat{T}}\left[\mathbb{E}_{s _{j+1}\sim\hat{T}(s_{j},a_{j})}[V^{\pi}_{T,R}(s_{j+1})]\right]$ $$

where $R_{j}$ is the expected return of the first $j$ steps taken in $\hat{T}$. Therefore,

$$ $\displaystyle W_{j}-W_{j+1}$ $\displaystyle=\mathbb{E}_{s_{j},a_{j}\sim\pi,\hat{T}}\left[\mathbb{E}_{s_{j+1} \sim T(\cdot|s_{j},a_{j})}[V^{\pi}_{T,R}(s_{j+1})]-\mathbb{E}_{s_{j+1}\sim\hat {T}(s_{j},a_{j})}[V^{\pi}_{T,R}(s_{j+1})]\right]$ $\displaystyle\leq\mathbb{E}_{s_{j},a_{j}\sim\pi,\hat{T}}\left[\|T(\cdot|s_{j}, a_{j})-\hat{T}(\cdot|s_{j},a_{j})\|_{1}\cdot R_{max}\right]$ $$

under the boundedness assumption for $R$. Finally, we have:

$$ $\displaystyle V_{T,{R}}^{\pi}-V_{\hat{T},{R}}^{\pi}$ $\displaystyle=\sum_{j=0}^{H-1}W_{j}-W_{j+1}$ $\displaystyle=\sum_{j=0}^{H-1}\mathbb{E}_{s_{j},a_{j}\sim\pi,\hat{T}}\left[ \mathbb{E}_{s_{j+1}\sim T(\cdot|s_{j},a_{j})}[V^{\pi}_{T,R}(s_{j+1})]-\mathbb{ E}_{s_{j+1}\sim\hat{T}(s_{j},a_{j})}[V^{\pi}_{T,R}(s_{j+1})]\right]$ $\displaystyle\leq\sum_{j=0}^{H-1}\mathbb{E}_{s_{j},a_{j}\sim\pi,\hat{T}}\left[ \|T(\cdot|s_{j},a_{j})-\hat{T}(\cdot|s_{j},a_{j})\|_{1}\cdot R_{max}\right]$ $\displaystyle=\mathbb{E}_{\tau\sim d^{\pi}_{\hat{T}}}\left[\sum_{s_{j},a_{j} \in\tau}\|T(\cdot|s_{j},a_{j})-\hat{T}(\cdot|s_{j},a_{j})\cdot R_{max}\|_{1}\right]$ $$

∎

###### Lemma A.4 (Pessimistic Transition Model) .

Under event $\mathcal{E}_{T}$, for all $\pi\in\Pi,\tilde{R}\in\mathcal{F}_{R}$:

$$ $V_{\hat{T},\tilde{R}-u_{T}}^{\pi}\leq V_{T,\tilde{R}}^{\pi}.$ $$

###### Proof.

$$ $\displaystyle V_{T,\tilde{R}}^{\pi}$ $\displaystyle=V_{\hat{T},\tilde{R}}^{\pi}-(V_{\hat{T},\tilde{R}}^{\pi}-V_{T, \tilde{R}}^{\pi})$ $\displaystyle\geq\mathbb{E}_{\tau\sim d^{\pi}_{\hat{T}}}\left[\tilde{R}(\tau) \right]-\mathbb{E}_{\tau\sim d^{\pi}_{\hat{T}}}[u_{T}(\tau)]$ $\displaystyle=\mathbb{E}_{\tau\sim d^{\pi}_{\hat{T}}}\left[\tilde{R}(\tau)-u_{ T}(\tau)\right]$ $$

where we have used the telescoping lemma (Lemma A.3), and where $u_{T}(\tau)=\sum_{(s,a)\in\tau}u_{T}(s,a)\geq\sum_{(s,a)\in\tau}\|\hat{T}(
\cdot|s,a)-T(\cdot|s,a)\|_{1}\cdot R_{max}$ under event $\mathcal{E}_{T}$.
∎

###### Lemma A.5 (Pessimistic Reward Model) .

Under event $\mathcal{E}_{R}$, for all $\pi\in\Pi,\tilde{T}\in\mathcal{F}_{T}$:

$$ $V_{\tilde{T},\hat{R}-u_{R}}^{\pi}\leq V_{\tilde{T},R}^{\pi}.$ $$

###### Proof.

$$ $\displaystyle V_{\tilde{T},R}^{\pi}$ $\displaystyle=V_{\tilde{T},\hat{R}}^{\pi}-(V_{\tilde{T},\hat{R}}^{\pi}-V_{ \tilde{T},R}^{\pi})$ $\displaystyle=\mathbb{E}_{\tau\sim d^{\pi}_{\tilde{T}}}\left[\hat{R}(\tau) \right]-\mathbb{E}_{\tau\sim d^{\pi}_{\tilde{T}}}\left[\hat{R}(\tau)-R(\tau)\right]$ $\displaystyle\geq\mathbb{E}_{\tau\sim d^{\pi}_{\tilde{T}}}\left[\hat{R}(\tau)- u_{R}(\tau)\right]$ $$

where we have used the fact that $|\hat{R}(\tau)-R(\tau)|\leq\sum_{s,a\in\tau}|\hat{R}(s,a)-R(s,a)|=\sum_{(s,a)
\in\tau}u_{R}(s,a)=u_{R}(\tau)$ under event $\mathcal{E}_{R}$.
∎

Combining the above two lemmas gives the following result:

###### Corollary A.1 .

Under events $\mathcal{E}_{T}$ and $\mathcal{E}_{R}$, for all $\pi\in\Pi$:

$$ $V_{\hat{T},\hat{R}-u_{T}-u_{R}}^{\pi}\leq V_{T,R}^{\pi}.$ $$

This justifies the overall objective considered in our pessimistic policy optimization procedure in Section 4.

###### Lemma A.3 (Telescoping Lemma) .

###### Proof.

###### Lemma A.4 (Pessimistic Transition Model) .

###### Proof.

###### Lemma A.5 (Pessimistic Reward Model) .

###### Proof.

###### Corollary A.1 .

### A.3 Suboptimality lower bound: a counterexample

Let $\pi^{*}_{offline}=\text{argmax}_{\pi\in\Pi}\min_{\tilde{T}\in\mathcal{T}}V_{
\tilde{T},R}^{\pi}$ denote the optimal offline policy, which has access to the ground-truth reward function. In this section, we ask whether its suboptimality $\epsilon_{T}=V^{\pi^{*}}_{T,R}-V^{\pi^{*}_{offline}}_{T,R}$ is a lower bound for the suboptimality of our learned policy $\hat{\pi}^{*}$ after preference elicitation.

##### Counterexample.

Consider the MDP illustrated in Figure 4. Assume the following MLE estimate and uncertainty function for both the transition and reward models:

$$ $\displaystyle\hat{T}(s_{1}|s_{0},a_{0})=0.5;$ $\displaystyle u_{T}(s_{0},a_{0})=0.4$ $\displaystyle\hat{T}(s_{1}|s_{0},a_{1})=0.5;$ $\displaystyle u_{T}(s_{0},a_{1})=0.1$ $\displaystyle\hat{r}(s_{1})=\hat{r}(s_{2})=0.5;$ $\displaystyle u_{R}(s_{1})=u_{R}(s_{2})=0.5$ $$

Assuming access to the learned transition model and the true reward function, we pessimistically estimate the value of both actions:

$$ $\displaystyle V^{a_{0}}_{\hat{T}_{inf},R}$ $\displaystyle=0.1\cdot 1+0.9\cdot 0=0.1$ $\displaystyle V^{a_{1}}_{\hat{T}_{inf},R}$ $\displaystyle=0.6\cdot 0+0.4\cdot 1=0.4$ $$

Thus, we have: $\pi_{offline}^{*}(s_{0})=\text{argmax}_{a}V^{a}_{\hat{T}_{inf},R}=a_{1}$. The offline policy picks the suboptimal action since the worst-case returns of this action are lower than those estimated for $a_{0}$. Evaluating this policy in the real environment, we get $\epsilon_{T}=V^{\pi^{*}}_{T,R}-V^{\pi^{*}_{offline}}_{T,R}=0.6\cdot 0+0.4\cdot
1
=0.4$.

We now estimate the optimal policy in the learned transition and reward model.
Applying pessimism with respect to both models, we get an equal estimated value of 0 0 for both actions $a_{0}$ and $a_{1}$. If policy optimization converges to $\hat{\pi}^{*}=a_{0}$, we reach the suboptimality $V^{\pi^{*}}_{T,R}-V^{\hat{\pi}^{*}}_{T,R}=0.8\cdot 1+0.2\cdot 0=0.8>\epsilon_{T}$.

This example demonstrates that $\epsilon_{T}$ is not a lower bound for the suboptimality of $\hat{\pi}^{*}$, as policy $\hat{\pi}^{*}$ can achieve lower suboptimality than $\pi^{*}_{offline}$ if errors in transition and reward model estimation compensate each other.

Figure: Figure 4: Tabular MDP. The environment starts in state $s_{0}$ and has horizon $H=1$. Transition probabilities from state $s_{0}$ are given for the two binary actions $a_{0},a_{1}$ (which send the agent to the other state with complementary probability).

### A.4 Suboptimality of OPRL: Proof of Theorem 5.1

#### A.4.1 Suboptimality Decomposition

Recall that $\hat{T}_{inf},\hat{R}_{inf}=\text{argmin}_{\tilde{T}\in\mathcal{T},\tilde{R}
\in\mathcal{R}}V_{\hat{T},\hat{R}}^{\pi}$ denote the pessimistic transition and reward models, such that $\hat{\pi}^{*}=\text{argmax}_{\pi\in\Pi}V_{\hat{T}_{inf},\hat{R}_{inf}}^{\pi}$. We have:

$$ $\displaystyle V^{\pi^{*}}-V^{\hat{\pi}^{*}}$ $\displaystyle=V^{{\pi}^{*}}_{T,R}-V^{\hat{\pi}^{*}}_{T,R}$ $\displaystyle=(V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}}_{\hat{T}_{inf},\hat{R}_{inf}})- (V^{\hat{\pi}^{*}}_{T,R}-V^{{\pi}^{*}}_{\hat{T}_{inf},\hat{R}_{inf}})$ $\displaystyle\leq(V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}}_{\hat{T}_{inf},\hat{R}_{inf} })-(V^{\hat{\pi}^{*}}_{T,R}-V^{\hat{\pi}^{*}}_{\hat{T}_{inf},\hat{R}_{inf}})$ $\displaystyle\leq V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}}_{\hat{T}_{inf},\hat{R}_{inf}},$ (5) $$

where we have first used the optimality of $\hat{\pi}^{*}$ (stating that $V^{\hat{\pi}^{*}}_{\hat{T}_{inf},\hat{R}_{inf}}\geq V^{\pi}_{\hat{T}_{inf},
\hat{R}_{inf}}$, for all $\pi$) and then the pessimism principle (stating that $V^{\hat{\pi}^{*}}_{\hat{T}_{inf},\hat{R}_{inf}}\leq V^{\hat{\pi}^{*}}_{T,R}$).

Finally, we decompose the last term above as follows:

$$ $\displaystyle V^{\pi^{*}}-V^{\hat{\pi}^{*}}$ $\displaystyle\leq\underbrace{(V^{{\pi}^{*}}_{T,\hat{R}_{inf}}-V^{\pi^{*}}_{ \hat{T}_{inf},\hat{R}_{inf}})}_{\text{transition term}}+\underbrace{(V^{\pi^{* }}_{T,R}-V^{\pi^{*}}_{T,\hat{R}_{inf}})}_{\text{reward term}}$ (6) $$

We further analyze each term in the following sections.

#### A.4.2 Analysis of the transition term

In this section, we now upper bound the transition term defined in Equation 6.

###### Lemma A.6 (Lemma 4, Zhan et al. [ 2023a ] ) .

Under the event $\mathcal{E}_{T}$, with probability $1-\delta$, we have for all $\tilde{T}\in\mathcal{T}$, for all $\tilde{R}\in\mathcal{G}_{R}$, for all $\pi$:

$$ $\mathbb{E}_{d^{{\pi}}_{T}}[\tilde{R}(\tau)]-\mathbb{E}_{d^{{\pi}}_{\tilde{T}}} [\tilde{R}(\tau)]\leq HR_{max}C_{T}(\mathcal{F}_{T},\pi)\sqrt{\frac{c_{T}\log( H\mathcal{N}_{\mathcal{F}_{T}}(1/N_{o})/\delta)}{N_{o}}},$ $$

where $c_{T}>0$ is a constant.

###### Proof.

From the telescoping lemma (Lemma A.3), we have:

$$ $\displaystyle V_{T,\tilde{R}}^{\pi}-V_{\tilde{T},\tilde{R}}^{\pi}$ $\displaystyle\leq R_{max}\mathbb{E}_{\tau\sim d^{\pi}_{{T}}}\left[\sum_{s_{j}, a_{j}\in\tau}\|T(\cdot|s_{j},a_{j})-\tilde{T}(\cdot|s_{j},a_{j})\|_{1}\right]$ $\displaystyle\leq HR_{max}\mathbb{E}_{(s,a)\sim d^{\pi}_{{T}}}\left[\|T(\cdot| s,a)-\tilde{T}(\cdot|s,a)\|_{1}\right]$ $\displaystyle\leq HR_{max}C_{T}(\mathcal{F}_{T},\pi)\sqrt{\mathbb{E}_{(s,a) \sim D_{offline}}[\|T(\cdot|s,a)-\tilde{T}(\cdot|s,a)\|_{1}^{2}]}$ $$

Under event $\mathcal{E}_{T}$, by Lemma A.2, we have, with probability $1-\delta$, for all $\tilde{T}\in\mathcal{T}$:

$$ $\mathbb{E}_{(s,a)\sim D_{offline}}[\|T(\cdot|s,a)-\tilde{T}(\cdot|s,a)\|_{1}^{ 2}]\leq\frac{1}{N_{o}}c_{T}\log(H\mathcal{N}_{\mathcal{F}_{T}}(1/N_{o})/\delta)$ $$

This concludes our proof.

∎

###### Lemma A.6 (Lemma 4, Zhan et al. [ 2023a ] ) .

###### Proof.

#### A.4.3 Analysis of the reward term

Next, we upper bound the reward term defined in Equation 6.

As in Zhan et al. [], we consider the following value function: $V^{\pi}_{T,R}=\mathbb{E}_{\tau\sim d^{\pi}_{T}}[R(\tau)]-\mathbb{E}_{\tau\sim d
_{pref}}[R(\tau)]$, where $d_{pref}$ is a fixed reference trajectory distribution. This baseline subtraction, which doesn’t affect either the optimal policy or the analysis of the transition term, is needed as the approximated confidence set is based on the uncertainty in preference between two trajectories, not in the reward of a single one.

###### Definition A.1 (Preference concentrability coefficient) .

The concentrability coefficient w.r.t. reward classes $\mathcal{F}_{R}$, a target policy $\pi^{*}$ and a reference trajectory distribution $d_{pref}$ is defined as:

$$ $C_{R}(\mathcal{F}_{R},\pi^{*})=\frac{\mathbb{E}_{\tau_{1}\sim d^{\pi^{*}}_{T}, \tau_{2}\sim d_{pref}}\left[u_{P_{R}}(\tau_{1},\tau_{2})\right]}{\mathbb{E}_{ \tau_{1},\tau_{2}\sim\mathcal{D}_{offline}}\left[u_{P_{R}}(\tau_{1},\tau_{2}) \right]}$ $$

Note that, for the purpose of our analysis, our definition differs from that of Zhan et al. [] who instead consider the max ratio of difference in rewards term: $|R(\tau_{1})-R(\tau_{2})-\tilde{R}(\tau_{1})+\tilde{R}(\tau_{2})|$ over the entire function class $\mathcal{F_{R}}$.

###### Lemma A.7 .

Let trajectories for preference elicitation be sampled uniformly from the offline dataset. Under the event $\mathcal{E}_{R}$, with probability $1-\delta$, we have for all $\tilde{T}\in\mathcal{G}_{T}$, for all $\tilde{R}\in\mathcal{R}$, for all $\pi$:

$$ $V^{\pi^{*}}_{T,R}-V^{\pi^{*}}_{T,\hat{R}_{inf}}\leq 2\kappa C_{R}(\mathcal{F}_ {T},\pi)\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{R}}(1/N_{p})/\delta)}{ N_{p}}},$ $$

where $c_{R}>0$ is a constant and $\kappa=\sup_{r\in[-R_{max},R_{max}]}\frac{1}{\sigma^{\prime}(r)}$ measures the degree of non-linearity of the sigmoid function.

###### Proof.

$$ $\displaystyle V^{\pi^{*}}_{T,R}-V^{\pi^{*}}_{T,\hat{R}_{inf}}$ $\displaystyle=\mathbb{E}_{\tau\sim d^{{\pi}^{*}}_{T}}[R(\tau)]-\mathbb{E}_{ \tau\sim d_{pref}}[{R}(\tau)]-\mathbb{E}_{\tau\sim d^{{\pi}^{*}}_{T}}[\hat{R}_ {inf}(\tau)]+\mathbb{E}_{\tau\sim d_{pref}}[\hat{R}_{inf}(\tau)]$ $\displaystyle=\mathbb{E}_{\tau_{1}\sim d^{{\pi}^{*}}_{T},\tau_{2}\sim d_{pref} }[R(\tau_{1})-R(\tau_{2})]-(\hat{R}_{inf}(\tau_{1})-\hat{R}_{inf}(\tau_{2}))]$ $\displaystyle\leq\kappa\mathbb{E}_{\tau_{1}\sim d^{{\pi}^{*}}_{T},\tau_{2}\sim d _{pref}}[|P_{R}(\tau_{1}\succ\tau_{2})-P_{\hat{R}_{inf}}(\tau_{1}\succ\tau_{2} )|]$ $\displaystyle\leq\kappa\mathbb{E}_{\tau_{1}\sim d^{{\pi}^{*}}_{T},\tau_{2}\sim d _{pref}}[u_{P_{R}}(\tau_{1},\tau_{2})]$ $\displaystyle=\kappa C_{R}(\mathcal{F}_{T},\pi^{*})\mathbb{E}_{\tau_{1},\tau_{ 2}\sim\mathcal{D}_{offline}}[u_{P_{R}}(\tau_{1},\tau_{2})]$ (7) $$

where $\kappa=\sup_{r\in[-R_{max},R_{max}]}\frac{1}{\sigma^{\prime}(r)}$ measures the degree of non-linearity of the sigmoid function. In the first inequality, we have applied the mean value theorem, under Assumption 3.2. In the second inequality, we have used the definition of uncertainty function $u_{P_{R}}$ as we know $\hat{R}_{inf}\in\mathcal{R}$.

Now, under event $\mathcal{E}_{R}$, by Lemma A.2, we have, with probability $1-\delta$ for all $\tilde{R}\in\mathcal{R}$:

$$ $\mathbb{E}_{(\tau_{1},\tau_{2})\sim\mathcal{D}_{pref}}[\|P_{R}(\tau_{1}\succ \tau_{2})-P_{\tilde{R}}(\tau_{1}\succ\tau_{2})\|_{1}^{2}]\leq\frac{c_{R}\log( \mathcal{N}_{\mathcal{F}_{R}}(1/N_{p})/\delta)}{N_{p}},$ (8) $$

where $c_{R}>0$ is a constant. This implies the following upper bound for the preference uncertainty function:

$$ $\mathbb{E}_{(\tau_{1},\tau_{2})\sim\mathcal{D}_{pref}}[u_{P_{R}}(\tau_{1},\tau _{2})]\leq 2\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{g}}(1/N_{p})/ \delta)}{N_{p}}}$ (9) $$

Under uniform sampling, the distribution of preferences in $\mathcal{D}_{pref}$ is that of the offline dataset:

$$ $\mathbb{E}_{(\tau_{1},\tau_{2})\sim\mathcal{D}_{offline}}[u_{P_{R}}(\tau_{1}, \tau_{2})]=\mathbb{E}_{(\tau_{1},\tau_{2})\sim\mathcal{D}_{pref}}[u_{P_{R}}( \tau_{1},\tau_{2})]$ $$

Thus,

$$ $V^{\pi^{*}}_{T,R}-V^{\pi^{*}}_{T,\hat{R}_{inf}}\leq 2\kappa C_{R}(\mathcal{F}_ {T},\pi)\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{R}}(1/N_{p})/\delta)}{ N_{p}}}.$ $$

∎

###### Lemma A.8 .

Let trajectories for preference elicitation be sampled through uncertainty sampling from the offline dataset. Under the event $\mathcal{E}_{R}$, with probability $1-\delta$, we have for all $\tilde{T}\in\mathcal{G}_{T}$, for all $\tilde{R}\in\mathcal{R}$, for all $\pi$:

$$ $V^{\pi^{*}}_{T,R}-V^{\pi^{*}}_{T,\hat{R}_{inf}}\leq 2\alpha\kappa C_{R}( \mathcal{F}_{T},\pi)\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{R}}(1/N_{p })/\delta)}{N_{p}}},$ $$

where $c_{R}>0$ is a constant and $\alpha\leq 1$.

###### Proof.

The proof follows closely that of Lemma A.7. We introduce the preference concentrability coefficient defined for a general preference dataset:

$$ $C_{R}^{\prime}(\mathcal{F}_{R},\pi^{*})=\frac{\mathbb{E}_{\tau_{1}\sim d^{\pi^ {*}}_{T},\tau_{2}\sim d_{pref}}\left[u_{P_{R}}(\tau_{1},\tau_{2})\right]}{ \mathbb{E}_{\tau_{1},\tau_{2}\sim\mathcal{D}_{pref}}\left[u_{P_{R}}(\tau_{1}, \tau_{2})\right]}$ $$

We start from Equation 7:

$$ $\displaystyle V^{\pi^{*}}_{T,R}-V^{\pi^{*}}_{T,\hat{R}_{inf}}$ $\displaystyle\leq\kappa\mathbb{E}_{\tau_{1}\sim d^{{\pi}^{*}}_{T},\tau_{2}\sim d _{pref}}[u_{P_{R}}(\tau_{1},\tau_{2})]$ $\displaystyle=\kappa C_{R}^{\prime}(\mathcal{F}_{T},\pi^{*})\mathbb{E}_{\tau_{ 1},\tau_{2}\sim\mathcal{D}_{pref}}[u_{P_{R}}(\tau_{1},\tau_{2})]$ $\displaystyle\leq 2\kappa C_{R}^{\prime}(\mathcal{F}_{T},\pi^{*})\sqrt{\frac{c _{R}\log(\mathcal{N}_{\mathcal{F}_{g}}(1/N_{p})/\delta)}{N_{p}}}$ $$

where we have used Equation 9.

Now consider the dataset of uncertainty-sampled preferences $\mathcal{D}_{pref}$. By definition, we have:

$$ $\mathbb{E}_{\tau_{1},\tau_{2}\sim\mathcal{D}_{pref}}\left[u_{P_{R}}(\tau_{1}, \tau_{2})\right]\geq\mathbb{E}_{\tau_{1},\tau_{2}\sim\mathcal{D}_{offline}} \left[u_{P_{R}}(\tau_{1},\tau_{2})\right]$ $$

Thus, we have: $C_{R}^{\prime}(\mathcal{F}_{T},\pi^{*})\leq C_{R}(\mathcal{F}_{T},\pi^{*})$. In other words, we can write: $C_{R}^{\prime}(\mathcal{F}_{T},\pi^{*})=\alpha C_{R}(\mathcal{F}_{T},\pi^{*})$, where $\alpha\leq 1$. This concludes our proof.
∎

We now conclude the proof of Theorem 5.1 under events $\mathcal{E}_{R}$ and $\mathcal{E}_{T}$.

From Lemma A.6, we upper bound the transition term:

$$ $V^{{\pi}^{*}}_{T,\hat{R}_{inf}}-V^{\pi^{*}}_{\hat{T}_{inf},\hat{R}_{inf}}\leq HR _{max}C_{T}(\mathcal{F}_{T},\pi^{*})\sqrt{\frac{c_{T}\log(H\mathcal{N}_{ \mathcal{F}_{T}}(1/N_{o})/\delta)}{N_{o}}}$ $$

From Lemmas A.7 and A.8, we upper bound the reward term:

$$ $V^{\pi^{*}}_{T,R}-V^{\pi^{*}}_{T,\hat{R}_{inf}}\leq 2\alpha\kappa C_{R}( \mathcal{F}_{T},\pi^{*})\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{R}}(1/ N_{p})/\delta)}{N_{p}}},$ $$

where $\alpha=1$ for uniform sampling or $\alpha\leq 1$ for uncertainty sampling.

Combining with Equation 6, we obtain Theorem 5.1.

###### Definition A.1 (Preference concentrability coefficient) .

###### Lemma A.7 .

###### Proof.

###### Lemma A.8 .

###### Proof.

### A.5 Suboptimality of Sim-OPRL: Proof of Theorem 6.1

#### A.5.1 Suboptimality Decomposition

We decompose the suboptimality slightly differently to Equation 5, introducing the optimal offline policy (optimal in the pessimistic model under the true reward function): $\pi^{*}_{offline}=\text{argmax}_{\pi\in\Pi}V_{\hat{T}_{inf},R}^{\pi}$.

$$ $\displaystyle V^{\pi^{*}}-V^{\hat{\pi}^{*}}$ $\displaystyle=V^{{\pi}^{*}}_{T,R}-V^{\hat{\pi}^{*}}_{T,R}$ $\displaystyle=(V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}_{offline}}_{\hat{T}_{inf},\hat{R }_{inf}})-(V^{\hat{\pi}^{*}}_{T,R}-V^{{\pi}^{*}_{offline}}_{\hat{T}_{inf},\hat {R}_{inf}})$ $\displaystyle\leq(V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}_{offline}}_{\hat{T}_{inf}, \hat{R}_{inf}})-(V^{\hat{\pi}^{*}}_{T,R}-V^{\hat{\pi}^{*}}_{\hat{T}_{inf},\hat {R}_{inf}})$ $\displaystyle\leq V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}_{offline}}_{\hat{T}_{inf}, \hat{R}_{inf}}$ $\displaystyle=(V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}}_{\hat{T}_{inf},R})+(V^{\pi^{*}} _{\hat{T}_{inf},R}-V^{\pi^{*}_{offline}}_{\hat{T}_{inf},\hat{R}_{inf}})$ $\displaystyle\leq\underbrace{(V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}}_{\hat{T}_{inf},R })}_{\text{transition term}}+\underbrace{(V^{\pi^{*}_{offline}}_{\hat{T}_{inf} ,R}-V^{\pi^{*}_{offline}}_{\hat{T}_{inf},\hat{R}_{inf}})}_{\text{reward term}}$ (10) $$

where we have followed the same analysis as in Section A.4.1 and used the optimality of $\pi^{*}_{offline}$ in the last inequality.

The analysis of the transition term is identical to the above (Section A.4.2). We analyze the reward term next.

#### A.5.2 Analysis of the reward term

###### Lemma A.9 (Optimal Offline Policy In Set) .

Let $\Pi_{offline}$ denote the following set of near-optimal pessimistic policies, under the pessimitic transition model $\hat{T}_{inf}$ and the reward confidence set $\mathcal{R}$:

$$ $\displaystyle\Pi_{offline}=\{\pi\>|\>\pi=\text{argmax}_{\pi\in\Pi}\mathbb{E}_{ \tau\sim d^{\pi}_{\hat{T}_{inf}}}\big{[}\tilde{R}(\tau)\big{]}\>\forall\tilde{ R}\in\mathcal{R}\}$ $$

Under event $\mathcal{E}_{R}$, we have $\pi^{*}_{offline}\in\Pi_{offline}$.

###### Proof.

Recall the definition of $\pi^{*}_{offline}$: $\pi^{*}_{offline}=\text{argmax}_{\pi\in\Pi}V_{\hat{T}_{inf},R}^{\pi}$. Note that there is no need to consider the preference baseline term in $V^{\pi}$ when building $\Pi_{offline}$ since it is independent of the policy.
Under event $\mathcal{E}_{R}$, we have $R\in\mathcal{R}$. Thus, $\pi^{*}_{offline}\in\Pi_{offline}$.
∎

###### Lemma A.10 .

Under event $\mathcal{E}_{R}$, we have, with probability $1-\delta$:

$$ $V^{\pi^{*}_{offline}}_{\hat{T}_{inf},R}-V^{\pi^{*}_{offline}}_{\hat{T}_{inf}, \hat{R}_{inf}}\leq 2\sqrt{\kappa^{2}c_{R}/N_{p}\log(\mathcal{N}_{\mathcal{F}_{ R}}(1/N_{p})/\delta)}$ $$

###### Proof.

$$ $\displaystyle V^{\pi^{*}_{offline}}_{\hat{T}_{inf},R}$ $\displaystyle-V^{\pi^{*}_{offline}}_{\hat{T}_{inf},\hat{R}_{inf}}$ $\displaystyle=(V^{\pi^{*}}_{\hat{T}_{inf},R}-V^{\pi^{*}}_{\hat{T}_{inf},\hat{R }})+(V^{\pi^{*}}_{\hat{T}_{inf},\hat{R}}-V^{\pi^{*}}_{\hat{T}_{inf},\hat{R}_{ inf}})$ $\displaystyle=\mathbb{E}_{\tau\sim d^{{\pi}^{*}_{offline}}_{\hat{T}_{inf}}}[R( \tau)]-\mathbb{E}_{\tau\sim d_{pref}}[{R}(\tau)]-\mathbb{E}_{\tau\sim d^{{\pi} ^{*}_{offline}}_{\hat{T}_{inf}}}[\hat{R}_{inf}(\tau)]+\mathbb{E}_{\tau\sim d_{ pref}}[\hat{R}_{inf}(\tau)]$ $\displaystyle=\mathbb{E}_{\tau_{1}\sim d^{{\pi}^{*}_{offline}}_{\hat{T}_{inf}} ,\tau_{2}\sim d_{pref}}[R(\tau_{1})-R(\tau_{2})]-\mathbb{E}_{\tau_{1}\sim d^{{ \pi}^{*}_{offline}}_{\hat{T}_{inf}},\tau_{2}\sim d_{pref}}[\hat{R}_{inf}(\tau_ {1})-\hat{R}_{inf}(\tau_{2})]$ $\displaystyle\leq\kappa\mathbb{E}_{\tau_{1}\sim d^{{\pi}^{*}_{offline}}_{\hat{ T}_{inf}},\tau_{2}\sim d_{pref}}[P_{R}(\tau_{1}\succ\tau_{2})-P_{\hat{R}_{inf} }(\tau_{1}\succ\tau_{2})],$ $$

where $\kappa=\sup_{r\in[-R_{max},R_{max}]}\frac{1}{\sigma^{\prime}(r)}$ measures the degree of non-linearity of the sigmoid function. We have applied the mean value theorem, under Assumption 3.2.

As $R_{inf}\in\mathcal{R}$, we have: $P_{R}(\tau_{1}\succ\tau_{2})-P_{\hat{R}_{inf}}(\tau_{1}\succ\tau_{2})\leq u_{P
_{R}}(\tau_{1},\tau_{2})$.

Let $d_{pref}$ correspond to the distribution of the preference data, which consists of rollouts from exploratory policies within the learned environment model: $d_{pref}=d^{\pi_{1}}_{\hat{T}_{inf}}/2+d^{\pi_{2}}_{\hat{T}_{inf}}/2$. Recall that the near-optimal policy set $\Pi_{offline}$ includes policy $\pi^{*}_{offline}$ (Lemma A.9) and that $\pi_{1},\pi_{2}$ are the two more exploratory policies within this set:

$$ $\mathbb{E}_{\tau_{1}\sim d^{{\pi}^{*}_{offline}}_{\hat{T}},\tau_{2}\sim d_{ pref}}[u_{P_{R}}(\tau_{1},\tau_{2})]\leq\max_{\pi_{1},\pi_{2}\in\Pi_{offline}} \mathbb{E}_{\tau_{1}\sim d^{\pi_{1}}_{\hat{T},\tau_{2}\sim d^{\pi_{2}}_{\hat{T }}}}[u_{P_{R}}(\tau_{1},\tau_{2})].$ $$

Now, under event $\mathcal{E}_{R}$, by Lemma A.2, we have, with probability $1-\delta$ for all $\tilde{R}\in\mathcal{R}$:

$$ $\mathbb{E}_{(\tau_{1},\tau_{2})\sim\mathcal{D}_{pref}}[\|P_{R}(\tau_{1}\succ \tau_{2})-P_{\tilde{R}}(\tau_{1}\succ\tau_{2})\|_{1}^{2}]\leq\frac{c_{R}\log( \mathcal{N}_{\mathcal{F}_{R}}(1/N_{p})/\delta)}{N_{p}},$ $$

where $c_{R}>0$ is a constant. This implies the following upper bound for the preference uncertainty function:

$$ $\mathbb{E}_{(\tau_{1},\tau_{2})\sim\mathcal{D}_{pref}}[u_{P_{R}}(\tau_{1},\tau _{2})]\leq 2\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{g}}(1/N_{p})/ \delta)}{N_{p}}}.$ $$

Thus, we obtain:

$$ $\displaystyle V^{\pi^{*}_{offline}}_{\hat{T}_{inf},R}-V^{\pi^{*}_{offline}}_{ \hat{T}_{inf},\hat{R}_{inf}}$ $\displaystyle\leq 2\kappa\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{g}}(1 /N_{p})/\delta)}{N_{p}}}.$ $$

The resulting sample complexity of $\mathcal{O}(\frac{\kappa^{2}d}{\epsilon^{2}})$ matches that of active preference learning within a known environment [Saha et al., , Chen et al., ].

∎

We now conclude the proof of Theorem 6.1 under events $\mathcal{E}_{R}$ and $\mathcal{E}_{T}$.

From Lemma A.6, we upper bound the transition term:

$$ $V^{{\pi}^{*}}_{T,R}-V^{\pi^{*}}_{\hat{T}_{inf},R}\leq HR_{max}C_{T}(\mathcal{F }_{T},\pi^{*})\sqrt{\frac{c_{T}\log(H\mathcal{N}_{\mathcal{F}_{T}}(1/N_{o})/ \delta)}{N_{o}}}.$ $$

From Lemma A.10, we upper bound the reward term:

$$ $V^{\pi^{*}_{offline}}_{\hat{T}_{inf},R}-V^{\pi^{*}_{offline}}_{\hat{T}_{inf}, \hat{R}_{inf}}\leq 2\kappa\sqrt{\frac{c_{R}\log(\mathcal{N}_{\mathcal{F}_{R}}( 1/N_{p})/\delta)}{N_{p}}}.$ $$

Combining with Equation 10, we obtain Theorem 6.1.

###### Lemma A.9 (Optimal Offline Policy In Set) .

###### Proof.

###### Lemma A.10 .

###### Proof.

## Appendix B Implementation Details

We trained all models on two 64-core AMD processors or a single NVIDIA RTX2080Ti GPU. The total wall-clock time for running all experiments presented in this paper amounted to less than 72 hours.

##### Transition and Reward Function Training.

For all baselines, transition and reward models were implemented as linear classifiers (for the Star MDP) or as two-layer perceptions with ReLU activation and hidden layer dimension 32 (Gridworld and Sepsis environments). Training was carried out for two or one epochs for the transition and reward models respectively, with the Adam optimizer [Kingma and Ba, ] and a learning rate of $10^{-3}$.

We provide a more detailed practical algorithm for Sim-OPRL in Algorithm 3. For both our method and baselines relying on uncertainty sets (OPRL and PbOP), we estimated uncertainty sets by training models initialized with different random seeds on different bootstraps of the data (sampling 90% of the data with replacement). We consider ensembles of size $|\mathcal{T}|=|\mathcal{R}|=5$ for both transition and reward models. Hyperparameters $\lambda_{T},\lambda_{R}$ control the degree of pessimism in practice and could be considered equivalent to adjusting margin parameters $\beta_{T},\beta_{R}$ in our conceptual algorithm proposed in Section 4. Since the exact values prescribed by our theoretical analysis cannot be estimated, the user must set these parameters themselves. Hyperparameter optimization in offline RL is a challenging problem [Levine et al., ]; for our experiments, we simply set $\lambda_{T}=0.5$, $\lambda_{R}=0.1$ (StarMDP, Gridworld) and $\lambda_{T}=\lambda_{R}=1$ for the Sepsis environment.

Figure: Algorithm 3 Sim-OPRL: Practical Algorithm

##### Near-Optimal Policy Set and Exploratory Policies.

Both Sim-OPRL and PbOP require constructing a set of near-optimal policies within a learned model of the environment. Note that the PbOP algorithm in Chen et al. [] proposes to construct the near-optimal policy set by considering all policies that have a preference greater than $1/2$ over all other policies in $\Pi$, under a transition and preference uncertainty bonus. This is infeasible to estimate in practice; we modified the algorithm to allow for practical implementation. The motivation in building the set of plausibly optimal policies remains the same, but the theoretical guarantees may not hold.

We build $\Pi_{offline}$ by maintaining a policy model for all $\tilde{R}\in\mathcal{R}$, i.e., each element of the reward ensemble. Policy models are optimized to maximize returns under the transition model $\hat{T}$ and the reward function $\tilde{R}-\lambda_{T}u_{T}$ (Sim-OPRL) or $\tilde{R}+\lambda_{T}u_{T}$ (PbOP). Next, the most exploratory policies are identified by generating 10 rollouts of each of the candidate policies within the learned (SimOPRL) or true (PbOP) model. The trajectories $(\tau_{1},\tau_{2})$ maximizing the preference uncertainty function $u_{P_{R}}(\tau_{1},\tau_{2})$ are used for preference feedback. In PbOP, the trajectories are then added to the trajectories buffer and the transition model is retrained for 20 (Star MDP, Gridworld) or 200 steps (Sepsis).

##### Preference Feedback Collection.

Preference labels are provided through the ground-truth reward function associated with every environment. As stated in Section 4, for computational efficiency, we sample preferences in batches of 4 (Star MDP, Gridworld) or 100 (Sepsis) to reduce the number of model updates needed.

##### Policy Optimization.

Policy optimization stages, both in estimating optimal policy sets in Sim-OPRL and PbOP and in outputting final policies, are carried out exactly through linear programming for the Star MDP and Gridworld using cvxopt [Diamond and Boyd, ], based on code from Lindner et al. [], and using Proximal Policy Optimization [Schulman et al., ] implemented in stable-baselines3 [Raffin et al., ] for the Sepsis environment. In the latter case, after every preference collection episode, reward and policy models were trained from the checkpoint of the previous iteration, for only 20 steps to minimize computation.

##### Baselines and Ablations.

We implement both OPRL baselines within our model-based offline preference-based algorithm described in Section 4. Uncertainty sampling is taking the pair with maximum preference uncertainty over 45 pairs for every sample, to reduce the load of computing preference uncertainty over the entire trajectory buffer.

Our ablation study for Figure 2(b) is conducted as follows. For Sim-OPRL without pessimism in the output policy, we output the policy that maximizes the value function under the MLE estimate of the transition and reward function, $\hat{T}$ and $\hat{R}$, after preference acquisition. For Sim-OPRL without pessimism in the simulated rollouts, we estimate the optimal policy set $\Pi_{offline}$ in the MLE estimate of the transition model instead of its pessimistic counterpart. Finally, for Sim-OPRL without optimism in the simulated rollouts, we generate rollouts from any two policies in $\Pi_{offline}$ instead of the most explorative ones.

## Appendix C Environment Details

Figure: Figure 5: Star MDP illustrated in Figure 1. Transition probabilities are 0.9 for all solid arrows. Omitted actions or complementary transitions keep the state unchanged.

##### Star MDP.

We illustrate the transition dynamics underlying the Star MDP in Figure 5. Transition probabilities are 0.9 for all depicted solid arrows, and leave the state unchanged otherwise. Other actions also keep the state unchanged with probability 1. Episodes have length $H=3$ and start from $s_{0}$. Unless specified otherwise, the offline dataset $\mathcal{D}_{offline}$ consists of 40 trajectories which only cover states $(s_{0},s_{1},s_{3})$ and $(s_{3},s_{1},s_{2})$.

Figure: Figure 6: Gridworld environment. Rewards at every state are indicated if non-zero. Transition probabilities are 0.9. Thick lines indicate an obstacle, through which state transitions have probability zero.

##### Gridworld.

We illustrate the gridworld environment in Figure 6. The environment consists of a $4\times 4$ grid with states associated with different rewards, including a negative-reward region in the top-right corner, a high-reward but unreachable state, and a moderate-reward state at the bottom right corner. Each episode starts in the top-left corner. Transition probabilities for each of the four actions (top, left, bottom, right) are 0.9 for the intended direction, and 0.1 for the others; and action stay remains in the current state with probability 1. Transitions beyond the grid limits or through obstacles have probability zero, with the remainder of the probability mass for each action being distributed amongst other directions equally. The offline dataset contains 150 episodes and the behavioral policy is $\epsilon$-optimal with noise $\epsilon=0.1$. Episodes have length $H=10$.

##### Sepsis Simulation.

The sepsis simulator [Oberst and Sontag, ] is a commonly used environment for medically-motivated RL work [Tang and Wiens, ]. We use the original authors’ publicly available code: [https://github.com/clinicalml/gumbel-max-scm/tree/sim-v2/sepsisSimDiabetes](https://github.com/clinicalml/gumbel-max-scm/tree/sim-v2/sepsisSimDiabetes) (MIT license). The state space consists of five discrete observational variables (heart rate, blood pressure, oxygen concentration, glucose, diabetes status) and the action space consists of three different binary treatment options (antibiotic administration, vasopressor administration, mechanical ventilation). The probability that each treatment affects the value of each vital sign is determined by Oberst and Sontag [] to reflect patients’ physiology. The ground truth reward function is sparse and only assigns a positive reward of $+1$ to surviving patients and a negative reward of $-1$ if death occurs (3 or more abnormal vitals) during their stay. The offline trajectories dataset includes 10,000 episodes following an $\epsilon$-optimal policy with noise $\epsilon=0.1$ and the episode length is $H=20$.

## Appendix D Additional Results

We include additional results in this section.

Figure: (a) As a function of offline dataset size.
Refer to caption: x6.png

In Figure 7, we report the accuracy of the transition and preference model achieved for the Star MDP as we vary the size of optimality of the offline dataset. Accuracy is measured against all possible state transitions and over 100 pairs of random trajectories (random combinations of the 5 states and 4 actions in a sequence of $H=3$). This complements our analysis in Sections 7 and 3. We see a steady improvement in both transition and reward model quality as we increase the amount of observational data in Figure 7(a), which explains the observed dependence of $N_{p}$ on $N_{o}$ in Figure 3(a).

In Figure 7(b), we notice low model performance at both extremes of the x-axis. When the dataset is fully optimal, we find that all trajectories involve the same sequence of actions and states, so learning a transition or reward model from this data is challenging. We reach a similar conclusion at the other end of the spectrum at high density ratios, where the coverage the optimal states reduces. We reach highest performance for both models at intermediate values, when diversity of the observational data is high.

Still, it is important to stress that the highest accuracy of both models does not necessarily translate to the best-performing policy: good performance on the distribution induced by the optimal policy is more important, as formalized by the concentrability coefficients.

Next, we plot performance as a function of preferences sampled for our two additional environments in Figure 8. We reach similar conclusion to those drawn from the Star MDP in Section 7: within the offline preference elicitation approaches, OPRL with uniform sampling is the least efficient, OPRL with uncertainty sampling performs better, and Sim-OPRL even better. The PbOP method naturally reaches a superior policy with fewer samples as it allows environment interaction and can thus improve its estimate of the transition model in parallel to learning the preference function.

Figure: (a) Gridworld.
Refer to caption: x8.png

## Appendix E Broader Impact

Better preference elicitation strategies for offline reinforcement learning have the potential to facilitate and improve decision-making in real-world safety-critical domains like healthcare or economics, by reducing reliance on direct environment interaction and reducing human effort in providing feedback. Potential downsides could include the amplification of biases in the offline data, potentially leading to suboptimal or unfair policies. Thorough evaluation is therefore crucial to mitigate this before deploying models in such real-world applications. In addition, human preferences may not be fully captured by binary comparisons. As noted in our conclusion, we hope that future work will explore richer feedback mechanisms to better model complex decision-making objectives.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Based on your theoretical analysis, could you discuss how you expect the performance will change on dataset of varying optimality?
2. Could you present experiment results on other dataset for the Cheetah environment, such as medium, medium-expert and expert, to support your discussion?
soundness: 3
strengths: The idea of using simulated rollouts in preference queries is a natural but unexplored idea in the literature of PbRL. One strength of this paper is that, the authors show the effectiveness in terms of sample complexity both theoretically and empirically.
summary: This paper studies preference-based reinforcement learning (PbRL) in offline setting, in which the agent utilizes a fixed trajectory dataset for policy learning and can query humans for preference feedback. In particular, the authors propose to sample preference queries by rolling out trajectory data using learned models of MDPs. The authors provides theoretical guarantees for the sample complexity of their proposed strategy and verify it on simple control tasks.
weaknesses: My concern is about the quality of learned policies. While I agree with the optimality criterion mentioned in 3.2, I think to ensure the practical value of the proposed strategy, it is important to include evaluations for offline dataset of varying optimality. This is because for high-dimensional tasks, under a fixed budget of offline trajectories, the coverage over state-action space and the optimality of the behavior policy, can be conflicting objectives. The state-action space is less covered by good behavior policies, yet this reduced coverage can raise concerns on learned transition model. See detailed question below.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: Strengths:
1. This paper provides a good theoretical analysis of preference elicitation with the offline datasets. It bounds the value difference between the optimal policy under the estimated transition model and the true optimal policy. Such bounds are achieved by decomposing the loss from the model estimation and the reward estimation.
2. Experiments show the proposed methods outperform other algorithms in several environments.
3. This paper conducted an ablation study to show the importance of pessimistic with respect to the transition model.
summary: This paper uses the offline dataset to learn the environment model. They do not assume they have access to the reward in the offline data set. Such offline datasets contribute to the overall learning by providing an estimation of the transition probability. This paper provides a theoretical analysis of reinforcement learning with offline datasets to achieve preference elicitation. The experiments show their algorithms outperform other algorithms in several environments. They also conducted an ablation test to show the importance of pessimistic with respect to the transition model.
weaknesses: Weaknesses:

1. The experiment environments are relatively simple. The grid world is quite small. It is interesting to try to extend this to more challenging reinforcement learning benchmarks.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - I do not quite understand “An advantage of sampling from the offline buffer, however, is that it is not sensitive to the quality of the model” in L346. What does “the model” refer to?
- Should $N_T$ in the second equation in L369 be $N_R$?
soundness: 3
strengths: - This paper focuses on the preference elicitation problem on offline RL, which attracts wide attention recently from many fields (such as RLHF for LLMs).
- This paper has theoretical results on the proposed algorithm with some high-level insights (e.g., pessimism for dynamics and optimism for reward modeling).
- This paper has practical algorithm designs and good empirical results.
summary: The paper presents Sim-OPRL, an offline preference-based reinforcement learning algorithm that addresses the challenge of acquiring preference feedback in a fully offline setup. It leverages a learned environment model to elicit preference feedback on simulated rollouts, balancing conservatism and exploration. The main idea is to employ a pessimistic estimation for the transition dynamics (based on the offline dataset) for the OOD issue, and use an optimistic estimation for the reward model (based on the preference elicitation data). The benefit of using simulated rollouts is to avoid wasting preference budget on trajectories with low rewards. The authors provide theoretical guarantees on sample complexity and demonstrate the empirical performance of a practical version of Sim-OPRL across various environments, showing its superiority over previous baseline methods (OPRL and PbOP).
weaknesses: - **Complexity of Implementation:** The algorithm's reliance on learning several accurate dynamics model might be challenging in practice, especially if the model fails to capture the true dynamics. Moreover, Sim-OPRL requires the trajectory rollouts using the dynamics model and the error may accumulate, which poses higher requirements for the dynamics model. Do the authors have any idea on how to design practical algorithms with less computational overhead (e.g., estimating multiple models) and on more complex environments (e.g., when it is hard to learn an accurate dynamics model).
- **Lack of study on the dependence on quality of offline data and feedback:** The performance of Sim-OPRL may be heavily dependent on the quality and coverage of the offline dataset. For the experiments in on the tasks listed in Table 2, how are the offline datasets are collected? Are they expert datasets (so the concentrability coefficients are small)? How the feedback is generated in the experiments? How would the algorithm perform when we vary the feedback quality?
- Minor: What is ``\hat{R}_\text{inf}``? I can guess it is pessimistic reward, but ``\hat{R}_\text{inf}`` and ``\hat{T}_\text{inf}`` are not formally defined.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. How does the complexity of the reward function impact the performance of Sim-OPRL? Have you (or do you plan to) test the algorithm with environments that are characterized by more complex, multi-objective, or non-linear reward functions? If the method is agnostic to the reward function (aside from sparsity) it would help to show that as well.
2. Can you provide more details on the sensitivity of Sim-OPRL to its hyperparameters, such as the pessimism and optimism parameters? How do you recommend tuning these parameters in practice? It may be insightful to include ablation testing in the appendix that demonstrates the sensitivity (or robustness) to hyperparameter selection, especially as this could drastically affect the real-world viability of the algorithm. 
3. Are there any other algorithms that would serve as a effective and informative baseline for Sim-OPRL? If not, would it be possible to run experiments that demonstrate learning performance on naive methods?
4. Could you please clarify the rationale behind limiting the experiments to the selected datasets and environments? Are there specific challenges that restrict the application of the method to a broader range of environments and dataset combinations? If there are no such constraints, additional experimental results would be valuable. Conversely, if limitations do exist, it would be beneficial to outline what they are, the reasons for their existence, and why they do not compromise the method's overall effectiveness and practical utility.
5. Generally speaking could the authors please explain the motivations for the setting further? Specifically, would it be practical to compare the results of Sim-OPRL to running standard offline RL algorithms (CQL, IQL, TD3_BC etc.) on the offline dataset directly? If not, why not?
soundness: 4
strengths: 1. The authors provide strong theoretical guarantees on the sample complexity of their approach, ensuring that the algorithm is both efficient and reliable. Additionally, the empirical results across various environments demonstrate the practical effectiveness and scalability of Sim-OPRL, showing it outperforms existing methods in terms of sample efficiency and policy performance.
2. Sim-OPRL incorporates a pessimistic approach to handle out-of-distribution data, ensuring robustness to model uncertainty. This is particularly important in offline settings where the data may not cover the entire state-action space. Being robust to OOD data makes the algorithm far more applicable to ‘real-world’ problems/settings.
3. The paper makes a compelling case due to their incorporation of theoretical and empirical evidence. To back up their theoretical insights, they conduct extensive experiments across two different environments. This provided empirical data confirms the practical applicability and robustness of Sim-OPRL, illustrating its effectiveness in scenarios where direct environment interaction is not feasible.
4. The attached code is well-written and largely self-documenting, with a clear and logical internal structure. This design not only facilitates ease of use for other users looking to implement the Sim-OPRL algorithm but also made the process of reviewing the practical implementation and validating the experiments straightforward and efficient. This made the review process much easier.
summary: The paper addresses the challenge of applying RL to real-world scenarios where direct interaction with the environment is impractical or unsafe. Traditional learning methods require environment interactions, which can be risky in certain fields (like healthcare applications). The paper proposes an algorithm called Sim-OPRL, an offline PBRL learning algorithm that learns from preferences without needing online interaction. This algorithm uses a learned environment model to simulate rollouts and gather preference feedback, balancing pessimism for out-of-distribution data and optimism for acquiring informative preferences. The paper formalizes the problem of preference elicitation in offline RL, proposes a novel algorithm, and provides theoretical guarantees on its sample complexity.

The paper also demonstrates the effectiveness of Sim-OPRL through empirical validation in various environments, including a gridworld and a sepsis simulation. The results show that Sim-OPRL outperforms an existing baseline algorithm (OPRL) in terms of sample efficiency and policy performance. The paper shows that by leveraging simulated rollouts, their algorithm efficiently learns the optimal policy while minimizing the number of human queries required.
weaknesses: 1. The paper’s empirical section does not properly consider different baseline algorithms to compare theirs with. The only algorithm that the authors use as a baseline is OPRL. This severely limits the ability to fully assess the relative performance and advantages of Sim-OPRL. To rectify this, The authors should consider including a wider array of offline PBRL algorithms/frameworks in their experiments.
2. The paper demonstrates promising results in the demonstrated environments, but it lacks validation in more complex and realistic settings. To strengthen the evidence of the algorithm’s practical applicability, the authors should evaluate Sim-OPRL on several different datasets. One example could be MuJoCo style datasets. Other relevant papers in the field construct preference datasets from the D4RL offline benchmark. These datasets provide a more challenging and ‘closer to real world’ testbed. Evaluation on such environments (in conjunction with adding more baseline algorithms) could result in a better assessment of the algorithm’s robustness, scalability, and generalizability.
3. The paper demonstrates the algorithm’s performance in relatively small-scale environments. Empirically, it does not seem to address scalability to larger, more complex environments. Due to the smaller scale test environments (Gridworld & Sepsis), the the actual scalability of the algorithm (particularly in real-world deployments outside of benchmarks) remains unclear. 
4. As the authors state, for the sepsis environment, the requisite number of preference samples is rather large, due to the sparse reward function. This seems like an inherent limitation, which they posit could be solved by warm-starting the reward model. It would be interesting to see this data and how it affects performance. If a sparse reward function is a true limitation of the Sim-OPRL method, the authors should show more experiments demonstrating that this can be 'worked around' by performing warm starts. This could also help to further justify the real world applicability of the algorithm.

---

confidence: 2
contribution: 2
initial_rating: 8
presentation: 3
questions: /
soundness: 3
strengths: 1. They delve into very interesting setups: offline RL with preference feedback.

2. Their theoretical results are solid and show he advantage of their proposed preference elicitation algorithm over prior methods.

3. They propose a practical algorithm for implementation and extensive experiments show that their method outperform prior methods in several environment.
summary: This paper delves into offline reinforcement learning from preference feedback and proposes an offline preference elicitation method to simulate trajectories from the learned environment model instead of sampling trajectories directly from the offline dataset. They provide theoretical justification for the previous RL with preference feedback method and show that their proposed method can effectively reduce the sample complexity upper bound. They also propose an empirical algorithm and show it can outperform prior methods and achieve SOTA on offline PbRL setups without access to the ground truth rewarded. They finally iid ablation studies to show the importance of incorporating the principle of pessimism.
weaknesses: I do not see any big issues.

Meta-review: This paper studies preference-based reinforcement learning (PbRL) in offline setting, a relatively unexplored area.
The authors show the sample complexity effectiveness of their approach.
The main contribution of the paper is theoretical; however, they derive a practical algorithm that works well in practice.
Perhaps the major weakness is that empirical experiments concerns environments that are quite simple.

Decision: Accept (Poster)