Hindsight Preference Learning for Offline Preference-based Reinforcement Learning

## Abstract

Abstract Offline preference-based reinforcement learning (RL), which focuses on optimizing policies using human preferences between pairs of trajectory segments selected from an offline dataset, has emerged as a practical avenue for RL applications. Existing works rely on extracting step-wise reward signals from trajectory-wise preference annotations, assuming that preferences correlate with the cumulative Markovian rewards. However, such methods fail to capture the holistic perspective of data annotation: Humans often assess the desirability of a sequence of actions by considering the overall outcome rather than the immediate rewards. To address this challenge, we propose to model human preferences using rewards conditioned on future outcomes of the trajectory segments, i.e. the hindsight information . For downstream RL optimization, the reward of each step is calculated by marginalizing over possible future outcomes, the distribution of which is approximated by a variational auto-encoder trained using the offline dataset. Our proposed method, Hindsight Preference Learning (HPL) , can facilitate credit assignment by taking full advantage of vast trajectory data available in massive unlabeled datasets. Comprehensive empirical studies demonstrate the benefits of HPL in delivering robust and advantageous rewards across various domains. Our code is publicly released at https://github.com/typoverflow/WiseRL .

## 1 Introduction

Although reinforcement learning (RL) has demonstrated remarkable success in various sequential decision-making tasks (Vinyals et al., ; Ye et al., ), its application in real-world scenarios remains challenging for practitioners, primarily due to two key reasons. First, modern RL methods typically require millions of online interactions with the environment (), which is prohibitively expensive and potentially dangerous in embodied applications (Levine et al., ). Second, reward engineering is necessary to align the induced behavior of the policy with human interests (Gupta et al., ; Knox et al., ).
However, tweaking the reward requires substantial efforts and extensive task knowledge of the real-world scenarios.
Reward hacking frequently occurs when the reward is improperly configured, leading to unintended consequences (Skalse et al., ).

There are several research directions aiming for addressing above challenges (Knox and Stone, ; Ng and Russell, ; Sadigh et al., ), among which *offline Preference-based RL (offline PbRL)* has gained increasing attention recently (Hejna and Sadigh, ; An et al., ; Kang et al., ).
In offline PbRL, an offline dataset is collected by deploying a behavior policy, after which human labelers are required to provide relative preferences between two trajectory segments selected from the offline dataset. Offline PbRL significantly reduces the burden on human effort given that labeling preferences between trajectories is considerably easier compared to crafting step-wise reward signals.
It has proven effective in large-scale applications including fine-tuning large language models (Ouyang et al., ; Touvron et al., ; Rafailov et al., ).

Figure: Figure 1: Illustration of the reward learning procedure in HPL. Unlike previous methods, HPL first generates embeddings $z_{t}$ to encode the future part of the segments and optimize a reward function $r_{\psi}$ which is conditioned on the $s_{t}$, $a_{t}$ and the future $z_{t}$ using the Bradley-Terry model.
Refer to caption: x1.png

Offline PbRL typically follows a two-phase paradigm: 1) learning a reward function that aligns with human preferences using a small labeled preference dataset; 2) applying the reward function to label a massive unlabeled dataset and performing policy optimization (Christiano et al., ).
For the first phase, existing methods employ Bradley-Terry model (Bradley and Terry, ) to learn *step-wise* reward signals from *trajectory-wise* preferences based on the *Markovian reward assumption*: the preference correlates with the cumulative rewards of each trajectory (Christiano et al., ).
However, as previous works (Kim et al., ) have unveiled, such an assumption is technically flawed and limited since humans evaluate the trajectory segments from a global perspective, making the obtained reward bear an implicit dependence on the future part of the segment. Consider the case of purchasing lottery tickets as an example. Although the expected return is negative, the payoff can be substantial if one wins. However, when assigning credits, we should allocate higher rewards to purchasing tickets only when we are certain of winning, rather than unconditionally encouraging such behavior.

In light of this, we propose *Hindsight Preference Learning (HPL)* to account for such dependence on future information.
The key idea of HPL is to develop a *hindsight preference model*, which models human preferences using a reward function conditioning on the state $s$, action $a$ and the future trajectory starting from $(s,a)$, i.e. the hindsight information.
In particular, given a $H$-length trajectory $\sigma_{1:H}=(s_{1},a_{1},s_{2},a_{2},\dots,s_{H},a_{H})$, the reward of $s_{t},a_{t}$ for $1\leq t\leq H$ is given by $r(s_{t},a_{t}|\sigma_{t:t+k})$. When labeling the unlabeled dataset, a scalar reward signal is computed for each state-action pair by marginalizing over all possible hindsight information, $r(s,a)=\int_{\sigma}p(\sigma|s,a)r(s,a|\sigma)d\sigma$.
To deal with the high-dimensional nature of hindsight information, we pre-train a variational auto-encoder to efficiently represent the hindsight information, making the above marginalization feasible in practice. The reward learning procedure of HPL is illustrated in Figure 1.

HPL has two key benefits over prior works. First, by leveraging hindsight information in preference modeling, it captures the implicit holistic perspective of human preference labeling, addressing the key limitation of the Markovian reward assumption adopted in previous works.
Second, the two-phase paradigm of offline PbRL might become less effective if there is a substantial distribution mismatch between preference and unlabeled dataset. HPL can take advantage of the unlabeled dataset by learning a prior over future outcomes. This allows for incorporating the information of trajectory distribution carried by the unlabeled dataset, thus delivering robust and advantageous rewards when labeling the unlabeled dataset.
We provide extensive evaluations to verify these benefits.

## 2 Preliminaries

### 2.1 Markov Decision Process

In standard RL, an agent interacts with an environment characterized by a Markov Decision Process (MDP) $\langle\mathcal{S},\mathcal{A},r,T,\gamma\rangle$. Here, $\mathcal{S}$ and $\mathcal{A}$ represent the state space and action space respectively, while $r(s,a)$ denotes the reward function, $T(s^{\prime}|s,a)$ is the transition function, and $\gamma$ is the discounting factor.
The value function defines the expected cumulative reward by following a policy $\pi(a|s)$,

$$ $\displaystyle V^{\pi}(s)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t}, a_{t})|s_{0}=s\right]\,.$ (1) $$

The primary objective of RL algorithms is to find an optimal policy that maximizes $\mathbb{E}_{s_{0}\sim\mu_{0}}[V^{\pi}(s_{0})]$, where $\mu_{0}$ is the initial state distribution.

### 2.2 Offline Preference-Based Reinforcement Learning

In this work, we consider the problem of learning an optimal decision-making policy from a previously collected dataset with *preference feedback*.
In its generalist framework, the ground-truth reward is not given in the data. Instead, the learner is provided with a *preference dataset* and a massive *unlabeled dataset*, and follows a two-phase paradigm: 1) *reward learning*, learn a reward model $r$ with the preference data;
and
2) *reward labeling*, apply $r$ to label the unlabeled dataset in order to perform policy optimization with a large amount of data.

Let $\sigma=(s_{0},a_{0},s_{1},a_{1},\dots,s_{|\sigma|},a_{|\sigma|})$ denote a consecutive segment of states and actions from a trajectory.
The preference dataset $\mathcal{D}_{\rm p}=\{(\sigma_{i}^{0},\sigma_{i}^{1},y_{i})\}_{i=1}^{|\mathcal
{D}_{\rm p}|}$ consists of segments pairs with preference label given by the human annotators.
The preference label is given by: $y_{i}=1$ if $\sigma^{0}\succ\sigma^{1}$ and $y_{i}=0$ if $\sigma^{1}\succ\sigma^{0}$,
where we use $\sigma\succ\sigma^{\prime}$ to denote $\sigma$ is more preferred than $\sigma^{\prime}$.
We assume that all segments have the same length $|\sigma|=H$.
The unlabeled dataset $\mathcal{D}_{\rm u}$, contains reward-free trajectories $\{\sigma_{i}\}_{i=1}^{|\mathcal{D}_{\rm u}|}$ collected by some behavior policy $\pi_{\beta}$. In practice, we usually have $|\mathcal{D}_{\rm u}|\gg|\mathcal{D}_{\rm p}|$ as collecting human annotations is more time-consuming and expensive compared to collecting unlabeled trajectories.

To learn a reward function $r$, a common approach is to assume a probabilistic preference model and maximize the likelihood of the preference dataset,

$$ $\displaystyle\mathcal{L}(\psi)=-\sum_{(\sigma^{0},\sigma^{1},y)\in\mathcal{D}_ {\rm p}}(1-y)\log P(\sigma^{0}\succ\sigma^{1};\psi)+yP(\sigma^{1}\succ\sigma^{ 0};\psi)\,,$ (2) $$

where $P(\sigma\succ\sigma^{\prime};\psi)$ is the preference model parameterized by the parameters $\psi$.
For the probabilistic preference model, most existing methods adopt the Markovian reward assumption (Christiano et al., ; Shin et al., ; Hu et al., ):

$$ $\displaystyle\rho_{\rm MR}(\sigma;\psi)=\sum_{(s,a)\in\sigma}r_{\psi}(s,a)\,.$ (3) $$

That is, the *preference strength* of a segment $\sigma$ correlates with its cumulative rewards.
Applying the Bradley-Terry model (Bradley and Terry, ) leads to the following *Markovian Reward (MR)* preference model,

$$ $\displaystyle P_{\rm MR}(\sigma^{0}\succ\sigma^{1};\psi)=\frac{\exp(\rho_{\rm{ MR}}(\sigma^{0};\psi))}{\exp(\rho_{\rm{MR}}(\sigma^{0};\psi))+\exp(\rho_{\rm{ MR}}(\sigma^{1};\psi))}\,.$ (4) $$

Plugging (4) into (2) yields a practical learning objective for learning the reward function.
Finally, we can label $\mathcal{D}_{\rm u}$ with the learned reward model for any $(s,a)\in\sigma,\sigma\in\mathcal{D}_{\rm u}$.
The resulting labeled dataset can be used for policy optimization with any offline RL algorithms, such as IQL (Kostrikov et al., ) and AWAC (Nair et al., ).

## 3 Hindsight Preference Learning

In this section we introduce a new preference model designed to address the limitations of the MR preference model by utilizing hindsight information.
We begin with an illustrative example that serves as the primary motivation for our approach, followed by a detailed explanation of the formalization and implementation of the proposed method.

### 3.1 Motivating Example: The Influence of the Future

To elucidate the influence of the future in preference modeling, we take the gambling MDP from Yang et al. () as an example (Figure 2).
An agent begins at $s_{1}$ with two actions: $a_{1}$ and $a_{2}$. Choosing the high-risk action $a_{1}$, the agent transitions to a rewarding state $s_{\rm good}$ with probability of $10\%$, but is more likely ($90\%$) to a penalizing state $s_{\rm bad}$.
Alternatively, the safer and actually optimal action $a_{2}$ consistently leads to a neutral state $s_{\rm avg}$, yielding a reward of 0 0.
Suppose we are to extract the reward function using the provided dataset, where preferences are labeled based on the ground-truth reward:

$$ $\displaystyle\mathcal{D}=\left\{\begin{aligned} &((s_{1}\rightarrow a_{1} \rightarrow s_{\rm good}\rightarrow a_{3}),&(s_{1}\rightarrow a_{2}\rightarrow s _{\rm avg}\rightarrow a_{3}),\quad&y=0)\\ &((s_{1}\rightarrow a_{1}\rightarrow s_{\rm good}\rightarrow a_{3}),&(s_{1} \rightarrow a_{2}\rightarrow s_{\rm avg}\rightarrow a_{3}),\quad&y=0)\\ &((s_{1}\rightarrow a_{1}\rightarrow s_{\rm good}\rightarrow a_{3}),&(s_{1} \rightarrow a_{1}\rightarrow s_{\rm bad}\rightarrow a_{3}),\quad&y=0)\\ &((s_{1}\rightarrow a_{1}\rightarrow s_{\rm bad}\rightarrow a_{3}),&(s_{1} \rightarrow a_{2}\rightarrow s_{\rm avg}\rightarrow a_{3}),\quad&y=1)\\ \end{aligned}\right\}.$ $$

Applying the MR preference model (4) to this dataset would likely yield a reward function where $r_{\psi}(s_{1},a_{1})>r_{\psi}(s_{1},a_{2})$,
because a larger proportion of trajectories involving $a_{1}$ lead to the rewarding state $s_{\rm good}$ (our experiments also validate this in Figure 3). However, this violates rationality as selecting $a_{2}$ offers a higher return in expectation.
This failure can be attributed to the inappropriate credit assignment inherent in the MR preference model (4):
a preference for $(s_{1}\rightarrow a_{1}\rightarrow s_{\rm good}\rightarrow a_{3})$ will assign credits to both $r_{\psi}(s_{1},a_{1})$ and $r_{\psi}(s_{\rm good},a_{3})$ equally, leading to over-estimated utilities for $a_{1}$. To address this issue, a natural approach is to condition the reward of $(s_{1},a_{1})$ on the future outcome of the segment (i.e. $s_{\rm good}$ or $s_{\rm bad}$), so that $a_{1}$ is encouraged only when it leads to a favorable outcome $s_{\rm good}$. The conditional reward function $r_{\psi}(s,a|\sigma_{\rm future})$ thus answers the critical question: If my future is determined to be $\sigma_{\rm future}$, how advantageous it is for me to choose action $a$ at $s$?

Figure: Figure 2: A gambling MDP that illustrates the potential failure modes of the MR preference model.
Refer to caption: x2.png

When applying $r_{\psi}(s,a|\sigma_{\rm future})$ to label data, we can marginalize over all possible future segments according to some prior distribution $p_{\rm prior}(\cdot|s,a)$ to get the value $r_{\psi}(s,a)=\mathbb{E}_{\sigma\sim p_{\rm prior}(\cdot|s,a)}[r_{\psi}(s,a|
\sigma)]$.
Note that the prior distribution can be estimated using the unlabeled offline dataset, which is unbiased concerning the environment transition and the behavior policy. Take the gambling MDP again as an example, the marginalization would decrease the utility of action $a_{1}$ since most of the time ($90\%$) the agent will arrive at the bad state.
Besides, when there exists a mismatch between the data distribution of the preference dataset $\mathcal{D}_{\rm p}$ and that of the unlabeled dataset $\mathcal{D}_{\rm u}$ (which is a common occurrence because of the improvement of the policy or the non-uniform sampling for preference pairs), it is found that the reward from existing methods may not align with the RL agent’s interests (Hu et al., ; Liu et al., ).
The conditional reward model provides an effective solution to this issue by enabling the marginalization of the reward function $r_{\psi}$ using a data distribution that is more closely aligned with the on-policy data, thereby improving the alignment between learned behaviors and desired outcomes.

### 3.2 Hindsight Preference Model

We now present *Hindsight Preference Model (HPM)*, a novel preference model that incorporates future information in preference modeling.
As opposed to (3), HPM assumes that the preference strength of a trajectory segment $\sigma=(s_{0},a_{0},s_{1},a_{1},\dots,s_{H})$ is determined by

$$ $\displaystyle\rho_{\rm HPM}(\sigma;\psi)=\sum_{(s_{t},a_{t})\in\sigma}r_{\psi} (s_{t},a_{t}|\sigma_{t:t+k})\,,$ (5) $$

where $\sigma_{i:j}$ denote the subsequence of $\sigma$ between step $i$ and $j$.(^1^11We use $\sigma_{t:t+k}$ to simplify the notations. For $t+k>H$, the subsequence will be clipped to $\sigma_{t:\min(t+k,H)}$.)
That is, in HPM the reward function $r_{\psi}$ not only takes the current state $s_{t}$ and action $a_{t}$ as input, but also depends on the $k$-step future outcome $\sigma_{t:t+k}$.
Then given a segment pair $(\sigma^{0},\sigma^{1})$, HPM model their preference by

$$ $\displaystyle P_{\rm HPM}(\sigma^{0}\succ\sigma^{1};\psi)=\frac{\exp(\rho_{\rm {HPM}}(\sigma^{0};\psi))}{\exp(\rho_{\rm{HPM}}(\sigma^{0};\psi))+\exp(\rho_{ \rm{HPM}}(\sigma^{1};\psi))}\,.$ (6) $$

In practice, directly implementing this conditional reward $r_{\psi}(s_{t},a_{t}|\sigma_{t:t+k})$ is challenging due to the high-dimensional nature of the $k$-step segment $\sigma_{t:t+k}$.
We address this issue by compressing the segment into a compact embedding rather than operating directly in the raw space of trajectory.

Figure: Algorithm 1 Hindsight Preference Learning (HPL)

### 3.3 Future Segment Embedding

We propose to compress future segments $\sigma_{t:t+k}$ into a compact embedding vector $z_{t}$ by training a conditional Variational Auto-Encoder (VAE) (Kingma and Welling, ).
The architecture of our model consists of three components: the encoder $q_{\theta}$, the decoder $p_{\theta}$, and a learnable prior $f_{\theta}$, which can be jointly optimized with the *Evidence Lower Bound (ELBO)*:

$$ $\displaystyle\log p(\sigma_{t:t+k}|s_{t},a_{t})$ (7) $\displaystyle\geq\mathbb{E}_{q_{\theta}(z_{t}|s_{t},a_{t},\sigma_{t:t+k})} \left[\log\frac{p_{\theta}(\sigma_{t:t+k},z_{t}|s_{t},a_{t})}{q_{\theta}(z_{t} |s_{t},a_{t},\sigma_{t:t+k})}\right]$ $\displaystyle=\mathbb{E}_{q_{\theta}(z_{t}|s_{t},a_{t},\sigma_{t:t+k})}\left[ \log p_{\theta}(\sigma_{t:t+k}|s_{t},a_{t},z_{t})\right]-\operatorname{KL} \left[q_{\theta}(z_{t}|s_{t},a_{t},\sigma_{t:t+k})\|f_{\theta}(z_{t}|s_{t},a_{ t})\right]$ $\displaystyle\stackrel{{\scriptstyle\text{def}}}{{=}}-\mathcal{L}_{\rm ELBO}(s _{t},a_{t},\sigma_{t:t+k};\theta).$ $$

Following the pre-training phase, the VAE can be utilized for both reward learning and reward labeling.
In the context of reward learning, the embedding $z_{t}$ can be employed as a substitute for $\sigma_{t:t+k}$ in preference modeling:

$$ $\displaystyle\rho_{\rm HPM}(\sigma;\psi)=\sum_{(s_{t},a_{t})\in\sigma}r_{\psi} (s_{t},a_{t}|\sigma_{t:t+k})\approx\sum_{(s_{t},a_{t})\in\sigma}r_{\psi}(s_{t} ,a_{t}|z_{t})\,.$ (8) $$

Here, the embedding is obtained using the encoder $z_{t}\sim q_{\theta}(\cdot|s_{t},a_{t},\sigma_{t:t+k})$.
Plugging this into the Bradley-Terry model gives an approximation of HPM (6).
We then once again utilize the preference dataset along with the cross-entropy loss, as defined in (2), to optimize $r_{\psi}$.
During the reward labeling phase,
we compute the reward using the prior distribution $f_{\theta}$:

$$ $\displaystyle r_{\psi}(s_{t},a_{t})=\mathbb{E}_{z_{t}\sim f_{\theta}(\cdot|s_{ t},a_{t})}\left[r_{\psi}(s_{t},a_{t},z_{t})\right]\approx\frac{1}{N}\sum_{l=1} ^{N}r_{\psi}(s_{t},a_{t},z_{t}^{(l)})\,,$ (9) $$

where $z_{t}^{(1)},z_{t}^{(2)},\ldots,z_{t}^{(N)}$ are i.i.d. samples from $f_{\theta}$. This approach ensures a robust approximation of the expected reward, facilitating effective reward shaping based on learned preferences.

We train these models using the unlabelled dataset $\mathcal{D}_{\rm u}$. This offers two benefits. Firstly, $\mathcal{D}_{\rm u}$ typically encompasses a substantial volume of data, which enhances model performance. Secondly, the scalar reward is obtained by marginalizing over the prior distribution $f_{\theta}$ during the reward labeling phase. Precisely aligning $f_{\theta}$ with $\mathcal{D}_{\rm u}$ can significantly enhance the stability of this marginalization process, particularly in instances of distributional shifts between $\mathcal{D}_{\rm u}$ and $\mathcal{D}_{\rm p}$.

#### Practical Implementation.

We employ the GPT architecture (Brown et al., ) for the encoder $q_{\theta}$ due to its expressivity in sequence modeling.
Given an input segment
$\sigma=(s_{0},a_{0},s_{1},a_{1},\dots,s_{H},a_{H})$,
we concatenate $s_{t}$ and $a_{t}$ together as a single token. In each attention layer, we apply the anti-causal attention mask which restricts each token’s attention to itself and subsequent tokens, ensuring that the output token $z_{t}$ encapsulates the forward-looking information starting from time step $t$.
The decoder network $p_{\theta}$ reconstructs $\sigma_{t:t+k}$ using the embedding $z_{t}$ and $(s_{t},a_{t})$.
In our implementation, $p_{\theta}$ takes inputs of $s_{t},a_{t},z_{t}$ and a time interval $\Delta t\in[0,1,...,k]$, and predicts $(s_{t+\Delta t},a_{t+\Delta t})$. This parameterization facilitates the parallel decoding of the entire trajectory by processing all specified intervals in a single forward pass. Finally, the prior $f_{\theta}$ is parameterized as an MLP network which receives $(s_{t},a_{t})$ and outputs a distribution over the embedding space.

### 3.4 Overall Framework of HPL

Putting everything together, we outline *Hindsight Preference Learning (HPL)* in Algorithm 1. HPL can be divided into three stages: 1) pre-training a VAE to embed future segments, using data from the unlabeled dataset $\mathcal{D}_{\rm u}$; 2) training the conditional reward function $r_{\psi}$ with the preference dataset $\mathcal{D}_{\rm p}$; and finally 3) label the unlabeled dataset with (9), followed by applying any offline RL algorithms for policy optimization.

## 4 Related Work

Preference-based Reinforcement Learning. Human preferences are easier to obtain compared to well-calibrated step-wise rewards or expert demonstrations in some domains, making them rich yet easy source of signals for policy optimization. Christiano et al. () utilizes the Bradley-Terry model to extract reward function from human preferences and lays the foundation of using deep RL to solve complex tasks. Based on this, several methods (Lee et al., ; Ibarz et al., ; Liang et al., ) further improves the query efficiency by incorporating techniques like pre-training and relabeling. OPRL (Shin et al., ) further proposes principled rules for query selection and provides baseline results using existing offline datasets. On the other hand, some works bypass the need for a reward model. IPL (Hejna and Sadigh, ) achieves this by expressing the reward with value functions via the inverse Bellman operator, while OPPO (Kang et al., ) uses hindsight information matching (HIM) to conduct preference learning in compact latent space. FTB (Zhang et al., ) employs powerful generative models to diffuse bad trajectories to preferred ones. DPPO (An et al., ) and CPL (Hejna et al., ), although with different starting points, both directly optimize the policy by relating it to the preferences.

Human Preference Modeling. To extract utilities from human preferences for policy optimization, we need preference models to establish the connection between preferences and utilities. A common approach is to use the Bradley-Terry model (Christiano et al., ) and hypothesizes that preference is emitted according to the sum of Markovian rewards, while Preference Transformer (Kim et al., ) and Hindsight PRIOR (Verma and Metcalf, ) extend this by using the weighted sum of non-Markovian rewards. Besides, another line of research proposes that human preference is decided by the sum of optimal advantages in the segment (Knox et al., , ), rather than the rewards. In this paper, we focus on the influence of the future and consider the sum of future-conditioned rewards.

Leveraging Hindsight Information. Hindsight information can provide extra supervision during training. For example, HER (Andrychowicz et al., ) proposes to relabel the transitions to allow sample-efficient learning in sparse-reward tasks. Prior works have also explored learning representations by predicting the future (Furuta et al., ; Xie et al., ; Yang et al., ), and such representations facilitate downstream tasks such as policy optimization (Furuta et al., ; Xie et al., ), preference modeling (Kang et al., ), exploration (Jarrett et al., ), and credit assignment (Harutyunyan et al., ).

## 5 Experiments

We evaluate HPL as well as other methods with various benchmarks. Specifically, we selected two tasks (Hopper and Walker2D) from Gym-MuJoCo locomotion (Brockman et al., ), two tasks (Hammer and Pen) from the Adroit manipulation platform (Kumar, ), and four tasks (Drawer-Open, Button-Press, Plate-Slide and Sweep-Into) from Meta-World Benchmark (Yu et al., ). For Gym-MuJoCo tasks and Adroit tasks, we select datasets from the D4RL Benchmark (Fu et al., ) and mask the reward labels as the unlabeled dataset $\mathcal{D}_{\rm u}$, while the annotated preference dataset $\mathcal{D}_{\rm p}$ is provided by Kim et al. (). For Meta-World tasks, we used the datasets released by Hejna and Sadigh () as $\mathcal{D}_{\rm u}$ and $\mathcal{D}_{\rm p}$. It is worthwhile to note that for Gym-MuJoCo and Adroit tasks, the preference label is generated by real human annotators, while for Meta-World tasks it is synthesized based on trajectory return. More details about the datasets and how the preference annotations are generated can be found in Appendix A.

For baseline methods, we consider 1) Oracle, which uses the oracle step-wise reward for policy optimization; 2) Supervised Fine-Tuning (SFT), which imitates the preferred segments; 3) MR, which uses the Bradley-Terry Model to extract Markovian rewards from the preferences; 4) Preference Transformer (PT) (Kim et al., ), which uses a transformer and bidirectional layer to model the reward; and 5) Inverse Preference Learning (IPL) (Hejna and Sadigh, ), which removes the need of reward modeling using inverse Bellman operator. More details about the implementations can be found in Appendix B. For evaluation metrics, we report the normalization score calculated by D4RL and the success rate for Meta-World tasks.

**Table 1: Normalized averaged score for locomotion tasks (top) and manipulation tasks (bottom). In the table, "hop" is abbreviated for the Hopper task, "walk" for Walker2D, "ham" for Hammer, "m" for medium, "r" for replay, "e" for expert, "h" for human, "c" for cloned. The reference scores for MR and PT are from Kim et al. (), while those for IPL are from Hejna and Sadigh (). For the rest numbers, we use our own implementations and report the average and the standard deviation of the performances across 10 evaluation episodes and 5 seeds. We bolded values that are within $95\%$ of the top-performing method among our implemented versions.**
| Dataset | Oracle | SFT | MR | PT | IPL | HPL |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ref. | reimpl. | ref. | reimpl. | ref. | reimpl. |  |  |  |  |
| hop-m-r | $97.4$ | $22.2$ | $11.56$ | $64.3_{\pm 18.2}$ | $84.54$ | $77.4_{\pm 8.0}$ | $73.57$ | $56.1_{\pm 20.3}$ | $\bm{83.0_{\pm 14.4}}$ |
| hop-m-e | $107.4$ | $5.2$ | $57.75$ | $86.3_{\pm 21.6}$ | $68.96$ | $78.7_{\pm 27.8}$ | $74.52$ | $67.8_{\pm 18.0}$ | $\bm{104.0_{\pm 7.7}}$ |
| walk-m-r | $82.2$ | $9.0$ | $72.07$ | $\bm{69.5_{\pm 1.7}}$ | $71.27$ | $64.0_{\pm 15.2}$ | $59.92$ | $42.3_{\pm 17.4}$ | $64.1_{\pm 8.9}$ |
| walk-m-e | $111.7$ | $0.4$ | $108.32$ | $90.8_{\pm 9.0}$ | $110.13$ | $102.2_{\pm 17.5}$ | $108.51$ | $\bm{106.1_{\pm 4.6}}$ | $\bm{108.9_{\pm}{0.5}}$ |
| Dataset | Oracle | SFT | MR | PT | IPL | HPL |  |  |  |
| pen-h | $78.5$ | $36.4$ | $14.1_{\pm 9.0}$ | $11.2_{\pm 4.5}$ | $11.5_{\pm 11.6}$ | $\bm{70.9_{\pm 23.2}}$ |  |  |  |
| pen-c | $83.4$ | $31.1$ | $13.8_{\pm 4.8}$ | $11.9_{\pm 13.3}$ | $12.3_{\pm 6.6}$ | $\bm{33.1_{\pm 19.6}}$ |  |  |  |
| ham-h | $1.8$ | $0.3$ | $0.2_{\pm 0.0}$ | $0.2_{\pm 0.3}$ | $0.0_{\pm 0.0}$ | $\bm{4.3_{\pm 4.7}}$ |  |  |  |
| ham-c | $1.5$ | $\bm{2.6}$ | $0.0_{\pm 0.1}$ | $2.0_{\pm 4.6}$ | $0.1_{\pm 0.1}$ | $0.3_{\pm 0.0}$ |  |  |  |
| drawer-open | - | $0.42$ | $\bm{0.92_{\pm 0.10}}$ | $0.39_{\pm 0.24}$ | $0.54_{\pm 0.26}$ | $\bm{0.95_{\pm 0.07}}$ |  |  |  |
| button-press | - | $0.26$ | $0.61_{\pm 0.04}$ | $0.38_{\pm 0.24}$ | $0.58_{\pm 0.11}$ | $\bm{0.70_{\pm 0.14}}$ |  |  |  |
| plate-slide | - | $0.26$ | $0.38_{\pm 0.08}$ | $0.29_{\pm 0.27}$ | $0.34_{\pm 0.29}$ | $\bm{0.43_{\pm 0.13}}$ |  |  |  |
| sweep-into | - | $0.24$ | $0.31_{\pm 0.10}$ | $0.22_{\pm 0.13}$ | $0.14_{\pm 0.15}$ | $\bm{0.37_{\pm 0.11}}$ |  |  |  |

### 5.1 Benchmark Results

Our first experiment investigates the capability of HPL in standard offline PbRL settings using both $\mathcal{D}_{\rm u}$ and $\mathcal{D}_{\rm p}$. For policy optimization, we used IQL for all methods except for SFT. We found certain design choices such as reward normalization can have a significant effect on the performance, so we included the reference score (denoted as ref.) from the original paper for some algorithms and the score of our implementation (denoted as reimpl.) for fair comparisons.

The results are listed in Table 1. We also implement variants that use AWAC for policy optimization, and the results are deferred to Appendix E.1. Overall, HPL consistently outperforms other baselines both in locomotion tasks and manipulation tasks, especially in complex domains like the pen task. The promising performance validates the effectiveness of HPL for learning from human preferences.

### 5.2 Tasks with Preference Distribution Shift

As we illustrated in Section 3.1, the distribution mismatch between the preference dataset $\mathcal{D}_{\rm p}$ and the unlabeled dataset $\mathcal{D}_{\rm u}$ may affect credit assignments. We take the gambling MDP (Figure 2) as a sanity check to see whether HPL can deliver better results. We used the dataset $\mathcal{D}$ in Section 3.1 as the preference dataset $\mathcal{D}_{\rm p}$, and additionally collected $\mathcal{D}_{u}$ by randomly choosing between $a_{1}$ and $a_{2}$. Afterwards, we compare the reward values of $(s_{1},a_{1})$ and $(s_{1},a_{2})$ given by both MR and HPL. We run both algorithms for 500 random seeds and plot the results in Figure 3. In the figure, each point stands for one trial and its coordinate stands for $r_{\psi}(s_{1},a_{1})$ and $r_{\psi}(s_{1},a_{2})$ respectively. We note that every trial of both methods has achieved $100\%$ accuracy for predicting the preference labels, so we focus on the quality of rewards. As discussed in Section 3.1, successful credit assignment should try to assign lower values to $r(s_{1},a_{1})$, i.e. the point should lie in the above triangular area. While HPL steadily assigns higher rewards to $(s_{1},a_{2})$, MR over-estimates the rewards for $(s_{1},a_{1})$ in almost half of the cases. Although it is still possible that MR can find the optimal policy through RL optimization, we emphasize that HPL delivers a much more robust and advantageous utility value that will facilitate the subsequent policy optimization.

Figure: Figure 3: The rewards values given by the MR method and HPL. Each dot represents one trial and its coordinates are the estimated reward values.
Refer to caption: x3.png

To validate HPL’s capability of learning better rewards in the face of the preference distribution shift on a larger scale, we constructed a series of tasks by combining $\mathcal{D}_{\rm u}$ and $\mathcal{D}_{\rm p}$ collected by different behavior policies. For example, "hopper: med-e $\rightarrow$ med" means we used $\mathcal{D}_{\rm p}$ with medium-expert quality and $\mathcal{D}_{\rm u}$ with medium quality. The performance curves are presented in Figure 4. In such mismatched scenarios, HPL performs better than all baseline algorithms in most of the tasks. Besides, HPL demonstrates faster convergence speeds and more stable performances, which validates that the reward from HPL is more robust and shaped for downstream RL optimization.

Figure: Figure 4: The performance curves of HPL and baseline methods in tasks with mismatched datasets. We report the average (solid line) and the standard deviation (shaded area) of each algorithm across 5 random seeds and 10 evaluation episodes for each seed.
Refer to caption: x4.png

### 5.3 Analysis of HPL

In this section, we examine each part of HPL to gain a deeper understanding of each design choice.

Figure: Figure 5: Left: The rendered image of one raw trajectory selected from the offline dataset (top row) and the reconstruction by the VAE (bottom row). Right: The relationship between the log-probabilities of segments and their embeddings.
Refer to caption: x5.png

Future Segment Embedding and the VAE Structure. HPL relies on the VAE structure to generate compact embeddings for future segment representation and sampling. Consequently, our first analysis investigates the quality of these embeddings.
Figure 5 displays the images of a trajectory segment from the offline dataset (top left) and its reconstruction by the VAE (bottom left). It is important to note that both the encoding and reconstruction processes are based on states and actions, rather than pixel observations. The VAE reconstruction is highly accurate, indicating that the embedding $z_{t}$ effectively compresses the relevant information.
In the right figure, we select one $(s_{t},a_{t})$ from the offline dataset and simulate trajectories from that point using a behavior cloning policy $\pi_{\beta}$. Subsequently, we plot the relationship between the log probability of each trajectory $\log p(\sigma_{t:t+k}|s_{t},a_{t})=\sum_{i=1}^{k}\log\pi_{\beta}(a_{t+i}|s_{t+
i})$) and the log probability of the embedding $\log p(z_{t}|s_{t},a_{t})=\log f_{\theta}(z_{t}|s_{t},a_{t})$). The positive correlation observed between these two log validates the efficacy of sampling from $f_{\theta}$.

Ablation on the Future Length $k$. The parameter $k$ controls the length of future segments encoded into the embedding $z$. As illustrated in Figure 6(a) (full tasks in Figure 10), extending $k$ generally enhances performance, highlighting the benefits of future conditioning. However, as $k$ exceeds a certain threshold, we witness fluctuations and decreases in the performances, probably due to the challenges of representing longer future segments accurately.

Figure: (a) Ablation on $k$
Refer to caption: x7.png

Scaling with Dataset Sizes. We conduct experiments to evaluate the scalability of MR and HPL with varying dataset sizes. In Figure 6(b) (full tasks in Figure 12), we adjust the size of $\mathcal{D}_{\rm u}$ from 10% to 100% of its total capacity and observe that HPL consistently outperforms MR across all proportions of unlabeled data. Figure 6(c) (full tasks in Figure 11) illustrates the scaling trends of HPL and MR with different numbers of preference queries. While both methods exhibit upwards trend in success rates, HPL is significantly better. These experiments collectively confirm the scalability of HPL.

## 6 Conclusions and Discussions

This paper focuses on extracting rewards from human preferences for RL optimization. Unlike previous methods that assume the preference is determined by the sum of Markovian rewards, our method, HPL, instead employs a new preference model that correlates the preference strength with the sum of rewards which are conditioned on the future outcome of this trajectory. By marginalizing the conditional reward over the prior distribution of future outcomes induced by the vast unlabeled dataset, HPL produces more robust and suitable reward signals for downstream RL optimization.

Limitations. The primary limitation of the current version of HPL lies in its failure to exploit the full potential of the learned VAE. Although the VAE functions as a generative model, it has not been employed to augment the unlabeled offline dataset through sampling. Additionally, the VAE architecture holds potential for other capabilities, such as uncertainty estimation (Liang et al., ) and the identification of diverse preferences (Xue et al., ), which have yet to be explored. Future work will focus on extending the HPL framework to fully harness these capabilities and thereby improve its overall performance.

## Appendix A Tasks and Datasets

### A.1 Tasks

We evaluated the HPL algorithm on different environments, including Gym-MuJoCo [Brockman et al., ], Adroit [Kumar, ], and Meta-World [Yu et al., ]. These tasks range from basic locomotion to complex manipulation. Among them, Gym-MuJoCo and Meta-World are released with MIT license, while Adroit is released with the Apache-2.0 license.

Gym-MuJoCo. We selected the Hopper and Walker2D tasks from the Gym-MuJoCo environment. The goal of the Hopper task is to control a single-legged robot to hop forward, with primary rewards based on forward speed, energy consumption, and a survival bonus for maintaining stability. The Walker2D task involves controlling a bipedal robot to walk forward, while the rewards are designed based on the forward speed, control penalties, and a survival bonus. The key challenge in both tasks is to maximize the forward distance while maintaining the robot’s stability.

Adroit. We chose the Hammer and Pen tasks from the Adroit environment. These tasks require controlling a 24-DoF simulated Shadow Hand robot to perform precise manipulations. The Hammer task involves using the robot to hammer a nail, with rewards given for successful strikes and penalties for misses or ineffective actions. The Pen task requires the robot to rotate a pen, rewarding successful rotations and penalizing failures or instability. Adroit tasks emphasize high precision and the complexity of robotic hand manipulations.

Meta-World. We selected multiple manipulation tasks from the Meta-World environment, including drawer-open, button-press, plate-slide, and sweep-into. These tasks require a Sawyer robotic arm to perform multi-step operations. For example, the drawer-open task involves grasping and pulling open a drawer, the button-press task requires accurately pressing a designated button, the plate-slide task involves pushing a plate to a specified location, and the sweep-into task requires sweeping objects into a target area. The reward structure in these tasks is designed as a combination of sub-tasks, providing partial rewards for each sub-task completed and a total reward for successfully completing the entire task. Meta-World tasks highlight the shared structure between tasks and the sequential nature of complex operations.

### A.2 Unlabeled Offline Datasets

Figure: Figure 7: From left to right, the figures show the Hopper task from Gym-MuJoCo, the Hammer task from the Adroit platform, and the button-press and plate-slide tasks from Meta-World.
Refer to caption: x10.png

For the unlabeled offline dataset $\mathcal{D}_{\rm p}$, we used the datasets provided in D4RL [Fu et al., ] for Gym-MuJoCo and Adroit tasks and the datasets from Hejna and Sadigh [] for Meta-World tasks.

Gym-MuJoCo Datasets. The datasets for Hopper and Walker2D tasks were obtained through online training and include *medium*, *medium-replay*, and *medium-expert* datasets. The *medium* dataset was generated by training a policy using Soft Actor-Critic [], stopping early when the policy reached a medium performance level, and collecting 1 million samples from this partially trained policy. The *medium-replay* dataset contains all samples observed in the replay buffer during training until the policy reaches medium performance. The *medium-expert* dataset was created by mixing equal amounts of expert demonstration data and medium-level policy data, aiming to test the algorithm’s performance with varying data quality. All of these datasets can be obtained following the APIs provided by D4RL, and the datasets are licensed with the CC BY 4.0 license.

Adroit Datasets. We used data for the Hammer and Pen tasks. These datasets include *human*, *expert*, and *cloned* datasets. The *human* dataset consists of a small number of demonstrations collected from human experts, with each task containing 25 trajectories. The *expert* dataset comprises a large amount of expert data collected from fine-tuned RL policies. The *cloned* dataset is generated by training an imitation policy on the demonstration data, running this policy, and mixing the generated data with the original demonstrations in a 50-50 ratio. This data generation method simulates a real-world scenario where a small amount of human demonstration data is augmented through imitation learning. All of these datasets can be obtained following the APIs provided by D4RL, and the datasets are licensed with the CC BY 4.0 license.

Meta-World Datasets. We used the preference-annotated dataset from Hejna and Sadigh [] and converted it into an unlabeled offline dataset by discarding the reference label. These datasets can be obtained following the original source of Hejna and Sadigh []. The authors did not specify the license of the datasets. However, the codes are released with the MIT license so we speculate the datasets inherit the MIT license as well since they are released together. In the following paragraphs, we detail the construction of these preference datasets based on the details provided by their original creators.

### A.3 Preference Datasets

For the preference dataset $\mathcal{D}_{\rm u}$, we selected the human-annotated datasets from Kim et al. [] for Gym-MuJoCo and Adroit tasks, and the synthetic datasets from Hejna and Sadigh [] for Meta-World tasks. The datasets are released alongside with the codes ([https://github.com/csmile-1006/PreferenceTransformer](https://github.com/csmile-1006/PreferenceTransformer) and [https://github.com/jhejna/inverse-preference-learning](https://github.com/jhejna/inverse-preference-learning) respectively). The authors did not specify the license of the datasets. However, the codes are both released with the MIT license so we speculate the datasets inherit the MIT license as well since they are released together. In the following paragraphs, we detail the construction of these preference datasets based on the details provided by their original creators.

For Gym-MuJoCo datasets, preferences were collected from actual human subjects. Specifically, human annotators watched the rendered videos of segments and selected the segment they believed was more helpful in achieving the agent’s goal. Each segment lasted 3 seconds (100 frames). Human annotators can prefer one of the segment pairs or remain neutral by assigning equal preference to both segments.
The annotators are instructed to make decisions based on some criteria. For the Hopper task, the robot is expected to move to the right as far as possible while minimizing energy consumption. Segments, where the robot lands unstably, are rated lower, even if the distance traveled is longer. If two segments are nearly tied on this metric, the one with the greater distance is chosen. For the Walker2D task, the goal is to move the bipedal robot to the right as far as possible while minimizing energy consumption. Segments where the robot is about to fall or walks abnormally (e.g., using only one leg or slipping) are rated lower, even if the distance covered is longer. If two segments are nearly tied on this metric, the one with the greater distance is chosen. For the *medium-replay* offline dataset, there are 500 queries, while for the *medium-expert* offline dataset, there are 100 queries in total. The segment length for all datasets is $H=100$.

The Meta-World datasets included script preferences came from Hejna and Sadigh []. First, the datasets included 100 trajectories of expert data for the target task, adding Gaussian noise with a standard deviation of 1.0. Then, the datasets included 100 trajectories of sub-optimal data by running the ground truth policy with noise on different randomizations of the target task, and another 100 trajectories of sub-optimal data by running the ground truth policy of different tasks within the target domain with noise. Finally, the datasets included 100 trajectories generated using uniform random actions. Each Meta-World task dataset contains 200,000 time steps. The preference datasets were constructed by uniformly sampling segments and assigning preference labels based on the total rewards of the segments.

## Appendix B Algorithm Implementations

In this section, we detail the implementations of both HPL and the baseline algorithms used in this paper.

### B.1 Preference Learning Methods

Markovian Reward (MR). The MR method optimizes a markovian reward function $r_{\psi}(s,a)$ using the Bradley-Terry model and the preference dataset $\mathcal{D}_{\rm p}$. The hyper-parameters for MR are listed in Table 2. It is worth noting that we add a final activation layer to the reward network to scale the reward to $[0,1]$. We find that without such activation, the performance of RL severely deteriorates in some of the Gym MuJoCo tasks. We suspect that this is related to the survival instinct [Li et al., ] in offline RL, i.e. in environments with terminal conditions, negative rewards tend to incline the agent to terminate the trajectory by selecting those dangerous out-of-distribution actions. Based on this observation, we decided to activate the reward values with ReLU for the Hopper and Walker2D tasks while leaving them unchanged for other tasks without environmental terminations. Such activation is shared across MR, PT and HPL. However, one may argue that the activation implicitly imposes an inductive bias on the obtained reward, which may not align with the ground truth. So we also add the reference scores in Section 5.1 for Gym MoJoCo tasks for comprehensive comparisons.

**Table 2: Hyper-parameters for MR.**
| hidden dimension for $r_{\psi}$ | 256 |
| --- | --- |
| # of hidden layers for $r_{\psi}$ | 2 for Gym MuJoCo tasks and 3 for others |
| final activation | ReLU for Gym MuJoCo tasks and Identity for others |
| optimizer | Adam |
| learning rate | 3e-4 |
| training steps for $r_{\psi}$ | 50k |

Preference Transformer (PT). The Preference Transformer employs a causal transformer followed by a bi-directional attention layer to estimate the reward values. By using the causal transformer, the states and actions can attend to historical tokens and thus the reward can utilize the historical information. The final bi-directional attention layer uses the attention scores as the weights of the rewards at each time step. The authors found PT can identify and place more emphasis on those critical states and actions. We also re-implemented the Preference Transformer based on the original Jax implementation provided by the authors, and the hyper-parameters are listed in Table 3. Note that we do not use any validation to select the reward model.

**Table 3: Hyper-parameters for PT.**
| attention embedding dimension | 256 |
| --- | --- |
| # of attention layers | 3 |
| # of attention heads | 1 |
| dropout rate | 0.1 |
| final activation | ReLU for Gym MuJoCo tasks and Identity for others |
| optimizer | Adam |
| learning rate | 3e-4 |
| learning rate warm-up steps | 10k |
| training steps for $r_{\psi}$ | 100k |

Inverse Preference Learning (IPL). IPL removes the need for learning a reward model, by expressing the reward using the value functions $Q(s,a)$ and $V(s)$ of the RL agent:

$$ $r(s,a)=Q(s,a)-\gamma\mathbb{E}_{s^{\prime}\sim p(s^{\prime}|s,a)}\left[V(s^{ \prime})\right].$ (10) $$

By substituting Eq (10) into Eq (4), the loss objective $\mathcal{L}_{\rm MR}$ provides guidance to increase the Q values of preferred states and actions. We also re-implemented IPL in this paper, and keep the hyper-parameters of IPL the same as the ones used in the original paper.

Hindsight Preference Learning (HPL). The key components of HPL are the conditional reward model $r_{\psi}$ and the VAE. We list the hyper-parameters of these modules in Table 4. The hyper-parameters are kept the same as listed in Table 4 unless otherwise noted.

**Table 4: Hyper-parameters for HPL.**
| VAE | attention embedding dim of encoder $q_{\theta}$ | 256 |
| --- | --- | --- |
| # of attention layers | 3 |  |
| # of attention heads | 1 |  |
| dropout rate | 0.1 |  |
| hidden dim for decoder $p_{\theta}$ | 256 for MuJoCo/Adroit, 512 for Meta-World |  |
| # of hidden layers for decoder $p_{\theta}$ | 256 |  |
| hidden dim for prior $f_{\theta}$ | 256 for MuJoCo/Adroit, 512 for Meta-World |  |
| # of hidden layers for prior $f_{\theta}$ | 2 |  |
| dimension of embedding $z$ | 128 |  |
| posterior/prior distribution | categorical distribution |  |
| learning rate | 3e-4 |  |
| training steps | 250k |  |
|  | KL loss coefficient | 0.1 |
|  | encoded future segment length $k$ | 5 |
| $r_{\psi}$ | hidden dims of $r_{\psi}$ | 256 |
| # of hidden layers for $r_{\psi}$ | 3 |  |
| final activation | ReLU for MuJoCo, Identity for others |  |
| optimizer | Adam |  |
| learning rate | 3e-4 |  |
| training steps | 100k |  |
|  | marginalization samples $N$ | 20 |

### B.2 RL Policy Optimization

For those methods that follow the two-phase paradigm as we discussed in Section 2.2, we use Implicit Q-Learning (IQL) [Kostrikov et al., ] for policy optimization with the learned reward model. The hyper-parameters for IQL are shared across various reward learning methods for fair comparisons. We list the hyper-parameters in Table 5.

**Table 5: Hyper-parameters for IQL.**
| expectile $\tau$ | 0.7 for MuJoCo, 0.75 for others |
| --- | --- |
| inverse temperature | 0.333 |
| clipping threshold for $\exp A$ | 100.0 |
| discounting factor | 0.99 |
| soft update for target networks | 0.005 |
| policy network | $\texttt{MLP}(\text{dim}(\mathcal{S}),256,256,256,2*\text{dim}(\mathcal{A}))$ |
| policy distribution | tanh-squashed diagonal Gaussian |
| critic network | $\texttt{MLP}(\text{dim}(\mathcal{S})+\text{dim}(\mathcal{A}),256,256,256,1)$ |
| value network | $\texttt{MLP}(\text{dim}(\mathcal{S}),256,256,256,1)$ |
| optimizers for all networks | Adam |
| learning rates for all networks | 3e-4 |
| training steps | 500k |

### B.3 Supervised Fine-tuning

Finally, we provide details about the implementations of the Supervised Fine-tuning (SFT) method used in the experiment section. For SFT, we use the preferred trajectory segments to perform behavior cloning. The behavior cloning process maximizes the log probability of the policy selecting the preferred segment. Thus, SFT methods fail to leverage the vast offline datasets, which is identified as a key advantage of offline PbRL methods. The hyper-parameters of SFT can be found in Table 6.

**Table 6: Hyper-parameters for SFT.**
| policy network | $\texttt{MLP}(\text{dim}(\mathcal{S}),256,256,256,\text{dim}(\mathcal{A}))$ |
| --- | --- |
| policy distribution | deterministic (Dirac distribution) |
| optimizer | Adam |
| learning rates | 3e-4 |
| training steps | 500k |

## Appendix C Experimental Setups

In this section, we provide additional details for the main results in Section 5.

Benchmark results (Table 1). We use the full amount of preference datasets and unlabeled datasets as detailed in Section A for MuJoCo tasks and Adroit tasks. For Meta-World tasks, we take the first 500 queries as the preference dataset $\mathcal{D}_{\rm p}$ and the first 5000 queries as the unlabeled dataset $\mathcal{D}_{\rm u}$. The $\mathcal{D}_{\rm p}$ and $\mathcal{D}_{\rm u}$ matches each other in terms of the data source.

Results of the mismatched tasks (Figure 4). We created a series of tasks by cross-over the preference datasets and the unlabeled datasets, as detailed in the main text. Besides, we use the full amount of the datasets, without further selection.

Scaling trends with varied sizes of $\mathcal{D}_{\rm u}$ (Figure 6(b)). In this set of experiments, we keep the setups and the hyper-parameters exactly the same as in Table 1, except for the sizes of the unlabeled dataset. Specifically, we select the first 1k, 2.5k, 5k, 7.5k and 10k trajectories from the dataset (which corresponds to 10%, 25%, 50%, 75%, 100% of the total capacity of $\mathcal{D}_{\rm u}$).

Scaling trends with varied sizes of $\mathcal{D}_{\rm p}$ (Figure 6(c)). In this set of experiments, we keep the setups and the hyper-parameters exactly the same as in Table 1, except for the sizes of the preference dataset. We select the first 100, 200, 300, 400, 500, and 1000 queries from the dataset, as depicted in the figure.

## Appendix D Disclosure of computational resources and efficiency

Throughout the experiments, we evaluate HPL as well as other baseline methods with workstations equipped with NVIDIA RTX 4090 cards. The running time of each method for the button-press task in the Meta-World environment is presented in Figure 8.

Figure: Figure 8: The running time of HPL and each baseline methods, using button-press from Meta-World as an example.
Refer to caption: x11.png

## Appendix E Supplimentary Experiment Results

Due to the limited space of the main text, we present additional supplementary results in this section.

### E.1 Benchmark Results of AWAC Variants

**Table 7: Normalized averaged score for AWAC variants of HPL and baseline algorithms. In the table, we use the same abbreviations for tasks as in Table 1. We report the average and the standard deviation of the performances across 10 evaluation episodes and 5 seeds, and bold the values that are within $95\%$ of the top-performing methods among all methods except for the Oracle.**
| Dataset | Oracle | SFT | MR-AWAC | PT-AWAC | IPL-AWAC | HPL-AWAC |
| --- | --- | --- | --- | --- | --- | --- |
| hop-m-r | $97.4$ | $22.2$ | $31.2_{\pm 0.2}$ | $68.7_{\pm 18.3}$ | $69.8_{\pm 13.6}$ | $\bm{94.6_{\pm 3.1}}$ |
| hop-m-e | $107.4$ | $5.2$ | $70.9_{\pm 34.5}$ | $\bm{93.3_{\pm 13.9}}$ | $55.3_{\pm 17.2}$ | $\bm{98.0_{\pm 15.9}}$ |
| walk-m-r | $82.2$ | $9.0$ | $63.1_{\pm 9.1}$ | $\bm{77.6_{\pm 5.4}}$ | $8.9_{\pm 11.0}$ | $71.2_{\pm 4.0}$ |
| walk-m-e | $111.7$ | $0.4$ | $\bm{91.7_{\pm 38.7}}$ | $76.7_{\pm 47.4}$ | $46.3_{\pm 52.8}$ | $\bm{95.1_{\pm 8.5}}$ |
| Gym average | $99.7$ | $9.2$ | $64.2$ | $79.1$ | $45.1$ | $\bm{89.7}$ |
| pen-h | $78.5$ | $35.4$ | $10.5_{\pm 9.9}$ | $0.0_{\pm 3.9}$ | $11.7_{\pm 8.2}$ | $\bm{44.7_{\pm 27.6}}$ |
| pen-c | $83.4$ | $31.1$ | $8.9_{\pm 11.8}$ | $12.9_{\pm 14.3}$ | $13.0_{\pm 17.6}$ | $\bm{36.4_{\pm 27.8}}$ |
| ham-h | $1.8$ | $0.3$ | $0.3_{\pm 0.5}$ | $0.0_{\pm 0.1}$ | $0.0_{\pm 0.2}$ | $\bm{4.7_{\pm 5.9}}$ |
| ham-c | $1.5$ | $\bm{2.6}$ | $0.1_{\pm 0.2}$ | $0.1_{\pm 0.1}$ | $0.1_{\pm 0.1}$ | $0.2_{\pm 0.0}$ |
| Adroit average | $41.3$ | $17.4$ | $5.0$ | $3.3$ | $6.2$ | $\bm{21.5}$ |
| drawer-open | - | $0.42$ | $0.77_{\pm 0.28}$ | $0.56_{\pm 0.29}$ | $0.58_{\pm 0.19}$ | $\bm{0.89_{\pm 0.07}}$ |
| button-press | - | $0.26$ | $\bm{0.78_{\pm 0.14}}$ | $0.67_{\pm 0.23}$ | $0.66_{\pm 0.26}$ | $0.69_{\pm 0.12}$ |
| plate-slide | - | $0.26$ | $0.35_{\pm 0.25}$ | $0.07_{\pm 0.10}$ | $\bm{0.52_{\pm 0.18}}$ | $0.47_{\pm 0.21}$ |
| sweep-into | - | $0.24$ | $0.30_{\pm 0.19}$ | $0.10_{\pm 0.12}$ | $0.24_{\pm 0.11}$ | $\bm{0.49_{\pm 0.09}}$ |
| Meta-World average | - | $0.30$ | $0.55$ | $0.35$ | $0.50$ | $\bm{0.64}$ |

In Section 5.1, the results are obtained by using IQL [Kostrikov et al., ] as the policy optimization algorithm. However, given that IQL relies on expectile regression rather than the policy for bootstrapping, it may not fully reveal the potential shortcomings of the learned rewards. Additionally, the choice of expectile could significantly affect the outcomes. In this section, we instead implement AWAC, another offline reinforcement learning (RL) algorithm that integrates policy into bootstrapping, to both HPL and the baseline algorithms. This approach aims to provide a more thorough assessment of reward quality.

The results are listed in Table 7. Similar to HPL-IQL, HPL-AWAC demonstrates stable and consistent advantages over the baseline methods in most of the tasks. Overall, HPL-AWAC achieves the best performance on average across these three task suites.

### E.2 Ablation on the Effect of Ensembling Reward Models

Figure: Figure 9: Learning curve of HPL and MR variants with ensemble reward networks. We report the average and the standard deviation of the performances across 10 evaluation episodes and 5 seeds.
Refer to caption: x12.png

One could argue that the success of HPL can be attributed to the marginalization step (Eq (9)), which implicitly ensembles reward models to yield improved rewards for subsequent RL optimization. As revealed by previous literature, reward model ensembling does bring benefit to the credit assignment by characterizing the aleatoric uncertainty and thus facilitating active knowledge acquisition [Liang et al., ] or promoting pessimism [Coste et al., ]. In this section, we ablate the effect of reward model ensembling by applying the ensembling trick to the MR method. Specifically, we set the number of reward ensembles to 5 and 20 respectively, and keep other configurations the same as the experiments in Section 5.1. These reward models are trained with the same dataset, differing only in their initializations. When labeling the dataset, we take the average of the outputs of the ensembles as the reward. Note that this can be considered as an implementation of OPRL, which also employs the ensembling technique.

Figure: Figure 10: Success rate of HPL with various encoded future length $k$. We report the average and the standard deviation of the performances across 20 evaluation episodes and 10 seeds.
Refer to caption: x13.png

As witnessed in Figure 9, despite the ensembling technique, the performance of MR still falls behind HPL. This justifies that, naively ensembling reward models which are trained via different initialization does not benefit the credit assignment. On the other hand, HPL ensembles reward values conditioned on different future outcomes based on their prior probabilities, producing more reliable and advantageous rewards.

### E.3 Ablation on Future Length k 𝑘 k italic_k

The parameter k controls the lengths of future segments encoded into the embedding. At the extreme of $k\to 0$, HPL theoretically degenerates to MR as the conditional reward $r_{\psi}$ contains no information about the future.

Figure 10 illustrates the performance of HPL across various values of $k$ for all tasks in Meta-World. As $k$ increases from zero, the performance generally improves, supporting the efficacy of future conditioning. However, beyond a certain threshold, further increases in $k$ lead to performance declines and fluctuations. This phenomenon may be attributed to the incapability of modeling excessively long trajectory segments with the VAE structure.

Figure: Figure 11: Success rate of HPL and MR with various sizes of $\mathcal{D}_{\rm p}$. We report the average and the standard deviation of the performances across 20 evaluation episodes and 10 seeds.
Refer to caption: x14.png

Figure: Figure 12: Success rate of HPL and MR with various sizes of $\mathcal{D}_{\rm u}$. We report the average and the standard deviation of the performances across 20 evaluation episodes and 10 seeds.
Refer to caption: x15.png

### E.4 Scaling with Dataset Sizes

In this section, we investigate the performance of HPL as well as MR with various dataset sizes. We vary the sizes of the preference dataset $\mathcal{D}_{\rm p}$ and the unlabeled dataset $\mathcal{D}_{\rm u}$, and plot the curve of the performances in Figure 11 and Figure 12, respectively. While both methods exhibit an upward trend in success rates as the dataset sizes $|\mathcal{D}_{\rm p}|$ and $|\mathcal{D}_{\rm u}|$ grow, HPL outperforms MR in almost every configuration. These experiments collectively confirm the superiority and scalability of HPL.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: - Why do the segment encoder q, decoder p, and prior f all use the same notation, θ, for their parameters?
- How does the distribution mismatch between Dp and Du cause problems, and how does HPL address this issue (through improved credit assignment) in the main benchmark experiments? Is there any analysis on these tasks other than the example of gambling MDP?
soundness: 3
strengths: - The paper introduces a framework for hindsight preference learning, with a practical implementation using a VAE architecture.
- I think using the example of a gambling MDP greatly helps in understanding the motivation for using hindsight preference learning.
- Experimental results support the effectiveness of HPL.
summary: The paper introduces a new preference learning method that utilizes hindsight information. By incorporating future states into the reward through a VAE structure, HPL can generate more robust rewards and overcome the limitations of Markovian rewards.
weaknesses: - Although I understand the benefits of hindsight preference learning over Markovian rewards, the authors do not clearly position their paper within the broader research on non-Markovian rewards. As mentioned in the related work, both the Preference Transformer and Hindsight PRIOR  assume non-Markovian rewards and consider the full trajectory when calculating rewards. Given this, why does the paper primarily motivate its approach as an improvement over Markovian rewards, despite the existence of several approaches that do not rely on this assumption?
- For example, what is the fundamental difference between this paper and the Hindsight PRIOR paper, as Hindsight PRIOR considers “the importance of a state in a trajectory given that a future state was reached” [1]? Please clarify the novelty of this work in comparison to other hindsight-based PbRL papers.
- The gambling example is very helpful in illustrating the motivation. However, I am concerned that in more stochastic settings, hindsight modeling may introduce noise and add excessive complexity. The ablation study on k, while focusing on a different message, seems to somewhat support this concern.

References 

[1] Verma, M., & Metcalf, K. (2024). Hindsight PRIORs for Reward Learning from Human Preferences.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: The preference signals on the MetaWorld platform do not actually take into account future information. Why is the algorithm proposed in this paper considered superior?
soundness: 2
strengths: **Originality**: The paper studies a very important problem in PbRL: the nature of human annotations, which is inherently more complex than scripted annotations. By proposing a VAE structure with marginalization techniques, this paper a novel and interesting method.


**Quality**: Experiments are conducted on existing human-annotated dataset where the proposed method shows greater performance.


**Clarity**: The paper is well written with illustrative examples, clear text and figures.


**Significance**: The studied problem: the mismatch between annotations and chosen reward models is of very importance and shall be beneficial for the community.
summary: This paper studies preference-based reinforcement learning (PbRL). Notably, it proposes to consider human annotations from a holistic perspective; that is the annotations would include the considerations of future outcome instead of in-trajectory information only. Inspired by this mismatch, authors propose to model human preference with hindsight information via a VAE structure, which is further marginalized for rewarding labeling and following RL downstream tasks. Experiments are conducted on annotations from human and scripted teacher and advantageous performance is demonstrated.
weaknesses: The primary concern lies in the **mismatch** between motivation and downstream methods. 

The paper aims to incorporate additional future information when learning the $r_{\psi}$. However, actually, the $r_{\psi}$ that takes future information into account should be the Q-function rather than the reward function. Or more rigorously, the $r_{\psi}$ actually encompasses far more information than reward function, but it is not necessarily a Q-function. When downstream algorithms utilize the $r_{\psi}$, the $r_{\psi}$ is considered as reward function, employing a method that does not consider future information.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: In addition to questions about potential weaknesses, I would like to raise a few more questions:
- When computing the reward using the prior distribution, how does the effect depend on the number of samples? (N=20 in appendix)
- What about using both preference data and unlabeled data when training the VAE? Wouldn't this improve the representation across multiple points?
soundness: 3
strengths: - The problem to be solved is clearly specified, and although the approach is simple, it effectively addresses aspects that previous papers have overlooked.
- In the experiments, HPL outperforms the existing SOTA algorithms on most datasets.
summary: The authors study the problem of offline preference-based RL. One major problem with existing SOTA approaches, as cited by the authors, is the evaluation of trajectory segments from a global perspective, making the obtained reward implicitly dependent on the future part of the segment. The contributions of the paper can be summarized as follows:
- The authors propose HPL, which models human preferences using a reward function conditioned on the state, action, and future trajectory, referred to as hindsight information.
- HPL leverages the trajectory distribution of the unlabeled dataset to learn a prior over future outcomes, providing robust and advantageous rewards during labeling.
weaknesses: - I understand that the use of a gambling MDP in Section 3.1 highlights the importance of this study. However, given the very short trajectory length (two steps) in the example, it remains unclear whether hindsight information was effectively applied to achieve accurate credit assignment using future trajectory information.
- Figure 10 shows that HPL is already 2 to 10 times slower than other models. There seem to be several reasons for this slower speed. 
     - Equation 8 requires summing all steps of each trajectory, which likely takes considerable time.
     - A significant amount of time is spent on sampling in Equation 9.
     - If the running time includes VAE modeling time, it is expected to take even longer.
- The core idea of HPL seems to lie not in using future trajectory to train the model, but rather in the learned VAE that represents the future trajectory. As the authors mentioned, when the future length exceeds a certain threshold, it becomes difficult to accurately represent longer future segments, which lowers HPL's performance. If future information is available, the performance should ideally be more accurate. How might this limitation be addressed?

Minor comments:

- typo on line 403: ''$\hat{P}(s_{good}|s_{1},a_{1})$ and $\hat{P}(s_{good}|s_{1},a_{1})$" -> "$\hat{P}(s_{good}|s_{1},a_{1})$ and $\hat{P}(s_{bad}|s_{1},a_{1})$"

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. The experiments utilize a customized split of the original dataset, using medium-expert quality for the preference dataset and medium quality for the unlabeled dataset. Has there been any evaluation of preference shift on real-world, large-scale datasets?
2. Equation 9 includes the number of samples N in the reward function. How does N influence performance, and what is the practical range for this parameter?
3. The concept of utilizing future segments appears similar to graph search and retrieval-based offline RL methods [1,2]. Could HPL achieve comparable results?
[1] Deyao Zhu, Li Erran Li, and Mohamed Elhoseiny. Value memory graph: A graph-structured world model for offline reinforcement learning. arXiv preprint arXiv:2206.04384, 2022. 
[2] Yin Z H, Abbeel P. Offline Imitation Learning Through Graph Search and Retrieval. arXiv preprint arXiv:2407.15403, 2024.
soundness: 4
strengths: 1. The paper addresses a critical challenge in offline preference-based reinforcement learning, presenting well-motivated arguments and clearly illustrating the reward modeling issue through compelling examples.
2. The authors make the insightful observation that human preferences often stem from final outcomes. Their proposed hindsight preference learning leverages future segments to better model these preferences, resulting in a more robust reward function, particularly for datasets with outcome shifts due to multiple modalities or approaches. The integration of conditional VAE makes the algorithm practical for complex tasks like visuomotor policy learning.
3. The experimental evaluation strongly supports the core ideas, encompassing multiple popular benchmarks and competitive baselines. The comprehensive ablation studies examining future segment length, dataset sizes, and reward ensemble provide valuable insights into HPL's advantages.
summary: This paper introduces Hindsight Preference Learning (HPL), a novel reward model learning method for offline preference-based reinforcement learning. The authors address the issue of distribution shift between preference-labeled and unlabeled datasets, which can bias the Markovian reward assumption. To tackle this problem, they propose a reward function that is conditioned not only on states and actions but also on future segments. To handle the high-dimensional nature of future segments, the authors employ a conditional VAE to transform them into embedding vectors that better integrate with the reward function. The effectiveness of the method is demonstrated through experiments on Gym-MuJoCo, Adroit, and Meta-World benchmarks, comparing against existing preference-based reinforcement learning algorithms. The results show significant improvements under preference distribution shift, while ablation studies validate the effectiveness of both the VAE and hindsight reward function.
weaknesses: 1. The VAE implementation is limited to encoding future segments, whereas it could potentially be extended to data augmentation, prior distribution modeling, and other applications.
2. While the authors focus on "preference distribution shift" between labeled and unlabeled datasets, they assume stability in other distributions. The paper does not adequately address potential shifts in visual appearance, dynamics, and sensor noise - common challenges in real-world datasets.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 2
strengths: 1. The paper is clearly written, easy to understand, and provides detailed, well-functioning code.
2. The proposed VAE method incorporates hindsight future information from trajectories into the reward model, which effectively mitigates the reward learning issue caused by the distributional shift between preference data $D_p$ and offline data $D_u$.
3. The experimental performance is impressive, and the ablation studies are thorough.
summary: This paper introduces Hindsight Preference Learning (HPL), a novel method for offline PbRL. It addresses the shortcomings of existing methods that rely on Markovian reward assumptions, by incorporating future trajectory outcomes in reward learning. HPL uses a variational auto-encoder (VAE) to encode future information and improves credit assignment by considering the future consequences of actions. The authors provide extensive empirical results across a variety of tasks (locomotion and manipulation) and demonstrate that HPL outperforms several state-of-the-art PbRL methods.
weaknesses: 1. The motivation example might be unreasonable: As far as I know, in the same task, $D_u$ is often sampled from $D_p$, so they follow the same marginal distribution. When such an extreme example occurs, it indicates a significant distribution difference between $D_u$ and $D_p$. Although Section 5.3 demonstrates the effectiveness of HPL under distributional shift, this example may lack generality.
2. In fact, this method can be viewed as leveraging the unlabeled dataset to improve the robustness of the reward model. The idea is somewhat similar to previous work [1], which learns the reward model from labeled data and then labels the unlabeled data to enhance offline RL performance. A further theoretical analysis, such as exploring the theoretical performance bounds of learning the hidden dynamics from the unlabeled data in the reward model, would enrich the paper (though this could be very challenging, so it's just a suggestion).
3. In Table 7, the performance of OPPO and CPL is quite poor. Is this mainly because the preference data $D_p$ is too small?
4. In Figure 6c, reducing the size of $D_p$ leads to worse performance. Why is that?
5. The baselines and ablations lack verification of the core idea of the paper: If the authors aim to verify that the performance improvement comes from the richer environment dynamics provided by the unlabeled data, they should compare with an ablation version where the VAE learns future representations from $D_p$ itself rather than from $D_u$, to further clarify the source of the performance gain. Additionally, in Table 1, a bi-directional transformer version of PT could be used to model future information as another baseline for comparison.
[1] The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning. ICLR23.

Meta-review: **summary**

This paper introduces HPL, which incorporates human annotations that consider future trajectory outcomes rather than just in-trajectory information. By leveraging VAE to encode high-dimensional future segments into embedding vectors, HPL conditions the reward function on states, actions, and future outcomes, effectively addressing distribution shifts between labeled and unlabeled datasets. Empirical results across diverse tasks demonstrate that HPL significantly outperforms state-of-the-art PbRL methods, providing robust and advantageous performance under preference distribution shifts. 

**strengths**

* Simple yet effective method with clear motivation
* Extensive experiments on diverse benchmarks (e.g., Gym-MuJoCo, Adroit, Meta-World) demonstrate that HPL outperforms state-of-the-art PbRL methods.
* The paper is well-written with clear text, illustrative figures, and thorough explanations.

**weaknesses**

* The distinction between HPL and existing works like Hindsight PRIOR, which also considers the importance of states based on future outcomes, is not clearly described.
* Ambiguous design choice in utilizing reward. The proposed reward function includes additional future trajectory information, but this information is treated as a standard reward function during downstream tasks. This approach may overlook the impact of this added information, potentially leading to biases or mismatches in reinforcement learning algorithms.
* The effectiveness of HPL heavily depends on the quality of future trajectory representation via the VAE. As future segments grow longer, the VAE struggles to encode them accurately, which diminishes performance. The paper lacks a clear strategy to mitigate this limitation and improve handling of long future trajectories.

**decision**

Even though the proposed idea is interesting, several design choices are ambiguous and similarity with prior works limits the originality of the work. During the discussion phase, some comments were partially addressed, but the paper requires revision to fully resolve these issues.

Decision: Reject