Two-Step Offline Preference-Based Reinforcement Learning with Constrained Actions

## Abstract

Abstract Preference-based reinforcement learning (PBRL) in the offline setting has succeeded greatly in industrial applications such as chatbots. A two-step learning framework where one applies a reinforcement learning step after a reward modeling step has been widely adopted for the problem. However, such a method faces challenges from the risk of reward hacking and the complexity of reinforcement learning. To overcome the challenge, our insight is that both challenges come from the state-actions not supported in the dataset. Such state-actions are unreliable and increase the complexity of the reinforcement learning problem at the second step. Based on the insight, we develop a novel two-step learning method called PRC: preference-based reinforcement learning with constrained actions. The high-level idea is to limit the reinforcement learning agent to optimize over a constrained action space that excludes the out-of-distribution state-actions. We empirically verify that our method has high learning efficiency on various datasets in robotic control environments.

## 1 Introduction

Deep Reinforcement Learning (DRL) is a learning paradigm for solving sequential decision-making problems and has rich real-world applications (Sutton & Barto, ; Luong et al., ; Haydari & Yılmaz, ). However, traditional DRL approaches require numerical reward feedback during learning, which in practice can be hard to design or obtain. In contrast, non-numerical preference feedback is more accessible in most cases (Wirth & Fürnkranz, ). As a result, preference-based reinforcement learning (PBRL) that only requires preference feedback has become a more realistic learning paradigm (Christiano et al., ). Recently, as a special instance of PBRL, reinforcement learning from human feedback (RLHF) that utilizes human preference has drawn much attention and achieved great success in many NLP tasks (Ouyang et al., ). Although a majority of current PBRL works focus on the online learning setting (Ibarz et al., ; Lee et al., ), the PBRL learning in the offline setting is more realistic Rafailov et al. (). In the online learning setting, the learning agent continuously interacts with the learning environment to collect new preference data throughout the training process; In the offline learning setting, the agent receives a batch of preference data before training starts. Compared to the online learning setting, the offline learning setting is more accessible as it does not require customized feedback throughout the training process. It is critical to investigate how to make PBRL learning efficient and practical in the offline setting. Particularly, we are interested in the most popular learning framework that is widely applied in various PBRL literature and industrial applications, which we call ‘two-step learning.’

Two-step learning framework: In a typical two-step learning framework (Ibarz et al., ; Christiano et al., ), the first learning step is reward modeling, where the learning agent approximates the reward model of the environment that can best explain the observations from the training dataset. The second learning step is a reinforcement learning phase where the learning agent learns a policy based on the reward model acquired in the previous step. There are multiple reasons behind the vast popularity of the two-step learning paradigm: 1. similar to the motivation of inverse reinforcement learning (Arora & Doshi, ), the learned environment model provides a succinct and transferable definition of the task; 2. the method is modular in that each phase of learning can be implemented by existing well-developed methods. Despite its popularity, this method faces two main challenges.

- •
Reward over-optimization/Pessimistic learning: Pessimistic learning has always been the key challenge in offline learning settings. Due to the distribution mismatch between the trajectories induced by the policy and that of the dataset, the agent cannot properly evaluate all possible possible policies.
- •
Reinforcement learning complexity: Implementing Reinforcement learning efficiently is known to be challenging Dulac-Arnold et al. (). In addition, for preference feedback, the reward model is learned from indirect preference signals rather than actual true rewards. The learned reward model may look very different from the actual reward, making the training step more unstable.

The challenges are empirically verified in Section 5. To solve the first challenge, the most popular approach is to apply a penalty on the evaluated performance of a policy during the reinforcement learning step. The penalty is the KL divergence between a policy and the behavior policy that can best describe the training data distribution. Under the penalty, the agent is encouraged to stay close to the dataset distribution while trying to find a policy that optimizes the learned reward model. However, the reinforcement learning problem in this problem may not be easy to solve due to the second challenge, which is empirically verified in Section 5.

The approaches to solving the second challenge mainly focus on the bandit setting for NLP applications. The bandit setting is a special case of RL that has no state transition. Current methods include direct preference alignment Rafailov et al. () and rejection sampling Liu et al. (), both of which cannot be directly applied to the general RL setting. To the best of our knowledge, no existing two-phase learning approach considers tackling both challenges simultaneously.

Our contributions. In this work, we propose a novel two-step learning PBRL algorithm PRC that tackles the above two main challenges simultaneously. Our key insight is that both challenges are induced by the same element: the state actions that are out of the dataset distribution. For pessimistic learning, the agent must avoid the policies that are more likely to visit such state-actions. These policies are not well covered by the dataset, and the agent cannot evaluate their performance accurately. Meanwhile, these out-of-distribution state-actions contribute to the complexity of the learning problem. Therefore, our insight is to constrain the action space to exclude all such out-of-distribution state-actions. Based on the insight, the key step in our approach is to constrain the action space to include only the actions with a high probability of being sampled by a behavior clone policy that represents the dataset action distribution. In other words, our method only focuses on the policies that are in the neighborhood of the behavior clone policy. As a result, our method not only inherits the pessimistic idea of behavior regularization but also reduces the complexity of the corresponding reinforcement learning problem. In the experiments, we construct offline PBRL datasets based on standard offline RL benchmark D4RL Fu et al. (). The details of dataset construction can be found in Section 5. We empirically verify that on the offline datasets we construct, our approach can always find policies of higher performance than the behavior policies of the dataset, suggesting that the improvement in our PBRL training is reliable.

## 2 Related Work

Offline Reinforcement learning: There have been many studies on the standard offline reinforcement learning setting with reward feedback (Levine et al., ; Cheng et al., ; Yu et al., ), in contrast to preference feedback considered in our setting. The key idea behind offline reinforcement learning is pessimism which encourages the learning agent to focus on the policies that are supported by the dataset (Cheng et al., ). However, the gap between learning from reward feedback and preference feedback is not clear at present.

Online PBRL: Preference-based reinforcement learning is popular in the online learning scenario. Various types of preference models and preference labels have been considered in the tabular case (Fürnkranz et al., ; Wirth et al., ). Similar to the works that consider the general case, in our work we consider the preference models to be Bradley-Terry models and output binary preference labels (Ibarz et al., ; Lee et al., ). The two-phase learning approach mentioned in Section 1 is prevalent in the online PBRL setting. Note that directly applying online methods to the offline scenario is also likely to be inefficient even when it is possible (Van Hasselt et al., ), as the online learning approaches do not need to be pessimistic.

Offline PBRL: Offline preference-based reinforcement learning is a relatively new topic. Zhan et al. () propose an optimization problem following the two-phase learning framework and theoretically prove that the solution to the problem is a near-optimal policy with respect to the training dataset. However, how to solve the optimization problem in practice remains unknown. Concurrently, Zhu et al. () formulates a similar optimization problem under the linear assumption on the environment and proposes a way to solve the problem, but the results cannot be extended to the general case without the linear assumption. Kim et al. () study methods for learning the utility models based on preference provided by humans. They propose to use a transformer-based architecture to predict the utility function behind the preference model. Rafailov et al. () propose a method that directly learns a policy without learning a utility model explicitly. Unlike the setting considered in our work, they focus on the NLP task and require trajectories in each pair to have the same prompt (initial state). Hejna & Sadigh () propose a method that learns a policy and the Q function of the environment instead of the utility function. They also require access to an additional trajectory dataset that only contains trajectories to make their algorithm efficient. Our work focuses on developing a practical and efficient two-phase learning approach for the offline PBRL setting that only has access to a preference dataset with no requirement on the trajectories in the dataset.

RLHF: Reinforcement learning from human feedback is a special instance of PBRL that its preference labels are provided by humans. It has been a popular topic recently. Success in various robotic control and NLP taasks have been achieved by fine-tuning pre-trained models through reinforcement learning based on preference feedback from human (Ouyang et al., ; Bai et al., ). These studies focus on solving specific tasks instead of the general problem considered in our work.

## 3 Preliminary

### 3.1 Offline Preference Reinforcement Learning Problem

We consider an offline preference-based reinforcement learning setting (Zhan et al., ). An environment is characterized by an incomplete Markov Decision Process (MDP) without a reward function $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P})$ where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, and $\mathcal{P}$ is the state transition function. A policy $\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ is a mapping from a state to a probability distribution over the action space, representing a way to behave in the environment by taking the action at a state. A deterministic policy is a special kind of policy that maps a state to a single action. We use $\Pi_{\mathcal{S},\mathcal{A}}$ to denote the class of all policies, and $\Pi^{D}_{\mathcal{S},\mathcal{A}}$ to denote the class of deterministic policies. A trajectory of length $t$ is a sequence of states and actions ($s_{1},a_{1},\ldots,s_{t},a_{t},s_{t+1})$ where $a_{i}\in A,s_{i+1}\sim P(\cdot|s_{i},a_{i})\forall i\in[t]$. A preference model $F$ takes a pair of trajectories $\tau_{1},\tau_{2}$ as input, and outputs a preference $\sigma\in\{\succ,\prec\}$, indicating its preference over the two trajectories, i.e. $\tau_{1}\succ\tau_{2}$ or $\tau_{1}\prec\tau_{2}$. In this work, following the conventions, we consider the preference models that are Bradley-Terry (BT) models (Bradley & Terry, ). Specifically, there exists an utility function $u:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, such that the probability distribution of the output preference of $F$ given $\tau_{1},\tau_{2}$ satisfies:

$$ $\Pr\{\tau_{1}\succ\tau_{2}\}=\frac{\exp(\sum_{(s,a)\in\tau_{1}}u(s,a))}{\exp( \sum_{(s,a)\in\tau_{1}}u(s,a))+\exp(\sum_{(s,a)\in\tau_{2}}u(s,a))}$ (1) $$

We define the performance of a policy $\pi$ on a utility function $u$ as the expected cumulative utility of a trajectory generated by the policy: $\mathbb{E}_{\tau\sim(\pi,\mathcal{P})}[\sum_{s,a\in\tau}u(s,a)]$. An offline dataset $D=\{(\tau^{1}_{1},\tau^{1}_{2},\sigma^{1}),\ldots,(\tau^{N}_{1},\tau^{N}_{2},
\sigma^{N})\}$ consists of multiple preference data over different trajectory pairs. We use the term ‘behavior policy’ to represent how the trajectories are generated in the dataset. A learning agent is given access to the incomplete MDP $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P})$ of the environment and a dataset $D$ whose preferences are generated by a preference model $F$. The learning goal is to learn a policy that has high performance on the utility function $u$ of the preference model $F$, and we say a learning algorithm is efficient if it can learn such a high-performing policy.

### 3.2 Traditional PPO Learning with KL-regularization

Here, we introduce the prevalent two-step learning framework that has been widely applied for solving PBRL problems Christiano et al. () and is closely related to our algorithm in this work. In general, the algorithm aims at solving the optimization problem below:

$$ $\displaystyle\operatorname*{arg\,max}_{\pi\in\Pi_{\mathcal{S},\mathcal{A}}} \mathbb{E}_{\tau\sim(\pi,\mathcal{P})}\sum_{(s,a)\in\tau}[\hat{u}(s,a)]-\alpha \cdot\text{KL}(\pi,\pi_{b})$ (2) s.t. $\displaystyle\hat{u}=\arg\min_{u}\mathcal{L}_{u}(u,D)$ $\displaystyle\pi_{b}=\arg\min_{\pi\in\Pi_{b}}\mathcal{L}_{\pi}(\pi,D)$ $$

Here, $\alpha$ is a parameter to control the degree of pessimism. $\hat{u}$ is a learned reward model and $\pi_{b}$ is a behavior imitation policy. $\mathcal{L}_{u}(u,D)$ is a loss function to evaluate the quality of the utility model on interpreting the preference labels in the dataset. $\mathcal{L}_{\pi}(u,D)$ is a loss function to evaluate how well the behavior imitation policy reproduces the behavior demonstrated by the dataset.

To solve the optimization problem, the first step is to find the optimal reward model and behavior imitation policy that minimize their corresponding losses. The second step is to use a reinforcement learning algorithm such as PPO to solve for the policy that maximizes the optimization goal. Pessimism is achieved through the KL regularization in the optimization goal, which encourages the policy not to be very different from the behavior imitation policy. Formally, Algorithm 1 below represents a typical way to solve the optimization problem in Eq 2.

Figure: Algorithm 1 Typical two-step training with KL-regularization

## 4 Preference Based Reinforcement Learning on a Constrained Action Space (PRC)

### 4.1 General PRC Algorithm

First, we introduce our main algorithm PRC. In Alg 2, we formally show the general form of PRC.

Figure: Algorithm 2 Preference Based Reinforcement Learning on a Constrained Action Space

Here, we abuse the notation of $\pi(a|s)$. If the action space is discrete, then $\pi(a|s)$ represents the probability of the policy to generate the response $a$ at the state $s$. If the action space is continuous, the $\pi(a|s)$ becomes the probability density. In the first step, the learning agent learns a behavior policy that can best imitate the behavior demonstrated in the dataset and then learns a reward model that best interprets the dataset’s preference label. This is the same as the first step in the typical two-step learning framework Alg 1. In the second learning step, the agent learns the policy supported on a clipped action space that maximizes the cumulative return on the learned reward model. At this step, the learning agent only considers the high probability actions according to the behavior policy.

Formally, let the incomplete MDP of the environment be $(\mathcal{S},\mathcal{A},\mathcal{P})$. Let the behavior policy and reward model learned in the first step be $\pi_{0}$ and $\mathcal{R}$ respectively. We define a constrained action space $\mathcal{A}^{\prime}$ based $\mathcal{A}$ and $\pi_{0}$. Let $\mathcal{A}^{\prime}|s$ be the set of actions $\mathcal{A}^{\prime}$ at a state $s$. The constrained action space $\mathcal{A}^{\prime}$ consists of all actions that have a probability higher than a threshold to be sampled from the behavior policy at a state $s$: $\mathcal{A}^{\prime}|s=\{a:\pi_{0}(a|s)\geq p\}$. Then, at the second learning step, the agent solves an RL problem where the MDP is $(\mathcal{S},\mathcal{A}^{\prime},\mathcal{P},\mathcal{R})$.

### 4.2 Analysis

Here, we show that the reinforcement learning step in PRC is essentially optimizing the reward function under a special behavior regularization constraint. Recap the prevalent PBRL framework in Alg 1. The pessimism in the algorithm is achieved by using KL regularization, which encourages the policy to be not very different from the behavior policy. Here, we consider a different regularization as below.

$$ $C_{p}(\pi,\pi_{b})=\begin{cases}-\infty&\text{if }\exists(s,a)\in(\mathcal{S}, \mathcal{A}),\pi(a|s)>0,\pi_{b}(a|s)<p\\ 0,&\text{otherwise}\end{cases}$ $$

The optimization problem under such a regularization is $\operatorname*{arg\,max}_{\pi\in\Pi_{\mathcal{S},\mathcal{A}}}\mathbb{E}_{\tau
\sim(\pi,\mathcal{P})}\sum_{(s,a)\in\tau}[\hat{u}(s,a)]-\alpha\cdot C_{p}(\pi,
\pi_{b})$.
Under such a regularization, the optimal policy can only be supported in a constrained action space where the probability density of the behavior policy is greater than the threshold $p$, as otherwise, it will suffer from an infinite penalty from the regularization.

Next, we discuss why choosing such a regularization is reasonable. First, unlike the soft constraint, such as KL regularization, our regularization is a hard constraint that forces the policy to stay close to the behavior policy. It is conceptually more conservative in that it explicitly decreases the possibility of reward hacking by outputting some policy that is not close to the behavior policy. Second, it reduces the complexity of finding the optimal policy under the regularization. This can make the reinforcement learning step more accessible and reduce the optimization loss at this step.

### 4.3 Practical Implementation

Here, we introduce a practical implementation of our PRC algorithm, which is later used in our experiments. First, we can train a deterministic policy $\pi_{0}$ that has a high probability of reproducing the behavior demonstrated by the dataset. The action given by the deterministic policy can be thought of as the center of the distribution of the actual underlying behavior policy of the dataset. Since we have no prior knowledge about the shape of the distribution of the behavior policy, we may only infer that an action is more likely to be sampled by the behavior policy if it is closer to the center of its distribution. Then, we can approximate the space of $\mathcal{A}^{\prime}$ as a box whose center is at the deterministic policy. The radius of the action space is a hyper-parameter related to the degree of pessimism.

Formally, Alg 3 is a practical implementation for PRC algorithm. Lines 1-2 learn a utility model and a behavior clone policy, which a standard supervised learning approach can achieve. The loss functions are set in Eq 3 below. In section 5, we empirically verify that training on a constrained action space is an efficient pessimistic learning approach, and the reinforcement learning is also much easier on a constrained action space.

$$ $\displaystyle\mathcal{L}_{u}(u,D)$ $\displaystyle=-\sum_{(\tau_{1},\tau_{2},\sigma)\in D}\log\frac{\exp(\sum_{(s,a )\in\tau^{*}}u(s,a))}{\exp(\sum_{(s,a)\in\tau_{1}}u(s,a))+\exp(\sum_{(s,a)\in \tau_{2}}u(s,a))}$ (3) $\displaystyle\mathcal{L}_{\pi}(\pi,D)$ $\displaystyle=\sum_{(s,a)\sim D}\|\pi(s)-a\|_{2}$ $$

Lines 3-5 implement an RL environment with a constrained action space. For a policy $\pi_{\text{clip}}$ supported on $\mathcal{S}$, $\mathcal{A}^{\prime}$, if it outputs an action $a^{\prime}\in\mathcal{A}^{\prime}$, then its corresponding actual policy outputs action $f(s,a^{\prime})$, and the state transition follows $\mathcal{P}(s,f(s,a^{\prime}))=\mathcal{P}^{\prime}(s,a^{\prime})$, where $\mathcal{P}^{\prime}(s,a^{\prime})$ is the state transition in Line 5. Therefore, we can search for the optimal clipped policy $\pi^{*}$ from $\Pi_{\mathcal{S},\mathcal{A}^{\prime}}$ on the MDP with state space $\mathcal{S}$, action space $\mathcal{A}^{\prime}$, state transition $\mathcal{P}^{\prime}$, and output the corresponding actual policy of $\pi^{*}$. In practice, we can solve this by applying the SAC or PPO algorithm on the MDP $\mathcal{M}^{\prime}=(\mathcal{S},\mathcal{A}^{\prime},\mathcal{P}^{\prime},
\hat{u})$.

Figure: Algorithm 3 PRC practical implementation

## 5 Experiments

### 5.1 Setup

Dataset:
Following previous studies Hejna & Sadigh (), we construct our offline preference dataset from D4RL benchmark Fu et al. (), and generate synthetic preference following the standard techniques in previous PBRL studies (Kim et al., ; Christiano et al., ). Based on the definition of a standard offline preference-based reinforcement learning setting in Section 3, given a reward-based dataset from D4RL, we construct a preference-based dataset through the following process:

- 1.
Randomly sample pairs of trajectory clips from the D4RL dataset. Following previous studies (Christiano et al., ), the length of the clip is set to be $20$ steps.
- 2.
For each pair of trajectory clips, compute the probability of a trajectory to be preferred based on the reward signals. To ensure consistency between different datasets, the reward signals are regularized to be bound in $[-1,1]$.
- 3.
For each pair of trajectory clips, randomly generate a preference label through a Bernoulli trial with the probability computed above.
- 4.
Return the preference dataset consisting of the trajectory clip pairs and the corresponding preference labels.

We consider various datasets from D4RL to represent the general learning scenarios. The robot control environments we choose include ‘Hopper’, ‘HalfCheetah’, and ‘Walker’. The types of trajectories we choose include ‘Medium’, ‘Medium-Replay’, and ‘Medium-Expert’. To make the number of state-action in the dataset aligned with that of the D4RL benchmark, the total number of trajectory pairs with a preference label is $1\times 10^{6}$.

Baseline Algorithms:
Here, we consider multiple two-step learning algorithms as baselines and a state-of-the-art attack in the related offline PBRL setting, which are listed below:

- 1.
Two-step learning with KL-regularization: We adopt the traditional two-step learning baseline with KL-regularization as introduced in Section 3. To make it a stronger baseline and the comparison fairer, we allow the algorithm to initialize from the behavior clone policy.
- 2.
Naive two-step learning: To show a naive baseline where pessimistic learning is entirely absent, we adopt a naive two-step learning baseline, which is the same as the traditional two-step learning above except that there is no regularization at all during the reinforcement learning step. Note that we also allow the algorithm to initialize from the behavior clone policy in this baseline.
- 3.
Reward modeling followed by standard offline RL algorithms: Another simple yet efficient two-step learning approach for offline PBRL problems is combining reward modeling with a standard reward-based offline RL algorithm. First, a reward model is learned from the preference dataset. Then, the reward model is used to provide scalar reward labels to the state-action in the dataset. Finally, we apply a standard offline RL algorithm IQL Kostrikov et al. () training on the dataset with scalar reward labels.
- 4.
Oracle: The oracle is trained with true rewards instead of preference labels. Here, we apply IQL training on the base D4RL dataset with true reward signals. Note that the information from the reward signals is strictly more than that from the preference signals in this case. The oracle should be considered as an upper bound on the performance of an offline PBRL algorithm.
- 5.
Inverse preference learning (IPL) Hejna & Sadigh (): IPL is the state-of-the-art learning algorithm for a related offline PBRL setting that requires a preference dataset and a behavior dataset. To avoid underestimating the performance of IPL, we apply this algorithm to our PBRL setting and allow it to check the behaviors in the full D4RL dataset.

To ensure a fair comparison, all the methods that require reward modeling share the same learned reward model during training.

Training setup: To learn a reward model, we follow a standard supervised learning framework. Specifically, we use a multilayer perceptron (MLP) structure for the neural network to approximate the utility model. The neural network consists of $3$ hidden layers, and each has $64$ neurons. We use a tanh function as the output activation so that the output is bound between $[-1,1]$.

To train a deterministic behavior clone policy, the neural network we use to represent the clone policy has the same structure as that for the utility model. To train a stochastic behavior clone policy, the network outputs the mean and standard deviation of a Gaussian distribution separately. The network is an MLP with $3$ hidden layers each with $64$ neurons. The last layer is split into two for the two outputs. For the mean output, we use a linear function for the last layer. For the standard deviation, we use a linear function and an exp activation for the last layer.

In the reinforcement learning step, we use either the SAC algorithm or the PPO algorithm, depending on the dataset. The neural network for the actor is the same as the network for training a stochastic behavior clone policy. The neural network for the critic is the same as the network for training the reward model.

### 5.2 Learning Efficiency Evaluation

Here, we compare the efficiency of PRC against other baseline algorithms on different datasets. To straightforwardly compare the performance of different learning algorithms, we show the performance of the best policy learned by a method during training.

The scores in Table 1 are the standard D4RL score of the learned policies. The results show that the PRC algorithm has high learning efficiency. It is generally more efficient than other baselines and sometimes even competes with the oracle. Our algorithm performs better than other two-step learning baselines initialized from the behavior clone policy. This observation indicates that starting from the behavior clone policy or its neighborhood is not enough to achieve high learning efficiency, and training on the constrained action space is the key to high learning efficiency for PRC.

**Table 1: Comparison between the performance of different learning methods.**
| Dataset | Oracle | PRC | Behavior Clone | Naive two-step | KL two-step | IPL | RM |
| --- | --- | --- | --- | --- | --- | --- | --- |
| HalfCheetah-Medium | 47.3 $\pm$ 0.2 | 47 $\pm$ 0.5 | 41.7 $\pm$ 1.0 | 40.1 $\pm$ 0.5 | 41.9 $\pm$ 0.1 | 42.7 $\pm$ 0.1 | 43.2 $\pm$ 0.1 |
| HalfCheetah-Medium-Replay | 46.1 $\pm$ 0.1 | 43.3 $\pm$ 0.2 | 32.9 $\pm$ 9.2 | 31.9 $\pm$ 0.8 | 32.9 $\pm$ 0.9 | 34.9 $\pm$ 3.1 | 40.1 $\pm$ 0.7 |
| HalfCheetah-Medium-Expert | 92.1 $\pm$ 0.7 | 78.4 $\pm$2.9 | 44.5 $\pm$ 3.6 | 40.9 $\pm$ 0.2 | 40.4 $\pm$ 0.5 | 41.7 $\pm$ 1.0 | 48.2 $\pm$ 0.8 |
| Hopper-Medium | 76.1 $\pm$ 1.2 | 71 $\pm$ 7.5 | 51.4 $\pm$ 3.9 | 67.5 $\pm$ 2.4 | 73.5 $\pm$ 4.0 | 72.4 $\pm$ 7.4 | 67.2 $\pm$ 0.3 |
| Hopper-Medium-Replay | 76.7 $\pm$ 5.3 | 47 $\pm$ 19.5 | 40.8 $\pm$ 8.5 | 49.4 $\pm$ 7.4 | 53.1 $\pm$ 2.2 | 39.5 $\pm$ 17.5 | 32.2 $\pm$ 0.4 |
| Hopper-Medium-Expert | 113.1 $\pm$ 0.4 | 100.2 $\pm$ 9.5 | 46.7 $\pm$ 10.9 | 67.3 $\pm$ 7.1 | 71.4 $\pm$ 10.9 | 76.9 $\pm$ 8.1 | 97.1 $\pm$ 5.2 |
| Walker2d-Medium | 84.5 $\pm$ 0.3 | 84.4 $\pm$ 0.8 | 73.5 $\pm$ 1.7 | 81 $\pm$ 0.8 | 79.5 $\pm$ 2.1 | 80.8 $\pm$ 1.6 | 81.9 $\pm$ 2.5 |
| Walker2d-Medium-Replay | 83.1 $\pm$ 2.3 | 87.9 $\pm$ 6.1 | 11.6 $\pm$ 0.6 | 49.8 $\pm$ 7.3 | 45.4 $\pm$ 0.1 | 49.3 $\pm$ 3.5 | 71.8 $\pm$ 6.4 |
| Walker2d-Medium-Expert | 111.5 $\pm$ 0.4 | 110.4 $\pm$ 0.6 | 95.9 $\pm$ 3.1 | 93.4 $\pm$ 4.9 | 92.1 $\pm$ 2.3 | 107.5 $\pm$ 3.0 | 105.7 $\pm$ 6.9 |
| Sum Totals | 730.5 | 669.6 | 439 | 521.3 | 530.2 | 545.7 | 587.4 |

Next, we empirically analyze why PRC should be an efficient learning algorithm from the aspect of pessimistic learning and reduced reinforcement learning complexity.

### 5.3 Pessimism Effectiveness

Here, we empirically verify that by training on a constrained action space, the PRC algorithm achieves effective pessimistic learning.

In Figure 1, we show some representative examples. For the PRC algorithm, during training, the performance trend of the learned policies measured on the learned reward model is aligned with that measured on the true reward. This suggests that the pessimistic learning in the PRC algorithm is effective: it mainly considers the policies that are supported by the dataset so that the agent can evaluate their relative performance accurately. In contrast, for naive two-step learning and KL-regularized two-step learning, we find multiple cases where the trends can even be opposite when evaluated on the reward model and on the true reward. These results suggest pessimistic learning is not efficient enough in the baseline methods compared to PRC.

Figure: Figure 1: Comparison between the trend of the performance of the learned policies on the learned (simulated performance) and true reward models (true performance) during training. An algorithm is not pessimistic enough if the two trends are not aligned.
Refer to caption: extracted/5954829/figures/reward_hack.png

### 5.4 Reinforcement Learning Efficiency on Constrained Action Space

Here, we empirically verify that reinforcement learning is much easier on the constrained action space. In this case, we focus on the performance of the learned policies evaluated on the learned reward model.

We observe that in most cases, the performance of the learned policies in the PRC method is much higher than in the baseline two-step learning methods. In Figure 2, we show some representative examples. It is clear that while the baseline method struggles with low-performing policies, the PRC method learns policies of much higher performance.

Figure: Figure 2: Comparison between the performance of the learned policies on the learned reward models during training. The reinforcement learning complexity is less in a setting if the simulated performance is high.
Refer to caption: extracted/5954829/figures/complexity.png

### 5.5 Reinforcement Learning Complexity in PBRL

Here, we empirically show that training on a reward model learned from preference signals is harder than that from true reward signals. Given the same D4RL dataset, we train two reward models based on the true reward signals and synthetic preference signals. Then, we train an efficient RL algorithm on the two rewards and compare the learning efficiency in the two cases.

The results in Figure 3 show that when learning on the reward model trained from true rewards, the RL agent can quickly learn some policies that have high performance on the reward model. In comparison, it can take much longer for the agent to find a good policy on the reward model from preference signals, and sometimes, the agent may not be able to find a decent policy after a lot of training epochs. This indicates that it is harder to learn from a reward model that is trained on preference signals instead of true reward signals.

Figure: Figure 3: For each dataset, a pair of reward models are trained on the true rewards and preference signals. The same RL algorithm is applied to learn on both reward models. The learning difficulty on a reward model is less if the PPO algorithm can learn better policies according to the reward model.
Refer to caption: extracted/5954829/figures/challenge.png

## 6 Conclusion and Limitation

In this work, we propose a novel two-step learning algorithm PRC for the offline PBRL setting. We empirically show that the PRC has high learning efficiency and provide evidence for why it is a more efficient two-step learning algorithm than others. Our framework is limited to the typical offline learning setting and does not answer the question of which trajectories are more worthy of receiving preference labels. Our experimental evaluation is limited to continuous control problems, the standard benchmark in RL studies.

## 7 Reproducibility

In the main paper, we explain the setting of the problem we study. The codes we use for the experiments can be found in the supplementary materials.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. The PCA algorithm is mentioned in Section 5.3, but it is not explained or referenced earlier in the paper. Could you clarify what the PCA algorithm refers to in this context, and how it relates to the overall methodology?
2. In Section 5.4, it is stated that "the reinforcement learning complexity is reduced when the simulated performance is high." However, I observed that the baseline performance is also quite high. Could you provide further explanation on how we should interpret Figure 2 and the relationship between simulated performance and RL complexity in this context?
3. In the experiments within Section 5.2, your method significantly exceeds the upper bound you set for the Walker2d-Medium-Replay dataset. Given that the action space is constrained, this result seems theoretically unlikely. Could you clarify how your method achieves such results, and whether this outcome is consistent with the constraints imposed by your approach?
4. In Section 4, it appears that Algorithm 3 does not directly address a hard constraint problem during the reinforcement learning process. Instead, it seems to perform a data processing step directly on the dataset. Could you explain how this aligns with or addresses the hard constraints mentioned earlier in the method description?
soundness: 2
strengths: 1. **Strong Empirical Performance** The empirical results presented in the experiment section demonstrate that PRC significantly outperforms other baseline methods.
2. **Higher Training Efficiency** PRC shows superior learning efficiency compared to other baselines. This efficiency is well-supported by evidence, clearly explaining why PRC is a more effective two-step learning algorithm.
3. **Satisfactory Writing** The overall structure and writing of the paper are satisfactory. It is well-organized, reader-friendly, and generally easy to follow.
summary: The author discusses the success of preference-based reinforcement learning (PBRL) in offline settings, particularly in industrial applications like chatbots. A common approach in this area is a two-step learning framework, where reinforcement learning follows a reward modeling step. However, this method faces challenges related to reward hacking and the complexity of reinforcement learning. The author identifies that these challenges stem from state-actions not supported by the dataset, as these state-actions are unreliable and complicate the learning process. To address this issue, the author proposes a novel method called PRC (preference-based reinforcement learning with constrained actions), which limits the reinforcement learning agent to optimizing within a constrained action space that excludes out-of-distribution state-actions. The method is empirically shown to achieve high learning efficiency across various robotic control datasets.
weaknesses: 1. **Insufficient Novelty.**
The contributions and novelty of using a two-step learning framework in the PBRL problem are limited. The primary innovation in this work lies in the use of a constrained action space. However, 1) this modification to the original PBRL objective appears to be marginal, and 2) the paper lacks both theoretical and principled justification for the overall performance improvements.
2. **Ambiguous Experimental Results.**
Some of the experimental results and figures are difficult to interpret, leading to confusion. There is a lack of in-depth analysis regarding the model's performance and the underlying reasons for its superior results. A more detailed study is necessary to clarify these points (please refer to the specific question below).
3. **Technical Issues.**
The core of the proposed method is to constrain the action space to include only actions with a high probability of being sampled by a behavior cloning policy. However, it remains unclear how the method guarantees that policies with higher performance than those in the dataset can always be found. This raises concerns about the robustness of the approach.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see Weaknesses.
soundness: 2
strengths: The authors identify and address two key issues in conventional two-step learning algorithms for offline PbRL. The paper is well-motivated and well-organized.
summary: This paper introduces PRC, a two-step learning algorithm for offline PbRL that addresses two major challenges, reward hacking and the complexity of RL with preference feedback. PRC addresses these issues by constraining the agent’s action space to actions with a high probability of being sampled from the dataset’s behavior policy. The authors empirically demonstrate PRC’s effectiveness across various robotic control tasks, showing that it achieves better performance and learning efficiency compared to other two-step learning algorithms.
weaknesses: 1. The authors should expand the related work section, as recent key studies on both online and offline PbRL, as well as RLHF for LLMs, are currently missing.

2. Especially, DPPO [1] is a two-step algorithm for offline PbRL that learns a preference predictor followed by direct policy learning through a newly proposed preference score based on policy-segment distance, thereby avoiding reward learning and RL. Additionally, CPL [2] directly learns a policy without both reward modeling and RL. Both approaches address the same challenges as this paper, indicating that this is not the first study to tackle these issues.

3. I am concerned that with unbounded regularization, as mentioned in line 240, optimization might become unstable. Additionally, could excessive conservatism risk resulting in a policy that is unable to achieve high rewards?

4. Further discussion on function $f$ and $\mathcal{P}^\prime$ are necessary to improve clarity and avoid confusion.

5. I recommend including training and evaluation configurations, such as hyperparameters for each method and details on the evaluation process (e.g., number of runs over random seeds), in the appendix for reproducibility.

6. The paper lacks quantitative results for experiments on pessimism effectiveness. Additionally, if these results are based on a single run, I suggest conducting multiple runs and illustrating the findings. This suggestion applies to all the experiments in section 5.3, 5.4, 5.5.

7. Several SOTA baselines in offline PbRL, such as PT [3], DPPO [1], and CPL [2], are missing from the evaluation.

8. Sections 5.4 and 5.5 include observations but provide few discussion of the results.

[1] An, Gaon, et al. "Direct preference-based policy optimization without reward modeling." Advances in Neural Information Processing Systems 36 (2023): 70247-70266.

[2] Hejna, Joey, et al. "Contrastive preference learning: Learning from human feedback without rl." arXiv preprint arXiv:2310.13639 (2023).

[3] Kim, Changyeon, et al. "Preference transformer: Modeling human preferences using transformers for rl." arXiv preprint arXiv:2303.00957 (2023).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Please see the above weakness.
soundness: 1
strengths: The paper considers a significant issue in offline PbRL: reward over-optimization, which should be appreciated. To assess the performance of the proposed method, the authors conducted some empirical evaluations on D4RL environments.
summary: The paper proposes constraining the action space to make the policy not far from the dataset for offline preference-based reinforcement learning (PbRL). The authors claim that limiting the action space can help reduce reward hacking and RL complexity. Some experiments conducted on D4RL environments show the performance of the proposed method, compared with some simple baseline methods.
weaknesses: 1. The core concept of constraining the action space to mitigate reward hacking and over-optimization is not entirely novel. The modification over conventional offline PbRL is very slignt and lacks considerations. Merely restricting the action space based on the offline dataset could introduce several issues, such as limiting the policy’s ability to explore alternative, potentially optimal actions. This paper does not account for the potential negative consequences of such a simplistic restriction, which may lead to unintended limitations in policy performance and adaptability.
2. The paper lacks a clear structure, with an excessive focus on background information and well-known techniques. The authors should place greater emphasis on detailing their own methods, providing both theoretical insights and empirical analysis to strengthen the contribution of their approach.
3. The experiments are not convincing and comprehensive, as the authors compare their method only to basic baselines in a limited set of simple environments. Additionally, the presentation of the results is unclear. For example, the authors show the performance of the best policy learned by a method during training, which is actually improper. In Table 1, certain values are bolded without explanation. Moreover, the final scores should be reported using normalized rewards rather than raw sums to enable more meaningful comparisons. Furthermore, Figures 1, 2, and 3 lack consistency in their x-axis labeling, making cross-comparisons difficult. Consolidating the results for each task into a single figure and displaying all methods would improve clarity and readability.
4. Overall, this paper does not meet the standards expected for this conference. Significant improvements are needed in both the methodology and experimental evaluation to adequately address the mentioned challenges in offline PbRL.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: 1. How do the authors estimate the threshold $p$ for the probability density of the behavior policy? Also, as I recognize this threshold as an important parameter, why there are no empirical studies on altering this $p$?
2. I recommend the authors provide explicit expressions (e.g., a math equation) of simulated performance and true performance in the experiment section (Sec 5.1).
3. What is BR-PAC in Figure 2? Why do the authors evaluate the performance of the learned policies on the learned reward models instead of ground-truth reward models? Why does the simulated performance go down in the rightmost subfigure in Figure 2? I do not quite understand the caption, ‘The reinforcement learning complexity is less in a setting if the simulated performance is high.’ I invite the authors to clarify this point.
soundness: 2
strengths: PRC handles pessimism by constraining actions to those covered in the dataset, improving policy stability.
summary: The paper proposes PRC (Preference-Based Reinforcement Learning with Constrained Actions) to address challenges in two-step offline preference-based reinforcement learning (PbRL). PRC mitigates reward hacking and instability by constraining the action space to well-represented actions in the dataset. This reduces the complexity of the reinforcement learning phase and improves efficiency. Empirical evaluations on robotic control tasks (D4RL benchmark) demonstrate PRC’s superior performance compared to traditional methods.
weaknesses: 1. This paper is indeed incremental, and the contributions (both theoretically and empirically) are not sufficient enough to be presented at this conference. For theory, the authors claim their method can mitigate reward hacking and reduce the complexity of RL, but no quantified analysis is presented. For the experiment, the authors do not offer indicators of the two key contributions.

2. The analysis in Section 4.2 is not enough. More content should be included, such as theoretical analysis of improved efficiency regarding the behavior policy probability density threshold $p$ and the extent to which reward hacking can be mitigated. 

3. Further, the experiments need more clarity.