SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gaon An*
white0234@m11ab.snu.ac.kr
Junhyeok Lee*
riman314@mllab.snu.ac.kr
Xingdong Zuo
NAVER
xingdong.zuo@navercorp.com
Norio Kosaka
Line Corporation
kosaka.norio@linecorp.com
Kyung-Min Kim
kyungmin.kim.ml@navercorp.com
Hyun Oh Song†
hyunoh@mllab.snu.ac.kr
Preference-based reinforcement learning (PbRL) is an approach that enables RL agents to learn from preference, which is particularly useful when formulating a reward function is challenging. Existing PbRL methods generally involve a two-step procedure: they first learn a reward model based on given preference data and then employ off-the-shelf reinforcement learning algorithms using the learned reward model. However, obtaining an accurate reward model solely from preference information, especially when the preference is from human teachers, can be difficult. Instead, we propose a PbRL algorithm that directly learns from preference without requiring any reward modeling. To achieve this, we adopt a contrastive learning framework to design a novel policy scoring metric that assigns a high score to policies that align with the given preferences. We apply our algorithm to offline RL tasks with actual human preference labels and show that our algorithm outperforms or is on par with the existing PbRL methods. Notably, on high-dimensional control tasks, our algorithm surpasses offline RL methods that learn with ground-truth reward information. Finally, we show that our algorithm can be successfully applied to fine-tune large language models.
Deep reinforcement learning has been successful in solving various decision-making tasks where a well-defined reward function is available [34, 35, 50, 5, 54]. However, in many real-world tasks, it can be challenging to design a quantitative reward function that accurately reflects the desired behavior, particularly when the task involves human evaluation. Preference-based RL (PbRL) seeks to provide an alternative solution by enabling agents to learn from preference information between pairs of trajectory segments [1, 10]. PbRL has gained considerable interest in recent years as making a relative judgment is much easier than providing a real-valued score, which makes human feedback much more viable [39, 38, 52, 45, 56].
*First two authors have equal contributions
Figure 1: An overview of the difference between our approach (below) and the baselines (top). Our approach does not require modeling the reward from the preference predictor as our policy optimization algorithm can learn directly from preference labels.
Figure 2: Predicted reward vs. true reward on the Hopper environment when using a reward model from PbRL [27]. The reward model fails to accurately capture the underlying reward structure.
Recent PbRL methods take a two-step approach: they first learn a reward model from the given preference dataset and then run off-the-shelf reinforcement learning algorithms on top of the learned reward model [10, 31, 40]. However, acquiring an accurate reward model only from preference labels, typically provided by human teachers, poses a significant challenge as it is unclear how to extract the underlying reward structure from preference. Current methods rely on modeling the reward with certain specific assumptions, though there are some concerns regarding whether those assumptions hold in practice [13, 27].
Alternatively, predicting the preference itself is comparatively more straightforward since we have direct access to training labels, allowing us to leverage powerful techniques from supervised learning. Building upon this observation, we introduce a PbRL algorithm that bypasses the need for reward function modeling by directly learning from preference labels. Our approach begins by devising a policy scoring metric that assigns high scores to policies aligning with the provided preference dataset. Concretely, the PbRL objective is formulated as a contrastive learning problem, guiding the learned policy to be closer to more preferred trajectory segments while distancing itself from the less preferred ones [9, 20]. Furthermore, we enhance the performance of the preference predictors from previous works by introducing a novel prediction smoothness regularizer. Experiment results on offline RL settings with actual human preference labels show that the proposed algorithm outperforms or is on par with the baselines on all of the tasks considered [16]. Notably, in high-dimensional control tasks, our algorithm outperforms offline RL methods that utilize ground-truth reward information. Moreover, our preliminary experiments show that our algorithm can be successfully applied for fine-tuning large language models. Our official code is available at https://github.com/snu-mllab/DPPO.
2.1 Preference-based reinforcement learning
Reinforcement learning considers an environment formulated as a Markov Decision Process (MDP) defined by a tuple  $(\mathcal{S},\mathcal{A},T,\mathcal{R},p_0,H)$ , where  $\mathcal{S}$  is a state space,  $\mathcal{A}$  is an action space,  $T(\mathbf{s}'|\mathbf{s},\mathbf{a})$  is the state transition dynamics,  $\mathcal{R}(\mathbf{s},\mathbf{a})$  is the reward function,  $p_0(\mathbf{s})$  is the initial state distribution, and  $H$  is the time horizon. The goal of reinforcement learning is to learn a policy  $\pi$  that optimizes the expected return:
$$
J (\pi) = \mathbb {E} _ {\mathbf {s} _ {0} \sim p _ {0}, \mathbf {a} _ {t} \sim \pi (\cdot | \mathbf {s} _ {t}), \mathbf {s} _ {t + 1} \sim T (\cdot | \mathbf {s} _ {t}, \mathbf {a} _ {t})} \left[ \sum_ {t = 0} ^ {H} r _ {t} \right].
$$
Conventional RL assumes the reward information  $(r_t)$  is given and uses this to optimize their policy. However, finding a suitable reward metric can be costly in many real-world scenarios.
For example, if the goal task is to train a robot to scramble an egg, it would be unclear how to design a reward function that captures all the desired properties. Instead, PbRL assumes the supervision is given in the form of preference. Concretely, for a pair of trajectory segments  $(\sigma^0,\sigma^1)$  where  $\sigma^i = (\mathbf{s}_0^i,\mathbf{a}_0^i,\mathbf{s}_1^i,\mathbf{a}_1^i,\dots ,\mathbf{s}_k^i,\mathbf{a}_k^i)$ , a (human) teacher indicates which segment it prefers. The preference label  $y$  is given as 0 if  $\sigma^0$  is preferred, i.e.,  $\sigma^0\succ \sigma^1$ . In a similar manner,  $y$  is set to 1 if  $\sigma^1$  is preferred and 0.5 if the two are equally preferred.
To learn a reward model  $\widehat{r}$ , prior works assume the preference depends on the value of the underlying rewards summed over each timestep [10, 6, 31, 40]:
$$
\widehat {P} [ \sigma^ {0} \succ \sigma^ {1}; \psi ] = \frac {\exp \left(\sum_ {t = 0} ^ {k} \widehat {r} \left(\mathbf {s} _ {t} ^ {0} , \mathbf {a} _ {t} ^ {0} ; \psi\right)\right)}{\exp \left(\sum_ {t = 0} ^ {k} \widehat {r} \left(\mathbf {s} _ {t} ^ {0} , \mathbf {a} _ {t} ^ {0} ; \psi\right)\right) + \exp \left(\sum_ {t = 0} ^ {k} \widehat {r} \left(\mathbf {s} _ {t} ^ {1} , \mathbf {a} _ {t} ^ {1} ; \psi\right)\right)},
$$
where  $\psi$  denotes the learnable parameters of the reward model. Given a dataset  $D_{\mathrm{pref}}$  of preference triples  $(\sigma^0,\sigma^1,y)$ , the reward model is trained by minimizing the cross-entropy loss between the preference predictions and the ground-truth labels:
$$
\ell_ {\widehat {r}} (\psi ; \mathcal {D} _ {\text {p r e f}}) = - \underset {(\sigma^ {0}, \sigma^ {1}, y) \sim \mathcal {D} _ {\text {p r e f}}} {\mathbb {E}} \left[ (1 - y) \log \widehat {P} [ \sigma^ {0} \succ \sigma^ {1}; \psi ] + y \log \widehat {P} [ \sigma^ {1} \succ \sigma^ {0}; \psi ] \right]. \tag {1}
$$
After training, any standard RL algorithm can be used to maximize the expected return under the learned reward model. Especially in the offline PbRL setting, we assume there exists a small dataset  $\mathcal{D}_{pref}$  with preference labels along with a much larger unlabeled dataset  $\mathcal{D}$  without any reward or preference labels [27, 49]. A typical approach involves utilizing  $\mathcal{D}_{pref}$  to learn the reward model and applying the model to label  $\mathcal{D}$ .
2.2 Contrastive learning
The goal of contrastive learning is to learn representations where similar sample pairs are close to each other while dissimilar pairs are far apart. For an anchor sample  $\mathbf{x}$  (e.g., an image), suppose we have a positive sample  $\mathbf{x}^{+}$  (e.g., the same image with data augmentation applied) and a set of negative samples  $\{\mathbf{x}_i^{-}\}_{i=1}^m$  (e.g., samples from other images). To learn an encoder  $f$ , contrastive learning minimizes the following loss:
$$
\ell_ {f} \left(\mathbf {x}, \mathbf {x} ^ {+}, \left\{\mathbf {x} _ {i} ^ {-} \right\} _ {i = 1} ^ {m}\right) = - \log \frac {\exp \left(f (\mathbf {x}) ^ {\mathsf {T}} f (\mathbf {x} ^ {+})\right)}{\exp \left(f (\mathbf {x}) ^ {\mathsf {T}} f (\mathbf {x} ^ {+})\right) + \sum_ {i = 1} ^ {m} \exp \left(f (\mathbf {x}) ^ {\mathsf {T}} f (\mathbf {x} _ {i} ^ {-})\right)},
$$
where a dot product of two representations  $(e.g., f(\mathbf{x})^{\top}f(\mathbf{x}^{+}))$  is considered as the similarity score between the two samples.
3 Learning directly from preference
While prior PbRL approaches adopt a two-step procedure involving the construction of a reward model using the preference data, the fidelity of the learned reward model in reflecting the original reward function remains uncertain. The main challenge lies in extracting the underlying rewards from the given preference. Previous works assume that the preference for a trajectory segment can be represented as an average of the underlying rewards [10, 21]. However, considering PbRL typically assumes human teachers provide the preference labels, it is unclear whether human preferences can also be modeled in this manner. For instance, humans may focus on a specific subset of the segment while ignoring other parts [24, 27]. Empirically, we find that the reward models learned from preferences often fail to accurately capture the underlying reward structure, as illustrated in Figure 2. This issue is problematic since within the current PBRL framework, the quality of the learned policy relies heavily on the quality of the learned rewards.
Meanwhile, predicting the preference itself is a more straightforward task as we have explicit labels to train with and can therefore apply powerful tools from supervised learning [18, 11, 36]. Based on this observation, our goal is to introduce a PbRL algorithm that directly learns with preference information without the need for reward modeling, as illustrated in Figure 1. To achieve this, we design a policy scoring metric that yields a high score when a policy aligns with the given preference dataset. In other words, we assume that a desirable policy should be closer to  $\sigma^0$  than  $\sigma^1$  if  $\sigma^0\succ \sigma^1$ . To formulate this property into an estimizable objective, we first define the closeness between a policy and a trajectory segment.
Figure 3: An overview of the score calculation process. To score a given policy, (1) the first step is to calculate the distance between each transition tuple and the policy. (2) Second, these distances are aggregated to a policy-segment distance through a predefined aggregation function. (3) Finally, we obtain the score value by contrasting the policy-segment distances according to their preference.
3.1 Policy-segment distance
We define the distance between a policy and a trajectory segment as an aggregation of the distance between a policy and each transition tuple in the trajectory segment. Concretely, the policy-segment distance can be expressed as
$$
d \left(\pi , \sigma^ {i}\right) = \operatorname {A G G} \left(d _ {\mathbf {s a}} \left(\pi , \mathbf {s} _ {0} ^ {i}, \mathbf {a} _ {0} ^ {i}\right), \dots , d _ {\mathbf {s a}} \left(\pi , \mathbf {s} _ {k} ^ {i}, \mathbf {a} _ {k} ^ {i}\right)\right),
$$
where  $d_{\mathbf{sa}}$  denotes the policy-transition tuple distance and AGG denotes an aggregation function. There can be multiple ways for instantiating  $d_{\mathbf{sa}}$ . For simplicity, we employ the expected  $\ell_2$  distance between the policy action and the trajectory action:  $d_{\mathbf{sa}}(\pi, \mathbf{s}, \mathbf{a}) = \mathbb{E}_{\tilde{\mathbf{a}} \sim \pi(\cdot|\mathbf{s})} [||\tilde{\mathbf{a}} - \mathbf{a}||_2]$ . Similarly, we opt to use the mean operator as the aggregation function. To sum up, our policy-segment distance function becomes
$$
d(\pi ,\sigma^{i}) = \frac{1}{k + 1}\sum_{t = 0}^{k}\left(\underset {\tilde{\mathbf{a}}\sim \pi (\cdot |\mathbf{s}_{t}^{i})}{\mathbb{E}}\left[||\tilde{\mathbf{a}} -\mathbf{a}_{t}^{i}||_{2}\right]\right).
$$
3.2 Preference score metric
Using the policy-segment distance defined above, we can now build a score metric that satisfies our desired property. Given a preference triple  $(\sigma^0,\sigma^1,y)$ , assume that  $\sigma^0$  is preferred over  $\sigma^1$ , i.e.,  $y = 0$ . We want to assign a high score if a policy is closer to  $\sigma^0$  than  $\sigma^1$ . This condition can be expressed in terms of the policy-segment distance as  $\sigma^0\succ \sigma^1\Rightarrow d(\pi ,\sigma^0) < d(\pi ,\sigma^1)$ . To capture this condition across multiple segment pairs into a single metric, we adopt a contrastive learning formulation:
$$
S (\theta ; \mathcal {D} _ {\text {p r e f}}) = \underset {(\sigma^ {0}, \sigma^ {1}, y) \sim \mathcal {D} _ {\text {p r e f}}} {\mathbb {E}} \left[ (1 - y) \cdot s \left(\pi_ {\theta}, \sigma^ {0}, \sigma^ {1}\right) + y \cdot s \left(\pi_ {\theta}, \sigma^ {1}, \sigma^ {0}\right) \right] \tag {2}
$$
$$
\mathrm {s . t .} s (\pi , \sigma^ {i}, \sigma^ {j}) = \log \frac {\exp (- d (\pi , \sigma^ {i}))}{\exp (- d (\pi , \sigma^ {i})) + \exp (- d (\pi , \sigma^ {j}))},
$$
where  $\theta$  denotes the learnable parameters of the policy  $\pi$ . In terms of contrastive learning, we can interpret  $\exp \left(-d\left(\pi_{\theta},\sigma^i\right)\right)$  as representing the similarity between the policy  $\pi_{\theta}$  and the segment  $\sigma^i$ . Then,  $\pi_{\theta}$ ,  $\sigma^0$ , and  $\sigma^1$  are each considered the anchor, the positive sample, and the negative sample, assuming  $\sigma^0\succ \sigma^1$ . Note that a high score can only be achieved if the policy is closer to more preferred trajectory segments.
While Equation (2) is sufficient to express our desired property, a minor drawback is that the score function is indifferent to the increase or decrease of the distances in the same magnitude. To understand this in detail, let us first denote  $d^{i} = d(\pi_{\theta}, \sigma^{i})$  for brevity. Then, for a preference triple  $(\sigma^{0}, \sigma^{1}, y)$  with  $y = 0$ ,
$$
s \left(\pi_ {\theta}, \sigma^ {0}, \sigma^ {1}\right) = - d ^ {0} - \log \left(\exp (- d ^ {0}) + \exp (- d ^ {1})\right) \approx \max  \left\{0, d ^ {0} - d ^ {1} \right\}. \tag {3}
$$
Figure 4: Predicted preference of overlapping segments from a single trajectory. In detail, we measure  $\widehat{P}[\sigma^i \succ \sigma^{i+1}]$ , where  $\sigma^i = (\mathbf{s}_i, \mathbf{a}_i, \ldots, \mathbf{s}_{i+k}, \mathbf{a}_{i+k})$ .
Algorithm 1 Direct Preference-based Policy Optimization
Input: Unlabeled dataset  $\mathcal{D}$ , preference dataset  $\mathcal{D}_{\mathrm{pref}}$ , learning rate  $\eta_{\phi}$  and  $\eta_{\theta}$ , number of training steps  $M$  and  $N$ , and regularization parameters  $\lambda, \nu$ .
Initialize network parameters  $\phi$  and  $\theta$
for step  $= 1$  to  $M$  do
Update the predictor parameter:
$$
\phi \leftarrow \phi - \eta_ {\phi} \nabla_ {\phi} \ell_ {\hat {P}} (\phi ; \mathcal {D} _ {\text {p r e f}}, \mathcal {D})
$$
for step  $= 1$  to  $N$  do
Update the policy parameter:
$$
\theta \leftarrow \theta + \eta_ {\theta} \nabla_ {\theta} S (\theta ; \mathcal {D}, \phi , \lambda)
$$
Therefore, the score value remains the same when the distances increase or decrease in the same magnitude. In other words, the score for the distances  $(d^0,d^1)$  and  $(d^{0} + \alpha ,d^{1} + \alpha)$  are identical, indicating there is no penalty when a policy deviates from even the preferred trajectory segment. To solve this, we add a regularizing factor  $\lambda \in (0,1)$  that decreases the score when the overall scale of the policy-segment distances increases:
$$
S (\theta ; \mathcal {D} _ {\text {p r e f}}, \lambda) = \underset {(\sigma^ {0}, \sigma^ {1}, y) \sim \mathcal {D} _ {\text {p r e f}}} {\mathbb {E}} \left[ (1 - y) \cdot s (\pi_ {\theta}, \sigma^ {0}, \sigma^ {1}; \lambda) + y \cdot s (\pi_ {\theta}, \sigma^ {1}, \sigma^ {0}; \lambda) \right] \tag {4}
$$
$$
\begin{array}{l} \text {s . t .} s (\pi , \sigma^ {i}, \sigma^ {j}; \lambda) = \log \frac {\exp (- d (\pi , \sigma^ {i}))}{\exp (- d (\pi , \sigma^ {i})) + \exp (- \lambda d (\pi , \sigma^ {j}))}. \end{array}
$$
If we set  $\lambda$  smaller than 1 and plug in this new formulation to Equation (3), it is easy to find out that the score will decrease when the overall distances increase. The resulting score calculation process for Equation (4) is illustrated in Figure 3.
3.3 Policy optimization with preference predictor
We can directly optimize a policy with preference labels by maximizing the score function in Equation (4). However, to leverage the unlabeled dataset  $\mathcal{D}$ , we train a preference predictor using the labeled dataset  $\mathcal{D}_{\mathrm{pref}}$ . We formulate this process as a simple binary classification problem and use the cross-entropy loss to optimize the predictor  $\widehat{P}$ :
$$
\begin{array}{l} \ell_ {\widehat {P}} (\phi ; \mathcal {D} _ {\text {p r e f}}, \mathcal {D}) = - \underbrace {\underset {(\sigma^ {0} , \sigma^ {1} , y) \sim \mathcal {D} _ {\text {p r e f}}} {\mathbb {E}} \left[ (1 - y) \log \widehat {P} \left[ \sigma^ {0} \succ \sigma^ {1} ; \phi \right] + y \log \widehat {P} \left[ \sigma^ {1} \succ \sigma^ {0} ; \phi \right] \right]} _ {\text {P r e f e r e n c e C o r r e c t n e s s}} \tag {5} \\ + \nu \underbrace {\underset {(\sigma , \sigma^ {\prime}) \sim \mathcal {D}} {\mathbb {E}} \left[ \left(\widehat {P} [ \sigma \succ \sigma^ {\prime} ; \phi ] - 0 . 5\right) ^ {2} \right]} _ {\text {P r e f e r e n c e S m o o t h n e s s}}, \\ \end{array}
$$
where  $\phi$  denotes the learnable parameters of the preference predictor  $\widehat{P}$ . The first term resembles Equation (1), but the difference is that here we directly model the preference predictor instead of modeling the reward function. The second term is a smoothness regularizer which guides the predictor to have a similar preference against two largely overlapping segments. Concretely, given  $\sigma = (\mathbf{s}_i,\mathbf{a}_i,\dots ,\mathbf{s}_{i + k},\mathbf{a}_{i + k})$ , we sample  $\sigma^{\prime}$  from the same trajectory as  $(\mathbf{s}_{i + \alpha},\mathbf{a}_{i + \alpha},\dots ,\mathbf{s}_{i + \alpha +k},\mathbf{a}_{i + \alpha +k})$ , where  $\alpha \sim \mathcal{N}(0,m^2)$  with  $m\ll k$ . We observe that if no smoothness regularization is applied  $(\nu = 0)$ , the preference can vary significantly between two almost identical segments, as illustrated in Figure 4. This behavior is undesirable as a human teacher will unlikely be able to even identify the difference between the two segments.
Figure 5: DPPO vs. BC for two example trajectory segments on the walker2d-medium-replay dataset of D4RL Gym. DPPO successfully avoids the behavior from the unpreferred trajectory segment while BC also clones the unpreferred behavior.
After training the preference predictor with Equation (5), we can train our policy with the unlabeled dataset  $\mathcal{D}$  by sampling pairs of trajectory segments and labeling their preferences:
$$
\begin{array}{l} S (\theta ; \mathcal {D}, \phi , \lambda) = \underset {(\sigma^ {0}, \sigma^ {1}) \sim \mathcal {D}} {\mathbb {E}} \left[ (1 - \widehat {y}) \cdot s \left(\pi_ {\theta}, \sigma^ {0}, \sigma^ {1}; \lambda\right) + \widehat {y} \cdot s \left(\pi_ {\theta}, \sigma^ {1}, \sigma^ {0}; \lambda\right) \right], \\ \text {s . t .} \widehat {y} = \mathbb {1} \left\{\widehat {P} \left[ \sigma^ {0} \succ \sigma^ {1}; \phi \right] > 0. 5 \right\}. \\ \end{array}
$$
Algorithm 1 summarizes the full process of our PbRL algorithm. Figure 5 shows an example of two policies each learned using our algorithm and behavior cloning (BC), which shows that vanilla BC suffers from imitating the behavior of unpreferred trajectory segments while our algorithm successfully distances from it. We name our preference-based policy optimization algorithm as DPPO, an abbreviation for Direct Preference-based Policy Optimization. The sections below evaluate the performance of DPPO in the offline setting.
4.1 Offline RL experiment details
Following recent PbRL works, we evaluate our algorithm on the offline setting which assumes a large unlabeled dataset  $\mathcal{D}$  is given along with a much smaller preference-labeled dataset  $\mathcal{D}_{\mathrm{pref}}$  [49, 27]. We evaluate our algorithm on D4RL, a standard benchmark for offline RL, with preference datasets generated by actual human teachers [16]. We did not experiment on Antmaze, a widely used task in D4RL, due to a crucial bug in its environment implementation (please refer to Appendix F for more details). Below is a brief description of the tasks considered in our experiments:
Gym D4RL Gym provides datasets for the Gym locomotion environments where the goal task is to move forward while maintaining body balance. D4RL Gym contains various types of datasets each from a different data collection process. Among those datasets, we focus on the medium-replay and medium-expert datasets, which contain trajectories with the most diverse quality.
Adroit pen Adroit tasks involve controlling a 24-DoF robotic hand to perform tasks such as grabbing a pen or hammering a nail [44]. Adroit tests if offline RL methods can learn from human demonstrations on these high-dimensional tasks. We focus on the pen task since standard offline RL algorithms fail to achieve reasonable performance on other tasks, even with the reward information.
Kitchen Kitchen requires learning a 9-DoF robot manipulation task [19]. Similar to Adroit, the reward function is sparse and the dataset contains human demonstrations. However, Kitchen requires solving multiple sub-tasks sequentially (e.g., opening the microwave, then turning off the switch) in a single episode. While Kitchen has three types of datasets (complete, partial, and mixed), we consider the latter two as complete only contains successful trajectories with almost identical quality.
For the Gym hopper, Gym walker2d, and Adroit pen tasks, we utilize publicly available human preference datasets released by [27]. For the other tasks, we generate a new preference dataset as there are no preference datasets available. To collect the human preference datasets for each task, we strictly adhere to the protocol outlined by [27]. This involves defining a set of desirable behaviors and instructing the human teacher to label preferences accordingly. For example, a desired behavior in the Hopper environment would be to maintain body balance and prevent falling down. The size of the resulting preference datasets ranges from 100 to 500 samples, depending on the specific task. For more details regarding the dataset generation process, please refer to Appendix A.