SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joey Hejna
jhejna@cs.stanford.edu
Reward functions are difficult to design and often hard to align with human intent. Preference-based Reinforcement Learning (RL) algorithms address these problems by learning reward functions from human feedback. However, the majority of preference-based RL methods naively combine supervised reward models with off-the-shelf RL algorithms. Contemporary approaches have sought to improve performance and query complexity by using larger and more complex reward architectures such as transformers. Instead of using highly complex architectures, we develop a new and parameter-efficient algorithm, Inverse Preference Learning (IPL), specifically designed for learning from offline preference data. Our key insight is that for a fixed policy, the  $Q$ -function encodes all information about the reward function, effectively making them interchangeable. Using this insight, we completely eliminate the need for a learned reward function. Our resulting algorithm is simpler and more parameter-efficient. Across a suite of continuous control and robotics benchmarks, IPL attains competitive performance compared to more complex approaches that leverage transformer-based and non-Markovian reward functions while having fewer algorithmic hyperparameters and learned network parameters. Our code is publicly released<sup>1</sup>.
Reinforcement Learning (RL) has shown marked success in fixed and narrow domains such as simulated control [20] and game-playing [37]. When deploying RL in more complex settings, like in robotics or interaction with humans, one often runs into a critical bottleneck: the reward function. Obtaining reward labels in the real world can be complex, requiring difficult instrumentation [48, 57] and painstaking tuning [56] to achieve reasonable levels of sample efficiency. Moreover, despite extensive engineering, reward functions can still be exploited by algorithms in ways that do not align with human values and intents [21], which can be detrimental in safety-critical applications [5].
Instead of hand-designing reward functions, contemporary works have attempted to learn them through expert demonstrations [1], natural language [34], or human feedback [47, 3, 53]. Recently, reward functions learned through pairwise comparison queries—where a user is asked which of two demonstrated behaviors they prefer—have been shown to be effective in both control [13, 47, 32] and natural language domains [51]. This is often referred to as Reinforcement Learning with Human Feedback (RLHF). Reward functions learned via RLHF can directly capture human intent, while avoiding alternative and more expensive forms of human feedback such as expert demonstrations. Preference-based RL algorithms for RLHF often interleave reward-learning from comparisons with off-the-shelf RL algorithms.
While preference-based RL methods discover reward functions that are aligned with human preferences, they are not without flaws. Learned reward functions must have adequate coverage of
$^{1}$ https://github.com/jhejna/inverse-preference-learning
both the state and action space to attain good downstream performance. Consequently, learning the reward function can be expensive, usually requiring thousands of labeled preference queries. To mitigate these challenges, recent works have proposed improving learned reward functions by adding inductive biases before optimization with RL. Hejna and Sadigh [22] pretrain reward functions with meta-learning. Park et al. [43] use data augmentation. Early et al. [15] and Kim et al. [28] make the reward function non-Markovian using recurrent or large transformer sequence model architectures respectively. Such approaches increase the upfront cost of preference-based RL by using additional data or compute. Moreover, these techniques still combine reward optimization with vanilla RL algorithms. Ultimately, this just adds an extra learned component to already notoriously delicate RL algorithms, further increasing hyper-parameter tuning overhead. Preference-based RL approaches often end up training up to four distinct neural networks independently: a critic (with up to two networks), an actor, and a reward function. This can be problematic as prediction errors cascade from the reward function, to the critic, and ultimately the actor causing high variance in downstream performance. To address these issues, we propose a parameter-efficient algorithm specifically designed for preference-based RL that completely eliminates the need to explicitly learn a reward function. In doing so, we reduce both complexity and compute cost.
The key insight of our work is that, under a fixed policy, the  $Q$ -function learned by off-policy RL algorithms captures the same information as the learned reward function. For example, both the  $Q$ -function and reward function encode information about how desirable a state-action pair is. This begs the question: why do we need to learn a reward function in the first place? Our proposed solution, Inverse Preference Learning or IPL, is an offline RL algorithm that is specifically designed for learning from preference data. Instead of relying on an explicit reward function, IPL directly optimizes the implicit rewards induced by the learned  $Q$ -function to be consistent with expert preferences. At the same time, IPL regularizes these implicit rewards to ensure high-quality behavior. As a result, IPL removes the need for a learned reward function and its associated computational and tuning expense.
Experimentally, we find that even though IPL does not explicitly learn a reward function, it achieves competitive performance with complicated Transformer-based reward learning techniques on offline Preference-based RL benchmarks with real-human feedback. At the same time, IPL consistently exhibits lower variance across runs as it does not suffer from the errors associated with querying a learned reward model. Finally, under a minimal parameter budget, IPL is able to outperform standard preference-based RL approaches that learn an explicit reward model.
Our work builds upon literature in reward learning, preference-based RL, and imitation learning.
Reward Learning. Due to the challenges associated with designing and shaping effective reward signals, several works have investigated various approaches for learning reward functions. A large body of work uses inverse RL to learn a reward function from expert demonstrations [1, 41, 46], which are unfortunately difficult to collect [27, 2, 35] or often misaligned with true human preferences [7, 31]. Subsequently, reward learning techniques using other simpler forms of feedback such as scalar scores [29] and partial [38] or complete rankings [11, 8] have been developed. One of the simplest forms of human feedback is pairwise comparisons, where the user chooses between two options. Often, pairwise comparison queries are sampled using techniques from active learning [47, 9, 14]. However, to evaluate learned reward functions, these methods rely on either RL or traditional planning algorithms which are complex and computationally expensive. Our approach takes a simpler perspective that is parameter-efficient by combining reward and policy learning. Though it is not the focus of our work, IPL could additionally leverage active learning techniques for selecting preference data online.
Preference-based Deep Reinforcement Learning. Current approaches to preference based deep RL train a reward function, and then use that reward function in conjunction with a standard reinforcement learning algorithm [13, 33, 50]. Several techniques have been developed to improve the learned reward function, such as pre-training [25, 32], meta-learning [22], data augmentation [43], and non-Markovian modeling. Within the family of non-Markovian reward modeling [6], recent approaches have leveraged both LSTM networks [15] and transformers [28] for reward learning. But, these methods still rely on Markovian offline RL algorithms such as Implicit Q-Learning (IQL) [30] for optimization. Ultimately, this makes such approaches theoretically inconsistent as the policy learning
component assumes the reward to be only a function of the current state and action. All techniques for learning the reward function in combination with standard RL methods [20, 49] end up adding additional hyper-parameter tuning and compute cost. IPL on the other hand, is directly designed for RL from preference data and eliminates the reward network entirely. Other recent works also consider contrastive objectives instead of RL [26, 23].
Recently, works in natural language processing have applied ideas from preference-based RL to tasks such as summarization [51, 54], instruction following [42], and question-answering [40]. The RLHF paradigm has proven to be powerful even at the massive scale of aligning large language models. In this regime, learned reward models are massive, making an implicit reward method like IPL more attractive. In fact, IPL in a contextual bandits setting recovers concurrent work by Rafailov et al. [45] on implicit reward modeling in LLMs (see Appendix A). While we focus on control in our experiments, we hope our work can inform future explorations in language domains.
Imitation Learning. Our work builds on foundational knowledge in maximum entropy (MaxEnt) RL [58] and inverse RL [59]. Recent works in MaxEnt inverse RL have used the mapping between  $Q$ -functions and reward functions under a fixed policy. Specifically, Garg et al. [18] show that the regularized MaxEnt inverse RL objective from Ho and Ermon [24] can be re-written using the  $Q$ -function instead of a reward function and Al-Hafez et al. [4] stabilize their approach. While the relationship between  $Q$ -functions and rewards has been used for MaxEnt inverse RL, we study this relationship when learning from preference data. While both problems seek to learn models of expert reward, the data differs significantly — preference-based RL uses comparisons instead of optimal demonstrations. This necessitates a greatly different approach.
3 Inverse Preference Learning
In this section, we first describe the preference-based RL problem. Then, we describe how, leveraging techniques from imitation learning, we can remove the independently learned reward network from prior methods. This results in a simpler algorithm with lower computational cost and variance in performance.
3.1 Preference-Based RL
We consider the reinforcement learning (RL) paradigm where an agent seeks to maximize its expected cumulative discounted sum of rewards in a Markov Decision Process (MDP). Standard off-policy RL algorithms, do so using state, action, reward, and next state tuples  $(s, a, r, s')$ . In preference-based RL, however, the reward function  $r$  is unknown, and must be learned from human feedback. Thus, Traditional preference-based RL methods are thus usually separated into two stages: first, reward learning, where  $r_E$  is estimated by a learned reward function  $r_\theta$ , and second, reinforcement learning, where a policy  $\pi(a|s)$  is learned to maximize  $\mathbb{E}_{\pi}[\sum_{t=0}^{\infty} \gamma^t r_\theta(s, a)]$  with  $\gamma$  as the discount factor. Though our method combines these two phases, we use the building blocks of each and consequently review them here.
Preference Learning. First, similar to prior works [13, 32], we assume access to preference data in the form of binary comparisons. Each comparison is comprised of two behavior segments,  $\sigma^{(1)}$  and  $\sigma^{(2)}$ , and a binary label  $y$  indicating which of the two was preferred by an expert. As in Wilson et al. [53], each behavior segment is simply a snippet of a trajectory of length  $k$ , or  $\sigma = (s_t, a_t, s_{t+1}, a_{t+1}, \ldots, a_{t+k-1}, s_{t+k})$ . Increasing  $k$  can provide more information per label at the cost of potentially noisier labels. The label  $y$  for each comparison is assumed to be generated by an expert according to a Bradley-Terry Preference model [10]:
$$
P _ {r _ {E}} \left[ \sigma^ {(1)} \succ \sigma^ {(2)} \right] = \frac {\exp \sum_ {t} r _ {E} \left(s _ {t} ^ {(1)} , a _ {t} ^ {(1)}\right)}{\exp \sum_ {t} r _ {E} \left(s _ {t} ^ {(1)} , a _ {t} ^ {(1)}\right) + \exp \sum_ {t} r _ {E} \left(s _ {t} ^ {(2)} , a _ {t} ^ {(2)}\right)}, \tag {1}
$$
where  $r_E(s_t, a_t)$  is again the expert's unknown underlying reward model. We use the subscript  $r_E$  on probability  $P$  to indicate that the preference distribution above results from the expert's reward function. Let the dataset of these preferences be  $\mathcal{D}_p = \{(\sigma^{(1)}, \sigma^{(2)}, y)\}$ . To learn  $r_E$ , prior works in preference-based RL estimate a parametric reward function  $r_\theta$  by minimizing the binary-cross-entropy over  $\mathcal{D}_p$ :
$$
\mathcal {L} _ {p} (\theta) = - \mathbb {E} _ {\sigma^ {(1)}, \sigma^ {(2)}, y \sim \mathcal {D} _ {p}} \left[ y \log P _ {r _ {\theta}} \left[ \sigma^ {(1)} \succ \sigma^ {(2)} \right] + (1 - y) \log \left(1 - P _ {r _ {\theta}} \left[ \sigma^ {(1)} \succ \sigma^ {(2)} \right]\right) \right]. (2)
$$
Standard Preference-Based RL
Phase 1 Reward Learning
Phase 2 Reinforcement Learning
Inverse Preference Learning
Single Learning Phase
Figure 1: A depiction of the difference between standard preference-based RL methods and Inverse Preference Learning. Standard preference-based RL first learns a reward function, then optimizes it with a blockbox RL algorithm. IPL trains a  $Q$  function to directly fit the expert's preferences. This is done by aligning the implied reward model with the expert's preference distribution and applying regularization.
This objective results from simply minimizing  $\mathbb{E}_{\mathcal{D}_p}[D_{\mathrm{KL}}(P_{r_E}||P_\theta)]$ , the KL-divergence between the expert preference model and the one induced by  $r_\theta$ , effectively aligning it with the expert's preferences. We note that some other works in preference-based RL focus on learning an improved model  $r_\theta$  to address the reward learning part of the problem [43, 28]. However, these methods still use off-the-shelf RL algorithms for the policy learning part of the problem.
Reinforcement Learning. Common off-policy RL methods learn a policy  $\pi$  by alternating between policy evaluation (using the contractive Bellam Operator  $\mathcal{B}_r^\pi$ ) to estimate  $Q^\pi$  and policy improvement, where the policy  $\pi$  is improved [52]. Concretely, after repeated application of  $\mathcal{B}_r^\pi$  as
$$
\left(\mathcal {B} _ {r} ^ {\pi} Q\right) (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim p (\cdot | s, a)} \left[ V ^ {\pi} \left(s ^ {\prime}\right) \right], \tag {3}
$$
the policy can be improved by maximizing  $Q$ . In some settings, the Bellman operator  $\mathcal{B}_r^*$  corresponding to the optimal policy  $\pi^*$  can be used directly, removing the need for the policy improvement step. In these cases, we can simply extract  $\pi^*$  from the resulting  $Q^*$ .
To learn the optimal policy, two-phase preference based RL methods rely on recovering the optimal  $r_E$  in the reward learning phase before running RL. This potentially propagates errors from the estimated  $r_\theta$  to the learned  $Q$ -function and ultimately learned policy  $\pi$ . In practice, it would be more efficient to eliminate the need for two separate stages. In the next section, we show how this can be done by establishing a bijection between reward functions  $r$  and  $Q$ -functions.
3.2 Removing The Reward Function
In this section, we formally describe how the reward function can be removed from offline preference-based RL algorithms. Our key insight is that the  $Q$ -function learned by off-policy RL algorithms in fact encodes the same information as the reward function  $r(s, a)$ . Consequently, it is unnecessary to learn both. First, we show how the reward function can be re-written in terms of the  $Q$  function allowing us to compute the preference model  $P_Q$  induced by the  $Q$ -function. Then, we derive an objective that simultaneously pushes  $Q$  to fit the expert's preferences while also remaining optimal.
Consider fitting a  $Q$  function via the Bellman operator  $\mathcal{B}_r^\pi$  for a fixed policy  $\pi$  until convergence where  $\mathcal{B}_r^\pi Q = Q$ . Here, to encode the cumulative discounted rewards when acting according to the policy, the  $Q$ -function depends on both  $r$  and  $\pi$ . This dependence, however, is directly disentangled by the Bellman equation. By rearranging it (Eq. (3)), we can solve for the reward function in terms of  $Q$  and  $\pi$ . This yields the so-called inverse soft-Bellman operator:
$$
\left(\mathcal {T} ^ {\pi} Q\right) (s, a) = Q (s, a) - \gamma \mathbb {E} _ {s ^ {\prime}} \left[ V ^ {\pi} \left(s ^ {\prime}\right) \right]. \tag {4}
$$
In fact, for a fixed policy  $\pi$  the inverse-Bellman operator is bijective, implying a one-to-one correspondence between the  $Q$  function and the reward function. Though this was previously shown in maximum entropy RL [18], we prove the general case Lemma 1 in Appendix A.
Intuitively, this makes sense: when holding the policy constant, only the reward function affects  $Q$ . We abbreviate the evaluation of  $(\mathcal{T}^{\pi}Q)(s,a)$  as  $r_{Q^{\pi}}(s,a)$  to indicate that  $r_{Q^{\pi}}$  is the unique implicit reward function induced by  $Q^{\pi}$ . Prior works in imitation learning leverage the inverse soft-Bellman operator to measure how closely the implicit reward model  $r_{Q^{\pi}}$  aligns with expert demonstrations [18]. Our key insight is that this equivalence can also be used to directly measure how closely our  $Q$  function aligns with the expert preference model without ever directly learning  $r$ .
Consider the Bradley-Terry preference model in Equation (1). For a fixed policy  $\pi$  and its corresponding  $Q^{\pi}$ , we can obtain the preference model of the implicit reward function  $P_{Q^{\pi}}[\sigma^{(1)} > \sigma^{(2)}]$  by substituting the inverse Bellman operator into Equation (1) as follows:
$$
P _ {Q ^ {\pi}} \left[ \sigma^ {(1)} > \sigma^ {(2)} \right] = \frac {\exp \sum_ {t} \left(\mathcal {T} ^ {\pi} Q\right) \left(s _ {t} ^ {(1)} , a _ {t} ^ {(1)}\right)}{\exp \sum_ {t} \left(\mathcal {T} ^ {\pi} Q\right) \left(s _ {t} ^ {(1)}, a _ {t} ^ {(1)}\right) + \exp \sum_ {t} \left(\mathcal {T} ^ {\pi} Q\right) \left(s _ {t} ^ {(2)}, a _ {t} ^ {(2)}\right)}. \tag {5}
$$
This substitution will allow us to measure the difference between the preferences implied by  $Q^{\pi}$  and those of the expert. To minimize the difference, we can propagate gradients through the preference modeling loss (Equation (2)) and the implicit preference model  $P_{Q^{\pi}}$  (Equation (5)) to  $Q$  just as we would for a parameterized reward estimate  $r_{\theta}$ . Unfortunately, naively performing this substitution is insufficient to solve the RL objective for two reasons.
The Optimal Inverse Bellman Operator. First, we have used an arbitrary policy  $\pi$ , not the optimal one, for converting from  $Q$ -values to rewards. Though the  $Q$ -function may imply the expert's preferences, the corresponding policy could be extremely sub-optimal. To fix this problem, we need to use the optimal inverse bellman operator  $\mathcal{T}^*$  to ensure the extract  $Q$ -function corresponds to that of  $\pi^*$ . For this step, we can use any off-policy RL-algorithm that converges to the optimal policy! If the algorithm directly estimates the  $\mathcal{B}_r^*$ , the corresponding  $\mathcal{T}^*$  can be estimated using the target from  $\mathcal{B}_r^*$ , or
$$
\left(\mathcal {T} ^ {*} Q\right) (s, a) = Q (s, a) - \gamma \mathbb {E} _ {s ^ {\prime}} \left[ V ^ {\operatorname {t a r g}} \left(s ^ {\prime}\right) \right] \text {w h e r e} V ^ {\operatorname {t a r g}} (s) \text {i s e s t i m a t e d a s i n} \mathcal {B} _ {r} ^ {*}.
$$
In many cases, however, computing the optimal bellman operator  $\mathcal{B}_r^*$  is infeasible. Instead, many modern off-policy RL algorithms use policy improvement to converge to the optimal policy. These methods, like Haarnoja et al. [20] use  $Q^{\pi}$  to estimate a new policy  $\pi'$  such that  $Q^{\pi'} \geq Q^{\pi}$ . By repeatedly improving the policy, they eventually converge to  $Q^*$ . Thus, by repeatedly improving the policy according to these algorithms, we can eventually converge to the optimal policy and can thus estimate corresponding optimal inverse bellman operator by using  $V^{\mathrm{targ}}(s) = \mathbb{E}_{a \sim \pi(\cdot|s)}[Q(s,a)]$  in the above equation.
Regularization. Given we can estimate  $\mathcal{T}^*$  using targets from  $\mathcal{B}_r^*$  or policy improvement, we can fit the optimal  $Q$ -function by minimizing the following loss function
$$
\mathcal {L} _ {p} (Q) = - \mathbb {E} _ {\sigma^ {(1)}, \sigma^ {(2)}, y \sim \mathcal {D} _ {p}} \left[ y \log P _ {Q ^ {*}} [ \sigma^ {(1)} > \sigma^ {(2)} ] + (1 - y) \log (1 - P _ {Q ^ {*}} [ \sigma^ {(1)} > \sigma^ {(2)}) \right].
$$
where  $P_{Q^{*}}$  is given by substituting  $\mathcal{T}^* Q$  into Eq. (5). Unfortunately, optimizing this objective alone leads to poor results and may not converge when using RL algorithms that depend on policy improvement. This is because the above objective is under constrained due to the invariance of the Bradley-Terry preference model to shifts. By examining Eq. (1), it can be seen that adding a constant value to all rewards does not change the probability of preferring a segment. However, shifting the reward function by a constant does change the  $Q$ -function. RL algorithms using policy improvement monotonically increase the  $Q$ -function until reaching the maximum at  $Q^{*}$ . Thus as the implicit reward continues to increase,  $Q^{*}$  will continue to increase and may never be reached. To resolve this issue, we insure that the optima of the preference loss is unique by introducing a convex regularizer  $\psi(\cdot)$  on the implicit rewards  $r_{Q^{\pi}} = \mathcal{T}^{\pi}Q$ , giving us the regularized preference loss:
$$
\mathcal {L} _ {p} (Q) = - \mathbb {E} _ {\sigma^ {(1)}, \sigma^ {(2)}, y \sim \mathcal {D} _ {p}} \left[ y \log P _ {Q ^ {*}} \left[ \sigma^ {(1)} > \sigma^ {(2)} \right] + (1 - y) \log \left(1 - P _ {Q ^ {*}} \left[ \sigma^ {(1)} > \sigma^ {(2)}\right)\right) \right] + \lambda \psi (\mathcal {T} ^ {*} Q) \tag {6}
$$
In practice we choose  $\psi$  to be a form of L2 regularization as is commonly done in imitation learning [18, 4] to prevent unbounded reward values.  $\lambda >0$  is a hyperparameter that controls the strength of regularization. Besides allowing us to guarantee convergence, regularization has a number of benefits. It can help center the implicit reward near zero, which has been shown to beneficial for RL [16]. Moreover, it encourages more realistic implicit rewards. For example, a reward function might change rapidly by large values when only small perturbations are applied to the state or action. Though such reward functions might be unrealistic, they are completely valid solutions of the inverse-Bellman
operator. Adding regularization can help penalize large deviations in reward unless they drastically reduce the preference loss. Thus, the first term of Eq. (6) encourages the  $Q$ -function to match the expert's preferences, while the second term smooths the implied reward function and makes it unique.
Our final algorithm, which we call Inverse Preference Learning (IPL) fits the optimal policy corresponding to the regularized expert reward function by repeatedly minimizing  $\mathcal{L}_p(Q)$  (Eq. (6)) and improving the value target used  $V^{\mathrm{targ}}$  with the update step from any off-policy RL algorithm. In this manner, IPL performs dynamic programming through the inverse bellman operator until convergence. In Appendix A, we prove the following Theorem.
Theorem 1 Given an off-policy RL algorithm that converges to the optimal policy  $\pi_r^*$  for some reward function  $r$  and regularizer  $\psi$  such that Eq. (2) is strictly convex, IPL converges to  $\pi_{r^*}^*$  corresponding to reward function  $r^* = \arg \min_r\mathbb{E}_{\mathcal{D}_p}\left[D_{KL}(P_{r_E}||P_\theta)\right] + \lambda \psi (r)$ .