Title: Inverse Preference Learning: Preference-based RL without a Reward Function
Venue: NeurIPS 2023
Accepted: yes
Keywords: preference learning

preference-based reinforcement learning

human-in-the-loop reinforcement learning
Abstract: Reward functions are difficult to design and often hard to align with human intent. Preference-based Reinforcement Learning (RL) algorithms address these problems by learning reward functions from human feedback. However, the majority of preference-based RL methods na\"ively combine supervised reward models with off-the-shelf RL algorithms. Contemporary approaches have sought to improve performance and query complexity by using larger and more complex reward architectures such as transformers. Instead of using highly complex architectures, we develop a new and parameter-efficient algorithm, Inverse Preference Learning (IPL), specifically designed for learning from offline preference data. Our key insight is that for a fixed policy, the $Q$-function encodes all information about the reward function, effectively making them interchangeable. Using this insight, we completely eliminate the need for a learned reward function. Our resulting algorithm is simpler and more parameter-efficient. Across a suite of continuous control and robotics benchmarks, IPL attains competitive performance compared to more complex approaches that leverage transformer-based and non-Markovian reward functions while having fewer algorithmic hyperparameters and learned network parameters. Our code is publicly released.

Inverse Preference Learning: Preference-based RL without a Reward Function

Abstract

Reward functions are difficult to design and often hard to align with human intent. Preference-based Reinforcement Learning (RL) algorithms address these problems by learning reward functions from human feedback. However, the majority of preference-based RL methods naïvely combine supervised reward models with off-the-shelf RL algorithms. Contemporary approaches have sought to improve performance and query complexity by using larger and more complex reward architectures such as transformers. Instead of using highly complex architectures, we develop a new and parameter-efficient algorithm, Inverse Preference Learning (IPL), specifically designed for learning from offline preference data. Our key insight is that for a fixed policy, the 𝑄 -function encodes all information about the reward function, effectively making them interchangeable. Using this insight, we completely eliminate the need for a learned reward function. Our resulting algorithm is simpler and more parameter-efficient. Across a suite of continuous control and robotics benchmarks, IPL attains competitive performance compared to more complex approaches that leverage transformer-based and non-Markovian reward functions while having fewer algorithmic hyperparameters and learned network parameters. Our code is publicly released 1 .

1 Introduction

Reinforcement Learning (RL) has shown marked success in fixed and narrow domains such as simulated control [20] and game-playing [37]. When deploying RL in more complex settings, like in robotics or interaction with humans, one often runs into a critical bottleneck: the reward function. Obtaining reward labels in the real world can be complex, requiring difficult instrumentation [48, 57] and painstaking tuning [56] to achieve reasonable levels of sample efficiency. Moreover, despite extensive engineering, reward functions can still be exploited by algorithms in ways that do not align with human values and intents [21], which can be detrimental in safety-critical applications [5].

Instead of hand-designing reward functions, contemporary works have attempted to learn them through expert demonstrations [1], natural language [34], or human feedback [47, 3, 53]. Recently, reward functions learned through pairwise comparison queries-where a user is asked which of two demonstrated behaviors they prefer-have been shown to be effective in both control [13, 47, 32] and natural language domains [51]. This is often referred to as Reinforcement Learning with Human Feedback (RLHF) . Reward functions learned via RLHF can directly capture human intent, while avoiding alternative and more expensive forms of human feedback such as expert demonstrations. Preference-based RL algorithms for RLHF often interleave reward-learning from comparisons with off-the-shelf RL algorithms.

While preference-based RL methods discover reward functions that are aligned with human preferences, they are not without flaws. Learned reward functions must have adequate coverage of

both the state and action space to attain good downstream performance. Consequently, learning the reward function can be expensive, usually requiring thousands of labeled preference queries. To mitigate these challenges, recent works have proposed improving learned reward functions by adding inductive biases before optimization with RL. Hejna and Sadigh [22] pretrain reward functions with meta-learning. Park et al. [43] use data augmentation. Early et al. [15] and Kim et al. [28] make the reward function non-Markovian using recurrent or large transformer sequence model architectures respectively. Such approaches increase the upfront cost of preference-based RL by using additional data or compute. Moreover, these techniques still combine reward optimization with vanilla RL algorithms. Ultimately, this just adds an extra learned component to already notoriously delicate RL algorithms, further increasing hyper-parameter tuning overhead. Preference-based RL approaches often end up training up to four distinct neural networks independently: a critic (with up to two networks), an actor, and a reward function. This can be problematic as prediction errors cascade from the reward function, to the critic, and ultimately the actor causing high variance in downstream performance. To address these issues, we propose a parameter-efficient algorithm specifically designed for preference-based RL that completely eliminates the need to explicitly learn a reward function. In doing so, we reduce both complexity and compute cost.

The key insight of our work is that, under a fixed policy, the 𝑄 -function learned by off-policy RL algorithms captures the same information as the learned reward function. For example, both the 𝑄 -function and reward function encode information about how desirable a state-action pair is. This begs the question: why do we need to learn a reward function in the first place? Our proposed solution, Inverse Preference Learning or IPL, is an offline RL algorithm that is specifically designed for learning from preference data. Instead of relying on an explicit reward function, IPL directly optimizes the implicit rewards induced by the learned 𝑄 -function to be consistent with expert preferences. At the same time, IPL regularizes these implicit rewards to ensure high-quality behavior. As a result, IPL removes the need for a learned reward function and its associated computational and tuning expense.

Experimentally, we find that even though IPL does not explicitly learn a reward function, it achieves competitive performance with complicated Transformer-based reward learning techniques on offline Preference-based RL benchmarks with real-human feedback. At the same time, IPL consistently exhibits lower variance across runs as it does not suffer from the errors associated with querying a learned reward model. Finally, under a minimal parameter budget, IPL is able to outperform standard preference-based RL approaches that learn an explicit reward model.

2 Related Work

Our work builds upon literature in reward learning, preference-based RL, and imitation learning.

Reward Learning. Due to the challenges associated with designing and shaping effective reward signals, several works have investigated various approaches for learning reward functions. A large body of work uses inverse RL to learn a reward function from expert demonstrations [1, 41, 46], which are unfortunately difficult to collect [27, 2, 35] or often misaligned with true human preferences [7, 31]. Subsequently, reward learning techniques using other simpler forms of feedback such as scalar scores [29] and partial [38] or complete rankings [11, 8] have been developed. One of the simplest forms of human feedback is pairwise comparisons, where the user chooses between two options. Often, pairwise comparison queries are sampled using techniques from active learning [47, 9, 14]. However, to evaluate learned reward functions, these methods rely on either RL or traditional planning algorithms which are complex and computationally expensive. Our approach takes a simpler perspective that is parameter-efficient by combining reward and policy learning. Though it is not the focus of our work, IPL could additionally leverage active learning techniques for selecting preference data online.

Preference-based Deep Reinforcement Learning. Current approaches to preference based deep RL train a reward function, and then use that reward function in conjunction with a standard reinforcement learning algorithm [13, 33, 50]. Several techniques have been developed to improve the learned reward function, such as pre-training [25, 32], meta-learning [22], data augmentation [43], and nonMarkovian modeling. Within the family of non-Markovian reward modeling [6], recent approaches have leveraged both LSTM networks [15] and transformers [28] for reward learning. But, these methods still rely on Markovian offline RL algorithms such as Implicit Q-Learning (IQL) [30] for optimization. Ultimately, this makes such approaches theoretically inconsistent as the policy learning

component assumes the reward to be only a function of the current state and action. All techniques for learning the reward function in combination with standard RL methods [20, 49] end up adding additional hyper-parameter tuning and compute cost. IPL on the other hand, is directly designed for RL from preference data and eliminates the reward network entirely. Other recent works also consider contrastive objectives instead of RL [26, 23].

Recently, works in natural language processing have applied ideas from preference-based RL to tasks such as summarization [51, 54], instruction following [42], and question-answering [40]. The RLHF paradigm has proven to be powerful even at the massive scale of aligning large language models. In this regime, learned reward models are massive, making an implicit reward method like IPL more attractive. In fact, IPL in a contextual bandits setting recovers concurrent work by Rafailov et al. [45] on implicit reward modeling in LLMs (see Appendix A). While we focus on control in our experiments, we hope our work can inform future explorations in language domains.

Imitation Learning . Our work builds on foundational knowledge in maximum entropy (MaxEnt) RL [58] and inverse RL [59]. Recent works in MaxEnt inverse RL have used the mapping between 𝑄 -functions and reward functions under a fixed policy. Specifically, Garg et al. [18] show that the regularized MaxEnt inverse RL objective from Ho and Ermon [24] can be re-written using the 𝑄 -function instead of a reward function and Al-Hafez et al. [4] stabilize their approach. While the relationship between 𝑄 -functions and rewards has been used for MaxEnt inverse RL, we study this relationship when learning from preference data. While both problems seek to learn models of expert reward, the data differs significantly - preference-based RL uses comparisons instead of optimal demonstrations. This necessitates a greatly different approach.

3 Inverse Preference Learning

In this section, we first describe the preference-based RL problem. Then, we describe how, leveraging techniques from imitation learning, we can remove the independently learned reward network from prior methods. This results in a simpler algorithm with lower computational cost and variance in performance.

3.1 Preference-Based RL

We consider the reinfrocement leraning (RL) paradigm where an agent seeks to maximize its expected cumulative discounted sum of rewards in a Markov Decision Process (MDP). Standard off-policy RL algorithms, do so using state, action, reward, and next state tuples ( 𝑠, 𝑎, 𝑟, 𝑠 ′ ) . In preference-based RL, however, the reward function 𝑟 is unknown, and must be learned from human feedback. Thus, Traditional preference-based RL methods are thus usually separated into two stages: first, reward learning, where 𝑟 𝐸 is estimated by a learned reward function 𝑟 𝜃 , and second, reinforcement learning, where a policy 𝜋 ( 𝑎 | 𝑠 ) is learned to maximize E 𝜋 [ ∑︁ ∞ 𝑡 = 0 𝛾 𝑡 𝑟 𝜃 ( 𝑠, 𝑎 )] with 𝛾 as the discount factor. Though our method combines these two phases, we use the building blocks of each and consequently review them here.

Preference Learning. First, similar to prior works [13, 32], we assume access to preference data in the form of binary comparisons. Each comparison is comprised of two behavior segments, 𝜎 ( 1 ) and 𝜎 ( 2 ) , and a binary label 𝑦 indicating which of the two was preferred by an expert. As in Wilson et al. [53], each behavior segment is simply a snippet of a trajectory of length 𝑘 , or 𝜎 = ( 𝑠 𝑡 , 𝑎 𝑡 , 𝑠 𝑡 + 1 , 𝑎 𝑡 + 1 , . . . , 𝑎 𝑡 + 𝑘 -1 , 𝑠 𝑡 + 𝑘 ) . Increasing 𝑘 can provide more information per label at the cost of potentially noisier labels. The label 𝑦 for each comparison is assumed to be generated by an expert according to a Bradley-Terry Preference model [10]:

where 𝑟 𝐸 ( 𝑠 𝑡 , 𝑎 𝑡 ) is again the expert's unknown underlying reward model. We use the subscript 𝑟 𝐸 on probability 𝑃 to indicate that the preference distribution above results from the expert's reward function. Let the dataset of these preferences be D 𝑝 = {( 𝜎 ( 1 ) , 𝜎 ( 2 ) , 𝑦 )} . To learn 𝑟 𝐸 , prior works in preference-based RL estimate a parametric reward function 𝑟 𝜃 by minimizing the binary-cross-entropy over D 𝑝 :

Standard Preference-Based RL

Inverse Preference Learning

Phase 1

Reward Learning

Phase 2

Reinforcement Learning

𝑟 𝜃

𝜎 (1)

𝜎 (2)

Black Box

RL

Algorithm

𝜋(𝑎|𝑠)

𝑄(𝑠, 𝑎)

Single Learning Phase

𝑉 𝜋 (𝑠)

𝑟(𝑠, 𝑎) = 𝑄

𝑠, 𝑎

-𝛾𝑉 𝜋 (𝑠 ′

)

Regularization

Preference

Distribution

Implicit Reward

𝜎 (1)

𝜎 (2)

This objective results from simply minimizing E D 𝑝 [ 𝐷 KL ( 𝑃 𝑟 𝐸 | | 𝑃 𝜃 )] , the KL-divergence between the expert preference model and the one induced by 𝑟 𝜃 , effectively aligning it with the expert's preferences. We note that some other works in preference-based RL focus on learning an improved model 𝑟 𝜃 to address the reward learning part of the problem [43, 28]. However, these methods still use off-the-shelf RL algorithms for the policy learning part of the problem.

Reinforcement Learning. Common off-policy RL methods learn a policy 𝜋 by alternating between policy evaluation (using the contractive Bellam Operator B 𝜋 𝑟 ) to estimate 𝑄 𝜋 and policy improvement, where the policy 𝜋 is improved [52]. Concretely, after repeated application of B 𝜋 𝑟 as

the policy can be improved by maximizing 𝑄 . In some settings, the Bellman operator B ∗ 𝑟 corresponding to the optimal policy 𝜋 ∗ can be used directly, removing the need for the policy improvement step. In these cases, we can simply extract 𝜋 ∗ from the resulting 𝑄 ∗ .

To learn the optimal policy, two-phase preference based RL methods rely on recovering the optimal 𝑟 𝐸 in the reward learning phase before running RL. This potentially propagates errors from the estimated 𝑟 𝜃 to the learned 𝑄 -function and ultimately learned policy 𝜋 . In practice, it would be more efficient to eliminate the need for two separate stages. In the next section, we show how this can be done by establishing a bijection between reward functions 𝑟 and 𝑄 -functions.

3.2 Removing The Reward Function

In this section, we formally describe how the reward function can be removed from offline preference-based RL algorithms. Our key insight is that the 𝑄 -function learned by off-policy RL algorithms in fact encodes the same information as the reward function 𝑟 ( 𝑠, 𝑎 ) . Consequently, it is unnecessary to learn both. First, we show how the reward function can be re-written in terms of the 𝑄 function allowing us to compute the preference model 𝑃 𝑄 induced by the 𝑄 -function. Then, we derive an objective that simultaneously pushes 𝑄 to fit the expert's preferences while also remaining optimal.

Consider fitting a 𝑄 function via the Bellman operator B 𝜋 𝑟 for a fixed policy 𝜋 until convergence where B 𝜋 𝑟 𝑄 = 𝑄 . Here, to encode the cumulative discounted rewards when acting according to the policy, the 𝑄 -function depends on both 𝑟 and 𝜋 . This dependence, however, is directly disentangled by the Bellman equation. By rearranging it (Eq. (3)), we can solve for the reward function in terms of 𝑄 and 𝜋 . This yields the so-called inverse soft-Bellman operator:

In fact, for a fixed policy 𝜋 the inverse-Bellman operator is bijective, implying a one-to-one correspondence between the 𝑄 function and the reward function. Though this was previously shown in maximum entropy RL [18], we prove the general case Lemma 1 in Appendix A.

Intuitively, this makes sense: when holding the policy constant, only the reward function affects 𝑄 . We abbreviate the evaluation of (T 𝜋 𝑄 )( 𝑠, 𝑎 ) as 𝑟 𝑄 𝜋 ( 𝑠, 𝑎 ) to indicate that 𝑟 𝑄 𝜋 is the unique implicit reward function induced by 𝑄 𝜋 . Prior works in imitation learning leverage the inverse soft-Bellman operator to measure how closely the implicit reward model 𝑟 𝑄 𝜋 aligns with expert demonstrations [18]. Our key insight is that this equivalence can also be used to directly measure how closely our 𝑄 function aligns with the expert preference model without ever directly learning 𝑟 .

Consider the Bradley-Terry preference model in Equation (1). For a fixed policy 𝜋 and its corresponding 𝑄 𝜋 , we can obtain the preference model of the implicit reward function 𝑃 𝑄 𝜋 [ 𝜎 ( 1 ) ≻ 𝜎 ( 2 ) ] by substituting the inverse Bellman operator into Equation (1) as follows:

This substitution will allow us to measure the difference between the preferences implied by 𝑄 𝜋 and those of the expert. To minimize the difference, we can propagate gradients through the preference modeling loss (Equation (2)) and the implicit preference model 𝑃 𝑄 𝜋 (Equation (5)) to 𝑄 -just as we would for a parameterized reward estimate 𝑟 𝜃 . Unfortunately, naïvely performing this substitution is insufficient to solve the RL objective for two reasons.

The Optimal Inverse Bellman Operator. First, we have used an arbitrary policy 𝜋 , not the optimal one, for converting from 𝑄 -values to rewards. Though the 𝑄 -function may imply the expert's preferences, the corresponding policy could be extremely sub-optimal. To fix this problem, we need to use the optimal inverse bellman operator T ∗ to ensure the extract 𝑄 -function corresponds to that of 𝜋 ∗ . For this step, we can use any off-policy RL-algorithm that converges to the optimal policy! If the algorithm directly estimates the B ∗ 𝑟 , the corresponding T ∗ can be estimated using the target from B ∗ 𝑟 , or

In many cases, however, computing the optimal bellman operator B ∗ 𝑟 is infeasible. Instead, many modern off-policy RL algorithms use policy improvement to converge to the optimal policy. These methods, like Haarnoja et al. [20] use 𝑄 𝜋 to estimate a new policy 𝜋 ′ such that 𝑄 𝜋 ′ ≥ 𝑄 𝜋 . By repeatedly improving the policy, they eventually converge to 𝑄 ∗ . Thus, by repeatedly improving the policy according to these algorithms, we can eventually converge to the optimal policy and can thus estimate corresponding optimal inverse bellman operator by using 𝑉 targ ( 𝑠 ) = E 𝑎 ∼ 𝜋 (·| 𝑠 ) [ 𝑄 ( 𝑠, 𝑎 )] in the above equation.

Regularization. Given we can estimate T ∗ using targets from B ∗ 𝑟 or policy improvement, we can fit the optimal 𝑄 -function by minimizing the following loss function

where 𝑃 𝑄 ∗ is given by substituting T ∗ 𝑄 into Eq. (5). Unfortunately, optimizing this objective alone leads to poor results and may not converge when using RL algorithms that depend on policy improvement. This is because the above objective is under constrained due to the invariance of the Bradley-Terry preference model to shifts. By examining Eq. (1), it can be seen that adding a constant value to all rewards does not change the probability of preferring a segment. However, shifting the reward function by a constant does change the 𝑄 -function. RL algorithms using policy improvement monotonically increase the 𝑄 -function until reaching the maximum at 𝑄 ∗ . Thus as the implicit reward continues to increase, 𝑄 ∗ will continue to increase and may never be reached. To resolve this issue, we insure that the optima of the preference loss is unique by introducing a convex regularizer 𝜓 (·) on the implicit rewards 𝑟 𝑄 𝜋 = T 𝜋 𝑄 , giving us the regularized preference loss:

In practice we choose 𝜓 to be a form of L2 regularization as is commonly done in imitation learning [18, 4] to prevent unbounded reward values. 𝜆 > 0 is a hyperparameter that controls the strength of regularization. Besides allowing us to guarantee convergence, regularization has a number of benefits. It can help center the implicit reward near zero, which has been shown to beneficial for RL [16]. Moreover, it encourages more realistic implicit rewards. For example, a reward function might change rapidly by large values when only small perturbations are applied to the state or action. Though such reward functions might be unrealistic, they are completely valid solutions of the inverse-Bellman

operator. Adding regularization can help penalize large deviations in reward unless they drastically reduce the preference loss. Thus, the first term of Eq. (6) encourages the 𝑄 -function to match the expert's preferences, while the second term smooths the implied reward function and makes it unique.

Our final algorithm, which we call Inverse Preference Learning (IPL) fits the optimal policy corresponding to the regularized expert reward function by repeatedly minimizing L 𝑝 ( 𝑄 ) (Eq. (6)) and improving the value target used 𝑉 targ with the update step from any off-policy RL algorithm. In this manner, IPL performs dynamic programming through the inverse bellman operator until convergence. In Appendix A, we prove the following Theorem.

Theorem 1 Given an off-policy RL algorithm that convergences to the optimal policy 𝜋 ∗ 𝑟 for some reward function 𝑟 and regularizer 𝜓 such that Eq. (2) is strictly convex, IPL converges to 𝜋 ∗ 𝑟 ∗ corresponding to reward function 𝑟 ∗ = arg min 𝑟 E D 𝑝 [ 𝐷 KL ( 𝑃 𝑟 𝐸 | | 𝑃 𝜃 )] + 𝜆𝜓 ( 𝑟 ) .

The proof of the theorem essentially relies on the fact that for a fixed policy 𝜋 , we can optimize L 𝑝 ( 𝑄 ) (Eq. (6)) to fit 𝑟 ∗ . Then, we can update the policy (or target values 𝑉 targ ) and optimize L 𝑝 ( 𝑄 ) again. Because 𝑟 ∗ is unique, we fit 𝑟 ∗ again the second time, but the 𝑄 -function has improved. There are many choices of regularizers where this holds. In tabular settings if 𝜓 ( 𝑟 ) = 𝑟 2 , L 𝑝 ( 𝑄 ) reduces to L2 regularized logistic regression, which is strictly convex, guaranteeing convergence (Appendix A).

Effectively, IPL removes the need to learn a reward network, while still converging to similar solution as other preference-based RL algorithms. Learning a reward network requires more parameters and a completely separate optimization loop, increasing compute requirements. Moreover, an explicit reward model introduces a whole new suite of hyper-parameters that need to be tuned including the model architecture, capacity, learning rate, batch size, and stopping criterion. In fact, because human preference data is so difficult to collect, many approaches opt to use simple accuracy thresholds instead of validation criteria to decide when to stop training 𝑟 𝜃 [32]. All of these components make preference-based RL unreliable and high-variance. On the other hand, our method completely removes all of these parameters in exchange for a single 𝜆 hyper-parameter that controls the regularization strength. Though we have theoretically derived IPL, in the next section we provide practical recipes for applying it to offline preference-based RL.

3.3 IPL for Offline Preference-based RL

In offline preference-based RL, we assume access to a fixed offline dataset D 𝑜 = {( 𝑠, 𝑎, 𝑠 ′ )} of interactions without reward labels generated by a reference policy 𝜇 ( 𝑎 | 𝑠 ) in addition to the preference dataset D 𝑝 . Common approaches to offline RL seek to learn conservative policies that do not stray too far away from the distribution of data generated by 𝜇 ( 𝑎 | 𝑠 ) . This is critical to prevent the policy 𝜋 from reaching out-ofdistribution states during deployment which can be detrimental to performance. In this section, we detail a practical version of IPL that uses the

X QL offline RL algorithm [19]. X QL fits the KL-constrained RL objective

where 𝛼 controls the magnitude of the KL-divergence penalty. The X QL algorithm directly fits the optimal 𝑄 -function using the optimal soft-Bellman operator [19, 55]

In practice, 𝑉 targ is estimated using the linex loss function over the current 𝑄 -function. Thus, to fit the optimal 𝑄 -function, IPL with X QL alternates between minimizing the preference loss L 𝑝 ( 𝑄 ) (Eq. (6)), and updating a learned value function 𝑉 until they converge to 𝑄 ∗ and 𝑉 ∗ . Note that we are not limited to using just D 𝑝 . Though the preference modeling part of L 𝑝 ( 𝑄 ) can only be optimized with preference data D 𝑝 , the value function can be updated with offline data as well. In the presence

of additional offline data, we find that updating the value function using D 𝑝 ∪ D 𝑜 leads to better performance. We approximate L2 regularization with the regularizer 𝜓 ( 𝑟 ) = E D 𝑝 ∪D 𝑜 [ 𝑟 ( 𝑠, 𝑎 ) 2 ] , which imposes an L2 penalty across the support of the data. While one might try to use weight decay to emulate L2-regularization, doing so is difficult in practice as T ∗ 𝑄 depends on both the 𝑄 network and the target network. We find that weighting the regularization equally between D 𝑝 and D 𝑜 performs well. After 𝑄 and 𝑉 have converged, we can extract the policy using the closed form relationship 𝜋 ∗ ( 𝑎 | 𝑠 ) ∝ 𝜇 ( 𝑎 | 𝑠 ) exp (( 𝑄 ∗ ( 𝑠, 𝑎 ) -𝑉 ∗ ( 𝑠 ))/ 𝛼 ) for KL-constrained RL as in Garg et al. [19], Peng et al. [44]. The full algorithm for IPL with X QL can be found in 1.

Though we have shown how IPL can be instantiated with X QL, it is fully with other offline RL algorithms. In fact, IPL can also be used with online RL algorithms like SAC [20]. Critically, this makes the IPL framework general, as it can remove the need for reward modeling in nearly any preference-based RL setting. This makes IPL simpler and more efficient. In the next section, we show that IPL can attain the same performance as strong offline preference-based RL baselines, without learning a reward network.

4 Experiments

In this section, we aim to answer the following questions: First, how does IPL compare to prior preference-based RL algorithms on standard benchmarks? Second, how does IPL perform in extremely data-limited settings? And finally, how efficient is IPL in comparison to two-phase preference-based RL methods?

4.1 Setup

As discussed in the previous section, though we use a KL-constrained objective for our theoretical derivation, in practice we can construct versions of IPL based on any offline RL algorithm. In our experiments we evaluate IPL with Implicit Q-Learning (IQL) [30], since it has been used in prior offline preference-based RL works. This allows us to directly compare IPL by isolating its implicit reward component and using the same exact hyper-parameters as prior works. Using IPL with IQL amounts to updating the value function according to the asymmetric expectile loss function instead of the linex loss function. Concretely, this can be done by replacing the value update in Algorithm 1 with min 𝑉 E 𝐵 𝑝 ∪ 𝐵 𝑜 [︁ | 𝜏 -𝟙 ( 𝑄 ( 𝑠, 𝑎 ) -𝑉 ( 𝑠 ) < 0 )| ( 𝑄 ( 𝑠, 𝑎 ) -𝑉 ( 𝑠 )) 2 ]︁ where 𝜏 is the expectile.

Inspired by Park et al. [43], we introduce data augmentations that sample sub-sections of behavior segments 𝜎 during training. While such augmentations are inapplicable to non-Markovian reward models, we find that they boost performance for Markovian reward models while also reducing the total number of state-action pairs per batch of preference data. This is important as IPL needs data from both D 𝑝 and D 𝑜 to regularize the implicit reward function. Additional experiment details and hyper-parameters can be found in the Appendix.

4.2 How does IPL perform on preference-based RL benchmarks?

Wecompare IPL to other offline preference-based RL approaches on D4RL Gym Locomotion [17] and Robosuite robotics [36] datasets with real-human preference data from Kim et al. [28]. We compare IQL-based IPL, with the same hyper-parameters, to various baselines that learn a reward model 𝑟 𝜃 before optimization with IQL. Markovian Reward or MR denotes using a standard Markovian MLP reward model, like those used in Christiano et al. [13] and Lee et al. [32]. Note that this is also equivalent to T-REX [11] for offline RL. Non-Markovian Reward or NMR denotes using the non-Markovian LSTM based reward model from Early et al. [15]. Preference Transformer (PT) is a state-of-the-art approach that leverages a large transformer architecture to learn a non-Markovian reward and preference weighting function. B-REX uses bayesian optimization to fit a linear reward function from predefined features [12], which in our case are random Gaussian projections of the states and actions. For fairness, we also compare against our own implementation of IQL with a Markovian Reward function that uses the same data augmentation as IPL.

Our results are summarized in Table 1. Starting with the first column, we see that preference-based RL methods are able to match IQL with the ground truth reward function in many cases. On, several tasks, however, the MR implementation from Kim et al. [28] fairs rather poorly. The nonMarkovian methods, (NMR and PT) improve performance. It is worth noting that on many tasks our

Table 1: Average normalized scores of all baselines on human-preference benchmarks from Kim et al. [28]. For the D4RL locomotion tasks 'hop' corresponds to hopper, 'm' to medium (training the data generating agent to 1/3 expert performance), 'r' to replay buffer data, and 'e' to data from the end of training. For the Robomimic tasks lift and can, 'ph' corresponds to proficient human data and 'mh' to multi-human data of differing optimality. The first four columns are taken from Kim et al. [28]. 'reimpl.' is our reimplementation of Markovian Reward with IQL. The 'Avg Std' row shows the average standard deviation across all eight environments. We run five seeds and report the final performance at the end of training like Kostrikov et al. [30]. Bolded values are within 95% of the top performing method. Note that standard devaition values in the table were rounded for space. On some tasks IPL achieves higher performance earlier in training, which is not reflected above (See Appendix B). We find that IPL outperforms PT on many environments, and also performs similarly to our implementation of MR despite not training a reward function.

implementation of a MR (sixth column) performs far better than reported in Kim et al. [28], likely due to our careful tuning of 𝑟 𝜃 and use of data-augmentations. Our method, IPL, achieves competitive performance across the board.

In general, IPL with IQL performs on-par or better than both our implementation of MR and PT in most datasets despite not learning a separate reward network. Specifically, IPL has the same performance or better performance than our MR implementation on six of eight tasks. More importantly, IPL does extremely well in comparison to Preference Transformer's reported results. On five of eight tasks IPL performs better than PT while having over 10 times fewer parameters, making IPL far more efficient. To be consistent with Kim et al. [28], we report results after a million training steps but performance for IPL often peaks earlier (see learning curves in the Appendix). For example, with early stopping IPL also outperforms PT on 'hop-m-r'. We posit that this is because the 𝑄 -function in IPL is tasked with both fitting the expert's preference model and optimal policy simultaneously, making both the policy and reward function non-stationary during training. In some datasets, this was more unstable.

IPL also has the lowest average standard-deviation across seeds, meaning it yields more consistent results than explicit reward methods. For standard two-phase preference-based RL algorithms, errors in the reward model are propagated to and exacerbated by the 𝑄 function. IPL circumvents this problem by not explicitly learning the reward.

Finally, in Table 2, we consider various design decisions of IPL. Augmentations provide a strong boost in the robotics environment, but offer only minor improvements in locomotion. Removing regularization, however, is detrimental to performance. This is likely because without regularization, the implicit reward values can continue to increase, leading to exploding estimates of 𝑄 . Finally, we show that IPL is compatible with other offline RL algorithms by combining it with X QL [19]. We find that with X QL, IPL performs even better on some tasks, but worse on others. Finally, in Appendix B, we also show that IPL can be combined with online preference-based RL algorithms like PEBBLE [32].

4.3 How does IPL scale with Data?

Collecting preference comparisons is often viewed as the most expensive part of preference-based RL. To investigate how well IPL performs in data limited settings, we construct scripted preference datasets

of four different sizes for five tasks from the MetaWorld benchmark [56] used in prior preferencebased RL works [32, 22]. We then train on the preference data D 𝑝 by setting D 𝑜 = {( 𝑠, 𝑎, 𝑠 ′ ) ∈ D 𝑝 } and use the same hyper-parameters for all environments and methods where applicable. Our results are summarized in Table 3. Again, IPL is a strong reward-free baseline. We find that at all data scales, IPL performs competitively to our implementation of MR (IQL with a learned Markovian reward) and consistently outperforms it in Button Press and Assembly. Increasing the amount of preference data generally improves performance across the board. However, as we generate queries uniformly at random some preference datasets may be easier to learn from than others, leading to deviations from this trend in some cases. As in the benchmark results in Table 1, IPL exhibits lower variance across seeds and tasks, in this case at three of four data scales.

4.4 How efficient is IPL?

One benefit of IPL over other preference-based RL methods is its parameter efficiency. By removing the reward network, IPL uses fewer parameters than other methods while achieving the same performance. In Table 4, we show the number of parameters for each method used in the last two sections. Preference Transformer uses over ten times more parameters than IPL, and the LSTM-based NMR model from Early et al. [15] uses nearly twice as many. When dealing with a limited compute or memory budget, this can be important. To exacerbate this effect, we consider an extremely parameter efficient version of IPL, denoted 'IPL (64)' in Table 4, based on Advantage Weighted Actor Critic (AWAC) [39] which eliminates the second critic and value networks used in IQL [30] and uses a two-layer 64-dimensional MLP. We then compare this parameter-efficient IPL to MR with the same parameter budget which results in 'MR (35)', a 35-dimensional MLP. Results are depicted on the left of Fig. 2. MR trained with a smaller network is unable to adequately fit the data, resulting in lower performance. Only after increasing the network size past that of IPL can MR begin to match performance.

Aside from parameter efficiency, IPL is also 'hyper-parameter efficient'. By removing the reward network, IPL removes a whole set of hyper-parameters associated with two phase preference based RL methods, like reward network architecture, learning rate, stopping criterion, and more. In the

50000

100000

150000

200000

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Parameter Equalization

50000

100000

150000

200000

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

MR Reward Steps

1000

10000

100000

50000

100000

150000

200000

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

IPL Regularization Weight

0.4

2

10

MetaWorld Drawer Open Ablations with 4000 Queries

MR (35)

MR (64)

IPL (64)

middle of Fig. 2 we show how the performance of MR is affected when the reward function is over or under fit. Choosing the correct number of steps to train the reward model usually requires collecting a validation set of preference data, which is costly to obtain. Instead of this, IPL only has a single regularization parameter, 𝜆 . The right side of Fig. 2 shows the sensitivity of IPL to 𝜆 . We find that in many cases, varying 𝜆 has little effect on performance unless it is perturbed by a large amount.

5 Conclusion

Summary. We introduce Inverse Preference Learning, a novel algorithm for offline preference-based RL that avoids learning a reward function. Our key insight is to leverage the inverse soft-Bellman operator, which computes the mapping from 𝑄 -functions to rewards under a fixed policy. The IPL algorithm trains a 𝑄 -function to regress towards the optimal 𝑄 ∗ while at the same time admitting implicit reward values that are consistent with an expert's preferences. Even though IPL does not require learning a separate reward network, on robotics benchmarks it attains competitive performance with preference-based RL baselines that use twice to ten-times the number of model parameters.

Limitations and Future Work. A number of future directions remain. Specifically, the implicit reward function and policy learned by IPL are both non-stationary during training, which sometimes causes learning to be more unstable than with a fixed reward function. This is a core limitation future work could address by better mixing

policy improvement and preference-matching steps to improve stability. More broadly, implicit reward preference-based RL methods are not limited to continuous control or binary feedback. Applying implicit reward techniques to other forms of feedback or extending IPL to language-based RLHF tasks remain exciting future directions.

Acknowledgments and Disclosure of Funding

This work was supported by ONR, DARPA YFA, Ford, and NSF Awards #1941722 and #2218760. JH is supported by by the National Defense Science Engineering Graduate (NDSEG) Fellowship Program. We would additionally like to thank Div Garg and Chris Cundy for useful discussions.

References

Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning , 2004.

Baris Akgun, Maya Cakmak, Karl Jiang, and Andrea L Thomaz. Keyframe-based learning from demonstration. International Journal of Social Robotics , 4(4):343-355, 2012.

Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , 2011.

Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. LS-IQ: Implicit reward regularization for inverse reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= o3Q4m8jg4BR .

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 , 2016.

Fahiem Bacchus, Craig Boutilier, and Adam Grove. Rewarding behaviors. In National Conference on Artificial Intelligence , 1996.

Chandrayee Basu, Qian Yang, David Hungerman, Mukesh Sinahal, and Anca D Draqan. Do you want your autonomous car to drive like you? In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI , pages 417-425. IEEE, 2017.

Erdem Bıyık, Daniel A Lazar, Dorsa Sadigh, and Ramtin Pedarsani. The green choice: Learning and influencing human decisions on shared roads. In 2019 IEEE 58th conference on decision and control (CDC) , pages 347-354. IEEE, 2019.

Erdem Biyik, Nicolas Huynh, Mykel J. Kochenderfer, and Dorsa Sadigh. Active preferencebased gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS) , July 2020.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning , pages 783-792. PMLR, 2019.

Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning , pages 1165-1177. PMLR, 2020.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , 2017.

Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with a novel acquisition function. Autonomous Robots , 39(3):389-405, 2015.

Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ramchurn. Non-markovian reward modelling from trajectory labels via interpretable multiple instance learning. In Advances in Neural Information Processing Systems , 2022.

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations , 2020. URL https: //openreview.net/forum?id=r1etN1rtPB .

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iqlearn: Inverse soft-q learning for imitation. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=Aeo-xqtb5p .

Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=SJ0Lde3tRL .

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , 2018.

Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems , 30, 2017.

Joey Hejna and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop RL. In Conference on Robot Learning , 2022.

Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without rl. arXiv preprint arXiv:2310.13639 , 2023.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems , 2018.

Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference-guided policy optimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 15753-15768. PMLR, 23-29 Jul 2023. URL https: //proceedings.mlr.press/v202/kang23b.html .

Rebecca P Khurshid and Katherine J Kuchenbecker. Data-driven motion mappings improve transparency in teleoperation. Presence , 24(2):132-154, 2015.

Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for rl. In International Conference on Learning Representations , 2023.

WBradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference on development and learning , pages 292-297. IEEE, 2008.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , 2022.

Minae Kwon, Erdem Biyik, Aditi Talati, Karan Bhasin, Dylan P Losey, and Dorsa Sadigh. When humans aren't optimal: Robots that collaborate with risk-aware humans. In 2020 15th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages 43-52. IEEE, 2020.

Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning , 2021.

Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preferencebased reinforcement learning. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track (round 1) , 2021.

Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. Inferring rewards from language in context. arXiv preprint arXiv:2204.02515 , 2022.

Dylan P Losey, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg, and Dorsa Sadigh. Controlling assistive robots with learned latent actions. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 378-384. IEEE, 2020.

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL) , 2021.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning multimodal rewards from rankings. In Conference on Robot Learning , pages 342-352. PMLR, 2022.

Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. {AWAC}: Accelerating online reinforcement learning with offline datasets, 2021. URL https://openreview.net/ forum?id=OJiM1R3jAtZ .

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 , 2021.

Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning , 2000.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022.

Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Representations , 2022.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290 , 2023.

Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI , volume 7, pages 2586-2591, 2007.

Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems , 2017.

C. Schenck and D. Fox. Visual closed-loop control for pouring liquids. In International Conference on Robotics and Automation , 2017.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Daniel Shin and Daniel S Brown. Offline preference-based apprenticeship learning. arXiv preprint arXiv:2107.09251 , 2021.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325 , 2020.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT Press, 2018.

Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems , 2012.

Jeff Wu, Long Ouyang, Daniel M Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862 , 2021.

Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations , 2023.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning , 2020.

Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real world robotic reinforcement learning. In International Conference on Learning Representations , 2020.

Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy . Carnegie Mellon University, 2010.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

Appendix

A Theory

A.1 Proofs

Lemma 1 For any fixed policy 𝜋 the inverse bellman operator T 𝜋 establishes a bijection between 𝑟 and 𝑄 . Moreover, for any 𝑟 , 𝑄 = (T 𝜋 ) -1 𝑟 is the unique fixed point of the Bellman operator B 𝜋 𝑟 . (Adapted from Garg et al. [18])

Proof . Let 𝑃 𝜋 be the stochastic transition matrix for the MDP corresponding to a fixed policy 𝜋 . In vector form, the inverse bellman operator becomes 𝑟 = T 𝜋 𝑄 = ( 𝐼 -𝛾𝑃 𝜋 ) 𝑄 . We can establish a bijection between 𝑄 and 𝑟 by showing that ( 𝐼 -𝛾𝑃 𝜋 ) is invertible. As 𝑃 𝜋 defines a valid probability distribution over next stat-action pairs and 𝛾 < 1 , we have that | | 𝛾𝑃 𝜋 | | < 1 . Thus, its Neumann series convergences, which implies the existence of ( 𝐼 -𝛾𝑃 𝜋 ) -1 . So, 𝑄 = ( 𝐼 -𝛾𝑃 𝜋 ) -1 𝑟 and a bijection exists. Using this, we can also show a 1-1 mapping with the bellman operator under reward 𝑟 . We have 𝑄 = ( 𝐼 -𝛾𝑃 𝜋 ) -1 𝑟 = (T 𝜋 ) -1 𝑟 = B 𝜋 𝑟 𝑄 at the fixed point of B 𝜋 𝑟 .

Theorem 1 Given an off-policy RL algorithm that convergences to the optimal policy 𝜋 ∗ 𝑟 for some reward function 𝑟 and regularizer 𝜓 such that Eq. (2) is strictly convex, IPL converges to 𝜋 ∗ 𝑟 ∗ corresponding to reward function 𝑟 ∗ = arg min 𝑟 E D 𝑝 [ 𝐷 KL ( 𝑃 𝑟 𝐸 | | 𝑃 𝜃 )] + 𝜆𝜓 ( 𝑟 ) .

Proof. We prove this statement in the Tabular setting, first for algorithms that use policy improvement. Let 𝑄 𝑡 ∈ R | 𝑆 × 𝐴 | and 𝜋 𝑡 indicate the Q-function and policy after 𝑡 iterations. Let 𝑄 0 = 1 /( 1 -𝛾 ) min 𝑆 × 𝐴 𝑟 ( 𝑠, 𝑎 ) . The inverse bellman operator tells us, in vector form, that 𝑟 = ( 𝐼 -𝛾𝑃 𝜋 ) 𝑄 where 𝑃 𝜋 is the transition matrix. Let 𝑟 ∗ = arg min 𝑟 E 𝐷 𝑝 [ 𝑦 log 𝑃 𝑟 + ( 1 -𝑦 ) log ( 1 -𝑃 𝑟 )] + 𝜆𝜓 ( 𝑟 ) , or the minimizer of the preference loss with regularizer 𝜓 such that we converge to a unique 𝑟 ∗ .

At each step of IPL, we substitute the inverse bellman operator into the preference loss and optimize. Thus at convergence, ( 𝐼 -𝛾𝑃 𝜋 𝑡 ) 𝑄 𝑡 = 𝑟 ∗ uniquely due to Lemma 1. Then, there are two cases based on the type of RL algorithm.

If our RL algorithm can directly estimate B ∗ 𝑟 , then we are done. This is because we have assumed convergence, and thus T ∗ 𝑄 = 𝑟 ∗ . By the bijection established in Lemma 1, we have that 𝑄 = (T ∗ ) -1 𝑟 ∗ which by also Lemma 1 is the unique fixed point of B ∗ 𝑟 ∗ which is 𝑄 ∗ 𝑟 ∗ . Thus, we have recovered the optimal 𝑄 function for 𝑟 ∗ from which the optimal policy 𝜋 ∗ 𝑟 ∗ can be extracted.

If we have an RL algorithm that uses policy improvement, we consider multiple steps of IPL. If the RL algorithm guarantees convergence via policy improvement, then we use 𝜋 𝑡 and 𝑄 𝑡 . to obtain a new policy 𝜋 𝑡 + 1 . Using 𝜋 𝑡 + 1 we can obtain the transition matrix 𝑃 𝜋 𝑡 + 1 . Finally, we optimize the preference loss again using 𝑃 𝜋 𝑡 + 1 in the inverse Bellman operator to obtain 𝑄 𝑡 + 1 . At convergence ( 𝐼 -𝛾𝑃 𝜋 𝑡 + 1 ) 𝑄 𝑡 + 1 = 𝑟 ∗ holds. As 𝑟 ∗ is unique, 𝑄 𝑡 and 𝑄 𝑡 + 1 are both Q-functions for the reward function 𝑟 ∗ , just under different policies. We know from the definition of policy improvement, that 𝑄 𝜋 𝑡 + 1 ≥ 𝑄 𝜋 𝑡 necessarily, and thus 𝑄 𝑡 + 1 ≥ 𝑄 𝑡 for any 𝑡 . Convergence is possible as according to Lemma 1, 𝑄 ∗ = (T ∗ ) -1 𝑟 is the a fixed point of B ∗ 𝑟 .

Proposition 1 If 𝜙 ( 𝑟 ) = 𝑟 2 , then IPL converges to the optimal policy corresponding to 𝑟 ∗ = arg min 𝑟 E D 𝑝 [ 𝐷 KL ( 𝑃 𝑟 𝐸 | | 𝑃 𝜃 )] + 𝜆𝜓 ( 𝑟 ) in tabular settings.

Proof. The preference-based loss function with L2 regularization can be viewed as L2 regularized logistic regression by writing the logits as a dot-product between a preference comparison vector 𝑥 comprised of -𝛾 𝑡 , 𝛾 𝑡 and 0 terms and a reward function 𝑟 . The Hessian for this objective is then 𝑋 𝑇 𝐷𝑋 + 𝜆𝐼 where 𝐷 𝑖𝑖 = logistic ( 𝑥 𝑖 ⊤ 𝑟 )( 1 -logistic ( 𝑥 𝑖 ⊤ 𝑟 )) , which is positive definite. Thus the problem is strictly convex and 𝑟 ∗ is unique, so IPL converges to its optimal policy by Theorem 1. Note that to guarantee this we must regularize 𝑟 across the entire state-action space, analogous to regularizing the weight vector in logistic regression.

A.2 Connections to DPO

Concurrent work, called Direct Preference Optimization (DPO) [45] also remove the need for explicit reward modeling for learning from preferences, but do so in the contextual bandits setting for language

models. Thus, DPO is limited to preference queries segment length 1 that must start from the same state . IPL is in fact, a more general version of DPO that does not have these restrictions. Specifically, IPL with X QL recovers the same exact policy as DPO when applied to the contextual bandits setting.

Within the bandits setting, there is no 'next-state' and 𝑉 ∗ ( 𝑠 ′ ) is removed, and the inverse bellman operator becomes just 𝑄 ( 𝑠, 𝑎 ) = 𝑟 ( 𝑠, 𝑎 ) . The optimal X QL policy is 𝜋 ∗ = 𝜇 ( 𝑎 | 𝑠 ) 𝑒 𝑄 ∗ ( 𝑠,𝑎 ) / 𝑍 ( 𝑠 ) where 𝑍 is the partition function. By rearranging, T ∗ 𝑄 = 𝑄 ∗ ( 𝑠, 𝑎 ) = log 𝜋 ( 𝑎 | 𝑠 ) 𝜇 ( 𝑎 | 𝑠 ) + 𝑍 ( 𝑠 ) . We can plug this into the preference model induced by Q in Eq. (5). In the RLHF setting, the partition function cancels since we assume the context to be the same between preferences. This exactly results in the DPO algorithm, showing the DPO is in fact just an instantiation of IPL for contextual bandits.

A.3 IPL with Rankings

IPL can easily be extended to rankings using a Plackett Luce model. Consider permutations 𝜏 over 𝐾 segments:

where 𝑑 𝑘 = ∑︁ 𝐾 𝑗 = 𝑘 exp ∑︁ 𝑡 𝛾 𝑡 𝑟 𝐸 ( 𝑠 𝜏 𝑗 𝑡 , 𝑎 𝜏 𝑗 𝑡 ) . Then, we make the same substitution using the inverse bellman operator giving us the permutation model 𝑃 𝑄 implied by the Q function, and run maximum likelihood estimation over the model with the preference loss L 𝑝 ( 𝑄 ) = E 𝜏 ∼D 𝑝 [︁ log 𝑃 𝑄 ( 𝜏 ) ]︁ + 𝜆𝜓 ( 𝑟 ) .

B Results

We divide this part of appendix into four different sections following the results section. Each section additionally provides hyper-parameters used for IPL in that section. The first section, setup, contains detailed information on the experimental setup and hyper-parameters used. The second section on benchmark results gives full learning curves for the experiments in Section 4.2. The third section provides full learning curves for the MetaWorld and Data-scaling experiments. The final Appendix section provides extended ablations.

B.1 Setup

Here we provide the full algorithmic outline of IPL using Implicit Q-Learning [30] that mimics our implementation. While in practice the policy 𝜋 could be extracted at the end of training, we do it simultaneously as in [30] in order to construct learning curves.

Algorithm 2: IPL Algorithm (IQL Variant)

Note that above we write the temperature parameter 𝛽 as done in IQL, instead of how it is usually done, using 𝛼 in the denominator [19, 44].

When sampling batches of preference data 𝐵 𝑝 ∼ D 𝑝 , we take sub-samples of each segment 𝜎 of length 𝑠 . For a sampled data point ( 𝜎 ( 1 ) , 𝜎 ( 2 ) , 𝑦 ) , we sample start ∼ Unif [ 0 , 1 , 2 , ...𝑘 -𝑠 ] and then let take 𝜎 = 𝑠 start , 𝑎 start , ..., 𝑠 start + 𝑠 . We use the same start value across the entire batch.

Given that we run experiments using MLPs, all of our experiments were run on CPU compute resources. Each seed for each method requires one CPU core and 8 Gb of memory.

B.2 Benchmark Results

Here we provide details for our experiments on the preference-based RL benchmark from Kim et al. [28]. We use the same hyperparameters as Kim et al. [28] and Kostrikov et al. [30] as shown in Table 5.

Gym-Mujoco Locomotion . Hopper and Walker2D agents are tasked with learning locomotion policies from datasets of varying qualities taken from the D4RL [17] benchmark. Preference datasets were constructed by Kim et al. [28] by uniformly sampling segments. Preference datasets for 'medium' quality offline datasets contain 500 queries, while preference datasets for 'expert' quality offline datasets contain 100 queries. Segment length 𝑘 = 100 for all datasets, and were subsampled to length 𝑠 = 64 by IPL and our MR (reimpl). Evaluation was preformed over 10 episodes every 5000 steps. Full learning curves are shown in Fig. 3.

RoboMimic . The RoboMimic datasets contain interaction data of two types: ph - proficient human and mh - multihuman. The multi-human data was collected from human demonstrators of mixed quality. The robot is tasked with learning how to lift a cube (lift) or pick and place a can (can). Preference datasets were again taken directly from Kim et al. [28]. Preference datasets of size 100 with segment lengths 𝑘 = 50 , randomly sub-sampled to length 𝑠 = 32 were used for the ph datasets. Preference datasets of size 500 with segment lengths 𝑘 = 100 , randomly sub-sampled to length 𝑠 = 64 were used for the mh datasets. Evaluation was performed over 25 episodes every 50000 steps. Full learning curves are shown in Fig. 4.

Online Experiments . We also test a combination of IPL with PEBBLE [32] on a few tasks in the MetaWorld benchmark. Results can be found in Fig. 5

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0.2

0.4

0.6

0.8

1.0

Score

Hopper Medium Replay

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0.2

0.4

0.6

0.8

1.0

1.2

Hopper Medium Expert

0.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.3

0.4

0.5

0.6

0.7

Score

Walker2D Medium Replay

0.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.6

0.7

0.8

0.9

1.0

1.1

Walker2D Medium Expert

D4RL Mujoco Locomotion with Human Preferences

PT

MR

MR (reimpl.)

IPL

0.2

0.4

0.6

0.8

1.0

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Lift PH

0.2

0.4

0.6

0.8

1.0

1e6

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Can PH

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Can MH

Robomimic with Human Preferences

PT

MR

MR (reimpl.)

IPL

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Plate Slide, 20k Feedback

PEBBLE

IPL

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Drawer Open, 20k Feedback

Online MetaWorld with Scripted Preferences

B.3 Data Scaling Results

Experiments for data scaling were conducted on the MetaWorld benchmark from Yu et al. [56]. Offline datasets for five different MetaWorld tasks were constructed as follows: Collect 100 trajectories of expert data on the target task using the built in ground truth policies with the addition of Gaussian noise of standard deviation 1.0. Collect 100 trajectories of sub-optimal data by running the groundtruth policy for a different randomization of the target task with Gaussian noise 1.0. Collect 100 trajectories of even more sub-optimal data by running the ground truth policy of a different task with Gaussian noise standard deviation 1.0 in the target domain. Finally, collect 100 trajectories with uniform random actions. As MetaWorld episodes are 500 steps long, this results in 200,000 time-steps of data. We then construct preference datasets by uniformly sampling segments from the offline dataset and assigning labels 𝑦 according to ∑︁ 𝑡 𝑟 ( 𝑠 ( 1 ) 𝑡 , 𝑎 ( 1 ) 𝑡 ) > ∑︁ 𝑡 𝑟 ( 𝑠 ( 2 ) 𝑡 , 𝑎 ( 2 ) 𝑡 ) where 𝑟 is the ground truth reward provided by metaworld. We then train using only the data from D General architecture hyper-parameters were taken from Lee et al. [32], Hejna and Sadigh [22] which also use the MetaWorld benchmark, but for online preference-based RL. Full-hyper parameters are shown in Table 6. We run 20 evaluation episodes every 2500 steps. Full learning curves are shown in Fig. 6. When reporting values in Table 3, we choose the maximum point on the learning curves which average across five seeds. This provides results as if early stopping was given by an oracle, which is less optimistic than averaging the maximum of each seed as done in Mandlekar et al. [36].

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Button Press, 500

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Button Press, 1000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Button Press, 2000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Button Press, 4000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Drawer Open, 500

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Drawer Open, 1000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Drawer Open, 2000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Drawer Open, 4000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Sweep Into, 500

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Sweep Into, 1000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Sweep Into, 2000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Sweep Into, 4000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Plate Slide, 500

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Plate Slide, 1000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Plate Slide, 2000

50000

100000 150000 200000

0.0

0.2

0.4

0.6

0.8

1.0

Plate Slide, 4000

50000

100000 150000 200000

Training Steps

0.00

0.02

0.04

0.06

0.08

0.10

Success Rate

Assembly, 500

50000

100000 150000 200000

Training Steps

0.00

0.02

0.04

0.06

0.08

0.10

Assembly, 1000

50000

100000 150000 200000

Training Steps

0.00

0.02

0.04

0.06

0.08

0.10

Assembly, 2000

50000

100000 150000 200000

Training Steps

0.00

0.02

0.04

0.06

0.08

0.10

Assembly, 4000

MetaWorld with Scripted Preferences

MR (reimpl.)

IPL

B.4 Ablations

In this section we provide additional ablations on both the benchmark datasets and MetaWorld datasets. We keep the hyperparameters the same, except for the parameter-efficient experiments.

Benchmark IPL Ablations. We include results of full ablations for IPL on the benchmark tasks in Table 7. We additionally provide comparisons between IPL and MR + IQL with and without data augmentation in Fig. 7.

0.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.2

0.4

0.6

0.8

1.0

Score

Hopper Medium Replay

0.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.2

0.4

0.6

0.8

1.0

1.2

Score

Hopper Medium Expert

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.4

0.5

0.6

0.7

0.8

Success

Can PH

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Success

Can MH

Human Preferences Benchmark Data-Aug Ablation

MR

MR+Aug

IPL

IPL+Aug

Hyper-parameter Sensitivity. We run hyper-parameter sensitivty results for the human-preference benchmark datasets in Fig. 8. The top row depicts the sensitivity for IPL to the value of 𝜆 . The bottom row depicts the sensitivity of MR to the number of timesteps the reward function is trained for.

Parameter Efficiency. For the parameter-efficient experiments only we use an efficient version of IPL based on AWAC [39] to additionally remove the need for learning value network. AWAC uses

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Score

Walker2D Medium Replay

0.1

0.5

2.5

0.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.4

0.6

0.8

1.0

1.2

Score

Walker2D Medium Expert

0.1

0.5

2.5

50000

100000 150000 200000

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Drawer Open, 500

0.4

2

10

50000

100000 150000 200000

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Drawer Open, 4000

0.4

2

10

Regularization Ablation with IPL

0.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.3

0.4

0.5

0.6

0.7

0.8

Score

Walker2D Medium Replay

1000

10000

100000

0.0

0.2

0.4

0.6

0.8

1.0

Training Steps

1e6

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Score

Walker2D Medium Expert

1000

10000

100000

50000

100000 150000 200000

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Drawer Open, 500

1000

10000

100000

50000

100000 150000 200000

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Drawer Open, 4000

1000

10000

100000

Reward Training Steps Ablation on Benchmark Tasks with MR

a policy-evaluation, policy imporvement style approach. The outline of this variant is given below:

Algorithm 3: IPL Algorithm (AWAC Variant)

For this version of IPL, we use 𝜆 = 0 . 5 . All other hyper-parameters remain the same as in Table 8 except the architectures. For the parameter-efficiency experiments only we use MLPs consisting of two dense layers with either dimension 64 or dimension 35. Running MR with a two-layer MLP of dimension 35 has almost exactly the same number of parameters as IPL-AWAC with two-layer MLPs of dimension 64. We include full results for the parameter-efficiency experiments in Table 8. We find that on Drawer Open and Sweep Into, IPL outperforms both MR (64) and MR (35). In these environments, performance increases from MR (35) to MR (64) indicating that the expressiveness of the 𝑄 -function and policy are limiting performance. For the same budget, IPL is able to perform better. In Button Press, the simplest task, we find that MR (64) actually over-fits more than MR (35) and MR (64) ends up performing worse. In Plate Slide, all methods perform similarly independent of parameter count. We omit Assembly because of its low success rate at all data scales.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a way to learn from preferences in the offline setting without learning a reward model. The main insight is that reward and Q functions are interchangeable and the policy learning using preference can be formulated directly as a function of Q which represents the reward function implicitly. This allows for increased learning performance without requiring to train a seperate reward function when doing RL with preferences.\n\nSTRENGTHS:\nStrengths: \n\n1. The paper presents a new way to do policy learning from preference data without requiring learning an intermediate reward function.\n2. The experiments demonstrate that their method IPL outperforms baselines that learn from offline preferences in a number of prior tasks that include simulated MuJoCo locomotion tasks and manipulation tasks.\n\nWEAKNESSES:\n1. Missing theoretical underpinnings: A number of questions come up with the proposed method that should be addressed:\n    1. Is the learned policy optimal for the implicit Q function?\n        \n        Equation 6 is the probability that one trajectory is preferred by other under the policy $\\pi$ with the reward function $T^\\pi Q$ . Replacing $T^\\pi Q$  with $T^* Q  = Q(s,a)-\\gamma E[V^*(s')]$ , it is claimed that the preference learning objective will fit the optimal value function. There seems to be jump to this claim without a proof. In my opinion, it is important to show how minimizing Equation after line 201 leads to soft optimal $Q^*$ with ground truth reward function $r^E.$\n        \n    2. Large regularization weight\n        \n        A large regularization weight (1 and sometimes even larger than the main preference loss e.g in fig 2) is used in the work which makes it more confusing as to what combined objective is being optimized. While the intuition given by the authors makes sense, it might be great to show how this regularization does not change the stationary point of $Q^*$.\n        \n2. Clarification on novelty:\nIt is mentioned multiple times in the paper that:\n”Line 163: Our key insight is that Q-function learned by off-policy RL algorithm in face encodes the same information as the reward function”\n\nThis is not a new insight. It has been presented in multiple prior works [1,2,3] where a change of variables for reward is performed to remove the intermediate step of reward learning for imitation learn and the mapping from reward to Q has been studied theoretically in the work of [4] and more recently [5]. It seems the novelty is to add the preference loss to the IQ-Learn loss function.\n\n3. Prior literature on reward learning:\n    \n    Three baselines are used to compare the method IPL - MR, LSTM and PT. A number of other prior works exist which aim to do reward learning from preferences [6,7,8] and would be interesting to compare experimentally to and discuss the relationship in the paper. \n    \n4. Gap between proposed method and practical method: The paper discusses learning an optimal value function using the linex loss of XQL[9] but in the experiment section the value update is performed using IQL. Preference loss uses soft-optimal-Q whereas the policy update step in IQL updates for optimal Q. There seems to be a disconnect between the proposed method and practical method which might be important to address.\n5. Experiments: \na. How is the preference dataset constructed? Elaborating that in the main paper can increase the understanding considerably.\n    \n    b. Limited experiments: Only 1 baseline is compared against in the meta-world task and 4 simulated domains in total are considered in the locomotion and robomimic tasks. With limited theoretical understanding of the method, it might be important to establish the method empirically with previous baselines in order to show its merits.\n    \n\n[1] Kostrikov, Ilya, Ofir Nachum, and Jonathan Tompson. \"Imitation learning via off-policy distribution matching.\" *arXiv preprint arXiv:1912.05032* (2019).\n\n[2] Nachum, Ofir, and Bo Dai. \"Reinforcement learning via fenchel-rockafellar duality.\" *arXiv preprint arXiv:2001.01866* (2020).\n\n[3] Ma, Yecheng Jason, et al. \"Smodice: Versatile offline imitation learning via state occupancy matching.\" *arXiv e-prints* (2022): arXiv-2202.\n\n[4]Garg, Divyansh, et al. \"Iq-learn: Inverse soft-q learning for imitation.\" *Advances in Neural Information Processing Systems* 34 (2021): 4028-4039.\n\n[5] Sikchi, Harshit, Amy Zhang, and Scott Niekum. \"Imitation from Arbitrary Experience: A Dual Unification of Reinforcement and Imitation Learning Methods.\" *arXiv preprint arXiv:2302.08560* (2023).\n\n[6]: Brown, Daniel, et al. \"Safe imitation learning via fast bayesian reward inference from preferences.\" *International Conference on Machine Learning*. PMLR, 2020.\n\n[7]: Chen, Letian, Rohan Paleja, and Matthew Gombolay. \"Learning from suboptimal demonstration via self-supervised reward regression.\" *Conference on robot learning*. PMLR, 2021.\n\n[8]: Sikchi, Harshit, et al. \"A ranking game for imitation learning.\" *arXiv preprint arXiv:2202.03481* (2022).\n\n[9]: Garg, Divyansh, et al. \"Extreme Q-Learning: MaxEnt RL without Entropy.\" *arXiv preprint arXiv:2301.02328* (2023).\n\nQUESTIONS:\n1. Gradient through target Q: When implementing the regularization on  $T^* Q  = Q(s,a)-\\gamma E[V^*(s')]$ it seems that only Q(s,a) is updated which means that reward is not being made to be zero centered and rather Q is made to be zero-centered.\n2. XQL vs IQL: The main paper discusses XQL and the experiments use IQL. Is there a reason behind this disconnect? Are there XQL experiments to validate?\n\nUpdate: Thanks to the authors for responding to a number of my questions and providing additional baselines. I have updated my score (4->5) and will reiterate on my score upon further discussion with reviewers. I hope to see the clarification on novelty and the additional baselines in the updated paper.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of learning a policy given offline pairwise preference data. The proposed algorithm, inverse preference learning, directly learns a value function without explicitly learning the reward function first. This setup simplifies the common two-step approach of first inferring the reward function in a supervised manner and then learning the policy using RL. Across a range of simulated robotics experiments, it is shown that this method performs at least as well as more complex approaches that have significantly more (hyper-)parameters.\n\nSTRENGTHS:\nThe paper is well-motivated and clearly written. The presented idea is novel, simple and well-executed. The resulting algorithm is easy to implement, has much fewer (hyper-)parameters compared to prior work, and seems to perform on par or better than prior work. Source code is also available. Given the increasing interest in reinforcement learning with human feedback (RLHF) for robotics and natural language problems, this paper should be of high interest for the community.\n\nFinally, I really appreciate that the authors invested time to improve the Markovian Reward (MR) baseline. This by itself is a nice little contribution, and I hope that future work will adopt this much stronger baseline.\n\nWEAKNESSES:\nThe presentation of the results in the tables could be improved. While in Table 1 the results are based on the final success rate at the end of training, it seems that the results in Table 2 are based on the best success rate achieved throughout training (particularly apparent for the Drawer open tasks). I found this to be a bit misleading, especially since IPL seems to heavily suffer from the problem that the success rate deteriorates over time. Furthermore, I didn't fully understand why some results in the tables are bold and others are not. For example, in the Assembly task with 500 queries (Table 2), it seems that IPL is not significantly better than MR.\n\nMinor comments:\n1. I initially got a bit confused as to why $L_r$ depends on $D_o$ and $D_p$. Perhaps you could remind the reader that the preferences in $D_p$ are also (sub-)trajectories.\n2. Typos:\n- line 89: \"offlien\" -> \"offline\"\n- line 101: \"Our work build on\" -> \"Our work builds on\"\n- Eq. 3: Subscripts t missing for $a, s$ in $\\pi$ and $\\mu$\n- Eq. 4: $p(\\cdot|s|a)$ -> $p(\\cdot|s,a)$\n- line 158: \"to learned\" -> \"to the learned\"\n- line 199: \"KL-constrianed\" -> \"KL-constrained\"\n- line 223: \"thus\" -> \"is [thus]\"\n- line 283: \"that use\" -> \"that uses\"\n- line 305: \"Data\" -> \"data\"\n- line 315: \"form\" -> \"from\"\n- line 308/319: \"Preference-based\" -> \"preference-based\"\n\nQUESTIONS:\n1. Could you please clarify the results in Table 2? I would suggest that both Table 1 and Table 2 either only report the final success rate or report both the final and best success rate. \n2. Could you please double-check/clarify why certain numbers in the results are bold and others are not?\n\nLIMITATIONS:\nOne of the original motivations of inverse RL was that the reward function provides a succinct and transferable definition of the task (Abbeel and Ng, 2004). While recent approaches seem to no longer learn \"succinct\" reward functions, one could argue that transferability is still a good reason to explicitly learn a reward function. Thus, a potential limitation of the presented approach is that the ability to use the learned reward function to learn different policies gets lost since the value function depends on a particular policy.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new algorithm, inverse preference learning, to learn from (offline) preferences between behavior segments. In particular, they show that human preferences can be modeled using only the Q-function, therefore eliminating the need to learn a separate reward function. The authors show this approach can succeed in a variety of environments.\n\nSTRENGTHS:\nThis paper has several strengths:\n- They study the important problem of preference-based RL.\n- This paper makes the novel insight that human preferences can be modeled using only the Q-function, and introduce an algorithm based on this insight.\n- They experimentally validate different aspects of their algorithm over a wide variety of tasks.\n\nWEAKNESSES:\n**W1.** The motivation that reward-modeling has a large cost is a little bit weak (especially with regards to memory). For example, InstructGPT uses a 6B reward model and a 175B policy. The reward model is therefore only a small fraction of the memory cost. In addition, one of the main hyperparameters of reward-modeling that the authors say is difficult to tune is the stopping criteria. However, the stopping criteria for reward model training is straightforward in most papers: Simply monitor the validation loss on a held-out fraction of comparison data. \n\n**W2.** This paper mainly focuses on offline RL, yet the most impactful RLHF applications are trained online, where behavior data is collected from the policy. It seems like IPL can be trained in an online manner as well, but it is not clear from the presentation.\n\nQUESTIONS:\n**Q1.** Can this method be used in an online setting, where new data is generated according to the policy?\n\n**Q2.** How do MR and IPL perform without data augmentation? And why does MR receive such a large boost based on this augmentation?\n\n**Q3**. Are the reimplemented training setup and the training setup from [25] exactly the same? If not,  PT and LSTM should be reimplemented for fair comparison.\n\n**Q4.** What is the bolding scheme in Table 1?\n\n**Q5.** Is there any way to experimentally validate the claim that IPL is more hyperparameter efficient? For example, random/grid hyperparamter search with an equivalent budget? From Figure 2, it looks like MR is actually more hyperparameter efficient than IPL.\n\nLIMITATIONS:\nYes, the authors do discuss limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRLHF pipelines typically consist of (1) training a reward model over human preference data and (2) using this trained reward model with a well-known RL method. This two stage training is computationally expensive. The authors of this paper develop an algorithm \"Inverse Preference Learning\" to directly learn the $Q, V$ functions, which can be easily used to extract an aligned policy. By directly learning the value functions, this approach bypasses learning a reward model, which is expensive to train, and prone to problems like reward hacking.\n\nQUESTIONS:\nOn Page 2, the authors mention - \"This can be problematic as prediction errors cascade from the reward function, to the critic, and ultimately the actor causing high variance in downstream performance.\" Can we quantify these prediction errors more formally, to establish how serious this issue is?\n\nOnly having two options in the preference data seems limiting. Can this idea be applied to the case where the preference is amongst more than two options? Eg. maximizing $p(a\\succ b\\succ c\\succ d)$ rather than $p(a\\succ b)$.\n\nOn Page 6, line 214, the authors say \"While such a reward function seems unrealistic\", here, why do we want rewards to be necessarily continuous? In many cases, human defined rewards are discontinuous (eg. 0-1 type of reward where the reward is 0 everywhere except when the agent reaches the goal, when it is 1).\n\nThe authors mention on Line 53-54 that \"the key insight of our work is that, under a fixed policy, the Q-function learned by off-policy RL algorithms captures the same information as the learned reward function\". This is not an insight developed in this paper, but rather a result from [1] (see Lemma 3.2 in [1]). Moreover, the result is for soft-Q functions, and not the standard Q function. I would recommend to make these points clear in the manuscript.\n\nFinally, there is parallel work [2] that also tries to bypass reward learning similar to this approach. It might be good to acknowledge, but I leave this decision to the authors.\n\nSuggestions related to language and typos:\n1. Page 2, 2nd last line - \"offlien\" -> \"offline\"\n2. Page 6, 3rd line - should \"another perturbation of size \\epsilon\" be \"another perturbation of size \\epsilon prime\"?\n\nReferences:\n1. IQ-Learn: Inverse soft-Q Learning for Imitation, Garg et al. (2022)\n2. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafailov et al. (2023)\n\nEDIT (16 Aug 2023): Updated score from 6->7.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}