Inverse Preference Learning:
Preference-based RL without a Reward Function
Joey Hejna 1 Dorsa Sadigh 1

Abstract

difficult instrumentation (Schenck & Fox, 2017; Zhu et al.,2020) and painstaking tuning (Yu et al., 2020) to achievereasonable levels of sample efficiency. Moreover, despite extensive engineering, reward functions can still be exploitedby algorithms in ways that do not align with human values
and intents (Hadfield-Menell et al., 2017), which can be
detrimental in safety-critical applications (Amodei et al.,2016).

Reward functions are difficult to design and often hard to align with human intent. Preferencebased Reinforcement Learning (RL) algorithms
address these problems by learning reward functions from human feedback. However, the majority of preference-based RL methods naı̈vely
combine supervised reward models with off-theshelf RL algorithms. Contemporary approaches
have sought to improve performance and querycomplexity by using larger and more complex reward architectures such as transformers. Insteadof using highly complex architectures, we develop
a new and parameter-efficient algorithm, Inverse
Preference Learning (IPL), specifically designed
for learning from offline preference data. Our key
insight is that for a fixed policy, the 𝑄-function
encodes all information about the reward function,effectively making them interchangeable. Using
this insight, we completely eliminate the needfor a learned reward function. Our resulting algorithm is simpler and more parameter-efficient.
Across a suite of continuous control and robotics
benchmarks, IPL attains competitive performancecompared to more complex approaches that leverage transformer-based and non-Markovian reward
functions while having fewer algorithmic hyperparameters and learned network parameters.

Instead of hand-designing reward functions, contemporary
works have attempted to learn them through expert demonstrations (Abbeel & Ng, 2004), natural language (Lin et al.,2022), or human feedback (Sadigh et al., 2017; Akrour
et al., 2011; Wilson et al., 2012). Recently, reward functions
learned through pairwise comparison queries—where a user
is asked which of two demonstrated behaviors they prefer—
have been shown to be effective in both control (Christiano
et al., 2017; Sadigh et al., 2017; Lee et al., 2021a) andnatural language domains (Stiennon et al., 2020). This isoften referred to as Reinforcement Learning with Human
Feedback (RLHF). Reward functions learned via RLHF candirectly capture human intent, while avoiding alternativeand more expensive forms of human feedback such as expert demonstrations. Preference-based RL algorithms for
RLHF often interleave reward-learning from comparisonswith off-the-shelf RL algorithms.
While preference-based RL methods discover reward functions that are aligned with human preferences, they are notwithout flaws. Learned reward functions must have adequate
coverage of both the state and action space to attain good
downstream performance. Consequently, learning the reward function can be expensive, usually requiring thousandsof labeled preference queries. To mitigate these challenges,
recent works have proposed improving learned reward functions by adding inductive biases before optimization with
RL. Hejna & Sadigh (2022) pretrain reward functions withmeta-learning. Park et al. (2022) use data augmentation.
Early et al. (2022) and Kim et al. (2023) make the reward
function non-Markovian using recurrent or large transformer
sequence model architectures respectively. Such approaches
increase the upfront cost of preference-based RL by using
additional data or compute. Moreover, these techniques still
combine reward optimization with vanilla RL algorithms.
Ultimately, this just adds an extra learned component to

1. Introduction
Reinforcement Learning (RL) has shown marked successin fixed and narrow domains such as simulated control(Haarnoja et al., 2018) and game-playing (Mnih et al.,2013). When deploying RL in more complex settings, likein robotics or interaction with humans, one often runs intoa critical bottleneck: the reward function. Obtaining reward labels in the real world can be complex, requiring
1 Stanford University. Correspondence to: Joey Hejna <jhe-jna@cs.stanford.edu>.
Interactive Learning with Implicit Human Feedback Workshop at
ICML 2023

1

Inverse Preference Learning

already notoriously delicate RL algorithms, further increasing hyper-parameter tuning overhead. Preference-based RL
approaches often end up training up to four distinct neuralnetworks independently: a critic (with up to two networks),an actor, and a reward function. This can be problematic
as prediction errors cascade from the reward function, tothe critic, and ultimately the actor causing high variance indownstream performance. To address these issues, we propose a parameter-efficient algorithm specifically designed
for preference-based RL that completely eliminates the needto explicitly learn a reward function. In doing so, we reduce
both complexity and compute cost.

2017; Kwon et al., 2020). Subsequently, reward learningtechniques using other simpler forms of feedback such as
scalar scores (Knox & Stone, 2008) and partial (Myers et al.,2022) or complete rankings (Brown et al., 2019; Bıyık et al.,2019) have been developed. One of the simplest forms ofhuman feedback is pairwise comparisons, where the user
chooses between two options. Often, pairwise comparison
queries are sampled using techniques from active learning(Sadigh et al., 2017; Biyik et al., 2020; Daniel et al., 2015).
However, to evaluate learned reward functions, these methods rely on either RL or traditional planning algorithms
which are complex and computationally expensive. Our approach takes a simpler perspective that is parameter-efficientby combining reward and policy learning. Though it is not
the focus of our work, IPL could additionally leverage active
learning techniques for selecting preference data online.

The key insight of our work is that, under a fixed policy,
the 𝑄-function learned by off-policy RL algorithms captures the same information as the learned reward function.
For example, both the 𝑄-function and reward function encode information about how desirable a state-action pair is.
This begs the question: why do we need to learn a reward
function in the first place? Our proposed solution, Inverse
Preference Learning or IPL, is an offline RL algorithm thatis specifically designed for learning from preference data.
Instead of relying on an explicit reward function, IPL directly optimizes the implicit rewards induced by the learned
𝑄-function to be consistent with expert preferences. At thesame time, IPL regularizes these implicit rewards to ensurehigh-quality behavior. As a result, IPL removes the need fora learned reward function and its associated computational
and tuning expense.

Preference-based Deep Reinforcement Learning. Current
approaches to preference based deep RL train a reward function, and then use that reward function in conjunction with astandard reinforcement learning algorithm (Christiano et al.,2017; Lee et al., 2021b; Shin & Brown, 2021). Several techniques have been developed to improve the learned reward
function, such as pre-training (Ibarz et al., 2018; Lee et al.,2021a), meta-learning (Hejna & Sadigh, 2022), data augmentation (Park et al., 2022), and non-Markovian modeling.
Within the family of non-Markovian reward modeling (Bacchus et al., 1996), recent approaches have leveraged both
LSTM networks (Early et al., 2022) and transformers (Kimet al., 2023) for reward learning. But, these methods still
rely on Markovian offlien RL algorithms such as Implicit
Q-Learning (IQL) (Kostrikov et al., 2022) for optimization.
Ultimately, this makes such approaches theoretically inconsistent as the policy learning component assumes the reward
to be only a function of the current state and action. All
techniques for learning the reward function in combination
with standard RL methods (Haarnoja et al., 2018; Schulman
et al., 2017) end up adding additional hyper-parameter tuning and compute cost. IPL on the other hand, is directly
designed for RL from preference data and eliminates the
reward network entirely.

Experimentally, we find that even though IPL does not explicitly learn a reward function, it achieves competitive performance with complicated Transformer-based reward learning techniques on offline Preference-based RL benchmarks
with real-human feedback. At the same time, IPL consistently exhibits lower variance across runs as it does not
suffer from the errors associated with querying a learned
reward model. Finally, under a minimal parameter budget,
IPL is able to outperform standard preference-based RL
approaches that learn an explicit reward model.

2. Related Work

Recently, works in natural language processing have applied
ideas from preference-based RL to tasks such as summarization (Stiennon et al., 2020; Wu et al., 2021), instruction
following (Ouyang et al., 2022), and question-answering(Nakano et al., 2021). The RLHF paradigm has proven to
be powerful even at the massive scale of aligning large language models. In this regime, learned reward models are
massive, making an implicit reward method like IPL more
attractive. While we focus on control in our experiments, wehope our work can inform future explorations in language
domains.

Our work builds upon literature in reward learning,
preference-based RL, and imitation learning.
Reward Learning. Due to the challenges associated withdesigning and shaping effective reward signals, several
works have investigated various approaches for learning
reward functions. A large body of work uses inverse RL tolearn a reward function from expert demonstrations (Abbeel& Ng, 2004; Ng et al., 2000; Ramachandran & Amir, 2007),
which are unfortunately difficult to collect (Khurshid &Kuchenbecker, 2015; Akgun et al., 2012; Losey et al., 2020)
or often misaligned with true human preferences (Basu et al.,Imitation Learning. Our work build on foundational2

Inverse Preference Learningknowledge in maximum entropy (MaxEnt) RL (Ziebart,2010) and inverse RL (Ziebart et al., 2008). Recent works
in MaxEnt inverse RL have used the mapping between 𝑄functions and reward functions under a fixed policy. Specifically, Garg et al. (2021) show that the regularized MaxEnt
inverse RL objective from Ho & Ermon (2016) can be rewritten using the 𝑄-function instead of a reward function
and Al-Hafez et al. (2023) stabilize their approach. While
the relationship between 𝑄-functions and rewards has beenused for MaxEnt inverse RL, we study this relationshipwhen learning from preference data. While both problems
seek to learn models of expert reward, the data differs significantly — preference-based RL uses comparisons instead of
optimal demonstrations. This necessitates a greatly different
approach.

to be generated by an expert according to a Bradley-Terry
Preference model (Bradley & Terry, 1952):

𝑃 𝐸 [𝜎 ( 1 ) ≻ 𝜎 ( 2 ) ] =

Í
( 1)
( 1)
𝑡 𝑟𝐸 (𝑠𝑡 ,𝑎𝑡 )
Í
( 1)
( 1)
( 2)
( 2)
𝑡 𝑟𝐸 (𝑠𝑡 ,𝑎𝑡 )+exp 𝑡 𝑟𝐸 (𝑠𝑡 ,𝑎𝑡 )
exp

exp

Í

,

(1)
where 𝑟 𝐸 (𝑠𝑡 , 𝑎 𝑡 ) is again the expert’s unknown underlying
reward model. We use the subscript 𝐸 on probability 𝑃 to
indicate that the preference distribution above results fromthe expert’s reward function. Let the dataset of these preferences be D 𝑝 = {(𝜎 ( 1 ) , 𝜎 ( 2 ) , 𝑦)}. To learn 𝑟 𝐸 , prior works
in preference-based RL estimate a parametric reward function 𝑟 𝜃 by minimizing the binary-cross-entropy over D 𝑝 :hi
L 𝑝 (𝜃) =E 𝜎 (1) , 𝜎 (2) ,𝑦∼D 𝑝 𝑦 log 𝑃 𝜃 𝜎 ( 1 ) ≻ 𝜎 ( 2 )

h
i
(1)
(2)
+ (1 − 𝑦) log 1 − 𝑃 𝜃 𝜎 ≻ 𝜎.

3. Inverse Preference Learning
In this section, we first describe the preference-based RL
problem. Then, we describe how, leveraging techniquesfrom imitation learning, we can remove the independently
learned reward network from prior methods. This results
in a simpler algorithm with lower computational cost andvariance in performance.

(2)

This objective results from simply minimizing
E D 𝑝 [𝐷 KL (𝑃 𝐸 ||𝑃 𝜃 )], the KL-divergence between theexpert preference model and the one induced by 𝑟 𝜃 ,effectively aligning it with the expert’s preferences. We
note that some other works in preference-based RL focus
on learning an improved model 𝑟 𝜃 to address the reward
learning part of the problem (Park et al., 2022; Kim et al.,2023). However, these methods still use off-the-shelf RL
algorithms for the policy learning part of the problem.

3.1. Offline Preference-Based RL
We consider the offline reinforcement learning (RL)
paradigm where an agent seeks to maximize its expectedcumulative discounted sum of rewards in a Markov Decision
Process (MDP) given an offline dataset D𝑜 comprised ofstate, action, and next state tuples (𝑠, 𝑎, 𝑠′ ) generated by an
unknown behavior policy 𝜇(𝑎|𝑠). However, unlike in thestandard offline RL paradigm, in preference-based offline
RL we do not assume access to reward labels in D𝑜 . Instead,the expert reward function 𝑟 𝐸 (𝑠, 𝑎) is unknown and mustbe learned from human feedback. Traditional preferencebased RL methods are thus usually separated into two stages:
first, reward learning, where 𝑟 𝐸 is estimated, and second,
reinforcement learning,where a policy 𝜋(𝑎|𝑠) is learnedÍ𝑡 𝑟 (𝑠, 𝑎)] with 𝛾 as the discountto maximize E 𝜋 [ ∞𝛾𝐸
𝑡=0factor. Though our method combines these two phases, we
use the building blocks of each and consequently review
them here.

Common approaches to offline RL seek to learn conservative policies that do not stray too far away from the distribution of data generated by 𝜇(𝑎|𝑠). This is critical to preventthe policy 𝜋 from reaching out-of-distribution states during
deployment which can be detrimental to performance. A
common solution to this problem is to use a constrained
objective (Kumar et al., 2020). In our derivations, we focus
on the constrained objective from Garg et al. (2023):
"∞
#
∑︁ 
𝜋(𝑎|𝑠)
max E 𝜋𝛾 𝑡 𝑟 (𝑠𝑡 , 𝑎 𝑡 ) − 𝛼 log.
(3)𝜋𝜇(𝑎|𝑠)
𝑡=𝑡 ′The second term subtracted from the reward 𝑟 enforces aKL-divergence constraint with the behavior policy 𝜇, encouraging the policy to remain near the dataset. Off-policy
RL algorithms commonly use the contractive soft-Bellman
operator,

First, similar to prior works (Christiano et al., 2017; Leeet al., 2021a), we assume access to preference data in theform of binary comparisons. Each comparison is comprisedof two behavior segments, 𝜎 ( 1 ) and 𝜎 ( 2 ) , and a binary
label 𝑦 indicating which of the two was preferred by anexpert. As in Wilson et al. (2012), each behavior segment is simply a snippet of a trajectory of length 𝑘, or
𝜎 = (𝑠𝑡 , 𝑎 𝑡 , 𝑠𝑡+1 , 𝑎 𝑡+1 , . . . , 𝑎 𝑡+𝑘−1 , 𝑠𝑡+𝑘 ). Increasing 𝑘 canprovide more information per label at the cost of potentiallynoisier labels. The label 𝑦 for each comparison is assumed(B𝑟𝜋 𝑄) (𝑠, 𝑎) = 𝑟 (𝑠, 𝑎) + 𝛾E𝑠′ ∼ 𝑝 (· |𝑠 | 𝑎) [𝑉 𝜋 (𝑠′ )],𝜋(𝑎|𝑠)
where 𝑉 𝜋 (𝑠) = E𝑎∼ 𝜋 (· |𝑠) 𝑄(𝑠, 𝑎) − 𝛼 log, (4)𝜇(𝑎|𝑠)
for policy evaluation where 𝑄 is the state-action value function and 𝑉 is the state value function. Recent works inoffline RL have shown that the optimal 𝑄-function, 𝑄 ∗ , for3

Inverse Preference Learning(1). For a fixed policy 𝜋 and its corresponding 𝑄 𝜋 , we canobtain the preference model of the implicit reward function
𝑃𝑄 𝜋 [𝜎 ( 1 ) ≻ 𝜎 ( 2 ) ] by substituting the inverse soft-Bellman
operator into Equation (1) as follows:

Equation (3) can be attained directly using the optimal softBellman operator B𝑟∗ via the closed form optimal
 soft-valuefunction 𝑉 ∗ (𝑠) = 𝛼 log E𝑎∼𝜇 (· |𝑠) 𝑒 𝑄 (𝑠,𝑎)/𝛼 (Garg et al.,2023; Xu et al., 2023). After repeatedly applying B𝑟∗ , thepolicy can easily be extracted from 𝑄.

𝑃𝑄 𝜋 [𝜎 ( 1 ) > 𝜎 ( 2 ) ] =To learn the optimal policy, two-phase preference based
RL methods rely on recovering the optimal 𝑟 𝐸 in the reward learning phase before running offline RL. This potentially propagates errors from the estimated 𝑟 𝜃 to learned
𝑄-function and ultimately learned policy 𝜋. In practice, itwould be more efficient to eliminate the need for two separate stages. In the next section, we show how this can bedone by establishing a bijection between reward functions 𝑟
and 𝑄-functions.

Í
( 1)
( 1)
exp 𝑡 ( T 𝜋 𝑄) (𝑠𝑡 ,𝑎𝑡 )
Í
( 1)
( 1)
( 2)
( 2)
𝜋
𝜋
𝑡 ( T 𝑄) (𝑠𝑡 ,𝑎𝑡 )+exp 𝑡 ( T 𝑄) (𝑠𝑡 ,𝑎𝑡 ).

(6)This substitution will allow us to measure the difference
between the preferences implied by 𝑄 𝜋 and those of theexpert. To minimize the difference, we can propagate
gradients through the preference modeling loss (Equation(2)) and the implicit preference model 𝑃𝑄 𝜋 (Equation (6))
to 𝑄—just as we would for a parameterized reward estimate𝑟 𝜃 . Unfortunately, naı̈vely performing the substitution is
insufficient to solve the objective in Equation (3). This isbecause we have used an arbitrary policy 𝜋, not the optimal
one, for converting from 𝑄-values to rewards. Next, weshow how we can form the optimal inverse soft-Bellman
operator T ∗ to ensure the policy extracted from the learned
𝑄-function is indeed optimal.

3.2. Removing The Reward Function
In this section, we formally describe how the reward
function can be removed from offline preference-based RL
algorithms. Our key insight is that the 𝑄-function learned
by off-policy RL algorithms in fact encodes the same
information as the reward function 𝑟 (𝑠, 𝑎). Consequently,
it is unnecessary to learn both. First, we show how the
reward function can be re-written in terms of the 𝑄 function
allowing us to compute the preference model 𝑃𝑄 inducedby the 𝑄-function. Then, we derive an objective thatsimultaneously pushes 𝑄 to fit the expert’s preferences
while also remaining optimal.

The optimal inverse soft-Bellman operator corresponds tothe standard optimal soft-Bellman operator B𝑟∗ , which isformed by using the optimal soft-value function insteadof 𝑉 𝜋 (𝑠) (Garg et al., 2023). Thus to construct T ∗ wesubstitute the form of 𝑉 used in B𝑟∗ into Equation (5):(T ∗ 𝑄) (𝑠, 𝑎) = 𝑄(𝑠, 𝑎) − 𝛾E𝑠′ [𝑉 ∗ (𝑠′ )]hhii= 𝑄(𝑠, 𝑎) − 𝛾E𝑠′ 𝛼 log E𝑎∼𝜇 (· |𝑠) 𝑒 𝑄 (𝑠,𝑎)/𝛼 .

Consider fitting a 𝑄 function via the Bellman operator B𝑟𝜋
for a fixed policy 𝜋 until convergence where B𝑟𝜋 𝑄 = 𝑄.
Here, to encode the cumulative discounted rewards whenacting according to the policy, the 𝑄-function depends on
both 𝑟 and 𝜋. This dependence, however, is directly disentangled by the Bellman equation. By rearranging it (Equation(4)), we can solve for the reward function in terms of 𝑄 and
𝜋. This yields the so-called inverse soft-Bellman operator:(T 𝜋 𝑄) (𝑠, 𝑎) = 𝑄(𝑠, 𝑎) − 𝛾E𝑠′ [𝑉 𝜋 (𝑠′ )].

expÍAbove we use the specific form of 𝑉 ∗ (𝑠) for our KLconstrained objective, but in practice others can be used
for different objectives. This makes our algorithm amenable
to nearly any RL algorithm, like SAC (Haarnoja et al., 2018),
so long as its value estimates converge to those of the desired
policy. Thus in practice we simply update our estimates of𝑉 according to any offline RL algorithm that converges toan optimal policy. For our KL-constrianed objective, thismeans fitting 𝑉 according to the linear exponential (linex)
loss function from Extreme Q-Learning (XQL) (Garg et al.,2023). In Algorithm 3 we show this step written out. Thus,
to fit the optimal 𝑄-function, we minimize the following
loss function:L 𝑝 (𝑄) = E 𝜎 (1) , 𝜎 (2) ,𝑦∼D 𝑝 𝑦 log 𝑃𝑄∗ [𝜎 ( 1 ) ≻ 𝜎 ( 2 ) ]

(
1)
(
2)
+ (1 − 𝑦) log(1 − 𝑃𝑄∗ [𝜎 > 𝜎 ) .

(5)

In fact, it has been shown that for a fixed policy 𝜋, the soft
inverse-Bellman operator is bijective, implying a one-toone correspondence between the 𝑄 function and the reward
function (Garg et al., 2021). Intuitively, this makes sense:
when holding the policy constant, only the reward function
affects 𝑄. We abbreviate the evaluation of (T 𝜋 𝑄) (𝑠, 𝑎) as𝑟 𝑄 𝜋 (𝑠, 𝑎) to indicate that 𝑟 𝑄 𝜋 is the unique implicit reward
function induced by 𝑄 𝜋 . Prior works in imitation learningleverage the inverse soft-Bellman operator to measure how
closely the implicit reward model 𝑟 𝑄 𝜋 aligns with expert
demonstrations. Our key insight is that this equivalence
can also be used to directly measure how closely our 𝑄
function aligns with the expert preference model without
ever directly learning 𝑟.

Unfortunately, optimizing this objective alone leads to poor
results due to the unconstrained nature of the implicit reward
function. In practice, explicit reward learning approaches
often constrain the learned reward function with either Tanh
activations (Lee et al., 2021a) or normalization (Kim et al.,2023). These techniques, however, are inapplicable to our

Consider the Bradley-Terry preference model in Equation
4

Inverse Preference Learning

Inverse Preference LearningStandard Preference-Based RL
𝜎 (1)
𝑟𝜃
𝜎 (2)

Black Box
RL
Algorithm

𝜎 (1)Preference
Distribution

𝑉 𝜋 (𝑠)𝜎 (2)𝑟(𝑠, 𝑎) = 𝑄 𝑠, 𝑎 − 𝛾𝑉 𝜋 (𝑠 ′ )Implicit Reward

𝜋(𝑎|𝑠)𝑄(𝑠, 𝑎)Regularization
Phase 1
Reward Learning

Phase 2
Reinforcement Learning

Single Learning Phase

Figure 1. A depiction of the difference between standard preference-based RL methods and Inverse Preference Learning. Standard
preference-based RL first learns a reward function, then optimizes it with a blockbox RL algorithm. IPL trains a 𝑄 function to directly
fit the expert’s preferences. This is done by aligning the implied reward model with the expert’s preference distribution and applying
regularization.

L (𝑄) = L 𝑝 (𝑄) + 𝜆L𝑟 (𝑄), where 𝜆 is a hyper-parameterthat controls the regularization strength. L 𝑝 (𝑄) encourages
the 𝑄 function to implicitly match the the expert’s preference model under the optimal policy. L𝑟 (𝑄) makes thesolution unique and smooths it. We find that weighting
the regularization equally between D 𝑝 and D𝑜 performs
well. Note that L 𝑝 (𝑄) is only computed on preference data
D 𝑝 . In practice, optimizing L 𝑝 (𝑄) requires estimating(T ∗ 𝑄) and consequently 𝑉 ∗ . Thus, during optimization, we
alternate between updating our learned 𝑄-network and alearned value 𝑉-network towards 𝑉 ∗ . After 𝑄 and 𝑉 have
converged, we can extract the policy using the closed form
relationship 𝜋 ∗ (𝑎|𝑠) ∝ 𝜇(𝑎|𝑠) exp ((𝑄 ∗ (𝑠, 𝑎) − 𝑉 ∗ (𝑠))/𝛼)
for KL-constrained RL as in Garg et al. (2023); Peng et al.
(2019).

implicit algorithm. Instead, we introduce L2 regularization
to the implicit reward, or (𝑟 𝑄∗ (𝑠, 𝑎)) 2 = ((T ∗ 𝑄) (𝑠, 𝑎)) 2 , asis commonly done in imitation learning (Garg et al., 2021;
Al-Hafez et al., 2023) to prevent unbounded reward values.
We find that this also has a number of practical benefits.
First, the solution to the Bradley-Terry preference model
is non-unique, as any constant shift in all reward values
does not change the probability of preferring any given
segment. L2 regularization makes the solution unique andcenters the induced reward near zero, which has shown tobe beneficial for RL (Engstrom et al., 2020). Second, regularization encourages more realistic implicit rewards. For
example, consider a reward function in continuous control
that changes from -100 to 100 when one a small perturbation of size 𝜖 is applied to 𝑠 and 𝑎, but changes back to 100
when another perturbation of size 𝜖 is applied. While such a
reward function seems unrealistic as it is unduly affected bysmall changes in states and actions, it is a completely valid
solution of the inverse soft-Bellman operator. Adding L2
regularization discourages this discontinuous behavior bypenalizing large deviations in implied reward unless they
drastically reduce the preference loss. Finally, the benefitsof implicit reward regularization naturally propagate to thelearned 𝑄-function. L2 regularization explicitly penalizes
neighboring 𝑄 and 𝑉 values from getting too far, smoothing
the value landscape. This helps prevent exploding 𝑄-values
as estimates of 𝑉 typically increase throughout the course oflearning. Thus, to encourage regularization across the entire
state and action space, we use the following regularization
objective over 𝑄 which uses data from both D 𝑝 and D𝑜 :L𝑟 (𝑄) = E𝑠,𝑎∼D𝑜 ∪D 𝑝 ((T ∗ 𝑄) (𝑠, 𝑎)) 2 .

Algorithm 1 IPL Algorithm (XQL Variant)
Input : D 𝑝 , D𝑜 , 𝜆, 𝛼for 𝑖 = 1, 2, 3, ... do
Sample batches 𝐵 𝑝 ∼ D 𝑝 , 𝐵𝑜 ∼ D𝑜Update 𝑄: min𝑄 E 𝐵 𝑝 [L 𝑝 (𝑄)] + 𝜆E 𝐵 𝑝 ∪𝐵𝑜 [L𝑟 (𝑄)]
Update 𝑉: min𝑉 E 𝐵 𝑝 ∪𝐵𝑜 [𝑒 𝑧 − 𝑧 − 1]
where 𝑧 = 𝑄(𝑠, 𝑎) − 𝑉 (𝑠))/𝛼Finally, extract 𝜋(𝑎|𝑠) via:
max 𝜋 E D 𝑝 ∪D𝑜 [𝑒 (𝑄 (𝑠,𝑎) −𝑉 (𝑠) )/𝛼 log 𝜋(𝑎|𝑠)]

We find that in the presence of additional offline data in D𝑜 ,updating the policy and/or value function with data fromboth D𝑜 and D 𝑝 performs better, just like with the regularization term. Ultimately, our optimization scheme learns a
𝑄 function for an optimal policy whose implied reward isconsistent with 𝑟 𝐸 , up to regularization, without ever learning the reward network. Practically, this has a number of

Our final 𝑄-learning objective thus the sum of the preference loss function under T ∗ 𝑄 and the regularization loss, or5Inverse Preference Learning

4.2. How does IPL perform on preference-based RL
benchmarks?

benefits. Learning a reward network requires more parameters and a completely separate optimization loop, increasing
compute requirements. Moreover, an explicit reward model
introduces a whole new suite of hyper-parameters that need
to be tuned including the model architecture, capacity, learning rate, batch size, and stopping criterion. In fact, because
human preference data is so difficult to collect, many approaches opt to use simple accuracy thresholds instead ofvalidation criteria to decide when to stop training 𝑟 𝜃 (Leeet al., 2021a). All of these components make preferencebased RL unreliable and high-variance. On the other hand,
our method completely removes all of these parameters in
exchange for a single 𝜆 hyper-parameter that controls theregularization strength. Despite removing these components,
in the next section we will experimentally demonstrate that
IPL performs just as well as explicit reward approaches.

We compare IPL to other offline preference-based RL
approaches on D4RL Gym Locomotion (Fu et al., 2020)
and Robosuite robotics (Mandlekar et al., 2021) datasetswith real-human preference data from Kim et al. (2023). We
compare IQL-based IPL, with the same hyper-parameters,
to various baselines that learn a reward model 𝑟 𝜃 before
optimization with IQL. Markovian Reward or MR denotes
using a standard Markovian MLP reward model, like thoseused in Christiano et al. (2017) and Lee et al. (2021a).
Non-Markovian Reward or NMR denotes using thenon-Markovian LSTM based reward model from Early et al.
(2022). Preference Transformer (PT) is a state-of-the-art
approach that leverages a large transformer architecture tolearn a non-Markovian reward and preference weighting
function. Finally, we compare against our own implementation of IQL with a Markovian Reward function that use
the same data augmentation as IPL.

4. Experiments
In this section, we aim to answer the following questions:First, how does IPL compare to prior preference-based RL
algorithms on standard benchmarks? Second, how does
IPL perform in extremely data-limited settings? And finally,
how efficient is IPL in comparison to two-phase preferencebased RL methods?

Our results are summarized in Table 1. Starting with thefirst column, we see that preference-based RL methods areable to match IQL with the ground truth reward function in
many cases. On, several tasks, however, the MR implementation from Kim et al. (2023) fairs rather poorly. The nonMarkovian methods, (NMR and PT) improve performance.
It is worth noting that on many tasks our implementationof a MR (fifth column) performs far better than reported
in Kim et al. (2023), likely due to our careful tuning of 𝑟 𝜃and use of data-augmentations. Our method, IPL, achieves
competitive performance across the board.

4.1. Setup
As discussed in the previous section, though we use a
KL-constrained objective for our theoretical derivation, in
practice we can construct versions of IPL based on any
offline RL algorithm. In our experiments we evaluate IPL
with Implicit Q-Learning (IQL) (Kostrikov et al., 2022),
since it has been used in prior offline preference-based
RL works. This allows us to directly compare IPL byisolating its implicit reward component and using thesame exact hyper-parameters as prior works. Using
IPL with IQL amounts to updating the value function
according to the asymmetric expectile loss function
instead of the linex loss function. Concretely, this can bedone by replacing
the value update in Algorithm 3 with
min𝑉 E 𝐵 𝑝 ∪𝐵𝑜 |𝜏 − 𝟙(𝑄(𝑠, 𝑎) − 𝑉 (𝑠))| (𝑄(𝑠, 𝑎) − 𝑉 (𝑠)) 2
where 𝜏 is the expectile.

In general, IPL performs on-par or better than both our
implementation of MR and PT in most datasets despite
not learning a separate reward network. Specifically, IPL
has the same performance or better performance than our
MR implementation on six of eight tasks. More importantly, IPL does extremely well in comparison to Preference
Transformer’s reported results. On five of eight tasks IPL
performs better than PT while having over 10 times fewer
parameters, making IPL far more efficient. To be consistent
with Kim et al. (2023), we report results after a million
training steps but performance for IPL often peaks earlier(see learning curves in the Appendix). For example, withearly stopping IPL also outperforms PT on “hop-m-r”. We
posit that this is because the 𝑄-function in IPL is tasked
with both fitting the expert’s preference model and optimal
policy simultaneously, making both the policy and reward
function non-stationary during training. In some datasets,
this was more unstable.

Inspired by Park et al. (2022), we introduce data augmentations that sample sub-sections of behavior segments 𝜎during training. While such augmentations are inapplicable
to non-Markovian reward models, we find that they boost
performance for Markovian reward models while also reducing the total number of state-action pairs per batch ofpreference data. This is important as IPL needs data fromboth D 𝑝 and D𝑜 to regularize the implicit reward function.
Additional experiment details and hyper-parameters can befound in the Appendix.

IPL also has the lowest average standard-deviation acrossseeds, meaning it yields more consistent results than explicit
reward methods. For standard two-phase preference-based
RL algorithms, errors in the reward model are propagated6

Inverse Preference Learning

Datasethop-m-rhop-m-ewalk-m-r
walk-m-e
lift-phlift-mh
can-phcan-mhAvg Std

IQL(Oracle)83.06 ± 15.80
73.55 ± 41.47
73.11 ± 8.07
107.75 ± 2.02
96.75 ± 1.83
86.75 ± 2.82
74.50 ± 6.82
56.25 ± 8.78
10.95

MR
(from Kim et al.)11.56 ± 30.27
57.75 ± 23.70
72.07 ± 1.96
108.32 ± 3.87
84.75 ± 6.23
91.00 ± 2.82
68.00 ± 9.13
47.50 ± 3.51
10.2

LSTM
(from Kim et al.)57.88 ± 40.63
38.63 ± 35.58
77.00 ± 3.03
110.39 ± 0.93
91.50 ± 5.42
90.75 ± 5.75
62.00 ± 10.90
30.50 ± 8.73
13.87

PT
(from Kim et al.)84.54 ± 4.07
68.96 ± 33.86
71.27 ± 10.30
110.13 ± 0.21
91.75 ± 5.90
86.75 ± 5.95
69.67 ± 5.89
50.50 ± 6.48
9.08

MR
(reimpl.)70.20 ± 35.0
102.97 ± 5.55
68.79 ± 5.64
109.07 ± 1.30
98.84 ± 2.33
90.04 ± 4.45
76.40 ± 3.67
53.6 ± 7.86
8.23

IPL
(Ours)
73.57 ± 6.67
74.52 ± 10.11
59.92 ± 5.11
108.51 ± 0.60
97.60 ± 2.94
87.20 ± 5.31
74.8 ± 2.40
57.6 ± 5.00
4.77

Table 1. Average normalized scores of all baselines on human-preference benchmarks from Kim et al. (2023). For the D4RL locomotion
tasks “hop” corresponds to hopper, “m” to medium (training the data generating agent to 1/3 expert performance), “r” to replay bufferdata, and “e” to data from the end of training. For the Robomimic tasks lift and can, “ph” corresponds to proficient human data and “mh”
to multi-human data of differing optimality. The first four columns are taken from Kim et al. (2023). “reimpl.” is our reimplementation ofMarkovian Reward with IQL. The “Avg Std” row shows the average standard deviation across all eight environments. We run five seeds
and report the final performance at the end of training like Kostrikov et al. (2022). On some tasks IPL achieves higher performance earlier
in training, which is not reflected above (See Appendix). We find that IPL outperforms PT on many environments, and also performs
similarly to our implementation of MR despite not training a reward function.

to and exacerbated by the 𝑄 function. IPL circumvents thisproblem by not explicitly learning the reward.
4.3. How does IPL scale with Data?

Preference Queries

Collecting preference comparisons is often viewed as themost expensive part of preference-based RL. To investigate
how well IPL performs in data limited settings, we constructscripted preference datasets of four different sizes for fivetasks from the MetaWorld benchmark (Yu et al., 2020) usedin prior Preference-based RL works (Lee et al., 2021a; Hejna& Sadigh, 2022). We then train on the preference data
D 𝑝 by setting D𝑜 = {(𝑠, 𝑎, 𝑠′ ) ∈ D 𝑝 } and use the same
hyper-parameters for all environments and methods whereapplicable. Our results are summarized in Table 2. Again,
IPL is a strong reward-free baseline. We find that at all datascales, IPL performs competitively to our implementation of
MR (IQL with a learned Markovian reward) and consistently
outperforms it in Button Press and Assembly. Increasing theamount of preference data generally improves performanceacross the board. However, as we generate queries uniformly
at random some preference datasets may be easier to learn
from than others, leading to deviations form this trend insome cases. As in the benchmark results in Table 1, IPL
exhibits lower variance across seeds and tasks, in this caseat three of four data scales.

500

1000

2000

4000

Button Press

MR
IPL

66.0 ± 8.0
53.3 ± 8.5

49.3 ± 12.1
60.1 ± 12.8

54.7 ± 26.8
70.2 ± 2.5

78.3 ± 9.2
90.2 ± 6.5

Drawer Open

MR
IPL

65.9 ± 9.962.1 ± 4.8

87.2 ± 5.2
78.7 ± 12.4

89.7 ± 6.4
89.5 ± 5.0

94.6 ± 3.9
96.6 ± 1.3

Sweep IntoMR
IPL

33.0 ± 5.7
34.5 ± 2.3

46.2 ± 6.0
48.2 ± 7.2

63.2 ± 13.7
58.8 ± 7.4

70.8 ± 7.9
65.9 ± 6.7

Plate Slide

MR
IPL

54.6 ± 5.3
52.9 ± 4.8

57.2 ± 4.5
55.8 ± 2.2

23.9 ± 18.8
55.4 ± 3.1

55.2 ± 3.0
54.9 ± 2.8

AssemblyMRIPL

0.6 ± 0.7
0.9 ± 0.6

0.7 ± 1.0
1.5 ± 1.5

0.0 ± 0.0
1.7 ± 1.9

2.6 ± 2.8
5.5 ± 5.2

Avg Std

MR
IPL

5.9
4.2

5.76
7.22

13.14
3.98

5.36
4.5

Table 2. Results on five MetaWorld tasks at four different preference data scales. We run five seeds for each method, and moredetails can be found in the Appendix. IPL performs the same or
better than IQL with a Markovian reward model on the majority oftasks and preference data scales without training a reward model.

ing the same performance. In Table 3, we show the numberof parameters for each method used in the last two sections.
Preference Transformer uses over ten times more parameters than IPL, and the LSTM-based NMR model from Early
et al. (2022) uses nearly twice as many. When dealing witha limited compute or memory budget, this can be important.
To exacerbate this effect, we consider an extremely parameter efficient version of IPL, denoted “IPL (64)” in Table 3,
based on Advantage Weighted Actor Critic (AWAC) (Nairet al., 2021) which eliminates the second critic and value
networks used in IQL (Kostrikov et al., 2022) and uses
a two-layer 64-dimensional MLP. We then compare thisparameter-efficient IPL to MR with the same parameter4.4. How efficient is IPL?
One benefit of IPL over other Preference-based RL methods
is its parameter efficiency. By removing the reward network,
IPL uses fewer parameters than other methods while achiev7

Inverse Preference Learning

MetaWorld Drawer Open Ablations with 4000 Queries

1.0

Parameter Equalization

MR Reward Steps

1.0

1.0

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.2

0.2

Success Rate

0.8

0.0

50000 100000 150000 200000

Training Steps

MR (35)

MR (64)

0.0

1000
10000
10000050000 100000 150000 200000

IPL (64)

IPL Regularization Weight

0.4

0.4
2
10

0.2
0.0

Training Steps

50000 100000 150000 200000

Training Steps

Figure 2. Left: Performance comparison with different parameter numbers. MR (35) has the same parameter budget as IPL (64). MR (64)has over twice as many. We see that with the same number of parameters as IPL, MR is unable to adequetly fit the data and performs
poorly. Middle: MR when the reward function is trained for a varying number of steps – with too few the reward model under-fits, andwith too many it over-fits, both leading to worse performance. Right: IPL with different regularization strengths. On the drawer open task,
performance is largely unaffected. For more ablations, see the Appendix.

Method
PT
NMR
MR
IPL
MR (64)
IPL (64)
MR (35)

Params2942218508746348426278537
34892
14025
14012

5. Conclusion
Summary. We introduce Inverse Preference Learning, anovel algorithm for offline preference-based RL that avoids
learning a reward function. Our key insight is to leverage
the inverse soft-Bellman operator, which computes the
mapping from 𝑄-functions to rewards under a fixed policy.
The IPL algorithm trains a 𝑄-function to regress towards the
optimal 𝑄 ∗ while at the same time admitting implicit reward
values that are consistent with an expert’s preferences. Even
though IPL does not require learning a separate reward
network, on robotics benchmarks it attains competitive
performance with preference-based RL baselines that use
twice to ten-times the number of model parameters.

Table 3. Parameter counts for different methods. The bottom threerows are for the limited parameter budget experiments in Section
4.4.

Limitations and Future Work. A number of future directions remain. Specifically, the implicit reward function
and policy learned by IPL are both non-stationary during
training, which sometimes causes learning to be more unstable than with a fixed reward function. This is a core
limitation future work could address by better mixing policy
improvement and preference-matching steps to improve stability. More broadly, implicit reward preference-based RL
methods are not limited to continuous control or binary feedback. Applying implicit reward techniques to other forms of
feedback or extending IPL to language-based RLHF tasksremain exciting future directions.

budget which results in “MR (35)”, a 35-dimensional MLP.
Results are depicted on the left of Figure 2. MR trained witha smaller network is unable to adequately fit the data, resulting in lower performance. Only after increasing the network
size past that of IPL can MR begin to match performance.
Aside from parameter efficiency, IPL is also “hyperparameter efficient”. By removing the reward network,
IPL removes a whole set of hyper-parameters associatedwith two phase preference based RL methods, like reward
network architecture, learning rate, stopping criterion, andmore. In the middle of Figure 2 we show how the performance of MR is affected when the reward function is overor under fit. Choosing the correct number of steps to train
the reward model usually requires collecting a validation
set of preference data, which is costly to obtain. Insteadof this, IPL only has a single regularization parameter, 𝜆.
The right side of Figure 2 shows the sensitivity of IPL to𝜆. We find that in many cases, varying 𝜆 has little effecton performance unless it is perturbed by a large amount.
Due to space constraints, extended results for this section
are included in the Appendix.

References
Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse
reinforcement learning. In International Conference on
Machine Learning, 2004.
Akgun, B., Cakmak, M., Jiang, K., and Thomaz, A. L.
Keyframe-based learning from demonstration. International Journal of Social Robotics, 4(4):343–355, 2012.
8

Inverse Preference Learning

Akrour, R., Schoenauer, M., and Sebag, M. Preferencebased policy learning. In Joint European Conference
on Machine Learning and Knowledge Discovery in
Databases, 2011.

Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos,F., Rudolph, L., and Madry, A. Implementation matters in deep rl: A case study on ppo and trpo. In
International Conference on Learning Representations,2020. URL https://openreview.net/forum?
id=r1etN1rtPB.

Al-Hafez, F., Tateo, D., Arenz, O., Zhao, G., and Peters,
J. LS-IQ: Implicit reward regularization for inverse reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=o3Q4m8jg4BR.

Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine,
S. D4rl: Datasets for deep data-driven reinforcement
learning. arXiv preprint arXiv:2004.07219, 2020.

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. Concrete problems in ai safety.
arXiv preprint arXiv:1606.06565, 2016.

Garg, D., Chakraborty, S., Cundy, C., Song, J., and Ermon, S. Iq-learn: Inverse soft-q learning for imitation.
In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openreview.
net/forum?id=Aeo-xqtb5p.

Bacchus, F., Boutilier, C., and Grove, A. Rewarding behaviors. In National Conference on Artificial Intelligence,
1996.

Garg, D., Hejna, J., Geist, M., and Ermon, S. Extremeq-learning: Maxent RL without entropy. In The EleventhInternational Conference on Learning Representations,2023. URL https://openreview.net/forum?
id=SJ0Lde3tRL.

Basu, C., Yang, Q., Hungerman, D., Sinahal, M., andDraqan, A. D. Do you want your autonomous car todrive like you? In 2017 12th ACM/IEEE International
Conference on Human-Robot Interaction (HRI, pp. 417–
425. IEEE, 2017.

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Softactor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International
Conference on Machine Learning, 2018.

Bıyık, E., Lazar, D. A., Sadigh, D., and Pedarsani, R. The
green choice: Learning and influencing human decisions
on shared roads. In 2019 IEEE 58th conference on decision and control (CDC), pp. 347–354. IEEE, 2019.

Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., andDragan, A. Inverse reward design. Advances in neural
information processing systems, 30, 2017.

Biyik, E., Huynh, N., Kochenderfer, M. J., and Sadigh, D.
Active preference-based gaussian process regression for
reward learning. In Proceedings of Robotics: Science andSystems (RSS), July 2020.

Hejna, J. and Sadigh, D. Few-shot preference learning forhuman-in-the-loop RL. In Conference on Robot Learning,2022.

Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons.
Biometrika, 39(3/4):324–345, 1952.

Ho, J. and Ermon, S. Generative adversarial imitation learning. Advances in neural information processing systems,
29, 2016.

Brown, D., Goo, W., Nagarajan, P., and Niekum, S. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International
conference on machine learning, pp. 783–792. PMLR,
2019.

Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., andAmodei, D. Reward learning from human preferencesand demonstrations in atari. In Advances in Neural Information Processing Systems, 2018.

Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg,
S., and Amodei, D. Deep reinforcement learning fromhuman preferences. In Advances in Neural Information
Processing Systems, 2017.

Khurshid, R. P. and Kuchenbecker, K. J. Data-driven motion
mappings improve transparency in teleoperation. Presence, 24(2):132–154, 2015.

Daniel, C., Kroemer, O., Viering, M., Metz, J., and Peters, J.
Active reward learning with a novel acquisition function.
Autonomous Robots, 39(3):389–405, 2015.

Kim, C., Park, J., Shin, J., Lee, H., Abbeel, P., and Lee,K. Preference transformer: Modeling human preferencesusing transformers for rl. In International Conference on
Learning Representations, 2023.

Early, J., Bewley, T., Evers, C., and Ramchurn, S. Nonmarkovian reward modelling from trajectory labels via
interpretable multiple instance learning. In Advances in
Neural Information Processing Systems, 2022.

Knox, W. B. and Stone, P. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE
international conference on development and learning,pp. 292–297. IEEE, 2008.
9Inverse Preference Learning

Nair, A., Dalal, M., Gupta, A., and Levine, S. {AWAC}:
Accelerating online reinforcement learning with offline
datasets, 2021. URL https://openreview.net/
forum?id=OJiM1R3jAtZ.

Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. In International
Conference on Learning Representations, 2022.

Kumar, A., Zhou, A., Tucker, G., and Levine, S.
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim,
Conservative q-learning for offline reinforcement
C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al.
learning.
In Larochelle, H., Ranzato, M., HadWebgpt: Browser-assisted question-answering with husell, R., Balcan, M., and Lin, H. (eds.), Advances
man feedback. arXiv preprint arXiv:2112.09332, 2021.
in Neural Information Processing Systems, volume 33, pp. 1179–1191. Curran Associates, Inc.,
Ng, A. Y., Russell, S. J., et al. Algorithms for inverse
2020. URL https://proceedings.neurips.
reinforcement learning. In International Conference on
cc/paper_files/paper/2020/file/
Machine Learning, 2000.
0d2b2061826a5df3221116a5085a6052-Paper.
pdf.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright,
C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama,Kwon, M., Biyik, E., Talati, A., Bhasin, K., Losey, D. P.,
K., Ray, A., et al. Training language models to foland Sadigh, D. When humans aren’t optimal: Robots
low instructions with human feedback. arXiv preprintthat collaborate with risk-aware humans. In 2020 15tharXiv:2203.02155, 2022.
ACM/IEEE International Conference on Human-Robot
Park, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., and Lee, K.
Interaction (HRI), pp. 43–52. IEEE, 2020.
Surf: Semi-supervised reward learning with data augmenLee, K., Smith, L., and Abbeel, P. Pebble: Feedbacktation for feedback-efficient preference-based reinforceefficient interactive reinforcement learning via relabelingment learning. In International Conference on Learning
experience and unsupervised pre-training. In InternaRepresentations, 2022.
tional Conference on Machine Learning, 2021a.
Peng, X. B., Kumar, A., Zhang, G., and Levine, S.
Advantage-weighted regression: Simple and scalable
Lee, K., Smith, L., Dragan, A., and Abbeel, P. B-pref:
off-policy reinforcement learning.
arXiv preprint
Benchmarking preference-based reinforcement learning.
arXiv:1910.00177,
2019.
In Conference on Neural Information Processing Systems
Datasets and Benchmarks Track (round 1), 2021b.
Ramachandran, D. and Amir, E. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pp. 2586–2591,
Lin, J., Fried, D., Klein, D., and Dragan, A. Infer2007.
ring rewards from language in context. arXiv preprint
arXiv:2204.02515, 2022.
Sadigh, D., Dragan, A. D., Sastry, S., and Seshia, S. A.
Active preference-based learning of reward functions. In
Losey, D. P., Srinivasan, K., Mandlekar, A., Garg, A., andRobotics: Science and Systems, 2017.
Sadigh, D. Controlling assistive robots with learned latent actions. In 2020 IEEE International Conference on
Schenck, C. and Fox, D. Visual closed-loop control for
Robotics and Automation (ICRA), pp. 378–384. IEEE,
pouring liquids. In International Conference on Robotics
2020.
and Automation, 2017.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., andKlimov, O. Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.

Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C.,
Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., andMartı́n-Martı́n, R. What matters in learning from offline human demonstrations for robot manipulation. In
Conference on Robot Learning (CoRL), 2021.

Shin, D. and Brown, D. S. Offline preference-based apprenticeship learning. arXiv preprint arXiv:2107.09251,
2021.

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A.,
Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing
atari with deep reinforcement learning. arXiv preprint
arXiv:1312.5602, 2013.

Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe,
R., Voss, C., Radford, A., Amodei, D., and Christiano,
P. Learning to summarize from human feedback. arXiv
preprint arXiv:2009.01325, 2020.

Myers, V., Biyik, E., Anari, N., and Sadigh, D. Learning
multimodal rewards from rankings. In Conference on
Robot Learning, pp. 342–352. PMLR, 2022.

Wilson, A., Fern, A., and Tadepalli, P. A bayesian approachfor policy learning from trajectory preference queries.
10

Inverse Preference Learning

In Advances in Neural Information Processing Systems,
2012.
Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe,
R., Leike, J., and Christiano, P. Recursively summarizing books with human feedback. arXiv preprint
arXiv:2109.10862, 2021.
Xu, H., Jiang, L., Li, J., Yang, Z., Wang, Z., Chan, V. W. K.,
and Zhan, X. Offline rl with no ood actions: In-sample
learning via implicit value regularization. In International
Conference on Learning Representations, 2023.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn,C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In
Conference on Robot Learning, 2020.
Zhu, H., Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh,
A., Kumar, V., and Levine, S. The ingredients of realworld robotic reinforcement learning. In International
Conference on Learning Representations, 2020.
Ziebart, B. D. Modeling purposeful adaptive behavior withthe principle of maximum causal entropy. Carnegie Mellon University, 2010.
Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al.
Maximum entropy inverse reinforcement learning. In
Aaai, volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.

11

Inverse Preference Learning

Appendix
We divide the appendix into four different sections following the results section. Each section additionally provides
hyper-parameters used for IPL in that section. The first section, setup, contains details information on the experimental
setup and hyper-parameters used. The second section on benchmark results gives full learning curves for the experiments in
Section 4.2. The third section provides full learning curves for the MetaWorld and Data-scaling experiments. The final
Appendix section provides extended ablations.

A. Setup
Here we provide the full algorithmic outline of IPL using Implicit Q-Learning (Kostrikov et al., 2022) that mimics our
implementation. While in practice the policy 𝜋 could be extracted at the end of training, we do it simultaneously as in(Kostrikov et al., 2022) in order to construct learning curves.

Algorithm 2 IPL Algorithm (IQL Variant)
Input : D 𝑝 , D𝑜 , 𝜆, 𝛼for 𝑖 = 1, 2, 3, ... do
Sample batches 𝐵 𝑝 ∼ D 𝑝 , 𝐵𝑜 ∼ D𝑜Update 𝑄: min𝑄 E 𝐵 𝑝 [L 𝑝 (𝑄)] + 𝜆E 𝐵 𝑝 ∪𝐵𝑜 [L𝑟 (𝑄)]
Update 𝑉: min𝑉 E 𝐵 𝑝 ∪𝐵𝑜 |𝜏 − 𝟙(𝑄(𝑠, 𝑎) − 𝑉 (𝑠))| (𝑄(𝑠, 𝑎) − 𝑉 (𝑠)) 2
Update 𝜋: max 𝜋 E D 𝑝 ∪D𝑜 [𝑒 𝛽 (𝑄 (𝑠,𝑎) −𝑉 (𝑠) ) log 𝜋(𝑎|𝑠)]

Note that above we write the temperature parameter 𝛽 as done in IQL, instead of how it is usually done, using 𝛼 in the
denominator (𝐺𝑎𝑟𝑔𝑒𝑡 𝑎𝑙., 2023; 𝑃𝑒𝑛𝑔𝑒𝑡 𝑎𝑙., 2019).
When sampling batches of preference data 𝐵 𝑝 ∼ D 𝑝 , we take sub-samples of each segment 𝜎 of length 𝑠. For a sampled
data point (𝜎 ( 1 ) , 𝜎 ( 2 ) , 𝑦), we sample start ∼ Unif[0, 1, 2, ...𝑘 − 𝑠] and then let take 𝜎 = 𝑠start , 𝑎 start , ..., 𝑠start+𝑠 . We use thesame start value across the entire batch.
Given that we run experiments using MLPs, all of our experiments were run on CPU compute resources. Each seed for eachmethod requires one CPU core (two hyper-threads) and 8 Gb of memory.

B. Benchmark Results
Here we provide details for our experiments on the preference-based RL benchmark from Kim et al. (2023). We use thesame hyperparameters as Kim et al. (2023) and Kostrikov et al. (2022) where applicable as shown in Table 4.
Gym-Mujoco Locomotion. Hopper and Walker2D agents are tasked with learning locomotion policies from datasets ofvarying qualities taken from the D4RL (Fu et al., 2020) benchmark. Preference datasets were constructed by Kim et al.
(2023) by uniformly sampling queries and labeling a subset of them. For all locomotion tasks the segment length. Preference
datasets for “medium” quality offline datasets contain 500 queries, while preference datasets for “expert” quality offline
datasets contain 100 queries. Segment lengths 𝑘 = 100 for all datasets, and were subsampled to length 𝑠 = 64 by IPL andour MR (reimplementation). Evaluation was preformed over 10 episodes every 5000 steps. Full learning curves are shownin Figure 3.
RoboMimic. The RoboMimic datasets contain interaction data of two types: ph — proficient human and mh – multihuman.
The multi-human data was collected from human demonstrators of mixed quality. The robot is tasked with learning how
to lift a cube (lift) or pick and place a can (can). Preference datasets were again taken directly from Kim et al. (2023).
Preference datasets of size 100 with segment lengths 𝑘 = 50, randomly sub-sampled to length 𝑠 = 32 were used for theph datasets. Preference datasets of size 500 with segment lengths 𝑘 = 100, randomly sub-sampled to length 𝑠 = 64 wereused for the mh datasets. Evaluation was performed over 25 episodes every 50000 steps. Full learning curves are shown in
Figure 4.
12Inverse Preference Learning

D4RL Mujoco Locomotion with Human Preferences

Hopper Medium Replay

1.0

0.8

0.8

Score

1.0

0.6

0.6

0.4

0.4

0.2

0.2
0.0

0.2

0.4

0.6

Hopper Medium Expert

1.2

0.8

1.0

1e6

Walker2D Medium Replay

0.0

0.2

0.4

0.6

0.8

Walker2D Medium Expert

1.0

1e6

1.1

0.7

1.0

0.6

Score

0.9

0.5

0.8

0.4
0.7
0.3
0.0

0.2

PT

0.4

MR

0.6

Training Steps

MR (reimpl.)

0.8

1.0

0.6

1e6
IPL

0.0

0.2

0.4

0.6

Training Steps

0.8

1.0

1e6

Figure 3. Full learning curves on the D4RL locomotion benchmark with human preferences.

Common Hyperparameters
Parameter
Locomotion Robomimic
𝑄, 𝑉, 𝜋 Arch
2x 256d
2x 256d
Learning Rate
0.00030.0003
Optimizer
Adam
Adam
𝛽
3.0
0.5
𝜏
0.7
0.7
D𝑜 Batch Size 256256
D 𝑝 Batch Size 88Training Steps 1 Mil
1 Mil
𝑘
100
100, 50
Subsample 𝑠
64
64, 32MR Hyperparameters
Parameter
Locomotion Robomimic𝑟 𝜃 Arch
2x 256d
2x 256d𝑟 𝜃 LR
0.00030.0003𝑟 𝜃 Optimizer Adam
Adam𝑟 𝜃 Steps
20k
20k

IPL Hyperparameters
Parameter
Locomotion Robomimic
𝜆0.5
4

Table 4. Hyperparameters used for the benchmark experiments. We can see that IPL has fewer hyperparameters.

13

Inverse Preference Learning

Robomimic with Human Preferences

Lift PH

1.0

Lift MH

1.0
0.8

Success Rate

0.8
0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.2

0.4

0.6

0.8

Can PH

1.0

1.0

0.0

1e6

0.2

0.4

0.8

1.0

0.8

1.0

Can MH

1.0

1e6

0.8

Success Rate

0.8

0.6

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.2

PT

0.4

MR

0.6

Training Steps

MR (reimpl.)

0.8

1.0

0.0

1e6
IPL

0.2

0.4

0.6

Training Steps

Figure 4. Full learning curves on the RoboMimic benchmark with human preferences.

14

1e6

Inverse Preference Learning

Common Hyperparameters
Parameter
Value
𝑄, 𝑉, 𝜋 Arch
3x 256d
Learning Rate
0.0003
Optimizer
Adam
𝛽
4.0
𝜏
0.7
D 𝑝 Batch Size 16
Training Steps 200k𝑘
25
Subsample 𝑠
16

MR Hyperparameters
ParameterValue𝑟 𝜃 Arch
3x 256d𝑟 𝜃 LR
0.0003𝑟 𝜃 Optimizer Adam𝑟 𝜃 Steps
20k

IPL Hyperparameters
Parameter
Locomotion
𝜆
0.5

Table 5. Hyper-parameters used in the MetaWorld data scaling experiments.

C. Data Scaling Results
Experiments for data scaling were conducted on the MetaWorld benchmark from Yu et al. (2020). Offline datasets for fivedifferent MetaWorld tasks were constructed as follows: Collect 100 trajectories of expert data on the target task using thebuilt in ground truth policies with the addition of Gaussian noise of standard deviation 1.0. Collect 100 trajectories of
sub-optimal data by running the ground-truth policy for a different randomization of the target task with Gaussian noise
1.0. Collect 100 trajectories of even more sub-optimal data by running the ground truth policy of a different task with
Gaussian noise standard deviation 1.0 in the target domain. Finally, collect 100 trajectories with uniform random actions. As
MetaWorld episodes are 500 steps long, this results in 200,000 time-steps of data. We then construct preference datasets byÍÍ
uniformly sampling segments from the offline dataset and assigning labels 𝑦 according to 𝑡 𝑟 (𝑠𝑡( 1 ) , 𝑎 𝑡( 1 ) ) > 𝑡 𝑟 (𝑠𝑡( 2 ) , 𝑎 𝑡( 2 ) )where 𝑟 is the ground truth reward provided by metaworld. We then train using only the data from D General architecture
hyper-parameters were taken from Lee et al. (2021a); Hejna & Sadigh (2022) which also use the MetaWorld benchmark, but
for online preference-based RL. Full-hyper parameters are shown in Table 5. We run 20 evaluation episodes every 2500
steps. Full learning curves are shown in Figure 5. When reporting values in Table 2, we choose the maximum point on thelearning curves which average across five seeds. This provides results as if early stopping was given by an oracle, which isless optimistic than averaging the maximum of each seed as done in Mandlekar et al. (2021).

D. Ablations
In this section we provide additional ablations on both the benchmark datasets and MetaWorld datasets. We keep thehyperparameters the same, except for the parameter-efficient experiments. We run hyper-parameter sensitivty results for thehuman-preference benchmark datasets in Figure 6. The top row depicts the sensitivity for IPL to the value of 𝜆. The bottom
row depicts the sensitivity of MR to the number of timesteps the reward function is trained for.
For the parameter-efficient experiments only we use an efficient version of IPL based on AWAC (Nair et al., 2021) toadditionally remove the value network. The outline of this variant is given below
Algorithm 3 IPL Algorithm (AWAC Variant)
Input : D 𝑝 , D𝑜 , 𝜆, 𝛼for 𝑖 = 1, 2, 3, ... do
Sample batches 𝐵 𝑝 ∼ D 𝑝 , 𝐵𝑜 ∼ D𝑜
Estimate 𝑉 as 𝑄(𝑠, 𝜋(𝑠))Update 𝑄: min𝑄 E 𝐵 𝑝 [L 𝑝 (𝑄)] + 𝜆E 𝐵 𝑝 ∪𝐵𝑜 [L𝑟 (𝑄)]
Update 𝜋: max 𝜋 E D 𝑝 ∪D𝑜 [𝑒 𝛽 (𝑄 (𝑠,𝑎) −𝑄 (𝑠, 𝜋 (𝑠) ) ) log 𝜋(𝑎|𝑠)]

For this version of IPL, we use 𝜆 = 0.5. All other hyper-parameters remain the same as in Table 6 except the architectures.
For the parameter-efficiency experiments only we use MLPs consisting of two dense layers with either dimension 64 ordimension 35. Running MR with a two-layer MLP of dimension 35 has almost exactly the same number of parameters
as IPL-AWAC with two-layer MLPs of dimension 64. We include full results for the parameter-efficiency experiments in15

Inverse Preference Learning

1.0

Button Press, 500

MetaWorld with Scripted Preferences

1.0

Button Press, 1000

1.0

Button Press, 2000

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

Success Rate

0.8

0.0
1.0

50000 100000 150000 200000

Drawer Open, 500

0.0
1.0

50000 100000 150000 200000

Drawer Open, 1000

0.0
1.0

50000 100000 150000 200000

Drawer Open, 2000

0.0
1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

Success Rate

0.8

1.0

50000 100000 150000 200000

Sweep Into, 500

1.050000 100000 150000 200000

Sweep Into, 1000

1.0

50000 100000 150000 200000

Sweep Into, 2000

0.0
1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

Success Rate

0.8

1.0

50000 100000 150000 200000

Plate Slide, 500

1.050000 100000 150000 200000

Plate Slide, 1000

1.0

50000 100000 150000 200000

Plate Slide, 2000

0.0
1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

Success Rate

0.8

0.0

Assembly, 500

50000 100000 150000 200000

Training Steps

MR (reimpl.)

0.0
0.10
0.08
0.06
0.04
0.02
0.00

Success Rate

0.10
0.08
0.06
0.04
0.02
0.00

50000 100000 150000 200000

IPL

50000 100000 150000 200000

Assembly, 1000

0.0
0.10
0.08
0.06
0.04
0.02
0.00

50000 100000 150000 200000

Training Steps

50000 100000 150000 200000

Assembly, 2000

50000 100000 150000 200000

Training Steps

0.0
0.10
0.08
0.06
0.04
0.02
0.00

Button Press, 4000

50000 100000 150000 200000

Drawer Open, 4000

50000 100000 150000 200000

Sweep Into, 4000

50000 100000 150000 200000

Plate Slide, 4000

50000 100000 150000 200000

Assembly, 4000

50000 100000 150000 200000

Training Steps

Figure 5. Full learning curves for the MetaWorld data scaling results with scripted preferences.

16

Inverse Preference Learning

Regularization Ablation with IPL

1.2

Walker2D Medium Expert

1.0

0.7

Score

Score

0.4

0.1
0.5
2.5

0.2
0.1
0.2

0.4

0.6

0.8

Training Steps

0.0

1e6

0.2

0.4

0.6

0.8

Training Steps

0.7

Walker2D Medium Expert

1.0

1.0

0.8

0.6

0.8

Training Steps

50000 100000 150000 200000

1.0

1e6

0.6

1000
10000
100000

0.5
0.4
0.0

0.2

0.4

0.6

0.8

Training Steps

0.2
0.0

1.0

1.0

Drawer Open, 4000

0.6
0.4

1000
10000
10000050000 100000 150000 200000

1e6

Training Steps

0.8

0.4

0.7

1000
10000
100000Drawer Open, 500

0.6

Score

Score

0.5

0.4

0.0

Training Steps

1.1

0.4
2
10

0.2

50000 100000 150000 200000

1e6

0.8

0.2

0.0

Success Rate

0.6

0.3

0.4
2
10

0.2

1.0

0.9

0.4

0.4

Reward Training Steps Ablation on Benchmark Tasks with MR

Walker2D Medium Replay

0.0

0.1
0.5
2.5

0.4

1.0

0.6

0.4

0.6

0.3

0.8

0.6

0.8

Drawer Open, 4000

0.8

Success Rate

0.5

1.0

0.8

1.0

0.6

Drawer Open, 500

Success Rate

Walker2D Medium Replay

Success Rate

0.8

Training Steps

0.2
0.0

1000
10000
10000050000 100000 150000 200000

Training Steps

Figure 6. Ablations on regularization strength 𝜆 for IPL (top row) and the number of reward steps for MR (bottom row). We see that IPL
is relatively consistent across different values of 𝜆. MR on the other hand, can vary greatly if the reward function under or over fits. In
Walker2D Medium Replay and Drawer Open, 500, we see that it can easily under-fit. In Walker2D Medium Expert it easily over-fits.

17

Inverse Preference Learning

Preference Queries

500

1000

2000

4000

MR (35)
MR (64)
IPL (64)
MR (35)
MR (64)
IPL (64)
MR (35)
MR (64)
IPL (64)
MR (35)
MR (64)
IPL (64)73.9 ± 8.9
54.2 ± 16.1
65.8 ± 13.3
13.4 ± 13.9
13.4 ± 19.0
89.8 ± 11.3
35.1 ± 8.9
31.1 ± 6.4
41.1 ± 14.2
55.2 ± 6.1
46.6 ± 21.9
54.9 ± 3.2

86.8 ± 8.2
42.6 ± 33.0
79.8 ± 18.1
12.6 ± 21.9
57.1 ± 31.2
93.2 ± 2.5
42.4 ± 9.9
55.8 ± 5.9
63.9 ± 8.0
51.1 ± 4.4
50.8 ± 0.6
49.4 ± 1.6

89.9 ± 14.4
67.1 ± 14.9
80.0 ± 17.3
15.5 ± 20.1
54.5 ± 31.7
99.5 ± 0.9
45.9 ± 9.6
49.6 ± 10.3
65.0 ± 12.0
53.0 ± 2.0
47.0 ± 2.5
45.2 ± 9.0

99.0 ± 1.0
43.4 ± 7.4
95.8 ± 5.2
18.4 ± 25.6
78.8 ± 12.2
95.5 ± 3.7
35.9 ± 4.1
56.4 ± 10.3
63.9 ± 11.8
48.9 ± 3.3
48.5 ± 4.6
48.8 ± 4.9

Button Press

Drawer Open

Sweep Into

Plate Slide

Table 6. Performance of different methods on the MetaWorld tasks under a limited parameter budget. MR (35) and IPL (64) have the same
number of parameters. The Assembly task is ommited due to low success rate. On Button Press, fewer parameters appears to performbetter as, due to the simplicity of the task, its easier for the bigger models to overfit. On Drawer Open and Sweep Into, we see consistent
gains from increasing the number of parameters in the network, and IPL performs best overall. On the Plate Slide task, all methods at
different parameter scales perform similarly.

Table 6. We find that on Drawer Open and Sweep Into, IPL outperforms both MR (64) and MR (35). In these environments,
performance increases from MR (35) to MR (64) indicating that the expressiveness of the 𝑄-function and policy are limiting
performance. For the same budget, IPL is able to perform better. In Button Press, the simplest task, we find that MR (64)actually over-fits more than MR (35) and MR (64) ends up performing worse. In Plate Slide, all methods perform similarly
independent of parameter count. We omit Assembly because of its low success rate at all data scales.

18