Inverse Preference Learning: Preference-based RL without a Reward Function
Authors: Joey Hejna
Dorsa Sadigh
Venue: NeurIPS 2023 poster (2023)
Keywords: preference learning
preference-based reinforcement learning
human-in-the-loop reinforcement learning
TL;DR: We design an offline preference-based RL algorithm that does not require learning a reward function.

Abstract: Reward functions are difficult to design and often hard to align with human intent. Preference-based Reinforcement Learning (RL) algorithms address these problems by learning reward functions from human feedback. However, the majority of preference-based RL methods na\"ively combine supervised reward models with off-the-shelf RL algorithms. Contemporary approaches have sought to improve performance and query complexity by using larger and more complex reward architectures such as transformers. Instead of using highly complex architectures, we develop a new and parameter-efficient algorithm, Inverse Preference Learning (IPL), specifically designed for learning from offline preference data. Our key insight is that for a fixed policy, the $Q$-function encodes all information about the reward function, effectively making them interchangeable. Using this insight, we completely eliminate the need for a learned reward function. Our resulting algorithm is simpler and more parameter-efficient. Across a suite of continuous control and robotics benchmarks, IPL attains competitive performance compared to more complex approaches that leverage transformer-based and non-Markovian reward functions while having fewer algorithmic hyperparameters and learned network parameters. Our code is publicly released.

Inverse Preference Learning: Preference-based RL without a Reward Function
Joey Hejna
Dorsa Sadigh Stanford University dorsa@cs .tt.nstrtaororrror edu
Stanford University jhejnancs stanford.
Abstract
Reward functions are difficult to design and often hard to align with human intent. Preference-based Reinforcement Learning (RL) algorithms address these problems by learning reward functions from human feedback. However, the majority of preference-basee baseed RL methods naïvely combine supervised reward models with off-theesheelf RI. algorithms. Contemporary approaches have sought to improve performance and query complexity by using larger and more complex reward architectures such as tramsformers. Instead of using highly complex architectures, we develop a new and parameter--effcieen algorithm, Inverse Preference Learning (IPL), specifically designed for learning from offline preference data. Our key insight is that for lixed pulicy, the Q-lunction encudes all information about the reward function, effectively making them interchangeable. Using this insight, we completely climinate the need for learned reward function. Our resulting algorithm is simpler and more parameter-efficies control and robotics benchmarks, IPL. attains competitive performance compared to more complex approaches that leverage transformer-based and non-Markovian reward functions while having fewer algorithmic hyperparameters and learned network parameters. Our code is publicly released
Across a suite of continuous
1 Introduction
Reinforcement Learning (RL) has shown marked success in fixed and narrow domains such as simulated control [20] and game-playing [37]. When deploying RL. in more complex settings, like in robotics or interaction with humans, one often runs into a critical bottleneck: the reward function. Obtaining reward labels in the real world can be complex, requiring difficult instrumentation [48, 57] and painstaking tuning [56] to achieve reasonable levels of sample efficiency. Moreover, despite extensive engineering. reward functions can still be exploited by algorithms in ways that do not align with human values and intents [21], which can detrimental safety-critical applications [5].
Instead of hand-designing reward functions, contemporary works have attempted to learn them through expert demonstrations [1]. natural language [34], or human feedback [47. 3, 533. Recently. reward functions learned through pairwise comparison queries where user is asked which of two demonstrated behaviors they prefer have been shown be effective in both control [13, 47, 32] and natural language domains [51]. This is often referred to Reinforcement Learning with Human Feedbuck (RI.FF. Reward functions learned via RIHF can directly capture human intent, while avoiding alternative and more expensive forms of human feedback such as expert demonstrations. Preference-basee RI. algorithms for RI. often interleave reward-learning from comparisons with off-the-shelf RL algorithms..
While preference- RL methods discover reward functions that are aligned with human pref- erences, they are not without llaws. Learned reward functions must have adequate coverage of
'nttps://githue http:g/thha..cccc/j/////a///aa/a htwp:///hhhoooaeeeeeeeeeeereaaee learmenn
37th Conference Neural Information Processing Systems (NeurIPS 2023).

---

both the state and action space to attain good downstream performance. Consequently, learning the reward function can be expensive, usually requiring thousands of labeled preference queries. To mitigate these challenges, recent works have proposed improving learned reward functions by adding inductive biases before optimization with RL. Hejna and Sadigh [22] pretrain reward functions with meta- veta-learning. Park et al. [43] use data augmentation. Early et al. [15] and Kim et al. [28] make the reward fumction non Markovian using recurrent or large transformer sequence model architectures respectively. Such approaches increase the upfront cost of preference-base
RL by using additional data or compute. Moreover, these techniques still combine reward optimization with vanilla RL algorithms. Ultimately, this just adds an extra learned component to already notoriously delicate RL algorithms, further increasing hyper-parameter tuning overhead. Preference- RL. approaches often end up training up to four distinct neural networks independently: a critic (with up to two networks), an actor, and a reward function. This can be problematic as prediction errors cascade from the reward function, to the critic, and ultimately the actor causing high variance in downstream per- formance. To address these issues, we propose a parameter-efficiet algorithm specifically designed for prelerence-buseed RL that completely climinates the need to explicitly learn a reward f'unction. In doing so, we reduce both complexity and compute cost.
The key insight of our work is that, under a fixed policy, the Q-function learned by off-policy RL algorithms captures the same information as the learned reward function. For example, both the Q-function and reward function encode information about how desirable state-action pair is. This begs the question: why we need to learn reward function in the first place? Our proposed solution, Inverse Preference I Learning or IPI., is an offline RI. algorithm that is specifically designed for learning from preference data, Instead of relying on an explicit reward function, IPL directly optimizes the implicit rewards induced by the learned Q-function to be consistent with expert preferences. At the same time, IPL regularizes these implicit rewards to ensure high-quality behavior. As a result, IPL. removes the need for learned reward function and its associated compulational und luming expense.
Experimentally, we find that even though IPL. does not explicitly learn reward function, it achieves competitive performance with complicated Transformer-basee reward learning techniques on offline Preference-bast based RL benchmarks with hal-human feedback. At the same time, IPL consistently exhibits lower variance across runs as it does not suffer from the errors associated with querying learned reward model, Finally, under minimal parameter budget, IPL. able to outperform standard preference-basel RL approaches that learn an explicit reward model.
2
Related Work
Our work builds upon literature in reward learning, preference-based RI, and imitation learning.
Reward Learning. Duc to the challenges associated with designing and shaping effective reward signals, several works have investigated various approaches for learning reward functions. A large body of work uses inverse RL to learn a reward function from expert demonstrations [1, 41, 46], which are unfortunately difficult to collect [27, 2, 35] or often misaligned with true human preferences [7, 31]. Subsequently, reward learning techniques using other simpler forms of feedback such as scalar scores [29] and partial [38] or complete rankings [11, 8] have been developed. One of the simplest forms of human feedback is pairwise comparisons, where the user chooses between two options. Often, pairwise comparison queries are sampled using techniques from active learning [47, 9, 14J. However, to evaluate leanndd reward functions, these methods rely on either RL or traditional planning algorithms which are complex and computationally expensive. Our approach takes a simpler perspective that is parameter-efficient by combining reward and policy learning. Though it is not the focus of our work. IPL could additionally leverage active learning techniques for selecting preference data online.
beased Deep Reinforcement Learning. Current approaches to preference based deep RL train a reward function, and then use that reward function in con conjunction with a standard reinforcement learning algorithm [13, 33, 50]. Several techniques have been developed reward function, such as pre-training 125, 32), meta-learning [22], data augmentation [43], and non- Markovian modeling. Within the family of non-Markovian reward modeling [6], recent approaches have leveraged both LSTM networks [15] and transformers [28] for reward learning. But, these methods still rely on Markovian offline RL algorithms such as Implicit Q-Learning (IQL) [30] for optimization. Ultimately, this makes such approaches theoretically inconsistent as the policy learning
Preference-
improve the learned

---

component assumes the reward to be only a function of the current state and action. All techniques for learning the reward function in combination with standard RL methods [20, 49| end up adding additional hyper-parameler tuning and compute cost, IPL on the other hand, is directly designed for RL from preference data and eliminates the reward network entirely, Other recent works also consider contrastive objectives instead of RL [26, 23].
Recently, works in natural language processing have applicd ideas from preference- based RL to tasks such as summarization [51, 54]. instruction following [42], and paradigm has proven to be powerful even at the massive scale of aligning large language models. In this regime, learned reward models are massive, making an implicit reward method like IPI. more attractive. In fact, IPL in a contextual bandits setting recovers concurrent work by Rafailov et al. |45] on implicit reward modeling in L.Mss (see Appendix A). While we focus on control in our experiments, hope our work can inform future explorations in language domains.
[40]. The RLHF
Imitation Learning, Our work builds on foundational knowledge in maximum entropy (MaxEnt) RL [58] and inverse RL [59]. Recent works in MaxEnt inverse RL have used the mapping between Q-functions and reward functions under a fixed policy. Specifically, Garg et al. [18] show that the regularized MaxEnt inverse RL objective from Ho and Ermon [24] can be re- written using the Q-function instead of a reward function and Al- Ma-efe et al. [4] stabilize their approach. While the relationship between Q-functions and rewards has been used for MaxEnt inverse RL., we study this relationship when learning from preference data. While both problems seek to learn models of expert reward, the data differs significantly weference-basel RL uses comparisons instead of optimal demonstrations. This necessitates a greatly different approach.
3 Inverse Preference Learning
In this section, we first describe the preference-basee RL problem. Then, we describe how. leveraging techniques from imitation learning, we can remove the independently learned reward network from prior methods. This results in a simpler algorithm with lower computational cost. and variance in performance.
3.1 Preference-Based RL
We consider the reinfrocement leraning (RL) paradigm where an agent seeks to maximize its expected cumulative discounted sum of rewards in Markov Decision Process (MDP). Standard off-policy RI. algorithms, do so using state, action, reward, and next state tuples (s, a, r. s'). In preference based RL, however, the reward function is unknown, and must be learned from human feedback. Thus, Traditional preference-bar based RI, methods are thus usually separated into two stages: first, reward learning, where rE is estimated by learned reward function re. and second, reinforcement learning. where a policy T(ass) is Icarned to maximize B, (Σ;", Y' (s.(s.c a)] with as the discounl factor. Though our method combines these two phases, we use the building blocks of each and consequently review them here.
Preference Learning. First, similar to prior works [13, 32]. we assume access to preference data in the form of binary comparisons. Each comparison comprised of two behavior segments, (1))) and (2)), and a binary label y indicating which of the two was preferred by an expert. As in Wilson et al. [53], each behavior segment is simply snippet of trajectory of length k, or o= (sy, dr, S1+1, dr+l............................ 414k-1, Sy4k). Increasing & can provide more information per label at the cost of potentially noisier labels. The label y for cach comparison is assumed to be generated by expert according to a Bradley-Teree -erry Preference model [10]:
(1)
exp Σ, (8 .a,
Por())(1)) [01)
(1)
exp E, rE(s))
(1) aa")) exp E, (ss (2) ((1))*
where (5; a,) is again the expert's unknown underlying reward model. We use the subscript on probability to indicate that the preference distribution above results from the expert's reward function. Let the dataset of these preferences be 2,,- ((0(1)) (2))) y)). To learn ry: prior works in
preference-based RL estimate a parametric reward function re by minimizing the binary-cross-entrops over Dp:
(0) -B 00 (. log Pro (0()) |120| (1 log Pro (0())) o(11)]] (2)
3

---

Standard Preference-Basser RI.
Inverse Preference Learning
o
Black Box RL Algorithm
Preference Distributian
(*())
Y's
(2)
(
r(s,a) = ((s,a) yV"(s))
o
Implicit Reward
s<(as))
Q
Regularization
Phase Reward Learning
Phase 2
Reinforcemeent Learning
Single Learning Phase
Figure 1: A depiction of the difference between standard preference-busee RL methods and Inverse Preference Learning. Standard preference- RL first learns reward function, then optimizes with a blockbox RL algorithm. IPL trains Q function to directly fit the expert's preferences. This is done by aligning the implied reward model with the expert's preference distribution and applying regularization.
This objective results from simply minimizing Ey,, [D,,,,,,,(L, [DKI (Pa)], the KL-divergence between the expert preference model and the one induced by re, effectively aligning it with the expert's preferences. We note that some other works in preference-basee model re address the reward learning part of the problem [43, 28]. However, these methods still use uff-the-shellf RL algorithms for the policy learning part of the problem.
RI focus on learning an improved
Reinforcement Learning, Common off-policy RL methods learn policy x by alternating between policy evaluation (using the contractive Bellam Operator 2, to estimate Q" and policy improvement. where the policy π is improved [52]. Concretely, after repeated application of B," as
(B (s. a) r(s, a) 1B,--pt-ssa)) (v)ss)
(3)
the policy can be improved by maximizing Q. In some settings, the Bellman operator 8 correspond- ing to the optimal policy can be used directly, removing the need for the policy improvement step. In these cases, we can simply extract x²" from the resulting Q".
To Iearn the optimal policy, two-phase preference based RL methods rely on recovering the optimal rg in the reward learning phase before running RL. This potentially propagates errors from the estimated ra the learned Q-function and ultimately lcarned policy œ. In practice. would be more efficient to climinate the need for two separate stages. In the next section, we show how this can be done by establishing bijection between reward functions and Q-ffuctionss.
3.2 Removing The Reward Function
In this section, we formally describe how the reward function can be removed from offline sefference-based RI. algorithms. Our key insight is that the Q-function learned by off-policy RI. algorithms in fact encodes the same information as the reward function r(s, a). Consequently. it is unnecessary learn both. First, we show how the reward function can be re-wrilten in terms of the Q function allowing us to compute the preference model Po induced by the Q-function. Then, we derive an objective that simultancously pushes Q fit the expert's preferences while also remaining optimal.
Consider fitting a Q function via the Bellman operator 3 for a fixed policy æ until convergence where BT Q = Q. Here, to encode the cumulative discounted rewards when acting according to the policy, the Q-function depends on both r and π. This dependence, however, is directly disentangled solve for the reward function in terms of
by the Bellman equation. By rearranging it (Eq. (3)), Q and л. This yields the so-called inverse soft- Betlman operator:
(5 (F*())s,4) ((s. [V*(((((s)))
(4)
In fact, for lixed policy the inverse- inverse-Bellman operator is bijective. implying one-to-one corre- spondence between the Q function and the reward function. Though this previously shown in maximum entropy RL [18], we prove the general case Lemma in Appendix A.

---

The Optimal Inverse Bellman Operator. First, we have used an arbitrary policy x, not the optimal one, for converting from Q-values to rewards. Though the Q-function may imply the expert's preferences, the corresponding policy could be extremely sub- sub-optimal. To fix this problem, we need to use the optimal inverse bellman operator T to ensure the extract Q-function corresponds to that of x". For this step, we can use any off-policy RL--algorithm that converges to the optimal policy! If can be estimated using the target from Intuitively, this makes sense: when holding the policy constant, only the reward function affects Q. We abbreviate the evaluation of (T, as rqa (s, a) indicate that rg= is the unique implicit reward function induced by Q*, Prior works in imiitation learning leverage the inverse soft-Bellman operator to measure how closely the implicit reward model rQ" aligns with expert demonstrations [18]. Our key insight is that this equivalence can also be used to directly measure how closely our Q function aligns with the expert preference model without ever directly learning r.
Consider the Bradley- Terry preference model in Equation (1). For a fixed policy л and its correspond- ing Q", we can obtain the preference model of the implicit reward function Po= [o(()) > x=2)] by subslituting the inverse Bellman operalor into Eyuation (1) as follows:
exp (T( 2.. (a)
Pe* (o > =00)
(5)
exp (T=)),))))( )."))= + exp E, (TTO)) (ap))
This substitution will allow us to measure the difference between the preferences implied by Q and those of the expert. To minimize the difference, we can propagate gradients through the preference modeling loss (Equation (2)) and the implicit preference model Po* (Equation (5)) to Q-just would for a parameterized reward estimate rg- Unfortunatcly, naively performing this substitution is insufficient to solve the RL objective for two reasons.
the algorithm directly estimates the By, the corresponding T B,, or
(T.))s.))) = Q(s. BB, [V/=)(s)]] where (s) is estimated as in By.
In many cases, however, computing the optimal bellman operator B is infeasible. Instead, many modern off-policy RL algorithms use policy improvement to converge to the optimal policy. These methods, like Haarnoja et al. [20] use e* to estimate a new policy x such that Q* 2 -a. By repeatedly improving the policy, they eventually converge to Q", Thus, by repeatedly improving the policy according to these algorithms, we can eventually converge to the optimal policy and can thus estimate corresponding optimal inverse bellman operator by using (1m)(s) = Ba-rr- ( [Q(( as) in the above equation.
Regularization. Given we can estimate ' using targets from B; or policy improvement, we can fit the optimal Q-function by minimizing the following loss function
(e) "r D>, log Po- x02) (1 y) log(1 - Po- (or())))) > =
-Po-lo")>a"")
where Po is given by substituting T'Q into Eq. (5). Unfortunately, optimizing this objective alone leads to poor results and may not converge when using RL algorithms that depend on policy improvement. This is because the above objective is under constrained due to the invariance of the Bradley-Terry preference model to shifts. By examining Eq. (1), it can be seen that adding a constant value to all rewards does not change the probability of preferring a segment. However, shifting the reward function by a constant does change the Q-function. RL algorithms using policy improvement monotonically increase the O-function until reaching the maximum at @'. Thus as the implicit reward continues to increase. Q" will continue increase and may never be reached. To resolve this issue, we insure that the optima of the preference loss is unique by introducing a convex regularizer (-() on the implicit rewards re= " giving us the regularized preference loss:
12) = B,""), ly log Po- (1_ > (2)2) + (1 y) log(1 Pa-lor"") - A (T'2) (6)
In practice we choose / to be a form of L2 regularization as is commonly done in imitation learning f18, 4| to prevent unbounded reward values. A > O is a hyperparameter that controls the strength of regularization. Besides allowing us to guarantee convergence, regularization has a number of benefits. It can help center the implicit reward near zero, which has been shown beneficial for RL [16]. Morcover, it encourages more realistic implicit rewards. For example, a reward function might change rapidly by large values when only small perturbations are applied to the state or action. Though such reward functions might be unrealistic, they are completely valid solutions of the anverse-Bellman
5

---

operator: Adding regularization can help penalize large deviations in reward unless they drastically reduce the preference loss. Thus, the first term of Eq. (6) encourages the Q-function to match the expertss preferences, while the second term smooths the implied reward function and makes it unique
Our final algorithm, which we call Inverse Preference Learning (IPL) lits the optimal policy corre- sponding to the regularized expert reward function by repeatedly minimizing Lp (Eq. (6)) and improving the value target used yoing with the update step from any off policy RL algorithm. In this manner, IPL performs dynamic programming through the inverse bellman operator until convergence. In Appendix A, we prove the following Theorem.
Theorem 1 Given an policy RL algorithm that convergences to the optimal policy x,, for some reward function and regularizer d such that Eq. (2) is strictly convex, IPL converges to Tę. corresponding to reward function r = arg min, By, DDki Pa)] + (r))..
The proof of the theorem essentially relies on the fact that for a fixed policy q, we can optimize {p( (Eq. (6)) to fit r". Then, we can update the policy (or target values (/mr)) and optimize Lp again. Because r" is unique, we fit r' again the second time, but the Q-function has improved. There are many choices of regularizers where this holds. In tabular settings if / (r) = Ep (2) reduces to 1.2 regularized logistic regression, which is strictly convex, guaranteeing convergence (Appendix A).
Effectively, IPI. removes the need to learn reward network, while still converging to similar solution as other preference-based RL algorithms. Learning a reward network requires more parameters and completely separate optimization loop, incrcasing compute requirements. Morcover, an explicit reward model introduces a whole new suite of arper-parameters that need to be tuned including the model architecture, capacity, learning rate, batch size, and stopping criterion. In fact, because human preference data is so difficult to collect, many approaches opt to use simple accuracy thresholds instead of validation criteria decide when to stop training ra [32]. All of these components make preference- rrefreence-based RL unreliable and high- varianer, On the other hand, our method completely removes all of these parameters in exchange for single A hyper-parameter that controls the regularization strength. Though we have theoretically derived IPI., in the next section we provide practical recipes for applying it oflline prefereece--ased RL.
3.3
IPL for Offline Preference-based RL
In offline preference-bused RL, we assume ac- cess to fixed offline dataset D. a(s,,, s')} of interactions without reward labels generated by reference policy p(a)s) in addition to the preference dataset D), Common approaches to offline RL seek to learn conservative policies that do not stray too far away from the distribu- tion of data generated by u(als). This is critical to prevent the policy æ from reaching out-of- distribution states during deployment which can be detrimental to performance. In this section, we detail a practical version of IPL that uses the
Algorithm 1: IPL Algorithm (XQL Variant)
Input:2),. D d. for i = 1.2. do Sumple batches Bp ~ Dp. DD B, Do Update Q: mino EB, [B, (Eq. (6)
Update V: miny EB,,B, -
where 2 Q(s,a)) V(s))/a Finally, extract r(als) via: max,, BDyun, (((ss,)-V(s))l log
XQL offline RL algorithm [19]. XQL fits the KL-constrained RL objective
Σ Y' (
a log m(a, p(a,ls))
max E N
where a controls the magnitude of the KL-divergence penalty. The XQL algorithm directly fits the optimal Q-function using the optimal sofi-Bellln operator [19. 55]
(x,))s.a) r(s,a) a) yB.. (v (s'),, where (x) = a log Ba-w(-1s) { [(is.a)aa]
In practice, ytagg is estimated using the linex loss function over the current Q-function. Thus, to fit. the optimal Q-function, IPL with XQL alternates between minimizing the preference loss (2) (Eq. (6)) and updating learned value function V until they converge to Q" and V Note that we are not limited to using just D,p- Though the preference modeling part of (Q) can only be optimized with preference data Dp the value function can be updated with oflline data as well. In the presence
6

---

of additional offiine data, we find that updating the value function using Dp U D, leads to better performance. We approximate I regularization with the regularizer (f(r) = BD [r((, a) which imposes an L2 penalty across the support of the data. While one might try use weight decay to emullate L2-regularization. doing so is difficult in practice as T depends on both the Q network and the target network. We find that weighting the regularization equally between Dp and D, performs well. After Q and V have converged, we can extract the policy using the closed form relationship x² (als) jr(a(s) exp ((Q* (s, a) V* (s)) for constraidd RL as in Garg al. [19], Peng et al. [44]. The full algorithm for IPL with XQOL can be found in 1.
Though have shown how IPI. can be instantiated with XQL., it is fully with other offline RI. algorithms. In fact, IPL can also be used with online RL algorithms like SAC [20]. Critically, this makes the IPL. framework general, as it can remove the need for reward modeling in nearly any preference-based RL setting. This makes IPL. simpler and more efficient. In the next section, we show that IPL can attain the same performance as strong oflline preference-based RL baselines, without learning reward network.
4 Experiments
In this section, we aim to answer the following questions: First, how does IPL compare prior preference-based RI. algorithms on standard benchmarks? Second, how does IPL. perform in extremely data-limited settings? And finally, how efficient is IPL in comparison to two-phase preference-base RL. methods?
4.1 Setup
As discussed in the previous section, though we use a KL-constrained objective for our theoretical derivation, in practice we can construct versions of IPI. based on any offline RI. algorithm. our experiments we evaluate IPL with Implicit Q-Learning (IQL) [30], since it has been used in prior oflline preference-busee RL works. This allows us to directly compare IPL by isolating its implicit as prior works. Using IPL with IQL
reward component and using the same exact hyper-parameteer amounts to updating the value function according to the asymmetric expectile loss function instead of the linex loss function. Concretely. this can be done by replacing the value update in Algorithm with miny EB,UB, 7 ((2(s, a) V(s) O)l (Q(s.a) - v(s))² where is the expectile.
Inspired by Park et al. [43], we introduce data augmentations that sample sub- sections of behavior segments or during training. While such augmentations are inapplicable to non- non-Markovian reward models, we find that they boost performance for Markovian reward models while also reducing the total number of state-action pairs per batch of preference data. This is important as IPL. needs data from both Dp and D, to regularize the implicit reward function. Additional experiment details and hyper-parameters can be found in the Appendix.
4.2 How does IPL perform on preference-based RL benchmarks?
We compare IPL to other offline preference-based RL approaches on D4RL Gym Locomotion [17] and Robosuite robotics [36] datasets with real-human preference data from Kim et al. [28]. We compare IQL-based IPL, with the same hyper-parameters, to various baselines that learn reward model re before optimization with IQL. Markovian Reward or MR denotes using a standard Markovian MLP reward model, like those used in Christiano et al. [13] and Lee al. [32]. Note that this is also equivalent to T-REX [1I] for offline RL. Non-Markovian Reward or NMR denotes using the Ma-MMarovvann LSTM based reward model from Early et al. [15]. Preference Transformer (PT) is statc-of-the thrtthaarttttte approach that leverages a large transformer architecture to learn a non- non-Markovian reward and preference weighting function. B B-REX uses bayesian optimization fit linear reward function from predefined features [12]. which in our case are random Gaussian projections of the states and actions. For fairness, we also compare against our own implementation of IQL with Markovian Reward function that uses the same data augmentation as IPL.
Our results are summarized in Table Starting with the first column, we see that preference-based RL methods are able to match IQL with the ground truth reward function in many cases. On. several tasks, however, the MR implementation from Kim et al. [28] fairs rather poorly. The non- Markovian methods, (NMR and PT) improve performance. It is worth noting that on many tasks our

---

TQL
MR
LSTM
PT
BREX
MR
IPL.
Dataset
(Oracle)
(from [28])
(from [28])
(from [28])
(reimpl.)
(reimpl.)
(Ours)
hop-m-r
83.06 -15.8
11,56+30.
57.88 +40.6
84.54 +4.1
62.0 +2033
70.20
73.57
hop e
73.55 =155
4237 123.7
38. 135.6
68.96
85.1 28.0
103.0 15.6
74.52.10.1
walk-m-r
73.11 18
72.07 12.00
77.00 13.0
71.27 133.3
10.3 15.4
68.79 65.6
59.92 15.1
walk-m-e
107.8
108.3
110,4
110.1 =0.2
99.62 e3.0
109.1 =1.
108.51 +0.6
lift ph
96.75
84.75 16.2
91,50
91.75 15,9
96.6 43.0
98.84 12.3
97.60 22,9
lift-mh
86.75
91.00+28
90.8 +5.8
86.75 +6.0
60.4 +25.1
90.04 -4.5
87.20 +5.3
can-ph
=4.50
69.00
62.00+10.9
69.67 +5.9
63.0
76.40=377
74.8
can-mh
56.25 8888
47.50 13.5
30.50 x877
50.50 16.5
30.4 123.0
53.6
57.6 15.0
Avg Std
10.95
10.2
13.87
9.08
13.77
8.23
4.8
Table 1: Average normalized scores of all baselines on human-preference benchmarks from Kim et al. [28]. For the D4RL. locomotion tasks "hop" corresponds to hopper, "m" medium (training the data generating agent to 1/3 expert performance), "r" to replay buffer data, and "e" to data from the end of training. For the Robomimic tasks lift and can, "ph" corresponds to proficient human data and "mh" to multi-human data of differing optimality. The first four columns are taken from Kim et al. Markovian Reward with IQL. The "Avg Std" row shows
[28]. "reimpl." is our reimplementation the average standard deviation across all eight environments. We run five seeds and report the final performance at the end of training like Kostrikov et al. [30]. Bolded values are within 95% of the top performing method. Note that standard devaition values in the table were rounded for space. On some tasks IPL achieves higher performance earlier training, which not reflected above (See Appendix B). We find that IPL. outperforms PT on many environments, and also performs similarly our implementation of MR despite not training reward function.
implementation of a MR (sixth column) performs far better than reported in Kim et al. [28], likely due to our careful tuning of ry and use of data-augmentations. Our method, IPI,, achieves competitive performance across the board.
In general, IPL with IQL performs on-par or better than both our implementationn MR and PT in most datasets despite not learning a separate reward network. Specifically, IPL has the same performance better performance than our MR implementation on six of eight tasks. More importantly, IPL does extremely well in comparison to Preference Transformer's reported results. On five of eight tasks IPL performs better than PT while having over 10 times fewer parameters, making IPL. far more efficient. To be consistent with Kim et al. [28], we report results after million training steps but performance for IPL, often peaks earlier (see learning curves in the Appendix). For example, with early stopping IPI. also outperforms PT on "hop-m-r". We posit that this is because the Q-function IPL is tasked with both fitting the expert's preference model and optimal policy simultaneously, making both the policy and reward function non-stationary during training. In some datasets, this was more unstable.
IPL also has the lowest average standard- -eviiationn across seeds, meaning it yields more consistent results than explicit reward methods. For standard two-phase preferenco-based RL algorithms, errors in the reward model are propagated to and exacerbated by the Q function. IPL circumvents this problem by not explicitly learning the reward.
Finally, in Table 2, we consider various design decisions of IPI.. Augmentations provide strong boost in the roboties environment, but offer only minor improvements in locomotion. Removing regularization, however, is detrimental to performance. This is likely because without regularization, the implicit reward values can continue increase, leading to exploding estimates of Q. Finally, we show that IPL is compatible with other offline RL algorithms by combining it with XQL [19]. We find that with XQL, IPL performs even better on some lasks. but worse on others. Finally, in Appendix B, we also show that IPL can be combined with online preference-brge RL algorithms like PEBBLE [32].
4.3 How does IPL scale with Data?
Collecting preference comparisons is often viewed as the most expensive part of preference-basel RL. To investigate how well IPL performs in data limited settings, we construct scripted preference datasets

---

Dataset
No Aug
1= 0
IPI.XQL
IPL
hop-m-r
70.46 +6.7
+0.41
80.4
73.57 +6.67
walk-m--
58.50
4.85
25.8
59.92
lift-mh
841
52.60
89.00
87.20
can-mh
53.2
13.8
59.0+5.
57.6+5.00
Table 2: Ablations for IPI. on the offline human-preference benchmark. We consider removing dara augmentation, removing regularization A == O. and other offline RL algorithms (XQL). Full results can be lound in Appendix B.
Preference Queries
500
1000
2000
4000
MR
66.0 =8,0
41.1
54.7 +26.8
78.3
Button Press
IPL
53.3 =8.5
60.1 #128
70.2 #2.5
90.2
MR
65.9 -9.9
87.2 -5.2
89.7 -6.4
94.6 -3.9
Drawer Open
IPL
62.1 -4.8
7127+12.4
89.5 +5.0
96.6=1.
MR
33.0 _5777
46.2 16.0
63.2 1137
70.8
Sweep Into
IPL
34.5 22.3
48.2 77,2
58.8 =7.4
65.9
MR
54.6
55.3
57.2 24.5
23.9 =18.8
55.2 :3.0
Plate Slide
IPL
52.9 _48
55.8 12,2
55.4 13.1
54.9 12.8
MR
0.6
0.7
0.0
2.6=288
Assembly
IPL
0.9 20.6
1.5
1.7 =1.9
5.5
MR
5.9
5.76
13.14
5.36
Avg Std
IPL
4.2
7.22
3.98
4.5
Table 3: Results on five MetaWorld tasks at four different preference data scales. We run five seeds for each method, and take the highest average performance across seeds from the learning curves. More details can be found in Appendix B. IPL. performs the same or better than IQL with a Markovian reward model the majority of tasks and preference data scales without training reward model.
of four different sizes for five tasks from the MetaWorld benchmark [56] used in prior preference- based RL works [32, 22J. We then train on the preference data Dp by sctting D, = {(s, a, E Dp and use the same byper-parametern for all environments and methods where applicable. Our results are summarized in Table 3. Again, IPL is a strong reward- free baseline. We find that at all data scales, IPL performs competitively our implementation of MR (IQL with learned Markovian reward) Button Press and Assembly. Increasing the amount preference generate queries uniformly at
and consistently outperforms data generally improves performance across the board. However, random some preference datasets may be easier to learn from than others, leading to deviations from this trend in some cases. As in the benchmark results in Table 1, IPI. exhibits lower variance across seeds and tasks, in this case at three of four data scales.
4.4 How efficient is IPL?
One henefit of IPL. over other preference-hased RI methods its parameter efficiency. By removing the reward network, IPL uses fewer parameters than other methods while achieving the same perfor- mance. In Table 4, we show the number of parameters for each method used in the last two sections, Preference Transformer uses over times more parameters than IPL, and the M-baee NMR model from Early et al. [15] uses nearly twice as many. When dealing with limited compute or memory budget, this can be important. To exacerbate this effect, consider an extremely parameter efficient version of IPL, denoted "IPL (64)" in Table 4. based on Advantage Weighted Actor Critic (AWAC) [39] which eliminates the second critic and value networks used IQL [30] and uses two-layer 64-dimensional MLP. We then compare this parameter-efficien IPL. to MR with the same parameter budget which results in "MR (35)", 25-dimensional MLP. Results are depicted on the left of Fig. 2. MR trained with smaller network is unable to adequately fit the data, resulting in lower per- formance. Only after increasing the network size past that of IPL. can MR begin match performance.
Aside from parameter elliciency, IPL is also "hyperrprrameter officient".. By removing the reward network, IPL removes whole of parame-paraeeters associated with two phase preference based RL methods, like reward network architecture, learning rate, stopping criterion, and more. In the

---

MetaWorld Drawer Open Ablations with 4000 Querics
Parameter Equalization
MR Reward Steps
IPL Regularization Weight
1.0
1.0
1.0
0.8
0.8
0.8
Success Rate
0.4
0.4
1000
U.4
0.2
0.2
10000
0.2
100000
10
50000
100000
150000
200000
50000
100000
150000
200000
50000 100000
150000 200000
Training Steps
Training Steps
Training Steps
MR (35)
MR (64)
IPL. (64)
Figure Left: Performance comparison with different parameter numbers, MR (35) has the same parameter budget as IPL (64), MR (64) has over twice as many. We see that with the same number of parametery as IPL. MR is unable to adequetly when the reward function is trained for varying number of steps with too few the reward model under-lits, and with too many it over-fits, both Icading to worse performance. Right: IPL with different regularization strengths. On the drawer open task. performance is largely unaffected. For more ablations, see the Appendix.
the data and performs poorly, Middle: MR
middle of Fig. we show how the performance of MR is affected when the reward function over or under fit. Choosing the correct number of steps to train the reward model usually requires collecting a validation set of preference data, which is costly to obtain. Instead of this, IPL only has a single regularization parameter, d. The right side of Fig. 2 shows the sensitivity of IPL to A. We find that in many cases, varying A has little effect on performance unless is perturbed by a large amount.
5 Conclusion
Summary. We introduce Inverse Preference Learning, novel algorithm for offline preference-based RL that avoids learning reward function. Our key insight is leverage the inverse heft-Bellman operator, which computes the mapping from Q- functions to rewards under fixed policy. The IPL algorithm trains Q-function to regress towards the optimal Q" while at the same time admitting implicit reward values that are consistent with expert's preferences. Even though IPI. does not require learning separate reward network, robotics benchmarks it attains competitive performance with preference-bused RL basclines that use twice to ten-times the number of model parameters.
Method
Params
PT
2942218
NMR
508746
MR
348426
IPL
278537
MR (64)
34892
IPL (64)
14025
MR (35)
14012
Table 4: Parameter counts for different methods. The bultom three rows are for the limited parameter budget experiments in Scction 4.4.
Limitations and Future Work. A number of future directions remain. Specifically, the implicit reward function and policy learned by IPL are both non-stationary during training. which sometimes causes learning be more unstable than with fixed reward function. This is a core limitation future work could address by better mixing policy improvement and meeeeeence-attchin reward preference-based RI, methods are not limited to continuous control or binary feedback. Applying implicit reward techniques to other forms of feedback or extending IPL language-based RL lasks remain exciting future directions.
steps improve stability. More broadly, implicit

---

mcknowledgments and Disclosure of Funding
This work was supported by ONR, DARPA YFA, Ford, and NSF Awards #1941722 and #2218760. JH is supported by by the National Defense Science Engineering Graduate (NDSEG) Fellowship Program. We would additionally like to thank Div Garg and Chris Cundy for useful discussions.
References
[1] Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference Machine Learning. 2004.
[2] Baris Akgun, Maya Cakmak, Karl Jiang, and Andrea L Thomaz. Keyframe-based learning from demonstration, International Journal of Social Robotics, 4(4):343-355, 2012.
[3] Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. 2011.
[4] Firas Al-Hafez, Davide Tatco, Oleg Arenz, Guoping Zhao, and Jan Peters. LS-IQ: Implicit reward regularization for inverse reinforcement learning. In The Eleventh International Con- ference on Learning Representations, 2023. URL attps://openreviee nwt/ o3Q4m8jg4BR.
[5] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safery. arXiv preprint arXiv: 1606.06565, 2016.
[6] Fahiem Bacchus, Craig Boutilier. and Adam Grove. Rewarding behaviors. In National Conference on Artificial Intelligence, 1996.
[7] Chandrayce Basu, Qian Yang, David Hungerman, Mukesh Sinahal. and Anca D Dragan. Do you want your autonomous car to drive like you? In 2017 12th ACM/IEEE International Conference Human-Roihe Interaction (HRI, pages 417 425. IEEE, 2017.
[8] Erdem Bryık, Daniel A Lazar, Dorsa Sadigh, and Ramtin Pedarsani. The green choice: Learning and influencing human decisions shared roads. In 2019 IEEE 58th conference on decision and control (CDC), pages 347-354. IEEE, 2019.
[9] Erdem Biyik, Nicolas Huynh, Mykel Kochenderfer, and Dorsa Sadigh. Active preference- based gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS). July 2020.
[10] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[11] Daniel Brown, Wonjoon Goo. Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference machine learning, pages 783-792. PMLR, 2019.
[12] Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scout Nickum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference Machine Learming, pages 1165-1177. PMLR, 2020.
[13] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
[14] Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with novel acquisition function. Autonomous Robots, 39(3):389-405, 2015.
[15] Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ranchurn. Non-markovian reward modelling from trajectory labels via interpretable multiple instance learning. In Advances in Neural Information Processing Systems, 2022.

---

[16] Logan Hngstromm Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study ppo and trpo. In International Conference on Learning Representations, 2020. URL https: / //openreviex wwtp://eootoooeeeeettttoo
[17] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. 2020.
[18] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq- learn: Inverse soll-q learning for imitation. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openrevee/ wwtp:wiwwwweeeeeeeeeee.ee..eeeee www.:neetooouaeee forummid=meeeoo -ettootttbb... xqtb5p.
[19] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon.
Extreme q-learning: Maxent
RL without entropy, In The Eleventh International Conference on Learning Representations. 2023, URL htwps:ttt///ecc /ppprr/v///peeeeeeeeeeeeeeeeeeee net//erunuti http://orummnnssssoeeedee. dd$SJOLde3RRL
[20] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018.
[21] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Starr J Russell, and Anca Dragan, Inverse reward design. Advances in neural information processing systems, 30, 2017.
[22] Joey Hejna and Dorsa Sadigh. Few-shot preference learning for human-in-the-logg RL. In Conference on Robot Learming, 2022.
|23] Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Hinn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without rl. arXiv preprint arXiv:2310.1366 2023.
[24] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[25] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane I Legg, and Dario Amodei, Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems, 2018
[26] Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference-guided policy optimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarleu, editors. Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 15753-15768. PMLR, 23-29 Jul 2023. URL https: / //proceedings. mir gtress/v202/kang281 html.
[27] Rebecca P Khurshid and Katherine J Kuchenbecker. Data-driven motion mappings improve transparency in teleoperation. Presence, 24(2):132-154, 2015.
[28] Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using translormers for rl. In International Conference on Learning Representations, 2023.
[29] W Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforce- ment. In 2008 7th IEEE international conference on development and learning, pages 292-297. IEEE, 2008
[30] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, In International Conference on Learning Representations, 2022.
[31] Minae Kwon, Erdem Biyik, Aditi Talati, Karan Bhasin, Dylan P Losey, and Dorsa Sadigh. When humans aren't It optimal: Robots that collaborate with risk-aware humans. In 2020 15th ACM/IEEE International Conference on 2020.
Interaction (HRI), pages 43-52, IEEE,

---

[32] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedhhckk-effiient interactive rein- forcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, 2021.
[33] Kimin Lec, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference- based reinforcement learning. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track (round I), 2021.
[34] Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. Inferring rewards from language in context, arXiv preprint wwxXi:22204.02511 2022.
[35] Dylan Losey, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg. and Dorsa Sadigh. Controlling assistive robots with learned latent actions. In 2020 IEEE International Conference on Rebotics and Automation (ICRA), pages 378-384, IEEH, 2020.
[36] Ajay Mandlekar, Danfci Xu, Josiah Wong. Soroush Nasiriany. Chen Wang, Rohun Kulkarni. Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[37] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, arXiv preprint 112.v:1312.5602, 2013.
[38] Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning multimodal rewards from rankings. In Conference on Robot Learning, pages 342-352. PMLR, 2022.
[39] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. (AWACI: Accelerating online reinforcement learning with offline datasets, 2021. URL https: https://openrrvieer .eet
forumPid-0J1MIR3AAC
[40] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-ansserring with human feedback. arXiv preprint urXiv:21 2021.
[41] Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
[42] Long Ouyang, Jelf Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin. Chong Zhang, Sandhini Agarwal, Katarina Slama. Alex Ray. et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203., 02155, 2022.
[43] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficcent preference-based reinforcement learning. In International Conference on Learning Representations, 2022.
[44] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wddinhtgg--wiighted regrrggssherrrssssssisi regression: Simple and scalable off policy reinforcement learning. arXiv preprint arXiv: 2019.
|45] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher 1) Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 18290, 2023.
[46] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. IJCAI, volume 7, pages 2586-2591 2007.
[47] Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017.
[48] C. Schenck and D. Fox. Visual closed- loop control for pouring liquids. In International Conference Robotics and Automation, 2017.
[49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint 17Xiv:1707.063444 06347007.677.000 2017.

---

|50| Daniel Shin and Daniel Brown. Offline preference-basel apprenticeship learning. arXiv preprint arXtv:2107.09251, 2021.
[51] Nisan Stiennon. Long Ouyang. Jeff Wu, Daniel M Ziegler, Ryan Lowe. Chelsea Voss. Alec Radford, Dario Amodei. and Paul Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.0/ 325, 2020.
|52| Richard S Sutton and Andrew G Barto. Reinforcement learning; An introduction. MIT Press, 2018.
[53] Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems, 2012.
[54] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 2021.
[55] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang. Zhaoran Wang. Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference Learning Representations, 2023.
[56] Tianhe Yu. Deirdre Quillen, Zhampeng Hc, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and cvaluation for multi-task and meta reinforvement learning. In Conference Robot Learning, 2020.
[57] Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real world robotic reinforcement learning. In International Conference Learning Representations, 2020.
[58] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Careegi Mellon University, 2010.
[59] Brian D) Ziebart, Andrew 1. Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pages 1433-1438. Chicago, II, USA, 2008

---

Appendix
A Theory
A.I Proofs
Lemma For any fixed policy π the inverse bellman operator T establishes bijection between r and Q. Moreover, for any r, Q (Ta))- is the unique fixed poins of the Bellman operator B,. (Adapted from Garg et al. 1181)
Proof. Let pa be the stochastic transition matrix for the MDP corresponding to a fixed policy x. In vector form, the inverse bellman operator becomes bijection between Q and r by showing that (/ - ypP) is invertible. As p defines a valid probability distribution over next stat-action pairs and y < 1, we have that |>| 1. Thus, its Neumann series convergences, which implies the existence of - yp*)= So, Q = (1 - and a bijection exists. Using this, can also show a 1-1 mapping with the bellman operator under reward We have Q = (
T (/ y/*) We can establish
= B Q at the fixed point of B,"
Theorem
Given an off- off-policy RL. algorithm that convergences to the optimal policy ny for some reward function and regularizer & such that Eq. (2) is strictly convex, IPL. converges to nt.- corresponding to reward function = arg min, By,, [Dxi(Prr (Frl P + (r))..
Proof. We prove this statement in the Tabular setting, first for algorithms that use policy improvement. Let Q, E IISxA and π, indicate the Q-function and policy after iterations. Let Qo 1/(1 y) minsxa u). The inverse bellman operator tells us, vector form, that = (1 y yPa Q where
pa is the transition matrix. Let r' arg min, ED, [y log P, (1 y) log(1 - P,)] Ad (r), or the minimizer of the preference loss with regularizer & such that we converge to a unique r".
At cach step of IPL, we substitute the inverse bellman operator into the preference loss and optimize. Thus convergence, (1- yp )², r" uniquely due Lemma 1. Then, there are two cases based on the type of RL algorithm.
If our RL algorithm can directly estimate B,, then convergence, and thus T-Q r" By the bijection established in Lemma 1, we have that Q = (T")-"' which by also lemma 1 is the unique fixed point of By. which is 2r... Thus, have recovered the optimal Q function for r" from which the optimal policy πỤ. can be extracted.
done. This is because have assumed
If we have an RL algorithm that uses policy improvement, we consider multiple steps of IPL. If the RL algorithm guarantces convergence via policy improvement. then we use n, and 2,. to obtain policy T Using π4+
obtain the transition matrix pr Finally, we optimize the preference loss again using p in the inverse Bellman operator to obtain 2,+1- At convergence (I - yP=--1)2-1 = r" holds. As r is unique, Q, and Q++1 are both Q- O-fuuctions for the reward function r", just under different policies. We know from the definition of policy improvement, that @T @ necessarily, and thus Q,+1 Q, for any Convergence is possible according to Lemma 1, Q" = =("*) y the a fixed point of B,
Proposition If ((r) = 12, then IPL converges to the optimal policy corresponding to r" arg min, Bo, [ rg [P) + Arb (r) in tobular seltings.
Proof. The preference based loss function with L2 regularization can be viewed as L2 regularized logistic regression by writing the logits as a dot-product between preference comparison vector x comprised of -y'.y" and () terms and reward function The Hessian for this objective is then XT + A7 where D; = logistic(x; (1) logistic(x;Tr)) which is positive definite. Thus the problem is strictly convex and r" is unique, so IPL. converges to its optimal policy by Theorem 1. Note that to guarantee this we must regularize across the entire state-action space, analogous to regularizing the weight vector in logistic regression.
A.2 Connections to DPO
Concurrent work, called Direct Preference Optimization (DPO) [45] also remove the need for explicit reward modeling for learning from preferences, but do so in the contextual bandits setting for language
15

---

models. Thus, DPO is limited to preference queries segment length 1 that must start from the same state. IPL in fact, more general version of DPO that does have these restrictions. Specifically, IPL with XQL recovers the same exact policy as DPO when applied to the contextual bandits setting.
Within the bandits setting, there is "next-state" and V (s') is removed. and the inverse bellman operator becomes just ((s,a) a(s,a). The optimal XQL policy is π²" p(alss)e (s,,)) (s,a)) (Z(s) where Z is the partition function. By rearranging. T Q" a) log slals)
Z(s). We plug this into the preference model induced by in Eq. (5). In the RLHF setting, the partition function cancels since we assume the context to be the same between preferences. This exactly results in the DPO algorithm. showing the DPO is in fact just an instantiation IPL for contextual bandits.
zials]
A.3 IPL with Rankings
IPI. can easily be extended to rankings using Plackett Luce model. Consider permutations over K segments:
[7 exp Σ FR (a75) Ids k=1
Pr. (T)
where d2 = Σ exp E, (r .a,"). Then, we make the same substitution using the inverse bellman operator giving us the permutation model Pe implied by the Q function, and run maximum likelihood estimation over the model with the preference loss (P( B.-2, 2, [log Pe(r)] (r).
Results
We divide this part of appendix into four different sections following the results section. Bach section additionally provides hyper-paramelers used for IPL in that section. The first section, setup, contains detailed information benchmark results gives full learning curves for the experiments in Section 4.2. The third section provides full learning curves for the MetaWorld and Data-scaling experiments. The final Appendix section provides extended ablations.
the experimental setup and hyper-parameters used. The second section
B.1 Setup
Here we provide the full algorithmic outline of IPL using Implicit Q-Learning [30] that mimics our implementation. While in practice the policy could be extracted at the end of training, we do simultancously as [30] in order to construct learning curves.
Algorithm 2: IPL Algorithm (IQL Variant)
Input: : 1) D.., for i= 1, 1,2,,,..... do Sample batches B, ~ Dp. B, ~ D.
Update Q: ming EB, (() xE (L,
Update V: miny BB, ||| (kr-1(((s,a)) V(s)) (Q(s, -) V(s))
Update max, EDD,D. (0810 V((.) log (()s))
Note that above we write the temperature parameter & as done in IQL, insteud of how it is usually done. using a in the denominator [19, 44].
When sampling batches of preference data Bp Dp- we take sub-samples of each segment of length s. For sampled data point (o-(1) (21 .y), we sample start Unif[O, 1,2, - s] and then let take Setrrs dstant: ... Ss We use the same start value across the entire batch.
Given that experiments using MLPs, all of our experiments were resources. Each seed for each method requires one CPU core and 8 Gb of memory.
CPU compute

---

B,2 Benchmark Results
Here we provide details for our experiments on the preference based RL benchmark from Kim et al. [28]. We use the same hyperparameters Kim et al. [28] and Kostrikov et al. [30] as shown in Table 5.
Gym-Mujoco Locomotion. Hopper and Walker2D agents are tasked with learning locomotion policies from datasets of varying qualities taken from the D4RL [17] benchmark. Preference datasets were constructed by Kim et al. [28] by uniformly sampling segments. Preference datasets for "medium" quality offline datasets contain 500 queries, while preference datasets for "expert" quality offline datasets contain 100 queries. Segment length 100 for all datasets, and were subsampled to length 64 by IPL and our MR (reimpl), Evaluation was preformed over 10 episodes every 5000 steps. Full learning curves are shown in Fig. 3.
The RohoMimic datasets contain interaction data of two types: ph proficient human and mh multihuman. The multi nulti-human data was collected from human demonstrators of mixed quality. The robot is tasked with learning how to lift cube (lift) or pick and place a can (can). Preference datasets were again taken directly from Kim et [28]. Preference datasets of size 100 with segment lengths & = 50, randomly sub-sampled to length = 32 were used for the ph datasets. Preference datasets size 500 with segment lengths 100, randomly sub-sampled to length = were used for the mh datasets. Evaluation was performed over 25 episodes every 50000 steps. Full learning curves are shown in Fig. 4.
RoboMimic:
Online Experiments. We also test combination of IPL with PEBBLE [32] on few tasks in the Meta World benchmark. Results can be found in Fig.
D4RL Mujoco Locomotion with Human Preferences
Hopper Medium Reploy
Hopper Medium Export.
12
1.0
1.0
0.8
0.4
0.2
0.0
0.2
0.4
0.8
1.0
0.0
0.2
CL.4
205
0.5
1.01
146
WalkerDD Medium Replay
Wa Medium Expert
1.1
0.4
0.3
0.0
Training Steps
1e6
Training Steps
1e6
PT
MR
MR (reimpl.)
IPL
Figure 3: Full learning curves on the D4RL. locomotion benchmark with human preferences.

---

Robomimic with lluaan Preferences
Lift PH
Lift MH
0.8
0.8
0.6
0.6
Supcess
0.0
0.0
0.6
1e6
1e6
Can PH
Can MH
0.8
0.8
8.6
0.0
0.0
Training Steps
1e6
Training Stops
1e6
(reimpl.)
Figure 4: Full learning curves on the RoboMimic benchmark with human preferences.
Online Mela with Scripted Preferences
Plate Slide, 20k Feedback
Drawer Open, 20k Feedback
Success Raie
0.8
0.4
0.4
PEBBLE
IPL
0.0
0.0
0.2
0.0 0.6 0.0 1.0
0.2
0.4
0.5
0.0
1.0
Training Steps
1e6
Training Steps
1e6
Figure 5; Online experiments for two of the most dillicult Meta World environments. Each method
(run on 3 seeds) gets feedback every 5000 steps for a total of 20k artificially queries. Notice that IPL has much lower variance across runs, particularly in Drawer.
Common Hyperparameters
MR Hyperparameters
Parameter
Locomotion
Robomimic
Parameter
Locomotion
Robomimic
Q, V. Arch
2x 256d
2x 256d
ra Arch
2x 256d
2x 256d
Learning Rate
0.0003
0.0003
re LR
0.0003
0.0003
Optimizer
Adam
Adan
ra Optimizer
Adam
Adam
B
3.0
0.5
re Steps
20k
20k
T
0.7
0.7
D. Batch Size
256
256
D), Batch Size
8
Training Steps
Mil
Mil
IPL Hyperparameters
&
100
100, 50
Parameter Locomotion Robomimic 0.5
Subsample 64
64, 32
Table 5: Hyperparameters used for the benchmark experiments. We can see that IPL. has fewer hyperparameters. For the XQL experiments we use 2 for locomotion and a = 5 for Robomimic. We left all other parameters the same.

---

B.3 Data Scaling Results
Experiments for data scaling were conducted the MetaWorld benchmark from Yu al. [56]. Offline datasets for five different MetaWorld tasks were constructed as follows: Collect 100 trajectories of expert data on the target task using the built in ground truth policies with the addition of Gaussian noise of standard deviation 1.0. Collect 100 trajectories of sub-optimal data by running the ground- truth policy for a different randomization of the target task with Gaussian noise 1.0. Collect 100 trajectories of even more sub-optimal data by running the ground truth policy of a different task with Gaussian noise standard deviation 1.0 in the target domain. Finally, collect 100 trajectorics with uniform random actions. As Meta World episodes are 500 steps long, this results in 200,000 time-steps of data. We then construct preference datasets by uniformly sampling segments from
the offline dataset and assigning labels y according to Σ, r(s, d; (T)) > Σ, r(s _a)) where is the ground truth reward provided by metaworld. We then train using only the data from D General architecture
were taken from Lee et al. [32], Hejna and Sadigh [22] which also use the Meta World benchmark, but for online preference- based RL. Full-hyper parameters are shown in Table 6, We run 20 evaluation episodes every 2500 steps. Full learning curves are shown in Fig. 6. When reporting values in Table we choose the maximum point on the learning curves which average across five seeds. This provides results as if carly stopping was given by an oracle, which is less optimistic than averaging the maximum of each seed as done Mandlekar al. [36].
MetaWorld with Scripted Preferences Press,
Press,
Button Press, 1000
Button Press, 4000
1.0
Sueress Kote
0.8
0.8
0.8
0.5
0.0
0.5
0.0
0.0
0.0
0.0
L000CK 200000
15000C 200000
50000 L000CC 200000
50000 150000 200000
Drawer Open, 500
Drawer Open, 10:00
Drawer Open, 2000
Drawer Open, 40000
1.0
1.0
1.0
1.0
Sucress Rate
8..
0.8
0.8
0.8
8.0
0.0
0.4
0.4
0.4
0.4
0.0
0.0
0.0
0.0
50oon
LONNOc 150tue 200nt
5uCCH 1QU000 150100 2000t)
SHoOn
1000000 1 550000 2000
5UCCHI 100000 150000 20000
Sweep Into, 500
Sweep Into, 1000
Sweep Into, 2000
Sweep Into, 4000
1.0
1.0
1.0
1.0
Singess Rate
8.0
0.6
0.0
0.0
0.4
6.2
0.0
0.0
0.0
0.0
SHOOI LOTNRO 15mino 2000mt
50008 LORIOC 150000 2000th
SICCH 1,T00OO 1 150000 2000th
5000E 100006 150000 2000I
Plate Slide, 500
Plate Slide, 1000
Plate Slide, 2000
Plate Slide, 4000
1.0
1.0
1.0
1.0
Seecesss Hate
0.6
0.8
0.8
0.7
6.2
0.2
50000 1000CC 15000C 200000
50000 1000CC 15000C 200000
50000 1000CC 15000C 200000
50000 1000CC 1500K 200000
Assembly,
5.0
Assembly, 1000
Assemblly, 2000
Assembly, 4000
0.10
0.10
0.10
0.10
Success Rale
306
0.06
0.06
0.06
0.04
0.04
0.04
0.06
0.02
0.02
0.02
0.00
0.00
0.00
0.00
50000
L000CC 150000 200000
50000
LOOGCC 15000C 200000
50000
200003
L00aC0 150000
Training Steps
Training Steps
Training Steps Training Steps
MR
(retmpp).)
IPL.
Figure 6: Full learning curves for the MetaWorld data scaling results with scripted preferences.

---

Common Hyperparameters
MR Hyperparameters Parameter
Parameter
Value
Value
Q. Arch
3x 256d
Arch
3x 256d
Learning Rate
0,0003
re LR
0.0003
Optimizer
Adam
Optimizer
Adam
B
4.0
re Steps
20k
0.7
Dp Batch Size
16
Training Steps
200k
IPL Hyperparameters Parameter
25
Value
Subsample s
16
A
0.5
Table 6: Hyper-parameters used in the Meta World data scaling experiments.
B.4 Ablations
In this section provide additional ablations both the benchmark datasets and MetaWorld datasets. We keep the hyperparameters the same, except for the parameter-effiiien experiments.
Benchmark IPL Ablations. We include results of full ablations for IPL on the benchmark tasks in Table We additionally provide comparisons between IPL and MR IQL with and without data augmentation in Fig. 7.
Dataset
No Aug
1=0
IPL-XQL
IPL
hop-m-r
70,46 + 6.73
10.41 + 2.26
80.4+
73.57 + 6.67
hop-m-e
51.26 17.46
52.81 + 7.45
54.3
74.52 10.11
walk-m-r
58.50 + 5.31
4.85 1 1.52
+.222
59.92 5.11
walk-m-e
108.91 0.18
58.77 = 15.75
73.16
108.51 + 0.60
lill-ph
2.50
8.72
9..40
97.60 + 2.94
lift-mh
4.11
52.60 10.07
10.07
89.00
87.20 + 5.31
can-ph
8.25
2
68.6
74.8 + 2.40
can can-mh
5.82
13.8 3.78.5773 5.73
59.0
57.6 5.00
Table 7: Extended IPL ablation results.
Jlopper Medium Benchmark Data-Aug Abletion Pil
Medium Replay
Mit
0.E
10
0.2
1.5
3.5 L06
1-2 Lue
0.1
1.2
mg
10.
Training
Steps
Steps
Lud
K53 Aug
Aua
Figure 7: IPL and MR+IQL with and without data augmentation across 5 seeds. We see that data augmentation makes a large difference, especially for MR+IQL in the hopper environment, while its effects are less for the robomimic Can datasets.
ppper-parameter Sensitivity. We run benchmark datasets in Hig. 8. The top row depicts the sensitivity for IPI. to the value of The bottom row depicts the sensitivity of MR to the number of timesteps the reward function is trained for.
sensitivty results for the human- ereaan-reeeerecce
Parameter Efficiency. For the parameter-efficient experiments only we efficient version of IPL based on AWAC [39] to additionally remove the need for learning value network. AWAC uses
20

---

Regularizahon Ablation with
WalkerrD Mediumm Replay
Walker2D Medium Expect
Drawer Open, 500
Drawer Ogen, 1030
1.2
1,9
1.0
L.0
0.8
0.8
0.5
Stegenss Rote
Stagnss Role
0.55
0.6
0.8
Sc0re
Scure
8/4
6.1
0.4
0.4
0.2
2.5
2.5
0.1
0.0
0.0
Training Steps
lefi
Training Steps
teli
To Training Stegs
T: Training Stegs
Reward Training Steps Ablation on Benchmark Tasks with MR
WalkerDD Medium Replay
Walker2D Medium Expect
Drawer Open, 500
Drawer Open, 4000
6.5
1.1
1.0
1.0
U.A
0.9
Stegoss Bale
Stegess Bale
0.5
0.8
060.1
0.7
8.4
6.4
0.4
1000
0.6
1000
0.2
1000
0.2
1000
6.3
10000
0.7
100000
100000
100000
100000
0.0
0.0
6.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.8
0.5
0.8
1.0
20000 LOM000
150000
200000
20000 100000 150000 200000
Training Steps
1w6
Training Steps
1w6
Training Steps
Training Steps
Figure 8: Ablations on regularization strength à for IPL (top row) and the number of reward steps for MR (bottom row). We that IPI. is relatively consistent across different values of A. MR the other hand, can vary greatly if the reward function under or over fits. In Walker2D Medium Replay and Drawer Open, 500, see that it can easily under-fit. In Walker21) Medium Expert it easily over-fits.
a policy-evaluation, policy imporvement style approach. The outline of this variant is given below: Algorithu IPL Algorithm (AWAC Variant)
Input: Do. d. for 1,2,3.... do Sample batches Bp ~ Dp B, D. Estimate V as Q(s, T())
Update Q: mino EB, Lp (2) tAE UB, (Q) Update max, En,,n, (QQis,))) ((s.)())))) logı((a(s)]
For this version of IPL, we use l = 0.5. All other hyper-parameters remain the same as in Table except the architectures. For the parameter-efficieeet
experiments only we use MLPs consisting of two dense layers with either dimension 64 or dimension 35. Running MR with a two-layer MLP of dimension 35 has almost exactly the same number of parameters IPL-AWAC with two-layer MLPs of dimension 64, We include full results for the parameter-efficienge experiments in Table We find that Drawer Open and Sweep Into, IPL outperforms both MR (64) and MR (35). In these environments, performance increases from MR (35) to MR (64) indicating that the expressiveness of the Q-function and policy are limiting performance. For the same budget, IPI. is able to perform better. In Button Press, the simplest task, we find that MR (64) actually over-fits more than MR (35) and MR (64) ends up performing worse. Plate Slide, all methods perform similarly independent of parameter count. We omit Assembly because of its low success rate at all data scales.
21

---

Preference Queries
500
1000
2000
4000
MR (35)
73,9 #8.9
86.8 +8.2
89.9 +144
99.0 =1.
Button Press
MR (64)
54.2+16.1
42,6
67.1 +14.9
43.4+74
IPI. (64)
65.8+133
79.8-18
80.0+17.
95,8+5.2
MR (35)
13.4+139
12,6+21.9
12.5
18.4
Drawer Open
MR (64)
13.4=19.0
37.1-31.2
53.5
78.8
IPL (64)
93.2+25
99.5
95.5
MR (35)
35.1 +8.9
42.4
49.
35.9 +4.1
Sweep Into
MR (64)
31.1
55,8
49.6+10.1
56.4+10.3
IPL (64)
41.1 =14.2
63.9 +8.0
65.0=12.0
63.9 +11.8
MR (35)
55.2
51.1
53.0
48.9 +3.3
Plate Slide
MR (64)
46.6+21.9
50.8 #116
47.0
48.5 #4.6
IPL (64)
54.9
49.4
45.2+9.0
48.8
Table 8: Performance of different methods on the MetaWorld tasks under limited parameter hudget. MR (35) and IPL (64) have the same number of parameters. The Assembly task is ommited due to low success rate. On Button Press, fewer paraneicrs appears to perform betler as, due to the simplicity of the task, its casier for the bigger models to overfit. On Drawer Open and Sweep Into. we see consistent gains from increasing the number of parameters in the network, and IPL performs best overall. On the Plate Slide task. all methods different parameter scales perform similarly.
22


Reviews:
[{"review_id": "jX21jn9mjL", "reviewer": "Reviewer_ggMH", "summary": "RLHF pipelines typically consist of (1) training a reward model over human preference data and (2) using this trained reward model with a well-known RL method. This two stage training is computationally expensive. The authors of this paper develop an algorithm \"Inverse Preference Learning\" to directly learn the $Q, V$ functions, which can be easily used to extract an aligned policy. By directly learning the value functions, this approach bypasses learning a reward model, which is expensive to train, and prone to problems like reward hacking.", "questions": "On Page 2, the authors mention - \"This can be problematic as prediction errors cascade from the reward function, to the critic, and ultimately the actor causing high variance in downstream performance.\" Can we quantify these prediction errors more formally, to establish how serious this issue is?\n\nOnly having two options in the preference data seems limiting. Can this idea be applied to the case where the preference is amongst more than two options? Eg. maximizing $p(a\\succ b\\succ c\\succ d)$ rather than $p(a\\succ b)$.\n\nOn Page 6, line 214, the authors say \"While such a reward function seems unrealistic\", here, why do we want rewards to be necessarily continuous? In many cases, human defined rewards are discontinuous (eg. 0-1 type of reward where the reward is 0 everywhere except when the agent reaches the goal, when it is 1).\n\nThe authors mention on Line 53-54 that \"the key insight of our work is that, under a fixed policy, the Q-function learned by off-policy RL algorithms captures the same information as the learned reward function\". This is not an insight developed in this paper, but rather a result from [1] (see Lemma 3.2 in [1]). Moreover, the result is for soft-Q functions, and not the standard Q function. I would recommend to make these points clear in the manuscript.\n\nFinally, there is parallel work [2] that also tries to bypass reward learning similar to this approach. It might be good to acknowledge, but I leave this decision to the authors.\n\nSuggestions related to language and typos:\n1. Page 2, 2nd last line - \"offlien\" -> \"offline\"\n2. Page 6, 3rd line - should \"another perturbation of size \\epsilon\" be \"another perturbation of size \\epsilon prime\"?\n\nReferences:\n1. IQ-Learn: Inverse soft-Q Learning for Imitation, Garg et al. (2022)\n2. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafailov et al. (2023)\n\nEDIT (16 Aug 2023): Updated score from 6->7.", "limitations": " ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": " ", "weaknesses": " ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review! The reviewer largely had theoretical questions about our work. We believe the answers to these questions will help all readers, and will correspondingly update the paper to include all the information below. \n\n**1. Cascading Errors**\n\nThe reviewer asked if we could better characterize the cascading error problem. First, cascading errors in Preference-based RL can be empirically observed in many prior works. For example, in PEBBLE [1] Figure 4, we see that confidence bands for preference-based methods are far larger than those of SAC with oracle rewards. Second, we can actually theoretically characterize this problem using arguments from [2] designed for behavior cloning. In their work, once an error is made a policy goes out of distribution and will subsequently only make errors. This “cascading” error framing can be applied to the networks used in PbRL. Assume that a network makes a prediction error with probability $\\epsilon$, and that all future networks necessarily make a prediction error when a previous one does, i.e. if the reward model makes a mistake, then the critic and actor also make a mistake. By [2] the total error across all networks can be bounded as $O(\\epsilon N^2)$, where $N$ is the number of networks. By reducing the number of networks by 1, IPL can *theoretically* lower this bound.\n\n**2. Only Two Forms of Data**\n\nThe reviewer stated that having two options in the preference data is limiting. We presented IPL only for binary preferences as it is the simplest case, but IPL can easily be extended to rankings using a Plackett Luce Model. Consider permutations $\\tau$ over $K$ segments: \n\n$P_{r_E}(\\tau) = \\prod_{k=1}^K  \\left(\\exp \\sum_t r_E(s^{\\tau_k}_t, a^{\\tau_k}_t)\\right) / d_k$   \n\nwhere $d_k = \\sum_{j=k}^K \\exp \\sum_{t} r_E(s^{\\tau_j}_t, a^{\\tau_j}_t)$\n\nThen, we make the same substitution using the inverse bellman operator giving us the permutation model implied by the Q function, and run maximum likelihood estimation over the model.  \n\n$L_p (r) = \\mathbb{E}_{\\tau \\sim \\mathcal{D}_p }\\left[ \\log P_r (\\tau) \\right]$\n\nWe will include a full derivation of IPL for rankings in the Appendix.\n\n**3. Regularizing the Reward Function**\n\nThe reviewer asked why we might want reward functions to be smooth instead of discontinuous, as in binary. We hypothesize that, though a human may design a binary reward function, human preferences are often smooth. For example, we might make judgments more smoothly across different comparisons, instead of attributing all reward to a single frame. Moreover, smooth reward functions have been shown to generally perform better [3], while sparse ones are harder to optimize. \n\n In our case the regularization is also necessary to remove the ambiguity of the Bradley Terry model – which can only recover $r_E$ up to a constant. See proof in the central reviewer response.\n\n**4. Key Insight**\n\nThank you for pointing this out – we intended to frame our key insight as extending the inverse-bellman operator from IQ-Learn. We will update the text to make it clear that our insight is applying this to the reward function in preference-based RL. \n\n**Soft-Q vs. Standard, Proof of Bijection**\n\nWhile we presented our method (IPL) with XQL, IPL can in fact use any RL update. We thought that presenting our method with XQL would make it easier to understand in the context of IQ-Learn, which originally introduced the inverse bellman operator and used soft Q-learning. IQ-Learn requires the soft-Q framework because a) it was developed for soft-inverse RL, and b) it guarantees that the saddle-point between the reward and policy is unique for inverse-RL. We used IQL in experiments to match baselines, and because XQL is often unstable.\n\nHowever, the bijection between Q-functions and reward functions under a fixed policy exists regardless if one uses soft-Q learning, allowing IPL to use any RL update. We will make this more clear in the final draft. Lemma 3.2 in IQ-Learn in fact does not depend on the soft-Q framework:\n\nLet $P^\\pi$ be the transition matrix for the MDP corresponding to a fixed policy $\\pi$. In vector form, the bellman equation becomes $Q = r + \\gamma P^\\pi Q$, or $r = (I - \\gamma P^\\pi) Q$. We can establish a bijection by showing that $(I - \\gamma P^\\pi)$ is invertible. $||\\gamma P^\\pi|| < 1$ by construction, as $P^\\pi$ is bounded by 1 as a probability distribution and $\\gamma < 1$, which guarantees that its Neumann series converges. This implies the existence of $(I - \\gamma P^\\pi)^{-1}$. Thus, $Q = (I - \\gamma P^\\pi)^{-1} r$ and a bijection exists. \n\n**Connections to DPO**\n\nWe are aware of the recent DPO work, which was released after the NeurIPS submission deadline. While IPL and DPO share some theoretical connections, we’d like to note that DPO is designed for and limited to the contextual bandit setting. This setting is appropriate for RLHF in LLMs; however, DPO would not apply to the more general setting of preferences over sequences of states and actions. IPL is strictly more general, as if you take IPL with XQL for bandits, you can exactly recover DPO.\n\nWithin the bandits setting, there is no  “next-state” and $V^*(s’)$ is removed, and the inverse bellman operator becomes just $Q(s,a) = r(s,a)$. The optimal XQL policy is $\\pi^* = \\mu(a|s) e^{Q^*(s,a)}/ Z(s)$ where $Z$ is the partition function. By rearranging, $TQ = Q^*(s,a) = \\log \\frac{\\pi(a|s)}{\\mu(a|s)} + Z(s)$. We can plug this into the preference model induced by Q in Eq 6 of IPL. In the RLHF setting, the partition function cancels since we assume the context to be the same between preferences. This exactly results in the DPO algorithm, showing the DPO is in fact just an instantiation of IPL. We will include this in the Appendix.\n\nThanks for finding typos!\n\n[1] Lee et al., PEBBLE. ICLR 2021.\n\n[2] Ross and Bagnell. Efficient reductions for imitation learning, 2010.\n\n[3] Ng, A. et al. Policy invariance under reward transformations ICML, 1999.\n"}, {"review_id": "Q1CQkculiK", "reviewer": "Reviewer_kKfF", "summary": "This paper proposes a new algorithm, inverse preference learning, to learn from (offline) preferences between behavior segments. In particular, they show that human preferences can be modeled using only the Q-function, therefore eliminating the need to learn a separate reward function. The authors show this approach can succeed in a variety of environments.", "questions": "**Q1.** Can this method be used in an online setting, where new data is generated according to the policy?\n\n**Q2.** How do MR and IPL perform without data augmentation? And why does MR receive such a large boost based on this augmentation?\n\n**Q3**. Are the reimplemented training setup and the training setup from [25] exactly the same? If not,  PT and LSTM should be reimplemented for fair comparison.\n\n**Q4.** What is the bolding scheme in Table 1?\n\n**Q5.** Is there any way to experimentally validate the claim that IPL is more hyperparameter efficient? For example, random/grid hyperparamter search with an equivalent budget? From Figure 2, it looks like MR is actually more hyperparameter efficient than IPL.", "limitations": "Yes, the authors do discuss limitations of their work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper has several strengths:\n- They study the important problem of preference-based RL.\n- This paper makes the novel insight that human preferences can be modeled using only the Q-function, and introduce an algorithm based on this insight.\n- They experimentally validate different aspects of their algorithm over a wide variety of tasks.", "weaknesses": "**W1.** The motivation that reward-modeling has a large cost is a little bit weak (especially with regards to memory). For example, InstructGPT uses a 6B reward model and a 175B policy. The reward model is therefore only a small fraction of the memory cost. In addition, one of the main hyperparameters of reward-modeling that the authors say is difficult to tune is the stopping criteria. However, the stopping criteria for reward model training is straightforward in most papers: Simply monitor the validation loss on a held-out fraction of comparison data. \n\n**W2.** This paper mainly focuses on offline RL, yet the most impactful RLHF applications are trained online, where behavior data is collected from the policy. It seems like IPL can be trained in an online manner as well, but it is not clear from the presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comments. \n\n**Weakness 1: Reward modeling cost**\n\nWhile reward models in RLHF in language models can be smaller, as in InstructGPT in control-based domains, the current trend is the opposite: reward models are getting bigger. Kim et al. [1] and Early et al [2] use reward models that are significantly larger than the policy and/or critic. In control, we often don’t have access to large amounts of applicable pre-training data, and thus the quality of the reward function is paramount to attaining good success. Moreover, while reward models in RLHF, like those in InstructGPT, provide feedback like a single-step contextual bandit at the end of generation, reward functions for control need to provide feedback at every step, analogous to “at each token”.  \n\nUsing validation loss as a stopping criterion for reward models as the reviewer suggests, while potentially accurate, can be expensive in practice. Human preferences in many domains (including control/robotics domains) are quite expensive to collect. In addition these preferences are often high-variance, requiring an even larger validation dataset to account for the high variance. \nIn addition, we note that IPLremoves the inherent hyper-parameters in reward modeling, such as the network architecture. \n\n**Weakness 2: Online RL**:\n\nWe agree that the online setting is also interesting, and there is nothing stopping IPL from being used online since it can be combined with any off-policy RL algorithm. To test an online version of IPL, we combine our framework with SAC and compare it to PEBBLE, a standard, state-of-the-art method for off-policy online Preference-based RL [3]. We include learning curves in Figure 1 of the attached document. On two of the harder Meta-World tasks from Table 2, plate-slide and drawer-open, we find that IPL trades blows with PEBBLE (out performing by a large margin on drawer-open) while exhibiting lower variance. \n\n**Questions**\n\n*Q1: Online Settings*. Yes! See our response to Weakness 2. \n\n*Q2: Data Augmentation*:  We ablate the use of augmentations for both IPL and MR+IQL in Hopper and Robomimic Can in Figure 2 of our uploaded page. We find that data-augmentation makes a substantial difference for MR+IQL on Hopper, but has a smaller effect in the robotics domains. The reviewer asks why this form of data-augmentation makes such a big difference. We would like to point out that this question was previously studied in SURF [5]. The fact that humans likely made similar judgements between subsets of segments is a good inductive bias for the reward function, helping its accuracy when relabeling the offline dataset.\n\n\n*Q3: Implementation*: We use the same exact datasets as [1] and the same evaluation procedure. We use the same exact network architectures, learning rates (and schedules). The primary difference in our implementation is the addition of data-augmentation, which we believe should be as it has already been shown to be effective in SURF [5]. Due to the use of data-augmentation, we also adjusted the reward training steps. Everything else remains identical. The data-augmentation techniques are inapplicable to PT [1] and the NMR baseline from [2], since they are sequence modeling approaches which require the whole data sequence. \n\nWe would like to additionally point out that Reviewer HQUc views our improved MR baseline to be “itself a nice little contribution”. \n\n*Q4: Bolding Scheme*: We bold results within 95% of the best performing method, as done in IQL [4]. We will include this clarification in all Table Captions.\n\n*Q5: Hyper-parameter Efficiency*: The reviewer asked us to further discuss the hyper-parameter efficiency of IPL. The only parameter we searched over when tuning IPL was the regularization coefficient. We include additional ablations on the regularization coefficient $\\lambda$ in our attached page, showing that in many cases, IPL is very robust to this value. We would also like to again highlight that  reward-modeling methods also have to choose an architecture for the reward network, its learning rate, batch size, etc. in addition to the stopping point parameter we ablate for MR+IQL and thus Figure 2 is not the complete picture. Nonetheless, we will reward our points surrounding hyper-parameter efficiency. \n\n[1] Kim et al.. Preference transformer. ICLR 2023.\n\n[2] Early et al.. Non-markovian reward modeling from trajectory labels. NeurIPS 2022.\n\n[3] Lee et al., PEBBLE. ICLR 2021.\n\n[4] Ilya Kostrikov et al.. Offline RL with implicit q-learning.ICLR, 2022.\n\n[5] Park et al. SURF: Semi-supervised reward learning with data augmentation. ICLR 2022.\n"}, {"review_id": "DtZWZTyfJ7", "reviewer": "Reviewer_HQUc", "summary": "This paper addresses the problem of learning a policy given offline pairwise preference data. The proposed algorithm, inverse preference learning, directly learns a value function without explicitly learning the reward function first. This setup simplifies the common two-step approach of first inferring the reward function in a supervised manner and then learning the policy using RL. Across a range of simulated robotics experiments, it is shown that this method performs at least as well as more complex approaches that have significantly more (hyper-)parameters.  ", "questions": "1. Could you please clarify the results in Table 2? I would suggest that both Table 1 and Table 2 either only report the final success rate or report both the final and best success rate. \n2. Could you please double-check/clarify why certain numbers in the results are bold and others are not?", "limitations": "One of the original motivations of inverse RL was that the reward function provides a succinct and transferable definition of the task (Abbeel and Ng, 2004). While recent approaches seem to no longer learn \"succinct\" reward functions, one could argue that transferability is still a good reason to explicitly learn a reward function. Thus, a potential limitation of the presented approach is that the ability to use the learned reward function to learn different policies gets lost since the value function depends on a particular policy.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated and clearly written. The presented idea is novel, simple and well-executed. The resulting algorithm is easy to implement, has much fewer (hyper-)parameters compared to prior work, and seems to perform on par or better than prior work. Source code is also available. Given the increasing interest in reinforcement learning with human feedback (RLHF) for robotics and natural language problems, this paper should be of high interest for the community.\n\nFinally, I really appreciate that the authors invested time to improve the Markovian Reward (MR) baseline. This by itself is a nice little contribution, and I hope that future work will adopt this much stronger baseline.", "weaknesses": "The presentation of the results in the tables could be improved. While in Table 1 the results are based on the final success rate at the end of training, it seems that the results in Table 2 are based on the best success rate achieved throughout training (particularly apparent for the Drawer open tasks). I found this to be a bit misleading, especially since IPL seems to heavily suffer from the problem that the success rate deteriorates over time. Furthermore, I didn't fully understand why some results in the tables are bold and others are not. For example, in the Assembly task with 500 queries (Table 2), it seems that IPL is not significantly better than MR.\n\nMinor comments:\n1. I initially got a bit confused as to why $L_r$ depends on $D_o$ and $D_p$. Perhaps you could remind the reader that the preferences in $D_p$ are also (sub-)trajectories.\n2. Typos:\n- line 89: \"offlien\" -> \"offline\"\n- line 101: \"Our work build on\" -> \"Our work builds on\"\n- Eq. 3: Subscripts t missing for $a, s$ in $\\pi$ and $\\mu$\n- Eq. 4: $p(\\cdot|s|a)$ -> $p(\\cdot|s,a)$\n- line 158: \"to learned\" -> \"to the learned\"\n- line 199: \"KL-constrianed\" -> \"KL-constrained\"\n- line 223: \"thus\" -> \"is [thus]\"\n- line 283: \"that use\" -> \"that uses\"\n- line 305: \"Data\" -> \"data\"\n- line 315: \"form\" -> \"from\"\n- line 308/319: \"Preference-based\" -> \"preference-based\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their in-depth review. We hope to have addressed all the reviewers' concerns, and would invite them to additionally examine the new experimental and theoretical results produced in response to other reviewers.\n\n**Clarification of Results in Table 2**\n\nIn our uploaded response page we have included evaluations at a fixed stopping point, like Table 1, and find the trend to be similar: IPL and MR+IQL perform very similarly, with IPL performing slightly better in 3 of the 5 tasks. We will change this to the main version of the Table in the final version. We hope this improves the consistency of our experiments. \n\n**Bolding Scheme**\n\nFollowing IQL [1], we bold values when they are within 95% of the best method. We will state this explicitly in all Table captions. For the specific example brought up by the Reviewer, in Table 2 on Assembly for 500 queries IPL gets 0.9, 95% of which is 0.855, so MR would not be bolded. We understand that this scheme has less significance when performance values are quite low, and can make note of as well in the Table caption.\n\n**Limitations**\n\nThank you for pointing this out! We agree that there are some settings where transferring a reward function may be more useful than transferring a Q-function – largely stemming from the fact that a Q-function is tied to an inherent policy. If one wants to collect sub-optimal data for a task, a reward function may be necesary. Many Meta-RL approaches are also designed to leverage reward functions instead of Q-functions. Reward functions learned from preferences could also be used to smooth sparse reward functions – a Q-function could not easily be used for smoothing (though the advantage could! Ng. 1999). We will note the additional limitation of the transferability of an implicit reward function in Section 5 and include this discussion.\n\nThank you for also finding typos! We have fixed them.\n"}, {"review_id": "A9lUQW2diS", "reviewer": "Reviewer_HK3n", "summary": "The paper proposes a way to learn from preferences in the offline setting without learning a reward model. The main insight is that reward and Q functions are interchangeable and the policy learning using preference can be formulated directly as a function of Q which represents the reward function implicitly. This allows for increased learning performance without requiring to train a seperate reward function when doing RL with preferences.", "questions": "1. Gradient through target Q: When implementing the regularization on  $T^* Q  = Q(s,a)-\\gamma E[V^*(s')]$ it seems that only Q(s,a) is updated which means that reward is not being made to be zero centered and rather Q is made to be zero-centered.\n2. XQL vs IQL: The main paper discusses XQL and the experiments use IQL. Is there a reason behind this disconnect? Are there XQL experiments to validate?\n\nUpdate: Thanks to the authors for responding to a number of my questions and providing additional baselines. I have updated my score (4->5) and will reiterate on my score upon further discussion with reviewers. I hope to see the clarification on novelty and the additional baselines in the updated paper.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Strengths: \n\n1. The paper presents a new way to do policy learning from preference data without requiring learning an intermediate reward function.\n2. The experiments demonstrate that their method IPL outperforms baselines that learn from offline preferences in a number of prior tasks that include simulated MuJoCo locomotion tasks and manipulation tasks.", "weaknesses": "1. Missing theoretical underpinnings: A number of questions come up with the proposed method that should be addressed:\n    1. Is the learned policy optimal for the implicit Q function?\n        \n        Equation 6 is the probability that one trajectory is preferred by other under the policy $\\pi$ with the reward function $T^\\pi Q$ . Replacing $T^\\pi Q$  with $T^* Q  = Q(s,a)-\\gamma E[V^*(s')]$ , it is claimed that the preference learning objective will fit the optimal value function. There seems to be jump to this claim without a proof. In my opinion, it is important to show how minimizing Equation after line 201 leads to soft optimal $Q^*$ with ground truth reward function $r^E.$\n        \n    2. Large regularization weight\n        \n        A large regularization weight (1 and sometimes even larger than the main preference loss e.g in fig 2) is used in the work which makes it more confusing as to what combined objective is being optimized. While the intuition given by the authors makes sense, it might be great to show how this regularization does not change the stationary point of $Q^*$.\n        \n2. Clarification on novelty:\nIt is mentioned multiple times in the paper that:\n”Line 163: Our key insight is that Q-function learned by off-policy RL algorithm in face encodes the same information as the reward function”\n\nThis is not a new insight. It has been presented in multiple prior works [1,2,3] where a change of variables for reward is performed to remove the intermediate step of reward learning for imitation learn and the mapping from reward to Q has been studied theoretically in the work of [4] and more recently [5]. It seems the novelty is to add the preference loss to the IQ-Learn loss function.\n\n3. Prior literature on reward learning:\n    \n    Three baselines are used to compare the method IPL - MR, LSTM and PT. A number of other prior works exist which aim to do reward learning from preferences [6,7,8] and would be interesting to compare experimentally to and discuss the relationship in the paper. \n    \n4. Gap between proposed method and practical method: The paper discusses learning an optimal value function using the linex loss of XQL[9] but in the experiment section the value update is performed using IQL. Preference loss uses soft-optimal-Q whereas the policy update step in IQL updates for optimal Q. There seems to be a disconnect between the proposed method and practical method which might be important to address.\n5. Experiments: \na. How is the preference dataset constructed? Elaborating that in the main paper can increase the understanding considerably.\n    \n    b. Limited experiments: Only 1 baseline is compared against in the meta-world task and 4 simulated domains in total are considered in the locomotion and robomimic tasks. With limited theoretical understanding of the method, it might be important to establish the method empirically with previous baselines in order to show its merits.\n    \n\n[1] Kostrikov, Ilya, Ofir Nachum, and Jonathan Tompson. \"Imitation learning via off-policy distribution matching.\" *arXiv preprint arXiv:1912.05032* (2019).\n\n[2] Nachum, Ofir, and Bo Dai. \"Reinforcement learning via fenchel-rockafellar duality.\" *arXiv preprint arXiv:2001.01866* (2020).\n\n[3] Ma, Yecheng Jason, et al. \"Smodice: Versatile offline imitation learning via state occupancy matching.\" *arXiv e-prints* (2022): arXiv-2202.\n\n[4]Garg, Divyansh, et al. \"Iq-learn: Inverse soft-q learning for imitation.\" *Advances in Neural Information Processing Systems* 34 (2021): 4028-4039.\n\n[5] Sikchi, Harshit, Amy Zhang, and Scott Niekum. \"Imitation from Arbitrary Experience: A Dual Unification of Reinforcement and Imitation Learning Methods.\" *arXiv preprint arXiv:2302.08560* (2023).\n\n[6]: Brown, Daniel, et al. \"Safe imitation learning via fast bayesian reward inference from preferences.\" *International Conference on Machine Learning*. PMLR, 2020.\n\n[7]: Chen, Letian, Rohan Paleja, and Matthew Gombolay. \"Learning from suboptimal demonstration via self-supervised reward regression.\" *Conference on robot learning*. PMLR, 2021.\n\n[8]: Sikchi, Harshit, et al. \"A ranking game for imitation learning.\" *arXiv preprint arXiv:2202.03481* (2022).\n\n[9]: Garg, Divyansh, et al. \"Extreme Q-Learning: MaxEnt RL without Entropy.\" *arXiv preprint arXiv:2301.02328* (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and effort in reviewing our work and have made a number of changes as a result. \n\n**Theoretical Underpinnings**\n\nThe reviewer raised a number of questions regarding the theoretical underpinnings of IPL, namely its optimality and regularization. We provide a proof of IPL’s convergence in our central response due to space constraints. It depends on a general proof of the bijection, which was included in our response to Reviewer ggMH, and shows that IPL returns the optimal policy for the regularized expert reward.\n\nWe hope the proof addresses the following concerns of the reviewer:\n\n1. IPL is formally characterized, increasing understanding across the board.\n2. We have shown that IPL optimizes for the regularized reward $r^*$\n3. We have shown that IPL converges to the optimal policy for this reward. \n\nThe reviewer also asked if the reward function we optimize is $r_E$ or not. $r^*$ is not equal to $r_E$. However, we would like to note that most preference based RL works do not optimize for $r_E$. For example, PEBBLE [2] bounds the reward values with a Tanh network in practice, which as a multiplicative clamp does not preserve $r_E$. Preference Transformer [1] normalizes reward by scaling it via the max and min episode, which is another multiplicative transform that does not preserve $r_E$. Like all these methods, the policy recovered by IPL is not exactly that of $r_E$, though they are likely similar. We will update the text to make this more clear, along with our proofs. \n\nThe reviewer also asked about large regularization coefficients. Empirically we found that the stronger the regularization, the more bounded the $Q$-function becomes. Practically we found that in some domains a larger regularization weight was useful for some tasks – and this might have to do with the data. With higher quality data, there might not be too much difference between the $Q$ values for a good and bad segment. However, There is also an intuitive argument – that humans have relatively smooth preferences.  For example, humans would probably attribute reward smoothly across a video clip instead of attributing it all to a single frame. \n\n**Clarification of Novelty**\n\nThank you for bringing this up. We apologize if the contribution statement was not clear. We have now made it clear that our contribution is the application of the inverse bellman operator to the Preference-based RL, not the inverse-bellman operator itself.\n\n**Prior Literature and Other Baselines**\n\nWe will make sure to include citations for these other, relevant works. However, each of these works seeks to solve is slightly different from the Preference-based RL problem that we deal with. To our knowledge, Preference Transformer is state of the art in all of our benchmark tasks, which is why we selected it as a strong baseline. \n\n“Safe Imitation Learning Via Fast Bayesian Reward Inference From Preferences” from Brown et al. is largely designed for image-based experiments (our benchmarks are state), uses a number of self-supervised losses, and uses entire demonstrations instead of trajectory snippets. However, most critically this method requires hundreds of online MCMC rollouts during training, making it inapplicable to our offline benchmarks. \n\n“Learning from Suboptimal Demonstration via Self-Supervised Reward Regression” by Chen et al. is not a preference-based RL method. Instead, it is an inverse-RL method that improves upon D-REX to generate preference data for improving the policy. They assume access to a set of demonstrations, and no rankings. Their method addresses a different complementary challenge than IPL (inverse RL) by generating preference data. In this way, it is complementary to IPL and could be used to generate more data. It also requires online samples (unlike IPL) in order to generate extra data from noised policies.\n\n“A Ranking Game For Imitation Learning” by Harshit et al. is also an online inverse RL method, which can leverage some preference data. It is designed to be used online with access to a set of demonstrations, unlike the methods we test against. \n\nWhile these baselines are highly relevant, they aren’t exactly designed for the same setup we are concerned with. Specifically, most either address the inverse-RL problem and require online samples.\n \n**Gap Between Proposed and Practical Method**\n\nSee response to reviewer ggMH under Soft-Q vs Standard. \n\n**Experiments**\n\n1. *How is the preference dataset constructed?* Our datasets are taken from PT [1]. They are constructed by sampling segments from existing offline benchmarks (D4RL, robosuite), and asking a human to choose their preferred segment. \n\n2.  *Limited Experiments* We compare all methods across eight standard benchmark tasks. The MetaWorld experiments were designed to show that across even more tasks and datasets scales IPL matches the performance of the best baseline that uses an explicit reward model (MR+IQL). This addresses the central motivation of our method – we can remove the reward function and still perform as well as the best baseline.\n\nIf the reviewer feels strongly about the MetaWorld tasks, we are happy to run them with Preference Transformer for the final paper, but are compute limited and do not have the bandwidth to finish them immediately as this would require 5 tasks * 4 data scales * 5 seeds = 100 large transformer reward models, and as an academic lab we don’t have the resources available to do that on a short time-scale. \n\n**Question 1: Grad in Regularization**\n\nWe apply a “stop gradient” on the target Q or V. The regularization $r^2 = (TQ)^2 = (Q(s,a) - \\gamma V(s’))^2$ thus encourages $Q(s,a) = \\gamma V(s’)$, which would imply that the implicit reward is zero. The $Q$-function is encouraged to be centered at the value. \n\n**Question 2: XQL vs IQL**\n\nSee response to reviewer ggMH under Soft-Q vs Standard. \n\n[1] Kim et al.. Preference transformer. ICLR 2023.\n\n[2] Lee et al., PEBBLE. ICLR 2021."}]

Author rebuttal:
Dear reviewers, 

Thank you all for your detailed feedback. We have responded to all reviewers individually with more content but wanted to make a global list of the major changes that we have made to the manuscript and additional experiments included in our allowed one page upload. 

**Theoretical**

A number of reviewers had questions about the theory and derivation of IPL. We have made a number of changes to improve our understanding of the method. 

1. Reviewers ggMH and HK3n asked about why IPL was derived with XQL but our experiments were run with IQL. Originally, we used XQL for our derivations to be consistent with the Inverse RL literature that has previously used the MaxEnt Framework, but used IQL for experiments to exactly match baselines. To remove this confusion, we have updated the method section with a new expanded derivation of IPL under any off-policy RL algorithm that works via policy evaluation and policy improvement steps, not just XQL.

2. We apologize if the contribution statement was not clear (Reviewers ggMH, HK3n). We have now made it clear that our contribution is the application of the inverse bellman operator to the Preference-based RL, not the inverse-bellman operator itself.

3. Reviewers ggMH and HK3n asked about the validity of IPL’s practical implementation given it does not necessarily use a MaxEnt RL algorithm. However, the bijection between $Q$ and $r$ for a fixed policy holds in the general case (not just soft Q-Learning). In our response to Reviewer ggMH we include a proof of this bijection, analogous to Lemma 3.2 in IQ-Learning, for the general case. 

4. Reviewer HK3n asked for a more theoretical understanding of IPL, which we believe will be of interest to all reviewers. We prove that IPL converges to the optimal policy for the expert reward function subject to regularization and show that regularization is necessary to guarantee this convergence. For space, this is included below and will be in the final version of the paper. 




**Experimental**

See uploaded pdf

1. Reviewers kKfF asked about IPL’s applicability to online settings. Since IPL works for any Off-Policy RL algorithm, it also works online. We compare it with PEBBLE on two of the harder Meta-World tasks from Table 2, and find that IPL trades blows with PEBBLE while exhibiting lower variance. 

2. Reviewers kKfF asked about our implementation of MR+IQL and data-augmentation. We ablate the use of augmentations for both IPL and MR+IQL in Hopper and Robomimic Can. We find that data-augmentation makes a substantial difference for MR+IQL on Hopper, but has a smaller effect in the robotics domains. 

3. Reviewers kKfF and HK3n asked about the regularization values used by IPL. We include additional ablations on the regularization coefficient $\lambda$, showing that in many cases, IPL is very robust to this value.

4. Reviewer HQUc asked that we change Table 2 to use the same evaluation criterion as Table 1. We have included evaluations at a fixed stopping point and find the trend to be similar: IPL and MR+IQL perform very similarly, with IPL performing slightly better in 3 of the 5 tasks. 

**Proof that IPL Converges to the Optimal Policy corresponding to the regularized expert reward**

First, note that *the Bradley-Terry Model is underspecified* and only recovers $r_E$ up to a constant shift, as constants are canceled by the Boltzmann. While constant shifts don’t change the optimal policy, it does change the Q-function. To prove convergence, we show that the sequence of Q-functions is increasing, as is standard practice for off-policy RL algorithms. This is only possible if the reward function does not shift when optimizing the preference loss. 

We prove this statement in the Tabular setting. Let $Q_t \in \mathbb{R}^{|S \times A|}$ and $\pi_t$ indicate the Q-function and policy after update $t$, respectively. Let $Q_0 = 1/(1 - \gamma) \min_{S \times A} r(s,a)$. The inverse bellman operator tells us, in vector form, that $r = (I - \gamma P^\pi)Q$  where $P^\pi$ is the transition matrix. Let $r^* = \arg \min_r \mathbb{E}_{D_p}[y \log P_r + (1-y) \log (1 - P_r) + \lambda r^2]$, or the minimizer of the regularized preference loss.

At each step of IPL, we substitute the inverse bellman operator into the preference loss and optimize. Thus at convergence, $ (I - \gamma P^{\pi_t})Q_t = r^*$ uniquely because of the bijection between $r$ and $Q$ under a fixed $\pi$ (See proof under rebuttal to Reviewer ggMH Soft Q vs Standard).

Then, we use any off-policy RL algorithm that guarantees convergence (ie $Q^{\pi_{t+1}} \geq Q^{\pi_{t}}$) to obtain a new policy $\pi_{t+1}$ from $\pi_t$ and $Q_t$. Using $\pi_{t+1}$ we can obtain the transition matrix $P^{\pi_{t+1}}$ in tabular settings. 

Finally, we optimize the preference loss again using $P^{\pi_{t+1}}$ in the inverse Bellman operator to obtain $Q_{t+1}$. At convergence $ (I - \gamma P^{\pi_{t+1}})Q_{t+1} = r^*$ holds. 

Due to regularization $r^*$ is unique, and thus $Q_t$ and $Q_{t+1}$ are both Q-functions for the reward function $r^*$, just under different policies. If we did not have regularization, this might not be the case. However, we know from the policy improvement step, that $Q^{\pi_{t+1}} \geq Q^{\pi_{t}}$ necessarily, and thus $Q_{t+1} \geq Q_{t}$ for any $t$. The base case holds because $ \forall \pi, Q_0 \leq Q^\pi$ by construction. This concludes the proof. 

Thus, by repeatedly optimizing the Preference Loss and improving the policy, we obtain the optimal policy corresponding to $r^*$. 

This can also be seen (very) informally by looking at Eq. 6 in our paper: If the policy improvement step works, then $V^{\pi_{t+1}}(s’) \geq V^{\pi_{t}}(s’)$, and thus the $Q(s,a)$ will need to *increase* to fit the same optima of the regularized preference loss. 

**Conclusion**

We hope that the increased theoretical and experimental rigor addresses the reviewers concerns. See individual responses for more clarifications. 

Thank you!

Meta-review: The reviews are certainly in favor. The results are promising. However, according to the reviews, a few aspects needs further improvements. First of all, both of the proposed method and its proof could use a more thorough and delicate discussion of related work. A few reviews mention its similarity to IQLearn and other prior work, and hence challenge this paper's novelty. The rebuttal does not convince most reviews into changing their opinion on this. Why the changes are needed, and what are the benefits could be better explained. We would like this paper to incorporate most of the related work mentioned in reviews in its final version. Secondly, a more thorough discussion of its limitation is needed. For example, as mentioned in a few reviews. There is definitely cost associated with no explicit reward function. What is the cost (e.g., transferability, learning stability), and why this trade-off is appealing in certain applications? Overall, most reviews believe this is a solid paper, and its results outweigh the concern over its novelty.
Decision: Accept (poster)