SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof of the theorem essentially relies on the fact that for a fixed policy  $\pi$ , we can optimize  $\mathcal{L}_p(Q)$  (Eq. (6)) to fit  $r^*$ . Then, we can update the policy (or target values  $V^{\mathrm{targ}}$ ) and optimize  $\mathcal{L}_p(Q)$  again. Because  $r^*$  is unique, we fit  $r^*$  again the second time, but the  $Q$ -function has improved. There are many choices of regularizers where this holds. In tabular settings if  $\psi(r) = r^2$ ,  $\mathcal{L}_p(Q)$  reduces to L2 regularized logistic regression, which is strictly convex, guaranteeing convergence (Appendix A).
Effectively, IPL removes the need to learn a reward network, while still converging to similar solution as other preference-based RL algorithms. Learning a reward network requires more parameters and a completely separate optimization loop, increasing compute requirements. Moreover, an explicit reward model introduces a whole new suite of hyper-parameters that need to be tuned including the model architecture, capacity, learning rate, batch size, and stopping criterion. In fact, because human preference data is so difficult to collect, many approaches opt to use simple accuracy thresholds instead of validation criteria to decide when to stop training  $r_{\theta}$  [32]. All of these components make preference-based RL unreliable and high-variance. On the other hand, our method completely removes all of these parameters in exchange for a single  $\lambda$  hyper-parameter that controls the regularization strength. Though we have theoretically derived IPL, in the next section we provide practical recipes for applying it to offline preference-based RL.
3.3 IPL for Offline Preference-based RL
In offline preference-based RL, we assume access to a fixed offline dataset  $\mathcal{D}_o = \{(s,a,s')\}$  of interactions without reward labels generated by a reference policy  $\mu (a|s)$  in addition to the preference dataset  $\mathcal{D}_p$ . Common approaches to offline RL seek to learn conservative policies that do not stray too far away from the distribution of data generated by  $\mu (a|s)$ . This is critical to prevent the policy  $\pi$  from reaching out-of-distribution states during deployment which can be detrimental to performance. In this section, we detail a practical version of IPL that uses the
XQL offline RL algorithm [19]. XQL fits the KL-constrained RL objective
Algorithm 1: IPL Algorithm (XQL Variant)
Input:  $\mathcal{D}_p, \mathcal{D}_o, \lambda, \alpha$
for  $i = 1,2,3,\ldots$  do
Sample batches  $B_{p}\sim \mathcal{D}_{p},B_{o}\sim \mathcal{D}_{o}$
Update  $Q$ :  $\min_{Q} \bar{\mathbb{E}}_{B_p}[\mathcal{L}_p(Q)]$  (Eq. (6))
Update  $V$ :  $\min_V \mathbb{E}_{B_p \cup B_o}[e^z - z - 1]$
where  $\dot{z} = Q(s,a) - V(s)) / \alpha$
Finally, extract  $\pi (a|s)$  via:
$$
\max _ {\pi} \mathbb {E} _ {\mathcal {D} _ {p} \cup \mathcal {D} _ {o}} [ e ^ {(Q (s, a) - V (s)) / \alpha} \log \pi (a | s) ]
$$
$$
\max _ {\pi} \mathbb {E} _ {\pi} \left[ \sum_ {t = t ^ {\prime}} ^ {\infty} \gamma^ {t} \left(r (s _ {t}, a _ {t}) - \alpha \log \frac {\pi (a _ {t} | s _ {t})}{\mu (a _ {t} | s _ {t})}\right) \right]
$$
where  $\alpha$  controls the magnitude of the KL-divergence penalty. The XQL algorithm directly fits the optimal  $Q$ -function using the optimal soft-Bellman operator [19, 55]
$$
(\mathcal {B} _ {r} ^ {*} Q) (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} [ V ^ {\mathrm {t a r g}} (s ^ {\prime}) ], \mathrm {w h e r e} V ^ {\mathrm {t a r g}} (s) = \alpha \log \mathbb {E} _ {a \sim \mu (\cdot | s)} \left[ e ^ {Q (s, a) / \alpha} \right].
$$
In practice,  $V^{\mathrm{targ}}$  is estimated using the linex loss function over the current  $Q$ -function. Thus, to fit the optimal  $Q$ -function, IPL with XQL alternates between minimizing the preference loss  $\mathcal{L}_p(Q)$  (Eq. (6)), and updating a learned value function  $V$  until they converge to  $Q^{*}$  and  $V^{*}$ . Note that we are not limited to using just  $\mathcal{D}_p$ . Though the preference modeling part of  $\mathcal{L}_p(Q)$  can only be optimized with preference data  $\mathcal{D}_p$ , the value function can be updated with offline data as well. In the presence
of additional offline data, we find that updating the value function using  $\mathcal{D}_p\cup \mathcal{D}_o$  leads to better performance. We approximate L2 regularization with the regularizer  $\psi (r) = \mathbb{E}_{\mathcal{D}_p\cup \mathcal{D}_o}[r(s,a)^2 ]$  which imposes an L2 penalty across the support of the data. While one might try to use weight decay to emulate L2-regularization, doing so is difficult in practice as  $\mathcal{T}^* Q$  depends on both the  $Q$  network and the target network. We find that weighting the regularization equally between  $\mathcal{D}_p$  and  $\mathcal{D}_o$  performs well. After  $Q$  and  $V$  have converged, we can extract the policy using the closed form relationship  $\pi^{*}(a|s)\propto \mu (a|s)\exp ((Q^{*}(s,a) - V^{*}(s)) / \alpha)$  for KL-constrained RL as in Garg et al. [19], Peng et al. [44]. The full algorithm for IPL with XQL can be found in 1.
Though we have shown how IPL can be instantiated with  $\chiQL$ , it is fully with other offline RL algorithms. In fact, IPL can also be used with online RL algorithms like SAC [20]. Critically, this makes the IPL framework general, as it can remove the need for reward modeling in nearly any preference-based RL setting. This makes IPL simpler and more efficient. In the next section, we show that IPL can attain the same performance as strong offline preference-based RL baselines, without learning a reward network.
In this section, we aim to answer the following questions: First, how does IPL compare to prior preference-based RL algorithms on standard benchmarks? Second, how does IPL perform in extremely data-limited settings? And finally, how efficient is IPL in comparison to two-phase preference-based RL methods?
As discussed in the previous section, though we use a KL-constrained objective for our theoretical derivation, in practice we can construct versions of IPL based on any offline RL algorithm. In our experiments we evaluate IPL with Implicit Q-Learning (IQL) [30], since it has been used in prior offline preference-based RL works. This allows us to directly compare IPL by isolating its implicit reward component and using the same exact hyper-parameters as prior works. Using IPL with IQL amounts to updating the value function according to the asymmetric expectile loss function instead of the linex loss function. Concretely, this can be done by replacing the value update in Algorithm 1 with  $\min_V\mathbb{E}_{B_p\cup B_o}\left[|\tau -\mathbb{1}(Q(s,a) - V(s) < 0)|(Q(s,a) - V(s))^2\right]$  where  $\tau$  is the expectile.
Inspired by Park et al. [43], we introduce data augmentations that sample sub-sections of behavior segments  $\sigma$  during training. While such augmentations are inapplicable to non-Markovian reward models, we find that they boost performance for Markovian reward models while also reducing the total number of state-action pairs per batch of preference data. This is important as IPL needs data from both  $\mathcal{D}_p$  and  $\mathcal{D}_o$  to regularize the implicit reward function. Additional experiment details and hyper-parameters can be found in the Appendix.
4.2 How does IPL perform on preference-based RL benchmarks?
We compare IPL to other offline preference-based RL approaches on D4RL Gym Locomotion [17] and Robosuite robotics [36] datasets with real-human preference data from Kim et al. [28]. We compare IQL-based IPL, with the same hyper-parameters, to various baselines that learn a reward model  $r_{\theta}$  before optimization with IQL. Markovian Reward or MR denotes using a standard Markovian MLP reward model, like those used in Christiano et al. [13] and Lee et al. [32]. Note that this is also equivalent to T-REX [11] for offline RL. Non-Markovian Reward or NMR denotes using the non-Markovian LSTM based reward model from Early et al. [15]. Preference Transformer (PT) is a state-of-the-art approach that leverages a large transformer architecture to learn a non-Markovian reward and preference weighting function. B-REX uses bayesian optimization to fit a linear reward function from predefined features [12], which in our case are random Gaussian projections of the states and actions. For fairness, we also compare against our own implementation of IQL with a Markovian Reward function that uses the same data augmentation as IPL.
Our results are summarized in Table 1. Starting with the first column, we see that preference-based RL methods are able to match IQL with the ground truth reward function in many cases. On several tasks, however, the MR implementation from Kim et al. [28] fairs rather poorly. The non-Markovian methods, (NMR and PT) improve performance. It is worth noting that on many tasks our
Dataset	IQL (Oracle)	MR (from [28])	LSTM (from [28])	PT (from [28])	BREX (reimpl.)	MR (reimpl.)	IPL (Ours)
hop-m-r	83.06 ±15.8	11.56 ±30.3	57.88 ±40.6	84.54 ±4.1	62.0 ±20.3	70.20 ±35.0	73.57 ±6.7
hop-m-e	73.55 ±41.5	57.75 ±23.7	38.63 ±35.6	68.96 ±33.9	85.1 ±8.0	103.0 ±5.6	74.52 ±10.1
walk-m-r	73.11 ±8.1	72.07 ±2.0	77.00 ±3.0	71.27 ±10.3	10.3 ±5.4	68.79 ±5.6	59.92 ±5.1
walk-m-e	107.8 ±2.2	108.3 ±3.9	110.4 ±0.9	110.1 ±0.2	99.62 ±3.0	109.1 ±1.3	108.51 ±0.6
lift-ph	96.75 ±1.8	84.75 ±6.2	91.50 ±5.4	91.75 ±5.9	96.6 ±3.0	98.84 ±2.3	97.60 ±2.9
lift-mh	86.75 ±2.8	91.00 ±2.8	90.8 ±5.8	86.75 ±6.0	60.4 ±25.1	90.04 ±4.5	87.20 ±5.3
can-ph	74.50 ±6.8	68.00 ±9.1	62.00 ±10.9	69.67 ±5.9	63.0 ±20.3	76.40 ±3.7	74.8 ±2.4
can-mh	56.25 ±8.8	47.50 ±3.5	30.50 ±8.7	50.50 ±6.5	30.4 ±23.0	53.6 ±7.9	57.6 ±5.0
Avg Std	10.95	10.2	13.87	9.08	13.77	8.23	4.8
Table 1: Average normalized scores of all baselines on human-preference benchmarks from Kim et al. [28]. For the D4RL locomotion tasks "hop" corresponds to hopper, "m" to medium (training the data generating agent to 1/3 expert performance), "r" to replay buffer data, and "e" to data from the end of training. For the Robomimic tasks lift and can, "ph" corresponds to proficient human data and "mh" to multi-human data of differing optimality. The first four columns are taken from Kim et al. [28]. "reimpl." is our reimplementation of Markovian Reward with IQL. The "Avg Std" row shows the average standard deviation across all eight environments. We run five seeds and report the final performance at the end of training like Kostrikov et al. [30]. Bolded values are within  $95\%$  of the top performing method. Note that standard deviation values in the table were rounded for space. On some tasks IPL achieves higher performance earlier in training, which is not reflected above (See Appendix B). We find that IPL outperforms PT on many environments, and also performs similarly to our implementation of MR despite not training a reward function.
implementation of a MR (sixth column) performs far better than reported in Kim et al. [28], likely due to our careful tuning of  $r_{\theta}$  and use of data-augmentations. Our method, IPL, achieves competitive performance across the board.
In general, IPL with IQL performs on-par or better than both our implementation of MR and PT in most datasets despite not learning a separate reward network. Specifically, IPL has the same performance or better performance than our MR implementation on six of eight tasks. More importantly, IPL does extremely well in comparison to Preference Transformer's reported results. On five of eight tasks IPL performs better than PT while having over 10 times fewer parameters, making IPL far more efficient. To be consistent with Kim et al. [28], we report results after a million training steps but performance for IPL often peaks earlier (see learning curves in the Appendix). For example, with early stopping IPL also outperforms PT on "hop-m-r". We posit that this is because the  $Q$ -function in IPL is tasked with both fitting the expert's preference model and optimal policy simultaneously, making both the policy and reward function non-stationary during training. In some datasets, this was more unstable.
IPL also has the lowest average standard-deviation across seeds, meaning it yields more consistent results than explicit reward methods. For standard two-phase preference-based RL algorithms, errors in the reward model are propagated to and exacerbated by the  $Q$  function. IPL circumvents this problem by not explicitly learning the reward.
Finally, in Table 2, we consider various design decisions of IPL. Augmentations provide a strong boost in the robotics environment, but offer only minor improvements in locomotion. Removing regularization, however, is detrimental to performance. This is likely because without regularization, the implicit reward values can continue to increase, leading to exploding estimates of  $Q$ . Finally, we show that IPL is compatible with other offline RL algorithms by combining it with XQL [19]. We find that with XQL, IPL performs even better on some tasks, but worse on others. Finally, in Appendix B, we also show that IPL can be combined with online preference-based RL algorithms like PEBBLE [32].
4.3 How does IPL scale with Data?
Collecting preference comparisons is often viewed as the most expensive part of preference-based RL. To investigate how well IPL performs in data limited settings, we construct scripted preference datasets
Dataset	No Aug	λ = 0	IPL-XQL	IPL
hop-m-r	70.46 ±6.7	10.41 ±2.26	80.4 ±2.13	73.57 ±6.67
walk-m-r	58.50 ±5.3	4.85 ±1.52	57.82 ±5.24	59.92 ±5.11
lift-mh	84.8 ±4.1	52.60 ±10.1	89.00 ±4.4	87.20 ±5.3
can-mh	53.2 ±5.8	13.8 ±5.7	59.0 ±5.0	57.6 ±5.00
Table 2: Ablations for IPL on the offline human-preference benchmark. We consider removing data augmentation, removing regularization  $\lambda = 0$ , and other offline RL algorithms (XQL). Full results can be found in Appendix B.
Preference Queries	500	1000	2000	4000
Button Press	MR	66.0 ±8.0	49.3 ±12.1	54.7 ±26.8	78.3 ±9.2
IPL	53.3 ±8.5	60.1 ±12.8	70.2 ±2.5	90.2 ±6.5
Drawer Open	MR	65.9 ±9.9	87.2 ±5.2	89.7 ±6.4	94.6 ±3.9
IPL	62.1 ±4.8	78.7 ±12.4	89.5 ±5.0	96.6 ±1.3
Sweep Into	MR	33.0 ±5.7	46.2 ±6.0	63.2 ±13.7	70.8 ±7.9
IPL	34.5 ±2.3	48.2 ±7.2	58.8 ±7.4	65.9 ±6.7
Plate Slide	MR	54.6 ±5.3	57.2 ±4.5	23.9 ±18.8	55.2 ±3.0
IPL	52.9 ±4.8	55.8 ±2.2	55.4 ±3.1	54.9 ±2.8
Assembly	MR	0.6 ±0.7	0.7 ±1.0	0.0 ±0.0	2.6 ±2.8
IPL	0.9 ±0.6	1.5 ±1.5	1.7 ±1.9	5.5 ±5.2
Avg Std	MR	5.9	5.76	13.14	5.36
IPL	4.2	7.22	3.98	4.5
Table 3: Results on five MetaWorld tasks at four different preference data scales. We run five seeds for each method, and take the highest average performance across seeds from the learning curves. More details can be found in Appendix B. IPL performs the same or better than IQL with a Markovian reward model on the majority of tasks and preference data scales without training a reward model.
of four different sizes for five tasks from the MetaWorld benchmark [56] used in prior preference-based RL works [32, 22]. We then train on the preference data  $\mathcal{D}_p$  by setting  $\mathcal{D}_o = \{(s, a, s') \in \mathcal{D}_p\}$  and use the same hyper-parameters for all environments and methods where applicable. Our results are summarized in Table 3. Again, IPL is a strong reward-free baseline. We find that at all data scales, IPL performs competitively to our implementation of MR (IQL with a learned Markovian reward) and consistently outperforms it in Button Press and Assembly. Increasing the amount of preference data generally improves performance across the board. However, as we generate queries uniformly at random some preference datasets may be easier to learn from than others, leading to deviations from this trend in some cases. As in the benchmark results in Table 1, IPL exhibits lower variance across seeds and tasks, in this case at three of four data scales.
4.4 How efficient is IPL?
One benefit of IPL over other preference-based RL methods is its parameter efficiency. By removing the reward network, IPL uses fewer parameters than other methods while achieving the same performance. In Table 4, we show the number of parameters for each method used in the last two sections. Preference Transformer uses over ten times more parameters than IPL, and the LSTM-based NMR model from Early et al. [15] uses nearly twice as many. When dealing with a limited compute or memory budget, this can be important. To exacerbate this effect, we consider an extremely parameter efficient version of IPL, denoted "IPL (64)" in Table 4, based on Advantage Weighted Actor Critic (AWAC) [39] which eliminates the second critic and value networks used in IQL [30] and uses a two-layer 64-dimensional MLP. We then compare this parameter-efficient IPL to MR with the same parameter budget which results in "MR (35)", a 35-dimensional MLP. Results are depicted on the left of Fig. 2. MR trained with a smaller network is unable to adequately fit the data, resulting in lower performance. Only after increasing the network size past that of IPL can MR begin to match performance.
Aside from parameter efficiency, IPL is also "hyper-parameter efficient". By removing the reward network, IPL removes a whole set of hyper-parameters associated with two phase preference based RL methods, like reward network architecture, learning rate, stopping criterion, and more. In the
MetaWorld Printer Open Ablations with 4000 Queries
Figure 2: Left: Performance comparison with different parameter numbers. MR (35) has the same parameter budget as IPL (64). MR (64) has over twice as many. We see that with the same number of parameters as IPL, MR is unable to adequately fit the data and performs poorly. Middle: MR when the reward function is trained for a varying number of steps – with too few the reward model under-fits, and with too many it over-fits, both leading to worse performance. Right: IPL with different regularization strengths. On the drawer open task, performance is largely unaffected. For more ablations, see the Appendix.
middle of Fig. 2 we show how the performance of MR is affected when the reward function is over or under fit. Choosing the correct number of steps to train the reward model usually requires collecting a validation set of preference data, which is costly to obtain. Instead of this, IPL only has a single regularization parameter,  $\lambda$ . The right side of Fig. 2 shows the sensitivity of IPL to  $\lambda$ . We find that in many cases, varying  $\lambda$  has little effect on performance unless it is perturbed by a large amount.
Summary. We introduce Inverse Preference Learning, a novel algorithm for offline preference-based RL that avoids learning a reward function. Our key insight is to leverage the inverse soft-Bellman operator, which computes the mapping from  $Q$ -functions to rewards under a fixed policy. The IPL algorithm trains a  $Q$ -function to regress towards the optimal  $Q^*$  while at the same time admitting implicit reward values that are consistent with an expert's preferences. Even though IPL does not require learning a separate reward network, on robotics benchmarks it attains competitive performance with preference-based RL baselines that use twice to ten-times the number of model parameters.
Limitations and Future Work. A number of future directions remain. Specifically, the implicit reward function and policy learned by IPL are both non-stationary during training, which sometimes causes learning to be more unstable than with a fixed reward function.
This is a core limitation future work could address by better mixing
Method	Params
PT	2942218
NMR	508746
MR	348426
IPL	278537
MR (64)	34892
IPL (64)	14025
MR (35)	14012
Table 4: Parameter counts for different methods. The bottom three rows are for the limited parameter budget experiments in Section 4.4.
policy improvement and preference-matching steps to improve stability. More broadly, implicit reward preference-based RL methods are not limited to continuous control or binary feedback. Applying implicit reward techniques to other forms of feedback or extending IPL to language-based RLHF tasks remain exciting future directions.
This work was supported by ONR, DARPA YFA, Ford, and NSF Awards #1941722 and #2218760. JH is supported by by the National Defense Science Engineering Graduate (NDSEG) Fellowship Program. We would additionally like to thank Div Garg and Chris Cundy for useful discussions.
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, 2004.
[2] Baris Akgun, Maya Cakmak, Karl Jiang, and Andrea L Thomaz. Keyframe-based learning from demonstration. International Journal of Social Robotics, 4(4):343-355, 2012.
[3] Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2011.
[4] Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. LS-IQ: Implicit reward regularization for inverse reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=o3Q4m8jg4BR.
[5] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.
[6] Fahiem Bacchus, Craig Boutilier, and Adam Grove. Rewarding behaviors. In National Conference on Artificial Intelligence, 1996.
[7] Chandrayee Basu, Qian Yang, David Hungerman, Mukesh Sinahal, and Anca D Draqan. Do you want your autonomous car to drive like you? In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI, pages 417-425. IEEE, 2017.
[8] Erdem Bıyık, Daniel A Lazar, Dorsa Sadigh, and Ramtin Pedarsani. The green choice: Learning and influencing human decisions on shared roads. In 2019 IEEE 58th conference on decision and control (CDC), pages 347-354. IEEE, 2019.
[9] Erdem Biyik, Nicolas Huynh, Mykel J. Kochenderfer, and Dorsa Sadigh. Active preference-based gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020.
[10] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[11] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783-792. PMLR, 2019.
[12] Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pages 1165–1177. PMLR, 2020.
[13] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
[14] Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with a novel acquisition function. Autonomous Robots, 39(3):389-405, 2015.
[15] Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ramchurn. Non-markovian reward modelling from trajectory labels via interpretable multiple instance learning. In Advances in Neural Information Processing Systems, 2022.