SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[50] Daniel Shin and Daniel S Brown. Offline preference-based apprenticeship learning. arXiv preprint arXiv:2107.09251, 2021.
[51] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325, 2020.
[52] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
[53] Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems, 2012.
[54] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021.
[55] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations, 2023.
[56] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, 2020.
[57] Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real world robotic reinforcement learning. In International Conference on Learning Representations, 2020.
[58] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
[59] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
A.1 Proofs
Lemma 1 For any fixed policy  $\pi$  the inverse bellman operator  $\mathcal{T}^{\pi}$  establishes a bijection between  $r$  and  $Q$ . Moreover, for any  $r$ ,  $Q = (\mathcal{T}^{\pi})^{-1}r$  is the unique fixed point of the Bellman operator  $\mathcal{B}_r^\pi$ . (Adapted from Garg et al. [18])
Proof. Let  $P^{\pi}$  be the stochastic transition matrix for the MDP corresponding to a fixed policy  $\pi$ . In vector form, the inverse bellman operator becomes  $r = \mathcal{T}^{\pi}Q = (I - \gamma P^{\pi})Q$ . We can establish a bijection between  $Q$  and  $r$  by showing that  $(I - \gamma P^{\pi})$  is invertible. As  $P^{\pi}$  defines a valid probability distribution over next stat-action pairs and  $\gamma < 1$ , we have that  $||\gamma P^{\pi}|| < 1$ . Thus, its Neumann series convergences, which implies the existence of  $(I - \gamma P^{\pi})^{-1}$ . So,  $Q = (I - \gamma P^{\pi})^{-1}r$  and a bijection exists. Using this, we can also show a 1-1 mapping with the bellman operator under reward  $r$ . We have  $Q = (I - \gamma P^{\pi})^{-1}r = (\mathcal{T}^{\pi})^{-1}r = \mathcal{B}_r^{\pi}Q$  at the fixed point of  $\mathcal{B}_r^{\pi}$ .
Theorem 1 Given an off-policy RL algorithm that converges to the optimal policy  $\pi_r^*$  for some reward function  $r$  and regularizer  $\psi$  such that Eq. (2) is strictly convex, IPL converges to  $\pi_{r^*}^*$  corresponding to reward function  $r^* = \arg \min_r\mathbb{E}_{\mathcal{D}_p}[D_{KL}(P_{r_E}||P_\theta)] + \lambda \psi (r)$ .
Proof. We prove this statement in the Tabular setting, first for algorithms that use policy improvement. Let  $Q_{t} \in \mathbb{R}^{|S \times A|}$  and  $\pi_{t}$  indicate the Q-function and policy after  $t$  iterations. Let  $Q_{0} = 1 / (1 - \gamma) \min_{S \times A} r(s, a)$ . The inverse bellman operator tells us, in vector form, that  $r = (I - \gamma P^{\pi})Q$  where  $P^{\pi}$  is the transition matrix. Let  $r^{*} = \arg \min_{r} \mathbb{E}_{D_{p}}[y \log P_{r} + (1 - y) \log (1 - P_{r})] + \lambda \psi(r)$ , or the minimizer of the preference loss with regularizer  $\psi$  such that we converge to a unique  $r^{*}$ .
At each step of IPL, we substitute the inverse bellman operator into the preference loss and optimize. Thus at convergence,  $(I - \gamma P^{\pi_t})Q_t = r^*$  uniquely due to Lemma 1. Then, there are two cases based on the type of RL algorithm.
If our RL algorithm can directly estimate  $\mathcal{B}_r^*$ , then we are done. This is because we have assumed convergence, and thus  $\mathcal{T}^{*}Q = r^{*}$ . By the bijection established in Lemma 1, we have that  $Q = (\mathcal{T}^{*})^{-1}r^{*}$  which by also Lemma 1 is the unique fixed point of  $\mathcal{B}_{r^*}^*$  which is  $Q_{r^*}$ . Thus, we have recovered the optimal  $Q$  function for  $r^*$  from which the optimal policy  $\pi_{r^*}^*$  can be extracted.
If we have an RL algorithm that uses policy improvement, we consider multiple steps of IPL. If the RL algorithm guarantees convergence via policy improvement, then we use  $\pi_t$  and  $Q_{t}$  to obtain a new policy  $\pi_{t + 1}$ . Using  $\pi_{t + 1}$  we can obtain the transition matrix  $P^{\pi_{t + 1}}$ . Finally, we optimize the preference loss again using  $P^{\pi_{t + 1}}$  in the inverse Bellman operator to obtain  $Q_{t + 1}$ . At convergence  $(I - \gamma P^{\pi_{t + 1}})Q_{t + 1} = r^{*}$  holds. As  $r^{*}$  is unique,  $Q_{t}$  and  $Q_{t + 1}$  are both Q-functions for the reward function  $r^{*}$ , just under different policies. We know from the definition of policy improvement, that  $Q^{\pi_{t + 1}} \geq Q^{\pi_t}$  necessarily, and thus  $Q_{t + 1} \geq Q_{t}$  for any  $t$ . Convergence is possible as according to Lemma 1,  $Q^{*} = (\mathcal{T}^{*})^{-1}r$  is the a fixed point of  $\mathcal{B}_r^*$ .
Proposition 1 If  $\phi(r) = r^2$ , then IPL converges to the optimal policy corresponding to  $r^* = \arg \min_r \mathbb{E}_{\mathcal{D}_p} [D_{KL}(P_{r_E}||P_\theta)] + \lambda \psi(r)$  in tabular settings.
Proof. The preference-based loss function with L2 regularization can be viewed as L2 regularized logistic regression by writing the logits as a dot-product between a preference comparison vector  $x$  comprised of  $-\gamma^t$ ,  $\gamma^t$  and 0 terms and a reward function  $r$ . The Hessian for this objective is then  $X^T D X + \lambda I$  where  $D_{ii} = \mathrm{logistic}(x_i \top r)(1 - \mathrm{logistic}(x_i \top r))$ , which is positive definite. Thus the problem is strictly convex and  $r^*$  is unique, so IPL converges to its optimal policy by Theorem 1. Note that to guarantee this we must regularize  $r$  across the entire state-action space, analogous to regularizing the weight vector in logistic regression.
A.2 Connections to DPO
Concurrent work, called Direct Preference Optimization (DPO) [45] also remove the need for explicit reward modeling for learning from preferences, but do so in the contextual bandits setting for language
models. Thus, DPO is limited to preference queries segment length 1 that must start from the same state. IPL is in fact, a more general version of DPO that does not have these restrictions. Specifically, IPL with  $\mathcal{XQL}$  recovers the same exact policy as DPO when applied to the contextual bandits setting.
Within the bandits setting, there is no "next-state" and  $V^{*}(s^{\prime})$  is removed, and the inverse bellman operator becomes just  $Q(s,a) = r(s,a)$ . The optimal XQL policy is  $\pi^{*} = \mu (a|s)e^{Q^{*}(s,a)} / Z(s)$  where  $Z$  is the partition function. By rearranging,  $\mathcal{T}^{*}\mathcal{Q} = \mathcal{Q}^{*}(s,a) = \log \frac{\pi(a|s)}{\mu(a|s)} +Z(s)$ . We can plug this into the preference model induced by  $Q$  in Eq. (5). In the RLHF setting, the partition function cancels since we assume the context to be the same between preferences. This exactly results in the DPO algorithm, showing the DPO is in fact just an instantiation of IPL for contextual bandits.
A.3 IPL with Rankings
IPL can easily be extended to rankings using a Plackett Luce model. Consider permutations  $\tau$  over  $K$  segments:
$$
P _ {r _ {E}} (\tau) = \prod_ {k = 1} ^ {K} \left(\exp \sum_ {t} r _ {E} (s _ {t} ^ {\tau_ {k}}, a _ {t} ^ {\tau_ {k}})\right) / d _ {k}
$$
where  $d_{k} = \sum_{j=k}^{K} \exp \sum_{t} \gamma^{t} r_{E}(s_{t}^{\tau_{j}}, a_{t}^{\tau_{j}})$ . Then, we make the same substitution using the inverse bellman operator giving us the permutation model  $P_{Q}$  implied by the Q function, and run maximum likelihood estimation over the model with the preference loss  $\mathcal{L}_{p}(Q) = \mathbb{E}_{\tau \sim \mathcal{D}_{p}}\left[\log P_{Q}(\tau)\right] + \lambda \psi(r)$ .
B Results
We divide this part of appendix into four different sections following the results section. Each section additionally provides hyper-parameters used for IPL in that section. The first section, setup, contains detailed information on the experimental setup and hyper-parameters used. The second section on benchmark results gives full learning curves for the experiments in Section 4.2. The third section provides full learning curves for the MetaWorld and Data-scaling experiments. The final Appendix section provides extended ablations.
Here we provide the full algorithmic outline of IPL using Implicit Q-Learning [30] that mimics our implementation. While in practice the policy  $\pi$  could be extracted at the end of training, we do it simultaneously as in [30] in order to construct learning curves.
Algorithm 2: IPL Algorithm (IQL Variant)
Input:  $\mathcal{D}_p, \mathcal{D}_o, \lambda, \alpha$   
for  $i = 1,2,3,\ldots$  do  
Sample batches  $B_p \sim \mathcal{D}_p, B_o \sim \mathcal{D}_o$   
Update  $Q$ :  $\min_Q \mathbb{E}_{B_p}[\mathcal{L}_p(Q)] + \lambda \mathbb{E}_{B_p \cup B_o}[\mathcal{L}_r(Q)]$   
Update  $V$ :  $\min_V \mathbb{E}_{B_p \cup B_o}\left[|\tau - \mathbb{1}(Q(s,a) - V(s))| (Q(s,a) - V(s))^2\right]$   
Update  $\pi$ :  $\max_{\pi} \mathbb{E}_{\mathcal{D}_p \cup \mathcal{D}_o}[e^{\beta(Q(s,a) - V(s))} \log \pi(a|s)]$
Note that above we write the temperature parameter  $\beta$  as done in IQL, instead of how it is usually done, using  $\alpha$  in the denominator [19, 44].
When sampling batches of preference data  $B_{p} \sim \mathcal{D}_{p}$ , we take sub-samples of each segment  $\sigma$  of length  $s$ . For a sampled data point  $(\sigma^{(1)},\sigma^{(2)},y)$ , we sample start  $\sim$  Unif[0, 1, 2, ...k - s] and then let take  $\sigma = s_{\mathrm{start}}, a_{\mathrm{start}}, \ldots, s_{\mathrm{start + s}}$ . We use the same start value across the entire batch.
Given that we run experiments using MLPs, all of our experiments were run on CPU compute resources. Each seed for each method requires one CPU core and 8 Gb of memory.
Here we provide details for our experiments on the preference-based RL benchmark from Kim et al. [28]. We use the same hyperparameters as Kim et al. [28] and Kostrikov et al. [30] as shown in Table 5.
Gym-Mujoco Locomotion. Hopper and Walker2D agents are tasked with learning locomotion policies from datasets of varying qualities taken from the D4RL [17] benchmark. Preference datasets were constructed by Kim et al. [28] by uniformly sampling segments. Preference datasets for "medium" quality offline datasets contain 500 queries, while preference datasets for "expert" quality offline datasets contain 100 queries. Segment length  $k = 100$  for all datasets, and were subsampled to length  $s = 64$  by IPL and our MR (reimpl). Evaluation was preformed over 10 episodes every 5000 steps. Full learning curves are shown in Fig. 3.
RoboMimic. The RoboMimic datasets contain interaction data of two types: ph — proficient human and mh – multihuman. The multi-human data was collected from human demonstrators of mixed quality. The robot is tasked with learning how to lift a cube (lift) or pick and place a can (can). Preference datasets were again taken directly from Kim et al. [28]. Preference datasets of size 100 with segment lengths  $k = 50$ , randomly sub-sampled to length  $s = 32$  were used for the ph datasets. Preference datasets of size 500 with segment lengths  $k = 100$ , randomly sub-sampled to length  $s = 64$  were used for the mh datasets. Evaluation was performed over 25 episodes every 50000 steps. Full learning curves are shown in Fig. 4.
Online Experiments. We also test a combination of IPL with PEBBLE [32] on a few tasks in the MetaWorld benchmark. Results can be found in Fig. 5
Figure 3: Full learning curves on the D4RL locomotion benchmark with human preferences.
B.2 Benchmark Results
Robomimic with Human Preferences
Can PH
Figure 4: Full learning curves on the RoboMimic benchmark with human preferences.
Online MetaWorld with Scripted Preferences
Figure 5: Online experiments for two of the most difficult MetaWorld environments. Each method (run on 3 seeds) gets feedback every 5000 steps for a total of  $20\mathrm{k}$  artificially queries. Notice that IPL has much lower variance across runs, particularly in Painter.
Common Hyperparameters
Parameter	Locomotion	Robomimic
Q, V, π Arch	2x 256d	2x 256d
Learning Rate	0.0003	0.0003
Optimizer	Adam	Adam
β	3.0	0.5
τ	0.7	0.7
D0Batch Size	256	256
DpBatch Size	8	8
Training Steps	1 Mil	1 Mil
k	100	100, 50
Subsample s	64	64, 32
MR Hyperparameters
Parameter	Locomotion	Robomimic
rθArch	2x 256d	2x 256d
rθLR	0.0003	0.0003
rθOptimizer	Adam	Adam
rθSteps	20k	20k
IPL Hyperparameters
Parameter	Locomotion	Robomimic
λ	0.5	4
Table 5: Hyperparameters used for the benchmark experiments. We can see that IPL has fewer hyperparameters. For the XQL experiments we use  $\alpha = 2$  for locomotion and  $\alpha = 5$  for Robomimic. We left all other parameters the same.
B.3 Data Scaling Results
Experiments for data scaling were conducted on the MetaWorld benchmark from Yu et al. [56]. Offline datasets for five different MetaWorld tasks were constructed as follows: Collect 100 trajectories of expert data on the target task using the built in ground truth policies with the addition of Gaussian noise of standard deviation 1.0. Collect 100 trajectories of sub-optimal data by running the ground-truth policy for a different randomization of the target task with Gaussian noise 1.0. Collect 100 trajectories of even more sub-optimal data by running the ground truth policy of a different task with Gaussian noise standard deviation 1.0 in the target domain. Finally, collect 100 trajectories with uniform random actions. As MetaWorld episodes are 500 steps long, this results in 200,000 time-steps of data. We then construct preference datasets by uniformly sampling segments from the offline dataset and assigning labels  $y$  according to  $\sum_{t} r(s_{t}^{(1)}, a_{t}^{(1)}) > \sum_{t} r(s_{t}^{(2)}, a_{t}^{(2)})$  where  $r$  is the ground truth reward provided by metaworld. We then train using only the data from  $\mathcal{D}$  General architecture hyper-parameters were taken from Lee et al. [32], Hejna and Sadigh [22] which also use the MetaWorld benchmark, but for online preference-based RL. Full-hyper parameters are shown in Table 6. We run 20 evaluation episodes every 2500 steps. Full learning curves are shown in Fig. 6. When reporting values in Table 3, we choose the maximum point on the learning curves which average across five seeds. This provides results as if early stopping was given by an oracle, which is less optimistic than averaging the maximum of each seed as done in Mandlekar et al. [36].
Figure 6: Full learning curves for the MetaWorld data scaling results with scripted preferences.
MetaWorld with Scripted Preferences
Common Hyperparameters
Parameter	Value
Q, V, π Arch	3x 256d
Learning Rate	0.0003
Optimizer	Adam
β	4.0
τ	0.7
Dp Batch Size	16
Training Steps	200k
k	25
Subsample s	16
MR Hyperparameters
Parameter	Value
rθArch	3x 256d
rθLR	0.0003
rθOptimizer	Adam
rθSteps	20k
IPL Hyperparameters
Parameter	Value
λ	0.5
B.4 Ablations
In this section we provide additional ablations on both the benchmark datasets and MetaWorld datasets. We keep the hyperparameters the same, except for the parameter-efficient experiments.
Benchmark IPL Ablations. We include results of full ablations for IPL on the benchmark tasks in Table 7. We additionally provide comparisons between IPL and MR + IQL with and without data augmentation in Fig. 7.
Table 6: Hyper-parameters used in the MetaWorld data scaling experiments.
Dataset	No Aug	λ = 0	IPL-XQL	IPL
hop-m-r	70.46 ± 6.73	10.41 ± 2.26	80.4 ± 2.13	73.57 ± 6.67
hop-m-e	51.26 ± 17.46	52.81 ± 7.45	54.3 ± 12.33	74.52 ± 10.11
walk-m-r	58.50 ± 5.31	4.85 ± 1.52	57.82 ± 5.24	59.92 ± 5.11
walk-m-e	108.91 ± 0.18	58.77 ± 15.75	75.16 ± 23.40	108.51 ± 0.60
lift-ph	98.0 ± 2.53	85.2 ± 7.71	98.40 ± 2.59	97.60 ± 2.94
lift-mh	84.8 ± 4.11	52.60 ± 10.07	89.00 ± 4.37	87.20 ± 5.31
can-ph	68.6 ± 8.25	25.4 ± 5.25	68.6 ± 7.66	74.8 ± 2.40
can-mh	53.2 ± 5.8	13.8 ± 5.73	59.0 ± 5.0	57.6 ± 5.00
Table 7: Extended IPL ablation results.
Figure 7: IPL and MR+IQL with and without data augmentation across 5 seeds. We see that data augmentation makes a large difference, especially for MR+IQL in the hopper environment, while its effects are less for the robomimic Can datasets.
Hyper-parameter Sensitivity. We run hyper-parameter sensitivity results for the human-preference benchmark datasets in Fig. 8. The top row depicts the sensitivity for IPL to the value of  $\lambda$ . The bottom row depicts the sensitivity of MR to the number of timesteps the reward function is trained for.
Parameter Efficiency. For the parameter-efficient experiments only we use an efficient version of IPL based on AWAC [39] to additionally remove the need for learning value network. AWAC uses
Figure 8: Ablations on regularization strength  $\lambda$  for IPL (top row) and the number of reward steps for MR (bottom row). We see that IPL is relatively consistent across different values of  $\lambda$ . MR on the other hand, can vary greatly if the reward function under or over fits. In Walker2D Medium Replay andDrawer Open, 500, we see that it can easily under-fit. In Walker2D Medium Expert it easily over-fits.
a policy-evaluation, policy imporvement style approach. The outline of this variant is given below:
Algorithm 3: IPL Algorithm (AWAC Variant)
Input:  $\mathcal{D}_p$ $\mathcal{D}_o$ $\lambda ,\alpha$    
for  $i = 1,2,3,\ldots$  do Sample batches  $B_{p}\sim \mathcal{D}_{p},B_{o}\sim \mathcal{D}_{o}$  Estimate  $V$  as  $Q(s,\pi (s))$  Update  $Q$  ..  $\min_Q\mathbb{E}_{B_p}[\mathcal{L}_p(Q)] + \lambda \mathbb{E}_{B_p\cup B_o}[\mathcal{L}_r(Q)]$  Update  $\pi$  ..  $\max_{\pi}\mathbb{E}_{\mathcal{D}_p\cup \mathcal{D}_o}[e^{\beta (Q(s,a) - Q(s,\pi (s)))}\log \pi (a|s)]$
For this version of IPL, we use  $\lambda = 0.5$ . All other hyper-parameters remain the same as in Table 8 except the architectures. For the parameter-efficiency experiments only we use MLPs consisting of two dense layers with either dimension 64 or dimension 35. Running MR with a two-layer MLP of dimension 35 has almost exactly the same number of parameters as IPL-AWAC with two-layer MLPs of dimension 64. We include full results for the parameter-efficiency experiments in Table 8. We find that on Painter Open and Sweep Into, IPL outperforms both MR (64) and MR (35). In these environments, performance increases from MR (35) to MR (64) indicating that the expressiveness of the  $Q$ -function and policy are limiting performance. For the same budget, IPL is able to perform better. In Button Press, the simplest task, we find that MR (64) actually over-fits more than MR (35) and MR (64) ends up performing worse. In Plate Slide, all methods perform similarly independent of parameter count. We omit Assembly because of its low success rate at all data scales.
Preference Queries	500	1000	2000	4000
Button Press	MR (35)	73.9 ±8.9	86.8 ±8.2	89.9 ±14.4	99.0 ±1.0
MR (64)	54.2 ±16.1	42.6 ±33.0	67.1 ±14.9	43.4 ±7.4
IPL (64)	65.8 ±13.3	79.8 ±18.1	80.0 ±17.3	95.8 ±5.2
Drawer Open	MR (35)	13.4 ±13.9	12.6 ±21.9	15.5 ±20.1	18.4 ±25.6
MR (64)	13.4 ±19.0	57.1 ±31.2	54.5 ±31.7	78.8 ±12.2
IPL (64)	89.8 ±11.3	93.2 ±2.5	99.5 ±0.9	95.5 ±3.7
Sweep Into	MR (35)	35.1 ±8.9	42.4 ±9.9	45.9 ±9.6	35.9 ±4.1
MR (64)	31.1 ±6.4	55.8 ±5.9	49.6 ±10.3	56.4 ±10.3
IPL (64)	41.1 ±14.2	63.9 ±8.0	65.0 ±12.0	63.9 ±11.8
Plate Slide	MR (35)	55.2 ±6.1	51.1 ±4.4	53.0 ±2.0	48.9 ±3.3
MR (64)	46.6 ±21.9	50.8 ±0.6	47.0 ±2.5	48.5 ±4.6
IPL (64)	54.9 ±3.2	49.4 ±1.6	45.2 ±9.0	48.8 ±4.9
Table 8: Performance of different methods on the MetaWorld tasks under a limited parameter budget. MR (35) and IPL (64) have the same number of parameters. The Assembly task is omitted due to low success rate. On Button Press, fewer parameters appear to perform better as, due to the simplicity of the task, its easier for the bigger models to overfit. On Printer Open and Sweep Into, we see consistent gains from increasing the number of parameters in the network, and IPL performs best overall. On the Plate Slide task, all methods at different parameter scales perform similarly.