SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Franka Kitchen environment provides six available sub-tasks, where the goal is to perform four of them in a designated order: open the microwave oven, move the kettle to the top left burner, turn on the light switch, and open the slide cabinet. The other two tasks, which are turning on the burner switch and opening the hinge cabinet, are dummy tasks without any positive reward. Figure 9 shows an example of a goal-related task and a dummy task. The robot on the left is moving a kettle, which is a goal-related task. On the other hand, the robot on the right is turning on a burner, which is a dummy task.
For each offline dataset (kitchen-partial and kitchen-mixed), we sample 100 pairs of trajectory segments and label them under the following guidelines:
- An agent that solves more goal-related tasks (in order) is preferred.
- If the two agents solved the same number of goal-related tasks, the agent that solved fewer dummy tasks is preferred.
- If the two agents solved the same number of goal-related and dummy tasks, the agent that was working on another goal-related task is preferred.
For walker-medium-replay, [27] originally uses a preference dataset of size 500, but we subsample 100 samples from this dataset as we find that a smaller dataset suffices for PbRL algorithms to achieve reasonable performance.
Figure 9: Example of a goal-related task (left) and a dummy task (right) in Franka Kitchen. The task on the left is to move a kettle and the task on the right is to turn on a burner.
B Offline RL experiment details
For each method, we evaluate its performance by collecting ten trajectories with online interaction and measuring the average return. Following the standard protocol of the offline RL literature, we rescale the average return by  $100 \cdot \frac{R - R_{\text{random}}}{R_{\text{expert}} - R_{\text{random}}}$ , where  $R$  denotes the (empirical) average return of the policy, and  $R_{\text{random}}$  and  $R_{\text{expert}}$  are the expected returns of a random policy and an online expert policy, respectively. We provide the implementation details for each PbRL method below.
B.1 PreferenceTransformer (PT)
We use the official implementation from the authors to train the reward model<sup>4</sup>. Then, for PT+IQL, we run IQL with the default hyperparameters for each environment using the official code<sup>5</sup>. Similarly, for PT+CQL, we run CQL with the default hyperparameters for each environment using the official code<sup>6</sup>. Table 4 and Table 5 lists the detailed hyperparameter settings for IQL and CQL.
Table 4: Hyperparameter settings for IQL in PT+IQL.
Task name	Learning rate	# Layers	Hidden layer size	τ	β	Dropout
Gym	3·10-4	2	256	0.7	3.0	-
Adroit pen, Kitchen	3·10-4	2	256	0.7	0.5	0.1
Table 5: Hyperparameter settings for CQL in PT+CQL.
Task name	Learning rate	# Layers	Hidden layer size	α	τ	Min Q Ver.
Gym	1·10-4	3	256	10.0	-	3
Adroit pen	3·10-4	3	256	1.0	5.0	2
Kitchen	3·10-4	3	256	1.0	5.0	3
B.2 DPPO (Ours)
For the preference predictor, we start from a GPT-2 based transformer architecture from PT and apply some modifications. First, we removed the preference attention layer proposed by PT. Also, since PT requires per-step reward predictions, their transformer output is restricted to a scalar value for each
<sup>4</sup>https://github.com/csmile-1006/PreferenceTransformer
<sup>5</sup>https://github.com/ikostrikov/implicit_q_learning
<https://github.com/aviralkumar2907/CQL
timestep. Instead, our preference model outputs a vector embedding per step, which is aggregated and forwarded into a final MLP layer for the preference score prediction. We train the predictor for 10,000 update steps on the preference dataset. Table 6 lists more details of the training of the preference predictor. For policy optimization, we use a 2-layer MLP architecture and update the policy for 1e6 update steps, following the experimental protocol of IQL. Unlike CQL and IQL which adopt a stochastic policy model, we use a deterministic policy model as we find that both models have no difference in empirical performance. More detailed settings for the policy optimization procedure are specified in Table 7.
Table 6: Hyperparameter settings of the preference predictor training process in DPPO (Ours).
Task name	Learning rate	# Layers	embedding dim	ν	m
All tasks	1·10-4	1	256	1.0	20
Table 7: Hyperparameter settings of the policy optimization process in DPPO (Ours).
Task name	Learning rate	# Layers	layer size	λ	Dropout
Gym	3·10-4	2	256	0.5	0.25
Adroit pen, Kitchen	3·10-4	2	256	0.1	0.25
C Offline RL results with more baselines
Our offline RL experiments in the main paper focus on baselines that apply value-based RL algorithms after modeling the reward, following [27]. However, another interesting line of work in offline RL is the behavior cloning-based RL algorithms, which typically learn a policy conditional to a target return or goal [8, 14]. Here, we provide the full experiment results including two behavior cloning-based baselines: %BC, which performs behavior cloning on trajectories with top-10% return values from the offline dataset, and RvS [14].
Table 8: Normalized average return on D4RL Adroit pen and Kitchen tasks, averaged over 5 seeds.  $\pm$  denotes the standard deviation.
Task Name	PT+%BC	PT+RvS	PT+CQL	PT+IQL	DPPO (Ours)
pen-human	19.4 ± 6.7	-1.8 ± 0.5	31.6 ± 3.3	53.0 ± 31.7	76.3 ± 14.4
pen-cloned	37.4 ± 7.5	-2.2 ± 0.3	18.3 ± 10.6	42.9 ± 24.4	75.1 ± 7.7
Average	28.4	-2.0	25.0	48.0	75.7
kitchen-mixed	40.9 ± 9.0	27.5 ± 9.4	12.3 ± 7.7	48.0 ± 11.9	52.5 ± 3.1
kitchen-partial	53.4 ± 9.0	26.0 ± 4.9	14.1 ± 13.0	40.2 ± 12.3	49.4 ± 5.7
Average	47.2	26.8	13.2	44.1	51.0
The results in Table 8 show that DPPO outperforms the behavior cloning-based baselines on all tasks, with the performance gap being especially large on the challenging pen tasks.
D Additional ablation on the smoothness regularizer
In recognition of the fact that our novel smoothness regularizer can also be integrated into any reward-modeling baselines, we assess how its application to the baselines would influence the overall performance and present results in Table 9.
Table 9: Normalized average return on D4RL Adroit pen and Kitchen tasks, averaged over 5 seeds.  $\pm$  denotes the standard deviation.
Task Name	PT+CQL	PT+CQL+ν	PT+IQL	PT+IQL+ν	DPPO (Ours)
pen-human	31.6 ± 3.3	18.3 ± 17.2	53.0 ± 31.7	53.7 ± 42.3	76.3 ± 14.4
pen-cloned	18.3 ± 10.6	32.7 ± 11.2	42.9 ± 24.4	49.8 ± 32.2	75.1 ± 7.7
Average	25.0	25.5	48.0	51.8	75.7
kitchen-mixed	12.3 ± 7.7	12.0 ± 5.0	48.0 ± 11.9	49.4 ± 5.2	52.5 ± 3.1
kitchen-partial	14.1 ± 13.0	11.4 ± 11.2	40.2 ± 12.3	49.4 ± 5.2	49.4 ± 5.7
Average	13.2	11.7	44.1	49.4	51.0
Although our smoothness regularizer does enhance the baseline methods' performance, this improved performance still lags behind our proposed method. This outcome suggests that the process of directly optimizing the policy through the use of preference information plays a critical role in the overall performance.
E RLHF experiment details
The fine-tuning procedure of RLHF is similar to the PbRL process discussed in our paper. First, a preference predictor is trained to assign high scores to texts that are more preferred by human teachers. Then, the reward for fine-tuning a language model is defined by combining the preference predictor and a regularizer on policy (language model) shift. Concretely,  $r \coloneqq r_{\phi} - \xi r_{\mathrm{KL}}$ , where  $r_{\phi}$  is the output of the preference predictor and  $r_{\mathrm{KL}}$  is the KL divergence between the output distributions of the fine-tuned model and the original (pretrained) model. This reward function aims to guide the fine-tuned model to generate outputs that are more preferred by human teachers while not deviating far from the original model. After defining the reward function, PPO [48] is applied to optimize the language model to maximize the expected reward. Recent works show that this fine-tuning process can effectively align the language models with human preference, for example by generating more helpful and harmless sentences.
We can define a DPPO-style score metric that resembles the reward function from the original RLHF process as
$$
\begin{array}{l} S _ {\mathrm {R L H F}} (\theta ; \mathcal {D}, \phi) := \underset {\sigma^ {0}, \sigma^ {1} \sim \pi_ {\theta} (\cdot | \mathbf {x}), \mathbf {x} \sim \mathcal {D}} {\mathbb {E}} \left[ (1 - \widehat {y}) \cdot s \left(\pi_ {\theta}, \sigma^ {0}, \sigma^ {1}\right) + \widehat {y} \cdot s \left(\pi_ {\theta}, \sigma^ {1}, \sigma^ {0}\right) \right] \\ - \xi_ {\mathbf {x} \sim \mathcal {D}} \mathbb {E} \left[ D _ {\mathrm {K L}} \left(\pi_ {\theta} (\cdot | \mathbf {x}), \pi_ {\theta_ {0}} (\cdot | \mathbf {x})\right) \right] \\ \end{array}
$$
$$
\mathrm {s . t .} \widehat {y} = \mathbb {1} \left\{\widehat {P} \left[ \sigma^ {0} \succ \sigma^ {1}; \phi \right] > 0. 5 \right\},
$$
where  $\mathcal{D}$  is a prompt dataset,  $\sigma^i$  is the output sequence sampled from the fine-tuned language model  $\pi_{\theta}, \pi_{\theta_0}$  is the original pretrained language model, and  $D_{\mathrm{KL}}$  is the KL divergence. Here we remove the regularizer  $\lambda$  as the KL divergence term naturally prevents the policy from deviating far from the preferred sequences. Now we can directly fine-tune the policy by maximizing this score metric.
We evaluate our resulting RLHF algorithm by fine-tuning a pretrained OPT-1.3b language model which has 1.3 billion trainable parameters [60]. To train the preference predictor, we use the HH-RLHF dataset which contains 161K pairs of human preference data about helpfulness and harmlessness [4]. We choose OPT-350m with 350 million trainable parameters as the architecture of the predictor model. We evaluate our RLHF algorithm by comparing it with the conventional RLHF that uses the same preference predictor model and performs PPO. Our algorithm was implemented on DeepSpeed-Chat, an open code base for RLHF training<sup>7</sup>.
We consider several evaluation metrics to measure the performance of our RLHF algorithm. First, we calculate the expected reward, which measures the empirical mean of  $r_{\phi}$  from the outputs of the fine-tuned model. Second, we measure the KL divergence between the outputs of the fine-tuned model and the original model. These two metrics evaluate how well the two competing objectives
$^{7}$ https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat
(maximizing the alignment with human preference and staying close to the original model) are optimized. Additionally, we perform an actual human evaluation on whether the fine-tuned models are more preferred than the original model. In detail, for a randomly sampled prompt from  $\mathcal{D}$ , we generate two responses each from the fine-tuned model and the original model. Then, we ask three human workers to choose which response they prefer more in terms of helpfulness, and designate the final preferred response by majority voting<sup>8</sup>. We repeat this trial 300 times for each fine-tuning method and measure the win rate of the fine-tuned models.
The evaluation results in Table 3 show that our RLHF algorithm achieves a higher average reward with lower KL divergence from the original model when compared to the conventional RLHF using PPO. This shows that DPPO exhibits a better trade-off between human preference alignment and closeness to the original model. Also, DPPO achieves a higher win rate versus the original model on actual human evaluation, which shows the fine-tuned model from our algorithm is more aligned with human preference. These results demonstrate that DPPO is a promising approach for addressing preference-based learning problems in NLP.
F Discussion on the Antmaze environment in D4RL
We acknowledge that many offline RL studies highlight experiment results from the D4RL Antmaze task to demonstrate the efficacy of their methods in more challenging environments. Yet, it must be noted that the official implementation of Antmaze has a critical bug in its goal-setting procedure<sup>9</sup>. Concretely, the offline datasets of Antmaze assume a multi-goal setting, where a random goal location is chosen at the start of each episode, while the actual environment (unintentionally) uses a single fixed goal for every episode. Consequently, the optimal policies with regard to the offline dataset and the environment differ: According to the offline dataset, the optimal policy is to quickly sweep through all the goal candidates, while within the environment context, the optimal policy is to move directly to the solitary fixed goal.
To fix this, we can modify the Antmaze environment to choose a random goal location for each episode, thereby aligning with the provided offline dataset. Surprisingly, after this modification, we observe that current offline RL algorithms struggle to achieve meaningful performance:
Table 10: Performance of offline RL algorithms on the original and fixed Antmaze environment.
Task Name	Orig. Antmaze	Fixed Antmaze
%BC	CQL	IQL	%BC	CQL	IQL
medium-play-v2	48.7 ± 3.3	44.7 ± 27.8	62.0 ± 4.1	10.5 ± 2.9	9.8 ± 4.8	1.5 ± 1.5
medium-diverse-v2	35.5 ± 6.0	24.4 ± 32.3	66.2 ± 6.3	10.4 ± 3.2	3.1 ± 3.3	1.8 ± 0.8
large-play-v2	12.2 ± 7.5	11.7 ± 11.1	59.8 ± 8.1	11.1 ± 1.4	5.5 ± 3.1	14.5 ± 4.7
large-diverse-v2	10.8 ± 3.6	5.5 ± 9.0	55.2 ± 6.5	8.0 ± 0.8	4.8 ± 1.3	11.2 ± 1.6
Due to this stark difference in the evaluation performance, we decided not to include the Antmaze results in our main paper. We hope the offline RL community recognizes this problem in future studies involving Antmaze.
G Resources
All our offline RL experiments were run on a single RTX 3090 GPU with 10 CPU cores (Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz). For the RLHF experiments, we used an A100 GPU with 10 CPU cores (AMD EPYC 7402 24-Core Processor).
<sup>8</sup>We used Amazon MTurk https://www.mturk.com/ to conduct our human evaluation experiments.
<sup>9</sup>https://github.com/Farama-Foundation/D4RL/issues/142
H Broad impact
Our proposed method can be used to train RL agents that better align with human preferences. This method has the potential to significantly facilitate the deployment of AI services that cater to the specific requirements of the general public. However, it is essential to recognize that the learned RL agent can be biased toward certain values (e.g., races, ethnic groups, religions, ...) if the preference dataset itself exhibits biases toward those values. Therefore, practitioners should exercise caution and thoroughly examine their preference dataset to identify and address any potential biases.