SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For simplicity, we define the training data for reward update as  $\mathcal{D} = \{(x_k,y_k)\}$ , where  $x_{k}$  and  $y_{k}$  denote the features of two trajectories included in the  $k^{th}$  query and its corresponding preference label provided from a human, respectively. Note that if the  $k^{th}$  query is  $(\sigma_i,\sigma_j)$ ,  $x_{k} = \phi (\sigma_{i}) - \phi (\sigma_{j})$ . We define the true risk of the reward model as  $R(\theta)\coloneqq \mathbb{E}[\ell (\theta^{*\top}(x),\theta^{\top}(x))]$  and the empirical risk of the reward model as  $\hat{R} (\theta)\coloneqq L_{pref} = (1 / |\mathcal{D}|)\sum_{x_k\in \mathcal{D}}\ell (\theta^{*\top}(x_k),\theta^{\top}(x_k))$ , where the cross-entropy loss and its minimizer are denoted as  $\ell$  and  $\theta^{*}$ , respectively. Assume that  $\| \phi \| _2$ ,  $\| \theta^{*}\|_{2}$ , and  $\| \nabla \theta \| _2$  are bounded with constants  $S$ ,  $D$ , and  $Q$ , respectively. Since RLHF involves an iterative process of policy and reward update, we distinguish the overall iteration  $T$  that represents the global iterative process, and the local iterations of policy and reward update,  $W$  and  $U$ , respectively.  $S_{M,T}$  represents the cumulative sum of the number of training data of the reward model from iteration 1 to  $T$ , with  $M$  segments compared per iteration. We introduce the concept of dependency graph and its maximum degree  $\Delta_{M,t}$  at iteration  $t\in [T] = \{1,2,\dots,T\}$  for further analysis, following previous work [15]. Detailed proofs for the following theorems, lemmas, and corollaries are in the supplementary material.
High Probability Feedback Efficiency. We find a high probability bound for  $S_{M,T}$  in terms of the expected values of  $S_{M}$  as described in Lemma 2. This implies that for sufficiently large  $T$  and  $N$ , the feedback efficiency averaged across  $T$  iterations is at least equal to  $p_N / (2N)$  in the proposed method with high probability. For sequential and root pairwise comparison, the lower bounds become 0.696 and 1, respectively, while the feedback efficiency for pairwise comparison is fixed to 1.
Lemma 2. For a sufficiently large  $T > (\max_t \Delta_{M,t} + 1) (M - 1) \ln(1 / \delta) / (\beta^2 M)$  and  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ ,  $S_{M,T}$  is greater than  $\beta TM / 2$ , where  $\beta$  denotes the expected number of pairs obtained from  $M$  segments divided by  $M$ .
Convergence Rate of the Empirical Risk. From the completeness assumption, data in  $\mathcal{D}$  should be linearly separable. According to prior work [24], the empirical risk converges with a rate of  $\mathcal{O}(1 / U)$ . Let  $n_B$  be the batch size of reward update. Then, after the global iteration  $T$ ,  $U = \lfloor S_{M,T} / n_B \rfloor$ .
From Lemma 2, we prove that with probability at least  $1 - \delta$  and a sufficiently large  $T$ , the empirical risk of the reward model converges at a rate of  $\mathcal{O}(2n_B / (\beta TM))$ . Therefore, higher  $\beta$  results in faster convergence. While  $(\beta, M)$  for pairwise comparison are fixed as  $(0.5, 2N)$ , the values for sequential and root pairwise comparison are approximated to  $(1.392, N + 1)$  and  $(2, N + 1)$ , respectively. Based on our analysis, the reward model is likely to converge faster in the order of root pairwise, sequential pairwise, and pairwise comparison in terms of the global iteration  $T$ .
Convergence Rate of the Generalization Bound. Based on prior work [15, 25], we derive a generalization bound with Rademacher complexity for learning the reward model under graph-dependence in  $\mathcal{D}$ . Theorem 3 shows the generalization bound of the reward model. Detailed proof of the theorem and definitions of  $F$  and  $B$  are presented in the supplementary material.
Theorem 3. Assume that a comparison algorithm generates a train set  $\mathcal{D}$  of size  $S_{M,T}$  with dependency graph  $G$  after  $T$  iteration. Then, for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ ,
$$
\forall \theta \in \mathcal {H}, R (\theta) \leq \hat {R} (\theta) + \sqrt {\left(\max  _ {t} \Delta_ {M , t} + 1\right) / S _ {M , T}} \cdot (2 F D + 3 B \sqrt {\ln (4 / \delta) / 2}), \tag {6}
$$
where  $\mathcal{H}$  is the parameter space of  $\theta$ , and  $F$  and  $B$  are functions of  $M$  with rates at  $\mathcal{O}(M^2)$ .
Theorem 3 addresses a critical aspect of the proposed method by proving that even with dependent data, the empirical risk of the reward model converges to the true risk with probability at least  $1 - \delta$ . This high probability bound contributes to the practical analysis of general RLHF models with graph-dependent data such as rankings. Additionally, we consider the trade-off between feedback efficiency  $(\propto S_{M,T})$  and data dependency  $(\propto \Delta_{M,t})$ . As shown in (6), lower  $\Delta_{M,t}$  and larger  $S_{M,T}$  lead to a tighter generalization bound of the reward model. However, data dependency increases in our settings as we augment more queries and enhance feedback efficiency. Hence, we demonstrate  $\Delta_{M,t}$  for sequential and root pairwise comparison in Lemma 3 and 4, respectively.
Lemma 3. Let  $G$  be a dependency graph generated by sequential pairwise comparison with  $M$  trajectory segments. Then, for any  $\delta \in (0,1)$ , the maximum degree of  $G$  is bounded as follows, with probability at least  $1 - \delta$ ,
$$
\max  _ {t \in [ T ]} \Delta_ {M, t} \leq 2 + \ln (1 + M / (\ln (1 / (1 - \delta)))) \tag {7}
$$
Lemma 4. Let  $G$  be a dependency graph generated by root pairwise comparison with  $M$  trajectory segments. Then, for any  $\delta \in (0,1)$ , the maximum degree of  $G$  is bounded as follows, with probability at least  $1 - \delta$ ,
$$
\max  _ {t \in [ T ]} \Delta_ {M, t} \leq M (2 - \delta) - 3. \tag {8}
$$
Corollary 3.1. For a fixed  $M \geq 2$ , the generalization bounds of the reward model for pairwise, sequential pairwise, and root pairwise comparison converge at rates of  $\mathcal{O}(\sqrt{\ln(T) / T})$ ,  $\mathcal{O}(\sqrt{(\ln(T))^2 / T})$ , and  $\mathcal{O}(\sqrt{\ln(T) / T})$ , respectively, with probability at least  $1 - 1 / T$ .
By applying Lemma 2, 3, and 4 to Theorem 3, we derive Corollary 3.1, which compares the generalization bounds of three methods. While generalization bounds of all methods are guaranteed to converge as  $T$  increases, root pairwise comparison demonstrates a faster convergence rate than sequential pairwise comparison, and has the same convergence rate as pairwise comparison.
Overall analysis of convergence. To sum up, root pairwise comparison theoretically surpasses the baseline as it proves better and equal convergence rates of both metrics, while requiring a half of samples than pairwise comparison. Sequential pairwise comparison achieves faster convergence of the empirical risk but slower convergence of the generalization bound compared to pairwise comparison. From the perspective of reward model convergence, prioritizing the feedback efficiency to speed up the convergence of the empirical risk is more critical than considering the generalization error in worst-case scenarios. In Section 5, we empirically show the superiority of root pairwise comparison over sequential and pairwise comparison.
We evaluate our method on robotic locomotion tasks from DeepMind Control Suite (DMControl) [3, 26] and robotic manipulation tasks from Meta-World [4]. We also demonstrate the proposed
Task	# feedback	Oracle	Pairwise	Sequential Pairwise	Root Pairwise
Walker Walk	0.4K	957.3 ± 2.3	862.5 ± 85.3	883.7 ± 71.9	907.5 ± 66.2
Cheetah Run	0.2K	886.9 ± 43.2	690.2 ± 93.2	715.8 ± 117.4	739.4 ± 100.8
Quadruped Walk	1K	843.3 ± 247.3	353.5 ± 183.9	479.5 ± 234.7	728.8 ± 274.1
Humanoid Walk	40K	272.5 ± 117.0	114.3 ± 80.5	141.3 ± 34.4	163.8 ± 71.6
Hopper Hop	4K	273.5 ± 47.0	26.2 ± 37.0	37.3 ± 59.8	100.1 ± 70.7
Table 2: Rewards after convergence in DMControl locomotion tasks. Two elements in each cell denote the average value and standard deviation of rewards across runs with 10 random seeds.
Task	# feedback	Oracle	Pairwise	Sequential Pairwise	Root Pairwise
Button Press	10K	99.3 ± 0.9	95.6 ± 7.6	97.0 ± 4.3	97.6 ± 5.6
Door Open	10K	100.0 ± 0.0	77.9 ± 41.3	77.4 ± 40.8	97.8 ± 5.0
Drawer Open	20K	99.9 ± 0.3	65.3 ± 41.6	72.0 ± 45.4	89.9 ± 31.6
Sweep Into	10K	88.8 ± 29.9	68.4 ± 35.2	55.7 ± 48.1	88.0 ± 31.0
Window Open	1K	99.9 ± 0.3	55.9 ± 45.1	66.1 ± 36.5	70.7 ± 40.8
Hammer	10K	91.5 ± 26.5	31.0 ± 24.7	30.8 ± 35.4	48.8 ± 41.2
Table 3: Success rates in Meta-World manipulation tasks. Two elements in each cell denote the average value and standard deviation of success rates across runs with 10 random seeds.
method in the real-world using a UR-5 robot for a block placing task. Experiment details and additional analyses with rendered results are provided in the supplementary material.
Tasks. Locomotion tasks from DMControl consist of walker walk, cheetah run, quadruped walk, humanoid walk, and hopper hop. Manipulation tasks from Meta-World consist of button press, door open, drawer open, sweep into, window open, and hammer. For each task, we train with 10 different seeds used in prior work [6, 10] and measure the average performance with a standard deviation. Evaluation metrics are chosen as the ground-truth return per episode for DMControl and the success rate for Meta-World. Additionally, we conduct experiments with real human feedback for 30 participants using policy and reward models pre-trained in the simulation environment. We also demonstrate a block placing task using a UR-5 robot in the real-world, by pre-training the policy in the Mujoco simulator [27] and fine-tuning in the real-world. Detailed experiment settings are provided in the supplementary material.
Baselines and Implementation Details. We use three baselines for all tasks: SAC [28], Meta-Reward-Net (MRN) [10], and PEBBLE [7]. Agents learned via SAC with the true reward functions are considered as the upper bound of all tasks. We use MRN and PEBBLE as the baseline methods using the traditional trajectory comparison method, pairwise comparison. MRN utilizes bi-level optimization for both reward and policy learning and achieves state-of-the-art among previous methods. We apply our sequential preference ranking algorithm on both MRN and PEBBLE to show the effectiveness of our algorithm. Results with MRN baseline are described in this section and results with PEBBLE baseline are provided in the supplementary material. We use unsupervised pre-training [7] for 9,000 steps for all experiments. The trajectory encoder is implemented using a two-layer feed-forward neural network, where the input dimension is the combined size of the state and action spaces, and the output dimension is set to 256. The linear reward model then takes the encoded feature and passes it through a single fully connected neural network. For each task, hyperparameters and implementation details are described in the supplementary material.
5.2 Simulation Experiments
Locomotion Tasks from DMControl. Results in Table 2 demonstrate the superiority of the proposed root pairwise comparison over the baseline pairwise comparison and sequential pairwise comparison in all locomotion tasks. The reward obtained after convergence is higher by an average of  $29.0\%$  compared to conventional pairwise comparison used in MRN [10]. Sequential pairwise comparison, on the other hand, does not exceed root pairwise comparison, but it improves performance against pairwise comparison by  $10.3\%$  on average. Based on the standard deviation of rewards after convergence across 10 different runs, root pairwise comparison shows higher stability than sequential
(a) Quadruped Walk (feedback=1K)
(b) Hopper Hop (feedback=4K)
(c)Drawer Open (feedback=20K)
(d) Window open (feedback=1K)
Figure 3: Comparison results in two locomotion tasks from DMControl and two manipulation tasks from Meta-World. All methods are implemented based on MRN. Each subfigure describes the comparison results among three trajectory comparison methods: pairwise (green), sequential pairwise (orange), and root pairwise (blue). Pink lines describe the oracle performance using SAC with the true reward.
(a) Average Rewards
(b) User Satisfaction Score
Figure 5: Training the Cheetah Run Task with Real Human Feedback. (a) The bar plot describes the ratio between the average rewards after training the policy with real human feedback. (b) The histogram describes distributions of user satisfaction scores for three trajectory comparison methods.
pairwise comparison. Reward graphs in Figure 3 (a) and (b) show that root pairwise comparison enables faster convergence of the policy learning.
Additionally, we assess the converged reward for various feedback maximums (0.1K, 1K, 2.5K, 4K) in the quadruped walk task, as illustrated in Figure 4. The graph reveals that both sequential and root pairwise comparison methods converge to higher average reward values and lower variance as the number of feedbacks increases. This implies that with a sufficient number of feedbacks, the amount of augmented preference data affects performance more than data dependency does. By taking advantage of learning a dense reward [5, 29, 30], root pairwise comparison outperforms the oracle baseline with a sufficient number of feedbacks.
Figure 4: Average reward per number of feedbacks after convergence.
Manipulation Tasks from Meta-World. Results in Table 3 show the comparison results with baseline in manipulation tasks. The proposed method using root pairwise comparison outperforms the methods using pairwise comparison and sequential pairwise comparison in all tasks, improving the overall success rate by  $25.0\%$  compared to the pairwise comparison baseline. Reward graphs in Figure 3 (c) and (d) show that root pairwise comparison enables faster convergence of the policy. Unlike the locomotion tasks, sequential pairwise comparison only improves the success rate by  $1.2\%$  over pairwise comparison. This may be due to the generalization error using sequential pairwise comparison. Comparing the standard deviation of success rates among 10 different runs, root pairwise comparison achieves the lowest standard deviation. This implies that root pairwise comparison is the most stable algorithm for manipulation tasks among three trajectory comparison methods.
Experiments with Real Human Feedback. To compare the performance and the user stress levels associated with three different methods: pairwise, sequential pairwise, and root pairwise comparison, we conduct experiments involving 35 real human participants. 5 participants trained the policy and the reward models from scratch with 200 human feedbacks. Other participants fine-tuned the models with 20 human feedbacks, where the models were pre-trained with simulated feedbacks. Figure 5 (a) shows that the proposed methods outperform the baseline in both training from scratch and fine-tuning experiments. Especially for training the models from scratch, root and sequential
Method	# feedback	Success Rate↑	True Reward↑	Episode Length↓	Reward Accuracy↑
Pairwise	0.8K	34.4	17.1	29.6	75.8
Sequential Pairwise	0.8K	41.0	20.1	28.4	80.9
Root Pairwise	0.8K	50.5	25.9	27.9	84.9
Table 4: Average performance in the real robot manipulation task. For each method, the agent is fine-tuned in the real world for 3,000 steps.
Figure 6: Block placing using a UR-5 robot.
pairwise comparison outperform conventional pairwise comparison by  $47.7\%$  and  $38.6\%$ , respectively. Quantitative results are provided and analyzed in the supplementary material. Additionally, as a user study, we asked the participants to rate their user satisfaction on a scale from one to five, with higher scores indicating greater satisfaction and lower stress levels. Figure 5 (b) illustrates that the average user satisfaction scores resulted in 2.20, 3.00, and 3.93 for pairwise, sequential, and root pairwise comparison, respectively. We perform paired-sample t-test and the differences between the scores are statistically significant for all cases, with the average p-value 0.035. Specifically, users responded that it is convenient to compare with one's most preferred trajectory. These results imply that root pairwise comparison is the least burdensome for real human users while achieving the highest performance.
5.3 Real Robot Experiments
Block Placing Task. To demonstrate our method in real-world environments, we conduct a block placing task using a real UR-5 robot. As illustrated in Figure 6, the goal of this task is to place a block at a given target position starting from a randomly chosen initial position. For faster training and stable movements of the agent, we train the policy with a deep Q-learning network (DQN) and discrete action space following prior work [31]. In this setting, an action is given by the moving direction  $\psi$ , where  $\psi$  is the heading direction of the end effector discretized into eight angle bins  $\{0, \frac{\pi}{4}, \frac{2\pi}{4}, \dots, \frac{7\pi}{4}\}$ . Once  $\psi$  is determined, the agent changes its pose and moves the end effector for a fixed distance in the angle of  $\psi$ . The policy is pre-trained in the simulator for 20,000 steps and fine-tuned in the real-world for 3,000 steps. The reward model is initialized before fine-tuning. Results in Table 4 show the comparison results with baseline after fine-tuning in the real-world. Using root pairwise comparison, the agent exhibits the highest success rate and the reward model achieves the fastest convergence with the highest accuracy among three trajectory comparison methods.
We propose a novel RLHF framework called SeqRank that utilizes sequential preference ranking, resulting in a substantial improvement in the feedback efficiency by at least  $39.2\%$ . Both sequential and root pairwise comparison enables faster estimation of the reward function. We also prove the convergence rates of the empirical risk and the generalization bound of the reward model for each method. Theoretical and experimental results show that both methods outperform conventional pairwise comparison, with root pairwise comparison showing the most substantial improvement against the baseline. The proposed method and theoretical analyses can be applied to other domains and serve as a practical solution to enhance the feedback efficiency and performance of RLHF. Potential negative societal impacts include amplifying existing biases present in human preferences. We hope our method encourages RLHF to be used for social good without violating ethics.
Limitations. As we experimentally show that the proposed root pairwise comparison achieves the highest reward accuracy among three trajectory comparison methods, we expect to find a tighter generalization bound of the reward model. As the generalization bound of logistic regression with dependent data is not fully studied yet, it will be our future work.
This work was in part supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01190, [SW Star Lab] Robot Learning: Efficient, Safe, and Socially-Acceptable Machine Learning,  $66\%$ , No. 2022-0-00331, Development of emotion recognition/generation-based interacting edge device technology for mental health care,  $17\%$ , and No. 2021-0-01341, AI Graduate School Program, CAU,  $17\%$ ).
[19] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems (NeurIPS), 2018. 3
[20] Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for preference-based reinforcement learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2020. 3
[21] Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2020.
[22] Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: reinforcement learning with trajectory preferences. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2021. 3
[23] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952. 4
[24] Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300, 2018. 7
[25] Massih-Reza Amini and Nicolas Usunier. Learning with partially labeled and interdependent data. Springer, 2015. 7
[26] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. 7
[27] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012. 8
[28] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning (ICML), 2018. 8
[29] Aleksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards to automate reinforcement learning. arXiv preprint arXiv:1905.07628, 2019. 9
[30] Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In International Conference on Machine Learning, pages 11436-11446. PMLR, 2020. 9
[31] Hogun Kee, Minjae Kang, Dohyeong Kim, Jaegoo Choy, and Songhwai Oh. Sdf-based graph convolutional q-networks for rearrangement of multiple objects. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023. 10