SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minyoung Hwang $^{1}$ , Gunmin Lee $^{1}$ , Hogun Kee $^{1}$ , Chanwoo Kim $^{2}$ , Kyungjae Lee $^{2*}$ , Songhwai Oh $^{1*}$   
{minyoung.hwang, gunmin.lee, hogun.kee}@rllab.snu.ac.kr, junior97@cau.ac.kr, kyungjae.lee@ai.cau.ac.kr, songhwai@snu.ac.kr
<sup>1</sup>Electrical and Computer Engineering and ASRI Seoul National University Seoul, 08826, Korea
$^{2}$ Department of Artificial Intelligence Chung-Ang University Seoul 06974, Korea
Reinforcement learning from human feedback (RLHF) alleviates the problem of designing a task-specific reward function in reinforcement learning by learning it from human preference. However, existing RLHF models are considered inefficient as they produce only a single preference data from each human feedback. To tackle this problem, we propose a novel RLHF framework called SeqRank, that uses sequential preference ranking to enhance the feedback efficiency. Our method samples trajectories in a sequential manner by iteratively selecting a defender from the set of previously chosen trajectories  $\kappa$  and a challenger from the set of unchosen trajectories  $\mathcal{U} \setminus \kappa$ , where  $\mathcal{U}$  is the replay buffer. We propose two trajectory comparison methods with different defender sampling strategies: (1) sequential pairwise comparison that selects the most recent trajectory and (2) root pairwise comparison that selects the most preferred trajectory from  $\kappa$ . We construct a data structure and rank trajectories by preference to augment additional queries. The proposed method results in at least  $39.2\%$  higher average feedback efficiency than the baseline and also achieves a balance between feedback efficiency and data dependency. We examine the convergence of the empirical risk and the generalization bound of the reward model with Rademacher complexity. While both trajectory comparison methods outperform conventional pairwise comparison, root pairwise comparison improves the average reward in locomotion tasks and the average success rate in manipulation tasks by  $29.0\%$  and  $25.0\%$ , respectively. Project page: https://rllab-snu.github.io/projects/SeqRank
Designing a suitable reward function in reinforcement learning often requires task-specific prior knowledge [1] and sufficient time to design the reward function to capture the true task objective. Without this effort, the agent may be easily led to suboptimal policies. These limitations motivate the use of reinforcement learning from human feedback (RLHF), which can directly learn from human's preferences without the need for a hand-crafted reward function [1]. A conventional way to learn a reward function in RLHF is pairwise comparison: querying human preference between two different trajectories as described in Figure 1. Despite the significant success of RLHF, conventional pairwise comparison requires a human to remember at least two trajectories to determine a single preference.
Figure 1: Overview. Reinforcement learning from human feedback captures human preference against agent trajectories. While the agent learns the control policy by interacting with the environment, it also learns the reward function from human feedback, given as binary labels for trajectory pairs.
In this regard, it is essential to develop an efficient comparison method that can obtain multiple preference data from a single human feedback, thus reducing a human's labeling effort. In this paper, we define the concept of feedback efficiency as the ratio between the number of trajectory pairs over the number of feedbacks, where the feedback efficiency of conventional pairwise comparison is set at a standardized level. We aim to develop a novel RLHF framework with greater feedback efficiency than existing methods.
The key idea of our approach is to utilize the preference relationships of the previous trajectory pairs. Bringing the nature of transitivity in human preferences [2], given three trajectories,  $\sigma_{i},\sigma_{j}$ , and  $\sigma_{k}$ , a human with preference over  $\sigma_{i}$  to  $\sigma_{j}$  and  $\sigma_{j}$  to  $\sigma_{k}$  also prefers  $\sigma_{i}$  over  $\sigma_{k}$ . Hence, if we maintain the history of preference relationships, we can easily augment preference data without additional human feedback. In the proposed method, the comparison process maintains two sets of trajectories. One collects previously compared trajectories, and the other includes remaining trajectories that have not yet been compared. We iteratively select and compare a defender trajectory from the set of previously chosen trajectories and a challenger trajectory from the set of unchosen trajectories. In this paper, we propose two trajectory comparison methods: (1) sequential pairwise comparison that selects the most recently sampled trajectory and (2) root pairwise comparison that selects the previously most preferred trajectory as the defender. For both methods, the agent can augment additional preference data from the comparison results due to the transitivity.
We show that the proposed sequential preference ranking substantially improves the feedback efficiency and significantly outperforms conventional pairwise comparison. We prove that the proposed method converges faster for the empirical risk of the reward model than conventional pairwise comparison. One drawback of the proposed method is that it generates dependencies within the original and the augmented preference data, which are both used to train the reward model. Still, we prove the generalization bound of the reward model with Rademacher complexity converges with rates of  $\mathcal{O}(\sqrt{(\ln(T))^2 / T})$  and  $\mathcal{O}(\sqrt{\ln(T) / T})$  for sequential and root pairwise comparison, respectively, where  $T$  is the number of iterations. We demonstrate the effectiveness of the proposed method in both locomotion tasks from Deepmind Control Suite (DMControl) [3] and manipulation tasks from Meta-World [4]. Among the proposed two trajectory comparison methods, root pairwise comparison shows the highest performance in all tasks, which is improved by  $29.0\%$  and  $25.0\%$  against the pairwise baseline in locomotion and manipulation tasks, respectively. Additional experiments using real human feedback validate the effectiveness of the proposed method, showing significant improvements in true reward, sample efficiency, feedback efficiency, and user satisfaction. Furthermore, our method successfully outperforms the baseline in terms of success rates in controlling a real UR-5 robot for a block placing task, demonstrating its practical applicability in the real world.
- We propose a novel RLHF framework that utilizes sequential preference ranking to enhance human feedback efficiency. We prove the proposed sequential and root pairwise comparison substantially improve the average feedback efficiency by  $39.2\%$  and  $100\%$ , respectively. An increase in feedback efficiency speeds up the estimation of the reward function.
- We derive the convergence rates of the empirical risk and the generalization bound of the reward model with Rademacher complexity using the proposed sequential and root pairwise comparison. We address the trade-off between feedback efficiency and data dependency required for successful reward learning.
- We empirically show that prioritizing the feedback efficiency is significantly important by evaluating in simulation and real-world environments. Both sequential and root pairwise
comparison outperform conventional pairwise comparison on average. Root pairwise comparison shows the most substantial improvement against the baseline by  $29.0\%$  and  $25.0\%$  in DMControl locomotion and Meta-World manipulation tasks, respectively.
Reinforcement Learning from Human Feedback. A common approach in reinforcement learning from human feedback, i.e., preference-based reinforcement learning, is to learn the reward function from explicit feedback, such as ratings or comparisons provided by human experts. Christiano et al. [5] present the human-in-the-loop reinforcement learning framework for robotic tasks. Lee et al. (2021a) [6] propose a benchmark that introduces simulated human teachers with various irrationalities. Recent work [7-10] introduce efficient RLHF algorithms. Lee et al. (2021b) [7] propose a method that improves Christiano et al. [5] via unsupervised pre-training and off-policy RL with relabeling. Park et al. [8] suggest a method that uses semi-supervised learning and data augmentation. Liang et al. [9] show that exploration based on uncertainty in learned reward functions helps to improve the performance with fewer samples and feedbacks. Liu et al. [10] propose a data-efficient RLHF framework incorporating bi-level optimization for reward and policy learning.
However, most of the previous studies still achieve low feedback efficiency as they do not take into account the nature of ranking through trajectory comparisons. Several recent work [11-13] suggest querying a complete ranking among multiple  $(\geq 3)$  trajectories, but this assigns the responsibility of ranking trajectories to humans, not robots. Requesting a human to remember multiple trajectories at once and provide best-of-multiple feedback [11] is also challenging due to human uncertainty [14]. To solve this challenge, we propose sequential preference ranking, which allows the agent to automatically rank trajectories while a human only provides pairwise comparisons based on one's preference. The key idea is to perceive piecewise ranks instead of ranking the trajectories as a whole.
Generalization Bounds for Learning Under Graph-Dependence. While data augmentation via sequential preference ranking can improve the efficiency of reward learning, this can lead to dependencies within the training data. However, traditional learning theories are based on the assumption that data are identically and independently distributed (i.i.d.). Hence, to analyze the generalization bound of the proposed method, we investigate the generalization bounds for learning with data dependencies. A common approach to consider data dependency is to construct a graph that characterizes the dependency relationship within the data [15]. Janson et al. [16] establish that the probability of the sum of graph-dependent random variables deviating from its expected value is bounded based on the fractional chromatic number of the dependency graph, by extending Hoeffding's inequality. Usunier et al. [17] present a concentration inequality by extending fractional Rademacher complexity, and prove generalization bounds for binary classification over interdependent data. Zhang et al. (2019) [18] propose McDiarmid-type concentration inequalities for Lipschitz functions of graph-dependent random variables and show that concentration relies on the forest complexity of the dependency graph. In this paper, we derive the generalization bound of the reward model with Rademacher complexity in terms of the maximum degree of the dependency graph and the number of training data, which includes augmented preference queries and labels.
3 Learning the Reward Function from Human Preferences
Following prior work [5, 7, 10, 19], we use a deterministic human model that provides preference as a binary feedback over a pair of trajectory segments. We define the true preference label  $y$  for a pair of segments  $(\sigma_i, \sigma_j)$  as follows:
$$
y = \left\{ \begin{array}{l l} 1, & \text {i f} \sum_ {t = 1} ^ {H} r \left(s _ {t} ^ {i}, a _ {t} ^ {i}\right) > \sum_ {t = 1} ^ {H} r \left(s _ {t} ^ {j}, a _ {t} ^ {j}\right) \\ 0, & \text {o t h e r w i s e ,} \end{array} \right. \tag {1}
$$
where  $r$  is the true reward function of a state-action pair  $(s_t^i, a_t^i)$  in  $\sigma_i$  at timestep  $t$  and  $H$  is the length of a segment. The true reward of a trajectory segment  $\sigma = (s_1, a_1, \dots, s_H, a_H)$  can be written as  $r(\sigma) = \sum_{t} r(s_t, a_t)$ . Similar to prior work [13, 20-22], we implement our algorithm with a linearly parameterized preference model. Assume that there exist a nonlinear trajectory encoder  $\phi: \mathcal{T} \to \mathbb{R}^d$  and a parameter  $\theta^* \in \mathbb{R}^d$  that represents the true reward function as  $r(\sigma) := \theta^{*\top} \phi(\sigma)$ , where  $d$  is the dimension of  $\theta^*$  and  $\mathcal{T}$  is a set of trajectory
(a) Three trajectory comparison methods.
(b) Sequential vs Root Pairwise Comparison
Figure 2: Trajectory comparison methods. (a) All three methods query a human to provide  $N$  feedback. Pairwise comparison samples  $2N$  trajectories to obtain  $N$  feedback, while sequential or root pairwise comparison samples  $N + 1$  trajectories. Despite sampling fewer trajectories, sequential or root pairwise comparison shows higher feedback efficiency than pairwise comparison by adopting sequential preference ranking. (b) Gray nodes illustrate fixed-length trajectory segments sampled from the replay buffer. Suppose the reward values for segments  $\sigma_{1}, \dots, \sigma_{10}$  are 2, 5, 1, 8, 6, 4, 3, 7, 9, 10, respectively. Black lines indicate actual pairs that receive true preference labels from human feedback. The upper node for each black line represents the preferred trajectory. For root pairwise comparison, orange nodes describe non-leaf nodes in the tree. Using sequential or root pairwise comparison, the agent can obtain augmented labels for non-adjacent pairs illustrated with purple lines.
segments. We estimate the reward using a learnable reward function  $\hat{r}_{\theta}$ , parameterized by  $\theta$ . Then, the estimated reward of  $\sigma$  can be written as  $\hat{r}_{\theta}(\sigma) = \sum_{t} \hat{r}_{\theta}(s_t, a_t) = \theta^{\mathsf{T}} \phi(\sigma)$ . Preference prediction is also interpreted as a binary classification problem with a class probability  $P(y = 1 | (\sigma_i, \sigma_j); \theta^*) = P_{\theta^*}[\sigma_i \succ \sigma_j] = \exp(\sum_{t} r(s_t^i, a_t^i)) / \sum_{k \in \{i,j\}} \exp(\sum_{t} r(s_t^k, a_t^k))$  based on the Bradley-Terry model [23], where  $\sigma_i \succ \sigma_j$  denotes that the  $i^{th}$  trajectory segment is preferable to the  $j^{th}$  trajectory segment. For training the reward model, we use  $P(\hat{y} = 1 | (\sigma_i, \sigma_j); \theta) = P_{\theta}[\sigma_i \succ \sigma_j] = \exp(\sum_{t} \hat{r}_{\theta}(s_t^i, a_t^i)) / \sum_{k \in \{i,j\}} \exp(\sum_{t} \hat{r}_{\theta}(s_t^k, a_t^k))$  to estimate the preference label  $\hat{y}$  of  $(\sigma_i, \sigma_j)$ . If  $P_{\theta}[\sigma_i \succ \sigma_j]$  is greater than 0.5,  $\hat{y}$  is 1. Otherwise,  $\hat{y}$  is 0. Using cross-entropy loss, we define the preference loss function  $L_{pref}$  as follows:
$$
L _ {p r e f} = - \mathbb {E} _ {((\sigma_ {i}, \sigma_ {j}), y) \in \mathcal {D}} [ y \log P _ {\theta} [ \sigma_ {i} \succ \sigma_ {j} ] + (1 - y) \log P _ {\theta} [ \sigma_ {i} \prec \sigma_ {j} ] ], \tag {2}
$$
where  $\mathcal{D}$  is the set of sampled trajectory segment pairs and their corresponding true preference labels.
In traditional RLHF methods, a common approach to sampling a segment pair is to randomly sample each segment from the replay buffer, resulting in the selection of irrelevant pairs. This independent sampling approach does not take into account any inherent relationships between the non-adjacent segments, where two segments are considered adjacent if they are compared directly from a human. In contrast, the proposed method adopts a sequential sampling approach that enables indirect inference of the relationship between non-adjacent segments. The proposed method allows for more comprehensive understanding of the preferences among trajectory segments, capturing both direct and indirect relationships.
4 SeqRank
We present SeqRank, which samples trajectories and compares them in a sequential manner for efficient reinforcement learning from human feedback. We also extend our key component, sequential pairwise comparison, to root pairwise comparison, which not only compares trajectories sequentially but also constructs a tree structure to rank trajectories in a more efficient way. Figure 2 (a) illustrates the difference between three trajectory comparison methods including conventional pairwise comparison and the proposed sequential and root pairwise comparison. We demonstrate the feedback efficiency and convergence of the reward model for each method. Regarding the notations used in our paper, a summary of symbols and terms can be found in Table 1 in the supplementary material.
We first make the following assumptions:
(Uniformness) Every trajectory is sampled uniformly from the replay buffer.
(Completeness) For any two segments  $\sigma_{i}$  and  $\sigma_{j}$ , “ $\sigma_{i} \succ \sigma_{j}$  or  $\sigma_{i} \prec \sigma_{j}$ ” is true.
(Transitivity) For any three segments  $\sigma_{i},\sigma_{j}$ , and  $\sigma_{k}$ , if  $\sigma_{i} > \sigma_{j}$  and  $\sigma_{j} > \sigma_{k}$ , then  $\sigma_{i} > \sigma_{k}$ .
Assuming  $N$  feedback per reward update, the agent samples the  $(n + 1)^{th}$  defender and challenger from  $\mathcal{K}_n = \{\sigma_1,\dots,\sigma_n\}$  and  $\mathcal{U}\backslash \mathcal{K}_n$ , respectively, where  $\mathcal{U}$  is the replay buffer. The defender becomes  $\sigma_{n}$  for sequential pairwise comparison and  $\arg \max_{\sigma \in \mathcal{K}_n}\theta^{\mathrm{T}}\phi (\sigma)$  for root pairwise comparison. After the  $N^{th}$  query, the agent receives  $N$  feedback in total. Figure 2 (b) illustrates the preference relationships among trajectory segments as a graph. Especially for root pairwise comparison, the challenger becomes a new root of the tree if it is preferred over the current root segment as described in Figure 2 (a). Otherwise, the challenger becomes a new child of the current root. The full procedure of our algorithm is provided in the supplementary material. Suppose  $S_M$  is the total number of trajectory segment pairs obtained from  $M$  trajectory segments.  $M = 2N$  for conventional pairwise comparison, and  $M = N + 1$  for sequential or root pairwise comparison. In the following section, we first demonstrate the feedback efficiency in terms of  $M$  and substitute it with the function of  $N$ .
4.1 Feedback Efficiency of the Algorithm
From the uniformness assumption, suppose the agent compares  $M$  trajectory segments with corresponding rewards as a random ordering of  $1, 2, \ldots, M$ . Based on the completeness and transitivity assumptions, the agent can augment non-adjacent segment pairs as illustrated in Figure 2 (b). Considering that the number of augmented pairs differs by the order of sampling trajectories with different rewards,  $S_{M}$  becomes a random variable indicating the total number of pairs collected from  $M$  trajectory segments. In this section, we demonstrate the average feedback efficiency of our method.
Definition 1. We define feedback efficiency as the ratio between the total number of trajectory pairs over the number of feedbacks. The expected feedback efficiency for  $N$  human feedback is  $\mathbb{E}[S_{N + 1}] / N$  for sequential and root pairwise comparison.
Sequential Pairwise Comparison. Following Definition 1, we use Lemma 1 to examine the expected number of segment pairs collected from sequential pairwise comparison.
Theorem 1. For any integer  $M \geq 2$ , the expected number of trajectory segment pairs collected from sequential pairwise comparison  $a_{M}$  is approximately linear to  $M$  as  $|a_{M} / M - 1 / (e - 2)| = o(1)$ .
Lemma 1. For any integer  $M \geq 2$ ,  $a_{M}$  can be induced from the following recurrence relation.
$$
\begin{array}{l} a _ {M} = \frac {\text {(t o t a l n u m b e r o f p a i r s f r o m a l l p o s s i b l e o r d e r i n g s o f t r a j e c t o r i e s)}}{\text {(n u m b e r o f a l l p o s s i b l e o r d e r i n g s o f t r a j e c t o r i e s)}} \tag {3} \\ = 2 - 2 / M! + \sum_ {j = 2} ^ {M - 1} a _ {M - j + 1} (2 j / (j + 1)! \\ \end{array}
$$
By solving the recurrence relation stated in Lemma 1, we derive Theorem 1 that  $a_{M}$  is approximately equal to a linear equation  $\alpha (M - \gamma)$  with an error less than  $\epsilon$ , for any  $M$  such that  $M!$  is greater than  $2\gamma/\epsilon$ .  $\alpha = 1/(e-2) \cong 1.392$  and  $\gamma > 1$ , respectively. By substituting a sufficiently large value of  $M \geq 100$ ,  $\gamma$  is calculated as 1.324, and the expected feedback efficiency of sequential pairwise comparison becomes  $1.392(M - 1.324)/(M - 1)$ , which converges to 1.392 as  $M$  increases. Moreover, for any  $M \geq 4$ , the linear approximation error of  $a_{M}$  is bounded by 0.003. Detailed proofs of Lemma 1 and Theorem 1 are in the supplementary material.
Comparing the proposed sequential pairwise comparison with conventional pairwise comparison that requires  $2N$  trajectory segments for  $N$  human feedback, we observe that sequential pairwise comparison only requires  $N + 1$  trajectory segments to query a human for  $N$  feedback. Additionally, using the proposed sequential pairwise comparison, the expected number of augmented queries is given by  $a_{N + 1} - N \cong 0.392N - 0.451$ .
Extension to Root Pairwise Comparison. We further extend sequential pairwise comparison to root pairwise comparison by exploiting a specific defender sampling strategy. The key idea is to compare the current segment with the most preferred one from the previously sampled segments. As described in Figure 2 (b), root pairwise comparison augments more pairs than sequential pairwise comparison by constructing a tree structure. Using root pairwise comparison, the average feedback efficiency is increased to two. Theorem 2 demonstrates the average feedback efficiency of our algorithm using root pairwise comparison. The detailed proof of Theorem 2 is in the supplementary material.
Method	M	Best	Average	Worst
pN	η	pN	η	pN	η
Pairwise	2N	N	1	N	1	N	1
Sequential Pairwise	N+1	N(N+1)/2	(N+1)/2	1.392(N-0.324)	1.392	N	1
Root Pairwise	N+1	N(N+1)/2	(N+1)/2	2(N+1-∑n=1^{N+1} \frac{1}{n})	2	N	1
Table 1: Trajectory comparison methods. We compare three trajectory comparison methods: pairwise, sequential pairwise, and root pairwise.  $\eta$  denotes the feedback efficiency.
Theorem 2. In root pairwise comparison, for any integer  $M \geq 2$ , the expected number of pairs  $b_{M}$  can be denoted by
$$
b _ {M} = 2 \left(M - \sum_ {n = 1} ^ {M} (1 / n)\right). \tag {4}
$$
Considering that  $\sum_{n=1}^{M} \frac{1}{n} < 1 + \int_{1}^{M} \frac{1}{x} dx = 1 + \ln M$ ,  $(\sum_{n=1}^{M} \frac{1}{n}) / (M - 1)$  converges to 0 as  $M$  increases. Therefore, root pairwise comparison achieves the expected feedback efficiency as  $b_{M} / (M - 1) \cong 2$ . This implies that the proposed root pairwise comparison achieves higher feedback efficiency than sequential pairwise comparison on average.
Table 1 shows the comparison results of trajectory comparison methods when the number of human feedback is fixed to  $N$ .  $M$  and  $p_N$  denote the number of sampled trajectories at each reward update session and the expected number of trajectory pairs obtained from  $N$  human feedback, i.e.,  $\mathbb{E}[S_N]$ , respectively. For sequential and root pairwise comparison methods,  $p_N$  is  $a_{N+1}$  and  $b_{N+1}$ , respectively. The baseline method using conventional pairwise comparison samples  $2N$  trajectories, which are approximately twice as many of sequential or root pairwise comparison methods. As shown in Table 1, sequential pairwise comparison achieves an average feedback efficiency of 1.392. Also, root pairwise comparison achieves the highest average feedback efficiency of 2.
4.2 Convergence of the Reward Model
Notations. Suppose  $(\phi(\sigma_i) - \phi(\sigma_j))$  is the input and  $\sigma_i \succ \sigma_j$  is the true label of a binary classification using a logistic regression model and a cross-entropy loss. Then,  $P_{\theta}[\sigma_i \succ \sigma_j]$  can be interpreted as a logistic regression model as follows:
$$
P _ {\theta} [ \sigma_ {i} \succ \sigma_ {j} ] = \frac {1}{1 + \exp (\sum_ {t} \hat {r} _ {\theta} (s _ {t} ^ {j} , a _ {t} ^ {j}) - \hat {r} _ {\theta} (s _ {t} ^ {i} , a _ {t} ^ {i}))} = \frac {1}{1 + \exp (- \theta \tau (\phi (\sigma_ {i}) - \phi (\sigma_ {j})))}. \tag {5}
$$