Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback
Authors: Minyoung Hwang
Gunmin Lee
Hogun Kee
Chan Woo Kim
Kyungjae Lee
Songhwai Oh
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning; Reinforcement Learning from Human Feedback; Preference-based Reinforcement Learning; Human-Robot Interaction
TL;DR: We propose a novel RLHF framework called SeqRank, which significantly improves the human feedback efficiency and performance of RLHF.

Abstract: Reinforcement learning from human feedback (RLHF) alleviates the problem of designing a task-specific reward function in reinforcement learning by learning it from human preference. However, existing RLHF models are considered inefficient as they produce only a single preference data from each human feedback. To tackle this problem, we propose a novel RLHF framework called SeqRank, that uses sequential preference ranking to enhance the feedback efficiency. Our method samples trajectories in a sequential manner by iteratively selecting a defender from the set of previously chosen trajectories $\mathcal{K}$ and a challenger from the set of unchosen trajectories $\mathcal{U}\setminus\mathcal{K}$, where $\mathcal{U}$ is the replay buffer. We propose two trajectory comparison methods with different defender sampling strategies: (1) sequential pairwise comparison that selects the most recent trajectory and (2) root pairwise comparison that selects the most preferred trajectory from $\mathcal{K}$. We construct a data structure and rank trajectories by preference to augment additional queries. The proposed method results in at least 39.2% higher average feedback efficiency than the baseline and also achieves a balance between feedback efficiency and data dependency. We examine the convergence of the empirical risk and the generalization bound of the reward model with Rademacher complexity. While both trajectory comparison methods outperform conventional pairwise comparison, root pairwise comparison improves the average reward in locomotion tasks and the average success rate in manipulation tasks by 29.0% and 25.0%, respectively. The source code and the videos are provided in the supplementary material.

Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback
Minyoung Hwang', Gunmin Lee', Hogun Kee', Chanwoo Kim", Kyungjae Lee²*. Songhwai Oh" (minyoung, hwang, gunmin. lee, hogun kee @rllab. smu ac.kr, jenior97@cau.co kr, kyungjae leeßai. cau.ac ac. kr, songhwai@enu ac.
Electrical and Computer Engineering and ASRI Seoul National University
2 Department of Artificial Intelligence Chung-Ang University Scoul 06974, Korea
Scoul, 08826, Korea
Abstract
Reinforcement learning from human feedback (RLHF) alleviates the problem of designing a task-specific reward function in reinforcement learning by learning it from human preference. However, existing RLHF models are considered inefficient as they produce only single preference data from each human feedback. To tackle this problem, propose novel RLHF framework called SeqRank, that uses sequential preference ranking to enhance the feedback efficiency. Our method samples trajectories in a sequential manner by iteratively selecting defender from the set of previously chosen trajectories X. and a challenger from the set of unchosen trajectories 2/ \ K., where 24 the replay buffer. We propose two trajectory comparison methods with different defender sampling strategies: (1) sequential pairwise comparison that selects the most recent trajectory and (2) root pairwise comparison that selects the most preferred trajectory from K.. We construct a data structure and rank trajectories by preference augment additional queries. The proposed method results in at least 39.2% higher average feedback efficiency than the baseline and also achieves a balance between feedback efficiency and data dependency. We examine the convergence of the empirical risk and the generalization bound of the reward model with Rademacher complexity. While both trajectory comparison methods outperform conventional pairwise comparison, root pairwise comparison improves the average reward locomotion tasks and the average success rate in manipulation tasks by 29.0% and 25.0%, respectively. Project page: https://rllab-st http://llab-su..ithuuuuu httplllbssnngguuubboooooeeeeeeee github.iooprojjett//eeegaaaa
Introduction
Designing a suitable reward function in reinforcement learning often requires task-specific prior knowledge [1] and sufficient time to design the reward function capture the true task objective. Without this effort, the agent may be easily led to suboptimal policies. These limitations motivate the use of reinforcement learning from human feedback (RLHF), which can directly learn from human's preferences without the need for a hand-crafted reward function [1]. A conventional way to learn reward function RLHF pairwise comparison: querying human preference between different trajectories as described in Figure 1. Despite the significant success of RLHF, conventional pairwise comparison requires a human remember least two trajectories to determine single preference.
"Corresponding Authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Current O Observation
r(01) =1
preference
Learn Policy
Learn Reward
01 0z
y=
T(O2) =0-
Human
Environment
Agent
Action
Figure 1: Overview. Reinforcement learning from human feedback captures human preference against agent trajectories. While the agent learns the control policy by interacting with the environment, it also learns the reward function from human feedback, given as binary labels for trajectory pairs.
In this regard, it is essential to develop an efficient comparison method that can obtain multiple preference data from single human feedback, thus reducing human's labeling effort. In this paper, we define the concept of feedback efficiency as the ratio between the number of trajectory pairs over the number of feedbacks, where the feedback efficiency of conventional pairwise comparison is set at a standardized level. We aim to develop novel RLHF framework with greater feedback efficiency than existing methods.
The key idea of our approach is to utilize the preference relationships of the previous trajectory pairs. Bringing the nature of transitivity in human preferences [2], given three trajectories, oa jj and ox human with preference over a; to oj and o, to ag also prefers o, over dg- Hence, if we maintain the easily augment preference data without additional human
history of preference relationships, feedback. In the proposed method, the comparison process maintains two sets of trajectories. One collects previously compared trajectories, and the other includes remaining trajectories that have not yet been compared. We iteratively select and compare a defender trajectory from the set of previously chosen trajectories and a challenger trajectory from the set of unchosen trajectories. In this paper, we propose two trajectory comparison methods: (1) sequential pairwise comparison that selects the most recently sampled trajectory and (2) root pairwise comparison that selects the previously most preferred trajectory as the defender. For both methods, the agent can augment additional preference data from the comparison results due to the transitivity.
We show that the proposed sequential preference ranking substantially improves the feedback effi- ciency and significantly outperforms conventional pairwise comparison. We prove that the proposed method converges faster for the empirical risk of the reward model than conventional pairwise com- parison. One drawback of the proposed method that it generates dependencies within the original and the augmented preference data, which are both used to train the reward model. Still, we prove the generalization bound of the reward model with Rademacher complexity converges with rates of (( ((v((I)()))/ and O ((VIn(T))/T for sequential and root pairwise comparison, respectively, where 7' is the number of iterations. We demonstrate the effectiveness of the proposed method in both locomotion tasks from Deepmind Control Suite (DMControl) [3] and manipulation tasks from Meta-World [4]. Among the proposed two trajectory comparison methods, root pairwise comparison shows the highest performance in all tasks, which is improved by 29.0% and 25.0% against the pair- wise baseline in locomotion and manipulation tasks, respectively. Additional experiments using real human feedback validate the effectiveness of the proposed method, showing significant improvements in true reward, sample efficiency, feedback efficiency, and user satisfaction. Furthermore, our method successfully outperforms the baseline in terms of success rates in controlling a real UR-5 robot for a block placing task, demonstrating its practical applicability in the real world.
The main contributions of this paper are as follows:
We propose novel RLHF framework that utilizes sequential preference ranking to enhance human feedback efficiency. We prove the proposed sequential and root pairwise comparison substantially improve the average feedback efficiency by 39.2% and 100%, respectively. An increase in feedback efficiency speeds up the estimation of the reward function.
We derive the convergence rates of the empirical risk and the generalization bound of the reward model with Rademacher complexity using the proposed sequential and root pairwise comparison. We address the trade-off between feedback efficiency and data dependency required for successful reward learning.
We empirically show that prioritizing the feedback efficiency is significantly important by evaluating in simulation and real-world environments. Both sequential and root pairwise

---

comparison outperform conventional pairwise comparison on average. Root pairwise comparison shows the most substantial improvement against the baseline by 29.0% and 25.0% in DMControl locomotion and Meta- Seta-World manipulation tasks, respectively.
Related Work
Reinforcement Learning from Human Feedback. A common approach in reinforcement learning from human feedback, i.e., preference-basee reinforcement learning, is to learn the reward function from explicit feedback, such as ratings or comparisons provided by human experts. Christiano el al. [5] present the human-in-the-loop reinforcement learning framework for robotic tasks. Lee ef al. (2021a) [6] propose benchmark that introduces simulated human teachers with various irrationalities. Recent work [7-10] introduce efficient RLHF algorithms. Lee al. (2021b) [7] propose method that improves Christiano et al. [5] via unsupervised pre- pre-training and off- RL with relabeling. Park al. [8] suggest a method that uses semi-supervised learning and data augmentation. Liang et al. [9] show that exploration based on uncertainty in learned reward functions helps to improve the performance with fewer samples and feedbacks. Liu et al. [10] propose data-efficient RLHF framework incorporating bi-level optimization for reward and policy learning.
However, most of the previous studies still achieve low feedback efficiency they do not take into account the nature of ranking through trajectory comparisons. Several recent work [11-13] suggest querying complete ranking among multiple (2 3) trajectories, but this assigns the responsibility of ranking trajectories to humans, not robots. Requesting a human to remember multiple trajectories once and provide best-of-multiple feedback [11] is also challenging due human uncertainty [14]. To solve this challenge, we propose sequential preference ranking, which allows the agent to automatically rank trajectories while a human only provides pairwise comparisons based on one's preference. The key idea is to perceive piecewise ranks instead of ranking the trajectories as a whole.
Generalization Bounds for Learning Under Graph-Dependencee via sequential preference ranking can improve the efficiency of reward learning. this can lead to dependencies within the training data. However, traditional learning theories are based the assumption that data are identically and independently distributed (i.i.d)). Hence, to analyze the generalization bound of the proposed method, investigate the generalization bounds for learning with data dependencies. A common approach to consider data dependency is to construct a graph that characterizes the dependency relationship within the data [15]. Janson et al. [16] establish that the probability of the sum of graph-dependent random variables deviating from its expected value is bounded based on the fractional chromatic number of the dependency graph, by extending Hoeffding's inequality. Usunier et al. [17] present concentration inequality by extending fractional Rademacher complexity, and prove generalization bounds for binary classification over interdependent data. Zhang et al. (2019) [18] propose McDiarmid -type concentration inequalities for Lipschitz functions of graph-dependent random variables and show that concentration relies on the forest complexity of the dependency graph. In this paper, we derive the generalization bound of the reward model with Rademacher complexity in terms of the maximum degree of the dependency graph and the number of training data, which includes augmented preference queries and labels.
While data augmentation
Learning the Reward Function from Human Preferences
Following prior work [5, 7, 10, 19), we use a deterministic human model that provides preference as binary feedhack over pair of trajectory segments. We define the preference label y for a pair of segments (a,, σ,) as follows:
1, if EE r(s,, ot) > ΣH r()),
0. otherwise,
where the true reward function of state-action pair (si, at) in os at timestep and H is the length of segment. The true reward of trajectory segment o = (S1, (1..,,1..8111 aH) can be written as r(a) = Σ, r(st, at). Similar to prior work [13, 20-22], we implement our al- gorithm with a linearly parameterized preference model. Assume that there exist a nonlinear trajectory encoder Φ T + IRd and a parameter 0" E Rd that represents the true reward function as r(a) := 8 ((T). where d is the dimension of 0" and T is set of trajectory
3

---

Replay Butfer
Sample traectorree
Sample N trajectories
Sample trajectories
Sequential Pairwise Comparison
Compane
the prev previously mostiretemed curentrajectory
On
25
Constructed Tree
Pairwise
Sequential P Pairwise
Root Painwise Comparison
Root airwise Comparison
Comparison
Comparison
(a) Three trajectory comparison methods.
(b) Sequential vs Root Pairwise Comparison
Figure 2: Trajectory comparison methods. (a) All three methods query human provide N feedback. Pairwise comparison samples trajectories obtain feedback, while sequential or rool pairwise comparison samples N''. + trajectories. Despite sampling fewer trajectories, sequential or root pairwise comparison shows higher feedback efficiency than pairwise comparison by adopting sequential preference ranking, (b) Gray nodes illustrate fixed- leee-lennthh trajectory segments sampled from the replay buffer. Suppose the reward values for segments di,... 0700 2,5, 1, 4, 3, 7, 9, 10, respectively. Black lines indicate actual pairs that receive preference lahels from human feedback. The upper node for each black line represents the preferred trajectory. For root pairwise comparison, orange nodes @ descrihe non-leaf nodes in the tree. Using sequential or root pairwise comparison, the agent can obtain augmented labels for non-adjacent pairs illustrated with purple lines.
segments. We estimate the reward using learnable reward function f.. parameterized by A. Then, the estimated reward of a can be written as fa(o) = E, fo(st,a) = @Te(o). Pref- erence prediction is also interpreted Ply = 1|(a,, oy)o,,))):0") = Pa- > az] = exp(E, r(st, at)) of))/E exp(E. r(st at)) based on the Bradley-Terry model [23], where o; > o, denotes that the 7th trajectory segment is prefer- able to the jth trajectory segment. For training the reward model, we use P(ğ 1|(a,, oy): Polo; > o₁l exp(E, (a(s), /f))//xe((.))) exp(2. fo(st, af)) estimate the preference label of (o,, v,). If Palo, > а,] is greater than 0.5. 1 is 1. Otherwise, I is Using cross-entropy loss, we define the preference loss function Lpref follows:
binary classification problem with class probability
Lyres
(y log Polo, > o,] (1- y) log Polo; < oz]]]-
(2)
where D the set sampled trajectory segment pairs and their corresponding true preference labels.
In traditional RLHF methods, a common approach to sampling segment pair is to randomly sample each segment from the replay buffer, resulting the selection of irrelevant pairs. This independent sampling approach does not take into account any inherent relationships between the non-adjacent segments, where two segments are considered adjacent if they are compared directly from a human. In contrast, the proposed method adopts a sequential sampling approach that enables indirect inference of the relationship between non-adjacent segments. The proposed method allows for more comprehensive understanding of the preferences among trajectory segments, capturing both direct and indirect relationships.
4 SeqRank
We present SeqRank, which samples trajectories and compares them sequential manner for efficient reinforcement learning from human feedback. We also extend our key component, sequential pairwise comparison, root pairwise comparison, which not only compares trajectories sequentially but also constructs tree structure to rank trajectories in more efficient way. Figure 2 (a) illustrates difference between three trajectory comparison methods including conventional pairwise com- parison and the proposed sequential and root pairwise comparison. We demonstrate the feedback efficiency and convergence of the reward model for each method. Regarding the notations used in paper, summary of symbols and terms can be found in Table I in the supplementary material.

---

We first make the following assumptions: (Uniformness) Every trajectory is sampled uniformly from the replay buffer.
(Completeness) For any two segments o; and oj. "o, oj or " is true.
(Transitivity) For any three segments di, oj and Øk, if oj > o and Øj > ak. then 0; > ok.
Assuming N feedback per reward update, the agent samples the (n + 1)th defender and challenger from Kn, = =o),..., Jn and 24 Kn. respectively, where U is the replay buffer. The defender becomes on for sequential pairwise comparison and arg max,ex," (T (T((O) for root pairwise comparison. After the N2h query, the agent receives N feedback in total. Figure 2 (b) illustrates the preference relationships among trajectory segments as a graph. Especially for root pairwise comparison, the challenger becomes a new root of the tree if it preferred over the current root segment as described in Figure (a). Otherwise, the challenger becomes new child of the current root. The full procedure of our algorithm is provided
the supplementary material. Suppose Sa the total number of trajectory segment pairs obtained from M trajectory segments. M = 2N for conventional pairwise comparison, and M N + 1 for sequential or root pairwise comparison. In the following section, we first demonstrate the feedback efficiency in terms of M and substitute it with the function of N.
4.1 Feedback Efficiency of the Algorithm
From the uniformness assumption, suppose the agent compares M trajectory segments with corre- sponding rewards as a random ordering of 1, ..,,,..M....... M. Based on the completeness and transitivity assumptions, the agent can augment non-adjacent segment pairs as illustrated in Figure 2 (b). Consid- ering that the number of augmented pairs differs by the order of sampling trajectories with different rewards, SM becomes a random variable indicating the total number of pairs collected from M trajectory segments. In this section, we demonstrate the average feedback efficiency of our method.
Definition We define feedback efficiency as the ratio between the total number of trajectory pairs over the number feedbacks. The expected feedback efficiency for N humanfeeddaat feedback is B[SN+1]/N for sequential and root pairwise comparison.
Sequential Pairwise Comparison. Following Definition 1, we use Lemma to examine the expected number of segment pairs collected from sequential pairwise comparison.
Theorem 1. For any integer M Z 2, the expected number of trajectory segment pairs collected from sequential pairwise comparison G. is approximately linear to M as |am/M - 1/(e - 22] = o[1)
Lemma For any integer M 2 2, aM can be induced from the following recurrence relation.
(total number of pairs from all possible orderings of trajectories)
am
(number of all possible orderings of trajectories)
M-11
= 2 - 2/M!
aM- 1+1 (2// +
j=2
By solving the recurrence relation stated in Lemma 1. we derive Theorem 1 that am is approximately equal to a linear equation a(M - -7) with an error less than e, for any M such that M! is greater than 27/e.
1/(e - 2) & 1.392 and > 1, respectively. By substituting sufficiently large value of M 2 100, y is calculated as 1.324, and the expected feedback efficiency of sequential pairwise comparison becomes 1.392 M 1.324) -)), which converges to 1.392 M increases. Moreover, for any M 2 4, the linear approximation error of ay is bounded by 0.003. Detailed proofs of Lemma and Theorem 1 are in the supplementary material.
Comparing the proposed sequential pairwise comparison with conventional pairwise comparison that requires 2N trajectory segments for N human feedback, we observe that sequential pairwise comparison only requires N + 1 trajectory segments to query a human for N feedback. Additionally, using the proposed sequential pairwise comparison, the expected number of augmented queries is given by dN-1 - N z 0.392NN - 0.451.
Extension to Root Pairwise Comparison. We further extend sequential pairwise comparison to root pairwise comparison by exploiting a specific defender sampling strategy. The key idea is to compare the current segment with the most preferred one from the previously sampled segments. As described in Figure 2 (b), root pairwise comparison augments more pairs than sequential pairwise comparison by constructing a tree structure. Using root pairwise comparison, the average feedback efficiency is increased to two. Theorem 2 demonstrates the average feedback efficiency of our algorithm using root pairwise comparison. The detailed proof of Theorem 2 is in the supplementary material.
5

---

Table 1: Trajectory compari- son methods. We compare thr- ee traeectory comparison meth- ods: pairwise, sequential pair- wise, and root pairwise. y de- N notes the feedback efficiency,
Method
M
Best
Average
Worst
PN
Ply
Py
Pairwise
2N
N
1
N
N
Sequential
Parrwis
N ++1
N(N + 1)/2
(NV + +1/2
1.392/N
0.324)
1.992
N
Pairwise
Root
N+1
N(N + 1)/2
(N 11/2
2
Theorem 2. In root pairwise comparison, for any integer M 2 2, the expected number of pairs bas can be denoted by
by = 2 E, M (
Considering that Σn < <1+ 1 de = 1+ In (E H 1) converges to 0 as M increases. Therefore, root pairwise comparison achieves the expected feedback efficiency as /M 1) 1) 2 2. This implies that the proposed root pairwise comparison achieves higher feedback efficiency than sequential pairwise comparison average.
-
Table shows the comparison results of trajectory comparison methods when the number of human feedback is fixed to N. M and pv denote the number of sampled trajectories at each reward update session and the expected number of trajectory pairs obtained from human feedback, i.c., [[Sw]. respectively. For sequential and root pairwise comparison methods, pN is aN+1 and 6N+11 respectively. The baseline method using conventional pairwise comparison samples 2N trajectories, which are approximately twice as many of sequential or root pairwise comparison methods. As shown in Table I, sequential pairwise comparison achieves an average feedback efficiency of 1.392. Also, root pairwise comparison achieves the highest average feedback efficiency 2.
4.2 Convergence of the Reward Model
Notations.
Suppose ((([;;)<{(o)) )) is the input and > oy is the true label of binary classification using a logistic regression model and a cross-entropy loss. Then, Polo, > oy] can be interpreted as a logistic regression model as follows:
1
Pola: > 0y] 1 Ex fo(s) 1 exp(( ,(o,)))
(5)
For simplicity, we define the training data for reward update as D = {( yx3} where uy and yh denote the features of two trajectories included the Athh query and its corresponding preference label provided from a human, respectively. Note that if the 1hth query is (o..0,). )), 24 = o(o,)) ((a)). We define the true risk of the reward model as R(0)= == E(e(o- (x),, (x:)) and the empirical risk of the reward model as R((0= := Lpref = (1/1D)) ExEE (xx), (Tx)), where the cross-entropy loss and its minimizer are denoted as l and 8", respectively. Assume that ||øl|2, ||0 I12. and || |||] bounded with constants S, D, and Q, respectively. Since RLHF involves iterative process of policy and reward update, we distinguish the overall iteration T that represents the global iterative process, and the local iterations of policy and reward update, W and U, respectively. SM,T represents the cumulative sum of the number of training data of the reward model from iteration to T, with M segments compared per iteration. We introduce the concept of dependency graph and its maximum degree AM, M, at iteration E [T] {1, "..,....' T'} for further analysis, following previous work [15]. Detailed proofs for the following theorems, lemmas, and corollaries are in the supplementary material.
High Probability Feedback Efficiency. We find a high probability bound for SM in terms of the expected values of SM described in Lemma 2. This implies that for sufficiently large T and N, the feedback efficiency averaged across iterations is at least equal to Pvv (2.N in the proposed method with high probability. For sequential and root pairwise comparison, the lower bounds become 0.696 and 1, respectively, while the feedback efficiency for pairwise comparison is fixed to 1.
Lemma 2. For sufficiently large T (max, AM, + (M - (n(1//8) M) and 8 E (0, 1), with probability at least 8, SM,T greater than (TT M/2, where 3 denotes the expected number of pairs obtained from M segments divided by M.

---

Convergence Rate of the Empirical Risk. From the completeness assumption, data in D should be linearly separable. According to prior work [24], the empirical risk converges with a rate of (1/UU). Let ni be the batch size of reward update. Then, after the global iteration T, U = [SM.r/BB)
From Lemma 2, prove that with probability at least 1 - 5 and sufficiently large T, the empirical risk of the reward model converges at a rate of O 8/ (8TM)), Therefore, higher 3 results in faster convergence. While (B, MM) for pairwise comparison are fixed as (0.5, 2.N). the values for sequential and root pairwise comparison are approximated to (1.392, N +1 and (2, N + +1), respectively. Based on our analysis, the reward model is likely to converge faster the order of root pairwise, sequential pairwise, and pairwise comparison in terms of the global iteration T.
Convergence Rate of the Generalization Bound. Based on prior work [15, 25], derive generalization bound with Rademacher complexity for learning the reward model under graph- dependence in D. Theorem 3 shows the generalization bound of the reward model. Detailed proof of the theorem and definitions of F and B are presented in the supplementary material.
Theorem 3. Assume that comparison algorithm generates a train set 'D of size SM,T with depen- dency gruph after T iteration. Then, for any 8 E (0, 1), with probability at least 1 - 5,
v0 H. R(O)= S R(()++ + v (mmxx, AM, + 1)/ (2F + 3B /2
(6)
where H. is the parameter space of O. and F and B are functions of M with rates at (M²))
Theorem 3 addresses a critical aspect of the proposed method by proving that even with dependent data, the empirical risk of the reward model converges the true risk with probability at least 1- This high probability bound contributes to the practical analysis of general RLHF models with graph-dependent data such as rankings. Additionally, we consider the trade-off between feedback efficiency (xx SM,r) and data dependency (x AM, As shown in (6), lower AM, and larger SM, lead to a tighter generalization bound of the reward model. However, data dependency increases in our settings as we augment more queries and enhance feedback efficiency. Hence, we demonstrate A: for sequential and root pairwise comparison in Lemma 3 and 4, respectively.
Lemma 3. Let G be a dependency graph generated by sequential pairwise comparison with M trajectory segments. Then, for any ō E (0, 1), the maximum degree of G is bounded as follows, with probability at least -5, 5,
[T) AM, S 2 + (M/(m(1((18)))
-1
(7)
Lemma 4. Let G be a dependency graph generated by root pairwise comparison with M trajectory segments. Then, for any E (0, 1), the maximum degree of G is bounded as follows, with probability at least -5, 8,
max A,, S M 5) -3
(8)
Corollary 3.1. For fixed M 2 2, the generalization bounds of the reward model pairwise, sequen- tial pairwise, and root pairwise comparison converge at rates of O( D(VIn(T)/T) O( ( /II(T))3/T) and O (IVIT(T)TT) respectively, with probability at least 1 - 1/T.
By applying Lemma 2, 3. and 4 to Theorem 3, we derive Corollary 3.1, which compares the generalization bounds of three methods. While generalization bounds of all methods are guaranteed to converge as 7 increases, root pairwise comparison demonstrates a faster convergence rate than sequential pairwise comparison, and has the same convergence rate as pairwise comparison.
Overall analysis of convergence. To sum up, root pairwise comparison theoretically surpasses the baseline it proves better and equal convergence rates of both metrics, while requiring a half of samples than pairwise comparison. Sequential pairwise comparison achieves faster convergence of the empirical risk but slower convergence of the generalization bound compared to pairwise comparison. From the perspective of reward model convergence, prioritizing the feedback efficiency to speed up the convergence of the empirical risk is more critical than considering the generalization error in worst-case scenarios. In Section 5, we empirically show the superiority of root pairwise comparison over sequential and pairwise comparison.
5 Experiments
We evaluate our method on robotic locomotion tasks from DeepMind Control Suite (DMControl)
[3, 26] and robotic manipulation tasks from Meta-Worke [4]. We also demonstrate the proposed
7

---

Teedback
Orack
Sequential Pairwise
Walker Walk
0.4K
957.3 =
862.5
907.5 + 66.2
Cheetah Run
0.2K
886.9
43.2
690.2
93.2
715.8 117.4
739.4 +: 100.8
Quadruped Walk
IK
843.3
247.3
353,5
183.9
479.5 = 234.7
728.8 + 274.1
Humanoid Walk
40K
272.5
117.0
114.3 +
80.5
141.3 + 34.4
163.8 + 71.6
Hopper Hop
4K
273.5 47.0
26.2 = 37.0
37.3 1 $9.8
100.1 + 70.7
Table 2: Rewards after convergence DMControl locomotion tasks. Two elements in each cell denote the average value and standard deviation of rewards across runs with 10 random seeds.
Task
Teedback
Oracle
Pairwise
Sequential Pairwise
Root Pairwise
Button Press
10K
99.3
95.6
97.0 + 4.3
97.6 5.6
Door Open
10K
100.0
77.9 = 41.3
77.4 + 40,8
97.8 5.0
Drawer Open
20K
99.9
65.3 + 41.6
72.0 + 45.4
89.9 31,6
Sweep Into
10K
88.8
29.9
68.4 + 35.2
55.7 + 48.1
88.0 + 31,0
Window Open
IK
99.9 0.3
55.9 + 45.1
66.1 + 36.5
70.7 40.8
Hammer
10K
91.5 26.5
31.0 + 24.7
30.8 +: 35.4
48.8 41.2
Table 3: Success rates in Meta-World manipulation tasks. Two elements in each cell denote the average value and standard deviation of success rates across runs with 10 random seeds.
method in the real-world using a UR-5 robot for a block placing task. Experiment details and additional analyses with rendered results provided in the supplementary material.
5.1 Experiment Settings
Tasks. Locomotion tasks from DMControl consist of walker walk, cheetah run, quadruped walk, humanoid walk, and hopper hop. Manipulation tasks from Meta-Worke consist of button press, door open, drawer open, sweep into, window open, and hammer. For each task, we train with 10 different seeds used in prior work [6, 10] and measure the average performance with standard deviation. Evaluation metrics are chosen as the ground-truth return per episode for DMControl and the success rate for Woold. Additionally, we conduct experiments with real human feedback for 30 participants using policy and reward models pre-trained in the simulation environment. We also demonstrate block placing task using a UR-5 robot in the real-world, by pre-training the policy in the Mujoco simulator [27] and fine-tuning in the real-world. Detailed experiment settings are provided in the supplementary material.
Baselines and Implementation Details. We use three baselines for all tasks: SAC [28], Meta- Reward-Net (MRN) [10]. and PEBBLE [7]. Agents learned via SAC with the true reward functions are considered as the upper bound of all tasks. We use MRN and PEBBLE as the baseline methods using the traditional trajectory comparison method, pairwise comparison. MRN utilizes bi-level optimization for both reward and policy learning and achieves state-of-the-arr among previous methods. We apply our sequential preference ranking algorithm on both MRN and PEBBLE to show effectiveness of our algorithm. Results with MRN baseline are described in this section and results with PEBBLE baseline are provided in the supplementary material. We use unsupervised pre- tre-training [7] for 9, 000 steps for all experiments. The trajectory encoder is implemented using two-layer
neural network, where the input dimension is the combined size of the state and action spaces, and the output dimension is set to 256. The linear reward model then takes the encoded feature and passes it through single fully connected neural network. For each task, hyperparameters and implementation details are described the supplementary material.
5.2 Simulation Experiments
Locomotion Tasks from DMControl. Results in Table 2 demonstrate the superiority of the proposed root pairwise comparison over the baseline pairwise comparison and sequential pairwise comparison in all locomotion tasks. The reward obtained after convergence is higher by an average of 29.0% compared to conventional pairwise comparison used MRN [10]. Sequential pairwise comparison, on the other hand, does not exceed root pairwise comparison, but it improves performance against pairwise comparison by 10.3% on average. Based on the standard deviation of rewards after con- vergence across 10 different runs, root pairwise comparison shows higher stability than sequential

---

(a) Quadruped Walk (feedback=1K)
(b) Hopper Hop
(c) Drawer Open (feedback=20k)
(d) Window open (feedback=1K)
(feedback=4K)
Figure 3: Comparison results in two locomotion tasks from DMControl and two manipulation tasks from Meta- results among three trajectory comparison methods: pairwise (green), sequential pairwise (orange), and root pairwise (blue). Pink lines describe the oracle performance using SAC with true reward.
Meta-World. World. All methods are implemented based MRN. Each subfigure describes the comparison
Pairwiee
Sequential
Root
20
Fine buning
10
Training with 200 fnedback
6:0
Average
3
(a) Average Rewards
(b) User Satisfaction Score
Figure Training the Cheetah Run Task with Real Human Feedback. (a) The bar plot describes ratio between the average rewards after training the policy with real human feedback. (b) The histogram describes distributions of user satisfaction scores for three trajectory comparison methods.
pairwise comparison. Reward graphs in Figure (a) and (b) show that root pairwise comparison enables faster convergence of the policy learning.
Additionally, we assess the converged reward for various feed- back maximums (O.1K, IK, 2.5K, 4K) in the quadruped walk task, as illustrated in Figure 4. The graph reveals that both sequential and root pairwise comparison methods converge to higher average reward values and lower variance as the num- ber of feedbacks increases. This implies that with a sufficient number feedbacks, the amount of augmented preference data affects performance more than data dependency does. By taking advantage of learning a dense reward [5, 29, 30), root pairwise comparison outperforms the oracle baseline with sufficient number feedbacks.
Jumman after Caroungercc
8.
SAC (oraci)) poinwise
soquancial [M-V Ve
200
1000 1568 7000
350
300
200
Number
Figure 4: Average reward per num- ber of feedbacks after convergence.
Manipulation Tasks from Meta-World. Results in Table show the comparison results with baseline in manipulation tasks. The proposed method using root pairwise comparison outperforms the methods using pairwise comparison and sequential pairwise comparison in all tasks, improving the overall success rate by 25.0% compared to the pairwise comparison baseline. Reward graphs in Figure 3 (c) and (d) show that root pairwise comparison enables faster convergence of the policy, Unlike the locomotion tasks, sequential pairwise comparison only improves the success rate by 1.2% over pairwise comparison. This may be due to the generalization error using sequential pairwise comparison. Comparing the standard deviation success rates among 10 different runs, root pairwise comparison achieves the lowest standard deviation. This implics that root pairwise comparison is the most stable algorithm for manipulation tasks among three trajectory comparison methods.
Experiments with Real Human Feedback. To compare the performance and the user stress levels associated with three different methods: pairwise, sequential pairwise, and root pairwise comparison, we conduct experiments involving 35 real human participants. 5 participants trained the policy and the reward models from scratch with 200 human feeedbacks. Other participants fine-tuned the models with 20 human feedbacks, where the models were pre-trained with simulated feedbacks. Figure (a) shows that the proposed methods outperform the baseline both training from scratch and fine-tuning experiments. Especially for training the models from scratch, root and sequential

---

Method
# feedback
Success Ratet
True Reward
Episode
Reward
Length
Accuracy
Pairwise 0.8K
34.4
17.1
29.6
75.8
Sequential Pairwise
0.8K
41.0
20.1
28.4
80.9
Root Pairwise
0.8K
50.5
25.9
27.9
84.9
Table 4: Average performance in the real robot manipulation task. For each method, the agent is fine-tuned in the real world for 3,000 steps.
Figure 6: Block placing using UR-5 robot.
pairwise comparison outperform conventional pairwise comparison by 47.7% and 38.6%, respectively. Quantitative results are provided and analyzed the supplementary material. Additionally, as a user study, asked the participants to rate their user satisfaction on scale from one five, with higher scores indicating greater satisfaction and lower stress levels. Figure 5 (b) illustrates that the average user satisfaction scores resulted in 2.20, 3.00, and 3.93 for pairwise, sequential, and root pairwise comparison, respectively. We perform paired-sample t-test and the differences between the scores are statistically significant for all cases, with the average p-value 0.035. Specifically, users responded that it is convenient to compare with one's 's most preferred trajectory. These results imply that root pairwise comparison is the least burdensome for real human users while achieving the highest performance.
5.3 Real Robot Experiments
Block Placing Task. To demonstrate our method in real-world environments, we conduct a block placing task using a real UR-5 robot. As illustrated in Figure 6, the goal of this task is to place block at a given target position starting from a randomly chosen initial position. For faster training and stable movements of the agent, we train the policy with a deep Q-learning network (DQN) and discrete action space following prior work [31]. In this setting, an action is given by the moving direction 1b where d is the heading direction of the end effector discretized into eight angle bins {0, x 2 4: ..... T= Once 1/ is determined, the agent changes its pose and moves the end effector for a fixed distance in the angle of th. The policy is pre-trained in the simulator for 20,000 steps and fine-tund tnnettned in the real-world for 3,000 steps. The reward model is initialized before fine-tuning. Results in Table 4 show the comparison results with baseline after fine-tuning the real-world. Using root pairwise comparison, the agent exhibits the highest success rate and the reward model achieves the fastest convergence with the highest accuracy among three trajectory comparison methods.
6 Conclusion
We propose novel RLHF framework called SeqRank that utilizes sequential preference ranking, resulting in substantial improvement in the feedback efficiency by least 39. 19.2%. Both sequential and root pairwise comparison enables faster estimation of the reward function. We also prove the convergence rates of the empirical risk and the generalization bound of the reward model for each method. Theoretical and experimental results show that both methods outperform conventional pairwise comparison, with root pairwise comparison showing the most substantial improvement against the baseline. The proposed method and theoretical analyses can be applied to other domains and serve as a practical solution to enhance the feedback efficiency and performance of RLHF. Potential negative societal impacts include amplifying existing biases present in human preferences. We hope our method encourages RLHF be used for social good without violating ethics.
Limitations. As we experimentally show that the proposed root pairwise comparison achieves the highest reward accuracy among three trajectory comparison methods, we expect to find a tighter generalization bound of the reward model. As the generalization bound of logistic regression with dependent data is not fully studied yet, it will be our future work.
Acknowledgments and Disclosure of Funding
This work was in part supported by Institute of Information & Communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. 2019-0-01 190, (SW Star Lab] Robot Learning: Efficient, Safe, and Soceally Machine Learning, 66%, No. 2022-0- 00331, Development of emotion recognitiion/genraation based interacting edge device technology for mental health care, 17%, and No. 2021-0-01344 AI Graduate School Program, CAU, 17%).
10

---

References
[1] Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, al. A survey of preferenee-based reinforcement learning methods. Journal of Machine Learning Research, 18 18(136)11-46; 2017.
[2] Michel Regenwetter, Jason Dana, and Clintin P Davis-Stober. Transitivity of preferences. Psychological review, 118(1):42, 2011.
[3] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 1001.v8801.0000 2018. 2.7
[4] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsca Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-tase, and meta reinforcement learning. In Proceedings of the Conference robot learning (CoRL), 2020. 2,
[5] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences, Advances in neural information processing systems (NeurIPS), 2017. 3,9
[6] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking geffrence-basse reinforvement learning. Neural Information Processing Systems Truck Datasets and Benchmarks (NeurlPS), 2021. 3,
[7] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficien interactive reinforcement learning via relabeling experience and unsupervised pre-training. Proceedings of the International Conference on Machine Learning (ICML), 2021. 8,8
[8] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi- supervised reward learning with data augmentation for feedback beefback-efficient preference preference-based reinforcement learning. Proceedings the International Conference Learning Representations (ICLR), 2022.
[9] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. Proceedings of the International Conference on Learning Representations (ICLR), 2022. 3
[10] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-nett Implicitly differentiable reward learning for preserence-based reinforcement learning. In Advances in Neural Information Processing Systems (NeurlPS), 2022. 8
[11] Malayandi Palan, Nicholas C Landolfi, Glcb Shevchuk, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. Proceedings of the Robotics: Science and Systems (RSS), 2019. 3
[12] Erdem Brysk, Daniel A Lazar, Dorsa Sadigh, and Ramtin Pedarsani. The green choice: Learning and influencing human decisions on shared roads. In Proceedings of the IEEF Conference Decision and Control (CDC). IEEE, 2019.
[13] Xianyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-hasee reinforcement learning with general function approximation. In Proceedings of the International Conference Machine Learning (ICMI), 2022.
[14] Erdem Bıyik, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. Proceedings of the Conference on Robot Learning (CoRL., 2019.
[15] Rui-Ray Zhang and Massih- Reza Amini, Generalization bounds for learning under graph-dependence: A survey, arXiv preprint arXiv:2203. 13534, 181:109272, 2022. 3, 6,
[16] Svante Janson, Tomasz Luczak, and Andrzej Rucinski. An exponential bound for the probubility of nonexistence of specified subgraph in random graph. Institute for Mathematics and its Applications (USA), 1988. 3
[17] Nicolas Usunier, Massih-Reds Reza Amini, and Patrick Gallinari. Generalization error bounds for classifiers trained with interdependent data. In Advances in Neural Information Processing Systems (NeurIPS). 2005.
[18] Rui Ray Zhang, Xingwu Liu, Yuyi Wang and Liwei Wang. Mcdiarmid-tyre inequalities for graph dependent variables and stability bounds. Advances in Neural Information Processing Systems (NeurIPS). 2019. 3

---

[19] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems (NeurIPS), 2018.
[20] Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for preference- reinforcement learning. Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2020.
[21] Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference- reinforcement learning with linite-time guarantees. Proceedings of the Advances in Neural Information Processing Systems (NewrIPS). 2020.
[22] Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: reinforcement learning with trajectory preferences. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS),
2021. 3
[23] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biomerrika, 39(3/4):324-345, 1952. 4
[24] Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint In0iv:1803.07555 2018. 7
[25] Massih Reza Amini and Nicolas Usunier. Learning with partially labeled and interdependent data. Springer, 2015.
[26] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm_ control: Software and tasks for continuous control. Software Impacts, 6:10002 2020. 7
[27] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
[28] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the International Conference on Machine Learning (ICML), 2018.
[29] Aleksandra Faust, Anthony Francis, arXiv preprint arXiv:1905.07622 2019. 9
Mehta. Evolving rewards to automate reinforcement learning.
[30] Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver. and Satinder Singh. What can learned intrinsic rewards capture? In International Conference on Machine Learning, pages 11436 1 1446. PMLR, 2020. 9
[31] Hogun Kee, Minjae Kang. Dohyeong Kim. Jaegoo Choy, and Songhwai Oh. Sdf-based graph convolutional g-networks for rearrangement of multiple objects. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023. 10
12


Reviews:
[{"review_id": "8S2rS2LkvD", "reviewer": "Reviewer_sxrB", "summary": "This paper presents SeqRank a framework for labelling user preferences in a ranked fashion taking advantage of the transitive property of human preferences.\n\nAuthors mathematically prove that their two variants of SeqRank (sequential pairwise comparison, and root pairwise comparison), achieve better feedback efficiency (defined as the ratio between the number of preference labels and the number of trajectories) than conventional pairwise comparisons. Additionally, authors prove that both methods converge to the same level of empirical risk and that root pairwise comparison converges at the same rate as the baseline pairwise comparison, whereas sequential pairwise converges slightly more slowly.\n\nEmpirical evaluations show that both variants of SeqRank beat the performance of a pairwise comparison using a recent Preference-based Reinforcement Learning algorithm  (Meta-Reward Net, MRN), in 5 Mujoco tasks and 6 MetaWorld tasks. Experiments show a correlation between feedback efficiency and preferences sample efficiency.\n\nAdditionally, authors prove the usefulness of SeqRank in a finetuning-from-human-feedack task as well as training an actual UR-5 robot on a block-placing task.", "questions": "1. PEBBLE does a selection of trajectory pairs based on the reward disagreement between three modules. How does this fit with SeqRank? Does it affect the uniformness assumption? Or does SeqRank simply work off the selected subset?\n2. In figure 4, are the task rewards normalised before being averaged? If not, tasks like Walker Walk (where SeqRank does really well) may be overrepresented.\n3. Section 5.3 is really compressed, to the point it is difficult to understand exactly what is the experiment. \n    * How are the preferences obtained?\n    * Is the reward also fine-tuned when training on the actual robot? What is meant by \"the reward model is initialized before fine-tuning\"?\n    * What is True Reward and Reward Accuracy in Table 4?\n\n**Nitpicks and suggestions:**\n* In the introduction, the paper claims that obtaining multiple preferences from a single trajectory reduces the cognitive load for the user. The experiments offer no evidence for this. I would rephrase as users prefer a more efficient method.\n* Consider simplifying the notation for the expectation for trajectory pairs which is variously $\\mathbb{E}[S_N]$, $p_N$, $a_N$, $b_N$  in different parts of the paper. I understand each is subtly different in each context, but perhaps a superscript could be used to denote the differences?\n* The description of the SeqRank algorithm is left in the appendix, is there some way to bring it back to the main text?\n* Mention in the main text that statistics about the population are in the appendix.\n* In figure 4, why is root pairwise comparison beating SAC?\n\n\n**Update after rebuttal:**\n\nRefer to author's rebuttal for answers.", "limitations": "Limitations are adequately addressed. I would point out that this method lowers the barrier of entry to control complex systems that would otherwise not be controlled by humans. This has safety and ethical implications that should be considered before deployment.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Paper presents an intuitive idea (rank preferences) backed up by robust mathematical analysis of its properties.\n* Compelling results in a wide variety of tasks, both in simulated and in real-world environments.\n* Feedback efficiency seems to correlate with sample efficiency, thus lowering the barrier of adoption for Preference-based Reinforcement Learning methods.", "weaknesses": "* Authors only test SeqReq on Meta-Reward Net. The paper would be more significant if  the same benefits observed for MRN+SeqRank were shown to carry over to other PbRL methods such as PrefPPO, PEBBLE [1].\n* In my opinion, the experiments with real humans do not help in establishing the significance of SeqRank. \n    * The relative improvement with root sequence comparison is of just 0.4% over the reward of the policy before fine-tuning. \n    * No measure of spread for the reward reported, so it is not possible to establish whether that improvement is significant or just noise. \n    * Regarding the questionnaires, authors should report whether the differences are statistically significant (preferably through a rank-based statistical test).\n* Beyond the above issues, I would have found it more significant to train the policy from scratch with actual human preferences and SeqRank as is common in the literature [1,2,3], rather than fine-tuning an already trained policy.\n\n\n[1] K Lee, L Smith, P Abbeel (2021) PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training. ICML.\n\n[2] PF Christiano, J Leike, T Brown, M Martic, S Legg, D Amodei (2017) Deep Reinforcement Learning from Human Preferences. NeurIPS.\n\n[3] B Ibarz, J Leike, T Pohlen, G Irving, S Legg, D Amodei (2018) Reward learning from human preferences and demonstrations in Atari. NeurIPS\n\n\n**Update after rebuttal**\n\nThe authors carried out additional experiments which addressed most of my concerns. In particular, the authors showed that i) SeqRank is compatible with uncertainty sampling, ii) that PEBBLE+SeqRank improves over vanilla PEBBLE, and  iii) that SeqRank outperforms pairwise comparison when training from scratch from human feedback. \n\nThe experiments with real human feedback from a pre-trained policy do not show a significant improvement over the baseline, but do show statistically significant higher user satisfaction scores for SeqRank.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the positive and thoughtful review by Reviewer sxrB, _e.g., robust mathematical analysis and compelling results in a wide variety of tasks_. We will address all raised concerns by the reviewer and revise the paper accordingly.\n\n## **Weaknesses**\n**Weak1: “Authors only test SeqRank on Meta-Reward Net. The paper would be more significant if the same benefits observed for MRN+SeqRank were shown to carry over to other PbRL methods such as PrefPPO, PEBBLE [1].”**\n\n> We thank Reviewer sxrB for suggesting additional experiments to evaluate the effectiveness of our method in various baselines. Based on the reviewer’s suggestion, we conducted experiments using the PEBBLE [1] baseline. Please refer to **Global Response to All reviewers - Additional Experiments - Exp1**. As shown in Table 1 and Table 2, we were able to improve the performance against PEBBLE. We will include experiment results using both PrefPPO and PEBBLE baselines for the final version of our paper.\n\n**Weak2: In my opinion, the experiments with real humans do not help in establishing the significance of SeqRank. … authors should report whether the differences are statistically significant.**\\\n**& Weak3: I would have found it more significant to train the policy from scratch with actual human preferences … rather than fine-tuning an already trained policy.**\n\n> We include statistical analysis for the real human experiment. Also, we trained the agent with real human feedback from scratch during the rebuttal period, without using any pre-trained policy or reward models. Please refer to **Global response to All reviewers - B.2. and Exp3**.\n\n## **Questions**\n**Question1: PEBBLE does a selection of trajectory pairs based on the reward disagreement between three modules. How does this fit with SeqRank? Does it affect the uniformness assumption? Or does SeqRank simply work off the selected subset?**\n\n>As Reviewer sxrB mentioned, uncertainty-based sampling methods such as disagreement sampling in PEBBLE can definitely be applied to SeqRank. We implement disagreement sampling by randomly sampling a subset of trajectories and choosing a trajectory with the highest disagreement with the current defender as the challenger. Implementation details and the experiment results will be included in the final version of the supplementary material. For the experiment results and analysis, please refer to **Global response to All reviewers - Additional Experiments - Exp2**.\\\n\\\nOn the other hand, disagreement sampling does affect the uniformness assumption, therefore we need some modifications on the assumptions of our theoretical analysis. We can first calculate the expected number of trajectory pairs S_M and data dependency based on the sampling strategy. Then, we can just use the convergence rate of the reward model’s empirical risk (L246) and the generalization bound (L255, Theorem 3) to theoretically demonstrate the effectiveness of SeqRank with disagreement sampling.\nEven for other types of sampling strategies, we can follow the same process and easily compare different types of sampling strategies for our work. We claim that Theorem 3 can be widely used to expand our work.\n\n**Question2: In figure 4, are the task rewards normalised before being averaged? If not, tasks like Walker Walk (where SeqRank does really well) may be overrepresented.**\n\n> Yes, the true task rewards from DMControl simulator are normalized before being averaged. We will include this detail in the experiment details in the supplementary material.\n\n**Question3: Section 5.3 is really compressed, to the point it is difficult to understand exactly what is the experiment. How are the preferences obtained? Is the reward also fine-tuned when training on the actual robot? What is meant by \"the reward model is initialized before fine-tuning\"? What is True Reward and Reward Accuracy in Table 4?**\n\n> The true reward is a hand-designed reward defined by human. Because it takes a long time to train a robot in the real world from scratch, we pre-trained the policy using RLHF with synthetic feedback from the true reward. Then, we fine-tuned the agent policy from the pre-trained model and trained the reward model from scratch. We chose to learn the reward model from scratch because there are some noises in real-world states and actions. We also used synthetic feedback from the true reward function for real-world experiments, due to time constraints. We changed the reward function used in previous work [2] to fit into our block-placing task. The reward accuracy measures the consensus between ensemble networks for the reward model. We will add the specific definitions of the terminologies in the final version of the supplementary material.\n\n## **Nitpicks and suggestions**\n>**Cognitive Load Statement**:\nYes, we will rephrase the statement. Please refer to **Global response to All reviewers - A.2. Overstatement Regarding Cognitive Load**.\n\n>**Notation Simplification**:\nWe agree with the reviewer’s suggestion to use a superscript to denote the differences between similar notations. We will change the notations as $S_M^*$, where $*$ will be pair, seq, and root for pairwise, sequential pairwise, and root pairwise comparison, respectively.\n\n>**Moving Algorithm to the Main Text**:\nYes, we will move the algorithm to the main paper. Please refer to **Global response to All reviewers - A.1. Moving Algorithm to Main Text**.\n\n>**Experiment Details**:\nYes, we will mention that statistics about the real human experiment participants are provided in the supplementary material.\n\n>**Figure 4 - why is root pairwise comparison beating SAC?**:\nPlease refer to **Global response to All reviewers - B.1. Additional Analysis for Figure 4 in the main paper**.\n\n[1] Lee et al., PEBBLE, ICML, 2021. (shortened due to characters limit)\\\n[2] Kee et al., \"Sdf-based graph convolutional q-networks for rearrangement of multiple objects\", International Conference on Robotics and Automation, 2023."}, {"review_id": "GI6Schaxht", "reviewer": "Reviewer_tocg", "summary": "This paper studies the idea of the feedback efficiency of RLHF, which the authors define as the ratio between the number of trajectory pairs divided by the number of feedbacks. To optimize such a metric, the authors propose new algorithms that decide which trajectories to compare. In particular, the authors propose a sequential pairwise comparison strategy, where the most recent trajectories are compared and a root comparison strategy, where the defender is the previously most preferred trajectory.\n", "questions": "**Q1.** What would be the impact of these algorithms in settings where only a small number of trajectories are comparable?\n\n**Q2.** Can you show RM convergence is faster (similar to figure 4)? The algorithms proposed in this paper should most directly affect the RM convergence.\n\n**Q3.** In figure 2, it seems like pairwise comparison would result in a more diverse training set. Could this diversity boost the performance of the reward model?", "limitations": "Yes, the authors address the limitations of the study.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper studies the important and timely problem of RLHF.\n- The paper has extensive experiments, including simulated and real world tasks, as well as using real human feedback.\n- The study on the labelers satisfaction is novel and very interesting.", "weaknesses": "**W1.** One major concern with this paper is that the claim of completeness, (that any two trajectories are comparable) is incorrect for the main RLHF application of LLM finetuning [1]. Since this is the claim is key to the paper, I think it makes the paper poorly motivated/placed in the literature.\n\n[1] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in Neural Information Processing Systems 35 (2022): 27730-27744.\nAPA\t\n\n**W2.** Another major issue is that the contribution of using the transitive property of human preferences is limited. This is a limited contribution since RLHF works typically design the labeling interface in such a way that the labeler will implicitly use the transitive property of human preferences. An example can be seen in Figure 12 of the InstructGPT paper [1].\n\n**W3,** The claim that cognitive overload is a critical drawback of RLHF seems like an overstatement. In fact, the labeler satisfaction and inter-labeler agreement in the InstructGPT paper (and other RLHF papers) is high (see Table 13 of [1]).\n\n**W4.** The presentation of the author's method is not very clear and the theoretical results are hard to follow. The main tool the authors use to illustrate their method is Figure 2, which is not exact enough. Moving the algorithm to the main text would be helpful.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate all feedback from Reviewer tocg and will incorporate the reviewer’s comments to revise our paper to be more sound and clear. Since the reviewer’s concerns are mainly grounded in the InstructGPT paper [1], we would like to mention three major differences between the robotics domain and the language domain RLHF.\n\n**1. Data and Feedback Complexity:**\nGathering data and feedback in robotics requires the assessment of complex physical interaction with the environment, whereas LLM relies on generative models.\\\n**2. Transitivity Property and Ranking:**\nApplying transitivity property via ranking feedback is more complex in robotics due to sequential actions, compared to the more straightforward application in LLM.\\\n**3. Generalization:**\nGeneralizing across tasks is more complex in robotics due to specific physical properties, whereas it is more feasible across tasks in LLM. Therefore, in the robotics domain RLHF, we deal with a specific fundamental task using a robot with fixed dynamics in a fixed environment.\n\n## **Weaknesses**\n**Weak1: One major concern with this paper is that the claim of completeness … incorrect for the main RLHF application of LLM finetuning [1]. … it makes the paper poorly motivated/placed in the literature.**\n\n>Based on the InstructGPT paper [1], RLHF is originally developed for training simple robots and Atari games, and has recently diverged into two parts: (1) its original purpose and (2) fine-tuning language models. Our primary focus is on training an agent for specific robotic tasks, like drawer opening with a robot arm (manipulator) with fixed dynamics in a fixed environment. In such contexts, preferences are understood as judgments on robot performance. Based on our knowledge, all previous RLHF work in robotics (a.k.a. PBRL) assume that any two trajectories are comparable in a specific task. Therefore, we consider the completeness assumption reasonable and the transitivity property of preference is applicable in our work.\n\n**Weak2: Another major issue is that the contribution of using the transitive property of human preferences is limited. This is a limited contribution since RLHF works typically design the labeling interface in such a way that the labeler will implicitly use the transitive property of human preferences. An example can be seen in Figure 12 of the InstructGPT paper [1].**\n\n>As Reviewer tocg mentioned, transitivity is a natural property of human preference, and humans implicitly use transitivity when they provide ranking feedback. While recent work [1] in the language domain RLHF uses ranking feedback, there isn’t a successful application of the ranking feedback in robotics yet due to the three differences between robotics and LLM listed at the top of this thread. Even if a human can provide ranking feedback, pairwise comparison requires less effort than ranking 3+ trajectories. Also, the proposed sequential preference ranking is more robust to noise than ranking feedback. Our methods are less affected by human uncertainty because the agent ranks preferences autonomously, not relying on human’s ranking feedback (L097).\n\\\nWe claim that Reviewer tocg’s concern that the transitivity property is already used in the LLM domain _emphasizes the motivation and importance of our work_ rather than limits our contribution, especially given the challenges of using ranking feedback in the robotics domain.\n\n**Weak3: The claim that cognitive overload is a critical drawback of RLHF seems like an overstatement. In fact, the labeler satisfaction and inter-labeler agreement in the InstructGPT paper (and other RLHF papers) is high (see Table 13 of [1]).**\n\n>Yes, we will revise the statement. Please refer to **Global Response to All Reviewers (Part I) -A.2**. However, the labeler satisfaction survey in InstructGPT is a domain-specific survey to rank language instructions, not robot trajectories. Considering the difference between the two domains, the survey does not imply that participants involved in robotics domain RLHF experiments could also be highly satisfied using ranking feedback. Also, while pairwise comparison is simpler than ranking feedback to users, we show that our method improves user satisfaction over conventional pairwise comparison.\n\n**Weak4: The presentation of the author's method is not very clear ... Figure 2 is not exact enough. Moving the algorithm to the main text would be helpful.**\n\n>Yes, we will move the algorithm to the main text for a clear description of the proposed method and revise Figure 2 to include how the reward model is trained from the trajectory pairs. Please refer to **Global Response to All Reviewers (Part I) -A.1**.\n\n## **Questions**\n**Question1: What would be the impact of these algorithms in settings where only a small number of trajectories are comparable?**\n>If two trajectories are too similar in many queries, we can apply \"equal\" labels and improve our algorithm for data augmentation. Please refer to the response to Reviewer Mtqb-Weak2.\n\n**Question2: Can you show RM convergence is faster (similar to figure 4)? The algorithms ... should most directly affect the RM convergence.**\n\n>Yes, as derived in **Convergence Rate of the Empirical Risk** (L242) in Section 4.2, one of our main contributions is that the reward model converges faster than conventional pairwise comparison. Specifically, root pairwise comparison achieves twice as fast, and sequential pairwise comparison achieves a 39.2% faster convergence rate of the reward model on average.\n\n**Question3: In figure 2, it seems like pairwise comparison would result in a more diverse training set. Could this diversity boost the performance of the reward model?**\n>Diversity depends on both (1) data dependency and (2) the number of data samples. Please refer to **Global Response to All Reviewers (Part V) - Exp4**.\n\n[1] Ouyang et al., \"Training language models to follow instructions with human feedback\", Advances in Neural Information Processing Systems, 2022."}, {"review_id": "cJTVf9EHl2", "reviewer": "Reviewer_Mtqb", "summary": "This paper introduces a novel approach, named SeqRank, to enhance the efficiency of reinforcement learning from human feedback. By incorporating sequential preference ranking, the proposed method significantly improves average feedback efficiency.  Evaluations demonstrate its superiority over existing baselines in simulated locomotion and manipulation tasks, as well as in real-world scenario with a robot manipulation task.", "questions": "1. In Section 5.2, you mentioned that Sequential pairwise comparison performs better than pairwise comparison on average. However, in several tasks, such as Sweep Into, pairwise comparison was better on average than sequential pairwise. Could the authors please explain why this discrepancy exists?\n\n2. In Figure 4, authors presented results for a specific task. Based on the values shown, it appears to be the Quadruped Walk task. However, it is not explicitly mentioned in the paper which task the results in Figure 4 correspond to. Could the authors please clarify this? \n\n3. Regarding the Experiments with Real Human Feedback: what are the standard deviations of the results. Could you provide information on the standard deviation for the reported data? Additionally, have the authors conducted any statistical hypothesis tests to validate the claims made based on the experiments with real human feedback?", "limitations": "See Weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper exhibits strengths in several areas. Firstly, it effectively identifies and defines relevant problem o feedback efficiency, Secondly, the paper provides a strong mathematical justification for the proposed method. The authors present a theoretical framework, underpinned by sound mathematical principles. Furthermore, the paper showcases evaluations conducted on a diverse set of tasks, including real-world experiments with a physical robot. The inclusion of real-world experiments adds a practical element, validating the effectiveness of the proposed solution in realistic settings.", "weaknesses": "One limitation of the paper is the lack of consideration for alternative sampling methods beyond pairwise sampling. While uniform pairwise sampling is a widely used, other sampling methods, such as Uncertainty-based sampling[1] perform better.\n\nAnother point that could have been addressed in the paper is the possibility to label comparisons as \"equal”.  It is worth mentioning that incorporating this labeling aspect could potentially lead to improvements in performance of the proposed method, but it is not clear how that could be done.\n\n[1] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. Neural Information Processing Systems Track on Datasets and Benchmarks(NeurIPS), 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the positive and constructive review by Reviewer Mtqb, _e.g., strong mathematical justification, validation of the effectiveness of the proposed solution in realistic settings_. We will address all raised concerns by the reviewer and revise the paper accordingly.\n\n## **Weaknesses**\n**Weak1: One limitation of the paper is the lack of consideration for alternative sampling methods beyond pairwise sampling. While uniform pairwise sampling is widely used, other sampling methods, such as Uncertainty-based sampling[1] perform better.**\n\n> As Reviewer Mtqb mentioned, the uncertainty sampling method is used in prior work PEBBLE [1]. We conduct additional experiments by applying uncertainty-based sampling to our method. Please refer to **Global response to All reviewers - Additional Experiments - Exp2**.\n\n**Weak2: Another point that could have been addressed in the paper is the possibility to label comparisons as \"equal”. It is worth mentioning that incorporating this labeling aspect could potentially lead to improvements in performance of the proposed method, but it is not clear how that could be done.**\n\n> In B-Pref [2], two trajectory segments are preferred equal if the difference between the rewards corresponding to the two trajectories is less than a threshold value.\nSuppose we sample 7 trajectory segments with the corresponding rewards [4.9, 5, 4, 3, 3.1, 7, 9]  for sequential pairwise comparison. If we apply our original sequential pairwise comparison, we can augment 4 indirect trajectory pairs. However, if we set the equal preference threshold as 0.2 and perform sequential pairwise comparison with additional transitivity assumptions (1) if A=B and B<C, A<C and (2) if A=B and B>C, A>C, we can augment 9 indirect trajectory pairs, which is more than twice as the original number of augmented trajectory pairs. Although our paper mainly focuses on basic settings without considering equal labels, we expect that our method can be combined with various types of simulated teachers suggested in PEBBLE.\n\n## **Questions**\n**Question1: In Section 5.2, you mentioned that Sequential pairwise comparison performs better than pairwise comparison on average. However, in several tasks, such as Sweep Into, pairwise comparison was better on average than sequential pairwise. Could the authors please explain why this discrepancy exists?**\n\n> As Reviewer Mtqb mentioned, sequential pairwise comparison outperforms pairwise comparison in all DMControl locomotion tasks while underperforms for door open, sweep into, and hammer tasks in Meta-World. For door open and hammer tasks, the differences between the average success rates are only 0.5 and 0.2, respectively, which could be seen as noise. Still, for sweep into task, the discrepancy gets higher because the true reward function from the simulation environment is not as good as other tasks. Based on the performance of the Oracle SAC model with true reward function in sweep into task, the success rate is 88.8%, which is the lowest among all manipulation tasks in Meta-World. With an improved true reward function that can represent human expertise, we expect that the discrepancy between pairwise and sequential pairwise comparison will be reduced, or sequential pairwise comparison will perform better than conventional pairwise comparison.\n\n**Question2: In Figure 4, authors presented results for a specific task. Based on the values shown, it appears to be the Quadruped Walk task. However, it is not explicitly mentioned in the paper which task the results in Figure 4 correspond to. Could the authors please clarify this?**\n> Yes, we apologize for the lack of information about the task. Figure 4 illustrates the experiment results in DMControl quadruped walk task with various number of feedbacks. Please refer to Global response to All Reviewers - B. Additional Experiments and Analyses - Additional Analysis for Figure 4 in the main paper.\n\n**Question3: Regarding the Experiments with Real Human Feedback: what are the standard deviations of the results. Could you provide information on the standard deviation for the reported data? Additionally, have the authors conducted any statistical hypothesis tests to validate the claims made based on the experiments with real human feedback?**\n\n> We include statistical analysis for the real human experiment. Please refer to **Global Response to All Reviewers (Part II)-B.2**. Also, we trained the agent with real human feedback from scratch during the rebuttal period, without using any pre-trained policy or reward models. Please refer to **Global Response to All reviewers - Additional Experiments - Exp3**.\n\n[1] Lee et al., \"PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training\", International Conference on Machine Learning, 2021.\\\n[2] Lee et al. \"B-pref: Benchmarking preference-based reinforcement learning\", Neural Information Processing Systems Track on Datasets and Benchmarks(NeurIPS), 2021."}, {"review_id": "U6tu9MwOcV", "reviewer": "Reviewer_f3sA", "summary": "The authors propose that a novel RLHF framework called SeqRank to improve the feedback efficiency via sequential preference ranking. They propose two methods: sequential pairwise comparison and root pairwise comparison.", "questions": "1 the proposed methods (sequential pairwise comparison, root pairwise comparison) is quite intuitive. Though it achieves better performance than standard pairwise comparison, it might be suboptimal.\n\n2 the definition of $S_M$ is not clear (number of pairs collected from $M$ trajectories). Here “pairs” refers to both directly compared pairs and indirectly inferred pairs (e.g., the number of lines in both black and purple in fig2(b)), is that correct?\n\n3 As for experiments, the numbers of feedbacks different from tasks. How do you choose this parameter? In figure 4, when the number of feedbacks is limited, it’s interesting to see the pairwise comparison outperform the two sequential methods. Is there any comment on this?\n\n4 As for the writing, since the proposed method aims to improve the feedback efficiency, which seems to be defined by the authors, I suggest the author provide some illustrations first to show its importance (e.g., higher efficiency will lead to higher performance) before discussing the proposed method is able to improve it. This might make it more clear.\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clear and writing is easy to follow. The main idea is theoretically sound and the experiment results are promising.", "weaknesses": "The proposed method seems to be relatively simple. But I think it’s okay since the authors prove its effectiveness both theoretically and empirically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the positive and thoughtful review by Reviewer f3sA, _e.g., clear and easy writing, theoretically sound idea, and promising experiment results_. We will address all raised concerns by the reviewer and revise the paper accordingly.\n\n## **Weakness**\n**Weak1: The proposed method seems to be relatively simple. But I think it’s okay since the authors prove its effectiveness both theoretically and empirically.**\n> We are pleased to Reviewer f3sA’s comments that the reviewer recognizes the theoretical soundness of the proposed method, and acknowledges the theoretical and empirical effectiveness of our method. Our intention is indeed to develop a method that is not only simple but also effective. We believe that this simplicity can be a strength, facilitating implementation and adaptability across different domains, while the robust theoretical and empirical results ensure that the proposed method significantly improves the performance in DMControl, Meta-World, and real-world robotic manipulation.\n\n## **Questions**\n**Question1: the proposed methods (sequential pairwise comparison, root pairwise comparison) is quite intuitive. Though it achieves better performance than standard pairwise comparison, it might be suboptimal.**\n\n> Yes, our work can be seen as suboptimal because it is a fundamental research about augmenting preference ranking data without additional human cost. However, we expect various applications of our work by combining the proposed sequential preference ranking with existing sampling methods, i.e., uncertainty-based disagreement sampling. To support our claim, we perform additional experiments by applying disagreement sampling to our method in **Global response to All reviewers - Additional Experiments - Exp2**. Additionally, we can improve the algorithm by using sequential/root pairwise comparison to augment preference data until the data dependency exceeds a threshold value. After exceeding the threshold value, we can switch the trajectory comparison method to pairwise comparison. By scheduling between two or more trajectory comparison methods, we will be able to develop an RLHF algorithm that effectively augments preference data with limited data dependencies.\n\n**Question2: the definition of $S_M$ is not clear (number of pairs collected from trajectories). Here “pairs” refers to both directly compared pairs and indirectly inferred pairs (e.g., the number of lines in both black and purple in fig2(b)), is that correct?**\n> Yes, the definition of $S_M$ includes both the number of directly compared pairs and indirectly compared pairs. We will revise the definition of $S_M$ described in L163 in the paper and Table 1 in the supplementary material as follows:\\\n\\\n$S_M$: the total number of segment pairs from $M$ trajectory segments including both directly and indirectly compared pairs.\n\n**Question3: As for experiments, the numbers of feedbacks different from tasks. How do you choose this parameter? In figure 4, when the number of feedbacks is limited, it’s interesting to see the pairwise comparison outperform the two sequential methods. Is there any comment on this?**\n\n> For the locomotion tasks in DMControl, we perform breadth-first search (BFS) for the number of feedbacks in the range of [200, 40000]. For the manipulation tasks in Meta-World, we perform BFS for the number of feedbacks in the range of [1000, 20000]. For agents with simple dynamics or tasks, we choose a low number in the range and for agents with complex dynamics or tasks, we choose a high number in the range.\\\n\\\nFor the experiments with real human feedback, we set the number of feedbacks as 20 for fine-tuning the policy and the reward model. For the real robot experiments, we perform BFS for the number of feedbacks in the range of [200, 800].\\\n\\\nIn Figure 4, we fix the frequency of reward learning as learning the reward at each 30,000 policy training steps for fair comparison among different numbers of feedback. For a small number of feedback, the increased data dependency affects the performance more than the increased feedback efficiency. In practical cases, we can learn the reward function more frequently to improve performance if the maximum number of feedback is as low as 100. Please refer to **Global Response to All Reviewers (Part I)-B.1**.\n\n**Question4: As for the writing, since the proposed method aims to improve the feedback efficiency, which seems to be defined by the authors, I suggest the author provide some illustrations first to show its importance (e.g., higher efficiency will lead to higher performance) before discussing the proposed method is able to improve it. This might make it more clear.**\n\n>Yes, we will add illustrations in Figure 1 in the main paper to describe that higher efficiency will lead to higher performance. This will definitely improve the understanding about the importance of feedback efficiency in reinforcement learning from human feedback (RLHF)."}]

Author rebuttal:
## **Global Response to All Reviewers (Part I)**

We sincerely appreciate all the reviewers for their thoughtful and constructive comments. We are encouraged that they found our motivation and idea clear, intuitive, and novel (sxrB, Mtqb, f3sA) to solve an important and timely problem in RLHF (tocg). We are glad they found our mathematical analysis on the reward model’s (1) convergence rate of the empirical risk, (2) convergence rate of the generalization bound, and (3) the trade-off between feedback efficiency and data dependency strong, robust, and sound (sxrB, Mtqb, fs3A). All reviewers agreed our method shows significant gains in performance across a wide variety of tasks including 11 different robotic tasks in simulation and block-placing task using a real UR-5 robot (sxrB, tocg, Mtqb, f3sA). We appreciate that they found our experiments with real human feedback and user study novel and very interesting (tocg). We will incorporate all feedback in the final version of our paper.

We address common reviewer comments as a **global response** and specific comments for each reviewer. We refer to line number **xxx** of the main paper as **Lxxx**. Keywords will be highlighted like `this` or **this**. All additional figures are in the `attached pdf`.

We believe that the revisions and additional experiments we have outlined will greatly enhance our paper's quality and impact, addressing the concerns raised. We hope that these responses and our commitment to improving the work will be positively received.

## **A. Responses to Comments for Contents**
**A.1. Moving Algorithm to Main Text (sxrB, tocg)**
>Reviewers sxrB and tocg suggested moving the algorithms to the main text to improve the clarity of our method. We will move Algorithm 1: Sequential Pairwise Comparison RLHF and Algorithm 2: Root Pairwise Comparison RLHF in the supplementary material to Section 4. SeqRank for a clear explanation of the proposed method. To save space, we will illustrate the common framework of both algorithms and specify the trajectory comparison algorithms for sequential and root pairwise comparison. We believe this will enhance the understanding of the proposed algorithms.

**A.2. Overstatement Regarding Cognitive Load (sxrB, tocg)**
>We appreciate the concern that high cognitive load (L030) may seem like an overstatement. For clarity, we will revise the introduction (L029-L033) as follows:\
_“Despite the significant success of RLHF, conventional pairwise comparison requires a human to remember at least two trajectories to determine a single preference. In this regard, it is essential to develop an efficient comparison method that can obtain multiple preference data from a single human feedback, thus reducing a human’s labeling effort.”_

## **B. Additional Experiments and Analyses**
### **B.1. Additional Analysis for Figure 4 in the main paper (sxrB,f3sA)**
>In Figure 4, we illustrate the average reward per number of feedbacks after convergence in DMControl quadruped walk task. We address two major concerns regarding the results in Figure 4.

**(1) When the number of feedback is limited (=100), why does pairwise comparison outperform sequential and root pairwise comparison? (f3sA-Question3)**
>In Figure 4, we fix the frequency of reward learning as learning the reward at each 30,000 policy training steps for fair comparison among different numbers of feedback. The results in Figure 4 imply that for a small number of feedback (=100), the increased data dependency affects the performance more than the increased number of augmented data. However, as the number of feedback gets sufficiently high, the number of augmented preference data affects the performance more than data dependency. Especially for root pairwise comparison, the performance when the number of feedback is 1000 is higher than the performance of pairwise comparison when the number of feedback is 4000. In practical cases, we can learn the reward function more frequently to improve performance if the maximum number of feedback is as low as 100.

**(2) When the number of feedback is high, why does root pairwise comparison outperform the oracle SAC model? (sxrB-Nitpicks and Suggestions 5)**
>A similar phenomenon is shown in previous work [1] that with a sufficient number of feedback, an RLHF agent performs better than an RL agent trained with true rewards. The authors claim that one possible cause of this outperforming behavior is that the reward learning process assigns positive rewards to all behaviors that are typically followed by high rewards. Numerous studies in RL [2, 3] also indicate that learning dense rewards can improve the overall performance of the agent compared to sparse rewards. We believe the strength of reinforcement learning from human feedback also appears in this result, that we can learn a plausible dense reward that potentially leads the agent to perform better.\
\
While RLHF can outperform the oracle model with a sufficient number of feedback, the proposed root pairwise comparison significantly boosts the speed to approach the oracle performance. This is because the high feedback efficiency of root pairwise comparison helps the agent learn the reward model faster than other methods. We address the convergence rate of the reward model in Section 4.2 on page 6 of the main paper.

`Global Responses to All Reviewers are continued (Part II - V) in the following official comments. Dear reviewers, please let the authors know if Part II - V are not visible. We expect the earliest access time would be Aug 9th 11PM PST.`

[1] Christiano et al., "Deep reinforcement learning from human preferences.", Advances in neural information processing systems, 2017.\
[2] Faust et al., "Evolving rewards to automate reinforcement learning.", Workshop on Automated Machine Learning at International Conference on Machine Learning, 2019.\
[3] Zheng et al., "What can learned intrinsic rewards capture?" International Conference on Machine Learning, 2020.

Meta-review: The meta reviewer went through the paper, reviews, responses and agrees with the majority of reviewers that this paper reached the bar of NeurIPS.
Decision: Accept (poster)