Title: Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning; Reinforcement Learning from Human Feedback; Preference-based Reinforcement Learning; Human-Robot Interaction
Abstract: Reinforcement learning from human feedback (RLHF) alleviates the problem of designing a task-specific reward function in reinforcement learning by learning it from human preference. However, existing RLHF models are considered inefficient as they produce only a single preference data from each human feedback. To tackle this problem, we propose a novel RLHF framework called SeqRank, that uses sequential preference ranking to enhance the feedback efficiency. Our method samples trajectories in a sequential manner by iteratively selecting a defender from the set of previously chosen trajectories $\mathcal{K}$ and a challenger from the set of unchosen trajectories $\mathcal{U}\setminus\mathcal{K}$, where $\mathcal{U}$ is the replay buffer. We propose two trajectory comparison methods with different defender sampling strategies: (1) sequential pairwise comparison that selects the most recent trajectory and (2) root pairwise comparison that selects the most preferred trajectory from $\mathcal{K}$. We construct a data structure and rank trajectories by preference to augment additional queries. The proposed method results in at least 39.2% higher average feedback efficiency than the baseline and also achieves a balance between feedback efficiency and data dependency. We examine the convergence of the empirical risk and the generalization bound of the reward model with Rademacher complexity. While both trajectory comparison methods outperform conventional pairwise comparison, root pairwise comparison improves the average reward in locomotion tasks and the average success rate in manipulation tasks by 29.0% and 25.0%, respectively. The source code and the videos are provided in the supplementary material.

Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback

Abstract

Reinforcement learning from human feedback (RLHF) alleviates the problem of designing a task-specific reward function in reinforcement learning by learning it from human preference. However, existing RLHF models are considered inefficient as they produce only a single preference data from each human feedback. To tackle this problem, we propose a novel RLHF framework called SeqRank, that uses sequential preference ranking to enhance the feedback efficiency. Our method samples trajectories in a sequential manner by iteratively selecting a defender from the set of previously chosen trajectories K and a challenger from the set of unchosen trajectories U \ K , where U is the replay buffer. We propose two trajectory comparison methods with different defender sampling strategies: (1) sequential pairwise comparison that selects the most recent trajectory and (2) root pairwise comparison that selects the most preferred trajectory from K . We construct a data structure and rank trajectories by preference to augment additional queries. The proposed method results in at least 39 . 2% higher average feedback efficiency than the baseline and also achieves a balance between feedback efficiency and data dependency. We examine the convergence of the empirical risk and the generalization bound of the reward model with Rademacher complexity. While both trajectory comparison methods outperform conventional pairwise comparison, root pairwise comparison improves the average reward in locomotion tasks and the average success rate in manipulation tasks by 29 . 0% and 25 . 0% , respectively. Project page: https://rllab-snu.github.io/projects/SeqRank

1 Introduction

Designing a suitable reward function in reinforcement learning often requires task-specific prior knowledge [1] and sufficient time to design the reward function to capture the true task objective. Without this effort, the agent may be easily led to suboptimal policies. These limitations motivate the use of reinforcement learning from human feedback (RLHF), which can directly learn from human's preferences without the need for a hand-crafted reward function [1]. A conventional way to learn a reward function in RLHF is pairwise comparison: querying human preference between two different trajectories as described in Figure 1. Despite the significant success of RLHF, conventional pairwise comparison requires a human to remember at least two trajectories to determine a single preference.

𝑟(𝜎 1 ) = 1

𝜎1

𝜎2

𝑟(𝜎2) = 0

Environment

Agent

Human

Learn

Policy

Learn

Reward

𝜎1

≻ 𝜎2

preference label

𝑦 = 1

Action

In this regard, it is essential to develop an efficient comparison method that can obtain multiple preference data from a single human feedback, thus reducing a human's labeling effort. In this paper, we define the concept of feedback efficiency as the ratio between the number of trajectory pairs over the number of feedbacks , where the feedback efficiency of conventional pairwise comparison is set at a standardized level. We aim to develop a novel RLHF framework with greater feedback efficiency than existing methods.

The key idea of our approach is to utilize the preference relationships of the previous trajectory pairs. Bringing the nature of transitivity in human preferences [2], given three trajectories, σ i , σ j , and σ k , a human with preference over σ i to σ j and σ j to σ k also prefers σ i over σ k . Hence, if we maintain the history of preference relationships, we can easily augment preference data without additional human feedback. In the proposed method, the comparison process maintains two sets of trajectories. One collects previously compared trajectories, and the other includes remaining trajectories that have not yet been compared. We iteratively select and compare a defender trajectory from the set of previously chosen trajectories and a challenger trajectory from the set of unchosen trajectories. In this paper, we propose two trajectory comparison methods: (1) sequential pairwise comparison that selects the most recently sampled trajectory and (2) root pairwise comparison that selects the previously most preferred trajectory as the defender. For both methods, the agent can augment additional preference data from the comparison results due to the transitivity.

We show that the proposed sequential preference ranking substantially improves the feedback efficiency and significantly outperforms conventional pairwise comparison. We prove that the proposed method converges faster for the empirical risk of the reward model than conventional pairwise comparison. One drawback of the proposed method is that it generates dependencies within the original and the augmented preference data, which are both used to train the reward model. Still, we prove the generalization bound of the reward model with Rademacher complexity converges with rates of O ( √ (ln( T )) 2 /T ) and O ( √ ln( T ) /T ) for sequential and root pairwise comparison, respectively, where T is the number of iterations. We demonstrate the effectiveness of the proposed method in both locomotion tasks from Deepmind Control Suite (DMControl) [3] and manipulation tasks from Meta-World [4]. Among the proposed two trajectory comparison methods, root pairwise comparison shows the highest performance in all tasks, which is improved by 29 . 0% and 25 . 0% against the pairwise baseline in locomotion and manipulation tasks, respectively. Additional experiments using real human feedback validate the effectiveness of the proposed method, showing significant improvements in true reward, sample efficiency, feedback efficiency, and user satisfaction. Furthermore, our method successfully outperforms the baseline in terms of success rates in controlling a real UR-5 robot for a block placing task, demonstrating its practical applicability in the real world.

The main contributions of this paper are as follows:

We propose a novel RLHF framework that utilizes sequential preference ranking to enhance human feedback efficiency. We prove the proposed sequential and root pairwise comparison substantially improve the average feedback efficiency by 39 . 2% and 100% , respectively. An increase in feedback efficiency speeds up the estimation of the reward function.

We derive the convergence rates of the empirical risk and the generalization bound of the reward model with Rademacher complexity using the proposed sequential and root pairwise comparison. We address the trade-off between feedback efficiency and data dependency required for successful reward learning.

We empirically show that prioritizing the feedback efficiency is significantly important by evaluating in simulation and real-world environments. Both sequential and root pairwise

comparison outperform conventional pairwise comparison on average. Root pairwise comparison shows the most substantial improvement against the baseline by 29 . 0% and 25 . 0% in DMControl locomotion and Meta-World manipulation tasks, respectively.

2 Related Work

Reinforcement Learning from Human Feedback. A common approach in reinforcement learning from human feedback, i.e., preference-based reinforcement learning, is to learn the reward function from explicit feedback, such as ratings or comparisons provided by human experts. Christiano et al. [5] present the human-in-the-loop reinforcement learning framework for robotic tasks. Lee et al. (2021a) [6] propose a benchmark that introduces simulated human teachers with various irrationalities. Recent work [7-10] introduce efficient RLHF algorithms. Lee et al. (2021b) [7] propose a method that improves Christiano et al. [5] via unsupervised pre-training and off-policy RL with relabeling. Park et al. [8] suggest a method that uses semi-supervised learning and data augmentation. Liang et al. [9] show that exploration based on uncertainty in learned reward functions helps to improve the performance with fewer samples and feedbacks. Liu et al. [10] propose a data-efficient RLHF framework incorporating bi-level optimization for reward and policy learning.

However, most of the previous studies still achieve low feedback efficiency as they do not take into account the nature of ranking through trajectory comparisons. Several recent work [11-13] suggest querying a complete ranking among multiple ( ≥ 3 ) trajectories, but this assigns the responsibility of ranking trajectories to humans, not robots. Requesting a human to remember multiple trajectories at once and provide best-of-multiple feedback [11] is also challenging due to human uncertainty [14]. To solve this challenge, we propose sequential preference ranking, which allows the agent to automatically rank trajectories while a human only provides pairwise comparisons based on one's preference. The key idea is to perceive piecewise ranks instead of ranking the trajectories as a whole.

Generalization Bounds for Learning Under Graph-Dependence. While data augmentation via sequential preference ranking can improve the efficiency of reward learning, this can lead to dependencies within the training data. However, traditional learning theories are based on the assumption that data are identically and independently distributed (i.i.d.). Hence, to analyze the generalization bound of the proposed method, we investigate the generalization bounds for learning with data dependencies. A common approach to consider data dependency is to construct a graph that characterizes the dependency relationship within the data [15]. Janson et al. [16] establish that the probability of the sum of graph-dependent random variables deviating from its expected value is bounded based on the fractional chromatic number of the dependency graph, by extending Hoeffding's inequality. Usunier et al. [17] present a concentration inequality by extending fractional Rademacher complexity, and prove generalization bounds for binary classification over interdependent data. Zhang et al. (2019) [18] propose McDiarmid-type concentration inequalities for Lipschitz functions of graph-dependent random variables and show that concentration relies on the forest complexity of the dependency graph. In this paper, we derive the generalization bound of the reward model with Rademacher complexity in terms of the maximum degree of the dependency graph and the number of training data, which includes augmented preference queries and labels.

3 Learning the Reward Function from Human Preferences

Following prior work [5, 7, 10, 19], we use a deterministic human model that provides preference as a binary feedback over a pair of trajectory segments. We define the true preference label y for a pair of segments ( σ i , σ j ) as follows:

where r is the true reward function of a state-action pair ( s i t , a i t ) in σ i at timestep t and H is the length of a segment. The true reward of a trajectory segment σ = ( s 1 , a 1 , ..., s H , a H ) can be written as r ( σ ) = ∑ t r ( s t , a t ) . Similar to prior work [13, 20-22], we implement our algorithm with a linearly parameterized preference model. Assume that there exist a nonlinear trajectory encoder ϕ : T → R d and a parameter θ ∗ ∈ R d that represents the true reward function as r ( σ ) := θ ∗ ⊺ ϕ ( σ ) , where d is the dimension of θ ∗ and T is a set of trajectory

Pairwise

Comparison

Comparison

Comparison

Sample

2𝑁

trajectories

Sample

𝑁 + 1

trajectories

Constructed Tree

𝟏

𝟐

𝟑

𝟒

𝟏

𝟐

𝟑

𝟒

𝟓

Compare

the previously most preferred & current trajectory

vs.

𝑵

𝑵+𝟏

𝟏

𝟐

𝟑

𝟒

𝟐

𝟑

𝟒

𝟓

𝑵

𝑵+𝟏

𝑵+𝟏

𝑵+𝟐

𝑵+𝟑

𝑵+𝟒

𝟐𝑵

𝟏

𝟐

𝟑

𝟒

𝑵

Sample

𝑁+1

trajectories

(a)

Three trajectory comparison methods.

Comparison

𝜎1

𝜎2

𝜎3

𝜎4

𝜎5

𝜎6

𝜎7

𝜎8

𝜎9

𝜎10

Comparison

𝜎5

𝜎6

𝜎10

𝜎8

high

reward

𝜎4

𝜎1

𝜎2

𝜎3

low

high

reward

low

𝜎9

𝜎7

(b)

Sequential vs Root

Pairwise Comparison

segments. We estimate the reward using a learnable reward function ˆ r θ , parameterized by θ . Then, the estimated reward of σ can be written as ˆ r θ ( σ ) = ∑ t ˆ r θ ( s t , a t ) = θ ⊺ ϕ ( σ ) . Preference prediction is also interpreted as a binary classification problem with a class probability P ( y = 1 | ( σ i , σ j ); θ ∗ ) = P θ ∗ [ σ i ≻ σ j ] = exp( ∑ t r ( s i t , a i t )) / ∑ k ∈{ i,j } exp( ∑ t r ( s k t , a k t )) based on the Bradley-Terry model [23], where σ i ≻ σ j denotes that the i th trajectory segment is preferable to the j th trajectory segment. For training the reward model, we use P (ˆ y = 1 | ( σ i , σ j ); θ ) = P θ [ σ i ≻ σ j ] = exp( ∑ t ˆ r θ ( s i t , a i t )) / ∑ k ∈{ i,j } exp( ∑ t ˆ r θ ( s k t , a k t )) to estimate the preference label ˆ y of ( σ i , σ j ) . If P θ [ σ i ≻ σ j ] is greater than 0 . 5 , ˆ y is 1 . Otherwise, ˆ y is 0 . Using cross-entropy loss, we define the preference loss function L pref as follows:

where D is the set of sampled trajectory segment pairs and their corresponding true preference labels.

In traditional RLHF methods, a common approach to sampling a segment pair is to randomly sample each segment from the replay buffer, resulting in the selection of irrelevant pairs. This independent sampling approach does not take into account any inherent relationships between the non-adjacent segments, where two segments are considered adjacent if they are compared directly from a human. In contrast, the proposed method adopts a sequential sampling approach that enables indirect inference of the relationship between non-adjacent segments. The proposed method allows for more comprehensive understanding of the preferences among trajectory segments, capturing both direct and indirect relationships.

4 SeqRank

We present SeqRank, which samples trajectories and compares them in a sequential manner for efficient reinforcement learning from human feedback. We also extend our key component, sequential pairwise comparison, to root pairwise comparison, which not only compares trajectories sequentially but also constructs a tree structure to rank trajectories in a more efficient way. Figure 2 (a) illustrates the difference between three trajectory comparison methods including conventional pairwise comparison and the proposed sequential and root pairwise comparison. We demonstrate the feedback efficiency and convergence of the reward model for each method. Regarding the notations used in our paper, a summary of symbols and terms can be found in Table 1 in the supplementary material.

We first make the following assumptions:

(Uniformness) Every trajectory is sampled uniformly from the replay buffer.

(Completeness) For any two segments σ i and σ j , ' σ i ≻ σ j or σ i ≺ σ j ' is true.

(Transitivity) For any three segments σ i , σ j , and σ k , if σ i ≻ σ j and σ j ≻ σ k , then σ i ≻ σ k .

Assuming N feedback per reward update, the agent samples the ( n +1) th defender and challenger from K n = { σ 1 , ..., σ n } and U\K n , respectively, where U is the replay buffer. The defender becomes σ n for sequential pairwise comparison and arg max σ ∈K n θ ⊺ ϕ ( σ ) for root pairwise comparison. After the N th query, the agent receives N feedback in total. Figure 2 (b) illustrates the preference relationships among trajectory segments as a graph. Especially for root pairwise comparison, the challenger becomes a new root of the tree if it is preferred over the current root segment as described in Figure 2 (a). Otherwise, the challenger becomes a new child of the current root. The full procedure of our algorithm is provided in the supplementary material. Suppose S M is the total number of trajectory segment pairs obtained from M trajectory segments. M = 2 N for conventional pairwise comparison, and M = N +1 for sequential or root pairwise comparison. In the following section, we first demonstrate the feedback efficiency in terms of M and substitute it with the function of N .

4.1 Feedback Efficiency of the Algorithm

From the uniformness assumption, suppose the agent compares M trajectory segments with corresponding rewards as a random ordering of 1 , 2 , ..., M . Based on the completeness and transitivity assumptions, the agent can augment non-adjacent segment pairs as illustrated in Figure 2 (b). Considering that the number of augmented pairs differs by the order of sampling trajectories with different rewards, S M becomes a random variable indicating the total number of pairs collected from M trajectory segments. In this section, we demonstrate the average feedback efficiency of our method.

Definition 1. We define feedback efficiency as the ratio between the total number of trajectory pairs over the number of feedbacks. The expected feedback efficiency for N human feedback is E [ S N +1 ] /N for sequential and root pairwise comparison.

Sequential Pairwise Comparison. Following Definition 1, we use Lemma 1 to examine the expected number of segment pairs collected from sequential pairwise comparison.

Theorem 1. For any integer M ≥ 2 , the expected number of trajectory segment pairs collected from sequential pairwise comparison a M is approximately linear to M as | a M /M -1 / ( e -2) | = o (1) .

Lemma 1. For any integer M ≥ 2 , a M can be induced from the following recurrence relation.

(total number of pairs from all possible orderings of trajectories)

By solving the recurrence relation stated in Lemma 1, we derive Theorem 1 that a M is approximately equal to a linear equation α ( M -γ ) with an error less than ϵ , for any M such that M ! is greater than 2 γ/ϵ . α = 1 / ( e -2) ≊ 1 . 392 and γ > 1 , respectively. By substituting a sufficiently large value of M ≥ 100 , γ is calculated as 1 . 324 , and the expected feedback efficiency of sequential pairwise comparison becomes 1 . 392( M -1 . 324) / ( M -1) , which converges to 1 . 392 as M increases. Moreover, for any M ≥ 4 , the linear approximation error of a M is bounded by 0 . 003 . Detailed proofs of Lemma 1 and Theorem 1 are in the supplementary material.

Comparing the proposed sequential pairwise comparison with conventional pairwise comparison that requires 2 N trajectory segments for N human feedback, we observe that sequential pairwise comparison only requires N +1 trajectory segments to query a human for N feedback. Additionally, using the proposed sequential pairwise comparison, the expected number of augmented queries is given by a N +1 -N ≊ 0 . 392 N -0 . 451 .

Extension to Root Pairwise Comparison. We further extend sequential pairwise comparison to root pairwise comparison by exploiting a specific defender sampling strategy. The key idea is to compare the current segment with the most preferred one from the previously sampled segments. As described in Figure 2 (b), root pairwise comparison augments more pairs than sequential pairwise comparison by constructing a tree structure. Using root pairwise comparison, the average feedback efficiency is increased to two. Theorem 2 demonstrates the average feedback efficiency of our algorithm using root pairwise comparison. The detailed proof of Theorem 2 is in the supplementary material.

Considering that ∑ M n =1 1 n < 1 + ∫ M 1 1 x dx = 1 + ln M , ( ∑ M n =1 1 n ) / ( M -1) converges to 0 as M increases. Therefore, root pairwise comparison achieves the expected feedback efficiency as b M / ( M -1) ≊ 2 . This implies that the proposed root pairwise comparison achieves higher feedback efficiency than sequential pairwise comparison on average.

Table 1 shows the comparison results of trajectory comparison methods when the number of human feedback is fixed to N . M and p N denote the number of sampled trajectories at each reward update session and the expected number of trajectory pairs obtained from N human feedback, i.e., E [ S N ] , respectively. For sequential and root pairwise comparison methods, p N is a N +1 and b N +1 , respectively. The baseline method using conventional pairwise comparison samples 2 N trajectories, which are approximately twice as many of sequential or root pairwise comparison methods. As shown in Table 1, sequential pairwise comparison achieves an average feedback efficiency of 1 . 392 . Also, root pairwise comparison achieves the highest average feedback efficiency of 2 .

4.2 Convergence of the Reward Model

Notations. Suppose ( ϕ ( σ i ) -ϕ ( σ j )) is the input and σ i ≻ σ j is the true label of a binary classification using a logistic regression model and a cross-entropy loss. Then, P θ [ σ i ≻ σ j ] can be interpreted as a logistic regression model as follows:

For simplicity, we define the training data for reward update as D = { ( x k , y k ) } , where x k and y k denote the features of two trajectories included in the k th query and its corresponding preference label provided from a human, respectively. Note that if the k th query is ( σ i , σ j ) , x k = ϕ ( σ i ) -ϕ ( σ j ) . We define the true risk of the reward model as R ( θ ) := E [ ℓ ( θ ∗ ⊺ ( x ) , θ ⊺ ( x ))] and the empirical risk of the reward model as ˆ R ( θ ) := L pref = (1 / |D| ) ∑ x k ∈D ℓ ( θ ∗ ⊺ ( x k ) , θ ⊺ ( x k )) , where the cross-entropy loss and its minimizer are denoted as ℓ and θ ∗ , respectively. Assume that || ϕ || 2 , || θ ∗ || 2 , and ||∇ θ || 2 are bounded with constants S , D , and Q , respectively. Since RLHF involves an iterative process of policy and reward update, we distinguish the overall iteration T that represents the global iterative process, and the local iterations of policy and reward update, W and U , respectively. S M,T represents the cumulative sum of the number of training data of the reward model from iteration 1 to T , with M segments compared per iteration. We introduce the concept of dependency graph and its maximum degree ∆ M,t at iteration t ∈ [ T ] = { 1 , 2 , ..., T } for further analysis, following previous work [15]. Detailed proofs for the following theorems, lemmas, and corollaries are in the supplementary material.

High Probability Feedback Efficiency. We find a high probability bound for S M,T in terms of the expected values of S M as described in Lemma 2. This implies that for sufficiently large T and N , the feedback efficiency averaged across T iterations is at least equal to p N / (2 N ) in the proposed method with high probability. For sequential and root pairwise comparison, the lower bounds become 0 . 696 and 1 , respectively, while the feedback efficiency for pairwise comparison is fixed to 1 .

Lemma 2. For a sufficiently large T > (max t ∆ M,t +1)( M -1) ln(1 /δ ) / ( β 2 M ) and δ ∈ (0 , 1) , with probability at least 1 -δ , S M,T is greater than βTM/ 2 , where β denotes the expected number of pairs obtained from M segments divided by M .

Convergence Rate of the Empirical Risk. From the completeness assumption, data in D should be linearly separable. According to prior work [24], the empirical risk converges with a rate of O (1 /U ) . Let n B be the batch size of reward update. Then, after the global iteration T , U = ⌊ S M,T /n B ⌋ .

From Lemma 2, we prove that with probability at least 1 -δ and a sufficiently large T , the empirical risk of the reward model converges at a rate of O (2 n B / ( βTM )) . Therefore, higher β results in faster convergence. While ( β, M ) for pairwise comparison are fixed as (0 . 5 , 2 N ) , the values for sequential and root pairwise comparison are approximated to (1 . 392 , N +1) and (2 , N +1) , respectively. Based on our analysis, the reward model is likely to converge faster in the order of root pairwise, sequential pairwise, and pairwise comparison in terms of the global iteration T .

Convergence Rate of the Generalization Bound. Based on prior work [15, 25], we derive a generalization bound with Rademacher complexity for learning the reward model under graphdependence in D . Theorem 3 shows the generalization bound of the reward model. Detailed proof of the theorem and definitions of F and B are presented in the supplementary material.

Theorem 3. Assume that a comparison algorithm generates a train set D of size S M,T with dependency graph G after T iteration. Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ ,

where H is the parameter space of θ , and F and B are functions of M with rates at O ( M 2 ) .

Theorem 3 addresses a critical aspect of the proposed method by proving that even with dependent data, the empirical risk of the reward model converges to the true risk with probability at least 1 -δ . This high probability bound contributes to the practical analysis of general RLHF models with graph-dependent data such as rankings. Additionally, we consider the trade-off between feedback efficiency ( ∝ S M,T ) and data dependency ( ∝ ∆ M,t ). As shown in (6), lower ∆ M,t and larger S M,T lead to a tighter generalization bound of the reward model. However, data dependency increases in our settings as we augment more queries and enhance feedback efficiency. Hence, we demonstrate ∆ M,t for sequential and root pairwise comparison in Lemma 3 and 4, respectively.

Lemma 3. Let G be a dependency graph generated by sequential pairwise comparison with M trajectory segments. Then, for any δ ∈ (0 , 1) , the maximum degree of G is bounded as follows, with probability at least 1 -δ ,

Lemma 4. Let G be a dependency graph generated by root pairwise comparison with M trajectory segments. Then, for any δ ∈ (0 , 1) , the maximum degree of G is bounded as follows, with probability at least 1 -δ ,

Corollary 3.1. For a fixed M ≥ 2 , the generalization bounds of the reward model for pairwise, sequential pairwise, and root pairwise comparison converge at rates of O ( √ ln( T ) /T ) , O ( √ (ln( T )) 2 /T ) , and O ( √ ln( T ) /T ) , respectively, with probability at least 1 -1 /T .

By applying Lemma 2, 3, and 4 to Theorem 3, we derive Corollary 3.1, which compares the generalization bounds of three methods. While generalization bounds of all methods are guaranteed to converge as T increases, root pairwise comparison demonstrates a faster convergence rate than sequential pairwise comparison, and has the same convergence rate as pairwise comparison.

Overall analysis of convergence. To sum up, root pairwise comparison theoretically surpasses the baseline as it proves better and equal convergence rates of both metrics, while requiring a half of samples than pairwise comparison. Sequential pairwise comparison achieves faster convergence of the empirical risk but slower convergence of the generalization bound compared to pairwise comparison. From the perspective of reward model convergence, prioritizing the feedback efficiency to speed up the convergence of the empirical risk is more critical than considering the generalization error in worst-case scenarios. In Section 5, we empirically show the superiority of root pairwise comparison over sequential and pairwise comparison.

5 Experiments

We evaluate our method on robotic locomotion tasks from DeepMind Control Suite (DMControl) [3, 26] and robotic manipulation tasks from Meta-World [4]. We also demonstrate the proposed

method in the real-world using a UR-5 robot for a block placing task. Experiment details and additional analyses with rendered results are provided in the supplementary material.

5.1 Experiment Settings

Tasks. Locomotion tasks from DMControl consist of walker walk, cheetah run, quadruped walk, humanoid walk, and hopper hop. Manipulation tasks from Meta-World consist of button press, door open, drawer open, sweep into, window open, and hammer. For each task, we train with 10 different seeds used in prior work [6, 10] and measure the average performance with a standard deviation. Evaluation metrics are chosen as the ground-truth return per episode for DMControl and the success rate for Meta-World. Additionally, we conduct experiments with real human feedback for 30 participants using policy and reward models pre-trained in the simulation environment. We also demonstrate a block placing task using a UR-5 robot in the real-world, by pre-training the policy in the Mujoco simulator [27] and fine-tuning in the real-world. Detailed experiment settings are provided in the supplementary material.

Baselines and Implementation Details. We use three baselines for all tasks: SAC [28], MetaReward-Net (MRN) [10], and PEBBLE [7]. Agents learned via SAC with the true reward functions are considered as the upper bound of all tasks. We use MRN and PEBBLE as the baseline methods using the traditional trajectory comparison method, pairwise comparison. MRN utilizes bi-level optimization for both reward and policy learning and achieves state-of-the-art among previous methods. We apply our sequential preference ranking algorithm on both MRN and PEBBLE to show the effectiveness of our algorithm. Results with MRN baseline are described in this section and results with PEBBLE baseline are provided in the supplementary material. We use unsupervised pre-training [7] for 9 , 000 steps for all experiments. The trajectory encoder is implemented using a two-layer feed-forward neural network, where the input dimension is the combined size of the state and action spaces, and the output dimension is set to 256 . The linear reward model then takes the encoded feature and passes it through a single fully connected neural network. For each task, hyperparameters and implementation details are described in the supplementary material.

5.2 Simulation Experiments

Locomotion Tasks from DMControl. Results in Table 2 demonstrate the superiority of the proposed root pairwise comparison over the baseline pairwise comparison and sequential pairwise comparison in all locomotion tasks. The reward obtained after convergence is higher by an average of 29 . 0% compared to conventional pairwise comparison used in MRN [10]. Sequential pairwise comparison, on the other hand, does not exceed root pairwise comparison, but it improves performance against pairwise comparison by 10 . 3% on average. Based on the standard deviation of rewards after convergence across 10 different runs, root pairwise comparison shows higher stability than sequential

(a)

(feedback=1K)

(b)

(feedback=4K)

(c)

Drawer Open

(feedback=20K)

(d)

Window open

(feedback=1K)

300

100

100

008

250

80

80

600

premad

200

150

uccess Rate

60

ccess Rate

60

400

100

40

40

200

50

20

20

0.0

0.2

0.4

Sep

0.6

0.8

1.0

1e6

0.0

0.2

0.4

Step

0.6

0.8

1.0

1e6

00

0.2

0.4

Step

0.6

0.8

1.0

1e6

0.0

0.2

0.4

sep

0.6

0.8

1.0

1e6

(a)

Average Rewards

1

2

3

4

5

(b)

User Satisfaction Score

Pairwise

Sequential

Root

Fine-tuning with 20 feedback

Trainingwith200feedback

200

400

600

800

AverageRewards

20

15

10

5

0

pairwise comparison. Reward graphs in Figure 3 (a) and (b) show that root pairwise comparison enables faster convergence of the policy learning.

Additionally, we assess the converged reward for various feedback maximums (0.1K, 1K, 2.5K, 4K) in the quadruped walk task, as illustrated in Figure 4. The graph reveals that both sequential and root pairwise comparison methods converge to higher average reward values and lower variance as the number of feedbacks increases. This implies that with a sufficient number of feedbacks, the amount of augmented preference data affects performance more than data dependency does. By taking advantage of learning a dense reward [5, 29, 30], root pairwise comparison outperforms the oracle baseline with a sufficient number of feedbacks.

900

Reward afterConvergence

800

700

009

500

400

SAC(oracle)

pairwise

300

sequential (M=N+1)

200

root (M=N+1)

0

500

1000

1500

2000

2500

3000

3500

4000

Number of Feedback

Manipulation Tasks from Meta-World. Results in Table 3 show the comparison results with baseline in manipulation tasks. The proposed method using root pairwise comparison outperforms the methods using pairwise comparison and sequential pairwise comparison in all tasks, improving the overall success rate by 25 . 0% compared to the pairwise comparison baseline. Reward graphs in Figure 3 (c) and (d) show that root pairwise comparison enables faster convergence of the policy. Unlike the locomotion tasks, sequential pairwise comparison only improves the success rate by 1 . 2% over pairwise comparison. This may be due to the generalization error using sequential pairwise comparison. Comparing the standard deviation of success rates among 10 different runs, root pairwise comparison achieves the lowest standard deviation. This implies that root pairwise comparison is the most stable algorithm for manipulation tasks among three trajectory comparison methods.

Experiments with Real Human Feedback. To compare the performance and the user stress levels associated with three different methods: pairwise, sequential pairwise, and root pairwise comparison, we conduct experiments involving 35 real human participants. 5 participants trained the policy and the reward models from scratch with 200 human feeedbacks. Other participants fine-tuned the models with 20 human feedbacks, where the models were pre-trained with simulated feedbacks. Figure 5 (a) shows that the proposed methods outperform the baseline in both training from scratch and fine-tuning experiments. Especially for training the models from scratch, root and sequential

pairwise comparison outperform conventional pairwise comparison by 47 . 7% and 38 . 6% , respectively. Quantitative results are provided and analyzed in the supplementary material. Additionally, as a user study, we asked the participants to rate their user satisfaction on a scale from one to five, with higher scores indicating greater satisfaction and lower stress levels. Figure 5 (b) illustrates that the average user satisfaction scores resulted in 2 . 20 , 3 . 00 , and 3 . 93 for pairwise, sequential, and root pairwise comparison, respectively. We perform paired-sample t-test and the differences between the scores are statistically significant for all cases, with the average p-value 0 . 035 . Specifically, users responded that it is convenient to compare with one's most preferred trajectory. These results imply that root pairwise comparison is the least burdensome for real human users while achieving the highest performance.

5.3 Real Robot Experiments

Block Placing Task. To demonstrate our method in real-world environments, we conduct a block placing task using a real UR-5 robot. As illustrated in Figure 6, the goal of this task is to place a block at a given target position starting from a randomly chosen initial position. For faster training and stable movements of the agent, we train the policy with a deep Q-learning network (DQN) and discrete action space following prior work [31]. In this setting, an action is given by the moving direction ψ , where ψ is the heading direction of the end effector discretized into eight angle bins { 0 , π 4 , 2 π 4 , ..., 7 π 4 } . Once ψ is determined, the agent changes its pose and moves the end effector for a fixed distance in the angle of ψ . The policy is pre-trained in the simulator for 20,000 steps and fine-tuned in the real-world for 3,000 steps. The reward model is initialized before fine-tuning. Results in Table 4 show the comparison results with baseline after fine-tuning in the real-world. Using root pairwise comparison, the agent exhibits the highest success rate and the reward model achieves the fastest convergence with the highest accuracy among three trajectory comparison methods.

6 Conclusion

We propose a novel RLHF framework called SeqRank that utilizes sequential preference ranking, resulting in a substantial improvement in the feedback efficiency by at least 39 . 2% . Both sequential and root pairwise comparison enables faster estimation of the reward function. We also prove the convergence rates of the empirical risk and the generalization bound of the reward model for each method. Theoretical and experimental results show that both methods outperform conventional pairwise comparison, with root pairwise comparison showing the most substantial improvement against the baseline. The proposed method and theoretical analyses can be applied to other domains and serve as a practical solution to enhance the feedback efficiency and performance of RLHF. Potential negative societal impacts include amplifying existing biases present in human preferences. We hope our method encourages RLHF to be used for social good without violating ethics.

Limitations. As we experimentally show that the proposed root pairwise comparison achieves the highest reward accuracy among three trajectory comparison methods, we expect to find a tighter generalization bound of the reward model. As the generalization bound of logistic regression with dependent data is not fully studied yet, it will be our future work.

Acknowledgments and Disclosure of Funding

This work was in part supported by Institute of Information & Communications Technology Planning &Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01190, [SW Star Lab] Robot Learning: Efficient, Safe, and Socially-Acceptable Machine Learning, 66%, No. 2022-000331, Development of emotion recognition/generation-based interacting edge device technology for mental health care, 17%, and No. 2021-0-01341, AI Graduate School Program, CAU, 17%).

References

Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research , 18(136):1-46, 2017. 1

Michel Regenwetter, Jason Dana, and Clintin P Davis-Stober. Transitivity of preferences. Psychological review , 118(1):42, 2011. 2

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018. 2, 7

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of the Conference on robot learning (CoRL) , 2020. 2, 7

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems (NeurIPS) , 2017. 3, 9

Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS) , 2021. 3, 8

Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. Proceedings of the International Conference on Machine Learning (ICML) , 2021. 3, 8

Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semisupervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. Proceedings of the International Conference on Learning Representations (ICLR) , 2022. 3

Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. Proceedings of the International Conference on Learning Representations (ICLR) , 2022. 3

Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. 3, 8

Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. Proceedings of the Robotics: Science and Systems (RSS) , 2019. 3

Erdem Bıyık, Daniel A Lazar, Dorsa Sadigh, and Ramtin Pedarsani. The green choice: Learning and influencing human decisions on shared roads. In Proceedings of the IEEE Conference on Decision and Control (CDC) . IEEE, 2019.

Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In Proceedings of the International Conference on Machine Learning (ICML) , 2022. 3

Erdem Bıyık, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. Proceedings of the Conference on Robot Learning (CoRL) , 2019. 3

Rui-Ray Zhang and Massih-Reza Amini. Generalization bounds for learning under graph-dependence: A survey. arXiv preprint arXiv:2203.13534 , 181:109272, 2022. 3, 6, 7

Svante Janson, Tomasz Luczak, and Andrzej Rucinski. An exponential bound for the probability of nonexistence of a specified subgraph in a random graph . Institute for Mathematics and its Applications (USA), 1988. 3

Nicolas Usunier, Massih-Reza Amini, and Patrick Gallinari. Generalization error bounds for classifiers trained with interdependent data. In Advances in Neural Information Processing Systems (NeurIPS) , 2005. 3

Rui Ray Zhang, Xingwu Liu, Yuyi Wang, and Liwei Wang. Mcdiarmid-type inequalities for graphdependent variables and stability bounds. Advances in Neural Information Processing Systems (NeurIPS) , 2019. 3

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems (NeurIPS) , 2018. 3

Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for preference-based reinforcement learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI) , 2020. 3

Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: reinforcement learning with trajectory preferences. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2021. 3

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952. 4

Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300 , 2018. 7

Massih-Reza Amini and Nicolas Usunier. Learning with partially labeled and interdependent data . Springer, 2015. 7

Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts , 6:100022, 2020. 7

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2012. 8

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning (ICML) , 2018. 8

Aleksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards to automate reinforcement learning. arXiv preprint arXiv:1905.07628 , 2019. 9

Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In International Conference on Machine Learning , pages 11436-11446. PMLR, 2020. 9

Hogun Kee, Minjae Kang, Dohyeong Kim, Jaegoo Choy, and Songhwai Oh. Sdf-based graph convolutional q-networks for rearrangement of multiple objects. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , 2023. 10

Peer reviews:
comments: SUMMARY:
The authors propose that a novel RLHF framework called SeqRank to improve the feedback efficiency via sequential preference ranking. They propose two methods: sequential pairwise comparison and root pairwise comparison.

STRENGTHS:
The paper is clear and writing is easy to follow. The main idea is theoretically sound and the experiment results are promising.

WEAKNESSES:
The proposed method seems to be relatively simple. But I think it’s okay since the authors prove its effectiveness both theoretically and empirically.

QUESTIONS:
1 the proposed methods (sequential pairwise comparison, root pairwise comparison) is quite intuitive. Though it achieves better performance than standard pairwise comparison, it might be suboptimal.

2 the definition of $S_M$ is not clear (number of pairs collected from $M$ trajectories). Here “pairs” refers to both directly compared pairs and indirectly inferred pairs (e.g., the number of lines in both black and purple in fig2(b)), is that correct?

3 As for experiments, the numbers of feedbacks different from tasks. How do you choose this parameter? In figure 4, when the number of feedbacks is limited, it’s interesting to see the pairwise comparison outperform the two sequential methods. Is there any comment on this?

4 As for the writing, since the proposed method aims to improve the feedback efficiency, which seems to be defined by the authors, I suggest the author provide some illustrations first to show its importance (e.g., higher efficiency will lead to higher performance) before discussing the proposed method is able to improve it. This might make it more clear.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach, named SeqRank, to enhance the efficiency of reinforcement learning from human feedback. By incorporating sequential preference ranking, the proposed method significantly improves average feedback efficiency.  Evaluations demonstrate its superiority over existing baselines in simulated locomotion and manipulation tasks, as well as in real-world scenario with a robot manipulation task.\n\nSTRENGTHS:\nThe paper exhibits strengths in several areas. Firstly, it effectively identifies and defines relevant problem o feedback efficiency, Secondly, the paper provides a strong mathematical justification for the proposed method. The authors present a theoretical framework, underpinned by sound mathematical principles. Furthermore, the paper showcases evaluations conducted on a diverse set of tasks, including real-world experiments with a physical robot. The inclusion of real-world experiments adds a practical element, validating the effectiveness of the proposed solution in realistic settings.\n\nWEAKNESSES:\nOne limitation of the paper is the lack of consideration for alternative sampling methods beyond pairwise sampling. While uniform pairwise sampling is a widely used, other sampling methods, such as Uncertainty-based sampling[1] perform better.\n\nAnother point that could have been addressed in the paper is the possibility to label comparisons as \"equal”.  It is worth mentioning that incorporating this labeling aspect could potentially lead to improvements in performance of the proposed method, but it is not clear how that could be done.\n\n[1] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. Neural Information Processing Systems Track on Datasets and Benchmarks(NeurIPS), 2021\n\nQUESTIONS:\n1. In Section 5.2, you mentioned that Sequential pairwise comparison performs better than pairwise comparison on average. However, in several tasks, such as Sweep Into, pairwise comparison was better on average than sequential pairwise. Could the authors please explain why this discrepancy exists?\n\n2. In Figure 4, authors presented results for a specific task. Based on the values shown, it appears to be the Quadruped Walk task. However, it is not explicitly mentioned in the paper which task the results in Figure 4 correspond to. Could the authors please clarify this? \n\n3. Regarding the Experiments with Real Human Feedback: what are the standard deviations of the results. Could you provide information on the standard deviation for the reported data? Additionally, have the authors conducted any statistical hypothesis tests to validate the claims made based on the experiments with real human feedback?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the idea of the feedback efficiency of RLHF, which the authors define as the ratio between the number of trajectory pairs divided by the number of feedbacks. To optimize such a metric, the authors propose new algorithms that decide which trajectories to compare. In particular, the authors propose a sequential pairwise comparison strategy, where the most recent trajectories are compared and a root comparison strategy, where the defender is the previously most preferred trajectory.\n\nSTRENGTHS:\n- This paper studies the important and timely problem of RLHF.\n- The paper has extensive experiments, including simulated and real world tasks, as well as using real human feedback.\n- The study on the labelers satisfaction is novel and very interesting.\n\nWEAKNESSES:\n**W1.** One major concern with this paper is that the claim of completeness, (that any two trajectories are comparable) is incorrect for the main RLHF application of LLM finetuning [1]. Since this is the claim is key to the paper, I think it makes the paper poorly motivated/placed in the literature.\n\n[1] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in Neural Information Processing Systems 35 (2022): 27730-27744.\nAPA\t\n\n**W2.** Another major issue is that the contribution of using the transitive property of human preferences is limited. This is a limited contribution since RLHF works typically design the labeling interface in such a way that the labeler will implicitly use the transitive property of human preferences. An example can be seen in Figure 12 of the InstructGPT paper [1].\n\n**W3,** The claim that cognitive overload is a critical drawback of RLHF seems like an overstatement. In fact, the labeler satisfaction and inter-labeler agreement in the InstructGPT paper (and other RLHF papers) is high (see Table 13 of [1]).\n\n**W4.** The presentation of the author's method is not very clear and the theoretical results are hard to follow. The main tool the authors use to illustrate their method is Figure 2, which is not exact enough. Moving the algorithm to the main text would be helpful.\n\nQUESTIONS:\n**Q1.** What would be the impact of these algorithms in settings where only a small number of trajectories are comparable?\n\n**Q2.** Can you show RM convergence is faster (similar to figure 4)? The algorithms proposed in this paper should most directly affect the RM convergence.\n\n**Q3.** In figure 2, it seems like pairwise comparison would result in a more diverse training set. Could this diversity boost the performance of the reward model?\n\nLIMITATIONS:\nYes, the authors address the limitations of the study.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SeqRank a framework for labelling user preferences in a ranked fashion taking advantage of the transitive property of human preferences.\n\nAuthors mathematically prove that their two variants of SeqRank (sequential pairwise comparison, and root pairwise comparison), achieve better feedback efficiency (defined as the ratio between the number of preference labels and the number of trajectories) than conventional pairwise comparisons. Additionally, authors prove that both methods converge to the same level of empirical risk and that root pairwise comparison converges at the same rate as the baseline pairwise comparison, whereas sequential pairwise converges slightly more slowly.\n\nEmpirical evaluations show that both variants of SeqRank beat the performance of a pairwise comparison using a recent Preference-based Reinforcement Learning algorithm  (Meta-Reward Net, MRN), in 5 Mujoco tasks and 6 MetaWorld tasks. Experiments show a correlation between feedback efficiency and preferences sample efficiency.\n\nAdditionally, authors prove the usefulness of SeqRank in a finetuning-from-human-feedack task as well as training an actual UR-5 robot on a block-placing task.\n\nSTRENGTHS:\n* Paper presents an intuitive idea (rank preferences) backed up by robust mathematical analysis of its properties.\n* Compelling results in a wide variety of tasks, both in simulated and in real-world environments.\n* Feedback efficiency seems to correlate with sample efficiency, thus lowering the barrier of adoption for Preference-based Reinforcement Learning methods.\n\nWEAKNESSES:\n* Authors only test SeqReq on Meta-Reward Net. The paper would be more significant if  the same benefits observed for MRN+SeqRank were shown to carry over to other PbRL methods such as PrefPPO, PEBBLE [1].\n* In my opinion, the experiments with real humans do not help in establishing the significance of SeqRank. \n    * The relative improvement with root sequence comparison is of just 0.4% over the reward of the policy before fine-tuning. \n    * No measure of spread for the reward reported, so it is not possible to establish whether that improvement is significant or just noise. \n    * Regarding the questionnaires, authors should report whether the differences are statistically significant (preferably through a rank-based statistical test).\n* Beyond the above issues, I would have found it more significant to train the policy from scratch with actual human preferences and SeqRank as is common in the literature [1,2,3], rather than fine-tuning an already trained policy.\n\n\n[1] K Lee, L Smith, P Abbeel (2021) PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training. ICML.\n\n[2] PF Christiano, J Leike, T Brown, M Martic, S Legg, D Amodei (2017) Deep Reinforcement Learning from Human Preferences. NeurIPS.\n\n[3] B Ibarz, J Leike, T Pohlen, G Irving, S Legg, D Amodei (2018) Reward learning from human preferences and demonstrations in Atari. NeurIPS\n\n\n**Update after rebuttal**\n\nThe authors carried out additional experiments which addressed most of my concerns. In particular, the authors showed that i) SeqRank is compatible with uncertainty sampling, ii) that PEBBLE+SeqRank improves over vanilla PEBBLE, and  iii) that SeqRank outperforms pairwise comparison when training from scratch from human feedback. \n\nThe experiments with real human feedback from a pre-trained policy do not show a significant improvement over the baseline, but do show statistically significant higher user satisfaction scores for SeqRank.\n\nQUESTIONS:\n1. PEBBLE does a selection of trajectory pairs based on the reward disagreement between three modules. How does this fit with SeqRank? Does it affect the uniformness assumption? Or does SeqRank simply work off the selected subset?\n2. In figure 4, are the task rewards normalised before being averaged? If not, tasks like Walker Walk (where SeqRank does really well) may be overrepresented.\n3. Section 5.3 is really compressed, to the point it is difficult to understand exactly what is the experiment. \n    * How are the preferences obtained?\n    * Is the reward also fine-tuned when training on the actual robot? What is meant by \"the reward model is initialized before fine-tuning\"?\n    * What is True Reward and Reward Accuracy in Table 4?\n\n**Nitpicks and suggestions:**\n* In the introduction, the paper claims that obtaining multiple preferences from a single trajectory reduces the cognitive load for the user. The experiments offer no evidence for this. I would rephrase as users prefer a more efficient method.\n* Consider simplifying the notation for the expectation for trajectory pairs which is variously $\\mathbb{E}[S_N]$, $p_N$, $a_N$, $b_N$  in different parts of the paper. I understand each is subtly different in each context, but perhaps a superscript could be used to denote the differences?\n* The description of the SeqRank algorithm is left in the appendix, is there some way to bring it back to the main text?\n* Mention in the main text that statistics about the population are in the appendix.\n* In figure 4, why is root pairwise comparison beating SAC?\n\n\n**Update after rebuttal:**\n\nRefer to author's rebuttal for answers.\n\nLIMITATIONS:\nLimitations are adequately addressed. I would point out that this method lowers the barrier of entry to control complex systems that would otherwise not be controlled by humans. This has safety and ethical implications that should be considered before deployment.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}