Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Authors: Xuening Feng
Zhaohui JIANG
Timo Kaufmann
Eyke Hüllermeier
Paul Weng
Yifei Zhu
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning from Human Feedback
Preference-based Reinforcement Learning
Human-in-the-loop Machine Learning
Primary area: reinforcement_learning->deep_rl

Abstract: Learning human objectives from preference feedback has significantly advanced reinforcement learning (RL) in domains where objectives are hard to formalize. 
However, traditional methods based on pairwise trajectory comparisons face notable challenges, including the difficulty in comparing trajectories with subtle differences and the limitation of conveying only ordinal information, limiting direct inference of preference strength. 
In this paper, we introduce a novel *distinguishability query*, enabling humans to express preference strength by comparing two pairs of trajectories. 
Labelers first indicate which of two pairs is easier to distinguish, then provide preference feedback only on the easier pair. 
Our proposed query type directly captures preference strength and is expected to reduce the cognitive load on the labeler. 
We further connect this query to cardinal utility and difference relations and develop an efficient query selection scheme to achieve a better trade-off between query informativeness and easiness. 
Experimental results demonstrate the potential of our method for faster, data-efficient learning and improved user-friendliness in RLHF benchmarks, particularly in classical control settings where preference strength is critical for expected utility maximization.

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Xuening Feng' Zhaohui Jiang'
Timo Kaufmann 23 Eyke Hillermeier 234
Paul Weng? Yifei Zhu
Abstract
PCQ
PCQ
Learning human objectives from preference feed- back has significantly advanced reinforcement learning (RL) in domains where objectives are hard to formalize. However, traditional methods based on pairwise trajectory comparisons face no- table challenges, including the diffieulty in com- paring trajectories with subtle differences and the limitation of conveying only ordinal information, limiting direct inference of preference strength. In this paper, we introduce a novel distinguishabil- ity query, enabling humans to express preference strength by comparing two pairs of trajectories. Labelers first indicate which of two pairs easier to distinguish, then provide preference feedback only on the easier pair. Our proposed query type directly captures preference strength and is ex- pected to reduce the cognitive load on the labeler. We further connect this query to cardinal utility and difference relations and develop an efficient query selection scheme achieve better trade- off between query informativeness and easiness. Experimental results demonstrate the potential of our method for faster, data-efficient learning and improved user-friendliness in RLHF bench- marks, particularly in classical control settings where preference strength is critical for expected utility maximization.
More distinguishable
(0.1)
Yrco (0.1)
of is preferred
PCQ
Distinguishability
Pairwise preference feedback уPсo
preference feedback d
Distinguishability query DQ (PCQo. PCQ1), and its feedback ynq = (d, yPCQ)
An illustration of the distinguishability query. PCQ refers the usual Pairwise Comparison Query for a pair of seg- ments (o", of )), and its feedback is yrco, indicating the preferred segment. Our proposed distinguishability query, DQ. consists of two candidate PCQs. Its feedback yoo (d, yисо) includes the choice of the more distinguishable PCQ, d, and its corresponding label, yroq-
Figure I.
models (OpenAI, 2022: Ouyang et al., 2022) to training robots for complex tasks (Christiano et al., 2017). The standard method asks human evaluators to compare pairs of trajectories, using these comparisons to learn a reward function that guides RL agent training. While this method is widely used and has shown impressive performance in simulated environments (Christiano et al., 2017; Lee et al., 2021 lb; Liang et al., 2022; Park al., 2022; Hu et al., 2024; Verma & Metcalf, 2024; Dong et al., 2024), it faces two key limitations: First, humans often struggle to compare trajec- tories, particularly when the differences are subtle, leading to inefficient queries and poor user experience. Second, pair- wise comparisons provide only ordinal information, leaving preference strength to be learned implicitly. This limita- tion is particularly problematic for reinforcement learning, which fundamentally requires cardinal utilities for expected utility maximization - the core principle guiding the agent's decisions under uncertainty.
1. Introduction
Learning human objectives from preference feedback has enabled reinforcement learning (RL) to tackle domains with hard-to-formalizee objectives, from fine-tuning language
'Shanghai Jiao Tong University, Shanghai, China Instittte for Informatics, LMU Munich. Munich, Germany 'Munich Center of Machine Learning (MCML). Munich, Germany "German Re- search Center for Artificial Intelligence (DFKI) Digital Innovation Research Center, Duke Kunshan University, Kunshan, China. Cor- respondence to: Paul Weng <paul weng @ dukckunshanncduu www.sshwaannnnn........
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we propose novel query type called the distin- guishability query (illustrated Figure 1), which addresses

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
these two critical limitations. In distinguishability query, the human first indicates which of two trajectory compar- isons is easier to distinguish and then provides preference feedback on only the selected pair as usual in preference- based RLHF methods. While traditional preference choice for a single pair of segments only conveys ordinal informa- tion, the additional choice of which comparison to evaluate allows us to infer preference strength, assuming that hu- mans prefer to provide feedback on the more distinguishable pair. This approach enables labelers to focus easier com- parisons and directly facilitates the learning of preference strength. Our proposed method, named DistQ. integrates distinguishability queries with an effective query selection scheme and a specialized learning objective.
psychology, and decision theory (Suppes & Winet, 1955; Krantz et al., 2006; Jansen et al., 2018). Prior work has shown that preferences elicited in RLHF, when noisy and under certain assumptions, can be used to infer cardinal utilities (Chan et al., 2021; Xu et al., 2020). This aligns with the empirical success of RLHF methods based on pairwise comparisons (Lee et al., 2021b; Liang et al., 2022; Park et al., 2022; Hu al., 2024; Dong et al., 2024). However, these assumptions about utility-dependent noise are strong and may not hold in practice. Moreover, since each com- parison is typically observed only once, models must rely on generalization across similar comparisons to infer pref- erence strength. This motivates us to explore more direct ways to elicit cardinal utilities.
Concretely, our contributions are as follows:
Distance Relations Cardinal utilities can be represented either as a real-valued function (unique up to positive affine transformations) or as a relation on pairs of outcomes. The latter formalism aligns closely with our proposed query type, where human labelers distinguish between two pairs of outcomes. Formally, human labeler preferences can be modeled using two relations R and D (Suppes & Winet, 1955; Jansen et al., 2018), where R represents preference relation and D a difference relation. If a pair of pairs sat- isfies ((a, b), (c, d)) E D, then exchanging b with a is at least as desirable as exchanging with c - mcaning a is more strongly preferred over than is over d. Several works establish axioms that determine utility function from such relation, unique up to positive affine transfor- mations (Alt, 1936; Suppes & Winet, 1955; Köbberling, 2006). Notable among these axioms are completeness and transitivity. When completeness is not satisfied, the relation only determines a set of compatible utility functions (Pivato, 2013).
1. We propose a new type of query, the distinguishability query, in the field of RLHF (see Section 4.1);
2. We establish connections between distinguishability queries and cardinal utility and distance relations stud- ied in related disciplines (see Section 2);
3. We design a query selection scheme for distinguisha- bility queries that aims to achieve a better trade-off between query informativeness and ease of answering (see Section 4.2);
4. We propose specific learning objective to improve reward learning by coupling cardinal and ordinal infor- mation (see Section 4.3);
5. We empirically demonstrate, on classic control tasks with synthetic oracle, that our method can achieve competitive performance in a more user-friendly man- ner compared to standard pairwise comparison meth- ods (see Section 5).
Eliciting Preference Strength Eliciting distinguishability for cardinal utility in RLHF remains underexplored. While Jansen et al. (2022) explore methods like time elicitation and label elicitation (the latter demanding absolute ratings, posing challenges (Yannakakis & Martínez, 2015) and bur- dening labelers), these are suited for finite outcome spaces with dense observations. DistQ instead employs relative dis- tinguishability queries and integrates neural network utility learning with probabilistic modeling and informativeness based query selection for generalization across vast trajec- tory spaces with learnable structure. Other methods infer cardinal strength from human response times (RTs), assum- ing RTs reflect preference strength through explicit cognitive modeling using the Drift Diffusion Model (DDM) (Ratcliff, 1978; Ratcliff & McKoon, 2008) or by leveraging RT's as predictive features. Shvartsman et al. (2024) use Gaus- sian Processes with either DDM approximations or RTs as stacking features, while Li et al. (2024) use EZ-DDMs (Wagenmakers et al., 2007) in linear bandits (assuming lin-
2. Related Work
Our work overlaps with three key areas: decision theory, RLHF approaches using pairwise comparisons, and ap- proaches that aim to reduce the burden on the human labeler. We briefly review each of these areas below.
Preference Strength Observed choices, which form the foundation of RLHF (Jeon et al., 2020), convey only or- dinal preferences, expressing information about rank but not about distance. With cardinal utilities, not only the order of utility values but also their differences are meaning- ful capturing preference strength, which is not expressed in purely ordinal utilities. Reinforcement learning funda- mentally requires such cardinal utilities for expected utility maximization (Von Neumann & Morgenstern, 1947). Car- dinal utilities have been extensively studied in economics,
2

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
ear utility). DistQ's explieit, second-order query operates differently: it asks humans to identify which of the two potential choices is easier, leveraging their meta-cognitive assessment of relative difficulty to allow an answer and provide strength signal while avoiding fully resolving the harder underlying comparison. This query type aims for greater robustness by avoiding direct reliance on absolute RT's and strong accompanying assumptions (e.g., DDM, lin- utility)- problematic in complex RLHF settings and its distinguishability signal readily extends standard loss functions.
reward function r(s, a) used train policy r[ass) (Lee et al., 2021b;a). The agent learns approximate reward function fgg fg a) from human preferences, implemented as an ensemble of N neural networks parameterized by ti (v1, WN). The policy T and reward f' are updated by alternating:
Step I (agent learning): The agent interacts with the environment using as to collect trajectories and up- dates the policy via RL, maximizing expected return under the current Fg..
Step 2 (reward learning): Preference queries are gener- ated from collected trajectories and used to update the reward model f& based on human feedback.
Reducing Burden of Human Labelers A major limi- tation of pairwise comparisons is their burden human labelers when the compared behaviors are similar or when neither option is clearly preferable. Prior work has ad- dressed this through multiple strategies: (1) pre-training, either in an unsupervised manner (Lee et al., 2021b) or us- ing demonstrations (Ibarz et al., 2018; Palan et al., 2019; Bıyık et al., 2022), (2) allowing labelers to abstain from answering queries (Lee et al., 2021a), and (3) query selec- tion strategies that aim to select queries that are easier for the human labeler to answer (Bryık et al., 2019). The first two strategies are complementary to our approach. Re- garding the third, which our approach falls into, Bıyık et al. (2019) propose to use information gain to select queries that are informative and easy to answer, implicitly prioritizing queries the human will be able answer and thus lead to the largest gain of information. While such an approach could be extended to distinguishability queries, we instead opted for a query selection scheme based on ensemble disagree- ment and prediction entropy for computational efficiency. In addition to this selection scheme, our distinguishability queries allow the labeler to actively choose easier queries themselves.
While any RL algorithm could be used in Step 1, we use the Soft Actor-Critic (SAC) algorithm (Haarnoj et al., 2018) in our experiments, following PEBBLE (Leee et al., 2021a).
Pairwise Comparison Queries in RLHF In RLHE, hu- man feedback is typically collected through pairwise com- parison queries (PCQ) (Christiano et al., 2017; Lee et al., 2021b). Given trajectory segments の" and a², represented by a sequence of states and actions, a PCQ can be denoted as (o", ot)). The oracle (c.g.. human labeler) gives preference through feedback frcq E {(1, 0), 10], where (1, 0) indi- cates segment のd is preferred over of and (0, 1) indicates the opposite. These query query-feedback triples (o oo ypcQ)
are stored in dataset DpcQ-
Preferences are linked to the reward function by means of a Trerr model (Bradley& & Terry, 1952), which as- sumes latent utilities p govern pairwise preferences through
exp(p')
Plo > 001 exp(x") + exp(x*)
In the context of RLHF, where the utility of a trajectory segment is defined as its return, the predicted probability of segment a² being preferred over 00 is
3. Preliminaries
Reinforcement Learning We consider a reinforcement learning (RL) setting characterized by a Markov Decision Process (MDP). The MDP is defined by AS,,, 7), where S and A denote state and action spaces, while P(s'|s, (o/ls,a),, a), r(s,a) a), and ? E (0, 1] represent the transition function, the reward function, and the discount factor. At each timestep I, the agent receives state 8; E S, takes action a, E A. receives reward r(st, az), and transitions to St+1 ~ P(5++1| a.). The return R2- = Eho >*(($++k, dx4x) is defined as the discounted cumulative sum of rewards from timestep 1, which is to be optimized.
exp E, 7' (0t, at) Pola' > 000 Eje(o.1) exp E. 7 (8,, at)
(1)
where (s at) for {0, 1} is the average output of the N reward networks F... for i E (I,...,N..
Given dataset Drco and corresponding predictions from Equation (1), reward learning is formulated as a classifica- tion problem (Christiano et al., 2017). The reward model Ae can be learned by minimizing the cross-entropy loss
C C EDooo
Σ }E(0,1}
Vico log (o' > - (2)
Reinforcement Learning from Human Feedback RLHF is a framework that aims to learn optimal agent behavior from human feedback (Christiano et al., 2017: Kaufmann et al., 2025). In this paper, we focus on inferring an unknown
where yPсQ (ufca- Vroo) is the feedback and Exorco is the (empirical) expectation for (a (2(o)))) ~ Drco-
3

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Select tup and bottom By easy PCQs
2pve
Pair PCQs based on easiness
La
Give feedback Qa. obtain
Randomly cut and pair
Select top My informative PCQs
Trajectory buffer B
2 Rag (Dog))
Informativeness based on Jurgerer variance
Easiness based smaller entropy
Larger DEUTERGN
Optimize PV
on Qoo
Figure 2. DistQ's query selection scheme for distinguishability queries. The process consists of multiple filtering steps: First, from set of randomly generated segment pairs Qp. we select a subset Qp with higher informativeness I. Then, we filter a subset Qpce based on casiness& & from Dpe. Both I and E are calculated from the current reward model fg. Finally, we construct 2a by pairing the PCQs in Opve, matching the top i-th easiest PCQ to the bottom (i + ng) > one. We ohtain Ebo with the labelers' feedback on Qd. which can then be utilized by our training objective for Fg. integrating information from distinguishability and preference feedback.
4. Method
erence feedback as yrco E {11, 0), 1)). We represent the distinguishabilitty query as DQ = (PCQo: PCQ1) = ((o), ob), (of. (of.ot)) The feedback to a distinguishability query yoo = (d, spCo) consists of two components: the dis- tinguishability preference feedback d {(1, (0, 1)} indicating which pairwise comparison query is more dis- tinguishable, and the pairwise preference feedback ypcQ to the selected more distinguishable pairwise comparison query. Such a query and corresponding feedback is rep- resented by (DQ, yDo) and stored in a dataset DDO- See Figure I for illustration.
We present DistQ, a novel RLHF approach that addresses limitations of traditional pairwise comparisons through dis- tinguishability queries. Our method aims to pose queries that are both informative for distance-aware reward learning and easier for human labelers to answer, enhancing both user experience and query efficiency. The approach consists of three key components: (I) the distinguishability query struc- ture that enables humans to express preference strength by comparing pairs of trajectory comparisons, (2) an efficient query selection method considering both informativeness and easiness, and (3) a specialized learning objective that leverages the richer feedback from distinguishability queries to improve reward learning.
We define the distinguishability measure M for pairwise comparison query (a o²) as
M(a", = - Er) (3)
The remainder of this section details these components. In Section 4.1, we introduce the structure and theoretical foun- dations of distinguishabilityy queries. Section 4.2 presents our efficient query selection scheme designed specifically for these queries. Finally, Section 4.3 develops a special- ized training objective that integrates both distinguishability and preference feedback. Together, these components form cohesive system with the aim of creating freenfriendly, informative queries while enriching the reward learning process, as illustrated in Figure 2.
Larger values indicate stronger distinguishability Note that M is symmetric. Given a learned reward model fv. distin- guishability M(a", ,o") can be approximated by
M Mp(o -') Σ (fo al) E E (4)
4.2. Selecting Informative and Easy-to-Answer Distinguishability Queries
4.1. Distinguishability Query
Given this newly proposed type of query, we design method (see Figure 2) to select distinguishability queries that are informative and also easy to answer for the labeler. Overall, we begin with selecting desirable pairwise compari- son queries and pair the selected ones into distinguishability queries.
Intuitively, in order to avoid posing unanswerable PCQs and to learn about preference strength, we propose providing the labeler with two such queries together as one distinguishabil- ity query. We let the labeler select the more distinguishable one that is easier to answer and then provide preference feedback for the chosen pairwise query. This approach ef- fectively combines query about ordinal preferences with one about preference strength, while simultaneously reduc- ing the burden on the human labeler.
Broadly speaking, we aim to select queries on which feed- back from the labeler could reduce our current predictive uncertainty while also ensuring that the labeler can easily provide feedback. This aligns with the concepts of epis- temic and aleatoric uncertainty (Hüllermeier & Waegeman, 2021): While the latter refers to inherent uncertainty due to randomness in the data-genng process (in our case the
Recall that in Section 3, the pairwise comparison query is denoted as PCQ = (a", (a",a") o²)) and the corresponding pref-
4

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
labeler's responses), the former is caused by the learner's limited knowledge of this process. Thus, while aleatoric uncertainty is irreducible, epistemic uncertainty can, in prin- ciple, be reduced through additional (training) information and therefore is a natural target for active learning and query construction (Nguyen et al., 2022). Since we assume the labeler gives stochastic feedback according to a Bradley- Terry model, aleatoric uncertainty is high when the utility difference is small and low when the utility difference is large.
where PJ =Pylo' oا-يه for j {0, 1] represents the average prediction from the ensemble reward model.
It is worthwhile to mention that although higher entropy values may also correspond to more uncertain predictions, and the entropy criterion has been used for pairwise com- parison query selection (Lee et al., 2021b), relying only on the highest entropy can result in queries that are nearest to the decision boundary and thus really hard for the labeler to answer. By selecting queries with lower entropy among those selected with the largest variance, we aim to select the queries that are as casy to answer despite the epistemic uncertainty involved.
Our selection scheme, therefore, favors queries that are infor- mative and hence epistemically uncertain, and meanwhile easy to answer by the labeler. Following previous work (Christiano et al., 2017; Lee et al., 2021b), we measure the informativeness using the variance of the prediction of the reward networks P.j, p. 's. A larger variance means larger dis- agreement between the reward networks and, thus, higher epistemic uncertainty. For query easiness, we measure it us- ing the average entropy of the Bemouuli distributions Pv. 's. Indeed, large entropy implies that the return difference of two segments small, which means that they are hard to distinguish. Next, we will explain our method step by step in detail.
Forming Distinguishabilit Queries After sorting the ny queries in 2po in decreasing order with respect to their easi- ness {a ,"),, the top dg and bottom Bg casiest queries are selected to form the set Opve. Then the top i-th easiest query is paired with the bottom (a + ng)-th easiest one to compose a distinguishability query. We apply this simple method so that easy queries are paired with less easy queries while still considering only the most informative queries overall.
4.3. Training with Distinguishability Queries
Informativeness Based on Variance As Figure 2 shows, given trajectory buffer B, we first randomly sample seg- ments from trajectories and pair them to obtain set 2p of candidate PCQs. With the current reward model fr.. we can compute the predicted pairwise preference probability in Equation (1) for each PCQ (o".o') 2p-
To learn from distinguishability feedback, assume that the labeler chooses between PCQ, (ag, ab) and PCQ; (of, (t) according to Bradley-Tene model. Using reward model fws the labeler's response can be predicted by:
exp.Malof.ot)
We define the informativeness ((o",ot) of a PCQ (a", as the variance of reward model prediction, which is
Ppl(of, ot) > (o) af)l = Σ exp ob)
(7)
hef0,1)
N Σ
= =
(PI. v -P)'
(5)
where (tf > (o8.ab) means that PCQ, is more distin- guishable than PCQp-
Under this assumption. we can then train the reward model with feedback d by formulating the distinguishability prefer- ence prediction also as a supervised classification problem. Similarly to Equation (2), we define the cross-entrops cooss-y loss for the distinguishability preference feedback d as
where P1. Pv, > و8] is the prediction solely from neural network fv, and p! = lo > o0] is the aver- age prediction from the ensemble reward model. Queries with higher variance indicate higher epistemic uncertainty of the current reward model, thus providing more informa- tion for reward learning. In this step, PCQs with top nı informativeness are finally selected from set Op for later steps.
CRew = -EDDo do log Ps [(o ot) > (af.
+ di log Ps [(o9. af) > (o of d.]]] (8)
Easiness Based on Entropy Let Qp be the set of the ny informative queries obtained in the last step. We then define the casiness E E(0 o²) of PCQ (a", ,²)) as the negative entropy of reward model prediction, which
where EDnO represents the (empirical) expectation for ((08:o)), (of, of): Voo) Dao Recall that distinguisha- bility feedback yoQ = (d, uPCQ) consists of both distin- guishability preference feedback d and pairwise preference feedback yPCQ- The latter feedback yPCQ can also be ex- ploited for reward training by writing loss CRey in Equa- tion (2) on the pairwise comparison query selected by d.
E(0", - -H(Pp)
Σ pd log pi_ 0=
(6)
5

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Query budgett
Query budgel 500
Query budget= = 10000
Query budgett - 400
D3
700
Epitedo Rawmrd
800
800
FGBRLS
Edisods Rewerdd
FEUELE
Success Rate (16)
600
600
SARF
500
BANT
60
4.0
hwin
Epitods
4.00
QPA
400
8
200
RUME
200
200
20
SAC
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.8
1.2
1.5
2.9
0.0
0.1
0.2
0.3
0.4
0.5
Environment ent Stepe (x109)
Environment Stepa
(x10f)
Environment Stapa (x109)
Envrronment Stupa (x10")
(a) Walker walk
Quadruped walk
(c) Humanoid stand
(d) Window open
budget 2000
Query budget 3000
Queery budged 10000
Quaryy budget 10000
100
100
100
Succosa Rada (N)
80
Succassa Ruta (S)
80
Sucoses Rate (16)
80
2.
50
60
60
60
MM
40
40
40
20
20
20
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.6
1.0
0.0
0.5
0.6
0.0
1.2
1.5
0.0
0.3
0.6
0.9
1.2
1.5
Environment Steps (x109)
Environment Steps (H20F)
Environment ent Steps (C:05))
Environment Stopa (x10%)
(e) Door unlock
(l) Door open
(g) Sweep into
(h) Disassemble
Figure 3. Performance comparison of DistQ against baselines on classic control tasks. Results show episode reward for DMControl Incomotion tasks (Figures 3()) to 3(c) and success rate for Meta- World manipulation tasks (Figures 3(d) to 3(h)). To account for DistQ's s different query structure, we show 'DQ' (same query budget as baselines) and 'DQ (half)' (half query budget) bracket the human effort for comparison. The solid curves and shaded regions represent the mean and standard deviation, respectivcly, across five runs.
Finally, we update the reward model ée by minimizing the linear combination of CRow and PCQ as
DeepMind Control Suite (DMControl) (Tassa et al., 2018) and robotic manipulation tasks from the Meta-World bench- mark (Yu et al., 2019).
CRew Xa + Xp
(9)
To quantitatively evaluate the performance of the involved RLHF methods, we follow a general setting where the agent has no access to the ground truth reward from the envi- ronment but can only receive synthetic feedback based on the ground truth reward from scripted labeler (Lee et al., 2021b). Given the feedback, the agent learns to solve the corresponding task guided by the underlying reward func- tion. The performance is then measured as the true average return for locomotion tasks and the success rate for manip- ulation tasks. We report the mean and standard deviation across five runs for all experiments.
where hyperparameters Aa and Ap denote the weight for <Rew and CRew respectively.
Our proposition, DistQ, corresponds to enhancing Step 2 (reward learning) described in Section 3. For simplicity, we do not change Step 1 (agent learning). The pseudo-code for the general procedure can be found in Algorithm 1 in Appendix C.
5. Experiments
In this section, we conduct experiments to investigate the following questions: (1) How do the proposed distinguisha- bility query and corresponding query selection method help with performance and query efficiency compared with state- of-the-art (SOTA) RLHF methods that only use pairwise comparison queries? (2) Are the pairwise comparison queries selected by our method easier to answer compared to those selected by the baseline methods? (3) How does each proposed technique contribute to the overall design?
Baselines For comparison, we adopt variety of SOTA methods in the field of RLHF, including PEBBLE (Lee et al., 2021b), SURF (Park et al., 2022), RUNE (Liang et al., 2022), MRN (Liu et al., 2022), and QPA (Hu et al., 2024). All of these baseline methods use pairwise compari- son queries. Regarding query selection, PEBBLE adopts an entropy-based method (i.c., smaller E), while SURF, RUNE, and MRN adopt atriance-based method (i.e., larger I). QPA selects queries randomly from a buffer of near-on- policy trajectories. All baselines are evaluated with the original settings listed in their paper. More details are pro- vided in Appendix A. What is more, considering that all these methods employ SAC for agent learning, we also mea-
5.1. Experimental Setup
Tasks Similar to prior works (Lee et al., 2021b;a; Park et al., 2022; Liang et al., 2022; Liu et al., 2022; Hu et al.,
2024), we consider a series of locomotion tasks from the
6

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Weongly Predicted Faedback
20
Query budget - 200, betch size + 20
50
Quary budget= - 500, batch size 50
20
Query budget - 250, batch size- 20
Quary budged + 500, batch size 50
45
DG g00
15
0.54
Whongyy Predicted
30
35
Weonggg Produced
25
0.1
25
70
20
6
15
10
70
100
160
190
100 130 160 190 220 250 280
40
100
130
190
10 40 70 100 130 160 220 250 209
Environment Stups (x107)
Environment (x10?)
Environment Steps (x100)
Environment Stups (x10?)
(a) Walker walk
(b) Quadruped walk
(c) Walker walk
(d) Quadruped walk
Figure 4. Number of incorrectly predicted pairwise comparison feedback across training iterations on locomotion tasks. Lower values indicate queries that are easier for the labeler answer, as the reward model's predictions better align with the labeler's preferences. Note that this figure includes results of ablation studies, as detailed in Section 5.5. The solid curves and shaded regions represent the mean and standard deviation, respectively, across five runs.
sure the performance SAC using the ground truth reward function as an upper bound of performance.
Locomotion Tasks from DMControl As shown in Fig- ures 3(a) to 3(c), DistQ with full budget largely outper- forms the baseline methods. This aligns with our expecta- tions, as distinguishability queries naturally provide richer information for reward learning, leading to superior reward models for guiding the agent. The primary exception is the challenging Humanoid stand task, where QPA outperforms DistQ with full budget slightly. Notably, Humanoid stand has not been addressed by most other baselines, prompting us to adopt QPA's hyperparameter settings of the total query budget and the query batch size in one feedback session for all methods on this specific task for a fair comparison. Addi- tionally, even with only half the budget, DistQ demonstrates competitive performance compared with most baselines (ex- cluding QPA). Across the other tasks, DQ (half) generally surpasses most baselines, with MRN on Quadruped walk being the other exception. The ability of DQ (half) to match outperform many of the baselines despite the reduced query budget demonstrates distinguishability queries are efficient while providing rich information and allowing for selective answering of easier queries.
Implementation We implement the distinguishability query and the query selection method on top of the widely adopted method PEBBLE (Lee et al., 2021b). We also adopt the same generie hyperparameter settings as PEBBLE without any tuning for a fair comparison. This implemen- tation is then evaluated and compared with all baselines. We argue that the proposed new query and corresponding query selection method can be implemented on top of any RLHF method utilizing pairwise comparison queries. See Appendix B for more details the implementation.
5.2. Benchmark Tasks with Unobserved Rewards
Figure shows the learning curves of DistQ and the five baselines on three locomotion tasks (Walker walk, Quadruped walk, and Humanoid stand) and five robotic manipulation tasks (Window open, Door unlock, Door open, Sweep into, and Disassemble). All baseline methods use budget of pairwise comparison queries indicated by "Query budget" in each subfigure.
DistQ shares PEBBLE as the common foundation with the baselines (see Appendix A). As these baselines focus mainly different aspects to improve query efficiency and perfor- mance, such exploration, augmentation of unlabeled data, and new training procedures for the agent, which are orthog- onal to DistQ, we expect that combining DistQ with each of the baselines could further improve performance, which would be worthwhile to explore for future work.
Note that for DistQ, although distinguishability query is composed of two pairwise comparison queries, we only ask the more distinguishable pairwise comparison query to the labeler. Therefore, we show the results of DistQ using both full budget (i.e., the DQ curve) and half budget (i.c., the DQ (half) curve), aiming for a fair comparison that lies somewhere in between these cases. In Figure 3(a), for example, the curves of all baseline methods are obtained by asking 200 pairwise comparison queries, while DQ (half) uses only 100 distinguishability queries. It is indeed difficult to set precise budget for DistQ to enable completely fair comparison. However, by presenting the results in this manner, we believe that the full and half budget settings provide a performance range of DistQ, which can serve as a meaningful comparison with baselines.
Robotic Manipulation Tasks from KerldWorld The re- sults in Figures 3(d) to 3(h) showcase similar phenomena to the locomotion results. DistQ with full budget still outper- forms all baselines and even converges to the same perfor- mance as SAC (yellow) or outperforms SAC. The version with half budget also exhibits better or similar performance compared with baselines with the exception of RUNE on Sweep into and SURF on Door unlock.
7

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
These results effectively demonstrate that the proposed DistQ can generally improve both performance and query ef- ficiency. We present the results numerically in Appendix D.I for a clearer comparison.
settings and results can be found in Appendix D.3.
Query budget 200
Query budget - 500
1000
1000
Epitodo Roward
500
800
5.3. Query Easiness
600
600
(1Q
Epieods
400
400
Typically, in reward learning, batch of pairwise compar- ison queries is selected and presented to human labeler during each feedback session. We consider queries to likely be difficult to answer when the reward model's prediction for a pairwise comparison is inconsistent with the labeler's actual response. The model, trained on prior feedback, should discern clear preferences. An inconsistency therefore suggests the query involved subtle differences, conflicting attributes, or unmastered nuances. These cognitively de- manding queries align with the concept of "wrong answers" that BIysk et al. (2019) describe, as they are more prone to human error and deliberation.
200
200
001t
0.0
0.2
0.4
0.5
1.0
Environment Steps (x11F))
Environment Stape (x:1f))
(a) Walker walk
(b) Quadruped walk
Figure 5. Ablation study evaluating different query selection strategies. Learning curves on locomotion tasks illustrate the comparison between our full method (DQ) and variants prioritiz- ing casiness then informativeness. (EI,, only informativeness (I), only easiness (E). The solid curves and shaded regions represent the mean and standard deviation, respectively, across five runs.
To investigate whether we ask casier pairwise comparison queries than baselines, we show curves of incorrectly pre- dicted feedback to pairwise comparison queries along the training process of various methods in Figures 4(a) and 4(b). Note that here DistQ with half budget (blue) also adopts half of the query batch size. We see that among all methods, DistQ with full budget (pink) always makes the fewest mis- predictions given the same number of pairwise comparison queries allowed to all baselines, which indicates that DistQ can really ask easier-to-ane to-wewerr queries and thus reduce the labeler's s effort.
1000
00
Quary budget
100
budget 10000
600
Seccees Rai
Egloods
400
200
0.4
Environment Steps (K110)
Environment (x10.))
(a) Quadruped walk
(b) Sweep into
Figure 6. Ablation study evaluating different PCQ pairing ap- proaches. Learning curves on locomotion tasks illustrate the comparison between our full method (DQ) and a variant employ- ing random pairing (rdd pairing). The solid curves and shaded regions represent the mean and standard deviation, respectively, across five runs.
5.4. User Study
To provide more direct validation of DistQ's 's effectiveness and user-friendliness. we conduct a user study involving real human labeler. This study compares DistQ against its backbone method, PEBBLE. For this evaluation, we design a novel task within the Quadruped environment: the agent is intended to learn to stand and then wave its right hind leg. The human participant provides feedback on the agent's s behavior based short video demonstrations, guiding the learning process for both methods.
1500
Query budget 200
1000
Query budget = 500
Epioada Rowaard
100
PoReLa
800
600
Eplecda
400
400
The agent's 's learned behavior is evaluated at the end of the training period for both DistQ and PEBBLE. The results highlight a significant performance difference: the agent trained with DistQ successfully performs the desired behav- ior in 100% of 10 evaluation rounds. In contrast, the agent trained with PEBBLE fails to perform the behavior in any round (0% success rate), often struggling even to achieve a stable standing posture. Furthermore, qualitative feedback from the participant indicates that providing feedback for DistQ is perceived as noticeably easier compared to PEB- BLE, suggesting a reduced cognitive burden on the human with DistQ. More comprehensive details of the user study
200
DG
200
0
PEBOLS
Eivironment Steps (x117)
Ervironment Steps (H117)
(a) Walker walk
(b) Quadruped walk
Figure 7. Ablation study evaluating different loss functions. Learn- ing curves on locomotion tasks illustrate the comparison between our full method (DQ), a variant using only the pairwise loss, and the bascline PEBBLE. The solid curves and shaded regions rep- resent the mean and standard deviation, respectively, across five runs.
8

---

Comparing Comparisons:
Informative and Easy Human Feedback with Distinguishability Queries
5.5. Ablation Studies
pairwise preference feedback (LPC)) ). To assess the contri- bution of L to the final performance, we compare three configurations in Figure 7: DistQ using the full b (pink), DistQ trained using only LPcq (blue), and the baseline PEB- BLE (green), which also uses only a pairwise preference loss. As anticipated, removing <Rew (the blue variant) im- pairs the performance compared to the full DistQ (pink), confirming the utility of distinguishability feedback. Sur- prisingly, though, DistQ trained only with LPCQ (blue) still outperforms PEBBLE (green). Given that both configura- tions rely solely on the same number of pairwise comparison queries, this indicates that the queries selected by DistQ are more effective for reward learning. We provide results of above ablation studies on more tasks in Appendix D.
To isolate the contributions of our proposed techniques to the overall performance of DistQ, we conduct ablation stud- ies focusing three key components: the query selection strategy, the PCQ pairing approach, and the newly designed loss function.
First, we examine the query selection strategy. Our full method (DQ, pink) prioritizes first selecting informative PCQs and then considers query easiness among those can- didates. We compare this against three variants: variant, EI (blue), which reverses this order by prioritizing easiness first and then informativeness; a variant, I (green), which selects PCQs based solely on informativeness; and a variant, E (orange), which selects PCQs based solely on easiness. Performance in terms of episode reward on two locomotion tasks is presented in Figure 5. To complement this, we also assess easiness of the PCQs selected by each variant by plot- ting the count of incorrectly predicted preference feedback in Figures 4(c) and 4(d), as discussed in Section 5.3.
6. Discussion
We introduce DistQ, a novel framework for RLHF designed to enhance both query efficiency and user-friendliness At its core, DistQ features the distinguishability query, a new type of human interaction, coupled with an efficient and user-friendly query selection method that strategically bal- ances the informativeness and easiness of queries. These key features, supported by a specifically designed loss function for reward learning, allow DistQ to learn more effectively from human feedback while minimizing the labeling effort. Experiments on a variety of locomotion and robotic manip- ulation tasks demonstrate that DistQ outperforms current oftte--f-the-rrt baselines in RLHF for control, particularly when considering the dual objectives of query efficiency and user- frienfriendly query design.
The El variant (blue), which prioritizes easy queries before informative ones, shows a significant decrease in episode reward on the considered tasks (Figure 5). This outcome suggests that an "easy first" approach can be detrimental as it may not provide sufficient information for robust reward learning. When selecting queries based only on informa- tiveness (I, green), the agent achieves strong, sometimes near-optimal, reward performance. However, this variant consistently generates the highest number of queries that are incorrectly predicted (Figures 4(c) and 4(d)), indicating that these queries would likely be most difficult for the la- beler. Conversely, selecting solely for easiness (E, orange) results in the fewest incorrect predictions (i.c., the easiest queries for the labeler) but leads to the poorest reward per- formance. These findings highlight a clear trade-off. DQ (pink) effectively balances these aspects. By first ensur- ing informativeness and then selecting for easiness from that informative pool, DistQ achieves strong performance while presenting queries that are demonstrably easier an- swer than those from the informativeness-on approach and more effective for learning than the "easy first" EI variant.
Limitations While DistQ demonstrates strong perfor- mance, future work could further extend the method and its evaluation and apply it to new domains such as language modeling. The current measures for informativeness and easiness, though effective, could potentially be refined or replaced with more sophisticated metrics, which might fur- ther optimize query selection. Furthermore, our present evaluation is focused on simulated control tasks. Extending the validation of DistQ to more realistic and diverse real- world applications would provide further insights into its capabilities and robustness.
Second, we investigate the PCQ pairing approach used to construct DQs. We compare our proposed pairing approach against a random pairing baseline ("'nd pairing"). As shown in Figure our pairing method significantly outperforms random pairing- This result underscores that the method used to combine PCQs into DQs influences their effective- ness, with our approach yielding DQs that better leverage human feedback for learning.
In summary, DistQ offers step towards more practical RLHF by directly addressing the need for queries that are both highly informative for the agent and easy to answer for human labelers. By advancing these dual objec- tives, we believe that DistQ contributes a valuable perspec- tive and a robust set of tools to develop more efficient and user-friendly RL.FF systems.
Conclusion
Finally, we ablate the components of our newly designed loss function, CRow (from Equation (9)), which integrates distinguishability preference feedback (C.tew) with standard

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Acknowledgements
model. In Proceedings of the International Conference Learning Representations (ICLR), 2024.
This work has been supported in part by the National Key R&D Program of China (Grant No. 2024YFC3017100), the program of National Natural Science Foundation of China (No. 62176154) which is research project funded by NetEase, and the LMUexcellent project funded by the Federal Ministry of Education and Research (BMBF) and the Free State of Bavaria under the Excellence Strategy of the Federal Government and the Länder as well as by the Hightech Agenda Bavaria. Eyke Hüllermeier has re- ceived funding from the European Union's Horizon Eu- rope research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 101073307. We also thank anonymous reviewers for critically reading the manuscript and suggesting substantial improvements.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, Soft Actor-Critic: Off-Policy Maximum Entropy Deep Re- inforcement Learning with a Stochastic Actor. In Pro-
ceedings of the International Conference on Machine Learning (ICML). PMLR, 2018.
Hu, X., Li, Zhan, X., Jia, Q.-S,, and Zhang, Y.-Q. Query- Policy Misalignment Preference- Based Reinforcement Learning. In Proceedings of the International Conference Learning Representations (ICLR), 2024.
Hüllermeier, E. and Waegeman, W. Aleatoric and epistemic uncertainty machine learning: An introduction to con- cepts and methods. Machine Learning, 110(3)457--56. 2021.
Impact Statement
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in Atari In Advances in Neural Infor- mation Processing Systems (NeurIPS), volume 31. Curran Associates, Inc., 2018
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Jansen, C., Schollmeyer, G., and Augustin, T. Concepts for decision making under severe uncertainty with partial ordinal and partial cardinal preferences. International Journal of Approximate Reasoning, 18:112-131, 2018.
References
Alt, F. Über die Meßbarkeit des Nutzens. Zeitschrift für Nationalékonomie/ / Journal of Economics, 762)1611-16,
1936. ISSN 0044-3158.
Jansen, C., Blocher, H., Augustin, T., and Schollmeyer, G. Information efficient learning of complexly structured preferences: Elicitation procedures and their application to decision making under uncertainty. International Jour- nal of Approximate Reasoning, 1 144:69-91, 2022.
Biyık, E., Palan, M., Landolfi, N. C., Losey, D. P., and Sadigh, D. Asking Easy Questions: A User-Friendly Approach to Active Reward Learning. In Proceedings of the Conference on Robot Learning (CoRI.. PMIR, 2019.
Biyık, E., Losey, D. P., Palan, M., Landolfi, N. C., Shevchuk, G., and Sadigh, D. Learning reward functions from di- verse sources of human feedback: Optimally integrating demonstrations and preferences. The International Jour- nal of Robotics Research, 41(1):45-67, 2022.
Jeon, H. J., Milli, S., and Dragan, A. Reward rational plicit) choice: A unifying formalism for reward learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. Curran Associates, Inc., 2020.
Reward-rational (im-
Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, E. survey of reinforcement learning from human feedback. Transactions Machine Learning Research, 2025. ISSN 2835-8856.
Bradley, R. A. and Terry, M. E. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4):324-34) 1952.
Chan, L., Critch, A., and Dragan, A. Human irrationality: Both bad and good for reward inference, 2021. preprint.
Köbberling, V. Strength of Preference and Cardinal Utility. Economic Theory, 27(2):355-39) 2006. ISSN 0938- 2259.
Christiano, P., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Process- ing Systems (NeurIPS), volume 30. Curran Associates, Inc., 2017.
Krantz, D. H., Suppes, P., and Luce, R. D. Foundations of Measurement. Volume 1: Additive and Polynomial Representations. Courier Corporation, 2006. ISBN 978- 0-486-45314-9.
Dong, Z., Yuan, Y. HAO, J., Ni, F. Mu, Y., ZHENG, Y., Hu, Lv, T., Fan, C., and Hu, Z. Aligndiff: Aligning diverse human preferences via behavior-customisable diffusion
Lee, K., Smith, L., Dragan, A., and Abbeel, P. B-Pref: Benchmarking Preference-Basee Reinforcement Learn- ing. In Proceedings of the Neural Information Processing
10

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), volume 1, 2021a.
Pivato, M. Multiutility representations for incomplete dif- ference preorders. Mathematical Social Sciences, 66(3): 196-220, 2013.
Lee, K., Smith, L. M., and Abbeel, P. PEBBLE: Feedback- Efficient Interactive Reinforcement Learning via Rela- beling Experience and Unsupervised Pre-training. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021b.
Ratcliff, R. theory of memory retrieval. Psychologi- cal Review, 85(2):59-108, 1978. doi: 10.1037/0033. 295X.85.2.59.
Ratcliff, R. and McKoon, G. The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks. Neural computation, 20(4):873-922 2008. doi: 10.1 22111622cco2208..1100 12-06-420.
Li, S., Zhang, Y., Ren, Z., Liang, C.. Li, N., and Shah, A. Enhancing preference-basee based linear bandits via hu- man response time. In Globerson, A., Mackey, L.., Bel- grave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 16852-16893, Curran Associates, Inc., 2024.
Shvartsman, M., Letham, B., Bakshy, E., and Keeley, S. Response time improves gaussian process models for per- ception and preferences. In Kiyavash, N. and Mooij. M. (eds.), Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), volume 244 of Proceedings of Machine Learning Research, pp. 3211-3226. PMLR, 15-19 Jul 2024.
Liang, X., Shu, K., Lee, K., and Abbeel, P. Reward Uncer- tainty for Exploration in Preference-Bane Based Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Suppes, and Winet, M. An Axiomatization of Utility Based on the Notion of Utility Differences. Management Science, 133-4):259-770, 1955.
Liu, R., Bai, F,, Du, Y., and Yang. Y. Meta-Reward-Ne Reward-Net: Implicitly Differentiable Reward Learning for Preference- Based Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS). Curran Asso- ciates, Inc., 2022.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T., and Riedmiller, M. DeepMind Control Suite, 2018 preprint.
Nguyen, V., Shaker, M., and Hüllermeier, E. How to mea- sure uncertainty in uncertainty sampling for active learn- ing. Machine Learning, 111(1):99-22,, 2022.
Verma, M. and Metcalf, K. Hindsight PRIORs for Reward Learning from Human Preferences. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
OpenAl. Introducing ChatGPT, 2022. enai. moom/inndxx cttttt////aattt//...oo (accessed 2024-05- 28).
httsst:::/ //op
Von Neumann, J. and Morgenstern, O. Theory of Games and Economic Behavior. Princeton University Press, edition, 1947.
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 35. Curran Associates, Inc., 2022.
Wagenmakers, E.-.,, Van Der Maas, H. L. J., and Grasman, R. P. P. P. An EZ-diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14(1):3-22, 2007. doi: 10.3758/BF03194023.
Xu, Y., Wang, R., Yang, L., Singh, A., and Dubrawski, A. Preference-baser Reinforcement Learning with Finite- Time Guarantees. In Advances Neural Information Processing Systems (NeurlPS), volume 33, Curran Asso- ciates, Inc., 2020.
Palan, M., Shevchuk, G., Landolfi, N. C., and Sadigh, D. Learning Reward Functions by Integrating Human Demonstrations and Preferences. In Proceedings of Science and Systems (RSS), volume 15, 2019.
Robotics:
Yannakakis, G. N. and Martínez, H. P. Ratings are Over- rated! Frontiers in ICT, 2:13, 2015.
Park, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., and Lee, K. SURF: Semi-Supervised Reward Learning with Data Augmentation for Bredback-Efficien Reinforcement Learning. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2022.
Yu, T. Quillen, D., He, 7.,, Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-Worid: A Benchmark and Evalu- ation for Multi-Task and Meta Reinforcement Learning. Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2019.
Preference-Based

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Yuan, Y., J. Ma, Y., Dong, Z.,, Liang, H., Liu, J., Feng, Z., Zhao, K., and ZHENG, Y. Uni-RLHF: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
12

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
A. Baselines
All baseline methods considered in this paper are under the RLHF framework as we explained in Section 3. this section, we provide more details about these baselines. Specifically, we summarize the query selection methods adopted by DistQ and baselines in Appendix A.
Table Query selection strategies employed by different methods.
Method Query Selection Strategy
DistQ
Larger variance lower entropy
PEBBLE
Larger entropy
SURF
Larger variance
RUNE
Larger variance
MRN
Larger variance
QPA
Random selection from near-on on--olicy buffer
A.1. PEBBLE
PEBBLE (Lee et al., 2021b) refines the preference-based reward learning paradigm introduced by Christiano et al. (2017) by substituting the original on-policy RL algorithm with the off-policy algorithm SAC. further enhances the sample efficiency and stability of the preference- baseed RL framework (as described in Section 3) through three primary mechanisms:
1. Unsupervised Pre-training: To generate diverse initial trajectories and accelerate subsequent reward learning, PEBBLE pre-trains the policy by maximizing the entropy of states encountered during exploration. This contrasts with random initialization and aims to provide a richer starting point for learning from preferences.
2. Entropy-based Query Selection: To select informative PCQs for human labeling, PEBBLE first samples a large batch of candidates. It then preferentially selects pairs that exhibit higher predictive entropy (H((p)) from the current reward model, prioritizing queries where the reward model is most uncertain.
3. Replay Buffer Relabeling: To ensure consistent reward signals for the off-policy SAC algorithm, which can be sensitive non-stationary rewards, PEBBLE periodically re-evaluates and updates the reward labels for transitions stored in its replay buffer using the latest learned reward model.
A.2. SURF
SURF (Park et al., 2022), building upon the PEBBLE framework, aims to reduce the quantity of human feedback required in RLHF while maintaining or even enhancing task performance. This is achieved by introducing data augmentation techniques designed to maximize the utility of available data. Specifically, SURF employs a semi-supervised learning (SSL) approach to generate pseudo-label. for unlabeled trajectory segments and proposes a tailored data augmentation strategy for PCQs. These mechanisms collectively enable SURF to significantly improve the query query-efficiency of RL.F.
A.3. RUNE
RUNE (Liang et al., 2022), which also integrates with the PEBBLE framework, introduces an intrinsic reward mechanism to enhance exploration. This intrinsic reward quantifies novelty by measuring the disagreement across ensemble of learned reward models. By incorporating the uncertainty of reward models directly into the agent's 's total reward signal, RUNE incentivizes the agent to explore regions of the state space where the reward function is least certain, thereby encouraging better exploration.
A.4. MRN
MRN (Liu et al., 2022) is a data-efficient RLHF framework that uses bi-level optimization for concurrent reward and policy learning, learning the Q-function and the policy at the inner level while adapting the reward model to the Q-function performance at the outer level. Specifically, MRN operates in two nested loops:
13

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Inner Loop:
The agent learns Q-function and corresponding policy using conventional RL algorithm,, based the current estimate of the reward function.
The reward function is adaptively updated to improve the consistency of the Q-function with the collected human preference data.
Outer Loop:
By optimizing the reward function based on its downstream impact on policy performance with respect human preferences, MRN aims to significantly improve data efficiency.
A.5. QPA
QPA (Hu et al., 2024) addresses the issue of query-policy misalignment observed in PEBBLE. The authors find that selecting PCQs solely to maximize the overall quality of the learned reward model may not translate improved policy performance, as such queries might not be relevant to the agent's s current learning needs. QPA tackles this by enforcing bidirectional alignment between the queries presented to the human and the agent's evolving policy, This is primarily achieved through two components:
Policy-Aligned Query Selection: Queries are preferentially selected from recent trajectories generated by the current policy. This ensures that human feedback is directly relevant the agent's ongoing behavior, making the feedback more impactful for policy learning.
Hybrid Experience Replay: A specially designed experience replay mechanism is used, which prioritizes recent, on-policy experiences.
B. Experimental Settings
HYPERPARAMETER
VALUE
HYPERPARAMETER
VALUE
General settings
Initial temperature
0.1
Hidden units per each layer
1024(DMControl)
256(Mcta-world)
Length of segment
50
# of layers
2(DMControl)
3(Meta-world)
Learning rate
0.0003
(Meta-world)
Batch Size
1024(DMContrrol
0.0005 (Walker)
512(Meta-worll)
0.0001 (Quadruped, Humanoid)
Optimizer Critic EMA T
Adam
Critic target update freq
2
0.005
(2) Frequency of feedhack
(0.9.0.999)
Discount 7 Maximum budget
0.99
5000 (Meta-world, Humanoid)
10000/50, 500/50, 200/20 (DMControl)
20000 (Walker)
# of queries per session
10000/50, 3000/30,
30000 (Quadruped)
2000/100, 400/10 (Meta-world)
# of ensemble models Non 3
# of pre-training steps
10000
Other settings DistQ
Loss weights (dd, Xp)
(1, 1)
Size of 2p
10x# of queries per session
Size of Qpe (nz)
5xf of queries per session
Size of Qpee (2n
2x# of queries per session
Table 2. Hyperparameters setting
B.1. Implementation Details
DistQ is implemented based on the PEBBLE framework. DistQ and all the baseline methods follow the general hyperpa- rameter configurations described Appendix B. For other specific hyperparameter settings of the basclines, we follow the corresponding publications and published source code.
14

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
For the evaluation tasks, choose the that adopted by the baselines with their corresponding settings of the querying process. This choice is motivated by two considerations: First, given the complex framework combining reward learning and RL training, it may take a lot of effort to tune the hyperparameters for each method to work for new tasks, Second, we try to ensure fair comparison with baselines by avoiding the possibility of choosing specific tasks and tuning hyperparameters that are beneficial to our method. In addition, this showcases the robustness of our method to some extent.
B.2. Computational Resources
For all experiments, we only need one GPU card to launch experiments. Experiments were carried out on different platforms, including GeForce RTX 3060 12G GPU + 48GB memory + Intel Core 17-10700F, or Geforce RTX 3060 12G GPU + 64GB memory + Intel Core 17-12700, or GeForce RTX 2070 SUPER + 32GB memory + Intel Core i7-9700, or GeForce RTX 2060 64GB memory + Intel Core i7-8700.
Note that the training time for specific method varies when different platforms. However, given the same computational platform, all evaluated methods take similar training time (no more than 12 hours even for the task requiring most queries). That is, introducing DistQ does not bring about excessive additional computational cost.
C. Algorithm
Algorithm DistQ
1: Randomly initialize policy model The and reward model Pro
2: Dataset for trajectories B <- 0
3: Dataset for distinguishability feedback 2DO <-
4: /PRE-TRAIN
5: Pre-training as PEBBLE's to obtain B, Te
6: for each iteration do
7: //REWARD LEARNING
8: if Iteration% == 0 then
9:
//SAMPLING QUERIES
10:
Randomly sample segment pairs 2p = ((o9,o')} from B
11:
Calculate informativeness of PCQs in 2p (Equation (5)). Qpc consists of the top my PCQs with larger Z
Calculate easiness of PCQs in 2pm (Equation (6)). 2pee consists of the top and bottom ng PCQs in terms of Sort PCQs in Qpne according E, and pair the i-th with the (i ng)-th to form Qd
14:
QDO' +- Qa with feedback
15:
QDo + QDo U QDo'
16:
I/TRAINING F. ON EXTENDED QDQ
17:
for each gradient step do
18:
Randomly sample minibatch {((of, od), (af, of), pP)} from Qbo
Optimize <Rew (Equation (9)) with respect
end for
21:
end if
22:
//COLLECT TRAJECTORIES
23:
for each timestep t do
24:
Collect interaction data by ~ (apls))) 8141 ~ P(344199,,4)
25:
Store B +- U (St, at, 8t+1)
26:
end for
I/AGENT LEARNING
28: for each gradient step do
29: Optimize T with minibatch ((.,a, randomly sampled from B
30:
end for
31:
end for
15

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
C.1. Pseudo-Code
Algorithm I lists the pseudo-code of our algorithm, differentiating between PEBBLE and our changes with the use of color (changes in orange).
D. Additional Experimental Results and Analysis
D.I. Numerical Experimental Results
To compare the performance of DistQ and other baseline methods shown in Figure more precisely, we list average values of the final performance over five runs for each method in Tables and 4, along with the corresponding standard deviation shown in parentheses. For each task, the highest average performance shown bold and the second highest is italic, except for the upper bound SAC.
Note that the table only shows the final performance of each method. The overall performance needs to be compared via both the learning curves and the numerical values. Tables 3 and 4 show that DistQ with full budget always achieves the best average final performance compared with other baselines on all considered tasks except Humanoid stand, which is consistent with Figure 3. In addition, DistQ with half budget also realizes superior or competitive performance considering all the other baselines on different tasks.
Task
Walker walk [200)
Quadruped walk [500]
Humanoid stand [10000]
SAC
968.42 (8.09)
894.08 (58.34)
542.32 (55.72)
DistQ
909.93 (25.22)
828.67 (56.95)
420.76 (60.15)
DistQ (half)
891.64 (72.68)
538.25 (236.65)
251 (198.28)
PEBBLE
795.28 (127.04)
398.45 (288.37)
233.66 (208.78)
SURF
756.56 (110.78)
257.82 (78.70)
6.69 (1.66)
RUNE
602.69 (301.13)
306.57 (240.40)
298. 15 (159.73)
MRN
877.63 (49.05)
615.08 (207.24)
261.18 18 (233.49)
QPA
874.97 (193.05)
413.22 (298.14)
456.17 (54.38)
Table Episode reward locomotion tasks, with query budget denoted the square brackets, corresponding Figures 3(a) 3(c).
Task
Window open [400]
Door unlock [2000]
Door open [3000]
Sweep into [10000]
Disassemble [10000]
SAC
1.00 (0.00)
1.00 (0.00)
1.00 (0.00)
1.00 (0.00)
0.00 (0.00)
DistQ
0.98 (0.04)
0.98 (0.04)
1.00 (0.00)
0.98 (0.04)
0.20 (0.44)
DistQ (half)
0.78 (0.43)
0.78 (0.43)
0.94 (0.13)
0.78 (0.43)
0.18 (0.40)
PEBBLE
0.54 (0.30)
0.50 (0.46)
0.46 (0.50)
0.70 (0.40)
0.00 (0.00)
SURF
0.80 (0.44)
0.92 (0.17)
0.60 (0.54)
0.58 (0.53)
0.02 (0.04)
RUNE
0.80 (0.39)
0.86 (0.19)
1.00 (0.00)
0.96 (0.08)
0.00 (0.00)
MRN
0.28 (0.42)
0.50 (0.47)
0.80 (0.44)
0.78 (0.43)
0.00 (0.00)
QPA
0.80 (8)
0.69 (0.33)
0.52 (0.45)
0.55 (0.41)
0.008 (0.008)
Table 4. Success rate manipulation tasks, with query budget denoted in the square brackets, corresponding to Figures 3(d) to 3(h).
D.2. Results of Ablation Study
In this section, we provide more results of the ablation study in Figure as mentioned in Section 5.5.
D.3. Detailed User Study
This section provides implementation details for the user study conducted in the Quadruped environment. Since DistQ builds on PEBBLE, an online RLHF framework, we implemented our own user study pipeline rather than using existing platforms designed for offline RLHF (Yuan et al., 2024). While developing systematic platforms for online RLHF remains
16

---

Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Query budget= = 10000
Query budget 10000
100
100
Success Rate (%)
80
Success Rate (%)
80
60
60
40
40
20
20
DQ (I)
DQ (painwise loss)
0
0
PEBBLE
0.0
0.3
0.6
0.9
1.2
1.5
0.0
0.3
0.6
0.9
1.2
1.5
Environment Steps (x 1x
Environment Steps (x10%)
(a) Sweep into
(b) Sweep into
Figure Learning curves of more ablation studies on manipulation task. The solid curves shaded regions represent the mean standard deviation, respectively, across five runs.
open problem for future work, custom implementation enables direct evaluation of method in human feedhack scenarios.
In our designed user study, the human labeler aims to train the agent to stand and wave its right hind leg. For both DistQ and PEBBLE, the labeler is asked 150 queries in total by the agent, which take about 3 hours to answer during the online interleaving of reward learning and agent learning. We evaluate the learned behavior of the agent at the end of training for both methods. Results show that with DistQ, the agent can successfully perform desired behavior. However, with PEBBLE, hardly even stands up. Videos of selected queries and evaluation of trained agents for both methods are available at http:///eeoddo.orgrrcooddd///// 56069999..
17


Reviews:
[{"review_id": "VDtMr07iDh", "reviewer": "Reviewer_ufMA", "summary": "* This paper proposes a novel human feedback type for RLHF and an algorithm allowing robots to learn reward functions from such human feedback.\n  * The novel feedback is that the robot first gives a human 2 pairs of trajectories, has the human choose the pair that is easier to choose, and then has the human choose one trajectory from that pair. In this way, from the 1st choice, the robot can infer the relative preference strength between the 2 pairs. From the 2nd choice, the robot can infer the preference. The benefit is that the robot can understand the preference strength in addition to the preference.\n  * The proposed algorithm, DistQ, allows a robot to learn from such feedback:\n    * Reward learning\n      * Given a trajectory buffer, the robot randomly chooses pairs of trajectories.\n      * The robot chooses the top n1 informative pairs based on variance.\n      * The robot chooses the top and bottom nE easy pairs based on entropy.\n      * The robot forms all the pairs, each of which has an easy trajectory with a hard trajectory.\n      * The robot sends all these pairs to the human for feedback and then uses the feedback to infer the reward function.\n    * Agent learning\n      * The robot uses the estimated reward function to do RL to optimize policy and collect trajectories into the buffer.\n      * Then, go back to the first step to update the reward function\n* The 1st experiment compared the RL performance, given a fixed budget, of the proposed method with 5 baseline methods in simulated robot locomotion and manipulation tasks. The proposed method with full budget (DistQ) outperformed all baselines, while the proposed method with half budget (DistQ(half)) outperformed some of the baselines. A user study is also conducted to show that DistQ is better than PEBBLE.\n* The 2nd experiment compared the query easiness, given a fixed budget, of the proposed method with 5 baseline methods in simulated robot locomotion and manipulation tasks. Both the proposed method with full budget (DistQ) and the proposed method with half budget (DistQ(half)) outperformed all baselines.\n* The 3rd ablation experiment shows that the algorithm design, including choosing n1 informative pairs, choosing nE easy and hard pairs, and assign an easy and a hard trajectories in one pair, are significant to the performance.\n\n## update after rebuttal\nI appreciate the authors' Rebuttal in addressing my concerns. I have adjusted my score accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. About experimental settings: \n\n(1) To address potential misunderstanding in the review, we first clarify some terminologies we used in our paper. One distinguishability query (DQ) consists of 2 pairwise comparison queries (PCQs). The human first chooses the more distinguishable PCQ and then chooses the preferred trajectory in this PCQ (line 202). The \"query budget\" in our paper refers to the number of PCQs (line 359) since all our baselines use PCQs. In the experiments section, for both DistQ and DistQ (half), the human **chooses the preferred trajectory only from the chosen PCQ in a DQ**, instead of \"also the not-chosen pair\" as stated in the review. That is, for each query in both settings, the human always makes 2 choices instead of 3. The difference between DistQ and DistQ (half) is that DistQ uses the same \"query budget\" as other baselines while DistQ (half) only uses half of the budget (line 364). See the first paragraph of Sec 5.2 for detailed instructions. We think both settings are consistent with our proposition. \n\n(2) We then discuss the fairness in our experimental evaluation. As stated in the review, for one DQ, the human makes 2 choices. We understand that this may be considered unfair when comparing with baselines using PCQs where the human only makes 1 choice. That's why we design DistQ (half), which only uses 100 DQs (=200 human choices). However, it's also unfair to only count the number of human choices since such choices for DQ and PCQ apparently provide different amount of information. Therefore, we consider DistQ with full budget. See our explanations in the first paragraph of Sec 5.2. \n\n(3) We next emphasize the effectiveness of DistQ. Given (2), it's hard to have a totally fair comparison among DistQ and baselines. The full and half budget settings actually provide a performance range of our method when compared with its baselines. Therefore, it's normal that DistQ (half) can't beat all its rivals in Fig 3. The fact that DistQ (half) can match or outperform baselines demonstrates in our opinion that it is possible to achieve good performance while asking informative and easier to answer queries. In addition, we believe that answering one DQ is easier than two PCQs (especially when not controlling for their hardness) based on our proposition. Besides, the goal of DistQ is to balance query informativeness and user-friendliness rather than only one of the two. Thus, we need to look at both performance (Fig.3) and query easiness (Fig.5, where our method outperforms others). \n\n(4) As for the suggested setting DistQ (half-half), we think this is identical to DistQ in our paper from the view of query quantification.\n\n2. About easiness measurement: \n\nWe consider the \"predicted feedback\" in terms of human side since we hope to quantify easiness by whether the human can provide correct feedback to queries. Similar query easiness measurement to our \"wrongly predicted feedback\" has also been used in previous work (**see the 2nd paper in the references list**). For the ground truth preference entropy of queries, it seems not very straightforward to see whether queries are easy to answer for humans compared with wrongly predicted number of feedback. But indeed, we acknowledge that this may also be a good measurement for easiness. We will use this entropy as a supplementary easiness measurement in our final version.\n\n3. About essential references: \n\nThank you for suggesting these recent works. We will discuss how our paper is related to them in our final version.\n\n4. About user study: \n\n(1) For each method, 150 queries are answered. We conduct 10 rounds of evaluation after one training (line 402-406). During one training, the human answers 150 queries. \n\n(2) The authors of PEBBLE conducted similar (but easier) user study on Quadruped agent, where 200 PCQs are needed. Besides, they only claimed that the agent could succeed but didn't report the success rate. PEBBLE fails to work in our experiments, because the setting is harder and with a more limited query budget.", "claims_and_evidence": "* The key claim is that the proposed query and algorithm can improve RLHF for robotic tasks. The empirical result supports this.\n* One limitation of the empirical result is about DistQ and DistQ(half). The proposed novel query requires the human to make 2 choices, first choosing the easy pair and then chooses the preferred trajectory in the pair. By contrast, the conventional query requires the human to make 1 choice, choosing the preferred trajectory in the pair. As a result, it is a bit tricky to compare the proposed algorithm with the proposed query with standard RLHF methods.\n  * The paper compared the following 2 variations of the proposed algorithm against the baseline methods:\n    * **DistQ(half)**: In each query, the human first chooses the easy pair, and then chooses the preferred trajectory in the chosen pair. And this query with 2 choices counts as 1 query in the \"query budget\" in the empirical study.\n      * +: This is consistent with the definition of the proposed query (Sec.4.1).\n      * -: Unfortunately, easy pairs contain a limited amount of information, as mentioned under Eq.6. Since the human only provides the preference feedback for the easy pair selected by the human, the overall performance could be limited. This might be why DistQ(half) does not seem to outperform many baseline methods in Fig.3.\n    * **DistQ**: In each query, the human first chooses the easy pair, and then chooses the preferred trajectory in the chosen pair **and also the not-chosen pair**. This query with 3 choices counts as 1 query in the \"query budget\" in the empirical study.\n      * +: This outperformed all baseline methods as in Fig.3.\n      * -: However, this is not consistent with the definition of the proposed query (Sec.4.1). As a result, I think it is not fair to compare DistQ with the baseline methods.\n  * In addition to the 2 variations considered by the authors in the empirical study, I think there is one more variation that could be interesting to consider:\n    * **DistQ(half-half)**: In each query, the human first chooses the easy pair, and then chooses the preferred trajectory in the chosen pair. This query with 2 choices counts as **2 queries** in the \"query budget\" in the empirical study.\n      * The reasoning behind this variation is that the human makes 2 choices, so consuming 2 units of the query budget. This reasoning is consistent with the paper's interest in the query easiness.\n  * Summary\n    * I think it is fair to compare **DistQ(half)** against baseline methods, but it does not seem to perform that well empirically.\n    * I think it is a bit unfair to compare **DistQ** against baseline methods.\n    * I think that the authors could consider **DistQ(half-half)**, which, in my opinion, is also fair to be compared against the baseline method. But I conjecture that it will not perform as well as **DistQ(half)**.", "methods_and_evaluation_criteria": "Method and evaluation criteria make sense. There are 2 limitations.\n* The 1st limitation is already discussed in `# Claims And Evidence*`.\n* The 2nd limitation is that Sec.5.3 measures the query easiness as to whether the robot's predicted feedback is consistent with the ground truth. This definition seems to define the easiness from the robot's view. However, the query easiness is supposed to be defined from the human's view. I think a better way to measure query easiness is to use ground truth preference entropy for this query as defined in Eq.6.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiment design, user study, and analyses make sense.", "relation_to_broader_scientific_literature": "The paper focuses on the robot RLHF, which is a hot topic and also a very relevant problem to machine learning, LLM, and robotics.", "essential_references_not_discussed": "* For Sec.2's Eliciting Preference Strength, two recent works explores the time elicitation in bandits (in the form of human response times), which can be relevant:\n  * Shvartsman, M., Letham, B., Bakshy, E., & Keeley, S. L. (2024, July). Response time improves gaussian process models for perception and preferences. In The 40th Conference on Uncertainty in Artificial Intelligence.\n  * Li, S., Zhang, Y., Ren, Z., Liang, C., Li, N., & Shah, J. A. (2024). Enhancing Preference-based Linear Bandits via Human Response Time. Advances in Neural Information Processing Systems, 37, 16852-16893.", "other_strengths_and_weaknesses": "## Strength\n* The problem is well-motivated\n* The paper is well-written.\n* The proposed query is creative.\n* The paper also contains a real user study to validate the method.", "other_comments_or_suggestions": "NA", "questions_for_authors": "* In the user study (Sec.5.4), there are 10 rounds. Based on App.D.3, in each round, the human answers 150 queries. So, in total, there are 1500 queries? Also, I am curious why PEBBLE has a successful rate of 0, which seems strange. Could you share some insights on why this is the case?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EMIthmgO5z", "reviewer": "Reviewer_pH8H", "summary": "This paper proposes Distinguishability Queries (DistQ), a new method for improving RLHF. DistQ reduces cognitive load by letting humans first choose which of two trajectory comparisons is easier to evaluate and then provide feedback on the easier pair. This approach captures both preference strength and ordinal information, improving the efficiency of learning reward functions. Experiments show that DistQ is more data-efficient and user-friendly than traditional methods, offering a better approach for RLHF in tasks with complex objectives.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. About more experimental environments: \n\nFor the current version of our paper, we selected representative control tasks of different difficulty levels (in terms of query budget needed to accomplish the task) and of different types (locomotion and robotic manipulation) to demonstrate the performance of our method. Following the reviewers' suggestions, **we now validate our method on more tasks, including the hard locomotion task Dog walk (hard), and robotic manipulation tasks Door open (easy) and Disassemble (hard)**. Results are shown in Fig 1 in this link (https://drive.google.com/drive/folders/1wR469npWztzkjyW0YF2H9C10LTI3wnTU). For the latter two tasks, our method can still perform better or competitively compared with other baselines. For the hard Dog walk task, however, all methods (including SAC with ground-truth reward function) fail to achieve it. A possible explanation is that, for such complex tasks, it takes a large number of queries and needs efforts to find workable hyperparameters for the backbone framework. Given the time limit, we haven't found proper settings for all methods and we've tried our best to test on as many tasks as possible. We will include more results in our final version if time permits.\n\n2. About more analysis and experiments: \n\nWe deeply appreciate your approval of our method. Besides synthetic feedback, we also conduct user study with real human providing feedback as discussed in Sec. 5.4. We provide both statistical results and cognitive feedback from the participants here, which evidently support our argument of effectiveness and user-friendliness of DistQ. Larger scale user study may take a large mount of human labor. We will find efficient ways to enrich our user study in future.\n\n3. About application to broader domains:\n\nDistQ is built on top of video-based queries and online interleaving of reward learning and agent learning, so the whole method can be straightforwardly adapted to domains like robotic arm experiments and games. As for offline RLHF tasks, we may need to adjust the implementation of the query selection criteria. But the proposed query type and high-level ideas for query selection can still be used in such settings.\n\n4. About discussion of more related papers:\n\nThank you for suggesting these recent related works. The first two mainly propose efficient and expandable platforms for RLHF experiments with real humans, which can be adopted for efficient large-scale user study. Considering your concerns about more extensive user study, we may utilize such platforms to perform user studies to further validate DistQ. The remaining two instead focus on refining RLHF approaches from different points of view. We will include a discussion about them in our final version.", "claims_and_evidence": "Yes, the claims presented in the text are all supported by experiments. I believe the aspect with some shortcomings is related to alleviating the burden on annotators, which may require more extensive user study.", "methods_and_evaluation_criteria": "Yes, the research question is very meaningful and compares a large number of benchmark algorithms, but the environments used are relatively limited.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The experimental design is scientific and mainstream, aligned with the state-of-the-art PBRL community, and compares with strong baselines.", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n- The paper is written very elegantly, and the research problem is highly significant. Providing annotators with easy and better annotation methods is extremely important.  \n- The aggregation of the distq component is innovative. The experimental results show some improvement over previous methods, and the authors analyze the effects of some design choices.\n\nWeaknesses:\n- I believe in the outstanding performance of distQ, and it indeed compared with many baseline algorithms. However, why wasn’t it tested in more environments (in terms of quantity or type, 9 envs in b-pref totally)? I think this is more important than comparing with more baseline algorithms. If distq could be validated on a broader range of experimental benchmarks, its impact would be greater.  \n- For me, the most interesting aspect of this article is its ability to provide annotators with Informative and Easy Human Feedback. This is very important for RLHF in real-world scenarios, but the article does not delve deeper into this point through more analysis and experiments. Most experiments use synthetic feedback, which makes it difficult to validate this interesting perspective. Appendix D.3 provides simple experimental results, but I believe this deserves more discussion in the main text. This is a valuable approach.  \n- Which part or all parts of DistQ can be applied to broader domains? For example, real-world robotic arm experiments, offline RLHF experiments, or Atari/Minecraft.  \n- Lack of discussion on papers related to improving the quality of annotation human feedback and reducing the burden of human feedback, such as: [1][2][3][4]    \n[1] Yuan Y, et al. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback[J]. arXiv preprint arXiv:2402.02423, 2024.  \n[2] Zhang L, et al. Crew: Facilitating human-ai teaming research[J]. arXiv preprint arXiv:2408.00170, 2024.    \n[3] Dong Z, et al. Aligndiff: Aligning diverse human preferences via behavior-customisable diffusion model[J]. arXiv preprint arXiv:2310.02054, 2023.    \n[4] Metz Y, et al. Reward Learning from Multiple Feedback Types[J]. arXiv preprint arXiv:2502.21038, 2025.", "other_comments_or_suggestions": "see above\n\n---\n**After rebuttal comment:\nThank you for the author's response. I believe the additional experimental results and demonstrations can enhance the quality of the paper, so I vote to accept it and raise the score from 3 to 4. I hope the author can include these additions in the next revised version, adding more experimental content and demonstrations to the experimental section of the main text. This will help better present DistQ.**", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BtfFsrtfo1", "reviewer": "Reviewer_mQWc", "summary": "This paper proposes a three-stage approach to optimizing the PbRL pipeline:  \n\n1. **Selecting the top N informative (based on the variance of reward ensembles) Pairwise Comparison Queries (PCQs)**  for comparison. This step explicitly enables the reward model to distinguish high-uncertainty pairs better, accelerating the reward learning process.  \n\n2. **Selecting the top and bottom M easy (based on entropy) PCQs** for re-pairing. This enhances the contrast of comparison pairs, thereby improving the sample efficiency of reward learning.  \n\n3. **Incorporating an additional distinguishability preference loss** to assist reward learning further.  \n\nIntuitively, all these improvements contribute positively to the PbRL pipeline. Comparative and ablation experiments demonstrate the effectiveness of these stages.\n\n## update after rebuttal\nDuring the rebuttal phase, the author supplemented many materials through anonymous links, such as new experiments on harder tasks, videos of learned policies, details of the user study, and explained some misunderstandings in the discussion section, which led me to increase my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. About experiments on more challenging tasks: \n\n**Please refer to point 1 in our response to reviewer pH8H**. Besides, it is worth mentioning that the suggested harder tasks are not evaluated in all our baselines either. Therefore, we need to determine workable hyper-parameters for all the methods as well as the backbone RL algorithm SAC, which can be time-consuming given the high difficulty of tasks. In our linked results, all methods follow their default hyper-parameter settings without tuning. \n\nAlso, there is possibility that these tasks are already challenging when a ground-truth reward function is available. Our method doesn't claim to improve the sample efficiency of the underlying deep RL algorithm. In contrast, we propose a method to circumvent the need of having to define a ground-truth reward function, by asking informative and easy-to-answer queries than other RLHF methods.\n\n2. Additional experiments with real human feedback:\n\nWe have conducted the suggested experiments with real human involved and explained details in Sec 5.4 and Appendix D.3. To have a comprehensive understanding of the user study, we also provided an anonymous link of videos of selected queries and evaluation of trained agents in Appendix D.3 in our original version of paper.", "claims_and_evidence": "Although the methods proposed in this paper are intuitively reasonable, I believe the experiments are insufficient to support these claims.  \n\n- The experiments are conducted on a very limited set of tasks, including two locomotion tasks and two manipulation tasks. Among them, Walker Walk and Quadruped Walk are relatively easy tasks in DMC, while Window Open and Sweep Into correspond to easy and medium-difficulty tasks in MetaWorld, respectively. If the authors could include more challenging tasks, such as **Humanoid/Dog** in DMC and **Shelf Place/Disassemble** in Meta-World, the experiments would be more convincing.  \n- As a PbRL algorithm, the experiments do not involve **human feedback**. Evaluating only on synthetic feedback fails to demonstrate the algorithm's effectiveness in real-world scenarios.", "methods_and_evaluation_criteria": "Yes. However, I suggest that the authors include additional experiments that go beyond merely maximizing reward and instead use human feedback to shape the agent's behavior, as Pebble did in Figure 6 of its paper.", "theoretical_claims": "This paper includes no proofs or theoretical claims.", "experimental_designs_or_analyses": "See **Claims And Evidence**.", "relation_to_broader_scientific_literature": "The key contributions of the paper are mainly related to preference-based RL.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "See **Claims And Evidence**.", "other_comments_or_suggestions": "No.", "questions_for_authors": "I have no questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No."}}, {"review_id": "CTIPrxY4iJ", "reviewer": "Reviewer_6jG4", "summary": "This paper proposes a new type of query in rlhf, the distinguishability query (DQ). Rather than directly comparing two sets of trajectories, the authors compare two sets of trajectories and selects the one that is easier to give feedback on. They then provide feedback on the easier pair, and can learn from that data. Experimental results show that the method can sometimes produce solid performance gains over the baseline, PEBBLE, but when equal amounts of data are used for the baseline and DQ the performance gain is not that large.\n\n## After Discussion\nAfter the discussion and reading the reviews of the other reviewers, I plan to maintain my score. The method is interesting and the paper seems solid overall. However, it is still unclear to me how to measure the cost of a distinguishability query compared to a normal RLHF query and what a fair comparison means for DQ versus other baselines. I don't really think the authors argument that \"it's also unfair to only count the number of human choices since such choices for DQ and PCQ apparently provide different amount of information\" makes sense, which is the essential claim all of their experimental results rely upon.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Claim about performance of DQ (half): \n\n(1) For query budget and fairness comparison discussion, please refer to **point 1-(1)&(2) in our response to reviewer ufMA**. \n\n(2) For effectiveness, please refer to **point 1-(3) in our response to reviewer ufMA**. Besides, we show in Fig 2 in https://drive.google.com/drive/folders/1wR469npWztzkjyW0YF2H9C10LTI3wnTU that DistQ learning from only distinguishability preference feedback (DQ_d loss) performs far behind DistQ learning from only pairwise preference feedback (DQ_pairwise loss), which demonstrates that DQ (half) actually obtains much less information from a seemingly \"roughly equal labeling budget\" compared with the baselines. In such case, performance of DQ (half) shown in Fig 3 could be recognized.\n\nAt last, to clarify, the comment in line 345 saying DQ (half) \"outperforms most baselines only except for MRN\" is only made on Quadruped walk\", which we think is consistent with Fig 3. As for other tasks, DQ (half) indeed has similar performance to some of the baselines, which is understandable given the above explanation.\n\n2. Overestimation from easy experimental setup:\n\n(1) We've conducted more experiments on harder control tasks. Please refer to **point 1 in our response to reviewer pH8H and mQWc**. Besides, as pointed out by reviewer mQWc, there also exist challenging control tasks to solve. We believe that for those tasks, subtle information in ambiguous queries does matter, and we demonstrate that our method can work.\n\n(2) Note that our method balances query efficiency and user-friendliness by selecting both informative (ambiguous) and easy-to-answer queries as introduced in Sec 4.2, instead of only selecting easy samples to train on. In our experiments (see Sec 5.5), we also conduct an ablation study to compare the performance of our method and an alternative approach that samples queries only based on easiness (DQ (E) curve in plots). The results show that our method considering both ambiguity and user-friendliness is much better than DQ (E).\n\n3. About testing DistQ on LLM Alignment:\n\nIn our current setting for DistQ, the reward model is an ensemble of three-layer neural networks with 256 hidden units, which is quite small compared with the reward model in LLM alignment. Therefore, it may need much more queries and computational resources for LLM alignment, which is hard for us to test given the limited time. \nWe believe the high-level idea of DistQ can be applied for LLM alignment, and we leave this extension to future work.", "claims_and_evidence": "The main claim that DQ can improve RLHF performance is somewhat supported. The experimental results show that DQ can outperform the baseline algorithms when it recieves a large budget then them (see DQ in figure 3). But when a roughly equal labeling budget is given for the baselines and DQ, (i.e. DQ (Half)), the results are not that good and can only really outperform the baseline in two of the four environments presented in Figure 3.\n\nIn the experiments section the authors claim that DQ (Half) outperforms the baselines in all except for MRN on Quadruped walk (line 345). However, this is a but overstated. DQ-half and other algorithms typically have very similar performance (at least in Figure 3) as well as overlapping confidence intervals.", "methods_and_evaluation_criteria": "The experiments and evaluation criteria do make sense. This paper works to improve RLHF, and some of the earliest papers on RLHF focused on continuous control [1]. They evaluate based on the average ground-truth reward received, which makes sense. However their experiments are all conducted on very simple continuous control environments. My concern is that their approach of only selecting easy samples to train on will work better in easy settings than in hard settings. This means their experimental setup may overestimate the utility of their method.\n\n[1] Christiano, Paul F., et al. \"Deep reinforcement learning from human preferences.\" Advances in neural information processing systems 30 (2017).", "theoretical_claims": "na", "experimental_designs_or_analyses": "Yes I did check the soundness of the experimental design. The experimental design and analysis seems both standard and solid. They compare their algorithm with many baselines, and they are careful to make sure the data labelling budget is comparable for different algorithms. In addition, they report five independent runs for each algorithm, and report standard deviation for all experiments.", "relation_to_broader_scientific_literature": "The idea of distinguishing between different queries is definitely a novel idea in RLHF. The paper does a good job of engaging with the literature.", "essential_references_not_discussed": "I do not know of any.", "other_strengths_and_weaknesses": "Strengths:\n- The idea of this paper is novel and interesting.\n- The writing and motivation for their method is very clear.\n- The experimental results are for the most part solid.\n\nWeaknesses:\n- The problem settings used in this paper are very easy problems. In continuous control we have a fairly straightforward goal, so it is fine to rely on non-ambiguous queries to learn it. However for more complex and open ended tasks such as LLM alignment, ambiguous queries may actually contain subtle and important information for reward learning.\n- DQ (half) does not really perform much better than most baselines.", "other_comments_or_suggestions": "none", "questions_for_authors": "Is there any way to test DistQ on LLM Alignment? Or, is it possible to consider a task where subtle information in ambiguous queries matters?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces a novel distinguishability query to improve RLHF for robotics tasks, allowing human annotators to express preference strength by comparing two pairs of trajectories. Labelers first indicate which pair is easier to compare, then provide preference feedback only on the easier pair. This approach captures both preference strength and ordinal information, improving the efficiency of learning reward functions. Experiments show that DistQ is more data-efficient and user-friendly than traditional methods (reducing the cognitive load on the labeler).

The reviewers agree that the proposed method, experiment design, user study, analyses make sense and evaluation criteria are sound. The paper is well-motivated and well-written, addressing a relevant research question.

The authors present new experiments on harder tasks, videos of learned policies, details of the user study, and explained some misunderstandings in the discussion section during the discussion period. The additional experimental results and demonstrations presented enhance the quality of the paper and these should be included in the finalized version. The paper will also greatly benefit from adding more discussion about the cost of distinguishability queries compared to normal RLHF queries (and why a totally fair comparison  among DistQ and baselines is hard --- as discussed in responses to reviewer concerns).
Decision: Accept (poster)