CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
Authors: Ni Mu
Hao Hu
Xiao Hu
Yiqin Yang
Bo XU
Qing-Shan Jia
Venue: ICML 2025 poster (2025)
Keywords: Preference-based RL
contrastive learning
Primary area: reinforcement_learning
TL;DR: We propose a contrastive learning method for offline PbRL that integrates preference information into the trajectory embedding space to select unambiguous queries.

Abstract: Preference-based reinforcement learning (PbRL) bypasses explicit reward engineering by inferring reward functions from human preference comparisons, enabling better alignment with human intentions. However, humans often struggle to label a clear preference between similar segments, reducing label efficiency and limiting PbRL’s real-world applicability. To address this, we propose an offline PbRL method: Contrastive LeArning for ResolvIng Ambiguous Feedback (CLARIFY), which learns a trajectory embedding space that incorporates preference information, ensuring clearly distinguished segments are spaced apart, thus facilitating the selection of more unambiguous queries. Extensive experiments demonstrate that CLARIFY outperforms baselines in both non-ideal teachers and real human feedback settings. Our approach not only selects more distinguished queries but also learns meaningful trajectory embeddings.

CLARIFY:
Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
Ni Mu '1
Hao Hu *
Xiao Hu
Yiqin Yang² 3
Bo Xu
Qing-Shan Jia
Abstract
directly from human feedback or demonstrations, avoiding the need for explicit reward engineering. Preference- based Reinforcement Learning (PbRL) stands out by using hu- man preferences between pairs of trajectory segments as the reward signal (Lee et al., 2021a; Christiano et al., 2017). This framework, based on pairwise comparisons, easy to implement and captures human intent effectively.
Preference- reinforcement learning (PbRL) bypasses explicit reward engineering by infer- ring reward functions from human preference comparisons, enabling better alignment with hu- man intentions. However, humans often strug- gle to label a clear preference between similar segments, reducing label efficiency and limiting PbRL's real-world applicability. To address this, we propose offline PbRL method: Contrastive LeAminn for ResolvIng Ambiguous Feedback (CLARIFY), which learns a trajectory embed- ding space that incorporates preference informa- tion, ensuring clearly distinguished segments are spaced apart, thus facilitating the selection of more unambiguous queries. Extensive experi- ments demonstrate that CLARIFY outperforms baselines in both non-ideal teachers and real hu- man feedback settings. Our approach not only selects more distinguished queries but also learns meaningful trajectory embeddings.
However, when trajectory segments are highly similar, it becomes difficult for humans to differentiate between them and identify subtle differences (Inman, 2006; Bencze et al., 2021). Therefore, existing PbRL methods (Lee et al., 2021a; Liang et al., 2022; Park et al., 2022) face this significant challenge of ambiguous queries: humans struggle to give a clear signal for highly similar segments, leading ambigu- ous preferences. As highlighted in previous work (Mu et al., 2024), this problem of ambiguous queries not only hinders labeling efficiency but also restricts the practical application of PbRL in real-world settings.
In this paper, we focus on solving this problem in offline PbRL by selecting larger proportion of unambiguous queries. We propose novel method, CLARIFY, which uses contrastive learning to learn trajectory embeddings and incorporate preference information, as outlined in Fig- ure 1. CLARIFY features two contrastive losses to learn a meaningful embedding space. The ambiguity loss lever- ages the ambiguity information of queries, increasing the distance between embeddings of clearly distinguished seg- ments while reducing the distance between ambiguous ones. The quadrilateral loss utilizes the preference information of queries, modeling the relationship between better and worse-performing segments as quadrilaterals. With theoret- ical guarantees, the two losses allow us to select more un- ambiguous queries based on the embedding space, thereby improving label efficiency.
1. Introduction
Reinforcement Learning (RL) has achieved remarkable suc- cess in various domains, including robotics (Kalashnikov et al., 2018; Ju et al., 2022), gaming (Mnih et al., 2015; Ibarz et al., 2018), and autonomous systems (Schulman, 2015; Bellemare et al., 2020). However, a fundamental chal- lenge in RL is the need for well-defined reward functions, which can be consuming and complex to design, espe- cially when aligning them with human intent. To address this challenge, several approaches have emerged to learn
"qqual contribution 'eejjing Key Laboratory Embodied Intel- ligence Systems, Department of Automation, Tsinghua University, Beijing, China "Monnhoo AI. Beijing, China 'The Key Labora- tory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. Correspondence to: Yiqin Yang syiqin.yange @ ia.ac.cn>>, Qing-Shan Jia <jiaqs @ tsinghua. tsinghua.cdu.cn>>.
Extensive experiments show the effectiveness CLARIFY. First, CLARIFY outperforms the state-of-the-art offline PbRL methods under non-ideal feedback from both scripted teachers and real human labelers. Second, CLARIFY sig- nificantly increases the proportion of unambiguous queries. We conduct human experiments to demonstrate that the queries selected by CLARIFY are more clearly distin- guished, thereby improving human labeling efficiency. Fi-
edu.en>>.
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
Train Trajectory Encoder f (o) through Contrastive Leamnnng
Select Clearly Distinguished Queries through Reject Sampling
Good segment
Probability
Segment embedding vector
Ambiguous
Ambiguuus query
Cleady- distinguished query
segment
Clearly Distingushhe
demp (l6) - (o
Non-idral teacher
Sample query (dp, with higher demb
Figure I: The framework of CLARIFY. 1) Train the trajectory embeddings via contrastive learning, incorporating preference information, 2) Using the embedding space, select clearly distinguished queries for non-ideal teachers via reject sampling.
nally, the visualization analysis of the learned embedding space reveals that segments with clearer distinctions are widely separated while similar segments are closely clus- tered together. This clustering behavior confirms the mean- ingfulness and coherence of the embedding space.
with the ambiguous query issue. However, the method of Mu et al. (2024) cannot be applied to offline settings. This paper focuses on addressing this issue in the offline PbRL..
Contrastive learning for RL. Contrastive learning is widely used self-supervised learning to differentiate be- tween positive and negative samples and extract meaningful features (Oord et al., 2018; Chen & He, 2021). In RL, has been applied representation learning in image-based RL (Laskin et al., 2020; Yuan & Lu, 2022) to improve learning efficiency, and to temporal distance learning (Myers et al., 2024; Jiang et al., 2024) to encourage exploration. In this pa- per, we apply contrastive learning to incorporate preference into trajectory representations for offline PbRL.
2. Related Work
Preference-based RL (PhRL), PbRL allows agents to learn from human preferences between pairs of trajectory segments, eliminating the need for reward engineering (Christiano et al., 2017). Previous PbRL. works focus on improving feedback efficiency via query selection (Ibarz et al., 2018; Biyik et al., 2020), unsupervised pretraining (Lee et al., 2021a; Mu al., 2024), and feedback augmenta- tion (Park al., 2022; Choi et al., 2024). Also, PbRI. has been successfully applied to fine-tuning large language mod- els (LLMs) (Ouyang et al., 2022), where human feedback serves as more accessible alternative to reward functions.
3. Preliminaries
Preference-based RL. In RL, an agent aims to maximize the cumulative discounted rewards in a Markov Decision Process (MDP), defined by a tuple (5, A, are the state and action space, P = P(- a) is the environ- ment transition dynamics, r = r(s, a) is the reward function, and y is the discount factor. In offline PbRL, the true reward function is unknown, and we have offline dataset D without reward signals. We request preference feedback p for two trajectory segments do and of length H sampled from D. p 0 denotes is preferred (a) > oo). = 1 denotes the opposite (01 < oo). In cases where segments are too similar to distinguish, we set p = no_cop, and this label is skipped during learning. The preference data (an, ,I,P) forms the preference dataset D,-
S and A
In offline PbRL, an agent learns a policy from an offline trajectory dataset without reward signals, alongside pref- erence feedback on segment pairs provided by teacher. Traditional methods (Shin et al., 2023) follow two- woo-phase process: training reward model from preference feedback, and then performing offline RI. Some works (Kim et al., 2022; Gao et al., 2024) focus on preference modeling, while others (Hejna & Sadigh, 2024; Kang et al., 2023) optimize policies directly from preference feedhack.
Learning from non-ideal feedback. Despite PbRL's po- tential, real-world human feedback is often non-ideal, growing area of concern. To address this, Lee et al. (2021b) proposes a model of non-ideal feedback. Cheng et al. (2024)
Conventional offline PbRL. methods estimate reward func- tion F = (s a) from D, and use it to train policy wo via offline RL. The Preference model, typically based on the Bradley- Terry model (Bradley & Terry, 1952), estimates the probability Py 10 > ou] as:
assumes random label errors, identifying outliers via loss function values. Xue et al. (2023) improves reward ro- bustness through regularization constraints. However, these approaches still rely on idealized models of feedback, which diverge from real human decision-making Mu et al. (2024) is perhaps the closest work to ours, addressing the challenge of labeling similar segments in online PbRL, which aligns
exp E, P (s) a!)
Polos ao] Σ Eie(0,1) exp E, (s af)
(I)
The following cross-entropy loss is minimized:
2

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
10
1.0
++
08
08
Nomallrae Retum
06
06
0..6
04
04
02
02
0.5
0.7
0.0
0.6
0.8
1.5
0.0
0.
0.4
046
06
1.0
0.0
(
(b)
(c)
Figure Illustration quadrilateral loss Lqund- (a) demonstration of the main idea of Lquad (b) An embedding
visualization of the intuitive example Section 4.1. (c) True embedding visualization results the benchmark experiments
on the "drawer-open" task, under skip rate 0.5 and 0.7.
similar trajectory segments, making the query ambiguous. This issue, validated in human experiments (Mu et al., 2024), hinders the practical application of PbRL. While previous work (Mu et al., 2024) tackles this issue in online PbRL settings, it remains unsolved offline settings.
min Preward
E (mo + plog Polor > ool|
(1 p) log Po > oi]
To tackle ambiguous queries offline PbRL, we propose novel method, Contrastive LeArning for ResolvIng Am- biguous Feedback (CLARIFY), which maximizes the se- lection of clearly-distigguiseed queries to improve human labeling efficiency. Our approach is based on contrastive learning. integrating preference information into trajectory embeddings. In the learned embedding space, clearly dis- tinguished segments are sell-separated, while ambiguous segments remain close, as detailed in Section 4.1. Based this embedding, introduce query selection method in Section 4.2 to select more unambiguous queries. The overall framework of CLARIFY is illustrated in Figure I and Algorithm 1, with implementation details provided in Section 4.3.
Contrastive learning is self- supervised learning approach, which learns meaningful rep- resentations by comparing pairs of samples. The basic idea to bring similar (positive) samples closer in the embedding space while pushing dissimilar (negative) samples apart. Early works (Chopra et al., 2005) directly optimize this objective, using the following loss function:
Contrastive learning.
N
Coontrastive
MI - 21112
2N
(3)
+ ((NAA max(0, (0, m - -112a (1ex-=;l12)²) 2/112)²
where 24 is the representation of sample i. Iy,=,, and (ww))))) are indicator functions, and m a pre-defined mar- gin. InfoNCE loss (Oord et al., 2018) is widely used in recent works, which is formulated as:
4.1. Representation Learning
In this subsection, we first formalize the problem of preference-basee representation learning. Our goal is to train an encoder 2 = fo(T), where a dieedddimessioall embedding of trajectory т. We leverage preference dataset Dp containing tuples (Øo, 11. p). where do and Ø1 tra- jectory segments, and p E {0, 1, no_cop) indicates the preference: = 0 if ao is preferred, = if o is preferred, and p no_cop if the segments are too similar to compare.
LIafONCE log exp(sim(22 24, 2t)t)
(4)
exp(siml 2x)/t)
where sim(...) is the similarity function, 2 denotes the pos- itive sample of 2;, and is temperature scaling parameter. In our work, we apply contrastive learning to model trajec- tory representations, incorporating preference information. fz(r) models 7, and its corresponding meference-based loss functions are detailed in Section 4.1.
The encoder
trajectory
A well-structured embedding space should maximize the distance between clearly distinguished segments while min- imizing the distance between ambiguous ones. In such space, trajectories with similar performance are close, while those with large performance gaps are far apart. This results in meaningful and coherent embedding space, where high-performance trajectories form one cluster, low-
4. Method
In this section, we start by introducing the issue of ambigu- ous queries: humans struggle to clearly distinguish between
3

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
performance trajectories form another, and intermediate trajectories smoothly transition between them. To achieve this, we propose two contrastive learning losses.
An intuitive example. To demonstrate the effectiveness of the quadrilateral loss, we conducted simple experiment. We generated 1000 data points with values uniformly dis- tributed between 0 and 1, initializing each data point's ' em- bedding as a 2-dim vector sampled from a standard normal distribution. We then optimized these embeddings using the quadrilateral loss. As shown in Figure 2(b), the learned embedding space exhibits a smooth and meaningful distribu- tion, with higher-valued data points transitioning gradually to lower-valued ones. Please refer to Appendix C.3 for experimental details.
Ambiguity loss Lamb- The first loss, called the ambiguity loss, directly optimizes this goal. It maximizes the distance between the embeddings of clearly distinguished segment pairs and minimizes the distance between ambiguous ones:
min Lamb
E Mza, 21)
(mT pp)-D,. (E(11
(5)
E (00.01-p)~DA Ppi panio dop
(20,
(20.21).. 21)]
4.2. Query Selection
where 20 = fo(oo), 21 = fo(o1), and f(.,)) is the distance metric.
This section presents query selection method based on rejection sampling to select more unambiguous queries. For given query (oo, p). we calculate the embedding dis- tance demb = {(f((o)) (a1)) between its two segments. For a batch of queries, we can obtain a distribution p(denb) of the embedding distances. We aim to manipulate this dis- tribution using rejection sampling increase the proportion of clearly distinguished queries.
However, relying solely on the ambiguity loss Lamb can cause several issues. First, when the preference dataset is small, continuous optimization of Lamb may lead to over- fitting. Additionally, Lamb alone can cause representation collapse, where ambiguous segments are mapped to the same point in the embedding space. This is because Lamb only leverages ambiguity information (whether segments are distinguishable) but ignores preference relations (which segment is better). The quadrilateral loss Lyuad, introduced below, mitigates this issue by acting as a regularizer, ensur- ing the embedding space better captures the full preference structure and improving representation quality.
We define the rejection sampling distribution (demm)) ) as follows. First, we estimate the density functions Petr (demb) and Pamb (demb) = 1 - (ete(dem)) for distil--ittgguishhee and ambiguous pairs, using the existing preference dataset Dp. These densities reflect the likelihood of observing distance demh for each type of segment pair. Next, compute new density function:
Quadrilateral loss Lqund- To address these issues, we introduce the quadrilateral loss Equad. which directly models preference relations between segment pairs, better capturing the underlying structure in the embedding space. Specif- ically, for two clearly distinguished queries (-+ and (o4,,o'), of,,'") where 0. and o', are preferred over a_ and o'_. we create a quadrilateral-shhng shaped relationship between their embeddings. By utilizing pairs of queries, the training data size grows from (n)) to (nn)), mitigating the overfitting issue caused by limited preference data.
max (0, Pete (ate(dems) Parnto (clemb))
Pi (demmt) I mmx (0, Petr (d') Pamm (cl) )) dil'
P2(demb) = Pete Pete(deem))/ / Pamb (demt) (Pete(d")/ Panb ( dddd
p(demb) = 0.5(p1 (dems) + P2 (2(ammb))
This density function emphasizes the distances where clearly distinguished segments are more frequent than ambiguous ones. Finally, we multiply this new density function by the original distribution p(demb) to obtain the rejection sampling distribution g(
The core idea fquad is illustrated in Figure 2(a). For each pair of clearly distinguished queries, treat 0+ and o'_, as "positive" samples, as they are preferred over a_ and o'_. the "negative" samples. The quadrilateral loss encourages the sum of distances between positive samples o., o', and between negative samples a_, o²_ to be smaller than the sum of distances within positive or negative pairs (i.e., between a+ and σ_, or o4 and a_). Formally, this is expressed as: min Equad=
q(demb)== = p(demb) p(demb),
(8)
which increases the likelihood of selecting queries with clearly distinguished segment pairs, improving labeling effi- ciency. For the remaining queries after rejection sampling, we follow prior work (Lee et al., 2021a; Shin et al., 2023) by selecting those with maximum disagreement.
E (a Ma' mr 1)- [((zt,z]]
+ +lxxt1,27 ,t,,,t/)= -f(z-, ,2-
4.3. Implementation Details
(6)
Embedding training. For embedding training in Section 4.1, we adopt Bi-directional Decision Transformer (BDT) architecture (Furuta et al., 2021a), where the encoder z
where 2t = folo4), = -o(004) = -f(lo)),2 fo(o'_)

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
Table I: Success rates on Metaworld tasks (the first 7 tasks) and episodic returns on DMControl tasks (the last 2 tasks), over 6 random seeds. We use skip rate e = 0.5, 0.7 and report the average performance and standard deviation of the last 5 trained policies. The yellow and gray shading represent the best and second-best performances, respectively.
Skip Rate
Algorithm
box-close
dial-turn
drawer-open
handle -pull-side
hammer
peg- peg-insert -side
sweep-into
cheetah-run
walker-walk
IQL
94,90 1.2
76.50
1.2
98.60 1 0.45
99.30 1.
72.00 2.30
88.40
1.30
79.40
1 1.91
607.46 1 k.s
830.15 m0os
MR
0.26 = o00
14.46
5:577
50.47
= 6.34
79.73 +: 12.19
0.14
= 0.06
9:23
2000
23.25
205.04
= 10.53
322.93 =: 161.77
OPRL.
8.25 Ms
$7.33
1 15.01
72.67 2.7
83,92 198
14.80 153
22,00
4.44
61.00
1.5
531.96
4:75
646.40
11:35
0.5
PT
0.15 2:00
30.54
1:34
61.64
= 10.7
89.75
607
0.11 2010
10:20
2.94
46.35
1M
384.59
19.06
599.43
19:43
OPPO
0.56 ooee
13.06
1: 1224
11.67
0.56
2.78 1:47
0.00
1000
15.56
11.57
346.01
1: 12107
311.27
1: 4273
LIRE
3.60 1.00=.m
46. 20 1
63.47
10:18
52.07
16.20 1537
21.60
57.47
1
553.61
789.18
76.77
CLARIFY
29.40
77.50
83,50
95.00)
26.75
24.25
68,00
617.31
796.34
MR
0.20
16.57
45.51
1015
74.82
0.06
5.21 2.3
17.22 :
234.77
306.39
13472
OPRL.
7.40 1
63.40 1.4
46.50
1. 18:3
84,00
1:1
5.00 2.76
23.75
-776
$2.00 1.333
513.94
1 15:45
664.16 1. 19417
0.18
25.64 1,40
64.72
18.4
74.94
0.09 0.05
8.21
4.89
28.52 #
429.19
44.92
647.66
19.53
OPPO
0.56 0.79
2.78 2.33
11.67
622
0.56
0.79
4.44 6.29
0.00
0:00
15,56 11.57
355.69
59.35
304.19
16:25
LIRE
7.67 16.79
38.40
+ 1.52
39.10
+ 6.78
50.60
31.59
14.25
7.30
23.40
3.40
56.00 + 6.40
514.75
1002
795.02
22:80
CLARIFY
17.50 6.87
79.40
3.83
78.60 10.52
95.00
1.10
28.75
15.58
23.00
1.22
59.67
10.09
$93.24 = 22:75
816.54 11.05
(T) and the decoder à = Jt/ (8,2) are Transformer- based. The model is trained with a reconstruction loss:
proposed objective. We present two key propositions show- ing that the losses Lquat and Lamb ensure: 1) margin separa- tion for distinguishable samples, and 2) convex separability of preference signals in the embedding space. The first property guarantees meaningful geometric embedding for selecting distinguishable samples, while the second ensures that the embedding space does not collapse and maintains a clear separation between positive and negative samples.
min Lrecon 0.0'
E TrD IT (s, (o(T)) (8,0)~T
Appendix provides more details on BDT. Also, to stabilize training, we constrain the 1.2 norm of the embeddings to be close to 1. Without this constraint, embeddings may either grow unbounded or collapse to the origin, both of which can cause training to fail:
Margin guarantees for disentangled representations. Proposition 5.1 formalizes how Lamb enforces geometric separation between trajectories with distinguishable value differences.
min Zoorm E max (1ll.
(10)
The final loss function combines these terms:
Proposition 5.1 (Positive Separation Margin under Optimal Ambiguity Loss). Let Dp be a distribution over labeled tra- jectory pairs (0o, F1,P) with E {0, 1, no cop}. Define P = {(00, 01) E {0, 13} and N = {(00. 1p=
C Crecon + Xamih Camhh + Eqund + AnormLnome (11)
where Aamb, Aquad, Anomn are hyperparameters. We use the L2 distance the distance metric ..,.).
no no.cop). Let
Rejection sampling. For rejection sampling in Section 4.2, following the successful discretization of low-dimensional features in prior RI. works (Bellemare et al., 2017; Furuta et al., 2021b), we discretize the embedding distance demb it into Roin intervals to handle continuous distributions.
Φ- = arg min Camb())
2, = (o:),
and set
d min
inf (co.o))E
f(z0. 21);
dmax sup E(20, 21),
Based the above discussion, the overall process of CLARR- IFY is summarized in Algorithm 1. First, we randomly sample query batch to pretrain the encoder and reward model. Next, we select queries based on the pretrained em- bedding space, update the preference dataset Dr and reward model, and retrain the embedding using the updated pp- Finally, we train the policy Te using a standard offline RL algorithm, such IQL (Kostrikov al., 2021). Additional implementation details are provided in Appendix C.2.
where f(zo, 1 zill. tfe class { } is continuous and sufficiently expressive, and supp(Dp) is compact in (oo, -ssppce,, then there exists 5 > 0 such that
dtin 2 dmax + +520 0.
Proof. Please refer to Appendix A. for detailed proof.
This proposition guarantees that the ambiguity-loss min- imizer ø² yields not only a correct ordering of pairwise distances but also enforces a strict margin 8 > 0:
5. Theoretical Analysis
This section establishes the theoretical foundation of CLAR- IFY's embedding framework and provides insights into the
inf 1 - -2 > (oo.01)EP
sup (Go.o))EN
| - =1 6.
5

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
Nor Lodd Norum
64-
64
22
22
07-
62
02
50:
60
24
26
20
01
01
04
01
69
60
62
64
66
6a
10
69
60
10
0.6
(a) dial-turn
(b) drawer-open
hammer
(d) walker-walk
Figure 3: Visualizations of the learned embedding spaces under e = 0.5, where segments with high returns (bright color) and low returns (dark color) are clearly separated into distinct clusters, with smooth transition between their centers.
In effect, CLARIFY's objective Lamh provably carves out uniform buffer 8 around all distinguishable trajectory pairs, endowing the learned embedding space with a nontrivial geometric margin that enhances robustness and separability.
sis to explain, available in Appendix A.3. Overall, CLAR- IFY's framework combines three principles: 1) Margin max- imization for query disambiguation, 2) Convex geometric separation for reliable hyperplane decisions, 3) Dynami- cally balanced contrastive gradients for stable embedding learning. Together, these properties ensure that the learned representation is both geometrically meaningful (aligned with trajectory values) and algorithmically useful (enabling efficient query selection).
Convex geometry of preference signals. The second propo- sition characterizes how Lquad induces convex separability in the embedding space. We show that minimizing this loss directly translates to constructing robust decision boundary between preferred and non-preferred trajectories:
Proposition 5.2 (Convex Separability). Assume the positive and negative samples are distributed in two convex sets C+ and C- in the embedding space. Let ju+ = TR[2*] p²====
6. Experiments
We designed our experiments answer the following ques- tions: Q1: How does CLARIFY compare other state-of- the-art methods under non-ideal teachers? Q2: Can CLAR- IFY improve label efficiency by query selection? Q3: Can CLARIFY learn a meaningful embedding space for trajec- tory representation? Q4: What is the contribution of each of the proposed techniques in CLARIFY?
B[2²] denote the class centroids. If Lquad is minimized with a margin 1) > then there exists a hyperplane H defined by:
H f2 Rd w + bb
such that for all 2 C+ and x E C
d(zt,H) 2 n and d(= HH) S -1
6.1. Setups
where d(z, H) (w b =)/l|u|| denotes the signed dis- tance from H.
Dataset and tasks. Previous offline PbRL studies often use D4RL (Fu et al., 2020) for evaluation, but D4RL is shown to be insensitive to reward learning due to the "survival in- stinct" (Li al., 2023), where performance can remain high even with wrong rewards (Shin et al., 2023). To address this, we use the offline dataset presented by Choi et al. (2024) with Metaworld (Yu et al., 2020) and DMControl (Tassa et al., 2018), which has been proven to be suitable for re- ward learning (Choi et al., 2024). Specifically, we choose complex Metaworld tasks: box-close, dial-turn, drawer- open, haande--uull-ste hammer, peg-insert-sice sweep-into, and 2 complex DMControl tasks: cheetah-run, walker-wall Task details are provided in Appendix C.I.
Proof. Please refer to Appendix A.2 for detailed proof. D
The second proposition characterizes the separability achieved in the embedding space. The centroid separation yr+ μ² acts as global discriminator, while the margin n ensures local robustness against ambiguous samples near decision boundaries. This result has two key implications: 1) The existence of a separating hyperplane H. guarantees linear separability of preferences, enabling simple query selection policies (e.g., margin-based sampling) to achieve high labeling efficiency. 2) The margin 7) directly quan- tifies the "safety gap" against ambiguous queries, where any query pair within 2n distance would be automatically filtered out unreliable. This mathematically substantiates our method's sss ability to actively avoid ambiguous queries during human feedback collection.
Baselines. We compare CLARIFY with several state-of- the-art methods, including Markovian Reward (MR), Pref- erence Transformer (PT) (Kim et al., 2022), OPRL (Shin et al., 2023), OPPO (Kang al., 2023), and LiRE (Choi et al., 2024). MR is based on Markovian reward model using MLP layers, serving the baseline in PT, which uses Transformer for reward modeling. OPRL leverages reward
To better illustrate the dynamics of the embedding space of the proposed losses, we provide a gradient- brasee analy-
6

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
CLARITY
CLARITY
OPRL
Table 2: Performance of CLARIFY and MR using different numbers of queries, under skip rate c = 0.5.
Query clarity rabio (96)
100
Accuracy (%)
of Queries
dial-tum
sweep-into
CLARIFY
MR
CLARIFY
MR
100
59.50 + 4.67
49.50 = 10.16
54.00 2.16
49.67 4:888
dial-turn hammmer
walker-walk
dial-turn
hammer
Wa wallerr walk
500
77.25 6.8
50.50 =: 5.98
68.67
= 1.70
56.67
= 4.08
Figure 4: Query clarity ratio and accuracy of human labels for CLARIFY and OPRL, with CLARIFY-selected queries are more clearly distinguished for humans.
1000
77.50
+ 3.0L
57.33
+: 5002
68.00 3.19
61.00
= 7.52
2000
77.80 1: 6.10
59.00
4: 5.7
68.75 1.48
63.25 1: 6.00
Table 3: Ratios of clearly-disiigguished queries under the non-ideal teacher with 0.5, for CLARIFY and baselines.
ensembles and selects queries with maximum disagreement. OPPO also learns trajectory embeddings but optimizes the policy directly in the embedding space. LiRE uses the listwise comparison to augment the feedback. Following prior work, use IQL (Kostrikov et al., 2021) to optimize the policy after reward learning. Also, we train IQL with ground truth rewards performance upper bound. More implementation details are provided in Appendix C.2.
dial-turn
hammer
sweep-into
walker-walk
CLARIFY MR OPRL PT 43.90%
76.33%
62.67%
68.67%
46,86 %
46.95%
50.33%
36.67%
36.28
31.67%
12.67%
25.60%
15.64%
49.33%
31.67%
37.90%
turn, hammer, and walker-walk. Labelers provide 20 prefer- ence labels per run over 3 seeds, selecting the segment best achieving the task (e.g.. upright posture walker-walk). Unclear queries are skipped. Appendix D details task objec- tives and prompts.
Non-ideal teacher design. Following prior works (Lee et al., 2021a; Shin et al., 2023), we use a scripted teacher for systematic evaluation, which provides preferences be- tween segments based on the sum of ground truth rewards Ti-- To better mimic human decision-making uncertainty, we introduce a "skip" mechanism. When the performance difference between two segments do. σ1 is marginal, that is,
We evaluate with:
1) Query clarity: the ratio of queries with human-provided preferences.
Σ rg)(s,a) Eis,a)en (assa)emTTasss00 a) < cH Tavg (12) (s.0)Eon
2) Accuracy: agreement between human labels and ground truth.
Figure 4 confirms CLARIFY's superior query clarity and accuracy, validating its effectiveness.
the teacher skips the query by assigning p = no cop. Here, H is the segment length, and rawg is the average ground truth reward for transitions in offline dataset D. We refer to e E (0, 1) as the skip rate, This model similar to the "threshold" mechanism in Choi et al. (2024), but differs from the "skip" teacher in B-Pref (Lee et al., 2021b), which skips segments with too small returns.
Embedding space visualizations. We visualize the learned embedding space using I-SNE in Figure 3. Each point repre- sents a segment embedding, colored by its normalized return value. For most tasks, high-performance segments (bright colors) and low-performance segments (dark colors) form distinct clusters with smooth transitions, indicating that the embedding space effectively captures performance differ- ences. The balanced distribution of points further demon- strates the stability and quality of the learned representation, highlighting CLARIFY's ability to create a meaningful and coherent embedding space.
6.2. Evaluation the Offline PbRL Benchmark
Benchmark results. We compare CLARIFY with baselines on Metaworld and DMControl. Table shows that skipping degrades MR's performance, even with PT. LiRE improves MR via listwise comparison, while OPRL enhances perfor- mance by selecting maximally disagreed queries. OPPO is similarly affected by skipping, performing on par with MR. In contrast, CLARIFY selects clearer queries, improving reward learning and achieving the best results in most tasks.
6.3. Human Experiments
Validation of the non-ideal teacher. To validate the ap- propriateness of our non-ideal teacher (Section 6.1), we conduct human labeling experiment, where labelers pro- vide preferences between segment pairs with varying return differences. The results in Figure 5 show that as the return difference increases, both the query clarity ratio and accu- racy improve. This suggests that when the return difference is small, humans struggle to distinguish between segments,
Enhanced query clarity. To assess CLARIFY's ability to select unambiguous queries, we compare the distinguish- able query ratio under a non-ideal teacher. Table 3 shows CLARIFY achieves higher query clarity across tasks.
We further validate this via human experiments on dial-
7

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
Query clarity raio (%)
Large Return Difference (RD)
Table 5: Performance of CLARIFY and OPRL on walker- walk task, under real human labelers.
Medium
Small
100
75
CLARIFY
OPRL.
50
Episodic Returns
420.75 :: 52.02
265.91 1: 33.57
20
Query Clarity Ratio (%)
63.33 + 8.50
53.33 + 624
dial-tum
hammer
walker-walk
Accuracy (%)
87.08 9.15
66.67 4
Large Return Difference (RD)
Medium RD
Small RD
Table 6: Performance of CLARIFY with and without opti- mizing Lamb and Lquad, under skip rate 0.5.
100
Accuracy
Lumb
Lquad
dial-turn
sweep-into
60
X
63.20
40.00
dial-tum
hammer
walker-walk
69.00 11:20
52.80 #: 17.01
Figure 5: Query clarity ratio and accuracy of human labels for queries with varying return differences (RD). For dial- turn and hammer, RD values 300, 100, and 10; for walker-walk, RD are 30, 10, and 1.
/ 71.25
62.20 4.92
/ 77.50 + 3.0
68.00 + 3.19
combination is crucial the method's success.
Table 4: Performance of CLARIFY and Par-based query selection method, under skip rate 0.5.
Ablation on the query selection. We compare two query selection methods: CLARIFY 's rejection sampling and density-based approach that selects the highest-density queries Pat(d), aiming for the most clearly distinguished queries. As shown in Table 4, the density-based method per- forms poorly, likely due to selecting overly similar queries, reducing diversity in the queries. In contrast, CLARIFY selects more diverse set of unambiguous queries, yielding better performance.
Query Selection
dial-tum
sweep-into
CLARIFY
77.50 + 301
68.00 1 3.19
Par-based
61.33 5.11
48.60 + 10.74
aligning with our assumption that small return differences lead to ambiguous queries.
Human evaluation. To evaluate CLARIFY's perfor- mance with real human preferences, we conduct experi- ments comparing CLARIFY with OPRL on the walker- walk task across 3 random seeds. Human labelers provide 100 feedback samples per run, with preference batch size M = 20. As shown Table CLARIFY outperforms OPRL in policy performance, suggesting that our reward models have higher quality. Additionally, queries selected by CLARIFY have higher clarity ratios and accuracy in human labeling, indicating that our approach improves the preference labeling process by selecting more clearly distin- guished queries. For more details on the human experiments, please refer to Appendix D. We believe these results indicate CLARIFY's potential in real-world applications, especially those involving human feedback, such as LLM alignment.
Enhanced query efficiency. We compare the performance of CLARIFY and MR using different numbers of queries. As shown in Table 2, CLARIFYY outperforms MR consis- tently, even if only 100 queries are provided. The result demonstrates CLARIFY's ability to make better use of the limited feedback budget.
Ablation on hyperparameters. As Figure 6 visualizes, when varying the values of Adiat Aquad, and Anoems the qual- ity of embedding space remains largely unaffected. This demonstrates the robustness of CLARIFY to hyperparame- ter changes, as small adjustments do not significantly alter the quality of learned embeddings.
7. Conclusion
This paper presents CLARIFY, a novel approach that en- hances PbRL by addressing the ambiguous query issue. Leveraging contrastive learning, CLARIFY learns trajec- tory embeddings with preference information and employs reject sampling select more elearly distinguished queries, improving label efficiency. Experiments show CLARIFY outperforms existing methods in policy performance and labeling efficiency, generating high-quality embeddings that boost human feedback accuracy in real-world applications, making PbRL. more practical.
6.4. Ablation Study
Component analysis. To assess the impact of each con- trastive loss in CLARIFY, we incrementally apply the ambi- guity loss Lamb and the quadrilateral loss fquad- As shown in Table 6, without either loss, performance is similar to OPRL. Using only Lamb yields unstable results due to overfitting early in training. When only Equad is applied, performance improves but with slower convergence. When both losses are used, the best results are achieved, showing that their
8

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
Acknowledgments
Cheng, J., Xiong, G., Dai, X., Miao, Q. Lv, Y., and Wang. F.-
Y. Rime: Robust preference based reinforcement learning with noisy preferences. arXiv preprint arXiv:2402.17257, 2024.
This work is supported by the Strategic Priority Re- search Program of the Chinese Academy of Sciences (N2X00027040200) the NSFC (No. 62125304, 62192751, and 62073182), the Beijing Natural Science Foundation (L233005), the BNRist project (BNR2024TD03000) and the 111 International Collaboration Project (B25027), The author would like to express gratitude to Yao Luan for the constructive suggestions for this paper.
Choi, H., Jung, S., Ahn, H.,, and Moon, Listwise re- ward estimation for offline preference-basee reinforce- ment learning. arXiv preprint arXiv:2408. 090.00940 2024.
Chopra, S., Hadsell, R., and LeCun, Y. Learning a sim- ilarity metric discriminatively, with application to face verification. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR ((),. IEEE, 2005.
Impact Statement
We believe that CLARIFY has the potential to significantly improve the alignment of reinforcement learning agents with human preferences, enabling more precise and adaptable Al systems. Such advancements could have broad soci- etal implications, particularly in domains like healthcare, education, and autonomous systems, where understanding and responding to human intent is crucial. By enhancing the ability of AI to align with diverse human values and preferences, CLARIFY could promote greater trust in AI technologies, facilitate their adoption in high-stakes envi- ronments, and ensure that Al systems operate ethically and responsibly. These developments could contribute to the re- sponsible advancement of AI technologies, improving their safety, fairness, and applicability in real-world scenarios.
volume 1, pp.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances neural information pro- cessing systems, 30, 2017.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Furuta, H., Matsuo, Y., and Gu, S. S. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:211 2 10364, 2021a.
Furuta, H., Matsushima, T., Kozuno, T., Matsuo, Y., Levine, S., Nachum, O., and Gu, S. S. Policy information capac- measure for task complexity in deep reinforcement learning. In International Conference on Machine Learning, pp. 3541 -3552. PMLR, 2021b.
References
ity: Information-thect
Bellemare, M. G., Dabney, W., and Munos, R. distribu- tional perspective reinforcement learning. In Inter- national conference machine learning, pp. 449-458.
Gao, C.-X., Fang, S., Xiao, C., Yu, Y., and Zhang, Z. Hind- sight preference learning for offline preference-based re- inforcement learning. arXiv preprint arXiv:2407.0445.. 2024.
PMLR. 2017.
Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-82, 2020.
Hejna, J. and Sadigh, D. Inverse preference learning: Preference-baie based without a reward function. Advances in Neural Information Processing Systems. 36, 2024.
Beneze, D., Szöllösi, Á., and Racsmány, M. Learning to distinguish: shared perceptual features and discrinination practice tune behavioural pattern separation. Memory, 29 (5):605-2211 2021.
Ibarz, B., Leike. J., Pohlen, T., Irving. G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Biyik, E., Huynh, N., Kochenderfer, M., and Sadigh, D. Active preference-based gaussian process regression for reward learning. In Robotics: Science and Systems, 2020.
Inman, M. Tuning in to how neurons distinguish between stimuli. PLoS Biology, 4(4)xe118, 2006.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 3334)333244345 1952.
Jiang, Y., Liu, Q., Yang, Y., Ma, X., Zhong, D., Bo, X., Yang,
J., Liang, B., Zhang, C., and Zhao, Q. Episodic novelty through temporal distance. In porrnsically-Mottivdd and Open- Ended Learning Workshop @ NeurIPS2024, 2024.
Chen, X. and He, K. Exploring simple siamese represen- tation learning. In Proceedings of the IEFE/CVF con- ference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Ju, H., Juan, R., Gomez, R., Nakamura, K., and Li, G. Transferring policy of deep reinforcement learning from
9

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
simulation to reality for robotics. Nature Machine Intelli- gence, 4(12):1077-1088) 2022.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv: 1el1:0880703346 2018.
Kalashnikov, D., Irpan, A., Pastor. P., Ibarz, J., Herzog, A., Jang, H.,, Quillen, D., Holly, E., Kalakrishnan, M., Van- houcke, V., al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference robot learming, pp. 651-673. PMLR, 2018.
Ouyang. L., Wu, J., Jiang. X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.. al. Training language models follow instructions with human feedback. Advances in neural information processing systems,
2022.
Kang, Y., Shi, D., Liu, J.. He. L., and Wang, D. Beyond reward: Offline preference-guider policy optimization. arXiv preprint arXiv:2305.1622 16217, 2023.
Park, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., and Lee, K. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based re- inforcement learning. arXiv preprint arXiv:2203. 100500, 2022.
Kim, C., Park, J., Shin, J., Lee, H., Abbecl, P., and Lee, K. Preference transformer: Modeling human preferences using transformers for rl. In The Eleventh International Conference on Learning Representations, 2022.
Schulman, J. Trust region policy optimization. arXiv preprint 1rXtr:1502.0547, 2015.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint WrXiv:2110006166 2021.
Shin, D., Dragan, A. D., and Brown, D. S. Benchmarks and algorithms for offline preference- preeeeence-based reward learning. arXiv preprint wrXiv:2301001992 2023.
Laskin, M., Srinivas, A., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp. 5639-5650, PMLR, 2020.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., et al. Deepmind control suite. arXiv preprint 1rXiv:10110000 2018.
Lec, K. Smith, L., and Abbeel, P. Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2/06.0509), 2021a.
Xue, W., An, B., Yan, S., and Xu, Z. Reinforcement learn- ing from diverse human preferences. arXiv preprint arXiv:2301. 2023.
Lee, K., Smith, L., Dragan, A., and Abbeel, B-pref: Benchmarking preference-based reinforcement learning. arXiv preprint arXiv:2111.03022 2021
Yu, T. Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100 1100. PMLR, 2020.
Li, A., Misra, D., Kolobov, A., and Cheng, C.-A. Survival instinct in offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Yuan, H. and Lu, Z. Robust task representations for of- fline rentforeinor learning via contrastive learning. International Conference on Machine Learning, pp- 25747-25759, PMLR, 2022.
Liang, X.,
Shu, K., Lee, K., and Abbeel, P. Reward uncer- tainty for exploration in preference-based reinforcement learning. arXiv preprint arXiv:2205.
2022.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., al. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533, 2015.
Mu, N., Luan, Y., Yang, Y., and Jia, Q-s. S-epoa: Overcom- ing the indistinguishabiliyy of segments with skill-driven preference-based reinforcement learning. arXiv preprint arXiv:2408.1 12030, 2024.
Myers, V., Zheng, C., Dragan, A., Levine, S., and Eysen- bach, B. Learning temporal distances: Contrastive succes- sor features can provide a metric structure for decision- making. arXiv preprint arXiv:2406. / 17098, 2024.
10

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
A. Additional Proofs and Theoretical Analysis
A.1. Proof of Proposition 5.1: Margin Lower Bound
Theorem A.1 (Proposition 5.1, restated). Let Dy be a distribution over trajectory-pairs (00, where p {0, 1, no.co). Define
P (1) e (0,1)),
N {(ou, pp= no.copp
(13)
Let 6* = arg min Camb( (6). Assume:
1. (Compactness)) The support of D, in (oo, 01) -sppec is compact, and each fo is continuous.
2. (Richness) For any two pairs (=², with (t) P and (=ō, = with (% E N, there exists an infinitesimal perturbation of ø that simultaneously changes E(z), ,z) and {(zu ;T) without affect other embeddings.
Write = for (ox) and define
min
inf (z0, 21), d sup (PP.
(1) (dg.oi)EN
(14)
Then there exists a constant such that
dmax + 0.
(15)
Proof. We first show that
d+
dmaxx
(16)
Set
A(Ø) == ( (ea.m)eel(((((o)ttt(((a)) B(Ø) == (o..))) ((fo)))e(((((ou))ff((())))
(17)
so that (omb(()) - A(ò) + B(0). Since Φ² is global minimizer,
Camb(o") S Lamb + AΦ)
+ s -A(Ø" + + + B(Ø + AΦ)
(18)
for every infinitesimal admissible Ac. Suppose for contradiction that
dmax Then there exist one distinguished pair (a ott) and one ambiguous pair (a, ,r) such that
d
(19)
( + .
(20)
Because é is strictly increasing in the Euclidean norm, we choose tiny 0 and perturb
= + +5 ut
z - - - fut, 20 + 20 = 21 + 2j +
u
(21)
where tt is the unit-vector from 2 to = (and similarly u"), as to increase {( 2t) by some 8' > 0 and decrease (25 x1) by some 8" > 0. By the richness assumption, this arises from some o² + Ad.
Under this perturbation,
A(d* Ad) A(d*) N= 8' (higher-order terms),
B(Ø + Aω) B(Φ*)
+ 8"
(22)
so
Lamb + Ao) - Camb((")
8'
8" <
0
(23)
NF
N
>0
for sufficiently small e. This contradicts the global optimality of Φ² Therefore
a+
d
(24)
mnin

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
claimed.
Then show the strict gap via compactness. Since the support of Dy is compact and for, é are continuous, the image sets
st [((20.21) (oo. 01) P}, 5= {E(=0. 21) (oo. σ1) N
(25)
compact and, by the separation result, disjoint in R. Two disjoint compact subsets of IR have strictly positive gap:
= inf g - sup s 0.
(26)
Therefore
d inf s
sup 8 + 6 = d 6,
(27)
completing the proof.
A.2. Proof of Proposition 5.2: Convex Separability
Proposition A.2 (Margin Guarantee for Lquad). Let C+ and C- Rd be convex, and write
B[2+], 1 = =E[=-]
n* 1 b= b= - 1111" 113).
(28)
Define the signed distance d((=) = (w = b)/ |w:| If d globally minimizes
+() Lquad = -15 {(= ) + ((<_,=-)
e(24 2(24122)
(29)
for every zt E + and = EC
d(z²) 2 7, d(z-) s -7,
(30)
where 1) = 111p ||-
Proof.
Writing out the expectation and differentiating under the integral sign, we can find
eLquad (w - (ut - -} = 0, Ob
dcquad 2 (b +11 ||x])13))
(31)
Dw
Hence, any global minimum,
w = n++ 4 = - (11ut |1,-111
(32)
direct expansion shows
w T z + [ - 1= 11=112 ||2 1 111))
(33)
so
|| - 11 11
1
d(u
2 l|
n, d(u -17.
(34)
2|| 11
Since d((z) is an affine function of and C+, C- are convex, the entire set C+ must lie on or beyond the level set (d = 11),
and C" on beyond {d ) Equivalently,
d(zt) n d(z-) n,
(35)
for every t E C+2 27 E C-, as claimed.
12

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
A.3. Gradient Dynamics of Lquad and Lmbb
The gradient expressions reveal two key mechanisms that govern the dynamics of the embedding:
Parametric Alignment (Equaa): The gradient 22 quid 2(=*' - induces relational drift, which does not move 2 towards fixed coordinates but instead navigates based the relative positions of its augmented counterpart z²' and the negative sample 2-'. This leads to self-supervised clustering effect, where trajectories with similar preference labels naturally group together in tight neighborhoods.
Discriminant Scaling (Camb): The distance-maximization gradient distinguishable pairs, -2(20 - 2 1), enforces repulsive dynamics, similar to the Coulomb forces between same-charge particles. contrast, the gradient for indistinguishable pairs, 2( 21), acts like spring forces, pulling uncertain samples toward neutral regions. This dynamic equilibrium prevents the embedding from collapsing while maintaining geometric coherence with the preference signals.
Together, these dynamics resemble potential, in physies: ones as repulsors, and ambiguous regions as saddle points. This resulting geometry directly facilitates our query selection objective.
trajectories act as attractors, low-preference
14
08
0.8
08
0.6
044
0.2
10
(a) 50.1,5)
(b) 10.1,
(c) (1,
(d) (5,
Figure 6: Embedding visualizations of the drawer-open task with e 0.5, under different hyperparameters settings. The
subfigure captions of are the values of (Aambr Aquad), and (Namb, Aquad) (0.1, 1) is the setting in experiments.
B. Additional Experimental Results
Ablation hyperparameters. As Figure 6 visualizes, when varying the values of Aamb Aquaa and Awams the quality of embedding space remains largely unaffected. This demonstrates the robustness of CLARIFY to hyperparameter changes, as small adjustments do not significantly alter the quality of learned embeddings.
C. Experimental Details
C.1. Tasks
The robotic manipulation tasks from Metaworld (Yu al., 2020) and the locomotion tasks from DMControl (Tassa et al., 2018) used in our experiments shown in Figure 7. The descriptions of these tasks follows.
Metaworld Tasks:
1. Box-close: An agent controls a robotic arm to close a box lid from an open position.
2. Dial-turn: An agent controls a robotic arm rotate dial to target angle.
3. Drawer-open: An agent controls a robotic arm grasp and pull open drawer.
4. Handle-pull-side: An agent controls a robotic arm to pull a handle sideways to a target position.
5. Hammer: An agent controls robotic arm to pick up a hammer and use drive nail into a surface.
13

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
Algorithm 1 The proposed offline PbRL method using CLARIFY embedding
Require:
Offline dataset D with no reward signal, total feedback number Nooral query batch size M, number of discretization Tinn number of gradient steps Ainits Remb, Rrwward coefficients Aamb- Aquad Anorm
// Reward learning
1: Initialize preference dataset D, - 0
2: Randomly select M queries {(To, 01, p) M. update Dp +- ((o0, σ1, p)} U DDp
3: Optimize the encoder fo based on Eq. by minit gradient steps
4: Optimize the reward model ng based on Eq. 2 by Reeward gradient steps
5: while total feedback Novaal do
6: Select M queries {(oo, p) based the query selection method in Section 4.2, update Dp + p))MM
Dp
7: Optimize the encoder f& based on Eq. 11 by Nemb gradient steps
8: Optimize the reward model ب based on Eq. 2 by Theward gradient steps
9: end while
// Policy learning
10: Label the offline dataset D using reward model to
11: Train the IQL policy va using the relabeled dataset D
6. Peg-insert-side: An agent controls robotic arm to insert peg into a hole from the side.
7. Sweep-into: agent controls a robotic arm sweep an object into target area.
DMControl Tasks:
1. Cheetah-run: A planar biped is trained control its body and run on the ground.
2. Walker-walk: A planar walker is trained to control its body and walk on the ground.
We the offline dataset from LiRE (Choi et al., 2024) for experiments. LiRE allows for control over dataset quality by adjusting the size of the replay buffer (replay buffer size data quality value 100000), which provides different levels of dataset quality. The dataset quality in our experiments differs from the one used in LiRE, as detailed in Table 7. The number of total preference feedback used in our experiments is detailed in Table 8.
(a) Box-close
(b) Dial-turn
(c) Drawer-open
(d) Hammer
(e) Handle-pull-side
(f) peg-insert-side
(g) Sweep-into
(h) Cheetah-run
Walker-walk
Figure 7: Nine tasks from Metaworld (a-g) and DMControl (h, i).
14

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
Table The dataset quality in our experiments. Specifically, to prevent all the methods from failing under our teacher with the skip mechanism, enhanced the dataset quality for several tasks.
Task
Value of LiRE
Value CLARIFY
Box-close
8.0
9.0
Dial-turn
3.5
3.5
Drawer-open
1.0
1.0
Hammer
5.0
5.0
Handle-pull-side
1.0
2.5
Peg-insert-side
5.0
5.0
Sweep-into
1.0
1.5
Cheetab-run
/
6.0
Walker-walk
1.0
1.0
Table 8: Total feedback number Nootll
Task
Value
Metaworld tasks
1000
Cheetah-run 500
malker-walk
200
C.2. Implementation Details
In our experiments, MR, PT, OPRL, LiRE, and CLARIFY are all two-step PbRL methods. In these methods, the reward model is first trained, followed by offline RL using the trained reward model. The reward models used in CLARIFY, MR, and OPRL share the same structure, as outlined in Table 9. We use the trained reward model to estimate the reward for every (s, a) pair in the offline RL dataset, and we apply min-max normalization to the reward values so that the minimum and maximum values are mapped to O and 1, respectively. We use IQL as the default offline RL algorithm. The total number of gradient descent steps in offline RL is 200,000, and evaluate the success rate episodic return for 20 episodes every 5,000 steps. For all baselines and our method, we run 6 different seeds. We report the average success rate episodic return of the last five trained policies. The hyperparameters for offline policy learning are provided in Table 9.
We follow the official implementations of MR and PT', OPPO?, and LiRE Note that LiRE treats queries with a too-small reward difference as equally preferred (p = 0.5), while in our setting, these queries are labeled as no.cop and excluded from reward learning.
For the BDT implementation in CLARIFY, we follow the implementation of HIM (Furuta al., 2021a)*, using the BERT architecture the encoder and the GPT architecture as the decoder.
The code repository of our method is:
The hyperparameters for both the baselines and our method are listed in Table 10.
C.3. Details of the Intuitive Example
To validate the effectiveness of proposed Lqund, we generated 1,000 data points with values uniformly distributed between 0 and 1. Each data point's 's embedding was initialized as a 2-dimensional vector sampled from standard normal distribution. To simulate the issue of ambiguous queries, we defined queries with value differences smaller than 0.3 as ambiguous. We
http:///hhbb.coom////mmell////ee Transfommer
http:///gihuu..coo///wwo//////// "hupp://githubb. "hanss//gituhh.om///////g/g/ com//fnnn//ggeeeerriieeedeed
15

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
then optimized these embeddings using the quadrilateral loss. To ensure stability during training, we imposed a penalty constrain the 1.2 norm of the embedding vectors to The learning rate was set to 0.1, and the hyperparameters Aquad and Anomm were set to and 0.1, respectively.
Table 9: Hyperparameters reward learning and policy learning.
Hyperparameter
Value
Optimizer
Adam
Learning rate
3e-4
Segment length H Batch size
50
128
Hidden layer dim
256
Reward model
Hidden layers
3
Activation function Final activation
ReLU
Tanh
# of updates rewward
50
of ensembles Reward from the ensemble models
3
Average
Query batch size M
50
Optimizer
Adam
Critic, Actor, Value hidden dim
256
Critic, Actor, Value hidden layers
2
Critic, Actor, Value activation function
ReLU
IQL
Critic, Actor, Value learning rate Mini-batch size
0.5
256
Discount factor
0.99
B
3.0
0.7
D. Human Experiments
Preference collection. We collect feedback from human labelers (the authors) familiar with the tasks. Specifically, they watched video renderings of each segment and selected the that better achieved the objective. Each trajectory segment was 1.5 seconds long, consisting of 50 timesteps. For Figure 5, labelers provided labels for 20 queries for each reward difference across 3 random seeds. For Figure 4, we ran 3 random seeds for each method, with labelers providing 20 preference labels for each run. For Table 5, we ran 3 random seeds for each method, with labelers providing 100 preference labels for each rum, and the preference batch size M = 20.
Prompts given human labelers. The prompts below describe the task objectives and guide preference judgments.
Metaworld dial-turn task.
Task Purpose:
In this task, you will be comparing two segments of a robotic arm trying to turn a dial. Your goal is to evaluate which segment performs better in achieving the task's objectives.
Instructions:
Step I: First, choose the segment where the robot's arm reaches the dial more accurately (the reach component).
Step 2: If the reach performance is the same in both segments, then choose the one where the robot's gripper closed more appropriately (the gripper closed component).
Step 3: If both reach and gripper closure are equal, choose the segment that has the robot's arm placed closer to the target position (the in in-place component).
16

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
Metaworld hammer task.
Task Purpose:
In this task, you will be comparing two segments where a robotic arm is hammering a nail. The aim is to evaluate which segment results a better execution of the hammering process.
Instructions:
Step 1: First, choose the segment where the hammerhead is in a hetter position and the nail is properly hit (the in-place component).
Step 2: If the hammerhead positioning similar in both segments, choose the one where the robot is better holding the hammer and the nail (the grab component).
Step 3: both the hammerhead position and grasping are the same, select the segment where the orientation of the hammer more suitable (the quaternion component).
DMControl walker-wall task.
Task Purpose:
In this task, you will compare two segments where bipedal robot is attempting to walk. Your goal to determine which segment shows better walking performance.
Instructions:
Step 1: First, choose the segment where the robot stands more stably (the standing reward).
Step 2: If both segments have the same stability, choose the one where the robot moves faster or more smoothly (the move reward).
Step 3: both standing and moving are comparable, select the segment where the robot maintains a better upright posture (the upright reward).
E. Introduction to Bi-directional Decision Transformer
Introduction to Hindsight Information Matching (HIM). Hindsight Information Matching (HIM) (Furuta et al., 2021a) aims to improve offline reinforcement learning by aligning the state distribution learned policies with expert demonstrations, Instead of directly imitating actions, HIM minimizes the discrepancy between the marginal state distributions of the learned and expert policies, ensuring that the agent visits states similar those in expert trajectories. This approach is particularly effective for multi-task and imitation learning settings, where aligning state distributions across different tasks can enhance generalization.
The main idea of BDT. BDT is designed for offline one-shot imitation learning, also known offline multi-task imitation learning. In this setting, the agent must generalize to unseen tasks after observing only a few expert demonstrations. Standard Decision Transformer (DT) faces challenges such scenarios due to its reliance on autoregressive action prediction and task-specific fine-tuning, In contrast, BDT improves generalization by utilizing bidirectional context, enabling to infer useful patterns from limited data. This makes BDT particularly effective for imitation learning tasks where the agent needs to efficiently adapt new environments with just small number of demonstrations.
BDT extends the Decision Transformer (DT) framework by incorporating bidirectional context, which improves sequence modeling in offline reinforcement learning. Unlike standard DT, which predicts actions autoregressively using causal attention, BDT adds an anti-causal transformer that processes the trajectory in reverse order. This anti-causal component enables BDT to use both past and future information, making it highly effective for Ktte-marginal matching and imitation learning.
17

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
The architecture of BDT. In this paper, utilize the encoder-decoder framework of BDT to learn the trajectory representation. The architecture of BDT consists of two transformer networks: causal transformer that models forward dependencies in the trajectory, and an anti-causal transformer that captures backward dependencies. This bidirectional structure allows BDT to extract richer temporal patterns. improving its ability to learn from diverse offline datasets.
18

---

CLARIFY: Contrastive Preference RL for Untangling Ambiguous Queries
Table 10: Hyperparameters of baselines and CLARIFY.
Hyperparameter
Value
Initial preference labels
M 50
OPRL
Query selection
Maximizing disagreement
Optimizer
AdamW
of layers
PT
# of attention heads
4
Embedding dimension
256
Dropout rate
0.1
Optimizer
Adam
Learning rate
le-4
x* learning rate
le-3
Number of layers 4
Number of attention heads
4
Activation function
ReLU
Triplet loss margin
OPPO
Batch size
256
Context length Embedding dimension Dropout
50
16
0.1
Grad norm clip
0.25
Weight decay
le-4
a
0.5
3
0.1
Reward model
Linear
LiRE
RLT feedback limit Q
100
Embedding dimension
16
Activation function
ReLU
Aamb
0.1
Aquai
Anorm
0.1
Batch size
256
Context length
50
CLARIFY
Dropout
0.1
Number of layers 4
Number of attention heads
4
Grad norm clip
0.25
Weight decay
le-4
Tiiii
20000
Nemb
2000
19


Reviews:
[{"review_id": "OVd1oU80j1", "reviewer": "Reviewer_DiW3", "summary": "This paper presents CLARIFY, an offline preference-based reinforcement learning (PbRL) algorithm, that leverages contrastive learning to organise the embedding space which is used to learn the reward function.\n\nDuring the reward-learning phase, CLARIFY alternates between learning a reward via Bradley-Terry and a contrastive objective that encourages preferred state-action pairs to cluster together.\n\nThis paper additionally shows that under CLARIFY: i) the distance between two trajectory embeddings is has a lower-bound, and that ii) there is a hyperplane that separates the embeddings of all preferred trajectories, and all dispreferred trajectories.\n\nExperiments show that Metaworld and DMControl tasks, where IQL policies using the reward learned by it CLARIFY outperforms in all but one scenario recent offline preference-based learning baselines (including OPRL, PT, OPPO and LiRE).\n\n**Post-rebuttal update**\n\nThe main concern about this paper was whether CLARIFY was robust to noisier labellers, with additional ablations showing that it indeed was. There were also additional concerns regarding the clarity of the text and the data flow of the method that were equally clarified during rebuttal.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for your valuable and detailed comments.\n**We conducted additional experiments and the results are shown in the [link](https://docs.google.com/document/d/e/2PACX-1vS7v9XEpXMFrH0skymO1RQUiXP2lcnnRoP114HpluBSSpvxE3vuRHNYJ1RwlggWB-rlihxrpdeVv53O/pub).**\n\n**C1.1: Robustness to non-ideal teachers.**\n\n**A for C1.1:**\nWe evaluated CLARIFY with a new flipping teacher that randomly assigns preferences for close-reward queries. Table 1 shows that CLARIFY outperforms OPRL by over 20% in success rate, confirming its robustness to varied non-ideal feedback.\n\n**C1.2: CLARIFY's behavior across $\\epsilon$ values.**\n\n**A for C1.2:** We evaluated CLARIFY with $\\epsilon$ in 0$\\sim$0.7. As shown in Table 2, at $\\epsilon=0.3$, CLARIFY outperforms OPRL by 8~18\\%. In contrast, at $\\epsilon=0$, CLARIFY's query selection reverts to random sampling, leading to performance comparable to MR. This suggests that CLARIFY excels when distinguishing a query is difficult.\n\n**C2.1: Additional component analysis.**\n\n**A for C2.1:** We conduct component analysis on DMControl tasks with $\\epsilon$=0.7. Tables 3 and 4 show that full CLARIFY ($\\mathcal L_\\text{amb}$ + $\\mathcal L_\\text{quad}$ + rejection sampling) consistently performs best, confirming component necessity.\n\n**C2.2: Comparison to PEBBLE.**\n\n**A for C2.2:** We compare CLARIFY to uncertainty-based sampling (PEBBLE). Table 5 shows that CLARIFY achieves better performance, demonstrating the effectiveness of our query selection strategy.\n\n**C2.3: Architecture impact.**\n\n**A for C2.3:** We illustrate our architecture in Figure 1, which shows that CLARIFY operates in two strictly decoupled phases. In the embedding training phase, the Bi-directional Decision Transformer encoder trains solely on trajectories using contrastive and reconstruction losses, without access to reward model or policy. In the reward learning phase, the reward model updates only on preference data without access to embedding information. This design prevents the encoder from leaking privileged information about future states or rewards.\n\n\n**Answer for Theoretical, Weaknesses, Other Comments:**\n- (**Theoretical**) Prop 5.2: corrected to $d(z^-,H)\\ge\\eta$.\n- (**Other Comments**) `no_cop` denotes cases where labelers consider queries too similar to specify a preference, resulting in a skipped label.\n- (**W1**) Query clarity ratio is defined as the proportion of clearly-distinguished queries to the total number of queries.\n- (**W2**) Clearly-distinguished queries are those where human preferences are clear (not `no_cop`).\n- (**W3**) Safety impact: A discussion of malicious use risks was added.\n\n**Q1, Q2, Q3, Q6: Clarity of the statement.**\n\n- **A for Q1:** Eq. 9 is integrated into the total training objective (Eq. 11) used in Algorithm 1 line 3. It is not a simple pretraining but trains the encoder continuously.\n- **A for Q2:** Distance metric $\\ell$ is the Euclidean distance. We conduct additional experiments to compare the Euclidean and Cosine distances. Figure 2 shows that points in the embedding trained with Cosine distance cluster together, while Euclidean distance is more suitable for trajectory embedding.\n- **A for Q3:** Query number: 1000 for MetaWorld tasks, 500 for cheetah-run, 200 for walker-walk (Table 10).\n- **A for Q6:** The thresholds of Large, Medium, and Small in MetaWorld and DMControl tasks are 300/100/10 and 30/10/1 respectively.\n\n**Q4: Performance with fewer queries.**\n\n**A for Q4:** We evaluate CLARIFY with 50 to 2000 queries. Table 6 shows a slight performance decline for CLARIFY with only 50 queries, though it still outperforms MR. Table 7 shows CLARIFY's performance with 100 queries on various tasks, which reaches about 70\\% of IQL's performance with ground truth rewards.\n\n**Q5: Statistical validation for walker-walk.**\n\n**A for Q5:** We conduct a statistical significance analysis for CLARIFY and OPRL.\n- Table 8 shows the 95% confidence intervals (CIs) of the query clarity ratio and label accuracy. The narrow intervals validate CLARIFY’s performance. Though walker-walk shows overlapping CI due to high environment stochasticity, CLARIFY's directional improvements in both metrics demonstrate the effectiveness of its query selection.\n- Additionally, we conducted independent two-sample t-tests comparing CLARIFY with OPRL. The experimental results in Table 9 show that CLARIFY achieves statistically significant improvements ($p<$0.05) in 5/6 tasks. These results confirm the advantage of CLARIFY's query selection over OPRL.\n\n**Q7: T-SNE visualization robustness.**\n\n**A for Q7:**\nThanks for pointing out this issue, and we conduct visualizations across multiple random seeds (Figure 3), revealing consistent clustering patterns, indicating robust embeddings. Additionally, we provide PCA (Figure 4) visualizations to support our conclusion.\n\nWe hope that our response has cleared most of your concerns.", "claims_and_evidence": "The following claims are made in the paper:\n\n**C1. CLARIFY provides better performance than current offline PbRL baselines under non-ideal teachers**\n\nThis claim is mostly supported by the thorough experiments in Table 1, where CLARIFY beats the baseline in all tasks except Metaworld's `peg-insert-side`.\n\nThe robustness of CLARIFY to different non-ideal scripted teachers is not sufficiently investigated however. The teacher presented in CLARIFY can skip trajectories whose ground-truth reward is close, but will never flip preferences (ie choosing a non-preferred trajectory over a preferred one), or skip preferences where the ground-truth rewards are large.\n\nEven with the presented scripted teacher, it is hard to gauge what is the effect of $\\epsilon$ in the expected performance. How does CLARIFY perform if $\\epsilon$ is ~0.3? What happens if $\\epsilon$ is zero (ie no trajectory is labelled as `no_cop`)? It is not necessary for CLARIFY to outperform the baselines in these situations, but it is important to characterise its behaviour so that the community can understand when is best to use CLARIFY.\n\n**C2. These improvements are both due to the space separation induced by CLARIFY, and the rejection sampling technique presented in the paper.**\n\nTable 4, and 6 shows that best results are obtained when all elements of CLARIFY ($L_{amb}$, $L_{quad}$, and rejection sampling) are active. However, this analysis is only carried out for two meta-world tasks and with $\\epsilon=0.5$. Other epsilon values and a few DMControl tasks should also be analysed.\n\nIn table 4, it would make sense to compare against an uncertainty-based sampling method like the one used in PEBBLE (Lee et al 2021a in the paper's bibliography).\n\nLastly, CLARIFY uses a Bidirectional Transformer, it is conceivable that the encoder of the transformer (which as far as I understand consumes the trajectories) is providing extra information to the policy. I would train implement the policy as a causal-decoder transformer only as an ablation to verify the model architecture is not behind the observed gains. \n\n(It is possible that my assumptions of how CLARIFY is implemented are mistaken, this paper could benefit from an architecture diagram).\n\n**Post-rebuttal update**\n\nDuring rebuttal new experiments were added to address the concerns above. In particular, it became clear that CLARIFY is quite robust to $\\epsilon$ changes and to other labellers (C.1). Similarly, experiments were added to show that the observed performance improvements were not due to the use of a bidirectional transformer (C.2)", "methods_and_evaluation_criteria": "Yes, the paper contains comparisons against recent baselines on commonly used tasks (MetaWorld and DMControl).", "theoretical_claims": "I did not have time to go through the proofs of Propositions 5.1 and 5.2, since they are contained in the appendices. But the claims derived from these propositions seem sound.\n\nI believe there is a typo in Proposition 5.2, $d(z^-, \\mathcal{H})$ should be $\\le \\eta$ rather than $\\le - \\eta$.", "experimental_designs_or_analyses": "All the analyses and ablations make sense, apart from the issues discussed above.", "relation_to_broader_scientific_literature": "CLARIFY is tackling the very challenging (and highly researched) problem of learning to solve a task without an explicit reward function from an offline dataset of interactions. In this context, the use of contrastive learning (which has been used elsewhere in Machine Learning to increase sample efficiency) is interesting.", "essential_references_not_discussed": "None that I could find.", "other_strengths_and_weaknesses": "**Other Strengths**:\n\n* The paper includes a comparison against actual human labellers for the `walker-walk` task (though unfortunately its performance is approximately half of the performance achieved with a non-ideal teacher, cementing the need of analysis with other non-ideal teachers).\n\n**Other Weaknesses**:\n\n* Query clarity ratio is not very clearly defined in the manuscript. I believe it is 1 - (ratio of `no_cop`) labels provided by human labellers?\n* Similarly, I am very confused by what is meant by clearly-distinguished queries in Table 3.\n* The Impact Statement is well thought out, but it does not mention possible negative consequences of offline PbRL (namely the ability for laymen to very easily program an agent to carry out malicious tasks).\n\n**Post-rebuttal update**\nAuthors addressed all the above issues. I would urge authors to adopt the definitions of `clearly-distinguished queries` and `clarity-ratio` present in the rebuttal.", "other_comments_or_suggestions": "* What does `no_cop` stand for?", "questions_for_authors": "* Q1. When is equation (9) used in Algorithm 1? Is it simply a pre-training for the bi-directional encoder?\n* Q2. The paper never really states what the distance metric $l$ in equation 5 is. I assume is simply the euclidean distance? Perhaps a more appropriate distance would be the cosine distance? In very high-dimensional spaces, it's very easy for two points to be very far apart.\n* Q3. What is the number of queries used for Table 1? This should be clearly stated.\n* Q4. How much do the results in Table 2 deteriorate under 50 queries? Does CLARIFY also ~77% of the IQL-with-ground-truth reward performance with only 100 samples for other tasks?\n* Q5. The error bars on Figure 4 are quite large, particularly for `walker-walk`, could you run a statistical significance analysis to verify that CLARIFY and OPRL are indeed different?\n* Q6. In Figure 5 what are the thresholds for Large Return Difference, Medium Return Difference, and Small Return Difference?\n* Q7. Using T-SNE visualisation to prove that the embeddings are robust to changes in the loss $\\lambda$ is very unconvincing. T-SNE has its own hyper-parameters and it may be ignoring important differences in the underlying space when projection to 2D.\n\n**Post-rebuttal update**\nAuthors addressed all the above issues. I would urge authors to adopt the definitions of `clearly-distinguished queries` and `clarity-ratio` present in the rebuttal. \n\nRegarding Q2, the research would likely benefit from further investigation of the effects of different distances (beyond a TSNE projection), but perhaps this could be left for further work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bOW9Cn5aEB", "reviewer": "Reviewer_1xXL", "summary": "This paper proposes an offline PbRL framework, CLARIFY, to address challenges arising from ambiguous queries. The method learns a trajectory embedding space through contrastive learning and utilizes the learned embedding to maximize the selection of clearly distinguished queries via rejection sampling, improving human labeling efficiency and achieving state-of-the-art results in offline PbRL settings.\n\n## Update After Rebuttal: I raised my score from 2 to 3 as the authors' rebuttal addressed most of my concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nThanks for your valuable and detailed comments.\nWe hope the following statement clear your concern.\n**We conducted additional experiments and the results are shown in the [link](https://docs.google.com/document/d/e/2PACX-1vQX0KIRCSWV8LrON718raf-d_BL75LRXMY5yB-Ts28kW0BZIVyWHan0kgw54vnZQtuxp1ODwe4IH1ws/pub).**\n\n\n\n**Claim, W1 and Q4: Embedding failure cases.**\n\n**A for Claim and Q4:**\nWe would like to clarify that Figure 2(c) in the original paper visualizes embeddings trained with both losses rather than using only quadrilateral loss.\n\nAs suggested, we visualize the failure modes in Figure 1 in the supplemental link.\nSpecifically, using only $\\mathcal L_\\text{amb}$ causes erroneous clustering, where low and high return trajectory embeddings are intermixed (Figure 1(a) in the supplemental link).\nIn contrast, using only $\\mathcal L_\\text{quad}$ yields densely packed embeddings with insufficient separation (Figure 1(b)). \nCombining both losses resolves these issues and leads to smooth, coherent clusters (Figure 1(c)).\nThis illustrates their complementary roles in structuring the embedding space.\n\n\n**Experimental Design 1 and Q5: Naive baseline comparison.**\n\n**A for Experimental Design 1 and Q5:**\nAs suggested, we compare CLARIFY with the naive approach that rules out pairs with `no_cop` preference labels, as shown in Table 1 in the supplement link.\nCLARIFY outperforms the naive approach (the \"Naive\" method in the table) by over 50\\% on most tasks. This demonstrates the effectiveness of CLARIFY.\n\n**Experimental Design 2, W2 and Q3: Computational costs.**\n\n**A for Experimental Design 2, W2 and Q3:**\n- As suggested, we analyze the computational cost and the effort required for hyperparameter tuning of CLARIFY, as shown in Table 2 in the supplement link.\nCLARIFY incurs moderate computational overhead, roughly 2-3 times that of OPRL, primarily due to embedding learning. \nWhile training time may increase, CLARIFY effectively identifies more clearly distinguished queries, accelerating the labeling process. \n- On the other hand, CLARIFY demands minimal hyperparameter tuning.\nIn the original paper, key hyperparameters ($\\lambda_\\text{amb},\\lambda_\\text{quad}$) were fixed across tasks with robust performance.\nTo support this, we conduct additional experiments to visualize the embeddings under various hyperparameter configurations, as in Figure 2 in the supplement link, illustrating the stability of embeddings under parameter variations. \nWe have added these results in the revised version.\n\n**Q1: Ambiguity's impact on PbRL.**\n\n**A for Q1:**\nAmbiguous queries hinder PbRL primarily in two ways:\n1. Labeling efficiency: Human teachers often struggle to differentiate between similar segments, leading to skipped labels (`no_cop`) that waste annotation effort. As in Table 3 of the original paper, only about 50% of queries receive clear preference labels, indicating significant inefficiencies.\n2. Reward learning accuracy: Labelers may produce random or incorrect preferences when segments are only marginally different. This can introduce errors to the reward model, ultimately degrading policy performance.\n\n**Q2: CLARIFY vs S-EPOA [1].**\n\n**A for Q2:**\nWhile both methods tackle ambiguous queries, they differ fundamentally in their approaches.\nS-EPOA addresses indistinguishability via unsupervised exploration and skill-based query selection in online settings.\nIn contrast, CLARIFY utilizes contrastive learning to create trajectory embeddings informed by preferences, which enables offline query filtering.\nOne of the key advantages of CLARIFY is that it eliminates the need for online exploration, making it particularly suitable for real-world applications, where interaction can be costly or risky.\nFurthermore, CLARIFY directly optimizes query distinguishability rather than relying on skill diversity as a surrogate.\nThis independence from skill diversity enhances CLARIFY's robustness in tasks with constrained skill exploration (e.g., limited state-space traversability), eliminating reliance on costly exploration for discovering diverse skills.\n\n**Answer to Essential References and Other Comments:**\nThank you for your keen attention to detail!\nWe have subscripted $\\mathcal L_\\text{amb}$ throughout the manuscript. Additionally, we have discussed [2] in the related work section.\n\nThanks again for the valuable comments. We sincerely hope our additional experimental results and explanation have cleared the concern.  More comments on further improving the presentation are also very much welcomed.\n\n[1] Mu, Ni, et al. \"S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning.\" arXiv preprint arXiv:2408.12130 (2024).\n\n[2] Liu, Tianqi, et al. \"Statistical Rejection Sampling Improves Preference Optimization.\" The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "The paper claims that the proposed embedding space is meaningful and coherent, as high-performance trajectories form one cluster, low-performance trajectories form another, and intermediate trajectories transition smoothly between them. The authors support this claim using Figures 2 and 3, which visualize the learned embedding spaces. My understanding is that Figure 2 represents the embedding space learned using only quadrilateral loss, while Figure 3 shows the embedding space learned with both ambiguity loss and quadrilateral loss. However, they appear similar. It would be helpful to highlight the failure points of the embedding space when using only ambiguity loss or only quadrilateral loss and demonstrate how combining these two losses addresses the problem.", "methods_and_evaluation_criteria": "The proposed methods and benchmark datasets are reasonable for the problem or application at hand.", "theoretical_claims": "The theoretical claims appear reasonable and effectively cover the core concept proposed in the paper.", "experimental_designs_or_analyses": "Adding a naive baseline that simply rules out ambiguous sample pairs with \"p = no_cop\" and comparing its performance with the proposed method would help readers better appreciate the effectiveness of the proposed approach.\n\nAdditionally, the proposed method introduces extra complexity to the PbRL algorithm by (1) learning an embedding space and (2) performing rejection sampling. Moreover, its final loss function involves several hyperparameters (e.g., $\\lambda_{\\mathrm{amb}}$, $\\lambda_{\\mathrm{quad}}$, $\\lambda_{\\mathrm{norm}}$). Comparing the computational costs and hyperparameter tuning efforts with other baselines would provide valuable insights into the true applicability of the proposed method.", "relation_to_broader_scientific_literature": "Designing effective reward learning within the PbRL framework is closely related to the broader scientific literature.", "essential_references_not_discussed": "There is potentially related literature that incorporates rejection sampling into PbRL or Preference Optimization, such as [1] Statistical Rejection Sampling Improves Preference Optimization, Liu et al., ICLR 2024.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is well-written and easy to follow.\n- The claims are largely supported by visualizations and theoretical analyses.\n- The paper evaluates the method on diverse benchmark datasets, demonstrating its real-world applicability.\n\nWeaknesses:\n\n- The effect of each loss term in the final proposed loss is not empirically studied in depth.\n- The computational overhead of learning the embedding and the effort required for hyperparameter tuning are not thoroughly analyzed.", "other_comments_or_suggestions": "There is a typo in Line 179: 'amb' should be subscripted, e.g., $\\mathcal{L}_{\\mathrm{amb}}$.", "questions_for_authors": "Q1. Why and how do ambiguous queries hinder the practical application of PbRL? Is this issue solely related to labeling efficiency, or does it also impact policy training within the PbRL framework?\n\nQ2. Could you elaborate on the differences between S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning, Mu et al., 2024, and the proposed method beyond the distinction between online and offline settings?\n\nQ3. How do the computational overhead and the effort required for hyperparameter tuning of the proposed method compare to those of other baselines?\n\nQ4. Could you visualize the failure points of the embedding space or the performance results when using only ambiguity loss or only quadrilateral loss and demonstrate how combining these two losses resolves the issue?\n\nQ5. How does the proposed method compare to a naive approach that simply rules out pairs with 'p = no_cop'?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xfEqwCG0IU", "reviewer": "Reviewer_znbK", "summary": "This paper presents CLARIFY, a method that selects unambiguous queries that humans can more easily label. It does this by learning a meaningful embedding space using two contrastive losses. This allows for weaker teachers to provide meaningful feedback on the selected trajectories. Experimental results in continuous control tasks show that CLARIFY can significantly improve labelling efficiency and improve policy performance.\n\n## After Rebuttal\nThe authors have nicely addresses my concerns. I will raise my score to a 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nThanks for finding our paper nice presentation, novel, thorough experiments and solid improvement. We hope the following statement clear your concern.\n\n**Claim and W1: Comparison to [1] and its offline applicability.**\n\n**A for Claim and W1:**\n- While both CLARIFY and [1] attempt to tackle ambiguous queries, their approaches are fundamentally different.\n[1] address ambiguous queries by learning diverse skills through unsupervised exploration. \nThe requirement for online exploration makes it incompatible with offline settings, where agents cannot interact with the environment.\nIn contrast, CLARIFY operates entirely offline by learning preference-informed trajectory embeddings through contrastive learning.\nThis eliminates the need for online exploration, making CLARIFY more suitable for real-world scenarios where environment interaction is costly, risky, or unavailable.\n\n- Additionally, CLARIFY optimizes query distinguishability directly via embedding distances, whereas [1] relies on skill diversity as a proxy.\nThis independence from skill diversity enhances CLARIFY's robustness in tasks with constrained skill exploration (e.g., limited state-space traversability), which eliminates the reliance on costly exploration for discovering diverse skills.\n\n**Methods 1: Task complexity and generalizability.**\n\n**A for M1:**\nWe appreciate the reviewer’s concern regarding task complexity.\nOur experiments include challenging tasks, such as MetaWorld's peg-insert-side, which require multi-step reasoning and are more complex than prior RLHF [2] evaluations focused on locomotion.\nCLARIFY significantly outperforms baselines across these tasks, as shown in Table 1 of the paper, demonstrating its effectiveness beyond simple settings.\n\n**W2: Importance of offline setting and query sampling.**\n\n**A for W2:**\nOffline reinforcement learning assumes agents learn solely from a fixed pre-collected dataset without environment interaction. It is critical for safety-sensitive domains (e.g., healthcare, autonomous systems), where real-time exploration is either unsafe or impractical.\nIn such cases, agents must rely on pre-collected datasets for learning, and thus, queries should be sampled from this static dataset rather than generated on-policy.\nThis aligns with CLARIFY's approach, which leverages offline datasets and contrastive learning to ensure the method's applicability in real-world, offline settings.\n\nWe sincerely thank the reviewer again for the timely and valuable comments. We hope that our response and additional experimental results have cleared most of your concerns.\n\n[1] Mu, Ni, et al. \"S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning.\" arXiv preprint arXiv:2408.12130 (2024).\n\n[2] Christiano, Paul F., et al. \"Deep reinforcement learning from human preferences.\" Advances in neural information processing systems 30 (2017).", "claims_and_evidence": "The only claim I have an issue with is their claim that Mu 2024 cannot be applied to offline settings (this claim is made on line 162). I have not looked into Mu 2024 deeply, but I do not see why it cannot be applied to offline settings with some simple modifications. Furthermore, I don't understand why the offline setting is important for the tasks the authors consider.\n\nThe main claims about the performance gains of Clarify are solid. They conduct many experiments and it seems like Clarify can indeed improve performance in most settings.", "methods_and_evaluation_criteria": "The experiments and evaluation criteria do make sense. This paper works to improve RLHF, and some of the earliest papers on RLHF focused on continuous control [1]. They evaluate based on the average ground-truth reward received, which makes sense. However their experiments are all conducted on relatively simple continuous control environments. One concern I have is that their approach of only selecting easy samples to train on will work better in easy settings than in hard settings. This means their experimental setup may overestimate the utility of their method.\n\n[1] Christiano, Paul F., et al. \"Deep reinforcement learning from human preferences.\" Advances in neural information processing systems 30 (2017).", "theoretical_claims": "no, I did not check them for correctness.", "experimental_designs_or_analyses": "Yes I did check the soundness of the experimental design. The experimental design and analysis is sound. They compare their algorithm with many baselines, and they conduct experiments in a wide variety of environments. In addition, they report five independent runs for each algorithm, and report standard deviation for all experiments. The main experimental results I am referring to are in Table 1.\n\nThey also show that CLARIFY results in queries that are more distinguishable to humans (Figure 4), and these results also seem sound.", "relation_to_broader_scientific_literature": "The authors approach seems novel and very relevant to data selection for RLHF, which is a popular area at the moment. I think the whole CLARIFY framework is novel, but it needs more detailed comparison to Mu 2024.", "essential_references_not_discussed": "Mu 2024 is cited, but not discussed in depth. Since Mu 2024 aims to accomplish a very similar goal as CLARIFY (albeit in the online setting), I think CLARIFY's novelty and contribution in comparison to Mu 2024 should be discussed in more detail.", "other_strengths_and_weaknesses": "Strengths\n- The writing is easy to understand, and the presentation is nice\n- The approach seems to be novel\n- The experiments are thorough and their approach shows solid improvement compared to baselines.\n\nWeaknesses:\n- The authors do compare in depth to Mu 2024 (why can’t it be included in the offline setting?)\n- There is no discussion of offline versus online approaches. Why do the authors sample queries from the offline dataset, rather than sampling new on-policy queries from the current policy? It seems like this would be a more effective approach in all of the experimental settings the authors consider.", "other_comments_or_suggestions": "none", "questions_for_authors": "See the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bCtwuuwTC4", "reviewer": "Reviewer_985K", "summary": "This paper address the query selection problem of PbRL. The authors propose a representation learning algorithm that embeds trajectories into high dimensional vectors and enlarge the distance between unambiguous trajectories pairs. The authors compare their proposed method with existing PbRL methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\n\nThanks for your valuable and detailed comments.\nWe hope the following statement clear your concern.\n**We conducted additional experiments and the results are shown in the [link](https://docs.google.com/document/d/e/2PACX-1vS0ZIKigh-syAaNtcr2Udzk8katEE6AtC0OA23Xveb1dUqzFtMws64U6o6GFUep_BTzQ0EaA770n88P/pub).**\n\n\n\n**Claim 1, Experimental Designs 1 and Q2: Preference collection and exploration-exploitation trade-off.**\n\n**A for Claim 1, Experimental Designs 1 and Q2:**\nThe preference labels in our study are collected over multiple rounds, with the embedding space updated iteratively as new queries are selected.\nAs for the exploration-exploitation trade-off, CLARIFY prioritizes exploitation by focusing on unambiguous queries, while its reject sampling strategy (Sec. 4.2) inherently supports exploration. This strategy diversifies the queries by sampling from a distribution, avoiding overemphasizing the \"clearest\" pairs.\n\nAs suggested, we conducted experiments to evaluate the exploration-exploitation trade-off.\nWe compared CLARIFY against two baselines: (1) pure exploration (random query selection, referred to as \"Random\") and (2) pure exploitation (maximizing the density for clearly-distinguished queries $\\rho_\\text{clr}(d_\\text{emb})$, referred to as \"Exploitation\").\nAs shown in Tables 1 and 2 in the supplement link, CLARIFY outperforms Random by over 20\\% in success rate, and surpasses Exploitation by over 15\\%. These results demonstrate an effective exploration-exploitation trade-off.\n\n\n\n**Claim 2: Label efficiency.**\n\n**A for Claim 2:**\nAs suggested, we evaluate CLARIFY's label efficiency from 50 to 2000. Table 3 in the supplement link shows that CLARIFY consistently outperforms baselines under various query numbers. \nIn addition, we observed that the performance of CLARIFY with 100 queries approaches that of MR with 1000 queries, demonstrating CLARIFY's effectiveness in achieving high performance with fewer labels.\n\n\n\n**Experimental Designs 2 and Q3: Reward fitting accuracy on the test set.**\n\n**A for Experimental Designs 2 and Q3:**\nAs suggested, we evaluate CLARIFY's reward fitting accuracy on the test set. Table 4 in the supplement link reflects the effectiveness of reward fitting, showing that CLARIFY attains about 3\\% higher accuracy than OPRL on most tasks.\n\n\n\n**W1: How Eq. 6 resolves representation collapse.**\n\n**A for W1:**\nThe quadrilateral loss $\\mathcal L_\\text{quad}$ prevents representation collapse by enforcing preference-aware geometry:  \n\n1. For clear preferences, it creates a hyperplane separating good and bad trajectories (Proposition 5.2).\n2. For ambiguous pairs, while $\\mathcal L_\\text{amb}$ minimizes their embedding distance, $\\mathcal L_\\text{quad}$'s contrastive gradient prevents trivial clustering of these pairs.\n\nTo support this, we conduct an ablation study on $\\mathcal L_\\text{amb}$, as shown in Table 5 in the supplement link. Removing $\\mathcal L_\\text{quad}$ degrades performance by about 10\\% on average, validating its necessity.\n\n\n\n**A for Other Comments:**\nThank you for your keen attention to detail! We have corrected the positive and negative samples to $(\\sigma_+, \\sigma^\\prime_+), (\\sigma_-, \\sigma^\\prime_-)$ in the revised manuscript.  \n\n\n\n**Q1: $\\mathcal L_\\text{reward}$ optimization.**\n\n**A for Q1:**\nTo elaborate, we present an additional figure (Figure 1 in the supplement link) that illustrates the architecture of CLARIFY.\nAs shown in the figure, the reward model is updated using the loss function $\\mathcal L_\\text{reward}$ (Eq. 2) during the reward learning phase, while the embedding space is updated according to Eq. 11 during the embedding learning phase.\nThese two phases are strictly decoupled. Thus, the loss function $\\mathcal L_\\text{reward}$ is not incorporated into Eq. 11.\nTraining the reward model solely with the Bradley-Terry loss is a standard approach in the PbRL literature [1,2,3].\nThis separation allows preference learning to focus on human feedback, while embeddings specialize in distinguishing trajectory pairs.\n\n\n\nThanks again for the valuable comments. We hope our response has cleared your concerns. We are looking forward to more discussions.\n\n[1] Lee, Kimin, et al. \"B-pref: Benchmarking preference-based reinforcement learning.\" arXiv preprint arXiv:2111.03026 (2021).\n\n[2] Shin, Daniel, et al. \"Benchmarks and Algorithms for Offline Preference-Based Reward Learning.\" Transactions on Machine Learning Research.\n\n[3] Cheng, Jie, et al. \"RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences.\" International Conference on Machine Learning. PMLR, 2024.", "claims_and_evidence": "The experiment results include a table for task performance and an ablation study shows that their method can increase the clarity of query. These results confirm the efficacy of their method.\n\nIn the meantime, this paper does not explicitly mention if the preference labels are collected in a single round or multiple rounds. If the preference labels are collected in multiple rounds, then the embedding space at round $T$ depends on previous rounds. In this setting, the method proposed in this paper can be considered as a exploitation strategy. In the meantime, exploration is also important: it seems that allocating label budget in ambiguous queries could be beneficial, as it might help us resolve the decision boundary. \n\nIf the preference labels are collected in multiple rounds, could you please results for label efficiency? In other words, how performance changes as we get more preference labels?", "methods_and_evaluation_criteria": "This paper compares methods using the task performance of offline RL, which is standard in this area.", "theoretical_claims": "No, I did not check the correctness of the proofs.", "experimental_designs_or_analyses": "Yes, I have checked the experiment results. \n\n1. This paper overlooks the analysis of exploitation and exploration tradeoff in the experiments.\n2. Since the author claims that their method mitigates overfitting issue, please provide results for the quality of reward learning, i.e. accuracy on test preference set.", "relation_to_broader_scientific_literature": "Embedding entire trajectories into a high-dimensional space and adjust the embeddings based on the proximity between trajectories is an interesting idea for the offline RL and the RL literature.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Ambiguous queries do not have labels of \"1\" or \"-1\", so they are not included in the samples used for minimizing eq. 6. The paper lacks explanation for why minimizing eq 6 resolves representation collapse of ambiguous queries.", "other_comments_or_suggestions": "1. In line 199-200, I think the \"positive\" samples should be $\\sigma_+$ and $\\sigma'_+$.", "questions_for_authors": "1. I notice that the preference loss $\\mathcal{L}_\\text{reward}$ does not appear in e.q. (11). How do you get the reward function from the trajectory embeddings? What is the reason of not learning the representations with both representation losses and reward learning losses?\n\n2. Are the preference labels collected in single round or multiple rounds? Could you please provide results for the exploitation-exploration trade-off?\n\n3. Can you provide results for reward fitting (i.e. accuracy on unseen preference samples)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces CLARIFY, an offline PBRL method that incorporates contrastive learning for reward learning from human preferences. The authors demonstrate that: i) contrastive learning ensures a lower bound on the distance between two trajectory embeddings, and ii) a hyperplane can effectively separate the embeddings of preferred and dispreferred trajectories. Experimental results on MetaWorld and DMControl tasks demonstrate the utility of the proposed method. 

Some reviewers raised concerns (e.g., robustness against noisier labelers and comparisons with Mu 2024), but most of these concerns were addressed during the rebuttal phase. I believe that accepting this work would make a valuable contribution to the community. Therefore, I recommend a weak accept.
Decision: Accept (poster)