Preference-based Policy Optimization from Sparse-reward Offline Dataset
Authors: Wenjie Qiu
Guofeng Cui
Shicheng Liu
Yuanlin Duan
He Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Offline Reinforcement Learning
Preference-based Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We introduce a contrastive framework that mitigates value overestimation in offline RL by training a policy to prefer successful trajectories over both observed and synthetic failures, leading to state-of-the-art results.

Abstract: Offline reinforcement learning (RL) holds the promise of training effective policies from static datasets without the need for costly online interactions. However, offline RL faces key limitations, most notably the challenge of generalizing to unseen or infrequently encountered state-action pairs. When a value function is learned from limited data in sparse-reward environments, it can become overly optimistic about parts of the space that are poorly represented, leading to unreliable value estimates and degraded policy quality. To address these challenges, we introduce a novel approach based on contrastive preference learning that bypasses direct value function estimation. Our method trains policies by contrasting successful demonstrations with failure behaviors present in the dataset, as well as synthetic behaviors generated outside the support of the dataset distribution. This contrastive formulation mitigates overestimation bias and improves robustness in offline learning. Empirical results on challenging sparse-reward offline RL benchmarks show that our method substantially outperforms existing state-of-the-art baselines in both learning efficiency and final performance.

Published as a conference paper at ICLR 2026
PREFERENCE-BASED POLICY OPTIMIZATION FROM SPARSE-REWARD OFFLINE DATASET
Wenjie Qiu'
Guofeng Cui Shicheng Liu? Yuanlin Duan'
He Zhu
Department of Computer Science, Rutgers University
2 Department of Electrical Engineering. Pennsylvania State University [wq37, ge 669, yuan in. duan, hz375) @rutgers.......... edu
ABSTRACT
Offline reinforcement learning (RL) holds the promise of training effective policies from static datasets without the need for costly online interactions. However, offline RL faces key limitations, most notably the challenge of generalizing unseen or infrequently encountered state-action pairs. When a value function is learned from limited data sparse-reward environments, it can become overly optimistic about parts of the space that are poorly represented, leading to unreliable value estimates and degraded policy quality. To address these challenges, we introduce a novel approach based on contrastive preference learning that bypasses direct value function estimation. Our method trains policies by contrasting successful demonstrations with failure behaviors present in the dataset, as well as synthetic behaviors generated outside the support of the dataset distribution. This contrastive formulation mitigates overestimation bias and improves robustness in offline learn- ing. Empirical results on challenging sparse-reward offline RL benchmarks show that method substantially outperforms existing late-of-the-are baselines in both learning efficiency and final performance.
INTRODUCTION
Offline reinforcement learning (RL) (Levine al., 2020; Prudencio et al., 2024) aims to learn high-quality decision policies purely from static datasets, without requiring additional environment interactions. This paradigm offers a compelling route for deploying RI. in real-world domains where data collection is costly, risky, or constrained- -ucra as robotics, healthcare, or recommendation systems. However, offline RL remains fundamentally challenging due to the distributional mismatch between the policy being learned and the limited data it learns from.
core issue lies in the extrapolation error that arises when learned value functions are queried on state-action pairs not well represented in the dataset. This is particularly problematic in sparse-reward settings, where the dataset may lack sufficient reward-bearing trajectories or behavioral diversity. become overly optimistic in poorly covered regions of the
As result, value-based methods state-action space, leading to unstable or suboptimal policies (Levine et al., 2020).
To address this, prior work has largely focused three classes of solutions. First, pessimism-basee approaches mitigate overestimation by explicitly penalizing uncertain or unsupported regions in the learned value function. Techniques such as conservative Q-learning (Kumar et al., 2020) or uncertainty-awarr backups enforce value suppression on out-of-distribution actions. However, these methods often rely on assumptions about the behavior policy and require careful calibration of the degree of pessimism, which becomes increasingly difficult in high -igensiinnsional or sparse settings (Liu et al., 2020; Xie et al., 2021). Second, regularization-bat based methods constrain policy updates to remain close to the behavior policy by adding policy divergence penalties (Wu et al., 2019; Kumar et al., 2019; Fujimoto Gu, 2021). While effective in well-covered datasets, these methods can be brittle when tuning the regularization strength and may fail to explore beyond suboptimal behaviors (Lee et al., 2021; Brandfonbrener et al., 2021; Lyu et al., 2022). Third, importance sampling based techniques, including DICE-style distribution correction (Cen et al., 2024; Lee al., 2021), attempt to re- re-weight observed rewards based on estimated marginal state-action densities. Although theoretically sound and behavior-agnostic these methods sensitive support mismatches and can suffer from

---

Published as a conference paper at ICLR 2026
high variance instability, especially when the data are limited reward signals are sparse (Xie et al., 2021; Li et al., 2022; Cen et al., 2024).
In this paper, propose a fundamentally different approach that avoids direct value function estimation altogether. We introduce PREFÓRL (PREFerence-baseee Optimization for Offline RL), a contrastive preference learning framework that optimizes policies by comparing successful (preferred) and unsuccessful (nonpreferred) behaviors from a static dataset. Previous work such as CPL (Hejna al., 2024) and DPPO (An et al., 2023) has considered this setting, although not explicitly in the offline RL context. Nonetheless, simply contrasting successful and unsuccessful behaviors does not resolve overestimation in datasets with limited actton coverage, since the policy can still become overly optimistic in poorly represented regions due to the absence of strong counterexamples. Crucially, in PREFORI, we extend the contrastive signal beyond failure behaviors present in the dataset to include synthetic behaviors generated outside the dataset's s support. By contrasting both types of behaviors against successful demonstrations, our method trains policies to imitate not just what succeeds, but to actively avoid what likely fails or lies outside the dataset's support.
This formulation enables us to sidestep the estimation pitfalls of value-based methods while di- rectly combating overestimation. Our empirical evaluation on challenging sparse-reward offline RL benchmarks shows that this contrastive approach leads to more stable learning and substantially outperforms existing tate--of-the-at offline RL baselines.
2 PROBLEM FORMULATION AND MOTIVATIONS
We formulate the reinforcement learning problem in the context of Markov Decision Process (MDP) M = (S,A, T.r.7:Po where S is the state space, A is the action space, T : S x A x S - [0, 1] is the transition probability function T(s' 8, a), r S A - R is the reward function, E (0, 1) is the discount factor, and pa S - 00 1] is the initial state distribution. A policy : x A + [0, 1] maps each state a distribution over actions. Let = {so, ao $1, .i..... denote trajectory sampled by interacting with the MDP under policy n, i.e.e. 8o Pu. ag ~ x(- st). 8141 ~ T(- St, az). Then, the discounted state-action distribution induced by
defined as 8, a al]. The goal is to learn policy (ala)s) that maximizes the expected discounted return: Bx..ag.s .l..s....>ddd ->00"'r( at)]. In offline RL, the agent does not have access to the environment M, and instead must learn a policy solely from a static dataset D collected from some (possibly unknown) behavior policy T3. We define D =
d' (8) = (1 7) 7 [1[s,
U Ti, where T; (( a. (i) (i) (8) ((21)) with N trajectories in total and T; denoting the length of the i-th trajectory. The empirical state-action distribution of the dataset is denoted d' a). which approximates (s,a)s. We consider the challenging setting of sparse reward offline RL, where informative reward signals are infrequent and the dataset predominantly consists of transitions with zero or low rewards, making it difficult to identify and generalize from successful behaviors. Formally, we assume D = (D=, D-), where D+ contains successful trajectories and D contains unsuccessful trajectories. For a trajectory T = {(st, at, Tt, $t41)) let R((T) = ΣT --= its cumulative return. We define D { E D R(T) > n), and D- = (T D S n), where 17 is threshold. For example, in many sparse-reward environments 17 = O, since trajectories that terminate in goal states receive a positive terminal reward, while those that do not yield zero cumulative return. We define dM (s) as the optimal state marginal, which can be viewed as state distribution successful trajectories D+ the dataset D.
denote
The Adyantage Preference Model. In Direct Preference Optimization (DPO) (Rafailov et al., 2023), a Bradley-Terry (BT) (Bradley & Terry, 1952) model is built on top of the hidden reward model Te given by expert users to capture the preferences of pairs of answers (v) V2) ~ (b(zz)) While DPO and other reinforcement learning from human feedback (RLHF) algorithms (Christiano et al., 2017) have shown strong performance for large language models which can be framed as contextual bandit problems-they are not directly suited for general RL tasks where trajectory- preferences are crucial for solving long-horizon problems. To that end, we define a trajectory of length n as T = (Sp. ag..... ,, Sp-1, an-1), and introduce the notion of a length-k representative segment, denoted by < Σ(T, k) = (So, âu ... $k-1, âg-1), which approximates the overall quality and semantics of its original trajectory 7 = T(c). Each ($,, à,) in the segment is sampled from т. with the constraint that their original indices (1) strictly increasing preserve temporal order. We
2

---

Published as a conference paper at ICLR 2026
denote segment-level preferences < > ST. which assume it reflects overall preference for their corresponding full trajectories, i.e., T(=*) T(s-). Recent work such as Knox et al. (2024) estimates such preferences by comparing partial discounted returns Σe y'r(s,, 0z) for trajectory segments. However, in settings with sparse or highly imbalanced rewards, this return-based signal may be too weak or misleading to support reliable comparisons. To mitigate this, we instead adopt an advantage-based preference model in Contrastive Preference Learning (CPL) (Hejna al., 2024), which focuses distinguishing successful behaviors not just based returns, but through their relative quality under advantage estimation:
exp E+ + ( ,4,) exp Exp AM 0x + exp - (1)
PA-IT* >T PA-ls" >5]=
where A denotes the optimal advantage function, and T = T(s and T complete trajectories. We use the shorthand "+" and "_" to denote the preferred / less preferred representative segments.
=T(s are two
Contrastive Preference Learning (CPL). Hejna et al. (2024) eliminates the hidden optimal advantage function A" the advantage-based preference model the context of maximum entropy RL (Ziebart, 2018; Ziehart et al., 2008; Haarnoja et al., 2017). The derivation is straightforward, as Ziebart (2018) provides critical insight, i.e., the optimal advantage function A and optimal policy n² (a|s) has direct relationship:
(s a) ologn (als),
assuming that the optimal advantage function is normalized J eAA da 1. This means that instead learning an implicit optimal advantage function, CPL can leverage the preference model to acquire the optimal policy directly. Given an offline preference dataset D (D+ DD),, the learning objective is to minimize the following loss function while increasing the likelihood of actions in the datasets.
exp E,+ 2'alogro(st at)
(Tg. D) = B(et >-D=- loo
exp Ee Aoolooge (5t (xt,)) + expx E. A'ologTo(S
where (st,5-) D denotes drawing a pair with ç+ ~ D+ ~~~ D= and X E (0, denotes the asymmetrie "bias" regularizer (An al., 2023) that dewn-weights the negative segments.
CREFERENCE-BASSE POLICY OPTIMIZATION
In Section 2, we reviewed contrastive preference learning (CPL) and its potential for effective policy learning sparse-reward offline RI. by leveraging static preference dataset Dpret- However, simply contrasting successful (preferred) and unsuccessful (non- (non-preferred) behaviors does not resolve overestimation datasets with limited state-action coverage, as the policy can still become overly optimistic in underrepresented regions due to the lack of strong counterexamples. In this section, we address the support mismatch issue by developing a practical offline RL algorithm called PREFORL for sparse-rewe reward offline datasets (Qiu, 2026).
3.1 DEGRADATION
To mitigate the support mismatch issue, our key idea is augment the offline dataset D with synthetically generated suboptimal trajectories, treated non-preferred examples. By training policies to prefer successful trajectories over both observed and synthetic failure cases, the framework encourages imitation of high-quality behavior while simultaneously improving robustness against failure modes and distributional drift.
Specifically, given pparse-reward offline dataset D= ( ",,,,D), we take a successful trajectory T+ E D+ and construct a corresponding less-predt -essfereer trajectory T by applying a controlled degradation operator either action-based (Lo) state-based (1*).
3

---

Published as a conference paper at ICLR 2026
a
a
S2
S a
T2 a
S
T1
Two variants of degradation. In the left figure, cetion-based degradation (4*) is applied to degrade the action a to ä using Gaussian noise. In the right figure, state-based degradation (+e) is used to degrade the action in trajectory by finding substitution action á correspond neighbor state in a less preferred trajectory 72. Red arrows with solid lines denotes the original actions, and the red arrows with dashed lines denotes degraded actions.
Figure 1:
Action-based Degradation Ja To achieve the requirement above, our action- degrada- tion method injects noises into the actions within the dataset (see Figure left). Given D U ai ())) we construct degraded dataset Die by adding Gaussian noise each action:
(1)
( + 4 ~NN0000221),
0f at
Die Ut(s
19) 19 )
(4)
Here, is tunable noise parameter whose impact analyzed in Appendix C. By construction, the original dataset of successful trajectories is preferred over its degraded counterpart:
D+
DLa
State-based Degradation te Instead of perturbing actions directly, state-based degradation con- structs suboptimal behavior by reassigning actions from nearby states (see Figure right). Given D - (()))) ai (8) D) we build degraded dataset Dla by, for each state s retrieving the action from a nearest-neighht neighhor state S6 recorded in the non-preferred dataset DD
(1) at)
D4e Ut(ol 1
(5)
In practice, nearest neighbors can be retrieved using Fuclidean distance in vector-based state spaces or feature-space distances in high-dimensional (e.g., image-based) environments. By construction, the dataset of successful trajectories preferred over its degraded counterpart:
D
D4s
The philosophy behind both degradation methods is to generate suboptimal datasets that serve as the basis for constructing preference datasets for policy optimization. State-based degradation Dls provides contrastive signals over nearby failure behaviors already present the dataset, whereas action-based degradation D4a introduces synthetic behaviors sampled outside the dataset's support. PREFORL can viewed as "squeezing" strategy: successful behaviors are sandwiched by synthetic degradations that bound what is preferable. By contrasting these degraded behaviors against successful trajectories, PREFORL trains policies not only to imitate what succeeds, but also to explicitly avoid behaviors that likely fail or fall outside the dataset's support.
4

---

Published as a conference paper at ICLR 2026
Algorithm
Preference-based Optimization for Offline RL (PREFORL)
Require: Policy parameters 0. offline dataset of trajectories D = (D+, D-), k-length representative
segment sampling function [T k), representative segment length k. temperature a, contrastive
bias X, discount factor y.
Ensure: Policy (s)))
for j = 1......... N do
DJ {. D = }
for m = 0, 0,1,....... M - do
Tm (...,, 0........ >D
[m,,, k) = (..., 81, at, 8r, σx)
b Build representative segment
DJ = D U(s}
> Collect preferred segments (+)
end for
Construct Dj- = DD U U.D
from D via Equation and 5
exp E.+ aL exp E. r'al+ + expx E. PaL=).
J+1 arg min
llo
IT
<tEDī
where -+ loggo ..a)).
b Update the policy m
end for
3.2
PREFORL Loss FUNCTION
Given a sparse-reward offline dataset D (D+, DD), ). we construct contrastive preference dataset:
Dpre: = (D+, D UD!))
Unlike CPL, which contrasts D with D-, PREFORL instead contrasts D with the degraded datasets Dls DDka to guide policy learning. The PREFORL loss function LPREFORI (To, Dpret) is defined as:
exp x.p ('alogmoStt,a
Eis s >oeDD
llog
(6)
exp Σ 7talogTs (3,, at) + expx E,- fallogTs a,))
Relation to BC and Offline RL. PREFORL leverages preference learning to address key limitations of Behavior Cloning (BC) (Pomerleau, 1988) and offline RL. BC merely imitates demonstrations and fails under distribution shift, while offline RL often suffers from value overestimation in sparse-reward datasets with limited coverage. By contrasting successful demonstrations with synthetic degradations outside the dataset's support, PREFORL mitigates overestimation and guides policies away from brittle behaviors toward more robust and reliable trajectories. In other words, contrastive training in PREFORL encourages policies to learn not just from what succeeds, but also from what fails or lies outside the support of the dataset's distribution.
3.3 ALGORITHM OVERVIEW
We present the overview of PREFORL in Algorithm 1. Given initial policy ng. in each iteration, we sample multiple preferred representative segments ç+ from D+, and build their corresponding less preferred degraded segments < At the end of each iteration, optimize policy wo using PREFORL loss function shows in Equation 6. Note that PREFORL is an offline algorithm that does requires online interaction with the environment.
3.4 THFORETICAL JUSTIFICATION.
Define the state marginals of d d=, and d' as ds)(s, d' and d' (s). Assume x² as the optimal policy whose induced state marginal distribution coincides with d' (s),, i.c., the distribution over states visited by successful trajectories in D+ The following bound on the performance gap between the learned and optimal policies established based on the above assumption Cen et al. (2024):
2Rmax
(rv (s) d (s)) + 2. E. (s) (Drv(T(-Is) x *<(ss))
V (po) - (po)l < 1 1-2
1 7
5

---

Published as a conference paper at ICLR 2026
Pen
Hamner
Door
Relocate
Figure Hand manipulation tasks in Adroit.
where Rinx max,,a [|((s, a) is the maximum reward. This shows that minimize Drv(a(-1s) || T (+|s) to optimize the learned policy т. Let PA-(sE Sg) = Bern(
AlsT
where Bern denotes Bernoulli distribution. Then the cross- entropy LPREFORL loss function in Equation 6 can be re- ce-written in terms of the advantage functions as follows:
and Palse SE) Bern( AGe
Atsp
CPREFORI (A, (pret)) Bist (E) DDrer [DL (Pa-(sE S6 P St sT))]
We show that Algorithm establishes connection between minimizing CPREFORL (A, D Poret) and minimizing the TV divergence between the learned policy ı and the expert policy x².
Lemma 3.1. Let π (als))
Alr.a)/a
Z(0) and x² (als) (x.a)) with softmax temperature >>
(-(a)
Suppose that the perturbed segments cover the full action space for each state s ~ d" Then:
CPREFORL (A, Dpret) +
Egndr D (+Is) (11) +
The assumption that the perturbed segments cover the action space ensures that segment preferences sufficiently constrain all state-action pairs. Therefore, minimizing the loss function encourages policy imitation of the distribution of successful trajectories D in the dataset D.
4 EXPERIMENTS AND EVALUATIONS
implemented our algorithm in tool called PREFORL. In this section, we evaluate PREFORL algorithm in various challenging domains including Meta MetaWorld World (Yu et al., 2020a), Adroit and Maze2D from D4RL (Fu et al., 2021) benchmark, and Sparse-MuJoCo proposed a previous offline RL work (Cen et al., 2024).
Adroit. The Adroit (Rajeswaran et al., 2018) domain is designed for controlling a 24-DoF simulated Shadow Hand robot complete different tasks. Demonstration of human experts and scripted controllers are given to evaluate the effectiveness of different RL or non-RL algorithms. In D4RL (Fu et al., 2021), Adroit is re-designed for offline RL setting only. We consider four tasks, i.e., pen, hammer, relocate and door (see Figure 2). In each task, three different types of datasets are provided evaluate the robustness of learning algorithm. Among them, two types of datasets are adopted from the original paper (Rajeswaran et al., 2018): 8):: human with 25 trajectories collected from human experts, large amount of expert demonstrations sampled from fine-tuned RL policy. Besides, each cloned is a mixing dataset which combines 50 percentage of expert demonstrations, and 50 percentage episodes sampled from a imitation policy trained on the demonstrations. We choose one imitation learning algorithm BC, and four offline RL algorithms (CQL, IQL, TD3+BC, CDE and ReBRAC) as baselines. Table denotes the normalized scores of PREFORL algorithms against other baselines Adroit tasks. Results indicate that PREFORL algorithm demonstrates competitive performance against other baselines and outperforms previous state-of-the-ee offline RL algorithm in majority of environments.

---

Published as a conference paper at ICLR 2026
Task
BC
CQL
IQL
TD3+BC
CDE
ReBRAC
CPL
PREFORL
pen-human
34.4
37.5
81.5=17.5
81.8+14.9
72.1
103.5:14.1
100.1+2.2
119.1=3.1
pen-cloned
56.9
39.2
77.2+17.7
61.4#19.3
42.1
91.8+21.7
91.242.2
92.0=3.3
pen-expert
85.1
107.0
133.6*16.0
146.0+7.3
105.0
154.1=5.4
130.9#3.2
144.8
door-human
0.5
9.9
3.1+2.0
-0.1
7.7
0,0+0.0
11.9#088
15,5+3,2
door-cloned
-0.1
0.4
0.8+1.0
0.1+0.6
0.1
1.1+2.6
3.6+3.5
16.3+0.7
door-expert
34.9
101.5
105.3+2.8
84.6+44.5
105.9
104.6*2.4
105.8=002
106.0#0.0
hammer-human
1.5
4.4
2.5
0.440.4
1.9
0.2+0.2
15.1
16.623.0
hammer-cloned
0.8
2.1
1.1
0.8a0.7
7.3
6.7+3.7
13.248.1
28.4+:.2
cammer-exxpen
125.6
86.7
129.610.5
117.0+30.9
126.3
133.8+0.7
128.3+0.3
128 6.0.2
relocate-human
0.0
0.2
0.1#0.1
-0.2+1..
0.3
0.0+0.0
0,6+0.0
0.920.3
relocate-cloned
-0.1
-0.1
0.2+0.4
-0.1
0.2
0.9+1.6
0.5+0.1
0.9:0.1
relocate-expert expert
101.3
95.0
106.5+2.5
107.3+1.6
102.6
106.6#3.2
110.2+0.4
111.2+0.7
Table 1: Normalized scores of PREFORL against other baselines on D4RL Adroit tasks. BC, CQL and IQL scores were taken from Fu et al. (2021), TD3+BC and ReBRAC scores were taken from Tarasov et al. (2023), and CDE scores were taken from Cen et al. (2024). Our reported results are averaged over 5 random seeds, and each data point consists of 20 evaluation trajectories.
Task
BCQ
CQL
IQ1.
TD3+BC
CDE
ReBRAC
CPI.
PREFORI.
halfcheetah-medium
57.8+13.2
11.64.11
76.6=5.8
41.66 11.6
82.0+8.
100.0
96.0+2.0
96.8+1.8
walker2d-medium
41,0=11.5
17.7210.4
19.5=4.2
21.0=16.7
53.0=11.7
42.0
85.316.1
98.0:3.5
hopper- mediumm
2.014.0
74.0&5.0
0.010.0
0.010.0
85.5
96.0
96.010.0
100.0:0.0
hulfcheetah- medimm--ddddeeeeeeee expert
24.849.8
4.245.8
95.4=4.2
0.040.0
95.242.9
0.0
47.3+4.6
100.020.0
87.0+13.4
61 61.6523.5
94.6=5.9
32,2422.8
97.0&2.8
36.0
100.0+0.0
100.0+0.0
hopper medium expert
20.0+11.0
94.8=2.8
22.0=10.8
97.0+1.4
98.4+3.6
Table 2: Success rate (in percent) of PREFORL against other baselines on Sparase-MuJoCo. CPL and PREFORL results are averaged over 5 random seeds, and each data point consists of 50 evaluation trajectories. Results of other baselines are taken from Cen et al. (2024) and Tarasov et al. (2023).
The Sparse-MuloCo benchmark is proposed in CDE (Cen et al., 2024) and originated from MuloCo domain in D4RL (Fu al., 2021) benchmark. Despite all episodes are collected from inherently dense-reward based environments, the quality of each trajectory can be classified into success and failed categories by examining the episode return. Following the settings in CDE, the return thresholds are set to be the 75- 75-percentile of all episode returns in each dataset. We set rewards to be for all failed trajectories in the lower 75 percent, whereas for other success trajectories. In evaluation, trajectory is considered successful when the return is above the threshold and failed otherwise. On Oparse-MuJoCo, we choose BCQ, CQL. IQL., TD3+BC and CDE as baselines. These offline RI. algorithms utilize different methods optimize policies or learn value functions, and all of them leverage sparse reward information. In Table 2, PREFORL demonstrates competitive performance against other offline RL algorithms in all domains, yet only utilizing reward information indirectly to construct sparse optimizing target. Details of experimental settings including dataset formulation and return thresholds can be found in Appendix H.
Sparse-MuJoCo.
Maze2D. The Maze2D domain includes navigation tasks aiming to instruct 2D agent to reach fixed goal position. Three maze layouts are provided with increasing difficulties, i.e., umaze, medium, and large (see figures in Table 3). Different from above-mentioned domains, the training data distribution in Maze2D differs from its evaluation distribution, and the lengths of the trajectories in the dataset varies. Specifically, in data collection process, starting position and a goal position are randomly sampled from valid positions in the maze, and an episode does not terminates if the agent reach the goal. Instead, a new goal is randomly sampled and the previous successful episode would be collected as if is an independent trajectory. In evaluation, an agent that always start from a fixed position is required to reach many goals possible within maximum episode steps. These goals will be substitute by a newly randomized one if reached. Table 3 demonstrates the average returns of PREFORL and other baselines (ReBRAC, CDE) Maze2D tasks. To enable fair comparison with CPL on Maze2D, we introduced unsuccessful trajectories by relabeling the goal regions in a subset of the original dataset trajectories. This generates explicit failure trajectories exclusively for CPL.
7

---

Published as a conference paper at ICLR 2026
UMaze
Mudicm
Large
ReBRAC CPL
0.71
0.34
1.19
0.54
0.34
CDE
1.05
0.57
0.55
PREFORL
2.22
0.67
0.63
Table Figures show navigation tasks in different mazes in Maze2D. In each maze, red and green balls denote start and goal positions. The table demonstrates average numbers of successes of PREFORL against other baselines on Maze2D tasks. All results are averaged over random seeds, and each data point consists of 50 evaluation trajectories. Note that environment, where denotes one successful contact to the sampled goal and otherwise.
sparse rewards in Maze2D
to contrast successful versus unsuccessful rollouts. The results shows that PREFORL outperforms the baselines in most environments, and can acquire high-quality policies consistently. It also shows that PREFORL performs well in both narrow (Adroit) and diverse (Maze2D) dataset distributions. Beyond Maze2D, additionally evaluate PREFORL on the AntMaze navigation benchmarks from D4RL (Fu et al., 2021). Detailed results are presented in Appendix G.
Meta World. The Meta (Yu et al., 2020a) is benchmark for meta-reinforcemmen learning and multi-task learning. It consists of 50 diverse and challenging robotic manipulation tasks. We select 16 diverse tasks from this benchmark and many of them are deemed most challenging tasks (Seo et al., 2022). Then, we use the provided scripted controller to sample 50 expert demonstrations for each selected environment. Since no reward signal recorded, this a typical learning-from demonstration problem. To evaluate the feasibility of applying PREFORL on high-dimensional environments, we set the observation space Meta World environments to be an 84 x 84 RGB image. We use BC the sole baseline, since standard offline RL algorithms typically fail in settings where only expert demonstrations are available and reward signals are absent. handle image- based observations, PREFORL and BC, and other training details are left in Appendix H. The evaluation results are shown in Table 4. The table shows, although BC is still strong baseline in nigh-dimensional goal-achieving tasks, the PREFORL algorithm outperforms it by a large margin in nearly every domain by using sparse and limited artificial preference signals.
pre- pre-trained ResNet-50 (He et al., 2015) the image encoder for both
Summary. In summary, PREFORL achieves strong performance across diverse high-dimensional control tasks and consistently outperforms BC, CPL, and other strong offline RL baselines (e.g.. ReBRAC, TD3+BC, CDE) in complex sparse-reward domains, demonstrating its ability to learn robust policies in the offline setting.
Our ablation studies (Appendix C) investigate the effect of the Gaussian perturbation variance and find that performance remains stable within a small-noise regime. This supports our analysis that must preserve locality, without requiring precise environment-specift tuning. Appendix D examines the impact of the degraded-to-preferee
dataset size, showing that PREFORL is highly insensitive to the ratio and performs reliably. Appendix E provides sensitivity analyses for two key hyperparameters, the contrastive bias coefficient and the representative segment length k. demonstrating that PREFORL maintains stable performance across broad range of settings without requiring careful tuning. Finally, Appendix reports additional comparisons against CPL and ReBRAC on World under dense-reward evaluation.
8

---

Published as a conference paper at ICLR 2026
Hammer
Peg Insert
Peg Unplug
Soccer
PREFORL BC
98.7 1 0.9 93.3 + 12.1
73.3 1 8.2
85.3 1: 3.4
67.7 + 4.1
49.0 + 5.0
67.3 +: 6.5
25.0 + 15.6
Wi ndow Open
Sweep
Di sassembll
Box Close
PREFORL BC
96.8 2.8
60.7 1 10.9
90.7 1: 4.1
89.3
91.3 + 3.9
43.3 + 11.1
82.7 + 3.1
77.0 + 12.2
Lever Pull
Drawer Open
Push Wall 1
Button Press
PREFORL BC
80.7 = 3.4 57.0 + 25.1
100.0 - 0.0 86.2 + 9.8
84.7 + 5.2
98.7 = 1.2
47.7 + 7.1
77.3 + 5.3
Stick Push
Stick Pull
Pick Wall
Soccer
PREFORL BC
100.0 + 0.0 96.6 + 0.9
59.3 - 5.2
48,0 3.3
96.8 + 3.0
86.2 + 9.8
41.3 + 3.0
25.0 + 15.6
Table 4: Success rate (in percent) of PREFORL against BC on 16 tasks from Meta World benchmark. 50 expert demonstrations are provided for each environment. We report the average results of PREFORL over 5 random seeds, and each data point consists of 50 evaluation trajectories.
RELATED WORK
Preference-based Reinforcement Learning. The mainstream preference-based RL (PbRL) meth- ods often involve learning reward model predict the scores from pairwise comparisons, then use this reward model to perform reinforcement learning for policy optimization (Christiano et al., 2017). Early work of PbRL demonstrate the feasibility of policy learning from preference signals to solve oower-dimensional problems (Wilson et al., 2012; Akrour et al., 2012; Busa-Fekett et al., 2014), recent works, however, are able to tackle control problems by training deep neural-network policies given sufficient preference labels (Sadigh et al., 2017; Biyik & Sadigh, 2018; Ibarz et al., 2018; Shin et al., 2023; III Sadigh, 2022). Within PbRL, Reinforcement Learning from Human Feedback (RLHF) is special and popular paradigm that align models with human intent. By eliminating the temporal structure RL, RLHF frame auto regressseee rext-generation as a contextual bandits problem, and many algorithms (Rafailov et al., 2023; Christiano et al., 2017; Ethayarajh et al., 2024; Shao et al., 2024) proven to work well in large-scale post-training of Large Language Models in general domains (Ouyang al., 2022; DeepSeek-AI et al., 2025a;b; Qwen al., 2025). Parallel

---

Published as a conference paper at ICLR 2026
to performance gains, the explainability of RLHF is also witnessing growing scholarly interest as burgeoning research direction (Liu et al., 2025).
Offline Reinforcement Learning. Similar to offline RL, work aims to optimize the policy solely from previously collected datasets without further interaction with the environment. This is particular useful in domains where online data collection is costly unsafe. As naive imitation learning algorithm, BC (Pomerleau, 1988) often struggles with data distributions that differ from those encountered during training. This also reveals the key challenge in offline RL: out-of-distribution (OOD) generalization. Several offiine RL. methods have been proposed to address the challenge. Behavior regularization approaches, such as BCQ (Fujimoto et al., 2019). BRAC (Wu al., 2019), BEAR (Kumar et al., 2019), IQL (Kostrikov et al., 2022) and ReBRAC (Tarasov et al., 2023), restrict learned policies stay close to the dataset's behavior distribution. Another line of work, uncertainty- aware approaches, including MOPO (Yu et al., 2020b), and CQL (Kumar et al., 2020), penalize actions with high uncertainty to mitigate the impact of distributional shift. Besides, Distribution Correction Estimation (DICE)-based methods like CDE (Cen et al., 2024) and OptiDICE (Lee et al., 2021) I) have been proposed to provide a direct behavior-agnostie estimation stationary distributions to tackle offline RL problems. Recently, decision transformer (Chen al., 2021) have introduced sequence modeling-basee based approaches, leveraging transformers to model trajectories directly. Approaches above optimize policies in various ways, yet they all focus on leveraging high-density transitions to construct learning objects. Our approach PREFORI, however, aggregates local experiences into a sparse preference signal, which implicitly encapsulates both step-wise wise knowledge for performing fine-grained control, and trajectory-level insight for discovering the optimal solution.
DISCUSSION
Limitations. For action-based degradation (t*), variance in the Gaussian noise should be tuned accordingly. Although a sufficiently small value is well-suited in practice, knowing the action range always preferred. For state-based degradation (!+), the computation overhead is not negligible as searching valid neighbors are time- time-consuming. A brute-force, exact search is computationally prohibitive for large-scale datasets, which would severely limit our method's scalability. To bypass this computational bottleneck, we deliberately employ efficient Approximated Nearest Neighbor (ANN) algorithm via FAISS library (Douze et al., 2024). This choice represents a practical trade- off between computational speed and retrieval precision. While empirical results suggest this approximation does not harm final performance, we acknowledge that our method's 's effectiveness is implicitly dependent the efficiency and quality of the ANN search algorithm.
Conclusion. In this work, we present a preference- based RL algorithm called PREFORL. Through extensive experiments, we demonstrate our approach can significantly outperform traditional imitation learning or offline RI. algorithms in sparse-reward offline dataset. By leveraging synthetically generated negative examples, method effectively mitigates value overestimation and learns robust policies, establishing new state-of-th -fte---rttt and a promising direction for offline policy learning. For future work, we aim generalize PREFORL systematic framework for generating synthetic preference data. We plan to explore its application in LLM post-training by synthesizing negative responses to regularize the model's output distributions, thereby mitigating reward hacking without additional human labeling. Furthermore, we will explore more effective and efficient degradation and sampling strategies to automatically curate high-quality preference pairs, reducing the reliance on costly human annotations for aligning foundation models.
REPRODUCIBILITY STATEMENT
We have included the instructions to reproduce our results in the supplementary material. The code is available https: // / thub. cow//U- RU-Automated wrrooniinnnrrrgPPPeeee
ACKNOWLEDGMENT
We thank the anonymous reviewers for their comments and suggestions. This work was supported by NSF Awards #CCF-2124155 and #CCF-2525293.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Riad Akrour, Mare Schoenauer, and Michèle Sebag. April: active preference learning- baannnggbbased rein- forcement learning. In Proceedings of the 2012th European Conference on Machine Learning and Knowledge Discovery in Databases Volume Part II, ECMLPKDD'I 12, pp. 116-131, Berlin, Heidelberg, 2012. Springer- Verlag. ISBN 9783642334856.
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Min Kim, and Hyun Oh Song. Direct preference-based policy optimization without reward modeling. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https: /ttps////ppeeeeeeeeee www.leewwweeeeeeeeeeee.eeeeeee.e forum? id=FkÄwlqBuyd
Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference on Robot Learning. volume of Proceedings of Machine Learning Research, pp. 519-52992222 528. PMLR, 29-31 Oct 2018. URL https:: httttt:://prrceeeeeee http//pocceddnnggg/e mlr, press / /t///////ii// ht1///iiiii/
html.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 3943/4) 1952. ISSN 00063444, 14643510. URL
http:/// /ttp://wwwww ftw.esssse jstor. .rgg /ttg//table////// /tssstaaaleee22222// 2334029.
David Brandfoobrener. Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. In Marc" Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 4933-4946, 2021.
Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference- based reinforcement learning: evolutionary direct policy search using preference-based racing algorithm. Mach. Learn., 97(3)3227-31, December 2014. ISSN 0885-6125. doi: 10.1007/ 5458-8.
s10994-001-5458-- URL https:////// /ttp::///dii. org/orggg//o 10///10.10 1.0.00007//00 /t////109944--0
Zhepeng Cen, Zuxin Liu, Zitong Wang, Yihang Yao, Henry Lam, and Ding Zhao. Learning from sparse offline datasets via conservative density estimation. In The Twelfth International Conference on Learning Representations, 2024. URL id= 1d=44M000gPTX.
net/ forum?
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: /ttps///pppeeeeeeeeeeeeee net/ forum? id=a 7A=MAAMdd.
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL bttps::/pproceee/e wtwpeeediiinn...neiieeeeeeeeeeee ceppurips.ccpaappeee files /appp/22p////////////// pdf.
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li. Ziyi Gao, Aixin Liu, Bing Xue. Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu. Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L..
11

---

Published as a conference paper at ICLR 2026
Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou. Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shumfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye. Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui L.i, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xic, Zhengyan Zhang, Zhewen Hao. Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning, 2025a.
URL https:tp////.m / aaaiiviiiiiv//.ia//..aa/aaaavvat org/ abs 20t./251...11 12948.
DeepSeek-AL, Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang, Peng Zhang. Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. I.. Jin, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma. Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang. Wenjun Gao, Wengin Yu, Wentao Zhang. X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun. Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Ying Tang. Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu. Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song. Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025b. URL htt.t::s:/ /ttp://aaxvvvvvv org/ abs / 2 212.... 19437.
Matthijs Douze Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanun Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela, Kto: Model alignment as prospect theoretic optimization, 2024. URL https:
01306.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021. URL https: http::::///////////o////o/asss/a
Scott Fujimoto and Shixiang Gu. A minimalist approach to offline reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:/ http::s///ppeeevieeeeee net /wo.mmirriii Q32U7dzwXpc.
12

---

Published as a conference paper at ICLR 2026
Scott Fujimoto, David Meger, and Doina Precup. Off- Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https:ttpt..... http://axxvv.g.o
/1512 03385.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. The Twelfth International Conference on Learning Representations, 2024. URL https://appnreet http///ppeeeeveeeeeeeeeeeeeeeeee net /oor//formmm/mm ddiiiiij
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gareet (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: /ttppse///pooceeiiiee ngs ngs. neurips wtw.iisscpppepeeeee pocp/pppr_fille wtt.fflllspapp//20111... file/ /cbe 9
ha-Paper.por .pdf.
Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference leamning for human-in in-the-loop RL. In 6th Annual Conference on Robot Learning, 2022. URL https: /otp:////e///eeee.e///o. net/
W. Bradley Knox, Stephane Hatgis-Kessell Serena Booth, Scott Nickum, Peter Stone, and Alessan- dro G Allievi. Models of human preference for learning reward functions. Transactions on Machine Learning Research, 2024. ISSN 2835-8856 URL https: http:/:://ppeeeeeeeeee net/ forum?
id=hpKJkVoThY,
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit
q-learning. In International Conference on Learning Representations, 2022. URL https: // //openrevieww.o net. http:/iwwweeet/ooeeieeeeeeeee 92
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Curran Associates Inc., Red Hook, NY. USA. 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1179-1191. Curran Asso- ciates, Inc., 2020. URL httpttsps///pccceeeineeeeoeoeeee neurips cc /paper files/iee/eeeeee //ape/ Papeer.aapp.... pdd.
Jongmin Lee, Wonseok Jeon, Byung-Jun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020. URL https:// arxiv. org/ 20000000... 01643.
Gen Li, Laixi Shi, Yuxin Chen, Yuejic Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. CoRR, abs/2204.05275,, 2022.
Shicheng Liu, Siyuan Xu, Wenjie Qiu, Hangfan Zhang, and Minghui Zhu. Explainable reinforcement learning from human feedback improve alignment. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025. URL https: http://ppeeeeeeeeee n forum?
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell. Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020,
NeurlPS 2020, December 6-12, 2020, virtual, 2020.
13

---

Published as a conference paper at ICLR 2026
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongging Lu. Mildly conservative q-learning for offline reinforcement learning. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Anmual Conference Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, IA, USA, November 28 December 2022, 2022.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information P Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387100
Dean A. Pomerleau. Alvinm: An autonomous land vehicle in a neural network. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 1. Morgan- Kaufmann, 1988. URL https: http:///oooeee/eeeeeeeeeeeeeeece 1988/ welBffile//e 2ww.a287 httpa///2..bbb880eeeeeaaaaaaaaa0
htw.rcceedinggs.eeuiiieoeee neurips cc fttpar/eeeepppppp.eeeeeee/eeeeee files/papes
Rafael Figueiredo Prudencio. Marcos Ricardo Omena Albuquerque Máximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Trans. Neural Networks Learn. Syst., 1533):10237 2024. doi: 10. 10.1109/NNLS.2022. 3250269. URL httph::p///// http:::://doo.ooggee. tttp:/ii.ooooooooooooooo.oo..oo. http//....111111111111 11t. TNNLS 2023. 3250269.
Wenjie Qiu. From Reasoning to Acting: Program Synthesis Approach to Reinforcement Learning. PhD thesis, 2026. URL
www.iriibirie....... rutgers. edu/ login?
isetat aefeesootao-acciiogooooaaaaasste ottp::/ccc..........aaaaaaooarre
2F329226621642FS 28 20wFacccontn11101133333333
Copyright Database copy- right ProQuest LLC; ProQuest does not claim copyright in the individual underlying works; Last updated 2026-01-12.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL httpps://at /ttps//axxiii orgg alss 2412 15155.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: // //ooenreeeew. net. ott.:rreeeeeeeeee/ooooeoooo----e www.mmmid--PPSSISaa.a
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. Proci Proceedings of Robotics: Science and Systems (RSS), 2018.
Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017. URL https: semant www.csshhhllaoo Cotp/oopusI::
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In 6th Annual Conference on Robot Learning, 2022. URL https:/ opwnrevieww net/ / forum? deediid-fonnno 2d 8HOJV..
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Y. K. Li. Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https: wwg.iiioog///a....oooooooooo.ooa abs// 2402. 03300.
14

---

Published as a conference paper at ICLR 2026
Daniel Shin, Anca Dragan, and Daniel S. Brown. Benchmarks and algorithms for offline preference- based reward learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
ottpp//www.......eeooooaaaaaaaa. Forum?id-TeuxXIISS..
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: httpss//ppeneevee.. net forum?idd
vgGWslLeEw.
Aaron Wilson, Alan Fem, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL https: htt.ttpss//pccceeding..... neurips. ccppppppppcppppp..pppcpaepppppee Sasaaperr_fleeeeeeeeeeeeeeeeeees fillsspapper 2012/f / 6 (7cc
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 2019. URL
http://wwwwvv.........oooo abs /ttp//991..1111111
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consisten consistent pessimism for offline reinforcement learning. In Marc Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 6683-6694, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1094-1100, PMLR, 30 Oct-01 Nov 2020a. URL https: htttt:::/ppoceedingg.. /ptpc/peeccegs...e/ppppppo mlr prwppppsssss..... wwtpp//eweeeeeeeeeeeeeeeeeeeeeee y000 html.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 14129-14142. Curran Associates, Inc., neurips wwtpaper_aeeeeeeeeeeeee fecaper_ffles_ wtt.f/feaeeeee//a////// 2020b. URL https::/prooeee obbbefffaaaa Paniddaa-pprrppd..
Brian D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Max- imum Causal Entropy. 2018 doi: 10. 00.118//11/6722622.11.
URL https: Adaptiiee
//kilthub. cmu,
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference Artificial Intelligence- Volume 3, AAAI08,, pp. 1433-1438, AAAI Press, 2008 ISBN 9781577353683.
15

---

Published as a conference paper at ICLR 2026
THE USE OF LARGE LANGUAGE MODELS
We utilized Google's Gemini, a large language model, to assist in the writing process of this paper. Its use was strictly limited to improving grammar, clarity, and phrasing for better readability, The LLM did not contribute to any of the core research ideas, methodologies, or results presented.
A PROOF OF LEMMA 3.1
Lemma 3.1 Let (aas) = A
Z(a) and (a|s)
< (4,0)/a
(+(+) with softmax temperature a > 0.
Suppose that the perturbed segments cover the full action space for each state d". Then:
CPREFORL (A, DPREFORL) -0
E Exnd' (Dv(="(+ls) (( +
Proof. By definition,
LPREFORL (A, Drref) = E (st (5 pret [DKL (P^r (5k 56x Pa(st (5,)
and each KL term nonnegative. Hence the assumption
CPREFORL(A, D Ppret) + 0
implies
DEL P + for Ppref-a.e. segment pair.
Since PA. and PA Bernoulli distributions, (PA [
0 if and only if their success probabilities coincide (in preference learning, "success" means "prefer the positive segment"). Thus, for almost all (st S
Pa- (st :5 = ,
Using the logistic parameterization
Palst SE) = Bern
(err(o(A(s)t) - A(sE)) ))),
AAs eA(s))
where a is the sigmoid function, the equality PA- P implies
(A Ar = d(A(st) A(sE))
Since is strictly monotone, obtain
A (st (s = A(c,)))))))) A(sE) for aprer-a.c. segment pair.
(7)
For each state in the support of d', let (st; (r) range over dataset occurrences with 8, 8. Each such action is perturbed by Gaussian noise,
= + C2.
~~~ N00,00))
Define
A = { [a + 8t =
By assumption, the perturbed segments cover the full action space for each state s ~ d'. Thus, Asav dense in neighborhood of the action values relevant to d"
For a segment and its Saussian-perturiie counterpart,
k-1
\ (7 = Σ y (A"(s) A" (st,
1=
and the same identity holds for Å. Define the per-step discrepancy
A (815 (4, 0. (Á(se, ax) A(st, (A* (s1, at) - A² A²(s,,
16

---

Published as a conference paper at ICLR 2026
Per Equation 7, for almost all segment pairs,
7 a7)
Because the noises E are independent across the pairs (a,, a, arise from independent Gaussian perturbations. The identity above must hold for all such combinations. By standard uniqueness arguments for continuous functions under product Gaussian measures, each term must vanish:
A (st; = 0 for almost all (st, at a,).
Thus,
A(s. a) a( (s, a) Ass a-) for a.c. (s, a) and A Pick any reference action 0g E A and define c(s) = A(s, an) - (s, ao). Then on the dense set, A(s,a) A" c(s).
This extends to:
A(s,a) A*(s,a) for -aea.e. and a.
Thus, we have:
Egnd' DIV
(a,aaa)
Alaajaa
Eg
Drv
Ev ed
E edlaarlaa
(ld((xa)tc(o))/e
~ Exnd
Drv
Eare edd'a")
O.
B TRAINING CURVES
The training curves of full dataset experiments are shown in Figure From the figure we can observe that the PREFORL converges quickly. The training curves are also stable, especially in the expert datasets training. Note that PREFORL utilizes a very sparse contrastive learning optimizing target, that expect the number of gradient steps is ten times smaller than other offline RL methods (Cen et al., 2024; Tarasov et al., 2023).
C NOISES IN ACTION-BASED DEGRADATION
In Section 6, we discussed that for action-based degradation (L"), variance a in the Gaussian noise should be tuned accordingly. To determine suitable variance for PREFORL in each environment, we conduct several hyperparameter searches on the MetaWorld and Adroit tasks (in Figure 4). In all experiments, assume we have access to the ground truth action range of each environment. This is not strictly necessary but always preferred, as approximated action ranges given by offline datasets are imprecise and conservative. The conservative estimation may lead insufficient exploration, especially when datasets distributions are narrow.
We select two environments in MetaWorld: Reg-Unplus ug and levels of variances. We use x% to denote variance a is percentage of the environment action range. Results demonstrate that when the variance is set to be a reasonably small number, i.c., around 1% to 2%, the success rates for both environments achieve their highest level. If is too small (0.5%), the distinction between original actions to degraded actions may be too subtle to differentiate. This leads to insufficient exploration and optimization. Nevertheless, setting aggressive noise (e.g., 5% and 8%) may also impair the PREFORL algorithm. Applying such aggressive noises may break the semantics of the trajectories. As reflected in the proof of Lemma 3.1, PREFORL implicitly recovers per-state
and six different
17

---

Published as a conference paper at ICLR 2026
Pen
Hammer
120
120
Normalized Score
100
100
80
Expert
80
Cloned
60
Human
60
40
Expert
Cloned
20
Human
0.0
2.0
0.0
1.5
2.0
2.5
Relocate
Door
Normalized Score
80
80
Expert
Expert
60
Cloned
60
Cloned
Human
Human
40
40
20
20
0.0
2.5
0.0
1.5
2.0
Gradlent Steps
x10*
Gradlent Steps
x10*
The x-axis denotes number of gradient steps and the y-axis denotes the normalized scores. We use 5 random seeds for each environment, and each data point consists of 50 evaluation trajectories. The shadow regions denote the standard deviation of mean values across different seeds.
Figure 3: The training curves of PREFORL.
advantage differences from segment-level preferences, which requires action perturbations be local so that the differences A (s, a) A" (s (s.a') capture the local geometry of A (s, +); large-variance perturbations violate this assumption and break this identifiability step. The lemma clarifies why the noise variance must be reasonably small in order to satisfy the locality requirement in our analysis.
We select all four expert datasets in Adroit: Pen-Expert, Door-Expert, Coameer-Exper and Relocate- Expert, and evaluate the effect of four different "small" variances accordingly. In Figure 4, the success rates for Adroit tasks keep unchanged across different levels Gaussian noises. The result shows that the success rates are stable across a range of Gaussian noise levels, long as remains in the small noise regime. This indicates that the perturbation variance does not require significant anvironment-spectin tuning.
D ABLATION STUDY OF DEGRADED DATASET SIZE
We performed an ablation study varying the size of the degraded dataset by changing the degraded- to-preferred ratio ([D U Dl / from to 10, and also considered the limiting case where degraded trajectories performance remains stable across all settings: the normalized scores vary by less than 1-2 points on every Adroit task, and no consistent trend emerges as the ratio increases. This indicates that PREFORL is insensitive to the size degraded samples and does not require large or exhaustively constructed degraded dataset.
generated on-the-fly (oo) during training loops. As shown in Table 5,
This empirical robustness is consistent with our theory. The theoretical condition in Lemma 3.1 requires that the noise distribution has full support over local neighhorhood in the action space-not
18

---

Published as conference paper at ICLR 2026
Impact of Variance on MetaWorld
Impact of Variance on Adroit
-00
63.4
MAA
Dep-bppt
150
144.8
143.3
Baant
24070
42.4
Success Rate
60
73.3
128.6
128.0
125
106.9
110.0
60
100
136.2
1067
105
0.5%
1.0%
15%
2.0%
5.0% 8.0%
100
0.5%
1.0%
1.5%
2.5%
Variance
Variance
Figure 4: The left figure denotes the effect of noise level on two Meta tasks. The right figure denotes the effect of noise level on Adroit tasks with expert dataset. Results are averaged over run seeds, and each data point is collect by 50 evaluation trajectories.
Task
Ratio oc
Ratio
Ratio =
Ratio 10
door-expert
106.0
106.1
105.2
105.9
hammer-expert
128.6
128.3
128.4
128.3
pen-expert
144.8
143.1
144.0
143.3
relocate-expert 111.1
111.0
109.3
111.1
Table 5; Impact of the degraded-to to- -o--rfeferedd sample ratio Adroit tasks. Scores are normalized and averaged over 3 seeds; oo denotes on-the-fly degraded sample generation.
that the dataset enumerates this support exhaustively. Gaussian perturbation already satisfies this condition. In practice, drawing modest number of perturbed trajectories provides sufficient "proba- bilistic coverage" to estimate the expectation in the preference loss.
E SENSITIVITY ANALYSIS
We include sensitivity analyses for both the contrastive bias parameter À and the representative segment length k, and the results shown in below tables are included in Tables and the revised manuscript.
As shown in Table 6, PREFORL is robust to the choice of over a wide range of values (0.25, 0.5, 1.0). Performance varies only slightly across settings and remains consistently strong all Adroit tasks. Our default choice
0.5.
Similarly, Table 7 demonstrates that PREFORL is insensitive the segment length k. Across E [20, 50, 100, 150} (with total sequence length fixed at 200), performance on pen-human and relocate-expert remains stable, with only minor fluctuations. Our default setting is k 100. Overall, these ablations show that PREFORL does not require fine-tuning of either À or k; the method is stable across a broad range of hyperparameter choices.
F COMPARSION AGAINST CPL AND REBRAC ON METAWORLD
We include comparisons against CPL and ReBRAC on MetaWorld tasks in Table 8. For each task, the dataset contains 50 demonstrations. We intentionally augmented the demonstration dataset with environment reward information to give ReBRAC stronger supervisory signal, whereas PREFORL does not rely on any dense reward annotations.
Both CPL and ReBRAC struggle on the demonstration-only MeraWorld tasks evaluated, whereas PREFORL consistently achieves 90-100% success. On disassemble-v2 and tickk-pshh-v,, both CPL and ReBRAC fail completely, while PREFORL reaches 90.7% and 100%, respectively. On door-open-v2 and hammer-v2. where CPL or ReBRAC show stronger performance, PREFORL still matches exceeds them, achieving 100% and 98.7%. These results suggest that, least in the
19

---

Published as a conference paper at ICLR 2026
Task
À = 0.25
À 0.5
À =
pen-human
113.9 2.4
119.1 3.1
116.4 + 3.2 110.5 112.2
relocate-expert
110.7
door-expert
105.7 0.1
106.0
105.5 0.3
hammer-expert
127.7 + 0.4
128.6 0.2
122.5 + 0.5
Table 6: Ablation study on contrastive bias X for Adroit tasks.
Segment Length (k)
pen-human
relocate-expert
20
119.0 = 0.8
110.6 + 1.1
50
118.7 3.9
110.6 + 1.1
100
119.1 = 3.1
112.2 = 0.7
150
121.1 5.19
111.7 1.9
Table 7: Ablation study segment length for Adroit tasks (with total sequence length fixed at 200).
subset of World environments we examined, preference- based learning with locally degraded actions provides more reliable supervision signal than methods that rely on inferring value functions from sparse or heterogeneous demonstrations. In the failure cases for ReBRAC, we found that higher learned reward estimates critic values do not necessarily correlate with higher task success.
G EVALUATION ON ANTMAZE ENVIRONMENTS
We selected subset of the AntMaze suite (umaze-diverse-v? large-play-v2, and large-diverse-v2) where existing offline RL baselines, particularly IQL and ReBRAC, are less effective according to their results. The results reported in Table 9.
PREFORL achieves competitive performance on two of the three difficult tasks. particular, large- p2ay-v2 is the setting where preference-basee learning provides the clearest benefit: demonstrations in "play" datasets are noisy and locally inconsistent. PREFORL relies on local preference signals induced by small action degradations rather than global return estimation, allowing it to extract reliable supervision even when the offline data are highly suboptimal. On the other hand, PREFORL less effective large-diverse-v2. This dataset contains highly heterogeneous demonstrations that traverse many disconnected regions of the maze with less state overlap. As result, the local preference pairs used by PREFORL become sparse in the parts of the state space relevant for reaching the goal, making it difficult for the model to propagate local improvements across the entire maze. In contrast, ReBRAC performs well on this task because leverages global value-function regularization that benefits from the broad coverage in the diverse dataset. We believe this result provides useful insight into the strengths and limitations preference-bae based offline RL.
H EXPERIMENT DETAILS
We use x A100 80G Nvidia GPUs for experiments. In this section, we discuss more experiment details including task formulation and dataset construction, as well training details including network architectures and hyperparameters.
H.1 ADROIT
Tasks. The Adroit in D4RL (Fu et al., 2021) contains four manipulation tasks (pen, hammer; door and relocate), and three types of datasets (expert, human and cloned). In human setting, 25 suman-generaln high-quality trajectories are collected in each dataset. In expert datasets, a scripted controller is used to generate 5K successful trajectories for generating offline dataset. However, cloned a special dataset that contains 5K both success and failed trajectories, as half of the episodes are collected from expert demonstrations, and the other half are sampled from suboptimal
20

---

Published as a conference paper at ICLR 2026
Task Name
CPL
ReBRAC
PREFORL
disassemble-v2
0.0
0.0
90.7
stick-push-v2
0.0
0.0
100.0
door-open-v2
100.0
84.4
100
hammer-v2
100.0
0.0
98.7
Table 8: Comparison of CPI., ReBRAC, and PREFORI. on Meta-World Tasks.
Dataset
BC
TD3+BC
AWAC
CQL
IQL
ReBRAC
DT
PREFORL
umaze-diverse-v2
47.344.1
44.8+11.6
54.8+8.0
37.3+3.7
54.3+5.5
83,547.0
51.820.4
68.943.2
large-play-v2
0.0=0.0
0.0=0.0
0.0=0.0
20.847.3
42.0a4.5
52.3=29.0
0.0+0.0
53,2+14.2
large-diverse-22
0.0=0.0
0.0+0.0
0.0+0.0
20.5+13.2
30.3+3.6
64.0#5.4
0.0+0.0
26.1
Table 9: Average scores selected D4RL AntMaze tasks. PREFORL results averaged over seeds.
imitation policy. The Adroit dataset is typical narrow distribution dataset because the tasks in Adroit contains relatively fixed goals and traces. This makes it suitable for both action-based and state-based degradation. In practice, we use Approximated Nearest Neighbor (ANN) search method IndexIVFE
implemented FAISS (Douze al., 2024) to search 10 nearest neighbor states. If any state with 10 percent less reward is found, we use its corresponding action as the degraded action to perform contrastive learning.
Hyperparameters. PREFORL algorithm.
The hyperparameters listed in Table 10 are used train Adroit policies using
H.2 METAWORLD
Tasks. We set tasks MetaWorld with image-based observations; hence, we need a network structure to process the RGB image. We choose pre-trained ResNet-50 (He et al., 2015) model as the image encoder for both BC and PREFORL Unlike many previous works, we do not freeze the ResNet model during training.
The hyperparameters listed in Table are used to train Meta World policies using PREFORI algorithm.
Hyperparameters.
H.3 MAZE2D
The hyperparameters listed in Table 12 are used to train Maze2D policies using PREFORL algorithm.
Hyperparameters.
H.4
MUAOOE
Tasks. We adopt "-v2" tasks in D4RL for Sparse-MuJoCo domains. We convert dense rewards in the offline dataset into sparse rewards as stated in the main text, and return thresholds for each environment are listed in Table 13. Note that perform binary judgment in evaluation, i.e., trajectory is considered successful if, and only if, its return exceeds the corresponding threshold.
Hyperparameters.
The hyperparameters listed in Table 14 are used train MuJooo
policies using PREFORI algorithm.
"odeel is available at: http://dwwlllaa..yyytttoo/meeeee
21

---

Published as a conference paper at ICLR 2026
Description
Value
Discount factor, 7
1.0
Biased regularizer value
0.1
Contrastive bias X
0.5
Contrastive segments length I
64
Batch size
20
Number of gradient steps
15000
Learning rate
0.0003
Degradation operators
Action-based to, State-based Ja
Variance action-based Ja
1%
Nearest neighbor search in state-based 4a
IndexIVFFlat
Number of probes
Condition cond in state-based +*
Reward is least 10% smaller MLP (1 024, 1024, 1024)
Policy network Activation
ReLUU
Table 10: Hyperparameters of PREFORL in training Adroit tasks.
Description
Value
Discount factor
1.0
Biased regularizer value Contrastive bias
0.1
0.5
Contrastive segments length
100
Batch size
64
Number of gradient steps Learning rate
15000
0.0003
Degradation operators
Action-based La 1%
Variance in action-based
Image encoder
Pre-trained ResNet-500 502
Policy network Activation
MLP (1024, 1024, 1024)
ReLU
Table 11: Hyperparameters of PREFORL in training Meta World tasks.
Description
Value
Discount factor
1.0
Biased regularizer value a Contrastive bias À
0.1
0.5
Contrastive segments length
100
Batch size
64
Number of gradient steps Learning rate
500
0.0003
Degradation operators
Action-based La
Variance a in action-based Ja
1%
Policy network Activation
MLP (1024, 1024, 1024)
ReLU
Table 12: Hyperparameters of PREFORL in training Maze2D tasks.
22

---

Published as a conference paper at ICLR 2026
Task
Return Threshold
halfcheetah medici
4909,
malker2d-medium
3697.8
hopper-medium
1621.5
10703.4
walker2d-medae med-imm-ddeeeerrt
4924.8
hopper-mecdiumeepprr
3561,9
Table 13: The return thresholds for Sparse-MuJooo tasks.
Description
Value
Discount factor
1.0
Biased regularizer value a Contrastive bias À
0.1
0.5
Contrastive segments length
100
Batch size
64
Number of gradient steps Learning rate
15000
0.0003
Degradation operator
Action-based La
Variance in action-based
1%
Policy network Activation
MLP (1024, 1024, 1024)
ReLU
Table 14: Hyperparameters of PREFORL in training Sparse-MuJoCo tasks.
23


Reviews:
[{"review_id": "MdU5Lr0G6J", "reviewer": "Reviewer_oreG", "summary": "This paper proposes PREFORL, a preference-based offline reinforcement learning method that addresses value overestimation in sparse-reward settings. The approach trains policies via contrastive learning between successful demonstrations and synthetic degraded trajectories generated through action perturbation or state-based substitution. Core contributions include: A degradation framework augmenting sparse offline datasets, (2) A preference optimization loss bypassing explicit value estimation, and (3) Theoretical analysis linking the loss to policy imitation. Evaluations on Adroit, Sparse-MuJoCo, Maze2D, and MetaWorld benchmarks show PREFORL outperforms offline RL/imitation baselines in success rates and normalized scores.", "questions": "- The proposed paradigm includes both a plug-in data-augmentation pipeline and a corresponding contrastive training pipeline. Can this paradigm be implemented on other BC-based offline-RL methods like Decision Transformer[1]?  I am glad to see how this paradigm performs on the DT backbone with different scales of data and models.\n- Why is the state-degradation dataset constructed with the nearest neighbor state instead of directly adding noise like the action-degradation? I think there should be a comparison study of these two different state-degradation manners.\n\n[1] Decision Transformer: Reinforcement Learning via Sequence Modeling", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Novel degradation framework towards both action and state level.\n- Comprehensive experimental validation.\n- Complete theoretical analysis.\n- Good writing for easy understanding.", "weaknesses": "- More navigation tasks (e.g. Antmaze-umaze/medium/large-diverse/replay), as well as offline RL baselines, should be performed. \n- The ablation study of the degraded dataset size is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BcROOsLT6P", "reviewer": "Reviewer_aGGo", "summary": "This paper presents PREFORL (PREFerence-based Optimization for Offline RL), a novel contrastive preference learning framework designed to train robust policies from sparse-reward offline datasets. The method aims to bypass the core challenges of conventional offline RL, specifically the extrapolation error and overestimation bias that plague value-based methods in data-limited, sparse-reward settings.", "questions": "The CPL baseline is excluded from the Maze2D evaluation, with the justification that the dataset lacks unsuccessful trajectories. Given that PREFORL's core novelty is to augment these negative examples, can you make a direct comparison showing that CPL fails while PREFORL succeeds due to the synthetic degradation? It would have been a more powerful demonstration.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The key idea is fundamentally original: avoiding direct, unstable value function estimation (the common bottleneck in sparse-reward offline RL) by transforming the task into a more robust contrastive learning problem.\n\n2. The introduction of two controlled degradation operators ($\\mathcal{D}^{\\perp a}$ and $\\mathcal{D}^{\\perp s}$) is a highly creative mechanism for generating meaningful synthetic negative examples.", "weaknesses": "1. For state-based degradation ($\\mathcal{D}^{\\perp s}$), the computation overhead associated with Nearest Neighbor Search is acknowledged to be non-negligible and potentially time-consuming for large-scale datasets. A brute-force, exact search is computationally prohibitive.\n\n2. The performance of the action-based degradation method is highly sensitive to the choice of the noise variance ($\\sigma$), which limits its robustness. This requires manual tuning per environment (or environment domain) to find the reasonably small number (e.g., $1\\%$ to $2\\%$) that maximizes the success rate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yKVOrb3P6m", "reviewer": "Reviewer_k6nG", "summary": "This paper proposes a novel method named PREFORL, which utilizes a contrastive learning framework to learn preferences from successful trajectories and synthetically degraded trajectories, aiming to address the value overestimation problem in sparse-reward offline reinforcement learning.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The idea is novel, introducing the concept of preference learning into offline RL to mitigate overestimation.\n\n2) It designs a scheme for generating negative trajectories and validates the algorithm's effectiveness through extensive experiments.", "weaknesses": "1) The underlying theory and mechanism explaining why introducing the preference learning framework alleviates overestimation are unclear.\n\n2) The method for generating negative trajectories and the selection of the contrastive bias parameter vary across different tasks, and their impact on performance remains unknown.\n\n3) The proof for Lemma 3.1 is not rigorous, as the approximation between $\\hat{A}$ and $A^*$ is unreasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k0n2MAUUPo", "reviewer": "Reviewer_uvFa", "summary": "This work presents a preference-based RL algorithm, which trains policies by contrasting successful demonstrations with failure behaviors present in the dataset. Experiments on offline benchmarks with sparse rewards validate the effectiveness of the proposed method.", "questions": "How is MetaWorld configured to use sparse rewards? By default, MetaWorld provides dense reward settings.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a contrastive preference learning framework to bypass direct value function estimation.\n- This paper provides both empirical and theoretical analyses.\n- The proposed approach outperforms baselines in various benchmarks.", "weaknesses": "- The motivation is not adequately supported by evidence. The authors claim that existing methods are sensitive to support mismatches and prone to high variance or instability, particularly when data are limited or rewards are sparse, but no empirical results or references are provided to support these statements.\n- The paper does not provide results with other competitive baselines on MetaWorld, such as PREFORL [1] and CPL [2].\n- There is no sensitivity analysis on the representative segment length $k$ and the contrastive bias $\\lambda$.\n\nReferences:\n\n[1] Tarasov et al. \"Revisiting the Minimalist Approach to Offline Reinforcement Learning\", NeurIPS, 2023.\n\n[2] Hejna et al. \"Contrastive Preference Learning: Learning from Human Feedback without RL\", ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)