OPRIDE: Efficient Offline Preference-based Reinforcement Learning via In-Dataset Exploration
Authors: Yiqin Yang
Hao Hu
Yihuan Mao
Jin Zhang
Chengjie Wu
Yuhua Jiang
Xu Yang
Runpeng Xie
Yi Fan
Bo Liu
Yang Gao
Bo XU
Chongjie Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Agent Systems
Partial Observability
Diffusion Models
Primary area: reinforcement learning

Abstract: Preference-based reinforcement learning (PbRL) can help avoid sophisticated reward designs and align better with human intentions, showing great promise in various real-world applications. However, obtaining human feedback for preferences can be expensive and time-consuming, which forms a strong barrier for PbRL. In this work, we address the problem of low query efficiency in offline PbRL, pinpointing two primary reasons: inefficient exploration and overoptimization of learned reward functions. In response to these challenges, we propose a novel algorithm, Offline PbRL via In-Dataset Exploration (OPRIDE), designed to enhance the query efficiency of offline PbRL. OPRIDE consists of two key features: a principled exploration strategy that maximizes the informativeness of the queries and a discount scheduling mechanism aimed at mitigating overoptimization of the learned reward functions. Through empirical evaluations, we demonstrate that OPRIDE significantly outperforms prior methods, achieving strong performance with notably fewer queries. Moreover, we provide theoretical guarantees of the algorithm's efficiency. Experimental results across various locomotion, manipulation, and navigation tasks underscore the efficacy and versatility of our approach.

Published as a conference paper at ICLR 2026
OPRIDE: OFFLINE PREFERENCE-BASER REINFORCE- MENT LEARNING VIA IN-DATASET EXPLORATION
Yiqin Yang'; Hao Hu Yihuan Mao", Jin Zhang", Chengjie Wu², Yuhua Jiang? Xu Yang", Runpeng Xie Yi Fan'l Bo Liu?, Yang Gao', Bo Xu', Chongjie Zhang 'he Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences
TTinnguua University "Moonshot AI Amazon
$ University of Arizona
"Wahington University St. Louis yigin. yange aanf0ia.... ac..c
ABSTRACT
Preference-based reinforcement learning (PbRL) can help avoid sophisticated re- ward designs and align better with human intentions, showing great promise in various real-world applications. However, obtaining human feedback for pref- erences can be expensive and time-conee time-consuming, which forms strong barrier for PhRL. In this work, we address the problem of low query efficiency in offline PbRL, pinpointing two primary reasons: inefficient exploration and overoptimiza- tion learned reward functions. In response to these challenges, we propose a novel algorithm, Offline PbRL via In-Dataset Exploration (OPRIDE), designed enhance the query efficiency of offline PbRL. OPRIDE consists of two key features: principled exploration strategy that maximizes the informativeness of the queries and a discount scheduling mechanism aimed at mitigating overoptimization of the learned reward functions. Through empirical evaluations, we demonstrate that OPRIDE significantly outperforms prior methods, achieving strong performance with notably fewer queries. Moreover, we provide thcoretical guarantees of the al- gorithm's efficiency. Experimental results across various locomotion, manipulation, and navigation tasks underscore the efficacy and versatility of our approach.
INTRODUCTION
Reinforcement learning (RL) has proven effective across a range of sequential decision-making tasks, from mastering games like Go (Silver et al., 2016) to controlling complex systems such as robots (Ahn et al. 2022) and plasma reactors (Degrave et al. 2022). However, in many real-world applications, designing appropriate reward function is daunting challenge, these tasks often involve objectives that are difficult to formalize with numerical rewards (Yang et al.) [201]] 2023).
Preference- based RL (PbRL) (Akrour et al. [0122] Christiano et al. (017) has emerged promising paradigm, leveraging human feedback in the form of pairwise preferences, which are inherently more interpretable yet still information-rich. This paradigm allows agents learn from relative judgments rather than numerical reward signals, significantly reducing the complexity of reward design. Recent advancements in PbRL have illustrated its efficacy enabling agents to learn novel behaviors (Christiano et al. 2017: Kim et al.] (023) and in achieving better alignment with human preferences (Ouyang et 2022 Yang al. al.[2025) which are often difficult to encapsulate in reward function. Despite these ad advantages, PbRL methods still face critical challenges, particularly acquiring human feedback efficiently. Querying human preferences is both time-consuming and resource-intensive. limiting the scalability of PbRL in real-world applications.
To address this challenge, we propose Offline PbRL via In-Dataset Exploration (OPRIDE), novel algorithm designed to systematically enhance the query efficiency of offline PbRL, depicted in Figure OPRIDE introduces a principled exploration strategy that identifies the most informative
"These authors contributed equally.
"The work does relate their position Amazon.

---

Published as a conference paper at ICLR 2026
Preference query: Dpret + Drret ((EAA),
(rht, (22,,222,0 ,22,0t)
Truin reward functions prof Dataset annocation
Annotator
Train M ensembles of rewards and Q functions with Drref
(rk),
Value Estimation V
Query Selection via In-Dutaset Exploration
x K Iteration
Variance-based adjustment
Selection: (1g Tw)
Reward Free (tifes
Preference Data Dpes
Policy extruction
12
Trajoctorics
17, Estimation Pacometery
4 Potimation with
Vg 13,
t
Pammsrr Viz
Dillarenee
Phase Query Selection
Phase 2 Policy Extraction
The procedure OPRIDE consists of two phases. In the first offline phase, we select query based on exploration mechanism. The blue circles and red triangles A represent the value estimation Ve, and V2 respectively. the second stage, we first learn an reward function based on the preference dataset and then annotate the reward-free dataset. Next, we adjust the discount factor to reduce the impact of noise in the reward learning.
Figure 1:
queries by analyzing value differences between trajectories, ensuring that each query maximally contributes to learning the optimal policy. Additionally, to prevent overoptimization of the learned reward function (Gao et al. 2023 Zhu et al., 2024), particularly in regions with high uncertainty, we incorporate discount factor scheduling mechanism that dynamically adjusts the discount based on the variance in the reward estimation. Based on the pessimistic property of the smaller discount factor, we can address the overestimation issue of the value function and, subsequently, better policy performance and higher query efficiency.
Experimental evaluations diverse locomotion and manipulation tasks, including AntMaze (Fu et al., 2020) and Meta- World (Yu et al. 2019), demonstrate the efficacy of our approach in achieving strong performance with significantly fewer queries compared state-of-the-art baselines. Remarkably, our method achieves compelling results with as few as ten queries on Meta-World tasks, underscoring its efficiency and scalability. Furthermore, we provide theoretical insights into the efficiency of our algorithm, demonstrating that our exploration strategy is provably efficient under mild assumptions.
Our contributions are threefold: (1) We introduce OPRIDE, novel offline PbRL algorithm that achieves superior query efficiency through in-dataset exploration; (2) We conduct extensive ablation studies that highlight the effectiveness of each component, providing insights into the factors driving query efficiency; and (3) We provide theoretical analyses establishing the provable efficiency of our algorithm involving principled exploration strategy under mild assumptions.
1.1 RELATED WORK
Preference-based RL. Various methods have been proposed to leverage human preferences (Akrour
al. 2012 Ibarz et al. 2018 and have demonstrated success in tackling complex control tasks (Chris-
Tiano el al. 2017]
[ Lee et al. 2021) and in aligning large language models (Stiennon er al. 2020
Ouyang et al. 2022, Rafailov et 2023 2024). In the realm of offline Preference-basee Rein-
forcement Learning (PbRL). a benchmark including several basclines (e.g., disagreement based method) introduced by OPRL. (Shin et al. 2023), which selects queries based on disagreement between the reward models and is inefficient in determining the optimal policy. Kim et al. (2023) apply Transformer models to effectively capture preferences for better credit assignment, Kang et al. (2023) present direct approach to learning policy based on preferences. A recent work by Lindner query selection method for PbRL, using the Laplacian
et al. 2021) proposes an information-dires approximation and the Hessian matrix for posterior computation. In contrast, method selects queries to maximize the information gain about the optimal policy rather than the reward function, ensuring higher query efficiency.
2

---

Published as a conference paper at ICLR 2026
Our work is also related to recent two-stage offline PbRL methods that enhance reward learning or query design. CLARIFY (Mu et al.) employs contrastive learning to disambiguate preferences in noisy queries by refining trajectory representations, differing from direct exploration of value differences within the dataset. LiRE (Choi et al. 2024) introduces listwise ranking to replace pairwise queries, improving feedback efficiency through sequential comparisons. Differently, focus maximizing policy-relevant information per query via value ensemble disagreement. Additionally, we note emerging single-stage paradigms that bypass explicit reward modeling: IPL (Hejna & Sadigh 2023) and CPL (Hejna et al.) derive policies directly from preferences via inverse learning or contrastive objectives, while DPPO (An al. 2023) optimizes policies using preference logits without reward intermediates. These methods represent an alternative direction, whereas OPRIDE retains the two-stage structure to leverage established offline RI. algorithms.
In addition to empirical achievements, prior studies have also explored the theoretical aspects of PbRL. Pacchiano et al. (2021) propose provable PbRL algorithm tailored for linear MDPs, Chen et al. (2022) extend this approach to scenarios where the Eluder dimension is finite. Zhan al. (2023a) delve into the study of PbRL within an offline setting where a preference dataset is provided. Wang al. (023) propose an efficient randomized algorithm for PbRL in linear MDP and an efficient TS based algorithm for nonlinear cases with finite Eluder dimensions. /eeharri et al. (2023) provides PbRL algorithm with PAC guarantees. Novoseller al. (2020) proposes the dueling posterior sampling algorithm that has an information-theoretic guarantee. Xu et al. (2020) provide a gap-dependent analysis for preference-basee contextual bandit and imitation learning. Wu & Sun (2023) analyze the complexity of learning with uulittybased preferences and general preferences.
2
PRELIMINARIES
We consider infinite-horizon Markov Decision Processes (MDPs), defined by the tuple 7, P, P),,, with state space S, action space A, horizon H, transition function P S x A A(S) and reward function S x A [0, 1]. Without loss of generality, we assume fixed start state sp.
A (AA) specifies a decision deeision-making strategy in which the agent chooses actions adaptively based on the current state, that is, a ~ x(- (s. The value function V SS + R and the action-value function (Q-function) Q" x A - are defined as
policy
r(t at) 80 = s) (s, = Ex Σ r(s,, az) 80 8, 00 = a) (1)
(8) = E
where the expectation is w.r.t. the trajectory t induced by ा. We define the Bellman evaluation operator
(T" f),, a) = Ee [r(, +ff(s','') (''s,a')).
We use T², Q", and V to denote an optimal policy, the corresponding optimal Q-function and optimal value function, respectively. We have the Bellman optimality equation
(*(s) max Q' (s,a), Q*(s, a) -./-P(( [r(s, a) + +s)]
aEA
Meanwhile, the optimal policy nt satisfies (-(- = argmax, (Q* :), (+1 (+1s)) We aim to learn policy from the candidate policy class IT that maximizes the expected cumulative reward. Correspondingly, we define the performance metric as the sub-optimality compared with the optimal policy, i.c.,
SubOpt(z) Va (so) V=())).
2.1 BELLMAN CONSISTENT PESSIMISM
A unique challenge in offline RL is that the learned policy may induce state-action density that different from the data distribution μ4, which may lead to large extrapolation errors when we do not impose any coverage assumption μ.. Therefore, it is important to carefully characterize the distribution shift, which we measure using the coverage coefficient. Specifically, we adopt the one used in Xie et al. (2021) that considers the distribution shift of Bellman emrors:
3

---

Published as a conference paper at ICLR 2026
Definition 1 (Bellman shift coefficient (Xie et al. 2021 We define C(V; μa, Q. as follows to measure the distribution shift from an arbitrary distribution v to the data distribution μA, w.r.. Q and
11q T 2
C(Vi Q, max
11g T q 2
Here Q is the Q-function approximation class we consider. Intuitively, C(1 2, T) measures how well Bellman errors under и transfer between the distributions and μ.. For instance, a small value of C(d"; /4, Q, π enables accurate policy evaluation for т using data collected under μ.. Definition is generalization compared to prior works that is defined specific to linear function approximation (ggaraal et al., 2021; Jin et al. 2021). More generally, we have C(us T) S ||//pllso =
p(s.a) holds for any and Q. sup ((300)
2.2
PREFERENCE- BASED REINFORCEMENT LEARNING
To learn reward functions from preference labels, we consider the Bradley-Terry pairwise preference
model (Bradley & Terry 1952) as used by most prior works (Christiano al. 2017: Ibarz et al.) 2018 Palan et al. 2019). Specifically, the preference label between two given trajectories тÜ and 7j is defined as
P P Ty (R exp (R((T) (H(T) R(T:)) +1'
(5)
where (St, at) at) is a trajectory and R((T) E >' (s az) is the return function. To simplify the theoretical analysis, consider learning a return model instead of reward model. The return model R is trained to minimize the cross-entropy loss between the predicted preference and the ground truth with given preference dataset Prret as follows:
[olog (T x T2 R) + (1 log((1 P (T x T2 [RR))] (6)
Lee (R)
(r) (r) peet
where the ground truth label given by human labelers. We assume that the difference of return functions AR = (AR(T,,2)) Traj Traj + RERR E R. AR(r), = R(n) R(12)) has a finite Fluder dimension, which common general function approximation class in RI. literature Russo & Van Roy, 2013) Chen al. 2022).
Definition 2 (Eluder Dimension (Russo & Van Roy 2013)). Suppose F is a function class defined in X, the a-Eluder dimension dgg. F a) is the longest sequence {I1, 22, "I. E X such that there exists o' where v; is d-independent of (- , for all [m].
The above defined Eluder dimension is used establish a suboptimality upper bound for our proposed algorithm in Section 4 The following generalized linear preference model considered by many prior works (Pacchiano et al. 202T Zhan al. (023b) is a special case of finite Eluder dimension (Chen ct al. 2022). We include it to show that our analysis in Section is general.
Definition 3 (Generalized Linear Preference Model). In d-dimensional generalized linear models, the preference function can be represented as IP(Ti >- (0) = a((á(TI, T2), R)) where a is an increasing Lipschitz continuous function, & Traj Traj I2d is known feature map satisfying (i(TT,72))le & H and 0 E Rd is the unknown parameter.
METHOD
In this section, present our proposed algorithm, Offline Preference-based Reinforcement Learning with In-Dataset Exploration (OPRIDE), illustrated in Figure The key idea of OPRIDE is to enhance the query efficiency of offline PbRL by conducting optimistic exploration with in-dataset queries and then utilizing the learned reward function pessimistically with discount factor scheduling. The overall algorithm shown in Algorithm
4

---

Published as a conference paper at ICLR 2026
3.1 OFFLINE QUERY SELECTION WITH IN-DATASET EXPLORATION
Generating informative queries is crucial for calibrating the reward function. Various methods have been proposed to generate queries for offline preference-based RL, like disagreement- approaches (Christiano et al., 2017) and information-gain-ball
approaches (Wilson et al. [012] Shin al., 2023). but may lead the algorithm to focus on refining reward estimates in regions of the state space that are irrelevant to the optimal policy. This naturally leads to the idea of employing an exploration objective (Akrour et al. 2011) into offline query selection, where we maximize the information gain about the optimal policy rather than the reward function.
Inspired by principled exploration strategies for PbRL, analyzed in Section 4 we propose to use the difference of value differences as the exploration criteria. Specifically, we first train a set of reward functions {ro, JM i== using bootstapping. then train a set of value functions {Vxs, J using offline algorithms like IQL (Kostrikov et al.) 2021 Ma et al. (222)) with the reward functions. Finally, we select two trajectories Ti and T2 that maximize the difference of value differences between the two trajectories:
argmax argmax (V), Vo, (n)) - (V,(TT) - Vey (T2)) (mi.T))D (JE(M])
The reward function ræ, and the value functions Qo,, Vo, are iteratively updated after each preference query. Intuitively, Equation aims select queries that most effectively minimize the diameter of the uncertainty set for the value function. The diameter represents the maximum possible disagree- ment between any two candidate value functions on any two policies. Reducing this diameter is proportional to the information gain from a query. Therefore, minimizing this diameter is equiv- alent to maximizing the information gain for each query. This objective directly linked to the information gain via information ratio I' (Lu & Van Roy 2019; Russo & Van Roy| 2016). The
diameter of the uncertainty set is upper-bounded by P diim - T TToTT(R (RR:D) D) 2 - 5, where
diam R = max R max,, anen (T1) - R1 (x2)) (R2(m.) R2(TT))), and [(R; D) is the mutual information between the reward function class R and the query dataset D. Maximizing the information gain / directly corresponds to reducing the diameter. Mathematically, the sample complexity of reward function estimation is proportional
the Eluder dimension of the reward function class, d (R), while Equation 7 complexity relates to the Eluder dimension of the optimal value function class. It is often the case that (V") << dew (R). Please refer Section 4 for the complete theoretical analysis.
3.2 POLICY EXTRACTION WITH VARIANCE-BASED DISCOUNT SCHEDULING
After obtaining the preference feedback, we can train the reward function using the cross-entropy
loss in Equation 6 and annotate the reward-free free dataset D = {{(s"; > mm to obtain a labeled
dataset D {{(x af; PT) o where F 1/M DM Di=l To.- However, it is well-known that learned reward function is prone to overoptimization (Gao et al. 2023 Zhu et al. 2024). leading to overestimation of the value function and, subsequently, a suboptimal policy.
M
Learning from preference feedback is more vulnerable to this issue, as the feedback is binary and sparse. To solve this issue, we propose adjust the discount factor based on the variance of the value function estimates that serve as a stronger regulator. Using a smaller discount factor is known to provide pessimistic and robust guarantees and performs well in various settings like imitation learning (Liu et al. 2024). Specifically, we reduce the discount factor where there is higher variance in value estimation, thereby alleviating the impact of reward function overestimation.
(s. > Top m%((m (Var, (Qa, (sj, M)) [Bachl
Ysmall,
7(8
(8)
7/+
else
where 7 is the adjusted discount factor. Please note that if the variance of the value estimation for a data point is greater than the top m% in the batch, we consider that the reward function for this data point has overestimation noise and reduces the corresponding discount factor. Subsequently, we
can learn a corresponding Q-value function and extract the policy from the labcled datasets D by adopting the standard offline reinforcement learning algorithms, like IQL (Kostrikov et al. 2021). We also consider softer confidence discount mechanism, which is detailed in the Appendix
5

---

Published as a conference paper at ICLR 2026
4 THEORETICAL ANALYSIS
In this section, we investigate the theoretical guarantees for generating queries with an explorative objective. Specifically, we consider the strategy consisting of the following procedures: (1) construct a confidence set for the return function based on existing queries; (2) construct a candidate policy set using pessimistic value estimation as the criteria; and (3) select a pair of policies that maximize disagreement on values for new queries. A detailed strategy description is available in Algorithm 2
Construct Confidence Set. For the return function, can use the cross entropy loss in Equation to get the maximum likelihood estimator (MLE) for the return function Rg:
Rs argmin (R), RER
where (R) = Ei=(( LogP((; > 7;; R)) (1 - 0.) log P(r Tf; R))) the MLE loss.
Then we can constuct the confidence set for the reward function as follows:
k Ca(R) - - RER Σ ((RR)=REN E=1
(( R(T?)) ( - (+)))) S 3x
(10)
where Sa the confidence parameter to be specified later. Given the confidence set for the return function R. can subsequently construct confidence set for policies using a pessimistic value function. Specifically, we consider the pessimistic value function ÎR that leads to the worst-case value for the optimal policy over the Bellman uncertainty the value function. Please refer to Algorithm in Appendix A. for more details. The candidate policy set IIg is constructed as follows:
1I. =
n B R Cx(R), ã argmax (R( m).}. ( #Ell
(11)
Intuitively speaking, II, consists of policies that are possibly optimal within the current level of uncertainty over reward and dynamics. By constraining exploration policies in IIg. achieve proper exploitation by avoiding unnecessary explorations.
Selecting Exploratory Policies. For given pair of policies (I1, R2) in Th. determine their exploration power by measuring how much disagreement can be made for different reward functions in the confidence set. Specifically, select explorative policies via the following criteria:
k k argmax max #1. (,E,
- ((TR)
2011 R2
(n0TT PR
R
(12)
Intuitively, we choose two policies æı, 2á such that there Ri E C2 (R) that strongly prefers m over T2, and there is R2 E (R) that strongly prefer over TT. We sample two trajectories 7k,1 k,1, nh: h.? query the preference between them, and add the preference dataset. Choosing the pair of trajectories that maximize disagreement helps us explore efficiently. Then, we have the following theoretical guarantee:
Theorem 4. Let By ci / Iog( KJ K(RR)) and e = c2 / log(N Ii| )/N, where are upper bounded by
universal constants. Then the expected suboptimality of from Algorithm 2
Ct log( Q N(1 7)2
Rdgua (AR, 1/K) log (K AR)
SubOpt (T) 5 5O
(13)
K(1 (1-7)
Offline Errur
Preference Error
where is the degree of non-linearity of the link function C is the coverage coefficient in Definition N is the size of the offline dataset and K the number of queries.
Proof. See Appendix B for detailed proof.

---

Published as a conference paper at ICLR 2026
Algorithm Offline Preference-Baser Reinforcement Learning with In-Dataset Exploration
Input: Unlabeled offline dataset DD {Tn {(0] a d))))= query budget K, ensemble
number M
2: Initialized the preference dataset preef +- Ø.
3: for episode = 1, K do
4: Train ensembles reward network ra, with D Pret using Lee in Equation
5: Train M corresponding value functions Vi. Qo with each reward function ro,-
6: Select trajectories 7k:1 م that maximize the exploration objective according to Equation 7
7:
Receive the preference OF between ,k,1 and ff:2 and add to the preference dataset, i.e.,
Deret + Dpret U {(7 2h? ox)}.
8:
end for
9:
Annotate the unlabeled offline dataset D with the reward function 0 and obtain D.
10:
Adjust the discount facto 7 to y based on Equation g
11:
Extract policy मe via offline RL from D.
Output: The learned policy Te
Domain
Task lever-pull
OPRL
PT
PT+PDS
IDRL
OPRIDE
63.2.10.44
49.213.7
51.740.1
33.1
51.811..
-ide
3.5
16.810.1
12.4114
67.4+0.1
79.0
plate-slide
77.4
4.940.00
37.322.
79.613.5
79.914.6
push
10.6+1..5
16.745.0
1.8=0.4
30.7+:5.3
59.1=5.4
push-back
0.8+0.0
1.140.4
1.1=0.1
14.0#1.11
17.7+2.0
Metaworld
push-wall
7.414.2
74.8+14.
3.420.9
89.2433.2
102.2:1.2
reach
63.5+2.9
82,0+0.8
84.3+10.9
75.8
88.0
soccer
34.3+4.0
51.314.1
41.5=11.9
44.3+2.1
45.4+3.9
sweep-into
37.1
9.8+0.2
9.2-0.1
63.1
71.6=0.1
sweep
6.8+1.8
8.0+0..4
8.0
73.0+-2.8
78.5+1.0
Average
30.4+4.2
31,42299
25.0
57.0
65.3
Table 1: Performance of offline RL algorithm the reward-labeled dataset with different preference reward learning methods on the Meta-Wordh tasks over five random seeds.
Equation decomposes the suboptimality of Algorithm 2 into two terms nicely: the offline error term and the preference eтror term. The first error is due to the finite sample bias of the dataset, and the preference error due to the limited amount of preference queries. Compared to pure online learning, the preference error is reduced by factor of 1/(1 7). Therefore, querying with an offline dataset can be much more sample-efficient than pure online queries when > K. This is because the offline dataset contains rich information about dynamics and reduce the effective horizon of the problem (Hu et al. 2023). This also aligns with our empirical findings that ~ 10 queries are usually sufficient for reasonable performance offline settings.
EXPERIMENTS
In this section, we aim to answer the following questions: (1) How does our method perform on various navigation and manipulation tasks compared other offline PbRL methods? (2) How effective is the proposed exploration-bad based query selection and discounted- bbased pessimism? (3) How does method perform across different numbers of queries?
5.1 EXPERIMENTAL DETAILS
Environment Setup. We perform empirical evaluations on Meta-Worlh (Yu et al. 2019) and the Antmaze task on the D4RL benchmark (Fu et al.) (020).. the preference query, we use a segment length of 50 for all tasks. Our setup begins with dataset of unlabeled trajectories. We use
7

---

Published as a conference paper at ICLR 2026
Domain
Task
OPRI.
PT
PT+PDS
IDRL
OPRIDE
umaze
76.313.7
77.5+4.5
84.548.5
85.543.4
87.315.6
umaze-diverse
72.5+3.44
68.0+330
78.0++6.0
69.1+4.2
73.1+2.4
medium-play
0.0+0.00
63.5+00.5
72.5+6.5
63.8+..11
62.2+2.0
Antmaze
medium-diverse
0.010.0
63.5+4..5
58.014.0
65.714.1
69.415.2
large-play
7.310.9
6.5
9.0
18.743.4
27.5
large-diverse
6.942.4
23.5+10.5
8.5
14.3+2.5
21.5+1.5
Average
27.1
50.4
51.7:1.9
52.8-1..6
56.8:14.8
Table 2: Performance of offline RL algorithm the wwwrrd-lbblld dataset with different preference reward learning methods on the Antmaze tasks over five random seeds.
PDS
VDS Random Query
VDS Disagreement
OPRIDE
Task
PT
Random Query
(VDS+IDE)
bin-picking
31.9+66.2
53.4+19.0
71.9=9.0
78.5+17.8
93.3=3.2
button-press-
peepp-rsssss wall
58.8+0.9
59.4:10.9
7..2=1088
67.4+5.4
77.7=0.1
door-close
10.1
62.418.7
72.3+0.1
88.3+0.77
94.8 1.1
faucet-close
57.8:10.9
46.2:10.2
59.4
48.7+00.6
73.1=0.8 0.8
peg-insert-side
16.8=0.1
12.4
13.8+4.4
9.748.5
79.0=0.2
reach
82.0+0.8
84.3+0.9
83.3+0.1
86.611011
8.5
sweep
8.0++0.4
8.0++0.1
28.7+118
18.2+-2.9
78.5 1.0
Table 3: Ablation of the query selection module on the Meta-World tasks. We report the performance of offline RL algorithm on the reward-labeled dataset with various query selection and policy extration mechanism. IDE and VDS represent the In In-Dataset Exploration module and the Variance-based Discount Scheduling module proposed Section 3.1 respectively.
preference-basee RL method (c.g., OPRIDE) to select queries from this dataset. These queries are then used to train reward model. Subsequently, this learned reward model is used relabel the entire trajectory dataset with rewards, which is then used to train a policy with a standard offline RL algorithm. We adopt the normalized score metrie proposed by the D4RL benchmark. Please refer to Appendix for more experimental details.
Baselines. We compare OPRIDE with several State-of-the-art offline PbRL methods, including Offline treference-basee Reinforcement Learning (OPRL; Shin et al. 2023 Preference Trans- former (PT: Kim et al., 2023) and Information Directed Reward Learning (IDRL; Lindner et al.) 2021). To illustrate the effectiveness of our proposed variance-based discount, we compare our method with Provable Data Sharing (PDS: Hu al.] (202)) as baseline algorithm. For OPRL, query selection, as it yields the best performance. We adopt the same architecture in Preference Transformer (PT) for fair comparison.
employ disagreement-B
5.2 EXPERIMENTAL RESULTS
Answer to Question 1: To show that OPRIDE can generate valuable rewards with a few queries, we conducted comprehensive comparative analysis of OPRIDE against several baseline methods. utilizing Meta- Meta-World World and Antmaze tasks as our testing grounds. Specifically, we use budget of 10 reinforcement learning methods. Then, we let
queries on each task for all offline preference-ban all algorithms employ the IQL algorithm for subsequent offline training for fair comparison. The experimental results in Table [ and Table 2 are normalized episode returns averaged over five random seeds. In most tasks in Meta-World and Antmaze, OPRIDE demonstrates superior performance compared to baseline algorithms. Moreover, unlike IDRL, which relies on the Laplacian approxima- tion and the Hessian matrix for posterior computation, method leverages critic values for query selection, ensuring easier implementation and superior empirical performance, as demonstrated in comparative experiments.
8

---

To study the contribution of each component in our framework, we conduct several ablation studies to verify the effectiveness of each part, as shown Table Comparing our method with the VDS Random Query and the VDS + Di Disagreement baseline, baseline, Published as a conference paper at ICLR 2026
Domain
Tasks
Zero
Random
Negative
OPRIDE
coffee-push
7.6+4.33
5.8
0.710.1
59.4124.8
disassemble
9.3++0.4
16.8+7.3
10.1+002
26.6+:499
hammer
38.1+6.4
46.1+:244
22.6+188
50.3:+3.2
push
17.
34.4
4.642.33
59.1:15.4
Metaworld
push-wall
81.9:13.8
80.1
17.611.9
102.2::12
soccer
33.3=11.6
41.1+888
44.0=66.4 29.0+2.6
45.4
sweep
29.0+t0.2
24.9=
78.5=1.00
Table 4: Comparison between the survival instinct and OPRIDE.
hanmers22
colice
desserlie
OPRIDE FT FT 'DRL
Ratumn
Eslosle Benum
Groode 10
00
70
13
Figure 2:
Performance of offline preference-based RL algorithms with various queries. OPRIDE achieves a better query efficiency across tasks and number of queries.
We also compare OPRIDE with the recent research work Survival Instinct (Li et al. 2024 since they find that wrong rewards can also lead to good offline RL performance. Specifically, we used three types of rewards, the same the author: (1) zero: the zero reward, (2) random: labeling each transition with a reward value randomly sampled from Unif [0, 1]. and (3) negative: the negation of true reward. Then, we trained the same offline learning algorithm as OPRIDE on the labeled dataset. The experimental results in Table Lindicate that OPRIDE still outperforms these baselines in most tasks. We attribute the above experimental results the challenging nature of the dataset we created. Specifically, in Li et al. (2024), the perturbed script policy data accounts for 100% of the dataset. However, in our created dataset, the perturbed script policy data only accounts for 5% the dataset. We conduct additional experiments on Mujoco and Kitchen tasks. Please refer to Appendix for the complete experimental results.
Answer to Question 2:
that disagreement- based approaches offer little improvement over the random query selection while exploration criteria lead to vast performance improvement, showcasing that our method is able to collect useful information within a few queries. Comparing the PDS Random Query can conclude that while PDS helpful some tasks
and the VDS + Random Query baseline, like b bin siin-pickee ppiiiinvvv it fails to prevent reward overoptimization and makes the performance worse some other tasks. On the contrary, VDS Random Query is able improve over the PT baseline on most tasks, showing its robust ability to reduce reward overestimation.
We have conducted ablation experiments determine the sensitivity of the discount factor hyper- parameter. Specifically, we vary the "small values from 0.5 to 0.95 for the data points with the high variance. The experimental results in Table 7 in Appendix E indicate that 0.7~0.8 8 is an appropriate range for 7/mall. and the performance is robust across different "ismall values. We conduct additional ablation studies for the In-Dataset Exploration module, the Variance-based Discount Scheduling module and the hyperparameter m. Please refer to Appendix E for the complete results.
Answer to Question 3: investigate how the number of queries affects OPRIDE 's overall perfor- mance, vary the number of queries and compare our method with various baselines. The results presented in Figure demonstrate that OPRIDE achieves a superior query efficiency and signifi- cantly outperforms the baselines across various numbers of queries. In most tasks, OPRIDE achieves good performance with just ten queries, and its performance continues to improve as the number of queries increases. In contrast, the baseline methods require multiple times the number of queries to

---

Published as a conference paper at ICLR 2026
achieve performance par with OPRIDE (e.g., hanmer-vi Even with 20 queries, the baseline algorithm shows no significant improvement on some hard tasks (c.g., coffee-push-v2),
To improve computational efficiency, implement the following modifi- cations. (1) When training reward functions, instead of traversing the entire dataset, we sample S segments each iteration. (2) Furthermore, we utilize multi-head mechanism instead of separate training for each reward function. This means different reward functions share the same backbone, with only the last layer being distinct. Therefore, the computational cost of overall method is S x (1 + (M - 1)/L) x K instead of N x M x K, where L is the number of neural network layers. Please refer Appendix for the complete computational cost results.
Computational Cost:
CONCLUSION
This paper proposes new framework, in-dataset exploration, to improve query efficiency in offline PbRL. Compared with disagreement-basee based approaches, using an exploration strategy helps reduce the burden of learning accurate reward function in the low-return region, improving learning efficiency. Our proposed algorithm, OPRIDE, conducts in-dataset exploration by weighted trajectory queries, and principled exploration strategy deals with pairwise queries. Our method has provable guarantees, and our practical variant achieves strong empirical performance on various tasks. Compared to prior methods, our method significantly reduces the required queries. Overall, our method provides promising and principled way to reduce queries required from human labelers in PbRL.
REPRODUCIBILITY STATEMENT
We have provided the source code in the supplementary materials, which will be made public after the paper is accepted. We have provided theoretical analysis in the Appendix A and Appendix B We have also provided implementation details in the Appendix F
ACKNOWLEDGMENTS
This work is supported by the National Key R&D Program of China (No.20227D011640)) and Amazon Research Award.
REFERENCES
Agarwal, A., Kakade. S., Krishnamurthy, A., and Sun, W. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 33: 00095-20100 2020.
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98): 1-76, 2021.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., al. Do can, not i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Akrour, R., Schoenauer, M., and Sebag, M. Preference-based policy learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 201 1, Athens, Greece, September 5-9, 2011. I. Proceedings, Part 1 11, pp. 12-27. Springer, 2011.
Akrour, R., Schoenauer, M., and Sebag, M. April: Active preference learning-base reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML. PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part 11 23, pp. 116-131. Springer, 2012.
An, G., Lee, J., Zuo, X., Kosaka, N., Kim, K.-M., and Song, H. O. Direct iiseerence-based policy optimization without reward modeling. Advances in Neural Information Processing Systems, 36: 70247-70266, 2023.
10

---

Published as a conference paper at ICLR 2026
Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 -35, 1952.
Cai, Q.. Yang. Z., Jin, C., and Wang, Z. Provably efficient exploration in policy optimization. In International Conference on Machine Learning, pp. 1283-1294, PMLR, 2020.
Chen, X., Zhong, H., Yang, Z.., Wang. Z., and Wang, L. Human-in-the-loop: Provably efficient preference-basee reinforcement learning with general function approximation. In International Conference on Machine Learning, pp. 3773-3793. PMLR, 2022.
Choi, H., Jung, S., Ahn, H., and Moon, T. Listwise reward estimation for offline preference-bace reinforcement learning. International Conference on Machine Learning, pp. 8651-8671. PMLR, 2024.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414-419, 2022.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint MrXiv:2004.072/99 2020.
Gao, L.,, Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Hejna, and Sadigh, D. Inverse preference learning: Preference- rl without reward function. Advances in Neural Information Processing Systems, 36: 11880-18887, 2023.
Hejna, J., Rafailov, R., Sikchi, H.,, Finn, C., Niekum, S., Knox, W. B., and Sadigh, Contrastive preference learning: Learning from human feedback without reinforcement learning. In The Twelfth International Conference on Learning Representations.
Hu, H., Yang, Y., Zhao, Q., and Zhang, C. On the role of discount factor in offline reinforcement learning. International Conference on Machine Learning, pp. 9072-9098. PMLR, 2022.
Hu, H., Yang, Y., Zhao, Q. and Zhang, C. The provable benefits of unsupervised data sharing for offline reinforcement learning. arXiv preprint arXiv:2302. 13493, 2023.
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Jiang, N., Kulesza, A., Singh, S., and Lewis, R. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 international conference on autonomous agents and multiagent systems, pp. 1181-1189, 2015.
Jin, C., Allen-Zhu Z., Bubeck, S., and Jordan, M. I. q-learning provably efficient? Advances in neural information processing systems, 31, 2018.
Jin, Y., Yang, Z., and Wang, Z. pessimism provably efficient for offline rl? International Conference on Machine Leurning, pp. 5084-5096. PMLR, 2021.
Kang, Y., Shi, D., Liu, J., He, L., and Wang, D. Beyond reward: Offline preference-guided policy optimization. arXiv preprint arXiv:2305.162111 2023.
Kim, C., Park, J.. Shin, J., Lee, H., Abbeel, P., and Lee, K. Preference transformer: Modeling human preferences using transformers for rl. arXiv preprint arXhv:2303.00557, 2023.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2 10.00/66 2021.
11

---

Published as a conference paper at ICLR 2026
Lee, K., Smith, L. M., and Abbeel, P. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 6152-6163. PMIR, 18-24 Jul 2021. _ hst htttpps://prrooeee ings mlr. mlr. wtt.press//l33//eesm
lhtml
Li, A., Misra, D., Kolobov, A., and Cheng, C.-A. Survival instinct offline reinforcement learning. Advances in neural information processing systems, 36, 2024.
L Lindner, D., Turchetta, M. T'cciatschek,, S., Ciosek, K., and Krause, A. Information directed reward learning for reinforcement learning. Advances in Neural Information Processing Systems, 34: 3850-3862, 2021.
Liu, Y., Dong, W., Hu, Y., Wen, C., Yin, Z.-H., Zhang, C., and Gao, Y. Imitation learning from observation with automatic discount scheduling. arXiv preprint arXiv:2310.07433, 2023.
Liu, Y., Dong, W., Hu, Y., Wen, C., Yin, Z.-H., Zhang, C., and Gao, Y. Imitation learning from observation with automatic discount scheduling, 2024.
Lu. X. and Van Roy, in neural information processing systems, 32, 2019.
information-theoreee confidence bounds for reinforcement learning. Advances
Ma, X., Yang, Y., Hu, H., Liu, Q. Yang. J., Zhang, C., Zhao, Q. and Liang, B. Offline reinforcement learning with
episodic memory. arXiv preprint 10.:0290,,, 2021.
Mu, N., Hu, H., Hu, X., Yang. Y., XU, B., and Jia, Q.-. Clarify: Contrastive preference reinforcement learning for untangling ambiguous queries. In socty- International Conference Machine Learning.
Novoseller, E., Wei, Y., Sui, Y., Yue, Y., and Burdick, J. Dueling posterior sampling for preference- based reinforcement learning. In Conference on Uncertainty Artificial Intelligence, pp. 1029- 1038. PMLR, 2020.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Pacchiano, A., Saha, A., and Lee, J. Ducling rl: Reinforcement learning with trajectory preferences. arXiv preprint arXiv: 1122111.0400 1.04850, 2021.
Palan, M., Landolfi, N. C., Shevchuk, G., and Sadigh, D. Learning reward functions by integrating human demonstrations and preferences. arXiv preprint arXiv: 190.0899, 2019.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Di- rect preference optimization: Your language model is secretly a reward model. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728-5374) Curran Asso- ciates, Inc., 2023. URL.atppe://prrceednnsssss www.eeddwiinn....us.eeee.e.....e wwtprrips.ccpppppaaaa 2023/ f 00227111//5500066666777702202230 Porereeee pdf Paper Conference
files/paper)
Rafailov, R., Hejna, J., Park, R., and Finn, C. From g": Your language model is secretly q-function, 2024.
Russo, D. and Van Roy, B. Eluder dimension and the sample complexity of optimistic exploration. In NIPS. pp. 2256-22662 2256-2264. Citeseer, 2013.
Russo, D. and Van Roy, B. Learning to optimize via posterior sampling. Mathematics of Operations Research, 1221)):2211122) 2014.
Russo, D. and Van Roy, B. An iiformatiionthhorettt analysis of thompson sampling. Journal of Machine Learning Research, 17(68):1-30 2016.
Sekhari, A., Sridharan, K., Sun, W., and Wu, R. Contextual bandits and imitation learning via preference-based active queries. arXiv preprint 1rXiv:2307.12926 2023.
12

---

Published as a conference paper at ICLR 2026
Shin, D., Dragan, A. D., and Brown, D. Benchmarks and algorithms for offline preference-basee -ased reward learning. arXiv preprint arXiv:2301.01392, 2023.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, Panneershelvam, V., Lanctot, M., al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484488) 2016.
Stiennon, N., Ouyang, L.,, Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning summarize with human feedback. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3008-3021. Curran Associates, Inc.,
2020. URL htttps::/prroeeednggs...t file/ LLLeLLEBBBB366222228333556633333 cc.pppppp.cppppp....eeeeepp Files/paape:/2220
meuripe www.mneurriicoo
Wang. Y., Liu, Q., and Jin, C. Is rlhf more difficult than standard rl? arXiv preprint urXiv22306.14111 2023.
Wilson, A., Fern, A., and Tadepalli, A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, 25, 2012.
Wu, R. and Sun, W. Making rl with preference-basee feedback efficient via randomization. arXiv preprint arXiv:2310.145: 2023.
Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-connistent pessimism for offline reinforcement learning. Advances neural information processing systems, 34:6683-6694, 2021.
Xu, Y.. Wang, R., Yang, L.. Singh, A., and Dubrawski, A. Preference-basee reinforcement learning with finite-time guarantees. Advances Neural Information Processing Systems, 33: 33:18884-18794 2020.
Yang, Y., Ma, X., Li, C., Zheng, Z., Zhang. Q., Huang, G., Yang. J., and Zhao, Q. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-100 10012001222 2021.
Yang, Y., Hu, H., Li, W., Li, S., Yang, J., Zhao, Q. and Zhang, C. Flow control: Offline reinforcement learning with lossless primitive discovery. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10843-1085 2023.
Yang, Y., Wang, Q., Li, C., Hu, H., Wu, C., Jiang, Y., Zhong, D., Zhang, Z., Zhao, Q., Zhang, C., et al. Fewer may be better: Enhancing offline reinforcement learning with reduced dataset. arXiv preprint ar arXiv:2502. 18955, 2025.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning (CoRL), 2019. /Rtt::
org /1t.s.191........
Zhan, W., Uehara, M., Kallus, N., Lee, J. D., and Sun, W. Provable offline reinforcement learning with human feedback. arXiv preprint arXiv:2305. /48/6, 2023a.
Zhan, W., Uchara, M., Sun, W., and Lec, J. D. How to query human feedback efficiently rl? arXiv preprint ar 2rXhv:235..18505 2023b.
Zhu, B., Jordan, M. I., and Jiao, J. Iterative data smoothing: Mitigating reward overfitting and overoptimization rlhf. arXiv preprint arXiv:240J. 16335, 2024.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DETAILS
In this section, provide a detailed description for the theoretical version of OPRIDE as in Algorithm 2
Algorithm 2 Theoretical Analysis Version of OPRIDE
1: Input: Unlabeled offline dataset D, query budget K
2: Initialized preference dataset Dpect + 0.
3: for k= 1. K do
4: Calculate confidence set (R. for reward function based on Preer with Equation 10
5: Calculate pessimistic value function @(-) using Algorithm for each reward function C& (R.)
6: Construct the near-optimal policy set 1l using Equation
7: Select explorative policies Ti: "h within 1lg based on Equation 12
8:
Sample trajectories T T with selected policy mi 22
9:
Receive the preference OІ between Tg and oi and add it to the preference dataset
D Dpret t Dpret U (TT +2 ox)}-
10:
end for
Output: Average policy ā Ek= (T m
11:
A.1 DETAILS BELLMAN-CONSISTEET PESSIMISM (BCP; XIE ET AL.. 2021
In this section, consider Bellman--onsisteet Pessimism (BCP: Xie al. 2021
the backbone
algorithm, described in Algorithmß] is representative model-free offline a algorithm with theoretical guarantees. PEVI uses negative bonus I(:,-) over standard Q-value estimation ((. -) (Bv) to reduce potential bias due to finite data, where B is some empirical estimation of B from dataset D.
We use the following notion of g-uncertainty quantifier as follows to formalize the idea of pessimism.
Algorithm 3 Bellman consistent Bellmman-consisern
Pessimism (BCP)
1: Input: Offline Dataset Doir { = {(s) of) 1
reward function
2: Set the loss function as
K
T
Elg.d' D)
(ne(0 at) (r(s) at) + 1 (++1))))))))
=
(14)
:=1
3: Set the confidence set of value functions as
(TT,e) {oEV: ((a,1;;;)) min Elg' 94 ;;; se] d'EV
(15)
4: Compute pessimistic policy and value function
म = argmaxx min (s T).
(16)
*enl (eV(n.e)
and
argmin (51 7)
(17)
gEV(R.e)
5: Output: a and q.
Lemma 5. Under conditions Theorem, let C+ C(dy- Q, T) we have
Ct log 12
- 50
(18)
N - 7
14

---

Published as a conference paper at ICLR 2026
where ft is the output of Algorithm 3 with dataset off and return function R. Similarly, we have
C* C 2m 5 7)2
(())
(19)
where T is the output of Algorithm 4 with dataset Dogs policy and return function R.
Proof. This proof is mainly adapted from the proof of Theorem in Xie et al. (2021) to the finite- horizon case. For simplicity we only prove the first part of the lemma. The second part can be proved similarly using the pessimistic property of the value function T.
Using the optimality of we have
max
v(30: x)
min v(30. T) S max ((50, min ())
vE VERT.
we VERR.er
vE
vE
Now, let Umin (T) argmin Cr... v(0 T) and Umax (T) = argmax Qt... v T).
Using standard reward decomposition argument Cai et al. (2020), have
U1, U1,maz (TT) - U1,min (T) 11 (TT) + v1 (T) U1,min
Ul,max
Ed,
Uh.man
T 1,+
Uhhminn T Uh+ 1mmin)
1=1
Uhhmaa
T Un 1 1.xz 2 + Uh T Uh 1.minll2.t
A=1
H Cld"; V.
[1Uh
T Un T + Ilun,mn T TT uh+ m.mmml))l)....))
i=1
1 C(dd=;A 14, 7)
(20)
(21)
holds under event Eg in Lemma 14 and Eg in Lemma [5] The second inequality follows from the definition of C(d"; μ, V. π) and the last incquality follows from Lemma [4 and Lemma 15 Let π n and plug in the definition 6, we complete the proof.
Algorithm Bellman- Pessimism Evaluation
1: Input: Offline Dataset Dar { = ((sh oT reward function policy x
2: Set the loss function as
C(g. g', T: D) =
(gi(st,af)) att - (r(s), att + (5/ Tet1)))
(22)
1= t==0
3: Set the confidence set of value functions as
(T g gevv gE L(g,,m;;; min L(g', ,4 m; sej
(23)
v'Ev
4: Compute pessimistic value function as
q = argmin ((50, T).
(24)
gEV(T.()
5: Output: and q.
15

---

Published as a conference paper at ICLR 2026
B PROOF OF THEOREM 4
Theorem 6 (Restatement of Theorem Suppose (1) Q" 2, E and (2) q 2, VT e I1, q E Q. Also suppose the difference of return functions has finite Eluder dimension dga (AR, a) and the underlying distribution of the offline dataset admit a finite coverage coefficient Ct. Let 3k c1 Stylog(K AR))K and e c2vlog(2 /og((N [1| Q /)), where Cl c2 are universal constants. Then the expected suboptimality of i from Algorithm 2 upper bounded by
Su So (
Ct (oog(NNQ)IITT)
detu (AR, AR, (K|AR))
(25)
N(1- 2)
K(1 k(1->)
where N is the size of the offline dataset and K is the number of queries.
Remark 7. In Theorem O we consider finite function classes for policy 11, Q-value and return function R. However, it can be readily extended to infinite function classes by using the covering number of the function classes, as done in Chen et al. (2022): Xie et al. (202I).
l). We also remark that while consider the realizable and Bellman Bellman-complete setting where Q" Q and TQ E for simplicity, we can extend the result to approximate realizable and Bellman-complete setting as in Xie el al. 2
The suboptimality bound uses the Eluder dimension of the difference function class AR of the original return function class R. This is because the preference query between Ti and T2 and the absolute value for R(T) can be free to choose.
Remark
only determine R(T1) R(T2) from
Proof. For simplicity we let V VTT(s)).
For any return function R E CR.) and the policy गे BCP(D, R) generated by Algorithm 3 we have
(26)
R.
R.
VV
RR
UR. 0/
R
SVZ R. RR.
R. 07
RR- +0
<VT UR
Tp.
+0
+
R
SV
UR
max
UR.
R2
-7
R2
UR.
R (2)
V
RR- R. (oE
k.1
UR.
kk.1
Ra
pk.2
Ra
UR
(27)
which hold under event E in Lemma II The first inequality follows from the pessimistic property of o, the second inequality follows from the fact that is the optimal policy with respect to 0p.- The third inequality holds since R, R C& (R) and the last inequality follows from the definition of k.1, 74.2.
Following Lemma have for all policy w and reward function R, the following holds with 28:
probability at least
Ct ( Eoft-
V
N 7)
16

---

Published as a conference paper at ICLR 2026
Then we have
Vit Vi--
(28)
x5/12)
Vii-
Th- +
max R1. (2EC
UR
Un2
+ U +U
URa
S Eaft +
max RI, (R.
(VV)
(
VRg
kh
+ Vita
Vishi2)
Ri ) +
- VRi
+
G R2
+
RR2
- -
-VR2
+
UR
- Vista2) ))
S Eoff
+
max (R)
((V71)
-
12g
+
V
l2
V + 4Eott
=5ffff
max (2C
VR
Ra
+ V
V
(29)
Consider the online preference-based regret as
K
Reg(K)
V*
V
+ Va val.e).
(30)
we have
Reg(K) K)
(31)
K
max
(i
Vight 2h1
R2g
+
R2
- VR,
+ 55K
R. (2)
K
((VR.(Th.)) - ( + VRa (T - VR. (vt:2))
max
R2E
(V gk B1
- VR, (7k:1)) -V//5" - (Tn(1)))))
VR, (Th2))
-VVie VR (Th:2)) } + 5Kf
(VRi(Th:l) - VRa (T + VRa (T VRA (T
max
R1. (2)
K
16
log )
1+51 55K +5KEott
1
K
max
((Ra( - R((():2)) - (R2(Th.l) - R2(
(32)
Ry (2)
k=1
K
16
log 5) + 5KK
1 1-7
K
<aVvdARR Klo (K AR/R// AR|/)) 16,
log 8 5KE
(33)
The first inequality follows from Equation 29 The second inequality follows from Azuma-
Hoeffding's 's inequality and the fact that VR(T) Vi is martingale when 7 Please refer to Cai et al. (2020) for a detailed derivation. The last inequality follows directly from Lemma 12
Finally, set 6 1/K and follow standard argument for regret to PAC conversion (Jin et al. 201 8)
we can show that the expected suboptimality of average policy मे generated by Algorithm is upper bounded by
Ct log(NN))lIT))
dew (AR, 1/K) f/K]log log (KJAR))
SubOpt( (T) Soo V
+ ci
N (1))
K(1 -))
17

---

Published as a conference paper at ICLR 2026
Theorem 9 (Performance Guarantees with Pure Offline Queries). Suppose (1) Q" x II, and (2) q E Q Vπ E q q E Q. Also we suppose the difference of return functions has a finite Eluder dimension dra AR, a) and the undertying distribution of the offline dataset admits finite coverage coefficient C Let 3x universal constants. Then the expected suboptimality of म from Algorithm
ci / /log K AR))KK and 02 Davlog((1 11 2)/N, where e₁ 02 are
2 with pure offline queries
upper bounded by
(V CTlog( log( Q N(1 7)
denu (AR, 1/K) 1/K)logg log (K (ARR)
(og(N\ARI)
())
K(1 11-))
N(1 7)
(34)
where N is the size of the offline dataset, K is the number of queries and C' max, the distribution that generates the dataset D.
d (+) p(a)
where
Proof. The main difference between using pure offline queries and using online queries is that we have to use trajectories sampled from the dataset Gk.1, çk.2 instead online sampled trajectories
7k1 k.2, This incurs an additional performance gap of Egap = Colo(( N AR) since we need to
N(1-1)
refine our query policies within the covered policy set
{ max
(s) pss) S <C
covered
The proof for Egap is the same as standard offline guarantees, and are omitted for simplicity, Then similar the proof of Theorem 6 the regret can be bounded
Reg(K)
K
((R1(F)))) (7 (Re(Ft.1) 12(
max R (R)
+ K gap + 16
log ( 5K
(35)
1 1-7
Then following the proof of Theorem
we have
[ct (N VI
den (AR, 1/K) log (K) (RAR)
C Clog(NN
SubOpt(#) I co
+ c1
+ + V
N(1 7)
K[1 (1)
N(1 7)
18

---

Published as a conference paper at ICLR 2026
C PERFORMANCE GUARANTEES WITH PURE OFFLINE QUERIES
In pure offline settings, we have the following theorem.
Theorem 10 (Performance Guarantees with Pure Offline Queries). Suppose (1) Q E Q, T" E II, and (2) T 2T Vπ
11, E Q. Also suppose the difference of return functions has finite Eluder dimension d R, 0:) and the underlying distribution of the offline dataset admits a finite coverage coefficient Ct. Lel 8y. = C1 /log K AR)) and c22/ogg [ 1),,, where c1 c are universal constants. Then the expected suboptimality of म from Algorithm 2 with pure offline queries is upper bounded by
(V 1)) N(1-7)
SubOpt(R) R) so So
dgn (AR, 1/K) log (K (K|AR))
C' C'log(N\ARl)
K ((1-7)
N(1 7)
(36)
d" where μ
where the size of the offline dataset, K is the number of queries and C' max is the distribution that generates the dataset DD
p(s)
Proof. The main difference between using pure offline queries and using online queries is that we have to use trajectories sampled from the dataset Gk.1, fk.2 instead of online sampled trajectories
Rk1,, k.2 This incurs an additional performance gap of Egap Cllg((VAAA)) refine our query policies within the covered policy set
since we need to
N(-1)
Tovered {
d (8) S sc.
max
(ss)
The proof for Egau is the same standard offline guarantees, and omitted for simplicity. Then similar to the proof of Theorem [ the regret can be bounded as
K
Reg( K) S
max k= R. (R)
((R1(Ft.)) - R2(kki2)) - (R2(Fk.1) - R2((kk2)))
K
+ K
+ 16
log (3) + Eoll
(37)
1 1-7
Then following the proof of Theorem 6 we have
Ct log((N MI[[TI) N(1 - 7)²
deau (AR, 1/K) log (K) AR)
Clog(N (Nog(N)R)] (R)
SubOpt(#) S co
c1
K(1 7)
+
V N (1)
19

---

Published as a conference paper at ICLR 2026
D
AUXILIARY LEMMAS
Lemma 11. With probability at least 1 - 6, the following event & holds
Rd E CR)),
vk E [K],
where
Cx(R) =
(RE E R: R: ((R(T)) - - R(T2))) S ck log(())RR)//))
is an absolute constant and d'(2Raa)) is the degree of non-linearity of the link function a.
Proof. Using Lemma
16
we have that
Σ
P( > [| S 2 (R))6)))))). P(r R2)
1=1
Note that P(T)) T R) (R(TT) R(T2)). and R(r) is bounded by Rmax have
((((()) - (((=)) - (1 (T)) R (Ta))) S cx (og((411/)))
Then, by the union bound, we have the conclusion immediately. Lemma 12. Under event & in Lemma 77 holds that
K
Σ
(R1 (2() - (Re(t")) R2(T R2((x,2)) (O /6)
(38)
Under event E we have maxi Skek Lemma 16. where
Proof.
(Cx))) S 2 log (K /R|/6) by
1/2
Σ - ((())))((()))--))) fa(T))))
B ((riTa)a =
sup fIJJET
Therefore, following Lemmal
we have
k
(Ri() R((T (Ra(T (2(
Bitt,z)) .rfi (Rx)
/deeul (R, 8) log (K (K]RR/66)
(39)
Lemma 13 (Lemma 5 of Russo & Van Roy (0144). Let V (X,, be set of functions
bounded by C >> O, (Vi)zz1 and (21) be sequences such that V C V and x, E X hold 2 1. f E (S RR') and
Let Vlni. = -[f(z))...... .f.......ff)))))
R. let diam sup,,ves |lu-
be the diameter of S. Then, for any T 2 1 and o: > 0 it holds that
T
diam (Vile,) <a C(dAT) 28,
(40)
where 8T maxigisT diam (Velen.) and d dim (V)
20

---

Published as a conference paper at ICLR 2026
The following lemmas summarizes the results regarding General Function Estimator.
Lemma 14 (Theorem A.I in Xie et al. 2021)). ) For any II, let ou be defined as follows,
Gr == arg min sup
114 - T= 21/2u-
(41)
QEQ
admissible
Then the following event &g holds with probability as least 8:
139 log |
E(qm; D) S n - 7)
(42)
where E(g, D) = (. - ev L(d 9, D)
The following lemma shows that (g, m; D) could effectively estimate ||g T 11
Lemma 15 (Theorem A.2 in Xie et al. (2021)). For any π E g E Q, h E [H], and any > 0, if E(g, n. D) S e, Then the following event E3 holds with probability as least 1 ò:
231 log 1
12 S
Ve:= ED.
(43)
n(1
7)
Lemma 16 (Theorem in Agarwal al. 2020)), Fix 1), assume |F| 00 and f F.
Then with probability at least 1 0
2 EaND [xx,-) (x)
s 2log(1F110)
21

---

Published as a conference paper at ICLR 2026
E
ADDITIONAL EXPERIMENTAL RESULTS
Experiments on Meta- World
Table shows the complete experimental results Meta- World.
Task
OPRL
PT
PT+PDS
IDRL
OPRIDE
assembly-v2
10.1+0.5
10.2+00.7
12.8+0.6
10.3+1.9
14.2++1.3
basketball-v2
11.7+10.2
80.7+0.1
78.7+2.0
82.7+2.5
61.4422.
bin-picking-v2
82.045.6
31.91:6.2
53.4-19.0
84.7+2299
93.313.2
button-press-wall--22
51.711.66
58.8+0.9
59.4
69.0=1..
77.710.1
box-close-v2
15.0+0.7
17.7=0.1
17.2+0..3
16.9+0.6
16.8+0.4
coffee-push-v2
1.7#1.7
1.3+0.5
1.3:0.5
42.0=3.8
59.4=24.8
disassemble-v2
8.440.8
6.0.0.44
7.6+0.22
7.41199
26.64:499
door-close-v2
61.2411.
65.1
62.448.7
78.1+3.2
94.8
door-unlock-v2
79.2+2.3
73.7=5.4
73.6+4.8
71.2#2.9
71.0=2.3
drawer-open-v2
53.013.3
59.7#1.3
58,3+0.1
62.5+2.0
68.7+3.0
faucet-close-v2
60.8
57.8=0.9
46.240.2
61.513.2
73.1
hammer-v2
16,4+1.0
30.2+1.7
32.6+0.88
33.6+2.88
50.3+3.2
hand-insert-v2
5.2+3.22
18.7=0.1
20.3+0.6
41.9+2.7
61.8+4.9
handle-press-v2
28.714.0
27.910.2
28.2+0..2
28.0+0.4
28.7
lever-pull-v2
63.2+110.4
49.213.7
51.740.1
33.1
51.8
3.8
16.8+0.1
12,4+14
67.4+0.1
79.0=0.2
plate-slide-v2
17.4
4.940.00
37.3+2.3
79.6
79.914.6
push-v2
10.6+1.5
16.745.0
1.8+0.44
30.7+-5.3
59.1+5.4
push-back-v2
0.8++0.0
1.1+0.4
1.1+0.1
14.0+1.1
17.7+2.0
push-wall-v2
7.4+4.2
74.8+14.4
3.4+0.9
89.2+-3.2
102.2+1.2
reach-v2
63.5+2.9
82.0+0.8
84.3+0.9
7..8
0..
soccer-v2
34.31:..0
51.314.1
41.5+111.9
44.3#2.1
45.4
sweep-into-v2
17.1
9.8+0.2
9.240.1
63.1+3.5
71.6=1.1
sweep-v2
1.8
8.040..4
8.0+0.1
73.0#2.8
78.5
Table 5: Performance of offline RL algorithm on the reward-labeled dataset with different preference reward learning methods the Meta-World tasks.
We conduct wider range of experiments on MuJoCo and Kitchen tasks. The experimental results T that OPRIDE achieves superior performance compared with other baselines. The experimental results also demonstrate that the In-Dataset Exploration and Variance-based Discount Scheduling mechanisms we proposed can be effectively applied to different tasks.
Experiments on Mujoco and Kitchen
Domain
Tasks
OPRL
PT
PT+PDS
OPRIDE
hopper-medium
23.0++0.1
36.9+2.1
35.8
38.5+12.2
57.74-23.7
68.0+2.66
69.1+1.7
92.3+15.8
70.6+1.11
71.7+22.6
70.9=1.8
72.7+118
walker2d--mediun-expp
108.3413.8
109.44.0.3
108.410.5
110.3+0.2
Mujoco
halfcheetah- mediiu
41.9+-0.1
42.14..11
41.5+0.1
42.410.1
halfcheetah malfcheetah-meciim-eee expert
81.810..6
81.940.1
82,410.2
86.5
fitchen-particn
34.610.2
48.244.1
51.1-2.3
38.743.7
kitchen-ma mixed
46.9+0.11
42.5+t1.0
44.9+119
49.8
kitchen kitchen-partia partial
62.6+1177
47.5+2.5
49.8+4.55
63.7+1.1
Table 6: Performance of offline RL algorithm the reward-labeled dataset with different preference reward learning methods on the Mujoco tasks.
Ablation about In-Dataset Exploration module The choice emphasize value functions reward functions crucial due to their ability to guide policy optimization effectively. Intuitively, while maximizing the information gain concerning the reward function (e.g., difference the
22

---

Published as a conference paper at ICLR 2026
Ssmall
0.5
0.6
0.7
0.8
0.9
0.95
bin-picking
72.11:2399
87.8:2.77
93.3#:.2
84.619.4
74.8=33.5
70,949.4
button-press-wer wall
77.6+-0.3
77.4++0.3
77.7+0.1
71.0++0.7
69.2+00.9
68.1+9977
door-close
88.4++08
89.9+00.7
94.8+t111
90.0+2211
91.1+11.5
87.6+0.7
faucet-close 58.1
58.2
73.1
61.422.7
55.1
17.4
Table 7: Performance of offline RL algorithm on the reward-labeles dataset with various discount factor values "ssmall on the high variance data points.
reward function) can help learn a well-calibrated reward function, it can still be sample inefficient in determining the optimal policy since low-return regions. For instance, suppose we have actions di and d2 that lead to a terminal state 8p. and their immediate rewards are highly uncertain, ranging from [-1, 1]. And we have actions ag and ag that lead to high return states but yield known fixed immediate reward of zero. By maximizing the reward differences, determining the optimal policy, which will not choose and 02 at all. Theoretically, maximizing the information gain with respect to the reward function insufficient derive a performance guarantee for PbRL.
not interested in the accuracy of the reward function in
will compare d1 and ag but such comparison contains information in
We conduct additional ablation studies for these two mechanisms. The experimental results in Table show that maximizing information gain about the optimal policy can achieve better performance than the reward function.
Domain
Tasks
OPRIDE (Reward Difference)
OPRIDE (Value Function Difference)
bin binpiickig
78.5
93.3=3.2
button-pre wallon-presswwall door-close faucet-close
67.4=5..4
77.7=0.1
88.3++0.7
94.8
Metaworld
48.7++066
0..1
peg-insert-side reach
9.7
7.20=1.22
8..6
88.
sweep
18.2+2.9
7..
Table 8: Ablation study the metaworld tasks.
Ablation about Variance-basee Discount Scheduling module The choice of using pessimistic discount factor in offline draws on theoretical guarantees discussed in prior works (Jiang et al. 2015 Hu et al. (022). While prior methods may utilize smaller fixed discount factor (Jiang ct al. 2015) or tuned values in imitation learning (Liu et al.] (2023), ), our approach innovatively employs variance-based discount scheduling mitigate reward overestimation issues specific to offline Preference-basee RI..
A smaller discount factor serves dual purpose: it regulates optimality against sample efficiency trade-offs (Hu et al., (022) and aligns with model-based pessimism principles, ensuring robust policy learning. Conversely, multiplicative adjustments rewards lack theoretical grounding and often yield suboptimal performance, as evidenced in Table
Domain
Tasks
OPRIDE (Penalise Reward)
OPRIDE (Penalise Discount Factor)
bin-picking
53.4+19.0
93.3-13.2
62.448.7 button-pret press-wall
59.410.9
77.7+0.11
door-close
94,8+11.1
Metaworld
faucet-close
46.2+002
73.1
peg-insert-side reach
12.4+1.4
79.0
84.3+0.99
88.0
sweep
8.0.10.1
78.
Table 9; Ablation studies about penalizing rewards and the discount factor.
23

---

Published as a conference paper at ICLR 2026
Ablation for hyperparameter m We conduct ablation study the hyperparameter m, with the results presented in Table I0 Our findings indicate a clear trade-off. An excessively large m reduces the discount factor y for too many data points, leading Conversely, a value m that is too small provides an insufficient penalty for estimation uncertainty. Based this, recommend using larger for more complex tasks (which tend to have higher estimation noise) and smaller m for simpler tasks.
overly pessimistic value estimation.
m%
10%
20%
30%
40%
50%
bin-picking
71.4++3.3
85.9=3.9
93.3++3.2
88.7+22.6
73.2+
button-press-walll
67.2+22.1
70.2=0.6
77.7+0.1
77.5+003
77.5+0.6
door-close
87.9
89.6=1.0
94.8+1.11
89.6+0.8
87.3+0.9
faucet-close
56.8:22.6
1.6
73.1
58.7+119
57.612.1
Table 10: Ablation study for the hyperparameter m from 10% to 50%.
Softer confidence discount mechanism We investigate more adaptive, "soft" confidence discount mechanism and compared it with our current threshold-basee approach. Specifically, we implement continuous annealing strategy where the smaller discount factor, Vsmall is adjusted based on the variance of the ensemble's 's value function estimates, Var d. (s,a)) a)) In this setup, 'small decreases as the variance increases, governed by the formula:
7/small max a-
The experimental results, presented in Table I1 show that the performance of this continuous annealing approach is comparable to our hard-penalty method. Given its comparable performance and simpler implementation, we opted for the threshold-basee approach in our final model.
Tasks
Continuous annealing
Threshold-based
bin-pices bin-picking button-press-wall
94.173.7
93.3=3.2
77.4+0.3
77.7=0.1
door-close
94.3++1.7
94.8
faucet-close
71.6+0..
73.1=0.8
peg-insert-side
80.5
79.0=0.2
reach
87.5
88.0=0.5
sweep
7..8+
7.0
Table Ablation study for the discount factor mechanism on the Meta-World tasks.
Computational cost We present the computational cost OPRIDE on the GeForce RTX 3090 GPU device, as shown in Table [2] The reported time is the sum of the reward function training time and the policy training time. The experimental results indicate that as the number of ensembles M increases, the computational cost does not increase significantly. This is because have adopted multi-head mechanism instead of separate training, thereby saving training time.
M
2
5
10
bin-picking button-press-wall
1.1h
1.2h
1.3h
1.1h
1.1h
1.2h
door-close
1.1h
1.2h
1.3h
faucet-close
0.9h
1.0h
1.2h
reg-insert-side
1.0h
1.1h
1.3h
Table 12: Computational Cost for OPRIDE with ensemble M, where is the hour. The report time is the sum of the reward function training time and the policy training time.
24

---

Published as a conference paper at ICLR 2026
Ablation study of query number We conduct experiments with various query numbers. The experimental results Table 13 show that performance gains accelerate most rapidly between 0.1 1% and 0.2% of queries. Beyond 0.3%, improvements gradually taper off, ultimately stabilizing at around 1 1%. At this point, adding further queries yields only marginal performance benefits.
Query Number
1 (BC)
10 (0.1%)
13
20 (0.2%)
30 (0.3%)
50 (0.5%)
100 (1.0%)
Hammer-v2
13.243.3
50.3+3.2
55,442.6
75.912.2
79.122.6
79.5+2.7
79.9
Coffce--push-v2
2.5
56.7+2.8
57.612.55
55.3#3.1
58.943.7
59,413.5
59.5+3.4
Disassemble-v2
9.3
26.6++2.8
27.8 3.1
30.6::5.2
35.7=4.88
37.2+t4.6
38.3+1..1
Push-v2
1.2+0.6
50.9+1188
63.742211
76.8+2244
82.742.6
83.6+2233
84.0 #2.2
Table 13: Performance of OPRIDE with various query numbers.
Comparison with one-stage and sequential ranked list methods We compare method with one-stage framework (IPL. (Hejna & Sadigh 2023), CPI. (Hejna et al.)) and sequential ranked list method (LiRE) (Choi al. 2024). The experimental results in Table show that compared PT, one-stage framework and LiRE achieve better performance by either bypassing reward modeling or enhancing query sampling mechanisms. Mcanwhile, OPRIDE outperforms all of them, demonstrating that our iterative two-stage framework can explore more valuable queries.
Tasks
PT
IPL
CPL
LiRE
OPRIDE
lever-pull
49.243.7
50.242.1
50.1
51,2
51.8
peg-insert-side
16.810.1
53.810.4
54.7+0.3
63.1
79,0=00.2
plate-slide
4.940.0
60.915.6
61.714.8
68.344.3
79.914.6
push
16.7+5.00
38.5+4.2
39.243.9
45.5+3.7
59.1
push-back
1.1:0.44
9.8
9.341.8
13.2
17.7+2.0
push-wall
74.8+-1..4
85.8+4..7
87.3+3.8
90.24+3.6
102.2+1.2
reach
82.0+0.88
84.2+0.2
84.7+0.6
85.4+0.3
88.0+0.5
soccer
51.3:4.11
52.314.7
53.613.9
54.213.6
45,4
sweep-into
9,8+0.2
54.2210.3
56.810.3
62.910.2
71.6=10.1
sweep
8.0:#0.4
69.3+1.1
68.2+1.3
11.
1..
Table 14: Comparison with one-stage framework and sequential ranked list method.
Pixel-based and actual numan-in-the-loow environments we conducted experiments on pixel- based Atari environments rigorously test OPRIDE's applicability beyond vector-state domains. In addition, these experiments incorporated actual human preference feedback rather than synthetic scripted teachers. We recruited 15 human evaluators with prior gaming experience to provide trajec- tory comparisons through an intuitive interface. Each evaluator performed 50 pairwise comparisons per task, with queries selected by OPRIDE's exploration mechanism.
As shown in Table 15 OPRIDE achieves state-of-the-arr results across all five Atari games. Notably, these improvements persist despite the added complexity of image-based state representations and inherent noise in human labeling. This validates OPRIDE's robustness to visual inputs and its effectiveness with genuine human feedback. We attribute this success to two key design features: (1) The exploration strategy's focus segment-level value differences remains effective when states are represented as latent features from CNN encoders, and (2) The variance-based discount scheduling mitigates overoptimization risks amplified by noisy human labels.
25

---

Published as a conference paper at ICLR 2026
Tasks Pong Breakout
OPRL
PT
PT+PDS
IDRL
OPRIDE
9.6
9.4
8.541.8
15.3=1.1
17.8
125.9+14.2
79.9
86.3
153.7=14.5
256.7
Q"bert
7924.6=3776.1
7482.94353.4
6844.24+394.6
8294.1=359.7
13535.2+327.2
Seaquest
2784
2538.3+78.9 164.9+21.5
2459.6==44.5
2941.2-16922
3478.4#71.3
Asterix
155.8+24.6
146.3+27.4
357.4=30.1
426.9+28.7
Table 15: Experiments on pixel-based Atari tasks with momman-n--the-log
26

---

Published as a conference paper at ICLR 2026
F EXPERIMENT DETAILS
Experimental Setup For the Meta- World tasks, each dataset consists of 1000 trajectories. 50 trajectories of which collected by the corresponding scripted policy added with Gaussian noise (0, 0.8) to increase diversity, and the rest 950 trajectories are collected with a policy that is e-greedy variant the former noisy policy and select random actions with probability 0.8. For the Antmaze tasks, we use the standard dataset in the D4RL benchmark but remove the reward labels.
OPRL We use the official implementation D which uses 7 ensembles. Each ensemble is initially trained with randomly selected query and then performs 3 rounds of active querying and training, and in each round, query is acquired, making a total of 10 queries.
PT We the official implementation change the number of queries to 10.
We follow its original hyper-parameter settings, and
IDRL We use the official implementation
We follow its original hyper-parameter settings.
Survival Instinct We use the official implementation settings.
We follow its original hyper-paramedt
OPRIDE Our code is built on PT. We the same transformer architecture and hyper-parametee with PT. The ensemble number N is 2. The size of 'D is 10000. The offline pre-training step for Vi(---) in the Equation 10000 c, where is the e-th selected query. Please refer to Table 16 for detailed parameters.
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
IQL. parameter
0.7
IQL parameter
3.0
Query Number
10
OPRL
Value
Ensemble Number
7
OPRIDE
Value
Ensemble Number N
2
Size of D
10000
Offline Pre-training step
10000 x
Top m%
Top 30%
Ismall S
0.7
1000
Table 16: Hyper-parameters sheet of Algorithms.
27


Reviews:
[{"review_id": "D8KrPjGfk9", "reviewer": "Reviewer_qmux", "summary": "This paper proposes Offline PbRL via In-Dataset Exploration (OPRIDE), a novel algorithm designed to systematically enhance the query efficiency of offline PbRL. OPRIDE introduces a principled exploration strategy that identifies the most informative queries by analyzing value differences between trajectories, ensuring that each query maximally contributes to learning the optimal policy. Additionally, to prevent overoptimization of the learned reward function, particularly in regions with high uncertainty, OPRIDE incorporate a discount factor scheduling mechanism that dynamically adjusts the discount based on the variance in the reward estimation. Experiments are performed on diverse locomotion and manipulation tasks, including AntMaze and Meta-World. Results compared to existing baselines demonstrates improvements in query efficiency.", "questions": "None", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method shifts query generation from purely reward-based uncertainty to policy-level uncertainty, focusing on the difference of value differences. This provides a more direct way to improve policy-relevant information efficiency compared to standard reward uncertainty sampling.\n2. Using multiple reward and value function estimates improves robustness and helps quantify epistemic uncertainty in offline settings where exploration is limited.\n3. Dynamically adjusting the discount factor based on value variance provides a practical, adaptive mechanism to mitigate reward overestimation and stabilize learning in noisy preference data.\n4. Authors conducted comprehensive experimental evaluations, which demonstrate overall significant improvements compared to baselines.", "weaknesses": "1. Potential instability: Adaptive discount factor modulation could introduce nonstationarity in learning dynamics, especially when variance estimates fluctuate across batches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kzxmkwSbkf", "reviewer": "Reviewer_5bc5", "summary": "This paper proposes OPRIDE to improve query efficiency in offline PbRL through two main components: (1) in-dataset exploration that selects queries by maximizing value function disagreement between ensemble members, and (2) variance-based discount scheduling (VDS) that adjusts the discount factor based on ensemble prediction variance to mitigate reward overoptimization. The experimental evaluation demonstrates strong query efficiency with comprehensive ablation studies.", "questions": "1. Interpretation of survival instinct: The authors argue that using datasets with only ~5% perturbed trajectories makes their setup more challenging than [1]. However, this seems contradictory: survival instinct tends to strengthen when expert-like trajectories dominate, making learning easier under misspecified rewards. Thus, a low perturbation ratio likely makes the task easier, not harder. Clarifying this conceptual mismatch would improve the paper’s experimental claims.\n\n2. Connection between Eluder dimension and proposed method: What is the specific connection between Eluder dimension in Theorem 4 and the practical algorithm (Algorithm 1)?\n\n3. Variance and overestimation relationship: How does high variance in value estimation directly lead to overestimation? Overestimation is typically measured as the gap between predicted Q-values and actual discounted sum of rewards under the same reward function, or judged via expected Q-values. High variance reflects uncertainty but does not necessarily imply overestimation.\n\n4. Discount factor adjustment and overestimation: How does adjusting the discount factor in VDS specifically mitigate overestimation? Section 3.2 refers to pessimism, but the mechanism is unclear. Reducing the discount factor may induce more myopic value estimates rather than truly mitigating overestimation. Does this adjustment simply reduce horizon length, or is there a more fundamental connection?\n\n5. Heuristic nature of m% threshold: Using a static threshold (Top m%) to identify high-variance samples for discount adjustment is quite heuristic. What principled improvements could be made?\n\n6. Scalability concern (related to Weakness 4) : What solutions exist when for cases where only a single reward model can be used due to computational constraints?\n\n[1] Li, A., Misra, D., Kolobov, A., & Cheng, C. A. (2023). Survival instinct in offline reinforcement learning. Advances in neural information processing systems, 36, 62062-62120.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Proposed method demonstrates substantially higher query efficiency than prior work, achieving compelling results with only 10 queries across diverse domains.\n2. The paper identifies and addresses reward overestimation in offline PBRL, an issue that has been relatively overlooked in previous studies. \n3. The use of critic-based query sampling, instead of reward model-based sampling, is intuitive and directly targets policy-relevant queries.", "weaknesses": "1. Missing baselines: While the paper compares OPRIDE with methods such as OPRL and PT, it does not include more recent and orthogonal approaches that bypass reward modeling (IPL [1], DPPO [2], and CPL [3]) or enhance query sampling with sequential ranked lists (LiRE[4]). The lack of these comparisons weakens the experimental contribution.\n\n2. Concerns about iterative learning: Unlike conventional two-stage offline PbRL methods that learn V and Q only once during policy extraction, OPRIDE iteratively trains V and Q during reward learning. This may introduce additional computational overhead compared to standard two-stage approaches. Furthermore, plasticity loss and capacity loss of neural networks may prevent both reward and corresponding value estimators from properly adapting to updated reward estimates as new preference data arrive [5, 6].\n\n3. Limited experimental scope: All experiments rely on vector-based state inputs, which limits practical applicability in pixel-based environments. In addition, considering the focus of PbRL, the paper also should provide actual human-in-the-loop experiments.\n\n4. Scalability concerns with mandatory ensembles: The method depends on ensemble reward models to compute value difference and measure variance. While ensemble can improve robustness, this requirement become impractical when reward models are very large, limiting scalability. The paper uses only M=2, but even this may be prohibitive in certain real-world settings.\n\n[1] Hejna, J., & Sadigh, D. (2023). Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 18806-18827.  \n[2] An, G., Lee, J., Zuo, X., Kosaka, N., Kim, K. M., & Song, H. O. (2023). Direct preference-based policy optimization without reward modeling. Advances in Neural Information Processing Systems, 36, 70247-70266.  \n[3] Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., & Sadigh, D. Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning. In The Twelfth International Conference on Learning Representations.  \n[4] Choi, H., Jung, S., Ahn, H., & Moon, T. (2024, July). Listwise Reward Estimation for Offline Preference-based Reinforcement Learning. In International Conference on Machine Learning (pp. 8651-8671). PMLR.  \n[5] Lyle, C., Rowland, M., & Dabney, W (2022). Understanding and Preventing Capacity Loss in Reinforcement Learning. In International Conference on Learning Representations.  \n[6] Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., & Dabney, W. (2023, July). Understanding plasticity in neural networks. In International Conference on Machine Learning (pp. 23190-23211). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZQUl0krGUD", "reviewer": "Reviewer_7G3h", "summary": "This paper introduces OPRIDE, a novel algorithm designed to enhance query efficiency in offline PbRL. The authors demonstrate that OPRIDE achieves statistical efficiency by analyzing the suboptimality gap. Comprehensive experiments on Meta-World and D4RL benchmarks validate the superior performance of the proposed algorithm.", "questions": "- I'm intrigued by one observation: adding as little as 0.1% to 0.2% of additional data consistently improved performance without signs of leveling off. Could you clarify at what point this trend starts to plateau? For example, does the effect diminish around 1%, 3%, 5%, or even higher? At what point does adding more data stop producing meaningful gains? Additionally, could you provide any insight into why such a small fraction of data has such a significant impact?\n\nPlease include this experiment in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The query efficiency is an important problem in offline PbRL.\n- OPRIDE achieves SOTA in most tasks across Meta-World and Antmaze benchmarks.\n- The proposed method is solid with theoretical guarantees.", "weaknesses": "- There are differences between the theoretical Algorithm 2 and the practical Algorithm 1; however, I fully understand that this simplification is necessary for theoretical analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uatFo0FRtb", "reviewer": "Reviewer_1iWt", "summary": "The paper introduces OPRIDE, an offline PbRL framework designed to improve query efficiency and mitigate overoptimization. It comprises two key components: In-dataset exploration (IDE), which selects queries maximizing differences among value functions trained on bootstrapped reward models, and variance-based discount scheduling (VDS), which dynamically reduces the discount factor for high variance state-action pairs to mitigate overestimation and stabilize learning. The authors provide theoretical guarantees on query efficiency, and experiments on Meta-World and AntMaze benchmarks demonstrate that OPRIED achieves superior performance with only 10 queries compared to prior offline PbRL baselines.", "questions": "1. How high variance in value estimation directly relates to overestimation? Overestimation can instead be measured explicitly as the gap between predicted Q-values and actual returns, or indirectly detecting spikes in streaming critic outputs [7,8].\n\n2. If overoptimization and overestimation need to be mitigated, alternative strategies such as using minimum prediction among ensemble reward models, spectral normalization and periodic resets can serve as direct remedies [9, 10]. How does the proposed VDS compare to these existing techniques?\n\n3. How would the proposed framework behave under noisy or inconsistent human feedback, which is common in real-world preference learning settings?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The IDS mechanism is intuitive and well-motivated. Unlike disagreement-based methods that focus on reward uncertainty, IDS selects queries that maximize information gain about optimal policy by measuring value function difference. Table 3 empirically validates that IDS significantly outperforms random and disagreement-based selection.\n2. The method achieves compelling performance with only 10 queries across diverse tasks (Tables 1 & 2, Figure 2), significantly outperforming baselines.\n3. The paper provides thorough ablations examining IDS vs. baselines (Table 3), VDS effectiveness (Table 9), and sensitivity to hyperparameters (Tables 7, 10). These studies clearly demonstrate the contribution of each component.", "weaknesses": "1. This paper only compares two-stage methods such as OPRL and PT. However, recent advances in offline PbRL have introduced a distinct line of one-stage framework that learns policies without reward learning, including IPL [1] and CPL [2]. These approaches are generally more robust and often outperform two-stage reward learning frameworks, as they avoid the approximation errors introduced by learning a separate reward estimator. it does not suffer from approximation error in reward estimator learning. The authors should include comparisons with these one-stage methods to more convincingly demonstrate the superiority of the proposed approach.\n\n2. Please also refer other two-stage offline PbRL studies, such as CLARIFY [3] and LiRE [4], and discuss how they related to the proposed method in the Related Works section.\n\n3. Experiments with non-ideal and noisy annotator (real human) are missing. To validate the practicality of the proposed method in real-world scenarios, the authors should include experiments involving actual human feedback [3, 4].\n\n4. The experiments are conducted only in vector-based environments. While most offline PbRL studies do not include experiments with proprioceptive (pixel-based state) inputs, evaluating such setting is essential for real-sensory and visual domains. Please refer to recent online PbRL studies that demonstrated effectiveness in pixel-based environments for this setting [5,6].\n\nReferences\n\n[1] Hejna, J., & Sadigh, D. (2023). Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 18806-18827.\n\n [2] Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., & Sadigh, D. Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning. In The Twelfth International Conference on Learning Representations.\n\n[3] Mu, N., Hu, H., Hu, X., Yang, Y., XU, B., & Jia, Q. S. CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries. In Forty-second International Conference on Machine Learning.\n\n[4] Choi, H., Jung, S., Ahn, H., & Moon, T. (2024, July). Listwise Reward Estimation for Offline Preference-based Reinforcement Learning. In International Conference on Machine Learning (pp. 8651-8671). PMLR.\n\n[5] Park, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., & Lee, K. SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning. In International Conference on Learning Representations.\n\n[6] Metcalf, K., Sarabia, M., Mackraz, N., & Theobald, B. J. (2023, December). Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards. In Conference on Robot Learning (pp. 1484-1532). PMLR.\n\n[7] Van Hasselt, H., Guez, A., & Silver, D. (2016, March). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 30, No. 1).\n\n[8] Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P. L., & Courville, A. (2022, June). The primacy bias in deep reinforcement learning. In International conference on machine learning (pp. 16828-16847). PMLR.\n\n[9] Coste, T., Anwar, U., Kirk, R., & Krueger, D. Reward Model Ensembles Help Mitigate Overoptimization. In The Twelfth International Conference on Learning Representations.\n\n[10] Nauman, M., Bortkiewicz, M., Miłoś, P., Trzcinski, T., Ostaszewski, M., & Cygan, M. (2024, July). Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning. In International Conference on Machine Learning (pp. 37342-37364). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)