Improving Reward Models with Proximal Policy Exploration for Preference-Based Reinforcement Learning
Authors: Yiwen Zhu
Jinyi Liu
Pengjie Gu
Yifu Yuan
Zhenxing Ge
Wenya Wei
Zhou Fang
Yujing Hu
Bo An
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning; Preference-based Reinforcement Learning;
Primary area: reinforcement_learning
TL;DR: To enhance the reliability of the reward model for current policy improvement, we have developed the Proximal Policy Exploration (PPE) algorithm to increase the coverage of the preference buffer in areas close to the near-policy distribution.

Abstract: Reinforcement learning (RL) heavily depends on well-designed reward functions, which are often biased and difficult to design for complex behaviors. 
Preference-based RL (PbRL) addresses this by learning reward models from human feedback, but its practicality is constrained by a critical dilemma: while existing methods reduce human effort through query optimization, they neglect the preference buffer's restricted coverage  —
a factor that fundamentally determines the reliability of reward model. 
We systematically demonstrate this limitation creates distributional mismatch: reward models trained on static buffers reliably assess in-distribution trajectories but falter with out-of-distribution (OOD) trajectories from policy exploration. 
Crucially, such failures in policy-proximal regions directly misguide iterative policy updates.
To address this, we propose **Proximal Policy Exploration (PPE)** with two key components:
(1) a *proximal-policy extension* method that expands exploration in undersampled policy-proximal regions, and
(2) a *mixture distribution query*  method that balances in-distribution and OOD trajectory sampling.
By enhancing buffer coverage while preserving evaluation accuracy in policy-proximal regions, PPE enables more reliable policy updates.
Experiments across continuous control tasks demonstrate that PPE enhances preference feedback utilization efficiency  and RL sample efficiency over baselines, highlighting preference buffer coverage management's vital role in  PbRL.

Improving Reward Models with Proximal Policy Exploration for Preference-Based Reinforcement
Learning
Yiwen 2.2
Jinyi Liu
Pengjie Gu Yifu Yuan
Zhenxing Ge
Wenya Wei
Zhou Fang" I Yujing Hu Bo An
ZZhejiang University
Nanyang Technological University Tianjin University
Nanjing University
Nettase Fuxi AI Lab "Skywork Al
Abstract
Reinforcement learning (RL) heavily depends on well-designed reward functions, which are often biased and difficult to design for complex behaviors. Preference- based RL (PbRL) addresses this by learning reward models from human feedback, but its practicality is constrained by a critical dilemma: while existing methods duce human effort through query optimization, they neglect the preference buffer's s restricted coverage
factor that fundamentally determines the reliability reward model. We systematically demonstrate this limitation creates distributional mismatch: reward models trained on static buffers reliably assess in-distribution trajectories but falter with ploration. Crucially, such failures in iterative policy updates. To address this, we propose Proximal Policy Exploration (PPE) with two key components: expands exploration in undersampled distribution query method that balances in-distribution and OOD trajectory sam- pling. By enhancing buffer coverage while preserving evaluation accuracy in policy-proxima regions, PPE enables more reliable policy updates. Experiments across continuous control tasks demonstrate that PPE enhances preference feed- back utilization efficiency and RL sample efficiency over baselines, highlighting preference buffer coverage management's vital role in PbRL.
(OOD) trajectories from policy ex- regions directly misguide
(I) a proximal-polic extension method that regions, and (2) mixture
Introduction
In reinforcement learning (RL), the reward function is pivotal it specifies the learning objectives and guides agents toward desired behaviors. Traditional RL has made significant achievements in complex domains such gaming and robotics, largely due well-designed reward functions 2 Yet, constructing these functions presents significant challenges. Designing suitable reward functions that accurately encapsulate complex behaviors like cooking summarizing books is time-consuming Additionally, embedding social norms into these
and prone to human cognitive biases [ 5
functions remains unresolved 9
An emerging alternative that addresses some of these challenges preference-based reinforcement learning (PbRL). This approach bypasses the need for meticulously engineered rewards by lever- aging overseer preferences between pairs of agent behaviors, which is typically gathered from 15 16 I71. In PbRL, agents learn to optimize behaviors that align with
humans 110
13
'Equal contribution
I Corresponding author: Hang@>jj..du.co
Code available at: hitpp:: //tgt/tgttttt
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the demonstrated human preferences, offering a more intuitive and flexible method for performing desired behaviors.
Despite its advantages, PbRL typically requires extensive preference feedback, which can be labor- intensive, time-consuming and sometimes infeasible to gather, potentially limiting its applicability in real-world settings where rapid adaptation is essential [33 14 15]. To overcome these challenges, prior research has explored various strategies for improving feedback efficiency. These strategies include selecting the most informative queries to improve the quality of the learned reward function while minimizing the required teacher input [2
19 20] Also, techniques such as sampling based on ensemble disagreements, mutual information, or behavior entropy have been employed to target behaviors to refine the overall reward model more effectively [3] [8] (20). Moreover, QPA 21 ensures that queries and policy learning progress concurrently, significantly reducing feedback unrelated the current policy and enhancing feedback efficiency.
18|
Central to these methods is the so-called "preference buffer", dataset that stores pairs of trajectory segments along with human preference labels. Analogous the replay buffer in standard RL, the preference buffer collects and maintains the preference data that the reward model is trained on. While the term "preference buffer" is not yet universally standardized, managing and sampling from such a dataset is fundamental to almost all modern PbRL algorithms [T 13
However, these methods overlook the relationship between the preference buffer and the effectiveness of the reward model. Without understanding this relationship, the reward model may not generalize well to data outside the distribution of the preference buffer. This can lead the reward model to inaccurately evaluate out-of nut-of-distribution data, potentially resulting in misguided policy improvements.
To address this issue, we focus on enhancing the coverage of the preference buffer. Basically, our findings revealed that the learned reward model provides more precise evaluations for trajectories that fall within the preference buffer's distribution. This insight led us to develop the Proximal Policy Exploration (PPE) algorithm. Firstly, we need to train an -it-r---iitributionn (OOD) detection mechanism to evaluate whether newly encountered data from the environment falls outside the preference buffer's ' distribution. Using the OOD degree measurement of the current data, we employ the pentimal-policyy extension method, which encourages the agent to explore data that, while beyond the preference buffer's distribution, still aligns closely with the current policy. Furthermore, we have designed the mixture distribution query method, which not only actively queries data outside the preference buffer's s distribution but also queries a portion of the in-distribution data. The aim of this approach is to actively expand the preference buffer's coverage while avoiding a reduction in the reward model's evaluation accuracy for in-distribution trajectories due to insufficient volume of in-distribution data. By integrating these two methods, we are able to broaden the preference buffer's coverage and bolster the reliability of the reward model's evaluations for the near-policy distribution.
Overall, work systematically addresses key bottleneck in preference-bree based reinforcement learning: the limited generalization of the reward model due to insufficient coverage of the preference buffer. By explicitly expanding the buffer's 's distribution and ensuring reliable reward evaluation, approach enables more robust and efficient policy learning in practical settings.
Our main contributions are as follows:
1. We introduce an OOD detection mechanism to ascertain whether data falls outside the preference buffer's distribution, and formulate the behavior policy resolution constrained optimization problem for exploring such data.
2. For this constrained optimization problem, we provide closed-form approximation. Through this, we introduce the proximal-policy extension method in PPE, an analytical behavior policy that directly explores data outside the preference buffer's distribution. This approach actively enhances the coverage of the preference buffer.
3. We have found that the reliability of the reward model is heavily dependent on the data distribution; the reward model can only provide reliable assessments when there is sufficient data within the evaluated distribution. To address this, we propose a mixture distribution query method in PPF, which balances the volume of in-distribution and ofi-r--disttiuutioon query data, ensuring accurate evaluations by the reward model across different regions.

---

4. Extensive experiments across multiple benchmark tasks demonstrate that PPE significantly improves both exploration efficiency and final policy performance compared to existing preference-based RI. methods, validating the effectiveness and generality of our approach.
2 Preliminaries
Preference-based RL In PbRL, an agent that interacts with an environment in discrete time steps. At each time step the agent at state selects action dz based on its policy. Unlike traditional RI., where the environment returns a reward r(*,, at) evaluating the agent's behavior, PbRL. employs preference feedback. Here, teacher provides preferences between pairs of agent behaviors, which uses to learn proxy rewards that align with human preferences, guiding policy adjustment [0 12 22 23).
Formally, a behavior segment 7 consists of a sequence of time-indexed observations and actions {(s,, ae)....).. ..... ()))))) Given pair of segments (r²,r') r"), the teacher gives their preference feedback signal Up among these segments, identifying preferred behaviors or marking segments as equally preferred incomparable. The primary objective in PbRL is to train the agent to perform behaviors aligned with human with minimal feedback.
The PbRL. learning process involves two main steps: (1) Agent Learning: The agent interacts with the environment to collect experiences and updates its policy using existing RL algorithms to maximize the sum of proxy rewards. (2) Reward Learning: The reward model fy is optimized based on feedback received from the teacher, denoted as (r".T', Up) Dr. This cyclical process continually refines both the policy and the reward model, detailed Appendix A
OOD Detection Neural networks are known for making confident predictions, even when encoun- common approach for OOD detection involves fitting generative model to the dataset, which assigns high probability to in-distribution samples and low probability to OOD ones. Although effective for simple, unimodal data, these methods can become computationally intensive when dealing with more complex and multimodal data. An alternative approach trains classifiers to act as more sophisticated OOD detectors (27).
tering outtof-distribution (OOD) samples 24 23 26
In this study, we focus on Morse neural networks 28 which train generative model to produce an unnormalized density that equals to the dataset modes. We utilize this model to generate metric that assesses the extent to which current data deviates from the preference buffer distribution. Morse neural network produces an unnormalized density M(x) [0, 1] on an embedding space RR, attaining value of at mode submanifolds and decreasing towards 0 when moving away from the mode (28). The rate at which the value decreases controlled by a Morse Kernel. More details about the Morse neural network can be found in Appendix B
We formally define the preference buffer, DP, as the dataset containing all collected preference tuples, denoted as Dr = ((7), T.. Up.;) J h=11 Fach tuple consists of a pair of trajectory segments (70, 7²) and preference label Up (e.g., V/ E {0, 1}), indicating the preferred segment. The reward model, fu.. is trained exclusively on data sampled from this buffer. A core premise of our work is that the distribution of trajectories within DF is critical for the reward model's generalization. Therefore, our proposed methods are explicitly designed monitor and strategically expand the coverage of this buffer.
Preference buffer
In this paper we "preference buffer" (D²) as the dataset of labeled preference tuples (D" (To, Tı, Up)), where (To, TI) are trajectory (or segment) snippets and yp E (0, }} indicates which one is preferred. The reward model is trained exclusively on samples drawn from (DP), and our query policy/selection schemes explicitly aim to expand the coverage of (DP) in policy-proxima regions.
3 Method
In this chapter, we delve into the importance of preference buffer coverage for the reward model in study and discuss strategies to actively expand this coverage.
3

---

Standent Sexiationn of resuad modex
Evaluation region
Spuarmatt coreeloee www.oaatioonn comfiicce
Trinngg region
Training region
Training region
Training
Evaluatión region
1P
52 301
(a)
(b)
(d)
Figure 1: Observe the reward model's effectiveness in random walk task with grid world. (a)
Training the reward model with preference data generated from trajectory pairs within the training region marked by the red frame, and assessing the correlation between the proxy and ground truth returns across all trajectories in the evaluation region denoted by the green frame; (b) The variance in the proxy rewards associated with transitions inside and outside of the training region changes in the size of the training region; (c) The Spearman correlation coefficient between proxy returns and ground truth returns for all trajectories in various evaluation regions, using reward models trained with preference data from different training regions; (d) The Spearman correlation coefficient varies with the number of feedbacks used train the reward model different training regions.
3.1 Why Coverage is Important? A Motivating Example
We designed an experiment to observe the relationship between the effectiveness of the reward model and the coverage of transitions in the preference buffer used to train the reward model.
As shown in Figure I we set up an environment in a grid world where the robot can move in four directions: up, down, left, and right. Each cell in the grid world has an associated ground truth reward, and the ground-truth return for given trajectory calculated by summing the predefined, ground-truth rewards of each cell visited in that trajectory. It should be noted that Figure ((a) serves as schematic representation; in reality, the grid world is structured as 9x9 grid. Additionally, the horizontal axes in Figures [(b) and [[c] represent the side lengths of the respective region, while the horizontal axis in Figure I(d] represents the number of feedbacks. Specifically, 'region X' refers to square area with the lefer-left corner at (0,0), bounded by the segments from 0 to X on both axes. For example, 'region denotes the grid area from (0,0) (3.3).
We designated two areas within the grid world as the training and evaluation regions, as illustrated in Figure T(a) First, we uniformly sampled 1,000 trajectory pairs of length in the training region via random walk policy (i.e., uniformly sampling actions). We sampled with replacement, so duplicate trajectories are possible, which is acceptable as the experiment focuses on the spatial coverage of training data rather than the diversity of trajectories within that region. Based on the relative sizes of their ground truth returns, we assigned preference labels to these pairs and stored them in a preference buffer. Next, we trained a reward model using data from the preference buffer with a Tradley-Terry loss. Finally, we evaluated all trajectories of length 6 in the evaluation region using the learned reward model. The goal of this evaluation is not to assess a specific policy, but rather to assess the reward model's 'ss ability to correctly rank these pre-existing trajectories. To further analyze the effectiveness of the reward model, we assessed the correlation between the proxy returns computed by the reward model and the ground truth returns using the Spearman correlation coefficient.
Preference Buffer Coverage Should Be an Explicit Optimization Objective: Results in Figure I(c)
show that expanding the training region significantly enhances the reward model's ability accurately evaluate trajectories. For instance, the blue curve in Figure [[c] representing a tiny 1xl training region, shows sharp decrease in performance as the evaluation region expands. It starts with
high Spearman correlation when evaluated on the same 1x1 region but plummets as the evaluation region grows to include states the reward model has never seen. The slight rebound of Spearman correlation when the evaluation region becomes very large does not indicate genuine generalization; is a statistical artifact of rank correlation. As the region expands, the pool includes more obviously high- vs. low-return trajectories, which the model can still order correctly despite widespread OOD errors, nudging the correlation upward from a strongly negative regime toward weak correlation. This result strongly supports central claim that reward models generalize poorly to out-o--distributiin

---

trajectories. This finding highlights the critical importance of increasing the coverage of the preference buffer over the transition space, If the preference buffer does not comprehensively cover the transition distribution of the current policy,, the proxy rewards generated by the reward model may be unreliable, making policy optimization ineffective. Only with extensive buffer coverage can the reward model reliably evaluate a broader range of trajectories. Given these observations, it is essential to explicitly include preference buffer coverage as an optimization objective in the PbRL pipeline.
Transitions: To create the ensemble baseline, we trained multiple reward models (5 in our experiment) on the exact same preference buffer. Diversity was introduced only through different random network initializations and mini- batch shuffling during training, common technique for uncertainty estimation. As demonstrated in Figurd[<b) the variance in outputs from ensemble reward models, given the same transition input, does not distinguish whether the transition belongs to the training region. Therefore, RUNE cannot actively expand the preference buffer's 's coverage.
Ensemble Variance Cannot Identify
To explore how varying the amount of preference feedback affects performance, we designed the experiment in Figure I(d) such that the evaluation region is the same as the training region. This setup allows to isolate the impact of feedback quantity while keeping the regions fixed. Figure [(d] shows that, within the same training region, increasing the amount of feedback improves the evaluation accuracy of the reward model. This indicates that it is not sufficient to focus solely on collecting iitoo--distribution data; it also necessary ensure that new queries include enough in-distribution data, Maintaining this balance is crucial to prevent the reward model from inaccurately evaluating regions it has already explored.
Balancing In- mn-Distribution and -ustt--Distriuutionn Feedback is Crucial:
Consequently, to train reliable reward model, it is essential not only for the agent to actively explore OOD data expand the preference buffer coverage but also to ensure that there a sufficient amount of in-distribution data within the preference buffer.
3.2 How to Expand Coverage of Preference Buffer? Proximal Policy Exploration
we propose the PPE algorithm, which includes two modules: the proximal-policy extension method to enhance preference buffer coverage, and the mixture distribution query method balance the inclusion of in-distribution and out-of-distribution data. By leveraging transition uncertainty estimation, PPE combines these methods to develop more reliable reward model within the current policy distribution.
Based on the observations in Section 3.1
Leveraging Morse Neural Network for Transition Uncertainty Estimation Drawing inspiration from the work of (29), we propose fo as a perturbation model that generates an action fo a). This implies that à = a only when the pair (s, a) originates from the preference buffer DP, Simul- taneously, the preference buffer DP is composed of tuples (7", T _'. (p),, where each segment 7 is sequence of aate-action pairs {st, ..),..., ($1+ a(+H)}
= (fo(s, where (RBF(E),2)) = 32 =21l?
Based on this, we design the Morse Neural Network such that M (S az) valid only when {s,, a E DD, In particular, we utilize a Radial Basis Function (RBF) kernel 30 to shape the Morse Network, as illustrated [T.
Subsequently, we optimize this Morse Neural Network by minimizing the KL divergence between un- normalized measures [1], as detailed in 28 This can be expressed as Dzz (D",s, a)ll 10]l-MMa( (s, a)). Hence, in terms of σ, this implies minimizing the loss depicted in Eq. 2). Additional details can be found in Appendix C
((d)
1 Σ Da
X 2 1
all
1 M.
Σ a. (nnfooioo((A)
exp 2 (llo(()ow))all
Here, a, signifies an action sampled from a uniform distribution over the corresponding action space, denoted as Uniform(A). Furthermore, M represents the number of samples drawn from Uniform(AA) while N refers to the number of sampled (s, a) pairs from D The parameter is used to control the sensitivity of the Morse Neural Network to OOD transitions.
5

---

Expanding Preference Buffer Coverage via Proximal-Police Extension Method Observations from Figure [(c] suggest that expanding the coverage of the preference buffer can enhance the ability of the trained reward model in evaluating the quality of trajectories. Particularly during the RL training process, only when the trained reward model has strong ability to evaluate the quality of trajectories within the proximal policy distribution can the risk of misguidance in policy improvement be reduced. Therefore, expanding the coverage of the preference buffer for the proximal policy distribution can further optimize policy improvement in PbRL.
Drawing on this insight, we have designed the proximal -poiicy extension method, to actively en- courage the agent to explore data that falls outside the preference buffer distribution but within the vicinity of the current policy's 's distribution. The behavior policy g used for exploration, is designed such that the state-action pairs (s, a) it generates when interacting with the environment can support the distribution produced by the current target policy тт. Formally, the behavior policy = N (pe, Eg) is defined as the solution to the constrained optimization problem in Eq. (3..
E [M((8.))
s.t. (N E Er)) S.
Since we need to calculate the constrained optimization problem described in 3 in each interac- tion process, using readily available solvers would result a significant consumption of computational resources. Therefore, tighten the constraint conditions obtain closed-forn approximate solu- tion as shown in Proposition 3.1 This approach greatly reduces the computational cost of solving the constrained optimization problem, while achieving desired objective of encouraging exploration of data out of the preference buffer distribution near the current policy distribution. The detailed derivation is presented in Appendix(D)
Proposition 3.1. The behavior policy for exploration resulting from E3. has the form mE N (HE, Σg), where
22e Σn (o
a)}a=pr (s.Msssss)app
-T=
EE Er.
Ma (Z Er L Mo a) a)la=pr
Mixture Distribution Query Selection In the previous section, we introduced an exploration method that enables the agent to explore a broader range of transitions that are out of the preference buffer but near the current policy distribution. These newly discovered transitions are stored in the replay buffer. Therefore, it becomes essential to have a query selection method that can select those segments that are out of the preference buffer and store them in the preference buffer.
Algorithm 1 Mixture Distribution Query
Input: 7 E DeP, MT(T), query size b and mixture ratio K.
Output: [7": 116
for i 1 nb do
L ,0,71 ~ pout (T)
// Sample outside the distribution of DR
3 i t0 (1 R.b do pin (T)
[
77,7' [
// Sample inside the distribution of DP
return (7.. - 11b Piz
Additionally, as inspired by the phenomenon demonstrated in Figure [(d] we merely select those segments outside the preference buffer's 's distribution and store them in the preference buffer, implies that the volume of data in the in-distribution region will not undergo substantial expansion. As a result, the evaluation capability of the trained reward model in the in-distribution region may become less reliable due to the lack of sufficient data in this area.
Taking all these factors into account, we propose the mixture distribution query method. This method aims to actively select -ut-of-distribution data to increase the preference buffer coverage, while also selecting some in-distribution data for query. This method not only proactively increases the

---

coverage of the preference buffer but also boosts the volume of in in-distribution data, thereby ensuring the evaluation capability of the reward model in the in-distribution region.
Specifically, for all r E D we can express the degree of a segment of trajectory being out of the preference buffer distribution according to (q.(5), where Dr represents the data be queried. A higher value indicates that the data is more likely to be in-distribution.
(T(T)
Σ (,,a)ET
Ma(s,a).
|
(
The size of D is not large, typically (D (D). especially when combined with the policy-aligned
query technique proposed in QPA [21]. the quantity of D is further reduced. Under this premise. we can redistribute the sampling probability for r E DDP.
As shown in 6). designed to the degree in-distribution and out-of-distribune sampling according to the degree of in-distribution and out-of out-of-distribution We mixture ratio E [0, 1] to control the proportion of samples drawn from each distribution. larger x indicates a higher proportion of samples drawn from pout() (-).
probability density functions pin(-) and pout (.) according
respectively representing the probability of
1 MMM(TT)
MT(T)
(TT(T) = ErEDEE M
pout (T) EVEDO Ma(T')
(6)
It's worth noting that, mentioned in the preceding paragraph, we need calculate Ms(s, a) for each newly encountered (8, a) when using proximal-policy extension method. Therefore, by maintaining (Ma(s, ((8, a) E (DD) and updating it regularly, we can avoid recalculating Ma a)
when using the mixture distribution query, thus saving significant amount of overhead. The specific procedure illustrated in Algorithm [
Proximal Policy Exploration Algorithm In summary, the proximal-policy extension method and the mixture distribution query method complement each other. The use the mixture distribution query method mitigate potential issues that might arise from solely using the mexximal-polion extension method. The combination of these two methods forms our PPE algorithm, with the algorithmic process detailed in Algorithm 2
Algorithm Proximal Policy Exploration
Input: Query frequency K, feedback size per query b, mixture ratio x and Morse buffer Dm Unsupervised pretraining
for each iteration do
a~ (-(s)
// Sample action via prozimal-pole eatension, Eq. // Store the 00D metric of transition
{s, (o(s, a}} Dm
Store new transition (s, a) if iteration% = then
{ (0.-2 (1=x)b (T)
// Minture distribution query, Algorithm
[ 1=b+1
pout(r)
Query for preference { Store preference DD + DP U ( .11
10
for each gradient step do
DD Training the reward model
B- (70, y
// Sample minibatch preference data
12
13
Optimize loss of Md in Eq. w.rlt. o using
14
Relabel the reward in D
15
Relabel the OOD metric via Mo(-) for (s,a) E DeP
16
for each gradient step do Optimize Ty via SAC method
17
In Algorithm Dm {(x, a, Ma(s, a)) ) E DeP.. The parts highlighted in brown represent the additions made by algorithm compared to the basic algorithm framework.
7

---

Improvements in different algorithms typically focus on various stages: the data storage stage (Line 5, QPA (2I) the data selection for querying stage (Line QPA, B-Pref 21 [2] the reward model update stage (Line 12, SURF, PEBBLE 14 13) and the agent update stage (Line 17, RUNE, QPA 15 21). Our approach, however, primarily enhances the data exploration stage, offering the advantage of excellent compatibility with existing methods. Although we use the mixture distribution query method for data selection, it does not conflict with existing query methods. We can apply the mixture distribution query method as a post-processing step on the results of existing query methods to select suitable data for querying. This further demonstrates the compatibility of our approach.
In practical applications, PPE can be implemented as algorithmic plugin within an existing framework. This integration enhances the policy exploration process without requiring extensive modifications to the current framework.
Experiments
Our method, outlined in Section 3.2 is designed be orthogonal and highly compatible with existing strategies. Notably, our mixed distributed query technique does not interfere with the policy alignment query employed in the QPA method. This compatibility allows us seamlessly integrate PPE into the QPA algorithm for subsequent experiments. To simplify our discussion, we will directly refer to this integrated approach as PPE henceforth.
We conducted an evaluation of our method using the MetaWorld [32 and DMControl 33 bench-
marks. For comprehensive comparison, we selected several baselines, including PEBBLE [3] SURF [14,, RUNE [15], and the previous state-of-the-art method, QPA [21]. We conducted experi- ments using five different random seeds. For each task, we record the mean and standard deviation of the final 10 evaluation episodes aggregated across all random seeds. For a complete understanding of our experimental details, please refer to Appendix Moreover, we also made use of the official code repositories provided in the papers of the corresponding baseline algorithms for fair comparison.
4.1 Benchmark Task Performance
Table I: Performance comparison across benchmark tasks
Task
PEBBLE
SURF
RUNE
QPA
PPE
Walker-walk
453.43 = 159.43
661.01 = 91.72
414.62 182.16
796.08 147.94
908.09 + 55.30
Walker-run
188.21 + 79.86
237.65 = 116.85
251.48 = 104,98
416.52 = 222.01
520.18 + 101.72
Quadruped-walk
364.46: 138.98
554.20= 279.01
506.89 266.33
621.51 259.35
660.71 +: 179.67
Quadruped-run
319.35 = 118.78
285.51 114.13
231.88 59.57
386.90 118.41
431.32 115.87
Cheetab-run
545.77 + 130.00
556.78 + 59.323
508.60 186.06
578.89
133.14
644.91 + 30.37
Humanoid-stame
249.48 218.74
306.60=: 167.76
279.45 220.00
453.65: 23.75
577.12 + 30.93
Drawer-open
20.00 + 44.72
40.09 + 54.89
67.20 + 44.21
40.09 54.89
79.61 + 44.55
Sweep-into
75.45 43.11
57.81 53.09
99.62 + 0.56
80.67 27.00
96.47 8.47
Hammer
41.31 53.57
85.23 + 26.18
91.86 + 17.77
78.75 44.04
96.27 5.19
Door-open
72.42 +: 35.82
91.47 = 11.04
98.23:11.92
100.00 + 0.00
100.00 0.00
Door-unlock
99.65 0.55
99.60 0.55
100.00 + 0.00
98.43=1.14
100.00 0.00
Button-press
76.69
78.81 14.38
88.66 +7.13
29.28
5.43
Window-open
87.14 17.08
87.29 266.9
96.77=4.14
90.15+: 20.09
100.00 0.00
We evaluate PPE on six locomotion tasks from DMControl: Walker-walk, Walker-run, Cheetah-run, Humanoid-stame Quadruped-wand and Quadruped-ruu
Locomotion Tasks in DMControl Suite:
Robotic Manipulation Tasks in Meta We also test on challenging MetaWorld manipulation tasks: Hammer, tweep-into. Drawer-open, Door-open, Door-unlock, Button-press, and Window-ope open.
Following previous work [10 12 15 21,, we use the ground truth success rate as the evaluation
metric.
Unless otherwise noted, we fix the mixture ratio (N = 0.5) and the KI. constraint = 0.01) for all tasks and environments, with no per-task tuning.

---

As shown in Table PPE consistently achieves superior performance across all benchmark tasks, where bold values indicate statistically significant improvement over the second-best method This indicates that PPE can more effectively select and utilize feedback under limited query budgets. This supports our claim that expanding the coverage the preference buffer improves the reward model's evaluation capability and leads to more reliable policy updates.
Furthermore, our results show that PPE greatly improves feedback efficiency across complex tasks. While RUNE [5] underperforms on DMControl tasks, it achieves the second-best performance after PPE on Meta World tasks. We also find that the performance variance of PbRL algorithms is higher in MetaWorke than in DMControl, consistent with previous studies [2
14 21 15
4.2 Ablation Study
To further investigate the impact of each component in PPE, we conducted additional ablation experiments on the Walker-walk task. These experiments aim to provide empirical evidence for the parameter selection of PPE.
To assess the roles of the proximal-policy extension method (EXT) and the mixture distribution query method (MDQ) within PPE, we incrementally applied these methods to the backbone algorithm QPA. As shown in Figure ((a) using either EXT or MDQ alone does not result significant improvements. As described in Section 3.2 EXT and MDQ complement each other. Using only EXT increases the amount of out-of-preference buffer distribution data the replay buffer without directly enhancing the coverage of the preference buffer. Conversely, using only MDQ fails to introduce sufficient out- of- of-preference buffer distribution data into the preference buffer due to the lack of active exploration, thus not effectively strengthening the reward model. Therefore, the superior performance of PPE arises from the mutual compensation of the shortcomings EXT and MDQ.
1000
1000
800
500
Edtcodo Recurn
500
tello
400
400
0.6
400
Comstrint
CPA
0.3
0.0001
QPA /M KPEEEEETtbb.... (TOO))
0.5
0.01
CPA-+PPEEETII WDQ)
200
0.7
1.0
OPA W MD
100
100.0
0.5
0.0
0.3
0.1
0.5
Step
Step
(x30%)
(a) Impact of each technique in PPE
(b) Impact of Mixture Ratio xc
(c) Impact of boundaries to explore
Figure 2: Various ablation studies on the Walker-walk task. The dashed black line indicates the final feedback collection step. Shaded areas indicate standard deviation; the solid line shows the mean.
Next, we examine the effect of the mixture ratio MDQ with complete PPE, which determines the balance of in-distribution and duitt--distribution data in the preference buffer. As Figure shows, optimal performance is achieved at k 0.5. This result confirms that indiscriminate addition of nut-of-distribution data to the preference buffer can overextend distribution boundaries, undermining the reward model's effectiveness as it relies on both the coverage of the preference buffer and balanced data density across trajectory regions. Furthermore, exclusive in-distribution data sampling could hinder the reward model's adaptability to new distributions trajectories after policy updates.
Additionally, we conducted ablation experiments on the KL constraint in Eq. This parameter represents the exploration boundary of EXT for out-of-distribution data. As discussed in Appendix D theoretically, using EXT requires that the behavior policy and target policy do not differ significantly. This implies that e too large, performance cannot be theoretically guaranteed. Conversely, is too small, EXT loses its exploratory significance. Experimental results, shown in Figure (ig) confirm this property: both excessively large and small values of negatively impact the results. Therefore, recommend setting to 0.01.

---

Related Work
Houman-in-the-low Reinforcement Learning: cuiman-in-the-lowe RL leverages human preferences.
typically collected via comparative judgments, to guide agent behavior [34
I0 35
The main
challenge lies in the high cost and efficiency of preference acquisition 13 15 Existing methods
range from online preference modeling [6] to recursive optimization
Efficient query selection is crucial for improving feedback efficiency in PbRL. Traditional approaches use entropy-based metrics, feature distances, and spatial sampling
Query Selection in PhRL:
37 38 Recent work highlights query-policy misalignment as a key limitation, where selected queries may not be the most informative for improving the current policy [21]. The Query-Police Alignment (QPA) method [21) addresses
strategies such as K-medoids and Poisson disk sampling [T8] 20
this by selecting queries expected to maximally impact the policy. Our proposed PPE method complements this perspective. While QPA focuses on the immediate utility of query for policy optimization, PPE aims to improve the long-term quality of the preference buffer by encouraging exploration of diverse, underrepresented state-action regions. Thus, PPE can be integrated with frameworks like QPA, where PPE first guides the agent to discover novel behaviors, from which QPA can then select the most policy-relevant queries.
Exploration in RL: Balancing exploration and exploitation is fundamental in RI. 222 In the context of PbRL, methods like RUNE [15] primarily drive exploration using disagreement among an ensemble of reward models. The rationale is that high disagreement signals regions of high uncertainty about the true reward function, which are valuable to query. However, this uncertainty- -bsee exploration can be inefficient, as it may repeatedly explore regions that are difficult to model but not necessarily helpful for discovering new, high-reward behaviors. In contrast, our approach focuses on maximizing preference buffer coverage. This encourages the agent to visit novel state-action regions that are underrepresented in the preference data, thereby improving the global quality and robustness of the learned reward model, rather than just focusing on areas of high model uncertainty. More broadly, RI exploration strategies include uncertainty-driven,, intrinsic motivation, and information-theore methods 39 40
43
6 Conclusion and Discussion
This paper highlights the critical role of preference buffer coverage in the evaluative accuracy of re- ward models. Our findings indicate that reward model's accuracy is the highest for trajectories within the preference buffer's 's distribution and significantly decreases for out-o--distributiio trajectories. We introduce the PPE algorithm, which actively expands the preference buffer coverage to enhance the reliability of the reward model, comprising two complementary components: the proximal-policy extension method and the mixture distribution query method. These components synergistically work to expand the preference buffer coverage while balancing the inclusion of both in-distribution and out-of-disse of-distribution data. PPE provides a more reliable reward model, thereby reducing the potential of misleading policy improvements. PPE has demonstrated substantial gains in feedback and sample efficiency through extensive evaluations on the DMControl and Meta Korld benchmarks. These results underscore the importance of actively expanding preference buffer coverage in PbRI. research.
In this study, our main focus is enhancing the reward model's ' quality by actively expanding the preference buffer's coverage. However, our current query method does not consider the variations in information between different pairs of agent behaviors. As we advance our research, we plan to investigate advanced methods to boost feedback efficiency. We believe that considering factors such as data similarity and clustering traits can further refine and optimize our query method.
10

---

References
[1] Mnih, V., K. Kavukcuoglu, D. Silver, et al. Human-level control through deep reinforcement learning, Nature, (7540):529-55) 2015. 2015.
[2] Silver, D., J. Schritwwissr,, K. Simonyan, et al. Mastering the game of go without human knowledge. Nature, (7676):35--35, 2017.
[3] Degrave, J., Felici, J. Buchli, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, (7897):414-419) 2022.
[4] Wu, J., L. Ouyang, D. M. Ziegler, al. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109, 10862, 2021.
[5] Meneilld-Menell, D., S. Milli, P. Abbeel, et al. Inverse reward design. Advances in Neural Information Processing Systems, 2017.
[6] Abel, D., W. Dabney, A. Harutyunyan, et al. On the expressivity of Markov reward. Advances in Neural Information Processing Systems, pages 7799-7812, 2021.
[7] Li, J., X. Hu, H. Xu, et al. Mind the gap: Offline policy optimization for imperfect rewards. arXiv preprint arXiv:2302.0166 2023.
[8] Sorg. J. D. The optimal reward problem: Designing effective reward for bounded agents. Ph.D. thesis, University of Michigan, 2011.
[9] Amodei, D., C. Olah, Steinhardt, et al. Concrete problems in Al safety. arXiv preprint 2016.
arXiv:
[10] Christiano, P. J. Leike, T. Brown, et al. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 2017.
[11] Ibarz, B., Leike, T. Pohlen, et al. Reward learning from human preferences and demonstrations in Atari. Advances in Neural Information Processing Systems, 2018.
[12] Lee, K., L. Smith, A. Dragan, et al. B-pref: Benchmarking preference-based reinforcement learning. arXiv preprint arXiv:2111.03024 2021.
[13] Lee, K., L. Smith, Abbeel. Pebble: Feedhack-efficien interactive reinforcement learning via relabeling experience and unsupervised pre-training, arXiv preprint arXiv:2106.05091, 2021.
[14] Park, J., Seo, J. Shin, et al. Surf: Semi-supervised reward learning with data augmentation for feedhack-efficient preference-batee reinforcement learning. arXiv preprint arXiv:2203. 10050, 2022.
[15) Liang, X., K. Shu, K. Lee, et al. Reward uncertainty for exploration in preference-based reinforcement learning. arXiv preprint arXiv:2205. /2401, 2022.
[16] Shin, D., A. D. Dragan, D. Brown. Benchmarks and algorithms for offline preference-based reward learning. arXiv preprint arXiv:2301.0139) 2023.
[17] Tien, J., J. Z.-Y. He, Z. Erickson, et al. Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint vrXiv:2204.06601 2022.
[18] Biyik, E.,, D. Sadigh. Batch active preference-based learning of reward functions. In Conference on Robot Learning, pages 519-528. 2018.
[19] Sadigh, D., A. D. Dragan, S. Sastry, et al. Active preference-based learning of reward functions. In Robotics: Science and Systems XIII, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA, July 12-16. 2017.
[20] Biylk, E., N. Huynh, M. J. Kochenderfer, al. Active preference-based gaussian process regression for reward learning. arXiv preprint arXiv:2005.02575 2020.
[21] Hu, X., J. Li, X. Zhan, et al. Query-policy misalignment preference-basee baseed reinforcement learning, arXiv preprint arXiv:2305.17400, 2023.

---

[22] Sutton, R. S. Reinforcement learning: An introduction. A Bradford Book, 2018.
[23] Leike, J., D. Krueger, T. Everitt, al. Scalable agent alignment via reward modeling: research direction. arXiv preprint orXiv:/811.0787/1 2018.
[24] Nguyen, A., J. Yosinski, J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, pages 427-436. 2015.
[25] Goodfellow, Shlens, C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
[26] Lakshminarayanan, B., A. Pritzel, C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 2017.
[27] Lee, K., K. Lee, H. Lee, et al. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems, 2018.
[28] Dherin, B., H. Hu, J. Ren, et al. Morse neural networks for uncertainty quantification. arXiv preprint arXiv:2307.00667, 2023.
[29] Srinivasan, P., Knottenbelt. Offline reinforcement learning with behavioral supervisor tuning. arXiv preprint 20X3:22400.13399,20000
[30] Seeger, M. Gaussian processes for machine learning. International Journal of Neural Systems, (02):69-106, 2004.
[31] Amari, S.-i. Information Geometry and Its Applications. Springer, 2016.
[32] Yu, T., D. Quillen, Z. He, al. Meta-world: A benchmark and evaluation for multi-task task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100. 2020.
[33] Tassa, Y., Y. Doron, A. Muldal, et al. Deepmind control suite. arXiv preprint arXiv: 180600 2018.
[34] Akrour, R., M. Schoenauer, M. Sebag. Preference-based policy learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, pages 12-27.
[35] Stiennon, N., L. Ouyang, J. Wu, et al. Learning to summarize with human feedback. Advances Neural Information Processing Systems, pages 3008-3021, 2020.
[36] Pilarski, P. M., M. R. Dawson, T. Degris, et al. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In 2011 IEEE International Conference on Rehabilitation Robotics, pages 1-7. IEEE, 2011.
[37] Rdusseeun, L., P. Kaufman. Clustering by means of medoids. In Proceedings of The Statistical Data Analysis Based on The LI Norm Conference. 1987.
[38] Bridson, R. Fast poisson disk sampling in arbitrary dimensions. SIGGRAPH Sketches, (1):1, 2007.
[39] Bellemare, M., S. Srinivasan, G. Ostrovski, et al. Unifying count-based exploration and intrinsic
motivation. Advances in Neural Information Processing Systems, 2016.
[40] Ostrovski, G., M. G. Bellemare, A. Oord, et Count-based exploration with neural density models. In International Conference on Machine Learning, pages 2721-2730. 2017.
[41] Hazan, H., S. Kakade, K. Singh, al. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pages 2681-2691. 2019.
[42] Bai, C., L. Wang, L. Han, et al. Principled exploration via optimistic bootstrapping and backward induction. In Proceedings of the 38th International Conference on Machine Learning, pages 577-587. 2021.
12

---

[43] Liu. J., Z. Wang, Y. Zheng, et al. Ovd-explorer: Optimism should not be the sole pursuit of exploration in noisy environments. In -hirty-Eighth AAAI Conference on Artificial Intelligence. pages 13954-13966 13962. 2024.
[44] Bradley, R. A., M. E. Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, (3/4):324-34) 1952.
[45] Basu, S., S. Prasad. A connection between cut locus, thom space and morse-bott functions. Algebraic & Geometric Topology, (9):4185-4233 2023.
[46] Haarnoja, T., A. Zhou, K. Hartikainen, et al. Soft actor-critic algorithms and applications. arXiv preprint 1rXiv:1812.05905, 2018.
[47] Burda, Y., H. Edwards, A. J. Storkey, et al. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019.
13

---

A The Process of Reward Model Training in PbRL
Using a preference dataset D the reward model fy learns to assign higher proxy returns Gy. E. fo (st, (t) to preferred trajectories. Employing the Bradley-Terry model [44), the probability that one trajectory is preferred over another is computed as:
exp (E (s). (())))))))))))))))))))))))))..) EiE(O.)) exp (Ee (s af))
Pa(T +9)
The probability estimate Ps is used to minimize the cross-entropy between the predicted and true preference labels: Lce o [R, = (r" > 7²)3 Po(" > 7²) 1(22 (r' +7)) log (o > 79]] (8)
After optimizing the reward function Fie from human preferences, PbRL algorithms enable training of RL agents with standard RL algorithms, treating the proxy rewards from Fgg. as if they were ground truth rewards from the environment.
B Information About Morse Neural Network
Definition B.1 (Morse Kernel), A Morse Kernel is a positive definite kernel K. When applied in space Z = Rt the kernel K (221,2 x2) takes values in the interval [0, 1] and satisfies K K(21 (21, 22) = 1 if and only if 22.
All kernels of the form ((21, 22) = e D(x where ((::)) is a divergence [31, are considered Morse Kernels. this study, utilize the Radial Basis Function (RBF) Kernel,
KnBr(21, 22)
where À is a scale parameter of the kernel [30
Consider a neural network that maps from feature space X to latent space Z via function fø : X + Z. with parameters ø. Here, X e Rd and Z E R A Morse Kernel can be used to impose structure on the latent space.
Definition B.2 (Morse Neural Network). A Morse neural network is defined as function fo : X + Z combined with a Morse Kernel ((2,t) t). where C a target chosen as a hyperparameter the model. The Morse neural network is expressed as Ma(r) = - K (z),
According Definition B.1
Ma(x) takes values in the interval [0. 1]. When Ma(x) 0, corresponds to a mode that aligns with the level set of the submanifold of the Morse neural network. Additionally, 1- Ma (z) represents the certainty that the sample is from the training dataset, making (1( measure of the epistemic uncertainty of x.
The function log[1 (z)) quantifies a squared distance, d(.,)), between fa( and the nearest
mode in the latent space at m.
d(=) min d(z, m),
(10)
MEM
where M the set of all modes. This provides information about the topology of the submanifold and satisfies the Morse-Bol non-degeneracy condition 445
The Morse neural network exhibits the following properties:
1. (z(x) E [0, 1]:
2. (x(x) at mode submanifolds;
3. log M& (x)) O represents a squared distance that satisfies the Morse-Bott Bott non- degeneracy condition on the mode submanifolds;
4. Since Mx(x)) is an exponentiated squared distance, the function is distance-aware, meaning that as fx( + t, [1 Ms +.
14

---

C Derivation of The Loss Function For Morse Neural Network In PbRL
We achieve the measurement of whether the current data outside the distribution of DP using the Morse Neural Network by minimizing the KL divergence D&L (P a)). The detailed derivation process as follows:
(s)
]
min 8,a~DP
E
log
IE
1- Mo( - DD' 1-Mo(s,a)-DD (sp as)
1-Mals,
[A JEEd
min
E
log [1-M
[] [1-Mo(s,a)]
spanDr
(1))
min
Σ
log (ogrk(((((s.)..))) M
E
KRBF
N
8,0~Dr Dv
ay a---ooilom((A))
min
Σ
X2 (1)
all
1
Σ
exp
llfo(s;au)
M au ~Unifor((A)
8,0
D Proof of Propostion 3.1
Consider the formula for the KL divergence between two high-dimensional Gaussian distributions:
E), Er)) 2 [(m (()--r)) "T) (n- μT) log det(((E)))) + tr(E7')) n]
(12)
When DEL (NL(N(, E), (JT,, Er)) S é is employed as a constraint, the solution to the optimization problem argmar (a [Ma (s. is typically achieved through iterative means. However,
a.E
considering objective for the calculated μ. to more effectively explore data from the out- of-preference buffer distribution within the proximal policy region, and the real-time requirement for problem-solving with each agent-environment interaction, propose more efficient closed- form approximation to the original problem by appropriately tightening the constraint, as shown in Proposition 3.1
We introducing V Vr, and the tightened constraint can be expressed as:
DRi(N N Er)) e.
2 (u - T (μ pT) - log det, + n) Se.
(13)
- (Iu - uT) > (μ uT) S e.
Substituting this into Eq. 3) derive simplified optimization problem:
max
E
[M (8, a)).
a~ (p.ET)
(14)
s.t.( pT) ) 2 (μ PT) S 2e.
To address the problem in Eq. 14 we construct the following Lagrangian function:
" L = Mo -(( (N HT) 2e).
(15)
Deriving with respect to a yields:
V,L= VoMo -EET (m WT).
(16)
Setting V-L= =0 0, we find:
1 Er Va == pT Eor
(17)
15

---

By applying the KKT conditions, we deduce:
(M pr) ETT (μ
pr)) 2c 0.
(18)
8>
Further, via plugging Eq. 17 in Eq.
18
we can solve to obtain:
2 Σt V MoMa 0)
la=p))
ET' ET VaMo(s,,))=zn) =2e, E>
=² VaMa(s, )) Σr VaMMo(. a== E>0.
(19)
2e
- VaMols, 8) Er I Mo aols,
2E
Through (9. we find that & is a function of However, Eq 17 is a differential equation, which challenging to solve directly for μ. Therefore, we perform Taylor expansion [V. (s, a)la=pi
V. V.Mo(s, a) s VoMa(s,,) (aasprr + Va Mo a)
(u - pT).
(20)
a=pr
This implies that when j sufficiently close to pr,, we can approximate:
VaMo(s,a)la=pp
(21)
Since our goal is to increase the density of proximal policy data in the preference buffer, thereby enhancing the reward model's evaluation capability under the current policy distribution, this approxi- mation does not conflict with our objective and is indeed very fitting.
Thus, further, we can deduce:
V2e V. a)la=ur
p. = PT +
(22)
VaMo(s a) |?= En V. Mo(s, a))a=rr
Therefore, the exploration behavior policy N N(EE, EE) can be expressed as
V2e Mn V Mo a)la=rr
μE PT Mo a) I) pr En V Mo a)la=ur
and ΣE Σr.
(23)
E Additional Experiments
E.1
The p-values for the Welch's T-test
Table 2: Welch's t-test p-values for performance comparisons between PPE and other baselines throughout all training evaluations
Task Wlgorithm
PEBBLE
SURF
RUNE
QPA
Walker-walk
0.001
< 0.001
< 0.001
0.001
Walker-run
0.001
< 0.001
< 0.001
0.001
Quadruped-walk
0.001
< 0.001
< 0.001
0.2977
Quadruped-run
0.001
< 0.001
< 0.001
< 0.001
Cheetah-run
0.001
0.001
< 0.001
0.001
Humanoid stand
0.001
0.001
< 0.001
< 0.001
Drawer-open
0.001
0.0412
0.1675
0.0078
Hammer
0.001
< 0.001
0.3041
0.0160
Sweep-into
0.001
< 0.001
0.0184
< 0.001
16

---

To facilitate the intuitive demonstration of the performance advantages of the PPE algorithm in our main text experiments, we have also tabulated the p-values of the Welch's T-test comparing the performance of the PPE algorithm with that of other baselines in all evaluations during training in 2
For most tasks, the algorithms PEBBLE, SURF, RUNE, and QPA show significant differences from PPE (p-value 0.05), indicating that these algorithms perform significantly differently from PPE on these tasks.
In three cases, the p- p-value is greater than 0.05, such for RUNE in the Drawer-open task, RUNE in the Hammer task, and QPA the Quadruped-wan task. Although there is no significant difference, PPE remains one of the best methods.
E.2 Exploration Methods Across Different Backbones
we applied PPE and RUNE to QPA and PEBBLE, respectively. This approach not only verifies the compatibility of PPE but also highlights the performance differences of various exploration methods across different backbones.
As shown in Tables 3 and 4
Table 3: The Performance of Different Exploration Methods on PEBBLE
Task Allgritth
PEBBLE
PEBBLE+RUNE
PEBBLE+PPE
Walker-walk
453.43 + 159.43
414.62 1 182.16
499.73 1 82.75
Walker-run
188.21 1: 79.86
251.48 1 104.98
257.64 1 58.59
Quadruped-walk
364.46 138.98
506.89 + 266.33
451.06 + 223.27
Quadruped-run
319.35 = 118.78
231.88 + 59.57
373.09 149.10
Cheetah-run
545.77 +: 130.00
508.60 + 186.06
569.54 + 84.27
Humanoid-stand
249.48-: 218.74
279.45 220.00
357.13 + 76.15
Table 4: The Performance of Different Exploration Methods on QPA
Task Wlgorithm
QPA
QPA+RUNE
QPA+PPE
Walker-walk
796.08 + 147.94
704.39 + 133.45
908.09 - 55.30
Walker-run
416.52 222.01
429.66 + 173.62
520.18 + 101.72
Quadruped-walk
621.51::259.35
593.61 295.84
660.07 + 175.58
Quadruped-ne run
386.90 118.41
367.71 108.01
433.42 + 116.58
Cheetah-run
578.89 133.14
689.52 + 49.39
644.91 + 30.37
Humanoid-stand
453.65 + 23.75
419.74+: 27.38
577.12 2 + 30.93
E.3
Ablation Study on K
Table 5: Impact of Mixture Ratio
Walker-walk performance with 100 feedback instances
Episode Return
Episode Return
Episode Return
722.33 1 256.97
0.4
756.41 1: 215.27
696.62 1: 243.53
0.1
795.26 1 174.23
0.5
908.09 + 55.30
0.9
744.50 1: 173.46
0.2
688.53 + 212.85
0.6
714.03 + 230.37
1.0
616.22 + 106.39
0.3 710.22 + 187.74
0.7
834.91 + 103.28
Under the Walker-walk experiment setting with 100 feedback instances, we investigated the impact of the mixture ratio is on the experimental results, as shown Table 5 Based on these results, we set the mixture ratio to 0.5 for all subsequent experiments.
17

---

E.4
Ablation Study on the Various Components of PPE
We denote the proximal-poile policy extension method as EXT and the mixture distribution query method as MDQ. The specific details are recorded in T
Table 6: Performance Impact of PPE Components on Walker- Walk Task (100 Feedback Instances)
Algorithm
Episode Return
Algorithm
Episode Return
QPA
796.08 147.94
QPA+PPEIEXPP.MMDDOWW
908.09 - 55.30
OPA-PPELEXPP..MDO:wvb
689.33 194.50
QPA+PPEGGXPP./MMMDOO.w
685.03 + 346.27
E.5
Ablation Study on KL Constraint e
In Table we present the impact of different KL constraints on the performance
Table 7: Impact of various KL Constraint on Walker-walk performance (100 Feedback Instances)
KL Constraint
Episode Return
KL Constraint
Episode Return
le-4
806.67 = 137.
le-2
908.09 + 55.30
le-1
745.54 + 163.10
1e0
638.53 + 202. 73
le1
783.26 + 163.70
1e2
635.46 + 214.01
E.6
Ablation study on more environments
600
620
Eddoodo Rctum
RCC
1000
0.0
Constra t
200
UPA
0.3
0.0001
P KTowin #)
0.5
0.31
+ ND2
0.7
1.0
EXE NDQ.
1.0
100.0
Step
Enwironment (x110f)
Step
(a) Impact of each technique in PPE
(b) Impact of Mixture Ratio
(c) Impact of boundaries to explore
Figure 3: Various ablation studies the Walker-run task. The dashed black line indicates the final feedback collection step. Shaded areas indicate standard deviation; the solid line shows the mean.
000
80C
000
Fepaace Satum
futum
Sp0o0ce Rotur
400
Nisture
0.0
KL Constrant
200
OPA
230
0.1
0.5
200
0.01
0,0001
GPA IWWIMEEEXTTOO WD))
0.7
1.0
(MA+ NUG
10Q.0
Environment Step (K10P)
Etvironment (K-10P)
Etivino Stop (x300))
(a) Impact of each technique in PPT:
(b) Impact of Mixture Ratio
(c) Impact of boundaries to explore
Figure 4: Various ablation studies the Cheetah-run task. The dashed black line indicates the final feedback collection step. Shaded areas indicate standard deviation; the solid line shows the mean.
Due to space limitations in the main text, present additional ablation studies conducted on the Walker-run Cheetah-run environments in this section. The results of these experiments are shown in Figures 3 and 4 respectively.
18

---

The results from these additional environments corroborate the findings presented in Section 4.2 As shown Figures ((a) and (a) the full PPE method (MDQ+EXT) consistently outperforms variants using only partial components, as well as the baseline without PPE.
Furthermore, the hyperparameter sensitivity analyses reinforce choices. Figures 3(b) 4(b) (
and [(c)) that setting the mixture ratio = 0.5 and the KL constraint = 0.01 yiclds robustly superior performance. Although other parameter values occasionally achieve comparable results, this combination proves to be the most stable and effective across different tasks.
As discussed in Section 4.2, these hyperparameter choices well-grounded A moderate value for x is crucial for the reward model's effectiveness, as strikes a critical balance: it prevents the reward model from being undertrained on in-distribution trajectories, while also avoiding overfitting to out-of- distriibbbtoon data. Similarly, a moderate ensures meaningful exploration (EXT) without sacrificing theoretical performance guarantees. Finally, it is worth noting that do perform ablation study on the entropy coefficient the underlying SAC algorithm, as our implementation utilizes its automated tuning version [46).
E.7 Coverage Visualization
We collected 100 feedback instances during the learning process of the Walker walk task using the QPA and QPA+PPE methods. The state and action spaces of these (s, a) pairs were clustered into 10 and 20 groups, respectively, using KMeaas. We then used heatmaps illustrate how the coverage of the preference buffer changes as feedback increases.
QPA
QPA-PPE 20n
Figure 5: Distribution of actions in different discrete states after clustering. The horizontal axis represents the 20 clustered actions, and the vertical axis represents the 10 clustered states. The first and second rows show the changes in coverage of the preference buffer during training for the QPA and QPA+PPE methods, respectively.
200
GPA+FPE
Doverage
200
120
Figure 6: Changes in coverage of the 10x20 clustered (s, a) space for the preference buffer corre- sponding to the QPA and QPA+PPE methods under the same seed.
19

---

Figures 5 and 6 demonstrate that as the number of queries increases, the use of PQPA+PPE clearly enhances coverage compared to QPA.
E.8
Human Feedback Experiments
How to Obtain Genuine Human Preferences Online We achieve authentic interaction with humans in the process of obtaining human preferences through the code above. This involves presenting two sets of behavior segment videos to humans and requesting preference labels from them. The specific interaction interface is shown in Figure 7
Collecting Human Feedback
(aw_,,, s.,, aa, 1,2, physica_augt, plysics_aeg2)
franh,hnight, frank_width, channela phyaica_aagt [0,0] .shaps
Create video sriber
ansput_vidtl width frams_width width à The narged width ia tuicw the original.
output beight frune, be iight fpx
Save viden
sp-asrua (hhapp-=
for seg. index in range (phreccceec shapel [0]):
render the paira of segsenta and save the video
Create video writer using inagein
7o7otttt. rp4', fpe-tpe) Iserate over frazes.
for franco, franel zip(ptycice_eegll (oggte/_...g...eee)) phyesce,.oggleeg.tneee
liorizotallyy corbined_frarm
www.hatcc((frrrre frars!))
Vrite to the video file
uriter. wwtpapppedddtaee www.
Tabel ing Trow
provide labeling instruction query preferences
(halle(lba):nn):
prin: (*12"
print (*............
ing_iiddes)
prefereace:
0: segment is better
1:
better
True::
cbeck 1f it is 0/1/munber type preference
rational, labol ("Petferreero
number")
rationall_aabl. intt/rattooal__bbel) break
except:
print ("Krong label
Please 0/1/other number.
princ
buran, www.leleeo..eeeeeeee.o. indox] rational, Labol
labeling Falee
hard to judge
up Labole
10]
huzan, labels up. (bbucal labels, cancel, axisn0)
sa_t_i deleto(ea..... cancel, axisr0)
(el.te cancel, ax1s=0)
print query nusber:", (enthunan, labela))
return sa,t,1, sa.t.2, hunss_iabele resbape( 1,1)
Feedback numbor: 9
Preferance: nurber:
Figure 7: Through this interface, humans can provide preference labels for the agent's behavior.
20

---

Comparative Performance Analysis of QPA and PPE in Human Preference Experiments To empirically validate our algorithm's 's effectiveness, conducted human preference experiments with five volunteer annotators. Each volunteer was asked to provide preference labels based on videos of agents performing the cheetah-run task. Specifically, each annotator completed 100 preference annotations, organized into 10-label batches. After every 20,000 training steps, new batch of labels was collected. Notably, this experimental setting fully consistent with the preference labeling setup described in the paper for the cheetah-run task, as detailed in Table I0
As illustrated in Figure 8 which presents mean performance values (solid lines) with standard deviation ranges (shaded areas), our analysis reveals that PPE demonstrates superior performance compared to QPA. To facilitate comparative analysis, we have included demonstration videos of the final trained agents in the supplementary materials, showcasing the practical behavioral differences between PPE-trained and QPA-trained agents.
Our investigation further uncovered a critical distinction in annotation: the QPA-provided queries exhibited substantial similarity in trajectory samples (Figure (a)). This phenomenon may be attributed to QPA reducing the size of DDS, which consequently diminishes trajectory diversity in DDF a configuration that increases cognitive load and decision uncertainty for human annotators.
In contrast, PPE's active exploration mechanism systematically expands the policy pooccyyp sampling space, as evidenced in Figure [(b This strategic diversity enhancement in query selection (I) improves sample representativeness, and (2) reduces the cognitive burden on human annotators by presenting more distinguishable trajectory pairs. We identify this fundamental difference in query as a primary contributor to PPE's performance advantage this human preference experiments.
QPA
600
PPE
Episode Retum
400
0
0.0
0.2
0.4
0.6
0.8
1.0
Environmen: Step (x10*)
Figure 8: Performance comparison between PPE and QPA on the Cheetah- run task using real human preference feedback.
QPA
(b) PPE
Figure 9: Comparative visualization of trajectory pair selection during human preference queries. (a) QPA's homogeneous sampling results in human preference queries. (b) PPE's active exploration generates diverse, distinguishable samples.
About OOD Detection
F1
Why Morse Network?
Theoretical Analysis Our core investigation focuses on whether increasing the preference buffer coverage can optimize the performance of online PbRL.
21

---

This is supported by the motivation example in Section 3.1 the coverage visualization in Appendix E.7 and extensive experiments throughout the paper.
The Morse network in our work is only used to provide gradients to mÆ for exploring OOD data.
Compared to ensemble-based OOD detection mechanisms like RND 47 the Morse network offers
several advantages:
(I). The Morse network requires only single neural network, whereas RND requires multiple networks for OOD detection, thus reducing computational overhead.
(2). The output of the Morse network is within the range [0.1]. In contrast, the output of RND is derived from the difference between the neural network's output and a random network, and this output is not standardized. Specifically, the form of the Morse network used in this paper is:
min
1 N
Σ
x² 2
[[fo(s, a) all
1
Σ
exp (1oc(ssa))
(24)
M
Dr
Dr
If we were to use the RND form, it would be:
Σ [llfo(s.a) - frandom (s, a]]l
min N
(25)
8,0~DP
By setting the RND's random network frandom (s, a) a, matches the left part of the optimization objective of the Morse network used in our paper.
Therefore, the Morse network has the additional objective of moving away from out-of-distribution (s,a) and requires fewer neural networks compared to RND.
Experimental Analysis To strengthen the paper by including experimental results using other OOD detection mechanisms to demonstrate that PPE can be applied to other OOD detection methods, we have added a set of experimental results using RND, as shown below.
Table Comparison of PPE algorithm performance using different OOD detection methods.
Task
Morse Network
RND
drawer-open
79.61 + 44.55
79.52 + 44.53
Sweep-into
96.47 + 8.47
77.17 1 20.69
Hammer
96.27 5.19
77.41 16.71
F.2 Computational Cost
Discussion on fØ Firstly, In our study, we utilized a neural network with a 3x256 architecture to learn the function fo required for the Morse network, as described in (4).
Secondly, we do not rely on the specific outputs of the Morse network to determine whether data OOD. Instead, we only utilize the gradient V, Va,ss a) and use it basis for sampling data in the 'Mixture Distribution Query'. These applications do not demand high precision in the Morse network's outputs; they only require a relative distinction in magnitude between in-distribution and out-of-distribution data.
Lastly, Given that our dataset is not very large, especially when QPA is used the backbone with dataset size of only '10 x poissodeleength which does not impose significant stress on the neural network.
Considering computational costs, we only train the Morse network for an additional 200 iterations after completing after per query. It is noteworthy that in many tasks, QPA and SURF involve training the reward model thousands of times after per query. Therefore, our use of the Morse network effectively meets needs without incurring substantial additional computational overhead.
22

---

Experiments Results of Computational Cost We averaged the time required for QPA and QPA+PPE to train the reward model (and the Morse network) after five query phases on the walker_wald walk task, all conducted on the same machine.
Table 9; Average time comparison between QPA and QPA+PPE.
Method
Average Time (seconds)
QPA
45.29
QPA+PPE
50.42
While PPE does introduce additional computational overhead, training the Morse network, like the reward model, is only necessary after each query. The total number of queries varies by task. For instance, in the 'walker walk' task, we followed the QPA setup, requiring total of 100 preference feedbacks, with each query obtaining 10 preference feedbacks. Therefore, the overall training process does not significantly increase computational cost.
G More Details About The Motivating Example In Section 3.1
G.1 Meaning of Region
In Section 3.1 "region 1-9" refers to square regions depicted in Figure [(a] with the lower-left corner the origin. The grid is labeled from 0 to 9 both the horizontal and vertical axes, increasing from left to right and from bottom to top, respectively. For example, "region 3" denotes a grid area bounded by the segments from 0 to on both axes.
G.2 The Evaluation Region for Used in Figure 1(d)
The evaluation region used is the same as the training region. This figure intended to explore how varying the amount of preference feedback affects the performance of the reward model when both the evaluation and training regions are fixed.
H Implementation Details
H.1 Fundamental Process of PbRL
An overview of the components in typical PbRL setup can be provided as below:
a). Data collection
b). Data selection and preference labeling
c). Learning the reward model using preference labels (70,71, ,,,)p)
d). Optimizing Tr with the learned reward model via reinforcement learning methods
H.2 bouu Buffers
H.2.1 Functions of Various Buffers
DaP stores potential segments that might be selected during the "data selection and preference labeling" phase. Specifically, when selecting (7 7") for preference labeling, these segments are drawn from DcP
D the replay buffer, a fundamental concept in reinforcement learning, storing (St, at, ft, 8(+1) instead of the ground truth rt is used during the policy optimization phase with the learned reward model.
DD stores preference feedbacks (7 7.. yp) for learning the reward model.
Dm stores additional one-dimensional data Ma (s, a) for each (s, a) DeP, as shown in Eq. 5 It is used compute to assess the OOD degree of T.
23

---

H.2.2 Memory Usage
D is essential for all off-policy reinforcement learning algorithms replay buffer.
DD and DD necessary for existing online PbRL. methods.
Dr only requires storing additional one-dimensional value Ma(s, a) for each (s. a) in
DoP, which a minor addition performed in Algorithm
line 4
Therefore, PPE does not require significantly more memory compared to previous online PhRL methods.
H.3 Origin of the Code for Baseline Algorithms
To ensure faimess in our experiments, we used the original source code provided by the authors of each baseline algorithm. Specifically, the sources are follows:
BUBBLE,
material
RUNE http://ppg/ggtuub.ooooo/orrrrrr/
B-Pref: https://gituub.ommrlrrreeeeere
The only modification we made was to unify logging format during training. We changed QPA's logging from using wandb to the storage format used by the B-Pref framework, which is also used by PEBBLE, SURF, and RUNE.
H.4 Human Involvement
In stage b, algorithms typically select (r0,7)) pairs, which then submitted for human preference labeling. In most PbRL. implementations, scripts are typically used to simulate human preference labeling. Our paper follows the same setup.
The Mixture Distribution Query used only in stage select, as shown Algorithm T These selected pairs are then submitted for human preference labeling (Algorithm 2 line 8). This is the only stage that requires human involvement.
This process is consistent with what is described in PEBBLE (Algorithm 2, line 11), QPA [5] (Algorithm 1, line 6), and RUNE (Algorithm 1, line 9).
H.S How Were Preferences Elicited?
We used the same approach as PEBBLE, SURF, RUNE, and QPA, utilizing the B-pref framework [6 to script access to the ground truth reward, thereby simulating human preference labels.
H.6
Reproducibility and Compute Resource Details
To ensure full reproducibility of experiments, we provide comprehensive details on the computing environment used throughout our work. These details address typical concerns regarding the type of compute workers, memory, storage, and execution times, outlined below:
Hardware Specifications: We conducted all experiments dedicated server featuring an Intel Xeon processor with 64 cores. The server equipped with 512GB of RAM and 10TB of storage, ensuring that the dataset loading and large-scale computations can be handled efficiently.
GPU Configuration: The experimental environment includes eight NVIDIA 1080ti GPUs, each with 11GB of VRAM. Each training task is assigned an individual GPU, although multiple tasks may share single GPU sinultaneously to optimize resource utilization. This configuration is essential for achieving the reported experimental throughput.

---

On average. each task requires approximately hours of training time. This metric provides clear indication the computational budget needed for individual experimental runs. Notably, besides the standard training procedures detailed in our paper, no additional compute was required, as preliminary or failed experiments did not necessitate extra resources beyond what is reported.
Compute Time:
By providing these explicit details, we aim to facilitate clear understanding of the computational resources required to reproduce our experiments and assess the scalability and feasibility of the approach on similar hardware setups.
H.7 Crowdsourcing and Human Subject Research Details
Overview In study, conducted experiments involving human subjects collect preference labels for agent behaviors. Volunteers were recruited to participate in these experiments, and uppropriate compensation was provided accordance with the NeurIPS Code of Ethics, ensuring that all participants received at least the minimum wage of the country in which the data was collected.
Instructions and Interaction Interface Participants were presented with pairs of behavior segment videos and asked to indicate their preference between the two. The full text the instructions provided to participants is included in the supplementary material. The interaction was facilitated through a custom interface, as illustrated Figure 7 This interface displayed two videos side by side, and participants
prompted to select the segment they felt better matched the task description of "controlling the robot to run like an animal," " or to indicate if neither segment was clearly preferable.
Figure 7 shows the actual interface used for collecting human feedback. Participants were instructed as follows:
You will be shown two short video segments of agent behavior. Please watch both videos and indicate which one you prefer by entering '0' you prefer the left video, '1' if you prefer the right video, or another number if you cannot decide. If you cannot make a clear choice, the pair will be excluded from the analysis.
Compensation All participants were compensated for their time and effort. The compensation rate set to ensure compliance with ethical standards and local minimum wage regulations.
Ethical Considerations This study was conducted in accordance with the NeurIPS Code of Ethics. All participants provided informed consent prior to participation, and their data was anonymized to protect privacy.
H.8 Parameter for Tasks
Table 10: The hyperparameters of tasks
Frequency of feedback
Queries number per session
Hyper-parameter
Total feedback
Training Steps
Walker-walk
le2
2c4
lel
Se5
Walker-run
le2
2c4
lel
lc6
Cheetah-run
le2
2e4
lel
le6
Quadruped-walk
le3
3e4
le2
le6
Quadruped-run
le3
3e4
le2
le6
Humanoid-stame
le4
5e3
Sel
2e6
Drawer-open
4e3
5e3
2el
le6
Sweep-into
le4
5e3
5el
2e6
Hammer
le4
5e3
Sel
2c6
Door-open
3e3
5e3
3el
le6
Door-unlock
3c3
5c3
3el
le6
Button-press
2c3
5e3
25
le6
Window-open
4e2
5e3
lel
5e5
25

---

Determining the number of feedback instances for each task, the interval between queries, and the quantity of feedback per query can be quite challenging. We have summarized the experimental settings from the QPA [21] and SURF [14] papers in Table TO The experiments in our paper strictly adhere to the settings outlined in this table.
H.9 Parameter for Algorithms
Our method does not introduce many additional parameters, shown in Table 11 In this work, represents the KL divergence constraint between the behavior policy and the target policy Eq. (3). which determines the exploration boundary in our approach. The parameter À controls the sensitivity the mixture ratio that controls the proportion of samples drawn from each distribution. Additionally, we followed the parameter
of the Morse Neural Network. Lastly, K, mentioned Algorithm [
Table 11: The hyperparameters of PPE
Hyper-parameter
Value
Hyper-parameter
Value
Parameter OOD detection
KL constraint
le-2
Mixture ratio K
0.5
settings from the baseline papers 21 detailed in Tables [5
12
1
15
13
The specific parameter configurations are
Table 12: The hyperparameters SAC
Hyper-parameter
Value
Hyper-parameter
Value
Discount
0.99
Init temperature
0.1
Alpha learning rate Critic target update freq
le-4
Batch size
1024
Critic EMA
5e-3
5e44
Walker_walke
5e-4 Walker_walle
Cheetah_ run,
Cheetah_run run,
Critic learning rate
Actor learning rate
Walker run)
Walker run)
le-4 (Other tasks)
le-4 (Other tasks)
Critic hidden dim Critic hidden layer
1024
Actor hidden dim Actor hidden layer
1024
Critic activation function Optimizer
ReLU
Actor activation function
ReLU
Adam
Table 13: The hyperparameters of QPA
Hyper-parameter
Value
Hyper-parameter
Value
Size policy-aligned
10
Data augmentation ratio
20
buffer
Hybrid experience replay
Min/Max length of subsampled snippets
0.5
[35, 45]
sample ratio
Table 14: The hyperparameters of SURF
Happer-parameter
Value
Hyper-parameter
Value
Unlabeled batch ratio
Threshold
0.99
Min/Max length of cropped segment
Loss weight
[45, 55]
Segment length before cropping
60
Table 15: The hyperparameters of PEBBLE
Hyper-parameter
Value
Hyper-parameter
Value
Unsupervised pre-training steps
Length of segment
50
9000
Size of query selection buffer
100
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope, clearly highlighting the challenge of limited preference buffer coverage in PbRL. and presenting PPE as principled solution with demonstrated empirical benefits. The stated contributions are consistent with the methods and results described.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper discusses limitations in the "Conclusion and Discussion" section. Specificallyy, authors acknowledge that current query method does not account for the variations in information between different pairs of agent behaviors. They also mention that their approach could be further improved by considering factors such data similarity and clustering traits to refine and optimize the query method. This reflects an awareness of the limitations in feedback efficiency and the potential for further enhancement of this method.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
27

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
The paper provides detailed theoretical derivations and proofs for its main results. Specifically, in the Appendix D Proof of Proposition 3.1 the paper clearly states the optimization problem, the assumptions (such as the use Gaussian distributions and the tightening of the KL divergence constraint), and provides step-by-step derivation leading to closed-form solution. The derivation includes the construction of the Lagrangian, application of the KKT conditions, and a Taylor expansion to justify the approximation used. All key formulas are numbered and cross-referenced. The assumptions (c.g.. setting Σ Er for tractability and real-time requirements) the context of the proof. Therefore, the paper meets the requirements for providing a full set of assumptions and complete and correct proof for its theoretical results.
Justification:
explicitly stated and discussed in
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides all necessary information enable reproducibility of the main experimental results. Specifically, Algorithm presents the detailed algorithmic imple- mentation, AppendixH] describes the experimental setup and implementation details, and Appendix E.8 explains how real human preference feedback is incorporated. These sections together ensure that readers have access to the full methodology, parameter settings, and procedures required to replicate the experiments and verify the main claims and conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (c.g., in the case
28

---

of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code will be made publicly available after the paper is accepted, ensuring open access for reproducibility. Additionally, the supplemental material includes videos of agent behaviors learned from real human preference labels provided by volunteers, which as qualitative evidence of the method's effectiveness.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ca
wwwlic/guides// SotaassSoooSSuubsssiooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
CCodeSumiiss issi ssionnoiicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
29

---

Justification: The paper provides a clear description of the experimental setup in the main text (Section 4), including the benchmarks used (MrldWorld and DMControl), the tasks eval- uated, the baselines for comparison, the use of five random seeds, and the evaluation metrics (mean and standard deviation of the final 10 evaluation episodes). For further transparency and reproducibility, the paper refers readers to Appendix H which contains comprehensive details on experimental settings, such as hyperparameters, training procedures, and imple- mentation specifics. This ensures that all necessary information to understand and interpret the experimental results available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate
information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports the mean and standard deviation of the experimental results across five random seeds. Additionally, Appendix [E] presents the p-values from Welch's s T-test comparing the performance of the PPE algorithm with other baselines, offering formal assessment of statistical significance. These practices ensure that the reported results are statistically sound and that the main claims are supported by appropriate statistical analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Due to space limitations in the main text, detailed information regarding the comprehensive descriptions of the hardware environment will be provided in the appendix. This ensures that all necessary details are available to reproduce the experiments fully, while keeping the main paper concise.
30

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research presented in this paper adheres fully the NeurIPS Code of Ethics. The work involves algorithmic development and simulation experiments on standard continuous control benchmarks. There no foreseeable risks of harm, privacy violations, or misuse of the technology, and does not engage in any discriminatory or unethical practices. All data and code used comply with relevant ethical standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
This paper discusses the potential positive societal impacts of improving the efficiency and reliability of preference-based reinforcement learning (PbRL). Such advance- ments can enable the development of more effective and human-aligned Al systems across various domains, including robotics, personalized assistance, and autonomous systems. By reducing the human effort required design reward functions and enhancing policy learning from human preferences, this work contributes to making AI systems more adaptable and user-friendly.
Justification:
At the same time, we acknowledge potential negative societal impacts. For instance, preference-based RL systems are deployed without proper oversight, they may learn biased or unsafe behaviors due to flawed or insufficient human feedback. Moreover, misuse of these systems could result in unintended consequences, especially in safety-critical applications. We recognize the importance of managing preference buffer coverage and ensuring reliable reward modeling as essential measures to mitigate such risks.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
31

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of models or datasets that pose high risk of misuse. The work focuses on reinforcement learning methods and experiments using standard benchmark environments without releasing pretrained models or scraped datasets. Therefore, no specific safeguards for responsible release are necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The paper uses existing assets such as benchmark environments and datasets for reinforcement learning experiments. All original creators and sources of these assets are properly cited in the paper. The versions and URLs of the assets used are clearly stated where applicable. Additionally, the licenses and terms of use for these assets have been reviewed and respected in accordance with their respective requirements.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
32

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce or release any new assets (such as datasets or models) the time of submission. Although the authors plan to release the relevant code with detailed documentation after the paper is accepted, currently no new assets are provided. Therefore, this question is not applicable.
Guidelines:
The answer NA means that the paper does release new assets. Researchers should communicate the details of the dataset/code/mmodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable), You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The paper reports that volunteers were recruited provide preference labels for the human preference experiments and that appropriate compensation was given. Further- more, the instructions provided to participants are included in the supplementary material. Therefore, the paper fulfills the requirements regarding documentation of instructions and compensation for human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Although the experiment involved volunteers providing preference labels, their participation was limited to passively watching videos without any intervention collection of sensitive personal data. The study posed minimal to no risk to participants, and no sensitive or potentially harmful procedures were involved. Therefore, formal Institutional Review Board (IRB) approval was not sought, as the research falls under minimal-risk observational studies exempt from such review according to common ethical guidelines.
33

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The algorithms and experiments in this work do not involve any large language models (LLMs), and the core method development does not rely on LLMs as important, original, non-standard components. However, the proposed method is general and can potentially be applied to the post-training of LLMSs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to LLM policy (https://neur))) h2tp:/ppp/../coooeeeeeee//////// for what should or should not be described.
34


Reviews:
[{"review_id": "PZqtzqGUaA", "reviewer": "Reviewer_Eg1k", "summary": "This paper addresses a critical challenge in Preference-Based Reinforcement Learning (PbRL): the limited generalization of reward models trained on preference data with insufficient coverage. The core contribution is a novel algorithm, Proximal Policy Exploration (PPE), designed to explicitly and actively expand the coverage of the preference buffer. PPE consists of two main components: 1) a \"proximal-policy extension\" method that uses a Morse Neural Network to detect out-of-distribution (OOD) states and guides the policy to explore these novel regions under a KL-divergence constraint, and 2) a \"mixture distribution query\" method that balances the sampling of newly discovered OOD data and familiar in-distribution data for human labeling. This dual approach aims to improve the reward model's global accuracy without sacrificing its stability, ultimately leading to better policy performance. The authors validate their method on a suite of continuous control benchmarks, demonstrating performance gains over existing state-of-the-art PbRL algorithms.", "questions": "See weakness", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strength\n\nWell-motivated:  Through an intuitive \"motivating example,\" the authors identify two core challenges in PbRL: 1) the reward model needs to explore out-of-distribution (OOD) regions to expand its cognitive boundaries and improve generalization, and 2) simultaneously, the model must continuously review in-distribution data to prevent performance degradation on known regions. The proposed PPE algorithm, with its two core components (proximal-policy extension and mixture distribution query), solves these two key problems.\n\nStrong Ablation Studies: The paper's ablation studies are very thorough and convincing. The results clearly show that removing either component of PPE leads to a significant drop in performance. This proves that both problems initially identified by the authors are crucial, which in turn validates the correctness of their core motivation and the completeness of their method's design.\n\nHigh Clarity: The paper's organization is clear, the writing is fluent. From the problem statement to the method's description and the experimental validation, the entire process is coherent.\n\n### Weaknesses\n\n1.  Scope of OOD Detection: The OOD detection mechanism in Equation 2 appears to primarily focus on whether the action `a` is novel for a given state `s`. It is unclear how the novelty of the state `s` itself is factored in. If the agent encounters a completely new state but performs a very familiar action, the current method might fail to identify this transition as an OOD event that warrants exploration.\n\n2.  Unclear Objective of the Loss Function: The paper interprets the loss function in Equation 2 as minimizing a KL divergence. This is confusing, as $M_\\phi$ is not defined as a probability distribution. The form of Equation 2 strongly resembles a contrastive learning objective, such as the InfoNCE loss [1]. The paper would be much clearer if the authors either elaborated on the connection between their loss function and the contrastive learning framework or provided a more detailed explanation of their KL divergence objective.\n\n3.  Applicability to Language Models: The paper's experiments are focused entirely on continuous control tasks. Given the immense potential of preference-based learning for aligning large language models (LLMs), a natural question is whether the \"coverage-driven\" approach proposed here could be extended to the language domain. Discussing how this method might be adapted to discrete and high-dimensional spaces like text generation would broaden the impact of this work.\n\n[1] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML 2021.", "was_revised": true, "final_justification": "Some concerns have been addressed. But I doubt the method can extend to LLMs. It introduces an additional network, making scaling harder for already large models, and perturbing the logit space risks excessive, unproductive exploration. In games or robotics, such inefficiency may be tolerable, but for LLMs it often produces meaningless outputs that offer no training benefit. As related works have not evaluated on LLMs, I consider this concern to be of lower priority. So I maintain the score to 4 (weak accept).", "rebuttal": "Dear Reviewer Eg1k,\n\nThank you for your thorough review and insightful feedback on our manuscript. We are grateful for your positive comments on the motivation, ablation studies, and clarity of our work. Your constructive criticisms are invaluable, and we believe that addressing them will significantly improve the quality of our paper.\n\nBelow, we address each of the weaknesses you identified in a point-by-point manner.\n\n---\n\n### **Response to Weaknesses**\n\n**1. Weakness: Scope of OOD Detection**\n\nThank you for this insightful question. We apologize if this aspect was not made sufficiently clear in the manuscript. We would like to clarify that our out-of-distribution (OOD) detection mechanism does, in fact, account for the novelty of the state `s`.\n\nThe core of our OOD detection is the perturbation model $\\hat{a} = f_{\\phi}(s, a)$, which is a neural network that takes both the current state `s` and action `a` as input. The OOD score is then derived from the difference between the perturbed action $\\hat{a}$ and the original action `a`, specifically $M_{\\phi}(s,a)=1-K_{RBF}(f_{\\phi}(s,a),a)$.\n\nBecause $f_{\\phi}$ is a function of `s`, its output is conditioned on the state. If the agent encounters a state `s` that is OOD with respect to the states present in the preference buffer $\\mathcal{D}^p$, the neural network $f_{\\phi}$ is expected to generalize poorly. Consequently, it will likely produce a significant perturbation, meaning $f_{\\phi}(s, a)$ will be far from `a`, even if the action `a` itself is a common one. This will result in a high OOD score $M_{\\phi}(s,a)$, correctly identifying the transition $(s, a)$ as novel and warranting exploration.\n\nIn our revision, we will add a more detailed explanation in Section 3.2 to explicitly state that the novelty of the state `s` is implicitly captured by the perturbation model $f_{\\phi}(s, a)$, thereby ensuring the OOD detection mechanism is sensitive to novel states as well as novel actions.\n\n---\n\n**2. Weakness: Unclear Objective of the Loss Function**\n\nThank you for this very sharp observation and for suggesting the connection to contrastive learning. We agree that our explanation was insufficient and could lead to confusion.\n\nThe formulation of our loss function in Equation 2 is directly adopted from the work on Morse Neural Networks by Dherin et al. (2023) . As stated in our paper, the objective is to minimize the KL divergence *between unnormalized measures*, i.e., $D_{KL}(\\mathcal{D}^{p}(s,a)||1-M_{\\phi}(s,a))$, based on principles from Information Geometry (Amari, 2016). This is distinct from the standard KL divergence between probability distributions, and we apologize for not making this critical distinction clear.\n\nYour insight regarding its resemblance to contrastive learning is excellent and provides a very helpful intuition. Indeed, the loss function operates on similar principles:\n* The first term, $\\frac{\\lambda^{2}}{2}||f_{\\phi}(s,a)-a||^{2}$, encourages the model to produce minimal perturbation for in-distribution pairs from $\\mathcal{D}^p$, effectively aligning the representation with the target (akin to positive pairs).\n* The second term, involving a sum over uniformly sampled actions $a_u$, pushes the model's output for these \"negative\" pairs away from their corresponding actions, promoting uniformity over the action space for a given state.\n\nThis \"alignment and uniformity\" characteristic is the hallmark of contrastive learning.\n---\n\n**3. Weakness: Applicability to Language Models**\n\n\nThank you for raising this excellent point. We agree that extending our coverage-driven approach to the alignment of Large Language Models (LLMs) is a highly promising and important future direction. While our current experiments are in continuous control, the underlying principles of our method can be adapted to discrete domains like language.\n\nHere is a potential path for adaptation:\n* **OOD Detection for Language:** The state `s` would be the context (prompt and previously generated text), and the action `a` would be the next token. Our OOD detection mechanism, based on Morse Neural Networks, could operate on the continuous embedding space of these `(context, token)` pairs. The model $f_{\\phi}$ would be trained to identify generated tokens that are semantically novel or unusual given the context, relative to the examples in the preference buffer.\n* **Exploration in Discrete Spaces:** We acknowledge that our proximal-policy extension method (Equation 4), which relies on the gradient $\\nabla_{a}M_{\\phi}(s,a)$, is not directly applicable to the discrete token space. However, this can be addressed in several ways:\n    1.  **Exploration in Logit Space:** Exploration could be performed by adding a calculated perturbation to the pre-softmax logits, which exist in a continuous space. The OOD score $M_{\\phi}$ could guide the direction of this perturbation.\n    2.  **Modulating Sampling Probabilities:** The OOD score $M_{\\phi}(s, a)$ for each potential next token `a` could be used to directly modulate the final sampling probability distribution. For instance, we could up-weight the probabilities of tokens that lead to unexplored, OOD regions, thereby encouraging the LLM to generate more diverse and novel text for subsequent preference labeling.\n\nWe are very optimistic about this direction. In the \"Conclusion and Discussion\" section of our revised manuscript [see Section 6], we will add a detailed discussion on these potential adaptations for applying PPE to LLMs, explicitly mentioning the challenges and proposing these concrete solutions.\n\n---\n\nOnce again, we thank you for your valuable and constructive feedback. We are confident that by incorporating these revisions, we can significantly strengthen our paper and address your concerns. We hope that our revised manuscript will meet the standards for acceptance.\n\nSincerely,\n\nThe Authors"}, {"review_id": "FEQvDBcRz0", "reviewer": "Reviewer_X7X4", "summary": "The paper proposes the proximal policy exploration scheme that balances explicit exploration of out-of-distribution and in-distribution trajectories via Morse network-based uncertainty estimation. The authors argue that generalization of reward models is limited due to insufficient preference buffer coverage, and demonstrate this in a motivating example. The use of the Morse network as an OOD detector is taken from existing work, and similarly the exploration scheme itself is derived from the TD3-BST approach (Srinivasan & knottenbelt, 2024). They showcase their approach PPE on a series of continuous control tasks (DMControl & MetaWorld) with simulated preference feedback, and provide additional results in the supplementary material.", "questions": "- How many participants were recruited in the human feedback experiments? How many queries were used? Can you provide additional information about the experiment? In theory, this would be a very insightful and important experiment, but a lot of detail is missing\n- How does PPE compare to \"standard\" SAC? I know that SAC baselines have been reported in previous work, but i would appreciate it being mentioned at least in the appendix for the environments and hyperparamter settings reported in this paper", "limitations": "Limitations are not sufficiently discussed, e.g.\n- choice of hyperparameters, stability\n- experiments should be extended to other environments and tasks beyond control benchmarks\n- larger scale investigation of human preference feedback", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- As mentioned, the exploration-exploitation trade-off remains the fundamental challenge in RL, and so i think this research is highly valuable and potentially high impact\n- The reported empirical results are impressive, outperforming competing methods\n- The appendices are very comprehensive, and provide ample detail. I particularly like the inclusion of a justification for using the Morse network\n- Ablation studies for the effect of different algorithmic components are highly appreciated (Section 4.2)\n- The proposed algorithm seems widely applicable if Morse networks are scalable, the authors also investigate the integration of PPE into other base algorithms\n\nWeaknesses:\n- Different contributions seem intermingled in the algorithm: There is the uncertainty estimation via Morse networks (which is not really evaluated except for using the OOD detection performance in isolation), the exploration mechanism, and the query selection. Therefore, i think proper ablation studies are important. However, i find them insufficient for now. Using just a single environment (especially since the results seem to have large error bars) limits their significance. I would appreciate an extension of the experiments in Figure 2a)\n- As the topic of exploration is very prevalent in RL, i think the related work should discuss existing approaches (including the mentioned baselines) in more detail.\n- For me, this paper misses certain basic ablations as a sanity check (e.g. varying the entropy bonus coefficient, which is fine-tuned for PPE)\n- The motivating example is not very convincing for me. The experiment itself is lacking detail (i.e. the network architecture, training parameters, etc. are not described), and the outcome is pretty straight-forward, i.e., fact that ML models do not automatically extrapolate is clearly established\n- Source code is not provided\n\nWhile i see some flaws right now, i think the overall algorithm and results are fair, and i think many of my concerns could be addressed as part of a revision.", "was_revised": true, "final_justification": "The presented algorithm has clear merits, and the paper is well-crafted overall. However, there remain some limitations in terms of evaluation, which makes it difficult to fully assess the effectiveness of the method.", "rebuttal": "Dear Reviewer X7X4,\n\nWe would like to express our sincere gratitude for your time and effort in providing such a detailed and constructive review of our manuscript. Your insightful comments and suggestions are invaluable to us, and they have helped us identify key areas for improvement. We are encouraged that you find our research on the exploration-exploitation trade-off to be valuable and potentially high-impact, and that you appreciate our empirical results and comprehensive appendices.\n\nWe have carefully considered all your feedback and provide a point-by-point response below. We believe that by addressing these points, we can significantly strengthen our paper.\n \nResponses to Weaknesses\n1. Regarding the sufficiency of ablation studies:\n\nWe agree with you that proper ablation studies are crucial for disentangling the contributions of the different components of our proposed algorithm. We appreciate you acknowledging the importance of the ablation study presented in Section 4.2.\n\nAs you noted, the current ablation in Figure 2(a) was conducted on the Walker-walk task. Our rationale was to use this as a representative environment to demonstrate the synergy between the proximal-policy extension method (EXT) and the mixture distribution query method (MDQ). The results clearly show that using either component in isolation does not lead to significant improvements, and their complementary nature is what drives the superior performance of the full PPE algorithm.\n\nWe take your point about the statistical significance and the desire for broader evidence. In the revised version of our manuscript, we will expand our ablation studies to include additional environments. This will provide more robust evidence for our claims and better illustrate the individual contributions of each component of PPE.\n\n2. Regarding the level of detail in the Related Work section:\n\nThank you for your suggestion to expand the discussion on existing exploration approaches. In Section 5, under \"Exploration in RL\", we do provide a brief overview, noting that prior PbRL work has focused on reward model disagreement for exploration, while our approach focuses on maximizing preference buffer coverage. We also situate our work within the broader context of RL exploration strategies like uncertainty-driven and information-theoretic methods.\n\nHowever, we agree that a more detailed comparison with the exploration strategies used in our baseline methods would strengthen the paper. \n\n3. Regarding missing basic ablations (e.g., entropy bonus):\n\nThis is an excellent point. To ensure a fair comparison in our main experiments, we utilized the official code repositories of the baseline algorithms and their recommended hyperparameter settings wherever possible. Our method, PPE, is integrated into the QPA framework, and the underlying RL algorithm is SAC. The hyperparameters for SAC were kept consistent with the baseline to isolate the effects of our proposed exploration and query methods.\n\nWhile we did not perform a specific ablation on the SAC entropy bonus coefficient, we did conduct sensitivity analyses on the key new hyperparameters introduced by PPE, namely the mixture ratio $\\kappa$ and the KL constraint $\\epsilon$. We agree that an ablation on the entropy coefficient would be informative. ]\n\n4. Regarding the motivating example:\n\nWe appreciate your critical feedback on the motivating example in Section 3.1. While we acknowledge that the general principle of neural networks struggling with out-of-distribution (OOD) data is well-established, our goal with this experiment was more specific to the PbRL context. We aimed to systematically demonstrate that this \"distributional mismatch\" is a critical issue where reward models trained on static buffers fail on OOD trajectories from policy exploration, and that \"such failures in policy-proximal regions directly misguide iterative policy updates\".\n\nThe key takeaways we wanted to highlight were: \n• The necessity of making preference buffer coverage an explicit optimization objective, which is not standard practice.\n• A non-trivial finding that a common method for uncertainty estimation, ensemble variance, fails to distinguish between in-distribution and OOD transitions in this setting, as shown in Figure 1(b).\n• The crucial insight from Figure 1(d) that simply exploring OOD data is not enough; a balance between in-distribution and OOD feedback is essential for maintaining the reward model’s accuracy.\n\nTo address your concern about the lack of detail, we will add the full experimental parameters for this example (e.g., network architecture, training details) to the appendix in our revision to improve clarity and reproducibility.\n\n5. Regarding the availability of source code:\n\nWe apologize for not providing the code with the initial submission. As stated in our checklist, we are fully committed to open science and will make the complete source code publicly available upon acceptance of the paper.\n\nResponses to Questions\n1. Regarding the details of the human feedback experiments:\n\nThank you for your interest in this aspect of our work. The primary quantitative results in our paper are based on experiments with a simulated oracle to ensure controlled and extensive comparisons, as is common practice in the field. However, we agree that validation with real human feedback is very important.\n\nAs mentioned in the checklist, we did conduct experiments with real human participants. We recruited volunteers who were compensated for their time to provide preference labels. The full instructions provided to these participants are included in the supplementary material, and Appendix E.7 details how this feedback was incorporated. The results are provided as qualitative evidence in the form of videos of learned agent behaviors. We acknowledge that quantitative details such as the number of participants and queries were not in the main text. We will add a detailed description of the human feedback study, including the number of participants, the number of queries collected per participant, and other relevant experimental details, to the appendix in the revised version.\n\n2. Regarding the comparison to a standard SAC baseline:\n\nThe goal is not to outperform SAC trained with the ground-truth reward, but rather to learn a policy that is as close as possible to that upper bound while using a minimal amount of preference feedback. Previous work in PbRL has established the performance of SAC with ground-truth rewards on these benchmarks. However, for completeness and to provide a clear performance ceiling for our experimental setting, we agree that including this baseline is beneficial. \n \nResponses to Limitations\n\nWe thank you for pushing us to expand our discussion on the limitations of our work. This is crucial for positioning our contributions accurately. The current discussion in Section 6 acknowledges that our query method could be improved by considering the information content of behavior pairs. We will expand this section in our revision to incorporate the excellent points you raised.\n \nOnce again, we thank you for your detailed and insightful review. We are confident that by incorporating your suggestions, we can substantially improve the quality and clarity of our paper.\n\nSincerely,\n\nThe Authors"}, {"review_id": "Ikbb9uvVYJ", "reviewer": "Reviewer_1gpz", "summary": "This paper introduces a novel shifts-aware reward learning framework named SAMBO-RL to mitigate distribution shift in model-based offline RL. Theoretical and empirical results demonstrate the effectiveness of the approach across multiple benchmarks.", "questions": "Terminology\n- Is \"preference buffer\" an established term in the literature? If not, could the authors provide a clearer explanation or citation for this term?\n\nThe description of the experimental setup in Section 3.1 is vague, and some findings raise questions:\n- Is it correct that the training region is identical to the evaluation region when their sizes are equal (e.g., training region = 3 and evaluation region = 3)?\n- Fig. 1(b), line 146: the result that \"variance in outputs from ensemble reward models does not distinguish whether the transition belongs to the training region\" is surprising. How is the ensemble implemented? Were the ensemble members deliberately made diverse during training?\n- Line 124: how are ground-truth returns obtained? Returns $G_t$ depend on the sequence of rewards, which in turn depend on the states the agent visits and the actions it takes. Which policy is employed in the calculation of ground-truth returns?\n- Fig. 1(c): the blue curve improve as the evaluation region expands. Could the authors explain this phenomenon?\n- Line 129: when the training region is small, there must be a lot of duplicated trajectories while sampling 1000 trajectories. How is this handled?\n- Fig. 1(d), could the authors elaborate on how the conclusion in lines 151–153 is derived? The result is not surprising from a learning perspective, as this setup is only for in-distribution cases, and more data generally improves performance. The conclusion should clarify that more in-distribution data improves in-distribution performance.\n- Appendix G contains relevant details and should be integrated into the main text for better understanding.\n\nMethodology\n- line 191, how should the \"target policy\" be understood in the context of the framework?\n- Are the mixture ratio $k$ and KL constant $\\varepsilon$ sensitive to different tasks? If so, how are they tuned for each task?\n- In appendix F.1, how should the values in Table 8 be interpreted? Could the authors evaluate the method on more tasks? Is it correct that a separate Morse Network must be trained for each task? What is the type and size of the data used for training the Morse Network in the experiments?\n\nAblation study\n- Could the authors elaborate on the ablation setups: 1) without MDQ and with EXT, and 2) with MDQ and without EXT? My understanding is that MDQ relies on EXT, and sole EXT does not directly impact training. Providing two pseudocode snippets, similar to Algorithm 1, for these setups would be helpful.", "limitations": "See weaknesses and questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and easy to follow, with logical structuring and clear explanations of the methodology.\n- The proposed method is novel on addressing distribution shift in PbRL.\n- The paper includes sufficient technical details in both the main text and appendix, making the method comprehensible.\n\nWeaknesses\n- Some aspects of the methodology and experimental setup require further clarification. Specific details are outlined in the questions below.\n- The proposed method introduces additional hyper-parameters and models, which may limit its practicality and scalability in real-world applications due to the increased complexity and tuning requirements.", "was_revised": true, "final_justification": "I raise my score from \"Borderline reject\" to \"Borderline accept\". My initial concerns have been addressed. \nHowever, the revised manuscript still needs clearer presentation to eliminate misunderstandings I raised earlier. Additionally, the significance of the proposed method is limited by inuring additional hyper-parameters, as I pointed out in my initial comment: \"The proposed method introduces additional hyper-parameters and models, which may limit its practicality and scalability in real-world applications due to the increased complexity and tuning requirements.\" For these reasons, I would suggest a \"borderline accept\" to this submission.", "rebuttal": "Dear Reviewer 1gpz,\n\nWe would like to extend our sincere gratitude for your detailed and insightful feedback on our manuscript. Your thoughtful comments and questions have been invaluable in helping us identify areas for improvement. We are encouraged that you found our paper well-written, our method novel, and our technical details sufficient.\n\nWe have carefully considered all your suggestions and provide the following point-by-point responses to your questions and concerns. We hope these clarifications will address the issues you raised and demonstrate the value of our work.\n\n#### **1. Terminology: \"Preference Buffer\"**\nThank you for this question. The term \"preference buffer\" is a direct and intuitive adaptation of the standard \"replay buffer\" concept for the Preference-based RL (PbRL) setting. In PbRL, instead of storing state-action-reward-next-state tuples, the agent collects and learns from preference data. This data, consisting of pairs of trajectory segments and a human preference label, i.e., $(\\tau^0, \\tau^1, y_p)$, is stored in a dataset $\\mathcal{D}_p$. While the exact phrase \"preference buffer\" may not be universally standardized, the concept of a buffer or dataset for storing preferences is fundamental to nearly all modern PbRL algorithms. Foundational works like Christiano et al. (2017) and Lee et al. (2021a, 2021b) all rely on collecting and sampling from such a dataset of preferences to train the reward model. We will clarify this in the revised text by explicitly defining it as the dataset $\\mathcal{D}_p$ that stores preference tuples and citing these foundational works to ground the concept.\n\n#### **2. Experimental Setup in Section 3.1 (Motivating Example)**\n\nWe apologize for the lack of clarity. The training and evaluation regions are distinct spatial areas within the 9x9 grid world, as depicted in **Figure 1(a)**. The training region is centered, while the evaluation region expands outwards from the center. Therefore, even when their side lengths are equal (e.g., both are 3x3), they are not identical unless both are the full 9x9 grid. For instance, when the training region is 3x3 and the evaluation region is 3x3, the evaluation is performed on a 3x3 area that may or may not perfectly overlap with the training region, depending on the specific setup shown in Figure 1(a). We will revise the caption and text in **Section 3.1** to make this distinction clearer.\n\nThis is an excellent question that highlights a key motivation for our work. The ensemble was implemented using a standard approach: we trained multiple reward models (5 in our experiment) on the exact same preference buffer, with diversity introduced only through different random network initializations and different mini-batch shuffling during training. This is a common technique for estimating uncertainty (e.g., in RUNE). Our finding, as shown in **Figure 1(b)**, is that the variance produced by such an ensemble does not serve as a reliable indicator for distinguishing in-distribution (ID) from out-of-distribution (OOD) transitions. The variance can be high for complex ID transitions and low for simple OOD ones. This observation motivates our decision to use a dedicated OOD detection mechanism (the Morse Neural Network) rather than relying on ensemble variance for guiding exploration.\n\nThe ground-truth return for a given trajectory is calculated by summing the predefined, ground-truth rewards of each cell visited in that trajectory. In this specific experiment, trajectories are generated by a random walk (i.e., uniformly sampling actions), not by a learned policy. The goal is not to evaluate a policy, but to assess the reward model's ability to correctly rank pre-existing trajectories. Thus, the ground-truth return $G(\\tau)$ for a trajectory $\\tau$ is simply the sum of the true rewards encountered, and no single policy is employed to generate all trajectories for evaluation.\n\nThere may be a misunderstanding of the plot. In **Figure 1(c)**, the blue curve (representing a tiny 1x1 training region) shows a *sharp decrease* in performance as the evaluation region expands. It starts with a high Spearman correlation when the evaluation region is also 1x1 (i.e., evaluating on the same single cell it was trained on) and then plummets as the evaluation region grows to include states the reward model has never seen. This result strongly supports our central claim: reward models generalize poorly to OOD trajectories, and their reliability is fundamentally tied to the coverage of the preference buffer.\n\nWhen sampling 1,000 trajectory pairs from a small region, duplicates are indeed likely. We sampled with replacement. For this motivating experiment, the presence of duplicate trajectories in the preference buffer is not an issue. It simply means the reward model observes the same preference comparisons multiple times during training, which is standard. The key takeaway of the experiment is about the effect of the spatial *coverage* of the training data, not the diversity of trajectories within that covered region.\n\nWe agree that moving key details about the motivating example from the appendix to the main text would improve readability. In the revised manuscript, we will incorporate the most relevant information from Appendix G into **Section 3.1**, space permitting.\n\n#### **3. Methodology**\n\nIn our framework, the \"target policy\" $\\pi_T$  refers to the current actor's policy that we are actively trying to improve. In our implementation, which uses SAC (as mentioned in Algorithm 2), $\\pi_T$ is the current SAC policy. Our exploration policy, $\\pi_E$, is derived from $\\pi_T$ using the proximal-policy extension method to gather data that is most useful for the *next* update of $\\pi_T$. We will add this clarification to the text.\n\nThis is a crucial question regarding the practicality of our method. As shown in our ablation study in **Section 4.2** (Figures 2b and 2c), we investigated the sensitivity of these parameters. We found that performance is robust within a reasonable range of values. Optimal performance was achieved around $\\kappa=0.5$  and $\\epsilon=0.01$. Based on this, we used these fixed values for all tasks in our experiments without any per-task tuning. This indicates that while PPE introduces new hyperparameters, they are not overly sensitive and do not require a costly tuning process for each new task, thus supporting the practicality of our approach.\n\n* **Interpretation of Table 8:** We apologize for not having the appendix available to you. Table 8 in Appendix F.1 reports the performance of our Morse Network as an OOD detector on several tasks, using standard metrics like AUROC and AUPR. High values for these metrics indicate that the network is effective at distinguishing between in-distribution data (from the preference buffer) and out-of-distribution data, which is crucial for our method.\n* **Evaluation on More Tasks:** Yes, a separate Morse Network must be trained for each task, as the distribution of state-action pairs is task-specific.\n* **Training Data:** The Morse Network is trained using the state-action pairs $\\{(s,a)\\}$ available in the current preference buffer $\\mathcal{D}^p$, as described by the loss function in **Eq. (2)*. The size of this buffer grows as more feedback is collected. In our experiments, the training starts with an initial set of preferences and is periodically updated using all data in $\\mathcal{D}^p$. We will ensure these details are clearly stated in the appendix of the final version.\n\n#### **4. Ablation Study**\n Your understanding is perfectly correct, and this highlights the synergy between the two components. We will gladly add pseudocode for these ablations in the appendix of the revised paper.\n* **Ablation 1 (QPA + EXT w/o MDQ):** In this setup, the agent explores using our proximal-policy extension method (**EXT**) to gather OOD data into the main replay buffer. However, for querying, it uses a standard method (in our case, the policy-aligned query from QPA) which does not explicitly prioritize selecting this newly found OOD data for labeling. As a result, the *preference buffer's* coverage does not expand effectively, and the reward model's OOD generalization does not improve, leading to minimal performance gains.\n* **Ablation 2 (QPA + MDQ w/o EXT):** Here, the agent explores normally (via the SAC policy's stochasticity), without active OOD exploration from **EXT**. The **MDQ** mechanism is then used to select pairs for querying. However, since the replay buffer contains very little actively-sought OOD data, the OOD sampling part of MDQ ($P^{out}$) has a poor set of candidates to select from. This also leads to ineffective expansion of the preference buffer's coverage.\n\nAs you rightly pointed out, the two components are complementary. EXT provides the OOD data, and MDQ ensures this data is incorporated into the preference buffer to improve the reward model. This synergy is why the full **PPE** algorithm significantly outperforms either component alone, as shown empirically in **Figure 2(a)**.\n\n***\n\nOnce again, we thank you for your rigorous review and constructive suggestions. We believe that by incorporating these changes, we can significantly improve the clarity and impact of our paper.\n\nSincerely,\nThe Authors"}, {"review_id": "OtifinJoUe", "reviewer": "Reviewer_iwcb", "summary": "This paper studies the problem of preference-based reinforcement learning (PbRL), in which an agent is tasked with learning a reward model based on preference relations between trajectories provided by a “teacher”, while simultaneously learning a policy that maximizes the learned reward function. The authors propose the use of two techniques for (i) expanding agent exploration in regions of the MDP which are undersampled in the agent’s buffer, and for (ii) sampling trajectories from a mixture distribution of in-distribution and OOD trajectories. To implement the first technique, the authors use Morse neural networks and introduce an approximation to a constrained optimization problem formulation. The method is evaluated on the MetaWorld and DMControl benchmarks and compared to other state-of-the-art baselines.", "questions": "Below, I have questions and constructive feedback to the authors:\n\n- Section 3.2 is hard to follow and could be improved in terms of clarity and presentation. For instance, what motivates the choice of using Morse neural networks and an RBF kernel? What are the necessary assumptions for this choice to be valid? At a high level, what is the goal of Eq. 2?\n\n- How tight is the closed-form approximate solution in Eq. 4 for Eq. 3? It would be important to provide an approximation bound in Proposition 3.1 to understand how loose this approximation is.\n\n- In line 218, how is the dataset $D_{cp}$ collected?\n\n- In Section 4.1, how is the preference queried? That is, who/what is ranking the trajectories?\n\n- The paper is missing a more thorough discussion of the state-of-the-art techniques. For instance, how do PEBBLE, SURF, RUNE, and QPA differ from PPE? What is the main high-level idea behind each of these baselines?\n\n- “... we record the mean and standard deviation of the final evaluation episodes aggregated across all random seeds”. Is this evaluated with respect to which metric? Total reward? Discounted return? The caption of Table 1 should indicate what the numbers in the table are measuring.\n\nRegarding the claims, “As shown in Table 1, PPE consistently achieves superior performance across all benchmark tasks, indicating that PPE can more effectively select and utilize feedback under limited query budgets. This supports our claim that expanding the coverage of the preference buffer improves the reward model’s evaluation capability and leads to more reliable policy updates.”:\n\n- Why is this setting “under a limited query budget”? This is never explained.\n\n- The fact that PPE achieves a higher return does not necessarily imply that it is more effective in selecting and utilizing feedback, or that it is better due to the expansion of the coverage of the preference buffer. Those claims are only supported by the ablation study.\n\n- Based on Fig. 2(a) and Fig. 2(c), it seems that the full proposed method (red curve in Fig. 2(a)) is only able to outperform QPA when the KL constraint is set to 0.01 (orange curve in Fig. 2(c)). For all other values, the performance is similar or worse than QPA (blue curve in Fig. 2(a)). \nDoes the recommended value of 0.01 result in better performance in other domains, or is it necessary to tune this value separately in each domain?\n\n- Notice that some of the values in Table overlap in their [mean-std,mean+std] range, but only PPE results are in bold font, e.g., Quadruped-walk, Quadruped-run.\n\nOther minor issues:\n- Next, we trained a reward model using data from the preference buffer with a Bradley-Terry loss. -> Please add a reference.\n- “ RUNELiang et al. (2022)” -> Should use \\citep instead of \\citet here.\n- “... training process, Only w” -> incorrect capitalization after comma.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The findings regarding the importance of having a good coverage of the state-action pairs in the agent’s buffer when learning reward models\n- The proposed method, combined with the state-of-the-art method QPA, is able to outperform standard QPA (when properly tuned).\n\nWeaknesses:\n- The paper has clarity and presentation issues that make it difficult for the reader to follow it (see Questions below).\n- Some algorithmic decisions need more justification and better description, as well as the experimental setting employed in Section 4 (see below).", "was_revised": true, "final_justification": "I thank the authors for the clarifications and detailed responses. I increased my score to 4 in light of the answers to my questions. I strongly suggest that the authors incorporate these clarifications in the main text in case of acceptance.\n\nIn my opinion, the main problem with the paper is the lack of clarity in the original submission, which generated many questions by all reviewers regarding basic definitions and algorithmic details. This is a borderline paper in my opinion. I have seen papers with worst quality being accepted and papers with better quality being rejected in the past.", "rebuttal": "Dear Reviewer iwcb,\n\nWe would like to extend our sincerest gratitude for your detailed and insightful review of our manuscript. Your constructive feedback is invaluable, and we believe that addressing your comments will significantly improve the quality and clarity of our paper.\n\nWe are encouraged that you recognized the strengths of our work, particularly our findings on the importance of preference buffer coverage and the empirical performance improvements of our method. We have carefully considered all your suggestions and provide the following point-by-point responses to your questions and concerns.\n\n---\n\n### **Responses to Questions**\n\n**1. Clarity of Section 3.2 (Motivation for Morse NN, RBF Kernel, and Goal of Eq. 2)**\n\nThank you for pointing out the need for greater clarity in this section. We agree that a clearer explanation of our methodological choices is essential.\n\n* **Motivation for Morse Neural Networks:** Our primary goal is to actively explore regions that are undersampled by the current policy but still close to it. To achieve this, we first need a mechanism to quantify whether a given state-action pair $(s, a)$ is \"out-of-distribution\" (OOD) with respect to the data currently in our preference buffer $\\mathcal{D}^{p}$. As we discussed, Morse Neural Networks (Dherin et al., 2023)  provide a principled way to learn an unnormalized density function $M(x)$ that is explicitly designed for this purpose. The function value approaches 1 for in-distribution data (modes of the dataset) and decreases towards 0 for OOD data. This provides a direct and continuous measure of \"OOD-ness,\" which is precisely what we need for our exploration strategy.\n\n* **Motivation for the RBF Kernel:** We use the Radial Basis Function (RBF) kernel, as shown in Eq. (1)  to shape the Morse Network. This is a standard and well-established choice in kernel-based methods (Seeger, 2004). The RBF kernel is advantageous here because its value depends on the Euclidean distance, creating a smooth, localized \"bump\" around in-distribution data points. This property is ideal for generating the desired density field where the density value decreases as we move away from known data, which aligns perfectly with the objective of the Morse Network.\n\n* **Goal of Equation (2):** Equation (2) defines the loss function used to train our Morse Neural Network. As we state on page 5 (lines 172-174) , this loss is derived from minimizing the KL divergence between the distribution of our preference buffer data and the learned density model. The loss function has two main components:\n    1.  The first term, $\\frac{\\lambda^{2}}{2}||f_{\\phi}(s,a)-a||^{2}$, encourages the model to learn an identity mapping ($f_{\\phi}(s,a) \\approx a$) for state-action pairs $(s, a)$ that are **in-distribution** (i.e., from the preference buffer $\\mathcal{D}^{p}$). This makes the Morse value $M_{\\phi}(s,a)$ low, as desired for in-distribution samples.\n    2.  The second term involves sampling actions $a_{u}$ from a uniform distribution and penalizes the model if it assigns high density to these random (likely OOD) actions.\n    \n    In essence, the goal of optimizing Eq. (2) is to train a model $f_{\\phi}$ that can reliably distinguish between in-distribution and out-of-distribution state-action pairs by outputting a continuous OOD score via the Morse function $M_{\\phi}$.\n\nWe will revise Section 3.2 to incorporate these clarifications, ensuring the motivation behind our choices is more transparent to the reader.\n\n**2. Tightness of the Approximation in Eq. (4)**\n\nThis is an excellent question regarding the theoretical guarantees of our approximation. The closed-form solution in Proposition 3.1  is derived by making two simplifying assumptions, which are detailed in Appendix D : (1) we use a first-order Taylor expansion to approximate the objective function $M_{\\phi}(s,a)$, and (2) we tighten the KL divergence constraint.\n\nDeriving a tight, analytical approximation bound is theoretically challenging due to the nature of these assumptions. However, we empirically analyze the impact of this approximation in our ablation study on the KL constraint parameter $\\epsilon$, presented in Figure 2(c) . As we discuss on page 9 (lines 306-309) , the results confirm our theoretical intuition: performance degrades if $\\epsilon$ is too large (as the first-order approximation becomes inaccurate) or too small (as exploration is overly restricted).\n\nThe primary motivation for this approximation was to develop a computationally efficient method suitable for real-time application in the RL loop, avoiding the high overhead of iterative solvers at each timestep. Our experiments demonstrate that with a reasonably chosen $\\epsilon$ (e.g., 0.01), this approximation is highly effective in practice. We will add a brief discussion in the main text to acknowledge this trade-off between theoretical tightness and computational feasibility.\n\n**3. Collection of the Dataset $\\mathcal{D}^{cp}$**\n\nWe apologize for the lack of clarity on this point. The dataset $\\mathcal{D}^{cp}$ represents the **candidate pool of trajectories to be queried**. Our method is designed to be integrated with existing query selection strategies. In our experiments, we integrated PPE with the state-of-the-art QPA algorithm.\n\nTherefore, $\\mathcal{D}^{cp}$ is first populated using the **policy-aligned query technique from QPA** (Hu et al., 2023), which selects trajectories relevant to the current policy. Our mixture distribution query method (Algorithm 1) then acts as a **re-sampling post-processing step** on this candidate pool $\\mathcal{D}^{cp}$ to select the final trajectory pairs for which we request preference labels . We will revise the text to make this process explicit.\n\n**4. Preference Query Mechanism in Experiments**\n\nFor our main benchmark experiments in Section 4.1, we followed the standard protocol used in prior PbRL literature (e.g., Christiano et al., 2017 Lee et al., 2021b). The preferences are provided by a **simulated oracle**. This oracle has access to the ground-truth returns of the trajectories. When presented with a pair of trajectory segments $(\\tau^0, \\tau^1)$, it provides a preference for the segment with the higher cumulative ground-truth reward. This ensures our experimental results are reproducible and allows for a fair and controlled comparison against the baseline methods. We note this practice in our motivating example on page 3 (lines 130-131).\n\nTo validate our approach in a more realistic scenario, we also conducted supplementary experiments with **real human feedback**, as mentioned in Appendix E.7. We will clarify the use of a simulated oracle for the main experiments in Section 4.\n\n**5. Discussion of State-of-the-Art Baselines**\n\nThank you for this suggestion. We agree that a more explicit high-level comparison of the baselines would strengthen the paper.\n\n**6. Evaluation Metric in Table 1**\n\nWe sincerely apologize for this omission. You are correct that the metric should be clearly stated. We will revise the caption of Table 1 and the main text to clarify:\n\n* For the **DMControl** suite tasks, the reported values are the **average episode returns**.\n* For the **MetaWorld** tasks, following the evaluation protocol of previous work, the metric is the **ground truth success rate**, reported as a percentage.\n\n**7. Clarification of Claims**\n\n\"Limited query budget\":** In our experimental setup, the agent is allowed to query for a fixed total number of preferences throughout training. This is enforced by setting a query frequency $K$ and a total number of environment steps (as seen in Algorithm 2 and the x-axis of Figure 2. This finite number of available queries constitutes the \"limited budget.\" Our goal is to maximize performance within this constraint. We will make this definition explicit in the experimental setup section.\n\n**8. Generality of the KL Constraint $\\epsilon$**\n\nThis is an important question about the practical application of our method. For all experiments reported in Table 1, we used the **fixed value of $\\epsilon=0.01$ across all tasks and environments without any per-task tuning**. Our results show that PPE consistently outperforms the baselines across a diverse set of locomotion and manipulation tasks. This demonstrates that the recommended value of $\\epsilon=0.01$ is robust and generalizes well, alleviating the need for costly hyperparameter tuning for each new domain, which we consider a practical strength of our approach.\n\n**9. Bolding of Results in Table 1**\n\nThank you for your careful eye and for pointing out this inconsistency. Your point is absolutely correct; bolding should be reserved for results that are statistically significantly superior, not just numerically higher when standard deviations overlap. As mentioned in our checklist response, we have performed Welch's T-tests to assess statistical significance (Appendix E) . In the revised manuscript, we will correct Table 1 and ensure that a result is bolded only if it is statistically significant (e.g., p-value < 0.05) compared to the next-best baseline.\n\n**10. Minor Issues**\n\nWe appreciate you catching these details. We will make corrections in the revised manuscript.\n---\n\nOnce again, we thank you for your thorough and valuable feedback. We are confident that addressing these points will make our paper stronger, clearer, and more impactful.\n\nSincerely,\n\nThe Authors"}]

Author rebuttal:

Meta-review: The paper introduces PPE, a coverage-driven exploration and querying scheme for preference-based RL that targets policy-proximal OOD regions (via a Morse NN detector) and balances ID/OOD querying. On DMControl and MetaWorld, PPE improves over strong baselines (notably QPA), with ablations supporting the synergy of its two components.

The acceptance is contingent on the following camera-ready updates:
* Consolidate clarifications in main text: define “preference buffer”; detail motivating setup; explain ensemble baseline implementation and Figure 1(c) behavior; specify evaluation metrics; add human-feedback details.
* Strengthen related work: contrast PPE with PEBBLE, SURF, RUNE, QPA, and exploration literature; justify Morse NN choice and Eq. 2’s information-geometric loss.
* Ablations and reporting: extend component ablations to ≥1 more environment; include entropy-coefficient sensitivity; ensure bolding only for statistically significant wins; add learning curves where helpful.
* Reproducibility: release code upon publication; document hyperparameters; note fixed KL value and mixture ratio used across tasks.
Decision: Accept (poster)