STAR: Efficient Preference-based Reinforcement Learning via Dual Regularization
Authors: Fengshuo Bai
Rui Zhao
Hongming Zhang
Sijia Cui
Shao Zhang
bo xu
Lei Han
Ying Wen
Yaodong Yang
Venue: NeurIPS 2025 poster ()
Keywords: preference-based RL
Primary area: reinforcement_learning
TL;DR: An efficient PbRL method that mitigates overfitting and overestimation via dual regularization, enhancing feedback efficiency in both online and offline settings

Abstract: Preference-based reinforcement learning (PbRL) bypasses complex reward engineering by learning from human feedback. However, due to the high cost of obtaining feedback, PbRL typically relies on a limited set of preference-labeled samples. This data scarcity introduces two key inefficiencies: (1) the reward model overfits to the limited feedback, leading to poor generalization to unseen samples, and (2) the agent exploits the learned reward model, exacerbating overestimation of action values in temporal difference (TD) learning. To address these issues, we propose STAR, an efficient PbRL method that integrates preference margin regularization and policy regularization. Preference margin regularization mitigates overfitting by introducing a bounded margin in reward optimization, preventing excessive bias toward specific feedback. Policy regularization bootstraps a conservative estimate $\widehat{Q}$ from well-supported state-action pairs in the replay memory, reducing overestimation during policy learning. Experimental results show that STAR improves feedback efficiency, achieving 34.8\% higher performance in online settings and 29.7\% in offline settings compared to state-of-the-art methods. Ablation studies confirm that STAR facilitates more robust reward and value function learning. The videos of this project are released at https://sites.google.com/view/pbrl-star.

STAR: Efficient Preference-based Reinforcement Learning via Dual Regularization
Fengshuo 1,, Rui Zhao Hongming Zhang" Sijia Cui
Shao Zhang'
Bo Xu Lei Han Ying Wen' Yaodong Yang
Shanghai Jiao Tong University PKU-PsiBot Joint Lab 'Zhongguaneun Academy *eencent
National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences Institute for AI, Peking University
Abstract
Preference-based reinforcement learning (PbRL) bypasses complex reward en- gincering by learning from human feedback. However, due the high cost obtaining feedback, PbRL typically relies on limited set of preference-labed samples. This data scareity introduces two key inefficiencies:
(I) the reward
model overfits to the limited feedback, leading to poor generalization to unseen samples, and (2) the agent exploits the learned reward model, exacerbating overes- timation of action values in temporal difference (TD) learning. To address these issues, we propose STAR, an efficient PbRL method that integrates preference margin regularization and policy regularization, Preference margin regularization mitigates overfitting by introducing a bounded margin in reward optimization, preventing excessive bias toward specific feedback. Policy regularization boot- straps conservative estimate Q from well-supported actteaaction pairs in the replay memory, reducing overestimation during policy learning. Experimental results show that STAR improves feedback efficiency, achieving 34.8% higher performance in online settings and 29.7% in offline settings compared to state- of-the-art methods. Ablation studies confirm that STAR facilitates more robust reward and value function learning. The videos of this project are released at https: / /tt//stttssssss google cww/view/pbrl pttp//prr............
1 Introduction
Reinforcement learning (RL) [47, 5] has demonstrated remarkable success across various domains [L, 66, 69, 35, 56, 26]. However, its reliance well-designed reward functions remains a significant challenge, as defining rewards for complex tasks is often difficult [59, 39, 62]. For example, in robotic manipulation, re-orienting a cube with a dexterous hand requires quantifying intermediate states, which is trivtrivial [3, 4, 58, 70].
Preference- based reinforcement learning (PbRL) offers alternative by replacing explicit reward functions with human preferences over agent trajectories [24, 37, 28, 73, 57]. PbRL. follows an iterative process: (1) collecting human feedback train reward model and (2) optimizing a policy using this learned reward model. This iterative process aligns the agent with human-desired behaviors without the need for manually specified dense reward signals.
Despite its advantages, PbRL suffers from two fundamental inefficiencies when human feedback is limited. First, the reward model tends to overfit due to the small amount of preference data, leading to poor generalization [13, 36, 60, 61, 54]. Second, policy learning exacerbates this issue: TD-based optimization relies the learned reward model, and errors in reward estimation, combined
"Corresonndence to: yaodong.yane yang @ oku.edu.en. leihan @eennnss@@miii www.mll.mmm....mm.ooooeeooo rui.zhao ml gmail.com.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

with overestimated Q-value updates, cause unstable learning [13, 28, 64, 19]. These challenges are illustrated in Figure 1, which visualizes how reward overfitting and Q-value overestimation degrade PbRL. performance. Similar issues have been discussed in prior work [28, 30), highlighting the need for more feedhack-efficient solutions. Addressing these inefficiencies is crucial for making PhRL. viable in real-world applications, where extensive human feedback is impractical.
To address these challenges, we introduce STAR, a coadback-efficienn PbRL algorithm that integrates two complementary techniques: human preferenceS margin regulariza'Tion and policy regulA Rization. Preference regu- larization mitigates overfitting in reward learn- ing by introducing a bounded margin in the optimization objective of standard preference learning, preventing the model from overly focusing on limited feedback data. Policy reg- ularization reduces overestimation bias during RL training by estimating a conservative Q value using only transitions stored in replay
learned reward (regularization)
conservative optimal Q learned Q
oracle reward leurned reward
in-sample data
Figure 1: Illustration of two core inefficien- cies-reward learning (left) and value estimation (right)-under limited human feedback. Our pro- posed STAR integrates preference regularization and conservative estimation to mitigate these challenges. memory. This estimate is then used to constrain policy updates via Kullback-Leibler (KL) divergence, stabilizing learning. Our empirical evaluations across complex manipulation and locomotion tasks, in both online and offline PbRL settings, demonstrate that STAR consistently outperforms baselines. Notably, this advantage is particularly pronounced when human feedback is limited, highlighting STAR's s ability make efficient use of limited data for superior performance.
The key contributions of our work are as follows: (I) We propose STAR, a novel feedback-efficient PbRL algorithm that integrates preference margin regularization and policy regularization to improve learning stability and feedback utilization. (2) We demonstrate that STAR enhances feedback efficiency over existing PbRL methods across diverse online and offline tasks. Furthermore, human experiments reveal the emergence of novel behaviors driven by human feedback. (3) We provide in-depth analysis showing that preference margin regularization and policy regularization improve the accuracy of the reward model and mitigate Q-value overestimation, leading to improved policy optimization in feedback- -limited settings.
2 Related Work
Preference-based Reinforcement Learning. PbRL is a novel way to learn agents from human feedback without reward engineering. Instead, a human provides preferences between the agent's s behaviors, and the agent uses this feedback to perform the task. Christiano et al. [4] formulates a basic framework, and Ibarz et al. [17] utilizes imitation learning as the warm-start strategy to speed up PbRL. To further improve feedback efficiency, PEBBLE [24] replaces PPO [44] with SAC [11] to achieve data efficiency and combines unsupervised pre-training with the reward relabeling technique during learning. Building upon this foundation, Park et al. [37] introduces SURF, a semi- supervised reward learning framework that improves reward learning via pseudo-labels and temporal cropping augmentation. MRN [28] incorporates bi-level optimization for improving the quality of the Q function. QPA [16] improves feedback efficiency by addressing the issue of query-policy misalignment. Besides, some research has various considerations, such as skill extraction [55], intrinsic reward [27], meta-learning [14]. and these methods have improved the efficiency to certain extent. In addition to focusing on PbRL in online settings, a significant portion of research also concentrates it in offline settings. PT [21] leverages transformer architecture to learn non-Markovian reward and preference weighting function. IPL [13] directly optimizes the implicit rewards deduced from the learned Q-function, ensuring alignment with expert preferences. Kang et al. [20] models offline trajectories and preferences in a one-step process without reward learning. DTR [51] balances staying close the behavior policy with high-return trajectories and choosing actions with high reward labels. When dealing with language models, PbRL naturally facilitates the emergence of reinforcement learning from human feedback (RLHF) [36]. Before that, Stiennon et al. [45] have fine-tuned a summarizing policy following the PbRL paradigm. Our approach is orthogonal to previous approaches in that we use conservative estimate Q to regularize the neural Q- Q-function to mitigate overestimation and leverage the preference margin regularization to prevent overfitting of the reward model.
2

---

Preliminaries
3.1 Preference-based Reinforcement Learning
In the preference-based RI. framework, human preferences replace hand-craft rewards. Following the notation in Christiano et al. [4], a trajectory segment a is defined as a sequence of states and actions (5t+1, dt++1 .... St+kk 0c+k). Given pair of segments (o o²), a human provides preference label y, which is a distribution over {(1, 0), (0, 1), (0.5, ) indicating preference for o", o², or indifference.
The reward model F assigns scores to state-action pairs, and the cumulative reward over a segment is used to predict human preferences. Following the Bradley-Teny Terry model [2], the preference predictor formulated as:
Polo > -o'] Sigmoid Σ ao - Iro(sbrat)
where 0² > o² indicates のº is more consistent with human expectations than o².
The reward model is optimized by minimizing the cross-entropy loss between predicted and actual preferences:
((0) log Polo > of) y() Polot > o9].
Creward (1))
(2)
(a 1/)-DDpp
where D pref denotes the dataset of preference pairs.
3.2 TD3-Based PbRL Framework
Twin Delayed Deep Deterministic Policy Gradient (TD3) [9] is an off-policy actor-critic algorithm designed to address value overestimation deep reinforcement learning. When implementing PbRL with TD3, the learned reward model Fgr replaces the environment reward in the standard TD3 pipeline. The policy learning objective becomes maximizing
Ia BBDD [ Qe, (s, no(s))
where D is the replay buffer of transition samples, and Qe, is the j-th Q-function.
Each Q-function is trained minimize the TD with rewards from the learned model Ply.
Jq(0,) (E [Q0, (St, az) "
where the target value Ye (st at) + 7 min Q8, (s1+1, (5 + +e.. Here, 0 and denote the target networks for Q-functions and policy; E [0, 1) is the discount factor; and e ~ clip(N((0. a). -c, c) is the clipped noise added for target policy smoothing.
Method
In this section, we introduce STAR, flexible framework designed to enhance feedback efficiency in any PbRL approach. Figure 2 provides an overview of our method, which build dual regularization through three core components: preference regularization, conservative value estimation (Q), and policy regularization. These components work together to optimize feedback utilization and improve policy performance. The detailed procedures for both the online and offline settings are outlined in Algorithms and 2 in Appendix A.
4.1 Preference Regularization with Bounded Margin
Intuition and Analysis. We analyze the issue of reward over-optimization in preference-based lear- ing. As discussed in Section 1. optimizing Equation (2) encourages the reward model to align with human preferences. However, this objective implicitly drives the optimization toward an unbounded reward difference between preferred and rejected segments: Σ, (s. az) Σ, Fe (s), a.) + co. Although the predicted rewards do not become literally infinite in practice, this unbounded opti- mization tendency indicates that Equation (2) is an ill-posed objective. It encourages the model to become overconfident in predictions for training data, leading to poor generalization and the reward overooptimization problem. To address this, we introduce Preference Margin Regularization

---

(s, a, s
(2) reward learning with margin regularization
human preferences
(1) (s, a F2b (8, a),
(,, s')
policy learning
conservative estimate Q
T (a|s)
with regularization
value estimate Q8
environment space
replay memory
state visited
state unseen
Figure 2: Illustration of STAR. (1) Label rewards using P.g. (2) Update the reward model Fie via
bounded optimization objective from preference margin regularization. (3) Estimate conservative Qe and update Qp. (4) Regularize TH with QE.
(PMR), which enforces a more reasonable decision boundary between preferred and rejected seg- ments, thereby mitigating overfitting to training preferences. Detailed derivations and analyses are provided in Appendix B.1.
PMR Formulation. Drawing from margin learning theory [43, 46, 29], we propose PMR, which introduces margin parameter & to smooth the target distribution:
Msmooth
(1 E) - (1- v).
(5)
The PMR loss function becomes:
LYMR (v)
Σ Osmoouth (1) log Polo ol-i]]
(6)
(00,ar .D-
By setting the derivative of this loss to zero and solving for the optimal solution, we can show that PMR implicitly enforces a bounded margin:
EP)(d).0)) aj) - EFo(sh log
1 1-6
This bounded margin between preferred and rejected samples replaces the unbounded optimization objective standard preference learning, effectively preventing the reward model from becoming overly confident while maintaining sufficient discriminative power.
4.2 Policy Regularization with Conservative Q
Building upon analysis of reward overfitting, we now address the complementary problem of Q-function overestimation in PbRI. Even with a well-calibrated reward model from PMR, standard Q-learning can still lead overestimation, especially in regions with limited data.
Conservative Estimate QE- We introduce a conservative Q-function Qe that only utilizes well- supported state-action pairs from the current replay memory. This conservative estimation provides two main advantages: firstly, it enables further exploitation of samples in the replay memory; secondly, it prevents overestimation caused by extrapolation in unseen states and actions. Following principles from conservative Q-learning [23, 22, 10, 67, 68], we define constrained operator T:
T&Qe a:) = 3 log (,) exp(Qe(st,
(8)
where 3 > 0 is temperature parameter. Since state-action pairs sampled from the replay buffer, the operator focuses exclusively on in-distribution actions. For any 31 > B2 > O, we have TBi Qe < TT As 3 + 0o, T + E[[e), while 3 - 0 recovers the in-distribution greedy estimate, i.c., T + sQp(

---

The conservative Q-function Qş is trained by minimizing:
JQ(E) BrpwD (Q az) Fas (st, as) -eT
(9)
where in in-distribution transition defined as: 7.M (St, 0t, St+1, a(+1).
Policy Regularization. We leverage the conservative Q-function Qe to regularize the policy learning process. The key insight is Qe derive reference policy से that captures the current best estimate of action distributions given uncertainty: [(a(s) x exp(Qę(8, ( (x), where ((x) is the partition function ensuring normalization.
We regularize the learned policy Is toward this reference policy using the KL divergence. Combined with the TD3 policy loss in Equation (3), the overall objective becomes:
(+) ES [oo(s, (s)))) + DDu. (T (mo(-ss))))((()))))
(10)
where O controls the regularization strength. This formulation ensures that the policy optimization guided by both the current Q-function Qo and the conservative estimate Qg.
4.3 Theoretical Analysis
We provide theoretical analysis to show the properties of the conservative estimate, denoted as Q in finite statenaction space S x A. Unlike the standard Q-function Q, which bootstraps over the entire action space, Q bootstraps only in-distribution actions, thereby reducing extrapolation error from out- caatoo-ddisrributiin data. As result, Q provides a conservative (i.e., lower-bound) estimate of Q, and converges the optimal value data coverage increases. The proof of the following theorem is provided in Appendix B.2.
Theorem 4.1. Consider a tabular setting with funite state-action space S x A. Let Qt and Q denote the Q-values at iteration t, updated via the standard Bellman optimality equation and the in di-iistribution bootstrapping variant, respectively. Then, both sequences converge to fixed points Q" and Q", i.e., limy-+o Qt = Q* and lim Qi = Q- Furthermore, for all (s,a) E S x A. we have Q (s, a) I (s, a). with equality holding if all state-action pairs are visited.
Experiments
We design our experiments to evaluate the effectiveness of STAR under various conditions and to validate the impact of our key design components. Specifically, we aim to answer the following questions: Q1: Does STAR achieve strong performance with limited feedback in both online and offline settings? (Sec. 5.3) Q2: Can preference regularization mitigate overfitting in the reward model? (Sec. 5.4) Q3: Does policy regularization alleviate Q-value overestimation compared to prior methods? (Sec. 5.4) Q4: How do the two regularization components complement each other to improve overall learning efficiency? (Sec. 5.4)
Further details about the tasks used in our experiments are provided Appendix D.
5.1 Evaluation Setup
Tasks. We evaluate STAR on 18 tasks drawn from standard benchmarks. In the online setting, we use three locomotion tasks from the DeepMind Control Suite (DMControl) [52] and three robotic manipulation tasks from Meta-World [63, 65]. For the offline setting, we include eight challenging control tasks from D4RL. [7] and four robotic manipulation tasks from Robosuite [74]. These tasks span a diverse set of continuous control scenarios, varying in complexity and feedback characteristics.
Baselines. We compare STAR against several -tate-of-the-art methods, covering both online and offline PbRL settings. Online Methods: (1) PEBBLE[24]: combines unsupervised pretraining with reward relabeling to improve sample efficiency during policy learning. (2) SURF[37]: employs temporal data augmentation and pseudo-labels in semi-supervised framework enhance policy performance. (3) MRN [28]: introduces bi-level optimization for reward learning and represents current state-of-the-at in online PbRL. Offline Methods: (4) PT[21]: leverages transformer- based architectures to model non-Markovian rewards and peeference-weighting functions, capturing long-term dependencies in offline PbRL. (5) /PL[13]: directly optimizes implicit rewards derived
from the learned Q-function to align with expert preferences. (6) DTR[51]: balances staying close
5

---

PEBBLE (Ours)
(Reward)
1000
1000
1000
DMControl Roward
750
750
750
500
500
500
250
250
0.4
0.6
0.0
0.8
T.0
0.0
0.2
Env. Steps
x10
Env. Steps
x10
Env. Steps
x10"
(a) Cheetah Run (feedback=100)
(b) Walker Walk (feedback=200)
(c) Quadruped Walk (feedback=1k)
100
100
Success Rate (%)
Meta world
75
75
50
50
50
25
25
25
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Env. Steps
x10
Env.
Steps
x10
Env. Steps
x10
(d) Button Press (feedback=1000 100)
(e) Window Open (feedback=1000 100)
(l) Sweep Into (feedback=4kg
Figure 3: Evaluating curves of methods on locomotion tasks and robotic manipulation tasks. The solid line presents the mean values, while the shaded area indicates the 78% confidence interval over five runs. The red line is our method.
the behavior policy with high-return trajectories and choosing actions with high reward labels. Reward- Methods: Since PbRL lacks explicit rewards, we also report results from reward- based algorithms for reference. For online settings, we include (7) SAC[U]]; for offline settings, evaluate (8) [QL[22] and (9) TD3+BC [8]. These comparisons provide a reference for evaluating the performance of STAR relative methods with access to ground -rrth rewards.
Evaluation Metrics. We evaluate algorithms over five independent trials per task and report the mean and standard deviation of their performance. Meta-World tasks (Button Press, Window Open and Sweep Into) are assessed using success rate, as they involve goal-conditioned objectives, while DMControl tasks (Cheetah Run, Walker Walk and Quadruped Walk) and offline benchmarks are evaluated based on ground-truth episode return, which captures cumulative performance in continuous control settings.
5.2 Implementation details
In our experiments, we follow the basic setup employed by prior work [24, 37, 28], which includes unsupervised exploration and an uncertainty-base trajectory sampling strategy. All methods employ an ensemble of three reward models, with outputs bounded in [-1, using a hyperbolic tangent activation (details in Appendix C.I).
To enable systematic and reproducible evaluation, we adopt a scripted teacher, following [24, 37, 28), which generates preference labels by comparing pairs of trajectory segments based the environment's true reward function. These preferences accurately reflect the underlying rewards, enabling direct quantitative comparison across algorithms. Importantly, agents do not have access to ground-trutt -ruth rewards and rely solely on preference signals. For offline experiments, we use real human preference data from Kim et al. [21]. The number of preference pairs varies by task complexity: 100 pairs for tasks such as Cheetah Run, Button Press, and Window Open; 500 pairs for tasks like poooeer-ediuu--rplaa
and up 4000 pairs for more complex tasks like Sweep Into. The number of preference labels used per task is detailed in Appendix C.2.
To ensure fair comparisons, all methods are trained with the same network architecture and shared hyperparameters, including learning rate and batch size. The only differences lie in method-specific components such the loss function. We use publicly available inplementations for PEBBLE.

---

SURF, MRN, and IPL. Additional implementation details, including hyperparameter settings and network architecture, are provided in Appendix C.
5.3 Main Results
Our evaluation primarily focuses on training with limited feedback. This refers an insufficient num- ber of preference labels compared to the feedback required to achieve SAC algorithm performance.
Online Settings. We evaluate STAR's ' perfor- mance on three locomotion tasks from DM- Control and three robotic manipulation tasks from Meta-World, As shown in Figure 3, STAR outperforms the baselines in most tasks. In the Cheetah Run task, STAR achieves 96% of the best performance using only 100 prefer- ence labels, demonstrating efficient feedback usage. Building on PEBBLE the backbone, the comparison between the red (STAR) and green (PEBBLE) curves in Figure 3 reveals a significant performance gap. This highlights the effectiveness of our techniques, including preference margin regularization and policy regularization, in enhancing performance with limited preference labels.
Table
Summary of normalized scores. Mean normalized scores (baseline / SAC) across tasks from DMControl and weta-World.
Task/Method
PEBBLE
SURF
MRN
STAR (ours)
DMControl
0.704
0,550
0.781
0.881
Meta-world
0.358
0.309
0.614
1,000
Average
0,531
0.430
0.698
0.941
(34.8% t)
Table Averaged normalized scores of all baselines on AntMaze, Gym-Muuoco locomotion tasks, and success rate on Robosuite tasks. All agents training use the same real human preferences dataset from Kim et al. [21]. We train the STAR and IPL., and report the average and standard deviation averaged over 15 episodes. The term 'reward' refers to the use of ground-truth rewards.
Dataset
IQL (reward)
TDS+BC (reward)
MR
PT
IPL
DTR
STAR (ours)
nimmaz-meddum-play--22
73.88 4.440
0,25 #043
31.13 1699
70.13
337
30.19
#49 67.12
422
69.0 1447
untmazze medium diverse v2
68.13 10.15
0,25 1043
19.38 1938
65.25
539
24.21
1812
63.23
18
67.0 174
untmazo-large-play-22
48.75 4.35
0.0 #00
24.25 1423
42.38
858
12.46
45.93
1433
50.67 102
44.38
0.0 +:00
5.88
19.63
0.0 200
47.10
1
48.0
antmaze-v22 total
$8.79
0.13
20.16
16.72
55.85
58.67
heppe--ndduu--oplay-y..
83.06 = 15,00
64.42 21:52
11.56 30:27
84.54
a: 4.07
73.57 #:00
82.12 #21.88
85.29 =311
happer-
73.55 4m
101.17 9:0n
57.75
73.00
68.96
11.96
74.52 4:0
91.13 4:7000
96.75
4:431
walker2d mea.au2u--umiiuu-p-a-----
73.11 k.000
85.62 4.01
72.07 1.6
71.27
10.00
59.92 s.1.
73.00
73.91
133
walker 2d meeeeredd--mm--eeeeeeee
107.75
110.03
108.32
110.13
108.51 06
108.39 3
110.12 0:24
locomotion- v2
84.37
90.31
62.43
K3.72
79.13
88.66
91.52
pi
96.75 a: L.B
84.75 2.225
91.75 a 5,00
97.6 =29
94,51 = 3,60
98.0 4:
lift-mh
86.75 2.883
91.00 4:400
86.75
5:555
87.2 =:3
92.69 4:4
93.0 #34
can ph
74.50 6.822
68.00 1 9.15
69.67
5.99
74.8 224
67.12 1.966
70.0 13.56
can mh
56.25 1.77
47.50
50.50
$7.6
42.71
47.0
robosuite
72.81
77.0
To further illustrate the results, we summarize the mean normalized scores (baseline score / SAC score) across tasks from DMControl and state-of-the-an (MRN) by 34.8%, indicating STAR significantly improves the feedback efficiency of PbRL methods across a range of complex tasks.
in Table 1. It shows that STAR surpasses the
Offline Settings. We benchmark STAR against several offline PbRL algorithms using the D4RL [7] and Robosuite [74] robotics datasets, incorporating real-human preference data from Kim et al. [21]. To avoid relying on out-of-sample actions, extract the policy using advantage weighted regression [38, 53, 34]. Our evaluation includes comparisons with two state-of-the-art offline PbRL algorithms, IPL and PT, as well as two significant offline RL algorithms. Since TD3 serves as our foundational RL algorithm, we also present results from TD3 enhanced with Behavior Cloning (TD3+BC) [8].
Additionally, we compare different reward model architectures: MR (MLP-based), LSTM (LSTM- based), and PT (Transformer based), corresponding to models proposed by Christiano et al. [4]. Early et al. [6], and Kim al. [21], respectively. The results in Table show that STAR outperforms all baselines most tasks. On D4RL, our method achieves average performance improvement of 29.7% compared to sfate-of-the-ar methods. Notably, it exhibits a clear advantage in challenging tasks such as antmaze-large where it nearly matches the performance of IQL with task rewards and
7

---

Normalized Reward fs... (s)
Normalized Reward P(s. a.)
True Reward
2
True Reward
Learned Reward (Ours)
Learned Reward (Ours)
(Ours
Reward (Ours weo
0
0-
2
400
600
800
1000
100
200
400
500
Env. Steps
Env. Steps
(a) Cheetah Run
(b) Window Open
Figure 4: Quality of Learned Reward. Time series plots of the normalized learned reward (dashed line) and the tround-truth reward (solid line) obtained from rollouts of policy trained with STAR.
Cheetah Run
Button Press
Quadruped Walk
Sweep Into
Tru SURF
Toue MRN
True STAR
100
750
60
Success Rate (%)
Average Value
50
75
700
Reward
40
650
50
30
600
25
20
550
10
0.0
0.01
0.05
0.1
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Smoothing
Env.
Steps
x x10
(b) Value Estimation Accuracy Figure 5: Ablation studies evaluating (a) the impact of the parameter & in preference margin regular- ization, and (b) the overestimation bias value estimates.
(a) Preference Regularization
even surpasses IQL certain cases. These results highlight the effectiveness of our approach across diverse tasks in offline settings.
5.4 Ablation Studies
Quality of Learned Reward. To evaluate the impact preference regularization on reward model quality. we compare the learned reward model with the ground-trute -ound-truthhae reward function on Cheetah Run and Window Open tasks. As shown Figure 4, we plot time series curves for our method (STAR), its ablated variant without PMR, and the ground-truth reward. The results demonstrate that STAR produces a reward function that more closely tracks the ground-truth reward across time steps. In contrast, the variant without PMR exhibits higher variance and deviates significantly from the true reward, indicating that PMR contributes to improved reward model calibration.
Accuracy of Value Estimation. We assess the accuracy of value estimation in STAR by tracking the value estimate trajectory during the learning process on Cheetah Run. Figure 5b charts the average value estimate true value computed as the average discounted return obtained by following the current policy using the ground truth reward. A clear overestimation bias is observed during learning, with SURF overestimating by 32% and MRN by 25%. When constrained by the policy regularizer, STAR significantly reduces overestimation bias just 7%, leading to a more accurate Q-function. This reduction in bias directly improves value estimation, enabling a more stable and effective learning process PbRL. Quantitatively, the overestimation error decreases by 72%, resulting in a more reliable learning trajectory and contributing to better policy performance across a variety of tasks.
10,000 states and compares it estimate of the true value. The
Contribution each technique. assess the individual contributions of each technique and their interaction effects, conduct ablation study on preference margin regularization (PMR) and

---

(a) Walker March
(b) Cheetah Prowl
(c) Hopper Backflip
(d) Cheetah High Jump
Figure Human Experiments. Novel skills learned from human feedback.
policy regularization (PR). Table 3 reports the performance on three tasks: Cheetah Run, Window Open, and Sweep Into. Removing either PMR PR leads notable performance drop across all tasks, with the absence both causing the most severe degradation. These results highlight that both techniques essential and complementary in improving the robustness and effectiveness of STAR.
We further investigate the impact of the parameter E in PMR. Figure 5a shows how varying & influences the perfor- mance of STAR. Interestingly, even a small value (e.g.. 0.01) leads to noticeable performance gains, indicat- ing the sensitivity of the reward model to minor regularization. However, ex- cessively large values introduce esti- mation bias, consistent with observations in prior work [12, 33]. In addition, we observe that tasks with more feedback (e.g., Quadruped Walk, Sweep Into) tend perform best with smaller values, whereas tasks with fewer feedback (e.g., Cheetah Run, Button Press) benefit from larger s. This suggests that the severity of reward overfitting diminishes with increased feedback, reducing the need for aggressive smoothing.
Table 3: Contribution of Each Technique. Performance on Cheetah Run measured by episode return, while Window Open and Sweep Into measured by success rate.
Task
Ours
Ours w/o PMR
Ours wlo PR
Ours w/o both
Cheetah Run
713.5
609.2
$79.7
491.5
Window Open
65.0
50.0
41.2
10.0
Sweep Into
81.2
59.7
51.4
41.6
Effect of Feedback Quantity. To fur- ther investigate how the number of feed- back signals influences performance, we evaluate all methods on Walker Walk with varying amounts of feedback N (100, 200, 500, 1000}. As shown in Fig- ure 6, performance improves across all methods feedback increases, likely due to more accurate reward model. Notably, STAR consistently outperforms all base- lines, even under limited feedback, high- lighting its ability to learn more reli- able Q-function and maintain strong per- formance with fewer preference labels.
Reward
800
SAC (upper bound)
PEBBLE
600
SURF
MRN
STAR (Ours)
200
400
600
800
1000
Number of Feedback
Figure 6: Effect of Feedback Quantity on Walker.
Human Experiments. We demonstrate that agents trained with STAR using human feedback can acquire novel and diverse skills, as illustrated Figure Specifically, we showcase: (a) Walker lowering its center of gravity and marching, (b) a Cheetah agent prowling, (c) a Hopper performing a backflip, and (d) Cheetah executing a high jump-ean trained with only human preference queries. These results underscore the effectiveness of human feedback in guiding complex behavior acquisition via STAR, particularly in settings where manual reward design non-trivial. Videos of all behaviors included in the supplementary material.

---

Conclusion
In this work, we introduce STAR, a novel preference-basee RL algorithm that significantly enhances feedback efficiency. By integrating preference margin regularization and policy regularization, STAR outperforms prior methods and achieves strong performance across a range complex tasks both online and offline settings. A key strength of our method lies in its ability learn effectively from limited number of preference labels. Our analysis reveals two main factors driving this efficiency: (1) mitigating overestimation bias through conservative policy regularization, leading to more accurate Q-function estimates; and (2) reducing reward model overfitting via preference margin regularization. Additionally, human experiments demonstrate that STAR enables agents acquire diverse and novel skills, reinforcing its practical potential. By improving the efficiency of PbRL, STAR takes step toward making preference-driien learning more applicable real-world robotic systems.
Limitation
Despite the significant improvement in feedhack efficiency achieved by STAR, this work primarily fo- cuses state-based inputs. Extending the method to handle partially observable or high-dimensional inputs, such as raw RGB observations, remains an important direction for future research. Moreover, all experiments are conducted in simulation. A key next step for future is to investigate the applicabil- ity of STAR to real-world RL, enabling robots to acquire manipulation skills directly on physical platforms.
Acknowledgments
This work is partially supported by National Key R&D Program of China (2024YFC3505402) and Project No. 20240313, Zhongguancun Academy, Beijing, China (100094). We sincerely thank the anonymous reviewers for their helpful comments and suggestions, which improved this paper.
10

---

References
[1] Fengshuo Bai, Hongming Zhang, Tianyang Tao, Zhiheng Wu, Yanna Wang, and Bo Xu. Picor: Multi-task deep reinforcement learning with policy correction. 37(6):6728-6736, 2023.
[2] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[3] Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. In Conference on Robot Learning (CoRL), volume 164, pages 297-307. PMLR, 08-11 Nov 2022.
[4] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
[5] Hao Dong, Zihan Ding, and Shanghang Zhang. Deep Reinforcement Learning: Fundamentals, Research and Applications. Springer Nature, 2020.
[6] Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ramchurn. Non-markovian reward modelling from trajectory labels via interpretable multiple instance learning. In Advances in Neural Information Processing Systems (NewrIPS), volume 35, pages 27652-27663, 2022.
[7] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[8] Scott Fujimoto and Shixiang (Shane) Gu. minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 20132-20145, 2021.
[9] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning (ICMI.), pages 1587- 1596. PMLR, 2018.
[10] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In International Conference on Learning Representations (ICLR), 2023.
[11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning (ICML), volume 80, pages 1861-1870, 2018
[12] Tong He, Zhi Zhang, Hang Zhang. Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In IEEE/CVF conference on computer vision and patlern recognition (CVPR), pages 558-567, 2019.
[13] Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based RL without reward function. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[14] Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for cemman-n--the-lowe
RL. In International Conference on Robot Learning (CoRL), pages 2014-2025, PMLR, 2023.
[15] Zhang-Wer Wei Hong, Tao Chen, Yen-Chen Lin, Joni Pajarinen, and Pulkit Agrawal. Topological experience replay. In International Conference Learning Representations (ICLR), 2022.
[16] Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Query-policy misalignment in siteeence-basee reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
[17] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
[18] Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence of stochastic iterative dy- namic programming algorithms. Advances neural information processing systems (NeurIPS),
6, 1993.
11

---

[19] Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-abilityy benchmarking of closed-loop end-to-end autonomous driving. Advances in Neural Information Processing Systems (NeurIPS), 37:819 37:819-844, 2024.
[20] Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference-guided policy optimization. In International Conference on Machine Learning (ICMI.), volume 202, pages 15753-15768, 23-29 Jul 2023.
[21] Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak I Lee, Pieter Abbeel, and Kimin Lee. Pref- erence transformer: Modeling human preferences using transformers for RI.. In International Conference Learning Representations (ICLR), 2023.
[22] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforeement learning with implicit q-learning. In International Conference on Learning Representations (ICLR), 2022.
[23] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1179-1191, 2020.
[24] Kimin Lee, Laura M Smith, and Picter Abbeel. Pebble: Feedback-efficiee interactive rein- forcement learning via relabeling experience and unsupervised pre-training In International Conference Machine Learning (ICML), volume 139, pages 6152-6163, 2021.
[25] Su Young Lee, Choi Sungik, and Sae- Young Chung. Sample-efficient deep reinforcement learning via episodic backward update. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[26] Yang Li, Shao Zhang, Jichen Sun, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan. Cooperative open-ended learning framework for zero-shot coordination. In International Conference on Machine Learning, pages 20470-20484. PMLR, 2023.
[27] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for explo- ration in preference-based reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
[28] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-nett Implicitly differ- entiable reward learning for preference-based reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 02270-22284, 2022.
[29] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. International Conference on Machine Learning (ICML), pages 507-516, 2016.
[30] Long Ma, Yuanfei Wang, Fangwei Zhong, Song-Chun Chun Zhu, and Yizhou Wang. Fast peer adaptation with context-aware exploration. arXiv preprint arXiv:2402.02468, 2024.
[31] Francisco S Melo. Convergence of q-learning: simple proof. Institute Of Systems and Robotics, Tech. Rep, pages 1-4, 2001.
[32] Neeraj Misra, Harshinder Singh. and Vladimir Hnizdo. Nearest neighbor estimates of entropy. American Journal of Mathematical and Management Sciences, 23(3-4):301-32 2003.
[33] Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[34] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprini axX1v:2006.09335, 2020.
[35] Yazhe Niu, Yuan Pu, Zhenjie Yang, Xueyan Li, Tong Zhou, Jiyuan Ren, Shuai Hu, Hongsheng Li, and Yu Liu. Lightzero: A unified benchmark for monte carlo tree search in general sequential decision scenarios. Advances in Neural Information Processing Systems (NeurIPS). 36:37594-37635, 2023.
12

---

[36]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 27730-27744, 2022.
[37] Jongjin Park. Youngeyo Seo, Jinwoo Shin, Honglak Lee, Picter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficien preference- based reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
[38] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In International Conference Machine Learning (ICML), page 745-750, 2007.
[39] Yuan Pu, Yazhe Niu, Zhenjie Yang. Jiyuan Ren, Hongsheng Li, and Yu Liu. Unizero: General- ized and efficient planning with scalable latent world models. arXiv preprint (.Vv:2206.10667, 2024.
[40] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[41] Egor Rotinov. Reverse experience replay. 2019.
[42] Max-Philipp B. Schrader. gym-sokoban.
attps://github httpa://gtthuu...cmmmm/araarrdd cow.mppSchaader///////
gym-sokoban, 2018.
[43] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 815-823, 2015.
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, has/1707.06347. 2017.
[45] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learming to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 3008-3021, 2020.
[46] Shan Suthaharan. Support vector machine. In Machine learning models and algorithms for big data classification: thinking with examples for effective learning, pages 207-235. Springer, 2016.
[47] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[48] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In IEEE/CVE conference on computer vision and pattern recognition (CVPR), pages 28 2818-2826, 2016.
[49] Csaba Szepesvári. Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning, 4(1):1-103, 2010.
[50] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 006600 2018
[51] Songjun Tu, Jingbo Sun, Qichao Zhang, Yaocheng Zhang, Jia Liu, Ke Chen, and Dongbin Zhao. In-dataset trajectory return regularization for offline preference-baser reinforcement learning. In AAAI Conference Artificial Intelligence (AAAI), volume 39, pages 20929 20937, 2025.
[52] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_ co__oolool: Software and tasks for continuous control. Software Impacts, 6: 100022, 2020.
13

---

[53] Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Advances Neural Information Processing Systems (NeurIPS), volume 31, 2018.
[54] Songsheng Wang, Rucheng Yu, Zhihang Yuan, Chao Yu, Feng Gao, Yu Wang, and Derek F Wong. Spec-vla: speculative decoding for aasionlanguage-actiinn models with relaxed accep- tance. arXiv preprint arXiv:2507.222. 22424, 2025.
[55] Xiaofei Wang, Kimin Lee, Kourosh Hakhamaneshi, Pieter Abbeel, and Michael Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback. In Conference on Robot Learning (CoRL), volume 164, pages 1259-1268, 08-11 Nov 2022.
[56] Xihuai Wang*, Shao Zhang*, Wenhao Zhang, Wentao Dong, Jingxiao Chen, Ying Wen, and Weinan Zhang. Zsc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination. The 38th Conference on Neural Information Processing Systems (NeurIPS 2024)
Track on Datasets and Benchmarks, 2024.
[57] Yuanfei Wang, Fangwei Zhong, Jing Xu, and Yizhou Wang. Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind. In International Conference on Learning Representations (ICLR), 2022.
[58] Yuanfei Wang. Xiaojie Zhang, Ruihai Wu, Yu Li, Yan Shen, Mingdong Wu, Zhaofeng He, Yizhou Wang, and Hao Dong. Adamanip: Adaptive articulated object manipulation environ- ments and policy learning. In International Conference on Learning Representations (ICLR), 2025.
[59] Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collec- tive robot reinforcement learning with distributed asynchronous guided policy search. In International Conference on Intelligent Robots and Systems (IROS), pages 79-86, 2017.
[60] Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. Llm4drive: A survey of large language models for autonomous driving. arXiv preprint arXiv:2311.01043 2023.
[61] Zhenjie Yang, Yilin Chai, Xiaosong Jia, Qifeng Li, Yuqian Shao, Xuekai Zhu, Haisheng Su, and Junchi Yan. Drivemoe: -fxture for vision- vision-language-e language- model in end-to-end autonomous driving. arXiv preprint arXi::2505 16278, 2025.
[62] Zhenjie Yang, Xiaosong Jia, Qifeng Li, Xue Yang, Maoqing Yao, and Junchi Yan. Raw2drive: Reinforcement learning with aligned world models for end-to-end autonomous driving (in carla v2). arXiv preprint arXiv:2505. 16394, 2025.
[63] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning ( (CoRL), volume 100, pages 1994-1100 PMLR, 2020.
[64] Hongming Zhang, Chenjun Xiao, Han Wang, Jun Jin, bo xu, and Martin Müller. Replay memory as an empirical MDP: Combining conservative estimation with experience replay. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[65] Hongming Zhang, Tongzheng Ren, Chenjun Xiao, Dale Schuurmans, and Bo Dai. Provable representation with efficient planning for partially observable reinforcement learning. In International Conference on Machine Learning (ICML), pages 59789, 2024.
[66] Hongming Zhang, Ke Sun, bo xu, Linglong Kong, and Martin Müller. A distance- based anomaly detection framework for deep reinforcement learning. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
[67] Hongming Zhang, Chenjun Xiao, Chao Gao, Han Wang, Martin Müller, et al. Exploiting the replay memory before exploring the environment: enhancing reinforcement learning through empirical mdp iteration. Advances in Neural Information Processing Systems (NeurIPS), 37: 85658-85693 2024.
14

---

[68] Hongming Zhang, Fengshuo Bai, Chenjun Xiao, Chao Gao, Bo Xu, and Martin Müller. 3- dqn: Improving deep q-learning by evolving the behavior. In International Conference on Autonomous Agents and Mulliagent Systems (AAMAS), pages 2317-2326, 2025.
[69] Ming Zhang, Shenghan Zhang, Zhenjje Yang, Lekai Chen, Jinliang Zheng, Chao Yang, Chuming Li, Hang Zhou, Yazhe Niu, and Yu Liu. Gobigger: A scalable platform for cooperative- competitive multi-agent interactive simulation. In International Conference Learning Representations (ICLR), 2023.
[70] Xiaojie Zhang, Yuanfei Wang, Ruihai Wu, Kunqi Xu, Yu Li, Liuyu Xiang, Hao Dong, and Zhaofeng He. Adaptive articulated object manipulation the fly with foundation model reasoning and part grounding. arXiv preprint (8t,:22577.1876,, 2025.
[71] Rui Zhao, Xu Liu, Yizheng Zhang, Minghao Li, Cheng Zhou, Shuai Li, and Lei Han. Craftenv: flexible collective robotic construction environment for multi- agget reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), page 1164-1172, 2023.
[72] Guangxiang Zhu, Zichuan Lin, Guangwen Yang, and Chongjie Zhang. Episodic reinforcement learning with associative memory. In International Conference on Learning Representations (ICLR), 2020.
[73] Xuekai Zhu, Daixuan Cheng, Dinghuai Zhang, Hengli Li, Kaiyan Zhang, Che Jiang, Youbang Sun, Ermo Hua, Yuxin Zuo, Xingtai Lv, et al. Flowrl: Matching reward distributions for Ilm reasoning. arXiv preprint arXiv:2509. 15207, 2025.
[74] Yuke Zhu, Josiah Wong, Ajay Mandlckar, and Roberto Martin-Martin. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12293, 2020.
15

---

Supplementary Material
Table of Contents
A Full Procedure of STAR
17
A.1
Overview
A.2
Discrete Settings
18
B Theoretical Analysis
19
B.1
Reward Overfitting Analysis
B.2
Proof of Theorem 4.1
20
C Experimental Details
23
C.I Preference-based RL Basic Settings
23
C.2 Feedback Experiments
24
C.3 Architecture and hyperparameters.
24
D Environment Specifications
24
D.1
Sokoban
25
D.2
CraftEnv
................................
25
D.3
Robotic tasks
26
D.4
D4RI benchmark
26
E
Full Experiments
E.1
Results on Discrete Settings
27
E.2
Ablation Studies
28
16

---

A Full Procedure of STAR
A.1 Overview
Our method builds on the PbRL framework PEBBLE [24], which serves as backbone for many online PbRL algorithms [37, 28]. In this paper, propose generic and flexible algorithm that can be seamlessly integrated with any PbRL algorithm and adapted to various settings, including discrete and continuous action spaces, well as online and offline environments. The detailed procedures of our method are presented in Algorithm for online settings and Algorithm 2 for offline settings.
For online settings, STAR introduces two distinct constrained operators for conservative Q-value estimation: max for discrete action spaces and T for continuous action spaces. These operators ensure more reliable policy learning by constraining overestimation in different action domains. In method first trains the reward model Fg using pre-collected human preference
offline settings, data, then proceeds to policy learning, carefully addressing the challenges of limited feedback and bias inherent in offline RL.
Algorithm STAR (Online)
Require: preference query frequency K, number of human's 's preference labels per session M
1: Initialize parameters of Qo, Tæ. Fø. Q& and preference dataset D <- 0
2: Initialize replay buffer B and To with unsupervised exploration
3: for each iteration do
4: Take action a ~ Ty and collect 8t+1
5: if' iteration % K = then
6:
// Query preference Sample pair of trajectories (o query human for y
7:
8:
Store preference data into dataset D + D DU [(o".'" " .9)
9:
// Reward learning
10:
Sample batch {(o o² })) from 2>
Optimize Equation (2) to update Fy
12:
Relabel the replay buffer B using Fv. end if
13:
14:
// Estimate conservative Q
Store transition (8s, as (8e, at); 8s+1) into replay buffer B
Drive Q via Equation (11) (discrete setting)
17:
Update Qe via Equation (9) (continuous setting)
18:
// Policy regular anguaarization
19:
Update Qo according to Equation (discrete setting)
20: Update Qo and rte according to Equation (4) and Equation (3), respectivelyycconiiuuus setting)
21:
end for
Ensure: policy Tt
Algorithm 2 STAR (Offline)
Require: preference dataset DD dataset B
1: Initialize parameters of Qo, Tg. Fg
2: // Reward learning
3: Optimize Equation (2) update For with preference data from D
4: Label the dataset via Fe
5: for each iteration do
6: Sample batch (s, Fo a), from B
7: // Estimate conservative Q
8:
Drive via Equation (11) (discrete setting)
9:
Update Q; via Equation (9) (continuous setting)
10:
// Policy angularization.
11:
Update Qe according to Equation (diseret setting)
12:
Update Qo and गe according to Equation (4) and Equation (3), cespmuuively.coomiiuuuus setting)
end for
Ensure: policy T&
17

---

A.2 Discrete Settings
In online settings with discrete action spaces, we provide an efficient implementation for bootstrapping a conservative estimate Q By leveraging well-supported state-action pairs from the current replay memory, we generate conservative Q-value estimates that mitigate overestimation risks. Specifically, we structure the replay memory as graph for discrete action spaces, following similar approach to Zhu et al. [72], Hong et al. [15], Zhang et al. [64]. This graph-based structure allows us to efficiently perform conservative estimations by identifying connections between state-action pairs, enabling rapid updates without incurring significant computational overhead.
We structure the replay memory as dynamic and directed graph, denoted as G = EV. Each vertex in this graph represents state paired with its associated action value estimation (s, -). forming the vertex set: V = {s|(s, Q(s. .)]]. Each directed edge represents a transition from state to state s' via action a, storing both the estimated reward f (s, a) and the transition count ((s, a, s'). These elements are essential for updating the model based on experience. The edge set is represented as E = {s 4 saI(a, (w. a). (s, a, s'), a)s).))}. To achieve efficient querying, each vertex and edge is assigned unique key through a hash function, ensuring query time complexity of ((1). This allows the graph to handle dynamic updates without significant overhead. Each vertex e contains an action set 0A(ss), which tracks the actions executed in state facilitating in-sample updates and improving learning efficiency. Similar to conventional replay memory, this graph stores the most recent experiences while maintaining a fixed memory size to prevent excessive memory usage.
Conservative Estimate Q.
This graph updating contains two primary components: estimation updating and reward relabeling. When new transition (s, a, (s, s') is observed, a new vertex and edge are added to the graph based on the previously outlined structure, initializing ((s, a) = 0 and setting N(s, a, 5') = If the edge for the transition already exists, the visit count is incremented: N(s. a, a,s') < NN, as. allowing the graph to adapt to repeated experiences and refine its estimation over time.
For updating the action value estimate Q. we sample a subset of graph vertices OV C V in reverse order, similar to the techniques used by Rotinov [41], Lee et al. [25]. This reverse-order sampling improves the efficiency of the updates by leveraging the structure of the graph, allowing for rapid convergence of Q During this process, the max-operator is constrained to operate over the in-sample action set 0.A(s) rather than the entire action space. This ensures that the method remains focused on well-explored actions, reducing the risk of overestimation by avoiding out-of-sample actions.
The value iteration process is defined as:
a(s,a)+ + Σ p(s is, Di'sss,
(F((s,)))
Q(s',a d'E0((s')
max
(11)
S'ES
where the empirical transition dynamics p(s'|s, N(s, a, s)) the graph. This update rule ensures that unseen actions are not queried during value estimation, thereby preventing overestimation.
Eg N(s, a, are computed from
For reward relabeling, each time the reward model F is updated, all past experiences in the graph are relabeled according to the updated model. This maximizes the utility of historical data and accounts for non-stationary reward functions. While this relabeling process can be computationally intensive as the number of stored transitions grows, it allows the algorithm adapt to changing reward landscapes and ensures that the learning process remains up-to-date with the most accurate reward information.
Updating Qa. In scenarios with discrete actions, we define â as the Boltzmann policy derived from Q. where (s)) = Softmax,ea (s...) -)). This policy is inherently conservative, considering only the support set 0s{(s) for a given state s. Similarly, the policy æ, derived from the Qo network, is expressed as (s) Softmax..eaAt) (Q)(s.))) :)). Total loss is defined as follows:
Laiscrete (0) = IE Tere
(Qo(s,a) a) - w)²| + 1ifreg (0),
(12)
where y = f a) + yax ((s', is the Q target, T (st, a, st+1 For a+)) is the transition and 17 is the weight factor.
18

---

B Theoretical Analysis
B.1 Reward Overfitting Analysis
Why Reward Overfitting? For reward learning, the logits output by the reward model trans- formed into probabilities using softmax function, ensuring that the resulting probabilities sum 1. The loss function in equation (13) is used to optimize re follows:
(o .W) ((0) log Polo x o'1 + v(1) log Poloo x d01].
Lreward (5)
(13)
In the optimal scenario, aim for the predicted probability of the preferred segment to be while the rejected segment's 's predicted probability should be O. This can be represented mathematically as:
exp F (sf. aQ))
exp(E. (5). a.)) + expl E (st,
(14)
exp E, P (s), a()) exp(E. fs a.)) exp (Ee F (s),
0
Expanding the softmax expressions in equation (14) yields the following analytical solution:
(o = eep Σ
Rss (sf. + expl Σ ast,
expl
Σ (52 ag)
log
expl
(s ai)) + expl > (ols a/))
(15)
exp( (
=0
Σ (o -00
Upon examining this analytical solution in equation (15), we observe that for the reward model to predict optimal behavior, the preferred segments must be assigned a constant value, while the rejected segments approach negative infinity. However, achieving this theoretical optimal solution is practically impossible, as would lead to overfitting. In practice, this results in the reward model being overly confident in its predictions, causing poor generalization to data.
Why preference margin regularization effect? To address this issue, introduce margin technique inspired by Szegedy et al. [48]. By introducing a bounded objective in reward optimization, constrain the reward model, preventing it from overoptimizing the loss function and ensuring that the analytical solution remains bounded. The specific update rule is as follows:
(,-E, if rd (0.5, 0.5), otherwise.
v(i.j) =
(16)
where margin factor.
Under preference margin regularization, we aim for the predicted probability of the preferred segment to be - E, while the rejected segment's predicted probability should be E. This can be represented mathematically as:
exp Pos (sf.a)))
(sf. af) + exp(E a()) =1 -
exp
(17)
explE, E, Pat (ff at))
exp E, Pr a!)) + exp E, Po (s) (s),
19

---

Expanding the softmax expressions in equation (17) yields the following analytical solution:
exp( Σ (sl. (1 E) exp(
Fa a)) + (1 -)
E) * expl 2 Fo (st.
exp(
Σ Pu(sf:a.) (1 E) expl
Pols), a)))
Σ Feelsl. ai) - (Fo(s) log
1 1
(18) From the derivation in equation (18), observe that after applying preference margin regularization, the objective shifts from maximizing the difference between preferred and rejected samples to maintaining a controlled margin. Specifically, preference margin regularization introduces a bounded margin of log (-=2) between positive and negative samples. This ensures that we do not overly optimize the loss during reward learning, which could lead to overfitting.
The key insight behind preference margin regularization is that limits the optimization process, preventing the model from becoming overly confident in its predictions by capping the margin between preferred and rejected samples. By keeping the margin bounded, preference margin regularization helps stabilize the learning process, reducing the risk of overfitting and improving the model's 's ability to generalize to unseen states.
B.2 Proof of Theorem 4.1
In this section, we analyze the property of Q in finite state-action space S A. The proof of lim; Qi = Q" has been well-established in previous work [40, 18, 31]. Then the proof of lim; Qi Q" is similar. We first prove the empirical Bellman operator Equation (21) is y-contraction operator under the supremum norm. Then when updating in a sampling manner as Equation (22), it can be considered as random process. Borrowing auxiliary result from stochastic approximation, we prove satisfies the conditions that guarantee convergence. Finally, to prove Q' lower-bounds Q", we rewrite Q* (s, a) Q' (s, a) based on the standard and empirical Bellman operators. When the data covers the whole state-action space, naturally have Q* == =Q"
For proof simplicity, we 3 denotes policies that interact with the environment and form the current replay memory. We first show existing results for Bellman learning in Equation (20), and then prove Theorem 4.1 in three steps. The Bellman (optimality) operator B defined as:
(BQ) = Σ P(s'ls, a)(r + max Q(s',a())
(19)
a'
s'ES
Previous works have shown the operator B is a y-contraction with respect to supremum norm:
||QQ
BQ2 ooc S >1111 - Q21loo:
the supremum norm ||e||00 maxigiçd a |us], d is the dimension of vector Following Banach's 's fixed-point theorem, Q converges to optimal action value Q* we consecutively apply operator B to Q, lim,- (B)"Q= Q"
Further, the update rule in Equation (20), i.e. Q-learning, is sampling version that applies the y-contraction operator B to Q
((s.a) + r(s,a) + max a(s'aa').
(20)
a'
can be considered as a random process and will converge to Q", lim4++00 Qi = Q", with some mild conditions [49, 40, 18,31]. 31].
Similarly, we define the empirical Bellman (optimality) operator B as:
(B))
Σ P(s
max Q(s', d')]. )
(21)
x'ES
And the sampling version is:
((3,a) + +
max a''0(u')'')>>
((s'aa'),
(22)
20

---

We split Theorem 4.1 into three lemmas. We first show B is a c- operator under supremum norm, thus converges to optimal action value Q', lim (B)"* Q- Then we show the sampling-bast based update rule in Equation (22) converges to Q", lim Q Q-, Finally, we show Q' lower-bounds Q", Q* a) Q* a) S 0, (, a) E S x A. And when the data covers the whole state-action space, i.e. B(a > for all state-action pairs, we naturally have Q" (8, a) = (s,a).
Lemma B.1. The operator defined in Equation (21) is a n-contraction operator under supremum norm,
[B B0 S 2 221loo
Proof. We can rewrite |BQ - B02lloo
||BQ|
BQ211o0
Σ (1)11,,)(r+
(((5',d)) P(s)s, + ag:S(as $1)00
max 22(6',
max
MAX
(
E'ES
max
Σ P(s'(s,))
[))
(1))
(2)5
8,0
of (1:(()))')))
d2:0(a5(=')30
S'ES
max 5,0
Σ
P(s'|s, a)
max
@i(s',
max
02(s', ab))
a): 1')
<4:((ab(='>>0
EES
S max y
Σ
P(s'ss,
|s,
max ):)()
-)( - -Q2(s', a)
s.a
SES
Σ P(s'ls, a
(((,5)) @2(5,a)
max y
{,0:3(((3}00
s'ES
E
P(5's, 221lo
max y
EES
= Q1 - 121
where the last line follows from Eves P()'(s,)) 1.
To show the sampling based update rule in Equation (22) converges to Q', we borrow auxiliary result from stochastic approximation [40, 18].
Theorem B.2. The random process {A,} taking values in IR and defined as
(+1(()) = (1 (x))) + (x())
(23)
converges to zero w.p.1 under the following assumptions:
(1) S $ 1, E, ax(x) co and Σ. (z) < xo;
(2) [E[F S >1].,|/.. with 7 <
(3) Var (F.( S C(1 ). C >
W is norm. In our proof it is a supremum norm.
Proof. See Robbins and Monro [40], Jaakkola et al. [18].
Lemma B.3, Given any initial estimation Qo. the following update rule: Q((s(41,a) Qc(s (24)
Qu+1(se, ai) Qe(si, a.) + ar( ar +
max
4:3(olec1)>00
converges w.p.1 I to the optimal action-value function Q
0 a < [)((s,a)= =0 and E < <0,
for all (s,a) SxA
21

---

Proof. Based on Theorem B.2, we prove the update rule in Equation (24) converges.
Rewrite Equation (24)
ax) = (1 (s, (s)) ar) + an(ae, ae)lra +7 [e(s++1,a) 1 as)
a:3(al841130
Subtract (sc, 0t) from both sides:
Q++1 (82, (x) "" (st, ae)
(1 (4(s), (Q) - Q (st, at)) + 0x)re +y +y max
(e(s++1,a) - (8t, a:
-+,, at)
Let
ax(s,)) = -)
(25)
and
((s,a)=+++) max (':3(a)s')>0 a') -'(s"_a')-Q'(ss0)) (s,
(26)
We get the same random process shown in Theorem B.2 Equation (23). Then, proving lim Q: Q' is the same as proving (s,s, a) converges assumptions in Theorem B.2 are satisfied under the definitions of Equations (25) and (26).
with probability 1. We only need to show the
Theorem B.2 (1) is the same the condition in Lemma B.3. It is easy to achieve, for example, we can choose (x, a) = 1/t. For Theorem B.2 (2), we have
a)) = Σ P(s'ss,a) +7 +y
- (s,
max
a':3(a((s))
.'ES
(BQ - as)
((,Q))(s, - (s,
Thus,
[E][F(ss = 11(8Q.) - (BQ")) 1o,0
ST114; - -Q'1loo
=7llArllo0:
with 7 < 1.
For Theorem B.2 (3), we have
(ar(F((s))FF) = (8) E[F((5))
(s) (18)))
E(r + Y max
Q((s',d') - (s) - -
a'=)(a')='>00
Elr + ?
Q d') - (s,)²
a' a'=
Varlr (')
max
C,(5', a [Fr
We add and minus @+ term to make it close to the RHS in Theorem B.2 (3):
Var(r + max (8', a') + max )0)
max D,(s'a')) a=)8(a)(s))00
Since r is bounded, thus r + : 2n (6', a²) bounded. And clearly the second part m (i(s',a') a') max 2- (s', a') can be bounded by || with some constant. Thus, we have
[ar(F(ss) S C(1 | 12.)
for some constant C' > 0 under supremum norm. Thus, by Theorem B.2, A, converges to zero w.p.1, i.c., Qu converges to Q* w.p.1.
22

---

Lemma B.4. The value estimation obtained by Equation (21) lower-bounds the value estimation
obtained by Equation (19):
a) Qs, S
(27)
for all state-action pairs.
Proof. Following the definition of Equations (19) and (21), we can rewrite as
max
(s, @ Q" (s)
8.0
max (BQ" a) BQ" (s)
5,a
max(
Σ P(s'|s.a)lr+ +Y max
(''''aa')) - P(s'ss,a)) + max Q" (s', a')))
5,a
(':0(a()'>00
SEES
*ES
max
Σ
P(d'ls,am
max 8'
(''''aa)) - max (s',
8,0
E'ES
Σ
((s'|ssaannnn @ (s d') max ('' (s',
max
E'ES
max
Σ
max
(s',aa)
8,a
EES
max y
Σ
P(s' s a)
max (3. (s,
9,0
5,
EES
y max 8.0
( (5, (* a)) = max (a'x(Q(( (Q'(s((()) a) (s,
s,
where the last line follows from Eves P(s (s,a) a) 1. Then we have
max(Q"'(s.)) (", a)) S y max (s. Q*(ssa)) aa
8,
s xz max sa
(8, (' (s,a))
S x" max ( ()) (s.
5,
Take limit for both sides and since 0 < 7 < 1, we have max (s, a) - (s, < 0.
When B(als) 0 for all state-action pairs, the two contraction operators B and B are the same. And based on Banach's fixed-point theorem, there unique fixed point. Thus Q' (s, a) (s, a) for all state-action pairs., i.e., (* a) Q* a) 0, (8,a) a) E S x A holds when ((a)s) > 0 for all state-action pairs.
Then, we get Theorem 4.1 proved with Lemmas B.I, B.3 and B.4.
C Experimental Details
In this section, we outline the detailed implementation settings, including basic PbRL configurations, feedback setup in experiments, and other training details.
C.1 Preference-based RL Basic Settings
In this section, we provide additional details on unsupervised exploration and the uncertainty- based sampling scheme, as mentioned in Section 5. These techniques are crucial for improving feedback efficiency in algorithms, as referenced in Lee et al. [24]. To ensure fair comparison, all preference-based RL algorithms in our experiments incorporate both unsupervised exploration and uncertainty-based sampling.
Unsupervised Exploration, Unsupervised exploration in preference-based RI., introduced by Lee et al. [24], involves designing intrinsic reward based the entropy of the state, effectively
23

---

encouraging the agent to explore diverse states and generate varied behaviors. Specifically, it utilizes a variant of particle-bat baasel entropy [32] as a computationally convenient entropy estimation method.
Uncertainty-basee Sampling. Several sampling schemes exist, including uniform sampling, dis- agreement sampling, and entropy sampling. The latter are categorized uncertainty-baase sampling and have demonstrated superior performance compared to uniform sampling, both intu- itively and empirically. In our experiments, all methods (in the online settings) employ disagreement sampling schemes.
C.2 Feedback in Experiments
Number of Feedback. In the online setting, we use 100 preference pairs for Cheetah Run, Button Press, and Window Open; 200 pairs for Walker Walk; 300 pairs for Push-5x5-1, Push-6x6-1, and Push-7x7-1; 1000 pairs for Quadruped Walk, Push-5x5-2, Push-6x6-2, Push-7x7-2, Strip-shaped Building, Block-shaped Building, and Simple Two-Story Building; and 4000 pairs for Sweep Into.
In the offline setting, we use 100 preference pairs for taetaaz--mmmiuuplll------ antmaze- diverse-v2, weweeee-mmiiumeepee for hopper--eeiuum-rpllaee untmaze-large-play playee2lll222 and aalmaze-large-eaee di
meeiimmmddeeeettteee can-ph, and lift-ph; 500 pairs walker2d-medium seapliuuu-rellly-2.... can-mh, and lift-mh; and 1000 pairs for
Human Labels. We incorporate human feedback from subjects with prior experience in robotic tasks, as outlined in PT [21]". An informed human annotator evaluates each task by watching video clips of trajectory segments and selecting the one that more effectively achieves the task objective. Each segment lasts 3 seconds (equivalent to 100 time steps). If no clear preference is observed, the annotator may assign equal preference to both segments by selecting a neutral option.
C.3 Architecture and hyperparameters.
In this section, we describe the architecture of the neural networks used in the SAC algorithm, which serves as the baseline method. The actor in SAC consists of two layers with 1024 hidden units. The two Q networks in SAC share the same architecture as the actor. This consistent architecture helps streamline the optimization process across the actor and critic networks. The detailed neural network parameters and hyperparameters for SAC are shown in Table 8a. Additionally, Table 8b presents the distinct hyperparameters for PEBBLE and STAR.
Hyperparameter
Value
Hyperparameter
Value
Discount
0.99
Batch size
1024
(81, .2))
(0.9,0.999)
Initial temperature
0.1
Hidden layer units
1024
Critic target update freq
2
Activation Function
ReLU
Critic
0.005
Critic optimizer
AdamW
Learning rate
5e-4
Actor optimizer
AdamW
(a) Hyperparameters of SAC.
Hyperparameter
Value
Hyperparameter
Value
Length of segment
50
Reward model size
3
Learning rate
0.0003
Frequency of feedback
5000
Reward batch size
128
Number of train steps
le6
Reward update
200
Replay buffer capacity
le6
Scaling parameter (STAR)
6
Graph update batch size (STAR)
32
margin parameter & (STAR)
0.05
Regularizer weight (STAR)
0.1
(b) Hyperparameters of PEBBLE and STAR.
D Environment Specifications
In this section, we delineate the tasks utilized our experiments. For online settings, discrete tasks include Sokoban [42] and Craftenv [71], while continuous tasks encompass robotic manip-
24

---

ulation challenges from Meta-world [63] and locomotion tasks from the DeepMind Control Suite (DMControl) [50, 52]. Regarding offline settings, we incorporate control tasks from the D4RL benchmarks [7].
D.1 Sokoban
Sokoban [42], the Japanese word for 'a warehouse keeper', is puzzle video game, which is analogous to the problem of having an agent in a warehouse push some specified boxes from their initial locations target locations. Target locations have the same number of boxes. The goal of the game is to manipulate the agent to move all boxes to the target locations. Specifically, the game is played on rectangular grid called a room, and each cell of the room is either floor or a wall. At each new episode, the environment will be reset, which means the layout of the room is randomly generated, including the floors, the walls, the target locations, the boxes' initial locations, and the location the agent. We choose six tasks with different complexities from Push-5x5-1 to Push-7x7-2, which is shown in Figure 8. The numbers in the task name denote respectively the size of the grid and the number of boxes.
(a) Push-5x5-1
(b) Push-5x5-2
(c) Push-6x6-1
(d) Push-6x6-2
Figure 8: Visualization of puzzle tasks from Sokoban, which focuses on evaluating the capabilities of agents in spatial reasoning, logical deduction, and long-term planning.
State Space. The state space consists of all possible images displayed on the screen, Each image has the same size the map, and using the way of dividing each pixel of the image by 255 to normalize into [0,1], we preprocess the image before inputting.
Action Space. The action space of Sokoban has total of eight actions, composed of moving and pushing the box in four directions, which are left, right, up, down, push-left, push-right, push-up, push-down in detail.
Reward Setting, The agent gets punishment with -0.1 reward after each time step. Successfully pushing a box to the target location, can get reward, and all boxes are laid in the right locations, the agent can obtain an extra +10 reward. We set the max episode steps to 120, which means the cumulative reward during one episode ranges from -12 to 10 plus the number of boxes.
D.2 CraftEny
Craftenv [71], A Flexible Robotic Construction Enviromment, is a collection of construction tasks. The agent needs learn manipulate the elements, including smartcar, blocks, and slopes, to achieve a target structure through efficient and effective policy. Each construction task a simulation of the corresponding complex real-woot wal-world task, which is challenging enough for reinforcement learning algorithms. Meanwhile, the CraftEnv is highly malleable, enabling researchers to design their own tasks for specific requirements. The environment is simple use since it is implemented by Python and can be rendered using PyBullet. We choose three different designs of the building tasks, shown in Figure 9, evaluate our algorithm in CraftEnv.
State Space. We assume that the agent can obtain all the information in the map. Therefore, the state consists of all knowledge of smartcar, blocks, folded slopes, unfolded slopes' body, and unfolded slopes' foot, including the position and the yaw angle.
Action Space. The available actions an agent are designed based on real-world smartcar models, including a total of fifteen actions. Besides all eight directions moving actions, i.e. forward, buckward, left, right, left-forward, left-bat barkbaaadddd right-forward, and right-backward there are interaction- actions, designed to simulate the building process in the real world. Specifically, the agent can act lift
25

---

(a) The Strip-shaped Building
(b) The Block-shaped Building
(c) The Simple Two-Story Building
Figure 9: Visualization of building tasks from CraftEnv. From left to right are The Strip-shaped
Building, The Block-shaped Building, and The Simple Two-Story Building task respectively.
and drop actions to decide whether or not to carry the surrounding basic element, and can flod or unflod slopes to build the complex buildings. In addition, the actions of rotate-left and rotate-right control the agent to rotate the main body to the left and right, and stop action is just a non-action.
Reward Setting. CraftEnv is flexible environment mentioned above. We can specify our reward function for different construction tasks. For the relatively simple tasks of building with specified shape requirements,
use discrete reward, where some reward is given when part of the blueprint built. While, for building tasks with high complexity, various reward patterns should be designed to encourage the agent to build with different intentions.
D.3 Robotic tasks
In experiments, utilize robotic manipulation tasks from Meta-world [63] and locomotion tasks from the DeepMind Control Suite (DMControl) [50, 52], which is visualized in Figure 10. Meta-World comprises 50 diverse manipulation tasks with common structural framework, while DMControl offers a collection stable, rigorously tested tasks, serving as benchmarks for continuous control learning agents. These tasks are equipped with well-formulated reward functions that facilitate agent learning. For performance evaluation, Meta-world introduces an interpretable success metric for each task, which serves the standard evaluation criterion across various settings. This metric often hinges on the proximity of task-relevant objects to their final goal positions, quantified as ||o- S e, with e representing nominal distance threshold, such as 5 em. DMControl, conversely, employs episode returns for evaluation. We adopt txed-length episodes of 1000 timesteps proxy. Given that all reward functions are designed such that r is proximity goal states, learning curves that measure total returns can maintain consistent y-axis limits of [0, 1000]. This uniformity simplifies interpretation and facilitates averaging across tasks.
D.4 D4RL benchmark
AntMaze. AntMaze involves a navigation challenge where Mujoco Ant robot must locate and reach specified goal. The data for this task is derived from pre-trained policy designed to navigate various maze layouts, primarily focusing on two configurations: medium and large. The data are compiled using two distinct strategies: diverse and play. Diverse data sets are produced by the pre-trained policy, which utilizes random start and goal locations, whereas play data sets are generated with specific, intentionally selected goal locations. The task's reward structure is sparse, awarding points only when the robot is within a predefined proximity to the goal; otherwise, no reward granted.
Gym-Mujoco Locomotion. In Gym-Mujoco locomotion tasks, the objective is to manage simulated robots (Walker2d, Hopper) to advance forward while minimizing energy expenditure (action norm) to ensure safe behavior. Two data generation methodologies are employed: mediumm-xxper and medium-replay. The emphnedium-expert data sets have an equal mix of expert demonstrations and suboptimal Shartially-trineed demonstrations. The emphmediun replay buffer of a partially-trained policy. The robot's forward velocity, a control penalty, and a survival bonus all contribute to determining the task's 's reward.
data sets come from the
Robosuite Robotic Manipulation. In Robosuite's 's robotic manipulation tasks [74], various 7-DoF simulated hand robots perform distinct tasks. Our experiments utilize environments simulated with Panda by Franka Emika, focusing on two specific tasks: lifting cube (lifi) and relocating a coke can from table to designated bin (can). Data collection involves inputs from either single proficient teleoperator (ph) or six teleoperators with varying skill levels (mh). The task's 's reward sparse, with further details deferred to the original paper.
26

---

(a) Cheetah
(b) Walker
(c) Quadruped
(d) Button Press
(c) Window Open
(f) Sweep Into
Figure 10: Six tasks used for experiments. (a-c) DMControl tasks. (d-f) Meta-world tasks.
E
Full Experiments
E.1 Results Discrete Settings
detailed introduction and visualization of the six puzzle-solving tasks from Sokoban and the three flexible environments from CraftEnv are provided in Appendix D. We selected these tasks to cover range complexities for our experiments. Figure 11 shows the learning curves of the average episode return for STAR and the baselines across discrete tasks. In each task, SAC, utilizing the ground-trutt reward, serves as the upper performance bound. As seen Figure 11, STAR demonstrates rapid performance gains early in training across various tasks. Remarkably, in several tasks, STAR achieves close to SAC's performance while using significantly fewer human preference labels, indicating high feedback efficiency. For example, in the Strip-shaped building task, STAR surpasses PEBBLE's average performance using only 30% the total samples, highlighting its sample efficiency, In contrast, some baselines, affected by randomness in challenging tasks, show decline in performance as training progresses, This underscores STAR 's robustness maintaining performance even with fewer feedback labels. These results demonstrate that STAR substantially reduces the amount of feedback required to tackle complex tasks effectively, making it a highly efficient approach in PbRL.
PEBBLE
PEBALE
(Ours)
(Reward)
Push 5
12
12
Epiode Reeurm
Ugiande Retur:
Egioode Return
Ripaade Return
T.0
0.5
Steps
x110
Steps 10
Stege X10
Strgp 3<20
(a) feedback=50
feedback=100
(c) feedback=300
(d) feedback=500
Figure 12: Training curves of all methods with varying numbers of preference labels on Push-5x5-1.
The solid line presents the mean values, and the shaded area denotes the standard deviations over five runs.
27

---

PEBBLE
PEBBLE (Ours)
(Roward)
Newwad
Rewaed
Rewed
-6
-8-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Buv.
Steps
x100
Rav. Stope
x10
Rev. Steps
x100
(a) Push-5x5-1 (feedback=300)
(b) Push-6x6-1 (fcedback=300)
(c) Push-7x7-1 (feedback=3000
12
12
Htwward
Reward
Reward
0.0
0.2
0.1
0.6
0.5
0.0
0.2
0.4
0.8
18
0.2
0.4
0.6
0.5
Bov. Steps
x10"
Rav. Steps
x10
Ene, Stege
x10"
(d) Push-5x5-2 (feedback=11i 1000)
(e) Push-6x6-2 (feedback=1000)
(1) Push-7x74 (feedback=1000)
Craftenv
12
Roward
Reward
Boward
0
0.2
0.4
0.8
0.0
0.2
0.6
0.8
0.0
0.4
0.6
Reve
Strpe
x10
Bov. Steps
x10"
Env.
Steps
x10*
(g) Strip-shaped (feodback=1000)
(h) Block-shaped (feedback=1000)
(i) Two-Story (feedback=1000)
Figure 11: Evaluating curves of all methods on locomotion tasks and robotic manipulation tasks. The solid line presents the mean values, while the shaded area indicates the 78% confidence interval over five runs. The red line our method.
E.2 Ablation Studies
Impact of Preference Quantity. To assess how the amount human preferences affects STAR performance, conducted additional experiment using varying numbers of preference labels: N E 50, 100, 300, 500 on Sokoban tasks. The training curves of the average episode return for all methods shown in Figure 12. The results reveal clear trend: as the number of preference labels increases, STAR 's policy performance steadily improves. Notably, with sufficient preference labels, STAR approaches the performance upper bound set by SAC, highlighting its efficiency in leveraging feedback data. This trend suggests that while STAR performs well with limited preferences, providing more feedback enables it closely match optimal performance. However, as the learning curves indicate, the rate of improvement begins to taper off at higher preference counts, suggesting diminishing returns beyond certain threshold.
Computational Efficiency of STAR. When deploying algorithms in real-world applications, training time overhead is critical factor. In discrete settings, we store vertices in dictionary using hashes keys, enabling efficient state retrieval in (l1) time. Additionally, updates the graph are only required at the end of each episode, further minimizing computational overhead. In continuous settings, STAR reduces computational costs by allowing the actor update lower frequency compared to traditional methods. To quantitatively assess the time overhead, Table 5 summarizes training times (in hours) STAR and baseline methods on Push-7x7-2 and Cheetah Run. For instance, STAR demonstrates only slight increase in training time compared to PEBBLE, with an overhead of less than 3%. In contrast, SURF incurs noticeable average increase of approximately 23.4% due to the additional time required for labeling samples. These results demonstrate that STAR
28

---

Table Training time (hour) of various tasks. Each run conducted using the exact same hardware environment.
Task/Methods
PEBBLE
SURF
MRN
STAR (ours)
Push-7x7-2
2.78
3.56 (+28.1%)
3.11 (+11.9%)
2.84 (+2.2 ))
Cheetah Run
3.23
3.83 (+18.6%)
3,42 (+5.9%)
3.31 (+2.5%
Table 5: Parameter search for policy regularization.
7
0.1
0.2
0.5
Quadruped Walk
790
763
728
643
620
Button Press
47
50
39
28
24
Sweep Into
55
58
52
49
48
Cheetah Run
710
713
697
579
542
maintains favorable balance between performance and training efficiency without significantly increasing computational costs.
Sensitivity Analysis of the Parameter Policy regularization is crucial component of STAR. In our experiments, we performed a parameter search for and selected a stable value within the mid-range of the search space. The table below presents the performance of STAR Cheetah Run for different values of 17. For all experiments conducted in online settings, we set 17 to 6, as shown in Table 8b.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state the claims made in this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss this paper's limitation and potential future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
30

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provide detailed proof of our theorem.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
he paper provides detailed descriptions of the model architecture, training procedure, dataset, and evaluation metrics, enabling reproduction of the main results without requiring access to the code.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
31

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We include source code in supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/ wtt.ggudddeeeeoeeuumiiisesssssse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips.co. cctps.ccc/pii haablic/guides////eeee /tt///ooeeSbiii iesionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide the experimental setup and implementation details in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
32

---

7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the mean values with confidence interval.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
Justification:
We report the compute resources used in experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: hett://neuiipee wwwpuurp..ccppulll..e cttppblic/cccccccccccccccccccclc /Ethi iesuuidelines"
Answer: [Yes]
33

---

The research adheres to the NeurIPS Code of Ethics, with no ethical concerns related to data usage, human subjects, or potential misuse identified.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release models or data with high risk for misuse; thus, no special safeguards are necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring
34

---

that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
The paper does not use third-party assets that require attribution or license disclosure.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re- e--aakkaedd, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
The paper does not use third-party assets that require attribution or license disclosure.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
35

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer LLM policy for what should or should not be described.
36


Reviews:
[{"review_id": "EtdJOeC3nR", "reviewer": "Reviewer_Q2Zo", "summary": "Preference Based Reinforcement learning allows for learning without explicit reward signal; however gathering these preferences may be expensive. When only a small number of preferences are available, it is very easy for the learned reward model to overfit. Moreover, Q value learning also suffers from over-estimation when being bootstrapped with limited amount of off-policy data. The paper introduces two complementary regularization techniques for these pain-points. \n\n(1) Regularization of the reward model to be smooth adn creates a bounded margin between preferred/rejected samples ; thus preventing the reward model form becoming overconfident. \n\n(2) Regularization of the policy to be close to a reference policy which is obtained by bootstrapping from a \"pessimistic\" estimate of Q value. This pessimistic estimate is obtained by only consider in-distribution actions from the replay buffer; \n\nThe proposed approach consistently outperforms the baselines by a decent margin in both offline and online settings while not adding any significant computation overhead.", "questions": "- How was the subset of environments form D4RL chosen for the offline setting experiments. \n- Equation 9 uses Q_ξ(s_{t+1}, a_{t+1}). However in first glance it seems like it should be a sample from the empirical behavioral distribution of the dataset. If it is so may be it is worthwhile pointing that it should be sampled from μ_empirical(·|s_{t+1}) and directly taking it from the transition is an approximation. \n- Do the authors think that regularization will be more or less helpful when the preferences are noisy? it could be interesting to see how the regularization helps or suffers when the noise is added systematically in the preferences.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths:*\n- The proposed regularizations in their approach(STAR) are well motivated for the problem space and has been experimentally shown to be complementary through the ablation studies. \n- STAR outperforms the baselines across synthetic offline datasets, online environments as well as human preference settings.   \n- The authors provide theoretic guarrantees that the conservative Q-function lowerbounds the optimal Q function in tabular settings. \n- Experiments on quality of learned reward, value estimation accuracy and effect of feedback quantities were very welcome additions. \n- The work overall takes a step towards making PbRL more practical for smaller preference budgets. \n\n*Weakness:*\n- The \"pessimism\" of the Q-function greatly depends on the quality of the offline dataset, as it is with respect to the behavioral policy. The dataset which is anchored to preferences may not always consist of optimal or near optimal rollouts. It could just be Very bad and Very good rollouts. It would have been nice to see experiments with a controlled set of datasets with different levels of diversity of the trajectories (in terms of cumulative rewards) in is used for bootstrapping. \n- Since the regularizations are both independently motivated, it could be interesting to see the performance of IQL with learned reward function with and without PMR. Comparing the performance of IQL + PMR with STAR could reveal more about the quality of policy regularization in STAR.", "was_revised": true, "final_justification": "I am inclined to accept. The problem of being able to learn from preference directly has extensive applications. The regularization techniques have shown to be complementary to each other and provides meaningful improvement. Moreover. authors have also provided evidence that it may be useful / complementary to other offline RL policy regularization as well.", "rebuttal": "We sincerely thank Reviewer Q2Zo for the insightful and constructive feedback. The suggestions to investigate the impact of each component in offline settings and to assess robustness to noisy feedback are particularly valuable and have significantly helped us improve the paper. In response, we have conducted additional experiments and provide detailed, point-by-point replies below. We believe these additions further reinforce the contributions of our work.\n\n**Q1. Regarding the impact of offline dataset quality on STAR’s performance (Weakness #1).**\n> **A1**: Thank you for this excellent point. We agree that this is a critical aspect, especially in the offline setting. We address this concern from two perspectives, aligning with our online and offline experiments:\\\n**(1) Online Setting**: The issue of a static, low-quality dataset is naturally mitigated. The replay buffer is continuously populated with new trajectories from the learning agent. As cited in our paper [1, 2], PbRL algorithm typically employ an unsupervised exploration at the very begining to encourage a diverse set of trajectories enters the buffer. Thus, the growing dataset provides a rich empirical distribution for $Q_\\xi$ to learn from, preventing it from being anchored to a narrow or polarized subset of behaviors.\\\n**(2) Offline Setting**: This is where your concern is most critical. We intentionally evaluated STAR on D4RL datasets with varying quality and diversity to address this (i.e., expert, replay, diverse). Our strong performance across these varied datasets (as shown in our original manuscript) demonstrates that our policy regularization is effective even when the behavior policy is not unimodal or near-optimal. We will clarify this aspect in our experiment section to highlight how our dataset choices directly test this robustness.\n\n**Q2. Regarding the effect of two components of STAR on offline settings (Weakness #2).**\n> **A2**: Thank you for this excellent suggestion to further disentangle the effects of our two components. Following your advice, we conducted a new ablation study in the offline setting, comparing STAR against IQL using a reward model trained with and without our Preference Margin Regularization (PMR).\n> The results are summarized below:\n> |Method|IQL w/o PMR|IQL w/ PMR|**STAR (Ours)**|\n> | -|-|-|-|\n> |antmaze-large-diverse-v2|18.8|31.5|58.7|\n> |hopper-medium-expert-v2 |63.8|78.1|96.8|\n> |lift-ph|89.2|93.7|98.0|\n>\n>These results lead to two clear and important conclusions: \\\n>**(1) PMR is a Generally Effective Component**: Adding PMR to the reward learning process significantly boosts the performance of a strong baseline like IQL (e.g., 18.8 -> 31.5 on AntMaze). This confirms that PMR effectively mitigates reward overfitting and is a valuable contribution in its own right. \\\n>**(2) Policy Regularization Provides Crucial, Synergistic Gains**: Even when IQL is augmented with the improved PMR-trained reward model, STAR still substantially outperforms it across all tasks. This decisively demonstrates that our policy regularization provides a critical advantage beyond simply using a better reward function. It effectively tackles the value overestimation problem that persists even with a regularized reward, confirming the synergistic and significant contribution of our full STAR algorithm.\n>\n>We will add this valuable ablation study to the appendix of our paper to make these points clear.\n\n\n**Q3. Selection criteria for D4RL benchmark environments (Questions #1).**\n> **A3**: We selected our D4RL environments based on a principled approach aimed at ensuring a rigorous, comprehensive, and fair evaluation. Our criteria were:\\\n**(1) Comparability**: We chose environments (e.g., AntMaze, Hopper) that are standard benchmarks in both offline RL and recent offline PbRL works [3, 4], ensuring our results are directly comparable to the state-of-the-art.\\\n**(2) Task Diversity**: The selected tasks span a wide range of challenges, including navigation (AntMaze), locomotion (Hopper), and manipulation (Lift), to demonstrate the general applicability of STAR.\\\n**(3) Data Quality Diversity**: Crucially, and connecting to your first question (Q1), we deliberately selected datasets with varying quality levels (replay, expert, diverse). This was essential to test STAR's robustness and its core ability to learn effectively from the kind of suboptimal, mixed-quality data that is common in real-world offline scenarios. This principled selection ensures that our claims are supported by evidence from a representative set of challenging and diverse offline settings.\n\n\n**Q4. Clarify using an approximation in Eq.(9) (Questions #2).**\n> **A4**: Thank you for this sharp and accurate observation. You are correct. Using the single next action $a_{t+1}$ from the transition tuple $(s_t, a_t, s_{t+1}, a_{t+1})$ in Eq.(9) is a practical, one-sample Monte Carlo approximation of the expectation over the dataset's empirical policy $μ(·|s_{t+1})$. This is a standard and computationally efficient practice in the field, adopted by prominent offline RL algorithms like CQL [5] and IQL [6], as it avoids the complexity and potential estimation errors of explicitly modeling the behavior policy. To improve the paper's precision, we will add a sentence in Section 3.2 to explicitly clarify this point as you've suggested.\n\n**Q5. Regarding robustness to noisy feedback (Question #3)**\n> **A5**: This is a very important question. While robustness to label noise was not the primary focus of our work, we agree it's a key challenge for real-world PbRL. To explore it, we conduct experiments on `Cheetah Run` and `Button Press` with ratios of noisy data as {0\\%, 5\\% 10\\%, 15\\%, 20\\%, 25\\%}. We systematically injected label noise (flipping preferences) and compared STAR with our backbone (PEBBLE).\n> \n>Table 1. Performance of STAR on Cheetah Run with various level noise.\n>|Method/Noise|0\\%|5\\%|10\\%|15\\%|20\\%|\n>|-|-|-|-|-|-|\n>|PEBBLE|612|423|/|/|/|\n>| STAR |713|702|629|455|/|\n>\n>Table 2. Performance of STAR on Button Press with various level noise.\n>\n>|Method/Noise|0\\%|5\\%|10\\%|15\\%|20\\%|25\\%|\n>|-|-|-|-|-|-|-|\n>|PEBBLE|23\\%|/|/|/|/|/|\n>| STAR |95\\%|80\\%|67\\%|67\\%|40\\%|/|\n>\n>These results demonstrate that STAR is more robust when the preferences are noisy. While both methods degrade as noise increases, STAR consistently and significantly outperforms PEBBLE. For example, at 15% noise, STAR maintains a strong performance of 67\\%, whereas PEBBLE's performance has dropped sharply. This demonstrates that STAR's framework indeed provides substantial robustness against noisy human feedback. We will add this experiment and analysis to the appendix.\n\n**References:**\\\n[1] Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. NeurIPS, 2022.\\\n[2] SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. ICLR, 2022. \\\n[3] Inverse Preference Learning: Preference-based RL without a Reward Function. NeurIPS, 2023.\\\n[4] In-dataset trajectory return regularization for offline preference-based reinforcement learning. AAAI, 2025.\\\n[5] Conservative q-learning for offline reinforcement learning. NeurIPS, 2020.\\\n[6] Offline Reinforcement Learning with Implicit Q-Learning. ICLR, 2022."}, {"review_id": "L96FcvoRBG", "reviewer": "Reviewer_vhLh", "summary": "The paper introduces STAR, a novel and efficient PbRL algorithm designed to improve learning from limited human feedback by addressing two major challenges: reward model overfitting and Q-value overestimation. To tackle these issues, STAR incorporates two key techniques: preference margin regularization to prevent overfitting and policy regularization to mitigate overestimation during policy updates. Extensive experiments across both online and offline benchmarks demonstrate the superior performance and robustness of STAR compared to existing methods.", "questions": "1. In line 120, the authors claim that optimizing Equation (2) (i.e., maximum likelihood objective), drives the reward model to assign infinite differences between preferred and rejected segments. However, this statement is confusing. As shown in Equation (9) of [2], the likelihood objective has a global maximum with a finite reward estimate. The authors should clarify why there will be an infinite difference.\n\n2. Figure 5(a) is confusing. I wonder what is the meaning of the red and blue colors? Initially, I think the red color corresponds to the rewards, and the blue color corresponds to the success rate. However, it seems that there are four environments as shown in the legend, regardless of the color.\n\n[2] Newman, Mark EJ. \"Efficient computation of rankings from pairwise comparisons.\" Journal of Machine Learning Research 24.238 (2023): 1-25.", "limitations": "NA", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n1. The paper is well-written and easy to follow.\n2. The experiments are extensive in both online and offline settings, together with some ablation studies.\n\n**Weakness**\n1. The innovation is limited. The contribution lies primarily in applying the established concepts (i.e., margin regularization and conservative Q-value estimation) to the PbRL setting rather than introducing fundamentally new algorithms or theoretical frameworks.\n\n2. Although the paper emphasizes the efficiency of STAR in limited feedback, it lacks thorough ablation studies analyzing how performance changes with varying numbers of preference-labeled trajectories. Ablation studies on the number of trajectories would be beneficial.\n\n3. Although the paper includes some theoretical analysis on conservative Q-value estimation, this aspect has already been extensively studied in prior works like [1], offering not too much novelty. More critically, the paper lacks theoretical guarantees specific to the PbRL setting. In particular, it is not evident whether the objective with the introduced margin can ensure the optimality of the learned reward. And it does not provide formal analysis or generalization bounds that account for the limited preference feedback.\n\n[1] Kumar, Aviral, et al. \"Conservative q-learning for offline reinforcement learning.\" Advances in neural information processing systems 33 (2020): 1179-1191.", "was_revised": true, "final_justification": "While the authors address some of my concerns, I still have some reservations regarding the novelty, theoretical analysis, and certain potentially incorrect claims, as partly acknowledged by the authors. After reviewing the other reviewers' comments, I have decided to maintain my current score. However, I am not opposed to acceptance given the merits of the paper.", "rebuttal": "We sincerely thank Reviewer vhLh for the detailed and critical feedback. We appreciate the acknowledgment of our paper's **clarity and extensive experiments**. The concerns raised about novelty, theoretical guarantees, and the ablation study on feedback are crucial, and they have prompted us to conduct new experiments and significantly clarify the positioning and contributions of our work. We provide detailed point-by-point responses below and believe the resulting discussion strengthens our paper considerably.\n\n**Q1: Reguarding ablation on feedback quantity (Weaknesses #2).**\n> **A1**: We have discussed the effect of feedback quantity on performance of STAR in Fig.6. To further validate STAR's feedback efficiency, we conduct the extensive ablation study on the impact of feedback quantity on task `Window Open`. The results across multiple tasks are presented below, which are averaged over five seeds, and will be added to the paper.\n> \n>Table 1. Walker Walk (measured by episode return, results in Fig.6 of our origial manuscript) .\n> \n>|Method| 100 | 200 | 500 | 1000|\n>| -----| --- | --- | --- | --- |\n>|PEBBLE|612±87|766±78|814±45|954±32|\n>| SURF |497±38|540±34|768±27|949±25|\n>| MRN  |695±33|831±25|884±24|956±25|\n>| STAR |752±31|839±24|936±22|960±21|\n>\n>Table 2. Window Open (measured by success rate).\n>\n>|Method| 100 | 500 | 1000| 2000|\n>| -----| --- | --- | --- | --- |\n>|PEBBLE|10±6 |31±33|55±42|80±24|\n>| SURF |20±20|32±25|42±26|85±13|\n>| MRN  |27±23|47±21|89±9|96±6|\n>| STAR |65±11|76±12|96±6|98±4|\n>\n>These results clearly show that STAR's performance advantage is most pronounced in the critical low-feedback regime. For instance, in Window Open with only 100 preferences, STAR achieves a 65% success rate, more than doubling the performance of the next best method (MRN at 27%). This directly and empirically substantiates our central claim of superior feedback efficiency. As feedback increases, all methods improve, but STAR consistently learns faster and more stably. We will add this crucial study to our appendix.\n\n\n**Q2. Regarding novelty and theory (Weaknesses #1, #3).**\n> **A2**: We thank the reviewer for these critical questions on novelty and theoretical guarantees. We would like to position our contribution more clearly.\n> 1. **On Novelty: A Synergistic Solution to a Dual Problem.** We agree that the core components (margin concepts, conservative Q-learning) are not new in isolation. Our primary contribution is not in inventing these primitives, but in:\nFirst, identifying and diagnosing the dual failure modes in feedback-limited PbRL: (1) reward model overfitting and (2) policy overestimation exploiting that flawed reward. Prior work has often addressed only one of these.\nSecond, proposing a novel and synergistic synthesis (STAR) that holistically addresses both issues. The novelty lies in the insight that these two problems are coupled and require a coupled solution. PMR creates a more robust reward signal, and our policy regularization ensures the agent does not exploit the residual uncertainty in that signal. This specific, motivated combination for PbRL is new and, as our experiments show, highly effective.\n2. **On Theoretical Guarantees for PbRL.**\nWe agree with the reviewer on both points and wish to clarify the scope of our analysis.\\\n**(1) Conservative Q-Estimation**: Our Theorem 1 is not intended to be a novel theoretical result on its own. As the reviewer correctly notes, the properties of conservative updates are well-studied [1]. The purpose of Theorem 1 is to provide a formal justification within our paper's context, showing that our specific in-distribution bootstrapping mechanism indeed yields a conservative value function, thereby grounding its use as a regularizer for the policy.\\\n**(2) Guarantees for PMR and Generalization**: Providing formal generalization bounds for a complex system coupling preference learning and deep RL is a widely acknowledged open problem, and such an analysis is beyond the scope of this primarily empirical and algorithmic work. However, our motivation for PMR is theoretically principled. Standard likelihood maximization (Eq. 2) presents an ill-posed, unbounded optimization target for a neural network approximator (more on this in Q3). PMR replaces this with a well-posed, bounded target ($log((1-ε)/ε)$). While not a proof of reward optimality, this is a principled way to improve generalization and prevent overfitting, which is a key theoretical motivation for its use.\\\nWe will revise the manuscript to state this positioning more explicitly, emphasizing that our contribution is a novel algorithmic framework for PbRL, supported by strong empirical results and justified by established theoretical principles.\n\n**Q3. The \"Infinite Reward Difference\" claim in Eq.(2) (Questions #1).**\n> **A3**: Thank you for this very sharp and important question. In the classical statistical setting described by the Bradley-Terry model (as in Newman [2]), for a given finite dataset with a strongly connected comparison graph, the global MLE indeed corresponds to a set of finite score values.\n> \n> In this deep learning context, the issue arises from the nature of the loss function for each training sample. The standard cross-entropy loss for a single preference pair $(\\sigma_0,\\sigma_1)$ is equivalent to maximizing $\\log P[\\sigma_0>\\sigma_1]=\\log(\\text{Sigmoid}(\\hat{r}(\\sigma_0)−\\hat{r}(\\sigma_1)))$. This loss is only minimized as the logit—the reward difference $\\Delta \\hat{r}=\\hat{r}(\\sigma_0)−\\hat{r}(\\sigma_1)$—approaches infinity. A powerful neural network has the capacity to continuously push $\\Delta \\hat{r}$ higher for the pairs in its training batches, leading to an over-confident model that fits the training data perfectly but generalizes poorly. This is the \"reward over-optimization\" problem we address. Therefore, our claim is not about the existence of a global finite MLE on a static dataset, but about the ill-posed nature of the optimization target for a neural network approximator, which drives it towards unbounded differences. Our PMR directly tackles this by replacing the unbounded target with a principled, finite margin, thus making the learning problem well-posed and preventing overfitting. We sincerely apologize that this critical distinction was not made clear in our manuscript. We will revise Section 3.1 to explicitly state this, clarifying that our analysis pertains to the optimization of neural network-based reward models in deep PbRL, and discuss the connection to the classical view.\n\n**Q4. The \"Infinite Reward Difference\" claim in Eq.(2) (Questions #1).**\n> **A4**: Thank you for this sharp and important question. Your point is correct and gets to the heart of a subtle but critical distinction we failed to make clear in the paper. In the classical setting of the Bradley-Terry model on a static, connected dataset (as in Newman [2]), the MLE corresponds to finite scores.\n>\n> And our claim refers specifically to the optimization dynamics of a high-capacity neural network used as a reward function approximator. In this setting, the key issue is that the standard cross-entropy loss (Eq.(2)) drives the optimization towards an unbounded reward difference. As you insightfully noted, the objective \"wants\" the reward difference to be infinite because that is the only way for the loss on a given sample to approach zero.\n> While the reward values do not become literally infinite in practice, this unbounded optimization tendency is precisely what we define as an \"ill-posed\" objective. It encourages the model to become over-confident in its predictions for training data, leading to poor generalization and the reward over-optimization problem we aim to solve.\n> Our Preference Margin Regularization (PMR) directly addresses this by replacing the unbounded target with a principled, finite margin. This regularizes the model by giving it a well-posed, achievable target, thus preventing it from overfitting to the training preferences.\n>\n>We are grateful for this opportunity to clarify. We will revise Section 3.1 to explicitly draw this distinction between the classical statistical view and the deep learning optimization view, and incorporate this more precise explanation.\n\n**Q5. Clarification on Figure 5(a) (Questions #2)**\n> **A5**: We sincerely apologize for the confusion caused by Figure 5(a). The presentation was indeed unclear. Your initial interpretation was on the right track. The figure plots performance for four distinct tasks, but uses two different metrics mapped to two y-axes:\\\n**Red lines (left y-axis)**: Represent tasks measured by episode return (Cheetah Run, Quadruped Walk).\\\n**Blue lines (right y-axis)**: Represent tasks measured by success rate (Button Press, Sweep Into).\n>\n>This design was chosen to compactly display results but clearly sacrificed clarity. We will completely redesign this figure in the revised manuscript to ensure there is no ambiguity. Thank you for pointing this out.\n\n\n**References:**\\\n[1] Kumar, Aviral, et al. \"Conservative q-learning for offline reinforcement learning.\" Advances in neural information processing systems 33 (2020): 1179-1191.\\\n[2] Newman, Mark EJ. \"Efficient computation of rankings from pairwise comparisons.\" Journal of Machine Learning Research 24.238 (2023): 1-25."}, {"review_id": "6w6NbEBxNY", "reviewer": "Reviewer_qN2G", "summary": "This paper proposes STAR, an efficient preference-based reinforcement learning (PbRL) algorithm that addresses key challenges associated with limited human feedback—specifically, reward model overfitting and Q-value overestimation. STAR introduces two novel techniques: Preference Margin Regularization (PMR), which imposes a bounded margin to prevent the reward model from overfitting to limited preferences, and Policy Regularization, which uses a conservative Q-value derived from in-distribution data to guide stable policy updates. Through extensive experiments in both online and offline settings, STAR demonstrates superior feedback efficiency, outperforming existing state-of-the-art methods. Additionally, STAR enables the emergence of diverse and complex agent behaviors using minimal human supervision, making it a practical solution for real-world PbRL applications.", "questions": "B-Pref [1] has experimentally shown that label smoothing has limited effectiveness in PbRL. Then, what explains the performance gains observed with label smoothing in STAR? What key differences in STAR's design make label smoothing effective in this case?", "limitations": "Same as mentioned in the weakness section", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- Strengths\n    - The paper is structurally well-organized and easy to follow.\n    - The authors clearly explain the challenges of PbRL they aim to address and propose appropriate methods to tackle them.\n    - The paper effectively explains how the proposed method helps mitigate reward overfitting and Q-value overestimation..\n    - STAR demonstrates strong performance compared to baselines in both online and offline settings.\n- Weakness\n    - Limited novelty: The label smoothing technique used in PMR has already been explored in many prior PbRL studies [1] , and both Conservative Q-learning and policy regularization closely follow methods commonly used in offline RL.\n    - The experimental results for offline RL are insufficient. There has been considerable criticism in prior studies that D4RL is not suitable for evaluating PbRL algorithms, as strong performance can often be achieved even with simple reward functions[2-5]. Additionally, in the Robosuite tasks, STAR underperforms compared to IPL. Therefore, to convincingly demonstrate the effectiveness of STAR in offline settings, more experiments across diverse environments are needed.\n    - As shown in Figure 5(a), performance varies significantly depending on the parameter λ, which suggests that STAR is sensitive to hyperparameter settings.\n\n\n[1] Lee, Kimin, et al. \"B-pref: Benchmarking preference-based reinforcement learning.\" *arXiv preprint arXiv:2111.03026* (2021).\n\n[2] Shin, Daniel, Anca D. Dragan, and Daniel S. Brown. \"Benchmarks and algorithms for offline preference-based reward learning.\" arXiv preprint arXiv:2301.01392 (2023).\n\n[3] Li, Anqi, et al. \"Survival Instinct in Offline Reinforcement Learning.\" arXiv preprint arXiv:2306.03286 (2023).\n\n[4] Hu, Hao et al. “Unsupervised Behavior Extraction via Random Intent Priors.” arXiv preprint arXiv:2310.18687 (2023).\n\n[5] https://openreview.net/forum?id=EG68RSznLT&noteId=p4vdRSdjnC", "was_revised": true, "final_justification": "I have understood all of the author’s responses. There is no change from the initial score, and I will continue to maintain the tendency to accept.", "rebuttal": "We sincerely thank Reviewer qN2G for the constructive feedback and for recognizing our clear problem formulation, methodological contributions, and strong empirical results. We appreciate your positive assessment of the paper’s organization and clarity, as well as the effectiveness of STAR in addressing key challenges in PbRL. We address your concerns point-by-point below.\n\n**Q1. Regarding novelty and contribution (Weakness #1).**\n> **A1**: Thank you for this critical point on novelty. While we build upon established principles, we respectfully argue that STAR's novelty lies not in inventing its individual components, but in: \\\n> **First, identifying and diagnosing the dual failure modes in feedback-limited PbRL:** (1) reward model overfitting and (2) policy overestimation exploiting that flawed reward. Prior work has often addressed only one of these.\\\n**Second, proposing a novel and synergistic synthesis (STAR) that holistically addresses both issues.** The novelty lies in the insight that these two problems are coupled and require a coupled solution. PMR creates a more robust reward signal, and our policy regularization ensures the agent does not exploit the residual uncertainty in that signal. This specific, motivated combination for PbRL is new and, as our experiments show, highly effective.\n\n\n**Q2. Regarding Offline RL Experiments (Weakness #2).**\n> **A2**: Thank you for raising these important concerns about our offline evaluation. We acknowledge the ongoing community discussion regarding the suitability of some D4RL tasks for PbRL evaluation. We chose our experimental setup for the following principled reasons:\\\n> **(1) Comparability**: Using standard D4RL benchmarks is crucial for ensuring that our results are directly comparable to a large body of prior and concurrent work in offline PbRL [1, 2]. This allows for a fair assessment of STAR's standing in the field.\\\n> **(2) Demonstrating Clear Improvement**: While some D4RL tasks might be solvable with simple reward functions, our experiments show that existing SOTA offline PbRL methods still struggle on them, leaving a significant performance gap. For example, on `antmaze-large-diverse`, STAR achieves a score of 48.0, whereas the next best baseline is far lower (PT:19.6). This demonstrates that these tasks are not yet \"solved\" by current offline PbRL methods and that our contributions lead to substantial, meaningful gains over existing approaches.\\\n> **(3) Performance on Robosuite**: Regarding the Robosuite results, we wish to clarify the performance comparison. While IPL shows strong performance, STAR is highly competitive, achieving a average score of 77.0 vs. IPL's score of 79.3. This narrow margin on a complex manipulation task, combined with our superior performance on D4RL, showcases STAR's overall robustness. \n\n\n**Q3. Hyperparameter Sensitivity (Weakness #3).**\n> **A3**: This is an excellent observation regarding hyperparameter sensitivity. The plot in Figure 5(a) was intentionally designed to show the effect of the regularization parameter $\\lambda$ across a wide range to provide a complete picture of its impact. However, in practice, the effective operational range for such regularization parameters is typically narrow. As the plot shows, while extreme values can degrade performance, there is a stable and robust region (e.g., $\\lambda \\in [0.05,0.1]$ in our experiments) where STAR consistently achieves high performance. This level of sensitivity is common for regularization methods in deep RL. To make this clearer, we will revise the caption of Figure 5(a) to highlight this \"effective operational range\" and clarify that the wide range was chosen for analytical purposes, while the method remains robust within a practical tuning range.\n\n\n**Q4. Why does PMR work when B-Pref's label smoothing didn't? (Questions #1)**\n> **A4**: This is a fantastic and critical question that gets to the heart of our contribution. There are two key reasons why our PMR succeeds while the label smoothing in B-Pref [3] showed limited effect:\\\n> **(1) Different Experimental Focus:** The analysis in B-Pref [1] primarily focused on robustness to label noise (mistake feedback) with a relatively large amount of feedback. Our work, in contrast, is focused on improving feedback efficiency in the low-feedback regime, where overfitting is the dominant challenge. Label smoothing's primary benefit is as a regularizer against overfitting, an effect that is most crucial and visible when data is scarce.\\\n> **(2) Synergy with Policy Regularization (The Key Difference):** This is the most important reason. In STAR, PMR does not work in a vacuum. It is one half of a synergistic pair.\nB-Pref [1] applied label smoothing to the reward model but used PEBBLE as PbRL algorithm, which is not rubust to noise feedback.\n>\n> STAR's policy regularization is designed to solve this exact problem. PMR first creates a smoother, more generalizable reward function. Then, our policy regularization explicitly prevents the agent from greedily exploiting this (now better, but still imperfect) reward landscape.\n\n**References:**\\\n[1] Inverse Preference Learning: Preference-based RL without a Reward Function. NeurIPS, 2023.\\\n[2] In-dataset trajectory return regularization for offline preference-based reinforcement learning. AAAI, 2025.\\\n[3] B-Pref: Benchmarking Preference-Based Reinforcement Learning. NeurIPS, 2021."}, {"review_id": "j9WqkV6Maz", "reviewer": "Reviewer_LJRt", "summary": "This paper presents a novel feedback-efficient PBRL framework STAR. The authors identify two critical challenges in PbRL under limited human feedback: reward model overfitting and Q-value overestimation, which lead to unstable policy learning. To address these, STAR integrates preference margin regularization to mitigate reward overfitting and policy regularization based on conservative Q-value estimates to reduce overestimation bias. The experiments in both online and offline settings demonstrate the effectiveness of STAR.", "questions": "NA", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.The issue of reward overfitting addressed in this paper is both interesting and significant.\n\n2.The proposed method can be effectively applied in both online and offline settings simultaneously.\n\nWeaknesses:\n\n1.The policy regularization and conservative Q-estimation techniques proposed in this paper are entirely derived from [1], raising concerns about the novelty of the approach.\n\n2.The proposed method introduces three hyperparameters: $\\epsilon$ in reward learning, $\\beta$ in conservative Q and $\\eta$ in policy regularization. The high complexity of these parameters raises concerns about the method's generalizability. While the paper provides an analysis of the sensitivity to $\\epsilon$ (although earlier the paper uses $\\epsilon$ while in the ablation study, it is referred to as $\\lambda$), there is no sensitivity analysis for the other hyperparameters, and the experimental results reveal the vulnerability of $\\epsilon$.\n\n3.There is a lack of comparison with recent state-of-the-art methods. The paper only compares with methods prior to 2022 (with only IPL being a 2023 method). In online settings, newer methods such as QPA [2], and in offline settings, advanced methods like LiRE [3] (which is especially relevant as it also improves reward learning) and DTR [4], should be compared for a more comprehensive evaluation.\n\n4.The ablation study only compares the contribution of each component in the online setting but does not discuss the offline setting.\n\n5.The implementation is based on TD3, whereas most other methods are implemented using SAC. It is apparent that the random strategy is more convenient for calculating Eq. (8) and Eq. (10). The authors do not provide relevant analysis or experimental justification for this choice.\n\n[1] Garg, D., Hejna, J., Geist, M., & Ermon, S. (2023). Extreme q-learning: Maxent rl without entropy. ArXiv Preprint ArXiv:2301.02328.\n\n[2] Hu, X., Li, J., Zhan, X., Jia, Q.-S., & Zhang, Y.-Q. (2023). Query-policy misalignment in preference-based reinforcement learning. ArXiv Preprint ArXiv:2305.17400.\n\n[3] Zhu, M., Liu, Y., Zhang, L., Guo, J., & Mao, Z. (2024). LIRE: listwise reward enhancement for preference alignment. ArXiv Preprint ArXiv:2405.13516.\n\n[4] Tu, S., Sun, J., Zhang, Q., Zhang, Y., Liu, J., Chen, K., & Zhao, D. (2025). In-dataset trajectory return regularization for offline preference-based reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 39(20), 20929–20937.", "was_revised": true, "final_justification": "The authors provided detailed response and provided more crucial experiment results, which solves my major concern. If the authors commit to incorporating the suggested detailed revisions in the final version, I am willing to support this paper. Accordingly, I have raised my score to 4.", "rebuttal": "We sincerely thank Reviewer LJRt for the detailed feedback. We recognize the concerns raised regarding novelty, hyperparameter complexity. We have taken these points very seriously and, in response, have conducted four extensive new sets of experiments to address every major point. This includes: \\\n(1) New comparisons with the SOTA methods QPA and DTR.\\\n(2) New ablation study on hyperparameter sensitivity.\\\n(3) New ablation study on the contribution of each component in the offline setting.\\\n(4) New implementation and evaluation of STAR with an SAC backbone. \\\nWe believe these significant additions substantially strengthen our paper and directly address the reviewer's concerns, which we detail below.\n\n**Q1. Regarding novelty (Weakness #1).**\n> **A1**: We thank the reviewer for this critical point on novelty. We agree that our policy regularization component draws inspiration from prior work in offline RL, including XQL [1], which we cite. While we build upon established principles, we respectfully argue that STAR's novelty lies not in inventing its individual components, but in:\\\n> **First, identifying and diagnosing the dual failure modes in feedback-limited PbRL:** (1) reward model overfitting and (2) policy overestimation exploiting that flawed reward. Prior work has often addressed only one of these.\\\n**Second, proposing a novel and synergistic synthesis (STAR) that holistically addresses both issues.** The novelty lies in the insight that these two problems are coupled and require a coupled solution. PMR creates a more robust reward signal, and our policy regularization ensures the agent does not exploit the residual uncertainty in that signal. This specific, motivated combination for PbRL is new and, as our experiments show, highly effective.\n\n**Q2. Hyperparameter Sensitivity (Weakness #2).** \n> **A2**: Thank you for pointing out the hyperparameter complexity and the lack of sensitivity analysis. We apologize for the typo regarding $\\epsilon$ with $\\lambda$, it should be $\\lambda$ throughout the paper, and we will correct this in the revision. \n> \n> **(1) To address your concerns, we have conducted new sensitivity analyses for both $\\beta$ and $\\eta$. The results are shown below.**\n>\n> Table 1. Sensitivity to $\\beta$.\n>|$\\beta$| 0  | 1 | 5 | 10|\n>| ----- | -- | --|-- |-- |\n>|Cheetah Run |685|757|713|697|\n>|Button Press|45|53|56|52|\n>\n>Table 2. Sensitivity to $\\eta$.\n>| Task | 0 | 0.1 | 1 | 5 | 10 |\n>| -----|---|-----|---|---|----|\n>| Quadruped Walk|620 |643 |728 |763 |790|\n>| Button Press  |24  |28  |39  |50  |47 |\n>| Sweep Into    |48  |49  |52  |54  |55 |\n>\n>These results show that while performance is indeed sensitive to hyperparameters, there are broad, stable regions for each parameter where STAR performs robustly and effectively. For instance, performance is strong for $\\beta\\in[5, 10]$, $\\eta\\in[5,10]$. This demonstrates that STAR does not require excessively fine-tuning to be effective. We will add these new ablation studies to the appendix.\n>\n> **About $\\lambda$.** The plot in Figure 5(a) was intentionally designed to show the effect of the regularization parameter $\\lambda$ across a wide range to provide a complete picture of its impact. However, in practice, the effective operational range for such regularization parameters is typically narrow. As the plot shows, while extreme values can degrade performance, there is a stable and robust region (e.g., $\\lambda \\in [0.05,0.1]$ in our experiments) where STAR consistently achieves high performance. This level of sensitivity is common for regularization methods in deep RL. To make this clearer, we will revise the caption of Figure 5(a) to highlight this \"effective operational range\" and clarify that the wide range was chosen for analytical purposes, while the method remains robust within a practical tuning range.\n\n\n**Q3. Comparison with other SOTA (Weakness #3).** \n> **A3**:  Thank you for this crucial feedback on our experimental comparisons. Following your suggestions, we have run new experiments comparing STAR against the strong recent baselines you suggested: QPA [2] for online settings and DTR [3] for offline settings.\n> \n> **(1) Online Comparison vs. QPA [2].** We ran QPA in our low-feedback settings and STAR in the original QPA settings for a fair and comprehensive comparison.\n>\n>Table 1. Performance on QPA and STAR. For each task, the number of feedback used is indicated in parentheses.\n>|Task|QPA|STAR|\n>|-|-|--|\n>|Cheetah Run (100)|642|713|\n>|Walker Walk (200)|823|839|\n>|Quadruped Walk (1000)|625|796|\n>|Button Press (100)|45|95|\n>|Window Open (100)|36|65|\n>|Sweep Into (100)|54|83|\n>|\n>|Walker Run (250)|618|762|\n>|Quadruped Run (1000)|492|575|\n>|Drawer Open (3000)|65|73|\n>|Door Open (3000)|81|87|\n>\n>**(2) Offline Comparison vs. DTR [3]:** We compared STAR with the reported DTR results on the standard D4RL locomotion tasks. We follow the setup in DTR, where each task use 2000 feedback from Uni-RLHF.\n>\n>Table 2. Performance on DTR and STAR.\n>\n> |Task|DTR|STAR|\n> |-|-|--|\n> |hopper-medium-replay-v2|93.1|95.2|\n> |hopper-medium-expert-v2|101.2|112.2|\n> |hopper-medium-v2|89.2|91.2|\n> |walker2d-medium-replay-v2|70.4|74.5|\n> |walker2d-medium-expert-v2|107.2|111.9|\n> |walker2d-medium-v2|87.9|86.2|\n> |halfcheetah-medium-replay-v2|38.5|42.0|\n> |halfcheetah-medium-expert-v2|82.0|92.6|\n> |halfcheetah-medium-v2|41.7|44.4|\n> |Average|79.0|83.5|\n>\n> These new results clearly demonstrate that STAR significantly outperforms even the most recent and powerful SOTA methods in both online and offline settings. We were unable to run LiRE [4] due to the lack of a public implementation at the time of rebuttal, but we will add it to our related work and acknowledge it as a relevant concurrent approach. These new experiments have been added to the main paper.\n\n**Q4. Offline Ablation Study (Weakness #4).**\n> **A4**: This is an excellent point. Our original submission was missing this analysis. Following your advice, we have conducted a new ablation study on the contribution of each component in the offline setting. We compare STAR against variants without Preference Margin Regularization (w/o PMR) and without Policy Regularization (w/o PR).\n> The results are summarized below:\n> |Method|STAR w/o both|STAR w/o PMR|STAR w/o PR|STAR|\n> | -|-|-|-|-|\n> |antmaze-large-diverse-v2|18.8|31.5|36.7|58.7|\n> |hopper-medium-expert-v2 |63.8|78.1|84.3|96.8|\n> |lift-ph|89.2|93.7|96.2|98.0|\n>\n>The results clearly show that both components provide a significant and complementary contribution to performance in the offline setting. Removing either PMR or PR leads to a substantial performance drop, and removing both results in very poor performance. This confirms that the synergy between our two regularizers is critical for offline success. We will add this study to our appendix.\n\n**Q5. Choice of TD3 over SAC (Weakness #5).**\n> **A5**: Thank you for this insightful question about our choice of backbone algorithm. We chose TD3 for our primary implementation for two main reasons:\\\n>**(1) Algorithmic Simplicity:** TD3's deterministic policy update aligns more directly with the derivation of our policy regularization, which constrains the policy towards a reference policy derived from the conservative $Q_\\xi$. It avoids the additional complexity of managing the entropy term found in SAC.\\\n> **(2) Focus of Contribution**: Our goal was to demonstrate the effectiveness of our dual-regularization framework. Using a simpler, well-understood backbone like TD3 helps to clearly isolate the performance gains attributable to our proposed STAR components, rather than conflating them with the effects of entropy maximization. However, to demonstrate the generality of our approach, we have now implemented and evaluated an SAC-based version of STAR. The results show that our framework is highly effective with either backbone.\n>\n>|Task|STAR (SAC)|STAR(TD3)|\n>|-|-|-|\n>|Cheetah Run|728|713|\n>|Quadruped Walk|756|796|\n>|Button Press|92|95|\n>|Window Open |74|65|\n>\n>The results show that STAR is robust to the choice of the base RL algorithm, achieving strong performance with both SAC and TD3 backbones. We will add this experiment and discussion to the appendix.\n>\n**Reference:**\\\n[1] Extreme Q-Learning: MaxEnt RL without Entropy. ICLR, 2023.\\\n[2] Query-Policy Misalignment in Preference-Based Reinforcement Learning. ICLR 2024.\\\n[3] In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning. AAAI 2025.\\\n[4] LIRE: listwise reward enhancement for preference alignment."}]

Author rebuttal:

Meta-review: The paper looks at the issue of overfitting in the case of preference based learning. They break down the problem into two aspect: (a) overfitting in the reward-modeling phase, and (b) exaggeration of the q-value over-estimation issues through this during the RL phase. To remedy this, inspired by existing works on margin regularization, authors propose leveraging margin based BCE loss for reward modeling. Further, to remedy the over-estimation bias, authors propose using CQL (Kumar et al. 2020) with the reward model learned with margin regularization.

Reviewers mostly praised the work. Some questions were raised regarding, offline data selection, environment selection, feedback noise, hyper-paramater sensitivity, etc. Authors engaged in a constructive discussion and addressed most of the issues.

In terms of technical novelty, as some reviewers noted, the contributions are modest. However, I commend the authors for systematically studying the problem:

- exposition of the issue and presentation is well done.
- flow of arguments are logical and clear.
- Both aspects of the contribution are evaluated individually + together, and additional hyper-param ablations are presented.
- Several benchmarks and multiple baselines are used.
- I would have appreciated more than 5 trials for at least some of the smaller benchmarks. However, given that over a dozen domains have been used and the trend of the results are in the right direction, I think these results are still useful and will be of interest to the audience.

Minor: I would request the authors to replace the word “dual” with “double” in the title, as “dual regularization” sounds like using dual variables to do some regularization, which is not the case.
Decision: Accept (poster)