Policy Likelihood-based Query Sampling and Critic-Exploited Reset for Efficient Preference-based Reinforcement Learning
Authors: Jongkook Heo
Jaehoon Kim
Young Jae Lee
Min Gu Kwak
Youngjoon Park
Seoung Bum Kim
Venue: ICLR 2026 Poster (2026)
Keywords: preference-based reinforcement learning
robotic manipulation
locomotion
Primary area: reinforcement learning
TL;DR: We present an efficient preference-based reinforcement learning method which selects informative queries using policy likelihoods and mitigates reward overestimation caused by primacy bias.

Abstract: Preference-based reinforcement learning (PbRL) enables agent training without explicit reward design by leveraging human feedback. Although various query sampling strategies have been proposed to improve feedback efficiency, many fail to enhance performance because they select queries from outdated experiences with low likelihood under the current policy. Such queries may no longer represent the agent's evolving behavior patterns, reducing the informativeness of human feedback. To address this issue, we propose a policy likelihood-based query sampling and critic-exploited reset (PoLiCER). Our approach uses policy likelihood-based query sampling to ensure that queries remain aligned with the agent’s evolving behavior. However, relying solely on policy-aligned sampling can result in overly localized guidance, leading to overestimation bias, as the model tends to overfit to early feedback experiences. To mitigate this, PoLiCER incorporates a dynamic resetting mechanism that selectively resets the reward estimator and its associated Q-function based on critic outputs. Experimental evaluation across diverse locomotion and robotic manipulation tasks demonstrates that PoLiCER consistently outperforms existing PbRL methods. Our code is available at https://github.com/JongKook-Heo/PoLiCER.

Published as a conference paper at ICLR 2026
POLICY LIKELIHOOD-BASED QUERY SAMPLING AND CRITIC-EXPLOITED RESET FOR EFFICIENT PREFERENCE-BASED REINFORCEMENT LEARNING
Jongkook Heo', Jachoon Kim', Young Jae Lee², Min Gu Kwak?, Youngjoon Park',
Scoung Bum Kim!"
School of Industrial and Management Engineering, Korea University, 'Samsung Electronics,
3 Department of Health Information Management, University of Pittsburgh, 'L.G Al Research
1 (hikso1406, jhoon0418, sbkiml (erimm)ekoree wekorea.........................
2 jae601 leelleellle @eeessmungg..... com, "mix4140pits edu, "yj.parkelgresearchh
ABSTRACT
Preference-based reinforcement learning (PbRL) enables agent training without explicit reward design by leveraging human feedback. Although various query sampling strategies have been proposed to improve feedback efficiency, many fail to enhance performance because they select queries from outdated experi- ences with low likelihood under the current policy. Such queries may no longer represent the agent's 's evolving behavior patterns, reducing the informativeness of human feedback. To address this issue, we propose a policy likelihood baseee query sampling and critic-exploited reset (PoLiCER). Our approach uses policy likelihood-based query sampling to ensure that queries remain aligned with the agent's evolving behavior. However, relying solely on policy-aligned sampling can result in overly localized guidance, leading to overestimation bias, as the model tends to overfit early feedback experiences. To mitigate this, PoLiCER incorporates dynamic resetting mechanism that selectively resets the reward es- timator and its associated Q-function based on critic outputs. Experimental evalu- ation across diverse locomotion and robotic manipulation tasks demonstrates that PoLiCER consistently outperforms existing PbRL, methods. Our code available at wthttpp://gitthi com/ Jong/ook-gooooooo Heoo/PoiiEl
INTRODUCTION
Reinforcement learning from human feedback (RLHF) has emerged as powerful paradigm for learning complex behaviors without manually designing reward functions, with applications across diverse domains (Kaufmann et al. 2024; Ouyang et al. 2022 Xu et al. 2023). In the context of Markov decision processes (MDPs), this approach often referred to as preference-based reinforce- ment learning (PbRL) (Wirth et al. 2017). In PbRL, annotator compares segjettrress-Sequenee of state-action pairs- from the agent's 's history and indicates which preferred (Christiano et al.) 2017). Despite its advantages, PbRL faces two major challenges stemming from its dependence on sequentially collected human feedback. First, query-policy misalignment arises when sampled queries longer represent the current policy's behavior, reducing the feedback effectiveness (Hu et al. 2024). Second, primacy bias causes the reward estimator to overweight early feedhack, creat- ing persistent biases that distort the reward model throughout training.
Hu et al. (2024) argued that query-policy misalignment occurs when queries are selected from long- past experiences that no longer represent the current policy. This reliance on outdated information diminishes feedback effectiveness and limits its ability to guide policy updates. To address this is- sue, Hu et al.] (2024) introduced near on-policy sampling (NOS), which prioritizes queries from recent agent experiences based on the assumption that recency correlates with the current policy's behavior. While NOS proves effective in environments with repetitive behaviors, such as loco-
"Corresponding author.

---

Published as a conference paper at ICLR 2026
motion tasks (Tassa et al. 2018 T Lunyasuvunakool et al. 2020 where recent experiences remain
representative, it struggles in coal-conditioned environments like Meta- World Yu et al. 2020 In these settings, dynamic target locations make re- cent experiences unreliable indicators of current pol- icy. Figure illustrates this limitation by tracking policy likelihoods of replay buffer episodes during training across three categories: all past episodes (AlI), the most recent 30 episodes (Last 30), and the top 30 episodes ranked by current policy likelihood (Top 30). While recent episodes show better align- ment with the current policy than the full history, a widening gap emerges between recent and high- likelihood episodes, demonstrating that temporal re- cency becomes an increasingly poor proxy for pol- icy relevance. To overcome this limitation, we pro- pose policy likelihood-basee which selects queries based on their likelihood under the current policy rather than temporal recency. Our method ensures that selected queries remain relevant throughout training by continuously adapting the evolving policy.
Epicodes
Last Pyisodos
Top 30 Epicodes
query sampling (PLS),
15:00
Envfronment Steps
Figure 1: Policy likelihoods of replay buffer episodes in Meta-World Sweep Into (10,000/50), categorized into all past episodes (All), recent 30 episodes (Last 30), and top 30 episodes into current policy likelihood (Top 30). Temporal recency becomes an increasingly poor indicator of policy relevance as training progresses.
The second critical challenge in PbRL is primacy tendency of neural networks to overfit to early data when trained on a growing dataset (Nik-
ishin et al. 2022, Lyle et al., 2022 2023 Kaufmann
jet al., 2024). PbRL, the reward estimator suffers
from this bias it learns from continuously collected feedback. Early feedback. which is often trivial or ambiguous (Mu et al. 2025 Tu et al., (2055). can dominate the entire learning process. We found that this bias leads reward overestimation: the reward estimator assigns excessively high values to early-stage favored state- acttinstions, even when later feedback contradicts them. In particular, identified that solely relying PLS can highly localize the learning process, narrowing the pol- icy distribution and further exacerbating overestimation. The biased rewards then propagate to the Q-function, reinforcing outdated behaviors and distorting policy optimization.
To demonstrate that primacy bias in the reward estimator causes reward overestimation. we com- pared PEBBLE with PEBBLE + Reset-a variant that resets the reward estimator parameters at ev- ery feedback session, following standard primacy bias mitigating approaches (Ash & Adams, 2020) Nikishin et al.] [0222, D'Oro et al.) [2022). We quantified reward overestimation using the rate as (R - R)RRRRRRR 100%, where R and R denote the scaled estimated and true episode rewards, respec- tively. Figure 2 shows that periodic resetting sub- stantially improves performance while reducing re- ward overestimation. Without resetting, PEBBLE consistently overestimates rewards, plateauing at ap- proximately 150%. Furthermore. PEBBLE fails to distinguish fine- ine-grained differences among trajecto- ries, particularly for low-reward cases where reward overestimation is most severe (see Figure 14 in Appendix[D.2 [.2)p. These results demonstrate how early, potentially misleading feedback can dominate the entire learning process.
-Rmm
X0W03 2.000 (a) Midicn Epcsode Retum
X.900 250000
50/00
[h) Middiz Rewand Ovenstiaaoon Hate
Figure 2: Performance comparison of PEB- BLE and PEBBLE Reset. (a) Average episode return and (b) reward overestimation rate on Walker Walk (100/10), showing me- dian across ten runs. Primacy bias in the re- ward estimator leads to reward overestima- tion and performance degradation in PbRL.
While resetting the reward estimator at every feedback session proves effective at mitigating pri- macy bias-driven reward overestimation, this technique introduces two key challenges: (1) the high computational cost associated with frequent resets and (2) residual bias in the corresponding Q- function, which persists due to outdated reward estimates reinforced through temporal difference (TD) learning. To overcome these challenges, we propose critic-exploited reset (CER), a dynamic
2

---

Published as a conference paper at ICLR 2026
resetting strategy that resets both the reward estimator and Q-function based on critic outputs. The critic output serves as an indicator of overestimation by approximating expected cumulative rewards (Van Hasselt et al. 2016 Nikishin et al. 2022 Chen et al. 202I). By strategically monitoring critic outputs, we trigger resets only when necessary, thus mitigating bias while maintaining computa-
tional efficiency.
In summary. we address two key challenges in PbRL: query-policy misalignment in query sampling and primacy bias in reward estimator learning. We propose policy likelihood-based query sampling and critic-exploited reset (PoLiCER), with each component targeting a specific challenge. First, PLS ensures queries remain aligned with the current policy by selecting trajectories based on their likeli- hood rather than temporal recency. Second, CER mitigates primacy bias through dynamic resetting of both the reward estimator and Q-function based on critic outputs, providing computational eff- ciency while addressing residual bias. We evaluated Pol. LiCER on complex continuous control tasks in the DeepMind Control Suite (DMControl) (Tassa et al.) 2018 [uunyasuvunakool et al., (020) and Meta-World (Yu et al. 2020). Comprehensive experimental results demonstrate that PoLiCER sig- nificantly improves performance when integrated with PEBBLE (Lee et al. (021b) and outperforms other PbRL methods with complementary components.
2 RELATED WORKS
Preference-based Reinforcement Learning. Christiano et al. (2017) established the foundation for training deep RL agents using human preferences (Mnih et al. 2016 Schulman et al.) 2017).
Lee et al. b02 built upon this work, incorporating soft actor-critic (SAC) (Haarnoja et al. 2018
unsupervised pre-training. and reward relabeling enhance sample efficiency. Subsequent studies have pursued various directions to improve feedback efficiency: SURF (Park et al. (2022) leveraged unlabeled queries through semi-supervised reward learning. RUNE (Liang et al.][222 used reward
uncertainty for exploration, and MRN (Liu et al. 2022) applied bi-level optimization to reward learning. Other advances include robust learning under noisy preferences (Cheng al. al.] [202]] Huang et al., 2025), dynamics modeling (Metcalf et al.] (002),, preference credit assignment (Verma & Metcalf (024), and vision-language model (VLM)-based reward design (Wang et al.) 2024; Tu
eet al. (025). Among these approaches, our work is most closely related to QPA (Hu et al. (024), which identified query-pole query-policy misalignment. While QPA relies on recency for query sampling, showed that recent data selection inadequately captures policy relevance. Instead, propose sampling queries based on policy likelihoods for more effective and policy-relevant query selection. DUO (Feng et al. 2025) similarly leverages policy likelihood through rectified Z-scores, whereas we employ ranks of trajectory likelihoods without buffer normalization. Complementary to query selection methods, PPE (Zhu al. 2024) tackled query-policy misalignment through exploration strategy that adjusts the behavior policy near the current policy to expand data coverage in policy- relevant regions. In contrast, we directly address query-policy misalignment by identifying policy- relevant queries from the replay buffer based on their likelihoods, without requiring modifications to the exploration strategy or additional behavior policy.
Primacy Bias and Loss of Generalization Ability in Neural Networks. PbRL poses unique challenge beyond standard RL: periodic reward model updates create non non-stationary reward signals, while high reliance approximate rewards risks overoptimization (Kaufmann al. 2024). These issues are compounded by primacy bias, a key challenge in deep RL identified by Nikishin et al. (2022). who proposed periodic resets as mitigation strategy. D'Oro et al. (2022) later demonstrated that more frequent resets enable higher replay ratios, improving sample efficiency. More broadly, primacy bias relates to a wider set of neural network degradation phenomena, including capacity loss-the gradual loss of network adaptability to new targets (Lyle et al.) (202)) and plasticiry loss-the inability to overwrite prior predictions when input-output relationships change (Lyle al.,
2023]. These issues are particularly severe in RL due to its non-stationary nature (Igi et al.) 2021]
Proposed solutions to counteract these degradations include feature space regularization (Lyle e et al,
2022), loss landscape flattening (Lyle et al.) 2023 Lee et al. 2023 Foret et al. (021). plasticity
injection (Nikishin et al. 2023), and resetting inactive neurons (Nikishin et al. 2022 D'Oro et al.)
2022 Sokar et al. 2023 Xu et al.) 2024). Notably, NNumaa et al. (2024) identified resetting as
the most effective and robust intervention. Building upon these insights, we develop an adaptive
resetting mechanism tailored for PbRL..
3

---

Published as a conference paper at ICLR 2026
3 REWARD ESTIMATOR LEARNING WITH PREFERENCE DATASET
PbRL trains agents to align with human intent using human feedback. This approach replaces tra- ditional reward functions with learned reward estimator that captures human preferences. The process begins by collecting pairs of agent behavior trajectories and corresponding preference la- bels, where human annotators indicate which trajectory they prefer. The reward estimator is then trained to assign higher rewards to preferred trajectories and lower rewards to non con-preferred ones. This learned reward function guides the agent's s policy optimization standard RL..
At each feedback session, trajectory pairs are sampled for human evaluation. A trajectory is defined as fixed-length sequence of state-action pairs,
{(sg. ag); (81, 01), ($H-1, 0g 0H-1)} E (S A) where H is the trajectory length. For each pair (a" .. the human teacher provides binary preference label y, typically in one of three forms: (1,0) when preferring σ² (0. when preferring
at, or (0.5, 0.5) to indicate equal preference. Each preference instance (a" "e,o"", .,, v) is accumulated in dataset D = {(0,, ,,", } where N repressnts the total feedback collected.
Using this dataset D. the reward estimator Pg. S' - R is trained to align predicted rewards with human preferences. The preference probability is modeled using the Bradley- Terry model (Bradley
& Terry 1952):
expl (or Polst, ac))
Polo > -0)= exp( (c., (o) (st, az)) + exp ((s (St, az))
Herein, 01 > d0 denotes that a² preferred over d., Fv is trained to minimize the binary cross- entropy (BCE) loss between true preference labels and predicted preference probabilities:
Li -I.
log Po (a > o²) + v(1) log (a > o)),,
(2)
where y(i indicates the i-th component of y. This framework seamlessly integrates with various RI algorithms (Mnih et al., 2016) Schulman et al. 2017 Haarnoja al. 2018 Yarats et al. 2022). eliminating the need for manual reward design.
4 PROPOSED METHOD
In this section, we present PoLiCER, which consists of two key components: (1) policy likelihood- based query sampling (PLS) and (2) critic-exploited reset (CER). PLS addresses query-policy mis- alignment by selecting informative queries based their likelihoods under the current policy. CER mitigates overestimation caused by primacy bias by strategically resetting both the reward estimator and its corresponding Q-function the feedback dataset accumulates.
4.1 POLICY LIKELIHOOD-BASED QUERY SAMPLING
de
P())
VS
01
25
dg
K
Sumple1 trajecteries uniformll
Calcolate of trajectory segmenns
Delliiing
Sample seguents weights
lop Hidthhhsss
Figure 3: Overall framework of PL.. Log- hogllikelsoodd trajectories are used for sampling informa- tive queries relevant the current policy. The inverse rank of log-likelihoods serves as probability weights, and the hyperparameter a controls the sharpness of the distribution.
Unlike previous approaches that use recency as a proxy for policy relevance, our method directly quantifies how well each trajectory represents the current policy by computing its log-likelihood,
4

---

Published as a conference paper at ICLR 2026
ensuring alignment regardless of when the data collected. Figure iillsstrtes the overall pro- cess. At each feedback session, we begin by uniformly sampling 2 x L x trajectories, where K is the number of queries to extract and L is a scaling factor, following prior weighted sampling ap- proaches (Christiano et al. 2017] Lee et al. b021abb). We then compute the average log-likelihood of each trajectory under the current policy " as follows:
Σ (oo
log Th (at 82)-
H
The log-likelihood I; is proportional to the probability of trajectory o; under policy TT. with higher values indicate trajectories that better reflect current behavior of the agent. However, in continuous control tasks, log-likelihoods of probability densities. To ensure robustness, we use the inverse rank of fog-likelihoods as the sampling weight w; rather than raw values (Schaul al. 2016):
be highly sensitive to outliers because of the unbounded nature
((1) / Σ V
Here, controls the sharpness of the sampling distribution: lower values yield more uniform sam- pling, while higher values favor high-likelihood trajectories. Finally, preference queries are gener- ated by randomly pairing two sampled trajectories.
Compared to disagreement sampling (DS) (Christiano et al. 2017 Lee al. 2021a), our method is computationally efficient, requiring only 2 x L x K forward passes, while DS incurs a cost that N times higher (where is the number of reward estimators in an ensemble). The pseudo code for PLS is presented in Algorithm| Din Appendix [ Figure 13 in Appendix D.I confirms that PLSS selects queries more relevant to the current policy than NOS, with no practical increase in training time.
4.2 CRITIC-EXPLLOTER RESET
As the feedback dataset grows, the reward estimator overfits to early feedback due to primacy bias, am- plifying reward differences through the BCE loss in
100
Threshold Value (8.)
Equation 2 (Heo et al., (025). While underestima- tion less problematic since low-reward state-action pairs are naturally avoided (Fujimoto et al.] (018)) overestimation proves more detrimental by encourag- ing suboptimal behaviors. Although periodic resetting helps, naive approaches suffer from computational in- efficiency and residual Q-function bias. To address these challenges, we propose CER, a dynamic reset- ting strategy that leverages critic outputs indica- tors reward overestimation. Rather than resetting at fixed interval, CER monitors critic outputs against adaptive threshold to determine when intervention is necessary. We design the threshold to increase mono- tonically throughout training for two critical purposes: first, it accommodates genuine policy improvement by allowing higher Q-values as the policy optimizes; sec- ond, it reduces unnecessary resets over time, avoiding excessive training disruption.
40-
RG= 6=7.5, p=0.9 Bo=25, p=0.9
10
25
Number of Resets (i)
Figure Visualization of CER thresh- old 3 for different step sizes. Thresholds grow monotonically and saturate at upper bounds determined by 8. Lower 8 com- pensates for task complexity by maintain- ing tighter overestimation control.
Specifically, CER continuously tracks the maximum critic output Qmax observed across gradient steps since the last reset. At each feedback session, if Q exceeds the current threshold 3,, CER resets both the reward estimator ry and Q-function Qo before updating the reward function. In PbRL, rewards are typically bounded to [-1, via tanh activation, constraining critic outputs to |==رن = 100 (with discount factor = 0.99). This bounded nature enables tank-agi monitoring with consistent scale. We achieve monotonic threshold growth while ensuring a clear upper-bound through exponential decay schedule:
5

---

Published as a conference paper at ICLR 2026
B++1 3
(5)
is the initial step size, P E (0. controls decay rate, and denotes the number of resets performed. Here, 8 directly regulates threshold growth based on task complexity (e.g.. action dimension). This exponential decay allows the threshold to grow monotonically while gradually saturating at 80 + r If the threshold is not exceeded, training proceeds normally with the existing parameters. Figure illustrates this adaptive threshold mechanism for different 8 values.
where 8
After each reset, we increase the replay ratio to accelerate adaptation to updated reward estimates. Inspired by adaptive replay ratio scaling (Ma et al. 2024), we maintain low replay ratios during early training to preserve plasticity, then gradually increase them later for sample efficiency. Once feedback collection ends and reward learning completes, resets cease and the replay ratio returns to one, as further overrides become unnecessary. Additional details are provided in Algorithm 2 in Appendix B Empirically, CER achieves higher returns and lower reward overestimation than naively resetting the reward estimator at every feedback session (see Figure 15 in Appendix D.2).
CER grounded in three key theoretical principles and empirical findings: (1) resets eliminate ac- cumulated primacy bias and corresponding overestimation (Nauman et al. 2022). (2) critic outputs serve as indirect indicators of overestimation (Van Hasselt et al. [016] Fujimoto et al.. 2018 Nik- ishin et al. 2022), and (3) valid policy improvements increase Q-values monotonically (Sutton et al., 1998j. Combined with bounded rewards, these principles justify our monotonic threshold schedule with asymptotic saturation: initial low thresholds address early overestimation, while gradually in- creasing thresholds distinguish genuine policy improvements from bias-induced anomalies. Our empirical sensitivity analysis (Appendix D.7) further validates the robustness of CER across hyper- parameters. We formalize CER's theoretical advantage through value approximation error bounds, adapting the analysis from Hu et al. (2024).
Definition 1. Let OF be the Q-function of the current stochastic policy under the true reward r. For a distribution μu, define the segributiin-dependdee norm ll.fll := Ex=ullf(=)]] and let F denote the function approximator class. The intrinsic function approximation error is defined as
a = fe 1 aa
Lemma 1. Assume the reward overestimation error satisfies -ir--lla- S e. Further assume
the learned Q-function estimator Ô is the empirical risk minimizer onto F, that QF, E arg min fer 110 - fllør. Then !!Q Q7,, lal S a I-
2z
(Proof in Appendix A..
Lemma 2. Suppose CER reduces the reward overestimation error bound from E to EC, where ô = &c > 0 quantifies this improvement. Then CER tightens the value approximation error bound, guaranteeing an improvement of 26 (Proof follows directly from Lemma 1 by substituting Ec for and subtracting the bounds).
i-7
These results quantify CER's advantage: reducing reward overestimation by yields a Q-function approximation error reduction of 28 offers principled method quantify the benefits of mitigating primacy bias.
1-y. This provides theoretical justification for CER's ' design and
EXPERIMENTS
5.1
SET UP
PoLiCER is compatible with any PbRL algorithm. In our experiments, we implemented it on PEB- BLE (Lee et al. 2021b) as the baseline. For PLS, we set the temperature parameter 1 across all main experiments. For CER, the initial threshold was set to 8n = 25 with a decay rate p = 0.9. The initial step size was defined g 30
| to compensate for task complexity using action dimension
[A] (i.c., 8- 7.5 for all Meta-World tasks and 1.43 for DMControl Humanoid). To improve feedback efficiency, we applied multiple temporal data augmentation (TA) with ratio 20, fol- lowing prior works (Hu et al. 2024: Park et al.) 2022). The pseudo code of PoLiCER is presented in Algorithm 3 in Appendix B We evaluated proposed method three locomotion tasks from DMControl (Tassa et al., 2018 Tunyasuvunakool et al.] (020)) and four robotic manipulation tasks from
(Yu et al. 2020). Following prior studies, we used synthetic human annotator

---

Published as a conference paper at ICLR 2026
SAC (upper bound)
PEBBLE
SURF
RUNE
MRN
QPA
PoLiCER
Ep Batur
Enwinoemen Steas T
enviremment Stegs (ntoP)
Envirsment Steps Iwith
Wallen WaIk (1URYLU)
Cheetan Rur (LUOYLU)
Ptumancor Stand (10000)20)
189
05
62
0.5 Door Open (4000/62)
625
0.55 65
0.5
5.15 Drawer (5000150)
2:5 95
1.5
20
(5000/50)
Hammer (10000)
Figure 5: Learning curves using vector-based state inputs. First row: performance results for loco- motion tasks (Walker Walk, Cheetah Run, Humanoid Stand) in DMControl. Second row: results for robotic tasks (Door Open, Sweep Into, Drawer Open, Hammer) in world.
that provides preference feedback based on ground truth rewards in B-Pref benchmark (Lee al. 2021a). Comprehensive experimental details are provided in Appendix
5.2
SICTOR-BASEL CONTROL WITH SYNTHETIC PREFERENCES
For a comprehensive comparison, we included SAC (Haarnoja et al. 2018) trained with ground truth rewards, along with other representative PbRL methods: SURF (Park al.) 2022), RUNE (Liang it al.) 2022), MRN (Liu et al. (0222), and QPA (Hu et al. 2024). Each algorithm was evaluated every 10,000 steps with ten evaluation episodes per task. In DMControl, we reported average episode returns, while in Meta- Werad, we used average success rates. Figure 5 presents the learn- ing curves for all methods using vector-based state inputs in DMControl and WoradW Solid lines and shaded regions indicate the mean and standard deviation across ten runs. Vertical dashed gray lines mark the final feedback step, and horizontal dotted lines represent the final performance, computed by averaging the last ten evaluation scores for each method.
In DMControl, both PoLiCER and QPA showed significant performance improvements over PEB- BLE. While their average performances were comparable, PoLiCER exhibited superior robustness with lower variance, as reflected in the 95% bootstrapped confidence intervals (see Figure 16Jin Ap- pendix D.3]. In contrast, in Meta-World environments, most PbRL methods showed high variance and poor average success rate. PoLiCER, however, achieved over 80% success with consistently lower variance. The limited improvements of QPA in Meta-World suggest that its policy-aligned sampling strategy is ineffective in goal-conditioned tasks. Notably, in Drawer Open, none of the other methods exceeded a 60% success rate after feedback sessions ended, with nearly half of their runs resulting in 0% success. This indicates a failure to learn a generalized reward function that aligns feedback, which highlights the need for deeper analysis of PoLiCER's 's individual compo- nents, as discussed in Section 5.3
5.3 ABLATION STUDY COMPONENT ANALYSIS
We evaluated the impact of each PoLiCER component on overall performance by incrementally in- corporating PLS, CER, and TA. Figure 6 shows the learning curves and critic outputs across three representative tasks: Walker Walk, Door Open, and Drawer Open, to assess the effectiveness of each component. As shown in (a), both PLS and CER independently led to substantial per- formance gains. Although their combination was not always complementary, especially in Drawer Open, it remained generally effective in most environments. TA offered modest improvements: slightly increased average scores in Door Open and reduced variance in Drawer Open, showing bet- ter generalization by the reward estimator. However, (b) reveals that PLS can induce critic overestimation due to its localized policy guidance, phenomenon also observed in QPA (see Figure
7

---

Published as a conference paper at ICLR 2026
PEBBLE
POLCERR (IA: PLS: O, CER
POLICER (TA: X, PLS:X, CER 0)
POLICER (IA: PLS: O CER
PoLIERR
1,000
500
300
(a) Score
Eptooto Robur
Sucoess Rote (9)
1.15
0.36
0.36
0.
00
0.35
0.75
16
0.0
0.3%
05
0.36
Invironment Steas (x10%)
Enstronment Steps (x:00)
Environment Steps (X30P)
100
320
100
(b) Cnttic Output
20
Ch Outoutt
0.255
8250
0.275
2.000
0:25
2:50
0:75
1,00
0.25
8.50
1.75
3.00
Enviorment Steps (x104)
Environmmnt Stops Ex10P)
Enxtronment Stops Ix30
Waller Walk (100/1)
Door (4e004000
DraweE Open (5000/50)
Figure 6: Contribution of each component in PoLiCER across Walker Walk, Door Open, and Drawer Open. In both rows, solid lines and shaded areas represent the average and standard deviation. (a) Learning curves showing average episode returns success rates. (b) Critic outputs during training. used to indirectly assess overestimation.
17 in Appendix D.4). While this guidance improved average performance over PEBBLE, can also reinforce overestimation, ultimately leading suboptimal policies. CER helps mitigate this effect, enabling the agent to explore a broader policy space while still leveraging policy-aligned sampling.
Notably, in Drawer Open, CER alone achieved the highest success rate of approximately 100%, outperforming other combinations of components and PbRL methods. This result can attributed the misalignment between task success and rewards, particularly during the early and mid-training phases as shown in Figure ? In these phases, the true reward dis- tributions of successful and failed trajectories exhibit significant overlap, indicating that the true reward function does not consistently re- flect actual task completion (see rendered ex- amples in Appendix D.4). This discrepancy introduce noisy preferences in early train- ing stages, which may misguide both policy and query sampler. As result, agents relying solely on PLS showed high performance variance and lim- ited improvement in Drawer Open. In contrast, CER effectively reduced the influence of these early misaligned preferences, preventing the agent from being trapped in biased reward estimations and enabling more stable and consistent learning.
Saute
Figure 7: True reward distributions for successful and failed trajectories in Drawer Open under Po- LiCER different training stages. Early and mid- training show low-reward trajectories even among successful attempts, revealing misalignment be- tween task success and rewards.
5.4 PIXEL-BASED CONTROL WITH SYNTHETIC PREFERENCES
We expanded our experiments to pixel-based state inputs using synthetic preference annotators. Fol- lowing Park et al. (2022), we adopted DrQ-v2 (arrats et al. 2022) as the backbone RL algorithm and included its variant of PEBBLE, SURF, and QPA (see the details in Appendix C.2). Our method achieved higher performance compared to other methods, shown in Figure
Specifically. QPA
failed to improve performance compared to the baseline (PEBBLE) in Cheetah Run and Window Open. This result suggests that the reliance of QPA recent experiences for query sampling and
8

---

Published as a conference paper at ICLR 2026
D1Q-22 (upper bound)
PEBBLE
SURF
QIA
PoLiCER
1,001
1,903
140
800
60
Episcade betum
Episode Returm
Success Rote (%)
200
203
20
0.0
0.55
0.5
0.75
1.0
0.25
0.5
0.75
1.0
0.6
Number Fromes (x10f)
Number (X10")
Walker Walk (200110)
Cheetah (1000/50)
Window Open (800/20)
Figure 8: Learning curves two locomotion tasks (Walker Walk and Cheetah Run), and one robotic manipulation task (Window Open) using pixel-based state inputs.
policy updates may be less effective high-dimensional pixel-based environments, where visual complexity weakens the link between recency and relevance. In contrast, PoLiCER achieved ap- proximately 80% success rate with low variance in Window Open, doubling the performance of PEBBLE. These findings demonstrate that prioritizing policy relevance over recency becomes in- creasingly crucial in complex visual domains, and our strategic reset approach provides a clear advantage for robust and stable preference learning across diverse perceptual challenges. While PoLiCER requires moderately longer training time than baselines due to CER's increased replay ratio after resets, this computational overhead is modest relative the inherent rendering costs of pixel-based and is offiset by substantial performance improvements (see Appendix D.5 for detailed computational analysis).
5.5 HUMAN-IN-THE-LE EXPERIMENT
While synthetic annotators the B-Pref benchmark (Lee et al. 2021a) provide a fair comparison of PbRL methods, they may yield different trends and interpretations from human labels (Hu al., Kim et al. (203)). As shown in Section [.3] synthetic preferences do not always align with intended behaviors, which can lead misleading conclusions. To ensure more realistic evaluations, we conducted human- in- -oopp experiments using ten volunteers on Walker Walk and Window Open for both PEBBLE and PoLiCER. Following prior works (Hu et al., 2024, Heo et al., 2025). we included an equal preference option = (0.5, 0.5) to account for human inconsistencies. Detailed experimental procedures for these tasks are provided in Appendix D.6
2024
SAC (upper bound)
PEBRLE
1,000
100
Episode Return
Success
400
40
200
20
0.0
0.125
0.25
0.375
0.5
0.0
0.125
0.25
0.375
0.5
Steps (x10F)
Steps (x110f
WalkeWWWk
Window Open (240115)
Figure 9: Learning curves Walker Walk and Window Open using vector- bastor-baedddd state inputs with human annotators.
Figure 9 shows that PoLiCER improved feedback efficiency over PEBBLE. Although both agents performed similarly during the feedback collection phase, PoLiCER achieved significantly higher

---

Published as a conference paper at ICLR 2026
ground truth rewards and success rates training progressed. We examined how the two meth- ods produced qualitatively different learned behaviors. On Walker Walk, Figure [0 striking differences: the PEBBLE-trained agent primarily moved by crawling its knees with hunched posture, failing to develop natural walking behavior. In contrast, the PoLiCER-trained agent suc- cessfully learned coordinated bipedal locomotion with alternating leg movements and maintained an upright torso throughout motion. This behavioral difference suggests that PoLiCER more effec- tively captured and incorporated human preferences regarding natural walking motion, translating feedback into more naturalistic and biomechanically appropriate locomotion patterns. These human- un-the-lopp results provide critical validation that PoLiCER effectively narrows the gap between theoretical preference learning and practical implementation challenges, suggesting a promising ap- proach for preference-basee based learning in complex interactive scenarios.
(a) Agent trained with PEBBLE
(b) Agent trained with PoLiCER
Figure 10: Comparison of final behaviors learned in Walker Walk with human feedback. (a) PEB- BLE agent resorted to crawling with poor posture, while (b) PoLiCER agent demonstrated coordi- nated bipedal walking with proper balance and form.
CONCLUSION
In this study, we introduced PoLiCER, novel approach that addresses two key challenges in PbRL:
(1) query- pvery-policy misalignment in query sampling and (2) reward overestimation caused by primacy bias in reward learning. Our solution combines two complementary components: PLS ensures se- lected queries better reflect current policy behavior, while CER dynamically resets both the reward estimator and Q-function to mitigate primacy bias. Together, these components create a synergistic effect-PLS PLS provides more relevant feedback signals, and CER prevents bias accumulation when processing this feedback. Our theoretical analysis proves that CER s resetting strategy delivers mea- surable improvements: for every unit of reward overestimation reduced, the Q-function approxima- tion error improves by twice that amount, scaled by the discount factor. Extensive evaluations on DMControl and Meta-World benchmarks demonstrated that PoLiCER significantly outperformed existing PbRL methods, achieving nearly 100% success rate in Drawer Open where other methods struggled exceed 60%. These results highlight the effectiveness of our integrated approach, which aligns query selection with policy behavior while maintaining stable learning through strategic re- sets.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we described the implementation details in Appendix including all hyperparameters, network architecture, and training procedures (Table (HI2). The complete algorithms are provided in AlgorithmssTH33 Appendix B Source code is publicly avail-
able at httptttss//ttttttteee hwtp://aagoooo--eeeoooooooo.
LEGMONTESSEMENTS
This research was supported by Culture, Sports and Tourism R&D Program through the Korea Cre- ative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2024(Project Name: Development of Al-based large-scale automatic game verification technology to improve game production verification efficiency for small and medium-sized game companies, Project Num- ber: RS-2024-00393500, Contribution Rate: 100%)
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In M Ranzato, A Beygelzimer, Y Dauphin, S Liang, and Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 29304-29320. Curran Associates, Inc., files/paper/2221/
http://www..eeo....oooo..r.....o www.:newwricccoott 2021. URL htt.://eceaaaa110oooo//0111eeeee
Jordan Ash and Ryan P Adams. On warm-starting neural network training. In
H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp.
Curran Associates, Inc.,
2020. URL https::/ ngs neurips.............e ec cc/paper____pee fi ftlles/ppppe//000/ fi
Paper....... .dff
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(34))322 345, 1952.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without model. arXiv preprint arXhv:2101.05982 2021.
Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei- Yue Wang. RIME: Ro- bust preference- based reinforcement learning with noisy preferences. In Forty-firs International Conference on Machine Learning, 2024. URL
net forum?ids
BxAven S80
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems. volume 30. Curran Associates, Inc., 2017. URL https::/pprceddiggss www.wiiisw.g.....s.s............ ccc.pps..cce/pee cc/paper...pee....eeeeeeeeeee.. files/paper/2017/ File/ http:///ccccceeoe111022220000000
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022. 2022.
Xuening Feng, Zhaohui Jiang, Timo Kaufmann, Puchen Xu, Eyke Hüllermeier, Paul Weng, and Yifei Zhu. Duo: Diverse, uncertain, on on-policy query generation and selection for reinforcement learning from human feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025. doi: 10. 1609/aaai. v...i...3911.... v39116.33822
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar minimiza- tion for efficiently improving gemeralization. In International Conference on Learning Represen- fations, 2021. UR.. forem//orrmmiom 6mm mposs rM

---

Published as a conference paper at ICLR 2026
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- 2018.
cations. arXiv preprint ar
Jongkook Heo, Young Jae Lec, Jachoon Kim, Min Gu Kwak, Young Joon Park, and Scoung Bum Kim. Mixing corrupted preferences for robust and feedback-efficient preference-based reinforce- ment learning. Knowledge-Bond Based Systems, 309:1 12824, 2025.
Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya- Yi-Qin Zhang. Query-policy mis- alignment in preference-based reinforcement learning. In The Twelfth International Confer- ence on Learning Representations, 2024. URL http:://///pereeeee....o net. forum?
UoBymIwPJR
Shuaiyi Huang, Mara Levy, Anubhav Gupta, Daniel Ekpo, Ruijie Zheng, and Abhinav Shrivastava. Trend: Tri-teaching for robust preference-based reinforcement learning with demonstrations. In Proceedings of the IEEE International Conference on Roboties and Automation (ICRA), 2025.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Böhmer, and Shimon Whiteson. Tran- sient non-stationarity and generalisation in deep reinforcement learning. In 9th International Conference on Learning Representations, 2021.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. 2024.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lec. Prefer- ence transformer: Modeling human preferences using transformers for (RL). In The Eleventh International Conference on Learning Representations, 2023. URL https: / /owp///vpe....eeep/o//o
Sest/Eorum?id=eeee
Diederik Kingma and Jimmy Lei Ba. Adam: method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, 2015.
Hojoon Lee, Hanseul Cho, Hyunseung Kim, Dachoon Gwak, Joonkee Kim, Jaegul Choo, Se- Young Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient reinforce- ment learning. Advances in Neural Information Processing Systems, 36:62270-62295, 2023.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benehnarking preference- based reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Syy- lems Datasets and Benchmarks Track (Round 1), 2021a. URL URL htt.httts///ppeeeeeee
net/
Eorum?id-ps mkHF
Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning. pp. 6152 6163. PMLR, 2021b.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representa- fions, 2022. URL
Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-nett Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270-22233 2022.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in inforcement learning. International Conference on Learning Representations, 2022. URL
htttps://openrev/
net / forum?idee drun2lld=zkCeeekool007
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Beraaro Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pp. 23190-23211. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Guozheng Ma. Lu Li, Sen Zhang, Zixuan Liu, Zhen Wang. Yixin Chen, Li Shen, Xueqian Wang, and Dacheng Tao. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. In The Twelfth International Conference Learning Representations, 2024.
URI.tttps: http::://pppeeeeeeeeee net foteet/ffoommidddRRiisssss
Katherine Metcalf, Miguel Sarabia, Natalie Mackraz, and Barry-John Theobald. Sample-efficient preference-based reinforcement learning with dynamics aware rewards. In Conference on Robot Learning. pp. 1484-1532. PMLR, 2023.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillierap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928 1938,197,410 2016.
Ni Mu, Hao Hu, Xiao Hu, Yiqin Yang, Bo XU, and Qing-Shan Jia. CLARIFY: Contrastive pref-
erence reinforcement learning for untangling ambiguous queries. In Forty-second International Conference on Machine Learning, 2025. URL httph::t////......... httts:://ppereww /openreveiveeooeoe ew. net. forum? d=
1VOCPctm3n
Calarina Muslimani and Matthew E. Taylor Leveraging sub-optimal data for human-in-the-lon reinforcement learning. In The Thirteenth International Conference Learning Representationss 2025. URL
Michal Nauman, Michat Bortkiewicz, Piotr Miloš, Tomasz Trzcinski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning. In International Conference on Machine Learning, pp. 37342-373364 PMLR, 2024.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron Courville. The primacy bias deep reinforcement learning. In International conference on machine learning, pp. 16828-16847. PMLR, 2022.
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and Andre Barreto. Deep reinforcement learning with plasticity injection. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neu- ral Information Processing Systems, volume 36, pp. 37142-37159. Curran Associates, Inc., 2023. URL https:://proceeed.gss..eee neur.pss........ppeee..r...... cctpsscccccppeeeeeeee httpp//ssse.pppppppeeeea////e/pe
fites//apee///////////// Paper Conference. Conerreeeeeeeeeee..eeeeneeeee.e .df
File/
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Curran Associates, Inc., 2022. URL httrps::/proceeedingssttt Paneraaaa--pp--e--eeeeeeeeeee www.errneooeoee..eee..ao.ee pdf
Jongjin Park, Younggyo Sco, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efffcient preference-bate reinforcement learning. In International Conference on Learning Representations, 2022. URL
htttps://openreviiee net ttte://ommmmmmffm....00000
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In 4th International Conference on Learning Representations, ICLR 2016 Conference Track Pro- ceedings, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, pp- 32145-32168. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2026
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv:1801.000 000900 2018.
Songjun Tu, Jingho Sun, Qichao Zhang, Xiangyuan Lan, and Dongbin Zhao. Online preference- based reinforcement learning with self-augmented feedback from large language model. In Pro- ceedings of the 24th International Conference Autonomous Agents and Multiagent Systems, pp. 2069-2077, 2025.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6: 10:000 2020.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Mudit Verma and Katherine Metcalf. Hindsight PRIORs for reward learning from human pref- erences. In The Twelfth International Conference Learning Representations, 2024. URL
htttps://openreviiee Stwp:e//www.eettffoooolodL--eee -NeeoI
Yufei Wang, Zhanyi Sun, Jesse Zhang. Zhou Xian, Erdem Biyik, David Held, and Zackory Erick- son. RI-vim-f: Reinforcement learning from vision language foundation model feedhack. In International Conference on Machine Learning, pp. 51484-51501. PMLR, 2024.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fümkranz.. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136): 2017. URL http: http://mrrrrrpppppr.///es ////s//v1//11 6- 634. ht.l
Guowei Xu, Rujjie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daumé III, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openzevew.
Jaazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang. and Yux- iao Dong. Imagereward: Learning and evaluating human preferences for text-to-image genera- tion. In Oh, T Naumann, Globerson, K Saenko, M Hardt, and S Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 15903-15935. Curran Associates, Inc., 2023.
URL
n wttpsss.ccppppppppp..eeeees.sspe paper_ files//paerr/2233 Paper Conference pdf
cc
Fi htle:3336feffees1111003332222220
Denis Yarats, Rob Fergus, Alessandro Lazaric and Lerrel Pinto. Mastering visual continuous
trol: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https:
net www.fformiiiiooo Barprmmmiii_S--_ Sttp:iiio__S__eeeeeee -yees
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforce- ment learning. In Leslie Pack Kaelbling, Danica Kragie, and Komei Sugiura (eds.), Proceed- ings of the Conference Robot Learning, volume 100, pp. 1094-1100. PMLR, 2020. URL https:: httt::::/proceedinng
Yiwen Zhu, Jinyi Liu, Yifu Yuan, Wenya Wei, Zhenxing Ge, qianyi fu, Zhou Fang, Yujing Hu, and Bo An. Optimizing reward models with proximal policy exploration in preference-based reinforcement learning. In NeurIPS 2024 Workshop on Behavioral Machine Learning, 2024.
URL https::///// htttps://ppenrevien: net. formm? wwwrrmmiddimmieei111111
14

---

Published as a conference paper at ICLR 2026
A PROOFS
Proof of Lemma
1 Using the Bellman operator and Jensen's 's inequality, we have
10%, - Qflla- = ((,aaaaa (s, (x,
s (s Po(s,a) r(s,a)
+ (aa O7 -
(,((', 10, (s',a') (s')'
< (ssa)aa f (s, - r(s, B
s E + 1Q7 - 1r
Hence,
7) 1107, Q d S
=QQ Or la la
1
Let f* E arg minser 107 flla- so that 11Q ll a.P. Then, by the triangle inequality,
a inf 10 flla- $ +1Q7 fa JEF
< 1177 Q, aa + 11Q - f" 1aa
< - o..
Finally, decompose total error and use triangle inequality again:
10 QF al.- < 1|Q Q7, |aa- + 1107, - Q7 llat
= = Q7, a1 + inf
11 - -llaz
JEF
- - + a,²) a -
Remark our proof we used the relation a, S + which leads the looser bound a + T-y'
25 Under the more restrictive assumption that the function approximation enror is independent of
the reward function (i.c.. af a,),, the total error bound reduces to af + - which coincides with the result in Hu et al. 2024
B ALGORITHM DETAILSS
In this section, we present the detailed procedures of PoLiCER, which consists of two components:
(1) PLS and (2) CER. The complete implementation with PEBBLE is described in Algorithm 3
Algorithm PLS
1: Require: Hyperparameters: temperature parameter o, the number of queries per session k; and
scaling factor L
2: Input: Replay buffer without reward R, preference dataset D, and current policy n&.
3: Sample trajectories uniformly from replay buffer {o, ( 1 R
4: Calculate log-likelihood I, и (s.at)aa, log (T 8t) Vo, E T
5: Calculate sampling weight w; using the inverse rank log-likelihool 4
6: Sample 2 x à trajectories and pair them with probability P(i) w / E, m
7: Add & queries with corresponding human annotations to preference dataset D
15

---

Published as a conference paper at ICLR 2026
Algorithm 2 CER
1: Require: Hyperparameters: current threshold 3;, step size 6, decay rate and current replay
ratio
2: Input: Maximum critic output Qmar
3: if Qmax > 3g then
4: Increase replay ratio r+ = 1
5: Increase threshold B, = 3; + p
6: i+=1
7:
Reset parameters of critic 0, target critic 8 and reward estimator vb
8:
Qmaz f -0
9:
end if
Algorithm PoLiCER
1: Require: Hyperparameters: reward learning frequency F
2: Initialize parameters of policy ø, critic 6, target critic 6, and reward estimator vó
Initialize a preference dataset D <-
4: (Optional) Initialize replay buffer R and policy as with unsupervised exploration
5: Quar t-
6: for each iteration do
7: if' iteration %F then
D feedback session Algorithm b Algorithm
8:
Sample queries using likelihoods of current policy T Determine whether to reset or not using maximum critic output Qmaz
9:
10:
Optimize with Equation 2 and temporal data augmentation
11:
Relabel replay buffer R using fx end if
12:
13:
if' feedback session over then
revert replay ratio after all feedback provided
replay ratio r
15:
end if
for each gradient step in do
agent update with current replay ratio
17:
Sample random mini-batch B from R Optimize 0, and and log current critic outputs Qo(se, ae) B
18:
19:
if Qs a) > maa then
update maximum critic output for Algorithm)
20:
Qmax f Qs(s,a)
21:
end if
22:
end for
23:
Take action Oe (a se) and collect 8t+1
24: Store transition {st, a, 8141. (Sr, at)} in
25:
end for
C EXPERIMENTAL DETAILS
C.1 IMPLEMENTATION DETAILS FOR VECTOR- BASED CONTROL
We adopted SAC (Haarnoja al. 2018) for vector-based control, with the full of hyperparam- eters listed in Table LJ For the baseline PbRL method PEBBLE (Lee et al. 20216), which uses SAC as its backbone, we employed ensemble estimator comprising three multi-layer perceptron (MLP) models. Each MLP contained three hidden layers of 256 units with leaky rectified linear unit (ReL.U) activations. The outputs of each ensemble member were bounded within the range [-1, 1] using a hyperbolic tangent activation. We trained the models using BCE loss with the Adam optimizer (Kingma Baj 2015), a batch size 128, and a learning rate 0.0003 (see Table 2 These settings closely followed those described in B-Pref benchmark (Lee et al. 202 For other PbRL methods built upon PEBBLE SURF PPak et al. 2022). RUNE (Liang et al. 2022). MRN maintained the hyperparameters
in the PEBBLE configuration. The remaining method method-specific hyperparameters are provided in Tables 3
(Liu et al. 2022), and QPA (Hu al. 2024
to Table summarizes the settings of PoLiCER, which kept largely consistent across environ-
16

---

Published as a conference paper at ICLR 2026
ments ensure generalizability. All experiments conducted using random seeds, and the environments used are illustrated in Figure 11
aakker alk (100/10)
Cheetah Run (100/10)
Humanoid Stand (10000/50)
Door Open (4000/00)
Sweep Into (5000/50)
Drawer Open (5000/50)
Hammer (10000/50)
Figure 11: Visualization of test environments for vector-based control. The top row shows three locomotion tasks from DMControl: Walker Walk, Cheetah Run, and Humanoid Stand. The bottom row displays four manipulation tasks from Meta- Sota-:orld: Door Open, Sweep Into, Drawer Open, and Hammer.
Across all experiments, we used different query sampling strategies depending on the method. For most methods (except QPA and PoLiCER), we employed disagreement-baasee sampling (DS), which leverages the uncertainty of ensemble reward model to select informative queries (Christiano al. 2017 Lee et al. 2021b). Specifically, DS operates two steps: first, it randomly samples L K trajectory pairs from query buffer containing the 100 most recent episodes; then it computes the variance preference predictions across ensemble members for each pair. High variance indicates that the reward model is uncertain about which trajectory is better, signaling regions where the model would benefit from additional training data. The top K pairs with the highest variance are then selected for human annotation. Importantly, this variance measures model disagreement about the reward function, not the reliability of consistency of human annotation. Following prior studies, set 10 for this initial sampling pool.
Table Hyperparameters of SAC for vector-based control.
Hyperparameter
Value
Hyperparameter
Value
temperature
Hidden dimension
1.024 (DMControl),
Learning rate for agent
0.0005 (Walker Walk, Cheetah Run). 0.0001 (Humanoid Stand), 0.0003 (Meta-World
256 (orld--Worrd)
of layers for agent
2 (DMControl),
3 (Meta-Worid)
Target critic update frequency Optimizer
Batch size for agent
1 1,024 (DMControl),
Adam (Kingma & Ba. 2015 (099. 0.995)
512 (Meta- World)
Optimizer temperatures
Critic EMA rate Discount factor
0.005
0.99
Table 2: Hyperparameters of PEBBLE for rector-based control.
Hyperparameter
Value
Hyperparameter
Value
Batch size for oi Total feedbacks # of queries session
128
Length of segment
50 (DMControl),
100/10 (Walker Walk, Chectah Run).
(Meta-World) 20,000 (Walker Walk. Cheetah Run).
4,000/40 (Door Open).
Frequency of feedback
5,000/50 (Drawer Open, Sweep Into),
5,000 (Humaanoid Stand, Meta. World)
10,000450 (Humanoid Stand, Hammer)
Pro-training steps
9,000
Learning rate for 1o
0.0003
Data collection steps
1,000
Query huffer
For QPA, implemented NOS as specified in their original work (Hu et al. 2024). which required reducing the query buffer size to focus on more recent experiences (see Table 66 for specific buffer size). In contrast, PoLiCER used our PLS method, which selects queries based on their relevance
17

---

Published as a conference paper at ICLR 2026
Table Hyperparameters of SURF for vector-based control.
perpaaaraett
Value
Hyperparameter
Value
Unlabeled batch ratio Loss weight for unlabeled data A Threshold
Segment length before cropping
60 (DMControl).
35 (Meta-World)
0.999 (Cheetah, Sweep Into).
Min/Max length of cropped segment
[45. 55] (DMControl),
0.99 (Others)
(Meta-World)
Table 4: Hyperparameters of RUNE for vector-based control.
Hyperparameter
Value
Initial weight for intrinsic reward
0.05
Decaying rate
0.00001
Table 5: Hyperparameters of MRN for vector-based based control.
Hyperparameter
Value
Meta step frequency
1,000 (Walker Walk)
10,000 (Door Open. Sweep Into)
5,000 (Others)
Table 6: Hyperparameters of QPA for vector-based control.
Hyperparameter
Value
Hyperparameter
Value
Query buffer size
10 (DMControl),
Data augmentation ratio
20
60 (Door Open).
Hybrid experience replay sample ratio
0.5
30 (Others)
Table 7: Hyperparameters of PoLiCER for vector-based control.
Hy Hyperparameter
Value
Hyperparameter
Value
Sampling weight temperature &
1.0
Data augmentation ratio т
Initial reset threshold
Threshold growth decay o
0.9
Numerator of step size
30
Maximum replay ratio
4
Denominator of step size 8
[A]
the current policy without restricting the buffer size. This allowed PoLiCER maintain the default buffer size of 100 episodes while still ensuring policy-aligned queries.
For the CER component of PoLiCER, we carefully calibrated the reset thresholds based on theoreti- cal bounds. Since the maximum possible critic output is bounded by 1;ー 100 with reward scale r = 1 and discount factor 7 = 0.99, we set the threshold range to [-100, 100). Specifically, we used an initial threshold fg 25, decay rate p = 0.9, and step size | (scaled by action dimension), ensuring that reset thresholds remained appropriate across all environments while adapting to task complexity.
C.2 IMPLEMENTATION DETAILS FOR BRSSE CONTROL
We extended our evaluation of PoLiCER to pixel- based state inputs. as described in Section 5.4 Following the official SURF implementation for pixel- pixel-based control, we replaced the SAC back- bone with DrQ-v2 Yaarats et al. 2022) and adopted the hyperparameters detailed in Table comparison, we implemented pixel-based versions of PEBBLE and SURF according to Park et al. (022). We also incorporated QPA into our pixel-based experiments. However, since the original implementation does not support pixel-based inputs, we adapted it using hyperparameters consis- tent with its vector-based version, modifying only the data augmentation ratio т. Tables through 12 outline the hyperparameter differences across all methods. For PoLiCER implementation built upon DrQ-v2, we applied a selective reset strategy that only affected the last three MLP layers of O-function and reward estimator, while preserving the image encoder parameters (Nikishin al.
For
18

---

Published as a conference paper at ICLR 2026
2022 This approach maintains learned visual representations across resets. All pixel-based experi- ments were conducted across ten random seeds, with the test environments used illustrated in Figure
Walker (200100)
Chestah Run (1000500)
Window Open (800/00)
Figure 12: Visualization of test environments for pixel-based control. The left two images show locomotion tasks from DMControl: Walker Walk and Cheetah Run. The rightmost image displays manipulation task from Meta-Worldh Window Open.
Table 8: Hyperparameters of DrQ-v2 for pixel- control.
Hyperparameter
Value
Hyperparameter
Value
Replay buffer capucity
1,000,000
Optimizer
Adam Kingma & Ba 2015
Action repeat
2
Critic EMA rate
0.01
# of seed frames
4,000
Update frequency
2
Exploration steps n-step returns
2,000
Feature dimension
50
3 (Cheetah Run).
Hidden dimension
1,024
1 (Others)
Exploration std clip
0.3
Batch size
256 (Cheetah Run).
Exploration std schedule
(1.0. 0.1, 500,000) (Cheetah Run).
512 (Others)
(1.0, 0.1, 100,000) (Others)
Table 9: Hyperparameters of PEBBLE for pixel- pixel-based control.
Hyperparameter
Value
Hyperparameter
Batch for Total feedbacks /
16
Length of segment
50 (DMControl).
200/10 (Walker Walk).
25 (Meta-World)
# of queries per session
1,000550 (Cheetah Run).
Frequency of feedback
30,000 (DMControl),
800/20 (Window Open)
10,000 Merta-oold)
Leurning rate for v
0.0003
Query buffer size
20
Table 10: Hyperparameters of SURF for pixel-based control.
Hyperparameter
Value
Hy peperarraer
Value
Segment length before cropping
(Cheetah Run).
Min/Max length cropped segment
(Cheetah
54 (Walker Walk),
[48, 52] (Walker Walk),
(Window Open).
[23. 27] (Window Openi,
Threshold
0.99
Unlabeled batch ratio
5 (DMControl),
Loss weight for unlabeled data
(Cheetah Run), 0.1 (Others)
(Meta- World))
Table 11: Hyperparameters of QPA for pixel-based control.
Hy perparameter
Value
Hyperparameter
Value
Query hutfer Hybrid experience replay sample ratio
augmentation
0.5
Table 12: Hyperparameters of PoLiCER for pixel-based control.
Hy perpparraet
Value
Hyperparameter
Value
Sumpling weight threshold
1.0
Data samentation ratio
prowth decay nate
Numerator of step size
Maximum replay nioo r
(Meta- World),
Denominator of step size
[.4
4 contool
Query buffer
19

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 COMPARISON OF SAMPLING SCHEMES
We compared proposed PLS with two existing sample schemes: DS and NOS. DD selects the lop K trajectory pairs with the highest variance among ensemble members, while NOS prioritizes queries from recent episodes to mitigate query-policy misalignment (Christiano al. 2017 Hu
et al. 2024 Details for both methods are provided in Appendix C.I
To ensure fair comparison, we evaluated all three methods on Meta-World Sweep Into across ten runs with identical hyperparameters. We set total feedback budget of 10,000 preferences, delivered batches of 50 instances every 5,000 environment steps following the unsupervised pre-training phase. Performance was assessed using two metries: learning curves (success rate) and the average log-likes lieliiilioodd of sampled trajectories under the current policy each feedback session (Hu al.) 2024
0.0
80
-2.5
Success Rate
60
Log TT
-5.0
-7.
PEBBLE +DS
20
PEBBLE +NOS
PEBBLE +PLS
10.0
0.0
0.75 Environment Steps (x109)
0.25
0.5
1.0
0.0
0.25
0.5
0.75
1.0
Environment Steps (x10%)
(a) Success Rate Sweep Into
Policy Log-likelihoods: on Swaep Into
Figure 13: Comparison of three different query sampling strategies on Meta-World Sweep Into (10,000/50) with ten random seeds. The feedback session configuration is consistent with Figure 1 (a) Success rate learning curves over ten evaluation episodes, showing the mean and standard deviation across runs. (b) Average log- lggliikessoods of sampled trajectories under the current policy at each feedback session. Median and IQR are reported mitigate the influence of outliers.
As shown in Figure 13 PLS significantly improved the success rate by approximately 20% com- pared the competing methods and consistently achieved the highest policy log-likelihoods, While NOS demonstrated effectiveness early in training, its benefits diminished over time, with the gap in policy likelihood between NOS and DSS gradually narrowing. This suggests that recency becomes a less reliable proxy for policy relevance as task complexity increases. Importantly, PLS introduces no noticeable computational overhead. On RTX 4090 GPU with 19-13900KF CPU, PLS completed one million training steps in approximately 1.5 hours, comparable to the training time required by NOS.
Table 13: Actor entropy statistics for three sampling strategies Meta-Words Sweep Into (10,000/50) with ten random seeds.
DS
NOS
PLS
Mean
-3.688
-3.759
-3.758
Std
1.138
0.948
0.945
Max
2.236
2.209
2.267
Min
-4.271
-4.158
-4.135
One might consider that PLS could implicitly bias sampling toward low-entropy policies due to its localized guidance. However, policy entropy is controlled independently by the backbone RI.
20

---

Published as a conference paper at ICLR 2026
algorithm (i.e., SAC uses automatic entropy adjustment, and DrQ-v2 relies scheduled explo- ration noise). Therefore, the sampling strategies do not directly influence entropy dynamics. To empirically verify this, compared policy entropy statisties across three sampling strategies used in our experiments. As shown Table [3 entropy remains nearly identical regardless sampling methods, indicating that PLS does not induce low-entropy behavior.
D.2 FURTHER ANALYSIS REWARD OVERESTIMATION
We further analyzed the detrimental effect of reward overestimation caused by primacy bias, as demonstrates how overestimation persists in PEBBLE and overestimation rates, Figure presents prepssntss
discussed in Section 4.2 While Figure 2
through learning comparing estimated and true rewards over 1,000 collected episodes in Walker Walk. We carefully sampled these episodes ensure a uniform distribution of true rewards ranging from to 1,000 (see Table 14 for detailed statistics).
detailed visualization by
60
50
Estiattd
100
100
40
30
50
20
-50
-50
10
100
-100
200
400
600
800
1000
200
400
600
800
1000
Rewards
Rewards
(a) PEBBLE
(b) PEBBLE+Reset
Figure 14: Comparison between estimated rewards and true rewards for 1,000 collected episodes in Walker Walk, uniformly sampled across the full reward range. (a) PEBBLE exhibits strong reward overestimation, particularly in low-reward regions. (b) PEBBLE+Rese alleviates this problem, pro- ducing more accurate reward estimates across the full reward range.
The heatmap reveals that PEBBLE exhibited severe reward overestimation, particularly for low- reward trajectories, where early feedback disproportionately inflated reward estimates. In contrast, PEBBLE+Resel produced a more accurate and consistent relationship between estimated and true rewards. Additionally, PEBBLE struggled distinguish between high- and mid-reward trajectories because of compressed estimated values in those regions, limiting the reward estimator's 's ability to capture performance differences needed for effective policy optimization.
Table 14: Statistics for true episode rewards collected in Walker Walk.
Statistic
Value
Count
1,000
Average
500.168
Standard deviation
281.716
Minimum
19.826
25th percentile (Q1)
257.404
50th percentile (Q2)
500.595
75th percentile (Q3)
744.916
Maximum
989.187
As described in Section and 4.2 resetting the reward estimator at every feedback session helped mitigate reward overestimation. However, this approach alone proved insufficient because the Q- function retained biased estimates from previously overestimated rewards. Our CER method ad- dressed this by resetting both the reward estimator and Q-function when current critic outputs ex- ceeded dynamically increasing threshold.
21

---

Published as a conference paper at ICLR 2026
Reward Overssimmetin Rate (16)
PEBBLE PERBLEF-AR
400
PERBLE=CER
Epioodo Ratum
200
Cntc
200
200
20-
0.125
0.25
2500 3750
Steps (X10P)
Environment Steps (xC10f)
Environment Steps (XI09)
(a) Episode Retum
(b) Reward Overestimation Rate
(c) Critic Output
Figure 15: Performance comparison between PEBBLE, PEBBLE + Reset, PEBBLE CER on Walker Walk. (a) Learning curves showing the average episode return, with the mean and standard deviation across ten runs. (b) Reward overestimation rate during training, displaying median and IQR. (c) Critic output magnitudes representing Q-value estimates, reported mean and standard deviation. CER achieved both faster return gain and more effective mitigation of overestimation.
Figure demonstrates the effectiveness of CER. Figures 13 (a) and (b) reveal that agents trained with CER achieved higher returns and lower reward overestimation rates, respectively, compared to using reward estimator reset alone. Importantly, Figure [5| (c) shows that CER had significant im- pact on reducing Q-function overestimation, resulting in more accurate value estimates throughout training.
D.3 ADDITIONAL RESULTS FOR VECTOR-BASED CONTROL
IQM
Median
Mean
Optimality Gap
PoLiCER
QPA
MRN
RUNE
SURF
PEBBLE
0.45
0.60
0.75 0.90
0.6
0.8
0.45
0.60
0.75
0.90
0.30
0.45
0.60
(a) SAC normalized scores for DMControl tasks
Median
Mean
Optimality Gap
POLICER
QPA
MRN
SURF
PEBBLE
0.25
0.50
0.75
1.00
0.4 0.6
D.B
1.0
0.4
0.6
0.8
0.4
(b) SAC nomalized scores for Meta- W orld tasks
Figure 16: Aggregate performance metrics for all methods across ten independent runs. (a) Re- sults for three DMControl tasks. (b) Results for four Meta-World tasks. Each panel shows SAC- normalized IQM, Median, Mean OG with 95% stratified bootstrap confidence intervals. Lower OG values indicate performance closer to SAC with ground-truth rewards.
To ensure robust and reliable performance evaluation, we used rliable library (Agarwal et al., 2021) to compute several normalized metrics relative to SAC: interquartile mean (IQM), mean, median, and optimality gap (OG). IQM represents the average performance within the interquartile range (middle 50% of runs), effectively reducing influence of outliers. OG quantifies the deviation from
22

---

Published as a conference paper at ICLR 2026
optimal performance (SAC in our study), with lower values indicating closer alignment optimal results. We reported all metrics with 95% stratified bootstrap confidence intervals. Figures I6
(a) and (b) present the aggregated results across three DMControl tasks and four Seta-World tasks, respec- tively. Across both benchmarks, Pol LiCE consistently outperformed other PhRL. methods, achiev- ing the highest IQM and lowest OG in Meta- Meta-World. Although PoLiCER and QPA yielded similar average scores in DMControl, PoLiCER showed significantly narrower 95% CI bands, demonstrat- ing greater stability across tasks. The lower OG values further confirmed that PoLiCER remained closer to SAC performance, highlighting its efficiency under limited feedback conditions.
SAC (upger bound)
PEBBLE
SURF
RUNE
MRN
QPA
PoLICER
1.000
300
202
Sc.cess Rath [K]
800
(a)
Epsadde
400
40-
0.9
(K10f)
100
50
(b) Crbbe
Ortic Outputs
Crtic Outtutt
Outputs
20
40-
40-
Environment (4207)
Emvironment Steps (x:10f)
Enveronment Steps Ex10") Hammer (10000/50)
Humanoid Stand (10000/50)
Drawer Open (5000/50)
Figure 17: Results across ten runs on Humanoid Stand, Drawer Open, and Hammer. (a) Learning curves showing performance progression with horizontal dotted lines indicating final performance. (b) Critic outputs during training. In both rows, solid lines represent means, shaded areas show standard deviation.
Building on these aggregate results, we visualized the learning curves and corresponding critic output plots across all methods on three challenging HHumanoid Stand, Drawer Open, and Sweep Into (Figure [7p. Solid lines represent mean values, shade regions indicate standard deviation, and vertical dashed gray lines mark the last feedback step. Dotted lines in the learning curves denote the final performance, calculated by averaging the last ten evaluation scores. The critic output plots show that most baselines suffered from overestimation during the feedback phase, with value estimates inflated beyond actual performance (Van Hasselt et al., 2016). Notably, QPA exhibited the highest critic outputs, driven by its use of recent-ann and thus policy relevant- data for query sampling and Q-function updates, reinforcing localized guidance (Hu al., 2024). In contrast, despite also using policy-aligned sampling, PoLiCER maintained much more stable critic values. This highlights the effectiveness of CER in mitigating overestimation and stabilizing training (Nikishin et al. 2022). This enabled PoLiCER combine feedback efficiency with robustness, avoiding runaway value estimates while preserving policy relevance.
D.4 DETRIMENTAL EFFECT OF REWARD MISIDENTIFICATION
Figure [iillusstratss problematic cases in the Drawer Open environment where the true reward signal significantly diverged from actual task success, creating misleading preference feedback. Figure [8 (a) shows a successful task completion the robot eventually opened the drawer completely but received unexpectedly low true rewards because of time spent exploring before achieving the goal. Conversely, Figure [8 (b) depicts a clear failure case where the robot merely grasped the drawer handle without fully opening it, yet paradoxically received high true rewards.
23

---

Published as a conference paper at ICLR 2026
This reward-task misalignment led synthetic annotators generate inconsistent preference labels that poorly reflected actual task completion. As documented by Lee et al. (2021a), such noisy preferences severely impact learning by misguiding both policy updates and query selection processes. This problem created a detrimental feedback loop where the query sampler reinforced suboptimal behaviors based flawed reward signals. This phenomenon helps explain high perfor- mance variance and inconsistent learning outcomes observed across most baseline methods.
(a) Success case with true rewards of 1285.33
(b) Failure case with true rewards of 3013.11
Figure 18: Visualization of reward-task misalignment in Drawer Open. (a) Successful drawer open- ing that received unexpectedly low rewards. (b) Failed attempt that received disproportionately high rewards. This misalignment generated noisy synthetic preferences that hindered consistent policy learning.
D.5
COMPUTATIONAL OVERHEAD ANALYSIS FOR PIXEL-BASED CONTROL
Table 15: Wall-clock training time (hours) comparison for pixel- control tasks across PbRL methods.
Method
Walker Walk
Cheetah Run Window Open (0.8M frames) (800/20)
(1M frames)
(1M frames) (1000/50)
(200/10)
DrQ-v2 (upper bound)
3.47
2.53
7.24
PEBBLE
4.04
3,21
8.77
SURF
4.27
4.36
11.45
QPA
4.26
3.28
8.22
PoLiCER
5.88
4.68
9.35
We analyzed the computational overhead of PoLiCER for the pixel-based control experiments de- scribed in Section 5.4 Wall-clock training time measured across methods three representa- tive tasks using an XTX 4090 GPU with an 39-13900KF CPU. RL with pixel-basee states is inher- ently computationally expensive due environment rendering. DrQ-v2 employs action repeat of 2, executing each action for 2 environment steps with single rendering. In our setup, this results in approximately 11-12 seconds per 1,000 environment frames in DMControl and 14-15 seconds per 500 frames in Meta- Woold. This rendering cost dominates training time, with DrQ-v2 (ground-truth rewards) requiring 2.5-3.5 hours for 1M frames in DMControl and 7.2 hours for 0.8M frames in Meta-World Window Open. As shown in Table [5] all PbRL methods incur additional overhead from reward model updates, with SURF requiring extra computation for semi-supervised learning and PoLiCER introducing overhead through CER CER-triggered increased replay ratio after resets. Po- LiCER's overhead primarily stems from CER rather than PLS. PLS adds only 2 x L x K policy forward passes per feedback session (Appendix D.II). D.I.. substantially less than DS. CER dynamically increases the replay ratio after resets during the feedback collection phase, returning to once all feedback is collected (Algorithm 3 lines 13-15). Therefore, The overhead magnitude varies across feedback schedule and reset frequency. While PoLiCER incurred approximately 8-9% additional training time, it achieved approximately 80% success rate, doubling the performance of PEBBLE
24

---

Published as a conference paper at ICLR 2026
on Window Open. This overhead remains modest relative to the inherent rendering costs and reward learning that dominate pixel-bacc base-based// PbRL.
D.6 HUMAN- TH-- EXPERIMENT PROCEDURE
We recruited volunteers consisting of undergraduate and graduate students not majoring in RL or robotics, ensuring preference judgments reflect intuitive human understanding of natural motion without domain expertise. Each participant annotated trajectories for a single run of both PEBBLE and PoLiCER ensure unbiased comparison (Muslimani & Taylor 2025). At each feedback ses- sion, participants were presented with pairs of trajectory videos displayed side-by-side for direct comparison. Each video clip spanned segment length of 50, corresponding 2.5 seconds at 20 frames per second (fps). We used a lateral camera view at 96 x 192 resolution for Walker Walk, while employing a dual-view setup combining corner and top-down perspectives vertically into single 288 x 288 resolution for Window Open, as the side view alone proved insufficient for judging spatial relationships between the robot arm and window handle (refer to Figure 19). Participants could replay videos repeatedly before making a decision. After 10,000 steps of unsupervised pre- training (Lee et al. 202Ib), we collected preference labels at regular intervals throughout training. For Walker Walk, gathered 10 queries every 20,000 steps (totaling 100 labels), while Window Open required 15 queries every 10,000 steps (totaling 240 labels). These intervals correspond to ap- proximately 10 minutes of wall-clock time in both environments. Completing all feedback sessions required approximately 1,8 hours for Walker Walk and 2.6-3.0 hours for Window Open.
WalkerWalk
Window Open
Figure 19: Snapshot examples of query pairs human annotators in Walker Walk (left) and Window Open (right).
To ensure consistent and meaningful feedback, we provided clear task-specific criteria to annotators for each environment. Participants were instructed to view both trajectory clips simultaneously and select the relatively better trajectory. We informed them that their feedback was actively shaping the agent's ' learning and that both clips might show suboptimal behavior early in training. While partic- ipants were encouraged to choose the relatively better one even in such cases, an equal preference option was available for genuinely ambiguous cases where trajectories appeared indistinguishable in quality. Annotators were instructed to evaluate trajectories in the following priority order:
Walker Walk
If fallen, prefer the trajectory that recovers to standing position more rapidly
If moving forward, prefer coordinated alternating leg motion rather than limping or relying one leg
When standing moving forward, prefer maintenance of upright torso posture.
Window Open
At the initial stage with the window closed, prefer trajectories where the end-effector moves closer to the handle
25

---

Published as a conference paper at ICLR 2026
If the handle is contacted, prefer leftward sliding motion that fully opens the window (from corner view)
Throughout manipulation, prefer smooth and direct movements without excessive jitter or unnecessary detours
While pursued fair and controlled HiL experiment by providing concrete evaluation crite- ria, some exceptional cases not specified led participants to express varying preferences for similar query pairs, resulting mixed personalized preferences. We present two representative examples in Window Open below to identify critical considerations for improving guideline design and anno- tation consistency in real-human experiments. Figure 20 (a) illustrates state-action credit assignment ambiguity, where one trajectory showed active sliding motion with less opening currently while the other achieved greater opening but remained stationary: annotators disagreed on whether to priori- tize current action state. Figure 20 (b) shows heterogeneous stage comparison difficulty, where each trajectory performed well at different task stages (approaching sliding), making direct com- parison ambiguous without cross-stage evaluation priorities. These cases highlight the need for more detailed guidelines that explicitly specify stage-specific priorities and clarify state-action trade-offs in evaluation criteria. Even with such mixed preferences across annotators, PoLiCER demonstrated more consistent performance than PEBBLE by mitigating primacy bias through CER's s periodic resets, preventing the reward model from overfitting individual annotator preferences.
Almost sliding motion with partial opening
Nearly full opening stationary
(a) State-action credit assignment ambiguity
Approaching the window handle
Actively sliding the bandle
(b) Heterogenaous stage comparison difficulty
Figure 20: Representative examples of mixed personalized preference cases that led to inter- subject annotation disagreement: (a) State-action credit assignment ambiguity and (b) Heteroge- neous stage comparison difficulty.
D.7 EFFECT OF HYPERPARAMETERS
We investigated PoLiCER's sensitivity key hyperparameters in DMControl Walker Walk; the temperature parameter for PLS and the initial reset threshold Bo for CER. Figure 21 presents the final performance achieved using 100 feedback instances across runs for various hyperparameter settings.
For the sampling temperature {0.0, 1.0, 1.5, 2.0). our results showed that 1.0 yielded the best overall performance. This moderate a value effectively balanced prioritization of policy- aligned queries with sufficient exploration diversity. Very high alpho values (1.5 and 2.0) provided only marginal improvements, likely because of overly concentrated sampling that limited explo- ration. Although not shown the figure, we observed that higher a accelerated early-stage learning, but this advantage diminished over time compared uniform sampling (a 0.0).
For the initial reset threshold S6. we found a clear trade-off. Lower 30 triggered more frequent re- sets, which reduced long-term bias accumulation but introduced short-term instability in TD learn- ing (Nikishin et al.) 2023). Higher 3, reduced computational costs by decreasing reset frequency but allowed primacy bias from early, potentially low-quality feedback persist longer. Despite these competing factors, PoLiCER maintained consistent performance across broader range of 3o values, demonstrating robust performance without requiring precise hyperparameter tuning.
26

---

Published as a conference paper at ICLR 2026
1000
1000
800
Episode Netur
Episade Return
600
600
400
400
200
200
15
20
25
(a) Sampling weight temperature a
(b) Initial reset threshold Bo
Figure 21: Final performance Walker Walk for PoLiCER using 100 feedback instances across ten
runs: (a) Effect of sampling weight temperature a for PLS and (b) Effect of initial reset threshold 30 for CER. Error bars indicate standard deviation across runs.
E LARGE LANGUAGE MODEL USAGE
Large Language Model (Claude, developed by Anthropic) was used to aid and polish the writing of this manuscript. Specifically, the LLM used for:
Grammar checking and correction
Improving sentence clarity and readability
Suggesting alternative phrasings for better expression
The LLM used solely writing assistance tool and did not contribute research ideation, methodology development, experimental design, data analysis, or any core technical contributions. All research ideas, algorithmic innovations, experimental results, and scientific conclusions en- tirely the work of the human authors.
27


Reviews:
[{"review_id": "k6Yqh54Qiz", "reviewer": "Reviewer_i7YB", "summary": "This paper proposes PoLiCER, which introduces two mechanisms to improve preference-based reinforcement learning. 1) Policy Likelihood-based Sampling (PLS) selects feedback queries most aligned with the current policy. 2) Critic-Exploited Reset (CER) prevents reward overestimation from primacy bias by adaptively resetting the reward and critic networks. The goal of PLS is to ensure feedback queries remain representative of the current policy, avoiding outdated or irrelevant samples. Compared to recency-based methods, it directly measures alignment between data and current policy. It is more computationally efficient than disagreement sampling, requiring only $2 \\times L \\times K$ forward passes than $N$ for disagreement sampling. And it does not increase training cost. The goal of CER is to counteract primacy bias, where the reward estimator overfits to early feedback and inflates Q-values, leading to overoptimistic policies. It dynamically stabilizes reward learning by resetting networks only when critic overestimation is detected. Experiments are conducted on Meta-World and DMControl tasks. Authors compared to several existing baselines and show improvements.", "questions": "It seems that PoLiCER has more empirical performance gain in pixel-based environments, is there any intuition on why this happens compared to state-based environments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "PoLiCER offers several strengths over prior preference-based reinforcement learning methods such as disagreement sampling and recency-based query selection. Its Policy Likelihood-based Sampling (PLS) improves data–policy alignment by measuring how representative each trajectory is under the current policy, rather than assuming recency implies relevance. This allows the model to select feedback that directly reflects its current behavior, improving sample efficiency and policy convergence.\n\nAnother advantage is computational efficiency. Unlike ensemble-based disagreement sampling, which requires multiple forward passes per query, PLS operates with a fixed, small number of policy evaluations. Its inverse-rank likelihood weighting also provides robustness to outliers, enabling stable performance across diverse continuous control tasks. Together, these factors make PoLiCER both efficient and reliable in selecting informative feedback.\n\nPoLiCER’s second component, Critic-Exploited Reset (CER), effectively mitigates reward overestimation caused by primacy bias. Instead of using fixed reset intervals, CER dynamically resets the reward estimator and critic only when critic outputs exceed an adaptive threshold. This approach reduces overestimation while allowing normal learning to continue when stable, leading to better long-term returns and less training disruption.\n\nOverall the writing is clear and easy to follow. Experimental setups are well explained, together with baseline methods. In the DMControl suite, it achieved performance comparable to QPA and clearly outperformed earlier PbRL methods like PEBBLE, SURF, RUNE, and MRN. In the more challenging Meta-World benchmarks, PoLiCER further distinguished itself by achieving over 80% average success, significantly higher and more consistent than competing methods.", "weaknesses": "1. PLS depends on accurate policy likelihood estimation, which may be unreliable in highly stochastic or multimodal policies. \n2. The rank-based weighting, though robust, can blur distinctions between highly relevant and moderately relevant samples. \n3. CER’s adaptive resets introduce temporary instability as networks reinitialize, requiring careful tuning of replay ratios. \n4. Because PoLiCER omits ensemble-based uncertainty modeling, it may handle noisy or inconsistent human feedback less effectively than Bayesian or disagreement-based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "37AtCEXjna", "reviewer": "Reviewer_vBkx", "summary": "PoLiCER addresses two central challenges in preference-based RL: query-policy misalignment and reward overestimation caused by primacy bias. It proposes Policy Likelihood-based Sampling (PLS) to select queries aligned with the current policy and Critic-Exploited Reset (CER) to dynamically reset the reward model and critic when overestimation is detected.", "questions": "Does the PLS likelihood score bias toward low-entropy policies? It is recommended to add sensitivity experiments on policy entropy/temperature.\n\nIs a diversity constraint necessary?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper focuses on (1) query–policy misalignment in query sampling and (2) reward overestimation caused by primacy bias in reward learning, which are key issues of high importance in PBRL.\n\nThe design of the approach is reasonable, including the likelihood computation in PLS and the CRITIC-EXPLOITED RESET mechanism; it is simple and low-cost.\n\nIt shows clear performance advantages on both vector-observation and pixel-observation tasks, and provides human-in-the-loop validation along with fairly comprehensive ablations.", "weaknesses": "PLS is sensitive to policy entropy/scale and may prefer low-entropy trajectories, reducing diversity.\n\nThe discussion of related work could be more complete. PPE~[1] advocates proximal policy exploration to expand the coverage of the preference buffer and improve reward model quality; like this paper’s PLS, it aims to make labeled data/queries closer to the current policy, but PPE emphasizes active generation/exploration, whereas PLS passively selects from the replay buffer and ranks by likelihood. The difference and advantage of PLS over PPE should be discussed in the manuscript.\n\nMinor issues such as typos: “primary bias” vs. “primacy bias”; a unified wording is recommended.\n\n\n[1] Zhu, Y., ... . (2024). Optimizing reward models with proximal policy exploration in preference-based reinforcement learning. In NeurIPS 2024 Workshop on Behavioral Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4VH0slCQlp", "reviewer": "Reviewer_SQmv", "summary": "This paper presents a new method called PoLiCER to improve query efficiency in previous PbRL work. It proposes two improvements: policy likelihood sampling (PLS) aims to make the reward model focus on the trajectories that are likely under the current policy; critic exploited reset forces the reward estimator to reset occasionally to reduce overestimation bias that emerges from repeated feedback over the same region of behaviour. Experiments involved in locomotion and manipulation show that PoLiCER outperforms prior PbRL algorithms.", "questions": "1. What are the details in human-in-the-loop experiments? How did the volunteers provide feedback? I hope the authors can provide some examples to prove that their experiments are fair, i.e., this feedback may show personalized preferences or mixed preferences, and PoLiCER can handle them. Meanwhile, explain what factor will affect the process.\n\n2. In the main experiments, why is QPA better than PoLiCER in Hammer? PLS improves QPA's sampling method, so PoLiCER should definitely be better than QPA in all experiments, but the results are not.\n\n3. What is the time consumption for pixel-based tasks? Since these tasks generally take more time, it is important to report your time consumption and compare it with previous methods.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Preference-based RL is a growing area with practical relevance. This work provides a new perspective to improve the algorithms, promoting the development of the field.\n\n2. The motivation is clear, and the authors use experiments to demonstrate that these issues are indeed important, which enhances the readability and coherence of the paper.\n\n3. The two mechanisms the authors propose are beneficial to each other: reset the critic and reward model to make them more plastic to adapt to preference data that is more consistent with the current strategy.\n\n4. The authors provide empirical evaluation across multiple tasks (locomotion + robot manipulation), which helps test generality.\n\n5. The code is included in the Supplementary Material, demonstrating the reproducibility of this work.", "weaknesses": "1. PLS seems to lack some theoretical insight to support. It is not clear why directly maximizing the policy likelihood can find more informative trajectories.\n\n2. Human-in-the-loop experiments are relatively simple.\n\n3. In the main experiments, using TA as another trick to the algorithm may cause SURF to completely become an ablation of this method, and the additional setting of the ratio may lead to unfairness in the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)