Pref-GUIDE: Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learning
Authors: Zhengran Ji
Boyuan Chen
Venue: Accepted by TMLR ()

Abstract: Training reinforcement learning agents with human feedback is crucial when task objectives are difficult to specify through dense reward functions. While prior methods rely on offline trajectory comparisons to elicit human preferences, such data is unavailable in online learning scenarios where agents must adapt on the fly. Recent approaches address this by collecting real-time scalar feedback to guide agent behavior and train reward models for continued learning after human feedback becomes unavailable. However, scalar feedback is often noisy and inconsistent, limiting the accuracy and generalization of learned rewards. We propose Pref-GUIDE, a framework that transforms real-time scalar feedback into preference-based data to improve reward model learning for continual policy training. Pref-GUIDE Individual mitigates temporal inconsistency by comparing agent behaviors within short windows and filtering ambiguous feedback. Pref-GUIDE Voting further enhances robustness by aggregating reward models across a population of users to form consensus preferences. Across three challenging environments, Pref-GUIDE significantly outperforms scalar-feedback baselines, with the voting variant exceeding even expert-designed dense rewards. By reframing scalar feedback as structured preferences with population feedback, Pref-GUIDE, offers a scalable and principled approach for harnessing human input in online reinforcement learning.

Published in Transactions on Machine Learning Research (10/2025)
Pref-GUIDE: Continual Policy Learning from Real-Time Hu- man Feedback via -aseed Learning
Zhengran Ji Duke University
shengran.ji@duke.co edu
Boyuan Chen Duke University
boyuon. www.on..cen@duuke... edu
Project Website: http:///aaaaaoooooocccaooooooooa
Reviewed on OpenReview: http:://ppaaarrieeeeee////////er
Abstract
Training reinforcement learning agents with human feedback is erucial when task objec- tives are difficult to specify through dense reward functions. While prior methods rely offline trajectory comparisons to elicit human preferences, such data is unavailable in on- line learning scenarios where agents must adapt the fly. Recent approaches address this by collecting real-time scalar feedback to guide agent behavior and train reward models for continued learning after human feedback becomes unavailable. However, scalar feod- back is often noisy and inconsistent, limiting the accuracy and generalization learned rewards. We propose PREF-GUIDE, a framework that transforms real-time scalar feedback into preference-basee PREF-GUIDE Individual mitigates temporal inconsistency by comparing agent behav- iors within short windows and filtering ambiguous feedback. PREE-GUIDE Voting further enhances robustness by aggregating reward models across population of users to form con- sensus preferences. Across three challenging enviromments, PREF-GUIDE significantly out- performs cealar-feedbart baselines, with the voting variant exceeding even expert-designed dense rewards. By reframing scalar feedback as structured preferences with population feod- back, PREF-GUIDE offers scalable and principled approach for harnessing human input in online reinforcement learning.
data improve reward model learning for continual policy training.
Human-in-the-kee phase
Real-time human scalar feedback (noisy, inconsistent)
Temporal consistent preferences
Evaluator- specific reward model
Popplltionninfommee consensus reward model
fost-human-guideee continual training
Figure Pref-GUIDE. Real-time scalar human feedback is often inconsistent, noisy, and varios across individuals. PREF-GUIDE addresses this by (a) converting scalar feedback into local pairwise preferences to achieve temporal consistency, and (b) aggregating reward models across human evaluators to form consensus- based rewards. (c) These improvements yield more robust reward learning and enable effective continual policy training after human feodback becomes unaviilable.

---

Published in Transactions on Machine Learning Research (10/2025)
1 Introduction
Incorporating human feedback into reinforcement learning (RL) has become increasingly important for train- ing agents in environments where task goals are difficult to formalize Christiano et al. (2017); Cao et al.
(2021]: ps Rafailov et al. (2023); Ramesh et al. (2024); Zhang et al. (0024b); Ji et al. (2025). While manually designed reward functions can work well in constrained or well-specified settings, they often fail to capture
the full complexity of real-world objectives or align with human intent. In such cases, human feedback offers
a more flexible and adaplive alternative for guiding agent behavior.
Two major paradigms have emerged to leverage human feedback in RL. Preference-based RL Christiano
et al. [017); Cao et al. (2021;; Ouyang et al. (2022): RRfailov et al. (2023); ; Ramesh et al. (2024) uses
offline datasets of pairwise trajectory comparisons to train reward models that reflect human judgments.
These models are then used to generate reward signals during policy learning. Alternatively, real-time
scalar feedback MacGlashan et al. (2017); Arunugam et al. (2019); Knox & Stone (2009); Warnell et al.
20018) Zhang et al. (2024b) allows humans to directly influence the learning process by providing moment-to-
moment evaluations during agent interaction. GUIDE Zhang et al. (2024b), recent method in this category, introduces a two-phase framework: during the -nthe--n- phase, scalar feedback is combined with enviromment rewards to guide agent learning; in the subsequent post post-human guidance phase, regression model trained the collected human feedback continues to guide the agent after the human input is no longer available.
While effective in early demonstrations, both paradigms face critical limitations. Preference-based methods rely on rich trajectory sets and offline human labeling, making them unsuitable for real-time and interactive scttings Wirth et al. (2017a). Morcover, the co-cvolution of the policy and reward model can introduce instability Wirth et. al. (2017a). Real-time scalar feedback, while more adaptable, suffers from two key challenges, especially in obtaining an effective reward model for post- hitt--huaannuudnnce phase:
Human feedback is inconsistent over time. Evaluators naturally shift their expectations over the course of training Wang et al. (2022), which makes it difficult to learn stable reward functions from raw feedback. For example, in an object search task, exploratory behavior may initially be rewarded but later penalized once goal-directed behavior is expected. This temporal inconsistency introduces non-stationarity, making it difficult for regression model to generalize feedback into a stable reward model.
Human feedback is not always reliable. Variability in cognitive ability, fatigue, attention lapses, and evaluation Wang et al. (2022); ; Wirth et al. (2017a); Zhang et al. (2024a] by contribute to intra- and inter- evaluator noise. These inconsistencies reduce the signal-to-noise ratio in scalar feedback, ultimately degrading policy performance when learned directly.
Our goal in this study is to improve the post-human-guidance phase of real-time human-guided RL. While we adopt the same human-in-the-loog phase as GUIDE, our contributions focus on transforming and aggregating the collected human feedback to support more stable and effective continual training without requiring additional human input. To this end, we propose PREF-GUIDE (Figure D framework that transforms noisy real-time scalar feedback into structured preference data to train more robust reward models. PREF-GUIDE consists of two key components:
Pref-GUIDE Individual transforms scalar human feedback from each evaluator into temporally local pairwise preferences. This design is motivated by our observation (Section 4.1) that human feedback tends to be relatively consistent within short time intervals. By comparing agent behaviors within short temporal windows, we generate multiple preference pairs from each scalar signal, which effectively reuses feedback to create more training data while maintaining temporal coherence. We further introduce no-preference margin Lee et al. (2021b) to handle ambiguous or low-confidence foedback.
Pref-GUIDE Voting builds on this idea by aggregating individual reward models across a population of human evaluators. Since feedback quality varies across individuals due to noise or evaluator bias [Wang] ct al. (2025); Meyers et al. (2023); ; von Rueden et al. (2015). we use a consensus- relabeling stratogy extract a more robust reward signal. By combining predictions from multiple evaluator-specific models, we reduce individual noise and improve the robustness of the learned reward (Figure
2

---

Published in Transactions on Machine Learning Research (10/2025)
We evaluate PREF-GUIDE across three challenging visual RL environments Zhang et al. 2024a bp. where agents must act based on partial visual observations. Our results show that PREE-GUIDE Individual outperforms regression-basee baselines when feedback quality is high, and PREF-GUIDE Voting maintains strong performance across diverse user inputs. In more complex tasks, our method even surpasses policies trained with expert-designed dense rewards, demonstrating the power of structured and population preference learning from real-time human foodback. Extensive ablation studies further validate our design choices and provide insight into their contributions.
By converting noisy and unreliable human signals into structured and pppulation-aggregattn preference data, GUID- offers a scalable and principled framework to leverage human feedback in RL, towards more consistent improvements even after human supervision ends.
2
Related Work
2.1 Real-Time Hlman--n-tthe-Lan Reinforcement Learning
Incorporating real-time human feedback into RL has been studied through several paradigms, each grounded in different assumptions about the nature and role of feedback. One major class of methods, such as TAMER Knox & Stone (2009) and Deep TAMER Warnell et al. (2018), treats human input as one-step reward signal, allowing agents to update their policies myopically based on predicted human evaluations. A second line of work, such as COACH MacGlashan et al. (017)) and Deep COACH Arumugam et al. (2019), interprets feedback as an advantage estimate, providing a more temporally grounded signal that can reduce the effects of inconsistent. feedback. More recently, GUIDE Zhang et al.] (2024b) has shown that using real-time scalar human feedback directly as a dense reward signal can provide superior performance in challenging visual RL
tasks.
On the other hand, while prior approaches have shown the promise of real-time human feedback in RL, they typically assume contimuous human involvement. GUIDE overcomes this assumption by training regression reward model from the human feedback data to provide continual training signals after the human feedback is no longer available. However, there are two key limitations in GUIDE. First, GUIDE relies on point-wise scalar prodiction, which limits its ability to capture the evolving nature of human ovaluations. Socond, GUIDE's performance strongly correlates with the individual human evaluators, where agents guided by human evaluators with stronger cognitive skills in certain aspects perform much better. To have human- guided RL algorithms agnostic to human individual differences, such correlation is undesirable.
Our work PREF-GUIDE builds on the real-time human-guided RL framework proposed in GUIDE, but addresses these limitations through two novel algorithm designs. PREE-GUIDE considers that human evaluation criteria evolve over time by grounding scalar feedback into pairwise preference labels within a short time window. Moreover, to mitigate human biases and individual differences, PREF-GUIDE aggregates individual reward models into consensus-based labels, improving the robustness of the reward model based on the feedback from population users.
2.2 Preference- Based Reinforcement Learning
Preference- based RL (PbRL) Wirth et al.] (201bb) obtains human feedback to train RL agents by collect- ing pairwise trajectory comparisons and in inferring latent reward function by modeling human preferences Bradlcy & Terry| (1952); Christiano et al. (017). The learned reward model will be used to supervise policy learning with dense reward estimates.
A rich body of work has expanded the PbRL framework to improve data efficiency and label quality. This includes integrating human demonstrations for pretraining [barz et al.) (2018). introducing soft labels to capture uncertainty Cao et al. (2021), leveraging relabeling and unsupervised pretraining Lee el al. 2021a), and more recently, using foundation models to automate preference labeling Wang et al. (2024); Jian et al. (2025). These methods typically operate in an offline or evolution loop setting, where the agent's policy is periodically updated based on human feodback collected over batches of offline rollouts Christiano et al. 2017); Tharz et al. (2018): Lee et al. 2202a bj.
3

---

Published in Transactions on Machine Learning Research (10/2025)
However, such iterative co-evolution of policy and reward model can introduce training instability, as both the learned behaviors and leedback targets shift over time Wirth et. al. (2017a). Moreover, the requirement for human queries on offline datasets or parallel policy rollouts makes it unclear to apply PbRL approaches directly in real-time decision-making tasks.
Our method, PREE-GUIDE, adapts the strengths of PbRL. to the real-time feedback setting. Rather than requiring humans to explicitly compare trajectories, we transform real-time scalar feedback, naturally pro- vided in real-time settings, into temporally grounded pairwise preferences. This allows to retain the expressivity and consistency benefits of PbRL without the overhead of repeated and offline query loops, Furthermore, while most existing PbRL work either trains evaluator-ppecffcc models or pools preference data from all evaluators without addressing inter-evaluator variability, PREF-GUIDE introduces a population- level aggregation mechanism. Through PREF-GUIDE Voting, we combine individual reward models into soft consensus labels, improving the robustness of the learned reward signal continual learning against individual differences.
3 Preliminaries
3.1 Teme-Time Human- Kuidan-Guieed Reinforcement Learning
real-time human-guided RL, a humnn observer provides scalar feedback while watching the agent interact with the environment. Let 7 (sp. ag, 01,0....... ST, ar) E (S x A)" denote a trajectory consisting of states and actions, where 8t E S and a, A represent the state and action at time step t. The human provides a scalar evaluation RR for the trajectory т, reflecting their assessment of the agent's behavior to the task objective.
Different methods adopt different formats for scalar feedback. For instance, in TAMER Knox & Stone 2009 f is a discrete value selected from 0, to represent negative, neutral, or positive responses. GUIDE Zhang el al. (024b) further improves this by using a continuous value in the range [-1,1] to capture more nuanced human feedback on spectrum from negative to positive.
GUIDE Zhang et al. (2024b) algorithm. GUIDE is recent framework in real-time human-guided RL to enable higher performance and faster convergence of RL agents using real-time human feedback. There are two main phases in GUIDE:
Human-in-the-loop phase: The continuous value feedback signals from human evaluators are directly combined with sparse terminal rewards to guide the policy training. Meanwhile, these feedback signals are stored as a dataset Dreal-time {(T,, f.) for training a reward model with a regression loss.
Post-humann-uidannce phase: Once human evaluators exit. the loop, the reward model will be used to estimate human feedback to continue guiding the training.
3.2 Preference- Basedd Reinforcement Learning
Preference- based reinforcement learning (PbRL) is commonly used when pairwise preference labels are avail- able, either from existing offline datasets or from parallel rollouts that allow trajectory comparisons. Rather than relying on scalar feedback values, PbRL uses human judgments over pairs of trajectories to learn reward model. This reward model estimates latent reward values consistent with the provided preferences and can be integrated into standard reinforcement learning pipelines to guide policy optimization.
In PbRL, a preference dataset Dpref {(T,A,T, Mi) is collected by asking humans to compare pairs of trajectories. Each label ys indicates which trajectory is preferred: y; = if 7,A x T. = if > 7A and y 0.5 if the annotator expresses no preference.
We assume preferences are generated according to latent reward function re (S x A)T R, parameterized by 0. Following the Bradley-Terry model Bradley & Terry (1922), the probability that trajectory 7 is
4

---

Published in Transactions on Machine Learning Research (10/2025)
Preference
Pref-GUIDE Individual Reward Prediction
Pref-GUIDE Voting
Fox
Scalar
Reward Model
Preference
Reward Mode
Performance
Scaler
Preference
Feadback
Reward
d
Relabel reward via Voting
Observation
d
RL Agen
Environment
Pref-GUIDE Individual
Action
Reward Model
Pref-GUIDE Individual
Pref-GUIDE Voting
(c) Continual training in gest-human-guidanne phase
Figure 2: Method Overview. (a) PREE-GUIDE Individual converts rcal-time scalar feedback from each human evaluator into a localized preference dataset, then trains evaluator-specific reward models. (b) PREF-GUIDE Voting aggregates predictions from these individual models to relabel trajectory pairs through consensus voting, providing a population-informa informmed preference dataset and a robust reward model.
(c) The aggregated reward model is used to guide RL training during the post-human past-muman-guidance phase.
preferred over BB the softmax likelihood the reward model predictions:
exp(ro(T^))
P( > T") exp(re(rA)) + exp(ra(r#))
The reward model is trained by minimizing the binary cross-entropy loss over the set of preference pains:
Spret Σ [y log P(T > (.8) + yo) P( T()]
Once learned, the reward model ro can be used to generate dense reward estimates for new trajectories, enabling the use of standard RL algorithms for policy optimization.
4 Method
Our method, termed PREF-GUIDE, is designed to improve the post host-huma--guddane phase of real-time human-guided RL. While recent methods have focused on improving the initial phase where human evaluators provide real-time scalar feedback, our focus on what happens after the human is no longer available. This phase is critical for scaling real-time human-guided RI, as continuous human supervision, though intuitive and effective, is costly, time-consuming, and ultimately unsustainable. An overview of PREF-GUIDE is shown in Figure
To address this, PREF-GUIDE introduces two key improvements aimed at making better use of the lim- ited human feedback collected during training: (1) converting scalar feedback into temporally local pairwise preferences to enhance consistency and data efficiency, and (2) aggregating feedback across multiple human evaluators reduce individual noise and bias. These improvements are realized through two complementary modules: PREF-GUIDE Individual, which transforms individual fcodback streams into structured prefer- ence data, and PREE-GUIDE Voting, which combines reward models from multiple evaluators to generate robust, consensus- based labels. We describe each component in detail in the following sections.
Human guidance data. Our datasct Dreal-time comes from GUIDE Zhang et al. (0024)). which contains interactions from 50 human evaluators across three challenging visual RL environments: Bowling, Find Treasure, and Hide and Seek 1v1. The scale of this dataset is the largest human study so far in real-time human-guided RI. For each evaluator, the dataset includes 5 minutes of feedback in Bowling and minutes each in Find Treasure and Hide and Soek lv1. Each data point consists of a short trajectory comprising three consecutive image observations and the corresponding agent actions, paired with continuous scalar
5

---

Published in Transactions on Machine Learning Research (10/2025)
mastepp 800
TineStpp 411
Tirratiop:: 300
TimeStppo44b Human Feedbadic -0.23
TixSstp:: (60
TireStepr 1102
Human Feedbadc Fostarrie
Humen Feedbackc 0.60
Human "eedbard
Human Freebaae -0.01
Rexdingg
Find Trans
Hide and Beak 1v1
1.0
0.5
Human Reedbakk
Dimppision 2
Dimaniio2
Dimeraaion
-.5
1.0
Dimension
Dimension
Dimanscon 1
Figure 3: Human feedback is temporally inconsistent for similar trajectories. We visualize t-SNE embeddings of trajectory representations from evaluator 0 in all three tasks. Each point represents a trajec- tory, color-coded by it's corresponding scalar human feedback. Despite many trajectorics being behaviorally similar (i.e., embedded closely), their leedback values vary widely. This observation highlights the temporal drift and inconsistency in human evaluations, motivating our approach to convert scalar feedback into tem- porally local preference pairs.
feedback signal provided by the human. To support our continual learning experiments, we also utilized training checkpoints from the start of the post-human phase, which include saved replay buffers, model weights, and optimizer states.
4.1 Inspiration: Human Feedback is Inconsistent Over Time
When a human evaluator observes an agent learning, we hypothesize that the criteria used to provide scalar feedback may shift over time. Early in training, humans often reward exploratory or "trying" behaviors, but later they expect more goal-directed performance. As a result, the same agent behavior might be rated positively at one point and negatively at another.
To validate our hypothesis, we processed each human evaluator's Dreal-time by encoding the trajectories using pretrained image encoder, DINO Oquab et al. (2023), followed by a t-SNE Van der Maaten & Hinton (2008) projection for visualization. The colors represent different values of the human foedback. As shown in Figure there is no clear alignment, between similar trajectory embeddings and scalar feedback. This trend is less obvious in Bowling due to its relatively shorter horizon and rather static observations, compared with Find Treasure and Hide and Scek 1v1. However, it's clear in the other tasks that require longer horizons and explorations, human feedback is not consistent over time. This result supports our hypothesis that feedback criteria shift as humans adjust their expectations during guidance. Visualizations from other evaluators are provided in the Appendix A.I
This observation raises a concern for existing methods like GUIDE, which directly train regression model predict scalar feedback given trajectory input. If the same behavior is labeled inconsistently, regression-based models may fail to capture meaningful reward signals, resulting in unstable learning.
4.2 Pref-GUIDE Individual
To address the challenge temporal inconsistency in scalar feedback, we propose PREF-GUIDE Individual, which converts real-time scalar feedback Dreal-time into a preference- based dataset Dpref-
A naive approach to handling scalar feedback would be directly regress on across the full training sequence. However, this fails to account for the shift in human evaluation criteria over time, and learns from

---

Published in Transactions on Machine Learning Research (10/2025)
globally inconsistent and noisy signals. Our key insight is that, although feedback may drift over time, it tends to be relatively consistent within short local intervals. By focusing on temporally local comparisons and introducing mechanism to handle ambiguity, we construct more stable and informative training signals.
We introduce two techniques to realize this idea:
Moving Window Sampling: We define a window
W: = {(Ti,
containing n consecutive trajectory-feedback pairs. In practice, we set n = 10, which corresponds to 5 seconds of human guidance in our environments. Within this short window, we assume the human's cvaluation standard approximately stationary. We then generate (*) trajectory pairs from each window, assigning preference labels by directly comparing their scalar feedback values.
No Preference Range: Even within short windows, small feedback differences may not indicate meaningful preferences. Treating any difference as a strict ordering will lead to overfitting on noisy comparisons. To address this, we introduce a no-preference threshold 6, set to 5% of the total feedback range. the difference - < 6, the trajectories are treated as equally preferred and labeled with 0.5.
Using the resulting dataset Dpref we train an evaluator-specific reward model ( using the Bradley-Terry model from Section 3.2 for each evaluator j. This reward model is then used for continual learning once human feedback is no longer available. The full procedure is detailed in Algorithm
4.3 Pref-GUIDE Voting
While PREF-GUIDE Individual mitigates temporal inconsistency within single evaluator's feedback, it does not address the variability in feedback quality across individuals. In practice, human evaluators differ significantly in how they interpret agent behavior, the consistency of their feedback, and their attentiveness during the training process. As observed in GUIDE, some participants provide rich, informative feedback, while others are noisier, inconsistent, or even disengaged. When building real-world human-guided RT. systems, relying on any single individual introduces the risk of overfitting to that evaluator's unique biases, cognitive patterns, momentary lapses.
Our idea is to leverage the collective intelligence of multiple human evaluators to avoid the risk of overfitting. A straightforward approach might train a single reward model using pooled data from all evaluators. How- ever, this method fails to account for evaluator-specific noise and can conflate incompatible reward signals, leading unstable or diluted supervision.
To overcome this, we propose PREF-GUIDE Voting, which aggregates the predictions of independently trained evalnator-specffic reward models to produce consensus preference labels. By treating each model as an independent judgment source, we can extract commonalities across diverse feedback patterns and downweight idiosyneratic or unreliable signals. Our key hypothesis is that while any individual may be noisy, the aggregated signal across multiple evaluators captures more reliable preferences, resulting in a more robust reward function for continual policy learning. The full procedure detailed in Algorithm P
Specifically, instead of relying on the predicted reward from the original reward model, we query all evaluator- specific reward models га (S 50) for their reward predictions each other's input trajectories to cast vote based on their respective judgments. These votes are averaged and then normalized to yicld consensus label in the range of J0. 1], with intermediate values reflecting uncertainty or disagreement.
5 Experiments and Results
Our experiments aim to evaluate the elfectiveness of PREF-GUIDE in improving reward model learning for continual policy training. Our experiments were performed in three challenging visual RL experiments from GUIDE Zhang et al.] (2024b): Bowling, Find Treasure, and Hide and Scek 1v1. The tasks present partial observability and require long- morihorizon decision making, making them suitable for testing reward learning from human feedback.
7

---

Published in Transactions on Machine Learning Research (10/2025)
Bowling (Top 15)
Find Treasure (Top 15)
Hide and Seak TvT (Top 15)
70
70
00
60
50
Sucoess Hote (96)
Awarage Score
50
Guccess Rate
60
40
30
40
40
20
15
2%0
%
10
Bowling (Al 60)
Aind Treasure (A] 50)
Hile and Seek 1v1 (AI 50)
70
70
100
60
60
50
Success Fote (%)
Average Score
50
Sucosss Rote
60
40
30
40
40
GUIDE (human in the loop)
GUIDE
20
Pref- CUIfG Individuul
Pref-GUIDE Voting
DDPG
DDPG Heuristic
15
2%
Minutc
Minuto
Minute
Figure 4: Results. Each column shows a different task. The top row reports results using only the top 15 evaluators (high-quality feedback), while the bottom row includes all 50 evaluators (mixed feedback). Curves after the dash vertical line denotes the performance during post-human -guidance phase. PREF-GUIDE Individual outperforms GUIDE when the feedback quality is high, while PREF-GUIDE Voting gives the best results across all conditions and even surpasses expert-designed rewards more complex tasks.
Each run consisted of real-time -iman-in-the-loo phase, followed by a post-human-guidard phase for continual learning without human input. The human-in-the-loog phase lasted 5 minutes for Bowling and 10 minutes for Find Treasure and Hide and Seek 1v1. After this, agents continued train using learned reward models for 15 minutes for Bowling and 50 minutes for the other two tasks. To accurately track policy performance over lime, we saved the policies al regular intervals for performance evaluation: every minute for Bowling, and every 2 minutes for the other two tasks.
Following GUIDE, we adopt their grouping strategy in evaluations based on the cognitive test scores of human evaluators. Results are reported for two groups: the Top 15 evaluators and the All evaluators. It has been found that the top 15 evaluators who obtained higher cognitive test scores provide higher quality guidance training RL agents Zhang et al. (202bb). We conducted our ablation studics the top 15 group to isolate design contributions under controlled experiments.
Baselines and Evaluation Goals. We compare against the following baselines:
DDPG Lillicrap et al. (2015): the base reinforcement learning algorithm used in GUIDE as the RL backbone, trained only with sparse environmental rewards.
DDPG Heuristic: a version DDPG augmented with expert-designed dense rewards, applicable only Find Treasure and Hide and Sock 1vl. This serves as an approximate upper bound for the performance achievable with manually crafted rewards. Specifically, the dense rewards are designed based on distance measurements and exploration area. tracking.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Bowing
10min
15mrh
20mi
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.0
50
30
70
0.0
30
50
1.0
Find Treeaure
20min
1.0
40mh
1.0
60min
Peioerogg of Exdlatioos
0.8
0.8
0.4
0.4
0.2
0.2
0.2
0.0
50
40
60
0.0
50
1.0
Hido and Sook
20min
GUIDE
40ml
60mi
heficuuDE
0.8
0.8
Pref-quiE
0.6
0.6
0.6
0.4
0.4
0.0
75
55
95
0.0
75
Miesstnne
Figure 5: Pref-GUIDE Voting enhances robustness across different evaluators. Each subplot shows the percentage of evaluators (y-axis) whose agents reached specific performance milestones (x-axis) at different time points (title of each plot). Each row shows one task. GUIF-GUIDE Voting consistently enables a larger fraction of evaluators to train high-performing agents across milestones and time.
GUIDE Zhang et al. 2024b): the primary baseline, which uses regression model trained real-time scalar feodback to guide continual learning after human supervision ends.
Key Questions: These baselines allow to answer the following key questions:
Can PREF-GUIDE outperform existing ocalar-feedbade regression approaches in continual training?
How robust when foedback quality varies across evaluators?
Can learned rewards from human foedback rival or surpass expert-designed dense rewards?
5.1 Results
Results presented in Figure When trained using feedback from the Top 15 evaluators, PREF-GUIDE Individual significantly outperforms GUIDE in all environments. This confirms our hypothesis that con- verting scalar feedback into localized preferences leads to more consistent and stable reward models, given that the feedback of sufficient quality.
However, when evaluated across All evaluators, PREE-GUIDE Individual shows degraded performance in the Find Treasure environment, falling slightly below GUIDE. This highlights a key limitation of relying single-subject feedback, where biases and noise from low-quality evaluators can undermine learning, The next key question becomes how to tackle such challenges when group of evaluators without filtering is presented.
As shown in the results, PREE-GUIDE Voting indeed can maintain high performance across both evaluator groups. By aggregating reward models across evaluators, it effectively neutralizes individual biases and noise. Importantly, PREF-GUIDE Voting not only benefits the results when all evaluators are presented without pre-filtering but also performs the best even when the Top evaluators are selected with a cognitive test ranking. Furthermore, in both Find Treasure and Hide and Seek 1vl, PREF-GUIDE Voting not only

---

Published in Transactions on Machine Learning Research (10/2025)
Howling (lop
Hind Iresssue (lop
Hide and Sock TvT (lop
70
70
100
60
Averoige Soore
50
Success Hote (16)
50
Rata
60
40
(*uman-i--the-loop)
GUIDE
30
40
40
Prof- GUIDE Ind vidual
GUIDU Ind vidual
wil No. pwo.prefeeeeee Range
20
GUIDUI Ind Sampling wro Mixiing Window
Pruf-GUIDE Ind vidual
10
wio No. prefarence Range
Mivinng Window Sampling
15
20g
10
30 Minute
10
20
30 Minute
40
50
60
Minute
Figure 6:
We evaluate the two key design choices: moving window sampling and no preference range. Removing either component leads to a noticeable drop in performance.
surpasses GUIDE but even exceeds the performance of DDPG Heuristic, suggesting that collective human feedback, when properly structured, can yield reward models more effective than expert-designed rewards.
We further compared the performance of GUIDE, PREF-GUIDE Individual (ours), and PREF-GUIDE Voting (ours) in Figure 3 For each algorithm, we computed the proportion evaluators (y-axis) whose agents achieved given performance threshold (x-axis). Each row shows different task, and each column shows different training time points. PREF-GUIDE Voting consistently shows the highest performance in all settings. The results support our hypothesis that aggregating reward models at the population level yields more stable and robust supervision. This, in turn, leads to more reliable policy improvements given different evaluators.
5.2 Ablation Studies
Ablation studies Pref-GUIDE Individual. We conducted ablations to understand the importance of the two design choices in PREF-GUIDE Individual: (1) Moving Window Sampling, which localizes feedback comparisons, and (2) No Preference Range, which filters ambiguous pairs. As shown in Figure 6 removing either component leads to a noticeable drop in policy performance. Without the moving window, preferences are extracted from globally inconsistent foedback. The coheronce is reduced. Without the no-preference margin, the model overfits to subtle, noisy changes in the feedback values, mistaking noise for meaningful preferences. These results demonstrate that both components are essential for extracting consistent and robust training signals from scalar human feedback.
Ablation studies on Pref- GUIDE Voting. We also evaluated the design of our consensus relabeling strategy. Our method uses normalized vote aggregation where soft preference labels reflect the degree of agreement across reward models. We first compared this to a straightforward approach, which combined all the evaluators' data to train a single reward model and used this reward model for every evaluator's continual training. Furthermore, we compared this to simpler binary majority vote strategy, which ignores disagrecment magnitude and directly applics the majority vote dircction as the label.
As shown in Figure ? normalized aggregation yields consistently better performance across all three tasks. Even though combining all the data to train single reward model has better performance bowling, in more complex tasks, Find Trcasure and hide and Soek 1v1, it cannot outperform PREF-GUIDE Voting. Combining all the evaluators' data, although bringing more data during reward model training, also intro- duced conflicting labels on similar trajectories due to individual evaluator bias and noise to undermine the reward model. This suggests that capturing the confidence of the population, not just the summary of the majority, or combining all the data naively, results in smoother gradients and more informative training signals. These findings validate design choice for robust population-level reward learning.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Howling (lop
Hind Iressure (lop 15)
Hide and Sook TvT (lop
70
70
100
50
Success Hole (36)
Averoige Soore
50
60
40
30
40
40
20
GUIDE Voting
Fref-GUIDE Viting
wie harmelization
10
Pref-GUIDE individaal
10 Minute
15
20
10
20
30 Minute
40
50
60
10
20
30 Minute
40
50
60
Figure 7: We compared three reward model training variants using population leedback: (1) our method with normalized voting, (2) binary majority vote relabeling, and combining all feedback data into single training set. Normalized voting consistently outperforms others, highlighting the value of capturing the degree of agreement among evaluators rather than relying on binary or pooled labels.
Trajectory
Trajectory 2
C
Trajectory
Trajectory 2
0.78
0.22
0.07
0.93
D
vS
0.37
0.63
0.5
0.5
Figure 8: Examples of population voting on Find Treasure trajectories. (A-C) The population prefers exploratory behavior and movement toward discovered treasures. (D) No strong preference when both trajectorics are similarly exploratory. Normalized scores reflect consensus strength.
5.3 Qualitative Visualizations of Voting Preferences
To better understand the behavior of PREF-GUIDE Voting, visualize population-level preferences over several trajectory pairs from the Find Treasure environment. Figure shows that population preferences align well with task-rclevant behavior. For instance, in pair A, the population prefers the agent that actively explores the map over one that stalls near the starting area. In pairs B and C, the agent that moves toward the discovered treasure is preferred over one that moves away. In pair D, the population shows no preference between the two equally exploratory trajectories. These results indicate that the voting mechanism produces intuitive and consistent judgments.
5.4 Pref-GUIDE Voting Enables Reward Learning Agnostic to Individual Differences
To understand how different evaluator groups contribute the final reward model in PREF-GUIDE Voting, we analyzed the voting contributions of subgroups categorized by cognitive test scores: Top 15 (highest performers). Middle 20, and Bottom 15. Table 5.4 shows the proportion of each group's contributions to winning voles across three environments.
Interestingly, we find that cach subgroup's contribution roughly proportional its representation in the overall population. This suggests that no subgroup disproportionately dominates the voting outcomes and that the final preference labels reflect balanced aggregation across all evaluators.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Group
Bowling
Find Treasure
Hide and Seek 1v1
Top 15 (30%)
30.78%
29.85%
30.96%
Middle 20 (40%)
39.22%
40.38%
40.08%
Bottom (30%)
30.00%
29.77%
28.96%
Table 1: Distribution of winning voles across evaluator groups. Each group's contribution to the consensus aligns with its population size, indicating balanced influence across cognitive subgroups.
However, this observation raises a deeper question. When comparing the performance of PREF-GUIDE Voting wetween Top 15 and All 50 evaluators in Figure | they are quite similar. This not the case for other methods and baselines, where Top 15 evaluators always generate higher performance on the guided agents than All 50 evaluators. The question is: if the lower-performing groups in cognitive tests contribute equally to the voting outcomes, why does performance of Top 15 feedback alone?
Voting under All 15 evaluators still match the
We believe that the answer lies in the balancing effect of the consensus- baasen aggregation. While the Top 15 group alone yields stronger raw feedback signals, as reflected in the superior performance of PREF-GUIDE Individual on this subgroup, the PREF-GUIDE Voting mechanism can filter and dilute the noise intro- duced by less consistent evaluators, while still preserving useful signals from across the population. This trade-off results in robust reward model that performs similarly whether trained on just the Top 15 or on all 50 evaluators.
Our results suggest that PREF-GUIDE Voting makes the agent's performance more agnostic to the quality of individual human feedback. It balances signal and noise in a way that enables learning from a broad population, without being overly sensitive to individual differences among human evaluators.
6 Conclusion
We introduced PREF-GUIDE, a method that converts real-time scalar human feedback into preference- based data to train reward models for continual policy learning. PREF-GUIDE Individual mitigates temporal inconsistency by transforming contimuous feedback labels into moving-window window sampled preference labels. PREF-GUIDD Voting enhances robustness by aggregating reward models across individuals into consensus-driven signal. Across three visual RL environments, PREF-GUIDE Individual outperforms regression- baselines such as GUIDE when feedback is of high quality, and PREF-GUIDE Voting maintains strong performance even under noisy conditions, surpassing both GUIDE and expert-designed dense rewards. Together, these results highlight the scalability and stability of structured preference learning from real-time human input.
As for the limitation of our method, for the two design choices of PREF-GUIDE Individual: no-preference range and moving window, we assumed that the same set of parameters is sufficient model different human evaluators. Though our results showed that the same set of parameters performed well, these parameter selections may be further improved. For instance, more opti subject-specific no-preference ranges and moving window sizes could potentially be inferred through a warm-up phase of human guidance, Further- more, for every human evaluator, we trained a separate model. This model is rather small, and the additional training overhead is minimal. However, is possible that such a model can add more training overhead when the reward model size needs be large. Exploring more efficient reward models can be an interesting fu- ture direction. For PREF-GUIDE Voting, more algorithm improvements or more sophisticated aggregation methods can also be studied to incorporate the human guidance from all subjects.
References
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint Xin: 1902. 04257, 2019.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Zehong Cao, KaiChiu Wong, and Chin-Teng Lin. Weak human preference supervision for deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 32(12):5369 5378, 2021.
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodci. Doep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017.
Borja Ibarz, Jan Leike, Tobias Pohlen, GeolTrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Aduances in neural information processing systems. 31, 2018.
Zhengran Ji, Lingyu Zhang, Paul Sajda, and Boyuan Chen. Enabling multi-robot collaboration from single- human guidance. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 4272- 4279. IEEE, 2025.
Pingcheng Jian, Xiao Wei, Yanbaihui Liu, Samuel A Moore, Michael M Zavlanos, and Boyuan Chen. Lapp: Large language model feodback for preference-driven reinforcement learning. arXiv preprint arXiv:2504.15/77 2025.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge cupbure, pp. 9-16, 2009.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.065 050000 2021a.
Kimin Lee, Laura Smith, Anca Dragan, and Picter Abbeel. B-pref: Benchmarking preference-based rein- forcement learning. Xix preprinl arXiv:2111.0302 2021b.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra, Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.00 02971, 2015.
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International conference on machine learning, pp. 2285 2294. PMLR, 2017.
Maria Christina Meyers, Marianne van Woerkom, and Robin Bauwens. Stronger together: A multilevel study of collective strengths use and team performance. Journal of Business Research, 159:113728, 2023. ISSN 0148-2963. doi: http:://bb..rrr/100000//// octeec/c/arrttce////////66666333
URL httpt::://w...ccieeeeereeeeeee com/
Maxime Oquab, Timothée Darcet, Th6o Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Xiv preprint S5h:22344001900 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feodback. Advances in neural information processing systems, 35:27730-27744 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems,
2023.
Shyam Sundhar Ramosh, Yifan Hu, Iason Chaimalas, Viraj Mchta, Pier Giusoppe Sessa, Haitham Bou Am- mar, and Ilija Bogunovic. Group robust preference optimization in reward-free rlhf. Advances in Neural Information Processing Systems, 37:37100-37137 2024.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Christopher von Rueden, Sergey Gavrilets, and Luke Glowacki. Solving the puzzle of collective action through inter-individual differences. Philosophical Transactions of The Royal Society B Biological Sciences, 370, 12 2015. doi: 111098/rstb.2115.0000
Jiangtao Wang, Bin Guo, and Liming Chen. Human-in-tee in-the-ooooppp machine learning: macro-micro perspec- tive. orXiv preprint arXiv:2202. 10564, 2022.
Jianren Wang, Yifan Su, Abhinav Gupta, and Deepak Pathak. Evolutionary policy optimization. arXiv preprinl ar Xiv:25030 19037, 2025.
Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. Rl-vim-f: Reinforcement learning from vision language foundation model feedback. ar preprinl or 0338010
2024.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürukranz. survey preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(13)):1-46, 2017a. URL http: //jmlr.org/pan wwt..o/pppp...////eeoeooo--
634.html
reinforcement learning methods. Journal of Machine Learning Research, 18(136):1 46, 2017b. Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey preference-
Lingyu Zhang, Zhengran Ji, and Boyuan Chen. CREW: Facilitating human-AT teaming rescarch. Trans-
actions on Machine Learning Research, 2024a. ISSN 2835-8856. URL https://openrevier: www.:ewwwweeffrrrrrtee
1d=2RXvHRMmSi
Lingyu Zhang, Zhengran Ji, Nicholas R Waytowich, and Boyuan Chen. GUIDE: Real-time human- shaped agents. In A. Globerson, T. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomozak, and
C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 138959 138980. Curran Associates, Inc., 2024b. URL https://proceeddngs http://owwddnnnnoooooo...o..oooo fite:fefeappasseeeeereeeeeeeeeee orgpap/eeeeeettatttooeereeeooonr pdf
fillss/aperr/0221
14

---

Published in Transactions on Machine Learning Research (10/2025)
A Appendix
A.1 Visualization of Trajectories vs Human Feedback
We provided the visualization of trajectories vs human feedback for all evaluators in Figure 9 For each evaluator's plot (evaluator's id is on the top left corner for cach set of plots), the tasks are Bowling, Find Treasure, and Hide and Seek Ivl (from left to right).
Find Treassure
Hide and Sook
Hind Treaaure
Hide and Seek
Find
Hds and Seek
Find Trsasure
Hide and Seek
Bowling
1v1
Bowligg
1v1
Bowing
Treassure
1v1
Bowingg
1v1
3
19
18
20
21
22
29
1.0
26
0.5
Human Faedback
0.0
-0.5
-10
38
48
49
Dimension
Figure 9: Visualization of Trajectorics vs Human Foodback for All 50 Evaluators. There no clear alignment for most evaluators in the three tasks.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Furthermore, also visualize the trajectories that correspond to the neighbor points in the embedding space to further verify our hypothesis. In Figure 10 and T2 for each task, we uniformly selected 8 anchor trajectories, and found the closest trajectory to every one of them in t-SNE space that is at least 100 timesteps apart to avoid temporal similarity. We visualized these trajectorics along with the corresponding human leedback. As shown the resulting trajectories, despite that they are semantically and visually very similar or near identical within the same trajectory, the human feedback very distinct different timesteps. These observations show strong evidence that human feedback is not consistent.
TimeStep:: 150
Tm6Shpp:: 323
TimeStop:
TimaStopp SDV Hurren Feedback C.51
Human Feedzack 0.50
Human Feedback 0.28
Human Feadbeckk: 0.33
Timestap: 250
SimsShep: 397 Humen Feedbeck: 0:04
TimeStp:: Human Feecbaok: 0:48
TineStep: 365 Human Feedback 0.78
Human Feedback 0.83
TimeSkp::
TimeSop:: 242 Humar Facdbeekk: 0.32
Hurnen Feadbackk: 43,88
ImstShpp: 300 Feedbaaac 0.35
limelstap: 411 -0.73
TimeStsp Humna Foxsback 0.98
TineStep: Human Focdbade 0.87
TimeStop 350
Time:Sop:: 510 Human Feedbadlc
Human Feadbacic
Figure 10: Visualization of Bowling from Subject O. The trajectories that are semantically visually similar to each other at different timesteps receive very distinct human feedback values.
TimeStop: 418J
TimeStop: 380 Humon Foodback CLEI
TimeSapp Human Fenebbade 3.95
TareSlepr02 Humar Fandberkk: 0.18
Humon Foodhonk: 0.26
TimeSto:: 100
"Tmsflig: 567
IncSSip: 200
ThmsSSopp: 19
Fastbeck:
Human Feedbosic U.D4
Humer Feedbeck 0.48
Humer Heedbeok 0.00
llmoStopp: Human Feecback: 0.12
limaStop: Human Feedbaok: 0.25
TimeStep: 350
TireSteo8811 Human Foediback -..0
Humer Foodbask 0.08
TimaStep Human Feachack: 0.80
TimeStep 445
TimsStop Human Feartrack 0.84
TimeStap 296
Human Feedloacic 23
Human Focdbadk 0.18
Figure 11: Visualization of Find Treasure from Subject 0. The trajectories that are semantically visually similar to each other at different timesteps reccive very distinct human feedback values.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Timestec: 150
TimeShap: 420
TimeSipp:: 200
TimeStaci 0
Human Feedbock: 0.36
Hurnen
Humen Feasback 0,80
mestep: 100
Timr:Sooo::: 521
limeStap:: 400
TimeStp: 25
Human Feedback 0:38
Humen Fee:back: -0.15
Humen Fbodseenee
TimeStep: 250 Feecbecc
TimeStep Humen Feedbaakk 0..01
TimpShop 350
TimeSlep: 5ra
Humain Feedback -0.74
motiloo: SE3T
TineSkp: 1100
TrruCtepp 50
TimeS Foortback .0,21
Humman Heacitacko 0.38
+0C1
Human Feedbacic +0.1
Visualization of Hide and Seek lv1 from Subject 0. The trajectories that are semantically visually similar to each other different timesteps receive very distinct human feedback values.
Figure 12:
A.2 Pref-GUIDE Individual Algorithm
The algorithm of PREF-GUIDE Individual is summarized in Algorithm 1
Algorithm PREE-GUIDE Individual
Require: Real-time dataset Dreal-time = ((Ti, JO) window size n, no preference range 1: Initialize D Preff + 0, rg
for i = N n do Define window W;
[T)} 5= 4: for each pair (r (A), (7" {B)
+n-1
E W; such that A < B do
if [ - pB < then
Prref 5- pret
P U[((TA (()(TAA))))0.)))
clsc if f > fB then
D Dpref
pret U [[((T, (,,,,, 1)) 1))
else
10: D preff + prer U ,(TT,,11)))))))) )),
11: end if
12: end for
13:
end for
14: Train re using Dpref
15: Use тя for continual learning
A.3 Pref-GUIDE Voting Algorithm
The algorithm of PREF-GUIDE Voting is summarized in Algorithm
2
17

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 2 PREF-GUIDE Voting
Require: Real-time datasct Dreal-time {(T, fi)l=n, window size n, no preference range 8, evaluator-
specific PREF-GUIDE Individual reward models {
Initialize pree-vvoting <-) 0
for == to N - m+1 do
3: Define window W; {TTj, (j)
+
m=i
4: for each pair (f (T
E W; such that A < B do
5: Initialize vote count 6- 0
for j= = to S do
if Ire (r - (T < d then
(1)
7:
8:
c+
else if 18 (T4)
(1)
(T then
0
10:
ct c
11: else
12:
c
13:
end if
14: end for
D prret-voting U(((TA,BB), 5)
15: end for
16: end for
17: Train final voting-based reward model Rg using D pree--voting
18: Use Rs for continual learning
A.4 Experiment Details
Each experimental result was averaged over seeds. The dense rewards for Hide and Seek 1vl and Find Treasure were defined based on the visibility of the treasure or hider and the distance between the agent and the target.
A.5 Hardware Requirement
We conducted the experiments using NVIDIA A100 and NVIDIA RTX A6000. However, our experiment can run on a single desktop with a single GPU, such as NVIDIA RTX 4070.
18


Reviews:
[{"review_id": "sEHKSgBTXp", "reviewer": "Reviewer_9hr6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Pref-GUIDE, a framework for human-in-the-loop continual RL. Pref-GUIDE addresses two key limitations from standard HITL with dense rewards: (1) the rewards from a single evaluator is not consistent, and (2) the rewards from multiple evaluators are not consistent. To address the first limitation, the paper proposes to convert dense rewards in a small time window to pairwise preferences and then fit a reward model to the preference labels (Pref-GUIDE Individual). To address the second limitation, the paper proposes to combine the individual preference reward models using a voting mechanism, and then fit an aggregate reward model to the voting-relabeled preferences (Pref-GUIDE voting). The paper evaluates Pref-GUIDE on three visual RL domains: Bowling, Find Treasure, and Hide and Seek 1v1. Compared to scalar human reward baselines (GUIDE) and standard sparse / dense reward RL baselines (DDPG), Pref-GUIDE shows strong robustness to noisy rewards, and often surpasses the performance of expert-designed dense rewards. This offers a promising pathway towards scalable human-in-the-loop RL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is motivated by concrete limitations of existing human-in-the-loop RL methods, namely, reward ambiguity. The proposed methods are based on clear insights, i.e. that individual evaluators have stationary preferences within a short time window, and that different evaluators can be reconciled via a voting mechanism. The empirical results convincingly demonstrate the effectiveness of the proposed methods.", "audience_yn": "Yes", "audience_explanation": "Addressing the reward ambiguity problem in HITL is of immense interest to the machine learning and reinforcement learning communities.", "requested_changes": "1. The paper could benefit from a discussion of the limitations of the method, e.g., when does the time window assumption fail?\n2. Some experiment details are omitted, e.g., how many seeds are in each experiment? What are the compute requirements? What are the exact dense rewards used in DDPG-Heuristic? Adding these details would help the reader better calibrate the results.", "broader_impact_concerns": "The paper does not have ethical implications that warrant a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VB2onj84Oz", "reviewer": "Reviewer_Pf2B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces pref-GUIDE, a method for addressing human evaluator inconsistency and individual bias during real time feedback in preference based RL. This method builds on GUIDE and focus on the \"continual\" learning when the human evaluator is no longer available and only the dataset from the previous human-in-the-loop phase is available. The proposed method for addressing temporal inconsistency is by splitting human feedback into temporally adjacent windows and generating preference pairs from the windows. The proposed method for addressing individual bias is by training a reward model on each evaluator's data and aggregating the reward scores during continual policy training. Experiments are conducted on partially observable visual RL environments. The results demonstrate the advantage of the proposed methods over GUIDE and DDPG trained on true sparse environment rewards and expert designed heuristic rewards. Ablations demonstrate the necessity of the proposed improvements over GUIDE.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments show clear advantage of the proposed methods (e.g., figure 5). The ablations further demonstrate the contribution of each proposed components.", "audience_yn": "Yes", "audience_explanation": "This paper proses an improvement over an existing algorithm for real-time learning from human feedback, which is a topic of general interest. The focus on adapting to human bias and inconsistency is a popular topic within this community.", "requested_changes": "* What's \"real-time\" about this feedback paradigm? Is it because human evaluators need to give a feedback $f$ at every time step? Wouldn't this be too slow to be practical, since the frame rate of most tasks is too high for humans to make a judgement?\n* What exactly is the form of the reward model? Is it $R(\\tau) = \\sum_{t}R(s_{t}, a_{t})$?\n* Why would the moving window sampling help mitigate temporal inconsistency? I would imagine training a separate BT model on each temporal window would address inconsistency, at the expense of have potentially too many models. However, if you only use the windows to sample preference pairs, and train a BT model under the assumption that the reward model underlying all comparisons is the same, how can this address inconsistency?\n* For the ablation results in figure 6, is voting applied to generate the figure? Or is the data generated by different evaluators considered as a single evaluator for training the reward model here? I think the authors should clarify this. It seems to me that voting has the highest contribution to performance. If not the former, I think the authors should do an ablation on pref-GUIDE individual + voting with no temporal sampling and no preference range. \n* How exactly are the rewards aggregated for training the policy?\n\nMinor\n* How many seeds did you use to repeat each experiment? \n* What RL algorithm did you use for the proposed method?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fb6v1k3Yle", "reviewer": "Reviewer_gwVU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Consider the problem of real-time human-guided reinforcement learning. One algorithm that exists for this problem is called GUIDE [Zhang et al. (2024b)], and it consists of two stages: \n\n1. **Human-in-the-loop phase-** the agent learns from a combination of sparse terminal rewards and the real-time scores in [-1,1] from human evaluators.\n\n2. **Post-human-guidance phase-** after feedback stops, GUIDE trains a reward model on the collected data and uses its predictions as a dense reward signal to continue policy learning without further human input.\n\nThis paper improves the post-human-guidance phase of GUIDE by addressing two inherent problems: (a) inconsistent feedback from the same human evaluator over time, and (b) variability across evaluators (e.g., due to cognitive ability). \n\n**Contributions**\n\n- **Pref-GUIDE individual-** converts scalar feedback into preference-based comparisons while mitigating problem (a) using a moving-window sampling strategy under the assumption of approximate stationarity within each window.\n\n- **Pref-GUIDE Voting-** to address problem (b), it trains a separate model per evaluator and aggregates their predictions into a consensus reward model.\n\n- The authors test these methods on multiple RL environments: Bowling, Find Treasure, and Hide and Seek 1v1, and show that they improve performance compared to GUIDE significantly. \n- The paper demonstrates competitive performance with DDPG/DDPG heuristic, with a significant improvement in Bowling.\n\n\n**Strengths**\n\n- Pref-GUIDE Individual and Pref-GUIDE Voting provide solutions for temporal inconsistency and evaluator bias, and I’m unaware of other existing solutions for PbRL.\n\n- The paper is well-written.\n\n**Weaknesses**\n\n- The solutions suggested are relatively trivial.\n\n-  The only significant improvement over the DDPG + DDPG heuristic appears in a single environment (Bowling) out of the three tested.\n\n- In Pref-GUIDE Individual, pairs of scalar rewards in [-1,1] are translated to {-1,0,1}, where 0 indicates no preference. Two rewards are mapped into 0 iff their difference is at most 5%. The choice of a 5% threshold for mapping small reward differences to “no preference” seems arbitrary. In particular, the paper lacks justification and does not explore statistically grounded thresholds (e.g., based on standard deviation or percentiles).\n\n- [Minor] Pref-GUIDE Individual may only correct for linear drift. Nonlinear temporal bias could still degrade performance.\n\n\n- See more weaknesses in `Requested Changes’.", "claims_and_evidence_yn": "No", "claims_explanation": "In Section 4.1, the authors hypothesize that a trajectory rated positively at one point might be rated negatively later, but the only supporting evidence (Figures 3 and 9) do not track individual ratings as a function of time. Without any per-trajectory time-series analysis, such as plots of the same trajectory’s scores across sessions or statistical tests for rating drift, the hypothesis remains unsubstantiated. The authors should include direct temporal evidence to convincingly support their claim.", "audience_yn": "Yes", "audience_explanation": "I think PbRL is a topic of interest to the TMLR’s audience, and this paper might improve the state of the art in a relevant task and improves an existing algorithm in 3 tasks.", "requested_changes": "**Critical for acceptance**\n\n*Major*\n\n- In Section 4.1, the authors hypothesize that the same agent might rate the same trajectory positively in the beginning and negatively at a later stage. The only supporting evidence (Figures 3 and 9) does not account for temporal variation.  I think the authors should include per-trajectory rating time series or time-point analysis to validate this claim.\n\n- I think the authors should include the DDPG + DDPG heuristic baseline in Figures 5-7 for fair comparison.\n\n*Minor*\n\n- Figure 3 (and Figure 9 in the appendix) shows Bowling receiving relatively consistent feedback despite the claim of inconsistency across all tasks. I think the authors should add a discussion explaining this apparent contradiction.\n\n- Figure 4 shows significant improvement only for Bowling relative to DDPG/DDPG heuristic, yet Figure 3 suggests Bowling’s feedback is relatively consistent. I think the authors should reconcile these observations in the discussion.\n\n- I think the authors should add a discussion of computational overhead and scalability when training separate models per evaluator (I would assume it is not very significant, but it’s still missing). \n\n- I think the authors should add a discussion on how the window size was chosen.\n\n**Suggestions to strengthen the work**\n\n- I’m not surprised that Pref-GUIDE Voting yields a more stable and robust model. I think using smarter aggregation techniques such as boosting would be more beneficial.", "broader_impact_concerns": "There are no ethical implications of the work that I can think of.", "additional_comments": "T-SNE is popular in vision and high-dimensional numeric tasks, but can you provide evidence that it meaningfully captures similarity among Bowling trajectories? In particular, have you verified, quantitatively or via human judgment, that trajectories close in t-SNE space receive similar preference scores? Is it possible that some trajectories appear adjacent in the embedding but actually should have very different scores?\nTo support your claim of inconsistent feedback over time (Section 4.1), do you have score data from the same evaluator on identical trajectories at multiple time points?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)