GUIDE: Real-Time Human-Shaped Agents
Authors: Lingyu Zhang
Zhengran Ji
Nicholas R Waytowich
Boyuan Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Human-guided Reinforcement Learning
Real-time decision making
Primary area: human-AI_interaction

Abstract: The recent rapid advancement of machine learning has been driven by increasingly powerful models with the growing availability of training data and computational resources. However, real-time decision-making tasks with limited time and sparse learning signals remain challenging. One way of improving the learning speed and performance of these agents is to leverage human guidance. In this work, we introduce GUIDE, a framework for real-time human-guided reinforcement learning by enabling continuous human feedback and grounding such feedback into dense rewards to accelerate policy learning. Additionally, our method features a simulated feedback module that learns and replicates human feedback patterns in an online fashion, effectively reducing the need for human input while allowing continual training. We demonstrate the performance of our framework on challenging tasks with sparse rewards and visual observations. Our human study involving 50 subjects offers strong quantitative and qualitative evidence of the effectiveness of our approach. With only 10 minutes of human feedback, our algorithm achieves up to 30\% increase in success rate compared to its RL baseline.

GUIDE: Real-Time Human-Shaped Agents
Lingyu Zhang', Zhengran Ji', Nicholas R Waytowich", Boyuan Chen'
Duke University, 2 AAmy Research Laboratory
http://wwwr http://www.eeeraarrooo...caaaaaa httpe/iiiiiaaaa.a..a.oooaaaa.aa com/GUIDE
Abstract
The recent rapid advancement of machine learning has been driven by increasingly powerful models with the growing availability of training data and computational resources. However, real-time decision-makin tasks with limited time and sparse learning signals remain challenging. One way of improving the learning speed and performance of these agents is to leverage human guidance. In this work, we introduce GUIDE, framework for real-time human- moanannggiiedd reinforcement learning by enabling continuous human feedback and grounding such feedback into dense rewards to accelerate policy learning. Additionally, our method features simulated feedback module that learns and replicates human feedback patterns in online fashion, effectively reducing the need for human input while allowing continual training. We demonstrate the performance our framework on challenging tasks with sparse rewards and visual observations. Our human study involving 50 subjects offers strong quantitative and qualitative evidence of the effectiveness our approach. With only minutes of human feedback, our algorithm achieves up to 30% increase in success rate compared its RL baseline.
Introduction
48 that require understanding the problem, exploring different options, making decisions, refiming our understanding, and adjusting decisions accordingly. Due their inherent complexity, these tasks pose significant challenges
Many real-world tasks are real-time decision-making processes 7
for current machine learning systems. Effective exploration [6] 36 crucial for gathering informative data, especially in mission-critical tasks such search and rescue, disaster response, and medical emergencies where time is limited. Furthermore, open- upen-world problems often lack dense
labels and provide extremely sparse environment feedback signals 42 19 29
21 Therefore, agents
must reason over long horizons to make informed decisions 24 26
43
also known as human-in -nth--ootte--oop machine learning, has been proposed integrate human feedback into reinforcement learning (RL) agents. Various methods differ in how they obtain, format, and incorporate human feedback. Some approaches rely on full demonstrations
Human-guided machine learning 32
or 52 assuming that humans can directly control the agents and possess expert-level
32
34 or partial corrections [12 through imitation learning 32
inverse RL [3 52
task knowledge. Conversely, non-expert humans can often still judge the quality of the agent strategies,
leading to comparison-basee feedback, such as preferences 2 20 45
18 46
47
23
53 and
rankings 10 These offline methods, however, require large datasets and parallel prompts and, hence, cannot support real-time guidance, limiting their applicability in dynamic environments.
Discrete label feedback has shown promise for real-time human-guided RI,, where humans provide scalar rewards 28 4 or discrete feedback [25 44 (e.g., good, neutral, bad). Despite its promise, this approach is constrained by the need for significant human efforts and the less informative nature of
38th Conference Neural Information Processing Systems (NeurIPS 2024).

---

Fig. 1: We propose GUIDE as a novel framework for real-time human-shaped agents enabling con- tinuous feedback and continual im- provements without human trainers. We also aim to understand how indi- vidual differences affect their guided agents' performances.
the discrete feedback. Current research has demonstrated success 25 44 28
primarily in simple,
low-dimensional tasks with limited solution spaces. Moreover, high-quality human feedback is challenging [T]] to obtain due to variability in human guidance skills, cognitive biases, and individual differences. Most studies 25
50 involve small sample sizes (N 10), often including the designers themselves, which makes it difficult to quantitatively assess the applicability of the methods to the real world. Furthermore, how to keep refining the agent without continuous human input remains unclear.
44 28
We introduce GUIDE (Fig. D a framework for real-time human -uiiddd RL that enables continuous human feedback and grounds such feedback into dense rewards, thereby accelerating policy learning. GUIDE also includes parallel training algorithm that learns simulated human feedback model to effectively reduce human inputs and enable continued policy improvements without human feedback. Our experiments span three challenging tasks characterized by continuous action spaces, high- dimensional visual observations, temporal and spatial reasoning, multi-agent interactions, and sparse environment rewards. Our human studies involving 50 participants provide strong quantitative and qualitative support to the efficacy of our approach. Additionally, we conduct a series cognitive tests and analyses quantify individual differences among participants and explore how these differences correlate with agent learning performance. Our further analysis of the learned policies highlights the critical role of alignment in developing effective human-guided agents.
2 Related Work
Human-Guided Machine Learning Various computational frameworks have been proposed to integrate human feedback in guiding machine learning agents. Behavior cloning [32 33
trains a policy through supervised learning on human demonstrations assuming direct human control of the agents. Inverse reinforcement learning []]22] infers objectives from human demonstrations for police optimization, but it requires extensive human demonstrations and often struggles with accurate
objective inference and scalability. Human preference-basee learning 2 20 45
18
46
17 47 23
allows humans to score trajectories through comparisons, reducing the need for full demonstrations yet still depending on offline policy sampling. GUIDE differs by focusing on real-time decision-making tasks without relying on abundant offline datasets or parallel trajectory rollouts.
Real-Time Human-Guided Reinforcement Learning Our work is formalized into an RL framework to integrate real-time human feedback into the policy learning process. TAMER 25 learns to regress a reward model from human feedback, which is used as a typical reward function for policy learning. Deep TAMER [4] generalizes this approach to higher dimensional inputs and uses a neural network to parameterize both the policy and the reward model. COACH [2] and Deep COACH 4 address the inconsistency of human rewards observed during training by modeling human feedback as an advantage function in actor-critic framework. Deep COACH scales COACH handle higher- dimensional observations and more expressive policy representations. Recent advancements have further improved these algorithms, addressing various challenges in ground human feedback, such as different feedback modalities, stochasticity of feedback [ and continuous action space 37
However, existing real-time human-guided RL studies share several limitations. Simplifying human feedback to discrete sets (positive, neutral, negative) with rewards {1, -1) requires users focus on clicking or typing the correct options which can be distracting, Such simplification also causes information loss by ignoring intermediate values. This approach also introduces hyperparameters to

---

associate feedback to trajectories in time, requiring extensive tuning. Moreover, most studies validate their methods on simple tasks with discrete actions and low-dimensional observations. Furthermore, human studies are often limited to the designers themselves, a small group of subjects (N < 10, mostly N < 5), or a trained expert policy with RL to simulate humans. It remains unclear whether the existing approaches scale general populations. There is also a large gap in understanding how individual differences among human participants affect poliey training. Lastly, there is little discussion on continuing improve policies in the absence of human feedback.
Key Novelties of GUIDE Unlike previous approaches, GUIDE enables continuous human feedback, maximizing guidance information with minimal hyperparameter tuning. Our experiments cover challenging tasks with continuous action spaces, high- high-dimensional visual observations, spatial- temporal reasoning, and multi-agent interactions. GUIDE is also evaluated the largest sample of human subjects (N 50) among related studies, incorporating cognitive assessments to characterize individual differences and their impact guiding learning agents. Additionally, GUIDE includes parallel training algorithm to mimic human feedback for continued policy improvements when direct human feedback becomes unavailable.
Preliminary
3.1 Value-Based Reinforcement Learning
Value-based Reinforcement Learning is effective approach to solving decision-making problems. In value-based RL, key element is the action-value function (ss, a) which provides an estimate of the total expected return when taking an action a in state and following a specific policy thereafter 40 The policy æ is derived from Q by selecting the action that maximizes the action-value function:
π (s) = argmax Q(s, a)
The action-value function Q(s, a) at time step can be defined the expected sum of discounted future rewards with discounted factor 7:
8
Q(s =E T" Retk+1ls,d a]
(2)
Where Re4x+1 is the reward h + steps after the current step. In valued-based deep reinforcement learning, a neural network is used approximate the action-value function to tackle challenges such as high-dimensional and partial observations and complex decision boundaries. The training of these networks typically involves combination of Monte-Cart Carlo sampling and bootstrapping techniques, which leverage current estimates of Q to improve future predictions. The accuracy of the Q-function is crucial, especially in environments where reward signals are sparse and delayed, posing significant challenges to the efficiency and efficacy of the learning process.
3.2 Human feedback as Value Functions
A natural way for humans to guide agent's 's learning is to observe its inputs and provide feedback on its actions. This translates directly to incorporating human feedback into reinforcement learning by assigning human feedback as the myopic state-action value. Deep TAMER [44] is prominent human- guided RL framework that leverages this concept by enabling humans to offer discrete, time-stepped positive or negative feedback. To account for human reaction time, credit assignment mechanism maps feedback to a window of state-action pairs. neural network F trained to estimate the human feedback value, denoted as fa,a, for given state-action pair a). The policy then selects the action that maximizes this predicted feedback.
We constructed a strong baseline by enhancing Deep TAMER in numerous ways. First, the original Deep TAMER's 's greedy action selection method only works with discrete action spaces. We designed a continuous version of Deep TAMER while adopting ofate-oftth--an reinforcement learning im- plementation practices. We implement actor-critic framework to handle continuous action space. Here, the critic is the human feedback estimator parameterized by Ø: (s, a), directly estimating human feedback instead of target action value. The actor is parameterized by 0: Ae(s) = aiming
3

---

Human
Simulated Human
Continuous
State
Environment Feedback
(Predicted)
Human
Action
Feedback
Environment
Policy
The training consists of two stages: During the Human guidance stage, the human trainer observes the state and action taken by the agent and provides real-time continuous feedback. The feedback values grounded into per-step dense rewards and combined with the environment reward. Concurrently, we train a human feedback simulator that takes state-action pairs and regresses the feedback values. During the Automated guidance stage, the trained simulator stands in for the human and provides feedback to continue to improve the policy, effectively reducing human efforts and cognitive loads.
Fig. 2: GUIDE:
to maximize the critic's 'ss output. The combined objective is defined as Lo-Deep TAMER La + Lzz where:
Lg Fa(s, Ao(s)) L. || Po(s, A())) fe.Ao(a)ll2
We follow recent advancements in model architectures and optimization strategies, such as target net soft updates and using Adam optimizer instead of SGD. Our strong baseline not only enhances Deep TAMER continuous actions and recent RL practices but also maintains the core methodology of integrating real-time human feedback into the learning process.
4
GUIDE: Grounding Real-Time Human Feedback
4.1
Method Overview
Developing real-time decision-making agents presents significant challenges. These agents must explore high-dimensional observations, constantly adapt, and make accurate decisions with limited supervisory signals. Real-time human-guided RL seeks to harness human expertise and adaptability. However, existing algorithms often underutilize human expertise by solely relying on discrete feedback. Additionally, integrating human feedback with environmental rewards be complex, and human evaluation criteria often evolve during training.
This section introduces our framework GUIDE (Fig. 2 to address these limitations. GUIDE enables real-time human guidance using dense and continuous feedback. The agent directly perceives the environment through visual frames, selects and executes actions, and receives continuous human feedback per decision step. This feedback then converted into dense rewards that will integrated with sparse environment rewards for effective learning. Concurrently, a separate neural network consistently observes the state-action pairs and human feedback and learns to predict the provided human feedback. This learned model will eventually replace the human as the feedback provider, significantly improving the efficiency of human-guided RL while minimizing the required human guidance time.
4.2 Obtaining Continuous Human Feedback
Conventional weal-time human -uman-guided RL frameworks rely on feedback that is discrete in time and values. An example that we adopted as our baseline shown Fig. A)). We propose novel interface that enables human trainers provide continuous-valued feedback every time step. As illustrated in Fig. B), the trainer hovers their mouse over window to indicate their assessment of the agent's behavior.

---

Our method offers several advantages:
Natural Engagement:
Hovering is a more natural interaction compared to clicking buttons 39 fostering continuous human guidance without interrupting the training flow.
Expressive Feedback: Continuous values allow humans express nuanced assessments, capturing wider range of feedback compared to discrete options, shown in Fig.
Constant Training:
Unlike discrete-feedbact algorithms where the model update frequency is based on the feedback frequency, the continuous feedback ensures that the model continuously
learns and adapts.
Simplified Credit Assignment: our setting, reasonable to assume a constant human feedback delay, alleviating the need for complex credit assignment to map delayed feedback to specific state-action pairs. Such delayed association time window has been reported to have inconsistent values across different studies. In contrast, we used the same one-time delay factor for all our studies.
4.3 Learning from Joint Human and Sparse Environment Feedback
While human guidance offers
Faskiee
1200
Frequency 12 M 8 it
valuable super- vision, sparse
Froqu Jont
600
environment re-
wards or terminal rewards remain crucial for their easy access and the ability
-1.0 -0.30
08:00 0.00 1335 6.99
1.50
-15
-1.0 -0.30
-2.0
0.50
0.0
Feedback Value
Foodback Value
Fig. 3: (A) Conventional discrete feedback. (B) Our continuous feedback. The histograms indicate the feedback distribution provided by the same subject on the same task. Continuous feedback carries more information from the human trainer.
induce desired targets. However, directly using human feedback incorporating existing environment rewards challenging [44]. We propose straightforward but effective solution: convert the human feedback at each time step t into reward, denoted as fi = rrt This allows the seamless integration of environment rewards through simple addition: = hf + rerv This approach also enables leveraging off-the-shelf advanced RI. algorithms, which typically make minimal assumptions about the reward function. Our method can be seen as interactive reward shaping [30) which is a vastly effective approach in handling sparse and long-horizon tasks. However, designing dense reward functions directly can be difficult as it often requires significant prior knowledge, manual effort, and trial and error. Moreover, it lacks adaptability unforeseen scenarios. Our approach allows for real-time reward shaping through human guidance, capitalizing on human flexibility and intuition.
value function makes
4.4 Continue Training by Learning to Mimic Human Feedback
Human guidance can be time-consuming and cognitively demanding. To maximize the benefit of human input while minimizing their effort, we propose regression model that learns to mimic human feedback, acting as a surrogate for the human trainer (Fig. 2). This model allows for continual policy improvements even when human feedback is not available. Our idea based on the assumption that human feedback implicitly follows a mapping function: Hss a) = f. where and are the state and action observed by the human, and f is the assigned feedback value.
The key insight is that during human guidance, we can readily collect state-action pairs along with their assigned feedback values, which allows training a human feedback simulator, parameterized by a neural network (s, a) = f. by minimizing |lf fll2- This training can occur concurrently with human guidance to prepare to substitute at any time or offline for later deployment in continual training. To prevent overfitting, we held out out of 5 trajectories as a validation set. Once trained, the simulator can operate inference mode to provide feedback in the absence of the human trainer. This model makes assumptions about the human's 's specific feedback style patterns. Instead, it leams to provide feedback consistent with the human trainer, minimizing the shift in reward distribution.
5

---

Experiments
5.1 Environments
We conduct experiments the CREW 151 platform.
Bowling: A modified version of Atari Bowling. Each episode consists of 10 rolls. The agent receives a reward equal to the number of pins hit after each roll. The action space 3-dimensional: the initial ball position, the distance to start steering the ball, and the steer direction. Observation the visual image.
The agent is tasked to navigate through a partially observable maze to retrieve a treasure. The agents receive +10 reward upon reaching the treasure, and a constant -1 time penalty for each step taken. The observation space is a top-down accumulated view of where the agent has explored, initial- ized by square area around the agent. The action space is two-dimensional vector of the next destiny location, to which low-level planning algorithm will navigate the agent. The max episode length is 15 seconds.
Find Treasure:
1v1 Hide-and-Seek: A multi-agent one-on-one visual hide-and-seck task B3 14], where we aim to learn the seeker policy whose goal is to navigate through the maze and catch the hider. The hider policy is heuristic that will avoid obstacles and reflect when running into wall while moving away from the seeker if the seeker within a certain range. The reward, observation, and action spaces remain the same Find Treasure.
5.2 Experiment Settings
We allocated 10 minutes for consistent human feedback. We collected the checkpoints and human feedback data and performed continual training for another minutes. The model's performance was evaluated in a test environment every minute for Bowling and every minutes for Find Treasure and Hide-and-Sen (10 rolls); for Find Treasure and 1v1 Hide-and-Seek, we evaluate 100 episodes for every checkpoint. All test results are reported on unseen test conditions.
Evaluation Metrics
For every Bowling checkpoint, we evaluate for 1 game
Baselines We include two state-of-the-art reinforcement learning algorithms and a human-guide RL algorithm as baselines. We selected Deep Deterministic Policy Gradient (DDPG) [27) and Soft Actor-Critic (SAC) [22 due their superior performance many benchmarks. We compared GUIDE to the continuous version of Deep TAMER described in Sec. 3.2, denoted c-Deep TAMER under the same amount of human training time. We also run experiments on Find Treasure and Hide and Seek with heuristic feedback, in other words, dense reward. The feedback is an exploration reward when the target (treasure or hider) is not within the agent's 's visible view and a distance reward when it is. This can seen as an upper bound of our method, as this the "heavy engineered reward" by domain experts with no noise delay.
Human Subjects. We recruited 50 adult subjects to participate in the experiments under the approval of the Institutional Review Board. The subjects have no prior training or knowledge of the algorithms or tasks and have only been instructed to assess the performance of the agents. Inter-trial intervals are included between individual game experiments. For each session, the subject will receive $20 compensation.
5.3 Human Cognitive Tests for Individual Difference Characterization
In order to quantify individual differences among participants and explore how these differences affect the human-guided RL performance, we conducted a series cognitive tests to evaluate their cognitive abilities relevant to the training tasks before the human-guided RI. experiments. We describe the details of the cognitive tests as follows. All tests involve six trials for each participant.
Eye Alignment [35 (A): The subject is asked to align ball the left side of the screen with target on the right side accurately possible within five seconds without using any tools.

---

Time Left:
Countdown: 2s
Wich
Wiick Twee Let.
Whlc explan video Lafs
than Lett
Fig. 4: Cognitive Tests:
We conducted series of cognitive tests quantify how individual
differences among subjects affect their guided agents' performances. (A) Eye Alignment (B) Reflex (C) Theory of Behavior (D) Mental Rotation (E) Mental Fitting (F) Spatial Mapping
The score is calculated the negative average of the distances between the ball's and the target's horizontal positions across trials.
Reflex (Fig. (B): The subject is asked to click the screen as quickly as possible when it changes from yellow to green. Clicking while the screen is still yellow or failing to click within two seconds after the screen turns green is considered a failure. The score is calculated as the negative average of the reflex times, with failures being left out in the score calculation.
The subject observes red ball moving in unknown but fixed pattern for five seconds. Afterward, the ball pauses, and the subject must predict the ball's position one second after it resumes moving. The subject has two seconds to make this prediction. The score is calculated as the negative average the distances between the ball's actual positions and the subject's ' predicted positions across trials.
Theory of Behavior 15 (Fig. (C):
The subject is tasked with identifying the piece among three similar pieces that cannot be directly rotated to match the target piece within eight seconds. The score is calculated based on the accuracy of the subject's 's identifications across all trails.
Mental Rotation 38 (Fgg. (D):
Mental Fitting 38 (Fig. (E): The subject tasked with identifying the only piece among three similar pieces that can fit with the target piece within eight seconds. The score calculated based on the accuracy of the subject's identifications across all trials.
Spatial Mapping (Fig. (F): A video of an agent navigating maze with restricted field of view presented to the subject. After watching the video, the subject asked to identify the maze from selection of three similar mazes within five seconds. The score calculated based on the accuracy of the subject's identifications across all trials.
5.4 Implementation Details
RL backbone and model architecture. The state-of-the-art visual reinforcement learning algorithm transitioned from SAC [2 DDPG [27) due to its stronger performance enabled by the easy incorporation of mult returns and the avoidance of entropy collapse (9). Following this, we also select DDPG (27
Reinforcement Learning backbone for GUIDE. We used 3-layer convolutional neural network as the vision encoder. The actor, critic, and learned feedback model shared this encoder, and each follows it with 3-layer Multi-Layer Perceptron.
Hyperparameters We used an Adam optimizer with a fixed learning rate of 1e-4 for RL policy training, with a discount factor of 7 = 0.99. We applied gradient clipping setting max grad norm 1. For the learned feedback model, and employed early stopping based on the loss held-out trajectories. For Deep TAMER's credit assignment window, we used the same uniform [0.2, 4] distribution as in the original paper. We used
used the same Adam optimizer with le-4 learning rate

---

Dow lig (Rop 151
The Teassm (Tpp 151
Hrle and Seec IVG. (Nop 15)
(fice sonee
TPER
(Irce Sriee
leatd2
Hus Pasr
improvement backbonn
Impr emproybment over Eackbone
50
00
3-
15.0
223
373
0.3
25
12.5
15.1
75.0
Bowking
Fid Theesume (AB
1Ide Saec (AB
CARDE OUB0
02200
Hourbiic
Improyement over backtone
55
backbac
poit
Tine 21.
1900)
Fig. GUIDE performance compared with other baselines. In challenging tasks, GUIDE consistently outperforms all other baselines. Subjects with higher cognitive test scores also result in higher performance the learned agents as shown the top row (Top 15).
shorter window of [0.2, 1] for Find treasure and Hide-ande -nd-Se..-S For these more difficult navigation tasks, we stacked three consecutive frames as input.
Human delay factor. Human delay non-neglectable in human-guided RL. The time takes for human see and process the states and produce feedback signal varies across environments and individuals. Preliminary experiments with the authors suggested that shifting human feedback by 2 seconds for Bowling and 1 second for Find Treasure and Hide and Seek best aligned with the intended state-action pairs.
5.5 Experiment Results and Analysis
Human-Guider RL performance We report our results in Fig. 5 where the x-axis is the total training time, and the y-axis is either the score for Bowling or the success rate for Find Treasure and Hide-and-Seek The dashed line separating the first and second 10 minutes indicates the end of the human guidance and the beginning of the automated guidance. The first row reports the training results of the 15 human trainers who scored the highest the cognitive test. The second row is the average performance of all 50 human subjects. We observe that subjects who tested higher cognitive skills result in higher performing agents, as the trained Al performance approaches closer to the heuristic feedback upper bound. For the simple Bowling task, all methods perform similarly, with GUIDE having a slight advantage over DDPG. On the challenging Find Treasure and Hide-and-Seck task, GUIDE-DDPG scored up to 30 percent higher success rate than its RL counterpart, and 50 percent higher than c-Deep TAMER given the same amount of human guidance time. It is worth noting that in this time-critical setting, GUIDE-DDPG is able reach the same level of performance its RL baselines within half the training time. The average of all 50 untrained human trainers was able to surpass its RL baseline by large margin while also surpassing our greatly enhanced Deep TAMER on the challenging Find Treasure and Hide and Seek by up to 40 points in success rate.
Exploration Analysis The ability explore the state space considered one of the key aspects of successful sparse reward RL. We hypothesize one benefit of human guidance inducing efficient exploration behavior. We quantify exploration tendency Find Treasure by measuring the accu- mulated visible area ratio. After two minutes of human feedback within the GUIDE framework, the

---

2min
4min
6min
8min
10min
visible ratio
0.-
01-
20 33. 10 20 30
30 33
30.0 12 20
23 23
step
Fig. 6: Exploration behavior of GUIDE and DDPG agents. For each of the plots, the x-axis the step number through the course of an episode. The y-axis is the ratio between the area of the visible view and the entire input frame. We observe stronger tendency of exploration exhibited by the human-guided agent compared to the baseline RL agent.
agents have shown more efficient exploration behavior compared the RL baseline. As shown in Fig. 6, each plot corresponds one stage in training. For each plot, the x-axis the step number through the course of an episode, and the y-axis is the visible area ratio for that step averaged across 10 episodes. The GUIDE results are tested on the human subjects who ranked top 15 in cognitive tests. The DDPG results are tested on five different random seeds. The higher the curve, the faster the agent explores the maze. We find that the exploration ability increases throughout training for both DDPG and GUIDE, with GUIDE learning to explore faster in the first minutes. As the training progresses, exploration reaches its bottleneck utility, and DDPG and GUIDE converge to a similar exploration rate.
Human- Guided Trajectory
Unseen Trajectory
Eny Houristic
-1.00
-1.00
-1.00
-1.00
-1.00
+1.00
1.00
-1.00
-0.31
+0.64
+0.55
+0.65
Learned Heuristic
-0.38
-0.33
-0.27
-0.41
-0.54
+0.64
+0.58
-1.05
-0.8
+1.6
-1.1
Human-
Unseen Trajoctory
Guided Trajectory
Enw Human
-1.00
-1.00
-1.00
-1.00
Env
-1.00
-1.00
1.00
+0000
-0.51
+0.10
+0.08
-0.61
Learned Houristic
-0.13
-0.21
+0.17
+0.26
Heuristic
-0.89
+1.14
+.77
+0.62
Fig. 7: Qualitative visualization of the learned human feedback model. Our learned feedback model
able to generalize to unseen trajectories and provide effective feedback place of humans.
Learned Feedback Model Ablations We qualitatively analyze the effect of the learned feedback model by visualizing its predictions on unseen trajectories. As shown in Fig. our learned feedback model generalizes unseen trajectories and is able to provide effective feedback in place of humans.
Individual Differences vs. Performance To further characterize individual differences and how they af- fect policy training, we found a significant correlation between the cognitive
Bowingg
+(h...)
+0h.s.)
41"
+0....]
40.5.
+(**)
ranking of the human sub- jects and their trained AI performance, with Pear- son correlation coefficient of 7.522 and a p-value of 0,001, as shown in Fig. g Among all cognitive tests, the mental rotation test and Mental Fitting test are
Treasure
Hide
+bns..]
+1r.s.)
+0n..]
401
+tn.s.)
Total
-(nx.)
+fns..)
+P]
+1*]
+(na.)
+[*)
the strongest indicators of overall guided Al success. We also discovered relationships between individual tasks and cognitive tests. Notably, 1v1 Hide-and- Seek score is positively related reflex time. This likely due to the fact that Hide-amt and-Sandkk Seck is quickly evolving dynamic task, and faster response time will
Eye
Theerr
Retation
Rirees
Refter
Overall
Fig. 8: Correlation between cognitive test scores (normalized) and GUIDE training performance. The darker the color, the more statistically signifi- cant the correlation. "4" positive, "_": negative.

---

20min
Bowling
Pecerage of Subjets
Find Treasure
CM
Perortage Buggodo
1v1 Hide & Seek
Forertag oupioons
Fig. 9: Robustness to individual differences. The subplots from left to right show how performance evolves over training time. each subplot, the x-axis training milestones indicated by task scores, and the y-axis is the percentage of humans who were able to train the agent to reach the milestone at a given time. We find that GUIDE's robustness scales more complex tasks.
result in better matching in feedback and state-action pairs. Detailed correlation results can be found in the appendices.
Robustness to individual differences A critical aspect of coman-guided machine algorithms method's robustness to different human trainers. A generalizable algorithm should maintain a strong performance when human feedback patterns vary. However, this is rarely discussed by prior methods likely due to the small size of human participants. Here provide thorough analysis on GUIDE's performance across our 50 human trainers compared to c-Deep TAMER. Each subplot in Fig shows for each suman-guided algorithm, the percentage of human trainers (y-axis) that are able to train the agent to reach specific milestones indicated by task scores (x-axis). Panel A shows that for the Bowling task, c-Deep TAMER and GUIDE are close in performance. Panel B and shows that for Find Treasure and Ivl Hide-and-Sede able reach milestones using GUIDE than using c-Deep TAMER, demonstrating the scalability of the robustness of GUIDE individual differences.
the complexity of the game increases, more human trainers
Conclusion, Limitations and Future Work
We introduce GUIDE, novel framework for real-time buman-guided RL. GUIDE leverages continu- ous human feedback to effectively address limitations associated with sparse environment rewards and discrete feedback approaches. Furthermore, GUIDE incorporates a parallel training algorithm to learn a human feedback simulator, enabling continued policy improvements without ongoing human input. Our extensive experiments demonstrate the efficacy of GUIDE across challenging tasks while presenting human cognitive assessments understand individual differences.
While GUIDE offers significant advantages, it presents some limitations and exciting avenues for future work. First, our current evaluation focuses on tasks with moderate complexity. Future work can explore scaling GUIDE to highly complex environments and large-scale deployments. Moreover, we have quantitatively observed strong individual differences in guiding learning agents through extensive human studies. However, we have not explored how to mitigate such differences and account for human variability in feedback patterns. Furthermore, understanding how the human feedback simulator operates and interprets the agent's behavior remains open question. Future work will delve into explainable learning techniques to shed light on this process.
10

---

7 Acknowledgements
This work is supported part by ARL STRONG program under awards W911NF2320182 and W911NF2220113.
References
[1] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
[2] R. Akrour, M. Schoenauer, and M. Sebag. Preference-baset policy learning. In Machine Learning and Knowledge Discovery in Datahases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part I 11, pages 12-27 Springer, 2011.
[3] R. Arakawa, S. Kobayashi, Y. Unno, Y. Tsuboi, and S.-i. Maeda. Dqn-tamer: flumanin--the-lopy reinforcement learning with intractable feedback. arXiv preprint arXhv:1810.11748, 2018.
[4] D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman. Deep reinforcement learning from policy-dependent human feedback. 2019.
[5] M. Berkowitz, A. Gerber, C. M. Thurn, B. Emo, C. Hoelscher, and E. Stern. Spatial abilities for architecture: Cross sectional and longitudinal assessment with novel and existing spatial ability tests. Frontiers in psychology, 11:00963,, 2021.
[6] C. J. Boes. The history of examination of reflexes. Journal of neurology, 261(12):2264 2274, 2014.
[7] M. Bojarski, D. Del Testa, Dworakowski, B. Firner, Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, al. End to end learning for self-driving cars. arXiv preprint arXiv: (6lo::66000311 2016.
[8] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. /2894, 2018
[9] C. Burges, R. Ragno, and Q. Le. Learning to rank with nonsmooth cost functions. Advances in neural information processing systems, 19, 2006.
[10] Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li. Learning rank: from pairwise approach to listwise approach. In Proceedings of the 24th International Conference Machine Learning, ICML '07 page 129-136, New York, NY, USA, 2007. URL htttp:://dii: org. 111.111...111111111 1145/ 1273496. 1273513
[11] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.1522 2023.
[12] C. Chai and G. Li. Human-in-the-loop techniques machine learning. IEEE Data Eng. Bull., 43(3):37-52, 2020.
[13] B. Chen, S. Song, H. Lipson, and C. Vondrick. Visual hide and seek. In Artificial Life Conference Proceedings 32, pages 645-655. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info
2020.
[14] B. Chen, Y. Hu, R. Kwiatkowski, S. Song, and H. Lipson. Visual perspective taking for opponent behavior modeling. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13678-13685. IEEE, 2021.
[15] B. Chen, C. Vondrick, and H. Lipson. Visual behavior modelling for robotic theory of mind. Scientific Reports, 11(1):424, 2021.

---

[16] N. Chentanez, A. Barto, and S. Singh. Intrinsically motivated reinforcement learning. Advances in neural information procesing systems, 17, 2004.
[17] Christiano, J. Leike, T. Brown, M. Martic. S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[18] C. Daniel, O. Kroemer, M. Viering. J. Metz, and J. Peters. Active reward learning with a novel acquisition function. Autonomous Robots, 39:389-40$ 2015.
[19] R. Devidze, P. Kamalaruban, and A. Singla. Exploration-guided reward shaping for reinforce- ment learning under sparse rewards. Advances in Neural Information Processing Systems, 35: 9829-5842, 2022.
[20] Fürnkranz, Hüllermeier, W. Cheng, and S.-H. Park. Preference-based reinforcement learning: formal framework and policy iteration algorithm. Machine learning, 19:123-156. 2012.
[21] Y. Guo, J. Choi, M. Moczulski, S. Feng, S. Bengio, M. Norouzi, and H. Lee. Memory based rajjectoyyccondiiioned policies for learning from sparse rewards. Advances in Neural Information Processing Systems, 33:4333-4345 2020.
[22] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[23] B. Ibarz, J. Leike, T. Pohlen, G. Irving. S. Legg, and Amodei. Reward learning from human preferences and demonstrations in atari. Advances neural information processing systems, 31, 2018.
[24] M. Jaderberg, V. Mnih, W. M. Czamecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: 1.05397, 2016.
[25] W. B. Knox and P. Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pages 9 9-16, 2009.
[26] A. C. Li, Vaezipoor, R. T. Iearte, and A. Mcllraith. Challenges to solving combinatorially hard long-horizon deep rl tasks. arXiv preprint 2xX1:22006.01822 2022.
[27] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
[28] MacGlashan, M. Littman, Roberts, R. T. Loftin, Peng, and M. E. Taylor. Conver- gent actor critic by humans. 2016. URL htta:pps//ap....mmaiiccccoo.a.rr oorprattccccccll.....ooo/o/ooopo
semanticacholar
42747476
[29] G. Matheron, N. Perrin, and O. Sigaud. The problem with ddpg: understanding failures in deterministic environments with sparse rewards. arXiv preprint arXiv:9911.11679 2019.
[30] A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278-287, 1999.
[31] A. Y. Ng. S. Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, page 2, 2000.
[32] D. Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
12

---

[33] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artift- cial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[34] S. Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
[35] D. Scharre. Sage: A test to detect signs of alzheimer's and dementia. The Ohio State University Wexner Medical Center, 2014.
[36] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 7707.66347 2017.
[37] I. Sheidlower, E. S. Short, and A. Moore. Environment guided interactive reinforcement learning: Learning from binary feedback in high-dimensional robot task environments. In Proceedings of the s International Conference on Autonomous Agents and Multiagent Systems, pages 1726-1728, 2022.
[38] R. N. Shepard and J. Metzler. Mental rotation of three-dimensional objects. Science, 171(3972): 701-703, 1971.
[39] S. S. Sundar, S. Bellur, J. Oh, Q. Xu, and H. Jia. User experience of on-screen interaction techniques: An experimental investigation of clicking, sliding, zooming, hovering, dragging, and flipping. Hummnn-Compln
Interaction, 29(2):109-15) 2014.
[40] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[41] H. Tang, R. Houthooft, D. Foote, A. Stooke, O. Xi Chen, Y. Duan, Schulman,, F. DeTurek, and P. Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[42] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv: / 707.088/7.7777 2017.
[43] R. Wang, S. S. Du, L. Yang, and S. Kakade. Is long horizon rl more difficult than short horizon ri? Advances in Neural Information Processing Systems, 33:9075-9085, 2020.
[44] G. Warnell, N. Waytowich, V. Lawhern, and P. Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[45] A. Wilson, A. Fern, and P. Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, 25, 2012.
[46] C. Wirth, J. Fürnkranz, and G. Neumann. Model-free preference-basee based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[47] C. Wirth, R. Akrour, G. Neumann, and J. Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136): 1636)):1464 1-6, 2017.
[48] Z. Xiong, X.-Y. Liu, S. Zhong, H. Yang, and A. Walid. Practical deep reinforcement learning approach for stock trading. arXiv preprint arXiv:1811.07522 pages 1-7, 2018.
[49] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data- aata--u reinforcement learning. arXiv preprint arXiv:2107.09645. 2021.
[50] D. Yilmaz, B. Ugurlu, and E. Oztop. Human-in-the-logg training leads faster skill acquisition and adaptation in reinforcement learning- based robot control. In 2024 IEEE 18th International Conference Advanced Motion Control (AMC), pages 1-6. IEEE, 2024.
13

---

[51] L. Zhang. Z. Ji, and B. Chen. Crew: Facilitating human-ai teaming research. arXiv preprint arXiv:2408.00/70, 2024.
[52] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
[53] D. M. Ziegler, N. Stiennon, J. Wu, B. Brown, Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv: 08593, 2019.
14

---

A Computatonal Resources
All human subject experiments are conducted on desktops with one NVIDIA RTX 4080 GPU. All evaluations are run on headless server with NVIDIA RTX A6000 and NVIDIA RTX 3090 Ti.
B Fully Cognitive test-Guided AI performance Study
Mep 10 sudern
paller
Srans bast soen 20ll
Flen VC ITAM
Shart 18 cls
Coee sndeect
VLSMA
Way
Jipe SIN, PaEN CES
Siee 30 PaRke
Sc3t OJ PUBCE MAI
Z0e pe1i CED
Scat FORC
WhC
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our extensive experiment results support our claims in the abstract and introduction. We also summarize our contributions the Conclusion section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? [Yes]
Answer:
Justification: We discuss the limitations of this work in the Conclusion, Limitations and Future Work section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the
implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
Our paper does not include theoretical results or contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the full implementation details and settings in our experiment section. We will also open-source the entire code base, including algorithms and task environments for the broader community for full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construet the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will also open-souree the entire code base, including algorithms and task environments for the broader community for full reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
pobllc/gguides/Dddeuuuiisiiiiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/pult hwtppppulllgggiioo wtw.riddeee/oSSeumiieiieeeeeeeee iitmiissiinnn ssisnooiccyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the full implementation details in the paper including hyperparam- eters, task setups, and algorithms. We will also open-source the entire code base, including algorithms and task environments for the broader community for full reproducibility. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: To the best of our knowledge, we present the most extensive studies among related studies with N = 50 human subjects with detailed analysis. We report the full results with extensive evaluation runs and the means and standard deviations. For all other RL baselines, results are also reported with multiple random seeds with the mean and standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We report the experiments' compute resources in the our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and we follow this code. Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes]
Justification: This work does not fit into the stated negative societal impacts. We discuss the positive societal impacts the abstract and the introduction sections.
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious unintended uses
(c.g., disinformation, generating fake profiles, surveillance), fairness considerations
19

---

(e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Question:
Justification: this paper poses such risks. Our human studies have been approved by the Institutional Review Board, and we will not release or share any identifiable data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited all the previous works used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will open-source the entire code base for our algorithms, basclines, and experiment environments. We will also include detailed documentation.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Our human studies have been approved by the Institutional Review Board with all detailed documents. We have included many details as possible in the main texts. We plan to release our instruction videos for the human subjects as well. We have included the compensation in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our study has been approved by the Institutional Review Board and we have included the details in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
22


Reviews:
[{"review_id": "I2Afbe7Q5S", "reviewer": "Reviewer_fsCr", "summary": "This paper proposes a new framework, GUIDE, for learning from continuous human feedback in complex decision making domains with continuous action spaces. By framing human feedback as a state-action value function, the framework proposes to learn this function and to combine it additively with the (generally sparse) reward coming from the environment. The feedback is collected in a continuous fashion by asking participants to move their mouse up or down to indicate higher or lower feedback values. In a user study, the paper finds that training agents with this type of feedback yields better performing agents than baselines. After assessing participants in a suite of cognitive tests, it finds that participants that score higher on the cognitive tests trained better agents.", "questions": "In 3.2 you propose to extend Deep Tamer by using the human feedback estimator F(s,a) as a critic within an actor-critic framework. Thus, you treat human feedback here as an estimate of the future value of each state-action pair (a Q-value). In 4.3, you propose to use the human feedback signal as a reward, to be added to the environment reward. Does this mean you make a different assumption regarding human feedback in 4.3. than in 3.2? Or do you propose to use feedback on future value as a reward?\n\nI am confused by your claim that Deep Tamer uses DQN. Looking at algorithm 1 in the Deep Tamer paper, it’s policy greedily chooses the action that maximizes the learnt human value function H(s,a). I do not see any use of DQN.\n\nDid your c-DeepTamer baseline receive the same (sparse) environment reward as GUIDE? And if so, how did you use this reward signal in conjunction with a critic that is trained from human feedback only?\n\nWhy was the training of c-DeepTamer stopped after 5/10 minutes? Would it not be possible to continue training it using simulated human feedback (similar to GUIDE) or without human feedback?\n\nWithin the user study, did participants get time to familiarize themselves with each environment before data recording began? I imagine participants would have needed some time to figure out how to solve each task.\n\nIs equation (3) correct? What is the purpose of he additional $- F(s, A(s))$ term?", "limitations": "Limitations are discussed in the conclusion.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper introduces an interesting new way of collecting continuous human feedback, and shows significant improvement in two out of three tasks considered. The use of cognitive tests as part of the user study is interesting, and uncovers insightful correlation between subject performance and their cognitive test scores.", "weaknesses": "The assumptions regarding what human feedback represents do not seem consistent between section 3 and 4 (see Questions). Further, the treatment of the feedback collection is rather simple (added to environment reward function) and, especially if it does represent a signal regarding the future value of state-action pairs, heuristic. Relative to Tamer and Deep Tamer, which treated human feedback more consistently by using it directly as a proper state-action value function, this paper feels like a regression on that front. \n\nThe implementation of the c-DeepTamer baseline raise a number of questions (see Questions), which shake my confidence in it as a baseline, or as a proper representative for how well Deep Tamer should perform here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments. We would like to address all of your concerns and questions below with point responses:\n\n----\n>“The assumptions regarding what human feedback represents do not seem consistent between section 3 and 4 (see Questions). ”\n\nWe would like to clarify that c-DeepTamer and GUIDE use human feedback differently. c-DeepTamer, described in sec 3.2, is a baseline algorithm. It is an upgraded version of the original Deep Tamer[1], and follows the original Deep Tamer formulation to use human feedback as the myopic state-action value function. We did not modify how Deep Tamer uses human reward as a fair comparison. In GUIDE, we instead use human feedback as an additional immediate reward to the sparse environment reward.\n\n>“Further, the treatment of the feedback collection is rather simple (added to environment reward function) and, especially if it does represent a signal regarding the future value of state-action pairs, heuristic. Relative to Tamer and Deep Tamer, which treated human feedback more consistently by using it directly as a proper state-action value function, this paper feels like a regression on that front.”\n\nTamer and Deep Tamer can treat human feedback directly as a state-action value function because they ignore the sparse environment feedback. However, we believe that it is still useful to leverage the sparse environment feedback for RL training. In GUIDE, Our design choice to use human feedback as an additional reward function is based on this intuition to integrate both types of feedback together.\n\n>“In 3.2 you propose to extend Deep Tamer by using the human feedback estimator F(s,a) as a critic within an actor-critic framework. Thus, you treat human feedback here as an estimate of the future value of each state-action pair (a Q-value). In 4.3, you propose to use the human feedback signal as a reward, to be added to the environment reward. Does this mean you make a different assumption regarding human feedback in 4.3. than in 3.2? Or do you propose to use feedback on future value as a reward?”\n\nYes, we use human feedback differently in c-Deep Tamer and GUIDE. In Deep Tamer, human feedback is treated as a myopic state-action value. In GUIDE, we treat human feedback as a reward function, which we believe is a more appropriate and effective approximation, as evidenced by our experimental results. Moreover, this design choice makes it more natural to incorporate sparse environment rewards with the human reward in practice.\n\n>“I am confused by your claim that Deep Tamer uses DQN. Looking at algorithm 1 in the Deep Tamer paper, it’s policy greedily chooses the action that maximizes the learnt human value function H(s,a). I do not see any use of DQN.”\n\nDeep Tamer can be seen as DQN with $\\gamma=0$, where the target state-value function at the current time step is directly assigned by humans. We realize that this phrase may cause confusion. We will remove the DQN claim in the revised paper and instead describe as Deep Tamer learns a reward model to regress human feedback, which is then used for greedy action selection.\n\n>“Did your c-DeepTamer baseline receive the same (sparse) environment reward as GUIDE? And if so, how did you use this reward signal in conjunction with a critic that is trained from human feedback only?”\n\nc-DeepTamer only upgrades Deep Tamer to handle continuous actions, with extensive recent neural network designs and hyperparameter tuning. Following Deep Tamer, it does not use an environment reward. As stated above, Deep Tamer treats human feedback as a myopic state-action value, which is not really a reward, making it impractical to add an environment reward to it. Changing the reward setup in Deep Tamer will make it unfair to compare. One contribution of GUIDE is to provide a formulation that uses both human reward and sparse environment reward.\n\n>“Why was the training of c-DeepTamer stopped after 5/10 minutes? Would it not be possible to continue training it using simulated human feedback (similar to GUIDE) or without human feedback?”\n\nWe strictly follow the original Deep Tamer implementation which does not handle continual training. In the original Deep Tamer, the model weights are only updated every time when a new human feedback signal is received (see Algorithm 1, line 5 in the Deep Tamer paper[1]). When there is no human feedback available, Deep Tamer will not be improved.\n\nBoth c-Deep Tamer and GUIDE are trained with the same amount of human guidance time. In fact, one contribution of GUIDE is to provide a mechanism to simultaneously train a simulated feedback model for continual training once human guidance is not available. We would like to clarify that this is our contribution in GUIDE instead of an inheritance from Deep Tamer.\n\n>“Within the user study, did participants get time to familiarize themselves with each environment before data recording began? I imagine participants would have needed some time to figure out how to solve each task.”\n\nYes. Before each experiment, we showed the participants a short video introducing the goal of each game and a quick demonstration of how each game is played and how the feedback interface can be used.\n\n>“Is equation (3) correct? What is the purpose of the additional -F(s, A(s)) term?”\n\nMathematically, the equation (3) is correct. The additional term -F(s, A(s)) is the actor loss, which tries to maximize the feedback value predicted by the critic network, given an action selected by the actor A(s). We recognize that the presentation may cause confusion since the actor and critic loss only affect their corresponding networks separately. We will clarify this by separating these two terms and indicate model updates in our revised paper.\n\n----\n*references*\n\n[1] Warnell, Garrett, et al. \"Deep tamer: Interactive agent shaping in high-dimensional state spaces.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KIgk5oxkCx", "reviewer": "Reviewer_sJ5r", "summary": "The paper proposes a new framework for human-in-the-loop reinforcement learning, where the human and provide real-time and continuous feedback, and an algorithm where the learning agents uses the human feedback to accelerate policy learning. The paper conducted a user study of 50 subjects to demonstrate the effectiveness of the proposed framework in accelerating policy learning and improving success rates over RL and human-in-the-loop RL baselines. Optionally, a human feedback simulator can also be trained to mimic human feedback after a certain amount of time, reducing the amount of human input.", "questions": "- How does human-in-the-loop compare to BC or reward engineering for the same amount of human input?\n- How are the three tasks for evaluation chosen, and what are the motivations for such choices?\n\nOne motivation for doing human-in-the-loop RL is for tasks where designing a reward function or providing demonstrations are difficult, such as the backflip task. However, for the tasks examined in this paper, there seems to be other simple ways for the RL agent to learn from human input.", "limitations": "The paper focused on the final success rate of policy learning, but did not provide sufficient data from the user's perspective. For example, the user study failed to include a survey regarding whether the real-time feedback system feels easier to use than the discrete feedback system.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes human-in-the-loop RL where the human can provide real-time continuous feedback, which is a novel paradigm compared to mainstream existing work which focus on discrete feedback signals.\n- This work conducted a user study of 50 subjects, which is the largest among relevant works. This is a great contribution in assessing the effectiveness of human-in-the-loop RL.\n- The evaluation is done on three challenging tasks, and GUIDE outperform all the baselines by a large margin on the \"find treasure\" and \"high and seek\" tasks.\n- The paper provides a detailed individual difference characterization by conducting a series of human cognitive tests. Analysis of the human cognitive test data provides meaningful insights. These data can also be very useful in future work.", "weaknesses": "- The baselines are generally quite weak. Based on the experiment results, it is unclear whether real-time continuous feedback is necessarily the best way for humans to guide the policy learning. There might be intermediate points on the spectrum of conventional discrete feedback and full continuous feedback that provides the best tradeoff between amount of human input and effectiveness of guiding policy learning.\n- Whether the simulated human feedback is helpful is unclear. In both the \"bowling\" the \"find treasure\" task, the score does not increase much after switching to simulated feedback. It might be the case that the simulated human feedback only works for tasks where it is straightforward to model the reward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments. We are glad that the reviewer found our method to be novel, the scale of our human study to be “a great contribution,” and our human analysis to be insightful. We would like to address all of your concerns and questions below with point responses:\n\n----\n>“The baselines are generally quite weak. Based on the experiment results, it is unclear whether real-time continuous feedback is necessarily the best way for humans to guide the policy learning. There might be intermediate points on the spectrum of conventional discrete feedback and full continuous feedback that provides the best tradeoff between amount of human input and effectiveness of guiding policy learning.”\n\nWe agree that it will be interesting to explore the “optimal” feedback mode. Based on feedback from our human participants, the continuous feedback mode was not cognitively demanding, hence a full continuous feedback that enables maximum human input is a straightforward and effective choice. We will leave the exploration of blending discrete and continuous feedback as future work.\n\n>“Whether the simulated human feedback is helpful is unclear. In both the \"bowling\" the \"find treasure\" task, the score does not increase much after switching to simulated feedback. It might be the case that the simulated human feedback only works for tasks where it is straightforward to model the reward.”\n\nWe agree that the effectiveness of simulated human feedback will vary depending on the complexity of the environment and human feedback. The main purpose of the simulated feedback is to allow continual training without the human trainer and minimize the shift in reward distribution, which is a novel algorithm design. We will leave this interesting investigation of the relationship between task complexity and the effectiveness of simulated feedback as future work. \n\n>“How does human-in-the-loop compare to BC or reward engineering for the same amount of human input?”\n\nIt is unclear how to quantify the amount of human input for reward engineering. One advantage of GUIDE is that we do not require human subjects to have prior knowledge of RL. However, reward engineering typically requires domain expert knowledge and experience in designing rewards for tuning RL policies. On the other hand, BC typically assumes expert demonstrations, which also demand more cognitive load from the subjects and expert-level demonstrations. However, for complex tasks, humans may not easily provide high-quality demonstrations while they can still provide an assessment of the agent’s decisions. Therefore, we believe that BC and reward engineering are not suitable comparisons in our case.\n\n>“How are the three tasks for evaluation chosen, and what are the motivations for such choices?”\n\nThe bowling task is a classic environment commonly used in prior literature [1, 2, 3]. We included it to compare it with Deep Tamer. Find Treasure is a navigation task motivated by potential real-world applications of human-guided RL, such as search and rescue and disaster response. It also serves as an intermediate task before hide-and-seek, where the target does not move. Hide-and-seek is selected as a representative task involving adversarial competition. Most existing tasks in literature are discrete action tasks with low-dimensional states. We believe that our task selection is a large step forward in investigating the scalability of human-guided RL.\n\n>“One motivation for doing human-in-the-loop RL is for tasks where designing a reward function or providing demonstrations are difficult, such as the backflip task. However, for the tasks examined in this paper, there seems to be other simple ways for the RL agent to learn from human input.”\n\nDespite reward engineering that could exist for these tasks, our setting is different. Designing rewards requires expert knowledge and RL training experience. However, our setting does not assume such expert experience in our human subjects. Moreover, designing dense rewards for these tasks requires extra information that is not available to our agents, such as the position of the agent, the target object location (our environment only provides partial visual inputs where target objects may not be visible), and the location of the adversarial agent (our environment only provides partial visual inputs where the other adversarial agent are not always visible). Therefore, our setting does fulfill the suggestions from the reviewer where the reward function is difficult to design without such extra experience or extra exposed information.\n\n>“The paper focused on the final success rate of policy learning, but did not provide sufficient data from the user's perspective. For example, the user study failed to include a survey regarding whether the real-time feedback system feels easier to use than the discrete feedback system.”\n\nWe would like to clarify that our work also quantifies individual differences and their impact on guided AI performance, as in our cognitive studies in Fig. 5 and 7. Surveys about user experience can be subjective. One possible solution is to quantify physiological signals to measure their cognitive load and stress level, combined with surveys. We leave such exploration in future work. Though we did not have a formal survey for this question, verbal feedback from our subjects suggests that most of them did not find the continuous feedback interface cognitively demanding.\n\n----\n*references*\n\n[1] Warnell, Garrett, et al. \"Deep tamer: Interactive agent shaping in high-dimensional state spaces.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.\n\n[2] Park, Sung-Yun, Seung-Jin Hong, and Sang-Kwang Lee. \"Human Interactive Learning with Intrinsic Reward.\" 2023 IEEE Conference on Games (CoG). IEEE, 2023.\n\n[3] Xiao, Baicen, et al. \"Fresh: Interactive reward shaping in high-dimensional state spaces using human feedback.\" arXiv preprint arXiv:2001.06781 (2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YDZSsTvapX", "reviewer": "Reviewer_Xw63", "summary": "The paper proposes a new approach to reinforcement learning with human feedback in simple video games. The method relies on continuous human feedback that is provided by the human observer hovering their mouse over a window with a spectrum of positive and negative rewards. Unlike prior approaches, this method converts human feedback directly into a reinforcement learning reward with an added time delay. Moreover, the method includes a model that regresses states into observed human feedback, which allows for simulated human feedback. The effectiveness of the method is demonstrated in three simple games in a human study with 50 participants.", "questions": "1. Is it reasonable to assume constant reaction time of the human observer?\n\n2. How would your method deal with games with more ambiguous state values, such as Montezuma's Revenge?\n\n3. The human feedback regressor $\\bar{H}(s, a)$ uses the Markov assumption, but the human feedback might be dependent on their memory of prior states. What if the human feedback is non-Markov?", "limitations": "Limitations are partially addressed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose a simple way to incorporate continuous human feedback as a reinforcement learning reward with a constant time delay. The environment reward and human feedback are simply added to form the final reward function.\n\n2. It is demonstrated that human preference can be directly regressed from states and actions to provide simulated human feedback.\n\n3. The authors perform an extensive human study showing the effectiveness of their method. They also correlate the subject’s performance in a cognitive test with their ability to guide an RL agent.", "weaknesses": "There are three major unstated assumptions:\n\n1. The delay between an event appearing on the screen and the change in human feedback is constant (Question 1). The authors tune this constant for each environment. But, more complex environments might induce different delays as the human observer might need to think about what they saw.\n\n2. People are able to provide constant feedback (Question 2). This might not be true for more complex environments where certain states might have ambiguous values.\n\n3. The human feedback is Markov (Question 3). This might not be true in more complex games.\n\n## Detailed comments:\n\n* Equation 1 should have Q instead of A. Unless you want to define an advantage function A.\n* Equation 2 should have an upper-case Q.\n* The term $R_{t+k+1}$ in Equation 2 is not very clear.\n* The meaning of “We follow recent advancements in neural architectures and hyperparameter designs” on line 125 is not clear.\n* The rest of the paragraph on lines 125 - 127 is superfluous.\n\n## Minor comments:\n\n* Inconsistent spacing between text and citations.\n* Calling this approach a “computational framework” might be a bit redundant given the context of the conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments. We would like to address all of your concerns and questions below with point responses:\n\n----\n>“There are three major unstated assumptions: The delay between an event appearing on the screen and the change in human feedback is constant (Question 1). The authors tune this constant for each environment. But, more complex environments might induce different delays as the human observer might need to think about what they saw.”\n(Question1: “Is it reasonable to assume constant reaction time of the human observer?”)\n\nWe agree that human response time can have various delays. In GUIDE, the advantage of using continual feedback is that it can partially alleviate this issue by eliminating the need for time window association for feedback to state-action pairs. Such a time window is required in previous discrete feedback designs. However, modeling more fine-grained response delay can potentially further improve GUIDE. We leave the exploration of this aspect in future work.\n\n>“People are able to provide constant feedback (Question 2). This might not be true for more complex environments where certain states might have ambiguous values.”\n\nOur assumption is that humans have a preference for any given state-action pair, though the strength of the preference may vary. Continuous-valued feedback can model the strength of preference, e.g., provide a feedback value with a smaller magnitude to reflect weaker preferences. Our experiments suggest that our method surpasses past work on discrete feedback. One future research direction could be empowering the model to learn adaptive changes in human feedback over the course of trajectories or use multiple feedback modalities. We believe our framework can be a strong starting point for incorporating these improvements. We leave such exploration as future work.\n\n>(Question 2: “How would your method deal with games with more ambiguous state values, such as Montezuma's Revenge?”)\n\nOur method of incorporating human feedback can also encourage exploration behaviors which is critical for challenging tasks such as Montezuma’s Revenge. We have observed a similar conclusion in Find Treasure, where human feedback can help explore the environment while the target object is not yet available to the agent, as in Fig. 7.  Our method is orthogonal to approaches that aim to reduce the ambiguity of feedback specifications. Exploring methods to reduce such ambiguity can be an interesting future direction.\n\n>“The human feedback is Markov (Question 3). This might not be true in more complex games.”\n(Question3: “The human feedback regressor $\\bar{H}(s, a)$ uses the Markov assumption, but the human feedback might be dependent on their memory of prior states. What if the human feedback is non-Markov?”)\n\nWe agree that human feedback may not always be Markov for any task. As we mentioned in Section 5.4, for Find Treasure and Hide-and-Seek, we stacked three consecutive frames as input to both the RL backbone and the feedback regressor. We found this to be sufficient to model human feedback. For more complex tasks, it will be interesting to explore whether incorporating more prior history steps or using a memory-based neural network will improve human feedback modeling.\n\n>“Minor typo and wording issues.”\n\nWe thank the reviewer for pointing out some typos and wording issues. We will improve these in the revised version: 1) change A to Q in Equation 1; 2) use upper-case Q in Equation 2; 3) Define $R_{t+k+1}$; 4) list specific designs in line 125 and be specific for lines 125-127; 5) fix spacing in citations; 6) rename computational framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lbk4Yn0nh8", "reviewer": "Reviewer_JtyH", "summary": "The paper introduce GUIDE, a RLHF framework for real-time RLHF with online and continous human feedback. GUIDE translates the human feedback to dense reward. Addtionally, GUIDE includes a parallel training model that learns a simulated human feedback. By involving 50 participants annotation, GUIDE solves three typical challenging sparse reward environments.", "questions": "Can you provide more details about the annotator's annotations, such as the actual interface and the frequency of operations?", "limitations": "Yes, and yes.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is easy to read.\n2. GUIDE firstly proposed novel continous human feedback and is also evaluated human annotators.\n3. GUIDE demonstrated improvement compared to baselines in 3 environments, and analyzed cognitive tests and analyses.", "weaknesses": "1. My biggest concern comes from the practicality of GUIDE. From both theoretical and experimental perspectives, I find it hard to believe that such a simple continuous feedback model can be applied to real-world scenarios. For example, the paper states in line 38 that \"Current research has demonstrated success primarily in simple, low-dimensional tasks with limited solution spaces.\" However, the experiments conducted in the paper also involve environments where baseline algorithms like DDPG or SAC can converge with a good reward function after only about **10 minutes of training**. Moreover, according to the experimental results, GUIDE, which incurs a high cost of human feedback, does not outperform manually designed simple rewards (such as the distance to the target, I think it is not hard to design it). Therefore, despite the fact that the environments used do have continuous actions and image inputs, I believe these environments are not suitable for validating RLHF algorithms because the reward functions are easy to design and the tasks themselves are simple.\n2. The core argument of the paper is that continuous real-time feedback is extremely difficult to implement in practice. It requires annotators to constantly provide scalar rewards without pause, and such absolute value annotations are more susceptible to biases from different individuals. Pair-wise annotation is much easier than absolute value annotation and can be conducted asynchronously with the training process. If an AI agent needs to be trained for several days, the cost will be unacceptable.\n3. Although the paper suggests using model predictions to synthesize feedback, such a simple supervised learning regression objective is unlikely to accurately model the complex reward distribution. My reasoning is that predicting the relative goodness of A and B is easier than predicting scalar reward values, but there will still be many prediction biases.\n4. The definitions of various symbols in the paper are imprecise and confusing, for example:\n- What is the meaning of A(s, a) in Equation 1? Also, A(s) = a in the line 123, is it the same?\n- difference of Q(s, a) and q?\n- How to get the r^hf?\nThese typos make it very difficult to understand the details of the paper.\n5. There is a lack of discussion on recent related works in RLHF, such as:\n- [1] White D, Wu M, Novoseller E, et al. Rating-based reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(9): 10207-10215.  \n- [2] Yuan Y, Hao J, Ma Y, et al. Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback[J]. ICLR2024.\n- [3] Guan L, Verma M, Guo S S, et al. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation[J]. Advances in Neural Information Processing Systems, 2021, 34: 21885-21897.\n- [4] Guan L, Valmeekam K, Kambhampati S. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences[J]. ICLR2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments. We would like to address all of your concerns and questions below with point responses:\n\n----\n>Practicality: “I find it hard to believe that such a simple continuous feedback model can be applied to real-world scenarios. For example, the paper states in line 38 that \"Current research has demonstrated success primarily in simple, low-dimensional tasks with limited solution spaces.\" However, the experiments conducted in the paper also involve environments where baseline algorithms like DDPG or SAC can converge with a good reward function after only about 10 minutes of training.”\n\nWe respectively disagree with the reviewer on this observation. Fig.5 shows the opposite observation where baseline algorithms did not converge within 10 minutes and showed no sign of convergence in 20 minutes in our new environments, Find Treasure and Hide-and-Seek. Therefore, these tasks remain significantly challenging for the current RL baselines. GUIDE, on the other hand, was able to improve over baselines by a large margin.\n\n>“GUIDE, which incurs a high cost of human feedback, does not outperform manually designed simple rewards (such as the distance to the target, I think it is not hard to design it). Therefore, despite the fact that the environments used do have continuous actions and image inputs, I believe these environments are not suitable for validating RLHF algorithms because the reward functions are easy to design and the tasks themselves are simple.”\n\nAs discussed in the paper, hand-design dense rewards typically do not exist in real-world scenarios and require extensive experience in reward engineering and a practical understanding of RL training. However, we aim to enable human guidance in RL for a broader audience who most likely do not have RL experience or reward engineering. Moreover, dense reward designs, such as precise distance to the target, as the reviewer suggested, assume extra information that is not available to GUIDE, whom only perceives partial visual observations. Therefore, our environments are strong and challenging testbeds for developing human-guided RL algorithms. Furthermore, as prior works have only focused on simple bowling games and low-dimensional observations, we believe that our environment is a large step forward in investigating the potential and scalability of human-guided RL.\n\n>“The core argument of the paper is that continuous real-time feedback is extremely difficult to implement in practice. It requires annotators to constantly provide scalar rewards without pause, and such absolute value annotations are more susceptible to biases from different individuals. Pair-wise annotation is much easier than absolute value annotation and can be conducted asynchronously with the training process. If an AI agent needs to be trained for several days, the cost will be unacceptable.”\n\nOur argument is not that continuous real-time feedback is extremely difficult to implement in practice. Instead, we point out that it provides richer feedback and assignments to every state-action pair than previous discrete feedback without a high cognitive load. Our contribution is to enable grounding such novel feedback modality through GUIDE.\n\nAs discussed in our paper, while pair-wise annotation is a popular choice in RLHF, it requires parallel policy rollouts in an asynchronous manner or offline trajectory collections. Our setting is different – we focus on real-time decision-making tasks where no such asynchronous rollouts, offline trajectories, or a simulator that can be run multiple times with policy rollouts and resetting are available. Due to the significant difference in the setting, pair-wise annotation is out of scope for our study.\n\n>“Although the paper suggests using model predictions to synthesize feedback, such a simple supervised learning regression objective is unlikely to accurately model the complex reward distribution. My reasoning is that predicting the relative goodness of A and B is easier than predicting scalar reward values, but there will still be many prediction biases.”\n\nAs discussed in our paper, while pair-wise annotation is a popular choice in RLHF, such as recent LLM studies, it requires parallel policy rollouts in an asynchronous manner or offline trajectory collections. Our setting is different – we focus on real-time decision-making tasks where no such asynchronous rollouts, offline trajectories, or a simulator that can be run multiple times with policy rollouts and resetting are available. Due to the significant difference in the setting, pair-wise annotation is out of scope for our study.\n\nWe agree that more advancements can be made to improve the simulated feedback model. However, prior to our work, there have not been studies enabling continual training in real-time human-guided RL. Our work is the first step towards this with potential future work to improve the optimization designs to handle more complex reward distributions.\n\n>“What is the meaning of A(s, a) in Equation 1? Also, A(s) = a in the line 123, is it the same? Difference of Q(s, a) and q? How to get the r^hf?”\n\nThank you for catching the typos. We will correct both $A(s, a)$ and $q(s, a)$  to $ Q(s, a)$. $r^{hf}$ is the human feedback reward provided by the human.\n\n>“There is a lack of discussion on recent related works in RLHF, such as: …”\n\nWe thank the reviewer for pointing out relevant literature, and we will include these in our revised version.\n\n>Can you provide more details about the annotator's annotations, such as the actual interface and the frequency of operations?\n\nYes. A screenshot of our interface is shown in Fig3 (B). The user hovers their mouse over the window to provide feedback. Moving upwards indicates stronger positive feedback, and downwards indicates stronger negative feedback. The decision frequency of the games is set to 0.5s/step. Hence, human feedback values are collected every 0.5 seconds.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presented a method that the reviewers found novel / interesting, and that overall the experiments supported the papers claims. However, the paper elicited a wide variety of concerns and some spirited debate.

# Concerns of the reviewers who rated the paper the lowest
## Reviewer fsCr (rating=4)
This reviewer had some concerns about the overall technical novelty, and said
> "a very solid set of human subject experiments would have been able to compensate for this ... but ... the experiments presented at this point are not quite complete enough". 

They pointed to concerns about the validity of the c-DeepTamer baseline both in terms of whether it is appropriate and whether the number of gradient steps used is valid, which the authors rebutted by stating that their baseline
>  "strictly follows Deep TAMER in the same way gradients were updated" and "We believe that a stronger solution would be able to update the network as standard ML practice, which is exactly one of our contributions beyond the original DeepTamer work. However, if we modify the original DeepTamer work to include our contribution, our experiments will become an ablation study of our own method instead of comparing it with prior work.".

I think the authors rebutted this point sufficiently. Furthermore, this reviewer suggested [A,B] as additional baselines, which the authors responded to partly by saying "experiments involving real-time human interactions are costly to run." I understand this point, and agree that it weakens the paper (regardless of the cost to run the experiments). The paper did not claim to be superior to prior work in its "key novelties" paragraph or experimental section, so this weakness doesn't undermine the main contributions as stated in those sections, yet the abstract _does_ state
>  "up to 32% increase in success rate compared to state-of-the-art baselines"

which is ambiguous and potentially misleading, as it suggests _all relevant baselines_ (including [A,B]) were run. Therefore, while I think this criticism is valid, it doesn't undermine the key novelties as stated in the majority of the paper, yet it does undermine a novelty stated in the abstract. Thus, I recommend the paper be accepted and the abstract be revised to more accurately and less ambiguously state the performance improvement, to be consistent with the remaining claims of the paper.

## Reviewer JtyH (rating=3)
This reviewer and the authors got into a spirited debate. Near the end of this discussion, the reviewer said
> “The feedback method of GUIDE is somewhat novel but useless, it requires a high level of annotator focus and a considerable tolerance for errors, making it infeasible for long-term annotation tasks.”

The authors responded to this by stating
> "the reviewer’s comments are based on conjecture without evidence"
> "Our continuous feedback mechanism still performs stronger than the discrete feedback baselines by a large margin on all 50 subjects, while the discrete feedback is less robust against individual differences."

Thus, the authors rebutted this point satisfactorily, in my judgement. This reviewer also stated 
> "My core opinion is that real-world tasks are usually very complex and cannot be learned in just 10 minutes". 

The authors responded partly by stating that the task attempted was challenging. I think both of these claims -- the authors's and reviewer's -- are hard to agree with because they are not backed by quantitative evidence. While I agree with the reviewer that evaluation on longer-term tasks would be an interesting direction for future work (and a stronger proof of efficacy), I believe it is not necessary for this paper to be acceptable, as it doesn't conflict with the paper's main claims. The authors also stated 
> "1) we do not assume any expert level of task execution as most imitation learning algorithms do; 2) our setup aims to tackle novel problems in real-time instead of assuming plenty of time and many offline rollouts or replay buffers for humans to provide feedback. "

I think the paper needs to clarify these assumptions and differences, e.g L185-186 states their approach allows for "capitalizing on human flexibility and expertise", which contradicts (1) above.

Given the somewhat strong consensus on the novelty and that the paper's claims seem reasonably supported by the evidence presented, I recommend acceptance.
Decision: Accept (poster)