GUIDE: Real-Time Human-Shaped Agents

Authors: lingyu  zhang
zhengran  ji
nicholas r waytowich
boyuan  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:04:58

Abstract: The recent rapid advancement of machine learning has been driven by increasingly powerful models with the growing availability of training data and computational resources. However, real-time decision-making tasks with limited time and sparse learning signals remain challenging. One way of improving the learning speed and performance of these agents is to leverage human guidance. In this work, we introduce GUIDE, a framework for real-time human-guided reinforcement learning by enabling continuous human feedback and grounding such feedback into dense rewards to accelerate policy learning. Additionally, our method features a simulated feedback module that learns and replicates human feedback patterns in an online fashion, effectively reducing the need for human input while allowing continual training. We demonstrate the performance of our framework on challenging tasks with sparse rewards and visual observations. Our human study involving 50 subjects offers strong quantitative and qualitative evidence of the effectiveness of our approach. With only 10 minutes of human feedback, our algorithm achieves up to 30\% increase in success rate compared to its RL baseline.

Main paper: 
 introduction: 
 Many real-world tasks are real-time decision-making processes [7, 48] that require understanding the problem, exploring different options, making decisions, refining our understanding, and adjusting decisions accordingly. Due to their inherent complexity, these tasks pose significant challenges for current machine learning systems. Effective exploration [8, 16, 41, 36] is crucial for gathering informative data, especially in mission-critical tasks such as search and rescue, disaster response, and medical emergencies where time is limited. Furthermore, open-world problems often lack dense labels and provide extremely sparse environment feedback signals [42, 19, 29, 21] . Therefore, agents must reason over long horizons to make informed decisions [24, 26, 43] .Human-guided machine learning [32, 33, 34, 31, 1, 52] , also known as human-in-the-loop machine learning, has been proposed to integrate human feedback into reinforcement learning (RL) agents. Various methods differ in how they obtain, format, and incorporate human feedback. Some approaches rely on full demonstrations [32, 34] or partial corrections [12] through imitation learning [32, 33, 34] or inverse RL [31, 1, 52] , assuming that humans can directly control the agents and possess expert-level task knowledge. Conversely, non-expert humans can often still judge the quality of the agent strategies, leading to comparison-based feedback, such as preferences [2, 20, 45, 18, 46, 17, 47, 23, 53] and rankings [9, 10] . These offline methods, however, require large datasets and parallel prompts and, hence, cannot support real-time guidance, limiting their applicability in dynamic environments. Fig. 1: We propose GUIDE as a novel framework for real-time human-shaped agents enabling continuous feedback and continual improvements without human trainers. We also aim to understand how individual differences affect their guided agents' performances. the discrete feedback. Current research has demonstrated success [25, 44, 28, 4] primarily in simple, low-dimensional tasks with limited solution spaces. Moreover, high-quality human feedback is challenging [11] to obtain due to variability in human guidance skills, cognitive biases, and individual differences. Most studies [25, 44, 28, 4, 50] involve small sample sizes (N < 10), often including the designers themselves, which makes it difficult to quantitatively assess the applicability of the methods to the real world. Furthermore, how to keep refining the agent without continuous human input remains unclear.We introduce GUIDE (Fig. 1 ), a framework for real-time human-guided RL that enables continuous human feedback and grounds such feedback into dense rewards, thereby accelerating policy learning. GUIDE also includes a parallel training algorithm that learns a simulated human feedback model to effectively reduce human inputs and enable continued policy improvements without human feedback. Our experiments span three challenging tasks characterized by continuous action spaces, highdimensional visual observations, temporal and spatial reasoning, multi-agent interactions, and sparse environment rewards. Our human studies involving 50 participants provide strong quantitative and qualitative support to the efficacy of our approach. Additionally, we conduct a series of cognitive tests and analyses to quantify individual differences among participants and explore how these differences correlate with agent learning performance. Our further analysis of the learned policies highlights the critical role of alignment in developing effective human-guided agents. 
related work: 
 Human-Guided Machine Learning Various computational frameworks have been proposed to integrate human feedback in guiding machine learning agents. Behavior cloning [32, 33, 34] trains a policy through supervised learning on human demonstrations assuming direct human control of the agents. Inverse reinforcement learning [31, 1, 52] infers objectives from human demonstrations for police optimization, but it requires extensive human demonstrations and often struggles with accurate objective inference and scalability. Human preference-based learning [2, 20, 45, 18, 46, 17, 47, 23] allows humans to score trajectories through comparisons, reducing the need for full demonstrations yet still depending on offline policy sampling. GUIDE differs by focusing on real-time decision-making tasks without relying on abundant offline datasets or parallel trajectory rollouts.Real-Time Human-Guided Reinforcement Learning Our work is formalized into an RL framework to integrate real-time human feedback into the policy learning process. TAMER [25] learns to regress a reward model from human feedback, which is used as a typical reward function for policy learning. Deep TAMER [44] generalizes this approach to higher dimensional inputs and uses a neural network to parameterize both the policy and the reward model. COACH [28] and Deep COACH [4] address the inconsistency of human rewards observed during training by modeling human feedback as an advantage function in an actor-critic framework. Deep COACH scales COACH to handle higherdimensional observations and more expressive policy representations. Recent advancements have further improved these algorithms, addressing various challenges in ground human feedback, such as different feedback modalities, stochasticity of feedback [3] , and continuous action space [37] . However, existing real-time human-guided RL studies share several limitations. Simplifying human feedback to discrete sets {positive, neutral, negative} with rewards {1, 0, -1} requires users to focus on clicking or typing the correct options which can be distracting. Such simplification also causes information loss by ignoring intermediate values. This approach also introduces hyperparameters to associate feedback to trajectories in time, requiring extensive tuning. Moreover, most studies validate their methods on simple tasks with discrete actions and low-dimensional observations. Furthermore, human studies are often limited to the designers themselves, a small group of subjects (N < 10, mostly N < 5), or a trained expert policy with RL to simulate humans. It remains unclear whether the existing approaches scale to general populations. There is also a large gap in understanding how individual differences among human participants affect policy training. Lastly, there is little discussion on continuing to improve policies in the absence of human feedback.Key Novelties of GUIDE Unlike previous approaches, GUIDE enables continuous human feedback, maximizing guidance information with minimal hyperparameter tuning. Our experiments cover challenging tasks with continuous action spaces, high-dimensional visual observations, spatialtemporal reasoning, and multi-agent interactions. GUIDE is also evaluated on the largest sample of human subjects (N = 50) among related studies, incorporating cognitive assessments to characterize individual differences and their impact on guiding learning agents. Additionally, GUIDE includes a parallel training algorithm to mimic human feedback for continued policy improvements when direct human feedback becomes unavailable.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of this work in the Conclusion, Limitations and Future Work section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
value-based reinforcement learning: 
 Value-based Reinforcement Learning is an effective approach to solving decision-making problems. In value-based RL, a key element is the action-value function Q(s, a) which provides an estimate of the total expected return when taking an action a in state s and following a specific policy thereafter [40] . The policy π is derived from Q by selecting the action that maximizes the action-value function:EQUATIONThe action-value function Q(s, a) at time step t can be defined as the expected sum of discounted future rewards with discounted factor γ:EQUATIONWhere R t+k+1 is the reward k + 1 steps after the current step. In valued-based deep reinforcement learning, a neural network is used to approximate the action-value function to tackle challenges such as high-dimensional and partial observations and complex decision boundaries. The training of these networks typically involves a combination of Monte-Carlo sampling and bootstrapping techniques, which leverage current estimates of Q to improve future predictions. The accuracy of the Q-function is crucial, especially in environments where reward signals are sparse and delayed, posing significant challenges to the efficiency and efficacy of the learning process.A natural way for humans to guide an agent's learning is to observe its inputs and provide feedback on its actions. This translates directly to incorporating human feedback into reinforcement learning by assigning human feedback as the myopic state-action value. Deep TAMER [44] is a prominent human-guided RL framework that leverages this concept by enabling humans to offer discrete, time-stepped positive or negative feedback. To account for human reaction time, a credit assignment mechanism maps feedback to a window of state-action pairs. A neural network F is trained to estimate the human feedback value, denoted as fs,a , for a given state-action pair (s, a). The policy then selects the action that maximizes this predicted feedback.We constructed a strong baseline by enhancing Deep TAMER in numerous ways. First, the original Deep TAMER's greedy action selection method only works with discrete action spaces. We designed a continuous version of Deep TAMER while adopting state-of-the-art reinforcement learning implementation practices. We implement an actor-critic framework to handle continuous action space.Here, the critic is the human feedback estimator parameterized by ϕ: F ϕ (s, a), directly estimating human feedback instead of a target action value. The actor is parameterized by θ: A θ (s) = a, aiming The feedback values are grounded into per-step dense rewards and combined with the environment reward. Concurrently, we train a human feedback simulator that takes in state-action pairs and regresses the feedback values. During the Automated guidance stage, the trained simulator stands in for the human and provides feedback to continue to improve the policy, effectively reducing human efforts and cognitive loads.to maximize the critic's output. The combined objective is defined as L c-Deep TAMER = L θ + L ϕ , where:EQUATIONWe follow recent advancements in model architectures and optimization strategies, such as target net soft updates and using Adam optimizer instead of SGD. Our strong baseline not only enhances Deep TAMER to continuous actions and recent RL practices but also maintains the core methodology of integrating real-time human feedback into the learning process.4 GUIDE: Grounding Real-Time Human FeedbackQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification:We will also open-source the entire code base, including algorithms and task environments for the broader community for full reproducibility. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Justification: We present the full implementation details in the paper including hyperparameters, task setups, and algorithms. We will also open-source the entire code base, including algorithms and task environments for the broader community for full reproducibility. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
method overview: 
 Developing real-time decision-making agents presents significant challenges. These agents must explore high-dimensional observations, constantly adapt, and make accurate decisions with limited supervisory signals. Real-time human-guided RL seeks to harness human expertise and adaptability. However, existing algorithms often underutilize human expertise by solely relying on discrete feedback. Additionally, integrating human feedback with environmental rewards can be complex, and human evaluation criteria often evolve during training.This section introduces our framework GUIDE (Fig. 2 ) to address these limitations. GUIDE enables real-time human guidance using dense and continuous feedback. The agent directly perceives the environment through visual frames, selects and executes actions, and receives continuous human feedback per decision step. This feedback is then converted into dense rewards that will be integrated with sparse environment rewards for effective learning. Concurrently, a separate neural network consistently observes the state-action pairs and human feedback and learns to predict the provided human feedback. This learned model will eventually replace the human as the feedback provider, significantly improving the efficiency of human-guided RL while minimizing the required human guidance time.Conventional real-time human-guided RL frameworks rely on feedback that is discrete in time and values. An example that we adopted as our baseline is shown in Fig. 3(A ). We propose a novel interface that enables human trainers to provide continuous-valued feedback at every time step. As illustrated in Fig. 3 (B), the trainer hovers their mouse over a window to indicate their assessment of the agent's behavior.Our method offers several advantages:• Natural Engagement: Hovering is a more natural interaction compared to clicking buttons [39] , fostering continuous human guidance without interrupting the training flow. • Expressive Feedback: Continuous values allow humans to express nuanced assessments, capturing a wider range of feedback compared to discrete options, as shown in Fig. 3 . • Constant Training: Unlike discrete-feedback algorithms where the model update frequency is based on the feedback frequency, the continuous feedback ensures that the model continuously learns and adapts. • Simplified Credit Assignment: In our setting, it is reasonable to assume a constant human feedback delay, alleviating the need for complex credit assignment to map delayed feedback to specific state-action pairs. Such delayed association time window has been reported to have inconsistent values across different studies. In contrast, we used the same one-time delay factor for all our studies. While human guidance offers valuable supervision, sparse environment rewards or terminal rewards remain crucial for their easy access and the ability to induce desired targets. However, directly using human feedback as a value function makes incorporating existing environment rewards challenging [44] . We propose a straightforward but effective solution: convert the human feedback at each time step t into a reward, denoted as f t = r hf t . This allows the seamless integration of environment rewards through simple addition:r t = r hf t + r env t. This approach also enables leveraging off-the-shelf advanced RL algorithms, which typically make minimal assumptions about the reward function. Our method can be seen as interactive reward shaping [30] which is a vastly effective approach in handling sparse and long-horizon tasks. However, designing dense reward functions directly can be difficult as it often requires significant prior knowledge, manual effort, and trial and error. Moreover, it lacks adaptability to unforeseen scenarios. Our approach allows for real-time reward shaping through human guidance, capitalizing on human flexibility and intuition.Human guidance can be time-consuming and cognitively demanding. To maximize the benefit of human input while minimizing their effort, we propose a regression model that learns to mimic human feedback, acting as a surrogate for the human trainer (Fig. 2 ). This model allows for continual policy improvements even when human feedback is not available. Our idea is based on the assumption that human feedback implicitly follows a mapping function: H(s, a) = f , where s and a are the state and action observed by the human, and f is the assigned feedback value.The key insight is that during human guidance, we can readily collect state-action pairs along with their assigned feedback values, which allows training a human feedback simulator, parameterized by a neural network Ĥ(s, a) = f , by minimizing ||f -f || 2 . This training can occur concurrently with human guidance to prepare to substitute at any time or offline for later deployment in continual training. To prevent overfitting, we held out 1 out of 5 trajectories as a validation set. Once trained, the simulator can operate in inference mode to provide feedback in the absence of the human trainer. This model makes no assumptions about the human's specific feedback style or patterns. Instead, it learns to provide feedback consistent with the human trainer, minimizing the shift in reward distribution. 
environments: 
 We conduct our experiments on the CREW [51] platform.Bowling: A modified version of Atari Bowling. Each episode consists of 10 rolls. The agent receives a reward equal to the number of pins hit after each roll. The action space is 3-dimensional: the initial ball position, the distance to start steering the ball, and the steer direction. Observation is the visual image.Find Treasure: The agent is tasked to navigate through a partially observable maze to retrieve a treasure. The agents receive a +10 reward upon reaching the treasure, and a constant -1 time penalty for each step taken. The observation space is a top-down accumulated view of where the agent has explored, initialized by a square area around the agent. The action space is a two-dimensional vector of the next destiny location, to which a low-level planning algorithm will navigate the agent. The max episode length is 15 seconds.1v1 Hide-and-Seek: A multi-agent one-on-one visual hide-and-seek task [13, 14] , where we aim to learn the seeker policy whose goal is to navigate through the maze and catch the hider. The hider policy is a heuristic that will avoid obstacles and reflect when running into a wall while moving away from the seeker if the seeker is within a certain range. The reward, observation, and action spaces remain the same as Find Treasure.Evaluation Metrics We allocated 10 minutes for consistent human feedback. We collected the checkpoints and human feedback data and performed continual training for another 10 minutes. The model's performance was evaluated in a test environment every minute for Bowling and every 2 minutes for Find Treasure and Hide-and-Seek. For every Bowling checkpoint, we evaluate for 1 game (10 rolls); for Find Treasure and 1v1 Hide-and-Seek, we evaluate 100 episodes for every checkpoint. All test results are reported on unseen test conditions.Baselines We include two state-of-the-art reinforcement learning algorithms and a human-guided RL algorithm as baselines. We selected Deep Deterministic Policy Gradient (DDPG) [27] and Soft Actor-Critic (SAC) [22] due to their superior performance on many benchmarks. We compared GUIDE to the continuous version of Deep TAMER as described in Sec. 3.2, denoted as c-Deep TAMER under the same amount of human training time. We also run experiments on Find Treasure and Hide and Seek with heuristic feedback, in other words, a dense reward. The feedback is an exploration reward when the target (treasure or hider) is not within the agent's visible view and a distance reward when it is. This can be seen as an upper bound of our method, as this is the "heavy engineered reward" by domain experts with no noise or delay.Human Subjects. We recruited 50 adult subjects to participate in the experiments under the approval of the Institutional Review Board. The subjects have no prior training or knowledge of the algorithms or tasks and have only been instructed to assess the performance of the agents. Inter-trial intervals are included between individual game experiments. For each session, the subject will receive $20 compensation.In order to quantify individual differences among participants and explore how these differences affect the human-guided RL performance, we conducted a series of cognitive tests to evaluate their cognitive abilities relevant to the training tasks before the human-guided RL experiments. We describe the details of the cognitive tests as follows. All tests involve six trials for each participant.Eye Alignment [35] (Fig. 4 )(A): The subject is asked to align a ball on the left side of the screen with a target on the right side as accurately as possible within five seconds without using any tools. The score is calculated as the negative average of the distances between the ball's and the target's horizontal positions across trials.Reflex [6] (Fig. 4 )(B): The subject is asked to click the screen as quickly as possible when it changes from yellow to green. Clicking while the screen is still yellow or failing to click within two seconds after the screen turns green is considered a failure. The score is calculated as the negative average of the reflex times, with failures being left out in the score calculation.Theory of Behavior [15] (Fig. 4 )(C): The subject observes a red ball moving in an unknown but fixed pattern for five seconds. Afterward, the ball pauses, and the subject must predict the ball's position one second after it resumes moving. The subject has two seconds to make this prediction. The score is calculated as the negative average of the distances between the ball's actual positions and the subject's predicted positions across trials.Mental Rotation [38] (Fig. 4 )(D): The subject is tasked with identifying the piece among three similar pieces that cannot be directly rotated to match the target piece within eight seconds. The score is calculated based on the accuracy of the subject's identifications across all trails.Mental Fitting [38] (Fig. 4 )(E): The subject is tasked with identifying the only piece among three similar pieces that can fit with the target piece within eight seconds. The score is calculated based on the accuracy of the subject's identifications across all trials.Spatial Mapping [5] (Fig. 4 )(F): A video of an agent navigating a maze with a restricted field of view is presented to the subject. After watching the video, the subject is asked to identify the maze from a selection of three similar mazes within five seconds. The score is calculated based on the accuracy of the subject's identifications across all trials.RL backbone and model architecture. The state-of-the-art visual reinforcement learning algorithm transitioned from SAC [22] to DDPG [27] due to its stronger performance enabled by the easy incorporation of multi-step returns and the avoidance of entropy collapse [49] . Following this, we also select DDPG [27] as our Reinforcement Learning backbone for GUIDE. We used a 3-layer convolutional neural network as the vision encoder. The actor, critic, and learned feedback model shared this encoder, and each follows it with a 3-layer Multi-Layer Perceptron.Hyperparameters We used an Adam optimizer with a fixed learning rate of 1e-4 for RL policy training, with a discount factor of γ = 0.99. We applied gradient clipping setting max grad norm to 1. For the learned feedback model, we used the same Adam optimizer with 1e-4 learning rate and employed early stopping based on the loss on held-out trajectories. For Deep TAMER's credit assignment window, we used the same uniform [0.2, 4] distribution as in the original paper. We used Human delay factor. Human delay is non-neglectable in human-guided RL. The time it takes for a human to see and process the states and to produce a feedback signal varies across environments and individuals. Preliminary experiments with the authors suggested that shifting human feedback by 2 seconds for Bowling and 1 second for Find Treasure and Hide and Seek best aligned with the intended state-action pairs.Human-Guided RL performance We report our results in Fig. 5 , where the x-axis is the total training time, and the y-axis is either the score for Bowling or the success rate for Find Treasure and Hide-and-Seek. The dashed line separating the first and second 10 minutes indicates the end of the human guidance and the beginning of the automated guidance. The first row reports the training results of the 15 human trainers who scored the highest on the cognitive test. The second row is the average performance of all 50 human subjects. We observe that subjects who tested higher on cognitive skills result in higher performing agents, as the trained AI performance approaches closer to the heuristic feedback upper bound. For the simple Bowling task, all methods perform similarly, with GUIDE having a slight advantage over DDPG. On the challenging Find Treasure and Hide-and-Seek task, GUIDE-DDPG scored up to 30 percent higher success rate than its RL counterpart, and 50 percent higher than c-Deep TAMER given the same amount of human guidance time. It is worth noting that in this time-critical setting, GUIDE-DDPG is able to reach the same level of performance as its RL baselines within half the training time. The average of all 50 untrained human trainers was able to surpass its RL baseline by a large margin while also surpassing our greatly enhanced Deep TAMER on the challenging Find Treasure and Hide and Seek by up to 40 points in success rate.Exploration Analysis The ability to explore the state space is considered one of the key aspects of successful sparse reward RL. We hypothesize one benefit of human guidance is inducing efficient exploration behavior. We quantify exploration tendency on Find Treasure by measuring the accumulated visible area ratio. After two minutes of human feedback within the GUIDE framework, the We qualitatively analyze the effect of the learned feedback model by visualizing its predictions on unseen trajectories. As shown in Fig. 7 , our learned feedback model generalizes to unseen trajectories and is able to provide effective feedback in place of humans. The darker the color, the more statistically significant the correlation. "+": positive, "-": negative.To further characterize individual differences and how they affect policy training, we found a significant correlation between the cognitive test ranking of the human subjects and their trained AI performance, with a Pearson correlation coefficient of 7.522 and a p-value of 0.001, as shown in Fig. 8 . Among all cognitive tests, the mental rotation test and Mental Fitting test are the strongest indicators of overall guided AI success. We also discovered relationships between individual tasks and cognitive tests. Notably, 1v1 Hide-and-Seek score is positively related to reflex time. This is likely due to the fact that Hide-and-Seek is a quickly evolving dynamic task, and faster response time will In each subplot, the x-axis is training milestones indicated by task scores, and the y-axis is the percentage of humans who were able to train the agent to reach the milestone at a given time. We find that GUIDE's robustness scales to more complex tasks.result in better matching in feedback and state-action pairs. Detailed correlation results can be found in the appendices.Robustness to individual differences A critical aspect of human-guided machine algorithms is the method's robustness to different human trainers. A generalizable algorithm should maintain a strong performance when human feedback patterns vary. However, this is rarely discussed by prior methods likely due to the small size of human participants. Here we provide a thorough analysis on GUIDE's performance across our 50 human trainers compared to c-Deep TAMER. Each subplot in Fig 9 showsfor each human-guided algorithm, the percentage of human trainers (y-axis) that are able to train the agent to reach specific milestones indicated by task scores (x-axis). Panel A shows that for the Bowling task, c-Deep TAMER and GUIDE are close in performance. Panel B and C shows that for Find Treasure and 1v1 Hide-and-Seek, as the complexity of the game increases, more human trainers are able to reach milestones using GUIDE than using c-Deep TAMER, demonstrating the scalability of the robustness of GUIDE to individual differences. 
conclusion, limitations and future work: 
 We introduce GUIDE, a novel framework for real-time human-guided RL. GUIDE leverages continuous human feedback to effectively address limitations associated with sparse environment rewards and discrete feedback approaches. Furthermore, GUIDE incorporates a parallel training algorithm to learn a human feedback simulator, enabling continued policy improvements without ongoing human input. Our extensive experiments demonstrate the efficacy of GUIDE across challenging tasks while presenting human cognitive assessments to understand individual differences.While GUIDE offers significant advantages, it presents some limitations and exciting avenues for future work. First, our current evaluation focuses on tasks with moderate complexity. Future work can explore scaling GUIDE to highly complex environments and large-scale deployments. Moreover, we have quantitatively observed strong individual differences in guiding learning agents through extensive human studies. However, we have not explored how to mitigate such differences and account for human variability in feedback patterns. Furthermore, understanding how the human feedback simulator operates and interprets the agent's behavior remains an open question. Future work will delve into explainable learning techniques to shed light on this process. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: To the best of our knowledge, we present the most extensive studies among related studies with N = 50 human subjects with detailed analysis. We report the full results with extensive evaluation runs and the means and standard deviations. For all other RL baselines, results are also reported with multiple random seeds with the mean and standard deviations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: This work does not fit into the stated negative societal impacts. We discuss the positive societal impacts in the abstract and the introduction sections. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: this paper poses no such risks. Our human studies have been approved by the Institutional Review Board, and we will not release or share any identifiable data. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have properly cited all the previous works used in this paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.13. New Assets Question: Are new assets introduced in the paper well and is the documentation provided alongside the assets? Answer: [Yes] Justification: We will open-source the entire code base for our algorithms, baselines, and experiment environments. We will also include detailed documentation. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [Yes] Justification: Our human studies have been approved by the Institutional Review Board with all detailed documents. We have included as many details as possible in the main texts. We plan to release our instruction videos for the human subjects as well. We have included the compensation in the paper. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: All human subject experiments are conducted on desktops with one NVIDIA RTX 4080 GPU. All evaluations are run on a headless server with 8 × NVIDIA RTX A6000 and NVIDIA RTX 3090 Ti.NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Our extensive experiment results support our claims in the abstract and introduction. We also summarize our contributions in the Conclusion section. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper presented a method that the reviewers found novel / interesting, and that overall the experiments supported the papers claims. However, the paper elicited a wide variety of concerns and some spirited debate.

# Concerns of the reviewers who rated the paper the lowest
## Reviewer fsCr (rating=4)
This reviewer had some concerns about the overall technical novelty, and said
> "a very solid set of human subject experiments would have been able to compensate for this ... but ... the experiments presented at this point are not quite complete enough". 

They pointed to concerns about the validity of the c-DeepTamer baseline both in terms of whether it is appropriate and whether the number of gradient steps used is valid, which the authors rebutted by stating that their baseline
>  "strictly follows Deep TAMER in the same way gradients were updated" and "We believe that a stronger solution would be able to update the network as standard ML practice, which is exactly one of our contributions beyond the original DeepTamer work. However, if we modify the original DeepTamer work to include our contribution, our experiments will become an ablation study of our own method instead of comparing it with prior work.".

I think the authors rebutted this point sufficiently. Furthermore, this reviewer suggested [A,B] as additional baselines, which the authors responded to partly by saying "experiments involving real-time human interactions are costly to run." I understand this point, and agree that it weakens the paper (regardless of the cost to run the experiments). The paper did not claim to be superior to prior work in its "key novelties" paragraph or experimental section, so this weakness doesn't undermine the main contributions as stated in those sections, yet the abstract _does_ state
>  "up to 32% increase in success rate compared to state-of-the-art baselines"

which is ambiguous and potentially misleading, as it suggests _all relevant baselines_ (including [A,B]) were run. Therefore, while I think this criticism is valid, it doesn't undermine the key novelties as stated in the majority of the paper, yet it does undermine a novelty stated in the abstract. Thus, I recommend the paper be accepted and the abstract be revised to more accurately and less ambiguously state the performance improvement, to be consistent with the remaining claims of the paper.

## Reviewer JtyH (rating=3)
This reviewer and the authors got into a spirited debate. Near the end of this discussion, the reviewer said
> “The feedback method of GUIDE is somewhat novel but useless, it requires a high level of annotator focus and a considerable tolerance for errors, making it infeasible for long-term annotation tasks.”

The authors responded to this by stating
> "the reviewer’s comments are based on conjecture without evidence"
> "Our continuous feedback mechanism still performs stronger than the discrete feedback baselines by a large margin on all 50 subjects, while the discrete feedback is less robust against individual differences."

Thus, the authors rebutted this point satisfactorily, in my judgement. This reviewer also stated 
> "My core opinion is that real-world tasks are usually very complex and cannot be learned in just 10 minutes". 

The authors responded partly by stating that the task attempted was challenging. I think both of these claims -- the authors's and reviewer's -- are hard to agree with because they are not backed by quantitative evidence. While I agree with the reviewer that evaluation on longer-term tasks would be an interesting direction for future work (and a stronger proof of efficacy), I believe it is not necessary for this paper to be acceptable, as it doesn't conflict with the paper's main claims. The authors also stated 
> "1) we do not assume any expert level of task execution as most imitation learning algorithms do; 2) our setup aims to tackle novel problems in real-time instead of assuming plenty of time and many offline rollouts or replay buffers for humans to provide feedback. "

I think the paper needs to clarify these assumptions and differences, e.g L185-186 states their approach allows for "capitalizing on human flexibility and expertise", which contradicts (1) above.

Given the somewhat strong consensus on the novelty and that the paper's claims seem reasonably supported by the evidence presented, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a new framework, GUIDE, for learning from continuous human feedback in complex decision making domains with continuous action spaces. By framing human feedback as a state-action value function, the framework proposes to learn this function and to combine it additively with the (generally sparse) reward coming from the environment. The feedback is collected in a continuous fashion by asking participants to move their mouse up or down to indicate higher or lower feedback values. In a user study, the paper finds that training agents with this type of feedback yields better performing agents than baselines. After assessing participants in a suite of cognitive tests, it finds that participants that score higher on the cognitive tests trained better agents.
strength_weakness: strength_weakness: The assumptions regarding what human feedback represents do not seem consistent between section 3 and 4 (see Questions). Further, the treatment of the feedback collection is rather simple (added to environment reward function) and, especially if it does represent a signal regarding the future value of state-action pairs, heuristic. Relative to Tamer and Deep Tamer, which treated human feedback more consistently by using it directly as a proper state-action value function, this paper feels like a regression on that front. 

The implementation of the c-DeepTamer baseline raise a number of questions (see Questions), which shake my confidence in it as a baseline, or as a proper representative for how well Deep Tamer should perform here.
questions: questions: In 3.2 you propose to extend Deep Tamer by using the human feedback estimator F(s,a) as a critic within an actor-critic framework. Thus, you treat human feedback here as an estimate of the future value of each state-action pair (a Q-value). In 4.3, you propose to use the human feedback signal as a reward, to be added to the environment reward. Does this mean you make a different assumption regarding human feedback in 4.3. than in 3.2? Or do you propose to use feedback on future value as a reward?

I am confused by your claim that Deep Tamer uses DQN. Looking at algorithm 1 in the Deep Tamer paper, it’s policy greedily chooses the action that maximizes the learnt human value function H(s,a). I do not see any use of DQN.

Did your c-DeepTamer baseline receive the same (sparse) environment reward as GUIDE? And if so, how did you use this reward signal in conjunction with a critic that is trained from human feedback only?

Why was the training of c-DeepTamer stopped after 5/10 minutes? Would it not be possible to continue training it using simulated human feedback (similar to GUIDE) or without human feedback?

Within the user study, did participants get time to familiarize themselves with each environment before data recording began? I imagine participants would have needed some time to figure out how to solve each task.

Is equation (3) correct? What is the purpose of he additional $- F(s, A(s))$ term?
limitations: limitations: Limitations are discussed in the conclusion.
paper_summary: paper_summary: The paper proposes a new framework for human-in-the-loop reinforcement learning, where the human and provide real-time and continuous feedback, and an algorithm where the learning agents uses the human feedback to accelerate policy learning. The paper conducted a user study of 50 subjects to demonstrate the effectiveness of the proposed framework in accelerating policy learning and improving success rates over RL and human-in-the-loop RL baselines. Optionally, a human feedback simulator can also be trained to mimic human feedback after a certain amount of time, reducing the amount of human input.
strength_weakness: strength_weakness: - The baselines are generally quite weak. Based on the experiment results, it is unclear whether real-time continuous feedback is necessarily the best way for humans to guide the policy learning. There might be intermediate points on the spectrum of conventional discrete feedback and full continuous feedback that provides the best tradeoff between amount of human input and effectiveness of guiding policy learning.
- Whether the simulated human feedback is helpful is unclear. In both the "bowling" the "find treasure" task, the score does not increase much after switching to simulated feedback. It might be the case that the simulated human feedback only works for tasks where it is straightforward to model the reward.
questions: questions: - How does human-in-the-loop compare to BC or reward engineering for the same amount of human input?
- How are the three tasks for evaluation chosen, and what are the motivations for such choices?

One motivation for doing human-in-the-loop RL is for tasks where designing a reward function or providing demonstrations are difficult, such as the backflip task. However, for the tasks examined in this paper, there seems to be other simple ways for the RL agent to learn from human input.
limitations: limitations: The paper focused on the final success rate of policy learning, but did not provide sufficient data from the user's perspective. For example, the user study failed to include a survey regarding whether the real-time feedback system feels easier to use than the discrete feedback system.
paper_summary: paper_summary: The paper proposes a new approach to reinforcement learning with human feedback in simple video games. The method relies on continuous human feedback that is provided by the human observer hovering their mouse over a window with a spectrum of positive and negative rewards. Unlike prior approaches, this method converts human feedback directly into a reinforcement learning reward with an added time delay. Moreover, the method includes a model that regresses states into observed human feedback, which allows for simulated human feedback. The effectiveness of the method is demonstrated in three simple games in a human study with 50 participants.
strength_weakness: strength_weakness: There are three major unstated assumptions:

1. The delay between an event appearing on the screen and the change in human feedback is constant (Question 1). The authors tune this constant for each environment. But, more complex environments might induce different delays as the human observer might need to think about what they saw.

2. People are able to provide constant feedback (Question 2). This might not be true for more complex environments where certain states might have ambiguous values.

3. The human feedback is Markov (Question 3). This might not be true in more complex games.

## Detailed comments:

* Equation 1 should have Q instead of A. Unless you want to define an advantage function A.
* Equation 2 should have an upper-case Q.
* The term $R_{t+k+1}$ in Equation 2 is not very clear.
* The meaning of “We follow recent advancements in neural architectures and hyperparameter designs” on line 125 is not clear.
* The rest of the paragraph on lines 125 - 127 is superfluous.

## Minor comments:

* Inconsistent spacing between text and citations.
* Calling this approach a “computational framework” might be a bit redundant given the context of the conference.
questions: questions: 1. Is it reasonable to assume constant reaction time of the human observer?

2. How would your method deal with games with more ambiguous state values, such as Montezuma's Revenge?

3. The human feedback regressor $\bar{H}(s, a)$ uses the Markov assumption, but the human feedback might be dependent on their memory of prior states. What if the human feedback is non-Markov?
limitations: limitations: Limitations are partially addressed.
paper_summary: paper_summary: The paper introduce GUIDE, a RLHF framework for real-time RLHF with online and continous human feedback. GUIDE translates the human feedback to dense reward. Addtionally, GUIDE includes a parallel training model that learns a simulated human feedback. By involving 50 participants annotation, GUIDE solves three typical challenging sparse reward environments.
strength_weakness: strength_weakness: 1. My biggest concern comes from the practicality of GUIDE. From both theoretical and experimental perspectives, I find it hard to believe that such a simple continuous feedback model can be applied to real-world scenarios. For example, the paper states in line 38 that "Current research has demonstrated success primarily in simple, low-dimensional tasks with limited solution spaces." However, the experiments conducted in the paper also involve environments where baseline algorithms like DDPG or SAC can converge with a good reward function after only about **10 minutes of training**. Moreover, according to the experimental results, GUIDE, which incurs a high cost of human feedback, does not outperform manually designed simple rewards (such as the distance to the target, I think it is not hard to design it). Therefore, despite the fact that the environments used do have continuous actions and image inputs, I believe these environments are not suitable for validating RLHF algorithms because the reward functions are easy to design and the tasks themselves are simple.
2. The core argument of the paper is that continuous real-time feedback is extremely difficult to implement in practice. It requires annotators to constantly provide scalar rewards without pause, and such absolute value annotations are more susceptible to biases from different individuals. Pair-wise annotation is much easier than absolute value annotation and can be conducted asynchronously with the training process. If an AI agent needs to be trained for several days, the cost will be unacceptable.
3. Although the paper suggests using model predictions to synthesize feedback, such a simple supervised learning regression objective is unlikely to accurately model the complex reward distribution. My reasoning is that predicting the relative goodness of A and B is easier than predicting scalar reward values, but there will still be many prediction biases.
4. The definitions of various symbols in the paper are imprecise and confusing, for example:
- What is the meaning of A(s, a) in Equation 1? Also, A(s) = a in the line 123, is it the same?
- difference of Q(s, a) and q?
- How to get the r^hf?
These typos make it very difficult to understand the details of the paper.
5. There is a lack of discussion on recent related works in RLHF, such as:
- [1] White D, Wu M, Novoseller E, et al. Rating-based reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(9): 10207-10215.  
- [2] Yuan Y, Hao J, Ma Y, et al. Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback[J]. ICLR2024.
- [3] Guan L, Verma M, Guo S S, et al. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation[J]. Advances in Neural Information Processing Systems, 2021, 34: 21885-21897.
- [4] Guan L, Valmeekam K, Kambhampati S. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences[J]. ICLR2023.
questions: questions: Can you provide more details about the annotator's annotations, such as the actual interface and the frequency of operations?
limitations: limitations: Yes, and yes.

Comments: comment: Thank you for your response. We would like to address all of your concerns and questions below.

----
>“I believe the author misunderstood my point. My core opinion is that real-world tasks are usually very complex and cannot be learned in just 10 minutes. Therefore, longer training times and more complex environments pose significant challenges to the GUIDE method mentioned in the paper. Many RLHF methods use environments like Atari or Minecraft for training to demonstrate their effectiveness in complex settings. However, the author has not proven this point, and the simulated feedback reward section lacks sufficient and comprehensive experimental evidence. I am not fully convinced that GUIDE can promote the development of the RLHF community. “

We would like to ask for specific clarifications from the reviewer. Though we have emphasized that one of our tasks is a challenging Atari game, and the other two tasks are similar to sub-tasks in Minecraft, the reviewer kept referring to “many RLHF methods use environments like Atari or Minecraft.” To our knowledge, and as stated in our paper, state-of-the-art algorithms with the same settings as ours (real-time human-guided RL) use much simpler environments than ours. In fact, ours is the most challenging benchmark experiment. For instance, Deep Tamer uses bowling, which is included in our experiment, and Deep COACH[1] uses Minecraft (a drastically simplified version that is simply goal navigation in a 10 x 10 grid world with discrete actions). Ours involve continuous control from partial visual observations and competitive settings.

If the reviewer is referring to other work that uses Atari or Minecraft but does not fall into our same problem setting, such as preference-based offline setting or imitation learning, we would like to argue that this is not a fair request to ask us to compare or use the same tasks. Our setting is much more challenging – 1) we do not assume any expert level of task execution as most imitation learning algorithms do; 2) our setup aims to tackle novel problems in real-time instead of assuming plenty of time and many offline rollouts or replay buffers for humans to provide feedback. Therefore, we kindly ask the reviewer to point at specific literature that uses the same setting but uses Atari or Minecraft as their comparisons.

>“The feedback method of GUIDE is somewhat novel but useless, it requires a high level of annotator focus and a considerable tolerance for errors, making it infeasible for long-term annotation tasks.”

We disagree with the reviewer’s statement. We would like to point out that the reviewer’s comments are based on conjecture without evidence.  As stated above, from our results, there is no evidence showing that a continuous feedback interface poses a higher cognitive load than discrete feedback since both require the users to watch the agent continuously, while GUIDE does not require constant switching and clicking on different buttons, nor evidence showing that continuous feedback requires a considerable tolerance for errors. From our 50 human subject studies as the experiment with the largest scale, our results show the opposite. Our continuous feedback mechanism still performs stronger than the discrete feedback baselines by a large margin on all 50 subjects, while the discrete feedback is less robust against individual differences. We have demonstrated the effectiveness of our feedback mechanism in our experiment results, as pointed out above, contrary to the reviewer’s comments that our feedback method is “useless”.

----
*references*

[1] Arumugam, Dilip, et al. "Deep reinforcement learning from policy-dependent human feedback." arXiv preprint arXiv:1902.04257 (2019).
Thank you for your response. We are glad that the reviewer found the feedback mechanism to be elegant and novel. We would like to address all of your concerns and questions below.

----
> “I like the proposed mechanism for giving feedback. It is elegant and to my knowledge novel. Reviewer JtyH has a point by saying that complex RL tasks will need more significant amounts of feedback, but in my view any interactive reward learning method would. The idea of combining environment reward with human feedback, however, cannot be counted as a novel contribution considering earlier work in this area [A,B]. Overall, the technical contribution is thus limited.”


We agree that prior work has also used human feedback for reward shaping. However, we believe that our main contribution is using such a method to ground real-time dense continuous feedback and conduct large-scale human experiments to verify it. We will add relevant discussion of this literature in our revised manuscript.


> “in my view the experiments presented at this point are not quite complete enough. The c-DeepTamer baseline is artificially limited. My best understanding is that each bit of feedback was used only once in a gradient update, contrary to standard ML practice (please correct me if I'm wrong).”


To maintain a fair comparison, c-DeepTamer strictly follows Deep TAMER in the way gradients are updated. As described in the original Deep TAMER paper Algorithm 1, the gradient updates happen:

(a) whenever the human provides new feedback, using the new feedback information as the data sample; and

(b) at a fixed rate, using data sampled from the replay buffer.

We believe that a stronger solution would be able to update the network as standard ML practice, which is exactly one of our contributions beyond the original DeepTamer work. However, if we modify the original DeepTamer work to include our contribution, our experiments will become an ablation study of our own method instead of comparing it with prior work. Considering the expensive 50 human studies, which already cost 2 hours per subject and $1,000, running ablations is unfortunately not feasible. Our choice is to focus on comparing the previous literature in the same setting as our problem.


> “Further, c-DeepTamer is not able to make use of the environment's reward, raising questions about whether it is a fair baseline in the first place. Why not use a baseline like the [A,B] that could use the environment's reward? The paper could also have been made much stronger by comparing to alternative interactive methods for learning from humans, including learning from preferences [C] and learning from demonstrations (e.g. [B]).”


We agree that additional baselines would be interesting to compare with. However, experiments involving real-time human interactions are costly to run. Our current 50 human experiments include three environments, two algorithms, cognitive tests, and initial and ending setup, as traditional human experiments already cost nearly 2 hours. The total cost of the experiments is $1,000. Longer experiment sessions will not only be more costly but will also affect human feedback quality. Under the time constraint, we chose the baseline that had the closest setting to ours. Deep TAMER is the state-of-the-art in real-time human-guided RL through scalar feedback of state-action pairs. [A] is not real-time, as the human operator provides feedback to trajectories sampled from a replay buffer. They also only provide feedback to actions or states instead of state-action pairs. As the reviewer has mentioned, [B] learns from demonstration, and [C] learns from preference, which are different settings from learning from feedback.

We would like to clarify that our work did not argue that real-time continuous human feedback is the optimal modality to provide feedback. In particular, we did not argue that this modality is better or can replace demonstration or preference learning. For demonstration, it has a different assumption that humans may be experts in the tasks, and this does not fall into our problem setting. In fact, we observe that humans typically have difficulty performing very well in our challenging hide-and-seek task where things change very fast, and the observations are partial. Preference learning is offline, which assumes multiple rollouts are provided, and humans do not need to react in real-time. It is often applied to the whole trajectories instead of state-action pairs. Our real-time continuous human feedback proposes a different problem setup and can be an alternative modality. However, we leave the exploration of multiple modalities as future work. Therefore, from both the perspective of financial, time and human costs and the problem-setting perspective, we did not compare the baselines from different modalities or problem settings. Our experiments indeed show significant improvement over real-time discrete feedback, which supports our claims.
Thank you for your detailed response. Unfortunately I continue to see two weaknesses in the paper.

I like the proposed mechanism for giving feedback. It is elegant and to my knowledge novel. Reviewer JtyH has a point by saying that complex RL tasks will need more significant amounts of feedback, but in my view any interactive reward learning method would. This could have been explored more in the paper. The idea of combining environment reward with human feedback, however, cannot be counted as a novel contribution considering earlier work in this area [A,B]. Overall, the technical contribution is thus limited.

A very solid set of human subject experiments would have been able to compensate for this, but in my view the experiments presented at this point are not quite complete enough. The c-DeepTamer baseline is artificially limited. My best understanding is that each bit of feedback was used only once in a gradient update, contrary to standard ML practice (please correct me if I'm wrong). Further, c-DeepTamer is not able to make use of the environment's reward, raising questions about whether it is a fair baseline in the first place. Why not use a baseline like the [A,B] that could use the environment's reward? The paper could also have been made much stronger by comparing to alternative interactive methods for learning from humans, including learning from preferences [C] and learning from demonstrations (e.g. [B]).

I appreciate that the authors have clarified some of my concerns, and will raise my score accordingly, but I still feel that the paper falls short of the acceptance threshold.

[A] Xiao, Baicen, et al. "FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback." Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. 2020.  
[B] Brys, Tim, et al. "Reinforcement learning from demonstration through shaping." Proceedings of the 24th International Conference on Artificial Intelligence. 2015.  
[C] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in neural information processing systems 30 (2017).
I believe the author misunderstood my point. My core opinion is that real-world tasks are usually very complex and cannot be learned in just 10 minutes. Therefore, longer training times and more complex environments pose significant challenges to the GUIDE method mentioned in the paper. Many RLHF methods use environments like Atari or Minecraft for training to demonstrate their effectiveness in complex settings. However, the author has not proven this point, and the simulated feedback reward section lacks sufficient and comprehensive experimental evidence. I am not fully convinced that GUIDE can promote the development of the RLHF community. The feedback method of GUIDE is somewhat novel but useless, it requires a high level of annotator focus and a considerable tolerance for errors, making it infeasible for long-term annotation tasks.
Dear reviewer, thank you for your response. We believe there is clearly a misunderstanding of our work.

>“I still think that the GUIDE method is difficult to apply to reinforcement learning agent that require long term training (e.g., a few days, atari/procegen/minecraft),”

>“I have some concerns about GUIDE’s usability for real-world tasks”

We find these points raised by the reviewer to be puzzling and self-conflicting.
As you have mentioned your concern for real-world applicability, but the environments mentioned by you (e.g., atari/procegen/minecraft) are not real-world tasks. In fact, our bowling task is one of the challenging ones in atari. Both of our find treasures and hide-and-seek are similar to minecraft sub-tasks. Given our setup with partial visual observation, long-horizon planning, and difficult-to-design reward functions without extra info, we believe our tasks are not toy examples. **As stated above, this is evidenced by our results in Fig. 5 where SOTA RL struggles to converge while GUIDE improves them by a large margin.** We believe that this shows the exact potential of GUIDE. It is infeasible to train regular RL on an unrealistic amount of experience, this is exactly the purpose of using real-time human guidance for accelerating agent learning.

As shown in Fig 5 of our paper, **untrained human participants using GUIDE are able to reach the same level of performance as the RL baseline within half the training time.** We believe these are encouraging results and it shows the potential of real-time human guidance.

>“sustained manual loads for a few days would still be excessively costly and inconsistent”

We agree that human feedback is expensive and effectiveness varies across individuals. However, we addressed both of these issues by introducing a simulated feedback provider and conducting the largest scale analysis existing on the effect of individual differences on AI guidance. This has not been done before.

>“and a few of the toy experiments in the paper don’t support the author’s point of view.”

We would like to emphasize that RL baselines still struggle on the tasks that the reviewer categorized as “toy examples.” Again, shown in Fig 5, GUIDE surpassed the baseline by a large margin. **Overall, we don’t consider devaluing our contribution simply based on a gap already pointed out by our experiments and us showing strong results in scalability to be appropriate for scientific advancements.** We do not feel the reviewer’s points directly relate to our work.
Thank you for your reply, I still think that the GUIDE method is difficult to apply to reinforcement learning agent that require long term training (e.g., a few days, atari/procegen/minecraft), and even if it could be used, sustained manual loads for a few days would still be excessively costly and inconsistent. I have some concerns about GUIDE's usability for real-world tasks, and a few of the toy experiments in the paper don't support the author's point of view. Thanks a lot, I will clearly keep my score.
We appreciate your positive feedback! Thank you!
I thank the authors for addressing my questions. I increased the confidence score to 4. 

Considering the overall contributions and limitations of the paper, I will keep my other ratings.
Dear Reviewer,

Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above.

We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts!

Best regards,

Authors
Dear Reviewer,

Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above.

We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts!

Best regards,

Authors
Dear Reviewer,

Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above.

We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts!

Best regards,

Authors
Dear Reviewer,

Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above.

We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts!

Best regards,

Authors