Learning from Active Human Involvement through Proxy Value Propagation
Authors: Zhenghao Peng
Wenjie Mo
Chenda Duan
Quanyi Li
Bolei Zhou
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Machine Learning
Human-in-the-loop Reinforcement Learning
Safety
Sample Efficiency
Reward-free
TL;DR: We propose a human-in-the-loop policy learning method called Proxy Value Propagation that aligns human preferences through active human involvement and achieves highly efficiency and safety, without offline pretraining and reward engineering.

Abstract: Learning from active human involvement enables the human subject to actively intervene and demonstrate to the AI agent during training. The interaction and corrective feedback from human brings safety and AI alignment to the learning process. In this work, we propose a new reward-free active human involvement method called Proxy Value Propagation for policy optimization. Our key insight is that a proxy value function can be designed to express human intents, wherein state- action pairs in the human demonstration are labeled with high values, while those agents’ actions that are intervened receive low values. Through the TD-learning framework, labeled values of demonstrated state-action pairs are further propagated to other unlabeled data generated from agents’ exploration. The proxy value function thus induces a policy that faithfully emulates human behaviors. Human- in-the-loop experiments show the generality and efficiency of our method. With minimal modification to existing reinforcement learning algorithms, our method can learn to solve continuous and discrete control tasks with various human control devices, including the challenging task of driving in Grand Theft Auto V. Demo video and code are available at: https://metadriverse.github.io/pvp.

Learning from Active Human Involvement through Proxy Value Propagation
Zhenghao Pengl, Wenjie Mo Chenda Duan Quanyi Lit, Bolei Zhou University of California, Los Angeles, University of Edinburgh
Abstract
Learning from active human involvement enables the human subject to actively intervene and demonstrate to the AI agent during training. The interaction and corrective feedback from human brings safety and Al alignment to the learning process. this work, we propose new reward-free active human involvement method called Proxy Value Propagation for policy optimization. Our key insight is that a proxy value function can be designed to express human intents, wherein state- action pairs in the human demonstration are labeled with high values, while those agents' actions that are intervened receive low values. Through the TD-leaming framework, labeled values of demonstrated state-action pairs are further propagated to other unlabeled data generated from agents' exploration. The proxy value function thus induces policy that faithfully emulates human behaviors. Human- in to-the-llop experiments show the generality and efficiency of our method. With minimal modification existing reinforcement learning algorithms, our method can learn to solve continuous and discrete control tasks with various human control devices, including the challenging task of driving in Grand Theft Auto V. Demo video and code are available at: htttp::/metadriai paarrse.gitiin ttub.io/pup 1o/pvp
Introduction
Reinforcement learning (RL) has been successfully applied in many domains, ranging from board game Go [46), strategy game StarCraft II [42], autonomous driving (18), and even nuclear fusion (7]. Existing RL methods assume the manually designed reward functions can fully express human intents and preferences. However, the resulting agents might exhibit biased, misguided, or undesired behaviors due to faulty reward functions (23 40 20). Moreover, the poor sample efficiency as well the safety concern due the trial-and-error exploration prevent the real-world deployment of RL.
couman-in-the-lowe methods are promising to achieve alignment, learning efficiency, and safety. Human-in-the-loop policy learning relies human subjects to oversee the learning process of the autonomous agents, thus it can better align the learned behaviors with the preferences of humans compared with handcrafted reward functions. Different forms of human involvement in human-in. the-loop policy learning have been studied over the years. Human subjects can advise actions upon the requests of the robots (28) or provide preference-bas based feedback to assess the relative value 34 These methods learn from passive human involvement, where the human subjects do not provide real-time feedback and intervention during data collection. For safety-critica, tasks such as autonomous driving, safety is undoubtedly the first priority in human preference and the passive involvement methods yield unbounded risks in such settings. An increasing body of works focuses on active human involvement, where human subjects actively intervene and provide demonstrations during the execution time 17 47 29 26) With online correction and demonstration from human subjects, Al alignment and training-time safety of the system can be substantially enhanced.
of the collected trajectories 53 5
38, 52 36
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this work, we focus on learning from active human involvement and develop a simple yet effective method that can turn a common malue-based RL method into a reward-free human-in -n-th---ooo method with minimal modification. Our key insight is that we can learn proxy value function from active human involvement, such that the proxy values encode human intents and guide policy learning to emulate human behaviors. Specifically, we propose the Proxy Value Propagation (PVP) method which labels high Q values to human actions and low Q values to agent actions that are intervened by the human subjects. The proxy values are then propagated to unlabeled state-action pairs in the agent's 's exploration through TD-learning. Value-based RL methods soon learn policies that align with human intents because of the value-maximization nature. Experiments show that PVP can be successfully applied to both continuous and discrete action spaces, and achieve higher learning efficiency compared to baselines in various tasks, including driving in Grand Theft Auto V (GTA V). is also compatible with different forms of human control devices, including gamepad, driving wheel, and keyboard. We summarize our main contributions as follows:
1) We propose a simple yet effective method, Proxy Value Propagation, that can be integrated into existing RL algorithms to learn from active human involvement. Our method is reward-free and can be generalized across various task settings and human control devices.
2) The experiments show that the proposed PVP method enables superior performance and high learning efficiency in various tasks from the MiniGrid, MetaDrive, CARLA, to GTA V environment. User study further shows that PVP achieves better performance and is more user-friendly compared to other human-in-ther -n-the------ooopp baselines.
2 Related Work
Al alignment is one of the major issues in learning trustworthy intelligent agents for real-world applications. It is difficult to represent various human preferences into a scalar reward function in existing Reinforcement Learning (RL) methods (4) 6. Meanwhile, the manually designed reward function, which might be misaligned with human preferences, often leads to undesired behaviors (23 20 As a promising complement to RL, Human-in-the-loop Learning (HL) can overcome costly reward engincering and convey human intents to the learning process directly through human vevolvement. Compared to imitation learning (IL) [14 91 where the agent learns directly from high-quality human demonstration, HL methods benefit from interactive human involvement and feedback during the training, mitigating the possible distributional shift that usually happens when learning from offline data (39.
Preference-based RL. A large body of work focuses on learning human preference via ranking pair of trajectories generated by the learning agent (5] 38 52 36 22 [1]. InstructGPT (34) aligns language models by first supervised learning in demonstration and then finetuning by the reward learned from human preference feedback. Preference learning can be applied to the tasks that human can not conduct, such as moving six-legged Ant robot by assigning exact torque at each joint [5]. For those tasks that human can demonstrate, these methods do not fully utilize real-time feedback from human subjects during agent-environment interaction.
HL with Passive Human Involvement. Different from preference based RL, human subjects can provide direct feedback to the learning agent during training through passive human involvement. Some works learn policy from human--rovidde evaluative feedback, a Boolean flagging correct or wrong actions 19
3 32). This is similar to the intervention in our framework. However, in 32). humans provide high-level instructions, e.g. pointing to the left/right, while in PVP humans provide intervention and low-level demonstrations. The other line of work allows the neural policy to operate the robot and the human subjects can provide demonstration upon the requests from the learning agents (28 30 16). The expert policy will intervene when uncertainty is huge, where the agent uncertainty is estimated by the variance of actions [30]. These methods reduce the cost of human resources but have potential risks to human subjects since they do not fully control the system. For example, when human subjects use these algorithms to train autopilot AI, they are exposed to significant risks if they are in self-driving cars due unpredictable agent behaviors.
Learning from Active Human Involvement. For safety-critical tasks such as autonomous driving, the safety of both the controlled vehicles and the human subjects is the top priority. There are many works that allow human subjects to proactively involve the agent-environment interactions based on their own judgment to ensure safety, which we call active human involvement. Human subjects

---

can terminate the episode if aecr--accientall situation happens and such intervention policy can be
learned [56
43
35
54 50]. Recent studies explore active human involvement methods through
intervention and demonstration in the human-agent shared autonomy [27 30 17
47
26 16, 54).
However, previous methods do not fully utilize the power of human involvement. COACH 27 treats human labels as indications of advantage instead of simply as reward. Compared to COACH, our method accepts not only the feedback (the intervention signal) but also the human demonstration. Our method does not consider the time delay of human subjects explicitly as COACH does. Interactive imitation learning method (HG-DAgger) [1 does not leverage data collected by agents, while Intervention Weighted Regression (IWR) (29 does not suppress undesired actions likely intervened by human. Meanwhile, Expert Intervention Learning (EIL) 47 and IWR [29] focus on optimizing actions step-wise without considering the temporal correlation between steps. These drawbacks harm learning efficiency and thus incur more human involvement. Moreover, previous methods lack experiments to demonstrate the generalizability different task settings and human control devices.
3 Problem Formulation
Policy learning aims at finding a policy to solve the sequential decision-making problem, which is usually modeled by Markov decision process (MDP). MDP is defined by the tuple M = (8, A. P, do) consisting of state space S, an action space A, state transition function P x A + S. a reward function r S x A + [Rminn Rmax]. a discount factor E (0, 1), and an initial state distribution do S + [0, 1]. The goal of conventional reinforcement learning is to learn novice policy (als)) S x A + [0, 1] that can maximize the expected cumulative return: Tn = arg max=, Er-P,, Ei=o r'r(st; at)], wherein
(So, Oo, 8r, ar) is the trajectory sampled from trajectory distribution Ps. induced by Tn. dg and P. Here T. defines a stochastic policy, while deterministic policy can be denoted as [s] + A and its action distribution is Dirac delta distribution T., (a|s) = 5(a - Mn (s)).
The reward function imposes an assumption that the reward can fully reflect the intentions of the users and incentivize desired behaviors. However, this assumption may not always hold and the learned agent may obtain biased behaviors or figure out the loophole to finish the task (23) 40). Revisiting the primal goal when developing learning systems, we find the reward is not a necessity since what we really want to achieve is the realization of human preference in the learned behaviors and, as suggested by 40), the ultimate source of information about human preferences are human behaviors.
Imitation Learning (IL) methods directly learn n,, from human behaviors. Assuming human expert has a human policy TT (ax]s) : S A + [o, 1J. which outputs human action dy E A. Note that human action shares the same action space as novice action. IL learns from the trajectories generated by human policy Th ~ Px., and optimizes the novice policy to close the gap between Tn ~ Pa.. and The Instead of generating incorporate a human subject into the loop of training for providing online data. This can mitigate the distributional shift since the data generated with that of the novice policy 39). This can be modeled by introducing an intervention policy I(-|s, a,)) to describe human subjects' intervention behaviors. In earlier methods such DAgger (39, the intervention policy is a Bernoulli distribution and the control authority switches back and forth between the novice and the expert. It is unrealistic invite real human subject to be involved in such training. Later studies allow the human subjects to intervene and take full control 49 43 26 (54) which we call such setting learning from active human involvement. During training. a human subject accompanies the novice policy and can intervene with the agent by taking over the control to demonstrate desired behaviors. The intervention policy can be considered as a deterministic policy denoted by [(s, an) S x A + {0, 1} where an (-Is) 3) is agent's 's action. With notations above, the behavior policy mh that generates actions during training is:
offfine dataset and training novice policy against it [14 91. we can
has closer state distribution to
(a(ss)) (1 I(,, (s. - -m,(s)) + I(s, ((s.
(1)
With such model of active human involvement, we can now formulate objectives.
Task-specified metrics. Our primal goal is to find novice agents whose behaviors are well-aligned with human preferences. In this work, we inform the human subjects of the primal goal of the tasks, e.g. navigating to the destination in driving tasks. They are also aware of how task-specified metrics, such success rate and route completion provided by the test environments, are computed.
3

---

These metrics serve as a proxy for human preferences in evaluating trained agents' performance. Unlike prior work where these metries were used as rewards, our learning agent cannot access them. The only supervision sources in our method are human interventions, [(, a), and demonstrations, a₁ ~ TT (-1s).
Preference Alignment. In our method, humans can intervene at any time. Most interventions occur in near-accidental situations or when agents are performing poorly. Conversely, lack of intervention indicates alignment with human preferences. Hence, another goal is to develop a novice policy that minimizes human interventions during shared control. In the next section, we will discuss our insights and how we build a concise, general, and efficient learning method to achieve these objectives.
4 Method
We propose the Proxy Value Propagation (PVP) method which can transform a value-based RL method into an efficient reward-free -free human-in-the-loop policy optimization method that learns from active human involvement. PVP is compatible with various task settings, such as continuous and discrete action spaces, as well as various human control devices. In this section, we first summarize the basic workflow of value-based RL before introducing the motivation and the design of PVP. We then describe the implementation details.
The proposed human-in-the-loop method results from the minimum modification of existing reinforcement learning methods. Thus, we briefly introduce the background of related methods. Value-based RI optimizes the value function and policy iteratively. On the value function side, we denote the state-action value and state value of policy π as Q(s, a) = E [E t= 70r (se, 0z)] neural network is commonly used to estimate the value function with Bellman backup: Q(s, a) + r(s, a.) + ? max, Q(s', a'), where s' is the next state. To learn the value network Qo parameterized by 6, stochastic gradient descent on the temporal difference (TD) loss is conducted (1) ee)),.=') [Qa(s, a) (r(s, a) + max, Q3(s', a') d')) where Qa can be a delay-updated target network. In this work, we adopt the TD learning in the reward-free setting. Remove the reward in the TD loss, the TD loss becomes:
balue- RL:
and (s) [=-- a). respectively.
jiD (0) = (e,a,s))
Q0(s, a) - 7 max 2g(5",a')P
On the policy side, based on the learned value function, the deterministic policy nn parameterized by ø can be learned by maximizing the Q values: (()) = E. Q(s, (x(s The optimal policy is expected to maximize Q values:
(s()) arg max Q(s,a).
(3)
4.1 Proxy Value Propagation
We illustrate the active human involvement of PVP in Fig. During training, the human subject supervises the agent-environment interactions (Fig. A). Those exploratory transitions by the agent are stored in the Novice Buffer B, {(s, an 8 )) At any time, the human subject can intervene the free exploration of the agent by pressing a button in the control device (Fig. | B). While pressing the button, the human takes over the control and provides demonstration of how to behave. During human involvement, both human and novice actions will be recorded into the Human Buffer B, {(s, on, an, 8')). Concurrently with the human-agent shared control, our method keeps updating the novice policy by the novel Proxy Value Propagation mechanism (Fig. TC), which will be discussed later.
In the shared human-agent agent control, human intervention serves as a distinct indicator of suboptimal agent performance, which could result from the agent executing perilous actions exhibiting ineffective behaviors. Thus, the optimal policy learned by the agent should (1) strive to approximate the behaviors demonstrated by the human subjects and (2) avoid performing actions that intervened by humans.
The key insight of this work is that we can manipulate the Q values to induce desired behaviors, given that value-based RL. has the nature to seek maximmiaxiiizing policy as Eq. [3] As shown in Fig. [ C, for emulating human behavior and minimizing intervention, we sample data (s, any ag) from the

---

Agant's exploratory trajectory
Humen-iwwtved trajectory
Human action dg
O O
Novea
as, x')
(A) Human oversees agent's exploration
Agent action
Q(s,as) + +1
Q(+,a) >Q(''aa)
Recorr dp
Pi
Q(a, an) +
Temporal Loss
Gas s')
Proxy Value Loss
(B) Human intervenes and provides demonstration
(C) Label proxy values and propagate through buffers
Figure I: Illustration of Proxy Value Propagation. (A) Human oversees the agent's 's trial-and-error exploration with the environment. When the human subject does not intervene, the transitions will be recorded into the novice buffer B,, (B) When the human intervenes, both novice action an and human action ag will be recorded into the human buffer B, but only the human action will be applied the environment. (C) In training, we use the human buffer to compute proxy value loss and propagate the human intent knowledge all transitions via TD loss without access to the reward.
human buffer and label the Q value of the human action an with +1 and the novice action on with -1. This is achieved by fitting the network directly with PV loss:
(P)
E (s, an) - 11² + (Qa(s, an) + (s),))))
(4)
(s.an,an)
The transitions in the novice buffer are not intervened by the human subject, meaning they are aligned with human preferences. Mcanwhile, those transitions also contain information of the forward dynamics [24, (55). To exploit the information contained in these transitions, instead of discarding these data as in 17.. we propagate the proxy values to these states via TD learning in Eq. and use those transitions together with those human-involee transitions for the policy learning. The final value loss is evaluated as follows:
((Ø) = PP (0) + (8) (8,a aa
E
11Qo(s, an) 1 + 1Qo(s, a,) + 1{)
JBBB
(5)
1Qe(s, a) 7 max Qg(s', e²)
(,. )BB U
Then follow the policy update process outlined in the base RL methods.
4.2 Analysis
Connection to CQL. The proposed PVP method can be interpreted as adopting the Conservative Q-Learning (CQL) [21] objective for reward-free free and online learning settings. It augments the CQL objective with an extra L2 regularization term imposed on the Q-values for human-involved transitions. In our online learning setting, Eq, 5 can be reformulated as:
(0) E
(2 an) + (=, +2 + 2(Qa(s, an) - Qo(s, an)) + TD loss.
(6)
1.2 Regalarization on Q
CQL Loss
CQL was originally proposed to mitigate the problem of overestimated Q-values in offline RL settings. These overestimations often lead to suboptimal policies due the optimistic selection of actions with misleadingly high values. In our work, we deal with human actions and novice actions sampled from two different distributions, where overestimation might also occur. However, unlike CQL, PVP does not have access to a reward function, meaning the Q-values are not grounded in an estimation of true values. The additional L.2 regularizer therefore serves to impose constraints on the Q-values, helping to prevent unbounded growth and potential overfitting. In Sec. 5.4. we compare the learned proxy Q-values under both CQL and PVP objectives. Our results indicate that human and agent actions are more distinguishable when learned through PVP.
5

---

MetaDrive-Gamepad
CAReA-Wheel
GTA V-Keyboard
MinGGrid-Keybordd
1.0
PVP (Oumg)
10
PVP (Ours) TD3
1,
PVP (Ouns) DON
TD3
Roult Compeiton
Roult Comp
Success sate
0.6
9
0.2
0.2
0.2
0.0
Skips
145
Stops Tel
Stapo 1ee
Steps
1el
Figure 2: Evaluation of PVP under four different environments with human control devices, For each environment, we plot the test-time performance curve of the agent trained by the proposed PVP and the RL counterpart TD3. The x-coordinate is the total number of environment interactions, which indicates the time steps the training agent (in RL method) or the human agent system (in our method) experiences during training. Compared to the RI. counterpart, the proposed method achieves much higher performance with superior learning efficiency.
Alternative to Reward Assignment. On the other hand, a more straightforward idea than PVP is to assign reward of +I to human actions and to agent actions during intervention. Unfortunately, it is not practical since the Bellman backup is conducted on the transition triplet (s, a, s'), where one has to use future states' values estimate current values. Therefore, the reward must correspond the action from the behavior policy, the action a causes the transitions from s to s', In our context, during involvement, the action a₁ must come from human policy as the human subject taking control. Though we can assign +1 reward and compute value target in those human-involvinn to the agent's actions because we don't know the next states caused
transitions,
we have no way to assign by those actions and thus we can't 't compute the value target. It is also not practical to query the environment to get the next state s" ~ P(ss, a,,) as the a, is potentially danger or undesired action and replaying in the real-world environment is not feasible. In our preliminary experiment, we find the policy fails to learn anything regardless of the amount of human involvement provided. This is because the reward will be +1 for all the human- human-involving transitions and the learning agent will find a pitfall to maximize its rewards: it always demonstrates undesired behaviors so that humans will always take control, which yields a +1 reward.
4.3 Implementation Details
Base RL Methods. Our method can be implemented for both continuous and discrete action spaces by extending TD3 110 and DQN (31) with PV loss and the balanced buffer. While TD3 uses deterministic policy, DQN adopts epsilon-greeds exploration that makes the policy stochastic. We remove the action noise in DQN and simply follow the argmax rule to select actions. Therefore, our method enjoys deterministic novice policy in both cases. The primary reason is that according to the feedback of human subjects, stochastic novice makes human subjects experience excessive fatigue due to the difficulty in monitoring and correcting agents' noisy actions. This design choice makes our method more fser-friendly, shown in the user study in Sec. 5.3
Balanced Buffers. The intervention gradually becomes sparse as the agent learns to reduce human intervention. However, those sparse intervention signals contain even more important information on how to behave under critical situations. Previous method (26 stores agent data and human data into one buffer and samples them uniformly. Abusing the notations, the ratio between the transitions from the agent's exploration and from human involvement is [B,. |5,] in each SGD batch. The human demonstrations are overwhelmed by the amount of agent-generated trajectories,

---

Table I: Comparison of different approaches in Meybbarve-Keybbard The overall intervention rate is given besides the human data usage.
Training
Testing
Method
Human
Total
Total Safety Cost
Episodic Return
Episodic Safety Cost
Success Rate
Data
Data Usage
Usage
SAC PPO
IM
2.76K +
386.77 =35.1
0.73 11:8
0.82 10.18
IM
24.34K
13356K
335.39
1:2.41
3.41 a1.11
0.69
TD3
IM
1.74K
0.62K
318.12
#21.9
0.47 20.08
0.70 4:009
SAC-Lag PPO-Lag
IM
1.84K
+ 0
351.96
1101.8
0.72 #0.49
0.73 20:00
IM
11.64K + 4666
299.99
499,46
1.18 1083
0.51 1:017
CPO
IM
4.36K 2222K
194.06 1008
1.71 1.00
0.21 1:000
Human Demo.
30K
39
347.523
0.39
0.97
BC
30K (1.0)
113.32 100.21
2.171
20:5
0.073 1000
GAIL
30K (0.015)
2M
25.90K + 8.5K
81,51 9.43
1.308 0.23
0.0 00
HGG-Dagerr
39.0K (0.76)
51K
56
116.393
1.979
0.045
IWR
35.8K (0.79)
45K
52
226.221
1,457
0,465
HACO
19.2K (0.48)
40K
130
143.287
1.645
0.139
PVP w/o TD
13.5K (0.34)
40.5K
70
252.447
1,277
0.220
PVP Reward
12.8K (0.32)
40K
30
319.383
0.767
0.755
PVP (Ours)
14.6K (0.37)
40K
76.8 =3.3
353.636 #23.7
0.898 20.15
0.857 2004
leading to inefficient learning of those critical human behaviors and even catastrophic forgetting. For example, the driving policy sometimes fails to master acceleration at the beginning of an episode, though the human subject has already demonstrated the expected maneuver multiple times. This is because the demonstration of initial acceleration only lasts a short period of time and thus scarce in the buffer. To address this issue, balance the transitions coming from the human buffer and the novice buffer. In each training iteration, we sample two equally-sized batches b,, and by from B, and B, respectively, each has N/2 samples. N is the batch size for the policy update. By concatenating 6n and bR. our method can balance the data from the human's 's demonstration and from the agent's 's exploration, and hence the ratio between two types of data in each SGD batch keeps 1: 1. Therefore, in the initial acceleration example above, the balanced buffer recalls the acceleration behavior, preventing catastrophic forgetting.
5 Experiments
5.1
Experimental Setting
Tasks. We conduct experiments on various control tasks with different observation and action spaces. For continuous action space, we use three driving environments, MetaDrive safety benchmark (25]. CARLA Town01 [ and customized driving environment built upon Grand Theft Auto V (GTA V), popular video game. In these tasks, the agent needs to steer the target vehicle with low- level acceleration, braking, and steering, to reach its destination. Specifically, in MetaDrive safety environments, the agent needs avoid any crash the heavy-trafits scene with normal vehicles, obstacles, and parked vehicles. In MetaDrive, there exists a split of training and test environments, and we present the performance the learned agent in a held-out test environment. To examine our method with different observation modalities, and the bird-eye view image in CARLA as observation. For discrete action space, we use MiniGrid Two Room task [4, which involves agent exploration such moving toward a door and opening the door before reaching the destination. The observation MiniGrid the semantic map of the agent's s local neighborhood. Please refer Appendix E for more information about the environment setup.
use the sensory state vector in MetaDrive and GTA
Evaluation Metries. In MetaDrive safety benchmark, we report total safety cost as number of crashes during training, which reflects the number of potential dangers exposed to the human subject during training. We also report episodic return, episodic safety cost, and success rate as the test performance of the agents. Episodic safety cost is the average number of crashes in one
7

---

HACO
generated by HACO and PVP agents the Table 2:
Results of different approaches in
CARLA.
Human Data
Total Data
Route Complction
Success Rate
Method
PVP
PPO
IM
0.24 +
0.0 + 0.0
TD3
1M
0.11 + 0.0
BC
5K
0.42 0.8
0.20 + 0.10
Aggnnts
HG-DAgger
6.8K
24K
0.64
0.47
Figure 3: We visualize the action sequences
IWR
5.7K
24K
0.69
0.60
HACO
4.8K
24K
0.52
0,40
same MetaDrive map who are trained 40K steps, PVP has much smoother actions.
PVP (Ours)
6.6K
24K
0.92 1: 0.05
0.73 +: 0.08
episode. The success rate is the ratio of episodes in which agents reach the destination to the total test episodes. CARLA, we report route completion and success rate. Route completion the ratio of the traveled distance to the length of the complete route. GTA V uses route completion and MiniGrid uses success rate to measure the performance. Except for total safety cost, the aforementioned metries measure the test-time performance, which is tested when the agent runs independently without human involvement. For coman-in-the-loee
experiments, we also report the total number of human- human-involved involved transitions (human data usage) and the overall intervention rate, which is the ratio of human data usage to total data usage. These show how much effort humans make to teach the agents. We also design user study to measure the experience of human subjects in Sec. 5.3
Human Interfaces. To examine the generalizability of our method, we leverage multiple control devices; Xbox Wireless Controller (Gamepad), keyboard, and Logitech G29 Racing Wheel. We denote the MetaDrive tasks with three devices as MetaDrive- MetaaDive-Gameppd//Ke Haameaad/Keyborreee
As shown in Fig. 2 human subjects can takeover through control devices and monitor the training process through the visualization of environments on the screen. The Ethics statement is provided in Appendix A
Experimental Details. We implement most of the code with Stable-Baseling [37. Training results of various baselines in MetaDrive tasks are obtained from the open-source code by [26). The RL baselines are repeated 5 times with different random seeds, while other human-in-the-loog methods repeated fewer times due to limited human resources. In the training of the human-in-the-loog methods, a real human subject participates in each experiment and we do not use any simulated user input. During testing, there no form of human involvement. For each experiment, we evaluate each checkpoint in the environment for multiple runs and report the average task-specified metrics as the performance of this checkpoint. We report the performance of the best checkpoint as the result of the experiment. We provide the standard deviation if the experiments are repeated multiple runs in tables and figures. All experiments with humans are conducted on a local computer with an Nvidia GeForce RTX 3080. The local computer can support real-time simulation and training. Hyper-paramen partmmerrameess and other details are given in Appendix and G
Baselines. We test four native RL baselines: PPO 45 SAC 13, TD3 10 and DQN 31 We also test three safe RL baselines: Constraint Policy Optimization (CPO) PPO-Lagrangian [8 SAC-Lagrangian 112 In all baselines above, the reward function and cost function (for MetaDrive Safety Benchmark) are defined by the environment and can be accessed by the agents. We also test IL methods Behavior Cloning (BC) and GAIL [14].
-itttii--tt--ooppp methods that learn from active human involvement are tested: Human-Gart DAgger (HG-DAgger) [17), Intervention Weighted Regression (IWR) 29 and Human-Al Copilot Optimization (HACO) 26
5.2 Baseline Comparison
Comparing with RL Counterparts. Fig. shows the curves of test-time performance. In MetaDrive- Gamepad, our method achieves 350 returns in 37K steps. This takes about one hour in the real- world HI experiment. TD3 baseline fails to achieve comparable results even after 300K steps of training. In CARLA, PVP agents learn to drive within 30 minutes with our method, while TD3 cannot solve the task. In GTA PVP can solve the task with 1.2K human data usage and 20K total data usage. The whole experiment takes only 16 minutes. TD3 instead utilizes 300K steps to achieve similar
8

---

CCL
Table 3: User study result. The maximum score for each item is 5.
HG-DAgger
IWR
HACO
PVP
Compliance
3.0 0.
4.0+ 0s
3,0
4,8
400
0.5
Performance
2.2 1.0
3.7 4
+ 0.9
4.8 0.5
Stress
3.2 0.9
4.5 205
2.3 209
4.7
= 0.6
Figure 4: Evolution of proxy values.
Stass
performance. In MiniGrid tasks, our method successfully solves the tasks while vanilla DQN fails, showing that PVP can learn exploratory solution and can be incorporated into discrete action space. We also show experiments on one casier and one harder MiniGrid environment in Appendix F.2 where PVP greatly improves learning efficiency.
Comparing with cumaa--in-theloppe Baselines. Table Suggests all tested HL methods achieve extremely low safety violations in training compared to vanilla RL and Safe RL methods, empirically supporting the preference alignment of the active human involvement, we consider human prefer- ence is to avoid safety violation. Compared to other numan-in-the--e the lowest human efforts in terms of human data usage and overall intervention rate, while greatly outperforming baselines in testing performance. Since MetaDrive has training and test environment split, the result suggests PVP can learn high-quality agents with generalizability. Similar results are shown in CARLA in Table 2 Compared RL baselines, HL methods achieve decent success rates and route completion rates even with only 24K environmental interactions. Compared to HL baselines, PVP achieves the best route completion rate.
methods, method costs
Visualization. In Fig. we visualize the action sequences of the agents trained by PVP and a human-agent shared control baseline HACO [26). The angle and length of each arrow represent the steering and acceleration, respectively. The human subject's s actions are marked with yellow. Compared to HACO method, PVP agent produces smoother actions, which explains its high user study scores shown in the next section.
5.3 User Study
We design user study questionnaire to assess the experience of human subjects. Details provided in the human subject research protocol in Appendix B Three aspects are considered: (I) Compliance measures whether the behaviors of the agent satisfy human intents. For example, a highly compliant agent behaves like human such that the human subjects feel like they are completing objectives by themselves. (2) Performance is the subjective evaluation from human subjects whether the agent can solve the primal task, e.g. driving to the destination in navigation tasks. This score should be low if the agent cannot learn partieular behavior forgets it even though human subjects have taught the agent multiple times. (3) Stress gauges the cognitive cost human subjects pay to train the agent. A typical source of stress is the annoying oscillation and jitter the agent demonstrates. Unexpected behavior that requires human's 's instant reaction also creates stress. A lower score means more stress.
Table shows method is the most usir-lriendly method. On the one hand, we use deterministic novice policy that greatly alleviates the jitter and unexpected behaviors, reducing stress, On the other hand, our method masters human behaviors and suppresses undesired actions with the balanced buffer and proxy value, improving the user experience in compliance and performance.
5.4 Ablation Studies
"PVP w/o TD", disabling TD learning via setting (Q) = 0 significantly damages the performance of PVP, suggesting that propagating information from human- involved states to other states is critical to the success of PVP.
TD learning: As shown in Table
PVP with reward: Both MetaDrive and CARLA results in Table D and show that adding the environmental reward doesn't 't bring significant improvement in the learning performance, which might be caused by the fact that the native reward function might not be aligned with human preference.

---

Balanced buffer: We find that disabling balanced buffers (PVP w/o BB) makes the training unsta- ble and leads to poor performance. This design avoids the catastrophic forgetting when the agent- generated data overwhelms the human demonstra- tions as in HACO [26].
Table 4: Ablation studies in CARLA.
Human Data
Route Completion
Success
Method
Rate
HACO
4.8K
0.52
0.40
HACO w/o
5.1K
0.49
0.20
Novice buffer: We find that PVP without the Novice buffer (PVP w/o NB) yields poor perfor- mance. The agent data stored in the novice buffer contains information on human preference and the forward dynamics of the environment. Thus, PVP does not discard the agent exploratory data as opposed to HG-DAgger [17.
PVP w/o BB
2.8K
0.62
0.33
PVP w/o NB
4.2K
0.708
0.33
PVP w/ Rew.
4.4K
0.793
0.467
PVP w/ SP
12.3K
0.40
0.20
PVP w/ CQL PVP (Ours)
8,0K
0,622
0.266
6.6K
0.92 + 0.08
0.73 0.88
We implement PVP based on Soft Actor-critic [3 so that the novice policy is now a stochastic policy. As shown in the "PVP w/ SP" in Table 4 introducing randomness in novice actions greatly reduces the performance. The human subjects report that the novice agents with stochastic policy oscillate frequently, making it hard to respond when the agents suddenly drive toward the side road. HACO 26 has similar human-Al shared control as PVP, but it adopts a stochastic policy. For comparison, we also implement HACO without a stochastic policy. "HACO w/o SP" suggests deterministic policy can not bring significant improvement to HACO.
Stochastic policy:
Regularization on Q values: As discussed in Sec. 4.2 PVP objective can be interpreted as CQL with newly introduced 1.2 regularization term the Q values. We conduct the experiment evaluate the performance of the vanilla CQL objective with other PVP designs in our reward-free online learning settings. As shown in "PVP w/ CQL" in Table 4 CQL objective yields worse performance. This experiment shows that the vanilla CQL doesn't work in this human active involvement setting. As shown in Fig. + the proxy value in the vanilla CQL method has a much larger magnitude which makes the values of behavior actions (the actions applied to the environment) and agent actions hard to distinguish. PVP has smoother proxy values with a clear margin between behavior and novice Q. CQL does not set a bound for the proxy value, thus proxy values in those extreme human actions are reinforced without a bound, making the novice policy rapidly learn those extreme actions, whereas PVP has bounded proxy values, leading to more stable training and better overall performance.
6 Conclusion
Learning through active human involvement is promising approach enabling safe and efficient policy learning. In this work, we propose Prexy Value Propugation (PVP) that can effectively learn from the intervention and the corrective feedback from active human involvement. PVP can be seamlessly integrated into existing value-based RL methods and achieves highly efficient reward-free policy learning, without offline pretraining and reward engineering. show the proposed method achieves superior performance and better user experience across diverse environments with different action spaces and human control devices, showing that the learning from active human involvement is a efficient policy learning method aligning human preference.
couman-in-thh-llee
experiments
Limitations. (1) We only apply our method to two value-based RL methods. Advanced techniques such as exploration encouraging 3 and prioritized replay buffer 44 can added to further improve the result. (2) Our method is not applicable to tasks where humans can not provide demonstrations. (3) We assume that human always demonstrates desired actions. We will show in Appendix D that suboptimal human behaviors will damage learning. In this case, can define sparse cost function in the training environment and utilize constrained optimization [2] to penalize bad demonstrations. (4) We assume that human subjects are available and attentive throughout the entire training. While our method is proven to be effective even under heavy traffic environments, we plan to further enhance its sample efficiency. We will achieve this goal by conducting offline RL training and policy evaluation in the background or passively involving human subjects whenever the model is uncertain about the environment.
Acknowledgment: This work was supported by the National Science Foundation under Grant No. 2235012. The human experiment in this study is approved through the IRB#23-000116 at UCLA.
10

---

References
[1] David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans. Agent-agnostic human-in-the-loyy reinforcement learning. ArXiv preprint, abs/1701 .04079, 2017.
[2] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning. ICMI 7017, Sydney, NSW, Australia, 6-11 August 2017, volume of Proceedings of Machine Learning Research, pages 22-31. PMLR, 2017.
[3] Carlos Celemin and Javier Ruiz-del Solar. An interactive framework for learning continuous actions policies based on corrective feedback. Journal of Intelligent Robotic Systems, 95(1):77 97, 2019.
[4] Maxime Shevallir-Boosvery, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. atttp://ggite
com/ htt///xxxmmmbbec gym- minigrid 2018.
[5] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isahelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neurul Information Processing Systems 2017. December 4-9, 2017, Long Beach, CA. USA. pages 4299-4307, 2017.
[6] Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins. Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.086300 2020.
[7] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmalcki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897)4444-11, 2022.
[8] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1-16, 2017.
[9] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. International Conference on Learning Representations, 2018.
[10] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Jennifer G. Dy and Andreas Krause, editors, Proceedings of 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1582-1591. PMLR, 2018.
[11] Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang. and Subbarao Kambhampati. Widening the pipeline in huided reinforcement learning with explanation and context-aware data augmentation. Advances in Neural Information Processing Systems, 34, 2021.
[12] Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort, 2020.
[13] Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic critic: Off Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, ICMI. 2018, Stockholmsmäison Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR. 2018
[14] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gamett, editors, Advances in Neural In- formation Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4565 4573, 2016.
[15] Braden Hurl, Krzysztof Czarnecki, and Steven L. Waslander. Precise synthetic image and lidar (presil) dataset for autonomous vehicle perception. CoRR, abs/1905.00160. 2019.
[16] Ananth Jonnavittula and Dylan P Losey. Learning to share autonomy across repeated interaction. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1851-1858. IEEE, 2021.
[17] Michael Kelly, Chelsea Sidrane, Katherine Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083. IEEE, 2019.
and Mykel Kochenderfer.. Hg-dagger:

---

[18] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda. John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning drive in a day. In 2019 International Conference on Robotics and Automation (ICRA), pages 0248-8254, IEEE, 2019.
[19] W Bradley Knox and Peter Stone. Reinforcement learning from human reward: Discounting in episodic tasks. 2012 IEFE RO-MAN: The 2 IEEF international symposium on robot and human interactive communication, pages 878-885. IEEE, 2012.
[20] Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingenuity, DeepMind Blog, 2020.
[21] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement lcarning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balean, and Hsuan-Tien Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[22] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2100.05091, 2021.
[23] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint 1010011111.0787,,22 2018.
[24] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv preprint, abs/2005. 0bs/2005.01643,, 2020.
[25] Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 2022.
[26] Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. International Conference on Learning Representations, 2022.
[27] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang. David L Roberts, Matthew E Taylor, and Michael Littman. Interactive learning from policy-dependen human feedback. In International conference on machine learning, pages 2285-2294. PMI.R, 2017.
[28] Travis Mandel, Yun-En Liu, Emma Brunskill, and Zoran Popovic. Where to add actions human-in- the-loop reinforcement learning. In Satinder P. Singh and Shaul Markovitch, editors, Proceedingy of the Thirty- First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 2322 2328 AAAI Press, 2017.
[29] Ajay Mandlekar, Danfei Xu, Roberto Martin-Martin Yuke 7hu, Li Fei-Fei, and Silvio Savarese. Human- in-the-loop imitation learning using remote teleoperation. ArXiv preprint, abs/2012.06733, 2020.
[30] Kunal Menda, Katherine Briggs-Campher and Mykel J Kochenderfer. Ensembledagger: A bayesian approach safe imitation learning. In 2019 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 5041-5048. IEEF, 2019.
[31] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness. Mare G Bellemare. Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-55 533, 2015.
[32] Anis Najar, Olivier Sigaud, and Mohamed Chetouani. Interactively shaping robot behaviour with unlabeled human instructions. Autonomous Agents and Multi-Agent Systems, 34(2):1-35, 2020.
[33] lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4026-4034, 2016.
[34] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint Xr:3:2203.021 155, 2022.
[35] Erfan Pakdamanian, Shili Sheng, Sonia Bace. Scongkook Heo. Sarit Kraus, and Lu Feng. Deeptake: Prediction of driver takeover behavior using multimodal data. In Proceedings of the 2021 CHI Conference Human Factors in Computing Systems, pages 1-14, 2021.
12

---

[36] Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi. and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. In Roboticx: Science and Systems, 2019.
[37] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-haselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 8, 2021.
[38] Siddharth Reddy, Anca D Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. Robotics: Science and Systems, 2018.
[39] Stéphane Ross and Drew Bagnell, Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statisticx, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[40] Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019.
[41] Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-basee learning of reward functions. UC Berkeley, 2017.
[42] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. ArXiv preprint, 0901190220443, 2019.
[43] William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. In Proceedings of 17th International Conference on Autonomous Agents and MultiAgent Systems, pages Agents and Multiagent Systems, 2018
International Foundation for Autonomous
[44] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint / 2015.
[45] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. ArXiv preprint,
[46] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser. Ioannis Antonoglou. Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[47] Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew Schmiule, Mung Chiang, Peter Ra- madge, and Siddhartha Srinivasa. Learning from interventions. Robotics: Science and Systems (RSS), 2020.
[48] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID lagrangian methods. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings Machine Learning Research, pages 9133-9143. PMLR, 2020.
[49] Fan Wang, Zhou, Chen, Tingxiang Fan, Zhang, Jiangyong Li, Hao Tian, and Jia Pan. Intervention aided reinforcement learning for safe and practical policy optimization in navigation. In Conference on Robot Learning, pages 410-44 421. PMLR, 2018.
[50] Zizhao Wang, Xuesu Xiao, Bo Liu, Garrett Warnell, and Peter Stone Appli: Adaptive planner parameter learning from interventions. In 2021 IEEE international conference on robotics and automation (ICRA), pages 6079 6085. IEEE, 2021.
[51] Zizhao Wang, Xuesu Xiao, Garrett Warnell, and Peter Stone. Apple: Adaptive planner parameter learning from evaluative feedback. IEEE Robotics and Automation Letters, 6(4):7744 7749, 2021.
[52] Garrett Warnell, Nicholas R. Waytowich, Vernon Lawhern, and Peter Stone. Deep TAMER: interactive agent shaping in high-dimensional state spaces. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors. Proceedings of the Shcrty-Second/a AAAI Conference Artificial Intelligence. (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18) 18), and the 8th AAAI Symposiun on Educational Advances Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1545-1554. AAAI Press, 2018
[53] Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1 46, 2017.
13

---

[54] Yunkun Xu. Zhenyu Liu, Guifang Duan, Jiangcheng Zhu, Xiaolong Bai, and Jianrong Tan. Look before you leap: Safe model-based reinforcement learning with human intervention. In Conference Robot Learning, pages 332-341. PMLR, 2022.
[55] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How leverage unlabeled data in offline reinforcement learning. arXiv preprint arXiv:2202.01741, 2022.
[56] Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end simulated driving. In Proceedings of AAAI Conference Artificial Intelligence, volume 31, 2017.
14

---

A Ethics Statement
Human subjects get paid to participate in the experiments. They can pause or stop the experiment if any discomfort happens. No human subjects are injured because all tasks test in virtual simulation. Each experiment will not last longer than one hour and subjects will rest at least three hours after one experiment. During training and data processing, no personal information is revealed in the collected dataset or the trained agents. We have obtained IRB approval to conduct this project.
B Human Subject Research Protocol
Recruiting and Requirement. For our study, we recruit 5 human subjects. All of them are college students and have the age from 20 to 30 years. Furthermore, every participant are required to have valid driver's license and have experience playing video game. Participation our study is entirely voluntary. We ensure transparency by informing all subjects about the nature of the experiments and how their demonstrations would be used. Every subject provide written consent, confirming they are fully aware and in agreement. Additionally, the study is conducted with the IRB approval.
Onboarding Period. Participants are required to undergo a practice session, during which they drive under complete control to get sense of the control devices (wheel, gamepad and keyboard), the environment interface, the dynamics of each environment and how an episode will fail or success. Each subject get familiar with all the control devices and all the environments, which is indicating by performing at least successful episodes, before they participate in the main experiments.
Main Experiment. During the initial stages of the formal experiment, subjects are advised to retain full control of the agent for the first one or two episodes. Subsequently. they may begin to let the agents taking control and intervene as necessary. The objective in all driving experiments is twofold: firstly, safely navigate the vehicle its designated destination, and secondly, to ensure the vehicle's operation aligns with traffic regulations and human preferences.
Subjects encouraged perform intervention whenever they perceive the vehicle might be in dangerous situation, in violation of traffic rules, or whatever scenario the human subjects feel they wouldn't behave in the way the novice policies do.
To ensure data integrity and counter potential proficiency biases, the order of experiments with different control devices, tasks and training algorithms is randomized for each subject. By doing this we mitigate the bias that a subject might become more familiar with the task when experimenting different algorithms.
User Study Questionnaire. We design a user study questionnaire assess the experience of human subjects The questionnaire is provided in Appendix B Three aspects are considered:
Compliance measures whether the behaviors of the agent satisfy human intents. For example, highly compliant agent behaves like human such that the human subjects feel like they are completing objectives by themselves.
Performance is the subjective evaluation from human subjects whether the agent solve the primal task, e.g. driving to the destination in navigation tasks. This score should be low if the agent cannot learn a particular behavior forget it even though human subjects have taught the agent multiple times.
Stress gauges the cognitive cost human subjects pay to train the agent. A typical source of stress is the annoying oscillation and jitter the agent demonstrates. Unexpected behavior that requires human's 's instant reaction also creates stress. A lower score means more stress.
The same questions repeated for each algorithm the human subject experimenting on.
15

---

Compliance: Generally, do you think the agent trained with this method complies with your intention? The higher score the better.
Examples:
(+) Good: Agent drives as you so that you don't even need to take over.
(-) Bad: Sudden unexpected behavior makes you mad.
Choices: 1, 2, 3, 5,
Performance: Do you think the agent trained with this method learns fast and performs well in terms of solving the task? The higher score the better.
Examples:
(+) Good: The agent learns fast so I don't need to take over too much in the later period.
(-)
Bad: The agent forgets what it learns have re-teach it.
(-) Bad: The agent never learns a specific behavior like accelcrating or turning even though I have taught many times.
Choices: 1, 2, 3, 4,5
Stress: Do you think training with this agent tired stressed? The lower score the more fatigue and stress. A higher score means you are more relaxed.
Tiredness might come from many sources: Oscillating trajectory, unexpected behaviors, degrading performance that you have to re-teach, etc.
is possible that your agent is not performing well but you don't feel tired training it. On the other hand, it is possible that your agent has good performance but still causes fatigue due to unexpected behaviors.
Choices: 1, 2, 4,
C Demo Video
Please find our demo video in the supplementary material. This video shows the footage of human experiments and the comparisons between agents learned by the baselines and the proposed method. The video contains three sections:
1. The first section shows how learn the driving policy in CARLA task within 20 minutes. We also compare the behavior of agents learned from PVP and TD3 baseline.
2. In the second section, we show the footage of MetaDrive human experiment where the human subject uses gamepad as the control device. We present the behavior comparison between PVP and TD3 baseline.
3. In the third section, show the applicability of method to other tasks. PVP performs well in GTA and can drive smoothly on the highway. In the discrete control tasks, the behavior comparison between PVP and DQN baseline in MiniGrid Empty Room and Four Room are provided.
D Preference Alignment
Here provide conceptual framework to describe the compliance of human intention. First, introduce a ground-truth indicator C:: x A + 10 of the intention violation, denoting whether the action is undesired. C revealed to the learning algorithm.
C(s.a) (1, if violates human intention
0 otherwise.
We will derive the upper bound of the discounted occurrence of intent violation, measure of training time human intent compliance:
S₂ = = = E
as)
(8)
TMPP.
where Pa, denotes the probability distribution of trajectories deduced by the behavior policy mb-
16

---

During training, human subject shares control with the learning agent. The agent's policy is deterministic policy (s). the human's 's policy is stochastic policy TH (a|s). The human subject intervenes (s,, a) = True under certain state and agent's action On- The mixed behavior policy that produces the real actions to the environment is denoted as:
Ta(a)s) (1 [s,, pm (+) - (n,( + [(s, (n()))T((((a)))))
where we use Dirac delta distribution represent the deterministie novice policy.
Two important assumptions on the human subject are introduced:
Assumption D.1 (Error rate of human policy). During human-Al shared control, the probability that the human subject produces undesired action is bounded by small value < 1:
E s (-15 (-1s)
Cls, a) S.
(10)
Assumption D.2 (Error rate of intervention policy). During Auman-Al shared control, the probability that the human subject does not intervene when the action is undesired bounded by a small value < 1:
E Pro
(1 (1=
< k.
(11)
We introduce the following theorem and give the proof as follows.
Theorem D.3 (Upper bound of intent violation). The discounted occurrence of intent violation S of the behavior policy na is bounded by the error rate of the human action c, the error rate of the human intervention and the intervention rate +/ = Es-Pa, I(s, an):
1 S., 1
(nc Ew..
(12)
Proof. Consider Eq. 9 we have:
E
a(s,a)= E {[1 Ms (s) + + ((s,)((s))
E
C(s,a)}
s aovoell)))
amoT)(-1s)
I k KE E (()) =
P
(13)
The upper bound of S..
Sn
Σ J'C(s,a S Σ (k+
(n + Eb)
(14)
TMPP
1
E Environment Details
Sacocoad Matan WORGEOOC Roam -000
C0S7-0
Stop (Prezz El:
Taleedver: frre
Takeovar Cost::0 2895-12911
Tota] 0
Figure 5: MetaDrive Safety benchmark.
Figure 6: CARLA Town01.
17

---

Torowrd 0.3 Gouta conedaatios Spead 6.6
Figure 7: GTA Training Environment.
Figure 8: MiniGrid (Four Room).
Table 5: Summary of the experiment environments.
Training & Test Split
Environment
Human Input Device
Ohservation Format
Action Space
MetaDrive
Gamepad, Keyboard, Wheel
State Vector
Continuous
CARLA
Wheel
Bird-eye View Image
Continuous
No
GTA V
Keyboard
State Vector
Continuous
Yes
MiniGrid
Keyboard
Semantic Map
Discrete
No
To avoid the potential risks of employing human subjects in physical experiments, we benchmark different approaches in four virtual simulated environments. We conduct experiments various tasks with different observation and action spaces and human input devices. Table Summaarzes the differences.
Continuous action space environments. For continuous action space, we use MetaDrive Safety Benchmark (25), CARLA Town01 environment g and customized policy learning environment built upon Grand Theft Auto (GTA V). In these tasks, the agent needs to steer the target car with low-level acceleration, brake and steering and move toward the destination.
MetaDrive Safety Benchmark preserves the capacity evaluate the safety and generalizability in unseen environments since it uses procedural generation to synthesize an unlimited number of driving maps for the split of training and test sets, which is useful benchmark the generalization capability of different approaches in the context of safe driving. We train agents in the training set, which contains 50 different scenes, and roll out the learning agents in the test set, which contains another 50 unique scenes. At each episode, the scene (road network) and the spawn location of traffic vehicles and ego vehicle are randomized. We use sensory state vector in MetaDrive as the observation for agents and thus apply MLP network architecture. When running pure RL methods in MetaDrive Safety Benchmark,
penalty will be added the reward when crash happens. This for fair comparison with the safe RI methods who have access to the cost function directly. Specifically, we follow the default reward scheme MetaDrive. The reward function in MctaDrive safety benchmark is composed of four parts follows:
R Cdisp Raisp Cspeed Repeed + Ceallision Realiision Rterm-
(15)
1. The displacement reward: Raiep d, de-1, wherein the d, and de- denotes the longitu- dinal movement in meters the target vehicle Frenet coordinates of the target trajectory between two consecutive time steps. If the agent drives in the wrong way then the displace- ment reward will be multiplied by 11 The displacement reward provides dense reward encourage the agent move forward. We set Cdisp 1.
2. The speed reward: allow speed in current road km / respectively. If the agent drives in wrong way then the speed reward will be multiplied by - 1. We set Cspeed = 0.1.
Repeed = Ut /maa. where Ut, Umar denotes current speed and maximum
3. The collision reward: Rostission = collision with vehicle, human, or object happens. Otherwise, it is O. The coefficient Coolitsion 5.
18

---

4. The terminal reward: Rterm is non-zero only at the last time step. At that step, we set Ruiisp R Repend = Reatlision and assign Ryerm according to the terminal state. Rierm is set to + 10 if the vehicle reaches the destination (successes) and -5 if the vehicle drives out of the road.
For measuring the safety, collision to vehicles, obstacles, sidewalk raises cost +1 at each time step. The sum of cost generated in one episode is the episodic cost.
In CARLA, we train and test agents in the Town01 environment. There exist many predefined routes in the town with different spawn locations and destinations. The length and spawn point of each route is randomized for each episode. We use the bird-eye view image in CARLA as observation and thus CNN is used as the feature extractor. In CARLA environment the reward function follows the reward function in MetaDrive safety benchmark.
In GTA V, we manually pick start and end coordinates in the world map to form multiple routes in different scenes. We split those scenes to the training and test sets. The training set contains two scenes with straight roads, turns, and medium traffic. The test set contains one different scene. For each episode, the traffic condition is randomized by the game engine. In GTA V environment the reward function follows the reward function in MetaDrive safety benchmark. The terminations include arriving at the destination (success), crashing with objects or vehicles for more than five frames (failure), and timeout (failure). The discrete keyboard input will be translated into continuous steering and acceleration signals for controls. Our customized human-in-the-lowe compatible policy learning environment builds upon GTA V with a full set of well-defined observation, reward and termination conditions. The environment will be open-sourced and available to the community |
For these tasks, the reward function is composed of two parts: a sparse termination reward (+10 (+10 when reaching the destination) and a dense moving reward (the distance moving toward the destination within one step).
Discrete action space environment. For discrete action space, we test on MiniGrid Two Room task 4 MiniGrid Two Room is task requiring heavy exploration since the agent needs to move toward door and open the door before reaching the destination. The spawn locations, the destinations, door locations and the geometry of each room are randomized. The observation of MiniGrid is the semantic map of agent's 's local neighborhood. MiniGrid tasks only support using the keyboard as the input device. Only in the MiniGrid task, we render the agent's action in the environment so that the human can decide whether to take over or return back based on both the current state and agent's ' action. But this is not feasible in other tasks since other tasks require real-time responses from humans and there is not enough time for humans to observe agent's actions even if we plot those actions in the visualization interface. Following the default reward function as in original repository, in MiniGrid environment a sparse reward is used. When the agent reach the goal, +1 reward is given and otherwise the reward is always 0.
Real-time experiment. Note that the local computer we use for which has an Nvidia GeForce RTX 3080 gpu, can support real-time simulation and training. In MetaDrive and CARLA, the physics simulation is run 10Hz in the virtual world and in GTA the frequency is 30Hz. That is, each environmental step will cause the virtual world advancing 0.1 / 0.033 seconds. After each environment interaction, PVP updates its policy once, inferring and back- propagating one SGD batch. Our experience suggests that the local computer can effortlessly support concurrent running of the simulation with human-agee shared control as well as the background policy update at the frequency in wall-time higher than the simulation frequency in the virtual world. That is, our training can run at frequency higher than 10Hz / 30Hz so that the time-clapse is actually faster in the real world than in the virtual world. We will limit the FPS to the system frequency so that the human subjects experience realistic time-elapse.
-n-the-oooo experiments,
Control devices. CARLA tasks use Wheel, GTA V tasks use a keyboard, and MiniGrid tasks use keyboard (provide discrete control signals). In all devices, a button is configured to indicate intervention. There is another button in the devices that activates an emergency stop. If any discomfort happens, human subjects can pause or stop the experiment immediately.
'The customized environment builds upon prior efforts on the communication between GTA V engine and Python interface: htttps://ggii http:///tthuuu.ooo/ttttrrrre DeeperGTAV https: //github/ com/aittorsip/VP1loo
https:///ithub. com/ gdpinchina/
19

---

Extra Experimental Results
F1 Impact of Control Devices
Table 6: The impact of different human input devices in MetaDrive benchmark.
Training
Testing
Input Device
Method
Human
Total
Total Safety Cost
Episodic Return
Episodic Safety Cost
Success Rate
Data Usage
Data Usage
HACO
21.2K (0.53)
40K
42
250.039
1.453
0.355
Wheel
PVP
10.3K (0.26)
40K
12
336.657
1.543
0.808
HACO
28.4K (0.71)
40K
55
71.37
1.97
0.0
Gamepad
PVP
7.4K (0.19)
40K
21
356.99
1.31
0.920
HACO
19.2K (0.48)
40K
130
143.28
1.645
0.139
Keyboard
PVP
14.6K (0.37)
40K
76
353.636
0.898
0.857
Table 6 presents the experiment results with different input devices in MetaDrive benchmark. In all settings, PVP agents outperform bascline method, showing the generalizability of PVP on different control devices. We observe that HACO [26] has performance discrepancy with different input devices. When using Gamepad, human subjects tend to push and pull the stick the limits, producing extreme values. Extreme actions are particularly harmful to previous method as it does not incorporate the regularization terms Q function bound the Q values. When using the keyboard, the human subjects press arrow keys to indicate increasin//decreasing current steering/acceleration values for an increment. Therefore there will be fewer extreme values happening than using Gamepad, which explains why the baseline HACO performs better with the keyboard compared to Gamepad. Due to less extreme values, when using Steering Wheel, HACO achieves good performance.
case study in a toy environment. We retrieve the agents stored during the training of HACO and PVP in CARLA task. We test them in a straight road in CARLA town and plot their actuating signals in Fig. 9 In this task, the steering should be always close to zero. However, we find that as the training iterations increase, the HACO agents gradually demonstrate unstable steering and their steering deviating. human-Al shared control, such unstable behaviors force human subjects to involve frequently. In contrast, the PVP policies learn much better solution in lane keeping.
Throttle
Steering
0.6
PVP
HACO
0.05
0.4
-0.05
PVP
HACO
0.0
0.5
1.0
1.5
2.0
0.5
1.0
2.0 164
Steps
Steps
Figure Control signals straight road in CARLA.
Visualization of action sequences in training. Fig. 10 we present the visualization of the trajectories during human-robot shared control. Comparing the visualization of HACO and PVP, we find that PVP generates smoother trajectories. Stable and smooth agent actions greatly improve human subjects' experience and relieve their stress during human-robot shared control. We can also find that as the training goes, PVP requires less human involvement. These results explain the performance of PVP and is aligned with the behavior shown in the supplementary video.
20

---

HACO @ 10K steps
PVP 10K steps
Human's Actions
Agent s Actions
HACO @ 20K steps
PVP @ 20K steps
HACO @ 30K steps
PVP @ 30K steps
HACO @ 40K steps
PVP 40K steps
Figure 10:
In MetaDrive task, we use the top-down view to plot the trajectories of human-agent shared control. We use dense arrows to represent the actions that are applied to the environments. The arrow starts at the position of the car at that time step and its direction is the steering angle, projected into ego car's local coordination. The length of the arrow represents the acceleration. We use green and yellow arrows to denote agent's 's actions and human's 's actions, respectively.
21

---

F.2
Extra Results in MiniGrid
1.0
1.0
PVP (Ours)
PVP (Ours)
Success Rate
Success Rate
0.4
0.4
0.
02
02
0.2
(Ours)
DON
0.0
0.0
0.0
0.5
Steps
Steps
Steps
(a) MiniGrid-Enm -eee---mptt--tieee (oandom-6x6-vi
(b) MiniGrid- TwoRooms-vo
(c)
Figure 11: MiniGrid results.
In Fig. II we present the extra results in two additional MiniGrid environments. PVP achieves superior performance compared to RL baseline. Note that use CNN without recurrent module as the feature extractor. The performance of PVP can be further improved if we utilize the neural architecture with memory capability.
G Hyper-parameters
In MetaDrive safety benchmark 25 task, the observation state vector. There exists split of training and test environments in MetaDrive. We present the result of the learned agent performing in the test environment.
In CARLA g the observation is the bird-eye view image in [84, 84, 5) shape, where 5 is the number of semantic channels. We train and evaluate the agents in the same NoCrashTTwwn01 environment.
In GTA V, the observation is state vector containing 2D LiDAR scanning for 240 total sampling points with max range 50m [15). vehicle state variables (speed, throttle, steering, heading), and navigation state variables (distance to road borders, distance to the next navigation point, collision to objects). There exists split of training and test environments in GTA V. We present the result of the learned agent performing in the test environment.
In MiniGrid tasks 4 N2-S4-vO (Two Room) and MuniGriddMulliRoom--ae top-down view semantic map in shape [7, 3,
Ranoom-6x6-60 (Empty Room), MiniGrid-MultiRoom.
(Four Room), the observation
In MetaDrive and GTA V, we use a MLP with two hidden layers, each has 256 units and ReLU activation, as the network architecture for the value network and policy network.
For CARLA task, since the input image has the same size of [84, 84] pixels, CNN architecture with [16, 32, 64, 128, 256] filters in each layer. The corresponding kernel-size is [[4, 4], [3, 3], [3, 31, [3, 3], [4, 4]], and strides [3, 2, 2, 2, 4]. We use ReLU activation functions between each layer.
the same 5-layers
For MiniGrid tasks, we use a 3-layer CNN architecture with [16, 16, 32] filters in each layer. All three layers have kernel-size and there max-pooling layer between the first two layers. We use ReLU as activation functions between each layer.
22

---

Table 7: PVP (MetaDrive)
Table 8: PVP (CARLA)
Hyper-parameter
Value
Hyper-parameter
Value
Discounted Factor for Target Network Update
0.99
Discounted Factor y for Target Network Update
0.99
0.005
0.005
Learning Rate
0.0001
Learning Rate
0.0001
Steps hefore Learning Start
100
Steps before Learning Start
100
Steps per Iteration
Steps per Iteration
Gradient Steps per Iteration
Gradient Steps per Iteration
Train Batch Size
100
Train Batch Size
128
Value Bound
Q Value Bound
Table 10: PVP (MiniGrid)
Hyper-paramettr
Value
Table PVP (GTA V)
Discounted Factor
0.99
Hyper-parameter
Value
T for Target Network Update
0.005
Discounted Factor y for Target Network Update
0.99
Learning Rate
0,0001
0.005
Steps before Learning Start
50
Learning Rate
0.0001
Steps per Iteration
Steps hefore Learning Start
100
Gradient Steps per Iteration
Steps per Iteration
Target Network Update Interval
Gradient Steps per Iteration
Train Batch Size
256
Train Batch Size
100
Q Value Bound
Value Bound
Exploration Reducing Fraction
Random Action Probability Initial Value
Random Action Probability Final Value
Table 11: HACO (MetaDrive)
Table 12: HACO (Carla)
Hyper-parane parameter
Value
parametr
Value
Discounted Factor
0,99
Discounted Factor for Target Network Update
0.99
for Target Network Update
0.005
0.005
Learning Rate Actor
0.0003
Learning Rate Actor
0.0003
Learning Rate Critic
0.0003
Learning Rate Critic
0.0003
Learning Rate Entropy
0.0003
Learning Rate Entropy
0.0003
Steps before Learning Start
100
Steps before Learning Start
100
Steps per Iteration
Steps per Iteration
Gradient Steps per Iteration
Gradient Steps per Iteration
Target Network Update Interval
Target Network Update Interval
Train Batch Size
128
Train Batch Size
128
CQL Loss Temperature
1.0
CQL Loss Temperature
1.0
Table 13: TD3 (MetaDrive)
Table TD3 (Carla)
parameearr
Value
Hyper papeer-arammttr
Value
Discounted Factor
0.99
Discounted Factor y for Target Network Update
0.99
for Target Network Update
0.005
0.005
Learning Rate
0.0001
Learning Rate
0.0001
Steps before Learning Start
10000
Steps before Learning Start
10000
Steps Iteration
Steps per Iteration
Gradient Steps per Iteration
Gradient Steps per Iteration
Train Batch Size
100
Train Batch Size
100
23

---

Table 16: DQN (MiniGrid)
Hyper-parameter
Value
Table 15: TD3 (GTA V)
Discounted Factor for Target Network Update
0.99
Hyper peremearrmettr
Value
0.005
Discounted Factor y
0.99
Learning Rate
0.0001
т Target Network Update
0.005
Steps before Learning Start
Learning Rate
0.0001
Steps per Iteration
Steps before Learning Start
10000
Gradient Steps per Iteration
32
Steps per Iteration
Target Network Update Interval
Gradient Steps per Iteration
Train Batch Size
256
Train Batch Size
100
Exploration Reducing Fraction
0.3
Random Action Probability Initial Value
Random Action Probability Final Value
0.05
24


Reviews:
[{"review_id": "t87pzoqT4G", "reviewer": "Reviewer_e73G", "summary": "The authors present Proxy Value Propagation for policy optimization. They model a proxy value function so that human intents receive high value (i.e. actions for which the human has inputted as corrective actions over a policy) while on-policy agents have low values for actions that caused an intervention. When optimized via TD-learning they are able to stitch together labeled values of demonstrated state-action pairs with unlabeled data from agent exploration (claimed). They execute the proposed algorithm on a number of environments (e.g. MiniGrid, MetaDrive, CARLA and the Grand Theft Auto 5 driving task) and show significant improvements over baselines.", "questions": "- A collection of human demonstrations is assumed (line 143), however it's still not overly clear to me how this is used. Maybe this is already clear in the text but which Q function is actually pre-trained with this data? Presumably there needs to be some overlap between the agent and human preferences initially otherwise every state would induce a human intervention? Maybe spill out this warm up procedure in more detail.\n\n- It's also not entirely clear under what conditions the human is told to intervene? In continuous and high rate action and state spaces (particularly in driving applications), there is not one specific action that causes failure. Are humans instructed to let the agent diverge from an optimal trajectory by a specific metric amount (e.g. half a lane in the case of self driving)? Or is it up to the human operator under what conditions would they intervene? Is the method sensitive to such human ambiguity?", "limitations": "First NeurIPS paper where I can comfortably say the limitations section is well done :-) I have nothing else to add.\n\nA societal impact statement doesn't seem necessary for this application / paper.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Human-in-the-loop (with active human involvement) is one of the few tried-and-tested methods to fine tune imitation learning / RL methods on complex real-world domains. Improvements in this field are actually quite impactful to practitioners deploying these systems in production (unlike some other methods that rarely see use outside of the academic setting). To this end the problem is well posed and of interest to the community.\n\n- PVP is a conceptually simple algorithm. I'm pretty sure this paper is self contained and I have everything I need to re implement it.\n\n- The workflow of the algorithm is also practical. I can imagine using this algorithm in a real system with the scale properties such a system would require.", "weaknesses": "- nit: \"It is also compatible with different forms of human control devices, including gamepad, driving wheel, and keyboard\" - True of baselines (and many other methods) as well? Or alternatively, this seems like an obvious point.\n\n- nit: line 128: \"It is unrealistic to invite a real human subject to involve in such training.\" There are many real-world uses of Dagger. Be careful with over generalization.\n\n- I'm not sure the claim that unlabeled (s, a) pairs are stitched together with human expert data is all that well experimentally justified. Or put another way, if the formulation has some equivalency to CQL (with an additional L2 regularization term), in the limit that these regularizes are extremely strong, the method will just look like behavioral cloning. To claim that there is value propagation to unlabeled states I would want to see experiments to confirm this (because in my experience CQL does not actually stitch together sub-optimal trajectories as is often claimed).\n\n- Figure 2 results are quite compelling. Although you're comparing a random-exploration based RL method (and pretty vanilla one at that) against a human-in-the-loop method. This is apples to oranges. I'd also be curious to see what performance looks like against state-of-the-art offline RL or supervised imitation learning in these domains (there's some of this in Table 1, but not in this earlier teaser image).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Please find our responses as follows.\n\n---\n\n### Weaknesses\n\n> **W1:** *nit: \"It (the proposed PVP method) is also compatible with different forms of human control devices, including gamepad, driving wheel, and keyboard\" - True of baselines (and many other methods) as well? Or alternatively, this seems like an obvious point.*\n\n**Response:** \n\nDifferent control devices correspond to different action spaces and different human interaction behaviors. For instance, the steering wheel enables a smoother continuous action space compared to the keyboard’s discrete action space by pressing certain keys. These variations pose a generalizability issue for many baseline algorithms. \n\nFor example, in Appendix F.1, we show that HACO struggles with the gamepad compared to the steering wheel. On the contrary, PVP demonstrates consistent performance across all tested control interfaces. This highlights PVP's potential for real-world applications where control methods can vary significantly.\n\n---\n\n> **W2:** *nit: line 128: \"It is unrealistic to invite a real human subject to be involved in such training.\" There are many real-world uses of Dagger. Be careful with overgeneralization.*\n\n**Response:**\n\nThanks for pointing this out. Real human experiments are expensive, but are not unrealistic. For example, HG-Dagger has been used to train policies in a real car. We will change the wording accordingly, from ‘unrealistic’ to ‘expensive’.\n\n---\n\n> **W3:** *I'm not sure the claim that unlabeled (s, a) pairs are stitched together with human expert data is all that well experimentally justified. Or put another way, if the formulation has some equivalency to CQL (with an additional L2 regularization term), in the limit that these regularizations are extremely strong, the method will just look like behavioral cloning. To claim that there is value propagation to unlabeled states I would want to see experiments to confirm this (because in my experience CQL does not actually stitch together sub-optimal trajectories as is often claimed).*\n\n**Response:**\n\nSection 5.4 (Table 1 \"PVP w/o TD\") presents an ablation study showing that removing the TD loss can greatly damage the learning performance. It gives evidence that \"there is value propagation to unlabeled states\" since if we only use the PVP loss, there will be no value propagation to unlabeled states.\n\n---\n\n### Questions\n\n> **Q1:** *A collection of human demonstrations is assumed (line 143), however it's still not overly clear to me how this is used. Maybe this is already clear in the text but which Q function is actually pre-trained with this data? Presumably there needs to be some overlap between the agent and human preferences initially otherwise every state would induce a human intervention? Maybe spill out this warm up procedure in more detail.*\n\n**Response:**\n\nWe don’t assume there exist collections of offline human demonstrations beforehand. Therefore, we don't conduct pretraining or warm-up based on held-out human demonstrations. \n\nAs clarified in the human subject research protocol (please see the attachment in the common rebuttal), human subjects will provide several episodes of demonstration with full control at the beginning. PVP is efficient enough to learn a rudimentary policy for humans to interact with  after a few minutes of training. This can be seen in the supplementary video.\n\n---\n\n> **Q2:** *It's also not entirely clear under what conditions the human is told to intervene? In continuous and high rate action and state spaces (particularly in driving applications), there is not one specific action that causes failure. Are humans instructed to let the agent diverge from an optimal trajectory by a specific metric amount (e.g. half a lane in the case of self driving)? Or is it up to the human operator under what conditions would they intervene?*\n\n**Response:**\n\nIt is up to the human operator under what conditions they would intervene. We provide a one-page PDF on human subject research protocol in the common rebuttal for more details. We will update Appendix B to include the newly added content.\n\nDuring the onboarding period, the participants will be briefed about the objectives of all driving tasks: to safely maneuver the vehicle to its designated destination while ensuring the vehicle's operation aligns with traffic regulations and human preferences. The participants will practice each task with different control devices.\n\nThe human subjects will perform intervention whenever they perceive the vehicle might be in a dangerous situation, in violation of traffic rules, or in whatever scenario they feel they wouldn't behave in the way the novice policies do.\n\n"}, {"review_id": "SQ1ehNJsz7", "reviewer": "Reviewer_C5fY", "summary": "This paper introduces a reward-free approach termed Proxy Value Propagation (PVP) to facilitate safe and human-aligned reinforcement learning (RL). PVP assigns high Q values to human actions (the state-action pairs in the human demonstration) and low Q values to agent actions that necessitate human intervention. The authors conduct experiments across diverse environments, offering a comparison with existing RL and human-in-the-loop (HL) techniques, as well as the results of user studies.", "questions": "See the revision recommendations in the \"Weaknesses\" section. The authors are encouraged to address the aforementioned concerns, provide more detailed experimental information, and incorporate these insights into subsequent revisions.", "limitations": "This paper partially discussed the limitations, but there may still be some ethical concerns. It is worth considering whether the proposed method could be potentially abused by malicious human subjects, leading the AI to learn in an incorrect manner.  Can malicious human preferences be detected in a timely manner, and the data be promptly removed?", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and readable, and provides detailed appendices and codes to supplement the paper's algorithms, experiments, and implementation details.\n\n2. This paper presents a reward-free approach to improve learning from human active involvement, effectively utilizing all data generated during the human intervention, including both agent and human-generated data.\n\n3. The proposed method can be extended to various value-based RL Methods.", "weaknesses": "The training process of PVP seems to rely heavily on human involvement, whether through observation or intervention, which can be quite costly, especially when scaling up to more complex scenarios. However, this may not be an issue if the cost is lower than designing safe rewards and the performance is superior to basic RL methods.\n\nNonetheless, the experimental results and settings in the paper leave me puzzled, and I am unable to determine if the results sufficiently support the conclusions.\n\n1. I am unsure if the experimental setup for Base RL Methods is fair. Why is the Episodic Return so high, but the Success Rate is lower than that of humans? For instance, GT Sophy [1], a race car AI trained using Deep RL by Wurman, Peter R., et al., has already surpassed human champions. Additionally, Imamura, Ryuji, et al. [2] mention that \"all agents learn the policy in approximately 400 epochs\" and \"its score places it among the top 10% approximately 28,000 human players.\" Can the authors provide more detailed descriptions of the rewards for Base RL Methods? Although the reward settings are briefly described in the appendix, the calculation methods and weights for each reward are not listed. Unreasonable rewards may affect the performance of Base RL Methods and the fairness of comparisons.\n\n2. PVP can be considered as providing dense training signals to the agent, as the agent's training relies on human judgment for every (s, a) pair. Even if the action is not intervened, it still depends on human judgment, thus providing a positive and accurate training signal, leading to faster convergence. For Base RL Methods, can designing more reasonable dense rewards and properly adjusting the weight of crash penalties achieve the same effect? Based on the evaluation metrics, human preference seems to be focused on reducing crash. Can manually designed rewards also express human intentions and preferences? The author should compare the cost of manually crafting the reward with the cost of human intervention, which can highlight the advantages of PVP.\n\n3. I also have doubts about the experimental results for BC and GAIL. What was their training data? Why do human demos have a 97% success rate, while BC and GAIL have success rates of less than 1%? Are there any errors in the experiments?\n\n4. I am also puzzled about the user study results in Table 3. According to the appendix, the scores can only be 1, 2, 3, 4, or 5, with an upper limit of 5 points, and PVP has an average score of 4.8. Why is there a standard deviation of 0.5 when the scores are so high? Are there cases with very low scores, and did the authors analyze these low-score cases in detail? Bad cases can be potential safety hazards.\n\n5. The statement about PVP with reward results is unevidenced: \"which might be caused by the fact that the native reward function might not be aligned with human preference.\" Success rate is a metric of agent capability, so why can't an agent's capability be improved if it is not aligned with human preferences? Is there evidence to prove that RL cannot surpass humans in this task [1, 2]?\n\n6. Additionally, the description of human subjects is insufficient. What is the proficiency of the human subjects in the environments? Were they all novices or professionals? Was a standard test specification or guide provided before the test to ensure consistency in test objectives?\n\n7. Have the authors compared the repetition rates of (s, a) and (s, an) in the Novice Buffer and Human Buffer? This might be closely related to the reliability of human subjects, and learning may not be meaningful for unreliable humans.\n\n8. How much variation is there in (s, an) among different human subjects? When participants have different styles, how should this be handled? For example, if some participants think the current (s, a) is reasonable, but others think it needs intervention, how should this sample be treated?\n\n[1]  Wurman P R, Barrett S, Kawamoto K, et al. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 2022\n\n[2] Imamura R, Seno T, Kawamoto K, et al. Expert Human-Level Driving in Gran Turismo Sport Using Deep Reinforcement Learning with Image-based Representation, 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n---\n### Weaknesses\n\n> **W1:** *I am unsure if the experimental setup for Base RL Methods is fair. Why is the Return so high, but the Success Rate is lower than humans? For instance, GT Sophy has already surpassed human champions. Can the authors provide more detailed descriptions of the rewards?*\n\n**Re:** \n\nFirst of all, we should emphasize that the essence of this work is to use human online interactions to replace the heavy reward engineering to achieve human-AI alignment. \n\nIt is true that through sophisticated reward engineering for a specific task, like what [A] did, the resulting agent can achieve outstanding performance, but it requires extensive domain knowledge and enormous amount of experimentation. Even with a well-defined reward function, the training safety and efficiency of RL remain to be solved.  We demonstrate the success of our method across different tasks with high sample efficiency and safety under human guidance, without even the access to the environment reward in training time.\n\n**The detailed descriptions of the rewards.** \n\nPlease refer to the one-page PDF for the detailed formulations of the rewards, which contains four parts: The displacement reward $R_{disp}$, a dense reward to encourage the agent to move toward destination; The speed reward $R_{speed}$; The collision reward $R_{collision}$; and The terminal reward $R_{term}$. This reward function is dense, considers collision penalty, and encourages reaching the destination.\n\n**Why do human subjects have a higher success rate but lower return?**\n\nAs you can see from our reward function, high return is possible when the RL agents drive faster but fail before reaching the destination. On the contrary, human subjects might not adopt such high velocity in order to gain more control and make sure the destination is arrived.\n\n**Why can't RL agents surpass human performance in our experiments?**\n\nWe do not have a heavily engineered reward function and do not fine tune the weights of different reward terms for different tasks like what [A] did, which is expansive.\n\nOn the other hand, imagining we can have a well-engineered reward function for each task that can potentially lead to a super-human policy, the challenges of training-time safety as well as the training efficiency need to be addressed. Our proposed work addresses all these challenges.\n\n---\n\n> **W2:** *For RL Methods, can designing more reasonable dense rewards and adjusting the weight of crash penalties achieve the human-like behaviors?*\n\n**Re:** \n\nEven though the reward function we used is much simpler than [A], the $R_{disp}$ is a dense reward for task completion and $R_{collision}$ considers the crash penalties and they form a reasonable reward function. It is possible that grid-searching all possible reward weights can lead to better agents. However, such sophisticated reward engineering might need to be performed for each new task that might cost huge computational resources. Meanwhile, the training-time safety and the training efficiency of RL algorithms remain to be solved. \n\nThis paper focuses on achieving training-time safety and training efficiency and also the important AI alignment problem, not only on getting a high performing policy through reward engineering.\n\n---\n\n> **W3:** *The statement about PVP with reward results is unevidenced. Is there evidence to prove that RL cannot surpass humans in this task [A, B]?*\n\n**Re:** \n\nThe reward function in [A] consists of 8 terms and the weights of these 8 terms are differently configured for different driving courses (Extended Data Table 1 in [A]). The well-defined reward function and the careful adjustment of reward weights, with other techniques in distributed training, creates the huge success of this Nature paper. Compared to [A], the reward function we used is much simpler and might not be capable of fully capturing human preferences. For example, human subjects usually have more fine grained behaviors such as decelerating before crossroad or maintaining a safe distance from the front vehicles. We believe adopting the task-specific reward function with sufficient weights tuning can let the RL agent surpass humans in our tasks.\n\nWe want to point out that, even though we believe reward engineering can finally solve the tasks used in this work and might outperform human players as in [A, B], the training inefficiency and the possible risk during training are still concerning.\n\nIn this paper we want to demonstrate the idea that using human active involvement can better communicate human preferences to the learning system while ensuring learning efficiency and safety.\n\n---\n\n> **W4:** *Doubts about the experimental results for BC and GAIL. Are there any errors in the experiments?*\n\n**Re:** \n\nDataset size is a critical factor for imitation learning baselines. Table 1 in the submission presents the test performance of the BC and GAIL agents trained by 30K steps collected by human subjects.\n\nTo further address your question, we run a CQL baseline on the same 30K steps human dataset. Moreover, we use a trained RL agent as the expert to generate 250K transitions and train baselines on those 250K steps dataset. As shown in the below table, these baselines can achieve decent performance with sufficiently large datasets. Therefore, there are no errors in the implementation in Table 1 and it is the insufficient training data that causes the failure of  BC and GAIL.\n\n|  | Episodic Return | Episodic Cost | Success Rate |\n|---|---|---|---|\n| BC (30K) | 113.3 | 2.17 | 0.07 |\n| GAIL (30K) | 81.5 | 1.30 | 0.0 |\n| CQL (30K) | 156.4 | 1.71 | 0.21 |\n| BC (250K) | 362.1 | 0.13 | 0.57 |\n| GAIL (250K) | 309.6 | 0.68 | 0.60 |\n| CQL (250K) | 373.9 | 0.24 | 0.72 |\n\n---\n\n[A] Outracing champion Gran Turismo drivers with deep reinforcement learning\n\n[B] Expert Human-Level Driving in Gran Turismo Sport Using Deep Reinforcement Learning with Image-based Representation"}, {"review_id": "FgVmu4WpRK", "reviewer": "Reviewer_pvGG", "summary": "This paper focuses on human-in-the-loop for reward-free policy learning, wherein a human has the option to override the policy by taking over control when the agent attempts to perform risk behaviors. To solve this problem, this work presents Proxy Value Propagation (PVP), which directly assigns positive values (+1) to the actions induced by the human interruption and negative values (-1) to the actions the agent takes in specific states. By manipulating the values, PVP encourages the agent to approximate human behaviors while avoiding the actions intervened by human.\n", "questions": "- If the reward function is set to zero, why $Q=\\sum r_t$ can be set to $+1$ or $-1$? \n\n- For two adjacent time steps in an episode, if both $Q(s_1,a_1)$ and $Q(s_2,a_2)$ are set to 1, how can it satisfy Bellman equation as $Q(s_1,a_1) = \\gamma max_a Q(s_2,a)$?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n- The problem was well-motivated and the setting is realistic. \n\n- The empirical results suggest PVP outperforms a set of baselines involving imitation learning and other human-in-the-loop methods in terms of multiple measurements. Furthermore, the experiments with human subjects are convincing and extensive.\n\n- The proposed method is simple and efficient.\n", "weaknesses": "One major concern is that the technique of directly manipulating the value seems to be not well-motivated. As claimed in lines 178-180, the optimal policy should approximate the behaviors of human subjects while avoiding performing the intervened actions. Starting from this motivation, two natural ways are 1) assigning high rewards to human actions and low rewards to intervened policy actions, and 2) constraining the policy by manipulating the likelihood of the specific actions. Since the value function induced by TD learning measures the long-term performances starting from specific state-action pairs, directly assigning +1 or -1 to the value function seems to tell the agent that it tends to succeed or fail starting from the s-a pairs. However, human might only take control at some discrete time steps, and the subsequent behaviors after the interruption are unpredictable. Thus, the natural design is to assign manufactured rewards (+1/-1) to the specific transitions from my perspective. Though I believe that directly manipulating the value and propagating through TD-learning might be a better choice, it is worth comparing with the naive baseline mentioned above and further discussing the advantages of PVP.\n\nThough learning with online human interruptions is quite an interesting setting, it is pretty ideal to me as online learning is risky and expensive even with a human guardian in several domains (e.g., robotics). In terms of the driving case investigated in the paper, I believe that offline learning with human interruption datasets might be more realistic and interesting given numerous in-the-wild datasets. However, I would agree the investigation is out of scope concerning the current paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your review! Please find our responses as follows.\n\n\n---\n\n### Weaknesses\n\n> **W1:** *One major concern is that the technique of directly manipulating the value seems to be not well-motivated. As claimed in lines 178-180, the optimal policy should approximate the behaviors of human subjects while avoiding performing the intervened actions. Starting from this motivation, two natural ways are 1) assigning high rewards to human actions and low rewards to intervening policy actions, and 2) constraining the policy by manipulating the likelihood of the specific actions. Since the value function induced by TD learning measures the long-term performances starting from specific state-action pairs, directly assigning +1 or -1 to the value function seems to tell the agent that it tends to succeed or fail starting from the s-a pairs. However, humans might only take control at some discrete time steps, and the subsequent behaviors after the interruption are unpredictable. Thus, the natural design is to assign manufactured rewards (+1/-1) to the specific transitions from my perspective. Though I believe that directly manipulating the value and propagating through TD-learning might be a better choice, it is worth comparing with the naive baseline mentioned above and further discussing the advantages of PVP.*\n\n**Response:**\n\nThough the idea of assigning reward +1 to human's actions and -1 to intervention actions is straightforward, it doesn't work in practice. This is because in the framework of value-based RL, the reward participates in the learning via the TD loss:\n\n$\nJ^\\text{TD}(\\theta) = \\mathbb E_{(s, a, s')} | Q_\\theta(s, a) - (r(s, a) + \\gamma \\max_{a'} Q_{\\hat{\\theta}}(s', a')) |^2\n$\n\nYou can see that the Bellman backup is conducted on the transition triplet $(s, a, s')$, where you have to use future's value with current reward as an estimator of current value. Therefore, the reward must correspond to the action from the behavior policy, the action $a$ causes the transitions from $s$ to $s'$. In our context, during human involvement, the action $a = a_h$ must come from human policy as the human subject is taking control $s' \\sim \\mathcal P(s, a_h)$. And thus the reward will be $+1$ for those human-involving transitions.\n\nTherefore, we have no way to \"assign low rewards to intervening policy actions $a_n$\" because we need to know the future state $s'' \\sim \\mathcal P(s, a_n)$ caused by this action so that we can compute the TD target. However, we can only have the future state caused by human action. It is also not practical to query the environment to get the next state $s''$ as the $a_n$ is potentially a danger / undesired action and there is no way to replay it in the environment.\n\nIn our preliminary experiment, we have tried this idea and the policy failed to learn anything no matter how much human involvement is provided. The reason is the reward will always be $+1$ for the transitions that occur during human involvement. And the learning agent found a pitfall to maximize its reward: it always demonstrates undesired behaviors, e.g. making a sharp left turn toward the road edge, so that humans will always take control which causes +1 reward.\n\n\n---\n\n> **W2:** *Though learning with online human interruptions is quite an interesting setting, it is pretty ideal to me as online learning is risky and expensive even with a human guardian in several domains (e.g., robotics). In terms of the driving case investigated in the paper, I believe that offline learning with human interruption datasets might be more realistic and interesting given numerous in-the-wild datasets. However, I would agree the investigation is out of scope concerning the current paper.*\n\n**Response:**\n\nWe agree that offline learning is a very important setting that is worth a lot of effort. The active human involvement being tackled in this submission is an orthogonal research setting and has its unique value in safety critical systems. In the development and deployment of learning-based systems, such as self-driving cars, human-AI shared control is a common scenario as there is usually a human driver sitting behind the wheel preparing to take over the control. Our proposed method can effectively leverage the human data collected during such shared control.\n\nOffline learning can help augment PVP, especially when we want to scale up the proposed method. We can introduce a pretraining stage before the human-AI shared control in PVP. By doing so the pretrained base policy learns some basic skills from in-the-wild datasets to reduce the human cost while achieving human alignment in later stages when training with PVP.\n\n---\n\n### Questions\n\n\n> **Q1:** *If the reward function is set to zero, why $Q = \\sum r_t$ can be set to +1 or -1? For two adjacent time steps in an episode, if both $Q(s_1, a_1)$ and $Q(s_2, a_2)$ are set to 1, how can it satisfy Bellman equation as $Q(s_1, a_1) = \\gamma \\max_a Q(s_2, a)$?*\n\n**Response:**\n\nThe proxy value assignment is an auxiliary learning objective but not a hard constraint to the proxy value function. This is because apart from the PVP loss, we also have the TD loss and thus the proxy value might not regress to $\\pm 1$.\n\nIn the context of this work, the proxy value function is not an estimator of the discounted return, as we don't have a reward function at all. Instead, the proxy value function is a function measuring human preference and can be used to select human preferable actions when applying the argmax rule on it $a = \\arg\\max_a Q(s, a)$. That is to say, the proxy value function can induce desired behavior, while it may not necessarily hold a mathematical interpretation such as the estimator of the discounted return.\n"}, {"review_id": "mQRrHq2rcp", "reviewer": "Reviewer_iaMb", "summary": "The paper proposes an approach for learning from human interventions based on offline RL with pseudo-rewards. The paper assumes a shared autonomy setting where a human monitors an agents rollouts and can intervene whenever deemed necessary and provide corrective demonstrations. The algorithm relabels those interventions with positive Q value targets and the faulty policy actions that led to intervention with negative Q targets. It then trains the policy on these pseudo-rewards with offline RL. In extensive experiments on multiple driving simulators with differing human input devices, the approach leads to faster and more successful learning of desired behaviors than prior RL and learning-from-human-intervention approaches.", "questions": "(Q1): How could the proposed method be combined with pre-trained policies or ground truth rewards? \n--> the current method requires lots of human teaching, it would be beneficial if human teaching was only used to *supplement* autonomous agent learning from some external source of supervision. The one experiment in the paper that does combine it with external rewards seems unsuccessful -- how can this be fixed?\n\n(Q2): How can the proposed method be extended to situations where humans cannot directly provide demonstrations?\n--> eg for an ant robot it is impossible for humans to provide demonstrations (the authors mention this short-coming of the current approach in the limitations section). I am wondering whether you have any ideas how this could be extended to situations like the ant robot?", "limitations": "- add more details about human experiment design\n\n- add a (simulated) robotics manipulation environment\n\n- clarify concerns about the proposed objective\n\n\n## Summary of Review\n\nOverall, I think this is a strong paper. It combines a novel, yet simple and elegant approach with an extensive evaluation and comparison to numerous prior works. While I would like the authors to address the points raised in my review, I am happy to accept the submission.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- the paper's approach is elegant and simple, it leverages common offline RL techniques in combination with pseudo-reward labels and, due to it's simplicity, seems easily applicable to other domains\n\n- the empirical evaluation of the approach is comprehensive, with numerous baselines spanning the relevant related works, multiple environments and many meaningful ablations\n\n- the paper is well-written and easy to understand. it has intuitive visualizations for illustrating the approach, evaluation environments and qualitative results.\n\n", "weaknesses": "(A) **Concerns about training objective**: the training objective sets the **Q-value targets** of the human intervention trajectories to +1 and the Q-value targets of the policy actions in those intervention trajectories to -1. I have a few concerns about this objective:\n\n  (1) by setting the Q-value target instead of setting the reward, the objective does not allow reward propagation through these updates, e.g. from post-intervention states. At the same time the algorithm runs \"regular\" Q-updates with full propagation and adds both objectives. Why not set **rewards** instead of Q-targets to indicate good and bad behavior and only run the regular Q-update on all data samples? That seems more elegant than the current two-part objective\n\n  (2) the current objective assigns low Q-value to all policy actions during human intervention. However, the human intervention only provides a signal that the sequence of policy actions **before** the intervention were bad. It would for example be possible that the human \"over-intervenes\" and keeps providing demonstrations in states in which the policy's actions are already good again. In these cases the current objective would penalize good policy actions\n  \n(B) **Only navigation environments**: while the experimental evaluations are comprehensive, they are limited to navigation environments (driving and grid world navigation). It would strengthen the paper to show the same algorithm can work for teaching robot manipulation tasks, e.g. by performing teleoperation in a simulated robotic environment.\n\n(C) **Unclear human user experiments**: all experimental evaluations are performed with real humans in the pipeline. While this is generally positive, it means special care needs to be taken to ensure fair comparability between all methods, since different users may interact with systems differently, they learn and adapt to systems over time etc. Thus, human user studies require great care in execution and documentation, eg ensuring that the same users interact with all methods and baselines (to account for differences between individual users), the order of methods is randomized and all users are given some time to practice with the system (to account for adaptation effects) etc. The paper does not provide any details in this regard for the quantitative evaluations in Tables 1, 2 and 4. It would be important to provide more details on the design of all experiments involving human users to ensure fair comparison.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Please find our responses as follows.\n\n---\n\n### Weaknesses\n\n> **(A1)** *Why not set rewards instead of Q-targets?*\n\nPlease refer to the response to W1 of Reviewer pvGG.\n\n> **(A2)** *The current objective assigns low Q-value to all policy actions during human intervention. However, human intervention only provides a signal that the sequence of policy actions before the intervention were bad. It would be possible that the human \"over-intervenes\" and keeps providing demonstrations in states the policy's actions are already good again. In these cases the current objective would penalize good policy actions.*\n\n**Response:**\n\nLet's suppose the \"over-intervention\" happens and this means the human's action is identical to the agent's action $a_n = a_h$. At this step, PVP will try to assign high proxy value to $Q(s, a_h)$ (regressing to $+1$) and low proxy value to $Q(s, a_n)$ (regressing to $-1$). However because $a_h = a_n$, their proxy values will cancel each other:\n\n$\n\\min (Q(s, a_h) - 1 )^2 + (Q(s, a_n) + 1)^2 + \\text{TD Loss} \n\\equiv\n\\min 2Q^2(s, a) + \\text{TD Loss}\n $\n\nUnder this circumstance, the proxy value still prefers this action because those human-intervening actions have much lower value (near $-1$) while those good actions have values close to 0.\n\n---\n\n> *(B) Only navigation environments: while the experimental evaluations are comprehensive, they are limited to navigation environments (driving and grid world navigation). It would strengthen the paper to show the same algorithm can work for teaching robot manipulation tasks, e.g. by performing teleoperation in a simulated robotic environment.*\n\n**Response:**\n\nWe totally agree! It is a promising future direction to apply PVP in robot manipulation tasks. There is a huge challenge to existing RL algorithms in robot manipulation tasks since fine manipulation tasks such as threading a zip tie or juggling a ping pong ball is very hard to be characterized by a reward function without sufficient engineering efforts. The proposed PVP method can learn from human demonstration behaviors in an online manner and provide corrective feedback to the agent. Experimenting on Robot manipulation tasks will be our immediate future extension. We plan to utilize the teleoperation hardware developed in a recent work [A] to conduct the real-world experiment.\n\n\n[A] Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS’23\n\n---\n\n> *(C) Unclear human user experiments: all experimental evaluations are performed with real humans in the pipeline. While this is generally positive, it means special care needs to be taken to ensure fair comparability between all methods. Thus, human user studies require great care in execution and documentation, eg ensuring that the same users interact with all methods and baselines (to account for differences between individual users), the order of methods is randomized and all users are given some time to practice with the system (to account for adaptation effects) etc. It would be important to provide more details on the design of all experiments involving human users to ensure fair comparison.*\n\n**Response:**\n\nThank you for your advice! \n\nWe provide a one-page PDF on human subject research protocol in the common rebuttal, where we discuss the requirement to human subjects, the onboarding period and the information / advice we provide to the human subjects.\n\nFor your concern on the proficiency biases, we do randomize the order of experiments with different control devices, tasks and training algorithms for each participant. Therefore we can average out the \"adaptation effects\".\n\n---\n### Questions\n\n> *(Q1): How could the proposed method be combined with pre-trained policies or ground truth rewards? --> the current method requires lots of human teaching, it would be beneficial if human teaching was only used to supplement autonomous agent learning from some external source of supervision. The one experiment in the paper that does combine it with external rewards seems unsuccessful -- how can this be fixed?*\n\n**Response:** \n\nWith the recent successes on large decision models such as Robotic Transformer 2 (RT-2), pretraining policy on large datasets is a promising approach to further improve the proposed method.\n\nWe haven't conducted an experiment on this. We have an ablation study that uses both the environmental reward and human demonstration at the same time. Because two supervision signals are provided at the same time, they might not be fully aligned and this causes confusion to the model. For example, humans might want to slow down before a crossroad, while the speed reward in our reward function encourages the agent to move as fast as they can.\n\nIn future work, we can experiment with a natural idea: the two-stage training scheme. We can first pretrain policy with ground truth reward or imitation learning and in the second stage we can incorporate the proposed PVP method to finetune the policy to improve its preference alignment such as improving the safety-critical performance. \n\n---\n\n> *(Q2): How can the proposed method be extended to situations where humans cannot directly provide demonstrations? --> e.g. for an ant robot it is impossible for humans to provide demonstrations (the authors mention this short-coming of the current approach in the limitations section). I am wondering whether you have any ideas how this could be extended to situations like the ant robot?*\n\n**Response:** \n\nWe can learn a goal-conditioned policy first, which also connects to your first idea using pretraining. We will then conduct the human-AI shared control in the second stage: we can invite human subjects to give demonstrations on the goal space, instead of the low-level action space. PVP can thus learn a high-level control policy from human demonstration and intervention that produces high-level goals for the low-level controller.\n\n"}, {"review_id": "dduDpObXKL", "reviewer": "Reviewer_sDoy", "summary": "The work introduces Proxy Value Propagation (PVP), an approach to reinforcement-learning without ground-truth rewards by instead using human intervention as a signal of quality. The value of a state-action pair is determined by a human intervention, where interventions signal that the agent's behavior was bad (and we assume that the human's action was good). Compared to naive RL approaches or other human-intervention methods, such as HG-DAgger, PVP learns self-driving policies much faster and with fewer crashes than all other approaches. The work is also tested with a user-study, gathering demonstration data from real humans.", "questions": "1. Is there any drawback to the proposed approach to action labeling when compared to something like COACH [1]? Should PVP be considering the time-delay of human feedback?\n2. Line 187-188 states that the novice-policy (agent behavior) contains information of the forward dynamics. What does this mean?\n3. The addition to the CQL loss is likened to an L2 regularization on the Q-values from the agent, but later experiments adding this L2 penalty to CQL seem to perform much worse than PVP. Why is there a discrepancy between the two? Shouldn't CQL + L2 == PVP, as in Equation 6?\n4. It seems that PVP relies heavily on humans doing the initial exploration. Would something like LOKI [3] perform comparably, or is the value propagation part of the key to success?\n5. Please address the questions listed above concerning the user study. Additional detail should be provided.", "limitations": "The authors list and acknowledge several limitations. Mitigations to these are left to be the subject of future work.\n\nSocietal impact of the work is not explicitly addressed, though an ethics statement does appropriately address IRB for the study.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Technical contribution:\n* The proposed method works very well empirically, outperforming baselines with and without human intervention.\n* The relative contribution of different components of the PVP approach are tested with an ablation study in the CARLA domain.\n* Experiments cover several domains including continuous/discrete actions, different input mechanisms, and different state observations.\n* The proposed approach specifically considers balance between human/agent labels, making sure to consider human preferences even when there are relatively few demonstrations/corrections.\n\nExperiments:\n* The experiments are clearly described and appropriately test the proposed PVP method against baselines.\n* Human users are recruited and the human-intervention method is tested with real users, not synthetic/heuristic labeling.\n\nNovelty:\n* The paper compares well against prior work and the related works section covers most important previous approaches to the problem.\n* The experiments compare against several baselines in prior work.\n\nClarity:\n* The method is intuitive and easy to follow. Figures and tables clearly communicate the results.\n* The supplementary material provides additional detail where useful (hyper-parameters, control schemes, etc.).\n\nReproducibility:\n* Code is provided to reproduce all experiments.", "weaknesses": "Technical contribution:\n* Prior work in human-labeling and intervention has found that human labelers are slow to react, and this time-delay must be accounted for (e.g. in [1], the authors needed to use eligibility traces to account for _which_ agent action was incorrect). Strange that this did not surface as a problem in these studies.\n* The proposed treatment of agent vs. human Q-values could result is serious policy degradation if the human labeler makes mistakes or if the demonstrator is sub-optimal.\n* Fusion of PVP with environment reward resulted in _worse_ performance.\n* Removing epsilon greedy exploration from DQN effectively forces humans to do the exploration for PVP, which seems to work well in the driving simulator, but could be an issue in other domains.\n\nUser study:\n* Information on the study is lacking. How many participants? What about background in computer science or driving? Mean age? What sorts of instructions were provided? Were participants allowed to practice first? These details can be in the supplement if they are not critical to the analysis of the algorithm, but they should be included for a user study.\n\nNovelty:\n* The approach seems very similar to [1], which treats human labels as indications of advantage instead of simply as reward (as in DeepTAMER, which is cited in the paper). While comparison to [1] is perhaps not necessary (as the paper compares thoroughly to several state-of-the-art approaches), [1] should at least be mentioned.\n* Should be aware of other approaches to human-intervention labeling (e.g., [2], which also considered driving feedback, or [3] which performs BC and then RL)\n\n\n[1] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. 2017. Interactive learning from policy-dependent human feedback. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (ICML'17). JMLR.org, 2285–2294.\n[2] Mariah L. Schrum, Erin Hedlund-Botti, Nina Moorman, and Matthew C. Gombolay. 2022. MIND MELD: Personalized Meta-Learning for Robot-Centric Imitation Learning. In Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction (HRI '22). IEEE Press, 157–165.\n[3] Cheng, Ching-An, Xinyan Yan, Nolan Wagener and Byron Boots. “Fast Policy Learning through Imitation and Reinforcement.” Conference on Uncertainty in Artificial Intelligence (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n---\n> **W1:** *Prior work in human-labeling and intervention has found that human labelers are slow to react, and this time-delay must be accounted for. Strange that this did not surface as a problem in these studies.*\n\n**Re:** \n\nIndeed, we do not consider the time-delay in the current method. The problem is mitigated as PVP learns from active human involvement. If the agent learns a time-delayed policy, then human subjects will correct such behaviors in later training.\n\nThe eligibility traces in [1] allow feedback to affect earlier decisions. The TD loss in our system serves a similar role that can propagate proxy values (considering it as an implicit form of human feedback, as it is not hard-coded to $\\pm 1$ for each action) to previous steps.\n\n---\n> **W2:** *The proposed treatment of agent vs. human Q-values could result in serious policy degradation if the human labeler makes mistakes or if the demonstrator is sub-optimal.*\n\n**Re:** \n\nIn our empirical studies, we found that mistakes made by the human labeler can indeed impact the policy during the early stages of training. However, this degradation is usually smoothed out as training progresses, with correct human demonstrations mitigating the impact of early errors. \n\nDue to resource limits, we did not perform quantitative analysis on how the level of noise (or how the quality of demonstration) affects policy learning. As we shown in the analysis in Appendix D Theorem D.3, the training-time performance is bounded by the human’s performance in providing action and in intervention. The quality of demonstration and intervention does affect the performance of PVP.\n\n---\n> **W3:** *Fusion of PVP with environment reward resulted in worse performance.*\n\n**Re:** \n\nAs presented in the one-page PDF, the reward function we used for driving tasks has 4 terms. The displacement reward, the speed reward, the collision penalty and the termination reward. However, as we discussed in the experiment section, the environmental reward might not necessarily align with human preference. Human subjects usually demonstrated more fine grained behaviors such as decelerating before crossroad or maintaining a safe distance from the front vehicles. Those behaviors might contradict the speed reward as it encourages high speed. Such misalignment between reward functions and human preference leads to worse performance.\n\n---\n> **W4:** *Information on the human subject study is lacking.*\n\n**Re:** \n\nThank you for the advice! Section 1 in the one-page PDF discusses the requirements for human subjects, the onboarding period and the information / advice we provide to the human subjects.\n\nWe summarize the answers to your concerns:\n* We recruit 5 human subjects. \n* All of them are college students with an average age of 21.4 years old. \n* They go through an onboarding procedure before the main experiments when they can practice the tasks with different control devices. \n* The participants are informed by the objectives: driving safely to the destination and ensuring the behavior aligns with traffic regulations and human preferences.\n* The participants are encouraged to perform intervention whenever they perceive the vehicle might be in a dangerous situation, in violation of traffic rules, or in whatever scenario the human subjects feel they wouldn't behave in the way the novice policies do. \n\n---\n> **W5:** *The approach seems very similar to [1]. [1] should at least be mentioned. Should be aware of other approaches to human-intervention labeling.*\n\n**Re:** \n\nThank you for bringing the insightful works [1, 2, 3]. We will discuss these papers in the revision. Compared to COACH [1], PVP accepts not only the feedback (the intervention signal) but also the human demonstration. Besides, PVP uses TD loss to build a proxy value function instead of using the feedback directly to train the policy. However, PVP does not consider the time-delay of human subjects explicitly as COACH.\n\n---\n> **Q1:** *Is there any drawback when compared to something like COACH [1]? Should PVP be considering the time-delay of human feedback?*\n\n**Re:** \n\nWe do not consider the time-delay in the current method explicitly and this is indeed a limitation.\n\nHowever, PVP has two features that can mitigate this issue. The eligibility traces in COACH allow feedback to affect earlier decisions. The TD loss in our system serves a similar role that can propagate proxy values (considering it as an implicit form of human feedback, as it is not hard-coded to $\\pm 1$ for each action) to previous steps.\n\nMeanwhile, PVP trains a deterministic novice policy and thus its behavior is consistent in a short period of time. Human subjects can relatively easier to determine the intention of the novice policy and thus make an intervention accordingly, compared to DAgger or HACO where there exists stochasticity in the novice policy. Therefore the time-delay of human feedback might not be a large issue.\n\nLast but not least, as you can see in the human subject research protocol, the objective of the driving task we told the human subjects is to arrive at the destination safely while the behavior should be human-like. We don't require the human subjects to drive as fast as possible. Thus the requirement of real-time response is relaxed.\n\n---\n> **Q2:** *The addition to the CQL loss is likened to an L2 regularization on the Q-values from the agent, but experiments adding L2 penalty to CQL seem to perform much worse than PVP. Why is there a discrepancy between the two? Shouldn't CQL + L2 == PVP, as in Equation 6?*\n\n**Re:** \n\nThere might be a misunderstanding. In the ablation study section (Line 339 and Table 4), we conduct experiments to evaluate the performance of the pure CQL objective (without regularization) while other PVP designs remain the same. This experiment shows that pure CQL is not applicable to this human active involvement setting. We will modify the wording to address the misunderstanding.\n\n"}]

Author rebuttal:
In this common rebuttal, we provide a one-page PDF containing two sections: 

1. Human subject research protocol: including information on human subjects recruitment, onboarding procedure, the information we provide to them, the main experiment and the questionnaire. We will update Appendix B to include newly added content.
2. Reward function for driving tasks: describing the detailed reward function we used. It contains four terms, a dense displacement reward which measures the progress toward destination, a speed reward, a collision penalty and a termination reward.


Meta-review: The paper introduces PVP that leverages human intervention as a signal of action quality instead of relying on ground-truth rewards. PVP assigns high Q values to actions taken by humans and low Q values to actions requiring human corrections. By relabeling human interventions with positive Q value targets and penalizing the agent's erroneous actions, the policy is trained through offline reinforcement learning. 


**strengths**

* Extensive comparisons with prior work and multiple baselines are conducted in the experiments.
* Human experiments that ensure real-world applicability and avoid synthetic/heuristic labeling.
* The proposed method is presented in a clear and intuitive manner.
* The paper's approach is noted for its elegance and simplicity, making it potentially applicable to various domains.

**weaknesses/suggestions**

* It would be nice if the authors could clarify the details of the user study in the final version.
* Current experiments are limited to navigation environments. It would strengthen the paper to show the results on robot manipulation tasks.

In summary, the paper presents the PVP method as a compelling approach to reinforcement learning with human involvement, backed by comprehensive experiments and clear presentation, making it an impactful contribution to the field.
Decision: Accept (spotlight)