Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization
Authors: Quanyi Li
Zhenghao Peng
Bolei Zhou
Venue: ICLR 2022 Poster (2022)
Keywords: Human in the Loop
Safe Reinforcement Learning
Autonomous Driving

Abstract: Human intervention is an effective way to inject human knowledge into the training loop of reinforcement learning, which can bring fast learning and ensured training safety. Given the very limited budget of human intervention, it remains challenging to design when and how human expert interacts with the learning agent in the training. In this work, we develop a novel human-in-the-loop learning method called Human-AI Copilot Optimization (HACO).To allow the agent's sufficient exploration in the risky environments while ensuring the training safety, the human expert can take over the control and demonstrate how to avoid probably dangerous situations or trivial behaviors. The proposed HACO then effectively utilizes the data both from the trial-and-error exploration and human's partial demonstration to train a high-performing agent. HACO extracts proxy state-action values from partial human demonstration and optimizes the agent to improve the proxy values meanwhile reduce the human interventions. The experiments show that HACO achieves a substantially high sample efficiency in the safe driving benchmark. HACO can train agents to drive in unseen traffic scenarios with a handful of human intervention budget and achieve high safety and generalizability, outperforming both reinforcement learning and imitation learning baselines with a large margin. Code and demo video are included in the supplementary materials.

Published as a conference paper at ICLR 2022
EFFICIENT LEARNING OF SAFE DRIVING POLICY VIA HUMAN-AI COPILOT OPTIMIZATION
Quanyi ; Zhenghao Peng": Bolei Zhou²
CCentre for Perceptual and Interactive Intelligence. Thhe Chinese University of Hong Kong, University of California, Los Angeles
ABSTRACT
Human intervention is an effective way to inject human knowledge into the loop of reinforcement learning, bringing fast learning and training safety. But given the very limited budget of human intervention, it is challenging to design when and how human expert interacts with the learning agent in the training. In this work, we develop a novel human-in-the-loop learning method called Human-Al Copi- lot Optimization (HACO). To allow the agent's sufficient exploration in the risky environments while ensuring the training safety, the human expert can take over the control and demonstrate the agent how to avoid probably dangerous situa- tions or trivial behaviors. The proposed HACO then effectively utilizes the data collected both from the trial-and-error exploration and human's partial demonstra- tion to train high-performing agent. HACO extracts proxy state-action values from partial human demonstration and optimizes the agent to improve the proxy values while reducing the human interventions. No environmental reward is re- quired in HACO. The experiments show that HACO achieves a substantially high sample efficiency in the safe driving benchmark. It can train agents to drive in seen traffic scenes with a handful of human intervention budget and achieve high safety and generalizability, outperforming both reinforcement learning and imita- tion learning baselines with a large margin. Code and demo videos are available at: /tttps://decisiii
www.ggitthhu..... io/HACO//
INTRODUCTION
How to effectively inject human knowledge into the learning process is one of the key challenges to training reliable autonomous agents in safety-critical applications. In reinforcement learning (RL), researchers can inject their intentions into the carefully designed reward function. The learning agent freely explores the enviroument to collect the data and develops the desired behaviors induced by the reward function. However, RL methods bear two drawbacks that limit their applications in safety-critical tasks: First, the nature of trial-and-error exploration exposes RL agent to dangerous situations (Saunders et al., (017). Second, it is difficult to summarize all the intended behaviors to be learned into the reward function. Taking the driving vehicle as an example, an ideal policy should obtain a set of skills, such overtaking, yielding, emergent stopping. and negotiation with other hicles. It is intractable to manually design a reward function that leads to the emergence all those behaviors in the trained agent. To mitigate these two challenges, practitioners enforee the human intentions through imitation learning (IL.) where the agent is trained to imitate the expert-generated state and action sequences. During the demonstration, the premature agent does not interact with the risky environment and thus the training safety is ensured. High-quality expert demonstrations provide direct the optimal solution for the agent imitate from. However, paradigm suffers from the distributional shift problem (Ross& & Bagnell 2010, Ross et al.) (011] while the induced skills are not sufficiently robust with respect to changes in the control task (Camacho & Michie 1995).
learning is an alternative paradigm to inject human knowledge, where a human subject accompanies the agent and oversees its learning process. Previous works require the human to either passively advise which action is good (Mandel et al.) 2017) or evaluate the collected trajectories (Christiano et al. 2017 Guan et al. 202 Reddy et al,
Different from vanilla RL or II., human-in-the-
"Quanyi Li and Zhenghao Peng contribute equally this work.

---

Published as a conference paper at ICLR 2022
Safe Action &
Environment
pamal from human
obstade
intervantion
Agent Action a
Agent
Human Expert
control target
Intervention Cost C₁
trajectory from agant
A. Human-Al Copilot
B. Human Interface
C. Copilot Trajectory
method, human subjects will decide whether to intervene in the execution of the learning agent and demonstrate the correct action. Such intervention incurs a cost to the agent such that it learns to minimize the total cost during the training. B. The human interface for copilot during the training- The human subject can steer the wheel or press the paddle to start the intervention and then the demonstration data will be recorded. C. An illustrative driving trajectory where the green segments are generated by the agent and the blue segments are the demonstrations of the human expert overcome the dangerous situations.
Figure 1: A. In the proposed human-in-the-
2018 Warnell al. 2018 Christiano et al. 2017 Sadigh et al. 2017: Palan et al. 2019). This kind of passive human involvement exposes the Bl system to risks since the agent explores the environment without protection. Some other works require the human merely intervene in the exploration by terminating the episode (Saunders et al., 2018 Zhang & Cho (2016). but it is not practical to terminate and reset the environment instantly the real world (Xu et al. 2020). Intervening and taking over the control from the learning agent is a natural approach to safeguard the human-AI system (Kelly al., 2019 Spencer al. 2020). However, a challenge exhibited in previous works is the budget of human intervention. Since human cognitive resource is precious and limited, it is essential to carefully design when and how the human expert involves in the learning process that the human knowledge can be injected effectively.
In this work, we propose an efficient human- in- -n-the-loop learning method called Human-Al Copilor Optimization (HACO). The key feature of HACO is that it can learn minimize the human inter- vention and adjust the level of automation to the learning agent adaptively during the training. As shown in Figure A HACO allows the human expert take tive manner. If the human decides to intervene in the action of the agent, he/she should demonstrate the correct actions to overcome current undesired situations to the learning agent. The human inter- vention and the partial demonstration are two sources of informative training data. We use offline RL technique to maintain proxy value function of the human-Al mixed behavior policy even though the agent doesn't 't have the access to the environmental reward during training. To encourage the exploration in the state-action space permitted by human, we also maximize the entropy of action distribution of the agent if the agent is not taken over.
the human-Al system in proac-
Experiments in the virtual driving environments MetaDrive Li et al. 2021) and CARLA (Doso- vitskiy al. (017)) show that, with economic human budget, HACO outperforms RL and IL baselines with a substantial margin terms of sample efficiency, performance, safety, and general- izability in the unseen testing environment. Thus the human-Al Al copilot optimization is efficient learning paradigm to inject human knowledge in an online setting.
RELATED WORK
Learning from Demonstration. Passive imitation learning such as behavior cloning (Widrow) 1964 Osa al., 2018 Huang et al.] 2020 Sun al. 2020) and recently proposed offline RL methods (Kumar et al. 2020; Fujimoto et al. 2018 Wu et al. 2019) train agents from an out- of-the-shelf data set and guarantee the training safety, since no interaction with the environment is needed. Inverse RL methods (Ng et al., 2000; Abbeel & Ng. 2004 Fu et al., 2017 Bloem & Bambos 2014) learn a reward function from the human demonstration and then use to incentivize the agents to master the intended behaviors. Proposed more recently. GAIL (Ho & Ermon 2016) and its variants (Song et al. 2018 Sasaki et al. 2018 Kostrikov et al. 2018) and SQIL (Reddy et al. 2019) compare the trajectory similarity between agents and humans and thus require the agent to interact with the environment. Similar to RL methods, this paradigm exposes the agent to potentially dangerous situations.
2

---

Published as a conference paper at ICLR 2022
Human-in-the-lopp Learning Methods. Many works focus incorporating human in the train- ing loop of conventional RL or IL paradigms. DAgger (Ross et al. 2011) and its extended meth- ods (Kelly al.] 2019) Zhang & Cho 2016 Hoque et al. 2021) correct the compounding error (Ross) & Bagnell| 2010) of behavior cloning by periodically requesting expert to provide more demonstra- tion. Instead of proving demonstration upon requests, Human-Gated DAgger (HG-DAAger) (Kelly)
iet al., 2019). Expert Intervention Learning (EIL) (Spencer et al., 2020) and Intervention Weighted Regression (IWR) (Mandlekar et al. 2020) empower the expert intervene exploration and carry the agent to safe states. However, these methods do not impose constraints to reduce human inter- vention and do not utilize the data from the free exploration of the agent. Human subjects can also involve in the loop providing preferences based on evaluative feedback on two behavior sequences
generated by the agent (Christiano al. 2017 Sadigh et al. 2017 Palan et al., 2019 Ibarz et al., 2018
Cui & Niekum 2018).
Human-Al copilot or shared autonomy is more intimate form of the numan-in-the-lone methods. The Al agent and human are working together simultaneously to achieve a common goal. By giv- ing human guidance and feedback at run-time instantly, the explorable state and action spaces can be greatly narrowed down (Saunders al. 2018). The learning goal can further match the task objective by providing extra human feedback combined with reward function (Reddy et al.) 2018 Warnell al., 2018 Wu al., [02T] Cederborg al.) 2015 Arumugam al. [219)]. Human-AI Al copilot is helpful and practical when applying RI. to real world tasks where safety constraints must be satisfied (Garcıa & Fernández 2015; Amodei et al.) 2016; [hharadwwaj et al. 2020 AAlshiekh
iet al. 2018). In our previous work (Peng et al. 2021), we made attempt to develop a method called Expert-Guided Policy Optimization (EGPO) where PPO expert policy is involved to monitor the learning agent. The difference can be summarized as twofold: (1) We substitute the expert with human and design special mechanism to mitigate the delayed feedback error; (2) Based on the com- prehensive ablation study and prototyping, we remove redundant designs like takeover function and the need reward function, making the proposed method simple yet effective.
Reducing human burden is major challenge in comman-nn-the-ooe methods. A feasible solution is to Iearn an intervention function that imitates human intervention signals and stops the catastrophic actions of agents (Kelly et al. 2019 Zhang & Cho 2016 Saundeers et al. 2017 Abel et al. 2017), which can relieve the mental stress of the human subject during training. In this work, we devise our learning scheme explicitly to include the human cognitive cost as one of the objectives to minimize.
3 HUMAN-AI COPILOT OPTIMIZATION
In this section, introduce Human-Al Copilot Optimization (HACO), efficient learning algo- rithm that trains agents from human interventions, partial demonstrations and free exploration. For suman-in-the--e learning, it is essential to design when and how to engage human subjects. The major issue is the cognitive cost of the human subject (Zhang et al. 2021). Frequent querying might bring tremendous cognitive cost and exhaust the human expert, causing incorrect delayed feed- back that hinders the training. Thus the proposed pipeline aims to minimize the human intervention cost during the training, which reduces the reliance on the expert's demonstration over time and im- proves the learning agent's 's autonomy. The overall workflow of HACO is presented in Algorithm [
3.1 HUMAN-AI COPILOT TRAINING PARADIGM
We aim to learn an autonomous agent with policy T, (a, (s) that can make informed action o, in state As shown in Fig. D we frame the human-Al copilot paradigm that extends the standard reinforcement learning diagram by incorporating a human expert. At each step, the human expert oversees current state and decides whether to intervene. If necessary, he/she will execute human action On to overwrite the agent's action 6n- We denote the human intervention by a Boolean indicator T(s, an)) and thus the action applied to the environment is called the safe action à [(s, an )a, + (1 [(s, an))).. Denoting the human policy as The the actual trajectories occurred during training are derived from shared behavior policy Ti:
(a(sss) Tu (1ls) I [(s, a)) + (als))((ss,
Sore I(s, a" d')=, (a'ss)da is the probability of the agent choosing an action that will be rejected by the human.
wherein G(s) =
3

---

Published as a conference paper at ICLR 2022
We call the transition sequences during the takeover {(s,, an,t: ah,t: On, ((+,), as the partial demonstration. The partial demonstration and the free exploration transitions will be recorded in the replay buffer B and fed to the training pipeline. Note that we do not require to store environmental reward and cost into the buffer since the proposed method does not need them.
In the human-AI copilot training, the human is obligated to guide the agent learning and safeguard the learning process by proactively taking over the control if necessary. This paradigm rules out the dispensable states and mitigates the safety concern in free exploration of RL and active imitation learning methods (Ross et al. 201T). Different from previous offline RL works training from fixed
dataset (Bojarski et al. 2016 Ho & Ermon, 2016 Reddy et al., 2019 Kumar et al., 2020 Fujimoto et al. 2018 Wu et al. 2019) where no closed loop feedback is accessible, the human- Al copilot training produces partial demonstrations that contains the necessary human knowledge to overcome dangerous situations into the learning. The copilot nature alleviates the distributional shift prob- lem, since the human intervenes when the agent performs suspicious behaviors, so that there is a continuity of the state visitation between the agent and the expert.
In next section, we will introduce how we instantiate the human-Al copilot paradigm with a human- efficient algorithm that can effectively optimize the agent toward safe and high-performing policy.
3.2 LEARNING OBJECTIVES
We form three objectives that fully utilize the human data: (1) Agent should maximize a proxy value function Qs,, a) which reflects human intentions on how to finish the task. (2) Agent should explore thoroughly to visit the state-action subspace permitted by the human. Concretely, we maximize the action distribution entropy ((()(-))). (3) Agent should maximize the level of automation and reduce human intervention. Episodic human intervention is estimated by intervention value function Q' (s, a) based on the step-wise intervention cost C(s, a). Thus the overall learning objective of HACO becomes:
max E ((Q(s.a) + H(T)) Q'ss (s.
We then discuss the practical implementation of aforementioned design goals.
Proxy value function. HACO follows reward-free setting so we can't estimate the expected state- action value based on ground-truth reward function defined by the environment. We instead es- timate proxy value function Q(s, a: σ) (ő is model parameters) that captures the ordinal pref- erence of human experts, which implicitly reflects human intentions. We utilize the conservative Q-larniing (Kumar et al. 2020) and form the optimization problem of the proxy value function as:
min
E (a, ((....))) ().
(ss ((ss,an))04 (Q(ss ani Φ) Q(s, ani [))].
(3)
The above optimization objective be interpreted as being optimistic to the human's action as and pessimistic to the agent's s action an- The proxy value function learns to represent the high-value state-action subspace preferred by the human expert.
Entropy regularization. If the learning agent visits human-preferable subspace insufficiently dur- ing free explorable sampling, the states evoking high proxy value are rarely encountered, making the back-propagation of the proxy value to preceding states difficult and thus damaging the learning. To encourage exploration, we adopt the entropy regularization technique in (Haarnoja et al. 2018) and forms auxiliary signal to update the proxy value function apart from Eq.
min
E (st,
ly - a(s,, ,))? y リ=?
E a eonaa (:10e1)))
IQ (Q5(41)a': a log nn (a'lst41))
(4)
wherein âe is the executed action at state st, d' denotes the delay updated parameter of the target network, y is the discount factor. Since the environment reward is not accessible to HACO, we remove the reward term in the update target y. Combining Eq. 3 and Eq. 4 the formal optimization objective of the proxy value function becomes:
min [5 Q(se, de: 0))² + I(Se, (Q((s, an,e; ((, Φ) Q(si, an.t: ())].
(5)
4

---

Published as a conference paper at ICLR 2022
Algorithm 1:
The workflow of HACO during training
Initialize an empty replay buffer B while Training not finished do while Episode is not terminated do
On. ~ [ (-((s) Retrieve agent's s action
((s,, (n,a) < Human expert decides whether to intervene by observing current state St
if (st; (n,t) is True then
On,t +- TT. (-18)) Retrieve human's action
Apply 0n,z to the environment
else
Apply On,t the environment
11
if I (sr, (n,t) is True and I 1ntt-), is False then C(ss, Un,t) & Compute intervention cost following Eq.
13
else
14
C(s an,t) + Set intervention cost zero Record St, On,t. T(se (n,s) and th,t (if (st, (n,,)) to the buffer B
16
Update proxy value Q, intervention value Q' and policy according to Eq. respectively
Eq.
7
Eq.
Reducing human interventions. Directly optimizing the agent policy according the proxy value function will lead to failure when evaluating the agent without human participation. This is because (ss,a) represents the proxy value of the mixed behavior policy #6 instead of the learning agent's Tn due to the existence of human intervention. It is possible that the agent learns to deliberately abuse human intervention by always taking actions that violate human intentions, such as driving off the road when near the boundary, which forces human to take over and provide demonstrations. In this case, the level of automation for the agent is low and the human subject exhausts provide demonstrations. Ablation study result in Table [[c) illustrates this phenomenon.
To economically utilize human budget and reduce the human interventions over time, we punish the agent action that triggers human intervention in a mild manner by using the cosine similarity between agent's action and human's is action as the intervention cost function the form below:
On an
C(s, an)= ||
an Tn(-))).
|||l|
The agent will receive large penalty only when its action is significantly different from the expert action terms of cosine similarity.
A straightforward form of C' is constant -1 when human expert issues intervention. However, we find that there usually exists temporal mismatch between human intervention and faulty actions so that the intervention cost is given to the agent at delayed time step 4 6. It is possible that the agent's action an,t+e is a correct action that saves the agent itself from dangers but is mistakenly marked faulty action that triggers human intervention. In the ablation study, find that using the constant cost raises inferior performance compared to the cosine similarity.
As shown in Line 1 11-14 of Algorithm [ we only yield non-zero intervention cost at the first step of human intervention. This is because the human intervention triggered by the exact action an, indicates this action violates the underlying intention of human at this moment. Minimizing the chance of those actions will increase the level of automation.
To improve the level of automation, we form additional intervention value function Q' (s, a) as the expected cumulative intervention cost, similar to estimating the state-action value in Q-learning through Bellman equation:
Q' (st, On.1) C(st, an,t) +
1Q' (5041,
$11 B.11-Baat11 1 x w (le)))
This value function is used to directly optimize the policy.
Learning policy. Using the entropy-regulare vention value function Q' a), we form the the policy improvement objective as:
proxy value function Q(s, a) as well as the inter-
max E [Q(s,, a.)) - a log nn (a, |1; 0) Q' an)l, an ~ nn (-)=:))) 0 sy4B"
(8)
5

---

Published as a conference paper at ICLR 2022
Figure 2: Examples of the safe driving environments from MetaDrive used in the experiments.
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Task. We focus on the driving task in this work. This is because driving is an important decision making problem with a huge social impact, where safety and training efficiency are critical. Since many researches on autonomous driving employ human in real vehicle (Bojarski et al.) 2016]] Kelly et al. 2019), the human safety and human cogmitive cost become practical challenges that limit the application of learning-base based methods in industries. Therefore, the driving task is an ideal benchmark for the human-Al AI copilot paradigm.
Simulator. Considering the potential risks of employing human subjects in physical experiments, we benchmark different approaches the driving simulator. We employ a lightweight driving sim- ulator MetaDrive (Li et al. 202T), which preserves the capacity to evaluate the safety and gener- alizability in unseen environments. The simulator is implemented based on Panda3D (Goslin & Mine 2004) and Bullet Engine that has high efficiency well accurate physics-based 3D ki- neties. MetaDrive uses procedural generation to synthesize an unlimited number of driving maps for the split of training and test sets, which useful benchmark the generalization capability of different approaches in the context of safe driving. Some generated driving scenes are presented in The simulator is also extremely efficient and flexible so that we can run the human-Al copilot experiment in real-time. Though we mainly describe the setting of MetaDrive in this section, we also experiment on CARLA (Dosovisskiy et al. 2017) simulator in Sec.
Fig. 2
Training Environment. In the simulator, the task for the agent is to steer the target vehicle with low-level control signal, namely acceleration, brake and steering, to reach the predefined destination and receive success flag. The ratio of episodes where the agent successfully reaches the destination called the success rate. To increase the difficulty of the task, we scatter obstacles randomly each driving scene such movable traffic vehicles, fixed traffic cones, and warning triangles.
The observation contains (1) the current states such as the steering, heading, velocity and relative distance to boundaries etc., (2) the navigation information that guides the vehicle toward the desti- nation, and (3) the surrounding information encoded by a vector of 240 Lidar-like distance measures of the nearby vehicles.
Though HACO does not receive environmental reward during training, we provide reward function train baseline methods and evaluate HACO in time. The reward function contains dense driving reward, speed reward and a sparse terminal reward. The driving reward measures the longi- tudinal movement toward destination. We also reward agent according to its velocity and give the sparse reward +20 when the agent arrives at the destination.
Each collision to the traffic vehicles or obstacles yields +1 environmental cost. Note that HACO can not access this cost during the training. This cost is used to train safe RL baselines as well as for testing the safety of trained policies. We term the episodic cost as safety violation which the measurement on the safety of a policy.
We invite the human expert to supervise the real-time exploration of the learning agent with hands on the steering wheel, as shown in the Fig. B.. When dangerous situation is going to happen, the human takes over the vehicle by pressing the paddle besides the wheel and starts controlling the vehicle by steering the wheel and stepping the pedals.
Split of training and test sets. Different from the conventional RL setting where the agent is trained and tested in the same fixed environment, we focus on evaluating the generalization performance through testing the trained agents in separated test environments. We split the driving scenes into the training set and test set with 50 different scenes in each set. After each training iteration, we roll out the learning agent without guardian in the test environments and record success rate and safety violation given by the environment and present it in Table

---

Published as a conference paper at ICLR 2022
Table 1: The performance of different approaches in safe driving benchmark.
Total Training
Training Data Usage
Test Return
Test Safety
Test
Category
Method
Safety Violation
Violation
Success Rate
Expert
Human
358.19 19
0.164061
0.98
RL
SAC-RS
2.76K 0.95K
1M
386.77
1:3.1
0.73 11.18
0.82 10.18
PPO-RS
24.34K
IM
335.39
#12.41
3.41 #1.11
0.69
SAC-Lag
1.84K 0.49K
1M
351.96
1:10.88
0.72 1:049
0.73 1:0.9
Safe RL
PPO-Lag
11.64K
+ 4.16K
IM
299.99 1:49,6
1.18 1:0.83
0.51 10.17
CPO
4.36K #2.22K
IM
194.06
108.88
1.71 1:100
0.21 1:0.9
Offline RL.
CQL
36K
156.4 +31.94
6.82 13.1
0.11 #0:07
BC
36K
101.63 16606
1.00 1045
0.01 #0033
IL
GAII.
3.70K #2.3K
36K
136.08
18.73
4.42 11.89
0.13 20.03
Human-in- the-loop
HG-DAgger
38.35
50K
111.87
2.38
0.04
IWR
77.38
50K
299.78
3.39
0.64
Ours
HACO
30.14 + 11.36
30K'
349.25
+ 11:45
0.79 0.31
0.83 + 0.04
During HACO training, in 8316 497.00 steps out of the total 30K steps the human expert intervenes and overwrites the agent's 's actions. The whole training takes about 50 minutes.
Implementation details. We conduct experiments on the driving simulator and implement algo- rithms using RLLib (Liang et al., 2018), an efficient distributed learning system. When training the baselines, we host 8 concurrent trials in an Nvidia GeForce RTX 2080 Ti GPU. Each trial con- sumes 2 CPUs with parallel rollout workers. Except numan-in-the-loop experiments, all baseline experiments are repeated 5 times with different random seeds. The main experiments of HACO is conducted on a local computer with an Nvidia GeForee RTX 2070 and repeat 3 times. The ablations and baseline human-in-the-loop experiments repeat once due the limited human budget. One human subject participates in each experiment. In all tables and figures, we provide the standard deviation if the experiments are repeated multiple runs with different random seeds. Information about other hyper-parameters is given in the Appendix.
4.2 BASELINE COMPARISON
We compare our method to vanilla RI. and Safe RI. methods which inject the human intention and constraint through the pre-defined reward function and cost function. We test native RL. methods, PPO (Schulman et al.) (017) and SAC (Haarnoja et al. 2018). with cost added to the reward as
auxiliary negative reward, called reward shaping (RS). Three common safe RL baselines Constraint
Policy Optimization (CPO) (Achiam et al. 2017), PPO-Lagrangian (tookee et al., 2020). SAC- Lagrangian (Ha et al. 2020) are evaluated.
Apart from the RL methods, also generate human demonstration dataset containing one-hour expert's 's demonstrations where there are about 36K transitions in the training environments. For the high-quality demonstrations in the dataset, the success rate of the episodes reaches 98% and the safety violation is down to 0.16. Using this dataset, we evaluate passive II. method Behavior Cloning, active IL method GAIL (Ho & Ermon 2016) and offline RL method CQL (Kumar al.) 2020). We also run the Human-Gated DAgger (HG-DAgger) (Kelly et al.) 2019) and Intervention Weighted Regression (IWR) (Mandlekar al. 2020) as the baselines of human-in-the-loon methods based on this dataset and the human-Al copilot workflow.
Training-time Safety. The training-tims safety is measured by the total training safety violation, the total number of critical failures occurring in the training. Note that the environmental cost here different from the human intervention cost HACO. As illustrated in Table I and Fig. A HACO achieves huge success in training time safety. Apart from the empirical results, we provide proof to show the training safety can be bound by the guardian in Appendix. Under the protection of the hu- man expert, HACO yields only 30.14 total safety violations in the whole training process, two orders of magnitude better than other RL baselines, even though HACO does not access the environmental cost. IWR and HG- -Agger also achieve drastically lower training safety violations, showing the power of human in-the- methods. The most competitive RL baseline SSC-RS, which achieves similar test success rate, causes averagely 2767.77 77 training safety violations which much higher
7

---

Published as a conference paper at ICLR 2022
Training time Safety Violation
Test Success Rate
1.0
S/G-Lgg
SAC-R5
0.8
H400
9.6
Tpoocod Roao
9.
Sampled
1e0
Tell
A. Baseline Comparison
B. Leamming Dynamics
Figure 3: A. The performance of RL methods and HACO. HACO achieves superior training time safety and test success rate. B. Dynamics of human cognitive cost. The takeover rate is the propor- tion of the human takeover steps in an episode. In the training process, the takeover rate and the episodic intervention cost gradually reduce and the agent obtains higher level of autonomy.
than HACO. The active IL method GAIL also has significantly higher safety violations than HACO and its performance is unsatisfactory.
From the perspective of safety, we find that the reward shaping technique is inferior compared to the Lagrangian method, both for SAC and PPO variants. PPO causes more violations than SAC, probably due the relatively lower sample efficiency and slower convergence speed.
Sample Efficiency and Human Cognitive Cost. The human-Al system is not only protected so well by the human, but achieves superior sample efficiency with limited data usage. As shown in Fig. 3 and Table I we find that HACO is an order of magnitude more efficient than RL baselines. HACO achieves 0.83 test success rate by merely interacting with the environment in the 30K steps, wherein only averagely 8,316 steps the human provides safe actions as demonstration. During nearly 50 minutes of human-Al copilot, there are only 27% steps that the human provides demonstrations.
Human-in in- the-loop baselines IWR and HG-DAgger consume 50K steps of human budget and only IWR can achieve satisfactory success rate. By prioritizing samples from human intervention, IWR manages to learn key actions from human intervention to escape dangerous situations caused by the compounding error. Without re- re-weighting the human takeover data, HG-Dagger fails to learn from few but important human demonstrations. The learning curves of these two methods can be found in the Appendix.
Unlike the success of HACO, all the learning-frowd demmontttnttiinonnn methods fail with the dataset containing 36K transitions. Compared to IL methods which optimize agents to imitate exact actions at each time step, HACO considers the learning on the trajectory basis. We incentivize the agent to choose an action that can bring potential return in future trajectory, instead of only mimicking the expert's behaviors at each step. On the other hand, HACO gathers expert data in online manner through human-AI copilot, which better mitigates the distributional shift severe in offline RL methods.
Learning Dynamics. The intervention minimization mechanism in HACO reduces human cognitive cost. As shown in Fig. BB, the takeover rate gradually decreases in the course of learning. The curve of episodic intervention cost suggests that the human intervention frequency becomes lower and the similarity between agent's action and human's action increases. We also provide visualization of the learned proxy value function in the Appendix, showing that the learning scheme of HACO can effectively encode human preference into the proxy values.
4.3 ABLATION STUDY
Takeover Policy Analysis. We request the human subjects to try two intervention strategies. The first is to take over in low frequency and produce long trajectory at each intervention. In this way the intervention cost becomes sparse. The other strategy is to intervene more frequently and provide fragmented demonstrations. In Table 2(a), the experiment shows that the proposed HACO works better with dense human intervention signals. Agent trained with long trajectories achieves inferior success rate and episodic reward than agents trained with dense intervention signals.
8

---

Published as a conference paper at ICLR 2022
Table 2: The test performance when ablating components in HACO. All experiments here are con-
ducted by human subjects. Some experiments only repeat once due to limited human budget.
Test Return
Test Cost
Test Success Rate
Experiment
(a) Human intervenes less frequently
220.03
0.867
0.397
(b) W/o cosine similarity intervention cost
23.23
0.42
0.00
(c) W/o intervention minimization
84.77
1.07
0.00
НACO
349.25 11.45
0.79 = 0.31
0.83 + 0.04
Table 3: Comparison between HACO and PPO. Agents are trained in CARLA town and tested in
the unseen CARLA town 2.
Algorithm
Test Safety Violation
Test Return
Test Success Rate
Train Samples
PPO
80.84
1591.00
0.35
500,000
HACO
11.84
1579.03
0.35
8,000
Cosine Similarity Cost Function. As shown in Table b), we replace the intervention cost function in Eq. to a constant value +1 if human intervention happens. We find the agent learns to stay in the spawn points and does not move at all in test time. As discussed in Sec. 3.2 it is possible that the human intervenes in incorrect timing. This makes agent fail to identify how to drive correctly. Using the negative cosine similarity to measure the divergence between agent and human's actions alleviates this phenomenon since the human intervention penalty is down-weighted when the agent provides action that adheres human intention.
Intervention Minimization. As shown in Table when removing the intervention minimization mechanism, the agent drives directly toward the boundary. This is because the agent learns to abuse human expert to take over all the time, which increases proxy values but causes consistent out-of- the-road failures in testing. This result shows the importance of intervention minimization.
CARLA Experiment. To test the generality of HACO, we run HACO in the CARLA simula- tor (Dosovitskiy et al.] 2017). We use the top-down semantic view provided by CARLA as the input and 3-layer CNN the feature extractor for HACO and the PPO baseline. For PPO, the reward follows the setting described in CARLA and is based on the velocity and the completion of the road. We train HACO (with a human expert) and PPO in CARLA town 1 and report the test per- formance in CARLA town 2. Table that the proposed HACO can be successfully deployed in the CARLA simulator with visual observation and achieve comparable results. Also, it train the driving agent with a new CNN feature-extractor in 10 minutes with only 8,000 samples in the environment. The video is available at: bttps:://ecisionffot github 1o/HACO//
CONCLUSION
We develop an efficient human-in-the- learning method, Human-Al Copilot Optimization (HACO), which trains agents from the human interventions and partial demonstrations. The method incorporates the human expert in the interaction between agent and environment to ensure safe and efficient exploration. The experiments on safe driving show that the proposed method achieves su- perior training-time safety, outperforming RL and IL baselines. Besides, it shows high sample efficiency for rapid learning. The constrained optimization technique used to prevent the agent from excessively exploiting the human expert, which also decreases the takeover frequency and saves valuable human budget.
One limitation of this work is that the trained agents behave conservatively compared to the agents from RL baselines. Aiming to ensure the training time safety of the copilot system, human expert typically slow the vehicle down to rescue it from risky situations. This makes the agent tend to drive slowly and exhibit behaviors such as frequent yielding in the intersection. In future work, we will explore the possibility of learning more sophisticated skills.
Acknowledgments This project was supported by the Centre for Perceptual and Interactive Intelli- gence (CPII) Ltd under InnoHK supported by the Innovation and Technology Commission.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
The proposed Human-Al Copilot Optimization algorithm aims at developing a new human-friendly human-in-the-loop training framework. We successfully increase the level of automation after human-efficient training. We believe this work has great positive social impact which advances the development of more intelligent AI systems that costs less human burdens.
We employ human subjects to participate in the experiments. Human subjects can stop the exper- iment if any discomfort happens. No human subjects were harmed the experiments since we test in the driving simulator. The human subjects earn hourly salary more than average in our community. Each experiment lasts near one hour. Human participants will rest at least three hours after one experiment. During training and data processing, no personal information is revealed in the collected dataset or the trained agents.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans. Agent-agnostic human-in-the. loop reinforcement learning. arXiv preprint arXiv:1701.000
2017.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning-Void Volume 70, pp. 22-31. JMLR. org, 2017.
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv: 1606. 160.00666, 2016.
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint drXiv:1902.0425: 2019.
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinchart, Sergey Levine, Florian Shkurti, and Ani- mesh Garg. Conservative safety critics for exploration. arXiv preprint arXiv:2010. 14497, 2020.
Michael Bloem and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse re- inforcement learning. 53rd IEEE conference on decision and control, pp. 4911-4916. IEEE, 2014.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv: 1604.0730 2016.
Rui Camacho and Donald Michie. Behavioral cloning a correction. Al Magazine, 16(2):92-92, 1995.
Thomas Cederborg, Ishaan Grover, Charles L Isbell, and Andrea Thomaz. Policy shaping with human teachers. In Twenty-Fourts International Joint Conference on Artificial Intelligence, 2015.
Paul Christiano, Jan Leike, Tom B Brown. Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint arXiv: 010.110064474441 2017.
Yuchen Cui and Scott Niekum. Active reward learning from critiques. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 6907-6914. IEEE, 2018.
Alexey Dosovitskiy, Germán Ros, Felipe Codevilla. Antonio M. López, and Vladlen Koltun. CARLA: an open urban driving simulator. CoRR, abs/1711.03938, 2017. URL http://
rxxiv. httpiv.rrgabbb//11ll1 0.3998
10

---

Published as a conference paper at ICLR 2022
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv:1710.11228, 2017.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv: 2018.
Javier Garcia and Fernando Fernández. A comprchensive survey safe reinforcement learning.
Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Mike Goslin and Mark R Mine. The panda3d graphics engine. Computer, 12-1114, 2004.
Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. Advances Neural Information Processing Systems, 34, 2021.
Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Confer- ence on Machine Learning, pp. 1861-1870, 2018. 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. arXiv preprint arXiv:1606.03476, 2016.
Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S. Brown, and Ken Goldberg, Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning, 2021.
Junning Huang, Sirui Xie, Jiankai Sun, Qiurui Ma, Chunxiao Liu, Dahua Lin, and Bolei Zhou. Learning a decision module by imitating driver's control behaviors. In Proceedings of the Con- ference on Robot Learning (CoRL), 2020.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Proceedings of the 32nd Inter- national Conference on Neural Information Processing Systems, pp. 8022-8034, 2018.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference Machine Learning. Citeseer, 2002.
Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J Kochenderfer. Hg-
dagger: Interactive imitation learning with human experts. In 2019 International Conference Robotics and Automation (ICRA), pp. 8077-8083. IEEE, 2019.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminator-actorr actticc Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv: 180900295. 2018.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04777 2020.
Quanyi Li, Zhenghao Peng, Zhenghai Xue, Qihang Zhang, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. 2021.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox. Ken Goldberg. Joseph Gon- zalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pp. 3053-3062, 2018.
Travis Mandel, Yun-En Liu, Emma Brunskill, and Zoran Popović. Where to add actions in human- in-the-loop reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 31, 2017.
Ajay Mandlekar, Danfei Xu, Roberto Martin-Martin Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Human-in in-the-loop imitation learning using remote teleoperation. arXiv preprint 0666:3012.06633,, 2020.
11

---

Published as a conference paper at ICLR 2022
Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, pp. 2, 2000.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. arXiv preprini arXiv: 1811:86111.1 2018.
Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward func- tions by integrating human demonstrations and preferences. arXiv preprint arXiv:1906.00 08928, 2019.
Zhenghao Peng, Quanyi Li, Chunxiao Liu, and Bolei Zhou. Safe driving via expert guided policy optimization. In 5th Annual Conference on Robot Learning. 2021.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. Robotics: Science and Systems, 2018.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint 19X1:1905..1108, 2019.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference artificial intelligence and stafistics, pp. 661-6666 JMLR Workshop and Conference Proceedings, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learn- ing of reward functions. UC Berkeley, 2017.
Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. In International conference on learning representations, 2018.
William Saunders, Girish Sastry, Andreas Stublmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. arXiv preprint arXiv:1707.05173, 2017.
William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. Proceedings of the 17th Interna- tional Conference Autonomous Agents and MultiAgent Systems, pp. 2067-2069. International Foundation for Autonomous Agents and Multiagent Systems, 2018
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: vrXiv:1707.000 1707.06347, 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2018.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. arXiv preprint 1rXiv:1807.09936 2018.
Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew Schmittle, Mung Chiang, Peter Ramadge, and Siddhartha Srinivasa. Learning from interventions. In Robotics: Science and Systems (RSS), 2020.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR, 2020.
Jiankai Sun, Hao Sun, Tian Han, and Bolei Zhou. Neuro-symbolic program search for autonomous driving decision module design. In Proceedings of the Conference on Robot Learning (CoRL), 2020.
12

---

Published as a conference paper at ICLR 2022
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high- nigh-dimensional state spaces. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Bernard Widrow. Pattern recognition and adaptive control. IEEE Transactions on Applications and Industry, 83(74))269-277, 1964.
Jingda Wu, Zhiyu Huang. Chao Huang. Zhongxu Hu, Peng Hang, Yang Xing, and Chen Lv. Human- in-the-loop deep reinforcement learning with application autonomous driving. arXiv preprint arXiv:2104.07246, 2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 2019.
Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual learning of control prim- itives: Skill discovery via reset-games, Advances in Neural Information Processing Systems, 33, 2020.
Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end autonomous driving. arXiv preprint 1605:::005.0660 2016.
Ruohan Zhang, Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in leveraging human guidance for sequential decision Decision-making making tasks. Autonomous Agents and Multi-Agent Sys- tems, 35(2):1-39, 2021.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
The demonstrative video and the code of HACO and baselines are provided at: https://
decisionforce. www.ooooorr......u....... Prithhb..o/HHACO.
A MAIN THEOREM AND THE PROOF
In this section, we derive the upper bound of the discounted probability of failure of HACO, showing that we can bound the training safety with the guardian.
Theorem 1 (Upper bound of training risk). The expected cumulative probability failure Va, of the behavior policy To in HACO is bounded by the error rate of the human expert action e, the error rate of the human expert intervention K. and the tolerance of the human expert K"
76
VA S 1
E 1
K'}
wherein K' max, K(s) max, (x()) da 2 0 is called human expert tolerance.
The human expert tolerance K' will becomes larger, if human relieves its intervention and allows the agent to explore the environment more freely.
The proof is given as follows.
Notations. Before starting, we firstly recap and describe the notations. HACO, a human subject copilots with the learning agent. The agent's policy is Tns the human's policy is The Both policies produces action in the bounded action space A E IR [RIA|, The human expert decides to intervention under certain state and agent's action a,, The human intervention denoted by a Boolean function: [ss, a). The mixed behavior policy Tu that produces the real actions applied to the environment is denoted as:
Th (a(s))) n. (1 [s, a)) + TI (a)s)
wherein G(s) = So e (ss a' TT (a')ss (a'ls)dd ssdd' is a function which denotes the probability of choosing an action that will be rejected by the human.
Therefore, at given state, we can split the action space into two parts: where intervention will happen or will not happen if the agent sample action in it. We denote the confident action space as:
(a = {a T(ass) is False].
(10)
The confident action space contains the actions that will not be rejected by human expert state
We also define the ground- -ruth indicator Cst denoting whether the action will lead to unsafe state. This unsafe state is determined by the environment and is not revealed to learning algorithm:
(19 = [1, if s' = P(s'(s, a) is an unsafe state, otherwise.
(11)
Therefore, at given state the step-wise probability of failure for arbitrary policy π is:
C9t (s [0.
(12)
a>=x(11s)
Now we denote the cumulative discounted probability of failure as:
Va
Σ *C" ar), T~N
E
(13)
which counts for the chance of entering dangerous states current time step as well as in future trajectories deduced by the policy We use V.., Er Vz, (So) denote the expected cu- mulative discounted probability of failure of the human. Following the same definition as Vz.. we also write the expected cumulative discounted probability of failure of the behavior policy Vn, Er-m, Vn, (so) Er, Σ Css (se, ar).
Assumption. Now introduce two important assumptions the human expert.
14

---

Published as a conference paper at ICLR 2022
Assumption 1 (Error rate of human action). For all states, the step-wise probability of that the human expert produces an unsafe action bounded by a small value e 1:
E a-oTT (-1s)
C (s, S.
(14)
Assumption (Error rate human intervention). For all states, the step-wise probability of that the human expert does not intervene when agent produces unsafe action is bounded by small value Kt < 1:
/ [1 1(s aEA
C (s, S
(15)
aEAA(()
These two assumptions does not impose any constrain the structure of the human expert policy, Lemmas. We propose several useful lemmas and the correspondent proofs, which are used in the main theorem.
Lemma (The performance difference lemma).
Vv Van +
EE
E anoi
[Ara (s, a)].
(16)
Here the Pas means the states are subject to the marginal state distribution deduced by the behav- ior policy Tb. Ax, (s.a) is the advantage of the expert current state action pair: As, (s, a) (19(s. a) + (s)) (s) and s' = (P(s, a) is the next state. This lemma is proposed and proved by Kakade & Langford (202) and is useful original proposition, the and A represents the expected discounted return and advantage w.r.. the reward, respectively. However, we replace the reward with indicator Cat so that the value function VU, and Vv,, presenting the expected cumulative failure probability.
show the behavior policy's safety. In the
Lemma 3. The cumulative probability failure of the expert V (s) bounded for all state:
Vxa (5) S 1-7
Proof.
Following Assumption !
Va, (st)
E nh
1' 1'9
(sr,
Σ
7/t (C )r)
Σ 11
- 1- 7
6
(17)
n'
Theorem. We introduce the main theorem of this work above, which shows that the training safety related the error rate on action and the error rate on intervention of the human expert. The proof is given as follows.
Proof. We firstly decompose the advantage by splitting the behavior policy:
A (8, / JOEA To(a|s)AAm (s,
E
amFa())s)
= =fc (mn(a))) - (() ()) (8, T
da
(18)
Jaed
=/e Socdite) (Tn(a)s) a))da + G(s) avxa
[Ama (s, a)).
15

---

Published as a conference paper at ICLR 2022
The second term is equal according the definition of advantage. We only need to compute the first term. We expand the advantage into detailed form, we have:
E Aza (s, a-17)
[Tn(a]s) (s,
AEAA())
Tx(als)[C**(s
(8, ~Van - Vin, (s)]da
=
(19)
aEAn(()
(s,
[(a (8))da
=(als) (s)da.
(EAn(()
aEAn(s)
(EAA(()
(b)
(c)
Following the Assumption 1 the term (a) can be bounded as:
/
((als)C9*ss (s, (s,a)da
(9, S k.
(20)
as)..(s)
aEAA((s)
Following the Lemma] the term (b) can be written as:
/ (a (s')da SY
Va (s')da 1
da
7E K(s),
(21)
1
(EAA(s)
aEAA(s)
aEAn(()
wherein ((s) = As) da denoting the area of human-prefene region in the action space. It a function related to the human expert and state.
The term (c) is always non-negative, so after applying the minus to term (c) the negative term will always be <
Aggregating the upper bounds of three terms, we have the bound on the advantage:
PE K(s)
EE ATt (8, S K +
(22)
1-7
Now we put Eq.
22 as well as Lemma into the performance difference lemma (Lemma we
have:
1
V=. Va,
[Ar, (s,
?
sn s amto
+
1s +
7E
max K [s)]
(23)
7E2
[c + +
1-7 K7
wherein K" max, K(s) max, Jae an (x) da the human expert has higher tolerance then K" should be greater.
correlated the tolerance of the expert. If
Now have proved the upper bound of the discounted probability of failure for the behavior policy in our method.
16

---

Published as a conference paper at ICLR 2022
B VISUALIZATION OF LEARNED PROXY VALUE FUNCTION
Figure 4: Visualization of proxy Q value learned by HACO.
To understand how well the proxy value function learns, we visualize common scenarios in pairs of figures shown above. The left sub-figure of each pair shows top-down view of driving scenario, where a sequence of snapshots of the control vehicle is plotted, showing its trajectory. The right sub-figure of each pair overlaps the heatmap of proxy values in the top-down image. We manually position the vehicle in different location in the map and query the policy to get action and run the proxy Q function to get the value Q(s, a). Region in red color indicates the proxy value is low if the agent locates there and vice versa.
In Fig. 4 [), the agent performs a lane change behavior avoid potential collisions with a traffic vehicle which is merging into the middle lane. The region near the traffic vehicle has extremely low values and thus the agent has small probability to enter this area.
In Fig. 4bb), traffic cones spread in the left lane. The agent learns avoid crashes and the proxy value heatmap shows a large region of low values.
As shown in the trajectory in Fig. Cc). though the agent can choose to bypass the traffic vehicle in both left-hand side or right-hand side, it chooses the right-hand side. The heatmap shows that much higher proxy Q value is produced on right bypassing path compared to left path. This behavior resembles the preference human who prefers hand side detour.
In addition, in some ares where paths boundary is ambiguous such as the intersection, the agent manages to learn a virtual boundary in the proxy Q space for efficiently passing these areas, as shown in the Fig. d).
The proxy Q value distribution shown in this section not only explains the avoidance behaviors, but also serves good indicator for the learned human preference.
17

---

Published as a conference paper at ICLR 2022
C
DETAILS OF HUMAN-IN-THE-LOO BASELINES
Test Success Rate
0.8
HG- DAgger
0.4
0.3
0.0
Data Usage
Figure 5: Detailed learning curves of human-in-the-loop baselines.
We benchmark the performance of two human -nteeeiittttttteeee loophhe-looppp methods HG-DAgger (Kelly al. 2019) and IWR (Mandlekar et al., 2020). Both methods require warming up through behavior clonigg on pre-collected dataset. In practice, is not enough to initialize the policy with basic driving skills. Therefore, we use the pre-collected human dataset containing 30K transitions warm up the policies. After warming up. HG-DAgger and IWR then aggregate human intervention data to the training buffer and conduct behavior cloning again update policy for epochs. each epoch the human-Al system collects 5000 transitions. The above figure shows the learning curves of IWR and HG-DAgger As discussed in the main body of paper, we credit the success of IWR to the re-weighting of human intervention data, which not emphasized in HG-DAgger
find that using 10K 20K steps of human collected data
D
MORE ZOOM-IN PLOT OF THE LEARNING CURVES
Training-ting time Safety Violation
Test Success Rate
PPO-Lag
PPO-Lag
PPO-RS
1.0
SAC-Lag
SAC-Lag
SAC-RS
0.9
SAC-RS
HACO
HACO
0.6
0.4
0.0
Sampled Steps
Sampled Steps
Figure 6: The zoomed in learning eurves of RL baselines and HACO.
The above figures present the zoomed in learning curves of RL baselines and HACO, showing the superior sample efficiency of HACO compared to RL baselines.
18

---

Published as a conference paper at ICLR 2022
E HYPER-PARAMETERS
Table 4: HACO
Table 5: PPO/PPO-Lag
Jlyper-paraeeee
Value
Hyper-parameter
Value
Discounted Factor 7 for Target Network Update
0.99
KL Coefficient
0.2
0.005
À for GAE (Schulman et al.
2018
0.95
Learning Rate
0.0001
Discounted Factor
0.99
Environmental Horizon 7
1000
Number of SGD epochs
20
Steps before Learning Start
100
Train Batch Size
4000
Steps per Iteration
100
SGD mini batch size
100
Train Batch Size
1024
Learning Rate
0.00005
CQL Loss Temperature
10.0
Clip Parameter
0.2
Target Entropy
2.0
Cost Limit for PPO-Lag
Table 6: SAC/SAC-Lag/COL
Hyper-parameter
Value
Table 7: BC
Discounted Factor for target network update
0.99
0.005
Hyper-parametea
Value
Learning Rate
0.0001
Environmental horizon
1500
Dataset Size
36,000
Steps before Learning start
10000
SGD Batch Size
32
SGD Epoch
200000
Cost Limit for SAC-Lag
Learning Rate
0.0001
BC iterations for CQL
200000
CQL Loss Temperature / 5
Min Q Weight Multiplier
0.2
Table 8: CPO
Table 9: GAIL
syper--aarmeter
Value
Hyper-parameter
Value
KL Coefficient
0.2
for GAE (Schulman et al. 2018
0.95
Dataset Size
36,000
Discounted Factor 7
0.99
SGD Batch Size
64
Number of SGD epochs
20
Sample Batch Size
12800
Train Batch Size
8000
Generator Learning Rate
0.0001
SGD mini batch size
100
Discriminator Learning Rate
0.005
Learning Rate
0,00005
Generator Optimization Epoch
5
Clip Parameter
0.2
Discriminator Optimization Epoch
2000
Clip Parameter
0.2
Cost Limit
Table 11: IWR
Table 10: HG-DAgger
Hyper-parameter
Value
Hyper-parameter
Value
Initializing dataset size
30K
Initializing dataset size
30K
Number of data aggregation epoch
4
Number of data aggregation epoch
4
Interactions per round
5000
Interactions per round
5000
SGD batch size Learning rate
256
SGD batch size Learning rate
256
0.0004
0.0004
Re-weight data distribution
True
19


Reviews:
[{"review_id": "tlP0Ju4SNTo", "reviewer": "Reviewer_ohEo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I enjoyed reading this paper and found its results convincing. However, there is one missing experimental comparison that I would like to see before I can recommend an accept: prior work on human-in-the-loop imitation learning [2] proposes the Intervention-Weighted Regression (IWR) method, which tackles the same problem under the same assumptions as HACO. While HACO directly penalizes the agent for triggering human interventions, IWR takes a different approach based on dataset balancing. To judge the relative performance and contribution of HACO, it would be helpful to implement and run IWR on the driving task in Table 1 and Figures 3-4.\n\nMissing related work:\n 1. [Learning from Interventions](http://www.roboticsproceedings.org/rss16/p055.pdf)\n 2. [Human-in-the-Loop Imitation Learning using Remote Teleoperation](https://arxiv.org/abs/2012.06733)\n\nMinor comments:\n - Is there a typo in Equation 3? As written, the objective evaluates to zero. I'm also confused as to why offline RL is necessary to fit the Q-function, since we do not have a reward function in this setting and instead have partial demonstrations. Can't we just use behavioral cloning to fit an imitation policy?\n - Why does the training cost go *up* over time in the left panel of Figure 3?\n\nUpdate\n----\nThank you to the authors for adding the comparison to IWR. I have increased my score.\n", "summary_of_the_paper": "The paper proposes HACO, a human-in-the-loop reinforcement learning method that safely trains an agent to imitate expert behavior while minimizing the number of expert interventions required. The key idea is to have a human watch over the agent (e.g., in a simulated driving environment), and take control whenever the agent enters unsafe states. HACO uses offline RL to train the agent to imitate the actions taken by the human during these interventions. To discourage the agent from intentionally visiting unsafe states in order to trigger human interventions, HACO also assigns a negative reward to the state transitions preceding a human intervention. Experiments with human participants in a simulated driving task show that HACO trains the agent to achieve higher success rates (in a test environment, without a human in the loop) than baseline methods based on imitation learning and offline RL, while requiring less training data and incurring a lower cumulative training cost.", "summary_of_the_review": "Missing crucial comparison to prior method", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NNK8DM-bWxN", "reviewer": "Reviewer_irh4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Quality\n\nThe paper's results are quite interesting, and the experiments – as far as I can tell – seem well executed. I particularly appreciated the additional ablations provided in Table 2. The method seems relatively well motivated. The method proposed seems to do very well relative to the chosen baselines along the performance metrics examined.\n\n### Clarity & Limitations\nStylistically the paper could use more editing. There are various typos (some below) and many phrasings that could be improved. \n\nThere are also various things that could be improved in the results' clarity, or which seem to constitute limitations:\n- Who is intervening? 3 experts – how were they selected? While the training is happening, what are the experts doing? \"The whole training takes about 50 minutes.\" Is the expert providing actions for the whole time?\n- \"The main experiments of HACO repeat 3 times.\" -> what does this mean? With each human expert, you perform one training run?\n- Generally would be helpful to give more intuition as to why IL is being outperformed with much less data?\n- \"We split the driving scenes into the training set and test set with 50 different scenes in each set. At the beginning of each episode, a scene in the training or test set is randomly selected.\" -> The latter part of this phrase seems to suggest that you're training on the test set?\n- I found it confusing how in Section 3.3, different costs and rewards were defined one by one, without a high-level motivation. Before describing each one, I would state clearly that you are considering _various different metrics of performance_: \"Test Return\", \"Test Cost\" (which is different than just negative returns!), and \"Test Success Rate\". I would potentially propose changing the name from \"Cost\" to \"Safety Violations\", and from \"Success Rate\" to \"Goal completion\" to make it easier to disambiguate what the motivation between using these multiple metrics is. As currently phrased, they all seem synonymous with \"reward\" (or the opposite of it) on a first read.\n- In Section 3.3, I would switch to introduce cosine similarity first, and then say that in ideal case this is not necessary. I found presenting things in the current order more confusing.\n- The sentence \"Frequent querying might exhaust the human expert and brings tremendous cognitive costs (Zhang et al., 2021)\" is almost verbatim repeated both in section 3 and in section 2. I would remove it from 2.\n- \"Under the protection from the human expert, HACO yields only 30.14 total training cost in the whole training process, an order of magnitude better than the other baselines,\" -> if I'm reading Table 1 correctly, it's essentially 2 orders of magnitude better? The next best value seems to be 1840\n- Figure 3: it might be useful to have zoomed in (on the x axis) training curves in the appendix. Are the shadings of the training curves standard errors?\n\nA limitation seems to be that of implicit assumptions made by the method: using CQL as in eq. (3) is making relatively strong assumptions about the nature of expert intervention. While it seems to work well in practice, the form of eq. (6) also seems quite arbitrary, and it's unclear how one would extend it to other environments (e.g. with discrete action spaces).\n\nAdditionally, the lack of human-in-the-loop baselines seems potentially problematic – after all, the method is presented as a improvement in that line of work. The same could be said about the environment – given how promising this method seems to be in this context, it would be very interesting to see how it performs in other environments. While I realize that this is not something that can be addressed in the timeframe of rebuttals, it would significantly strengthen the paper.\n\nTypos:\n- \"To encourage the exploration in the area that does not _violating_ human intention,\"\n- \"Human-Gated DAgger (HG-DAgger) (Kelly et al., 2019) utilizes an expert to _intervene exploration_ and\"\n- \"Other forms of human participation _includes_ providing human preferences\"\n- \"rewarding the state-action pairs deduced _by human_ and penalizing those _by agent_.\"\n- \"The speed reward _is vehicle’s_ current speed\"\n- \"Since we test different approaches in the driving simulator, no injury would _happens_.\"\n\n### Originality\n\nAlthough I'm not an expert in this area, as far as I know, this seems like a novel contribution.", "summary_of_the_paper": "In this work, the authors propose a new algorithm for data-efficient human-in-the-loop learning, Human-AI Copilot Optimization. The main idea is to have experts intervene during training in cases in which unsafe situations arise. The HACO learned policy utilizes a multi-task objective: doing well relative to a learned value function (based on human interventions), keeping an exploratory policy, and keeping human interventions at a minimum. Experimentally, HACO seems to be able to drastically reduce the amount of number of environment training timesteps required to reach basic competencies to the agent in the test environment, while maintaining good task and safety performance.", "summary_of_the_review": "Overall, the novel method presented by this paper, HACO, seems to be able to greatly outperform safe RL, IL, and Offline RL baselines – leading it to being able to achieve relatively similar safety and performance metrics while incurring in many less unsafe situations during training time, and while using 2 orders of magnitude less data than RL approaches. While some portions of the paper and motivations for the method lack clarity and baselining is limited to non human-in-the-loop methods, the strength of the empirical results still suggests that the submission is potentially worth accepting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GKP-nvMPzEE", "reviewer": "Reviewer_cwDu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=== Strengths ===\n\n+ The paper is well written and the presentation is easy to follow.\n+ The overall idea and the human-in-the-loop setup are nice and suit very well with real-world applications.\n+ The presented approach is simple and effective, and has strong experimental performance compared to IL and is on-par with RL, and does not require extra reward function. It is much more sample efficient than the common RL approaches.\n+ The paper also presents a novel driving simulator with procedurally generated maps and active agents. The authors mention that the simulator will be released, which is a big plus.\n\n=== Weaknesses ===\n\n- I like the analysis that justifies the design choice of equation 6, but I think the proposed solution is slightly overfitting to the environment (for example, steering which the environment uses), since in general action space does not necessarily form a metric space.\n- There is a typo in equation 3.\n- The authors mention the method builds upon CQL, however judging from equation 5 it does not seem to strictly follow CQL, but looks more like a standard Q learning loss. I would appreciate it if the authors can clarify this.\n", "summary_of_the_paper": "This paper presents HACO, a human-in-the-loop learning algorithm that aims to learn imitative driving policy while minimizing the number of human interventions. HACO builds on CQL and operates under the no-reward assumption. HACO learns a proxy action-value function by penalizing the policy’s action and maximizing during human interventions. It additionally adds an entropy term to encourage exploration. The policy trains by maximizing the proxy action-value, and penalizing an accumulative intervention cost, computed using the cosine difference between the human and the policy actions. HACO is evaluated in a closed-loop driving simulator. HACO outperforms the selected imitation and offline RL baseline and is on par with RL methods which have access to environment rewards. It is also orders of magnitude more sample efficient than standard RL methods.", "summary_of_the_review": "This paper presents a simple yet very effective human-in-the-loop learning algorithm. The idea of minimizing human intervention is nice, and the presented approach has strong experimental performance. The novel simulator on which the presented approach is evaluated will be released, which is another plus. I recommend acceptance for this paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DpWcddM0Uov", "reviewer": "Reviewer_Mao2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper is well written with clear logic and goals. The experiments are comprehensive. The proposed method is compared with several types of baselines including RL, safe RL, offline RL, and IL. \n2. The method proposed for co-learning and intervention minimization is solid. \n\nWeakness:\n1. The description of the method can be improved. There seem to be some errors in the equations. For example, in equation (3), the two Q(s, \\hat{a}) are the same. \n2. I am not sure how to determine the convergence of the algorithm. Why HACO has much fewer steps than the baselines. Especially during the testing phase, why HACO also has fewer steps?\n3. The main part of the methodology is based on batch RL. The adding of intervention minimization seems to be a marginal contribution. \n4. The driving tasks are considered to be too easy. ", "summary_of_the_paper": "This paper proposed a method for driving policy learning based on human-AI copilot. The algorithm learns from human interventions and also tries to minimize the total efforts of human intervention. Comprehensive experiments and comparisons with multiple baselines show that the proposed algorithm can achieve high sample efficiency and reduce unsafe events. The contributions are in the design of the copilot learning method.", "summary_of_the_review": "The paper is solid and clear, with enough experimental support. I think it is higher than the bar. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new algorithm for augmenting RL training with human examples, and this is applied to learning safe driving policies. This algorithm is properly tested and compared to other relevant algorithms with favorable results. Reviewers agree that this is good work and that it should be published. Reviewers had multiple questions, which were in my opinion answered satisfactorily by the authors. Notably, the authors ran additional tests against other human-in-the-loop RL algorithms with good results. In sum, this seems to be a solid paper, worth accepting.
Decision: Accept (Poster)