Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization

1 INTRODUCTION . How to effectively inject human knowledge into the learning process is one of the key challenges to training reliable autonomous agents in safety-critical applications . In reinforcement learning ( RL ) , researchers can inject their intentions into the carefully designed reward function . The learning agent freely explores the environment to collect the data and develops the desired behaviors induced by the reward function . However , RL methods bear two drawbacks that limit their applications in safety-critical tasks : First , the nature of trial-and-error exploration exposes RL agent to dangerous situations ( Saunders et al. , 2017 ) . Second , it is difficult to summarize all the intended behaviors to be learned into the reward function . Taking the driving vehicle as an example , an ideal policy should obtain a set of skills , such as overtaking , yielding , emergent stopping , and negotiation with other vehicles . It is intractable to manually design a reward function that leads to the emergence of all those behaviors in the trained agent . To mitigate these two challenges , practitioners enforce the human intentions through imitation learning ( IL ) where the agent is trained to imitate the expert-generated state and action sequences . During the demonstration , the premature agent does not interact with the risky environment and thus the training safety is ensured . High-quality expert demonstrations provide direct the optimal solution for the agent to imitate from . However , IL paradigm suffers from the distributional shift problem ( Ross & Bagnell , 2010 ; Ross et al. , 2011 ) while the induced skills are not sufficiently robust with respect to changes in the control task ( Camacho & Michie , 1995 ) . Different from vanilla RL or IL , human-in-the-loop learning is an alternative paradigm to inject human knowledge , where a human subject accompanies the agent and oversees its learning process . Previous works require the human to either passively advise which action is good ( Mandel et al. , 2017 ) or evaluate the collected trajectories ( Christiano et al. , 2017 ; Guan et al. , 2021 ; Reddy et al. , ∗Quanyi Li and Zhenghao Peng contribute equally to this work . 2018 ; Warnell et al. , 2018 ; Christiano et al. , 2017 ; Sadigh et al. , 2017 ; Palan et al. , 2019 ) . This kind of passive human involvement exposes the human-AI system to risks since the agent explores the environment without protection . Some other works require the human to merely intervene in the exploration by terminating the episode ( Saunders et al. , 2018 ; Zhang & Cho , 2016 ) , but it is not practical to terminate and reset the environment instantly in the real world ( Xu et al. , 2020 ) . Intervening and taking over the control from the learning agent is a natural approach to safeguard the human-AI system ( Kelly et al. , 2019 ; Spencer et al. , 2020 ) . However , a challenge exhibited in previous works is the budget of human intervention . Since human cognitive resource is precious and limited , it is essential to carefully design when and how the human expert involves in the learning process so that the human knowledge can be injected effectively . In this work , we propose an efficient human-in-the-loop learning method called Human-AI Copilot Optimization ( HACO ) . The key feature of HACO is that it can learn to minimize the human intervention and adjust the level of automation to the learning agent adaptively during the training . As shown in Figure 1 A , HACO allows the human expert to take over the human-AI system in a proactive manner . If the human decides to intervene in the action of the agent , he/she should demonstrate the correct actions to overcome current undesired situations to the learning agent . The human intervention and the partial demonstration are two sources of informative training data . We use offline RL technique to maintain a proxy value function of the human-AI mixed behavior policy even though the agent doesn ’ t have the access to the environmental reward during training . To encourage the exploration in the state-action space permitted by human , we also maximize the entropy of action distribution of the agent if the agent is not taken over . Experiments in the virtual driving environments MetaDrive ( Li et al. , 2021 ) and CARLA ( Dosovitskiy et al. , 2017 ) show that , with an economic human budget , HACO outperforms RL and IL baselines with a substantial margin in terms of sample efficiency , performance , safety , and generalizability in the unseen testing environment . Thus the human-AI copilot optimization is an efficient learning paradigm to inject human knowledge in an online setting . 2 RELATED WORK . Learning from Demonstration . Passive imitation learning such as behavior cloning ( Widrow , 1964 ; Osa et al. , 2018 ; Huang et al. , 2020 ; Sun et al. , 2020 ) and recently proposed offline RL methods ( Kumar et al. , 2020 ; Fujimoto et al. , 2018 ; Wu et al. , 2019 ) train agents from an outof-the-shelf data set and guarantee the training safety , since no interaction with the environment is needed . Inverse RL methods ( Ng et al. , 2000 ; Abbeel & Ng , 2004 ; Fu et al. , 2017 ; Bloem & Bambos , 2014 ) learn a reward function from the human demonstration and then use it to incentivize the agents to master the intended behaviors . Proposed more recently , GAIL ( Ho & Ermon , 2016 ) and its variants ( Song et al. , 2018 ; Sasaki et al. , 2018 ; Kostrikov et al. , 2018 ) and SQIL ( Reddy et al. , 2019 ) compare the trajectory similarity between agents and humans and thus require the agent to interact with the environment . Similar to RL methods , this paradigm exposes the agent to potentially dangerous situations . Human-in-the-loop Learning Methods . Many works focus on incorporating human in the training loop of conventional RL or IL paradigms . DAgger ( Ross et al. , 2011 ) and its extended methods ( Kelly et al. , 2019 ; Zhang & Cho , 2016 ; Hoque et al. , 2021 ) correct the compounding error ( Ross & Bagnell , 2010 ) of behavior cloning by periodically requesting expert to provide more demonstration . Instead of proving demonstration upon requests , Human-Gated DAgger ( HG-DAgger ) ( Kelly et al. , 2019 ) , Expert Intervention Learning ( EIL ) ( Spencer et al. , 2020 ) and Intervention Weighted Regression ( IWR ) ( Mandlekar et al. , 2020 ) empower the expert to intervene exploration and carry the agent to safe states . However , these methods do not impose constraints to reduce human intervention and do not utilize the data from the free exploration of the agent . Human subjects can also involve in the loop providing preferences based on evaluative feedback on two behavior sequences generated by the agent ( Christiano et al. , 2017 ; Sadigh et al. , 2017 ; Palan et al. , 2019 ; Ibarz et al. , 2018 ; Cui & Niekum , 2018 ) . Human-AI copilot or shared autonomy is a more intimate form of the human-in-the-loop methods . The AI agent and human are working together simultaneously to achieve a common goal . By giving human guidance and feedback at run-time instantly , the explorable state and action spaces can be greatly narrowed down ( Saunders et al. , 2018 ) . The learning goal can further match the task objective by providing extra human feedback combined with reward function ( Reddy et al. , 2018 ; Warnell et al. , 2018 ; Wu et al. , 2021 ; Cederborg et al. , 2015 ; Arumugam et al. , 2019 ) . HumanAI copilot is helpful and practical when applying RL to real world tasks where safety constraints must be satisfied ( Garcıa & Fernández , 2015 ; Amodei et al. , 2016 ; Bharadhwaj et al. , 2020 ) . In our previous work ( Peng et al. , 2021 ) , we made attempt to develop a method called Expert-Guided Policy Optimization ( EGPO ) where a PPO expert policy is involved to monitor the learning agent . The difference can be summarized as twofold : ( 1 ) We substitute the expert with human and design special mechanism to mitigate the delayed feedback error ; ( 2 ) Based on the comprehensive ablation study and prototyping , we remove redundant designs like takeover function and the need of reward function , making the proposed method simple yet effective . Reducing human burden is a major challenge in human-in-the-loop methods . A feasible solution is to learn an intervention function that imitates human intervention signals and stops the catastrophic actions of agents ( Kelly et al. , 2019 ; Zhang & Cho , 2016 ; Saunders et al. , 2017 ; Abel et al. , 2017 ) , which can relieve the mental stress of the human subject during training . In this work , we devise our learning scheme explicitly to include the human cognitive cost as one of the objectives to minimize . 3 HUMAN-AI COPILOT OPTIMIZATION . In this section , we introduce Human-AI Copilot Optimization ( HACO ) , an efficient learning algorithm that trains agents from human interventions , partial demonstrations and free exploration . For human-in-the-loop learning , it is essential to design when and how to engage human subjects . The major issue is the cognitive cost of the human subject ( Zhang et al. , 2021 ) . Frequent querying might bring tremendous cognitive cost and exhaust the human expert , causing incorrect or delayed feedback that hinders the training . Thus the proposed pipeline aims to minimize the human intervention cost during the training , which reduces the reliance on the expert ’ s demonstration over time and improves the learning agent ’ s autonomy . The overall workflow of HACO is presented in Algorithm 1 . 3.1 HUMAN-AI COPILOT TRAINING PARADIGM . We aim to learn an autonomous agent with policy πn ( an|s ) that can make informed action an in state s. As shown in Fig . 1 , we frame the human-AI copilot paradigm that extends the standard reinforcement learning diagram by incorporating a human expert . At each step , the human expert oversees current state and decides whether to intervene . If necessary , he/she will execute human action ah to overwrite the agent ’ s action an . We denote the human intervention by a Boolean indicator I ( s , an ) and thus the action applied to the environment is called the safe action â = I ( s , an ) ah + ( 1 − I ( s , an ) ) an . Denoting the human policy as πh , the actual trajectories occurred during training are derived from a shared behavior policy πb : πb ( a|s ) = πn ( a|s ) ( 1− I ( s , a ) ) + πh ( a|s ) G ( s ) , ( 1 ) wherein G ( s ) = ∫ a′∈A I ( s , a ′ ) πn ( a ′|s ) da′ is the probability of the agent choosing an action that will be rejected by the human . We call the transition sequences during the takeover { ( st , an , t , ah , t , I ( st , an , t ) , st+1 ) , ... } as the partial demonstration . The partial demonstration and the free exploration transitions will be recorded in the replay buffer B and fed to the training pipeline . Note that we do not require to store environmental reward and cost into the buffer since the proposed method does not need them . In the human-AI copilot training , the human is obligated to guide the agent learning and safeguard the learning process by proactively taking over the control if necessary . This paradigm rules out the dispensable states and mitigates the safety concern in free exploration of RL and active imitation learning methods ( Ross et al. , 2011 ) . Different from previous offline RL works training from fixed dataset ( Bojarski et al. , 2016 ; Ho & Ermon , 2016 ; Reddy et al. , 2019 ; Kumar et al. , 2020 ; Fujimoto et al. , 2018 ; Wu et al. , 2019 ) where no closed loop feedback is accessible , the human-AI copilot training produces partial demonstrations that contains the necessary human knowledge to overcome dangerous situations into the learning . The copilot nature alleviates the distributional shift problem , since the human intervenes when the agent performs suspicious behaviors , so that there is a continuity of the state visitation between the agent and the expert . In next section , we will introduce how we instantiate the human-AI copilot paradigm with a humanefficient algorithm that can effectively optimize the agent toward safe and high-performing policy .

Summary: This paper presents HACO, a human-in-the-loop learning algorithm that aims to learn imitative driving policy while minimizing the number of human interventions. HACO builds on CQL and operates under the no-reward assumption. HACO learns a proxy action-value function by penalizing the policy’s action and maximizing during human interventions. It additionally adds an entropy term to encourage exploration. The policy trains by maximizing the proxy action-value, and penalizing an accumulative intervention cost, computed using the cosine difference between the human and the policy actions. HACO is evaluated in a closed-loop driving simulator. HACO outperforms the selected imitation and offline RL baseline and is on par with RL methods which have access to environment rewards. It is also orders of magnitude more sample efficient than standard RL methods.