Deep Q Learning from Dynamic Demonstration with Behavioral Cloning

1 INTRODUCTION . Deep reinforcement learning ( DRL ) methods have made great progress ( Mnih et al. , 2013 ; 2015 ; Silver et al. , 2017 ) when applied in several rule-based applications such as the Go game ( Silver et al. , 2016 ) . However , due to the diversity and uncertainty of complex systems , the establishment of a simulation environment is difficult to be consistent with the real-world system . Therefore , DRL algorithms usually fail in a direct application to many real-world scenarios . Meanwhile , a DRL model may produce an action that is sampled from a random policy when exploring the state-action space . However , random actions are not allowed in many real-world circumstances . For example , in autonomous driving experiments ( Kiran et al. , 2020 ) , a random policy may contribute to traffic congestion , even road accidents . Therefore , fitting to complex situations becomes one of the most urgent tasks for applying a DRL model for complicated decision-making tasks . It is noted that human experts have great advantages in learning efficiency and decision-making performance ( Tsividis et al. , 2017 ) . Incorporating expert knowledge is a potential solution to enhance the adaptability of DRL models for complex tasks ( Hester et al. , 2018 ; Matas et al. , 2018 ) . Nevertheless , the knowledge and experience of an expert are difficult to be modeled and described directly . One solution , attracting more and more attention , is to indirectly learn expert strategies by learning their decision trajectories , also known as demonstrations ( Schaal , 1997 ; Behbahani et al. , 2019 ; Ravichandar et al. , 2020 ) . Particularly , deep Q learning from demonstrations ( DQfD ) is a typical algorithm that succeeded in combining DRL with demonstrations ( Hester et al. , 2018 ) , which combines the temporal difference ( TD ) error in the traditional DDQN algorithm with supervised expert loss by constructing a hybrid loss function . Through a specially designed large margin supervised loss function ( Piot et al. , 2014a ; b ) , the DQfD method can guide and assist an agent to learn the expert ’ s knowledge by constantly steering the agent learning strategies closer to those represented by the demonstration . However , the DQfD model suffers from three major issues : ( 1 ) In the DQfD learning process , the trajectory data in the historical demonstration dataset is the only source for contributing expert loss values , which does not include the self-generated transitions of the trained agents . As a result , the DQfD algorithm merely relies on TD errors to improve the policy but the demonstration is idle when the self-generated transitions are sampled from the experience replay buffer , which reduces the efficiency of utilizing demonstrations . ( 2 ) According to the learning mechanism , static demonstrations are too limited to cover sufficient state-action space during the agent training process , especially when it is difficult or expensive to collect demonstrations in real-world applications . Also , with more newly-generated transitions added into the experience replay buffer , historical demonstrations would make smaller contributions to the policy improvement as their sampling probability becomes lower and lower . ( 3 ) The DQfD algorithm requires the learned policy to approximate the demonstration but ignores the imperfection of historical demonstrations and it is an imperfect demonstration that commonly exists in real-world applications . Perfect demonstration can always provide an appropriate guide but the imperfect demonstration is detrimental to the improvement of the policy when the learned policy is superior to the policy represented by the imperfect demonstration . To address the above issues , we propose a novel deep Q learning from dynamic demonstration with the behavioral cloning ( DQfDD-BC ) method . Fig . 1 illustrates the structure of the proposed approach . DQfDD-BC shares the same basic components of the DDQN algorithm ( Hasselt et al. , 2016 ) , including two Q networks and an experience replay buffer . There are an additional imitation learning ( IL ) ( Hussein et al. , 2017 ) module and an alterable demonstration in the replay buffer . Besides , the loss value is calculated considering both the output of the trained Q network and the BC model . Two main contributions are summarized as follows : ( 1 ) An IL model , named behavioral cloning ( BC ) ( Torabi et al. , 2018 ; Bühler et al. , 2020 ) , is proposed to generate expert loss to utilize all the transitions in the experience replay buffer . It first attempts to extract the experts ’ policy from the initial demonstration and allows the agent to provide reasonable actions when facing newly generated states . During the self-learning process , the agent ’ s actions are compared with those generated by the BC model through a self-designed expert loss function . The inclusion of the BC model allows the knowledge in the demonstrations to be sufficiently utilized in the training process and enables the model to cope with the states which experts have not ever encountered . The supervised learning process and self-learning process can promote each other . The supervised model provides a basic reference to guide the model adjustment . Meanwhile , the self-learning process keeps improving and overcomes the limitation of the BC model and suboptimal samples during the learning process . ( 2 ) An automatic update mechanism is proposed to adaptively enhance the BC model . In particular , new transitions are generated by the trained agents to fine-tune the BC model if the model achieves a relatively high-performance score . Such a mechanism tries to include more high-quality transition samples to improve the demonstration and avoids potential adverse impacts caused by imperfect demonstrations . In this study , we evaluate the proposed DQfDD-BC method on several gym environments ( Brockman et al. , 2016 ) . For comparison purposes , the DDQN and DQfD methods are used as the baselines ( Hasselt et al. , 2016 ; Hester et al. , 2018 ) . The experiments clearly demonstrate that DQfDD-BC surpasses all the baselines in terms of convergence speed as well as decision-making performance . The ablation experiments also show that both the proposed expert loss function together with the BC model and dynamic demonstration contribute significantly to the performance superior to the DQfDD-BC algorithm . 2 RELATED WORKS . In most cases , demonstration comes from the human or other kinds of ” experts ” , who can provide valuable information for different hard problems , such as robot control ( Schaal , 1997 ; Ravichandar et al. , 2020 ) , self-driving ( Bojarski et al. , 2016 ) and so on . As a kind of approaches which can make use of the demonstration , behavioral cloning ( Torabi et al. , 2018 ) , one of imitation learning ( Hussein et al. , 2017 ; Brown et al. , 2020 ) , has received extensive attention ( Zhang et al. , 2018 ; Bühler et al. , 2020 ) due to its significant advantages , such as fast learning speed , simplicity , and effective utilization of demonstration and so on . It constructs mappings from states to actions ( or distributions of actions ) in a supervised manner to model the policies represented by the demonstrations and achieves the goal of learning from a demonstration by minimizing various supervised losses . In a study by NVIDIA , the model obtained interesting results by minimizing the mean squared error between the steering command output by the network and the command of a human driver ( Bojarski et al. , 2016 ) . Chauffeurnet further enhances the performance of imitation learning in autopilot by synthesizing the worst scenarios ( Bansal et al. , 2018 ) . DAgger ( Ross et al. , 2011 ) is proposed to cope with the covariate shift problem and experts are required to respond to self-generated states so that totally new transitions are generated and they can expand the state-action space covered by demonstration . The goal of performance improvement can be achieved by adding new samples continuously during the learning process . However , the DAgger method requires an always-available expert to assist in labeling the data , which reduces the practicality of the method . Deeply AggreVaTeD ( Sun et al. , 2017 ) enables the DAgger to handle continuous action spaces by using deep neural networks but the weakness of DAgger was also preserved . Another typical class of imitation learning algorithms is generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ; Wang et al. , 2019 ) . Instead of mapping from state to action directly , GAIL learns from the demonstrations by introducing the generator and discriminator . The generator is used to generate state-action pairs and learn a policy , while the discriminator is used to distinguish whether a state-action pair is from the expert or the learned policy . The information contained in the demonstration is indirectly combined with policy improvement through the process of adversarial learning . GAIL shows good performance on the high-dimensional continuous-control problem ( Kang et al. , 2018 ; Song et al. , 2018 ) . Policy optimization with demonstrations ( POfD ) ( Kang et al. , 2018 ) utilizes the demonstration by an adversarial learning method and has made efforts on sparse and dense reward environments . Recently , demonstrations have been leveraged to tutor DRL to achieve better performance . Since the DRL reward is immediate feedback from the environment for evaluating the performance of the policy , reshaping the reward function through the demonstration is an effective approach to improve DRL performance ( Brys et al. , 2015 ; Suay et al. , 2016 ) . By transforming the form of the reward function , the difficulty of agent training has been reduced . Model-Based DRL ( MBRL ) has shown great power in some difficult tasks ( Kaiser et al. , 2019 ) and demonstration is also leveraged to deal with the MBRL problems ( Lambert et al. , 2019 ; Thananjeyan et al. , 2020 ) . All of the above methods yielded impressive results but most of them utilized demonstrations through reward shaping or imitating the demonstration . Another idea is to learn from demonstrations directly to improve the policy . The transitions in the experience replay buffer and demonstrations are both extracted by DRL agents ( Hester et al. , 2018 ; Oh et al. , 2018 ; Xiong et al. , 2019 ) . These approaches put the demonstration into the experience replay buffer and sample from the hybrid experience replay buffer . The DQfD algorithm enhances the DDQN ( Hasselt et al. , 2016 ) algorithm by keeping the demonstration in the replay buffer at all times . A designed supervised expert loss is generated when the transitions in the demonstration are sampled to update the parameters of the neural networks . The self-imitation learning ( SIL ) method chooses different loss functions and adds new trajectories to the demonstration . It updates the same neural network twice with the A2C loss function and the SIL loss function ( Oh et al. , 2018 ) . The DDPGfD algorithm ( Vecerik et al. , 2017 ) is similar to the DQfD and they both assist in enhancing the original algorithms through a hybrid loss function with the ” expert loss ” . The difference between the two is that the DDPGfD algorithm is based on the DDPG algorithm ( Lillicrap et al. , 2015 ) , which is designed to solve continuous action space problems . However , both DQfD and DDPGfD suffer from the problem of under-exploiting demonstration data ( Kang et al. , 2018 ) and our DQfDD-BC method has made new and comprehensive progress on solving the problem .

Summary: This paper proposes integrating deep Q-learning from dynamic demonstrations with a behavioral cloning model (DQfDD-BC). Compared with DQfD, the proposed approach introduces a behavior cloning model, which was first pre-trained by leveraging historical demonstrations and then updated using generated dynamic demonstration. The BC model is used in the expert loss function, where the DRL model's actions are compared with those obtained from the BC model for policy improvement guidance. The experimental results in OpenAI Gym environments show that the proposed approach adapts well to different demonstrations' imperfection levels and accelerates the learning processes. The ablation study also indicates that the new method improves the learning convergence performance compared with the original DQfD model.