Demonstration Actor Critic

1 INTRODUCTION . Reinforcement Learning ( RL ) aims at solving sequential decision-making problems by learning through interacting with environments in a trail-and-error way . In many real scenarios , the existence of expert demonstrations has been well perceived as a critical value to enhance the capability of reinforcement learning algorithms . Recent years have witnessed many studies exploring the paradigm of learning from demonstration ( LfD ) , which provides the learner with some demonstration data generated by expert policies . However , LfD yields a strong dependency on the assumption of demonstration optimality , which is usually inconsistent with the reality . To better integrate LfD with reinforcement learning , increasing efforts turn to reinforcement learning from demonstrations ( RLfD ) , with a relaxation to the demonstration optimality assumption , which can lead to significantly boost sample efficiency of the RL process . One major branch of RLfD proposes to leverage demonstration data in a supervised manner , by either using them to directly pretrain the policy ( Silver et al. , 2016 ) or supplement the learning target of the policy with a supervised objective when encountering the states in demonstration data ( Rajeswaran et al. , 2017a ) . Although appealingly simple and direct , such branch of RLfD unfortunately fails to fully exploit demonstration data as it can only provide supervision signal over those states observed in the demonstrations ( Brys et al. , 2015 ; Rajeswaran et al. , 2017a ; Reddy et al. , 2019 ) . To deal with such problem , another major branch of RLfD takes advantage of the demonstrations in reward shaping , by either designing the demonstration-oriented potential-based reward shaping function ( Brys et al. , 2015 ; Sun et al. , 2018 ) , or inducing implicit dynamic reward shaping through learning a discriminator from demonstrations , which can distinguish between demonstrations and self-generated data ( Zhu et al. , 2018 ; Kang et al. , 2018 ) . These methods can provide guidance on how to take actions , even for those states are not seen in the demonstrations . Particularly , these methods train an RL agent not only to imitate demonstrated actions when it encounters the demonstrated states , but also to reach demonstrated states , when it confronts states that are not observed in the demonstration data 1 ( Ho & Ermon , 2016 ; Reddy et al. , 2019 ; Wang et al. , 2019 ) . This is the core idea behind these reshaping reward based approaches . However , since the new adopted shaping reward yields no direct dependence on the current policy , this branch of methods , updating policy over demonstrated states in the same way as others by the reshaped value function , overlook the validity of such direct supervision for demonstrated states when learning the policy . In order to provide both guidance for all states as well as direct supervision for demonstrated states , we propose a new objective function with policy-dependent shaping reward . To demonstrate the theoretic soundness of this approach , we first present a convergence proof for policy iteration of the proposed objective , under the tabular setting given the assumption of the existence of an expert policy πE ( a|s ) . Furthermore , to cope with the problem of missing explicit expression of πE ( a|s ) in reality , we develop a new practical algorithm , called Demonstration Actor Critic ( DAC ) , by making several approximations that can be implemented using deep neural networks . Intuitively , if the current state is not included in the demonstration , the agent will learn to update the policy merely relying on the reshaped Q-value function . Otherwise , the agent will take advantage of both expert information and the reshaped Q-value function to update the policy . To demonstrate the effectiveness of our algorithm , we conduct experiments on the continuous physical locomotion tasks based on Mujoco ( Todorov et al. , 2012 ) in sparse-reward environments . In comparison with five strong and off-the-shelf baselines , the empirical results clearly show that our new DAC approach can attain consistent and significant improvements . Considering the recent concerns on reproducibility ( Henderson et al. , 2017 ) , all of our reported results are based on experiments run across a large number of seeds . The main contributions of this paper are summarized as : • We introduce a novel RLfD objective with policy-dependent shaping reward , which provide both guidance for all states as well as direct supervision signal over demonstrated states . • We derive a Demonstration Policy Iteration method with guaranteed convergence , under the tabular setting , by assuming the existence of the expert policy πE .. • We develop Demonstration Actor Critic ( DAC ) , a new practical algorithm to learn the policy for the continuous setting , given the missing expert policy in reality . • We conduct empirical experiments in a couple of popular continuous tasks in sparse-reward environments to demonstrate the advantage of DAC as it consistently outperforms state-ofthe-art baselines . 2 RELATED WORK . There is a growing interest in combining learning from demonstration ( LfD ) with reinforcement learning ( RL ) . Recently there are three popular approaches under this problem setting : 1 ) utilizing demonstration data by adopting value-based RL algorithms ; 2 ) leveraging demonstration data in a supervised manner ; 3 ) using the demonstrations to reshape the original reward function . For the first approach , they adopt value-based RL algorithms to utilize the demonstration data . Kim et al . ( 2013 ) proposes Approximate Policy Iteration with demonstrations ( APID ) , which uses expert demonstrations to define linear constraints that guide the optimization of Approximate Policy Iteration ( API ) . Piot et al . ( 2014 ) builds on a similar idea but integrate expert constraint directly into the minimization of the optimal Bellman residual ( OBR ) . Following this line , Chemali & Lazaric ( 2015 ) bases on classification-based policy iteration and proposes Direct Policy Iteration with Demonstrations ( DPID ) . More recently , thanks for the development of deep learning , DQfD ( Todd et al. , 2018 ) introduces LfD into DQN ( Mnih et al. , 2015 ) , using the same additional structured classification loss than previous works . Besides , DQfD also adds demonstration data into the replay buffer in the same way as self-generated data . It employs a refined priority replay mechanism ( Schaul et al. , 2016 ) and assigns additional priority to demonstration data . However , these methods are limited by applications with discrete action spaces , due to the usage of max operator over the 1Although current state-action pair ( st , at ) encountered by the agent may trigger low immediate reward r ( st , at ) due to its unexposure in the demonstration data , the long-term reward Q ( st , at ) is still likely to be high , especially if the agent can confront the demonstrated state-action pairs in later steps , so as to provides more reasonable guidance over state st. whole action space . DDPGfD ( Vecerik et al. , 2017 ; Nair et al. , 2017 ; Vecerik et al. , 2019 ) , which is built upon DDPG ( Lillicrap et al. , 2015 ) , extends DQfD to continuous action domain . Moreover , NAC ( Gao et al. , 2018 ) uses a unified loss function to process both off-line demonstrations and on-line experience based on the maximum entropy reinforcement learning framework . Nonetheless , treating demonstration data in the same way as self-generated experience usually requires a tremendous number of high-quality demonstration , which are difficult to collect at scale , as discussed in ( Kang et al. , 2018 ) . For the second approach , they attempt to leverage the demonstration data in a supervised manner . For instance , Silver et al . ( 2016 ) proposes to pre-train the policy with the demonstration data as a policy initialization step for further reinforcement learning , and Rajeswaran et al . ( 2017b ) augments the original policy loss with a behavior cloning loss during the policy training . Although appealingly simple and direct , such methods can only provide accurate supervision signal over those states that have been seen in the demonstrations . For the third approach , they pursue to reshape the original reward function in order to align with the experience from the demonstrations . Specifically , Brys et al . ( 2015 ) introduces a reward reshaping mechanism by defining a heuristic potential function based on non-normalized multi-variate Gaussian . Besides , Sun et al . ( 2018 ) uses expert ’ s value function as reward shaping , under the assumption of access to a reward-to-go oracle that provides an estimate of expert reward-to-go during training . Furthermore , Kang et al . ( 2018 ) introduces an implicit reward shaping via a parameterized discriminator , which aims to distinguish the demonstrated state-action pairs from self-generated pairs , and learn the policy with policy gradient methods . These methods can encourage the agent not only to imitate demonstrated actions , but also to visit demonstrated states . However , since the new adopted shaping reward yields no direct dependence on the current policy , this branch of methods pay rare attention to the validity of such direct supervision with respect to demonstrated states . To address this problem in the following of this paper , we develop an algorithm that can both provide guidance on all states and directly exploit the supervision signal on demonstrated states . As one of the most popular IL algorithms , Generative Adversarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) trains a discriminator to distinguish whether a state-action pair is from the expert or the learned policy . Meanwhile , GAIL optimizes the policy by maximizing expected return with respect to the reward function , which is based on that discriminator . Though effective for imitation learning , GAIL can not leverage the valuable reward signal given by the environment and may suffer from declining performance when the demonstration data is imperfect . By contrast , our algorithm can overcome such inherent limitation by introducing reward signals from the environment into the training process . Regarding the usage of discriminator , GAIL trains a discriminator to distinguish expert state-action pairs from other state-action pairs , while our method uses a discriminator to distinguish expert actions from other actions given expert states , which is totally different . Some of previous works ( Peters et al. , 2010 ; Azar et al. , 2012 ; Schulman et al. , 2015 ; Haarnoja et al. , 2017 ; Neu et al. , 2017 ; Abdolmaleki et al. , 2018 ; Haarnoja et al. , 2018 ; Geist et al. , 2019 ) studied the entropy regularization MDPs . Although there exist some similarities between our method and these works in terms of formulation , these methods are not very suitable for the RLfD problem studied in this paper . Particularly , these methods do not encourage the agent to reach demonstrated states ( states visited by the expert strategy ) explicitly , but it is a very important unique property of RLfD problem itself ( Ho & Ermon , 2016 ; Kang et al. , 2018 ; Reddy et al. , 2019 ) . By contrast , in our work , we delicately design a new policy-dependent shaping reward , in order to not only imitate demonstrated actions over these demonstrated states , but also reach demonstrated states , specifically for the RLfD problem . In addition , most of these works assume that the explicit expression of initial policy is available , but in our case , we can only access to expert demonstrations . To this end , we use the GAN technique to replace the necessary of explicit expression of πE , and take advantage of support estimation techniques to estimate the indicator function of suppπE ( s ) , which also leads to an obvious difference between these entropy regularization works and our work .

Summary: This paper proposes to mix reinforcement learning and imitation learning to boost the learning of an actor critic architecture. The authors use a regularized reward function that minimizes the divergence between the policy of the expert and the one followed by the agent. They use demonstrations obtained from a trained agent and experiment their method on several mujoco tasks.