Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling

Imitation learning , followed by reinforcement learning algorithms , is a promising paradigm to solve complex control tasks sample-efficiently . However , learning from demonstrations often suffers from the covariate shift problem , which results in cascading errors of the learned policy . We introduce a notion of conservativelyextrapolated value functions , which provably lead to policies with self-correction . We design an algorithm Value Iteration with Negative Sampling ( VINS ) that practically learns such value functions with conservative extrapolation . We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks . We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm , which is shown to outperform prior works in sample efficiency . 1 INTRODUCTION . Reinforcement learning ( RL ) algorithms , especially with sparse rewards , often require a large amount of trial-and-errors . Imitation learning from a small number of demonstrations followed by RL finetuning is a promising paradigm to improve the sample efficiency ( Rajeswaran et al. , 2017 ; Večerík et al. , 2017 ; Hester et al. , 2018 ; Nair et al. , 2018 ; Gao et al. , 2018 ) . The key technical challenge of learning from demonstrations is the covariate shift : the distribution of the states visited by the demonstrations often has a low-dimensional support ; however , knowledge learned from this distribution may not necessarily transfer to other distributions of interests . This phenomenon applies to both learning the policy and the value function . The policy learned from behavioral cloning has compounding errors after we execute the policy for multiple steps and reach unseen states ( Bagnell , 2015 ; Ross & Bagnell , 2010 ) . The value function learned from the demonstrations can also extrapolate falsely to unseen states . See Figure 1a for an illustration of the false extrapolation in a toy environment . We develop an algorithm that learns a value function that extrapolates to unseen states more conservatively , as an approach to attack the optimistic extrapolation problem ( Fujimoto et al. , 2018a ) . Consider a state s in the demonstration and its nearby state s̃ that is not in the demonstration . The key intuition is that s̃ should have a lower value than s , because otherwise s̃ likely should have been visited by the demonstrations in the first place . If a value function has this property for most of the pair ( s , s̃ ) of this type , the corresponding policy will tend to correct its errors by driving back to the demonstration states because the demonstration states have locally higher values . We formalize the intuition in Section 4 by defining the so-called conservatively-extrapolated value function , which is guaranteed to induce a policy that stays close to the demonstrations states ( Theorem 4.4 ) . In Section 5 , we design a practical algorithm for learning the conservatively-extrapolated value function by a negative sampling technique inspired by work on learning embeddings Mikolov et al . ( 2013 ) ; Gutmann & Hyvärinen ( 2012 ) . We also learn a dynamical model by standard supervised learning so that we compute actions by maximizing the values of the predicted next states . This algorithm does not use any additional environment interactions , and we show that it empirically helps correct errors of the behavioral cloning policy . When additional environment interactions are available , we use the learned value function and the dynamical model to initialize an RL algorithm . This approach relieves the inefficiency in the prior work ( Hester et al. , 2018 ; Nair et al. , 2018 ; Rajeswaran et al. , 2017 ) that the randomly-initialized Q functions require a significant amount of time and samples to be warmed up , even though the initial policy already has a non-trivial success rate . Empirically , the proposed algorithm outperforms the prior work in the number of environment interactions needed to achieve near-optimal success rate . In summary , our main contributions are : 1 ) we formalize the notion of values functions with conservative extrapolation which are proved to induce policies that stay close to demonstration states and achieve near-optimal performances , 2 ) we propose the algorithm Value Iteration with Negative Sampling ( VINS ) that outperforms behavioral cloning on three simulated robotics benchmark tasks with sparse rewards , and 3 ) we show that initializing an RL algorithm from VINS outperforms prior work in sample efficiency on the same set of benchmark tasks . 2 RELATED WORK . Imitation learning . Imitation learning is commonly adopted as a standard approach in robotics ( Pomerleau , 1989 ; Schaal , 1997 ; Argall et al. , 2009 ; Osa et al. , 2017 ; Ye & Alterovitz , 2017 ; Aleotti & Caselli , 2006 ; Lawitzky et al. , 2012 ; Torabi et al. , 2018 ; Le et al. , 2017 ; 2018 ) and many other areas such as playing games ( Mnih et al. , 2013 ) . Behavioral cloning ( Bain & Sommut , 1999 ) is one of the underlying central approaches . See Osa et al . ( 2018 ) for a thorough survey and more references therein . If we are allowed to access an expert policy ( instead of trajectories ) or an approximate value function , in the training time or in the phase of collecting demonstrations , then , stronger algorithms can be designed , such as DAgger ( Ross et al. , 2011 ) , AggreVaTe ( Ross & Bagnell , 2014 ) , AggreVaTeD ( Sun et al. , 2017 ) , DART ( Laskey et al. , 2017 ) , THOR Sun et al . ( 2018a ) . Our setting is that we have only clean demonstrations trajectories and a sparse reward ( but we still hope to learn the self-correctable policy . ) Ho & Ermon ( 2016 ) ; Wang et al . ( 2017 ) ; Schroecker et al . ( 2018 ) successfully combine generative models in the setting where a large amount of environment interaction without rewards are allowed . The sample efficiency of ( Ho & Ermon , 2016 ) has been improved in various ways , including maximum mean discrepancy minimization ( Kim & Park , 2018 ) , a Bayesian formulation of GAIL ( Jeon et al. , 2018 ) , using an off-policy RL algorithm and solving reward bias problem ( Kostrikov et al. , 2018 ) , and bypassing the learning of reward function ( Sasaki et al. , 2018 ) . By contrast , we would like to minimize the amount of environment interactions needed , but are allowed to access a sparse reward . The work ( Schroecker & Isbell , 2017 ) also aims to learn policies that can stay close to the demonstration sets , but through a quite different approach of estimating the true MAP estimate of the policy . The algorithm also requires environment interactions , whereas one of our main goals is to improve upon behavioral cloning without any environment interactions . Inverse reinforcement learning ( e.g. , see ( Abbeel & Ng , 2004 ; Ng et al. , 2000 ; Ziebart et al. , 2008 ; Finn et al. , 2016a ; b ; Fu et al. , 2017 ) ) is another important and successful line of ideas for imitation learning . It relates to our approach in the sense that it aims to learn a reward function that the expert is optimizing . In contrast , we construct a model to learn the value function ( of the trivial sparse reward R ( s , a ) = −1 ) , rather than the reward function . Some of these works ( e.g. , ( Finn et al. , 2016a ; b ; Fu et al. , 2017 ) ) use techniques that are reminiscent of negative sampling or contrastive learning , although unlike our methods , they use “ negative samples ” that are sampled from the environments . Leveraging demonstrations for sample-efficient reinforcement learning . Demonstrations have been widely used to improve the efficiency of RL ( Kim et al. , 2013 ; Chemali & Lazaric , 2015 ; Piot et al. , 2014 ; Sasaki et al. , 2018 ) , and a common paradigm for continuous state and action space is to initialize with RL algorithms with a good policy or Q function ( Rajeswaran et al. , 2017 ; Nair et al. , 2018 ; Večerík et al. , 2017 ; Hester et al. , 2018 ; Gao et al. , 2018 ) . We experimentally compare with the previous state-of-the-art algorithm in Nair et al . ( 2018 ) on the same type of tasks . Gao et al . ( 2018 ) has introduced soft version of actor-critic to tackle the false extrapolation of Q in the argument of a when the action space is discrete . In contrast , we deal with the extrapolation of the states in a continuous state and action space . Model-based reinforcement learning . Even though we will learn a dynamical model in our algorithms , we do not use it to generate fictitious samples for planning . Instead , the learned dynamics are only used in combination with the value function to get a Q function . Therefore , we do not consider our algorithm as model-based techniques . We refer to ( Kurutach et al. , 2018 ; Clavera et al. , 2018 ; Sun et al. , 2018b ; Chua et al. , 2018 ; Sanchez-Gonzalez et al. , 2018 ; Pascanu et al. , 2017 ; Khansari-Zadeh & Billard , 2011 ; Luo et al. , 2018 ) and the reference therein for recent work on model-based RL . Off-policy reinforcement learning There is a large body of prior works in the domain of off-policy RL , including extensions of policy gradient ( Gu et al. , 2016 ; Degris et al. , 2012 ; Wang et al. , 2016 ) or Q-learning ( Watkins & Dayan , 1992 ; Haarnoja et al. , 2018 ; Munos et al. , 2016 ) . Fujimoto et al . ( 2018a ) propose to solve off-policy reinforcement learning by constraining the action space , and Fujimoto et al . ( 2018c ) use double Q-learning ( Van Hasselt et al. , 2016 ) to alleviate the optimistic extrapolation issue . In contrast , our method adjusts the erroneously extrapolated value function by explicitly penalizing the unseen states ( which is customized to the particular demonstration offpolicy data ) . For most of the off-policy methods , their convergence are based on the assumption of visiting each state-action pair sufficiently many times . In the learning from demonstration setting , the demonstrations states are highly biased or structured ; thus off-policy method may not be able to learn much from the demonstrations . 3 PROBLEM SETUP AND CHALLENGES . We consider a setting with a deterministic MDP with continuous state and action space , and sparse rewards . Let S = Rd be the state space andA = Rk be the action space , and letM ? : Rd×Rk → Rd be the deterministic dynamics . At test time , a random initial state s0 is generated from some distribution Ds0 . We assume Ds0 has a low-dimensional bounded support because typically initial states have special structures . We aim to find a policy π such that executing π from state s0 will lead to a set of goal states G. All the goal states are terminal states , and we run the policy for at most T steps if none of the goal states is reached . Let τ = ( s0 , a1 , s1 , . . . , ) be the trajectory obtained by executing a deterministic policy π from s0 , where at = π ( st ) , and st+1 = M ? ( st , at ) . The success rate of the policy π is defined as succ ( π ) = E [ 1 { ∃t ≤ T , st ∈ G } ] ( 3.1 ) where the expectation is taken over the randomness of s0 . Note that the problem comes with a natural sparse reward : R ( s , a ) = −1 for every s and a . This will encourage reaching the goal with as small number of steps as possible : the total payoff of a trajectory is equal to negative the number of steps if the trajectory succeeds , or −T otherwise . Let πe be an expert policy 1 from which a set of n demonstrations are sampled . Concretely , n independent initial states { s ( i ) 0 } ni=1 from Ds0 are generated , and the expert executes πe to collect a set of n trajectories { τ ( i ) } ni=1 . We only have the access to the trajectories but not the expert policy itself . We will design algorithms for two different settings : Imitation learning without environment interactions : The goal is to learn a policy π from the demonstration trajectories { τ ( i ) } ni=1 without having any additional interactions with the environment . Leveraging demonstrations in reinforcement learning : Here , in addition to the demonstrations , we can also interact with the environment ( by sampling s0 ∼ Ds0 and executing a policy ) and observe if the trajectory reaches the goal . We aim is to minimize the amount of environment interactions by efficiently leveraging the demonstrations . Let U be the set of states that can be visited by the demonstration policy from a random state s0 with positive probability . Throughout this paper , we consider the situation where the set U is only a small subset or a low-dimensional manifold of the entire state space . This is typical for continuous state space control problems in robotics , because the expert policy may only visit a very special kind of states that are the most efficient for reaching the goal . For example , in the toy example in Figure 1 , the set U only contains those entries with black edges.2 To put our theoretical motivation in Section 4 into context , next we summarize a few challenges of imitation learning that are particularly caused by that U is only a small subset of the state space . Cascading errors for behavioral cloning . As pointed out by Bagnell ( 2015 ) ; Ross & Bagnell ( 2010 ) , the errors of the policy can compound into a long sequence of mistakes and in the worst case cascade quadratically in the number of time steps T . From a statistical point of view , the fundamental issue is that the distribution of the states that a learned policy may encounter is different from the demonstration state distribution . Concretely , the behavioral cloning πBC performs well on the states in U but not on those states far away from U . However , small errors of the learned policy can drive the state to leave U , and then the errors compound as we move further and further away from U . As shown in Section 4 , our key idea is to design policies that correct themselves to stay close to the set U . Degeneracy in learning value or Q functions from only demonstrations . When U is a small subset or a low-dimensional manifold of the state space , off-policy evaluation of V πe and Qπe is fundamentally problematic in the following sense . The expert policy πe is not uniquely defined outside U because any arbitrary extension of πe outside U would not affect the performance of the expert policy ( because those states outside U will never be visited by πe from s0 ∼ Ds0 ) . As a result , the value function V πe and Qπe is not uniquely defined outside U . In Section 4 , we will propose a conservative extrapolation of the value function that encourages the policy to stay close to U . Fitting Qπe is in fact even more problematic . We refer to Section A for detailed discussions and why our approach can alleviate the problem . Success and challenges of initializing RL with imitation learning . A successful paradigm for sample-efficient RL is to initialize the RL policy by some coarse imitation learning algorithm such as BC ( Rajeswaran et al. , 2017 ; Večerík et al. , 2017 ; Hester et al. , 2018 ; Nair et al. , 2018 ; Gao et al. , 2018 ) . However , the authors suspect that the method can still be improved , because the value function or the Q function are only randomly initialized so that many samples are burned to warm them up . As alluded before and shown in Section 4 , we will propose a way to learn a value function from the demonstrations so that the following RL algorithm can be initialized by a policy , value function , and Q function ( which is a composition of value and dynamical model ) and thus converge faster .

Summary: This work presents the value iteration with negative sampling (VINS) algorithm, a method for accelerating reinforcement learning using expert demonstrations.  In addition to learning an expert policy through behavioral cloning, VINS learns an initial value function which is biased to assign smaller expected values to states not encountered during demonstrations.  This is done by augmenting the demonstration data with states that have been randomly perturbed, and penalizing the value targets for these states by a factor proportional to their Euclidean distance to the original state.  In addition to the policy and value function, VINS also learns a one-step dynamics model used to select actions against the learned value function.  As the value function learned in VINS is only defined with respect to the current state, action values are estimated by sampling future states using the learned model, and computing the value of these sampled states.