Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration

1 INTRODUCTION . Reinforcement Learning ( RL ) is being considered for application to a variety of real-world settings that have very large state-action spaces , but even under the availability of accurate simulators determining how best to explore the environment is a major challenge . Oftentimes , the problem setting is such that intuitively obvious reward functions are quite sparse . Such rewards typically correspond to the attainment of a particular task , such as a robot attaining designated way points , with little fine-grain reward feedback on the intermediate steps taken to complete these tasks . The sparsity of rewards implies that existing RL approaches often do not make much headway into learning viable policies unless provided with carefully hand-tuned reward feedback by a domain expert . Many of these problems correspond to systems where data has been gathered over time using an empirically determined ( sub-optimal ) behavior policy , which contains information important to help bootstrap learning . An additional caveat is that this behavior data might only contain measurements of a subset of the true state . Is it possible to design an algorithm that has principled policy optimization for efficient learning , while also being able to explore by utilizing the data derived from a behavior policy ? A natural candidate to begin with is the framework of policy gradient algorithms ( Schulman et al. , 2015 ; Lillicrap et al. , 2016 ; Haarnoja et al. , 2018 ) , which performs really well in the dense reward setting . Can such a policy gradient algorithm be aligned with a behavior policy for guided exploration that is critical in the sparse reward setting ? In this work , we propose a principled approach for using the well known trust region policy optimization ( TRPO ) algorithm ( Schulman et al. , 2015 ) with offline demonstration data for guidance in a sparse reward setting . Our choice of the TRPO approach is motivated by its analytical tractability and superior performance . Our key insight is that we can utilize the trust region approach while being guided by the behavior data by virtue of two steps . The first step is identical to traditional TRPO to generate a candidate policy . In the second step , the objective is to find a policy closest to the behavior policy , subject to it being in the trust region of the candidate generated in the first step . Thus , the second step ensures that the policy chosen is always guided by the behavior policy , but the level of alignment with the behavior policy can be reduced by shrinking the trust region over time . We call our approach Learning Online with Guidance Offline ( LOGO ) to capture this two-step procedure . This principled approach enables us to provide analytical performance guarantees while achieving superior performance in a number of benchmark problems . Our main results are as follows : ( i ) LOGO can guarantee a performance improvement as long as the behavior policy is able to offer an advantage . Reducing the alignment with the behavior policy over time allows us to extract this advantage and then smoothly move beyond to find near-optimal policies , ( ii ) we provide a generalized version of the Performance Difference Lemma ( Kakade & Langford , 2002 ) wherein the stage reward function can depend on the policy itself , and use this result to determine a surrogate objective function for step 2 of LOGO that is straightforward to evaluate . This allows us to implement LOGO in the manner of two TRPO-like steps , enabling us to leverage the TRPO code base , ( iii ) we show on standard MuJoCo environments that LOGO trained with sparse rewards can attain nearly the same performance as an optimal algorithm trained with dense rewards . We go further and show that this excellent performance carries over to waypoint tracking by a robot over Gazebo simulations and real-world TurtleBot experiments , ( iv ) finally , we show that LOGO can also be used in the case where the demonstration data only contains a censored version of the true state ( incomplete state information ) by simply adding a projection to the available subset of the state space in step 2 of the algorithm . Again , we provide supporting evidence via excellence in both MuJoCo simulations , as well as obstacle avoidance by a TurtleBot that is trained with a behavior policy without a Lidar ( censored state ) , but is tested with it ( full state ) . 1.1 RELATED WORK . Our work is mainly related to two RL research areas : Imitation Learning ( IL ) : The goal of an IL algorithm is to imitate an ( expert ) policy using the demonstration data generated by that policy . Behavior cloning ( BC ) is a simple IL approach where the expert policy is estimated from the demonstration data using supervised learning . BC algorithms , however , suffer from the problem of distribution shift ( Ross et al. , 2011 ) . Inverse reinforcement learning ( IRL ) algorithms ( Ng & Russell , 2000 ; Ziebart et al. , 2008 ) estimate a reward function from the demonstration data and solve a forward RL problem using this reward function . Generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ) avoids the reward estimation problem by formulating the IL problem as a distribution matching problem , and provides an implicit reward for the RL algorithm using a discriminator ( Goodfellow et al. , 2014 ) . Most IL algorithms do not use reward feedback from the environment , and hence are restricted to the performance of the policy that generated the demonstration data . Our approach is different from pure IL , and we leverage online RL with ( sparse ) reward feedback for efficient learning . Learning from Demonstration ( LfD ) : The key idea of the LfD algorithms is to use demonstration data to aid online learning ( Schaal et al. , 1997 ) . Many works propose to exploit demonstration data by adding it to the replay buffer with a prioritized replay mechanism to accelerate learning ( Hester et al. , 2018 ; Vecerik et al. , 2017 ; Nair et al. , 2018 ) . Rajeswaran et al . ( 2018 ) combine a policy gradient algorithm with demonstration data by using a mix of behavior cloning and online RL fine tuning . Nair et al . ( 2020 ) propose AWAC algorithm to accelerate online RL by leveraging large amounts of offline data with associated rewards . Different from this , LOGO does not need the reward observations . Moreover , we give provable guarantees on the performance of LOGO whereas AWAC does not have any such provable guarantee ( further details are provided in Appendix G ) . Kim et al . ( 2013 ) propose a framework to integrate LfD and approximate policy iteration by formulating a coupled constraint convex optimization problem , where the expert demonstrations define a set of linear constraints . This approach is , however , limited to small problems with a discrete action space . The closest to our work is the PofD algorithm proposed by Kang et al . ( 2018 ) . PofD modifies the reward function by taking a weighted combination of the sparse reward from the online interaction and an implicit reward obtained from the demonstration data using a discriminator . Very different from this , we propose an intuitive and principled approach of using the offline demonstration data for guiding the online exploration during the initial phase of learning . Our two step approach enables us to provide a rigorous performance guarantee for the proposed LOGO algorithm , and to leverage trust region-based approaches to solve high dimensional problems with even sparser settings . We provide an extensive comparison between our algorithm and PofD in Section 5 . Offline RL : Recently , there has been many interesting works in the area of offline RL ( Kumar et al. , 2019 ; Fujimoto et al. , 2019 ; Siegel et al. , 2020 ; Wu et al. , 2019a ) which focus on learning a policy using only the offline data without any online learning or online fine-tuning . Different from these , LOGO is an online RL algorithm ( further details are provided in Appendix G ) . 2 PRELIMINARIES AND PROBLEM SETTING . 2.1 PRELIMINARIES . A Markov Decision Process ( MDP ) is denoted as a tuple < S , A , R , P , γ > , where S is the state space , A is the action space , R : S ×A → R is the reward function , P : S ×A× S → [ 0 , 1 ] is the transition probability function with P ( s′|s , a ) giving the probability of transitioning to state s′ when action a is taken at state s , and γ ∈ ( 0 , 1 ) is the discount factor . A policy π is a mapping from S to probability distribution over A , with π ( s , a ) specifying the probability of taking action a in state s. A policy π can generate state-action trajectory τ , where τ = ( s0 , a0 , s1 , a1 , . . . ) , s0 ∼ µ , at ∼ π ( st , · ) , st+1 ∼ P ( ·|st , at ) and µ is the initial state distribution . The infinite horizon discounted return of policy π is defined as JR ( π ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) ] . The goal of a reinforcement learning algorithm is to learn the optimal policy π ? = arg maxπ JR ( π ) . The value function of a policy π defined as V πR ( s ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] , is the expected cumulative discounted reward obtained by following the policy π starting from the state s. The action-value function of a policy π is defined similarly as QπR ( s , a ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , a0 = a ] . The advantage function is defined as AπR ( s , a ) = QπR ( s , a ) − V πR ( s ) . The discounted state visitation distribution for the policy π , denoted as dπ , is defined as dπ ( s ) = ( 1− γ ) ∑∞ t=0 γ tP ( st = s|π ) , where the probability is defined with respect to the randomness induced by π , P and µ. Definitions and Notations : The Kullback-Leibler ( KL ) divergence between two distribution p and q is defined as DKL ( p , q ) = ∑ x p ( x ) log p ( x ) q ( x ) . The average KL divergence between two polices π1 and π2 with respect to dπ1 is defined as Dπ1KL ( π1 , π2 ) = Es∼dπ1 [ DKL ( π1 ( s , · ) , π2 ( s , · ) ) ] . The maximum KL divergence between two polices π1 and π2 is defined as DmaxKL ( π1 , π2 ) = maxs DKL ( π1 ( s , · ) , π2 ( s , · ) ) . The total variation ( TV ) distance between two distribution p and q is defined asDTV ( p , q ) = ( 1/2 ) ∑ x |p ( x ) −q ( x ) | . The average TV distance and the maximum TV distance between two polices π1 and π2 are defined as Dπ1TV ( π1 , π2 ) = Es∼dπ1 [ DTV ( π1 ( s , · ) , π2 ( s , · ) ) ] and DmaxTV ( π1 , π2 ) = maxs DTV ( π1 ( s , · ) , π2 ( s , · ) ) , respectively .

Summary: This paper presents ‘LOGO’, an extension of the TRPO algorithm which enables additional learning guidance from offline, sub-optimal (possibly incomplete observation) demonstration data. By annealing away the learning contribution from the sub-optimal guidance policy data during training (with a learning schedule and corresponding hyper-parameter), LOGO utilises this data for guidance, rather than directly imitating it. Furthermore, because LOGO utilises the trust-region methodology, the authors are able to provide a theoretical analysis and lower bound on performance improvement each episode. The method shows promising performance on several MuJoCo continuous control tasks, as well as in a Gazebo TurtleBot simulation, which is also able to be transferred to a real-world robot.