Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration

1 INTRODUCTION . Reinforcement Learning ( RL ) is being considered for application to a variety of real-world settings that have very large state-action spaces , but even under the availability of accurate simulators determining how best to explore the environment is a major challenge . Oftentimes , the problem setting is such that intuitively obvious reward functions are quite sparse . Such rewards typically correspond to the attainment of a particular task , such as a robot attaining designated way points , with little fine-grain reward feedback on the intermediate steps taken to complete these tasks . The sparsity of rewards implies that existing RL approaches often do not make much headway into learning viable policies unless provided with carefully hand-tuned reward feedback by a domain expert . Many of these problems correspond to systems where data has been gathered over time using an empirically determined ( sub-optimal ) behavior policy , which contains information important to help bootstrap learning . An additional caveat is that this behavior data might only contain measurements of a subset of the true state . Is it possible to design an algorithm that has principled policy optimization for efficient learning , while also being able to explore by utilizing the data derived from a behavior policy ? A natural candidate to begin with is the framework of policy gradient algorithms ( Schulman et al. , 2015 ; Lillicrap et al. , 2016 ; Haarnoja et al. , 2018 ) , which performs really well in the dense reward setting . Can such a policy gradient algorithm be aligned with a behavior policy for guided exploration that is critical in the sparse reward setting ? In this work , we propose a principled approach for using the well known trust region policy optimization ( TRPO ) algorithm ( Schulman et al. , 2015 ) with offline demonstration data for guidance in a sparse reward setting . Our choice of the TRPO approach is motivated by its analytical tractability and superior performance . Our key insight is that we can utilize the trust region approach while being guided by the behavior data by virtue of two steps . The first step is identical to traditional TRPO to generate a candidate policy . In the second step , the objective is to find a policy closest to the behavior policy , subject to it being in the trust region of the candidate generated in the first step . Thus , the second step ensures that the policy chosen is always guided by the behavior policy , but the level of alignment with the behavior policy can be reduced by shrinking the trust region over time . We call our approach Learning Online with Guidance Offline ( LOGO ) to capture this two-step procedure . This principled approach enables us to provide analytical performance guarantees while achieving superior performance in a number of benchmark problems . Our main results are as follows : ( i ) LOGO can guarantee a performance improvement as long as the behavior policy is able to offer an advantage . Reducing the alignment with the behavior policy over time allows us to extract this advantage and then smoothly move beyond to find near-optimal policies , ( ii ) we provide a generalized version of the Performance Difference Lemma ( Kakade & Langford , 2002 ) wherein the stage reward function can depend on the policy itself , and use this result to determine a surrogate objective function for step 2 of LOGO that is straightforward to evaluate . This allows us to implement LOGO in the manner of two TRPO-like steps , enabling us to leverage the TRPO code base , ( iii ) we show on standard MuJoCo environments that LOGO trained with sparse rewards can attain nearly the same performance as an optimal algorithm trained with dense rewards . We go further and show that this excellent performance carries over to waypoint tracking by a robot over Gazebo simulations and real-world TurtleBot experiments , ( iv ) finally , we show that LOGO can also be used in the case where the demonstration data only contains a censored version of the true state ( incomplete state information ) by simply adding a projection to the available subset of the state space in step 2 of the algorithm . Again , we provide supporting evidence via excellence in both MuJoCo simulations , as well as obstacle avoidance by a TurtleBot that is trained with a behavior policy without a Lidar ( censored state ) , but is tested with it ( full state ) . 1.1 RELATED WORK . Our work is mainly related to two RL research areas : Imitation Learning ( IL ) : The goal of an IL algorithm is to imitate an ( expert ) policy using the demonstration data generated by that policy . Behavior cloning ( BC ) is a simple IL approach where the expert policy is estimated from the demonstration data using supervised learning . BC algorithms , however , suffer from the problem of distribution shift ( Ross et al. , 2011 ) . Inverse reinforcement learning ( IRL ) algorithms ( Ng & Russell , 2000 ; Ziebart et al. , 2008 ) estimate a reward function from the demonstration data and solve a forward RL problem using this reward function . Generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ) avoids the reward estimation problem by formulating the IL problem as a distribution matching problem , and provides an implicit reward for the RL algorithm using a discriminator ( Goodfellow et al. , 2014 ) . Most IL algorithms do not use reward feedback from the environment , and hence are restricted to the performance of the policy that generated the demonstration data . Our approach is different from pure IL , and we leverage online RL with ( sparse ) reward feedback for efficient learning . Learning from Demonstration ( LfD ) : The key idea of the LfD algorithms is to use demonstration data to aid online learning ( Schaal et al. , 1997 ) . Many works propose to exploit demonstration data by adding it to the replay buffer with a prioritized replay mechanism to accelerate learning ( Hester et al. , 2018 ; Vecerik et al. , 2017 ; Nair et al. , 2018 ) . Rajeswaran et al . ( 2018 ) combine a policy gradient algorithm with demonstration data by using a mix of behavior cloning and online RL fine tuning . Nair et al . ( 2020 ) propose AWAC algorithm to accelerate online RL by leveraging large amounts of offline data with associated rewards . Different from this , LOGO does not need the reward observations . Moreover , we give provable guarantees on the performance of LOGO whereas AWAC does not have any such provable guarantee ( further details are provided in Appendix G ) . Kim et al . ( 2013 ) propose a framework to integrate LfD and approximate policy iteration by formulating a coupled constraint convex optimization problem , where the expert demonstrations define a set of linear constraints . This approach is , however , limited to small problems with a discrete action space . The closest to our work is the PofD algorithm proposed by Kang et al . ( 2018 ) . PofD modifies the reward function by taking a weighted combination of the sparse reward from the online interaction and an implicit reward obtained from the demonstration data using a discriminator . Very different from this , we propose an intuitive and principled approach of using the offline demonstration data for guiding the online exploration during the initial phase of learning . Our two step approach enables us to provide a rigorous performance guarantee for the proposed LOGO algorithm , and to leverage trust region-based approaches to solve high dimensional problems with even sparser settings . We provide an extensive comparison between our algorithm and PofD in Section 5 . Offline RL : Recently , there has been many interesting works in the area of offline RL ( Kumar et al. , 2019 ; Fujimoto et al. , 2019 ; Siegel et al. , 2020 ; Wu et al. , 2019a ) which focus on learning a policy using only the offline data without any online learning or online fine-tuning . Different from these , LOGO is an online RL algorithm ( further details are provided in Appendix G ) . 2 PRELIMINARIES AND PROBLEM SETTING . 2.1 PRELIMINARIES . A Markov Decision Process ( MDP ) is denoted as a tuple < S , A , R , P , γ > , where S is the state space , A is the action space , R : S ×A → R is the reward function , P : S ×A× S → [ 0 , 1 ] is the transition probability function with P ( s′|s , a ) giving the probability of transitioning to state s′ when action a is taken at state s , and γ ∈ ( 0 , 1 ) is the discount factor . A policy π is a mapping from S to probability distribution over A , with π ( s , a ) specifying the probability of taking action a in state s. A policy π can generate state-action trajectory τ , where τ = ( s0 , a0 , s1 , a1 , . . . ) , s0 ∼ µ , at ∼ π ( st , · ) , st+1 ∼ P ( ·|st , at ) and µ is the initial state distribution . The infinite horizon discounted return of policy π is defined as JR ( π ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) ] . The goal of a reinforcement learning algorithm is to learn the optimal policy π ? = arg maxπ JR ( π ) . The value function of a policy π defined as V πR ( s ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] , is the expected cumulative discounted reward obtained by following the policy π starting from the state s. The action-value function of a policy π is defined similarly as QπR ( s , a ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , a0 = a ] . The advantage function is defined as AπR ( s , a ) = QπR ( s , a ) − V πR ( s ) . The discounted state visitation distribution for the policy π , denoted as dπ , is defined as dπ ( s ) = ( 1− γ ) ∑∞ t=0 γ tP ( st = s|π ) , where the probability is defined with respect to the randomness induced by π , P and µ. Definitions and Notations : The Kullback-Leibler ( KL ) divergence between two distribution p and q is defined as DKL ( p , q ) = ∑ x p ( x ) log p ( x ) q ( x ) . The average KL divergence between two polices π1 and π2 with respect to dπ1 is defined as Dπ1KL ( π1 , π2 ) = Es∼dπ1 [ DKL ( π1 ( s , · ) , π2 ( s , · ) ) ] . The maximum KL divergence between two polices π1 and π2 is defined as DmaxKL ( π1 , π2 ) = maxs DKL ( π1 ( s , · ) , π2 ( s , · ) ) . The total variation ( TV ) distance between two distribution p and q is defined asDTV ( p , q ) = ( 1/2 ) ∑ x |p ( x ) −q ( x ) | . The average TV distance and the maximum TV distance between two polices π1 and π2 are defined as Dπ1TV ( π1 , π2 ) = Es∼dπ1 [ DTV ( π1 ( s , · ) , π2 ( s , · ) ) ] and DmaxTV ( π1 , π2 ) = maxs DTV ( π1 ( s , · ) , π2 ( s , · ) ) , respectively .

Summary: In a regime where we have access to offline behavior data from a suboptimal policy (a heuristic, human demonstrations, etc) we can use that data and "trust region"-based methods (TRPO in this work) to nudge exploration in the right direction by keeping the learnable policy close to the behavioral one. While there are several works exploring this general idea, the particular way presented in this work is novel and achieves better results on a set of MuJoCo environment. The approach is also shown to work on a physical robot (Turtlebot).