Learning Self-Imitating Diverse Policies

1 INTRODUCTION . Deep reinforcement learning ( RL ) has demonstrated significant applicability and superior performance in many problems outside the reach of traditional algorithms , such as computer and board games ( Mnih et al. , 2015 ; Silver et al. , 2016 ) , continuous control ( Lillicrap et al. , 2015 ) , and robotics ( Levine et al. , 2016 ) . Using deep neural networks as functional approximators , many classical RL algorithms have been shown to be very effective in solving sequential decision problems . For example , a policy that selects actions under certain state observation can be parameterized by a deep neural network that takes the current state observation as input and gives an action or a distribution over actions as output . Value functions that take both state observation and action as inputs and predict expected future reward can also be parameterized as neural networks . In order to optimize such neural networks , policy gradient methods ( Mnih et al. , 2016 ; Schulman et al. , 2015 ; 2017a ) and Q-learning algorithms ( Mnih et al. , 2015 ) capture the temporal structure of the sequential decision problem and decompose it to a supervised learning problem , guided by the immediate and discounted future reward from rollout data . Unfortunately , when the reward signal becomes sparse or delayed , these RL algorithms may suffer from inferior performance and inefficient sample complexity , mainly due to the scarcity of the immediate supervision when training happens in single-timestep manner . This is known as the temporal credit assignment problem ( Sutton , 1984 ) . For instance , consider the Atari Montezuma ’ s revenge game – a reward is received after collecting certain items or arriving at the final destination in the lowest level , while no reward is received as the agent is trying to reach these goals . The sparsity of the reward makes the neural network training very inefficient and also poses challenges in exploration . It is not hard to see that many of the real-world problems tend to be of the form where rewards are either only sparsely available during an episode , or the rewards are episodic , meaning that a non-zero reward is only provided at the end of the trajectory or episode . In addition to policy-gradient and Q-learning , alternative algorithms , such as those for global- or stochastic-optimization , have recently been studied for policy search . These algorithms do not decompose trajectories into individual timesteps , but instead apply zeroth-order finite-difference gradient or gradient-free methods to learn policies based on the cumulative rewards of the entire trajectory . Usually , trajectory samples are first generated by running the current policy and then the distribution of policy parameters is updated according to the trajectory-returns . The cross-entropy method ( CEM , Rubinstein & Kroese ( 2016 ) ) and evolution strategies ( Salimans et al. , 2017 ) are two nominal examples . Although their sample efficiency is often not comparable to the policy gradient methods when dense rewards are available from the environment , they are more widely applicable in the sparse or episodic reward settings as they are agnostic to task horizon , and only the trajectorybased cumulative reward is needed . Our contribution is the introduction of a new algorithm based on policy-gradients , with the objective of achieving better performance than existing RL algorithms in sparse and episodic reward settings . Using the equivalence between the policy function and its state-action visitation distribution , we formulate policy optimization as a divergence minimization problem between the current policy ’ s visitation and the distribution induced by a set of experience replay trajectories with high returns . We show that with the Jensen-Shannon divergence ( DJS ) , this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped , dense rewards learned from these experience replays . This algorithm can be seen as self-imitation learning , in which the expert trajectories in the experience replays are self-generated by the agent during the course of learning , rather than using some external demonstrations . We combine the divergence minimization objective with the standard RL objective , and empirically show that the shaped , dense rewards significantly help in sparse and episodic settings by improving credit assignment . Following that , we qualitatively analyze the shortcomings of the self-imitation algorithm . Our second contribution is the application of Stein variational policy gradient ( SVPG ) with the Jensen-Shannon kernel to simultaneously learn multiple diverse policies . We demonstrate the benefits of this addition to the self-imitation framework by considering difficult exploration tasks with sparse and deceptive rewards . Related Works . Divergence minimization has been used in various policy learning algorithms . Relative Entropy Policy Search ( REPS ) ( Peters et al. , 2010 ) restricts the loss of information between policy updates by constraining the KL-divergence between the state-action distribution of old and new policy . Policy search can also be formulated as an EM problem , leading to several interesting algorithms , such as RWR ( Peters & Schaal , 2007 ) and PoWER ( Kober & Peters , 2009 ) . Here the M-step minimizes a KL-divergence between trajectory distributions , leading to an update rule which resembles return-weighted imitation learning . Please refer to Deisenroth et al . ( 2013 ) for a comprehensive exposition . MATL ( Wulfmeier et al. , 2017 ) uses adversarial training to bring state occupancy from a real and simulated agent close to each other for efficient transfer learning . In Guided Policy Search ( GPS , Levine & Koltun ( 2013 ) ) , a parameterized policy is trained by constraining the divergence between the current policy and a controller learnt via trajectory optimization . Learning from Demonstrations ( LfD ) . The objective in LfD , or imitation learning , is to train a control policy to produce a trajectory distribution similar to the demonstrator . Approaches for self-driving cars ( Bojarski et al. , 2016 ) and drone manipulation ( Ross et al. , 2013 ) have used human-expert data , along with Behavioral Cloning algorithm to learn good control policies . Deep Q-learning has been combined with human demonstrations to achieve performance gains in Atari ( Hester et al. , 2017 ) and robotics tasks ( Večerı́k et al. , 2017 ; Nair et al. , 2017 ) . Human data has also been used in the maximum entropy IRL framework to learn cost functions under which the demonstrations are optimal ( Finn et al. , 2016 ) . Ho & Ermon ( 2016 ) use the same framework to derive an imitation-learning algorithm ( GAIL ) which is motivated by minimizing the divergence between agent ’ s rollouts and external expert demonstrations . Besides humans , other sources of expert supervision include planningbased approaches such as iLQR ( Levine et al. , 2016 ) and MCTS ( Silver et al. , 2016 ) . Our algorithm departs from prior work in forgoing external supervision , and instead using the past experiences of the learner itself as demonstration data . Exploration and Diversity in RL . Count-based exploration methods utilize state-action visitation counts N ( s , a ) , and award a bonus to rarely visited states ( Strehl & Littman , 2008 ) . In large statespaces , approximation techniques ( Tang et al. , 2017 ) , and estimation of pseudo-counts by learning density models ( Bellemare et al. , 2016 ; Fu et al. , 2017 ) has been researched . Intrinsic motivation has been shown to aid exploration , for instance by using information gain ( Houthooft et al. , 2016 ) or prediction error ( Stadie et al. , 2015 ) as a bonus . Hindsight Experience Replay ( Andrychowicz et al. , 2017 ) adds additional goals ( and corresponding rewards ) to a Q-learning algorithm . We also obtain additional rewards , but from a discriminator trained on past agent experiences , to accelerate a policy-gradient algorithm . Prior work has looked at training a diverse ensemble of agents with good exploratory skills ( Liu et al. , 2017 ; Conti et al. , 2017 ; Florensa et al. , 2017 ) . To enjoy the benefits of diversity , we incorporate a modification of SVPG ( Liu et al. , 2017 ) in our final algorithm . In very recent work , Oh et al . ( 2018 ) propose exploiting past good trajectories to drive exploration . Their algorithm buffers ( s , a ) and the corresponding return for each transition in rolled trajectories , and reuses them for training if the stored return value is higher than the current state-value estimate . Our approach presents a different objective for self-imitation based on divergence-minimization . With this view , we learn shaped , dense rewards which are then used for policy optimization . We further improve the algorithm with SVPG . Reusing high-reward trajectories has also been explored for program synthesis and semantic parsing tasks ( Liang et al. , 2016 ; 2018 ; Abolafia et al. , 2018 ) . 2 MAIN METHODS . We start with a brief introduction to RL in Section 2.1 , and then introduce our main algorithm of self-imitating learning in Section 2.2 . Section 2.3 further extends our main method to learn multiple diverse policies using Stein variational policy gradient with Jensen-Shannon kernel . 2.1 REINFORCEMENT LEARNING BACKGROUND . A typical RL setting involves an environment modeled as a Markov Decision Process with an unknown system dynamics model p ( st+1|st , at ) and an initial state distribution p0 ( s0 ) . An agent interacts sequentially with the environment in discrete time-steps using a policy π which maps the an observation st ∈ S to either a single action at ( deterministic policy ) , or a distribution over the action space A ( stochastic policy ) . We consider the scenario of stochastic policies over high-dimensional , continuous state and action spaces . The agent receives a per-step reward rt ( st , at ) ∈ R , and the RL objective involves maximization of the expected discounted sum of rewards , η ( πθ ) = Ep0 , p , π [ ∑∞ t=0 γ tr ( st , at ) ] , where γ ∈ ( 0 , 1 ] is the discount factor . The action-value function is Qπ ( st , at ) = Ep0 , p , π [ ∑∞ t′=t γ t′−tr ( st′ , at′ ) ] . We define the unnormalized γ-discounted state- visitation distribution for a policy π by ρπ ( s ) = ∑∞ t=0 γ tP ( st = s|π ) , where P ( st = s|π ) is the probability of being in state s at time t , when following policy π and starting state s0 ∼ p0 . The expected policy return η ( πθ ) can then be written as Eρπ ( s , a ) [ r ( s , a ) ] , where ρπ ( s , a ) = ρπ ( s ) π ( a|s ) is the state-action visitation distribution . Using the policy gradient theorem ( Sutton et al. , 2000 ) , we can get the direction of ascent∇θη ( πθ ) = Eρπ ( s , a ) [ ∇θ log πθ ( a|s ) Qπ ( s , a ) ] . 2.2 POLICY OPTIMIZATION AS DIVERGENCE MINIMIZATION WITH SELF-IMITATION . Although the policy π ( a|s ) is given as a conditional distribution , its behavior is better characterized by the corresponding state-action visitation distribution ρπ ( s , a ) , which wraps the MDP dynamics and fully decides the expected return via η ( π ) = Eρπ [ r ( s , a ) ] . Therefore , distance metrics on a policy π should be defined with respect to the visitation distribution ρπ , and the policy search should be viewed as finding policies with good visitation distributions ρπ that yield high reward . Suppose we have access to a good policy π∗ , then it is natural to consider finding a π such that its visitation distribution ρπ matches ρπ∗ . To do so , we can define a divergence measureD ( ρπ , ρπ∗ ) that captures the similarity between two distributions , and minimize this divergence for policy improvement . Assume there exists an expert policy πE , such that policy optimization can be framed as minimizing the divergence minπD ( ρπ , ρπE ) , that is , finding a policy π to imitate πE . In practice , however , we do not have access to any real guiding expert policy . Instead , we can maintain a selected subsetME of highly-rewarded trajectories from the previous rollouts of policy π , and optimize the policy π to minimize the divergence between ρπ and the empirical state-action pair distribution { ( si , ai ) } ME : min π D ( ρπ , { ( si , ai ) } ME ) . ( 1 ) Since it is not always possible to explicitly formulate ρπ even with the exact functional form of π , we generate rollouts from π in the environment and obtain an empirical distribution of ρπ . To measure the divergence between two empirical distributions , we use the Jensen-Shannon divergence , with the following variational form ( up to a constant shift ) as exploited in GANs ( Goodfellow et al. , 2014 ) : DJS ( ρπ , ρπE ) = max d ( s , a ) , dE ( s , a ) Ẽρπ [ log d ( s , a ) d ( s , a ) + dE ( s , a ) ] + ẼρπE [ log dE ( s , a ) d ( s , a ) + dE ( s , a ) ] , ( 2 ) where d ( s , a ) and dE ( s , a ) are empirical density estimators of ρπ and ρπE , respectively . Under certain assumptions , we can obtain an approximate gradient of DJS w.r.t the policy parameters , thus enabling us to optimize the policy . Gradient Approximation : Let ρπ ( s , a ) and ρπE ( s , a ) be the state-action visitation distributions induced by two policies π and πE respectively . Let dπ and dπE be the surrogates to ρπ and ρπE , respectively , obtained by solving Equation 2 . Then , if the policy π is parameterized by θ , the gradient of DJS ( ρπ , ρπE ) with respect to policy parameters ( θ ) can be approximated as : ∇θDJS ( ρπ , ρπE ) ≈ Ẽρπ ( s , a ) [ ∇θ log πθ ( a|s ) Q̃π ( s , a ) ] , where Q̃π ( st , at ) = Ẽρπ ( s , a ) [ ∞∑ t′=t γt ′−t log dπ ( st′ , at′ ) dπ ( st′ , at′ ) + dπE ( st′ , at′ ) ] .. ( 3 ) The derivation of the approximation and the underlying assumptions are in Appendix 5.1 . Next , we introduce a simple and inexpensive approach to construct the replay memoryME using highreturn past experiences during training . In this way , ρπE can be seen as a mixture of deterministic policies , each representing a delta point mass distribution in the trajectory space or a finite discrete visitation distribution of state-action pairs . At each iteration , we apply the current policy πθ to sample b trajectories { τ } b1 . We hope to include inME , the top-k trajectories ( or trajectories with returns above a threshold ) generated thus far during the training process . For this , we use a priorityqueue list forME which keeps the trajectories sorted according to the total trajectory reward . The reward for each newly sampled trajectory in { τ } b1 is compared with the current threshold of the priority-queue , updatingME accordingly . The frequency of updates is impacted by the exploration capabilities of the agent and the stochasticity in the environment . We find that simply sampling noisy actions from Gaussian policies is sufficient for several locomotion tasks ( Section 3 ) . To handle more challenging environments , in the next sub-section , we augment our policy optimization procedure to explicitly enhance exploration and produce an ensemble of diverse policies . In the usual imitation learning framework , expert demonstrations of trajectories—from external sources—are available as the empirical distribution of ρπE of an expert policy πE . In our approach , since the agent learns by treating its own good past experiences as the expert , we can view the algorithm as self-imitation learning from experience replay . As noted in Equation 3 , the gradient estimator of DJS has a form similar to policy gradients , but for replacing the true reward function with per-timestep reward defined as log ( dπ ( s , a ) / ( dπ ( s , a ) + dπE ( s , a ) ) ) . Therefore , it is possible to interpolate the gradient of DJS and the standard policy gradient . We would highlight the benefit of this interpolation soon . The net gradient on the policy parameters is : ∇θη ( πθ ) = ( 1− ν ) Eρπ ( s , a ) [ ∇θ log πθ ( a|s ) Qr ( s , a ) ] − ν∇θDJS ( ρπ , ρπE ) , ( 4 ) where Qr is the Q function with true rewards , and πE is the mixture policy represented by the samples inME . Let rφ ( s , a ) = dπ ( s , a ) / [ dπ ( s , a ) + dπE ( s , a ) ] . rφ ( s , a ) can be computed using parameterized networks for densities dπ and dπE , which are trained by solving theDJS optimization ( Eq 2 ) using the current policy rollouts andME , where φ includes the parameters for dπ and dπE . Using Equation 3 , the interpolated gradient can be further simplified to : ∇θη ( πθ ) = Eρπ ( s , a ) [ ∇θ log πθ ( a|s ) [ ( 1− ν ) Qr ( s , a ) + νQr φ ( s , a ) ] ] , ( 5 ) where Qr φ ( st , at ) = −Ep0 , p , π [ ∑∞ t′=t γ t′−t log rφ ( st′ , at′ ) ] is the Q function calculated using − log rφ ( s , a ) as the reward . This reward is high in the regions of the S × A space frequented more by the expert than the learner , and low in regions visited more by the learner than the expert . The effective Q in Equation 5 is therefore an interpolation between Qr obtained with true environment rewards , and Qr φ obtained with rewards which are implicitly shaped to guide the learner towards expert behavior . In environments with sparse or deceptive rewards , where the signal from Qr is weak or sub-optimal , a higher weight on Qr φ enables successful learning by imitation . We show this empirically in our experiments . We further find that even in cases with dense environment rewards , the two gradient components can be successfully combined for policy optimization . The complete algorithm for self-imitation is outlined in Appendix 5.2 ( Algorithm 1 ) . Limitations of self-imitation . We now elucidate some shortcomings of the self-imitation approach . Since the replay memoryME is only constructed from the past training rollouts , the quality of the trajectories inME is hinged on good exploration by the agent . Consider a maze environment where the robot is only rewarded when it arrives at a goal G placed in a far-off corner . Unless the robot reaches G once , the trajectories inME always have a total reward of zero , and the learning signal from Qr φ is not useful . Secondly , self-imitation can lead to sub-optimal policies when there are local minima in the policy optimization landscape ; for example , assume the maze has a second goal G′ in the opposite direction of G , but with a much smaller reward . With simple exploration , the agent may fillME with below-par trajectories leading to G′ , and the reinforcement from Qr φ would drive it further to G′ . Thirdly , stochasticity in the environment may make it difficult to recover the optimal policy just by imitating the past top-k rollouts . For instance , in a 2-armed bandit problem with reward distributions Bernoulli ( p ) and Bernoulli ( p+ ) , rollouts from both the arms get conflated in ME during training with high probability , making it hard to imitate the action of picking the arm with the higher expected reward . We propose to overcome these pitfalls by training an ensemble of self-imitating agents , which are explicitly encouraged to visit different , non-overlapping regions of the state-space . This helps to discover useful rewards in sparse settings , avoids deceptive reward traps , and in environments with reward-stochasticity like the 2-armed bandit , increases the probability of the optimal policy being present in the final trained ensemble . We detail the enhancements next .

Summary: The paper proposes how previously experienced high reward trajectories can be used to generate dense reward functions for more for efficient training of policies in context of reinforcement learning. The paper does this by computing the state-action pair distribution of high rewarding trajectories in the replay buffer, and using a surrogate reward that measures the distance between this distribution and the current state-action pair distribution. The paper derives approximate policy gradients for this surrogate reward function. The paper then describes limitations of doing this: possibility of getting stuck in the local neighborhood of currently well-performing trajectories. It also describes an extension based on Stein variational policy gradients to diversify behavior of an ensemble of policies that are learned together. The paper shows experimental results on a number of MuJoCo tasks.