AUGMENTED POLICY GRADIENT METHODS FOR EFFICIENT REINFORCEMENT LEARNING

1 INTRODUCTION . Reinforcement Learning ( RL ) can be broadly classified in two categories : model-free RL , which uses experience from interacting with the environment to learn a value function , and model-based RL , which uses experience to approximate the environment by a model ( Deisenroth & Rasmussen ( 2011 ) ) . Recently , impressive results were achieved applying model-free RL aproaches to challenging tasks . A RL agent with a neural network ( NN ) based Q-value representation was introduced in Mnih et al . ( 2015 ) which was able to reach an equivalent or higher performance on many Atari games compared to humans by learning a state function based on raw pixels . Further modifications addressed the problem of Q-value overestimation ( van Hasselt et al . ( 2015 ) ) , dropping significant experience ( Schaul et al . ( 2015 ) ) and extended the approach to asynchronous methods ( Mnih et al . ( 2016 ) ) to improve the overall performance . Beside learning a value function , it is possible to use the gradient of the RL objective to update the policy ( Peters & Schaal ( 2006 ) , Sutton & Barto ( 2010 ) , Silver et al . ( 2014 ) , Lillicrap et al . ( 2015 ) ) . Actor-critic methods aim at combining the strong points of policy gradients and an approximated value function , in which the critic learns an approximated value function that is used to update the policy of the actor in the direction of its gradient ( Konda & Tsitsiklis ( 2000 ) , Schulman et al . ( 2015a ) , Mnih et al . ( 2016 ) , Gu et al . ( 2016 ) ) . In contrast , model-based RL can be significantly more data efficient . It is notable that a dynamics model is not necessarily globally valid . In guided policy search ( GPS ) , time-varying linear dynamics models are learned from multiple rollouts with diverse start and goal conditions ( Levine & Koltun ( 2013a ) , Levine & Koltun ( 2013b ) , Levine & Koltun ( 2014 ) ) . These trajectory samples are then used to fit a time-varying Gaussian dynamics model leading to a Gaussian distribution over trajectories , even for unknown dynamics ( Levine & Abbeel ( 2014 ) ) . The neural network policy is trained in a supervised fashion such that local policies are improved minimizing a cost function bounded by the trust region of the KL-divergence , and the weights of the NN policy are adjusted to mimick the trajectory distribution . Further improvements extended GPS to a broader robust state space by introducing a new policy representation which is called general motor reflex ( GMR ) ( Ennen et al . ( 2018 ) ) . In a first step , the state space is encoded to a latent state representation by a variational autoencoder . Then , a translation model predicts motor reflex parameters in order to adjust a Gaussian controller resulting in robust trajectories even outside the distribution of training samples . However , modelling errors can reduce the efficiency and benefit of model-based RL algorithms since model imperfections are exploited ( Deisenroth & Rasmussen ( 2011 ) , Schneider ( 1996 ) , Atkeson & Santamaria ( 1997 ) ) known as model-bias . With model-bias being a result of overfitting , one approach to solve this problem is to incorporate explicit uncertainty estimates by using a probabilistic dynamics model . Probabilistic inference for learning control ( PILCO ) ( Deisenroth & Rasmussen ( 2011 ) ) uses a Gaussian process ( GP ) to represent the dynamics of the environment . Such a model provides predictions with true Bayesian uncertainty estimates but suffer from the curse of dimensionality ( Calandra et al . ( 2014 ) ) . Recently , Bayesian neural networks ( BNN ) received much attention . Approximate Bayesian techniques , e.g . variational inference ( Graves ( 2011 ) ) or Monte-Carlo dropout ( Gal & Ghahramani ( 2015 ) ) , come at the drawback that correlations between parameters are not maintained , while scaling abilities of the Bayesian ’ gold standard ’ inference method Markov-Chain-Monte-Carlo ( MCMC ) are limited ( Hinton & Neal ( 1995 ) . Adding a regularization term to the loss function of a NN , e.g . Tikhonov regularization , permits maximum a posterior ( MAP ) estimates of the parameters since the solution of the regularized loss function is equivalent to the maximum of the posterior density function ( Bardsley ( 2012 ) ) . However , the regularized solution leads to a point estimator of the true posterior density . This insufficiency can be addressed when sampling from the true posterior distribution using a MCMC method providing additional information about the shape of the posterior but is computationally more demanding . Our approach is inspired by recent advances , Model-Ensemble Trust-Region Policy Optimization ( ME-TRPO ) ( Kurutach et al . ( 2018 ) ) , Model Based Meta Policy Optimization ( MB-MPO ) ( Clavera et al . ( 2018 ) ) , Model-assisted Boostrapped DDPG ( MA-BDDPG ) ( Kalweit & Boedecker ( 2017 ) ) , and Model-Based Value Expansion ( MVE ) ( Feinberg et al . ( 2018 ) ) . In ME-TRPO , a set of samples from the real environment is used to train an ensemble of NNs . In a second iteration , fictitious trajectory rollouts are generated to update the policy using Trust Region Policy Optimization ( TRPO ) ( Schulman et al . ( 2015a ) ) , i.e. , the policy update is only based on samples from the approximated dynamics model of the environment . In MB-MPO , the meta policy learning framework ’ model agnostic meta learning ’ ( Finn et al . ( 2017 ) ) is used to train a policy on an ensemble of learned dynamics models . The notion is that the meta-learning frame of the dynamics models is used to adapt quickly to any new dynamics model within one update step . The overall similarities of all dynamics models are incorporated in the meta-model that is then adjusted to create the fitted dynamics model . MA-BDDPG employs the same idea to augment the dataset to achieve a better data efficiency . In contrast to our approach , MA-BDDPG makes use of an actor-critic method in which the policy is represented deterministically . Updates in the actor and critic network follow the advances of Deep Q-Networks by refering to time-delayed target networks . A further difference is in the procedure for estimating uncertainty , i.e . Kalweit & Boedecker ( 2017 ) applies bootstrapping ( Efron & Tibshirani ( 1998 ) ) to get a distribution over Q-functions while we use an anchored ensemble of NNs . MVE improves value estimates assuming that an approximated dynamics model can be used in training a critic up to a depth H in which we are certain about the model accuracy . Contrary to our approach , MVE defines its dynamics model as a deterministic neural network . Our new algorithm MBPGE is a model-free RL algorithm that is augmented by a true Bayesian ensemble of dynamics models to generate fictitious trajectory rollouts to decrease the sample complexity . In contrast to ME-TRPO , we take advantage of a very recent inference advance based on randomised anchored MAP sampling that allows for a true Bayesian uncertainty estimate of the true posterior ( Pearce et al . ( 2018 ) ) . As a result , our stochastic dynamics model reduces the effect of policy overfitting as model inaccuracies will be addressed by highly random next state predictions . We have seen that ill-distributed datasets , i.e. , data points of some regions in the state space are overrepresented while others are underrepresented , might lead to enormous fictitious total rewards that diverge heavily from those of the real environment samples due to overfitting dynamics models . We tackle this issue by introducing a ratio of fictitious and real rewards so that our algorithm adjusts naturally to become more model-free if the dynamics model is inaccurate and acts in a more model-based way if the approximation of the environment is accurate . To summarize , our main contributions are : 1 . A novel model-free RL algorithm that uses a true Bayesian representation for the stochastic dynamics model and 2. a routine that adjusts the composition of the dataset for a policy gradient estimate by weighting the trajectory samples based on the ratio of the total rewards of the real environment and the approximated model . We derive our approach by reviewing the background in constrained policy gradient methods and randomized maximum a posterior sampling . Then , we present our augmented model-free reinforcement learning algorithm in detail . Finally , we evaluate our approach on challenging simulated tasks in a continuous control domain . 2 PRELIMINARIES . We formulate the underlying problem of our tasks as a discrete-time finite-horizon Markov decision process ( MDP ) , defined by the tuple ( S , A , P , r , ρ0 , γ , T ) . Here , we assume that our agent acts in a stochastic environment with continuous state space S ⊆ Rn and continuous action space A ⊆ Rm . Additionally , we assume that an unknown dynamics model P ( st+1|st , a ) is defined for our environment and we are given a reward function r ( st , at ) . Let ρ0 denote the initial state distribution , γ is the discount factor , T is the finite horizon and πθ ( a|s ) denotes our neural network policy , parameterized by θ , distributed over actions a , and conditioned to states s. The overall aim is to learn an optimal policy π such that the expected total reward J ( π ) = Eτ∼π [ ∑T t=0 rs ( st , at ) ] is maximized , where τ = s0 , a0 , ... , aT−1 , sT denotes a trajectory with s0 ∼ ρ0 , at ∼ πθ and st+1 = P ( st , at ) . 2.1 POLICY GRADIENT METHODS . The maximization problem of the RL objective can be solved by performing gradient ascent computing the derivative with respect to the policy parameters θ . The major problem of policy gradient methods is to find a good estimator for the gradient since the variance of the gradient estimator increases with the time horizon . Fortunately , a variance reduction scheme for policy gradients , generalized advantage estimator ( GAE ) ( Schulman et al . ( 2015b ) ) , reduces the variance significantly while only introducing a small bias . It is possible to trade-off between bias and variance by using a λ−return ( Sutton & Barto ( 2010 ) ) . Extending the idea of an n-step return , one can weight each n-step update by λn−1 and normalize the sum by ( 1− λ ) resulting in Rλt = ( 1− λ ) ∞∑ n=1 λn−1Rnt . ( 1 ) The finite horizon version assumes that all rewards after time step T equal 0 , leading to Rλt = ( 1− λ ) T−t−1∑ n=1 λn−1Rnt + λ T−t−1RT−tt . ( 2 ) Setting λ = 0 results in the biased single step return R ( 1 ) t with low variance , while λ = 1 corresponds to the unbiased Monte-Carlo return R ( ∞ ) t with high variance . GAE can be derived when estimating the advantage Aπ ( st , at ) = Qπ ( st , at ) − V π ( st ) with the λ−return . The gradient of the objective now turns to g : = ∇θE [ T∑ t=0 rt ] = E [ T∑ t=0 Ât∇θlogπθ ( at|st ) ] . ( 3 ) Usually , we intend to update our parameters only by small changes . However , instead of performing update steps in parameter space using the Euclidean distance , it is much more convenient to measure closeness between the current policy and the updated policy in terms of distances between distributions . The intuition behind is that distances between distributions , e.g . Kullback-Leibler divergence ( Kullback & Leibler ( 1951 ) ) or Hellinger distance ( Hellinger ( 1909 ) ) , are invariant to scaling or shifting . This type of approach is known as natural policy gradients ( Kakade ( 2001 ) , Sutton et al . ( 2000 ) , Peters & Schaal ( 2006 ) ) . Trust region policy optimization ( TRPO ) utilizes this idea and maximizes its ’ surrogate ’ objective function by constraining the allowed policy update step ( Schulman et al . ( 2015a ) ) max θ Êt [ πθ ( at|st ) πθold ( at|st ) Ât ] s.t . Êt [ KL [ πθold ( ∗|st ) , πθ ( ∗|st ) ] ] 5 δ . ( 4 ) TRPO builds on the concept of guaranteed monotonic improvement . The notion is that we can compute an upper bound for the error of diverging policies . Therefore , we can guarantee a policy improvement as long as we optimize the local approximation within a trusted region . The proof is given in Schulman et al . ( 2015a ) . PPO combines the appealing guarantee of improvement with a much simpler implementation ( Schulman et al . ( 2017 ) ) . Instead of maximizing the ’ surrogate ’ function in eq . 4 , it uses the following clipped objective J ( θ ) = Êt [ min ( rt ( θ ) Ât , clip ( rt ( θ ) , 1− , 1 + ) Ât ] . ( 5 ) Clipping the probability ensures that the probability ratio rt ( θ ) = πθ ( at|st ) πθold ( at|st ) is in the interval [ 1 − , 1 + ] , while taking the minimum of the clipped and unclipped objective leads to a lower bound of the objective with the same convenient effect as constraining the ’ surrogate ’ objective in TRPO .

Summary: This paper presents a new model-free + model-based algorithm, MBPGE, that trains a policy using a policy gradient algorithm on top of the learned models. Contrary to previous approaches, they use a true Bayesian distribution by means of the randomized anchorized MAP. Furthermore, they combine rollouts from the real environment and from the learned dynamics directly for the policy training, instead of relying just on the ones of learned dynamics, which induces a larger distributional shift.