Title: AUGMENTED POLICY GRADIENT METHODS FOR EFFICIENT REINFORCEMENT LEARNING Anonymous authors Paper under double-blind review

Abstract: We propose a new mixture of model-based and model-free reinforcement learning (RL) algorithms that combines the strengths of both RL methods. Our goal is to reduce the sample complexity of model-free approaches utilizing fictitious trajectory rollouts performed on a learned dynamics model to improve the data efficiency of policy gradient methods while maintaining the same asymptotic behaviour. We suggest to use a special type of uncertainty quantification by a stochastic dynamics model in which the next state prediction is randomly drawn from the distribution predicted by the dynamics model. As a result, the negative effect of exploiting erroneously optimistic regions in the dynamics model is addressed by next state predictions based on an uncertainty aware ensemble of dynamics models. The influence of the ensemble of dynamics models on the policy update is controlled by adjusting the number of virtually performed rollouts in the next iteration according to the ratio of the real and virtual total reward. Our approach, which we call Model-Based Policy Gradient Enrichment (MBPGE), is tested on a collection of benchmark tests including simulated robotic locomotion. We compare our approach to plain model-free algorithms and a model-based one. Our evaluation shows that MBPGE leads to higher learning rates in an early training stage and an improved asymptotic behaviour.

1. INTRODUCTION:

Reinforcement Learning (RL) can be broadly classified in two categories: model-free RL, which uses experience from interacting with the environment to learn a value function, and model-based RL, which uses experience to approximate the environment by a model (Deisenroth & Rasmussen (2011) ). Recently, impressive results were achieved applying model-free RL aproaches to challenging tasks. A RL agent with a neural network (NN) based Q-value representation was introduced in Mnih et al. (2015) which was able to reach an equivalent or higher performance on many Atari games compared to humans by learning a state function based on raw pixels. Further modifications addressed the problem of Q-value overestimation (van Hasselt et al. (2015) ), dropping significant experience (Schaul et al. (2015) ) and extended the approach to asynchronous methods (Mnih et al. (2016) ) to improve the overall performance. Beside learning a value function, it is possible to use the gradient of the RL objective to update the policy (Peters & Schaal (2006) , Sutton & Barto (2010) , Silver et al. (2014) , Lillicrap et al. (2015) ). Actor-critic methods aim at combining the strong points of policy gradients and an approximated value function, in which the critic learns an approximated value function that is used to update the policy of the actor in the direction of its gradient (Konda & Tsitsiklis (2000) , Schulman et al. (2015a) , Mnih et al. (2016) , Gu et al. (2016) ).

In contrast, model-based RL can be significantly more data efficient. It is notable that a dynamics model is not necessarily globally valid. In guided policy search (GPS), time-varying linear dynamics models are learned from multiple rollouts with diverse start and goal conditions (Levine & Koltun (2013a) , Levine & Koltun (2013b) , Levine & Koltun (2014) ). These trajectory samples are then used to fit a time-varying Gaussian dynamics model leading to a Gaussian distribution over trajectories, even for unknown dynamics (Levine & Abbeel (2014) ). The neural network policy is trained in a supervised fashion such that local policies are improved minimizing a cost function bounded by the trust region of the KL-divergence, and the weights of the NN policy are adjusted to mimick the trajectory distribution. Further improvements extended GPS to a broader robust state space by introducing a new policy representation which is called general motor reflex (GMR) (Ennen et al. (2018) ). In a first step, the state space is encoded to a latent state representation by a variational autoencoder. Then, a translation model predicts motor reflex parameters in order to adjust a Gaussian controller resulting in robust trajectories even outside the distribution of training samples. However, modelling errors can reduce the efficiency and benefit of model-based RL algorithms since model imperfections are exploited (Deisenroth & Rasmussen (2011 ), Schneider (1996) , Atkeson & Santamaria (1997) ) known as model-bias. With model-bias being a result of overfitting, one approach to solve this problem is to incorporate explicit uncertainty estimates by using a probabilistic dynamics model. Probabilistic inference for learning control (PILCO) (Deisenroth & Rasmussen (2011) ) uses a Gaussian process (GP) to represent the dynamics of the environment. Such a model provides predictions with true Bayesian uncertainty estimates but suffer from the curse of dimensionality (Calandra et al. (2014) ). Recently, Bayesian neural networks (BNN) received much attention. Approximate Bayesian techniques, e.g. variational inference (Graves (2011)) or Monte-Carlo dropout (Gal & Ghahramani (2015) ), come at the drawback that correlations between parameters are not maintained, while scaling abilities of the Bayesian 'gold standard' inference method Markov-Chain-Monte-Carlo (MCMC) are limited (Hinton & Neal (1995) . Adding a regularization term to the loss function of a NN, e.g. Tikhonov regularization, permits maximum a posterior (MAP) estimates of the parameters since the solution of the regularized loss function is equivalent to the maximum of the posterior density function (Bardsley (2012) ). However, the regularized solution leads to a point estimator of the true posterior density. This insufficiency can be addressed when sampling from the true posterior distribution using a MCMC method providing additional information about the shape of the posterior but is computationally more demanding.

Our approach is inspired by recent advances, Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) (Kurutach et al. (2018) ), Model Based Meta Policy Optimization (MB-MPO) (Clavera et al. (2018) ), Model-assisted Boostrapped DDPG (MA-BDDPG) (Kalweit & Boedecker (2017) ), and Model-Based Value Expansion (MVE) (Feinberg et al. (2018) ). In ME-TRPO, a set of samples from the real environment is used to train an ensemble of NNs. In a second iteration, fictitious trajectory rollouts are generated to update the policy using Trust Region Policy Optimization (TRPO) (Schulman et al. (2015a) ), i.e., the policy update is only based on samples from the approximated dynamics model of the environment. In MB-MPO, the meta policy learning framework 'model agnostic meta learning' (Finn et al. (2017) ) is used to train a policy on an ensemble of learned dynamics models. The notion is that the meta-learning frame of the dynamics models is used to adapt quickly to any new dynamics model within one update step. The overall similarities of all dynamics models are incorporated in the meta-model that is then adjusted to create the fitted dynamics model. MA-BDDPG employs the same idea to augment the dataset to achieve a better data efficiency. In contrast to our approach, MA-BDDPG makes use of an actor-critic method in which the policy is represented deterministically. Updates in the actor and critic network follow the advances of Deep Q-Networks by refering to time-delayed target networks. A further difference is in the procedure for estimating uncertainty, i.e. Kalweit & Boedecker (2017) applies bootstrapping (Efron & Tibshirani (1998) ) to get a distribution over Q-functions while we use an anchored ensemble of NNs. MVE improves value estimates assuming that an approximated dynamics model can be used in training a critic up to a depth H in which we are certain about the model accuracy. Contrary to our approach, MVE defines its dynamics model as a deterministic neural network.

Our new algorithm MBPGE is a model-free RL algorithm that is augmented by a true Bayesian ensemble of dynamics models to generate fictitious trajectory rollouts to decrease the sample complexity. In contrast to ME-TRPO, we take advantage of a very recent inference advance based on randomised anchored MAP sampling that allows for a true Bayesian uncertainty estimate of the true posterior (Pearce et al. (2018) ). As a result, our stochastic dynamics model reduces the effect of policy overfitting as model inaccuracies will be addressed by highly random next state predictions. We have seen that ill-distributed datasets, i.e., data points of some regions in the state space are overrepresented while others are underrepresented, might lead to enormous fictitious total rewards that diverge heavily from those of the real environment samples due to overfitting dynamics models. We tackle this issue by introducing a ratio of fictitious and real rewards so that our algorithm adjusts naturally to become more model-free if the dynamics model is inaccurate and acts in a more model-based way if the approximation of the environment is accurate. To summarize, our main contributions are:

1. A novel model-free RL algorithm that uses a true Bayesian representation for the stochastic dynamics model and 2. a routine that adjusts the composition of the dataset for a policy gradient estimate by weighting the trajectory samples based on the ratio of the total rewards of the real environment and the approximated model.

We derive our approach by reviewing the background in constrained policy gradient methods and randomized maximum a posterior sampling. Then, we present our augmented model-free reinforcement learning algorithm in detail. Finally, we evaluate our approach on challenging simulated tasks in a continuous control domain.

2. PRELIMINARIES:

We formulate the underlying problem of our tasks as a discrete-time finite-horizon Markov decision process (MDP), defined by the tuple (S, A, P, r, ρ 0 , γ, T ). Here, we assume that our agent acts in a stochastic environment with continuous state space S ⊆ R n and continuous action space A ⊆ R m . Additionally, we assume that an unknown dynamics model P (s t+1 |s t , a) is defined for our environment and we are given a reward function r(s t , a t ). Let ρ 0 denote the initial state distribution, γ is the discount factor, T is the finite horizon and π θ (a|s) denotes our neural network policy, parameterized by θ, distributed over actions a, and conditioned to states s. The overall aim is to learn an optimal policy π such that the expected total reward J(π) = E τ ∼π T t=0 rs(s t , a t ) is maximized, where τ = s 0 , a 0 , ..., a T -1 , s T denotes a trajectory with s 0 ∼ ρ 0 , a t ∼ π θ and s t+1 = P (s t , a t ).

2.1. POLICY GRADIENT METHODS:

The maximization problem of the RL objective can be solved by performing gradient ascent computing the derivative with respect to the policy parameters θ. The major problem of policy gradient methods is to find a good estimator for the gradient since the variance of the gradient estimator increases with the time horizon. Fortunately, a variance reduction scheme for policy gradients, generalized advantage estimator (GAE) (Schulman et al. (2015b) ), reduces the variance significantly while only introducing a small bias. It is possible to trade-off between bias and variance by using a λ-return (Sutton & Barto (2010) ). Extending the idea of an n-step return, one can weight each n-step update by λ n-1 and normalize the sum by (1 -λ) resulting in

EQUATION

The finite horizon version assumes that all rewards after time step T equal 0, leading to

EQUATION

Setting λ = 0 results in the biased single step return R

(1) t with low variance, while λ = 1 corresponds to the unbiased Monte-Carlo return R (∞) t with high variance. GAE can be derived when estimating the advantage A π (s t , a t ) = Q π (s t , a t ) -V π (s t ) with the λ-return. The gradient of the objective now turns to

g := ∇ θ E T t=0 r t = E T t=0 Ât ∇ θ logπ θ (a t |s t ) .

(3)

Usually, we intend to update our parameters only by small changes. However, instead of performing update steps in parameter space using the Euclidean distance, it is much more convenient to measure closeness between the current policy and the updated policy in terms of distances between distributions. The intuition behind is that distances between distributions, e.g. Kullback-Leibler divergence (Kullback & Leibler (1951) ) or Hellinger distance (Hellinger (1909) ), are invariant to scaling or shifting. This type of approach is known as natural policy gradients (Kakade (2001) , Sutton et al. (2000) , Peters & Schaal (2006) ). Trust region policy optimization (TRPO) utilizes this idea and maximizes its 'surrogate' objective function by constraining the allowed policy update step (Schulman et al. (2015a) )

EQUATION

TRPO builds on the concept of guaranteed monotonic improvement. The notion is that we can compute an upper bound for the error of diverging policies. Therefore, we can guarantee a policy improvement as long as we optimize the local approximation within a trusted region. The proof is given in Schulman et al. (2015a) . PPO combines the appealing guarantee of improvement with a much simpler implementation (Schulman et al. (2017) ). Instead of maximizing the 'surrogate' function in eq. 4, it uses the following clipped objective

EQUATION

)

Clipping the probability ensures that the probability ratio r t (θ) = π θ (at|st) π θ old (at|st) is in the interval [1 -, 1 + ], while taking the minimum of the clipped and unclipped objective leads to a lower bound of the objective with the same convenient effect as constraining the 'surrogate' objective in TRPO.

2.2. MODEL-BASED REINFORCEMENT LEARNING WITH AN ANCHORED ENSEMBLE OF DEEP NEURAL NETWORKS:

Contrary to model-free RL, model-based RL utilizes the interactions with the environment to learn an approximated model. Model-based approaches benefit from a significantly lower sample complexity. However, learning an accurate model of the environment can be very challenging for certain domains. As a consequence, model-based algorithms often show a worse asymptotic behaviour. To date, there are several diverse approaches to represent the dynamics model and a proper choice is often crucial. Dynamics model in RL must reliably fit to datasets with a low and high number of data points: during the first few iterations only a limited number of data points is available, causing expressive general function approximators, e.g. deep neural networks, to overfit. With progressing iterations simple function approximators tend to underfit complex system dynamics.

Bayesian regression aims to find the parameters θ of a function y * = f θ (x * ), which are likely to have generated the output Y given input X. This can be achieved by updating a prior distribution over the parameters of the neural network θ 0 ∼ p(θ) by using Bayes Theorem. If it is possible to compute the posterior distribution 

EQUATION

Previous work revealed that exploiting uncertainty estimation in RL can increase the sample efficiency dramatically (Deisenroth & Rasmussen (2011) ). Gaussian Processes (Rasmussen & Williams (2008)) perform well in low data regimes but do not scale to complex tasks with high dimensionality (Calandra et al. (2014) ). On the other hand, probabilistic NNs can approximate arbitrarily complex dynamics but come at the cost that a true Bayesian uncertainty estimate cannot be provided (Gal & Ghahramani (2015) ). The computation of the integral in eq. 7 can in general only be approximated numerically under a trade-off between accuracy and performance.

A Baysian neural network (BNN) can be approximated by using a diverse ensemble F = {f 1 , ..., f K } of NNs. Ensembling provides an uncertainty estimate in such way that the variance of the ensemble's prediction is considered to be its uncertainty. Technically, this approach is not Bayesian (Gal (2016) ). This drawback was addressed in Pearce et al. (2018) by regularizing the parameters about values drawn from a prior distribution which is called randomized anchored maximum a posterior sampling. This approach uses Bayes' rule to compute a multivariate posterior by

EQUATION

with N (µ prior , Σ prior ) as multivariate prior and N (µ like , Σ like ) as multivariate likelihood. When following the MAP approach µ M AP = µ post , we can use a parameterized mean of the prior µ prior,θ0 so that the true posterior distribution is matched with

EQUATION

According to Pearce et al. (2018) suitable parameters for θ 0 can be found by setting

EQUATION

)

EQUATION

When using NNs in an anchored ensemble, the loss function needs to be modified. The typical loss function in a neural network is defined as

L = 1 N ||y -ŷ|| 2 2 + 1 N ||Γ 1/2 θ|| 2 2 (12)

where ŷ denotes the predictions of the neural net, y is the vector of the corresponding labels, θ is the flattened vector of NN parameters, and Γ is the diagonal square matrix of a L 2 regularization. Minimizing the loss in eq. 12 results in parameters that are equal to MAP estimates with a zero mean normal prior. Eq. 12 can be modified in such way that the minimized parameters can be seen as MAP estimates with non-zero centered priors

EQUATION

)

The proof is provided in Pearce et al. (2018) . As a result, randomised anchored MAP sampling with NN leads to a good approximation of the true posterior. The algorithm for training an anchored ensemble with deep neural networks is given in Algorithm 1. In the initialization, an ensemble of K neural networks with different parameters θ j,0 is created. The parameters are drawn from a distribution over the network weights θ j,0 ∼ N (µ 0 , Σ 0 ) with µ 0 according to eq. 10 and Σ 0 as presented in eq. 11 (see Alg. 1, 3-8). During training, every individual neural network is trained with the training data X and the corresponding labels Y. The loss is computed according to eq. 13 (see Alg. 1, 9-11). Mean and variance of the true posterior distribution are estimated for the ensemble by

EQUATION

where f j denotes an individual NN within the set of K NNs of the ensemble, σ2 is an estimate of the noise of the training data, ŷ(x * ) is the predicted mean of the ensemble and σ2 y denotes the predicted variance of the ensemble at test point x * .

We make use of the uncertainty estimate in eq. 15 by sampling the next state prediction from the distribution predicted by the approximated BNN, i.e. s t+1 ∼ N ( ŷ(s t , a t ), σ2 y (s t , a t )). Consequently, the rollouts sampled from the learned dynamics model will diverge much in regions of high uncertainty which prevents the policy to overfit to optimistic model inaccuracies. The efficiency of this approach was empirically proven by Kurutach et al. (2018) . µ 0 = µ prior (eq. 10), Σ 0 = Σ prior (eq. 11)

6: sample θ j,0 from N (µ 0 , Σ 0 ) 7: create neural network f j with Γ, θ j,0 8: end for # Ensemble Training: 9: for j = 1 to K do 10: train f j with X, Y, loss in eq. 13 11: end for # Ensemble Prediction:

12: ŷ(x * ) = 1 K K j=1 f j (x * ) (eq. 14) 13: σ2 y (x * ) = I σ2 + 1 K K j=1 (f j (x * ) -ŷ) T (f j (x * ) -ŷ) (eq. 15)

3 AUGMENTED MODEL-FREE REINFORCEMENT LEARNING

In this section, we describe our gradient based model-free RL algorithm that uses BNNs with randomized MAP sampling as explained in the previous section. Our motivation is to address complex and high-dimensional real robotic tasks in continuous action and state spaces. This aim requires model-free RL algorithms to become more data efficient while maintaining the same asymptotic behaviour. Our approach, model-based policy gradient enrichment, achieves this goal by amending model-free RL with an ensemble of deep NN based dynamics models. Using randomized anchored MAP sampling our approach does not only compute a point estimate of the true posterior distribution but also accounts for the shape of the posterior distribution (Pearce et al. (2018) ). This does not only provide a true Bayesian uncertainty estimate, but also prevents overfitting in the approximated stochastic dynamics model.

3.1. THE MODEL-FREE REINFORCEMENT LEARNING FRAME:

The first step of our proposed algorithm follows the commonly accepted procedure of model-free RL algorithms in which we iteratively perform trajectory rollouts to collect data and estimate the gradient (see Alg. 2, 6-7). Policy improvement is performed following the clipped 'surrogate' objective shown in eq. 5 (see Alg. 2, 12) . Additionally, we use an advantage estimator (Schulman et al. (2015b) ) to reduce variance when estimating the policy gradient. However, we do not discard the collected samples of the real environment when performing a policy update. Instead, we add the samples to our dataset D in order to train our stochastic dynamics model. In MBPGE, the batch of trajectories U PPO needed for each unique policy update is sampled partially from the real environment (see Alg. 2, 7) and partially from the approximated dynamics model (see Alg. 2, 9). First, the trajectories τ will be collected from the real environment and added to U PPO and D. In a second step, the anchored ensemble of NNs is trained based on all trajectory samples of the real environment (see Alg. 2, 8). The complete algorithm is given in Algorithm 1. This ensemble is used to collect fictitious trajectories τ which are added to U PPO . Afterwards, a standard policy update according to proximal policy optimization (PPO) (Schulman et al. (2017) ) based on the batch U PPO will be performed. Depending on the composition of sampled trajectories in the sample batch U PPO , we can effect the influence of the stochastic dynamics model on the policy update. If the learned dynamics model is inaccurate, i.e. it is corrupted by a high model bias, we want our policy update to rely more on trajectories from the real environment. Contrary, we can use an accurate dynamics model to generate the major part of the update samples from the approximated environment to reduce the sample complexity of PPO. In order to control the amount of trajectories sampled from the real and approximated environment, we utilize a simple indicator for model inaccuracies. If the agent achieves on both the real-and approximated environment about the same total reward (i.e.

R(τ ) =:

T t=1 r(s t , a t )), it indicates that the dynamics model is accurate in the states which are likely to be explored by the policy. Hence, it is sufficient to sample the major part from the learned dynamics model. We use this notion to adjust the composition of trajectory samples drawn from the real and approximated environment by the following heuristic: The number ñk of trajectories from the learned dynamics model is set to

EQUATION

R(τ ) = 1 n k-1 n k-1 i=1 R(τ i ) R(τ ) = 1 ñk-1 ñk-1 i=1 R(τ i )

with k being the index of the current PPO iteration. The hyperparameter α ∈ [0, 1] controls the maximum of trajectories which can be collected from the learned dynamics model. Consequently, the number of trajectories sampled from the real environment is set to

EQUATION

where N r encodes a fixed number of trajectories utilized for a PPO update and n min is the minimal amount of real environment trajectories per PPO update. Algorithm 2 summarizes our approach.

4. RESULTS:

In this section, we present the evaluation results of our proposed augmented model-free RL algorithms. The experiments aim to compare the performance of our new method shown in Algorithm 2 to state-of-the-art model-free and model-based algorithms. We used the following algorithms:

• run a PPO update step with the data in U PPO 13: end for our experiments we would like to evaluate how our approach compares to state-of-the-art RL algorithms in terms of sample complexity and final performance. The results are performed for a fixed trajectory length of 200 time steps. In total, all algorithms perform 4e5 steps on the environment. Figure 1 depicts the comparison of our MBPGE algorithm to prior work on pure policy gradient based RL and MB-MPO that exceeds the performance of ME-TRPO as shown in Wang et al. (2019) . Furthermore, Figure 1 illustrates the total number of samples (the samples from the real environment plus fictitious samples) our algorithm has processed for optimization.

The results clearly show that MBPGE can outperform the available alternatives. Incorporating fictitious samples from a learned stochastic dynamics can dramatically improve the raw performance of MF-RL. The necessary amount of training samples from the real environment is greatly reduced while the asymptotic performance of MF-RL algorithms is maintained. The positive effect of the stochastic dynamics model becomes apparent by the increased learning rate. As a matter of fact, the averaged return rises significantly faster for MBPGE than for the pure model-free algorithm. We believe that the increased performance is caused by two reasons. First, having no prior on the actual task, the agent needs to explore how to distinguish between high and low-rewarded actions. Since the dataset of the observed experience in an early training stage is simply not accurate enough, the Baysian uncertainty estimate of the randomized anchored MAP in the anchored ensemble will have high variance. Consequently, the expected reward under high variance of the fictitious trajectory samples is very low, even though the mean would be exactly the same as in a deterministic dynamics model. This permits a policy update in the direction of the highest gradient we are certain about while avoiding catastrophic failures due to erroneously optimistic predictions. Second, with progressing data samples the dataset of collected experiences becomes more expressive resulting in a potentially more accurate dynamics model. Figure 1 (e) illustrates that the more certain the agent is about its dynamics model the more samples it generates in the fictitious environment. The less challenging the task, the more confident is the agent about its dynamics model and the higher the factor by which the total number of samples used for policy gradient estimates is upscaled (Hopper: 10.62, Swimmer: 4.62, Half-Cheetah: 2.44, Walker: 1.94). Even for the most challenging environment our approach can double the amount of data samples based on the learned dynamics model. As a result, the raw performance of MBPGE is significantly higher.

Designing a suitable heuristic for estimating the best amount of fictitious data samples is very challenging. We tested diverse heuristics with different trajectory lengths and found that the heuristic in eq. 16 performs best (cf. 2). Additionally, we find that the number of fictitious data samples used for a policy update step is, up to a certain degree, of minor importance for learning speed but effects the asymptotic behaviour. As a consequence, we introduced a hyperparameter α in eq. 16 to manually reduce the size of fictitious data samples. Improvement to this inconvenient solution is part of our future research.

5. CONCLUSIONS:

In this paper, we introduce model-based policy gradient enrichment, an algorithm that incorporates a stochastic dynamics model to augment the dataset for estimating the current policy gradient. Our method takes advantage of randomized anchored MAP that results in an ensemble of neural networks providing a true Baysian uncertainty estimate. We presented that our approach leads to significantly faster learning while maintaining the final asymptotic performance of plain model-free RL. We further show that the stochastic dynamics model is a suitable approach to reduce model bias leading to an increased learning rate in an early training stage.

Swimmer::

The basis of our Swimmer was the Swimmer-v2 from OpenAI Gym. Its observation vector consists of all angular joint positions and velocities and our only modification was that we extended the observation vector by the x position in space (pseudo code: x = sim.data.qpos[0]). We utilized the default reward function r(s t , s t-1 , a t ) = xt-xt-1 ∆t -0.0001 a t 2 2 and no kind of early termination criterion.

Ant::

The basis of our Ant was the Ant-v2 from OpenAI Gym. Its observation vector consists of all angular joint positions and velocities as well as some contact related states. Our modification was to remove the contact states from the observation vector and to add the x position in space (pseudo code: x = sim.data.qpos[0]) to it. We utilized the reward function r(s t , s t-1 , a t ) = xt-xt-1 ∆t + 1 -0.5 a t 2 2 . The only difference of it to the default reward function is that it has no contact penalty. We additionally utilized the default early termination criterion of Ant-v2.

A.2 HYPERPARAMETERS:

If not specified differently, we used the hyperparameters as listed in Table 1 (MBPGE), Table 2 (PPO), Table 2 (TRPO) and Table 3 (MB-MPO). For our plain PPO and TRPO experiments we have not defined a fixed number of trajectories to collect for a policy update. Instead, we collected trajectories until the batch contained a certain number of time steps T min . The Anchored Ensemble requires an assumption about the prior distribution. We have set the mean µ prior of the Gaussian prior distribution to zero and its variance Σ prior according to the Xavier initialization (Glorot & Bengio (2010) ) (η in = fan-in, η out = fan-out) Σ prior,i,j = 2 η in,i,j + η out,i,j

:



Figures and Tables:

the posterior distribution of new data points y * can be inferred by marginalizing over θ p(y * |x * , X, Y ) = (Y |X, θ)p(θ)dθ .

Randomized MAP sampling with anchored ensembles 1: input: Training data X & Y, vector test data point x * , prior mean µ prior and prior covariance Σ prior , ensemble size K, data noise variance estimate σ2 2: output: estimate of mean ŷ and variance σ2 y # Ensemble Initialization: 3: Γ = σ2 Σ -1 prior 4: for j = 1 to K do 5:

Trust Region Policy Optimization (Schulman et al. (2015a)): a model-free RL algorithm with constraint policy gradient update • Proximal Policy Optimization (Schulman et al. (2017)): a model-free RL algorithm with clipped 'surrogate' objective • Model-Based Meta-Policy-Optimization (Clavera et al. (2018)): a model-based RL algorithm that uses an ensemble of dynamics model to quickly adapt to new dynamcis model with one policy gradient step The evaluation process is performed on four simulated OpenAI Gym Benchmarking tasks (Brockman et al. (2016)), i.e., Hopper, Half-cheetah, Swimmer, and Walker, which are based on the the MuJoCo physics simulator (Todorov et al. (2012)). A detailed description of the environments is given in Appendix A.1 and the hyperparameter settings are listed in Appendix A.2. In Algorithm 2 Model-based Policy Gradient Enrichment 1: input: Batch size N r ; n min ; α; PPO hyperparameters; dynamics model hyperparameters 2: initialize D as an empty dataset 3: n 0 = N r 4: ñ0 = 1 5: for k = 0 to N iter do 6: initialize U PPO as an empty dataset 7: collect n k trajectories with the current policy π on the real environment and add them to U PPO and D 8: train anchored ensemble of NNs on D (Algorithm 1) 9: collect ñk trajectories with the current policy π on the approximated environment and add them to U PPO 10: compute ñk+1 with equation 16 11: compute n k+1 with equation 17 12:

Figure 1: Comparison of learning curves of our approach to other state-of-the-art RL algorithms. The horizontal axis denotes the number of samples, the vertical axis is the average return. The maximum trajectory length is set to 200 time steps. The bottom right figure compares the number samples from the real environment to the total number of used samples for optimization by MBPGE. The result of MBPGE-Deterministic refers to our algorithm in which a deterministic dynamics model was in use, i.e. the next state was not randomly sampled from a predicted distribution but instead chosen to be the point estimate given by the NN.

Figure2: Design study on hyperparameter α in eq.16 to manually restrict the maximum possible number of fictitious data samples. The performance curves are shown for three values, α = 1, α = 0.8, and α = 0.6. The results show that finding the optimal composition of the dataset for a policy gradient estimate is non-trivial.

Peer review (iclr2020):
### Summary This paper described a model-based RL method which uses learned dynamics models to augment the replay buffer used when training a PPO agent. Specifically, the agent learns an ensemble of dynamics models, and then performs a PPO updates using a mixture of trajectories sampled from the true environment and trajectories sampled from teh learned dynamics models. The fictitious trajectories are sampled from different dynamics models in the ensemble, which helps prevent the policy from exploiting the errors of a single model. The proportion of real/fake trajectories are adapted using the ratio of the real vs. predicted rewards: if the average returns are similar, then the proportions will be approximately equal, but if they are very different then more trajectories from the real environment will be used. Overall, I think there are some interesting elements to this paper but it is currently not ready for publication. I recommend weak reject for three reasons: limited novelty, the experiments as they stand are not very convincing and the writing needs work. #### Novelty The two stated contributions are: 1. the use of principled uncertainty estimates in model-based RL (an anchored ensemble) 2. the routine which adjusts the proportion of real vs. simulated trajectories for learning the policy. In my opinion, 1. is not much of a contribution since a number of works have already made use of different types of model uncertainty in the context of model-based RL. Here they use a slightly different version (the anchored ensemble, which is similar to the randomized priors used in DQNs) but this is more of a choice of implementation than a contribution in itself (a number of methods for estimating uncertainty exist, such as regular ensembles, dropout masks, BNNs, etc). The manner in which they use the uncertainty is to perform simulations with several different models to avoid overfitting to one, and this was already proposed in the ME-TRPO paper. The idea of 2. is interesting and in general, adapting between model-based and model-free RL regimes during execution seems to have potential. However the current method seems currently under-explored and heuristic. The fact that it is sensitive to the alpha hyperparameter (Figure 2) is a bit troubling. Is there a principled way which does not require tuning this hyperparameter? Or reformulating things so that it is less sensitive? #### Experiments Although the experiments show some improvement, they are not that convincing. The improvements do not seem statistically significant for Swimmer and Walker, and the asymptotic performance is worse than PPO for Hopper. There is only half-cheetah where the algorithm shows a clear improvment. Since one of the claimed contributions is the use of the anchored ensemble, there should be an ablation experiment showing this gives an improvement over a standard ensemble, but this is not included. #### Writing: One issue with the paper is that it spends *much* too long discussing related work and preliminaries. The first 5 pages are devoted to this and the proposed method is only introduced on page 6! Some detailed comments: - Paragraph 1 in the intro should be drastically cut. The sentence *Recently...challenging tasks* could be followed by references and then a sentence discussing the sample inefficiency and then moving on to the next paragraph. Most readers will be familiar with DQN, actor critic etc. - Same for paragraphs 2 and 3. A short discussion of the general idea behind model-based RL and its improved sample efficiency, the issue of the policy exploiting model errors, and some references are sufficient. - Similarly, section 2.1 is mostly unnecessary. It isn*t necessary to detail the updates for standard algorithms such as PPO/TRPO unless they are useful for the proposed algorithm. - Section 2.2 is also too long. They main idea is very simple and is summarized in Equation 13.