AUGMENTED POLICY GRADIENT METHODS FOR EFFICIENT REINFORCEMENT LEARNING
Authors: Kai Lagemann
Gregor Roering
Christoph Henke
Rene Vossen
Frank Hees
Venue:  (2020)
Keywords: model-free reinforcement learning
model-based reinforcement learning
Baysian neural network
deep learning
reinforcement learning

Abstract: We propose a new mixture of model-based and model-free reinforcement learning
(RL) algorithms that combines the strengths of both RL methods. Our goal is to reduce the sample complexity of model-free approaches utilizing fictitious trajectory
rollouts performed on a learned dynamics model to improve the data efficiency of
policy gradient methods while maintaining the same asymptotic behaviour. We
suggest to use a special type of uncertainty quantification by a stochastic dynamics
model in which the next state prediction is randomly drawn from the distribution
predicted by the dynamics model. As a result, the negative effect of exploiting
erroneously optimistic regions in the dynamics model is addressed by next state
predictions based on an uncertainty aware ensemble of dynamics models. The
influence of the ensemble of dynamics models on the policy update is controlled
by adjusting the number of virtually performed rollouts in the next iteration according to the ratio of the real and virtual total reward. Our approach, which we
call Model-Based Policy Gradient Enrichment (MBPGE), is tested on a collection of benchmark tests including simulated robotic locomotion. We compare our
approach to plain model-free algorithms and a model-based one. Our evaluation
shows that MBPGE leads to higher learning rates in an early training stage and an
improved asymptotic behaviour.

Under review as a conference paper at ICLR 2020
AUGMENTED POLICY GRADIENT METHODS FOR EFFICIENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose new mixture of model-based and model-free reinforcement learning (RL) algorithms that combines the strengths of both RL methods. Our goal is to re- duce the sample complexity model-free approaches utilizing fictitious trajectory rollouts performed on learned dynamics model to improve the data efficiency of policy gradient methods while maintaining the same asymptotic behaviour. We suggest to use special type of uncertainty quantification by stochastic dynamics model in which the next state prediction is randomly drawn from the distribution predicted by the dynamics model. As a result, the negative effect of exploiting erroneously optimistic regions in the dynamics model addressed by next state predictions based influence of the ensemble of dynamics models on the policy update is controlled by adjusting the number of virtually performed rollouts in the next iteration ac- cording to the ratio of the real and virtual total reward. Our approach, which we call Model-Based Policy Gradient Enrichment (MBPGE), is tested on a collec- tion of benchmark tests including simulated robotic locomotion. We compare our approach to plain model-free algorithms and model- based one. Our evaluation shows that MBPGE leads to higher learning rates in an early training stage and an improved asymptotic behaviour.
uncertainty aware ensemble of dynamics models. The
INTRODUCTION
Reinforcement Learning (RL) can be broadly classified in two categories: model-free RL, which uses experience from interacting with the environment to learn a value function, and model-based RL, which uses experience to approximate the environment by model (Deisenroth & Rasmussen (2011). Recently, impressive results were achieved applying model-free RL aproaches to challeng- ing tasks. A RL agent with a neural network (NN) based Q-value representation was introduced in Mnih et al. (2015) which was able to reach equivalent or higher performance on many Atari games compared to humans by learning a state function based on raw pixels. Further modifications addressed the problem of Q-value overestimation (van Hasselt al. (2015)) dropping significant experience (Schaul et al. (2015)) and extended the approach to asynchronous methods (Mnih et al.) (2016) improve the overall performance. Beside learning value function, it is possible to the gradient of the RL objective to update the policy (Peters & Schaal (2006), Suttonn & Barto (210)). Silver et al. (2014). Lillicrap et al.] (2015)). Actor-critic methods aim at combining the strong points of policy gradients and an approximated value function, in which the critic learns an approximated value function that used update the policy of the actor in the direction of its gradient (Konda & Tsitsiklis (2000), Sccuulann et al. (2015)) Mnih et al. (006)) Gu et al. 2016)) )
In contrast, model-based RL can be significantly more data efficient. It is notable that dynamics model is not necessarily globally valid. guided policy search (GPS), time-varying linear dynamics models are learned from multiple rollouts with diverse start and goal conditions (Levine) & Koltun| (2013a), Levine & Koltun (2013b), Levine & Koltun (2014). These trajectory samples are then used to fit time-varying Gaussian dynamics model leading to a Gaussian distribution over trajectories, even for unknown dynamics (Levine Abbeel (014)). The neural network policy is trained in supervised fashion such that local policies are improved minimizing cost function bounded by the trust region of the KL-divergence, and the weights of the NN policy are adjusted to mimick the trajectory distribution. Further improvements extended GPS to broader robust

---

Under review as a conference paper at ICLR 2020
state space by introducing new policy representation which is called general motor reflex (GMR) (Ennen et al. (2018)). In a first step, the state space is encoded to a latent state representation by variational autoencoder. Then, translation model predicts motor reflex parameters in order to adjust a Gaussian controller resulting in robust trajectories even outside the distribution of training samples.
However, modelling errors can reduce the efficiency and benefit of model-based RL algorithms since model imperfections are exploited (Deisenroth & Rasmussen (201T). [Schneider (196), Atkessn & Santamaria (1997)) known as model-bias. With midel-bias being result of overfitting, one approach to solve this problem is to incorporate explicit uncertainty estimates by using probabilistic dynamies model. Probabilistic inference for learning control (PILCO) (Deisenroth & Rasmussen (201T)) uses Gaussian process (GP) to represent the dynamics of the environment. Such a model provides predictions with true Bayesian uncertainty estimates but suffer from the curse of dimensionality (Calandra et al. (2014)). Recently, Bayesian neural networks (BNN) received much attention. Approximate Bayesian techniques, e.g. variational inference (Graves (20TT) or
Monte-Carlo dropout (Gal & Ghahramani (2015), come at the drawback that correlations between parameters are not maintained, while scaling abilities of the Bayesian 'gold standard' inference method
(MCMC) are limited (Hinton & Neal (1995). Adding
regularization term to the loss function of a NN, e.g. Tikhonov regularization, permits maximum posterior (MAP) estimates of the parameters since the solution of the regularized loss function is equivalent to the maximum of the posterior density function (Bardsley) (2012)). However, the regularized solution leads to a point estimator of the true posterior density. This insufficiency can be addressed when sampling from the true posterior distribution using a MCMC method providing additional information about the shape of the posterior but is computationally more demanding.
Our approach inspired by recent advances, Model-Ensemball Trust-Region Policy Optimization (ME-TRPO) (Kurutach et al. (2018)), Model Based Meta Policy Optimization (MB-MPO) CClavera) cct al. (2018) ) Model-assisted Boostrapped DDPG (MA-BDDPG) (Kalweit & Boedecker (2017)) )
and Model-Based Value Expansion (MVE) (Feinberg et al. (2118). In ME-TRPO, set of samples from the real environment is used to train an ensemble of NNs. In a second iteration, fictitious tra- jectory rollouts are generated to update the policy using Trust Region Policy Optimization (TRPO)
(Schulman et al. (2015a), i.e., the policy update is only based on samples from the approximated dynamics model of the environment. In MB-MPO, the meta policy learning framework 'model agnostic meta learning' (Finn et al.] (2017)) is used to train a policy on an ensemble of learned dynamics models. The notion is that the meta-learning frame of the dynamics models is used to adapt quickly to any new dynamics model within one update step. The overall similarities of all dynamics models are incorporated in the meta-model that then adjusted dynamics model. MA-BDDPG employs the same idea to augment the dataset to achieve better data efficiency. In contrast to our approach, MA-BDDPG makes use of an actor-critic method in which the policy is represented deterministically. Updates in the actor and critic network follow the advances of Deep Q-Networks by refering to time-delayed target networks. A further difference is in the procedure for estimating uncertainty. i.e. Kalweit & Boedecker (2017) applies bootstrapping
create the fitted
(Efron & Tibshirani (1998)) to get a distribution over Q-functions while we use an anchored ensemble of NNs. MVE improves value estimates assuming that an approximated dynamics model can be used in training a critic up to a depth H in which we are certain about the model accuracy. Contrary to our approach, MVE defines its dynamics model deterministic neural network.
Our new algorithm MBPGE is a model-free RL algorithm that is augmented by true Bayesian ensemble of dynamics models to generate fictitious trajectory rollouts to decrease the sample com- plexity. In contrast to ME-TRPO we take advantage of very recent inference advance based on randomised anchored MAP sampling that allows for a true Bayesian uncertainty estimate of the true posterior (Pearce et al. (2018). As a result, our stochastic dynamics model reduces the effect of policy overfitting as model inaccuracies will be addressed by highly random next state predictions. We have seen that ill ill-iistrbutted datasets, i.c., data points of some regions in the state space are overrepresented while others are underrepresented, might lead to enormous fictitious total rewards that diverge heavily from those of the real environment samples due to overfitting dynamics mod- els. We tackle this issue by introducing a ratio of fictitious and real rewards that our algorithm
2

---

Under review as a conference paper at ICLR 2020
adjusts naturally to become more model-free if the dynamics model is inaccurate and acts in more model-based way the approximation of the environment is accurate. To summarize, our main contributions are:
1. A novel model-free RL algorithm that uses a true Bayesian representation for the stochastic dynamics model and
2. a routine that adjusts the composition of the dataset for policy gradient estimate by weighting the trajectory samples based on the ratio of the total rewards of the real envi- ronment and the approximated model.
We derive our approach by reviewing the background in constrained policy gradient methods and randomized maximum posterior sampling. Then, present augmented model-free reinforce- ment learning algorithm detail. Finally, we evaluate our approach on challenging simulated tasks in continuous control domain.
2 PRELIMINARIES
We formulate the underlying problem of our tasks discrete-time finite-horizon Markov decision process (MDP), defined by the tuple (S,A, P.r. Po: 7, T). Here, we assume that our agent acts in a stochastic environment with continuous state space S C Rn and continuous action space A C RRm. Additionally, we assume that an unknown dynamics model P( a) defined for environment and distribution, y is the discount factor, T is the finite horizon and Ty (a|s) denotes our neural network policy, parameterized by 0, distributed over actions a, and conditioned states The overall aim is to learn optimal policy such that the expected total reward (TT) Brnx [T= rs(st, at)
given reward function r(s,, at). Let Po denote the initial state
maximized, where т = 80, ao ... dT-1, ST denotes trajectory with so ~ po. at me and 8141 P(st,a)).
2.1 POLICY GRADIENT METHODS
The maximization problem of the RL objective can be solved by performing gradient ascent com- puting the derivative with respect to the policy parameters 0. The major problem of policy gradient methods is to find good estimator for the gradient since the variance of the gradient estimator increases with the time horizon. Fortunately, variance reduction scheme for policy gradients, gen- eralized advantage estimator (GAE) (Schulman et al. (2015b)), reduces the variance significantly while only introducing a small bias. It is possible to tradc-off between bias and variance by using a X-retum (Sutton & Barto (2010)) Extending the idea of an n-step return, one can weight each n-step update by - and normalize the sum by (1 - A) resulting in
R =1 X)
R
(1)
n=1
The finite horizon version assumes that all rewards after time step T equal 0, leading
T-t-1
=
Σ
X
R
T- R Tt
(2)
n=1
Setting results in the biased single step return RO with low variance, while À 1 corre- sponds to the unbiased Monte-Carlo return 12(20) with high variance. GAB can be derived when estimating the advantage A (st, 0z) = Q (Ste V (st) with the X-retur. The gradient of the
objective now turns to
T
g = V
E
Σ rt
E
Σ
A (a
(3)
Usually, we intend to update our parameters only by small changes. However, instead of perform- ing update steps in parameter space using the Fuclidean distance, it is much more convenient to
3

---

Under review as a conference paper at ICLR 2020
measure closeness between the current policy and the updated policy in terms of distances between distributions. The intuition behind is that distances between distributions, e.g. Kullback-Leiin
divergence (Kullback & Leibler (195T)) Hellinger distance (Hellinger (909)). invariant to scaling or shifting. This type of approach is known as natural policy gradients (Kakade (2001),
SSutoon et al. (2000), PPterrs
(Pteers & Schaal (2006). Trust region policy optimization (TRPO) utilizes this idea and maximizes its 'surrogate' objective function by constraining the allowed policy update step
(Schulman al. (2015a)
(o(ar A. ood (0r))))
max R.
8.1. IR, [KT.[TT.. (*1s)), (o(+)s))]] S 88.
(4)
TRPO builds on the concept of guaranteed monotonic improvement. The notion is that we can compute an upper bound for the error of diverging policies. Therefore, we can guarantee policy improvement as long as we optimize the local approximation within a trusted region. The proof given in |Schulman et al. (2015a). PPO combines the appealing guarantee of improvement with much simpler implementation (Schulman et al. (2017)) Instead of maximizing the 'surrogate' function in eq. it uses the following clipped objective
(() E. (1i --,, +)
(5)
xo(ailes) is in the interval Clipping the probability ensures that the probability ratio re(0) Tesra 2e)
[1- 1 + e), while taking the minimum of the clipped and unclipped objective leads to lower bound of the objective with the same convenient effect as constraining the 'surrogate' objective in TRPO.
2.2 MODEL-BASED REINFORCEMENT LEARNING WITH AN ANCHORED ENSEMBLE OF DEEP NEURAL NETWORKS
Contrary to model-free RL, model-based RL utilizes the interactions with the environment to learn an approximated model. Model-based approaches benefit from a significantly lower sample com- plexity. However, learning accurate model of the environment can be very challenging for certain domains. As consequence, model-based algorithms often show worse asymptotic behaviour. To date, there are several diverse approaches to represent the dynamics model and proper choice is often crucial. Dynamics model in RL must reliably fit to datasets with a low and high number of data points: during the first few iterations only a limited number of data points available, causing expressive general function approximators, e.g. deep neural networks, to overfit. With progressing iterations simple function approximators tend to underfit complex system dynamies.
Bayesian regression aims to find the parameters 0 of a function y* fo(x*), which are likely to have generated the output Y given input X. This can be achieved by updating a prior distribution over the parameters of the neural network 0, p(0) by using Bayes Theorem. If it is possible to compute the posterior distribution
p (() P( 0)p(()
(6)
x(
the posterior distribution of new data points y* can be inferred by marginalizing over 0
ply * Ix+ |x+ X,Y) f (x11x,0)x(0)00
Previous work revealed that exploiting uncertainty estimation in RL can increase the sample effi ciency dramatically (Deisenroth Rasmussen (201T)) Gaussian Processes (Rasmussen & Williams 2008)) perform well in low data regimes but do not scale to complex tasks with high dimensionality Calandra et al. (2014). On the other hand, probabilistic NNs can approximate arbitrarily complex dynamics but come the cost that true Bayesian uncertainty estimate cannot be provided (Gal & Ghahramani (201)). The computation of the integral in eq. 7 can in general only be approximated numerically under trade-off between accuracy and performance.
A Baysian neural network (BNN) can be approximated by using a diverse ensemble F (Ji,.., JR} of NNs. Ensembling provides uncertainty estimate in such way that the variance
4

---

Under review as a conference paper at ICLR 2020
of the ensemble's prediction is considered to be its uncertainty. Technically, this approach is not Bayesian (aal (016)). This drawback was addressed in Pearce et al. (2018) by regularizing the parameters about values drawn from prior distribution which is called randomized anchored max- imum a posterior sampling. This approach uses Bayes' rule to compute a multivariate posterior by
N (Ipost Exostt X N (Uprior, Exprior W (pttkee E(kee
(8)
with N (ppriorr Eprior) as multivariate prior and M( E(iee as multivariate likelihood. When following the MAP approach AP = Mposts we can use parameterized mean of the prior pprioon8 so that the true posterior distribution is matched with
AMAP (6)) = ( + Sprtor) (I
- lske + E priorrprior
According to Pearce et al. (2018) suitable parameters for do can be found by setting E (pmar((0)) (0n) Ppost and (M Eosst resulting in On N(u, En) with
140 = Pprior
(10)
Σg
Σ prior
Σ prior like
E Eprior
(11)
When using NNs in anchored ensemble, the loss function needs to be modified. The typical loss function in a neural network is defined as
y 011
r 2012
(12)
where ý denotes the predictions of the neural net, y is the vector of the corresponding labels, 0 is the flattened vector of NN parameters, and r is the diagonal square matrix of T.2 regularization. Minimizing the loss in eq. 12 results in parameters that are equal to MAP estimates with a zero mean normal prior. Eq. 12 can be modified in such way that the minimized parameters can be seen as MAP estimates with non-zero centered priors
Lanchored
N y
+ N (0-000))))
(13)
The proof is provided in Pearce et al. 2018 As a result, randomised anchored MAP sampling with NN leads to good approximation of the true posterior. The algorithm for training an anchored ensemble with deep neural networks is given in Algorithm [ In the initialization, an ensemble of K neural networks with different parameters 0,,0 is created. The parameters are drawn from distribution over the network weights 0 ~ N(Ho: Σg) with μ4o according to eq. 10 and Eg as presented in eq. [ (see Alg. оل During training, every individual neural network trained with the training data X and the corresponding labels Y. The loss is computed according to eq. [3 (see
Alg.
9 11). 1). Mean and variance of the true posterior distribution estimated for the ensemble by
K
8(x*)
1
Σ
fx*))
(14)
K
1 k
(x(()) To?
K (1,(x)) -) - -))
(15)
where f, denotes individual NN within the set of K NNs of the ensemble, o? is estimate of the noise of the training data, ý (x**) is the predicted mean of the ensemble and oū denotes the predicted variance of the ensemble at test point æ=
We make use of the uncertainty estimate in eq. 15 by sampling the next state prediction from the dis- tribution predicted by the approximated BNN, I.e. $2+1 ~ "N(gss, az), (st, at)). Consequently, the rollouts sampled from the learned dynamics model will diverge much in regions of high uncer- tainty which prevents the policy to overfit to optimistic model inaccuracies. The efficiency this approach was empirically proven by Kurutach et al. (2018).
5

---

Under review as a conference paper at ICLR 2020
Algorithm Ranndmiized MAP sampling with anchored ensembles
input: Training data & Y, vector test data point x*, prior mean Mprior and prior covariance E priior ensemble size K, data noise variance estimate o?
2: output: estimate of mean y and variance oz
# Ensemble Initialization:
3: I 62- prior
4: for = 1 K do
5: μ40 = Mpriior (eq. 10 Σo = E Eprior (eq.
6: sample 3,,0 from N (14, En)
7: create neural network f, with r, 0.00
8: end for
# Ensemble Training:
9: for
to K do
10: train f, with X, Y, loss in eq.
end for
# Ensemble Prediction:
((x*) E/= fx(x*) (eq.
(xx)) 10 + R 1 (x)(x*) (fz(() ()) (eq. 15
3 AUGMENTED MODEL-FREI REINFORCEMENT LEARNING
In this section, describe our gradient based model-free RL algorithm that uses BNNs with ran- domized MAP sampling as explained in the previous section. Our motivation is to address complex and digh-diimesioonl real robotic tasks in continuous action and state spaces. This aim requires model-free RL algorithms to become more data efficient while maintaining the same asymptotic behaviour. Our approach, model-based policy gradient enrichment, achieves this goal by amending model-free RL with ensemble of deep NN based dynamics models. Using randomized anchored MAP sampling our approach does not only compute a point estimate of the true posterior distribu- tion but also accounts for the shape of the posterior distribution (Pearce et al. (2018). This does not only provide a true Bayesian uncertainty estimate, but also prevents overfitting in the approximated stochastic dynamics model.
3.1 THE MODEL-FREE REINFORCEMEN LEARNING FRAME
The first step of our proposed algorithm follows the commonly accepted procedure of model-free RL algorithms in which we iteratively perform trajectory rollouts collect data and estimate the
gradient (see Alg. 2 6 7
Policy improvement is performed following the clipped 'surrogate"
objective shown eq. (see Alg. 2 T2). Additionally, we use an advantage estimator (Schulman)
er al. (2055b) to reduce variance when estimating the policy gradient. However, we do not discard the collected samples of the real environment when performing a policy update. Instead, we add dataset D in order train stochastic dynamics model. In MBPGE, the batch of trajectories Uppo needed for each unique policy update sampled partially from the real environment (see Alg. 2 7 and partially from the approximated dynamics model (see Alg. [ 9.. First, the trajectories т will be collected from the real environment and added to Uppo and D. In second step, the anchored ensemble of NNs is trained based on all trajectory samples of the real environment (see Alg. 8). The complete algorithm is given in Algorithm[[ []] This ensemble is used collect fictitious trajectories F which are added Uppo. Afterwards, standard policy update according to proximal policy optimization (PPO) ISchulman et al. (2017)) based on the batch Uppo will be performed. Depending on the composition of sampled trajectories the sample batch Uppo, we can effect the influence of the stochastic dynamics model on the policy update. If the learned dynamics model is inaccurate, i.e. it is corrupted by a high model bias, we want our policy update to rely more on trajectories from the real environment. Contrary, we can an accurate dynamics model to generate the major part the update samples from the approximated environment to reduce the sample complexity of PPO. In order to control the amount of trajectories sampled from the real and approximated environment, we utilize a simple indicator for model inaccuracies. the agent achieves on both the real- and approximated environment about the same total reward (i.e.
the samples

---

Under review as a conference paper at ICLR 2020
((s,, R(T) = -(T)=Ei a,)), it indicates that the dynamies model is accurate in the states which likely to explored by the policy. Hence, it is sufficient to sample the major part from the learned dynamics model. We use this notion to adjust the composition of trajectory samples drawn from the real and approximated environment by the following heuristic: The number 7g of trajectories from the learned dynamics model set to
R(()) R(f)
R(F)) R(T)
n& max
min
aNv,1
(16)
n2-1
8p-1 R(A))
R(T)
Σ R(T))
R(f) = 7
=
nk-1
with k being the index of the current PPO iteration. The hyperparameter E [0. 1] controls the maximum of trajectories which can be collected from the learned dynamics model. Consequently, the number of trajectories sampled from the real environment is set to
Th = max N,- Tk; Romin)
(17)
where N, encodes a fixed number trajectories utilized for PPO update and Remin is the minimal amount of real environment trajectories per PPO update. Algorithm summarizes our approach.
RESULTS
In this section, we present the evaluation results of our proposed augmented model-free RL algo- rithms. The experiments aim to compare the performance of our new method shown in Algorithm state-of-the model-free and model-based algorithms. We used the following algorithms:
Trust Region Policy Optimization (Schulman et al. (2015a)): model-free RL algorithm with constraint policy gradient update
Proximal Policy Optimization (Schulman et al. 2017) ) a model-free RL algorithm with clipped 'surrogate' objective
model-based RL algo- rithm that uses an ensemble of dynamics model to quickly adapt to new dynamcis model with one policy gradient step
Model-Basse
Optimization (Clavera al. 2018 ):
The evaluation process is performed on four simulated OpenAl Gym Benchmarking tasks (Brock- man et al.] (2016)), i.e., Hopper, Half-cheetah, Swimmer, and Walker, which are based on the the MuloCo physies simulator (Todorov al. (2012)) A detailed description of the environments given in Appendix A.I and the hyperparameter settings are listed in Appendix A.2 In
Algorithm 2 Model-based Policy Gradient Enrichment
1: input: Batch size Ny; Amini a; PPO hyperparameters: dynamics model hyperparameters
2: initialize D as an empty dataset
3: no N,
4: 7og =1
Niter do 6: initialize Uppo an empty dataset
5: for k
7: collect ną trajectories with the current policy on the real environment and add them to
Uppo and D
8: train anchored ensemble of NNs on D (Algorithm I
9:
collect स. trajectories with the current policy the approximated environment and add
them to Uppo
10: compute ñę+1 with equation 16
11: compute ng+11 with equation 17
12:
PPO update step with the data in Uppo
13: end for

---

Under review as a conference paper at ICLR 2020
Axtrng Iechi
700
092
302
X.0
step
K10
step
X10
(a) Swimmer
(b) Hopper
(c) Walker
Avemze relumr
mumbrr ammples
20
200
dime stop
sumples from rea cnr
Half-Cheetah
(e) real fictitious samples
MBPGE-Deee Deterministic
Figure 1: Comparison of learning curves of approach other state-of-the-ee of-tr--rrt RL algorithms. The horizontal axis denotes the number of samples, the vertical axis is the average return. The maxi- mum trajectory length set 200 time steps. The bottom right figure compares the number samples from the real environment to the total number of used samples for optimization by MBPGE. The result of MBPGE-Determinisst
refers our algorithm in which deterministic dynamics model was in use, i.c. the next state was not randomly sampled from a predicted distribution but instead chosen be the point estimate given by the NN.
Average retuin
700
Average return
4000
600
3000
400
2000
200
1000
ORTE
ORTE
OUTS a 0.6
OUTS a 0.6
100
PPO
PPO
0
time step
time step x10
(a) Walker2D: maximum trajectory length 200
(b) Walker2D: maximum trajectory length = 1000
Figure 2: Design study on hyperparameter in 16 to manually restrict the maximum possible
number of fictitious data samples. The performance curves are shown for three values, a 1, 0.8, and a 0.6. The results show that finding the optimal composition of the dataset for policy gradient estimate is non-trivial.
8

---

Under review as a conference paper at ICLR 2020
experiments we would like evaluate how approach compares to ofate-of-hhe-ar RL algorithms in terms of sample complexity and final performance. The results are performed for fixed trajectory length of 200 time steps. In total, all algorithms perform 4e5 steps the environment. Figure 1 depicts the comparison of our MBPGE algorithm to prior work on pure policy gradient based RI. and MB-MPO that exceeds the performance of ME-TRPO in [Wang et al.J (2019). Furthermore, Figure [ illustrates the total number of samples (the sam- ples from the real environment plus fictitious samples) our algorithm has processed for optimization.
shown
The results clearly show that MBPGE can outperform the available alternatives. Incorporating fic- titious samples from learned stochastic dynamics can dramatically improve the raw performance of ME-RI.. The necessary amount of training samples from the real environment greatly reduced while the asymptotic performance of MF-RL algorithms is maintained. The positive effect of the stochastic dynamics model becomes apparent by the increased learning rate. As a matter of fact, the averaged return rises significantly faster for MBPGE than for the pure model-free algorithm. We believe that the increased performance is caused by two reasons. First, having no prior on the actual task, the agent needs to explore how to distinguish between high and rowrrewadeed actions. Since the dataset of the observed experience in an early training stage is simply not accurate enough, the Baysian uncertainty estimate of the randomized anchored MAP in the anchored ensemble will have high variance. Consequently, the expected reward under high variance of the fictitious trajectory samples very low, even though the mean would be exactly the same as in a deterministic dynamics model. This permits a policy update in the direction of the highest gradient we are certain about while avoiding catastrophic failures due erroneously optimistic predictions. Second, with progressing data samples the dataset of collected experiences becomes more expressive resulting in a potentially more accurate dynamics model. Figure [[e]] illustrates that the more certain the agent is about its dynamics model the more samples it generates in the fictitious environment. The less challenging the task, the more confident the agent about its dynamics model and the higher the factor by which the total number of samples used for policy gradient estimates is upscaled (Hopper: 10.62, Swimmer: 4.62. Half-Chectah: 2.44, Walker: 1.94). Even for the most challenging environment our approach can double the amount of data samples based on the learned dynamics model. As a result, the raw performance of MBPGE is significantly higher.
Designing a suitable heuristic for estimating the best amount of fictitious data samples is very chal- lenging. We tested diverse heuristics with different trajectory lengths and found that the heuristic in eq. [6]performs best (cf. [ Additionally, we find that the number of fictitious data samples used for a policy update step is, up certain degree, of minor importance for learning speed but effects the asymptotic behaviour. As a consequence, we introduced a hyperparameter a in eq. to manually reduce the size of fictitious data samples. Improvement to this inconvenient solution part of our future research.
CONCLUSIONS
In this paper, we introduce model-based policy gradient enrichment, an algorithm that incorporates a stochastic dynamics model to augment the dataset for estimating the current policy gradient. Our method takes advantage of randomized anchored MAP that results an ensemble of neural networks providing true Baysian uncertainty estimate. We presented that our approach leads to significantly faster learning while maintaining the final asymptotic performance of plain model-free RI. We further show that the stochastic dynamics model is a suitable approach to reduce model bias leading to an increased learning rate in an carly training stage.
ACKNOWLEDGMENTS
We thank Maren Bennewitz for helpful discussions and feeback, and Christian Lagemann for com- ments that greatly improved the manuscript.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Christopher G. Atkeson and Juan Carlos Santamaria. A comparison of direct and model-based re- inforcement learning- In IN INTERNATIONAL. CONFERENCE ON ROBOTICS AND AUTOMA- TION, pp. 3557-3564. IEEE Press, 1997.
Johnathan M. Bardsley, Meme-based image reconstruction with uncertainty quantification. SIAM Journal on Scientific Computing, 340):A1316-A1332, 2012. ISSN 1064-8275. doi: 10.1137/
11085760x.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Marc Peter Deisenroth. Manifold gaus- sian processes for regression, 2014.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinfrrmmeet learning via meta-policy optimization. CoRR, abs/ URL htttttt////xii>>> org abs/1809..... 05214
2018.
Marc Deisenroth and Carl Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In ICML, pp. 465-472, 01 2011.
Bradley Efron and Robert Tibshirani. An introduction to the bootstrap, volume 57 of Monographs on statisties and applied probability. Chapman & Hall, Boca Raton, Fla., [nachdr.] edition, 1998.
ISBN 978- 278-0412042317.
Philipp Ennen, Pia Bresenitz, René Vossen, and Frank Hees. Learning robust manipulation skills with guided policy search via generative motor reflexes. CoRR, abs/1 009/009995,1, 2018. URL
http:://arxii..rr//bb////// www.oorggggbboooooo 05714
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. CoRR, abs/1803.00101,, 2018. http:://aax org wtwpr/g/abb////// 00101
Chelsea Finn, Picter Abbeel, and Sergey Levine, Model-agnostic meta-learning for fast adaptation of deep networks. CoRR, abs/1703.03400, 2017. URL
wbs./or////bb.ao..oo.o....s...ea ab5/1703.
03400
Justin Fu, Sergey Levine, and Pieter Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. CoRR, abs/1509.06841, 2015. URL http:
06841
Yarin Gal. Uncertainty in deep learning. 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning, 2015.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (cds.), Proceedings of the Thirteenth Interna- tional Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL http::://roceeednnneee/// .or.mmmrr. wwt..m.pp.ee//////rrroee/o htt Ca. html
Alex Graves, Practical variational inference for neural networks. In Proceedings of the 24th In- lernational Conference on Neural Information Processing Systems, NIPS' 11, pp. 2348-2356, USA, 2011. Curran Associates Inc. ISBN 978-1-61839-599.1 http:t/:// //lt:://d... acm org/
Shixiang Gu, Timothy P. Lillicrap, Zoubin Ghahramani, Richard E. Turner, and Sergey Levine. Q- prop: Sample-efficient policy gradient with an off off-policy critic. CoRR, abs/1611.00227,, 2016.
htt.::tt::///rtt www.vvoorrggaaao aergg/abs/le 02247
10

---

Under review as a conference paper at ICLR 2020
E. Hellinger.
Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen. Journal für die reine und angewandte Mathematik (Crelles Journal), 1909(136):
210-271, 1909. ISSN 0075-4102. doi: 19.5515//nll.190.11....
Geoffrey E. Hinton and Radford M. Neal. Bayesian learning for neural networks. 1995.
Sham Kakade. natural policy gradient. volume 14, pp. 1531-1538, 01 2001.
Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continuous deep rein- forcement learning. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), Proceedings of the 1st Anmual Conference on Robot Learning, volume 78 of Proceedings of Machine Learn- ing Research, pp. 195- PMLR, 13-15 Nov 2017. URL http:::/prrocee..gg....... mllr.
Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In S. A. Solla, T. K. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems 12, pp. 1008-1014. MIT Press, 2000. URL http::/papers .ips. cc/paper/ pdf
1766-actor--rriicaaalooe
S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statis- tics, 22(1):79-8)) 86, 1951. ISSN 0003-4851. doi: 10.1214/aoms/ 1777296947
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. CoRR, abs/1802. 10592, 2018. URL htt.:tttt//axxi.orrg//
abs/1802.10592
Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 1071-1079. Curran Associates, Inc., 2014. URL http: http://paaprrsaaeeee nips wwpp.ccpppppprrr 5444 1144-Learring-neuue neural wewerrallntttttwwooooooeeooe newwork-policies with wow.ded-polccy-serreeeeeeee pdf
Sergey Levine and Vladlen Koltun. Guided policy search. In Sanjoy Dasgupta and David McAllester
(eds.). Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1-9, Atlanta, Georgia, USA, 17-19 Jun 2013a. mlr press wttp//////ee//ee////ee/ee html
PMLR. /ttphtt::///rooeeeiii
Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimiza- tion. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26. pp. 207- 215. Curran Associates, Inc., 2013b. URL httt::://papeeee http:/pppprrr..ippssccpppppppeee pdf]
Sergey Levine and Vladlen Koltun. Learning complex neural network policies with trajectory opti- mization. volume 3, 06 2014.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CORR, 09 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Joannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learn- ing. Nature, 51::529 EP -, 2015. doi: 10. 10.1008/nature14233 URL 1038/naturel 42366
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, abs/ 1o0./168..1783, 2016. URL URL httphttpp//axiiii http://ttw.....o///////22222/ 01783
Tim Pearce, Mohamed Zaki, Alexandra Brintrup, Nicolas Anastassacos, and Andy Neely. Uncer- tainty in neural networks: Bayesian ensembling, 2018.

---

Under review as a conference paper at ICLR 2020
J. Peters and S. Schaal. Policy gradient methods for robotics. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2219-2225, Oct 2006. doi: 10.1109/IROS. 2006.282564.
Lutz Prechelt. Early Stopping But When?. pp. 53 67. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-35289 8. doi: 1110077978-3-6422333 5. URL
hitpps:/ddo wtt.a//.org/1111007791001000 1007/978 342235289---
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning- Adaptive computation and machine learning. MIT Press, Cambridge, Mass., 3. print edition, 2008. ISBN 0-262-1825344
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. CoRR, abs/151 11111.0550 .05952, 2015.
Jeff G. Schneider. Exploiting model uncertainty estimates for safe dynamic control learning. In Proceedings of the 9th International Conference on Neural Information Processing Systems, NIPS 96, pp. 1047-1053, Cambridge, MA, USA, 1996. MIT Press. URL http::tp//// //dl. aam..ccccmmmmm... acm. org/eitation. wtt.aaaaa.aa..c..ooooooo........
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05477, 2015a. http::://aa....oo////ao/a//
05477
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. 000.0450,, 06 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, ubs/1707 .6347, 2017. URL http: org/ abs/ 1707. 06347
David Silver, Guy Lever, Nicolas Heess, Thomas Degris. Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Confer- ence on International Conference on Machine Learning Volume 32, ICML'114 pp. 1-387- 1-395. JMLR.org, 2014. URI. httt:://dl acm. www.:aaw......aaae.eeeeeeee wttp://iaaatooo..mm..d/o//////// 3044850
Richard Sutton, David Mcallester, Satinder Singh, Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Adv. Neural Inf. Process. Syst, 12, 02 2000.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. Bradford book. MIT Press, Cambridge, Mass., [nachdr.] edition, 2010. ISBN 9780262193988
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based con- trol. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. CoRR, abs/1500.064611, 2015. URL http::////// arxiv.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shun- shi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking mosel-based rein- forcement learning. CoRR, abs/1900.00057, 2019. URL http:: http://xxxxv.oorob/////0// 02657
12

---

Under review as a conference paper at ICLR 2020
Appendices
A EXPERIMENTAL DESCRIPTION
A.1 ENVIRONMENTS
Our method requires the reward of time step to be known function of the form 8t ae) or r(,, (t). Otherwise, would be necessary to introduce another regression model for learning the reward function. Hence, we had to manipulate the default environments slightly as described in the following.
Hopper:
The basis of our Hopper was the Hopper-v2 from OpenAl Gym. Its observation vector consists of all angular joint positions and velocities and our only modification was that we extended the observation vector by the æ position in space (pseudo code: sim.data.qpons data.qpos((O). We utilized the default reward function r(s,, St-1, 0c)
0.001 |a, and early termination criterion.
Walker:
The basis of our Walker was the Walker2d-v2 from OpenAl Gym. Its observation vector consists of all angular joint positions and velocities and our only modification was that we extended the observation vector by the v position in space (pseudo code: sim.data. (.t).. We utilized the default reward function r r8s,, 0.) 2t 11-1 +1 0.001 |ax112 l12 and early termination criterion.
A
HallCheetah:
The basis of our HalfCheetah was the HalfCheetah-v2 from OpenAl Gym. Its observation vector that we extended
consists of all angular joint positions and velocities and only modification the observation vector by the x position in space (pseudo code: = sim.data.qpos (pos(O).... We utilized the default reward function r(s;, St-1, 0x) -1- 0.1 || l2 and no kind of early termination criterion.
Swimmer:
The basis of our Swimmer was the Swimmer-v2 from OpenAl Gym. Its observation vector consists of all angular joint positions and velocities and our only modification was that we extended the sim. 1im.data.qpos(O]] We utilized the 0.0001 ||a; || and no kind of early termination
observation vector by the z position in space (pseudo code:
default reward function r8tt, 81- at) criterion.
zt -
At
Ant:
The basis of our Ant the Ant-v2 from OpenAl Gym. Its observation vector consists of all angular joint positions and velocities as well as some contact related states. Our modification was to remove the contact states from the observation vector and to add the position in space (pseudo code:
(om.data.qpos((0] to it. We utilized the reward function r(st, 8t-1, a.) ---2t-1 1 0.5 0.5||a, l| The only difference of to the default reward function that it has no contact penalty. We additionally utilized the default early termination criterion of Ant-v2.
Ar
A.2 HYPERPARAMETERS
not specified differently, we used the hyperparameters as listed in Table (MBPGE), Table 2 (PPO). Table 2 (TRPO) and Table 3 (MB-MPO) For plain PPO and TRPO experiments have not defined a fixed number of trajectories to collect for a policy update. Instead, we collected trajectories until the batch contained certain number of time steps T Twin- The Anchored Ensemble requires an assumption about the prior distribution. We have set the mean pprior of the Gaussian prior distribution to zero and its variance Σ Eprior according the Xavier initialization (Glorot & Bengio (2010)) (7)z fan-in, nott fan-out)
Epiior
Minis + Mout.i.j
13

---

Under review as a conference paper at ICLR 2020
within this work. The index denotes the NN layer and j the unit inside that layer.
PPO
Anchored Ensemble
MBPGE
AGAE 0.95
a²
10 5
N, 15
7 = 0.99
K
Tmmin =
M 64
e = 0.2
walker: a = 0.8
Adam stepsize: 0.0003
Table I: MBPGE hyper parameter settings.
PPO
Trpo
AGAE 0.95 7 0.99
AGAE = 0.95 7 y= 0.99
M = 64
max KL: 0.01
c c= 0.2
Tmin 2048
T Tmin 2048
Adam stepsize: 0.0003
Table 2: PPO and TRPO hyper parameter settings.
TRPO
MB-MPO
ÀGAB 0.95
inner adaption step size 0.001
7 = 0.99
max KL: 0.01
number of meta-optimization steps: 5
number of collected trajectories per algorithm iteration: 20 (real en- vironment)
40 trajectories were collected from the learned dynamics model for an inner adaption step and for the meta-policy optimization.
Table 3: MB-MPO hyper parameter settings.
Dynamics Model:
The NNs used for representing the dynamics model MBPGE and in MB-MPO consisted of hidden ReLU layers with 1000, 900, 800 and 700 units (2244000 parameters). (Fu et al. 2015 p. 5) described the positive effect of the second order Markov assumption (i.e. the next state depends on the current- and previous state and action) with regard the ability of NN learning strong nonlincarities, e.g. contacts. Most challenging robotic manipulation or locomotion problems which are typically solved by model free RI. include contacts, so the experiments we used NNs of the form f(s,, (1-11, A problem with this NN/BNN configuration is the very first prediction of a roll out which 8,-1 and a,-1 are not defined. We handled this problem by using second NN/BNN fo which trained only the very first roll out transitions (51, so, ao) (i.e. dynamic model only for predicting 81 = fo(so, do)). fo consisted hidden ReLU layers with units each (7268 parameters).
Furthermore, trained our dynamics models to predict As, 8t+1 instead of directly pre- dicting 8411- Additionally, we used the following regularization techniques:
Early stopping wit GI. criterion as specified (Prechelt 2012, p. 58)
Normalization of the training dataset that has a zero mean and unit variance
Shuffling of the training data
Weight decay with À 0.0001 (only for MB-MPO)
14

---

Under review as a conference paper at ICLR 2020
Policy:
For each tested algorithm we utilized a stochastic policy of the form aj ([((s)), σ), in which u(s,) was the output of NN (2 tanh layers with 128 units each) and standard deviation of each action. The network's ' parameters were initialized randomly according to the Xavier initialization (Glorot & Bengio (20T0)) and the initial standard deviation vector set to 0, = 1 vi E {l,..., Doj with Da being the dimension of the action vector.
vector containing the
15


Reviews:
[{"review_id": "SJeE8rx6YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary \n\nThis paper described a model-based RL method which uses learned dynamics models to augment the replay buffer used when training a PPO agent. \nSpecifically, the agent learns an ensemble of dynamics models, and then performs a PPO updates using a mixture of trajectories sampled from the true environment and trajectories sampled from teh learned dynamics models. \nThe fictitious trajectories are sampled from different dynamics models in the ensemble, which helps prevent the policy from exploiting the errors of a single model.\nThe proportion of real/fake trajectories are adapted using the ratio of the real vs. predicted rewards: if the average returns are similar, then the proportions will be approximately equal, but if they are very different then more trajectories from the real environment will be used. \n\nOverall, I think there are some interesting elements to this paper but it is currently not ready for publication. I recommend weak reject for three reasons: limited novelty, the experiments as they stand are not very convincing and the writing needs work. \n\n\n#### Novelty\nThe two stated contributions are: \n1. the use of principled uncertainty estimates in model-based RL (an anchored ensemble)\n2. the routine which adjusts the proportion of real vs. simulated trajectories for learning the policy.      \n\nIn my opinion, 1. is not much of a contribution since a number of works have already made use of different types of model uncertainty in the context of model-based RL. Here they use a slightly different version (the anchored ensemble, which is similar to the randomized priors used in DQNs) but this is more of a choice of implementation than a contribution in itself (a number of methods for estimating uncertainty exist, such as regular ensembles, dropout masks, BNNs, etc). The manner in which they use the uncertainty is to perform simulations with several different models to avoid overfitting to one, and this was already proposed in the ME-TRPO paper. \n\nThe idea of 2. is interesting and in general, adapting between model-based and model-free RL regimes during execution seems to have potential. However the current method seems currently under-explored and heuristic. The fact that it is sensitive to the \\alpha hyperparameter (Figure 2) is a bit troubling. Is there a principled way which does not require tuning this hyperparameter? Or reformulating things so that it is less sensitive?\n\n\n#### Experiments\n\nAlthough the experiments show some improvement, they are not that convincing. The improvements do not seem statistically significant for Swimmer and Walker, and the asymptotic performance is worse than PPO for Hopper. There is only half-cheetah where the algorithm shows a clear improvment. \n\nSince one of the claimed contributions is the use of the anchored ensemble, there should be an ablation experiment showing this gives an improvement over a standard ensemble, but this is not included.  \n\n\n\n#### Writing:\n\nOne issue with the paper is that it spends *much* too long discussing related work and preliminaries. \nThe first 5 pages are devoted to this and the proposed method is only introduced on page 6!\n\nSome detailed comments:\n- Paragraph 1 in the intro should be drastically cut. The sentence \"Recently...challenging tasks\" could be followed by references and then a sentence discussing the sample inefficiency and then moving on to the next paragraph. Most readers will be familiar with DQN, actor critic etc. \n- Same for paragraphs 2 and 3. A short discussion of the general idea behind model-based RL and its improved sample efficiency, the issue of the policy exploiting model errors, and some references are sufficient. \n- Similarly, section 2.1 is mostly unnecessary. It isn't necessary to detail the updates for standard algorithms such as PPO/TRPO unless they are useful for the proposed algorithm. \n- Section 2.2 is also too long. They main idea is very simple and is summarized in Equation 13. \n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1l3VHlRKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary ###\n\nThis paper focuses on model based reinforcement learning (RL). Specifically, the authors consider the setting of combining model based and model free RL algorithms by using the learned dynamics model to generate new data for training the model free algorithm. In order to capture the uncertainty of the environment and the model, the author applied Baysian neural network to learn the dynamics of the environment. The authors approximated the true Bayesian inference process by keeping an anchored ensemble of neural networks, where the prior and posterior of network weights are approximately Gaussian. The ensemble of dynamics model is then used to generate data to train a PPO[1] based agent. In order to prevent the agent from exploiting the learned dynamics model, the authors propose a heuristic way of balancing the amount of real data and model generated data by comparing the rewards.\n\nThe authors evaluate the proposed algorithm on simulated robotic locomotion environments in MuJoCo, and the results show that the proposed method has better same efficiency compared to baseline methods in some environments.\n\n\n### Review ###\n\nOverall I think this paper presents an interesting idea in combing model based and model free RL algorithms. The idea is very well presented and authors include empirical evidence to support the proposed method. However I do find a number of shortcomings that need to be addressed.\n\n\nPro:\n\n1. The idea for this paper is really well presented. The structure of the paper is well organized and the experiment results are easy to interpret.\n\n2.  The authors provide a detailed description of the configurations and the hyperparameters for each experiments. Such description would be very helpful if the results in this paper are to be reproduced.\n\n\nCon:\n\n1. I’m not convinced about the magnitude of novelty in this paper. The proposed method seems very similar to ME-TRPO[2] and it seems to me that the novelty comes from the application of Bayesian ensemble techniques and the generated data ratio tuning heuristics. While these variations might be important for the final performance of the proposed method, the paper does not include any ablation study to further justify the importance of these variations.\n\n2. I’m not convinced about some of the performance of some of the baseline methods presented in this paper. In this paper, MB-MPO[3] does not improve at all during training on Half-Cheetah environment. However, in the original MB-MPO paper, the algorithm does improve and the performance seems to be comparable to that of the proposed method in this paper.\n\n3. The experiment results are not very strong for the proposed method. In 3 of 4 environments, the proposed algorithm does not show much advantage over the baseline algorithms. The only environment in which the proposed method shows significant improvement is Half-Cheetah, and I believe that the baseline algorithms might not be properly tuned in this environment.\n\n4. The paper lacks certain baseline comparisons. There are many other model based RL algorithms developed recently, and it would be important to compare to these methods. Some examples would be ME-TRPO[2], SLBO[4] and MBPO[5].\n\n\nThe idea in the paper is well presented and carefully investigated. However, I am still not convinced about the novelty of the proposed idea and the magnitude of performance improvement. Therefore, I would not recommend acceptance before these problems are addressed. \n\n\n\nReferences\n\n[1] Schulman, John, et al. \"Proximal policy optimization algorithms.\" arXiv preprint arXiv:1707.06347 (2017).\n\n[2] Kurutach, Thanard, et al. \"Model-ensemble trust-region policy optimization.\" arXiv preprint arXiv:1802.10592 (2018).\n\n[3] Clavera, Ignasi, et al. \"Model-based reinforcement learning via meta-policy optimization.\" arXiv preprint arXiv:1809.05214 (2018).\n\n[4] Luo, Yuping, et al. \"Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees.\" arXiv preprint arXiv:1807.03858 (2018).\n\n[5] Janner, Michael, et al. \"When to Trust Your Model: Model-Based Policy Optimization.\" arXiv preprint arXiv:1906.08253 (2019).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklIkSDXqH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new model-free + model-based algorithm, MBPGE, that trains a policy using a policy gradient algorithm on top of the learned models. Contrary to previous approaches, they use a true Bayesian distribution by means of the randomized anchorized MAP. Furthermore, they combine rollouts from the real environment and from the learned dynamics directly for the policy training, instead of relying just on the ones of learned dynamics, which induces a larger distributional shift.\n\nThe paper readability could be improved. First of all, the introduction spans over 2 pages. The readability could be significantly improved by splitting the introduction into introduction (containing the motivation) and related work. Secondly, the preliminary section spans more than their approach, which hints that the original contribution is too incremental. All of the actual method is just combining section 2.1 and 2.2. Lastly, section 3.1 is hard to parse. The authors extend the paper to 9 pages when there was no need for it.\n\nIn terms of contribution, it seems that the main contribution is to use the method presented [1] and replace the ensemble with the ensemble of [2] and use PPO [3] instead of TRPO [4]. There is no further insight in the paper, but the fact that the ensemble of [2] works better than the classical ensemble (fact shown in [2]) and that PPO works better than TRPO (fact shown in [3]). The other contribution is to combine samples from the real environment and the from the learned models to train the policy. This however has actually been done in the context of model-based reinforcement learning, see [5]. It would be interesting that the authors compared against different heuristics to choose the ratio between imagined and real rollouts (i.e., constant, annealed, fraction of improvement [1], etc…). Overall, I don’t think that the contributions of this paper are enough for publication. \n\nRegarding the experiment section I strongly believe that the comparison is flawed. The results they report on MB-MPO do not match with the ones in [6, 7]. For instance, half-cheetah does not learn at all while it is an easier environment than Walker and Hopper (and there’s learning in those). Furthermore, given this they should also compare against ME-TRPO and ME-TRPO switching the TRPO with PPO, since this would be a fairer comparison. This section lacks also of a proper ablation analysis to identify how much each element of the proposed algorithm affects the performance the choice of Bayesian ensemble and heuristic to determine the amount of real samples and imagined ones.\n\nAt this stage, there is not enough contribution in terms of novelty nor delta in performance.\n\n\n\n[1] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel. Model-Ensemble Trust-Region Policy Optimization.\n[2] Tim Pearce, Felix Leibfried, Alexandra Brintrup, Mohamed Zaki, Andy Neely. Uncertainty in Neural Networks: Approximately Bayesian Ensembling.\n[3] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. Proximal Policy Optimization Algorithms.\n[4] John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel. Trust Region Policy Optimization\n[5] Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine. When to Trust Your Model: Model-Based Policy Optimization\n[6] Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, Pieter Abbeel. Model-Based Reinforcement Learning via Meta-Policy Optimization\n[7] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba. Benchmarking Model-Based Reinforcement Learning.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a hybrid model-free/model-based policy gradient method that attempts to reduce sample complexity without degrading asymptotic performance. They evaluate their approach on a collection of benchmark tests.

The reviewers raised concerns about limited novelty of the proposed approach and flaws in the evaluation. The authors need to compare to more baselines and ensure that the baseline algorithms are performing as previously reported. Even then, the reported improvements were small.

Given the issues raised by the reviewers, this paper is not ready for publication at ICLR.
Decision: Reject