Benchmarking Model-Based Reinforcement Learning
Authors: Tingwu Wang
Xuchan Bao
Ignasi Clavera
Jerrick Hoang
Yeming Wen
Eric Langlois
Shunshi Zhang
Guodong Zhang
Pieter Abbeel
Jimmy Ba
Venue:  (2020)
Keywords: Reinforcement learning
model based Reinforcement learning
Benchmarking

Abstract: Model-based reinforcement learning (MBRL) is widely seen as having the potential
to be significantly more sample efficient than model-free RL. However, research in
model-based RL has not been very standardized. It is fairly common for authors to
experiment with self-designed environments, and there are several separate lines of
research, which are sometimes closed-sourced or not reproducible. Accordingly, it
is an open question how these various existing algorithms perform relative to each
other. To facilitate research in MBRL, in this paper we gather a wide collection
of MBRL algorithms and propose over 18 benchmarking environments specially
designed for MBRL. We benchmark these algorithms with unified problem settings,
including noisy environments. Beyond cataloguing performance, we explore
and unify the underlying algorithmic differences across MBRL algorithms. We
characterize three key research challenges for future MBRL research: the dynamics
bottleneck, the planning horizon dilemma, and the early-termination dilemma.
Finally, to facilitate future research on MBRL, we open-source our benchmark.

Under review as a conference paper ICLR 2020
BENCHMARKING MODEL-BASED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Model-based reinforcement learning (MBRL) is widely seen as having the potential be significantly more sample efficient than model-free RI.. However, research in moodel-based RL has not been very standardized. It is fairly common for authors to experiment with self-designed environments, and there are several separate lines of research, which are sometimes flosed-sourced or not reproducible. Accordingly, it is an open question how these various existing algorithms perform relative to each other. To facilitate research in MBRL, in this paper gather a wide collection MBRL algorithms and propose over 18 benchmarking environments specially designed for MBRL. We benchmark these algorithms with unified problem settings, including noisy environments. Beyond cataloguing performance, we explore and unify the underlying algorithmic differences across MBRL algorithms. We characterize three key research challenges for future MBRL research: the dynamics bottleneck, the planning horizon dilemma, and the early-termination dilemma. Finally, to facilitate future research on MBRL, we open-source our benchmark
INTRODUCTION
Reinforcement learning (RL) algorithms are most commonly classified in two categories: model-free RL (MFRL), which directly learns a value function or policy by interacting with the environment, and model-based RL (MBRL). which uses interactions with the environment to learn model of it. While model-free algorithms have achieved success in areas including robotics (Lillicrap et al.
2015 Schulman et al. 2017 Heess al., 2017] Andrychowicz et al.) 2018), video-games (Mnih et al. 2016)) and character animation (Peng et al. 2018). their high sample complexity limits largely their application simulated domains. By learning model of the environment, model-based methods learn with significantly lower sample complexity. However, learning an accurate model of the environment has proven be a challenging problem certain domains. Modelling errors cripple the effectiveness of these algorithms. resulting in policies that exploit the deficiencies of the models, which is known as model-bias (Deisenroth & Rasmussen 201 T). Recent approaches have been able to alleviate the model-bias problem by characterizing the uncertainty of the learned models by the means of probabilistic models and ensembles. This has enabled model-based methods to match model -frel-frre asymptotic performance in challenging domains while using much fewer samples (Kurutach et al.) 2018
Chua et al. 2018 Clavera et al. 2018).
These recent advances have led to a great excitement in the field of model-based reinforcement learning. Despite the impressive results achieved, how these methods compare against each other and against standard baselines remains unclear. Reproducibility and lack of open-source code persistent problems in RI. (Henderson et al., 2018 Islam et al. 2017), which makes it difficult to compare novel algorithms against prior lines of research. In MBRL, this problem is exacerbated by the modifications made to the environments: pre-processing of the observations, modification of the reward functions, using different episode horizons. Such lack of standardized implementations and environments in MBRL makes it difficult to quantify scientific progress.
Systematic evaluation and comparison will not only further our understanding of the strengths and weaknesses of existing algorithms, but also reveal their limitations and suggest directions for future research. Benchmarks play a crucial role other fields research. For instance, MFRL has benefited
'The link to the code base invisible during reviewing process.

---

Under review as a conference paper ICLR 2020
greatly from the introduction of benchmarking code bases and environments such as rllab (Duan et al. 2016), OpenAl Gym (Brockman et al. 2016), and DM Control Suite (Tassa et al. 2018); where the latter two have been the de facto henchmarking platforms. Besides RI., benchmarking platforms
have also accelerated areas such as computer vision (Deng et al., 2009 Lin et al.) 2014), machine translation (Koehn et al. 2007) and speech recognition (Panayotov et al. 2015).
In this paper, we benchmark 11 MBRL algorithms and MFRL algorithms across 18 environments based on the standard OpenAI Gym (Brockman et al.) 2016). The environments, designed to hold the common assumptions in model-based methods, range from simple 2D tasks, such as Cart-Pole, to complex domains that usually not evaluated on, such Humanoid. The benchmark is further extended by characterizing the robustness of the different methods when stochasticity in the observations and actions is introduced. Based on the empirical evaluation, propose three main causes that stagnate the performance of model-based methods: 1) Dynamics bottleneck: algorithms with learned dynamics are stuck performance local minima significantly worse than using ground- truth dynamics, i.e. the performance does not increase when more data is collected. 2) Planning horizon dilemma: while increasing the planning horizon provides more accurate reward estimation, can result in performance drops due to the curse of dimensionality and modelling errors. 3) Early termination dilemma: early termination is commonly used in MFRL for more directed exploration, to achieve faster learning. However, similar performance gain are not yet observed in MBRL algorithms, which limits their effectiveness in complex environments.
2 PRELIMINARIES
We formulate all of our tasks as discrete-time finite-horizon Markov decision process (MDP), which is defined by the tuple AS,,, Po. H). Here, S denotes the state space, A denotes the action space, p(s la, s) : S A S [0, 1] is transition dynamics density function, r(s, a, s') S x A x S - IR defines the reward function, is the initial state distribution, is the discount factor, and И the horizon of the problem. Contrary to standard model-free RL. we assume access to analytic differentiable reward function. The aim of RL. is to learn an optimal policy that maximizes the expected total reward ((T)
Ea. 4rrpp
EE ?'r(st,
Dynamics Learning: MBRL algorithms are characterized by learning a model of the environment. After repeated interactions with the environment, the experienced transitions are stored in data- set D {(t,, at 8t-1) which is then used to learn dynamics function fo- In the case where ground-truth dynamics are deterministic, the learned dynamics function I' predicts the next state. In stochastic settings, it is common to represent the dynamics with a Gaussian distribution, i.e., p(sc+ilat, st) NN((se, at), (ss,, a:)) and the learned dynamics model corresponds to fo (Np(se, ar), Es)s),))))
3 ALGORITHMS
In this section, we introduce the benchmarked MBRL algorithms, which are divided into: 1) Dyna- style Algorithms, 2) Policy Search with Backpropagation through Time, and 3) Shooting Algorithms.
3.1 DYNA-STYLE ALGORITTMS
In the Dyna algorithm (uutonn 1990 1991abb b) training iterates between two steps. First, using the current policy, data is gathered from interaction with the environment and then used learn the dynamics model. Second, the policy is improved with imagined data generated by the learned model. Dyna algorithms learn policies using model-free free algorithms with rich imaginary experience without interaction with the real environment. Dyna can also be applied to tasks with image input as in world models (Ha Schmidhuber, (b).
Model-Enseminn Trust-Region Policy Optimization (ME-TRPO) (Kurutach et al., 2018): Instead of using a single model, ME-TRPO uses an ensemble of neural networks to model the dynamies, which effectively combats model-bias. The ensemble fe = [far ..., fox) } is trained using standard squared L2 loss. In the policy improvement step, the policy is updated using Trust- Regt-n Policy

---

Under review as a conference paper ICLR 2020
Optimization (TRPO) (Schulman et al. 2015), on experience generated by the learned dynamics models.
Stochastic Lower Bound Optimization (SLBO) (Luo et al.
2019 SLBO is variant of ME-TRPO with theoretical guarantees of monotonic improvement. In practice, instead of using single-step squared L2 loss, SLBO uses multi-step L2-norm loss to train the dynamics.
(MB-MPO) (Clavera et al. 2018); MB-MPO forgoes the reliance on accurate models by meta-learning policy that is able to adapt to different dynamics. Similar to ME-TRPO, MB-MPO learns an ensemble of neural networks. However, each model the ensemble is considered as a different task meta-train (Finn et al. 2017) on. MB-MPO meta-trains a policy that quickly adapts to any of the different dynamics of the ensemble, which is more robust against model-bias.
Model-Based
3.2 POLICY SEARCH WITH BACKPROPAGATION THROUGH TIME
Contrary to Dyna-style algorithms, where the learned dynamics models are used to provide imagined data, policy search with backpropagation through time exploits the model derivatives. Consequently, these algorithms are able to compute the analytic gradient of the RL objective with respect to the policy, and improve the policy accordingly.
Probabilistic Inference for Learning Control (PILCO) DDisennoth & Rasmussen 2011 Deisen roth et al., 2015] Kamthe & Deisenroth] (017): In PILCO, Gaussian processes (GPs) are used to model the dynamics of the environment. The dynamics model fn(s, a.) is a probabilistic and non- parametric function of the collected data D. The policy To is trained to maximize the RI objective by computing the analytic derivatives of the objective with respect to the policy parameters O. The training process iterates between collecting data using the current policy and improving the policy. Inference in GPs does not scale in high dimensional environments, limiting its application to simpler domains.
Iterative Linear Seaddratic-Gausian (iLQG) (Tassa et al., 2012): In iLQG, the ground-truth dynamics are assumed to be known by the agent. The algorithm uses a quadratic approximation on the RI. reward function and linear approximation on the dynamics, converting the problem solvable by linear-quaan quaaraqu regulator (I (LQQR) (Bemporad et al. 2002). By using dynamic programming, the optimal controller for the approximated problem is linear time-varying controller. iL.QG is a model predictive control (MPC) algorithm, where re-planning is performed at each time-step.
Guided Policy Search (GPS) (Levine & Abbeel 2014 Levine
2015 Zhang et al. 2016 Finn
et al. 20165 Montgomery & Levine 2016; Chebotar el 201 Guided policy search essentially distills the iLQ controllers mig into a neural network policy Te by behavioural cloning, which minimizes E[DKL ( (x) [e)] The dynamics are modelled to be Gaussian- linear time-varying. To prevent over-confident policy improvement that deviates from the last real-world trajectory, the reward function is augmented as f(st, a.) r(st, az) D/ ( (-1s:) (p(((ss)))) where p(-(se) is the passive dynamics distribution from last trajectories. In this paper, we use the MD-GPS variant (Montgomery
& Levine 2016).
Stochastic Value Gradients (SVG) (Heess et al. 2015): SVG tackles the problem of compounding model errors by using observations from the real environment, instead of the imagined one. To accommodate mismatch between model predictions and real transitions, the dynamics models in SVG probabilistic. The policy is improved by computing the analytic gradient of the real trajectories with respect to the policy. Re-parametrization trick used to permit back pack-propagation through the stochastic sampling.
3.3 SHOOTING ALGORITHMS
This class of algorithms provide way to approximately solve the receding horizon problem posed in model predictive control (MPC) when dealing with non-linear dynamics and non-convex reward functions. Their popularity has increased with the use of neural networks for modelling dynamics.
Random Shooting (RS) (Richards) 2005 Rao 2009): RS optimizes the action sequence auttr to maximize the expected planning reward under the learned dynamics model, i.e.,
maxa E 1Ett r(5/, a()]. In particular, the agent generates K candidate random sequences
3

---

Under review as a conference paper ICLR 2020
of actions from a uniform distribution, and evaluates each candidate using the learned dynamics. The optimal action sequence is approximated as the one with the highest return. RS agent only applies the first action from the optimal sequence and re-plans at every time-step.
Mode-Free Model-Based (MB-MF) NNgabannii et al., 2017): Generally, random shooting has worse asymptotic performance when compared with model-free algorithms. In MB-MF, the authors first train a RS controller TRS. and then distill the controller into a neural network policy m using DAgger (Ross et al. 2011), which minimizes DKL (TT(())), FRS). After the policy distillation step, the policy is fine-tune tuned using standard model-free algorithms. particular the authors use TRPO (Schulman et al. 2015).
Probabilistic Ensembles with Trajectory Sampling (PETS-RS and PETS-CEM) (Chua et al. 2018): In the PETS algorithm, the dynamics are modelled by ensemble of probabilistic neural networks models, which captures both epistemic uncertainty from limited data and network capacity, and aleatoric uncertainty from the stochasticity of the ground-trun dynamics. PETS-RS is the same as RS except for different modeling of the dynamics. In PETS- the online optimization problem solved using cross-enie cooss-y method (CEM) (De Boer al. 2005 Botev al. 2013 to obtain better solution. PETS-CEE CEM can also plan in latent space of image observations (Hafner et al. 20188.
3.4 MODEL-FREE BASELINES
In our benchmark, we include MFRL baselines to quantify the sample complexity and asymp- totic performance gap between MFRL and MBRL. Specifically. compare against representative MFRL algorithms including Trust-Region Policy Optimization (TRPO) (Schulman et al.] (201)). Proximal-Policy Optimization (PPO) (Schulman al., 2017, Heess et al. 2017), Twin Delayed Deep Deterministic Policy Gradient (TD3) (Fujimoto et al. 2018), and Soft Actor-Critic (SAC) (Haarnoja state-of-the-arr on-policy MFRL algorithms, and the latter two considered the state-of-the-art off-policy MFRL algorithms.
al. 2018). The former
4 EXPERIMENTS
In this section, we present the results of our benchmarking and examine the causes that stagnate the performance of MBRL methods. Specifically, we designed the benchmark to answer the following questions: 1) How do existing MBRL approaches compare against each other and against MFRL methods across environments with different complexity (Section |.3 2) Are MBRL algorithms robust against observation and action noise (Section and 3) What are the main bottlenecks in the MBRL methods?
Aiming to answer the last question, we present three phenomena inherent of MBRL methods, which we refer to as dynamics bottleneck (Section 4.5). planning horizon dilemma (Section (.6c).on4.)).... and early termination dilemma (Section 4.7
4.1 BENCHMARKING ENVIRONMENTS
Our benchmark consists of 18 environments with continuous state and action space based on OpenAl Gym (Brockman al.) 2016). We include full spectrum of environments with different difficulty and episode length, from CartPole to Humanoid. More specifically, we have the following modifications:
To accommodate traditional MBRL algorithms such as iLQG and GPS, we modify the reward function so that the gradient with respect to observation always exists or can be approximated.
We note that early termination has not been applied in MBRL, and we specifically have both the raw environments and the variants with early termination, indicated by the suffix ET.
The original Swimmer-v0 in OpenAl Gym was unsolvable for all algorithms. Therefore, modified the position of the velocity sensor so that it's easier to solve. We name this easier version as Swimmer while still keep the original one reference, named as Swimmer-v0.
For detailed description of the environments and the reward functions used, we refer readers to Ap- pendix A We also provide open-sourced version of the tasks based on Roboschool or Pybullet (AMD) 2014 Ellenberger 2018 Klimov & Schulman 2017), which refer to Appendix H

---

Under review as a conference paper ICLR 2020
-300
103
-30
10030
L00000
11003)
20030
J000
20300
200000
5000
Creologo
(a) Acrobot
(b) Swimmer
(c) HalfCheetah
PETS-CEM
PETS-RS TD8
ME- TRPO PILCO
MBMF
GPS
SAC
SLBO
SVG
MB MPO
Figure I: A subset of all 18 performance curve figures of the bench-marked algorithms. All the algorithms are run for 200k time-steps and with 4 random seeds. The remaining figures are in appendix
4.2 EXPERIMENT SETUP
Performance Metric and Hyper-pann paremeerr Search: Each algorithm is run with random seeds. In the learning curves, the performance is averaged with sliding window of 5 algorithm iterations. The error bars were plotted by the default Seaborn (Waskom 2010) smoothing scheme from the mean and standard deviation of the results. Similarly, in the tables, show the performance averaged across different random seeds with a window size of 5000 time-steps. We perform a grid search for each algorithm separately, which is summarized in appendix B For each algorithm, We show the results using the hyper-pppaaae parameters producing the best average performance.
Training Time: In MFRL, million time-step training is common, but for many environments, MBRL algorithms converge much earlier than 200k time-steps and it takes an impractically long time to train for million time-ster time-steps for some of the MBRL algorithms. We therefore show both the performance of 200k time-step training for all algorithms and show the performance of 1M time-step training for algorithms where computation is not major bottleneck.
4.3 BENCHMARKING PERFORMANCE
In T 1 summarize the performance of each algorithm trained with 200,000 time-steps. We also include some representative performance curves in Figure [ The learning curves for all the environments can be seen appendix C The enginecring statistics shown in Table ( include the computational resources, the estimated wall-clock time, and whether the algorithm is fast enough to run real-time test time, namely, if the action selection can be done faster than the default time-step of the environment. In Table 5 we summarize the performance ranking.
Shooting Algorithms: RS very effective on simple tasks such as InvertedPendullm, CartPole and Acrobot, but as task difficulty increases RS gradually gets surpassed by PETS-RS and PETS-CEM, which indicates that modelling uncertainty aware dynamics crucial for the performance. At same time, PETS-CEM, is better than PETS-RS in most of the environments, showing the importance of an effective planning module. However, PETS-CES Walker2D and SlimHumanoid, indicating that we need for more complex environments. MB-MF does not have obvious gains compared to other shooting algorithms, but like other model-free controllers, MB-MF can jump out of performance local-minima in MountainCar. Shooting algorithms are effective and robust across different environments.
search not as effective as PETS-RS in Ant, expressive and general planning module
Dyna-Style Algorithms: MB-MPO surpasses the performance of ME-TRPO in most of the environ- ments and achieves the best performance domains like HalfCheetah. Both algorithms seems to perform the best when the horizon is short. SLBO can solve MountainCar and Reacher very effi- ciently, but more interestingly in complex environment it achieves better performance than ME-TRPO and MB-MPO, except for SlimHumanoid. This category of algorithms horizon complex domains due to the compounding error effect.
efficient solve long
SVG: For the majority of the tasks, SVG does not have the best sample efficiency. But for Humanoid environments, SVG is very effective compared with other MBRL algorithms. Complex environments exacerbate compounding errors; SVG which uses real observations and a value function to look into future returns, is able to surpass other MBRL algorithms in these high- high-dimensional domains.
5

---

Under review as a conference paper ICLR 2020
Table Final performance for environments of the bench-marked algorithms. the algorithms are run for 200k time-steps. Blue refers to the best methods using ground truth dynamics, red to the best MBRL algorithms, and green to the best MFRL algorithms. The results show the mean and standard deviation averaged over 4 random seeds and window size of 5000 times-steps. "GT" indicates the model is using the ground-truth dynamics.
Pendulum
InvertedPendalum
Acrobot
CartPole
Mountain Car
Reacher
Random I1QQ
-202.6
249.3
-205.
13.6
-374.5 + 17.1
38.4 32.5
-105.1
-45.7
160.8 29.8
-0.0 0.0
-195.5 + 28.7
199.3 0.6
-35.9
8.3
-6.0
2.6
GE-CEM
170.5
35.2
-0.2 0.1
13.9 1. 40.5
199.9 0.1
-58.0
2.9
.3. 1: 1.2
-68.5
RS
164.4 9.1
.0.0 0.0+
.4.9
200.0 0.0-
-71.3 0.5
-27.1 0.6
PETS-CEM
167.4 53.0
-20.5 28.9
12.5 29.0+
199.5 2 3.0
-57.9
-12.3 2: 5.2
PETS-RS
167.9
-12.1 25.1
-71.5 44.6
-78.5 2.1
-00.1
ME-TRPO
177.3
-126.2 86.6
.68.1 6.7
160.1 69.1
-42.5 26.6
-334
GPS
162.7 7.6
-74.6 97.8
-19333 11.7
14.4 18.6
-00.6 32.1
-19.8 = 0.9
166.1 16.1+-2300
196.1 13.3
4.6
-13.2
SVG
141.4 62.4
-183.1
-79.7 6.6
82.1 31.9
-21.6 32.6
-11.0 1.0
MB-MPO
171.2
-0.0
-87.8
-30.6
-56
SLBO
173.5 2.5
-240.4 7.2
-75.6
78.0 166.6
44.1 6.8
4.11 0.1x
PPO
163.4 8.0
.40.8 + 21.0
-9.3
86.5 7.8
21.7 13.1
-19
TRPO
166.7
-27.6 15.8
-147.5 12.3 -64.3 6.9
41.3 15.7
-37.2 164
-10.1 10.1
TD3
161.4 + 14.4
-224.5 - 0.4
196.0 3.1
-6.00 1.2
-14.0 0,9
9.5
HalfCheetah
Swimmer-vo
Swimmer
Ant
Walker2D
Random
-288.3 65.8
1.2 11.2
-9.5 11.6
473.8 40.8
124.6 + 145.0
-2456.9
345.3
459.4
-118.22
GT-CEM
14777.2
13964.2
111.0
335.9 L.1
12115.3 209.7
226.0
178.6
7719.7
486.7
GT-RS
815.7 :: 38.5
35.8 3.0
42.2 5.3
2709.1 631.1
2519.0
469.8
137.6 -164
421.0 55.2
31.1 2.0
92.8 8.1
535.5 37.0
239.9 81.7
2060.3 228.0
MB-MF
126.9 72.7
51.8 1: 30.9
284.9
25.1
134.2 2: 50.4
85.7 27.7
2218.1 437.7
145.8
PETS RS
966.9 471.6
42.1 20.2
170.1 8.1
1852.1 141.0
130,0 148.1
312.5 1: 493.4-
ME-TRPO
282.2
21.1
GPS
52.3 41.7
14.5 5.6
-35.3
445.5 212.9
275.4 309.1
1730.. 441.7
PILCO
11..91266.00
-33.8
16.1
-18.7 10.3
770.7 153.0
N.
2693.8 484.4
SVG
336.6
77.2 99.0
75.2 85.3
3779
185.0 141.6
-4300 = 230.1
MB-MPO
3639.0 1185.8
85.0
98.9
268.5
125.4
705.8 147.2
30.3 22.3
-1545.9 216.5
SLBO
1097.7
125.2
123.3
PPO
17.2 84.4
38.0
306.8
321.0 51.2
80.1 17.3
-199366 234.1
TRPO
-12.0 = 85.5
37.9
215.7
10.4
323.3 24.9
116.8 + 47.3
228663 373.3
TD3
3614.3 82.1
40.4 8.3
331.1 0.9
956.1 66.9
259.7
-73.8 769.0
SAC
4000.7 + 202.1
41.2 4.6
309.8 4.2
506.7 165.2
2012.7 571.3*
-415. + 588.1
Walker2D-ET
Hopper
Hopper-ET
SlinHumanoid
SlinHumanoid-E7
Humanoid-E7
Random
-2.8 4.3
-2572.7 + 631.3
12.7.2..8
-1172.9 + 757.0
41.8 47.3
50.5 57.1
ILQG
229.0 74.7
1157.6 224.7
83.4 21.7
13225.2 = 1344,9
$20.0 240.9
255.0 94.6
GT-CEM
1654.9
676.0 361.5
668.0
GT-RS
207.9 :2:
-2467.2 = 55.4
209.5 46.8
8074.4 = 441.1
103.8
312.9 167.8
10.5
388.5
MB-MF
350.0 107.6
1047.4 1098.7
926.9
154.1
-1320.2 735.3
809.7
S7.5
776.8 62.9
-2.5 1. 6.8
1125,0 1 679.6
129.3 36.0
1472.4 738.3
355.1
157.1
110.8 91.0
PETS-RS
-0.8
1469.8 224.1
205.8 36.5
2055.1 = 771.5+
320.7
182.2
106.9 1 102.6
ME-TRPO
99.5 4.6
1272.5 500.9
4.9 4.0
-154.9 1. 534.3
76.1 1 8,8
72.9 1 8.9
-2303.9
PILCO
N.
1729.9
N, A.
N.
N.
N.
SVG
252.4 48.4
.87.9 1: 427.9
435.2 1: 163.8
1096.8 7910
1084,3 77.00
811.8 241.5
MB-MPO
-10.3
333.2 1189.7
8.3 +
674.4 982.2
115.5 319
73.1 23.1
SLBO
108.7 207.8
-741.7 + 734,1
805.7 142,4
-58 332.1
776.1 252.5
1377.0 150.4
306.1 17.2
100.88 +
758.0 + 62.0
-1466.7 278.5
TRPO
229.5 27.1
-2100.1 + 640.6
237.4 33.5
-1140.9 + 241.8
281.3 + 10.9
289.8 = 5.2
3299.7 1951.5
2245.3 232.4A
1057.1 29.5
1319.1
1070.0 168.3
147.7
SAC
2216.4 678.7
7264 + 675.5
1815.5 + 685.1
1328.4 468.2
843.6 313.1
1794.4 458.3+
PILCO: PILCO achieves one of the best sample efficiency low dimentional environments such as Cartpole, Pendulum and InvertePPendulum. But it fails in most other environments with bigger episode length and observation size, being unstable across random seeds and consuminn to train.
GPS: GPS has the best performance in Ant-ET, but cannot match the best algorithms in other environments. In the original GPS, the environment usually 100 time-step long, while most of our environments are 200 or 1000 time-step. Also GPS assumes several separate constant initial states, while our environments sample the initial state from a distribution. The deviation of trajectories between iterations can be the reason of GPS's performance drop.

---

Under review as a conference paper ICLR 2020
RS
MBMF
PETS
PETS-RS
METRRO
PILCO
MB-MPP
$1.B0
TRPO
SAC
Reacher2D
4.03
30.9
2.9
2.38
Choclah
8.83
1.05
15.3
6.76
5.23
N. A.
1.41
57.5
1.56
0.04
0.02
4.3
2.21
Ast
8.2
5.25
6.5
5.01
3.46
5.1
N. A.
1.49
55.2
5.46
0.07
0.05
3.6
3.15
Humanoid-FF
13.9
5.05
7.03
5.1
5,68
N.
N. A.
1.92
41.4
5.5
0.05
0.04
5.37
3.35
Stinhumanodd-ET
9.5
3.3
4.76
3.35
2.58
N.
6,86
0.03
3.13
4.05
Slimhumsuud
41.6
3.95
Real-time sime lesting
V
<
V
V
CPUGPU used
20/0
20/00
4/1
4/1
4/1
570
4/1
2/0
8/0
12/0
570
570
12/0
12/0
Table 2: Wall-clock time in hours for each algorithm trained for 200k time-steps.
MF baselines: SAC and TD3 are two very powerful baselines with very stable performance across different environments. In general model-free and model-based methods are two almost evenly matched rivals when trained for 200,000 time-steps.
MB with Ground-truth Dynamics: Algorithms with tround-truth dynamics can solve the majority of the tasks, except for some of the tasks such as MountainCar. With the increasing complexity of the environments, shooting methods gradually have much better performance than the policy search methods such as iLQG, whose linear quadratic assumption not a good approximation anymore. Early termination cause lot of troubles for model- algorithms, both with and without ground- truth dynamics, which is further studied in section 4.7
4.4 NOISY ENVIRONMENTS
In this section, we study the robustness of each algorithm with respect to the noise added to the observation and actions. Specifically, added Gaussian white noise the observations and actions with standard deviation 0, and a. respectively. In Table 3 we show the results for the HalfCheetah environment, for the full results we refer the reader to appendix D
HalfChezrah
B.QG
GEPETS
R8
PETS
ME-TRPO
SVG
MR-MPO
SLBBO
GG--R
PETS-RS
SAC
TD3
Original Performance
2142.6
147772
421
2795.3
2283.7
336.6
3639.0
1097.7
815.7
966.9
4000.7
3614,3
Change/ / 0.1
-3167.9
-1313877
-2748
-1974.3
-336.5
-1282.6
-2H54
-2718.6
Change d. 0.01
+2.1
-131.5
-2797
Change/ / d, 0.1
-1881.4
-3292.7
+24.8
-367.8
-963.9
-173.1
-26611
+495.5
-210.9
161.7
-470.2
642.2
Change/ 0.03
-1832.5
-1616.6
+213
-368.1
-160.8
-314.7
+79.7
-66.6
-170
50.5
-292.6
327.5
Table 3: The relative changes of performance of each algorithm in noisy HalfCheetah environments.
We use bold text to indicates a decrease of performance >10% of the performance without noise.
As expected, adding noise is in general detrimental the performance of the MBRL algorithms. ME-TRPO and SLBO are more likely to suffer from a catastrophic performance drop when compared shooting methods such as PETS and RS, suggesting that re-planning successfully compensates for the uncertainty. On the other hand, the Dyna-style method MB-MPO presents to be very robust against noise. Due to the limited exploration in baseline, the performance is sometimes increased after adding noise that encourages exploration.
4.5 DYNAMICS BOTTLENECK
We further run MBRL algorithms for 1M time-steps on HalfCheetah, Walker21), Hopper, and Ant environments to capture the asymptotic performance, as are shown in Table 4 and Figure 2 The results show that MBRL algorithms plateau at performance level well below their model-free counterparts and themselves with ground-trutt dynamics. This points out that when learning models, more data does not result in better performance. For instance, PETS's performance plateaus after 400k time-steps at a value much lower than the performance when using the ground truth dynamics. We also study the performance with different network capacity, well as using linear of RBF parameterization, as summarized in Appendix
The following assumptions can potentially explain the dynamics bottleneck. The prediction error accumulates with time, and MBRL inevitably involves prediction on unseen states. While techniques such as probabilistic ensemble were proposed capture uncertainty, can be seen empirically in our paper well as in Chua et al. (2018), that prediction becomes unstable and inaccurate with time. 2) The policy and the learning of dynamics is coupled, which makes the agents more prone to performance local-minima. While exploration and off-policy learning have been studied in Bellemare (Houhhooft al. 2016); Dearden et al. (1999); Wiering Schmidhuber] Hoothoooooo et al. (2016); Schaul
al. 2019 Fujimoto et al. (2018). it has been barely addressed on current model-based approaches.
7

---

Under review as a conference paper ICLR 2020
180
1000
2000
LAME
(a) HalfCheetah
(b) Ant
(c) Walker2D
(d) Hopper
PETS-CEM
TD8
ME- TRPO
SLBO
SAC
MB- MP-MPO
Figure 2: Performance curve for each algorithm trained for million time-steps.
GT-CEM
PETS PETS-CEM
ME-TRPO
MB-MPO
SLBO
TD3
SAC
HallCheetah
2875.9
1045.4
2041.4 932.7
5072.9
Walker2D
7719.7 1: 486.7
1931.7 667.3
-2917.1 1: 610,0
-1793.7 1: 80.6
1371.7 1: 2761.7
3293.6 1: 641.1
2911.0 2: 985.3
Hogper
3232.3 + 192.3
288.4 +: 988.2
948.0 + 854.3
.495.2 + 265.0
2963.1 + 323.4
2745.7 4: 546.7
3020.3 = 134.6
121153 209.7
1675.0 108.6
262.7 4 36.5
810.8 240.6
513.6 182.0
3073.8 4: 773.8
2989.9 1182.8
Table 4: Bench-marking performance for 1 million time-steps.
4.6 PLANNING HORIZON DILEMMA
One of the critical choices in shooting methods is the plan- ning horizon. In Figure 3 we show the performance of iLQG, CEM and RS, using the same number of candidate planning sequences, but with different planning horizon. We notice that increasing the planning horizon does not necessarily increase the performance, and more often instead decreases the perfor- mance. This happens both when using ground-truti dynamics and using learned dynamics. We argue that this is result of insufficient planning in a search space which increases expo- nentially with planning depth, i. e.. the curse of dimensionality, as is also observed in Vemula et al. (2019): Hafner et al. (2018). However, in more complex environments such as the ones with early terminations, short planning horizon can lead to catastrophic performance drop, which we discuss in appendix !] We further experiment with the imaginary environment length Dyna algorithms. We have similar results that increasing horizon does not necessarily help the performance, which is summarized in appendix F
CT-CEM-Ant Unk-CEM-Cheedee
GT ILQG Ant GT iLQQ Cheetoh
GT-RS-An
GT-RS- RS-Cheetah
Figure 3: The relative performance
with different planning horizon.
4.7 EARLY TERMINATION DILEMMA
Early termination, when the episode finalized before the horizon has been reached, is a standard technique used in MFRL algorithms to prevent the agent from visiting unpromising states or damaging states for real robots (Peng et al. 2018 2016) Merel et al., 2017]] Heess et al.. Brockman et al., 2016). When early termination is applied to the real environments, MBRL can correspondingly also apply early termination in the planned trajectories, or generate early terminated imaginary data. However, we find this technique hard to integrate into the existing MB algorithms. The results, shown in Table indicates that early termination does in fact decrease the performance for MBRL algorithms of different types. We further experiment with addition schemes incorporate early termination, summarized in appendix || However none of them were successful. We argue that to perform efficient learning complex environments, such Humanoid, early termination is almost necessary. We leave it as an important request for research.
RS
MB-MF
PETS-CEM
PETS-RS
ME-TRPO
GPS
PILCO
SVG
MB-MPO
SLBO
Mean rank
5.3/ 10
10
4.1/10
5/10
58/10
7.9/ 10
8.5/ 10
5.0/ 10
4,7/ 10
4.1/10
Modian rank
10
7/10
4/
5/10
6./
8
10/10
1/
4.5/
10
Table 5: The ranking of the MBRL algorithms the 18 benchmarking environments
5 CONCLUSIONS
In this paper, we benchmark the performance of wide collection of existing MBRL algorithms, evalu- ating their sample efficiency, asymptotic performance and robustness. Through systematic evaluation and comparison, we characterize three key research challenges for future MBRL research. Across this very substantial benchmarking. there is no clear consistent best MBRL algorithm, suggesting lots of opportunities for future work bringing together the strengths of different approaches.
8

---

Under review as a conference paper ICLR 2020
REFERENCES
Yunfei Bai Steve Baker Gino van den Bergen Jeff Bingham Nicola Candussi Erin Catto Lawrence
Chai Erwin Coumans Disney Animation Benjamin Ellenberger Christer Ericson Google Dirk Gregorius Marcus Hennix Jasmine Hsu MBSim Development Team Takahiro Harada Simon Hobbs John Hsu Ole Kniemeyer Jay Lee Francisco Leon lunkhound Vsevolod Klementjev Phil Knight John McCutchan Steven Peters Roman Ponomarev Nathanael Presson Gabor PUHR Arthur Shek Russel Smith Sony Jakub Stephien Marten Svanfeldt Jie Tan Pierre Terdiman Steven Thompson Tamas Umenhoffer AMD, Apple. Bullet physics sdk. https://github... 2014.
culleepphysiccs/blll
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint 180i 00177. 2018.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unitying count-based exploration and intrinsic motivation. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems 29, pp. 1471-1479. Curran Associates, Inc., 2016. URL http: http:/ppprrss.....c nigs. pa/pper httpttp:/ppppprssiiss....o/
pdf
Alberto Bemporad, Manfred Morari, Vivek Dua, and Efstratios N Pistikopoulos. The explicit linear
quadratic regulator for constrained systems. Automatica. 38(1):3-20 20, 2002.
Zdravko I Botev, Dirk P Kroese, Reuven Y Rubinstein, and Pierre L'Ecuyer. The cross-entropy method for optimization. In Handbook of statistics, volume 31, pp. 35-59. Elsevier, 2013.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 000.66000011440.. 2016.
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3381-3388. IEEE, 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamies models. arXiv preprint arXiv: 1805. 121 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. 2018.
Model-based reinfrrmmeet learning via meta-policy optimization. CoRR,
UtR.httt://arxxvvcoo org/ abs 18099 05214
fieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Richard Dearden, Nir Friedman, and David Andre. Model based bayesian exploration. In Proceedings of the Fifieenth Conference on Uncertainty in Artificial Intelligence, 9AI'99, pp. 150-159, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-614-9. URL http:://l. acm. org/ httg/iitaiionn.oo
2073814
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pp. 465-472, 2011.
Marc Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data- efficient learning in robotics and control. IEEE transactions pattern analysis and machine intelligence, 3702):408-433, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Yan Duan, Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329-1338, 2016.
Benjamin Ellenberger.
Pybullet gymperium.
ht:ps::/githab..cmm
pybullet -ymm 2018.

---

Under review as a conference paper ICLR 2020
C. Finn, M. Zhang. J. Fu. X. Tan, Z. McCarthy, E. Scharff, and S. Levine. Guided policy search code implementation, 2016a. URL http:://plrl ril.berkeley.edu/gps.
/pps Software available from
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp- 49-58, 2016b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. CoRR, abs/1703.03400, 2017. URL http:://arsi..org/aaaaaaaa 03400]
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:/80. 2018.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2450-2462, 2018a.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 2018b.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor, arXiv preprint arXiv: 012200 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint 2018.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944-2952, 2015.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610.0511 2016.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang. Ali Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprini arXiv: 077.0222766 2017.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelli- gence, 2018
Rein Houthooft, Xi Chen, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In D. D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1109-1117. Curran Associates, Inc., 2016. URL /tttt://papprs..e httppppppww.app.ppp.ppppppeeapae www.:icppppc.....e /aper// r wwt.varattiinn
1591-vime-ve
rmat. on max i mi z -x on.. pdf]
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of bench- marked deep reinforcement learorn tasks for continuous control, arXiv preprint / 2017.
Sanket Kamthe and Mare Peter Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. arXiv preprint arXiv: arXivv 1706,0649, 2017.
Oleg Klimov and J Schulman. Roboschool, 2017.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, pp. 177-180, 2007.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 1802. 10592, 2018.
Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071-1079, 2014.
Sergey Levine, Nolan Wagener, and Pieter Abbeel. Learning contact-rich manipulation skills with guided policy search. In 2015 IEEE international conference on robotics and outomation (ICRA), pp. 163-163. IEEE, 2015.
10

---

Under review as a conference paper ICLR 2020
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: arXiv:/509.00 05090099999 2015.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. ICLR, 2019.
Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning, pp. 1128-19331 2016.
William H Montgomery and Sergey Levine. Guided policy search via approximate mirror descent. In Advances in Neural Information Processing Systems, pp. 4008-4016, 2016.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynam- ics for model- based deep reinforcement learning with model-free fine-tuning. arXiv preprint arXiv: 170.0000..,566 2017.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206 5210. IEEE, 2015.
Xue Bin Peng, Glen Berseth, and Michiel Van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Transactions on Graphics (TOG), 35(4):81, 2016.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 373) 2018
Aravind Rajeswaran, Kendall Lowrey, Emanuel V Todorov, and Sham M Kakade. Towards gen- eralization and simplicity in continuous control. In Advances in Neural Information Processing Systems, pp. 6553-6564, 2017.
Anil V Rao. A survey of numerical methods for optimal control. Advances in the Astronautical Sciences, 135(1):497-525 2009.
Arthur George Richards Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured
prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 627-635, 2011.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: a source of plateaus in deep reinforcement learning, arXiv preprint arXiv:1904, 11455, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region
policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1889-1897. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Richard S Sutton. Integrated architectures for learning. planning, and reacting based on approximating dynamic programming. In Machine Learning Proceedings 1990, pp. 216-224. Elsevier. 1990.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160-163 1991a.
11

---

Under review as a conference paper ICLR 2020
Richard S Sutton. Planning by incremental dynamic programming, In Machine Learning Proceedings 199/, pp. 353-357. Elsevier, 1991b.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 4906-4913. IEEE, 2012.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 18Xiv=880.0000000 2018.
Anirudh Vemula, Wen Sun, and Andrew Bagnell. Contrasting exploration in parameter and action space: A zeroth-order optimization perspective. arXiv preprint arXiv:990.11103,, 2019.
Michael Waskom. Seaborn: statistical data visualization. htttp:/seaborn, pydata.
2010. Accessed: 2010-09-30.
Marco Wiering and Jürgen Schmidhuber. Efficient model-based exploration. In PROCEEDINGS OF THE SIXTH INTERNATIONAL CONFERENCE ON SIMULATION OF ADAPTIVE BEHAVIOR: FROM ANIMALS TO ANIMATS 6, pp. 223-228. MIT Press/Bradford Books, 1998.
Tianhao Zhang, Gregory Kahn, Sergey Levine, and Pieter Abbeel. Learning deep control policies for
autonomous aerial vehicles with mpc-guided policy search. 2016 IEEE international conference on robotics and automation (ICRA), pp. 528-535, IEEE, 2016.
12

---

Under review as a conference paper ICLR 2020
A ENVIRONMENT OVERVIEW
We provide an overview of the environments in this section. shows the dimensionality and horizon lengths those environments, and Table specifies their reward functions.
Environment Name
Observation Space Dimension
Action Space Dimension
Horizon
Acrobot
200
Pendulum
3
200
InvertedPendullm
4
100
CartPole
4
200
MountainCar
2
200
Reacher2D (Reacher)
11
50
HalfCheetah
17
1000
Swimmer-v0
8
1000
Swimmer
8
2
1000
Hopper
11
1000
Ant
28
1000
Walker 2D Humanoid
17
1000
376
17
1000
SlimHumanoid
45
17
1000
Table 6: Dimensions of observation and action space, and horizon length for most of the environments used the experiments.
Environment Name
Reward Function RG
Acrobot
cos 01,, cos (01,, + 02,t)
Pendulum InvertedPendullmm
cos 0g 0.1 sin 0, 0.102 - 0.001a²
03
CartPole
cos 0, 0.01x²?
MountainCar
position,
Reacher2D (Reacher)
distance,
HalfCheetah
2. 0.1 a e3
Swimmer-v0
4; 0.0001 a
Swimmer
4; 0.0001 a,
Hopper
2.
0.1|
3.0 X (2) 1.3)
Ant
2,
0.1
a -30 x (2: - 0.57)
Walker2D
4, 0.
a
3.0 (2: 1.3)2
Humanoid
50/3 d2- 0.1 a.
11
5c
6 X impact 5 bool <= 2 <= 2.0)
SlimHumanoid
50/3 x d1 0.1| + 5 x bool <= 2 <= 2.0)
Table 7: Reward function for most of the environments used in the experiments. The tasks specified by the reward functions are discussed in further detail in Section A.
A. ENVIRONMENT- SPECIIIC DESCRIPTIONS
In this section, we provide details about environment-specifie dynamics and goals.
Acrobot The dynamical system consists of pendulum with two links. The joint between the two links is actuated. Initially, both links point downwards. The goal is swing up the pendulum, such that the tip of the pendulum reaches given height. Let 01,,, 62 be the joint angles of the first (with one end fixed to hinge) and second link at time t. The 6-dimensional observation at time the tuple: (cos 01,e, sin 0r,t, cos 02 sin 02 01,,, 02.)). The reward is the height of the tip of the pendulum: R2, cos 01,e e cos (01, - + 02.2).
13

---

Under review as a conference paper ICLR 2020
Pendulum A single-linked pendulum is fixed on the one end, with an actuator located on the joint. The goal is to keep the pendulum at the upright position. Let 0, be the joint angle at time l. The 3-dimensional observation time is (cos fr, sin 0,, 6&) The reward penalizes the position and velocity deviation from the upright equilibrium, as well as the magnitude of the control input.
InvertedPendullum The dynamical system consists of a cart that slides on rail, and a pole connected through an unactuated joint to the cart, The only actuator applies force on the cart along the rail. The actuator force is a real number. Let dg be the angle of the pole away from the upright vertical position, and Ia be the position of the cart away from the centre of the rail at time t. The 4-dimensional observation at time is (I,, O,, ft, 0z). The reward -0 penalizes the angular deviation from the upright position.
CartPole The dynamical system of Cart-Pole is very similar to that of the Inverted Pendulum environment. The differences are: 1) the real-valued actuator input is discretized to -1, 1, with a threshold at zero; 2) the reward cos 0g - 0.01r, indicates that the goal is to make the pole stay upright, and the cart stay at the centre of the rail.
MountainCar A car is initially positioned between two "mountains", and can drive on one- dimensional track. The goal is to reach the top of the "mountain" on the right. However, the engine of the car not strong enough for it to drive up the valley in one go, so the solution is to drive back and forth to accumulate momentum. The observation at time is the tuple (position,, velocity:) ) where both the position and velocity are one-dimensional, with respect to the track. The reward at time is simply position.. Note that we use fixed horizon, so that the agent is encouraged to reach the goal as soon as possible.
Reacher2D (or Reacher) An arm with two links is fixed at one end, and is free to move on the horizontal 2D plane. There are two actuators, located at the two joints respectively. At each episode, a target is randomly placed on the 2D plane within reach of the arm. The goal to make the tip of the arm reach the target as fast as possible, and with the smallest possible control input. Let Og be the two joint positions, Ztorget,t be the position of the target, and Stip.t be the position of the tip of the arm at time 1, respectively. observation is (cos 0,, sin 1r, Starget,t, O,, Dtip.t - Ftarget,t). The reward at time is [iip,t - Ttorget,e| 2 a 112. where the first term is the Euclidean distance between the tip and the target.
HalfCheetah Half Cheetah is 2D robot with 7 rigid links, including 2 legs and torso. There are 6 actuators located at joints respectively. The goal is to run forward as fast as possible, while keeping control inputs small. The observation include the (angular) position and velocity of all the joints (including the root joint, whose position specifies the robot's sss position the world coordinate), except for the position of the root joint. The reward is the direction velocity plus penalty for control inputs.
Swimmer-v0
Swimmer-v0 is a 2D robot with 3 rigid links, sliding on a 2D plane. There are actuators, located on the 2 joints between the links. The root joint is located at the centre of the middle link. The observation include the (angular) position and velocity of all the joints, except for the position of the two slider joints (indicating the and y positions). The reward is the r direction velocity plus penalty for control inputs.
The dynamical system of Swimmer is similar to that of Swimmer v0, except that the root is located at the tip of the first link (i.e. the "head" of the swimmer).
Swimmer
Hopper Hopper is a 2D "robot leg" with 4 rigid links, including the torso, thigh, leg and foot. There are 3 actuators, located at the three joints connecting the links. The observation include the (angular) position and velocity of all the joints, except for the position of the root joint. The reward is the direction velocity plus penalty for the distance to a target height and control input. The intended goal is to hop forward as fast as possible, while approximately maintaining the standing height, and with the smallest control input possible. We also add an alive bonus of to the agents at every time-step, which is also applied to Ant, Walker2D.
14

---

Under review as a conference paper ICLR 2020
Ant Ant is 3D robot with 13 rigid links, including torso 4 legs. There are 8 actuators, 2 for each leg, located at the joints. The observation include the (angular) position and velocity of all the joints, except for the x and y positions of the root joint. The reward is the direction velocity plus penalty for the distance to a target height and control input. The intended goal is to go forward, while approximately maintaining the normal standing height, and with the smallest control input possible.
Walker2D Walker 2D is planar robot, consisting of 7 rigid links, including a torso and 2 legs. There are actuators, 3 for each leg. The observation include the (angular) position and velocity of all the joints, except for the x position of the root joint. The reward is the direction velocity plus penalty for the distance target height and control input. The intended goal to walk forward as fast as possible, while approximately maintaining the standing height, and with the smallest control input possible.
Humanoid Humanoid is 3D human shaped robot consisting of 13 rigid links. There are 17 actuators, located at the humanoid's 's abdomen, hips, knees, shoulders and elbows. The observation space include the joint (angular) positions and velocities, centre of mass based inertia, velocity, external force, and actuator force. The reward is the scaled x direction velocity, plus penalty for control input, impact (external force) and undesired height.
Slimodd
The dynamical system of Slim Humanoid similar that of Humanoid, except that the observation is simply the joint positions and velocities, without the center of mass based quantities, external force and actuator force. Also, the reward longer penalizes the impact (external force).
B
SEARCH AND ENGINEERING DETAILS
In this section, provide more detailed description of the hyper-parameter search for each algorithm. Note that we select the best hyper- peee-pereetter combination for each algorithm, but we still provide reference syper-parameter combination that is generally good for all environments.
B. ILQG
For iLQG algorithm, the hyper- spperpparaettes searched summarized in 8 While the recommended hyper-parametery usually have the best performance, they can result in more computation resources needed. In the following sections, number of planning trajectory is also refereed as search population size.
Hyper-parameter
Value Tried
Recommended Value
planning horizon max linesearch backtrack number iLQG update per time-step number of planning trajectory
10, 20, 30, 50, 100 1, 10, 15, 20
20
10
5, 10, 20
10
1, .... 10, 20
10
Table 8:
grid search options for iLQG.
B.2 GROUND-TRUTH CEM AND GROUND- RS
For the CEM and RS with ground-truth dynamics, we search only with different planning horizon, search population size. which include 10, 20, 30, 40, 50, 60, 70, 80, 90, 100. As also mentioned in planning horizon dilemma section 4.6 the best planning horizon is usually 20 to 30.
B.3 RS, PETS AND PETS-RS
We mention that in Nagabandi et al. (017). RS has very different hyper-parameter sets from the RS studied in PETS-RE -S Chua et al. 2018) The search of harame-assmettrs for RS is the same for
15

---

Under review as a conference paper ICLR 2020
Slyper-praamete
Value Tried
Recommended Value
planning horizon
10, 20, 30, .... 90, 100
30
search population size
500, 1000, 2000
1000
Table 9: Hyper-parameter grid search options for RS, CEM using ground-truth dynamics.
RS using ground-truth dynamics illustrated in the Table 9 the hyper-parameterd in Table IO For simpler environments, it is usually better to use planning horizon of 30. For environments such as Walker2D and Hopper, 100 the best planning horizon. We
The PETS and PETS is searched with
hyper-parameter
Value Tried
Recommended Value
planning horizon search population size
10, 30, 50, 60, 70, 80, 90, 100 50, 100, 500, 1000, 2000
30 100
500
elite size PETS combination
50, 100, 150
50
D-E, DE-E, PE-E, PE-TSinf, PE-TS1, PE-DS
PE-E
Table 10: Hyper-parameter grid search options for RS.
note that for the dynamics proppagtton combination, we choose PE-E not because of having the best performance. PE-E is among the best models, with comparable performance other combinations such as PE-DS, PE-TS1, PE-TSinf. However PE-E is very computation efficient compared to other variants. For example, PE-DS on PETS- CEMM costs hours for one random seed for HalfCheetah with planning horizon of 30 train for 200,000 time-steps. While PE-E usually only takes about hours, and is suitable for research. The best models for HalfCheetah uses planning horizon of 100, and takes about 15 hours.
We also note that with some trivial pre- pre-processing of the observation function, such as using sin and functions to the angle observation, the performance performance on the pre-processed task provided in the original PETS paper[Chua Chua et al. (2018), and compare it with the performance the provided task in the benchmark in Figure We note that for the modified environments in Chua et al. (2018), the performance can reach more than 10000 for 200k time-steps.
be much better for PETS. We include the
23:43
(a) The performance on both environments for PETS-CEM and PETS-RS
(b) The performance of PETS-CERS modified HalfCheetah
(c) The performance of PETS-RS modified HalfCheetah
Figure 4: The performance of HalfCheetah from our benchmark, and the HalfCheetah from the PETS paper Chua et al. (2018), which refer to as "Modified". We also include the performance of PETS-CEM and PETS-RS RS using different dynamics- proppagainoo combination on the modified HalfCheetah from Chua et al. (2018).
In the benchmark results in the main paper, PETS-RS uses the search scheme in Table 10 except for it does not have hyper-parameter of elite size.
B.4 MBMF
Originally MBMF was designed run for 1 million time-steps Nagabandi et al. (2017). Therefore, to accommodate the algorithm with 200,000 time-steps, we perform the search in Table I1
16

---

Under review as a conference paper ICLR 2020
Hyper-parae prpee-prraetter
Value Tried
Recommended Value
trust region method
TRPO, PPO
PPO
search population size
1000, 5000, 2000
5000
planning horizon time-steps per iteration model based time-steps
10, 20, 30
20
1000, 2000, 5000
1000
5000, 7000, 10000
7000
dagger epoch
100, 300, 500
300
Table Hyper-parameter grid search options for MBME.
B.5 METRPO AND SLBO
For METRPO and SLBO, we search for the following syper-paraamters. We note that for environ- ments with episode length of 100 or 200, we always use the same length for imaginary episodes. We also refer Appendix F for more details.
Hyper-parameter
Value Tried
Recommended Value
imaginary episode length
1000, 500, 200, 100
1000
TRPO iterations
1, 10, 20, 30, 40 5, 10, 20
20 / 40
network ensembles
5
Terminate imaginary episode
True, False
False
Table 12: Hyper-parameter grid search options for METRPO and SLBO.
B.6 GPS
The GPS is based on the code-base Finn et al. (2016a). We note that in the original code-base, the agent samples the initial state from several separate conditions. For each condition, there is not randomness of the initial state. However, in our bench-marking environments, the initial state is sample from a Gaussian distribution, which is essentially making the environments harder to solve.
Hyper-parameter
Value Tried
Recommended Value
time-stepp step per iteration
1000, 5000, 10000
5000
kl step
0.5, 1.0, 2.0, 0.5 5, 10, 20, 30
1.0
dynamics Gaussian mixture model clusters
20
policy Gaussian mixture model clusters
10, 20
20
Table 13:
grid search options for GPS.
B.7 PILCO
For PILCO, we search the following hyper-parameter in Table 14 unstable actoss random seeds. Also, it is quite common for PILCO algorithms add additional penalty in existing code-bases using human priors. We argue that is unfair to other algorithms and remove any additional reward functions. Also, for PILCO train for 200,000 time-steps, we have to use a data-set to increase training efficiency.
note that PIL.O is very
B.8 SVG
For SVG, we reproduce the variant of SVG-1 with experience replay, which claimed in Heess et al. (2015).
17

---

Under review as a conference paper ICLR 2020
Hyper- parameterr
Value Tried
Recommended Value
Optimizing Horizon
30, 100, 200, adaptive
100 or 30
episode per iteration
2. 4
data-set size
200, 1000, 2000, 20000, 40000, 10000
200 or 1000
Table 14: Hyper-parameter grid search options for PILCO.
Value Tried
Recommended Value
SVG learning rate data buffer size KL penalty
0.00003, 0.0001, 0.0003, 0.001 25000
0.0001
25000
0.0003, 0.001, 0.003
0.001
Table 15: Hyper-parameter grid search options for SVG.
B.9 MB-MPO
In this algorithm, we use most the peper--prameters in the original paper Clavera et al. (2018), except in the ones the algorithm is more sensitive to.
papee-parameter
Value Tried
Recommended Value
inner learning rate rollouts per task
0.0005, 0.001, 0.01
0.0005
10, 20, 30
20
MAML iterations
30, 50, 75
50
Table 16: Hyper-parameter grid search options for MB-MPO.
B.10 MODEL-FREE BASELINES
For PPO and TRPO, search for different time-steps samples in iteration. For SAC and TD3, we use the default values from Haarnoja et al. (2018) and Fujimoto et al. (2018) respectively.
Value Tried
Recommended Value
time-steps per iteration 1000, 2000, 5000, 20000
2000
Table 17: Hyper-parameter grid search options for model-free algorithms.
18

---

Under review as a conference paper ICLR 2020
C DETAILED BENCH-MARKING PERFORMANCE RESULTS
-30
AL.00
-20
7:35
015
FST5-C02
5000
5000
1ROM
F500 2ow
17009 1000
(a) Acrobot
(b) Pendulum
(c) InvertedPendulum
1180
VELVPO
14.00
$1.0
-20
RC
BAC
1030
-10
-03
AET32O
(125 CCM
7000 X000
1260
1000
20:00
2:00 1000
7500
J008
19000 -2:00 20000
2:00
2000
7800 JUNB
1800 10:00 20000
(d) CartPole
(c) MountainCar
(f) Reacher
$230
$1.0
60
MEECO
METEEO
1950
2000/
2000 J008 DIME 1500 tran zero
30
g HalfChectah
(h) Swimmer-vO
Swimmer
300
FR.00
DALOO
SEECT
-200
17940 21000
Hopper
(k) Hopper-ET
Ant
-50
250
1300
SL.0
SLEO
-30
000
2008 From
(m) Ant-ET
Walker2D
(0) Walker2D-ET
Figure 5: Performance curve for MBRL algorithms. There are still 3 more figures in continued Figure
In this appendix section, we include all the curves of every algorithms in Figure and Figure Some of the GPS curves and PILCO curves are not shown in the figures. We note that this is because their reward scale is sometimes very different from other algorithms.
19

---

Under review as a conference paper ICLR 2020
-30
ETO
500
100
5007 37:0 1000
2000
5000
7l00o
1000
10:0
153601
700000
(p) Humanoid-ET
( SlimHumanoid-ET
(r) Slimoumanoid
Figure 6: (Continued) Performance curve for MBRL algorithms.
MBMF
Average
4.41
7.47
4.01
144
44.68
0.04
0.031
3.87
Staddand devistion
2.16
0.74
3,96
1.45
1.40
7.22
N.
0.33
9.95
1.63
0.0172
0.0116
0.89
0.67
Table 18: Wall-clock time hours averaged for each algorithm trained for 200k time-steps from the original Table
In addition to Table we also have the performance of SVG trained for million time-steps. For the environments of HalfCheetah, Ant, Walker2D and Hopper, the performance respectively 578.7:439.1, 472.0 48.11 -1168.2:::377 and -783.4158008
D NOISY ENVIRONMENTS
In this appendix section, we provide more details of the performance with noise for each algorithm. In Figure and Figure g] we show the curves of different algorithms, and Table|T9ani Table 20 show the performance numbers at the end the of training. The pink color indicates decrease of performance, while the green color indicates increase of performance, and black color indicates a almost the same performance.
Cheetah
Cheetah, d.
Cheetah, d.
0.01
Cheelah, 0.
Cheetah, 0,
iLQR
2142.6 + 137.7
-25.3 127.5
187.2 102.9
261.2 = 106.8
310.1 + 112.6
GT-PETS
14777.2
13964.2
1638.5 188.5
9226.5 8893.4
11484.5 = 12264.7
13160.6 13642.6
GT-RS
815.7 1: 38.5
38.5
493.3 1: 38.3
604.8 42.77
645.7 39.6
RS
421.0
146.2 +: 19.9
423.1 28.7
445.8 19.2
442.3 26.0
MB-MF
126.9 72.7
16.1:887. 87.8
232.1 122.0
184.0 148.9
257.0 96.6
PETS
2795.3
1879.5 801.5
2410.3 844.0
2427.5 674.1
2427.2 1118.6
PETS-RS
966.9 471.6
217.0 193.4
814,9 678.6
1128.6+ 6-4.22
1017.5 734.9
ME-TRPO
2283.7 4
409.4 834.2
1396.9 + 834.8
1319,8 698.0
2122.9 889.1
GPS
41.7
-68 13.6
175.2 169.4
45.7
94.0 + 57.0
PILCO
267.0
-882.0
-275,4 164.6
.175.6 284.1
290.3
SVG
336.6 387.6
0.1 11122133
240.8 236.6
163.5 338.6
21.9 81.0
MB-MPO
3639.0 1185.8
2356.4
3635.5 + 1486.8
3372.9 1373
3718.7 922.3
SLBO
1097.7 166.4
212.5 279.6
1244. +: 604.0
1593.2 4: 2:5.00 265.0
731.1 + 215.8
PPO
17.2
-113,3
-83.1 117.7
-28.0 54.1
-35.5 87.8
TRPO
-12.0 85,5
-146.0 67.4
9.4 4. 57.6
-32.7 110.9
-70.9 71.9
TD3
3614.3 82.1
895.7 61.6
817.3 = 11.0
4256.5 + 117.4
3941.8 + 61.3
SAC
4000.7 1 202.1
1146.7 1 67.9
3869.2 88.2
3530.5 + 67.8
3708.1 +: 96,2
Table 19: The performance of each algorithm in noisy HalfCheetah (referred to in short hand as "Cheetah") environments. The green and red colors indicate increase and decrease in performance, respectively.
E PLANNING HORIZON DILEMMA GRID SEARCH
We note that we also perform the dilemma search with different population size with learnt PETS- CEM. We experiment both with the HalfCheetah our benchmarking environments, as well as the environments from Chua et al. (018)) 8), whose observation is further pre-processed. It can be seen from the figure that, planning horizon dilemma exists with different population size. We also show that observation pre- proceesessing affect the performance by large margin.
20

---

Under review as a conference paper ICLR 2020
(al) HalfCheetah PETS-CEM (a2) Pendulum PETS-CEM
(a3) CartPole PETS-CEM
(b1) HalfCheetah PETS-RS
(b2) Pendulum PETS-RS
(b3) CartPole PETS-RS
(cl) HalfCheetah RS
(c2) Pendulum RS
(c3) CartPole RS
(d1) HalfCheetah MBMF
(d2) Pendulum MBMF
(d3) CartPole MBMF
(el) HalfCheetah METRPO (e2) Pendulum METRPO
(e3) CartPole METRPO
Figure 7: The performance curve for algorithms with noise. We represent the noise standard deviation with "O" and "A" respectively for the noise added to the observation and action space.
F PLANNING HORIZON DILEMMA IN DYNA ALGORITHMS
In this section, we study how the environment length and imaginary environment length (or planning horizon) affect the performance. More specifically, we test with HalfCheetah and Ant, using different environment length form [100, 200, 500. 1000]. For the planning horizon, besides the matching
21

---

Under review as a conference paper ICLR 2020
2800
(fI) HalfCheetah GPS
(f2) Pendulum GPS
(f3) CartPole GPS
(g1) HalfCheetah TRPO
(g2) Pendulum TRPO
(g3) CartPole TRPO
1900
(h1) HalfCheetah PPO
(h2) Pendulum PPO
(h3) CartPole PPO
(il) HalfCheetah TD3
(12) Pendulum TD3
(i3) CartPole TD3
1950
(i1) HalfCheetah SAC
(i2) Pendulum SAC
(i3) CartPole SAC
Figure 8: (Continued) The performance curve for algorithms with noise. We represent the noise standard deviation with "O" and "A" respectively for the noise added to the observation and action space.
length, we also test all the length from [100, 200, 500, 800, 1000). The figures are shown in Figure 10 and the tables are shown in T1
Note that we also include planning horizon longer than the actual environment length for reference. For example, for the Ant with 100 environment length, we also include results using 200, 500, 800,
22

---

8536 18 Under review as a conference paper ICLR 2020
Pendulum, 2g
dg
Cart.Pole, 42
0.1
Can- 6g
iLQR
160.8 1. 29.8
-357.9
251.9
.2.2 166.5
200.3 0.5
197.8 1 2.9
200.4 1: 0.7
GT-PETS
26.2
199.5
GT-RS
171.5 = 31.8
125.2 40.3
157.8 = 39.1
201 0.0
200.2 +
201.0 = 0.0
RS
9.4
154.5
12.9
160.1 6.7
00100
197.7 4:4.
200.9
MB MB-MF
157.5 =: 13.2
162.8
14.7
163.9 8.5
199.7
152.3 48.3
137.9 48.5
PETS
167.4 =: 53.0
174.7
27.8
166.7 1 52.0
199.5 3.0
156.6 50.3
196.1 1 5.1
PETS-RS
58.6
ME-TRPO
177.3-199 1.9
173.3 1. 3.2
173.7 4.8
160.1 1: 69.1
174.9 1: 21.9
165.9 1: $8.5
GPS
168,9
= -0660 149
PILCO
-32.66 410.1
-211.6 212.1
168.9 30.5
-1.9 155.9
139.9 54.8
SVG
141.4 + 62.4
86.7 34.6
73.2
82.1 31.9
119.2 46.3
106.6 42.0
MB MB-MPO
171. 26.9
178.4-1222 22.2
183.8 19.9
199.3 2.3
65.1 542.6
198.2
SLBO
1735
171.1 1.5
173.6
78.0 166.6
-91.7 1: 8010
-1418 = 167.5
PPO
8.3
165.9 15.4
157.3 + 12.6
86.5
120.5
120.3 46.7
TRPO
166.7 7.3
167.5 1: 6.7
161.1 1: 13.0
47.3 1: 15.7
-572.3 1: 368.0
-818.0 = 288.1
TD3
7.2
3.1
1.0.444.
180.9
SAC
168.2 9.5
169.3 5.6
169.1 + 12.6
199.4 = 0.4
60.9 23.4
70.7 11.4
Table 20: The performance of each algorithm noisy Pendulum and Cart-Pole environments. The
green and red colors indicate increase and decrease in performance, respectively. Numbers in black indicate no sigmificant change compared to the default performance.
1358 1150 L4
1616 16-0-00-00
3305 1939-33
9196 61
55
3007
7720
2501
20
SS63 OT
SPG 544
1007
2458
007
1267.
1170.85
D000
30
1006.14
1600
$180
9276
1105
1106
4000
1312.20
3347.00
55005100
2900
LE96.64
1427.80
1106.25
2562.61
67
2720.51
2:17.46
8150.90
1472
220
1-800.8
1562.00
07402
000.24
1751.10
1487.
68
3300
600
8844.82
1247.48
0094.90
0770000
867.88
1774L14
2200L00
2577.16
700
1722
1117
1975
1170
1397.77
3791
7711
2400
803
1318
8638
800-37
2000
0.25..05
000
1850.60
1312.00
750 26
7750,
874.08
20100
1014.11
1663
D393
1002
1100
2750.51
80
30
40
80
80
plana
place
(b) HalfCheetah with pre-processing in Chua
(a) HalfCheetah.
et al. (2018)
Figure 9: The performance grid using different planning horizon and depth.
Environment
Original Length
Horizon-110 100
Horizon=200
Horizon=500
Horizon-800
Horizon= 1000
HalfChoctah
Eav-100
250.7 + 32.1
290.3 = 44.5
222.0 34.1
253.0 +: 22.3
243.7 + 41.7
HalfCheetah
Env-200
422.7 = 143.7
675.4 139.6
529.0 50.0
124.5
528.1 74.7
HalfCheetah
Eay-500
816.6 = 466.0
583.4 +: 392.7
399.2 = 250.5
986.9 1: 501.9
1062.7 182.0
HalfCheetah
Env-1000
1312.1 + 656.1
1514.2 + 1001.5
1522.6 456.3
1544.2 + 1349.0
2027.5 + 1125.5
Ant
Eav-100
1207.8 + 41.6
1142.2 25.7
1111.9 35.3
1103.7 = 70.9
1085.5 + 22.9
Ant
Env-200
1249.9 127.7
1172.7 36.4
1136.9 32.6
1079.7 37.3
10%6. = 18.6
Ant Ant Env-1000
Eav-500
1397.6 + 49.9
1319.1 + 50.1
1423.6 + 46.2
1287.3 + 118.7
1331.5 = 92.9
1666.2
201.9
1646.0 + 151.8
1680.7 255.3
1530.7
1647.2 118.5
Table 21: The performance for different environment length and planning horizon SLBO summa- rized in to a table. HalfCheetah and Ant were used in the experiments.
1000 planning horizon. As we can see, for the HalfCheetah environment, increasing planning horizon does not have obvious affects on the performance. In the Ant environments with different environment lengths, a planning horizon of 100 usually produces the best performance, instead of the longer ones.
23

---

Under review as a conference paper ICLR 2020
300
900
180
-28
The mus 2018
(a) HalfCheetah, Env Length 100
(b) HalfCheetah, Env Length 200
(c) HalfCheetah, Env Length 500
Loughh 20-Clociacco
13
Unee 2010
20111
2000 100R
1901. June 12A00 2.00 15000 Soue
(d) HalfCheetah, Env Length 1000
(c) Ant. Env Length 100
(f) Ant, Env Length 200
Bestoool 1h-in
2:00 C000 1080 1800 12600 19000 1700 2008
2:H 2008
2000 12400
18K0 1500 2000
(g) Ant,
(h) Ant,
Env Length 500
Env Length 1000
Figure 10: The performance curve for different environment length and planning horizon in SLBO. HalfCheetah and Ant used in the experiments.
G DYNAMICS NETWORK STRUCTURE AND CAPACITY
We also study how the structure or the capacity the dynamics network affect the performance. In Figure II we show the performance of PETS-CEM HalfCheetah using different networks to learn the dynamics. In Rajeswaran et al. (2017). the authors show that two simple networks, linear network and RBF network, can be used as the policy network, which obtains similar performance compared with using multi-layer perceptron (MIP) in model-free reinforcement learning. Therefore, we use linear network and RBF network to learn the dynamics in model-based reinforcement learning. We also test the performance using wider and deeper networks.
As we can from Figure 11 in model-based RL, linear network and RBF network lead to catastrophic performance drop, indicating multi-layer neural network (MLP) is needed to learn the features to model forward dynamics. On the other hand, increasing the capacity of the MLP by making it much deeper and wider does not seem to increase the performance either.
H NON-MuJOCO ENVIRONMENTS
To facilitate research, we also provide simulated agents that uses free physics engines other than MuloCo. Some of the results can be shown in Figure [2] We note that these environments have different reward and observation functions from the environments benchmarked in the main paper. Therefore the results can not used compare with the results of the environments based on MuJoCo.
24

---

Under review as a conference paper ICLR 2020
20000
Figure 11: The performance of PETS-CEM on HalfChectah using different network structures.
Siice
(a) Roboschool Reacher
(b) Pybullet Reacher
(c) Roboschool HalfCheetah
Figure 12: The performance curve for some of the tasks based on Roboschool or Pybullet Klimov & Schulman 2017): AMD 2014); Ellenberger (018).
We use the best super-parameters for the corresponding MuJoCo environments, which indicates potential performance gain can be obtained with more careful syper-parameter search.
I
EARLY TERMINATION
GT-CEM
GT-CEM-ET
GT-CEM-ET, 100
leamed-CEM
leamed-CEM-ET
Ant Hopper
12115.3 209.7
8074.2 + 210.2
4339.8 87.8
1165.5
226.9
162.6 142.1
3232.3 + 192.3
260.5 + 12.2
817.8
217.6
1125.0 679.6
801.9 194.9
Walker2D
7719.7 + 486.7
105.3 + 36.6
6310.3 55.0
-493.0 583.7
290.6 113.4
Table 22: The performance of PETS algorithm with and without early termination.
In this appendix section, we include the results of several schemes experiment with early termina- tion. The early termination dilemma is universal in all MBRL algorithms we tested, including Dyna- algorithms, shooting algorithms, and algorithm that performs policy search with pack-propagation through time. To study the problem, we majorly start with exploring shooting algorithms including RS, EESS-RS and PETS- -EM. which only relates to early termination during planning. In Table and Table 24 we also include the results that the agent does not consider being terminated in planning, if it will be terminated, which represent "Unaware".
GT-CEM
CT-CEM+ET-Unaware
GT-CEM-ET
GT-CEM-ET, 7 100
Ant Hopper Walker2D
12115.3 + 209.7
226.0 + 178.6
8074.2 + 210.2
4339.8 = 87.8
3232.3 + 192.3
256.8 + 16.3
260.5 + 12.2
817.8 217.6
7719.7 + 486.7
254.8 233.4
105.3 + 36.6
6310.3 = 55.0
Table 23: The performance using ground-truth dynamics for CEM.
For the algorithms with unknown dynamics, we specifically study PETS. We design the following schemes.
Scheme The episode will be terminated and the agent does not consider being terminated during planning.
25

---

Under review as a conference paper ICLR 2020
GT-RS
GT-RS-ET-Unaware
GT-RS-ET
GT-RS-ET, 100
Ant Hopper Walker2D
2709.1 631.1
2519.0 469.8
2083.8 537.2
2083.8 + 537.2
-2467.2 + 55,4
209.5 + 46.8
220.4 + 54.9
289.8 + 30.5
-1641.4 + 137.6
207.9 27.2
231.0 32.4
258.3 + 51.5
Table 24: The performance using ground-truth dynamics for RS.
Scheme A
Scheme B
Scheme D
Scheme C
Scheme E
Ant Hopper Walker2D
1165.5 = 226.9
81.6 + 145.8
171.0 + 177.3
110.8 + 171.8
162.6 + 142.1 684.1 157.2
1125.0 679.6
129.3 36.0
701.7 = 173.6
801.9 + 194.9
-493.0 = 583.7
-2.5 + 6.8
-79.1 + 172.4
290.6 + 113.4
142.8 + 150.6
Table 25: The performance of PETS- CEM using learned dynamics at 200k time- stees.
Ant-ET-Uswware
Ant-FT
Ant-ET-2xPorally
Ant-ET--SPPenllt
Ait-FT-IIxPetally
Ant-FT-20xPenadly
Ant-FT-RUPennlly
GFCCEM GE-RS
226.0 178.6
8074.2 210.2
1940,99 20519
S091.3 1831
7968.8 179.6
7968.9 181.5
7601.5 11408
2519.0
2085.8
6364
2591.1
763.2
2728.8 855.5
Table 26: The performance of agents using different alive bonus or depth penalty during planning.
Scheme B: The episode will be terminated early and the agent adds penalty in planning to avoid being terminated.
Scheme C:
The episode will be terminated, and the agent pads zero rewards after the episode is terminated during planning.
Scheme D: The same Scheme A except for that the episode will be terminated.
Scheme E: The same as Scheme C except for that agent allow interact with the environment for
extra time-steps (100 time-steps for example) to learn dynamics around termination boundary.
The results are summarized in Table 25 penalty during planning, whose results are shown in Table 26
We also study adding more alive bonus, i. e. more death
26


Reviews:
[{"review_id": "S1lyTWa2Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a systematic empirical evaluation of model-based RL algorithms on (mostly) continuous control environments from OpenAI Gym, with comparison to popular model-free algorithms. It identifies three challenges for model-based RL, learning the dynamics, selecting the planning horizon, and applying early termination to guide learning.\n\nA systematic comparison of model-based RL algorithms is missing from the literature, and I believe that this paper does a fairly thorough job of providing such a comparison. A wide range of algorithms are selected, and the environments are representative of those commonly used in the literature. The first two challenges identified have been recognized in the literature. For example, Vemula et al. (2019) [1] discuss the planning horizon in random search RL algorithms.\n\nHowever, I would like to see some results on the policy search algorithms such as PILCO in Section 4.5, even if they are on the simpler environments. Currently they are not represented in Table 4. \n\nMinor comment:\n1. There are several instances where the writing should be clarified, e.g. acronyms are not explained before they are used. For example, it would be helpful to the reader to define GT-CEM and GT-RS in Section 3, especially as Table 1 (page 6) comes before the text discussing those two algorithms (page 7). \n2. Table 1 is a bit difficult to parse. Maybe it could be split up, or some algorithms/environments deferred to the appendix.\n\n[1] Vemula, Anirudh, Wen Sun, and J. Bagnell. \"Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective.\" The 22nd International Conference on Artificial Intelligence and Statistics. 2019.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1gHh2vitS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper performs an empirical comparison of a number of model-based RL (MBRL) algorithms over 18 benchmarking environments. The authors propose a set of common challenges for MBRL algorithms.\n\nI appreciate the effort put into this evaluation and I do think it helps the community gain a better understanding of these types of algorithms. My main issue with the paper is that I don't find the evaluation thorough enough (for instance, no tabular environments are evaluated) and the writing still needs quite a bit of work. I encourage the authors to continue this line of work and improve on what they have for a future submission!\n\nMore detailed comments:\n- Intro: \"model-free algorithms ... high sample complexity limits largely their application to simulated domains.\" I'm not sure this is a fair criticism. Model-based are also mostly run on simulations, so sample efficiency is not necessarily the cause model-free are only run on simulations. Further, this statement kind of goes against your paper, since all your evaluations are on simulations!\n- Intro: \"2) Planning horizon dilemma: ... increasing planning horizon... can result in performance drops due to the curse of dimensionality...\" and \"similar performance gain[sic] are not yet observed in MBRL algorithms, which limits their effectiveness in complex environments\" goes against the point you made earlier on in the intro about sample efficiency.\n- Preliminaries: \"In stochastic settings, it is common to represent the dynamics with a Gaussian distribution\", this is only for continuous states. It would be nice if you could evaluate tabular environments as well.\n- Sec 4.1: \"we modify the reward funciton so that the gradient... exists...\" which environments were modified and how did they have to be modified?\n- Sec 4.1: You discuss early termination but have not defined what exactly you mean by it.\n- Fig 1: 12 curves is still a lot and really hard to make much sense of. A lot of the colors are very similar.\n- Sec 4.1: \"it takes an impractically long time to train for 1 million time-steps for some of the MBRL algorithms\" why?\n- Table 1 is a *lot* of numbers and colors and is really hard to make much sense of. There are also so many acronyms on the LHS it's difficult to keep track.\n- Table 2: What about memory usage?\n- Sec 4.4: \"Due to the limited exploration in baseline, the performance is sometimes increased after adding noise that encourages exploration.\" Why does this noise not help exploration in the baselines?\n- Sec 4.5: \"This points out that when learning models more data does not result in better performance.\" This seems like it's closely correlated with the particular form chosen for the model parameterization more than anything.\n- Fig 3: y-axis says \"Relative performance\", relative to what?\n- Sec 4.7: \"Early termination...is a standard technique... to prevent the agent from visiting unpromising states or damaging states for real robots.\" I've never seen this used as a justification for this.\n- Sec 4.7: \"Table 1, indicates that early termination does in fact decrease the performance for MBRL algorithms\", and then you say \"We argue that to perform efficient learning in complex environments... early termination is almost necessary.\" Those two statements contradict each other.\n\nMinor comments to improve writing:\n- When using citations as nouns, use \\citep so you get \"success in areas including robotics (Lillicrap et al., 2015)\" as opposed to \"success in areas including robotics Lillicrap et al. (2015)\" (the latter is what you have all throughout the paper).\n- Sec 3.1: s/This class of algorithms learn policies/These class of algorithms learn policies/", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Hyxszyn6Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper\nThe authors benchmark 11 model-based and 4 model-free RL algorithms on 18 environments, based on OpenAI Gym.\nThe performance in each case is averaged across 4 different seeds.  Computation time is also reported.\nFurthermore, the authors analyze the performance hit incurred from adding noise to the observations and to the actions\nFinally, the authors propose to characterize what hinders the performance of model-based methods, ie., what they call the dynamics bottlenecks, the planning horizon dilemma, and the early termination dilemma.\nAs a conclusion, it turns out no clear winner stands out, which motivates further development of model-based approaches.\n\nStrong and weak points\nThis is a very interesting empirical study, especially since\n- it includes a comparison with model-free algorithms,\n- it considers computational aspects and indicates what algorithms can be run in real-time,\n- the authors use open-source software (PyBullet) as simulators, which makes the study more reproducible (although the code has not been shared yet)\n\nBut,\n- 4 seeds averaged across is clearly low, given the well-known variance of RL algorithms. In fact, the std values in the tables prove this point. I understand the benchmark is heavy on computation, however, this would only have delayed the output of the numbers without requiring more work (and admittedly, been even more harmful for the ecology...).\n- Not as a criticism but rather a suggestion, it would have been useful to summarize the table 2 by comparing the algorithms using normalized values (mean = 0, std = 1) averaged across the environments. \n- One the (strongest) weak points for me remains the assumption of the given differentiable reward function, as learning the reward function might be challenging for the model, for instance when it is sparse.\n- It is also surprising that the authors did not benchmark against nor cite (Ha, David, and Jürgen Schmidhuber. “Recurrent world models facilitate policy evolution.” NeurIPS 2018), especially since the code is open-source. \n- I would have made the same remark for (Learning Latent Dynamics for Planning from Pixels,  Hafner et al.) but the paper does not seem to be not peer-reviewed. Edit: the paper is indeed published at ICML 2019. So the remark holds. \n\nQuestion:\n- could the author elaborate on the early termination? it is not precisely defined anywhere and yet, seems to be an important point.\n\nMinor\n- page 3, parentheses around Finn et al. (2017)\n- page 3, ILQG, “is an model”", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provide an extensive set of benchmarks for Deep Model-based RL algorithms.

This paper contains a large number of algorithms, environments, and empirical results. The reviewers all recognized the need for such a study to provide some clarity, insights, and common standards. The reviewers we concerned about several aspects of the implementation of the effort. (1) All the performance is based on 4 runs, smoothed curves, and default errors (often extensively overlapping). The paper cites Henderson et al, and yet does not follow the advice laid out therein. (2) The results were fairly inconclusive---perhaps to be expected---we didn't learn much (more on this below). (3) The paper has communication issues.

The overall approach taken was a bit perplexing. Some algorithms we given access to the dynamics. The reward functions were converted to diff. forms, and early stopping in a domain specific way was employed. This all seems like simplifying the problem in different ways so that some methods can be competitive, but it is not at all clear why. If we take the typical full rl problem and limit domain knowledge, many of these approaches cannot be applied and others will fail. Those are the results we want. One could actually view these choices are unfair to more general algorithms---algorithms that need diff rewards pay no price for this assumption. This also leads to funny things, for example, like using position as the reward in mountain car (totally non-standard, and invalid without discounting). The paper claims a method can solve MC, but that is unclear from the graph. The paper motivates the entire enterprise based on the claimed lack of standardization in the literature, but then proceeds to redefine classic control tasks with little discussion or explanation. 

The paper has communication issues. For example, all the domains are use continuous actions (and the others in the response highlight that is their main focus), but this is never stated in the paper. The paper refers to and varies "environment length", but this was not defined in the paper and has no obvious meaning. The tasks are presumably discounted but the the value of gamma is not specified anywhere in the paper (could be there, but I searched for a while). Pages of parameter settings in the appendix with many not discussed or their ranges justified.

This paper is ambitious, but I urge the authors to perhaps limit the scope and do less, and consider a slightly broader audience in both the writing and experiment design.   

Decision: Reject