Exploring Model-based Planning with Policy Networks
Authors: Tingwu Wang
Jimmy Ba
Venue:  (2020)
Keywords: reinforcement learning
model-based reinforcement learning
planning

Abstract: Model-based reinforcement learning (MBRL) with model-predictive control or
online planning has shown great potential for locomotion control tasks in both
sample efficiency and asymptotic performance. Despite the successes, the existing
planning methods search from candidate sequences randomly generated in the
action space, which is inefficient in complex high-dimensional environments. In
this paper, we propose a novel MBRL algorithm, model-based policy planning
(POPLIN), that combines policy networks with online planning. More specifically,
we formulate action planning at each time-step as an optimization problem using
neural networks. We experiment with both optimization w.r.t. the action sequences
initialized from the policy network, and also online optimization directly w.r.t. the
parameters of the policy network. We show that POPLIN obtains state-of-the-art
performance in the MuJoCo benchmarking environments, being about 3x more
sample efficient than the state-of-the-art algorithms, such as PETS, TD3 and SAC.
To explain the effectiveness of our algorithm, we show that the optimization surface
in parameter space is smoother than in action space. Further more, we found the
distilled policy network can be effectively applied without the expansive model
predictive control during test time for some environments such as Cheetah. Code
is released.

Published as a conference paper at ICLR 2020
EXPLORING MODEL-BASED PLANNING WITH POLICY NETWORKS
Tingwu Wang',2 Jimmy Ba!
1 Department of Computer Science, University of Toronto Vector Institute {t ttinguwwang, jba) (cs. toronto.
ABSTRACT
Model-based reinforcement learning (MBRL) with model-predictive control or online planning has shown great potential for locomotion control tasks in both sample efficiency and asymptotic performance. Despite the successes, the existing planning methods search from candidate sequences randomly generated the action space, which is inefficient in complex high-dimensional environments. In this paper, we propose a novel MBRL algorithm, model-based policy planning (POPLIN), that combines policy networks with online planning. More specifically, we formulate action planning at each time-step as an optimization problem using neural networks. We experiment with both optimization w.r.f. the action sequences initialized from the policy network, and also online optimization directly w.r.t. the parameters of the policy network. We show that in the MuJoCo benchmarking environments, POPLIN is about 3x more sample efficient than the previously state- of-the-art algorithms, such as PETS, TD3 and SAC. To explain the effectiveness of our algorithm, we show that the optimization surface in parameter space is smoother than in action space. Further more, we found the distilled policy network can be effectively applied without the expansive model predictive control during test time for some environments such as Cheetah. Code released here
INTRODUCTION
model-based reinforcement learning (MBRL) agent learns its internal model of the world, i.c. the dynamics, from repeated interactions with the environment. With the learnt dynamics, MBRL agent can for example perform online planning, interact with imaginary data, or optimize the controller through dynamics, which provides significantly better sample efficiency (Deisenroth & Rasmussen 2011 Sutton 1990; Levine & Abbeel 2014; Levine & Koltun 2013). However, MBRL algorithms generally do not scale well with the increasing complexity of the reinforcement learning (RL) tasks in practice. And modelling errors in dynamics that accumulate with time-steps greatly limit the applications of MBRL algorithms. As result, many latest progresses in RL has been made with model-free reinforcement learning (MFRL) algorithms that are capable of solving complex tasks at the cost of large number of samples (Schulman et al.) [017 Heess et al. 2017 Schulman et al. 2015) Mnih et al. 2013 Lillicrap et al. 2015 Haarnoja et al. 2018).
With the success of deep learning, a few recent works have proposed to learn neural network-based dynamics models for MBRL. Among them, random shooting algorithms (RS), which uses model- predictive control (MPC), shown to have good robustness and scalability (Richards) 2005). In shooting algorithms, the agent randomly generates action sequences, use the dynamics to predict the future states, and choose the first action from the sequence with the best expected reward. However, RS usually has worse asymptotic performance than model-free controllers (Nagabandi et al. 2017). and the authors of the the PETS algorithm (Chua et al. 2018) suggest that the performance of RS directly affected by the quality the learnt dynamics. They propose a probabilistic ensemble to capture model uncertainty, which enables PETS algorithm to achieve both better sample efficiency and better asymptotic performance than sfate-of- --thh-rrt model-free controllers in environments such as Cheetah. However, PETS is not as effective on environments with higher dimensionality.

---

Published as a conference paper at ICLR 2020
(al) Reward Surface
(a2) PETS Iter
(a3) PETS Iter 2
(a4) PETS Iter
(b1) Reward Surface
(b2) POPLIN Iter
(b3) POPLIN Iter 2
(b4) POPLIN Iter
We transform each planned candidate action trajectory with PCA into a 2D blue scatter. The top and bottom figures are respectively the visualization of PETS (Chua et al. 2018) and our algorithm. The red area has higher reward. From left to right, we show how candidate trajectories are updated, across different planning iterations within one time-step. As we can see, while both reward surface is not smooth with respect to action trajectory. POPLIN, using policy networks, has much better search efficiency, while PETS is stuck around its initialization. The details are in section 5.3
Figure I:
In this paper, we explore MBRL algorithms from different perspective, where we treat the planning each time-step as an optimization problem. Random search action space, as what being done in state-of-the-an MBRL algorithms such as PETS, is insufficient for more complex environments. On the one hand, we are inspired by the success of AlphaGo (Silver et al.) 2016 2017). where a policy network is used to generate proposals for the Monte-Carlo tree search. On the other hand, are inspired by the recent research into understanding deep neural networks (Nguyen & Hein 2017] Li et al. 2018: [Soudry& & Hoffer 2017). Deep neural networks, frequently observed in practices, is much less likely to get stuck in sub-optimal points. In Figure we apply principal component analysis (PCA) on the action sequences generated in each planning iteration within time-step. The reward surface of the action space is not smooth and prone to local-minimas. We argue that optimization in the policy network's parameter space will more efficient. Furthermore, we note that the state-of- oftee-of-the-ar MBRL algorithm with MPC cannot be applied real-time. We therefore experiment with different policy network distillation schemes for fast control without MPC. To sum up, the contribution of this paper is three-fold:
We apply policy networks to generate proposals for MPC in high dimensional locomotion control problems with unknown dynamics.
We formulate planning as optimization with neural networks, and propose policy planning in parameter space, which obtain state-of-the-an performance current bench-marking environ- ments, being about more sample efficient than the previous state-of-the-an art algorithm, such as PETS (Chua et al. 2018), TD3 (Fujimoto et al.) 2018) and SAC (Haarooj al. 2018).
We also explore policy network distillation from the planned trajectories. We found the distilled pol- icy network alone achieves high performance on environments like Cheetah without the expansive online planning.
2 RELATED WORK
Model-based reinforcement learning (MBRL) has been long studied. Dyna (Sutton 1990: 1991
algorithm alternately performs sampling in the real environments and optimize the controllers on the learned model of the environments. Other pioneering work includes PILCO (Deisenroth & Rasmussen] 201T), where the authors model the dynamics using Gaussian Process and directly optimize the surrogate expected reward. Effective it is solve simple environments, PII.CO
2

---

Published as a conference paper at ICLR 2020
heavily suffers the curse of dimensionality. In (Levine & Abbeel 2014 Levine & Koltun 2013
Levine et al. 2016 Chebotar et al. (Zhang et al. 2018), the authors propose guided policy search (GPS). GPS uses iLQG (Li & Todorov 2004] Todorov & Li 2005: Tassa et al. 2012) the local controller, and distill the knowledge into a policy neural network. In SVG (Heess et al. 2015
the authors uses stochastic value gradient so that the stochastic policy network can be optimized by
nack-propagation with off-policy data. Recently with the progress of model-free a algorithms such
as TRPO and PPO (Schulman et al. 2013 2017), Kurutach al. (2018) Luo et al. (019) propose modern variants of Dyna, where TRPO (Schulman et al. 2015) is used to optimize the policy network using data generated by the learnt dynamics. Concurrent to this work, Janner et al. (2019) further
use SAC (Haarnoja et al. 2018) to train the policy network, and gets state-of-the-ar performance on many tasks. At the same time, random shooting methods proposed by Nagabandi et al. (2017); Chua et al. (2018) have shown its robustness and effectiveness on benchmarking environments. PETS algorithm (Chua et al. 2018) is considered by many to be the state-of- sfate-of-the-ar shooting algorithm. which we discuss in detail in section 3 Dynamics is also used to obtain better value estimation to
speed up training (Gu et al. 2016 Feinberg et al., (0188) [Buckman et al. 2018). Latent dynamics
models using VAE (Kingma & Welling, 2013) commonly used solve problems with image
input (Ha & Schmidhuber] 22018a b: Hafner et al. 2018 Kaiser et al. 2019).
3 BACKGROUND
3.1 REINFORCEMENT LEARNING
In reinforcement learning, the problem of solving the given task is formulated as infinite-horizon discounted Markov decision process. For the agent, we denote the action space and state space respectively as A and S. We also denote the reward function and transition function as r(st, az) and [8 [s,, at), where & E S and ag E A are the state and action time-step t. The reward function is assumed known to the agent in this work. The agent maximizes its expected total reward ((x) = 12 rsst, at)] with respect to the agent's controller x.
3.2 RANDOM SHOOTING ALGORITIMM AND PETS
Our proposed algorithm is based on the random shooting algorithm (Richards 2005 In random shooting algorithms (Nagabandi et al. 2017 Chua et al., 2018), data-set of D (st, ae, St+1)} collected from previously generated real trajectories. The agent learns an ensemble of neural networks denoted as (s [82, ax), with the parameters of the neural networks denoted Φb. In planning, the agent randomly generates a population of K candidate action sequences. Each action sequence, denoted a {ao,..., --.- a, }, contains the control signals every time-steps within the planning horizon т. The action sequence with the best expected reward given the current dynamics network (5 |82, a₂) is chosen. RS, first action signal and re- re-plan at time-ster step. In PETS (Chua et al. 2018), the authors further use cross entropy method (CEM) (De Boer et al. [2005] best sequences from the last CEM iteration.
model-predictive control algorithm, only executes the
Botev al. 2013) to re-samples sequences near the
MODEL-BASED POLICY PLANNING
Algorithm 1 General POPLIN Framework
1:
while Training iterations not Finished do for 7th time-step of the agent do
In this section, we deseribe two variants of POPLIN: model-based policy planning in action space (POPLIN-A) and model-based policy plan- ning in parameter space (POPLIN-P). Following the notations section 3.2 we define the expected
3:
CEM planning as in section 4. 4.2
4:
Execute the first action from CEM.
5: end for
6: Dynamics update and policy distillation.
planning reward function at time-step i as follows:
7:
end while
E+t
R(si, a.) E
rsss az)
where sc+1 ~ fo(sc+199t az).
The action sequence a; = {aa 1.+1. a is generated by the policy search module, as later The expectation of predicted trajectories {si, 8it1, tn 8 is
described in Section 4.1 and 4.2
3

---

Published as a conference paper at ICLR 2020
estimated by creating P particles from the current state. The dynamics model fo fo (s az) used by 4th particle at time-step is sampled from deterministic or probabilistic ensemble models. To better illustrate, throughout the paper we denote this dynamics as a fixed deterministic model, i.e. f = fa. In practice the dynamics uses probabilistic ensemble models, which requires some trivial modifications to the math and refer readers PETS Chua al. (2018) for details.
4.1 MADEL-BASEE POLICY PLANNING ACTION SPACE
In model-based policy planning in action space (POPLIN-A), we use a policy network to generate good initial action distribution. We denote the policy network as (({)). Once the policy network proposes sequences of actions on the expected trajectories, we add Gaussian noise to the candidate actions and use CEM to fine-tune the mean and standard deviation of the noise distribution.
Similar to defining a; {ai, di+l ..... a.+r). ). we denote the noise sequence at time-step / with horizon T as 6; = 6ú, 6i+1, 61+r). We initialize the noise distribution as Gaussian distribution with mean 140 CEM iteration, we first sort out the sequences with the top { + expected planning reward, whose noise sequences are denoted as 18th ,.,, candidates, e..
and covariance Σo oft where of is the initial noise variance. In each
J. Then we estimate the noise distribution of the elite
E' Cov((o 8, 1) n² Mean ((8), (8.......... 1,.).
The elite distribution (n', E') in CEM algorithm is used update the candidate noise distribution a>= + aL". For every time-step, several CEM iterations are
as μ (1 a)μ + ap', - performed by candidate re-sampling and noise distribution updating. We provide detailed algorithm boxes in appendix A. We consider the following two schemes to add action noise.
POPLIN-A-Init: this planning schemes, we use the policy network only to propose the initial- ization of the action sequences. When planning at time-step i with observed state we first obtain the initial reference action sequences, denoted as å, {ā,, d. d }. by running the initial forward pass with policy network. At each planning time-step 4, where i < ( < i т, we have à x(6)), where & = ff( 6, 8, The expected reward given search noise ò, will
R(s 6,) =E
Σ r(s âe + 56) where St-1 folse+1 âe 5g).
(3)
POPLIN-A-Replan: POPLIN-A-Replan is a more aggressive planning schemes, which always re-plans the controller according the changed trajectory given the current noise distribution. If we had the perfect dynamics network and the policy network, then we expect re-planning to achieve faster convergence the optimal action distribution. But increases the risk of divergent behaviors. this case, the expected reward for each trajectory is
R(s; 8,) =
Σ
r(s,, (st) + +6)
where 8t-1 fo(sc+15s, 8z).
(4)
4.2 MODEL-BASED POLICY PLANNING PARAMETER SPACE
While planning in the action space is a natural extension of the original PETS algorithm, we found provides little performance improvement in complex environments. One potential reason is that POPLIN-A still performs CEM searching in action sequence space, where the conditions of convergence for CEM is usually not met. Let's assume that robot arm needs to either go left or right to get past the obstacle in the middle. In CEM planning in the action space, the theoretic distribution mean is always going straight, which fails to model the bi-modal action distribution.
Indeed, planning in action space is non-convex optimization whose surface has lots of holes and peaks. Recently, much research progress has been made in understanding why deep neural networks much less likely to get stuck in sub-optimal points JaanntsNNuyen Nguyen & Hein (2017); Li et al. (2018) (oudryy & Hoffer (2017). And we believe that planning in parameter space is essentially using deeper neural networks. Therefore, propose model-based policy planning in parameter space (POPLIN-P).
4

---

Published as a conference paper at ICLR 2020
Instead of adding noise in the action space, POPLIN-P adds noise in the parameter space of the policy network. We denote the parameter vector of policy network as 6, and the parameter noise sequence starting from time-step as Wi {wi,
The expected reward function is now
(8;; wi) E
Σ
Toonn (8t, Tot (si)) where 8++1 T
(5)
Similarly, we update the CEM distribution towards the following elite distribution:
E' Cov((w,; -..,...ff 1) pl Mean((w.) wet 1.... w
We can force the policy network noise within the sequence to be consistent, i.c. wi+1 Wi+r: which we name as POPLIN-P-Uni. This reduces the size of the flattened noise vector from (r+ 1) to [0), and is more consistent in policy behaviors. The noise can also be separate for each time-step, which we name as POPLIN-P-Sep. We benchmark both schemes in section 5.4
Stochastic policy network encourages explo- ration, and increases the robustness against the impact compounded model errors. POPLIN-P, which inserts exogenous noise into the parameter space, can be regarded as re-parameterized stochastic policy network, which natural combines stochastic policy network with planning.
Equivalence to re-paramcterized stochastic policy:
4.3 MODEL-PREDICTIVE CONTROL AND POLICY CONTROL
MBRL with online re-p re-planning or model-predictive control (MPC) is effective, but at the same time time-consuming. Many previous attempts have tried to distill the planned trajectories into policy network Levine & Abbeel (004)); Levine & Koltun (2013): Chebotar et al. (201)); iZhang et al. (2018), and control only with policy network. In this paper, define two settings of using POPLIN: MPC Control and Policy Control. In MPC control, the agent uses policy network during the online planning and only execute the first action. In policy control, the agent directly executes the signal produced by the policy network given current observation, just like how policy network is used in MFRL algorithms. We show both performance of POPLIN in this paper.
4.4 POLICY DISTILLATION SCHEMES
The agents iterate between interacting with the environments, and distilling the knowledge from planning trajectory into policy network. We consider several policy distillation schemes here, and discuss their effectiveness in the later experimental section.
Behavior cloning (BC): BC can be applied to POPLIN-A and POPLIN-P, by minimizing the squared L2 loss as Equation| 7 D is the collection of observation and planned action from real environment. When applying BC to POPLIN-P, we fix parameter noise of the network to be zeros.
min E., E (o()) [e. - all2.
Generative adversarial network training (GAN) Goodfellow et al. (2014): GAN can be applied POPLIN-P. We consider the following fact. During MPC control, the agent only needs to cover the best action sequence in its action sequence distribution. Therefore, instead of point-to-point supervised training such as BC, we can train the policy network using GAN:
min max Ex, RDD (og(D((s, + ESED, 10 log(1 - (s (s)),
where a discriminator D parameterized by sb is used, and we sample the random noise z from the initial CEM distribution o..
Setting parameter average (AVG): AVG also applicable to POPLIN-P. During interaction with real environment, also record the optimized parameter noise in to the data-set, i. D {(ss w)). And we sacrifice the effectiveness of the policy control and only use policy network as good search initialization. The new parameter is updated 0 0 + EweD
5

---

Published as a conference paper at ICLR 2020
1050
(a) Cheetah
(b) Ant
(c) Swimmer
(d) Acrobot
POPLIN-P
POPLIN-A
SAC
METRPO
RS
TD3
PETS
PPO
Figure 2: Performance curves on different bench-marking environments. 4 random seeds are run for each environment. The full figures of all 12 MuJoCo environments are summarized in appendix
Hopper
Swimmer
Cheetah-vD
POPLIN-P (ours)
12227.9 5652.
23301 + 320.9
2055.2 + 613.8
334.4
34.2
4238,0 + 11330
597.0 478.8
POPLIN-A (osrs)
4651.1 + 1088.5
1148A + 438.3
202.5 962.5
344,9 : 7.1
1562.8 +: 1136.7
-105.0
249.8
PETTS
2018
4204.5
789.0
114.9
12.6
22884
1019.
282.5
501.6
(FETREO (Ximinen
-744.8
500.9
104.6
-1699.3
TD3 Puiindd ol 201N
218.9 543.3
870.1 +: 283.8
1816.6 + 994.8
72.1
130.9
3015.7 : 969.8
-516.4 812.2
SAC
Faurogoj 2018
1745.9 + 839.2
548.1
146.6
788.3 - 738.2
204.6 - 69.3
34598 + 1326.6
164.5
13186
Training Time-stp
50000
Reacher3D
Pusher
Pendulum
InvertedPendalum
Acrobot
Caripole
POPLIN-P (ours)
(curss)
-29.0 d: 25.2
-55.8
23.1
167.9 1: 45.9
-0.0 1:0:2:00 0.9
23.2 1: 27.2
200.8 0.3
POPLIN.A (ouns)
-27.7
24.3
178.3 19.3
0.0
20.5 20.1
200.6 1.3
PETS (Chaa
43.6
23.5
-1.44
METREO Kurutach 2018
43.5 3.7
-98.5
12.6
174.8
6.2
-29.3 29.5
.78.7 5.0
138.5 1: 63.2
TD3
Pajicton 2013
-331.6 134.6
-216.4
+9.6
168.6
+ 12.7
-102.9 101.0
-76.5 10.2
409.2 + 928.8
SAC
Taarsojj
161.66 +: 43.7
-227.6
4.222
159.5
-0.2
.694
195.5 4:
Training Time-step
50000
50000
Table I: The training time-step varies from 50,000 to 200,000 depending on the difficulty of the tasks. The performance is averaged across four random seeds with the last 3 episodes.
5 EXPERIMENTS
compare POPLIN with existing algorithms. We also show the policy control In section 5.1
performance of POPLIN with different training methods in section 552 In section 5.3 we provide explanations and analysis for the effectiivness of our proposed algorithms by exploring and visualizing the planner's s reward optimization surface. In section 5.4] we study the sensitivity of our algorithms with respect to Syyper-paramene and show the performance of different algorithm variants.
5.1 MuJoCo BENCHMARKING PERFORMANCE
In this section, we compare POPLIN with existing reinforcement learning algorithms including PETS (Chua al. 2018). GPS Levine al. 2016), RS (Richards 2005). MBMF (Nagabandi et al.) 2017). TD3 (Fujmotoo et al. 2018) METRPO (Kurutach et al.j|011, 8). PPO Schulman et al. 2017 Heess et al. 2017). TRPO (Schulman et al. 2015) and SAC (Haarnoja et al.) 2018), which includes the most recent progress of both model-free and model-based algorithms. We examine the algorithms with 12 environments, which is a wide collection of environments from OpenAI Gym (Brockman et al. 2016) and the environments proposed in PETS (Chua et al. (2018), which are summarized in appendix A.2 Due the page limit and to better visualize the results, put the complete figures and tables in appendix A.3 And in Figure 2 and Table we show the performance of our algorithms and the best performing baselines. The
search summarized in appendix [A.31]
As shown in Table I POPLIN achieves state-of th--rrr performance in almost all environments, solving most of the them with 200,000 or 50,000 time-steps, instead of million time-steps commonly used in MFRL algorithms. POPLIN-A (POPLIN-A-BC-ReC) environments such as Pendulum, Cart-pole, Swimmer. But on complex environments such as Ant, Cheetah or Hopper, POPLIN-A does not have obvious performance gain compared with PETS. POPLIN-P POOPLIN-P-Sep-VVEE on the other hand, has consistent and stable performance among different environments. POPLIN-P is significantly better than all other algorithms in complex environments such as Ant and Cheetah. However, like other model-based algorithms, POPLIN cannot solve environments such Walker and Humanoid. the performance of POPLIN plateaus
has the best performance in simpler

---

Published as a conference paper at ICLR 2020
30
-05
10
200
Genstags
(a) Cheetah
(b) Pendulum
(c) Pusher
(e) Reacher31
MPC Control Policy Control
POPLIN-A A-BC
POPLIN-E P-GAN
POPLIN P-VVG POPLIN POPLIN-P-AVG
POPLIN-P-
POPLIN-A A--LI---ABCC
- POPLIN- P-GNN
POPLIN-P PCC
Figure 3: The MPC control and policy control performance of the proposed POPLIN-A, and POPLIN- P with its three training schemes, which are namely behavior cloning (BC), generative adversarial network training (GAN) and setting parameter average (Avg).
2010
13/00
1300
2010
3000
(a) PETS
(b) POPLIN-A
(c) POPLIN-P
Figure 4: The performance of PETS, POPLIN-A, POPLIN-P using different population size of candidates Cheetah. The variance of the candidates trajectory in POPLIN-P is set to 0.1.
quickly. Gradually model-free algorithms will have better asymptotic performance. We view this as bottleneck of our algorithms and leave it to future research.
5.2 POLICY CONTROL PERFORMANCE
In this section, we show the performance of POPLIN without MPC. To be more specific, we show the performance with the Cheetah, Pendulum, Pusher and Reacher3D, as shown in Figure] and we refer readers appendix A.4 for the full results.
We note that policy control is not always successful, and in environments such as Ant and Walker2D, the performance almost random. In simple environments such as Pusher and Reacher3D, POPLIN-A has the best MPC performance, but has worse policy control performance compared with POPLIN-P- At the same time, both POPLIN-P-BC and POPLIN-P-GAN P-ANN are able to
BC and POPLIN-P-GAN efficiently distill the knowledge from planned trajectory. Which one of POPLIN-P-BC and POPLIN-P- GAN is better depends on the environment tested, and they can be used interchangeably. This indicates that POPLIN-A, which uses deterministic policy network, is more prone to distillation collapse than POPLIN-P, which can be interpreted as using a stochastic policy network with reparameterization trick. DOPLIN-P-AVEE which only use policy network optimization initialization has good MPC performance, but sacrifices the policy control performance. In general, the performance of policy control lags behind MPC control.
5.3 SEARCH EFFECTIVENESS AND REWARD SURFACE
In this section, we explore the reasons for the effectiveness of POPLIN. In Figure we show the performance of PETS, POPLIN-A and POPLIN-P with different population sizes. As we can see, PETS and POPLIN-A, which are the two algorithms that add search noise in the action space, cannot increase their performance by having bigger population size. However, POPLIN-P is able to efficiently increase performance with bigger population size. We then visualize the candidates in their reward or optimization surface in Figure I We use PCA (principal component analysis) to transform the action sequences into 2D features. As local-mining and local-maxine maxima islands. The CEM distribution of PETS algorithm is almost fixed across iterations this surface, even if there are potentially higher reward regions. POPLIN is able
see, the reward surface is not smooth, with lots of
7

---

Published as a conference paper at ICLR 2020
(a) PETS Surface
(b) PETS Population
(c) POPLIN-A (d) POPLIN-A (e) POPLIN-P (1) POPLIN-P Surface
Population
Surface
Population
Figure 5: The reward optimization surface in the solution space. The expected reward is higher from color blue to color red. We visualize candidates using different colors defined in the legend. The full results can be seen in appendix A.7
(a) POPLIN-A
(b) POPLIN-P-Avg
(c) POPLIN-P-GAN
(d) PETS
Figure 7: The ablation study of of POPLIN-A, POPLIN-P-BC, POPLIN-P-Avg. POPLIN-P-GAN.
to efficiently search through the jagged reward surface, from the low-reward center to the high reward left-down corner. To further understand why POPLIN is much better at searching through the reward surface, we then plot the figures in the solution space in Figure 5 More specifically, we now perform PCA the policy parameters for POPLIN-P. As we see in Figure 5 (c), the reward surface in parameter space is much smoother than the reward surface in action space, which are shown in
Figure 5 (a), (b). POPLIN-P efficiently search through the smoother reward surface in parameter space.
PETS
POPLIN-A
In Figure 6 we also visualize the actions distribution in one episode taken by PETS, POPLIN-A and POPLIN- P using policy networks of different number of hidden layers. We again use PCA to project the actions into 2D feature space. As we can see, POPLIN-P shows a clear pattern of being more multi- with the use of deeper the network.
ams
9.24
-201.
0.13
POPLIN-P OLayer
POPLIN-P
9.3
100_
8.
5.4 ABLATION STUDY
Figure 6: Projected action distribution.
In this section, we study how sensitive our algorithms with respect to some of the crucial hyper- parameters, for example, the initial variance of the CEM noise distribution. We also show the performance of different algorithm variants. The full ablation study and performance against different random seeds are included in appendix AA.5) In F we show the performance of POPLIN-A using different training schemes. We try both training with only the real data samples, which we denote as "Real", and training also with imaginary data the agent plans into the future, which we denote "Hallucination". In practice, POPLIN-A-In performs better than POPLIN-A AO-planA-Reelan. which suggests that there can be divergent or overconfident update in POPLIN-A-Replan And training with or without imaginary does not have big impact on the performance. In Figure7((b) and (c), we also compare the performance of POPLIN-PA Pnni with POPLIN-P-Sep, where we show that POPLIN-P-SS PPPpp has much better performance than POPLIN-P CELLIN---Uni indicating the search is not efficient enough in the constrained parameter space. For POPLIN-P-AN of the noise distribution, the agent gets better at planning. However, increasing initial noise variance does not increase the performance of PETS algorithm, as shown in (b). (d). It is worth mentioning that ROPLIN-P-GAN is highly sensitive to the entropy penalty we add to the discriminator, with the 3 curves in Figur (c) using entropy penalty of 0.003, 0.001 and 0.0001 respectively.
with bigger initial variance
8

---

Published as a conference paper at ICLR 2020
CONCLUSIONS
In this paper, we explore efficient ways to combine policy networks with model-based planning. We propose POPLIN, which obtains state-of-the-ar performance the MuJoCo benchmarking environments. We study different distillation schemes to provide fast controllers during testing. More importantly, we formulate online planning as optimization using deep neural networks. We believe POPLIN will scale to more complex environments in the future.
REFERENCES
Zdravko I Botev, Dirk P Kroese, Reuven Y Rubinstein, and Pierre L' L'Ecuyer. The cross-entropy method for optimization. In Handbook of statistics, volume 31. pp. 35-59. Elsevier, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint orXiv:/606.01540 2016.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances Neural Information Processing Systems, pp. 8224-8234 2018.
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal. and Sergey Levine. Combining model-basee based and model-free updates for trajectory-centric reinforcement learning. In Proceedings of the 34th International Conference Machine Learning- Volle---ol 70, pp. 703-711. JMLR. org. 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. arXiv preprint arXiv: 1805. 12/ 2018.
Pieter- Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference machine learning (ICML-II), pp. 465-472, 2011.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning, arXiv preprint arXiv:1803.00101, 2018
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.09477, 2018.
Ian Goodfellow, Jean Pouget-Abadie. Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014.
Shixiang Gu, Timothy Lillicrap. Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning, pp. 2829-28323222 2016.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2450-2462, 201 8a.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: /803. /0122, 2018b. 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-polic maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint S.Vv:1801.01290 2018.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint 18Xiv: 1,0451,, 2018.

---

Published as a conference paper at ICLR 2020
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 1944-29522 2015.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence locomotion behaviours in rich environments. arXiv preprint arXiv:1707.0222 02286,, 2017.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. arXiv preprint Xrv: 08253, 2019.
Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. arXiv preprint arXiv: arxivv:9033 aeee://03000440 2019.
Diederik P Kingma and Max Welling. 1rXiv:1312.6114, 2013.
Auto-encoding variational bayes. arXiv preprint
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint 1802. 2018.
Sergey Levine and Picter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071-1079, 2014.
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference Machine Learning, pp. 1-9, 2013.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1333 2016.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, pp. 6389-6399, 2018.
Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In ICINCO (I), pp. 222-2299, 2004.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 10X.001009.0271,, 2015.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. ICLR, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Anusha Nagabandi, Gregory Kahn, Ronald Fearing, and Sergey Levine. Neural network dynam- ics for model-based deep reinforcement learning with model-free fine-tuning. arXiv preprint arXiv:1708.00 02566, 2017.
Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In Proceedings of the 34th International Conference on Machine Learning- Vollme 70, pp. 6603-2612. JMLR. org, 2017.
Arthur George Richards. Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp.
2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1889-1897, 2015.
10

---

Published as a conference paper at ICLR 2020
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiesser, Ioannis Antonogiou, Veda Panneershelvam,, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354339, 2017.
Daniel Soudry and Elad Hoffer. Exponentially vanishing sub-optimal local minima in multilayer neural networks. arXiv preprint arXiv: 2017.
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine Learning Proceedings 1990, pp. 216-224. Elsevier, 1990.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 204):160-163, 1991.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In Intelligent Robots and Systems (IROS), 2012 JEEE/RSJ International Conference on, pp. 4906-4913. IEEE, 2012.
Emanuel Todorov and Weiwei Li. A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005., pp. 300-300 IEEE, 2005.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference pp. 5026- 5033. IEEE, 2012.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured latent representations for model-based reinforcement learning. arXiv preprint 100X::18008.0105, 2018

---

Initialize policy network parameters dynamics network parameters Φ data-set Published as a conference paper at ICLR 2020
A
APPENDIX
A.1
ALGORITHM DIAGRAMS
To better illustrate the algorithm variants of our proposed methods, we summarize them in Algo-
rithm 2 3
Algorithm 2 POPLIN-A-Init
Initialize policy network parameters dynamics network parameters o, data-set
2: while Training iterations not Finished do for 8th time-step of the agent do
3:
Sampling Data
4: Initialize reference action sequence {ā,, di+t:..., 0i+r).
Using Equation
5:
Initialize action-sequence noise distribution. - 140,
of
6:
for jah CEM Update do
CEM Planning
7:
Sample action noise sequences {ō.} from (oomN((H E). for Every candidate 8; do
8:
Trajectory Predicting
9:
for t i to i + т, 81+1 = -o (94+1/88 a₁ = àt + 56)
10:
Evaluate expected reward of this candidate.
11:
end for
Fit distribution of the elite candidates n',''. Update noise distribution = (1 a)ı + aq', E = aZ
13:
end for
15:
Execute the first action from the optimal candidate action sequence.
end for
17:
Update & using data-set 'D
Dynamics Update >> Policy Distillation
18:
Update using data-set D
19:
end while
Algorithm 3 POPLIN-A-Rephe
2: while Training iterations not Finished do for 2th time-step of the agent do
3:
Sampling Data
4: Initialize action-sequence noise distribution. μoo 2 o
5:
for jth CEM Update do
CEM Planning
6:
Sample action noise sequences {ō,} from from.N(44 (N,, E).
7:
for Every candidate 5, do
Trajectory Predicting
8:
T, 8 = fa (81+1 1 (8t, ar (st) + Evaluate expected reward of this candidate. end for
for =
9:
11:
Fit distribution of the clite candidates as μ', E' Update noise distribution μ a)μ + ap', Σ (1 a)2
13:
end for
Execute the first action from the optimal candidate action sequence. end for
15:
16:
Update á using data-set
Dynamics Update Policy Distillation
17:
Update using data-set D
18:
end while
A.2 BENCH-MARKING ENVIRONMENTS
In the original PETS paper Chua et al. (2018), the authors only experiment with 4 environments, which are namely Reacher3D, Pusher, Cartpole and Cheetah. In this paper, we experiment with the more environments based on the standard bench-marking environments from OpenAI Gym Brockman et al. (2016). More specifically, we experiment with InvertedPendulum, Acrobot, Pendulum, Ant. Hopper, Swimmer, Walker2d. We also note that the Cheetah environment in PETS Chua al. (2018) different from the standard HalfCheetah-vl in OpenAl Gym. Therefore we experiment with both versions in paper, where the Cheetah from PETS is named "Cheetah" and the HalfCHeetah
12

---

Published as a conference paper at ICLR 2020
Algorithm 4 POPLIN-P
Initialize policy network parameters 6, dynamics network parameters o, data-set while Training iterations not Finished do for 8thh time-step of the agent do
2:
3:
Sampling Data
4: Initialize parameter-sequence noise distribution, Muu, E o¹
5: for jmh CEM Update do
CEM Planning
6:
Sample parameter noise sequences {w;} from (u, E). for Every candidate w, do
7:
Trajectory Predicting
8:
for = i to i T, St+1 = fo (s, aq -ooww (s:)) Evaluate expected reward this candidate. end for Fit distribution of the elite candidates as n/, E'. Update noise distribution μ (1 a)ı + ap', E =0 (1-
9:
10:
11:
13:
end for
14:
Execute the first action from the optimal candidate action sequence.
15:
end for
Update e using data-set
Dynamics Update Policy Distillation
17:
Update 0 using data-set 'D
end while
Cheetah
Ant
Hopper
Swimmer
Chestah-vO
Walker2d
Swimmer-v0
POPLIN-P
12227.9
5652.8
4235.0
597.0 478.8
POPLIN-A
4681.1
1088.5
1148.4 1. 438.3
202.5
962.5
344.9 1: 7.1
1562.8
1136.7
-105.0
249.8
26.7 13.2
PETS
4204.5
789.0
1165,5 :: 226.9
114.9 1.4.91:221.1 621.
326.2
12.6
2288.4 = 1019.0
282.5 4: 501.6
29.7 13.5
191.1
535.5
35.1
22.4
421.0 55.2
MBMF
-55.5 1: 62.5
134.2 1: 50.4
-104.44 1: 1098.7
110.7 4: 45.6
126.9 1.6.9.177 72.7
-2188 = 437.7
30.6 = 4,9
TRPO
412.4 + 33.3
323.3 24.9
-2100.1
640.6
47.8
11.1
-120 85.5
-2286.3 + 373.3
26.3 + 2.6
PPO
483.0 46.1
51.2
-103.8 + 1028.0
155.5
14.9
17.2
-1893.6 234.1
24.7
GPS
129.4 + 140.4
445.5 + +122.9
-768.5 + 200.9
-30.9 + 6,3
52.3 + 41.7
-1730. + 441.7
8.2 10.2
METRPO
707.1
+
104.6
2283.7
-1609.3
35.4
TD3
218.9 + 593.3
870.1
283.8
1816.6 + 994
72.1 + 130.9
3015.7
969.8
-5164 812.2
17.0 12.9
SAC
1745.9 839.2
548.1
146.6
788.3 738.2
204.6 69.3
3459.8
1326.6
164.5
1318.6
23.0 17.3
Random
-28422 + 83.3
478.0 47.8
-2768.0
571.6
-12.4 12.8
-312.4 + 44.2
-2450.1 + 406.5
Time-step
50000
200000
200000
50000
200000
200000
200000
Table 2: Performance of each algorithm environments based on OpenAl Gym Brockman et al. (016) MuJoCçTodorov et al. (2012) environments. In the table, we record the performance at 200,000 time-step.
Cheetah
Ant
Hopper
Swimmer
Cheetah-v0
Walker2d
Swimmer-v0 v0
POPLIN-P
0.944 : 0.079
0.932 : 0.128
0.919 1 0.112
0.936 : 0.086
0.927 = 0.227
0.968 0.15
0.928 + 0.115
POPLIN-A
0.395 0.057
0.177
0.582
0,962
0.078
0.33
PETS
0.363 : 0.002
0.091 0.466
0.566 : 0.113
0.916 : 0.032
0,538 0.204
0.87 = 0.157
0.743 = 0,338
RS
0.072 1 0.005
0.214
0.015
0.092 1 0.006
0.156 1 0.024
0.164 0.011
0.137 1 0.071
0.67 0.058
MBMF
0.025 + 0.002
0.054 + 0.02
0.355 + 0.2
0,377 + 0.114
0.105 0.015
0.088 + 0.137
0.765 0.123
TRPO
0.028 + 0.003
0.129 0.01
0.164
0.22:00.025
0.078 = 0.017
0.067 0.117
0.658 0.065
PPO
0.023 0.01
0.128 0.02
0.527
0.187
0.489 0.037
0.083
0.073
0.618 0.1
GPS
0.067 1 0.051
0,178 1 0.085
0.406 1: 0.037
0.023 1 0.016
0.09 1 0.008
0.278 + 0.138
0,205 = 0.255
METRPO
0.004 - 0.043
0.113 + 0.007
0.777
0.664 0.262
0.537 : 0.18
0.885 0.055
TD3
0.074 1: 0.061
0,348
0.114
0,876 0.181
0.28 0.327
0.683 0.194
0.62 :: 0.254
0.425 = 0,323
SAC
0.184 1 0.006
0.219
0.059
0.689 1 0.134
0.612 1 0.173
0.772 0.265
0.833 1: 0.412
0.575 0.433
Random
0.037 +
0.191 0.019
0.042
0.104
0.069 0.032
0.018 0.009
0.016 + 0.127
0.06 0.3
max, min
13000, 880
2500, 0
2500, 3000
360, .40
400, 4600
700, -5000
40, 0
Table 3: The normalized performance of Table 2
from OpenAl Gym is named as "Cheetah-vo". Empirically, Cheetah is much easier to solve than Cheetab-vO, as show in Table 2 and Table 4 We also include two swimmer, which we name as Swimmer and Swimmer-v0, which we explain in section A.2.1
13

---

Published as a conference paper at ICLR 2020
(a) Ant
(b) Acrobot
(c) Cartpole
LECTIRN
PELSO
(d) Swimmer
(e) Cheetah
(f) Hopper
gg) InvertedPendulum
(h) Swimmer-vo
(i) Reacher3D
1000
150
-000
12:00 1500 11h6
11000
110M
Walker2d
(k) Pusher
(I) Cheetah-v0
Figure 8: Full Performance of POPLIN-P, POPLIN-A and other sfate-of-the-ann art algorithms on 12 different bench-marking environments. In the figure, we include baselines such TD3, SAC, PPO, METRPO, PETS, RS and our proposed algorithm.
A.2.1 FIXING THE SWIMMER ENVIRONMENTS
We also notice that after update in the Gym environments, the swimmer became unsolvable for almost all algorithms. The reward threshold for solving around 340 for the original swimmer, but almost all algorithms, including the results shown in many published papers Schulman al. (2017). will stuck at the 130 reward local-minima. We note that this is due the fact that the velocity sensor the neck of the swimmer, making swimmer extremely prone to this performance Docal-miniman We provide a fixed swimmer, which we name as Swimmer, by moving the sensor from the neck to the head. We believe this modification is necessary to test the effectiveness of the algorithms.
14

---

Published as a conference paper at ICLR 2020
Reacher3D
Pusher
Pendulum
InvertedPendulum
Acrobot
Caripole
POPLIN-P
-29.0 = 25.2
25.2
-55.8 +: 23.1
167.9 45.9
+0.0
23.2 + 27.2
200.8 = 0.3
POPLIN-A
-27.7 = 25.2
-56.0 + 24.3
178.3 = 19.3
-0.0 0.0
20.5 + 20.1
200.6 = 1.3
PETS
-47.7 43.66
-52.7 1: 23.5
155.7 1: 79.3
377.8
-18.4 46.3
199.6 4.6
RS
-12
-146.4 1:
161.2
-..0
-12.5 14.3
201.0 0.0
MBMF
23.
15.2 -285.8
163.7 15.2
17.0
-146.8 29.9
22.5
TRPO
-176.5 24.3
-235.5
158.7 9.1
6.9
-291.2
46.3 6.0
PPO
-162.2 15.7
2.9
160.9 = 12.5
12.4
-205.4
68.8 +
GPS
-552.8 577.7
1.3
164.3
20.7
-114.3
-18.7 = 101.1
METRPO
-3..5
-98.5 12.6
174.8
-
-78.7 5.0
138.5 63.2
TD3
-33166
134.6
216.44
168.6 12.7
-102.9 101.0
-76.5 10.2
4009 928.8
SAC
-161.6 43.7
-227.6
42.2
159.5 12.1
-.22
-.9.4
195.5 8.7
Random
-183.1 41.5
-199.0 10.0
-249.5 = 228.4
205.99 12.1
-374.1 = 15.6
31.3 = 36.3
Time-step step
50000
50000
50000
50000
50000
50000
Table Performance of each algorithm on environments based on OpenAl Gym Brockman al.
(2016) classic control environments. In the table, we record the performance at 50000 time-step.
A.3 FULL RESULTS OF BENCH-MARKING PERFORMANCE
In this section, we show the figures of all the environments Figure 8 We also include the final see, POPLIN has consistently the best performance
performance in the Table 2 and 4 As among almost all the environments. We also include the time-steps we use on each environment for all the algorithms in Table and
A.3.1 PARERR-ARAMETERS
In this section, introduce the hyper-parameters search during the experiments. One thing to notice is that, for all of the experiments on PETS, POPLIN, we use the model type PE (probabilistic ensembles) and propagation method of (expectation). While other combinations of model type and propagation methods might result in better performance, they are usually prohibitively computation- ally expensive. For example. the combination of PE-DS requires training time of about 68 hours for one random seed, for PETS to train with 200 iteration, which is 200,000 time-step. As a matter of fact, PE-E actually one of the best combination many environments. Since POPLIN is based on PETS, we believe this is fair comparison for all the algorithms.
show the hyper-parameter search we perform for PETS in the paper Table For the hyper- parameters specific to POPLIN, we summarize them in 6 and
Hyper-parameter
Value Tried
Population Size
100, 200, .... 2000
Planning Horizon
30, 50, 100
Initial Distribution Sigma
0.01, 0.03, 0.1, 0.25, 0.3, 0.5
CEM Iterations
5, 8, 10, 20
ELite Size {
50, 100, 200
Table 5: Hyper-parameter grid search options for PETS.
Hyper-parameter parameter
Value Tried
Training Data
real data, hallucination data Replan, Init 0.001, 0.003, 0.01, 0.03, 0.1
Variant
Initial Distribution Sigma
Table Hyper-parameter grid search options for POPLIN-A.
15

---

Published as a conference paper at ICLR 2020
Hyper-parameter
Value Tried
Training Data
real data, hallucination data BC, GAN, Avg Uni, Sep
Training Variant
Noise Variant
Initial Distribution Sigma
0.001, 0.003, 0.01, 0.03, 0.1
Table 7: Hyper-parameter grid search options for POPLIN-P. We also experiment with using WGAN in Salimans al. (2016) train the policy network, which does results in good performance and is not put into the article.
A.4 FULL RESULTS OF POLICY CONTROL
Due to the space limit, we are not able to put of the results of policy control in the main article. More specifically, main article, as can be seen in 9 (b). Again, we note that POPLIN-P-BC and POPLIN-P-GIE comparable to each other, mentioned in the main article. POPLIN-P-BH and TOPLIN-P-GAN are the better algorithms respectively in Cheetah and Cheetah-vO, which are essentially the same environment with different observation functions.
add the figure for the original Cheetah-vO compared to the figures shown in the
are
3000
POPLESIARE
ROPEINENANE
PLRSES Avg
PUPLES PREAA
Kibry werd
1900
PEPE MU
PHALIN Picdh--e
A
300
Falky Ciaral
1003
301
Gkanrage
Tearraee
(a) Cheetah
(b) Chectah-v0
(c) Pendulum
-30
INPLESSADO
PNPLESGAA
POPLESWII
SINI CLEAEE
(e) Pusher
(f) Reacher3D
Figure 9: The planning performance and the testing performance the proposed POPLIN-A, and POPLIN-P with its three training schemes, which are namely behavior cloning (BC), generative adversarial network training (GAN) and setting parameter average (Avg).
A.5 ABLATION STUDY FOR DIFFERENT VARIANT OF POPLIN
In this section, we show the results different variant our algorithm. In Figure [] the performances of different random seeds are visualized, where show that POPLIN has similar randomness in performance to PETS. Additionally, we visualize POPLIN-P-BC in Figure I0 (b), whose best distribution variance for policy planning is 0.01, while the best setting for testing 0.03.
A.6 POPULATION SIZE
In Figure 12 include more detailed figures of the performance of different algorithms with different population size. One interesting finding is that even with fixed parameters of zeros, POPLIN- P can still performance very efficient search. This is indicating that the efficiency in optimization of
16

---

Published as a conference paper at ICLR 2020
Replie Sod Rogla Quak
1400
Styripart
Du Dar.
LPF Dicted
Tea
1900
Driston ud The
Cenne
MPC Ocitud
Sop Repiguabl
Fally Cantall
100
10.0
1000
2016
2010
2000
20
2000
3000
2014
3400
(a) POPLIN-A
(b) POPLIN-P-BC
(c) POPLIN-P-ANg
100
Dn. Rererd
200
Ouated
TruerS
Folce
-130
(d) PETS
(d) POPLIN-P-GAN
Figure 10: The performance of POPLIN-A, POPLIN-P-BC, POPLIN-P-AYY POPLIN-P-GAN using
different hyper-parameters. The tested environment is Cheetah.
6000
olgocitiam
POPLIN-P P--.---add-113
5000
POPLIN-P-ae 2-40
POPLIN- 3156
4000
POPLIX.P-ood--
POPLIN-A-weed A-1-NN----eed11- 1254
3000
POPLIN-Assed
POPLIN.A-weede 5-1
POPLIN-A A-a-EN---- 64666
2000
PR1S-aed-11331
7ETS-aad 2344
1000
9K118-
seedS 4107
1000
10000
30000
30000
00000
500000
timestops
(c) Random seeds
Figure 11: The performance of POPLIN-A, POPLIN-P, and PETS of different random seeds Cheetah environment.
POPLIN-P, especially of POPLIN-P-AVG, is the key reasons for successful planning. However, this scheme naturally sacrifices the policy distillation and thus cannot be applied without planning.
A.7 THE REWARD SURFACE OF DIFFERENT ALGORITIIM
In this section, we provide a more detailed description of the reward surface with respect the the solution space (action space for PETS and POPLIN-A, and parameter space for POPLIN-P) in Figure [3] 14
16 17 As we can see, variants of POPLIN-A are hetter at searching, but the reward surface is still not smooth. POPLIN-A-Replan is more efficient in searching than POPLIN-A-IE Init, but the errors in dynamics limit its performance. We also include the results for POPLIN-P using a 1-layer neural network in solution space in Figure 16 (g), (h). The results indicate that the deeper the network, the better the search efficiency.
We also provide more detailed version of Figure in Figure [8] We respectively show the surface for PETS, POPLIN-P-P using and 0 hidden layers. Their planned trajectories across different CEM updates are visualized in Figure T9 20 21 Originally in Figure we use the trajectories in iteration
17

---

Published as a conference paper at ICLR 2020
Parantia
Dgr
1328
308
208
20M
558
CEM
(b) POPLIN-A
(c) PVPG
Pepaoo
Frydoo
1000
2011
1000B
200,
2000
C2RO
2000
(d) POPLIN-P-BC
(e) POPLIN-P-ZeroWWegh
Figure 12: The performance of PETS, POPLIN-A, POPLIN-P-Avg, POPLIN-P-BC and POPLIN- whose network has fixed parameters of zeros. The variance the candidates trajectory a in POPLIN-P is set to 0.1. The tested environment is Cheetah.
Iteration
Iter-1
Iter-2
Iter-3
Iter-4
Iter-5
(a) PETS reward surface
(b) PETS candidates
Figure 13: Reward surface in solution space (action space) for PETS algorithm.
1. 3, 5 for better illustration. In the appendix, we also provide all the iteration data. Again, the color indicates the expected cost (negative expected reward), From left right, we show the updated the trajectories in each iteration with blue scatters.
18

---

Published as a conference paper at ICLR 2020
Iteration
Iter-1
Iter-2
Iter-3
Iter-4
Iter-5
(c) POPLIN-A AReplln reward surface
(d) POPLIN-A- BEILIIN-A-Rplie A-Rellnn candidates
Figure 14: Reward surface in solution space (action space) for POPL.N-A-Repi
Iteration
Iter-1
Iter-2
Iter-3
Iter-4
Iter-5
(e) POPLIN-A-Init reward surface
POPLIN-A-Init candidates
Figure 15: Reward surface in solution space (action space) for POPLIN-A-Iie
Iteration
Iter-1
Iter-2
Iter-3
Iter-4
Iter-5
gg)
HUPLIN-P-Layerl reward surface
(h) POPLIN-P-Layer)) candidates
Figure 16: Reward surface in solution space (parameter space) for POPLIN-P with 0 hidden layer.
19

---

Published as a conference paper at ICLR 2020
Iteration
Iter-1
Iter-2
Iter-3
Iter-4
Iter-5
Sonkia
000
(i) POPLIN-P reward surface
POPLIN-P candidates
Figure 17: Reward surface in solution space (parameter space) for POPLIN-P using hidden layer.
(a) PETS Surface
(b) POLINA Surface
(c) POLINA-LO Surface
Figure 18: The color indicates the expected cost (negative of expected reward). We emphasis that all these figures are visualized the action space. And all of them are very unsmooth. For the figures visualized in solution space, we refer to Figure 13
(al) PETS Iteration
(a2) PETS Iteration
(a3) PETS Iteration
(a4) PETS Iteration 4
(a5) PETS Iteration 5
Figure 19: The figures are the planned trajectories of PETS.
(b1) POPLIN Iteration
(b2) POPLIN Iteration
(b3) POPLIN Iteration 3
(b4) POPLIN Iteration 4
(b5) POPLIN Iteration 5
Figure 20: The figures are the planned trajectories of POPLIN-P using hidden layer MLP.
20

---

Published as a conference paper at ICLR 2020
(b2) POPLIN-LO (b2) POPLIN-LO
(b3) POPLIN-LO Iteration 3
(b4) POPLIN-LO Iteration 4
(b4) POPLIN-LO Iteration 5
Iteration
Iteration
Figure 21: The figures are the planned trajectories of POPLIN-P using hidden layer MLP.
21


Reviews:
[{"review_id": "SJeV9O1dYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions\n\nThis paper proposes a MBRL algorithm for continuous action space domains that relies on a policy network to propose an initial trajectory, which is then refined, either in the action space or in the policy space, using standard CEM.\n3 options are evaluated to update the policy network: a GAN approach, a regression approach (behaviour cloning) and a direct modifications of the policy based on the perturbation found by CEM.\n\n\nReview\n\nThe presentation of the method is thorough, as well as the motivations and the experiments. However, the presentation of the experimental result could gain in clarity. For example, plots in Figure 7 are totally unreadable when printed on paper, and even after zooming in the pdf it's difficult to tell what's going on. I think the text doesn't even mention on which environment these curves are obtained.\nI'd suggest some normalized performance ratio over the final performance of the base algorithm, to show the contribution of all the ingredients of the method.\n\nFor the results presented in section 5.1, it's unclear to me which exact variation of POPLIN-A and POPLIN-P have been selected.\n\n\nFinally the abstract reads \"We show that POPLIN obtains state-of-the-art performance in the MuJoCo benchmarking environments\" (similar claims are made elsewhere in the paper). I'd tone this down a little, since the said performance ist state-of-the-art only for the 200k frames regime, and as it's hinted at the end of 5.1, I assume the proposed approach suffers from the same shortcoming as other competing MBRL methods which is that their performance tends to plateau after a few hundred thousand samples. Overall, the SOTA performance in MuJoCo is held by MFRL methods, but typically require more training (up to 1M)\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hkxyvbc6Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents POPLIN, a novel model-based reinforcement learning algorithm, which trains a policy network to improve model-prediction control. The paper studies extensively how to utilize the policy, by planning in action space or planning in parameter space and how to train the policy, by behavioral cloning, by GAN or by averaging the results of CEM. The experiments show that the proposed algorithm can perform very well in MuJoCo tasks. \n\nOverall, the paper is well-written and the method is novel. The extensive experiments make the paper more convincing. \n\nQuestions:\n1. In the example of arm in the first paragraph of Section 4.2, although the mean is 0, the randomness in sampling will be the tie breaker. So \"failing\" is probably not the best word here. \n2. It seems that the policy network is deterministic. Why?\n3. Reparametrizable policy often requires fewer (noise) parameters to be optimized over. For example, suppose the policy outputs a multi-variate Gaussian distribution with diagonal covariance in R^10, then we only need to optimize over 10 parameters (the Gaussian noise).  Why optimize all parameters in the policy network, which makes optimization more difficult? \n4. Sec 5.1 Para 2: To my knowledge, the state-of-the-art model-based RL algorithm in MuJoCo environments is MBPO (https://arxiv.org/abs/1906.08253, NeurIPS 2019). \n5. What's the architecture of policy network? More importantly, how many parameters does the policy network have? It's really interesting to see that CEM works for such a high dimensional space. In an environment where a larger network is required, the optimization seems to be more difficult. \n6. In Ablation Study, what does \"imaginary data\" mean? \n7. I'm also curious to see how the objective of CEM improves. \n\nMinor comments:\n\n1. Sec 5.1 Para 2 L11: efficient -> efficiently. \n2. Are you talking about Figure 3 at Sec 5.2? If so, could you please add a link to the figure? \n3. A lot of default values need to be specified: What's the policy distillation method used in POPLIN-A/P in Table 1? Does POPLIN-A mean POPLIN-A-Replan or POPLIN-A-init? Does POPLIN-P mean POPLIN-P-Sep or POPLIN-P-Uni? \n4. Sec 4.1 Eq (2): \\delta_0...\\delta_\\xi are \\xi+1 sequences. \n5. Sec 5.3 Para 3: multi-model -> multi-modal. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1e3tFvstH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis work provides a novel model-based reinforcement learning algorithm for continuous domains (Mujoco) dubbed POPLIN. The presented algorithm is similar in vein to the state-of-the-art PETS algorithm, a planning algorithm that uses state-unconditioned action proposal distributions to identify good action sequences with CEM in the planning routine. The important difference compared to PETS is the incorporation of a parametric state-conditioned policy (trained on real data) in the planning routine to obtain better action-sequences (CEM is used to learn the \"offset\" from the parametric policy). The paper presents two different algorithmic ablations where CEM either operates in action space or parameter space (POPLIN-A and POPLIN-P respectively), in combination with different objectives to learn the parametric policy. The method is evaluated on 12 continuous benchmarks and compared against state-of-the-art model-based and model-free algorithms, indicating dominance of the newly proposed method.\n\nQuality\n\nThe quality of the paper is high. This is an experimental study and the number of benchmarks and baselines is far above average compared to other papers in that field. One minor point is that averaging experiments over 4 seeds only is usually not optimal in these environments, but in light of the sheer amount of baselines and benchmarks excusable. While the experimental results are impressive, the authors mention that asymptotic performance in Walker2d and Humanoid might not match the asymptotic performance of model-free baselines. This could be stated more clearly. Also, there are no Humanoid experiments in the paper despite mentioned in the text (2nd paragraph in Section 5.1)?\n\nClarity\n\nThe clarity of the paper can be in parts improved upon. For example, how does the \"policy control\" ablation (mentioned in Section 4.3) work precisely, i.e. the interplay between executing the parametric policy in the real world and harnessing the environment model? I assume the policy distillation techniques in Section 4.4 are different alternatives for the second-to-last lines in the pseudocodes from the appendix? Which one is the default used in Section 5.1? On a minor note, above Equation (7), a target network is mentioned---where does the target network occur in Equation (7)? There are some plots that do not mention the name of the environment, e.g. in Figure (4), but also some in the appendix. Furthermore, it could be stated more clearly that the reward function is assumed to be known. If the authors improve the clarity of their paper significantly, I am willing to increase my score further (presupposing that no severe issues arise in the discussion phase).\n\nOriginality\n\nAdding a parametric policy to PETS is not the most original idea, but clearly a gap in the current literature.\n\nSignificance\n\nThe experiments and the empirical results make the paper quite significant.\n\nUpdate\n\nThe authors improved the clarity of the paper. I therefore increase to 8. Section 4.4 paragraph \"Setting parameter average (AVG)\" can still be improved---does this go together with POPLIN-P-Uni from Section 4.2?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model-based policy optimization approach that uses both a policy and model to plan online at test time. The paper includes significant contributions and strong results in comparison to a number of prior works, and is quite relevant to the ICLR community. There are a couple of related works that are missing [1,2] that combine learned policies and learned models, but generally the discussion of prior work is thorough. Overall, the paper is clearly above the bar for acceptance.

[1] https://arxiv.org/pdf/1703.04070.pdf
[2] https://arxiv.org/pdf/1904.05538.pdf
Decision: Accept (Poster)