Evaluating Model-Based Planning and Planner Amortization for Continuous Control
Authors: Arunkumar Byravan
Leonard Hasenclever
Piotr Trochim
Mehdi Mirza
Alessandro Davide Ialongo
Yuval Tassa
Jost Tobias Springenberg
Abbas Abdolmaleki
Nicolas Heess
Josh Merel
Martin Riedmiller
Venue: ICLR 2022 Poster (2022)
Keywords: Model-based Reinforcement Learning
Planning
Robotics
Model Predictive Control
Learning

Abstract: There is a widespread intuition that model-based control methods should be able to surpass the data efficiency of model-free approaches. In this paper we attempt to evaluate this intuition on various challenging locomotion tasks. We take a hybrid approach, combining model predictive control (MPC) with a learned model and model-free policy learning; the learned policy serves as a proposal for MPC. We show that MPC with learned proposals and models (trained on the fly or transferred from related tasks) can significantly improve performance and data efficiency with respect to model-free methods. However, we find that well-tuned model-free agents are strong baselines even for high DoF control problems. Finally, we show that it is possible to distil a model-based planner into a policy that amortizes the planning computation without any loss of performance.

Published as a conference paper at ICLR 2022
EVALUATING MODEL-BASEL PLANNING AND PLAN- NER AMORTIZATION FOR CONTINUOUS CONTROL
Arunkumar Byravan*
Leonard Hasenclever*
Piotr Trochim"
Mehdi Mirza'
Alessandro Davide lalongo
Yuval Tassa"
Jost Tobias Springenberg'
Abbas Abdolmaleki"
Nicolas Heess*
*
Josh Merel
Martin Riedmiller
ABSTRACT
There is widespread intuition that model-based control methods should be able surpass the data efficiency of model-free approaches. In this paper we attempt to evaluate this intuition on various challenging locomotion tasks. We take a hybrid approach, combining model predictive control (MPC) with a learned model and model-free policy learning: learned policy serves as a proposal for MPC. We find that well-tuned model-free agents are strong baselines even for high DoF control problems but MPC with learned proposal distributions and models (both trained on the fly or transferred from related tasks) can significantly improve performance and data efficiency in hard poultittask/multiggoon settings. Finally, we show that it is possible to distil a model-based planner into policy that amortizes the planning computation without any loss of performance. Videos of agents performing different tasks can be seen our website
INTRODUCTION
In recent years, model-free RL algorithms have improved and scaled to the point where it is feasible to learn adaptive behavior for high- high-dimensional systems in diverse settings (Schulman et al. 2017) Heess et al. (017). Despite these improvements, there is widespread intuition that model based methods can further improve data efficiency. This has led to recent advances that demonstrate improved learning efficiency by leveraging model learning during policy training (Lowrey et al. 2019 Nagabandi et al. 20200 Hubert et al. [021]. However, there is also work which urges moderation in interpreting these results, by showing that well-tuned model-free baselines can compare favorably against some model-based approaches (Springenberg et al. 2020 van Hasselt et al., 2019).
In this paper, we focus on model-based RL with learned models and proposals ! We use model- predictive control (MPC) to improve the quality of behavior generated from a policy that serves as proposal for the planning procedure. When acting, the MPC-hased search procedure improves the quality of the data collected which serves sort of "active exploration". This data can then be used to improve the proposal, in effect consolidating the gains. This hybrid approach interpolates between model-free RL, on one side, and trajectory optimization with model on the other.
Fundamentally, the spectrum on which this hybrid approach situated reflects a trade-off in terms of reusability / generality versus compute cost at deployment time. Policies obtained by model-free RL often generalize poorly outside the situations they have been trained for, but are efficient to execute (i.c., they fully amortized). Models offer potentially greater generalization, insofar as the model is accurate over a broad domain of states, but it can be computationally costly to derive
'DeepMind
'Equal contributions. Correspondence to [abyravan, leonardh @ @google.com
1 Work done at DeepMind
$ Maa Planck Institute for Intelligent Systems, Tübingen, Germany and Computational and Biological Learning Group. University of Cambridge
Facebook Reality Labs
We use the term proposal mean "proposal distribution", which distribution over the space of actions.

---

Published as a conference paper at ICLR 2022
actions from models. Ultimately, the pure planning approach involving deriving actions from models is usually prohibitive and some form of additional knowledge is required to render the search space tractable for real-time settings. Coming from the model-free RL perspective, the natural approach is to learn a policy that serves as a relatively assumption-free proposal for the planning process.
While in most model-based RL publications the ambition is speed up policy learning for single task, we believe this is not necessarily the most promising setting, as it is difficult to dissociate the learning of dynamies model (which is potentially task independent) from that of value function (which is task specific). A different intuition is that model-based RL might not accelerate learning on a particular problem, but rather enable efficient behavior learning on new tasks with the same embodiment. In such transfer settings, we might hope that a dynamics model will offer complementary generalization to the policy, and we can transfer the policy, model, or both.
In this work, we validate these intuitions about the relative value of the different components of this system. For high-dimensional locomotion problems, we find that even with a good learned model successful MPC requires good proposal distributions to succeed. This effect is particularly pronounced in domains with multiple goals (or equivalently multiple different tasks). Building on this insight we show that it is sometimes possible to leverage learned models with a limited search budget to boost exploration and learn policies more efficiently. Policies that are trained by off-policy updates from data acquired through planning do not reliably perform well when used without planning in our experiments. To overcome this problem, and enable planner amortization into a policy that does not require planning test time, we propose training the policy with combination of a behavioral cloning objective (on MPC data) and an off-policy update with MPO (Abdolmaleki et al. [01 20). When transferring models and proposals other tasks we find only marginal improvements in data efficiency relative to a well-tuned model-free baseline. Overall, our results suggest that while MPC with learned models can lead to more data efficiency, and planners can be amortized effectively into compact policies, it is not a silver bullet and model-free methods are strong baselines.
2 BACKGROUND & RELATED WORK
Model-predictive control refers to the use of model-based search or planning over a short horizon for selecting an action. In order to make it computationally tractable, it is common "seed" the search at a given timestep with either an action from a proposal policy or an action obtained by warm- starting from the result of the previous timestep. In simulation, it has been demonstrated that a fairly generic MPC implementation can be effective for control of a relatively high DoF' humanoid (Tassa
ct al. 2012) and that MPC with learned models (Chua et al., 2018 Wang Bal 2020 Nagabandi
et al. 2020) can achieve data-efficient solutions to simple control problems. Real RC helicopter
control has also been achieved using an MPC approach that made use of a learned model (Abbeel
ct al. 2006). MPC approaches are gaining wider use in robotics (multicopters (Neunert et al. [0066]
Torrente et al. 2021). quadruped (Grandia et al. 2019 Sleiman et al. 2021), humanoids (Tassa
eet al. [014]] Kuindersma et al. 2016)), and dexterous manipulation (Nagabandi et al. (020)); but the
computational speed of the planner is a bottleneck for hardware deployment. There are different ways
around this, with the core practical solution being to plan in lower dimensional reduced coordinate models. Alternatively, POPLIN (Wang& & Ba 2020) explores learning proposals for MPC and planner distillation but is tested on simple tasks and does not leverage model-free RL for proposal learning. and LOOP (iikchi et al. 2022) uses an off-policy actor to control exploration for MPC.
A Amortized policies map states to actions quickly, but implicitly reflect considerable knowledge by being trained on diverse data. Model-free RL produces policies which amortize the knowledge reflected in rollouts required to produce them. Similarly, it is possible to produce diverse trajectories from a planner and distil them into single policy (Levine & Koltun, 2013) Mordatch & Todorov
2014 Mordatch et al. 2015 Pan et al. 2017, Xie et al.) 2021).
Reinforcement learning approaches with MPC have become more popular recently, and our work fits within this space. As noted in the introduction, previous work often emphasizes the role of amortization through the learning of value functions and models. TreePI (Springenberg et al. 2020). MuZero [Schrittwisser et al., 20200] Hubert et al., (021). SAVE (Hamrick et al. 2020). DPI (Sun et al. 2018) and MPC-Net (Carius et al. 2020) all perform versions of hybrid learning with model- based MCTS or planning being used to gather data which is then used to train the model and value function to accelerate learning. Other recently proposed algorithmic innovations blend MPC with learned
2

---

Published as a conference paper at ICLR 2022
value estimates to trade off model and value errors (Bhardwaj al. 2021). Here, primarily consider learning dynamics models to enable transfer to new settings with different reward functions.
Other uses of models unlike hybrid MPC-RL schemes have also been explored in the literature; however, they are not the focus of this work. Nevertheless we highlight two of these approaches: value gradients can be backpropagated through dynamics models to improve credit assignment (Nguyen & Widrow 1990 Heess et al., 2015 Amos et al. 2020) and it is possible to train policies on model rollouts improve data efficiency (Suttonn 1991 Janner et al., 2019). Recently, there has also been considerable effort towards exploring model-based offline RL methods in order to gain full value from offline datasets Yu et al.
2020 Argenson & Dulac-Arnldd 2021 Kidambi al. 2020
3 TASKS
In this paper consider number of locomotion tasks of varying complexity, simulated with the MuJoCo (Todoroy et al. (012)) physics simulator. We consider two embodiments: an 8-DoF ant from mmm_ontrol [Tuyaauvunakooo et al. (020)) and model of the Robotis OP3 robot with 20 degrees of freedom. For each embodiment, we consider three tasks; walking forward, walking backward and "so-to-target-pose" (GTTP), a challenging task that is the focus of our evaluation. In all tasks, the agent receives egocentric proprioceptive observations and additional task observations.
In the walking forward and backward tasks the agent is rewarded for maximizing forward (or backward) velocity in the direction of narrow corridor. For the OP3 robot we also include a small pose regularization term. The task is specified through a relative target direction observation.
The GTTP task, which builds on existing motion tracking cennsrructuur(Haaeneleeee et al. 2020), consists of either body on plane, with tar- get pose in relative coordinates as a spec-fp observation and proximity to the target pose re- warded. When the agent is within a threshold distance of the target pose (i.e. it achieves the current target). it gets a sparse reward and a new target is sampled. For the ant we target poses from policies trained on a standard go-to-target task (Tunyasuvunakool et al. 2020). For the OP3, we use poses from the CMU mocap database (cmu) (retargeted to the robot). We use thousands of different target poses; the agent has to learn to transition between them. Thus the GTTP task can be seen as either a single highly diverse task or as a multi-task setting with strongly related tasks and consistent dynamics. We believe the GTTP task should be particularly amenable to model-based methods: it combines a high-dimensional control problem with diverse goal distribution. This makes it hard to solve quickly with model-free methods. However, since all goals share the same dynamics, a dynamics model should help leverage the common structure. See Sec. [A] an in-depth motivation and Sec. F.5 for results on tasks from the DeepMind Control Suite (Tasaa et al. 2018).
Figure 1: Go-to-target-pose (GTTP) task with the OP3
& Ant. The agent has reach the target pose (blue).
METHOD
4.1
MODEL PREDICTIVE CONTROL (MPC) FOR BEHAVIOR GENERATION
We consider control obtained by model-based planning that refines initial action-samples from a proposal distribution. MPC executes the first action from the planned action sequence and iteratively re- plans after each timestep (c.f. actor loop in Alg. TT. In this scheme, actions are sampled either from the planner (PLANNER) with probability Palan or the proposal To (with parameters 6) with probability
1 - Pplan- Setting Pplan = 0 results in using only samples from the proposal and vice-versa for Pplan = 1. Setting Pplan = 0.5 leads to interleaved execution of proposal and planner actions, like the stochastic mixing of student and expert actions in DAGGER et al. 2011).
The PLANNER subroutine takes in the current state Sz, the proposal Ta (with parameters 0), a model m& (with parameters σ) that predicts next state 8 given current state s, and action a, the known reward function r r(*,, ag $t-1) and, optionally, a learned state-value function Vs(s) (with parameters e)) that predicts the return given state s. While consider primarily Sequential Monte Carlo based planner (SMC, Alg. 2 (Piché et al. 2019; Gordon et al.] 1993) and the Cross-Entropp Method (CEM,
Alg. 3 (Botev et al. 2013), other planners (Williams al. 2017) could be used. See Sec.
3

---

Published as a conference paper at ICLR 2022
Algorithm 1 Agent combining MPC with modcl-free RL
Given: Randomly initialized proposal To. (pre-trained or random) model ma, random critic Qv. optionally (pre-trained or random) task aassosgnttiic proposal p.. II Modules to be learned
Given: Known reward r. planning probability Palan. replay butfer B. MPO loss weight a, BC loss weight B. learning rates & optimizers (ADAM) for the different modules. Known modules and parameters
II MPC loop Asynchhocooyouussy on the actors
while True do
Initialize ENV and observe state so-
while episode not terminated do
If Choose between planner and proposal action depending on Pylan
II Use mixture of (pre-trained) task-agnostic proposal (pw) and learned proposal (Re) as proposal for proposal transfer exps (Sec. (..4. Use pre-trained model for model transfer exps (Sec. 5.3)
dg PLANNER(8; No, mø, T. V+) if x S Pplan TT(s))
where U
otherwise.
Step ENV(s.. ai) -> (8141, and write transition to replay buffer B end while end while
I/ Asynchronously on the learner
while True do
Sample batch B of trajectories, each of sequence length T from the replay buffer B
Update action-value function Qs based on B using Retrace (Munos et al. 2016). Update model my based B using multi-step loss in [Equation 20 (Sec. 4.2)
Update proposal me based on using [Equation 4 (Sec. 4.3)
Optionally, for from-scratch experiments, update agsk-agnostic proposal pa using behavioural cloning [Equation 1) on transitions in B.
end while
The hybrid agent in Alg. has several learnable components: the model mgg the proposal my as well as the (optional) value function Vo: these are learned based on data generated by the MPC loop the actor and we measure learning progress w.r.t environment steps collected. Note that we assume the reward function is known since it is under the control of the practitioner in most robotics applications.
4.2 LEARNING DYNAMICS MODELS
Successfully applying MPC requires learning good dynamics models. In this paper, we train predictive models my that take in the current state 8 and action a, and predict the next state 8t+1 = (5. a.). This model has both Iearned and hand-designed components; the task-agnostic proprioceptive observations are predicted via "black-box" neural networks while any non-proprioceptive task- specific observations (e.g. relative pose of the target) are calculated in closed form from the predicted proprioceptive observations together with a known kinematic model of the robot. The learned components are parameterized as set of deterministic feed-forward MLPs, one per observation group (e.g. joint positions), and predict the next observation from the current observations and action. This model trained end-to-end via a multi-step squared error (Eqn. 20 between an open loop sequence of predicted states and true states sampled from the replay buffer. We use deterministic models as they are more amenable to multi-step losses and an ablation study (Sec. F:4 of the supplement) as well as a recent benchmarking effort (Lutter et al.] 2021) found minimal differences between using deterministic vs stochastic model ensembles for model-based planning. Training happens from scratch, concurrently with policy learning, More details are in Sec. Cf the supplement.
4.3 LEARNING PROPOSAL DISTRIBUTIONS
As we will show, planning with a good dynamics model by itself is insufficient for solving challenging continuous control problems. That is, generating behavior according to the MPC loop Alg. [ with goal-unaware proposal fails to solve the challenging GTTP tasks even when planning with the ground-truth model and a known reward function. This is due to the fact that the action space is large, and the planning algorithm is limited by computational constraints and a finite horizon. In order for planning to succeed, we find it vital to provide planners with an action proposal distribution, which helps search by increasing the probability of sampling plausible, task-relevant actions. In addition, we show that the learned proposal can itself be deployed without planning at test time; this can be particularly useful in compute constrained settings where planning the robot may not be feasible.
4

---

Published as a conference paper at ICLR 2022
But what constitutes good proposal? A natural answer to this question is that a good proposal produces actions, for each state, that lead to task success, and the planner selects the best among already good candidate actions. A simple strategy to obtain such proposal is to iteratively amortize the planning process. That is, starting from any proposal, execute the planner for a given duration to return improved action and fit the returned action; this will sharpen the proposal and lead to the planner focusing on better actions in the next iteration. This idea has been used in the RL community before and underlies modern search based algorithms such as AlphaZero (Schrittwieser et al. 2020).
In the MPC setting we can formalize the idea as follows: let TT(als) refer to the stochastic mixture of PLANNER &t, Te. m r, Vi)) and (a(a)s) with probability Pplan (i.e., as actions are selected according to the MPC loop in Alg. D. And let Drg be dataset of states and actions collected by executing this policy- note that practice for data efficiency reasons we consider dataset agg- where मड is the average behavior distribution during training, which changes over time due to the proposal and model being learned. Clearly, assuming the planning procedure produces improved actions, we expect that मe is at least good as the average proposal policy Re. We can then improve our proposal by minimizing the forward KL divergence DS [K L( (-(s) [=([s))]] which leads to an improving average proposal over time, and is equivalent to maximizing:
Jnc( =
Ex [log To (a]s)),
i.e. the proposal is learned by amortizing the planner via behavioral cloning of the planner actions.
Unfortunately, in cases where the planner cannot find an improvement on the proposal (as is the case for random proposals in our experiments), the above objective may lead to premature convergence at sub-optimal behavior (see e.g. Wann & Ba (2020). To ameliorate this issue, we can consider hybrid updates that both amortize the planner, but also directly favor actions that lead an improvement in terms of cumulative return via an off off-po RL policy update. We will use the MPO (Abdolmaleki ctt al. 2018b) algorithm, although other recent RL algorithms could be used instead. More precisely, the MPO policy improvement step involves using learned action-value function Q (s.(ss) x
Eu Σx 2"re,,,,, (t) 8, ag = a), and optimizing the KI. constrained RL objective:
JNhr (g) = E. E [ (s, a]]l
s.t. ESED. [K][(([ss]] <.
(2)
It is known (see Abdolmaleki et al. 201 2018b) that the solution of this optimization is given in closed form as q(a () x no (b(a]s) expp("=e (s, a)/n), where is a dual variable optimized such that the KL- constraint on the policy is fulfilled. We can fit this improved policy by minimizing the KL divergence E.ED: K L(g(- (s) mo( (-((s))) [s)) which is equivalent to maximizing the weighted log-likelihood:
IMPo = E.EDa [Bng [exp(Q"*' (s,a)/n) log (a(as)]]]
(3)
where Te. is the last reference policy (fixed for this optimization). In practice MPO uses an additional trust-region constraint to stabilize learning. We can combine the two objectives (Eqns. and 3 for
improving the proposal via a simple weighting to obtain the complete objective
IMPO+BC (0) Jmro (8) + (J)c(0)
(4)
The full agent showing the actor and learner loops is shown in Alg. [ In our experiments we compare several variants corresponding to different choices of Pplan: and B. See Sec. for further details and rationale on our choice of MPO as the learning algorithm.
5 RESULTS
5.1 EVALUATING PRE-TRAINED MODELS AND PROPOSALS
In a first set of experiments we study how MPC (Alg. I Pplan = 1) performs on the locomotion tasks considered in the paper. We evaluate performance with two different models, the ground truth MuJoCo simulator and pre-trained model that is trained on data from a successful agent on the corresponding task (see Sec. (.2). We also consider two different proposal distributions: zero-mean Gaussian proposal and task-agnostic proposal that is pre-trained with a behavioral cloning objective on logged data from a successful agent (similar to the prior distribution in Galashov et al. (2019), see Sec. E.2) This pre-trained proposal depends only on proprioceptive information but not task specific observations, and therefore is a reasonable prior capturing average behavior. We assume known reward function but do not use learned value function. The best results of large hyper
5

---

Published as a conference paper at ICLR 2022
Ant forward
OP3
Model
Planner
Proposal
backward
GTTP
forward
backward
GTTP
Near-Optimal performance
1380
1580
650
R: 480
Rs 570
2 730
Gaussian
0++1
0+:1
197 +
5
-14+1
9
ground truth
CEM
Gaussian
780 + 4
771 +
400 2
13
11
10 =
ground truth
SMC
Gaussian
717 + 4
378 7
12
10 =
11
pre-trained
CEM
Gaussian
280 1 8
108 10
365
+ 16
9
3
19 2
pre-trained
SMC
Gaussian
231 7
8
411 15
411
-7 7
15
pre-trained
615
693
56
277 27
318 35
36
ground truth
CEM
pre-trained
1291 + 33
1218 1218::35
187
12
275 :
ground truth
SMC
pre-trained
1064 27
1207 27
373
112 + 14
472
- 23
178 =--=4
pre-trained CEM pre-trained
1222 + 41
1097 62
349 + 16
338 + 39
462
+ 50
100
pre-trained
SMC
pre-trained
1177 + 40
1095 + 40
420 14
247 + 12
473
+73+17
17
170 =
Table 1: MPC with the true reward function, MuJoCo simulator/pre-trainge trained dynamics model and fixed Gaussian/pre-trainet proposals. present best results from large sweeps over planner hyper parameters, compute budget, planner horizon and Gaussian proposal variance. Results use a horizon of 30 time-steps, 2000 samples and are averaged over 100 episodes. For further details, see Sec. F.T indicates settings where saw numerical issues with ground truth model and the CEM planner.
parameter sweep are shown in Table 1 together with performance of the two proposals and baseline performance of a successful agent whose data was used train the pre-trained model and proposal (labelled "Near-ptimal performance', Results from an additional sweep comparing performance of planning w.r.t computational budget and proposal use can be seen in Sec. F.1
When planning with zero-mean Gaussian proposal and the ground truth dynamics, both planners improve significantly on the performance of the Gaussian proposal for the simpler Ant tasks but not for the harder OP3 tasks (this may be in part because the OP3 tasks terminate if the robot falls, which makes planning hard). This result holds across a wide range of computational budgets (see subsection F.T). Planning with a pre-trained model and the Gaussian proposal performs similar the proposal itself on the harder OP3 tasks but there is a small improvement, albeit less than when planning with the ground truth dynamics, for the simpler Ant tasks. Specifically on the Ant GTTP task, planning leads to better rewards but fails to consistently achieve target poses and there remains a large qualitative difference between the planner results and near-optimal performance.
When planning with the pre-trained task agnostic proposal instead, the performance far exceeds the results from planning with the Gaussian proposal or executing the pre- pre-trained proposal without any planning. This highlights the need for a suitable proposal, especially for the high-dimensional OP3 tasks, further motivating our approach learn a proposal for MPC with model-free RI.. When planning with pre-trained proposal, using the ground truth dynamics or pre-trained model achieve similar performance, indicating that our pre-trained models are suitable for planning. Lastly, both planners perform similarly across tasks, though on the harder GTTP tasks SMC slightly outperforms CEM. We use SMC throughout the paper as makes better use of the proposal unlike CEM which uses only for plan initialization (see Sec. F. for full results).
5.2 LEVERAGING MPC WITH LEARNED MODEL AND PROPOSAL
Next, we evaluate different approaches to learning a proposal from scratch for MPC based on the approach discussed in Sec. 4.3 In particular, we consider the following variants:
MPO: MPO with a distributional critic similar Hoffman et al. (2000). Corresponds to 1,3 = 0. Ppian = 0 from
Alg. MPO Syyer--prraeed are tuned for each task (and listed in Sec. F.2 of the supplement).
MPC+MPO: MPC to collect data. MPO objective for learning (ar 1, = 0, Pptan 0.5).
MPO+BC: Adding MPO and BC objectives (a = 1, 3 > O, Palan = 0). where is tuned per task.
MPC+MPO+BC: MPC to collect data. Combined MPO+BC objective (a 1, 3 0, Pplan 0.5).
The model is trained from scratch for the MPC variants (see Sec. We choose Pplan 0.5 for all our MPC experiments as it worked better than Patan = 1.0, and use 250 samples and planning horizon of 10 for SMC (see Sec. F2 for ablations). We tune the BC objective weight / per task.

---

Published as a conference paper at ICLR 2022
Learning From Scratch
Ant GTTP Reward
Ant- GTTP Target pose count
Ant- BWD Walk Reward
1700
Num. poses reached
Episode return
600
Episode return
1200
800
400
300
0
10
20
30
0
10
20
30
0.5
1.0
1.5
2.0
OP3 GTTP Reward
0P3
GTTP Target pose count
0P3
BWD Walk Reward
800
650
Num. poses reached
Episode return
Episode retum
450
500
6
300
300
3
150
100
0
0
25
50
75
100
25
50
75
100
10
20
30
40
Env steps (millions)
MPC+MPO
MMP++CC
MPC+MPO+BC
Actor
Proposal (mean)
Executing the MPC actor (MFC+) or the learned policy (rest). Proposal (mean): Executing the learned proposal's mean action.
Figure 2: Performance of algorithmic variants in Sec. 5.2 when trained from scratch. Actor:
Figure (left & center columns) presents performance comparisons the go target pose tasks showing the reward and corresponding number of achieved target poses for each variant. MPC+ outperforms MPO significantly with regards actor performance (see left column, solid lines). However even though MP MPC+MPO led to higher reward trajectories earlier in the learning process, the proposal distribution by itself tended perform only as well as the MPO baseline. We speculate that this phenomenon is due to the increased off- off-policyness of the planner trajectories. Adding the BC objective mitigates this issue, improving the performance of both MPC and MPC+MPO. The resulting variant MPC+MPQ BCBC significantly outperforms all other baselines in terms of actor and proposal performance as well as learning speed. Additionally, we tried bootstrapping with the learned critic for planning, but did not observe significant improvements on the GTTP tasks (see Fig.
also tested our approach on simpler forward and backward walking tasks for both the Ant and OP3 bodies. Figure 2 (right column) shows the results from the backward walking tasks. For the Ant (top row), MPC+ significantly outperforms MPO early on during training but reaches similar asymptotic performance while for the OP3 (bottom row) the difference between MPO and MPC+MPO small throughout training. Interestingly, on these simpler tasks the proposal for MPC+MPO matches the actor performance and adding the BC objective gives only minor performance gains. We posit that a well-tuned implementation of the model-free MPO baseline achieves near-optimal performance on these tasks and provides a strong baseline that hard to beat both in terms of data efficiency and performance even with the true reward function and bootstrapping with a learned critic. We present further details as well as forward walking results showing similar trends the supplement (Fig. [3). Lastly, since the BC objective substantially improves results for both MPO and MPC+MPO we use the BC variants in all further experiments. Videos of learned behaviors can seen our website
Ablations:
We additionally ran several experiments ablating our design choices. The results of these experiments can be found in the supplement. Figure 10 and Figure I] show the results of varying
Pplan well as the number of samples used by the planner and the planning horizon, respectively. In
Figure 12 we show the effect of bootstrapping with the learned value function within MPC. Figure 17 and Figure 18 compare deterministic models with PETS-style stochastic models (Chua al. 2018
2 All learning curve results are averaged over seeds. See Sec. F.2 hyperparameter details and videos.
Recennt work has shown that similar combinations of MPO and BC objective work quite well across challenging online and offline-RI. settings (Abdolmaleki al.
7

---

Published as a conference paper at ICLR 2022
Model and/or Proposal Transfer Actor Performance
Ant GTTP to GTTP
Ant FWD Walk to GTTP
Ant GTTP to BWD Walk
675
1700
Episode return
Episode return
1200
Episode
800
400
450
450
0
10
20
30
0
10
20
30
0.5
1.0
1.5
2.0
OP3 GTTP to GTTP
OP3 FWD Walk to GTTP
OP3
GTTP GTT- to BWD Walk
750
650
Episode return
600
Episode return
600
Episode retum
450
400
400
300
150
100
100
0
10
20
30
40
50
10
20
30
40
50
10
20
30
Env steps (millions)
MPO++C
MPC MPO+
MPC BC Finetune
MPC MPO Frozen
Proposal From Scratch
Reloaded Proposal
Figure 3: Performance of algorithmic variants in Sec 5.3 with model and/or proposal transfer across tasks. Top bottom row: Ant OP3 results; Solid dashed lines: Model transfer (except MPO+BC) proposal transfer.
in our setting. Furthermore, in Figure 19 we report results using ensembles rather than single model. Finally, Figure 20 shows results the walker and humanoid tasks in the DeepMind Control Suite.
5.3 MODEL TRANSFER
In this section, we study how well models can transfer between tasks. We first explore to what extent learned model can boost performance on complex task. As control experiment transfer learned models from the OP3 GTTP task to the same task. We consider three different settings (in addition to the baseline MPO+BC) where a) no model is transferred and one is trained from scratch on the target task (MPC+MPO+BOL b) the transferred model is finetuned on the target task (MPC+MPO++CC) FinFFunnnnnne and e) the transferred model is kept frozen throughout training on In this experiment (Figure left column, solid lines)
the target task find little improvements learning speed or asymptotic performance when transferring a model vs learning from scratch. Transferring frozen model (which can fail generalize out of distribution data) performs slightly worse vs finetuning the transferred model. This trend also holds other transfer settings such transferring models from the forward walking task GTTP and transferring from the GTTP task to backward walking (Figure 3 center/right column, respectively, solid lines).
While this result is perhaps surprising, does agree with initial investigation of planning with learned models (Table T) where we saw poor performance on all our tasks without good proposal for the planner leverage. Transferring a good model still does not solve the initial exploration problem, especially on the harder tasks. Videos of learned behaviors can be seen here
5.4 PROPOSAL (AND MODEL) TRANSFER
In addition to transferring learned models across tasks we also consider transferring learned proposals. On each source task we train proposal that is dependent only on proprioceptive information and lacks task-specific information (similar to the one used |subsection 5.1) and is thus transferable across tasks. As simple approach to transfer pre-trained proposal we used it for action generation while keeping the learning objective unchanged. Concretely, we use a mixture of the reloaded proposal from source task and the learned proposal on the target task as our proposal distribution (πτ) for the MPC loop in algorithm|] I The mixture weight of the reloaded proposal is annealed linearly from to O in fixed number of learning steps (tuned per task); Sec. F33 of the supplementary material.
8

---

Published as a conference paper at ICLR 2022
Again, we first transfer the proposal trained on the GTTP task to the same task (iiuure 3 left column, dashed lines). Transferring the proposal leads to faster learning for both MPO+BC and MP C+MPO+BC as well as to a smaller extent, better asymptotic performance. Transferring the model and proposal does not lead to any additional
together (MP MMPC+MP*+C++*S netune, improvements, further strengthening our intuitions from the model transfer experiments.
Next we transfer proposals between different source and target tasks, which yielded some nuanced insights into the need for compatibility of the tasks. Figure (center column, dashed lines) shows the results when transferring a proposal from the forward walking task to the GTTP task and Figure 3 (right column, dashed lines) presents proposal transfer results from the GTTP task to backward walking. Interestingly, proposal transfer hurts both XPCCMPO+BB and MPO+BC in both these cases, especially for the high-dimensional OP3, leading to slower learning and lower final performance on both the target tasks. These results provide complementary insights regarding proposal transfer: there should be good overlap between the data distributions of the source and target tasks for proposal transfer to succeed. This is not the case in these transfer experiments; forward walking has very narrow goal distribution compared to GTTP making the resulting proposal far too peaked, and while proposal from the GTTP task would be quite broad (see website for some trajectory rollouts) it is highly unlikely to capture the behavior of walking backwards. Overall, these results provide encouragement that combining the right proposal, potentially trained on diverse set of tasks, together with model-basee planning can lead to efficient and performant learning on downstream tasks. For more transfer results and plots showing amortized policy performance see the supplement (Sec. F3
6 DISCUSSION
Our initial experiments highlighted that a good dynamics model is not enough to solve challenging locomotion tasks with model predictive control when computation time is limited, especially in tasks with high task and control complexity. In such settings a good proposal is necessary to guide the planner. Motivated by this finding we study different approaches to learning proposals for MPC, considering variants that combine the learning objective from MPO, a model-free RL algorithm, together with behavioral cloning objective for efficient planner amortization. We also evaluate transfer performance across tasks where either the proposal, learned model, or both are transferred. Overall our results show that for the locomotion domains considered in this paper, MPC with learned model and proposal can yield modest improvements in data efficiency relative to well-tumed model-free baselines. We found that the gains are larger as task and control complexity increase. On simpler walking tasks we saw very small improvements that are further diminished if an amortized policy desired at test time. On our most challenging task, the OP3 go to target pose task, we see both significantly faster learning speed and improved asymptotic performance.
common justification for model-based approaches is the intuition that models can more easily transfer to related tasks since the dynamics of body are largely task independent. We attempted to validate this intuition but had difficulty achieving large gains in data efficiency even when transferring to the same task. We speculate that this finding is related to the difficulty of planning with limited setting even with a near perfect model. the overall system
search budget in a multi- caalti-goal/multiiteet is limited by the lack of good proposal then model transfer by itself may have negligible effect. When transferring models and proposals to new simpler tasks we also did not find substantial benefits. There are a number of other potential pitfalls in this setting in addition the lack of good proposal. If a proposal leads to trajectories that are inconsistent with the state distribution on which the model was trained, MPC may add little value. As we observed when transferring from the go-to-target-pose tasks to walking backwards, an unsuitable proposal may limit asymptotic performance. Additionally, on tasks with fairly narrow goal distributions a well-tuned model-free method can perform just as well as a model-based agent. This suggests tasks with multi-task/multi--e settings provide good test bed to showease the strengths of model-based approaches and aid further research.
On the whole, the gains from MPC with learned models in our setting are meaningful, but not so dramatic as to be a silver bullet, a finding similar to (o/Springenberg et al. (2020); Hamrick et al. (2021). This paper focused on learning complex locomotion tasks from state features, using structured dynamics model as well as focusing on MPC as the way to leverage the model. There are a number of additional settings where models can be used differently and may aid transfer. For example, in partially observed tasks with pixel observations, transferring models and representations may lead to
improvements in data efficiency (Byravan et al. 2020 Hafner et al. 2018 2020

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like thank Jackie Kay, Alex X. Lee, Claudio Fantacci, Jessica Hamrick and Tom Erez as well as the rest of the team at Deepmind for insightful discussions and feedback. The data used in this project was obtained from mocap.cs.cmu.com cmw..ed. The database was created with funding from NSF EIA-0196217.
REFERENCES
Carnegie mellon university CMU graphies lab motion capture library. http::/mocapp: cmu edu/ Accessed: 2021-6-17.
Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Y Ng. An application of reinforcement learning aerobatic helicopter flight. In Proceedings of the 19th International Conference Neural Information Processing Systems, pp. 1-8, 2006.
Abbas Abdolmaleki, Jost Tobias Springenberg. Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller. Relative entropy regularized policy iteration. arXiv preprint arXiv:1812.022: 2018a.
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv, abs/ abss/8806.0el 806206920, 2018b.
Abbas Abdolmaleki, Sandy H Huang, Giulia Vezzani, Bobak Shahriari, Jost Tobias Springenberg, Shruti Mishra, Dhruva TB, Arunkumar Byravan, Konstantinos Bousmalis, Andras Gyorgy, et al. On multi-objeccive policy optimization as a tool for reinforcement learning. arXiv preprint arXiv:2/06.. 08199, 2021.
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. arXiv preprint arXiv:2008. 12775, 2020.
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. In International Confer- on Learning Representations, ICLR, volume 2021, 2021.
Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladien Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su. Rearrangement: A challenge for embodied ai. arXiv preprint wwx.iv:20110099900 2020.
Mohak Bhardwaj, Sanjiban Choudhury, and Byron Boots. Blending mpe & value function approxima- tion for efficient reinforcement learning. In International Conference on Learning Representations, ICLR, volume 2021, 2021.
Z I Botev, D Kroese, R Y Rubinstein, and P L'Ecuyer. The cross-entropy method for optimization. Handbook of Statist., 2013.
Arunkumar Byravan, Jost Tobias Springenherg, Abbas Abdolmaleki, Roland Hafner, Michael Neunert, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Imagined value gradients: Model-based policy optimization with tranferable latent dynamics models. In Conference Robot Learning, pp. 566-589. PMLR, 2020.
Jan Carius, Farbod Farshidian, and Marco Hutter. Mpc-net: A first principles guided policy search. IEEE Robotics and Automation Letters, 5(2):2897-29) 2904, 2020.
Albin Cassirer, Gabriel Barth-Maron, Eugene Brevdo, Sabela Ramos, Toby Boyd, Thibault Sottiaux, and Manuel Kroiss. Reverb: A framework for experience replay, February 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In S Bengio, H Wallach, H Larochelle, K Grauman, Cesa-Bias Bianchi and R Gamett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
10

---

Published as a conference paper at ICLR 2022
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojtek M Czarnecki, Yee Whye Tch, Razvan Pascanu, and Nicolas Heess. Information asymmetry in KL-regularized RL. In International Conference Learning Represen- fations, 2019.
Neil J Gordon, David Salmond, and Adrian FM Smith. Novel approach to nonlinear/non--aause bayesian state estimation. In IEE proceedings (radar and signal processing), volume 140, pp. 107-113. IET, 1993.
Ruben Grandia, Farbod Farshidian, René Ranftl, and Marco Hutter. Feedback mpe for torque- controlled legged robots. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4730-4737. IEEE, 2019.
Tuomas Haarojja Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: wtXiv:812.005005 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. November 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In Eighth International Conference on Learning Representations, April 2020.
Jessica B Hamrick, Victor Bapst, Alvaro GonzzlzzG Tobias Pfaff, Theophane Weber, Lars Buesing, and Peter W Battaglia. Combining q-learning and search with amortized value estimates. In International Conference on Learning Representations, ICLR, volume 2020, 2020.
Jessica Hamrick. Abram L Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Wither- spoon, Thomas Anthony, Lars Buesing, Petar Veličković, and Théophane Weber. On the role of planning in model-based deep reinforcement learning. International Conference on Learning Representations, 2021.
Leonard Hasenclever, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel. Comic: mentary task learning & mimicry for reusable skills. In International Conference on Machine Learning, pp. 4105 4115. PMLR, 2020.
Comple-
Nicolas Heess, Greg Wayne, David Silver. Timothy Lillicrap, Yuval Tassa, and Tom Erez. Learning continuous control policies by stochastic value gradients. In Proceedings the 28th International Conference on Neural Information Processing Pastems-Volumee 2, pp. 2944-2952, 2015.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint orXiv::707 000.022866 2017.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang. Bilal Piot, and Nando de Freitas. Acme: research framework for distributed reinforcement learning. June 2020.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonogiou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. arXiv preprini arXiv:2104.06303 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- Based policy optimization. In Advances in Neural Information Processing Systems, volume 32, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
11

---

Published as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Scott Kuindersma, Robin Deits, Maurice Fallon, Andrés Valenzuela, Hongkai Dai, Frank Permenter, Twan Koolen, Pat Marion, and Russ Tedrake. Optimization- baaend locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous robots, 40(3):429 455, 2016.
Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learning, pp. 1-9. PMLR, 2013.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. In International Conference on Learning Representations, ICLR, volume 2019, 2019.
Michael Lutter, Leonard Hasenclever, Arunkumar Byravan, Gabriel Dulac-Arnold, Piotr Trochim, Nicolas Heess, Josh Merel, and Yuval Tassa. Learning dynamics models for model predictive agents. September 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 5 518(7540):529-53,, 2015.
Igor Mordatch and Emo Todorov. Combining the benefits of function approximation and trajectory optimization. In Robotics: Science and Systems, volume 4, 2014.
Igor Mordatch, Kendall Lowrey, Galen Andrew, Zoran Popovic, and Emanuel V Todorov. Interactive control of diverse complex characters with neural networks. Advances in Neural Information Processing Systems, 28:3 123-13121400 2015.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G Bellemare. Safe and efficient off off-policy reinforcement learning. arXiv preprint / 02647, 2016.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112. PMLR, 2020.
Michael Neunert, Cedric De Crousaz, Fadri Furrer, Mina Kamel, Farbod Farshidian, Roland Siegwart, and Jonas Buchli. Fast nonlinear model predictive control for unified trajectory optimization and tracking. In 2016 IEEE international conference on robotics and automation (ICRA), pp. IEEE, 2016.
Derrick Nguyen and Bernard Widrow. The truck backer-upper: An example of self-learning in neural networks. In Advanced neural computers, pp. 11-19. Elsevier, 1990.
Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile autonomous driving using end-to-end deep imitation learning. arXiv preprint arXiv: (rXiv:1709.07172 2017.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. Transactions on Graphics (TOG), 2018
Alexandre Piché, Valentin Thomas, Cyril Ibrahim, Yoshua Bengio, and Chris Pal. Probabilistic planning with sequential monte carlo methods. In International Conference on Leurning Represen- tations. 2019. URL
net/ http://wwwww.eeeeeeeuteeeaeeea wow.mmidd-yetttneeee..
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, 2011.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with leamed model. Nature, 588(7839):604-666 2020.
12

---

Published as a conference paper at ICLR 2022
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off-policy with online planning, In Conference on Robot Learning, pp. 1622-1633. PMLR, 2022.
Jean-Pierre Sleiman, Farbod Farshidian, Maria Vittoria Minniti, and Marco Hutter. A unified mpe framework for whole-body dynamic locomotion and manipulation. JEEE Robotics and Automation Letters, 6(3):4688 4695, 2021.
Jost Tobias Springenberg, Nicolas Heess, Daniel Mankowitz, Josh Merel, Arunkumar Byravan, Abbas Abdolmaleki, Jackie Kay, Jonas Degrave, Julian Schrittwieser, Yuval Tassa, al. Local search for policy iteration in continuous control. arXiv preprint arXiv:20/0.05545, 2020.
Wen Sun, Geoffrey J Gordon, Byron Boots, and J Bagnell, Dual policy iteration. In S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa-Bianchi and R Garnett (eds.), Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 160):160-1 1991.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4906-4913. IEFF, 2012.
Yuval Tassa, Nicolas Mansard, and Emo Todorov. Control-limited differential dynamic programming. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1168-1175. IEEE, 2014.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXhv:1801.00690 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Guillem Torrente, Elia Kaufmann, Philipp Föhn, and Davide Scaramuzza. Data-driven mpe for quadrotors. IEEE Robotics and Automation Letters, 6(2):3769 3776, 2021.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020.
Hado Hasselt, Matteo Hessel, and John Aslanides. When parametric models in reinforce- ment learning? June 2019.
Tingwu Wang and Jimmy Ba. Exploring model- -ased planning with policy networks. In International Conference on Learning Representations, ICLR, volume 2020, 2020.
Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2): 344-357, 2017.
Kevin Xie, Homanga Bharadhwaj, Danijar Hafner, Animesh Garg, and Florian Shkurti. Skill transfer via partially amortized hierarchical planning. In International Conference on Learning Representations, ICLR, volume 2021, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 14129-14142. Curran Associates, Inc., 2020.
13

---

Published as a conference paper at ICLR 2022
A TASKS
We consider a number of locomotion tasks of varying complexity, simulated with the MuJoCo (Todorov et al.) 2012) physics simulator. We consider two embodiments: an 8-DoF ant from di_ mo__oottoll Junyasuvunakool et al. 2020) and a model of the Robotis OP3 robot with 20 degrees of freedom. For each embodiment, consider three tasks: walking forward, walking backward and (GTTP), a challenging task that is the focus of our evaluation. In all tasks, the agent receives egocentric proprioceptive observations (joint angles. joint, linear and angular velocities as well as end effector positions). In addition we provide the world z-axis in the robot's frame of reference. For the ant the proprioceptive observations are 37 dimensional. For the OP3 robot the proprioceptive observations are 64 dimensional. In addition to proprioceptive observations the agent also receives task specific observations that describe the task goal and differ by task.
poo-to-target-oosse
A.1 WALKING TASKS
In the walking forward and backward tasks the agent is rewarded for maximizing forward (or backward) velocity in the direction of narrow corridor:
U Gurrgtt
(5)
where u denotes the velocity of the robot and étarget denotes unit vector in the direction of desired movement (both in the frame of the agent). For the OP3 robot we also include a small pose regularization term to regularize towards walking pose. The agent observes the target direction Etarget E Rdd in it's egocentric frame reference as a task-specific observation for the walking tasks.
A.2 GO-TO-TARGET- POSE TASK
The GTTP task consists of either body plane, with a target pose in relative coordinates task-specific observation and proximity to the target pose rewarded. When the agent is within a threshold distance of the target pose (i.e. it achieves the current target), it gets a sparse reward and new target sampled. For the ant we use target poses from policies trained on standard go-to-target task (Tunyasuvunakool et al. (020).. For the OP3, we use poses from the CMU mocap database (emu) (retargeted to the robot). We use thousands of different target poses; the agent has to learn to transition between them. Thus the GTTP task can be seen as either a single highly diverse task or as a multi-task setting with strongly related tasks and consistent dynamics. We extend existing motion tracking infrastructure (Hasenclever et al.) 2020) to build our task.
Figure 4: Go-to-target-pose (GTTP) task with the OP3 Ant. The agent has to reach the target pose (blue).
The agent agent receives task specific observations that specify the target pose relative to the agent. Specifically, we use relative joint angles, as well as relative root position and relative positions and orientations of a number of different body parts, all expressed in the egocentric frame of the embodiment, These task observations are 107 dimensional for the ant and 163 dimensional for the OP3, respectively.
Task reward The task reward consists of two parts, a dense reward term that corresponds to how well the target pose is matched and a sparse reward that is added if the target pose 'achieved':
Tdense + S [Tadnnse > Fitresholdd for N steps),
where we use Fthreshold = 0.65, . = 1, S = 50 for the OP3 and l'tresholdd 0.8, N = 10, 50 for the ant. The rewards are similarly robot specific and will be described below. The rewards as well as the tolerance Tinneshold and N were tuned to give visually good behaviors with large-scale on-policy agent.
For the dense reward we use rewards of the following form:
(-(
(-("hoty) Mrubac cd)
)
))
(- Mmobatt
l'èense
0.6exp
Tpose + 0.4exp
(-(
0.3
2
14

---

Published as a conference paper at ICLR 2022
where d denotes the center of mass distance between the agent and the target, "osee is reward that depends chiefly on the robots joint configuration but less on the relative position to the target pose and Mobott robot specific multiplier. We use mohott 4 for the OP3 and mobox = 1 for the ant.
Ant pose reward For the ant we use the following pose reward:
T pose 0.5 exp (-dree) (0.Arpox + 0.6r 0.5 exp (-0.1dma)
where d denotes the relative root distance. pes and "qua terms penalizing deviations from the target pose terms of the positions and orientations of the individual MuloCo bodies that make up the Ant:
ppos exp (-0.85 Pbodiss
ref Pbodics l1²)
Fquat exp (-4 Abodics Abodies 112die
(10)
where Poodies and Poodies are the body positions the ant and the reference, respectively, and Ibodies and ref are the body positions of the ant and the reference, respectively,
OP3 pose reward For the OP3 use the following pose reward (similar Peng al. (2018);
Hasenclever et al. (2020)) with terms penalizing deviations in terms the center mass, the joint angle velocities, the end effector positions and the joint orientations:
? pose 0.1 com + Teel + 0. 0.15rapp + 0.657
The first term Teom penalizes deviations from the centre mass:
exp |-40 Poomll?) 112)
(11)
where Prom and peet are the positions of the centre of mass of the simulated character and the mocap reference, respectively. The second term Yvel penalizes deviations from the reference joint angle velocities:
Tvel exp (-0.1/lgvel
Gval 11
rel
(12)
where Qvel and qual are the joint angle velocities of the simulated character and the mocap reference, respectively. The third term Tapp penalizes deviations from the reference end effector positions:
exp [-160]| Papp 1peep11 112),
(13)
Tapp
where Papp and p are the end effector positions of the agent and the target pose, respectively. Finally, Fuuat penalizes deviations from the reference in terms of the quaternions describing the body orientations:
Toodies
exp (-2)1 Aquaa C ref
Phodies 11
(14)
where e denotes quaternion differences and Ghodies and gee are the joint quaternions of the agent and the target pose, respectively.
Planner rewards The reward terms above cannot be computed solely from observations available to the agent. principle would be possible to instead use reward terms that are only based on observations. However, in practice, planner rewards that correlated with but different from the task reward.
For the OP3 use the following reward for planning:
droot
)) (0.6exp
2 2| Pjotons! 112
0.5 0.9 exp (-droa) + 0.1 exp
10
dyoot
+ 0.5 exp
(15)
10
where J is the number of joints.
For the Ant we use similar reward with slightly different lengthscales for the planning reward:
0.5 (0.9exp (- dreat
drext
))
(0.6exp
2|]Pioints
p
jointss
ref
I|a
0.1 exp
40
J
dront
0.5 exp
(16)
15

---

Published as a conference paper at ICLR 2022
Target pose sampling At the beginning of an episode and whenever target pose is achieved sample a new target. The target pose is sampled uniformly from a reference set of target poses. For the ant this set is derived from expert policies that have been trained on a standard go-to-target task from dm_ dm_cont rol (Tunyasuvunakool et al. 2020). For the OP3 we use target poses from the CMU mocap datahase (emu).
To determine the position of the new target relative to the agent we sample both a heading relative current agent heading (sampled uniformly between -120" and 120%) and random distance. We also sample the orientation of the target randomly relative to the current agent orientation (heading shift sampled uniformly between 120 and 120%).
To make this challenging task a bit easier we use what we call an intra-episode curriculum to determine the distance of the target pose from the agent. We linearly scale up the random distance with each achieved target (up to some maximum value). Concretely, let U[a, b) be a uniform distribution and let M be the number of curriculum phases. Then after having achieved m target poses, the distance of the next target pose is drawn from U [mmin(m/MM Ja, min(m/ /M, 1)6.. We use M = 4 and a = 0.5, = 5 meters for the ant and = 0.05, = 1. meters for the OP3.
Motivating the go-to- target-postee task The go-to-target-pon task can be motivated and understood from a few distinct perspectives. Firstly, arises naturally as a temporally abstract version of a motion tracking task, and indeed we build our task as an extension of existing motion tracking infrastructure (Hasenclever et al., 2020) that is available as part of di_ mit_oootroll (Tunyasuvunakool et al. (020).
In tracking task the target pose and reward change every timestep such that the agent is rewarded for producing behavior that tracks a vime-na reference motion. In the GTTP task, the instruction and reward definitions can be similar the tracking task; however, the GTTP task involves fixed target pose for a number of timesteps until the target is achieved. Insofar as the policy that solves a tracking task an inverse dynamics model (s, s + a). the policy that solves the GTTP task is a temporally abstract inverse model.
From another perspective, the GTTP task the essential cel-reearrageement task that can be performed by any avatar in an empty environment. Batra et al. (2020) argue that a conceptual unification of goal-directed navigation tasks, visual servoing, and object manipulation tasks are that they all are rearrangement tasks that can be standardized as challenges for embodied control.
Rearrangement tasks vary along multiple dimensions, including temporal abstraction and degree of specification. A motion tracking task is essentially self-rearrangement challenge that involves a target pose being fully specified at every timestep. The GTTP task involves a target pose being specified with an unspecified offset and the policy must handle the temporal abstraction by closing in on the target pose. While both of these tasks fully specify the desired pose, one can alternatively consider versions of the tasks where the desired pose is not fully specified, the desired pose is about external objects rather than the pose of the body. To surmount the issues related to temporal abstraction, policies may, implicitly or explicitly, break problems down into subgoals. Curiously, goal-directed policies (or inverse models) also may themselves serve as effective abstractions for achieving subgoals, playing the role of lower-level controllers insofar as they can abstract away the capabilities required to execute movements to a subgoal.
even further where
We believe the GTTP task should be particularly amenable to model-based methods: it combines a high-dimensional control problem with diverse goal distribution. This makes it hard to solve quickly with model-free methods. However, since the dynamics are shared between all goal poses, a dynamics models should be beneficial in leveraging the common structure.
B PLANNERS
We consider two sample based planners, primarily a Sequential Monte Carlo based non-iterative planner (SMC) (Piché et al. 2019 Gordon et al. (993) and the Cross-Entropy Method (CEM) (Botev) et al. 2013). Our SMC and CEM planners are shown in algorithm 2 and respectively. We briefly describe the planners and discuss how they can be warm-started with a provided action proposal distribution.
SMC planner: The SMC planner (Alg. 2 is a non-iterative sample-based planner that maintains number of particles S corresponding to different model rollouts. All particles are initialized to
16

---

Published as a conference paper at ICLR 2022
start at the initial state Su- At each step of the rollout (upto horizon of H), an action is sampled from a proposal for each particle; this proposal can be either a fixed distribution or the learned proposal. These actions are then rolled out through the model to compute the next state, reward and optionally, a value estimate. The particles are then resampled according to the weighted exponentiated reward/advantage with temperature parameter т. The rollout performed upto horizon of H and the first action from a randomly chosen partiele (amongst the surviving particles at the final timestep) chosen as the action to executed. Unlike CEM, SMC samples actions from the proposal each step within the rollout and is therefore more suited to warm- warm-starting using learned proposal; we choose it as our planner primarily for this reason.
Algorithm 2 SMC planner
1: Given: state 8n. proposal Tp. model mhg reward r, planning horizon H, number of samples S, planner temperature and optionally value function V
2: {sp so $
I/ Initialise samples to the initial state (56 (() (t41 denote the ith particle up to time t
3: Let T0 (5)
{(
0p
(5)
4: for tr = O... do
5:
{ar (i) =o(-1s )=
II Sample actions
6: { ma(s
a
11
1/ Take model step.
7:
tre rss
1/ Evaluate rewards.
8: calculate advantage A
based on rewards and value function V
// Resampling
A///T) A /T)
9:
w. & exp
II Resampling weights proportional to exponentiated advantage
10: 11.32..... ig Categorical((w:)) (0)
Draw resampling indices II resample trajectories
11: Zoit +- 0it
(k)
(in)
Vk 10=1.... 1...S
12: end for
13: (11..... $s
Il Sample action index Return first action from a random particle
14:
return 0g
The CEM planner (Alg. 3) an iterative sample-based planner maintains distri- bution over action sequences that is usually parameterized as Gaussian with mean μ and standard deviation a. At the start of planning. the mean μ initialized an open-loop sequence of actions sampled from proposal distribution (either fixed or learned) and the standard deviation a is ini- tialized to dwnit In each iteration, S trajectories are sampled from this distribution and rolled out through the model and their returns are computed. The top F. fraction of trajectories, ranked by the return, are retained and their mean Helite and standard deviation Delite are computed. These are used to updated the mean μ and standard deviation via exponential average (with weights Omeon and asta). This procedure is repeated for iterations and after the final iteration, the first action from μ is executed in the environment. Unlike SMC, CEM uses the proposal only for plan initialization the first iteration and uses the distribution N )) for further trajectory samples.
CEM planner:
C MODELS
Successfully applying model predictive control requires learning good dynamics models. In this paper, we train single-step feedforward dynamics models m. that take in the current state and action a and predict the next state $1+1 ma az). This dynamics model has both learned and hand-designed components; the task-agnostic proprioceptive observations are predicted via "black-box" neural networks while any non-proprioceptive task-specific observations (e.g. relative pose of the target) are calculated in closed form from the predicted proprioceptive observations in combination with known kinematic model of the robot.
Fig. 5 (top) shows the network architecture of the learned components of the predictive dynamics model (m&). We use a deterministic, feed-forward MLP per proprioceptive observation i.e. one each for joint angles, joint velocities, linear velocities and proprioceptive observations and the action from the current timestep and predicts a delta change in the observation being predicted; this delta change scaled by an embodiment specific timestep
This MLP takes input all the
17

---

Published as a conference paper at ICLR 2022
Algorithm 3 CEM planner
Given: state proposal To, model mæ. reward r, planning horizon H, number of samples S, elite fraction E, noise standard deviation Onits number iterations and optionally value function Vi // Rollout proposal distribution using the model to initialize the plan.
(so, 0o. $1,.... H &- rollout faath_proposse (mo Tg. H)
f- [a0, 01 0.
II initial plan (H action dimension)
0 f- Oinit
for i 1. I do
for = ==1....d
II Sample candidate actions. // Evaluate candidate action sequences open loop according to the model and compute associated returns.
Pk: (H,
t-taaaauateeecctioooo patpiaattteactiiinneteeee Pk, H,r, V+p)
end for
Rank candidate sequences by reward and retain top F fraction.
Compute mean Pelite and per-dim standard deviation delte based on the retained elite sequences.
11 Ommean ) + Olmean/Meltee
I/ Update mean; Ommean 0.9 Update standard deviation; asid = 0.5
σ - (1 asa) + O
end for return first action in μ
(df 0.02 for Ant, dt = 0.03 for OP3) and added to the observation at the current timestep to get the prediction the next timestep:
$411= = s m² ae) dt
(17)
where st denotes the k-th predicted observation at time + 1, o is the k-th observation time t and m is the k-th MLP.
The only exception to this is for joint angle predictions, where we make a delta prediction on top of the joint velocity predictions get the predicted joint angles the next timestep:
g++1 qt + (m² (st, a.) + de+1) ) dt
(18)
Here, Tt and d++1 denote the observed and predicted joint angles and + respectively, m2 denotes the MIP that predicts the delta change in joint angles and d++1 is the predicted joint velocities +.+ 1. In effect, we implement an Fuler integration step for joint angles based on the predicted joint velocities 1)) and a correction term from the MLP m (s), at).
In the case of the that predicts the 3D positions and orientations (represented as quaternions) of different parts of the embodiment. The architecture of this model is shown in Fig. 5 (bottom). This network takes in only the joint angles of the robot and predicts the 3D positions and orientations of the robot's bodies via ferwaforward, deterministic MLP; as before, there are two separate MLPs, one each for the position and orientation predictions.
task, we have an additional learned "forward kinematics" model
C.I TRAINING
We train our models alongside the policy and critic using data collected from the MPC loop in Alg. D that is saved in the replay buffer. We sample batch of trajectories (51.7, -1,---1,,771 from this replay buffer, where T' is the trajectory length (set to 10 in all our experiments). This batch of trajectories also used for policy and critic learning.
Given a trajectory of length T, we perform an open-loop rollout with the model using the initial state 31 and the entire sequence of actions a1:T-1:
22 = m (81, m 01)
33 = m5(32 a2)
ST (5 -1
(19)
18

---

Published as a conference paper at ICLR 2022
dimfobs. k]
Proprio
256
256
256
obs k denates the k-th observation. The MLP takes in the action and all proprio observations and predicts the change in a the k-th observation as culput. This scaled by the task timestep (dt) and added to the k-th observation time get final prediction.
observations
obs_k (t+] predicted
Action (t)
obs_k
(t)
Forward Dynamics model
(separate MLP per observation predicts future observations based observations and action at time
dimibodies. ppos)
(or) dimt dimtbodies, quatt)
256
256
256
HIHHH
dense (FC) layer norm
elu
Joint angles (t)
concat
tanh
sum
Forward Kinematics model
(MLP that predicts end-effector postionss and orientations joint angies used only GTTP task, There are two separate ML Ps; one each for posltion and ortentation prediction respectively)
scale by dt
Figure 5: Network architectures of the forward dynamics and forward kinematics model. The forward dynamics model (top) takes as input the observations (s²) and the actions (a²) at time t, and predicts the observations at time 1+1 (8t+1). ). We use separate MI.P per predicted observation group that takes in the entire observation (s) input and predicts single ohservation group (8") ) as output (e.g. there is separate ML. that regresses joint angles, one that regresses joint velocities etc.). This done by predicting delta change in the observation that scaled by a task-specific timestep parameter (dl = 0.02 for Ant and di = 0.03 for OP3 tasks) and added to the current observation s to generate the final prediction ($ We use separate MLPs for each observation group decouplc the different loss terms and handle scale differences between the different observation groups (c.g. accolerations and velocities tend to have larger values compared to positions). As the name suggests, the forward kinematics model (bottom) takes only the joint angles as input and predicts the positions and orientations of different parts of the embodiment. There is a separate network for position predictions and one for orientation predictions (represented as quaternions). This network is used only for the go-to-target-pose task where the task-specific observations include the relative pose of the target's bodies; use the predictions of this network integrate these reference observations in closed form (see Sec. Cfoor an explanation).
where the first prediction 82 uses the true state 81 and subsequent predictions use the predicted state from the previous timesteps.
We then compute mean squared error between the predictions and the true states:
T
Jm T Σ
18
s;1/2
(20)
where 6; and 8; denote the predicted and true state respectively. This objective Jm is minimized for training the model parameters Φ; we use an Adam optimizer (Kingma & Bal 2014) with a fixed learning rate of le-4 for training, and use a sequence length of 10.
We use a similar training procedure for the forward kinematics model (used only for the GTTP task) but instead use the joint angles from the true states $1.7 as inputs the network as opposed to predictions. We compute a squared eтror between the true and predicted 3D positions and a quaternion difference for the between the true and predicted orientations; these are summed together and averaged across time to get the final objective for training. As before, we minimize this objective using an Adam optimizer (Kingma Bal 2014) with a fixed learning rate of le-4 and a sequence length of 10 for training the model parameters.
C.2 INTEGRATING MODEL PREDICTIONS FOR PLANNING
As mentioned earlier, only proprioceptive observations predicted by the learned models but the policy and critic depend on both proprioceptive and task-specific observations (see e.g. Fig. (6). Consequently, need way generate future task-specific observations given initial state and
19

---

Published as a conference paper at ICLR 2022
action sequence for multi-step planning to work. We do this by integrating task-specific observations in closed form based on the predictions from the learned dynamics (and kinematics) models. As specified in egocentric frame of reference do not need any external information for this integration other than the observations at the current timestep and the predictions from our learned models. We briefly describe the integration process for all our tasks next.
all our observations (proprio and task-specific)
For the walking tasks, the only task-specific observation be integrated is the relative direction of the target. Since the angular velocity of the robot is a proprioceptive observation (and the timestep di known) we can generate a change in orientation by Euler integrating this velocity from the unit quaternion. By rotating the relative target direction with this change in orientation we can obtain the predicted relative target direction at +
For the GTTP tasks, there are several task- bask-specific observations including the relative joint angle differences between the robot's current configuration and the target, the relative root position and the relative 3D positions and quaternions of the target bodies w.r.t the current configuration. The integration of relative joint angles is straightforward: we can subtract the Euler integrated joint velocities to the relative joint angles at to get the relative joint angles at + 1. Similarly, we can integrate the relative root position by adding the translation in the egocentric frame of reference (obtained by Euler integration of the linear velocities). The integration of the relative body positions and orientations is more involved, and uses a combination of the predicted translation, rotation (obtained by Euler integration of linear/angular velocities) and the observed & predicted positions and orientations of the robot (at and t + respectively); as before, this does not need any external information not available part of the egocentric observations. The procedure for predicting the full next state 8141 given the current state 8; and is as follows:
1. Predict all the proprioceptive observations via the learned dynamics model: 8/41 ma1(1,a)) ax); Vk.
2. (Only for the GTTP task) Predict the body positions and orientations t++ 1 via the predicted joint angles at - 1 (from step above).
3. Integrate the task-specific observations based on the predictions from the learned dynamics (and optionally, the kinematics) models.
This procedure is used within the PLANNER subroutine (see Alg. [ for generating future states via multi-step rollouts.
D MPO
Maximum A-Posteriori Policy Optimization (MPO) (Abdolmaleki et al. 2018b) continuous control RL algorithm that performs an expectation minimization form of policy iteration. There are two steps comprising policy evaluation and policy improvement. The policy evaluation step receives as input policy Ne and evaluates and action-value function Q7th (s,a) by minimizing the squared TD error:
(1++ 10 (4411,0411 - Q (st, az))
min E. (e 8
where i denotes the parameters of a target network (Mnih et al. from 0 and d= denotes the distribution over 8t, at S1+1, 0++1 induced by In practice use replay-buffer of samples in order to perform the policy evaluation step. The second step, policy improvement consists of optimizing the objective J(s, T) E,, [Q7 (s, a]] for states drawn from state distribution (ss). In practice the state distribution we draw samples from is the replay buffer. By improving J in all states s, we improve our objective. To do so, two step procedure performed.
2015) that are periodically updated
First, we construct non-parametric estimate such that J(s, q) 2 ((S,TR). This is done by maximizing J(s, q) while ensuring that the solution, locally, stays close the current policy g3i i.e. Ey(s) ( < This optimization has a closed form solution given as
g(als) me (alss) exp (s.a)/1
20

---

Published as a conference paper at ICLR 2022
where 7) is a temperature parameter that can be computed by minimizing a convex dual function (Ab dolmaleki et al. 201 8a). Second, we project this non-maraetttic representation back onto a param- eterized policy by solving the optimization problem - - arg min, By(s) [KL(g(a|s) [=] where 7k+1 is the new and improved policy and where one typically employs additional regulariza- tion (Abdolmaleki et al.. 2018b). Note that this amounts to supervised learning with samples drawn from ([]]s); see Abdolmaleki et al. (2018b) for details.
D.1 CHOICE OF MPO
We believe that the choice of the underlying learning algorithm (MPO) is not crucial for our results. In fact, we expect that using any state-of-the-ar off-policy algorithm such as Soft Actor Critic (SAC) (Haarooj et al. 2018) should lead to similar results across the tasks considered.
We choose MPO as our learning algorithm for two reasons. First, performance-winn it is comparable with state-of-the-an art off-policy actor-critic algorithms and has been thoroughly evaluated across variety of continuous control settings (Hoffman et al. 2020). Second, the behavioral cloning objective (Eqn. T add is compatible with the MPO objective (Eqn. (3) as they are both log- log-likelihood based objectives with a similar scale; setting a trade-off between the two is straightforward. Adding the BC loss MPO has also been explored in prior work (Abdolmaleki et al. 2021) with improvements in learning across different settings.
E EXPERIMENTAL SETUP
We work with a distributed set-up with 64 actors interacting with the environment, and a single learner that trains the policy, critic and model. The actors collect data via the MPC loop described in Alg. D where the probability of planning Pylon controls the tradeoff between executing the action from either the planner or the learned proposal mT. This data is then stored in a replay buffer in the form of trajectories of state, action, reward triplets ($1:T, 41:2'-1, T1:r). The learner samples batches from the replay buffer (we of million trajectories in all our experiments) and runs a learning update where the parameters of the policy, critic and model are updated in tandem. To study data efficiency in data-limited regime, we additionally limit the number of actor steps per learner update step, i.e. the learner is blocked if the actor has not sampled enough data and the actor blocked if the learner has not performed enough parameter updates. This is implemented using the reverb framework (Cassirer et al. [02TT) We did a sweep over this "rate limit" parameter and found the best settings to be 8 actor steps per parameter update for the walking tasks and 16 actor steps per parameter update for the GTTP tasks (increasing the values further led to significantly reduced performance). This is held fixed for all from scratch and transfer experiments with all variants.
a batch size of 256, sequence length T = 10 and replay buffer size
Alg. T describes the overall agent including the acting and learning loops which run in a distributed fashion with 64 actors and a single learner as mentioned earlier. The data from the actors which run MPC, warm-started with the learned policy (see MPC loop of Alg. I is added to the replay buffer; the learner asynchronously samples batches of data from this buffer and updates the parameters of the policy, critic, model and optionally, the aagostin proposal in an iterative fashion.
E.1 POLICY AND CRITIC TRAINING
Our policy architecture is presented in Fig. 6 (top left, (a)); the policy takes as input both the proprioceptive and task-specific observations and outputs the parameters of Gaussian distribution from which actions are sampled. We use a distributional critic (Fig. - bottom left, (b)) that takes as input both the observations and actions and returns a discrete distribution (101 bins) that parameterizes histogram of Q-values (from 0-300), similar to the one proposed in Hoffman et al. (2020).
Given a batch sampled from the replay buffer we compute Q targets using Retrace MMnos et al. 2016) with terminal value bootstrapped from a target critic network (both the target policy and critic network parameters are updated once every 200 steps). The use of a distributional critic makes value backups hard; we circumvent this by bootstrapping using the mean Valuue. We then convert the value targets into a two-hot representation (only two bins in the histogram are non-zero rest are zero), which fit with critic by minimizing the softmax cross-entropy loss between the critic
21

---

Published as a conference paper at ICLR 2022
dimtaction)
dense (FC)
elu
Proprio observations (t)
256
256
258
layer
concat
tanh
Task observations (t)
(c) Task-agnostic Proposal (Depends only on proprio. transferred across tasks)
dimfaction) 2
(a) Policy
256
Proprio abservations (t)
(b) Critic
Proprio
observations
256
256
256
258
256
101
Task observations (t)
Action (t)
Figure 6: Network architectures of the policy, critic and task-agnostic proposal used for transfer. (a) The policy network takes all observations as outputs and predicts the mean/std. dev. of the Gaussian action distribution. The mean is scaled by tanh and the standard deviation which is scaled via softplus and summed together with initial scale (Ie-4 our experiments). (b) The distributional critic takes all observations and the action time as input and predicts discrete distribution that made up of 101 bins (representing equally spaced Q-values from 0-300), and (c) The task-agnostic proposal only takes in the proprioceptive observations as input and predicts the parameters of a Gaussian distribution (similar to the policy network). Since this network does not take task-specific observations captures only the average behavior, and be freely transferred across tasks. We use this proposal for the initial planner experiments (Sec. [51 and the proposal transfer experiments (Sec. 5.4)
predictions and the two-hot targets. As discussed in the main text (Sec. 4.3 and Equ. we use a weighted combination of the model-free MPO objective and Behavioral Cloning objective (BC) for training the policy and critic. We use separate Adam optimizers (Kingma & Ba 2014) with learning rates of 3e-4 for training the policy and critic, and use Adam optimizer with learning rate of le-4 to optimize the temperature and dual variables in the MPO algorithm (Abdolmaleki et al. 2018b).
E.2 TRAINING A TASK-AGNOSTIC PROPOSAL
In addition to the policy and critic, we also train a task-agnostic proposal (Fig. center right, (c)) that takes only proprioceptive observations as input and returns Gaussian distribution for sampling actions. Since this proposal lacks any task-specific information it can only capture average behavior (e.g. walking randomly in the GTTP task, as opposed to directed walking to a target). Since proprioceptive observations are the same across tasks for a given embodiment this proposal can be transferred freely across tasks; we used this proposal for the initial planner experiments (Sec. 5.1 and the proposal transfer experiments (Sec. 5.4).
We train this task-agnostic proposal alongside our policy and critic using a Behavioral Cloning (BC) objective only. We sample batches from the replay buffer (which have both good and bad data) and fit the policy by maximizing the log-pro log-rrob of executed actions under the proposal distribution (similar to Eqn. Th. We an Adam optimizer with a learning rate of 1e-4 for training this task-agnostic proposal. We present a few videos showing the behavior learned by the task task-agnostic proposals the supplementary material and the website
22

---

we ran hyper parameter sweep over Published as a conference paper at ICLR 2022
Ant GTTP
Hortzon
Hortzon
Mortaon
Mortoon
Morizon
000
800
100
4
Inco
Number of zamp es 0og scale)
Gausdian proposai
Gaussiin propocai planning)
Sest agent perfor
Pre-tratned task agnostic
Fre trained task
proposal stanning!
Figure 7: Performance of planning with SMC on the Ant GTTP task using a temperature of 0.01. Performance with both a standard Gaussian proposal and pre-trained task-agnostic proposal improves when more samples used. Interestingly, they perform similarly across the board. This is likely because the pre-trained proposal (which can walk but not aware the target pose) tends to propose actions that would wander away from the goal whereas the Gaussian proposal gives jittery actions that will not lead to sustained movement of the ant's centre of mass. Since the initial target pose is quite close to the ant and the ant is stable even jittery movements yield relatively large return. This is reflected in the performance of the two proposals without planning. The Gaussian proposal, which does not lead much movement, achieves larger return than the pre-trained proposal which wanders off. See our website for a visualization of the behavior generated by task-agnostic proposal. 'Best agent performance' denotes the performance of the best RL agent from this work and should be interpreted very good performance. Even with large computational budget, planning does significandly worse than this agent. Results averaged over 100 episodes. The error bars show standard error.
Ant Walking
Walking bwardd Horizon
Welcing backouurd Horizon
Weilingg beccounnt Horlorn
Walking Hollann 5
Wale bad Horizon 50
0
Uniking frnward 1orizon
Walking for ward
Weiskr forward foon
Whiking forward Porizon
Walking forwart
1000
500
01
40
60
Number sampiio
performancel
trained thik agrostic proposal
crained agnostic prococal ino planning)
Figure 8: Performance of planning with SMC the Ant walking tasks using temperature of 0.1. Both standard Gaussian proposal and pre-trained task-agnostic proposal improve when more samples used, As expected, the pre-trained proposal outperforms the Gaussian proposal across the board. Unlike the GTTP task the pre- trained task task-agnostic proposal is able to walk approximately forward or backward but can drift. Thus. planning with this proposal does much better than using the Gaussian proposal (though the difference is reduced for larger computational budgets best RL agent trained on this task and should he interpreted as very good performance. Results averaged over 100 episodes. The crror hars show + standard error.
would expect). 'Best agent performance" denotes the performance of the
F FURTHER EXPERIMENTAL RESULTS
F.1 PLANNER RESULTS
For our result with pre-trained models and proposal (Table I the common planner parameters:
Planning horizon (H): 5, 10, 15, 20, 25, 30
Number of samples (S): 50, 100, 200, 500, 1000, 2000
For the Gaussian proposal we also swept over the standard deviation of the Gaussian (0.5, 1.0 and 2.0). In addition we also swept over planner-specifie parameters. For SMC:
23

---

Published as a conference paper at ICLR 2022
Planner temperature (r): 0.001, 0.01, 0.1
For CEM:
Elite fraction (E); 0.15, 0.3
Number of iterations (7): 1, 2,
Noise standard deviation (dinse): 0.1, 0.25, 0.5, 1.0
Table presents the best results for each planner, which use a planning horizon of H = 30 and number of samples S 2000. We note here that best performance with CEM is obtained with I > 1, which effectively means that CEM uses a significantly larger computational budget than our SMC planner which is non-iterative; in spite of this SMC is still quite competitive with CEM across all tasks as can be seen from the results in T
In further set of experiments we evaluated the performance of SMC with wider range of samples and ground truth dynamics on the Ant walking. Ant GTTP and OP3 GTTP tasks. The results are shown in Figures 7 8 9
In the simpler, lower dimensional Ant domains, increasing the number of samples used by the planner improves performance, irrespective of the proposal distribution used for planning. In the higher-dimensional OP3 GTTP task the results are qualitatively different. When using longer horizons and more samples from a pre-trained but goal-agnostic, proposal the performance increases. However even with 25600 samples from a standard Gaussian proposal the planner fails to make any progress. This is due to the difficulty of sample-based optimization in high-dimensionas spaces, which further validates our choice of the use of learned proposal distribution for the planner. Overall the planner results show the challenges of planning in challenging high dimensional tasks even with access to ground truth models, rewards and large computational budgets. Using a proposal is crucial for progress on these domains.
GP3 Horizon
Hortbon
Horbzon 10
Horizon
Horizon 50
200
150
100
50
Number of samples (log Scale)
Gausstan tre-tr task osti
Gaussian (nd planning)
www.trrraledd agnostic proposal pla planning
Figure 9: Performance of planning with SMC on the OP3 GTTP task using a temperature of 0.01. With the pre-trained task-agnostic proposal performance improves when more samples are used. As with the Ant GTTP task this proposal can avoid falling and walk in random directions which provides significant reward compared to the Gaussian proposal which can casily cause the robot to fall, state from which it cannot escape. Consequently, with a standard Gaussian proposal even 25600 samples do not lead to any meaningful task progress; on such high-dimensional tasks random sampling would rarely lead to any interesting behavior appearing. With the standard Gaussian proposal the planner reaches a target pose only in 1-2% of episodes by chance. The performance of the best RL agent trained this task is around 730 (not shown) which is far better than anything achieved by planning (with or without the proposal). Results are averaged over 100 episodes. The error bars show = 1 standard error.
While the best planner only results use a high computational budget as presented above, we use SMC with S = 250 samples and planning horizon 10 for all our learning experiments for improved learning speed (wall-clock time) and to ensure that our computational budgets are realistic with respect to real-world considerations (assuming a 20Hz run-rate of the actor). Interestingly, this setting does not result in performance reduction compared to those with more compute as show in ablation experiments (Fig. [10) and IT). Details on other planner hhperr-pramene parameters for individual experiments are detailed below.
F.2 FROM SCRATCH RESULTS
In initial experiments we determined the number of actor steps per learner step (using the MPO baseline). We tried 4, 8 and 16 actor steps per learner step and found that the performance degraded
24

---

Published as a conference paper at ICLR 2022
OP3 GTTP From Scratch Sweep over num samples (N), planner prob (Pptan)
Reward
Target Pose Count
750
Num. poses reached (Actori
N-250
Episode retur (Actor)
600
N-500
N=1000
400
Palan - 0.5
Palen 1100
200
0
10
20
30
40
50
10
20
30
40
50
Env steps (millions)
Figure 10: Performance of MPC+MPO+RC when learning from scratch the OP3 GTTP task with sweep over different numher of samples for the SMC planner (N) and probability of planning (Pplan from MPC loop of Alg. D. Left: Plot comparing reward vs numher environment interactions, Right: Plot comparing the numher target poses reached vs number of environment interactions. For a fixed planning horizon H = 10, increasing the number of samples does not make significant difference in performance (compared to the default N 250, solid blue line). Increasing the probability of planning Ppten 1.0 (always planning) leads slightly worse performance compared intericaved planner and policy exccutions (Pptan 0.5). Results averaged over
seeds.
OP3 GTTP From Scratch Sweep over planning horizon (H), planner prob (Palen)
Target
Count
750
12
Num. poses reached (Actor)
H=5
Episode return (Actor)
N=10
500
H=20
H=30
300
Pala
Palon =1.0
100
10
20
30
40
50
20
40
50
Env steps (millions)
Figure 11: Performance of MPC+MPC+BC when learning from scratch on the OP3 GTTP task with sweep over different planning horizons for the SMC planner (H) and probability of planning (Ppton from MPC loop in Alg. D Left: Plot comparing reward number of environment interactions, Right: Plot comparing the number of target poses reached number of environment interactions. For fixed number of samples N 250. changing the planning horizon also does not have a significant impact on performance data efficiency compared (solid blue linc); there is slight improvement with = 20 but this leads
to the default setting of to significantly slower experiments w.r.t wall clock time so we choose H 10 in our experiments. At lower horizons H 5, always planning (Pplan 1.0) does as well interleaved planning and policy execution (Ppian 0.5) but there's significant difference higher horizons over two seeds.
10 (see left plot). Results averaged
when using lower than actor steps per learner step the walking tasks and 16 actor steps per learner step for the GTTP experiments. We used these settings for all subsequent experiments.
tuned each algorithmic variant independently per task, running sweeps over MPO hyper param- eters and, where applicable, BC objective weight 3 and planner temperature We show the best results of each variant in the figures. For MPO
we ran sweeps over the constraint sweep several different settings for the additional trust-region constraint in the policy mentioned in the discussion of Equutton 3, In practice, MPO constraints the and variance of the policy separately, with constraint parameters
parameter 0.1, 0.5 in Equation 2] Additionally
25

---

Published as a conference paper at ICLR 2022
Learning From Scratch Sweep over Value Bootstrapping
GTTP
GTTP
700
725
Episode return
600
Episode return
500
500
400
300
10
20
30
25
50
75
100
Ant FWD Walk
Ant BWD Walk
1700
1700
Episode return
Episode return
1200
1200
800
800
400
400
0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
Env steps (millions)
MPC+
-MPC+MPO+BC
Value bootstrapping False
Value bootstrapping
Figure 12: Performance of MPC+MPO and MPC MPO+RC when learning from scratch on different Ant/OP3
tasks, when running a sweep over hootstrapping with the value function planning (Alg. [ Top row: Perfor- mance on the GTTP tasks with and without value bootstrapping (solid and dashed lines respectively). Learning a value function can be hard for the the value function does not improve performance, and in fact, hurts performance (compare dashed and solid lines). Bottom row: Performance the simpler Ant walking tasks, where value bootstrapping does improve performance and data efficiency, Results are averaged over four seeds.
style GTTP task, and consequently bootstrapping with
the harder OP3 GTTP task
Walking Forward From Scratch
1700
525
MPO
400
MPC+MPO
Episode return
1200
MPO+BC
300
MPC+MPO+BC
800
Episade
200
400
Actor
Proposal (mean)
0.5
2.0
30
Env steps (millions)
Figure 13: Performance of the various algorithmic variants in Sec 5.2 when trained from scratch the Forward Walking task. Left columm: Ant, Right column: OP3 results. MPC+MFO improves upon MPO for both Ant and OP3 on the actor (solid lines) but the amortized policy has lower performance (red, dotted lines) than the actor. MP MPC+BC and MPC-MP D+3C improve on the performance of their non-BC counterparts, and the amortized policy of the MPC+MPO+ BC matches surpasses the corresponding actor performance (especially the OP3 task). See Sec. 5.2 and Fig. for further description of the results when learning from scratch.
e, and ey. This helps avoid premature convergence in some cases. See Abdolmaleki al. 2018b
for details. We used éx 10-3 and swept over 3 values from 6,. 5. 10-1 101 55 10 10
26

---

Published as a conference paper at ICLR 2022
depending on the task. For algorithmic variants involving BC we ran hyper-parameed sweeps varying the BC objective weight 3 = 0.001, 0.1, For algorithmic variants involving MPC we ran sweeps over the planner temperature = 0.001, 0.01, 0.1 (T 0.01 worked best for all tasks except the Ant walking tasks where r = 0.1 worked best).
We use SMC with S = 250 samples and planning horizon of H 10 for all our experiments with MPC. We determined these to be the best planner settings based on a hyperparameter sweep over planner parameters when learning from scratch with MPC on the OP3 GTTP task; these were also chosen with real-world considerations to ensure that planning could be run at 20Hz on practical systems. Fig. pressentt the results when training MP C+MPO+BC with different number of samples S for the SMC planner with fixed horizon of 10, and the best temperature setting (r = 0.01) from the from scratch experiment (see Fig. 2 Increasing the number of samples or setting Pplan 1.0 does not make a big difference. Fig. shows related experiment, but with a sweep over the planning horizon H instead with 5 250 samples. Once again, there little variation in performance with a longer or shorter horizon.
We also did a quick sweep over bootstrapping with the value function during planning, training MPC+MPO+BC from scratch several tasks with and without bootstrapping. When bootstrapping from the value function, we use the learned critic and average the Q values from the given state and 10 actions sampled from the policy at the given state to compute the value V. Fig. 12 presents this ablation. In general, boostrapping with the value function helps on easier tasks such as walking (Fig. 12 bottom row) but hurts performance the harder GTTP task (Fig. [2] top row) where learning a good value function can be difficult. We use the best settings for each task in our experiments (no bootstrapping for the GTTP tasks and bootstrapping for the walking tasks).
We additionally present results for learning from scratch on the walking forward tasks in Fig. 13 As with the backward walking results, MPC+MPO does better than MPO early on during learning and the addition of the BC objective improves performance compared to the non-BC counterpart especially on the OP3 task. All variants converge to similar asymptotic performance.
F.3 TRANSFER RESULTS
Model transfer: As discussed in the main text we test two variants of model transfer where the transferred model is kept frozen on the target task SEPC+MPO+BCC BCFFOCCC or finetuned on the target task (MPC+MPO+BC+E') (intrFinentnne along with a baseline where we learn the model from scratch. For all the model transfer experiments we transfer the models from the best performing MPC+MPO+BC BC agent on the source task (and its hyperparameter settings). For all transfer settings except GTTP to GTTP we transfer only the forward dynamics model; for GTTP to GTTP we transfer both the forward dynamics and kinematics models. For finetuning we use an Adam optimizer with a learning rate of le-4 for both models (similar to from scratch training).
Proposal transfer: In the proposal transfer experiments we transfer the task-agnostic proposal which depends only proprioceptive observations to the target task. We use mixture of this reloaded proposal from the source task and the learned amortized policy (which has access to all observations) on the target task as the proposal distribution (TT) for planning in algorithm [ The mixture weight of this reloaded proposal is annealed linearly from 1 to 0 in a fixed number of learning steps M. At the start of learning on the target task we sample exclusively from the reloaded task-agnostic proposal on the actor and as learning progresses we sample less and less from this proposal; when the mixture weight reaches 0 we stop sampling from the reloaded proposal and revert to using only the amortized policy on the target task as the proposal distribution on the actor.
We tuned the hyper-parametes M (in terms of learner steps) which controls the slope of this annealing of the mixture probability on all our transfer tasks and presented results from the best setting of M. We choose all other parameters based on the best from scratch results on the target task, with the BC objective added (MPC-MPC+BC).
The per-task sweeps are:
OP3 GTTP to GTTP: M 1.25e5, 2.5c5, 5e5, 1c6, 2c6
Ant GTTP to GTTP: M 1,522,, 31250, 62500, 125000, 250000
OP3 Forward Walk to GTTP: M = 1,25e5, 2.5e5, 5e5, le6, 2e6
27

---

Published as a conference paper at ICLR 2022
Model and/or Proposal Transfer Amortized Proposal (Mean) Performance
Ant GTTP to GTTP
Ant AnD Walk to GTTP
Ant GTTP to BWD Walk
725
1700
Episode return
650
Episode return
650
1200
Episode
800
550
550
400
450
450
0
10
20
30
0
10
20
30
0.5
1.0
1.5
2.0
OP3 GTTP to GTTP
OP3 FWD Walk to GTTP
OP3 GTTP GTTP to BWD Walk
750
Episode return
600
Episode return
600
Episode retum
450
400
400
300
150
100
100
0
10
20
30
40
50
10
20
30
40
50
10
20
30
Env steps (millions)
MPO++C
MPC++M
+PCC BC Finetune
MPC MPC+ Frozen
-Proposal From Scratch
Reloaded Proposal
Figure 14: Performance of the amortized proposal (mean) for all the algorithmic variants in Sec 5.3 model and/or proposal transfer transfer (except MPO+BC) / proposal transfer. The amortized policies converge to similar (or slightly higher) performance compared to the actor performance results shown in Fig. As explained in Sec. [.3 model transfer leads to fairly small performance improvements across tasks. Proposal transfer (Sec. [54) helps for transfer from the GTTP task the GTTP task itself (left column) but can hurt performance when there mismatch in state/goal distributions between the source and target tasks, especially for the higher-dimensioona OP3 (Forward walking to GTTP center column GTTP to Backward walking right column).
45.4 with
tasks. Top bottom row: Ant / OP3 results; Solid dashed lines: Model
Transfer- - GTTP to FWD Walk Actor performance
Ant
OP3
1700
525
MPO+BC
400
MPC+MPO+BC
Episode return
1200
Episade return
of
MPC-MPO+BC+Finetume
300
MPC+MPO+
800
200
Proposal From Scratch
100
Reloaded Proposal
0.5
1.0
1.5
2.0
0
20
30
steps (millions)
Figure 15: Actor performance for different model and proposal transfer variants from Sec. 5.3 and Sec. 5.4 when transferring models and/or proposals from the GTTP task to the Forward walking task. For the simpler Ant (left column), transferring the model does not lead to performance improvements but transferring the proposal does speed up learning significantly.
OP3 (right column), transferring the proposals or models leads to faster learning at the beginning but results in convergence to sub-optimal policy with lower performance compared to the baseline MPO.
Ant Forward Walk to GTTP: M = 15625, 31250, 62500, 125000, 250000
OP3 GTTP to Forward/Backwart Walk: M = 15625, 3 31250, 62500, 125000
Ant GTTP to Forward/Backwand Walk: M 5000, 10000, 20000, 40000
28

---

Published as conference paper at ICLR 2022
Transfer GTTP FWD Walk Amortized Proposal (Mean) performance
OP3
400
MPC+MPO+8C
Episade return
1200
Episode return
BRC+MMO++CC+Finetuuc
300
800
Proposal Scratch
100
Reloaded Proposal
0
30
Env steps (millions)
Figure 16: Performance of the amortized policy (mean) when transferring models and/or proposals from the GTTP task the Forward walking task. Amortized policy matches (or slightly outperforms) the performance of the planner both during learning asymptotically (see Fig. to compare against actor performance).
Additional transfer results: Fig. 14 shows the performance the learned amortized policy for the different transfer variants several of our transfer tasks. This is the counterpart to Fig. which shows the actor performance on the same tasks; general the amortized policy able to match or exceed the performance of the actor for all these settings due the addition of the BC objective. Finally, Fig. [5] and Fig. [6] show the performance of the different variants when models and/or proposals trained the GTTP task and transferred to the Forward Walking task. As in previous transfer results (Sec.15.3 (5.3), model transfer (solid lines) does not lead to significant improvements w.r.t learning speed or asymptotic performance. Similar the results Sec. Scc.5.4] proposal transfer (dotted lines) can help speed up learning early on but can also potentially hurt asymptotic performance especially on the more complex OP3 tasks.
F.4 MODEL ABLATION USING A STOCHASTIC MODEL ENSEMBLE
In our work we primarily used deterministic dynamics models (Sec. which is in contrast with some of the recent literature on model-based reinforcement learning that advocates for ensembles of stochastic models (e.g. PETS (Chua et al.) 2018)) In this section we ablate this design choice. In this ablation, we use the same architecture as for our deterministic models but with Gaussian output distributions (parameterized by a mean and log- og-variance) instead of single predictions. Similar to PETS (Chua et al. 2018), bound the predicted variance to ensure stability and use single step negative log-likelihood loss for training the stochastic models instead of a multi-step loss as for the deterministic models.
For ensembles, we initialize each member randomly and train each ensemble member on a randomly selected subset (of half the batchsize) of each batch to encourage diversity of model predictions. For planning with the stochastic ensemble, we split the different particles SMC equally across the ensemble members; each particle the same ensemble member throughout entire rollout (similar to the TSco approach from PETS (Chua et al. 2018)). This approach ensures that we keep the overall computational budget for planning the same irrespective of the ensemble size.
The results of this ablation study are presented in Fig. 17 and Fig. 18 where we compare the performance of MPC and MPC+MPO+BD when learning from scratch all our tasks. Fig. 17 shows the performance of the actor; using ensembles of stochastic models does not lead to a significant difference performance on the simpler Ant tasks and provides small advantage on the OP3 tasks. This advantage is reduced when looking at the performance of the amortized proposal in Fig. [8] only the performance of MPC+MPO on the OP3 GTTP task is slightly improved through the use of the stochastic ensemble. All results used ensemble size of 3 except the results the Ant GTTP task which uses a single stochastic model.
We additionally ran ablation on the size of the stochastic ensemble, considering ensembles of 1,3 and models. Fig. 19 shows both the actor and amortized proposal performance of MPC+MPO and MPC+MPO+ED the OP3 GTTP task. We observed meaningful difference between 1, 3 and ensemble members with the same computational budget for planning; similar results were observed for all other tasks considered in this paper.
29

---

Published as a conference paper at ICLR 2022
Learning From Scratch
Actor Probabilistic (vs) Deterministic Model
Ant GTTP Reward
Ant BWD Walk Reward
Ant FWD Walk Reward
1700
1700
Episode return
600
Episode return
1200
Episode retur
1200
800
800
400
400
300
0
10
20
30
0
0.5
1.0
1.5
2.0
0
0.5
1.0
1.5
2.0
OP3 GTTP Reward
OP3 BWD Walk Reward
OP3 - FWD Walk Reward
800
650
525
Episode return
Episode return
Episode retum
400
450
500
300
300
200
300
150
100
25
50
75
10
20
30
40
10
20
30
40
Env steps (millions)
MPC+MPO
MMCC+MP++BC
Deterministic
Probabilistic
Figure 17: Actor performance of MPC+MPO (red) and EMCC*MP+BC (green) when trained from scratch using PETS style stochastic model ensemble (dotted lines) the deterministic model (solid lines) used in main results. There is little difference to using a stochastic ensemble OP3 GTTP using stochastic model does slightly better but does not change the results qualitatively (this improvement is reduced when looking at the performance of the amortized proposal; see Fig. 18) Results were obtained using ensemble of three Gaussian dynamics models for all tasks except the Ant GTTP task where single Gaussian dynamics model dynamics model vs an ensemble (see Fig. 19 for a comparison of different ensemble sizes on OP3 GTTP task). Results are averaged over 4 seeds for the GTTP tasks and seeds for the Walking tasks.
deterministic model on most tasks; on
used. In practice, there is difference using single Gaussian
Overall, these results validate our choice of deterministic models showing that a single well structured deterministic model can perform well as stochastic ensemble across challenging locomotion tasks involving high-dimensional, non-linear contact dynamics.
F.5 RESULTS ON DEEPMIND CONTROL SUITE
addition to the tasks presented in the paper we also evaluated our approach on externally published and open-sourced locomotion tasks which are part of the DeepMind Control Suite (Tassa et al.) 2018). Specifically, we tested our approach on three locomotion tasks from the Deep.indd Control Suite: stand, walk and run, with embodiments: 6-DOF wa ker (18-D state space) and 21-DOF humanoid (54-D state space) forming a total of six tasks.
We first tuned MPO (Abdolmaleki al. 2018b) to match the performance of externally published benchmarks in the literature (Hoffman et al.) 2020) across all these tasks. Next, we evaluated our approach of combining MPC with MPO (using deterministic model) these tasks; Fig 20 shows the performance of all the algorithmic variants in Sec. [.2 on these six tasks. All variants perform similarly on the simpler walker embodiment; there no advantage using MPC across all three tasks for this embodiment. On the humanoic d, the MPC variants slightly outperform the model-free MPO variants, especially on the challenging humanoid-2am broadly similar to those presented in the main text and further lend strength to the central message of the paper; while model-based methods can perform really well on high-dimensional continuous control tasks, well-tuned model-free methods are strong baselines. Only on the most challenging tasks and, especially, in multi catiiiooll/mluiii-oeee settings, do model- -ased methods substantially outperform their model-free counterparts.
task. Overall though, the results are
30

---

Published as a conference paper at ICLR 2022
Learning From Scratch Amortized Proposal (mean) Probabilistic (vs) Deterministic Model
Reward
Ant
Ant
Reward
700
1700
1700
Episode return
600
Episode return
Episode return
1200
1200
450
400
400
300
0
10
20
30
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
GTTP
Reward
OP3 BWD Walk Reward
OP3 FWD Walk Reward
800
650
525
700
Episode return
Eoisode return
Episode return
400
500
300
300
200
150
100
100
0
25
50
75
10
20
30
40
10
20
30
40
Env steps (millions)
MPC+MPO
-MPC+MPO
Deterministic
Probabilistic
Amortized proposal (mean) performance for MPC+MPG (red) and MPC+MPO+DE (green), when trained from scratch using PETS style probabilistic model ensemble (dotted lines) vs the deterministic model (solid lines) used in main results. Using stochastic model ensemble instcad of a deterministic model does change the results significantly. Results averaged over seeds for the GTTP tasks and 2 seeds for the Walking tasks.
Figure 18:
OP3 GTTP
From Scratch Effect of ensemble size
Actor
Proposal (mean)
800
800
600
600
Episode return
Episode retur
400
400
MPC+ MPO
MPC+ MPO BC
200
200
25
50
75
25
50
75
steps (millions)
Figure 19: Performance of MP C-CC+MFO O (solid lines) and MPC+MPO+BC (dotted lines) when learning from scratch the OP3 GTTP task with PETS style probabilistic model ensemble, where sweep the size of the ensemble. Using 3 models the ensemble does lead significant difference in performance for both the actor (left) and the amortized proposal (right). Results are averaged over four seeds.
31

---

Published as a conference paper at ICLR 2022
Learning From Scratch Control Suite
walker-stand
walker-walk
walker-run
1000
1000
1000
Episode return
800
800
Episode return
800
600
600
Episode
400
400
400
200
200
200
0
0
0.25
0.5
0.75
1.0
0.25
0.5
0.75
1.0
0.25
0.5
0.75
1.0
humanoid-stand
humanoid-walk
humanoid-run
1000
1000
1000
Episode retur
800
Episode retur
800
Episode return
800
600
600
600
400
400
400
200
200
200
0
2.5
5.0
7.5
10.0
2.5
5.0
7.5
10.0
2.5
5.0
7.5
10.0
Env steps (millions)
MPO
MPC++MP
MPP+BC
MPC
Actor
Proposal (mean)
Figure 20: Performance of the algorithmic variants in Sec. 5.2 when learning from scratch on six control suite tasks across two embodiments. On the simpler wa ker embodiment all variants perform similarly. On the challenging humanci- d embodiment the MPC variants slightly outperform MPO especially the hardest humanoid é-run task. All results averaged over three seeds, and the best results from parameter sweep also swept over bootstrapping from learned critic while planning for these tasks and found no significant difference with and without bootstrapping; the presented results are without value bootstrapping.
(see F.2 for parameters).
32


Reviews:
[{"review_id": "w-a7YEwgYGh", "reviewer": "Reviewer_Bjo9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- demonstrates the benefit of combined model-learning and proposal learning, in support of MPC, on simple & a challenging GTTP task\n- provides a greater understanding about model-transfer and planner-transfer, across tasks\n- makes us rethink some of the conventional assumptions about model-based RL\n\nWeaknesses:\n- only MPC-settings are considered;  in an ideal world, the reader would also be able\n  to understand how these methods stack up against model-based methods that use the model\n  in different ways.\n- the paper could comment on the possible relative benefits retained by MPC approaches in general, \n  over policy-based methods, in terms of adapting to out-of-distribution states.\n  \n\nThe paper can in part be seen as an \"understanding\" paper, one that provides insights into\nthe interplay between planning and model-free control, and how they can be integrated.\nThis remains and underexplored area.  Relatedly, the relative benefits of model-learning\nin task-specific and for new-tasks with shared dynamics also needs to be better understood, \nas noted by the paper.\n\nThe paper builds on an interesting parameterized task, the go-to-target-pose (GTTP).\nSuch parameterized-goal tasks are likely to provide more insight than fixed tasks.\n\nI'd love to see a structured abstract diagram or table that could somehow capture the design space\nof model-based planning methods, and the general assumptions that motivate the different design choices.\nCaveat:  I don't know if such a diagram is realizable.\n\nre: section 2, other uses of models\nPerhaps older refs to backprop-through-time (BPTT) would help communicate that BPTT has been\nused even before the rediscovery of backprop, i.e., state adjoints used in control.  Or even just go back\nto the Nguyen & Widrow \"Truck Backer-upper\" from 1990.\n\n\"it is possible to train policies on model rollouts to improve data efficiency (Janner et al)\"\nIt might make more sense to simply cite DYNA here, as one among many?\n\nThe connections between MPO and SAC could be made more explicit.\nOr the differences, if these are relevant for the results presented in the paper.\n\nThe idea of separating out proprioceptive information from other goal-related state information\nis a good one.  It is in some sense specific to a certain class of MDP, however, so that could\nbe worthwhile clarifying (although will be obvious to those working on robotics. It also points\nto some form of control hierarchy, where new tasks won't need to leverage significantly new proprioceptive states\nand actions.  Is the global orientation wrt vertical i.e., basic IMU info, also modeled as being part of the proprioceptive information?\n(ok, I now see the answer is \"yes\" in App A).\n\nFigure 2:  The middle column of graphs, giving Target pose counts, would be better replaced\nby simply another task.  The results are highly correlated with the reward graphs in the first column,\nand a tired reader will mistake the results as being yet more training curve graphs. \n\nFigure 3:  The overall title of \"Model and/or Proposal Transfer -- Actor Performance\" is confusing.\nAnd perhaps the legend could be improved, i.e., MO+BC could be relabeled as \"MO+BC (no model)\"\nand \"Proposal from scratch\" could be relabeled \"model transfer\"; and \"Reloaded Proposal\" could become \"Proposal Transfer\".\n\nIt could be Worthwhile noting in the main text that SMC is more conputationally efficient than CEM.\nThis detail is currently in App E.1, but would be worthwhile being part of the Table 1 discussion.", "summary_of_the_paper": "The paper provides a thorough, detailed evaluation of model-based RL, as realized via MPC-based planning.\nIn particular, it examines: (a) how it can be improved via learned policies that act as MPC proposal distributions;\nand (b) to understand and disentangle the benefits of (i) model transfer and (ii) proposal transfer,\nwhen using model-based RL. Comparisons are performed using both CEM and SMC samplers. \nFor the locomotion-related benchmarks considered, MPC with a learned model & proposal can provide modest improvements\nwrt a well-tuned model-free baseline (MPO). However, for a more challenging go-to-target-pose (GTTP) task,\nthey offer significantly improved learning speed and final performance.\nThe benefits of model-transfer are rather marginal, even when transferred to the same task, \ncounter-to-common-belief-and-intuition. \n", "summary_of_the_review": "This is a reasonably thorough evaluation of common assumptions related to model-based RL, particularly in the context of MPC methods. The ideas are tested on a range of problems, ranging from the simple to the challenging, i.e., the go-to-target-pose (GTTP) task. The documented benefits of learned proposals on difficult tasks such as GTTP, as evaluated using both CEM and SMC, are also a worthwhile contribution.  The empirical work is solid.  A better understanding of model-based methods in continuous control settings is important for the RL community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FN8alBbbYl0", "reviewer": "Reviewer_54KF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is very clearly written, well argued, and well executed.    The experiments are well-designed and clearly illuminate the points being made.  The paper adds an increment to the collective knowledge of the field about RL methods.\n\nI also find it deeply unexciting and not at all imaginative.    I would have been more enthusiastic about a less well executed paper with a really new idea.   Surely the fact that transfer isn't working very well tells us that we need some new insights (or I guess, an argument about why all of our intuition about transfer in problems like these is wrong).   Is there a way to improve the planner so that it's less stupid, for example by learning landmarks or making it hierarchical?  Could we make the proposal distribution depend on the goal in the GTTP tasks?\n\nI have some small comments/questions, but none of them are critical:\n- In the early parts of the paper it would be good to clarify what you mean by \"learning proposals\" (it's made clear later)\n- I found it jarring that the word \"proposal\" is used frequently when really \"proposal distribution\" is what is intended;  to me a \"proposal\" would be a single sample from a \"proposal distribution.\"\n- When you discuss \"target poses\" do you really mean \"target configurations\"?   (This is ambiguous, I guess, but in robotics it's most common to use \"pose\" for a 6DOF pose of a rigid object and \"configuration\" for anything more complicated.   \n- Do you think anything different would have happened if the goals were a bit more naturalistic (e.g., get the centroid of the agent into a region)---it's rare in any natural situation or application that the entire configuration of a complex agent would need to be copied.\n- The design of the model was interesting---I was eager to read more about it in the paper body.\n- I am concerned that stochasticity was \"waved away\" without enough consideration.  In many important domains, the distribution of possible outcomes is not at all well summarized by the mean.  It seems important to at least acknowledge this.\n- It was a bit confusing to read the phrase \"task-agnostic proposal pre-trained with a behavioral-cloning objective.\"   It's hard to imagine a situation in which behavioral-cloning is task-agnostic unless, for example, you train on BC data from a whole mixture of tasks.  Perhaps that's what you did.\n\n(To the degree that I lack confidence it is with respect to the detailed novelty of this paper---I don't keep up with this sub-part of the literature and so I can't attest to whether or not this was already all well-known, for example.  But I think I understand the methods described at a sufficient level and get the experiments and their overall message.)", "summary_of_the_paper": "This paper performs a careful study of the advantages of integrating model-based components into an RL system.  In particular, they clearly distinguish between the effects of learning a predictive model and learning a proposal distribution for a forward-sampling planner.   They sensibly focus on what seems like the \"sweet spot\" for model-based methods:  transfer of a model from one task to another in a domain with the same basic dynamics.   They find that, when using forward-sampling planners, the proposal distribution is the driving factor of success, so that learning and transferring a good model does not, in itself, improve performance much.", "summary_of_the_review": "Whether or not to accept this paper depends on what we think conference publications are for.  \n\nThis paper contributes some clear and well justified knowledge.  But the increment is small and it seems likely to me that substantially new and different techniques will arise soon and render this irrelevant.    It seems unlikely that this paper will inspire or excite new research.\n\nUltimately, this decision is above my pay grade, and depends on the program chairs' views about the role of paper acceptance.\n\nMy guess is that a good but not hugely exciting paper is thought to be on the positive side, so I'll lean positive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Cp6Fw0vuo-X", "reviewer": "Reviewer_i15j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPros:\n1. The quality of the experiment section is very good.\nThere are proper baselines, and the environments are well designed.\nThe videos are also very good, providing ideas how well the algorithms are.\n\nThe difficulty of the environments are much higher than the ones we currently use for model-based algorithms.\nI think they can be good baselines if the code is released along with the algorithm as a default baseline for future research.\n\n2. The related work section is very adequate.\n\nCons:\n\n1. Some details of the paper are missing.\nAnd since MPO consists of a very important role in the paper, it would be necessary to include a preliminary section.\n\n2. No code is released, and it is almost impossible to reproduce the results presented in the paper.\nFor the reproductivity, I don't think other researchers can reproduce the algorithms without access to the engineering details.\n\n3. It seems that the most obvious novelty of the paper is the use of MPO as the proposal policy network.\nIt seems rather arbitrary and I wonder how other model-free algorithms would have fit in the framework.\n", "summary_of_the_paper": "\nIn this paper, the authors follow the line of research of adding policy in online model-planning.\nThey consider multi-task / multi-goal environments, and use MPO as the proposal network, which improves the performance.\n", "summary_of_the_review": "\nI think in general the algorithm is very good. The environments are pretty interesting.\nThe release of code will be quite important to this project if it aims to encourage future research.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3JoTJot5ABO", "reviewer": "Reviewer_CeZd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n--\n- The paper investigates a topic that is interesting and timely\n- The results illustrate settings in which the quality of the policy proposals are very important to the final performance\n- The proposed hybrid method is straightforward\n- The paper includes exhaustive ablations\n\nWeaknesses\n--\n- W1 Some writing issues: **W1.1 Missing proposal terminology** and **W1.2 Dubious justification of model determinism**\n- W2 The presented evidence doesn't fully validate the claims: **W2.1 Missing evidence of computational burden**,  **W2.2 Narrow set of environments used**, and **W2.3 Narrow set of tasks used**\n\nWeakness 1:\n--\n-  **W1.1 Missing proposal terminology** The terminology \"proposal\" used in the abstract and introduction needs a clear definition if it's going to be used here. Is it a single action sampled from a policy? Is it a stochastic policy? Unfortunately, the current introduction forces the reader to guess.\n- **W1.2 Dubious justification of model determinism** S4.2 The use of deterministic models here is fine, but I'd argue that it's a reasonable assumption mainly because of determinism (I assume) in the true environment dynamics. If the MDPs investigated had significant stochasticity in the environment dynamics, then perhaps it would not be a great assumption, and performance would be significantly worse. S4.2 should be explicit about the degree of stochasticity in the actual environment dynamics. If the degree is insignificant (or none), it'd be great to get the authors' take on whether they agree that the main reason deterministic dynamics models work well is simply that the modeling class can represent the true dynamics models employed in the target MDPs.\n\nWeakness 2:\n--\n- **W2.1 Missing evidence of computational burden** S4.3 The claim that planning in these tasks cannot be performed alone with the ground-truth dynamics model and reward function requires more detail and requires more evidence. It is fair to assume that with enough search time/compute, a long-enough planning horizon, and access to ground-truth dynamics and rewards, the tasks can be solved e.g. via brute force search. Thus, the claim needs to be more specific about the computational restrictions that necessitate the use of a proposal-generating policy. This issue also appears in the first paragraph of S6 -- there's no actual discussion or experimentation of what constitutes 'limited' computation time. Because this claim is central to the paper's message -- that a good dynamics model is insufficient for transfer -- the paper would be much stronger if it presented evidence across a clear regime of computational time constraints that the result holds. For example, a plot of Performance vs. Computation time, with the 'optimal' and 'best proposal-based method' performance as single points (or sets of points), accompanied by the performance of MPC with GT dynamics and rewards with different computational budgets plotted as a curve. By the paper's claims, the latter MPC curve should be significantly below the 'optimal' and 'best proposal-based method' for reasonable computational regimes, and above the 'best proposal-based method' when an 'unreasonable' amount of compute is assumed (if feasible).\n- **W2.2 Narrow set of environments used** The employed set of multi-task evaluation environments in the paper is quite small. While it's good that the single-task environments are presented in the appendix, without a broader set of multi-task environments, we cannot draw broad conclusions about these methods in other multi-task settings\n- **W2.3 Narrow set of tasks used** The multi-task evaluation is limited to tasks in which the multi-task nature is target-goal reaching. Results presented on different MDPs in which the reward function variation is more sophisticated than goal reaching would provide more support for the claims.\n\nMinor weaknesses\n--\n- The DAGGER reference in 4.1 is too tangential (neither necessary nor particularly useful).\n- S4.3 Near \"We can then improve our proposal by minimizing the KL divergence\", the optimization variables need to be clarified, since it's unclear if the $\\pi_\\theta$ contained within $\\tilde{\\pi}_{\\mathcal B}$ is optimized. However, Eq. 1 clarifies this, so perhaps just call it a 'forward' KL divergence.\n- S4.1 says more details about the planner are in Sec B of the supplement, but the supplement didn't contain a .pdf (I found Sec B in the appendix of the .pdf)\n- Fig 2 needs more details in the caption or in the main text to describe the difference between 'actor' and 'proposal'. My understanding is that the 'actor' is the MPC actor if MPC is used, or equivalent to the policy (and the 'proposal') if MPC is not used.\n- Footnote 2 requires a citation or evidence for the claim \"MPO has a known problem with shrinking policy variances\". Furthermore, this hypothesis could be tested simply by adding a policy-entropy bonus to objective function, no?\n", "summary_of_the_paper": "This paper's goal is to evaluate the sample-efficiency of planning-based and policy-based approaches, as well as hybrid versions of each, in several multi-task continuous control environments. The paper shows that for the tasks investigated and the specific computational budget allocated to the planner, the planner performs significantly worse without a good proposal distribution (policy sampler), even when employed with ground-truth access to dynamics and rewards. Then, the paper presents multi-task experiments with variations of their proposed hybrid planning-amortized method that concurrently learns a proposal distribution, model, and critic; these experiments show that the MPC-based variants outperformed a model-free method to varying degrees.", "summary_of_the_review": "While the research question and initial evidence are promising, the main issue with the paper is Weakness 2 above, particularly W2.1. Evidence is required to support the claimed computational burden of planning with ground-truth dynamics model and rewards. Furthermore, the multi-task experimental results are only conducted in two environments (W2.2) and only for goal-directed tasks (W2.3), which doesn't support the claim that the quality of the proposal distribution is critical to planning in *general* **multi-task** settings; it instead only constitutes evidence that the quality of the proposal distribution is critical to planning in *some* **goal-directed tasks**. \n\nIn my opinion, this missing evidence weakens the paper slightly below the bar of acceptability.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper examines the advantage of using models in RL.  The authors' rebuttals convinced us of the value of the paper.
Decision: Accept (Poster)