Model-Based Offline Planning
Authors: Arthur Argenson
Gabriel Dulac-Arnold
Venue: ICLR 2021 Poster (2021)
Keywords: off-line reinforcement learning
model-based reinforcement learning
model-based control
reinforcement learning
model predictive control
robotics

Abstract: Offline learning is a key part of making reinforcement learning (RL) useable in real systems. Offline RL looks at scenarios where there is data from a system's operation, but no direct access to the system when learning a policy. Recent work on training RL policies from offline data has shown results both with model-free policies learned directly from the data, or with planning on top of learnt models of the data. Model-free policies tend to be more performant, but are more opaque, harder to command externally, and less easy to integrate into larger systems. We propose an offline learner that generates a model that can be used to control the system directly through planning. This allows us to have easily controllable policies directly from data, without ever interacting with the system. We show the performance of our algorithm, Model-Based Offline Planning (MBOP) on a series of robotics-inspired tasks, and demonstrate its ability leverage planning to respect environmental constraints. We are able to find near-optimal polices for certain simulated systems from as little as 50 seconds of real-time system interaction, and create zero-shot goal-conditioned policies on a series of environments.

Published as a conference paper at ICLR 2021
MODEL-BASED OFFLINE PLANNING
Arthur Argenson aa bargggoglle com Google Research
Gabriel Dulac-Arnold dulacarnol 1 @egoogle.oo com Google Research
ABSTRACT
Offline learning is a key part of making reinforcement learning (RL) useable in real systems. Offline RL looks at scenarios where there data from a system's 's operation, but no direct access to the system when learning policy, Recent work on training RL policies from offiin data has shown results both with model-free policies learned directly from the data, or with planning on top of learnt models of the data. Model-free policies tend to be more performant, but are more opaque, harder to command externally, and less easy to integrate into larger systems. We propose an offline learner that generates a model that can be used to control the system directly through planning. This allows to have easily controllable poli- cies directly from data, without ever interacting with the system. We show the per- formance of our algorithm, Model-Based Offline Planning (MROP) on series of robotics-inspired tasks, and demonstrate its ability to leverage planning to respect environmental constraints. We are able to find near-optimal polices for certain simulated systems from as little as 50 seconds of real-time system interaction, and create zero-shot goal-com condiionot policies on series of environments.
INTRODUCTION
Learnt policies for robotic and industrial systems have the potential to both increase existing sys- tems' efficiency robustness, as well as open possibilities for systems previously considered too complex to control. Learnt policies also afford the possibility for non-experts to program controllers for systems that would currently require weeks of specialized work. Currently, however, most ap- proaches for learning controllers require significant interactive time with system to be able to converge to a performant policy. This is often either undesirable or impossible due to operating cost, safety issues, or system availability. Fortunately, many systems are designed to log sufficient data about their state and control choices to create a dataset of operator commands and resulting system states. In these cases, controllers could be learned offline, using algorithms that produce a good controller using only these logs, without ever interacting with the system. In this paper we propose such algorithm, which we call Model-Based Offline Planning (MPOP), which is able to learn poli- cies directly from logs of a semi-performant controller without interacting with the corresponding environment. It is able leverage these logs generate more performant policy than the one used to generate the logs, which can subsequently be goal- coal-conditioned or constrained dynamically during system operation.
Learning from logs of a system is often called 'Offline Reinforcement Learning" (Wu et al.
2019
Peng et al.
2019;
Pujimoto al., 2019
Wang et al.
2020) and both model-free (Wu et al. 2019
Wang et al.
2020
Fujimoto et al.) 2019 Peng et al.
2019) and model- based (Yu et al.) 2020 Ki-
dambi al. 2020) approaches have been proposed to learn policies in this setting. Current model-
based approaches, MOPO (Yu al. (020) and MoREL (Kidambi et al. (020), learn model to train a model-free policy in a Dyna-like (Sutton& & Barto 2018) manner. Our proposed approach, MBOP, model-based approach that leverages Model-Predictive Control (MPC) (Rault et al. 1978) and ex- tends the MPPI (Williams et al. 7b) trajectory optimizer to provide a goal or reward-conditiona policy using real-time planning. It combines three main elements: learnt world model, learnt behavior-cloning policy, and a learnt fixed-horizon value-function.
MBOP's key advantages are its data-efficiency and adaptability. MBOP is able to learn policies that perform better than the demonstration data from as little as 100 seconds of simulated system time (equivalent to 5000 steps). A single trained MBOP policy can be conditioned with reward function,

---

Published as a conference paper at ICLR 2021
goal state, as well as state- based constraints, all of which can be non-stationary, allowing for easy control by a human operator or a hierarchical system. Given these two key advantages, we believe it to be a good candidate for real-world use in control systems with offline data.
We contextualize MBOP relative to existing work in Section 2 and describe MBOP in Section 3 In Section 4.2 demonstrate MBOP's performance on standard benchmark performance tasks for offlin RL., and in Section 4.3 we demonstrate MBOP's performance in zero-shot adaptation to varying task goals and constraints. In Section [4.4] we perform an ablation analysis and consider combined contributions of MBOP's 's various elements.
2 RELATED WORKS
Model-Based approaches with neural networks have shown promising results recent years. Guided Policy Search (Levine & Koltun 2013) leverages differential dynamic programming trajectory optimizer on locally linear models, and caches the resulting piece-wise policy in a neural network. [Williams al.] (201bb) show that simple model-based controller can quickly learn to drive a vehicle on a dirt track, the BADGR robot (Kahn et al., (020) also uses
2017a) with learned model to learn navigate to novel locations, Yang et al. (2020) show good results learning legged locomotion policies using MPC with learned models, and (Ebert et al. 2018) demonstrate flexible robot arm controllers leveraging learned models with image-based goals. [Silver et al. (2016) have shown the power of additional explicit planning in various board games including Go. More recently planning- baasdd algorithms such as PlaNet (Hafner et al. (0019b have shown strong results in pixel-based continuous control tasks by leveraging latent variational RNNs. Simpler approaches such as PDDM (Nagabandi et al.) 2020) or PETS (Chua et al.) 2018) have shown good results using full state information both in sim- ulation and on real robots. MBOP is strongly influenced by PDDM (Nagabandi et al. 2020) (itself an extension on PETS (Chua et al.) 2018)). in particular with the use of ensembles and how they leveraged during planning. PDDM was not designed for offline use, and MBOP adds a value func- tion composition well as a policy prior during planning to increase data efficiency and strengthen the set of priors for offline learning. It leverages the same trajectory re-weighting approach used in PDDM and takes advantage of its beta-mixture of the T trajectory buffer.
Path Integral (MPPI) (Williams et al.
Both MoREL (Kidambi et al., 2020) and MOPO Yuu et al.] 2020) leverage model-based ap- proaches for offline learning. T This is similar to approaches used in MBPO (Janner al. 2019) and DREAMER (Hafner et al.) 2019a), both of which leverage a learnt model to learn a model-free controller. MoREL and MOPO, however, due to their offline nature, train their model-free learner by using a surrogate MDP which penalizes for underlying model uncertainty. They do not use the models for direct planning on the problem, thus making the final policy task-specific. MOPO demonstrate the ability of their algorithm to alter the reward function and re-train new policy ac- cording to this reward, but cannot leverage the final policy to dynamically adapt to an arbitrary goal or constrained objective. Mattushhma et al. (2020) use a model- policy for deployment effi- cient RL. Their use case is a mix between offline and online RL, where they consider that there is a limited number of deployments. They share similarity in the sense that they also use behavior- cloning policy Ty to guide trajectories in a learned ensemble model, but perform policy improvement steps on a parametrized policy initialized from Ty using a behavior-regularized objective function. Similarly to MoREL and MOPO their approach learns parameterized policy for acting in the real system.
The use of a value function to extend the planning horizon of a blanning-based policy has been previ- ously proposed by Lowrey et al. (2018) with the POLO algorithm. POLO uses a ground-truth model (e.g. physics simulator) with MPPI/MPC for trajectory optimization. POLO additionally learns an approximate value-function through interaction with the environment which is then appended to op- timized trajectories to improve return estimation. Aside from the fact that MBOP uses an entirely approximate & learned model, it uses a similar idea but with a fixed-horizon value function to avoid bootstrapping, and separate heads of the ensemble during trajectory optimization. BC-trained poli- cies as sampling priors have been looked at by POPLIN (Wang & Ba 2019). POPLIN does not use value bootstrapping, and re-samples an ensemble head at each timestep during rollouts, which likely provides less consistent variations in simulated plans. They show strong results relative to a series of model-basee and model-free approaches, but do not manage to perform the Gym Walker envi-
2

---

Published as a conference paper at ICLR 2021
ronment. Additionally, they are overall much less data efficient than MBOP and do not demonstrate performance in the offline setting.
Task-time adaptation using model-based approaches has been considered previously in the model- based literature. [u et al. (219)) look at mixing model-free and model-based approaches using notions of uncertainty to allow for adaptive controllers for non-stationary problems. [Rajeswaran et al. (2020) use game-theoretic framework to describe two adaptive learners that are both more sample efficient than common MBRL algorithms, as well as being more robust non-stationary goals and system dynamics. MBOP is able to perform zero-shot adaptation to non-stationary goals and constraints, but does not provide mechanism for dealing with non-stationary dynamies. If brought into the on-line settings, approaches from these algorithms such as concentrating on recent data, could however be leveraged to allow for this.
Previous approaches all look at various elements present in MBOP but none consider the full combi- nation of BC prior on the trajectory optimizer with a value-function initialization, especially in the case of full offline learning. Along with this high-level design, many implementation details such as consistent ensemble sampling during rollouts, or averaging returns over ensemble heads, appear to be important for a stable controller from our experience.
3 MODEL-BASED OFFLINE PLANNING
Our proposed algorithm, MBOP MMoell-Based Offline Planning), is model-based RL algorithm able to produce performant policies entirely from logs of a less-performant policy, without ever interacting with the actual environment. MBOP learns a world model and leverages particle-based trajectory optimizer and model-preeicctvv control (MPC) to produce a control action conditioned on the current state. It can be seen an extension of PDDM (Nagabandi et al. 2020), with behavior- cloned policy used as a prior on action sampling, and a fixed-horizon value function used to extend the planning horizon.
In this following sections, we introduce the Markov Decision Process (MDP) formalism, briefly explain planning-based approaches, discuss offline learning, and then introduce the elements of MBCP before describing the algorithm full.
3.1 MARKOV DECISION PROCESS
Let us model our tasks as Markov Decision Process (MDP), which can be defined as a tuple (S, A. p,r, r,r,), where an agent is in a state E S and takes an action E A at timestep t. When in state 8t and taking an action at, an agent will arrive in a new state 8t41 with prob- ability p(s++1 (st, ac), and receive reward r(st, de, St+1). The cumulative reward over full episode called the return R and can be truncated to a specific horizon as Rg- Generally re- inforcement learning and control aim to provide an optimal policy function x² S - A which will provide an action de in state s; which will lead to the highest long-term return: N (st) = arg max,c.A Ees 2'r(St, x² (s.),, where y is time-wiss discounting factor that we fix to ? 1, and therefore only consider finite-horizon returns.
3.2 PLANNING WITH LEARNED MODELS
large body of the contemporary work with MDPs involves Reinforcement Learning (RL) Sutton & Barto (2018) with model-free policies Mnihh et al. (2015); Lillicrap et al. (2015); Schulman et al. (2017) Abdolmaleki et al. (2018). These approaches learn some form of policy network which provides its approximation of the best action de for a given state 8, often as single forward-pass of the network. MBOP and other model-based approaches Deisenroth & Rasmussen (201T); Chua ct al. (2018); Williams et al. (2017b); Hafner et al. (2019b); Lowrey et al. (2018); Nagabandi et al. (2020) are very different. They learn an approximate model of their environment and then use a planning algorithm to find high-return trajectory through this model, which is then applied to the environment
[
This is interesting because the final policy can be more easily adapted to new
This approach is often called Model-Based Reinforcement Learning (MBRL) in the literature, but we chose to talk more generally about planning with learned models as the presence of reward is not fundamentally necessary and the notion of reinforcement much less present.
3

---

Published as a conference paper at ICLR 2021
tasks, be made to respect constraints, or offer some level of explainability. When bringing learned controllers to industrial systems, many of these aspects are highly desireable, even to the expense of raw performance.
3.3 OFFLINE LEARNING
Most previous work in both reinforcement learning and planning with learned models has assumed repeated interactions with the target environment. This assumption allows the system to gather increased data along trajectories that are more likely, and more importantly to provides counter- factuals, able to contradict prediction errors in the learned policy, which is fundamental to policy improvement. In the case offline learning, we consider that the environment is not available dur- ing the learning phase, but rather that we are given dataset D of interactions with the environment, representing a series of timestep tuples (st; dey ra,, St+1). The goal is provide performant policy given this particular dataset D. Existing RL algorithms do not easily port over to the offline learn- ing setup, for varied set of reasons well-covered in Levine al. (2020). In our work, we use the real environment to benchmark the performance of the produced policy. It is important to point out that oftentimes there is nevertheless a need to evaluate the performance of given policy π without providing access to the final system, which is the concern of Off Policy Evaluation (OPE) Precup 22000); [Nacuum et all] (219)) and Offline Hyperparameter Selection(OHS) Paine et al. 2020 which outside the scope of our contribution.
3.4 LEARNING DYNAMICS, ACTION PRIORS, AND VALUES
MBOP uses three parameterized function approximators for its planning algorithm. These are:
1. fm A R. a single- siggeettmesteppp model of environment dynamics such that (Pr, St+1) (s, a.). This is the model used by the planning algorithm to roll out potential action trajectories. We will use fm (5,, az), to denote the state prediction and Im (8t: ot), for the reward prediction.
2. fo : :S SSX - A. behavior-cloned policy network which produces a, fo(#,, ae-1), 1), and is used by the planning algorithm as prior to guide trajectory sampling.
3. JR S x A + R is truncated value function, which provides the expected return over fixed horizon RH of taking specific action in a state RH fR(s),
Each one is bootstrap ensemble (Lakshminrayyaaan et al. 2017) of K feed-forward neural net- works, thus fm is composed of fmm vi E [1, K], where each fü is trained with a different weight initialization but from the same dataset D. This approach has been shown to work well empirically to stabilize planning (Nagabandi et al. 2020 Chua et al.] 2018)). Each of the ensemble member networks is optimized to minimize the Le loss on the predicted values in the dataset D in a standard supervised manner.
3.5 MBOP-POLTE
MBCP uses Model-Predictive Control (Rault et al., 1978) to provide actions for each new state as a, = (s)).. MPC works by running fixed-horizon planning algorithm at every timestep, which returns a trajectory T of length H. MPC selects the first action from this trajectory and returns it as planning algorithm is effectively black box to MPC, although in our case we have the MPC loop carry around a global trajectory buffer T. high-level view of the policy loop using MPC is provided in Algorithm
a, This
The MBOP-Policy loop is straightforward, and only needs to keep around T at each timestep. MPC well-known be surprisingly simple yet effective method for planning- control. Finding a good trajectory is however more complicated, as we will see in the next section.
3.6 MBOP-TRANE
MBCP-Trajopp extends ideas used by PDDM (Nagabandi et al. 2020) by adding a policy prior (provided by ft) and value prediction (provided by TR). The full algorithm is described in Algorithm
4

---

Published as a conference paper at ICLR 2021
Algorithm High-Level MBCP-Police
1: Let D be a dataset of E episodes
2: Train f'm fo, fn on D
3: Initialize planned trajectory: T 10..... ,0-1I..
4: for t 1..00 do
5: Observe 8t
6: '* MBOF -Tra Trap ,1, Sms So, fr)
& Update planned trajectory T" starting at To. b Use first action To as ((:))
7: T
8: end for
Algorithm 2 MBOP-Traajo
1: procedure MBCP-TRAJOP:(6.P2 fo, H, N, 2,
2: Set RN 0N
& This holds our N trajectory returns. This holds our action trajectories length H.
3: Set A N. = ÜN.H
4:
5:
for 1..N do
Sample N trajectories over horizon H. Use consistent ensemble head throughout trajectory.
6:
[ I= mod K
7:
$1 8, ag = To, R
8:
for = 1..H do
9:
~ NN0,,0²
d, fi(ss, 44 +
Sample current action using BC policy. Beta-mixture with previous trajectory T. b Sample next state from environment model. Take average reward over all ensemble members.
11:
An, = (1 3)a+ + (mitttittt.
8141 f'n (8t, An,)).
13:
R= R ΣK fin (8t, Anclr
end for
E 1k(sH:1, 15: R. R + K
1
Append predicted return and store.
16:
end for
T E RRa An,th Vt E 10, H -
17:
Generate return-weighted average trajectory.
EN enR
18:
return T"
19:
end procedure
2 In essence, MBCP-Tra is iterative sindeddshooting trajectory optimizer with refinement. MBCP-TRR -Bopotra rolls out N trajectories of length H using fm as an environment model. As fm is actually an ensemble with K members,
denote the 1th ensemble member fh Line of Alg. 2 allows the nith trajectory to always use the same /th ensemble member for both the BC policy and model steps. This use of consistent ensemble members for trajectory rollouts is inspired by PDDM. We point out that f' models return both state transitions and reward, and so we denote the state component as Jm (5e; (z), and the reward component as ( d:),-
The policy prior fů is used to sample an action which is then averaged with the corresponding action from the previous trajectory generated by 1B0P-Trajope jopt. By maintaining T' from one MPC step to another we maintain a trajectory prior that allows us to amortize trajectory optimization over time. The 3 parameter can be interpreted as a form of learning rate defining how quickly the current optimal trajectory should change with new rollout information (Wagener et al. 2019). We did not find any empirical advantage to the time-correlated noise in Nagabandi et al. (2220), instead opting for i.i.d. noise.
As opposed to the BC policy and environment model, reward model is caleulated using the average over all ensemble members to calculate the expected return R,, for trajectory At the end of a trajectory, we append the predicted return for the final state and action by averaging over all members of fn- The decision to take average of returns using the ensemble heads was also inspired by the approach used in Nagabandi et al. (2020).
Once have set of trajectories and their associated return, generate average action for timestep t by re-weighting the actions of each trajectory according their exponentiated return, as in Nagabandi al. (2020) and Williams al, (2017b) (Alg 3 Line 17).
5

---

Published as a conference paper at ICLR 2021
RWGGrppeeeSeeingeee Performance
RLUIQuadrupes///a Performance
000
100
593
Coledd Satum
600
Edlost
600
300
HOOP
200
D4Ca Painta
Dasa To
(a) Performance RLU RWRL Cartpole Dataset
(h) Performance RLU RWRL Quadruped Dataset
RLUWWker/WWak Performance
DoRLLndrat/Derrttpp Performance
CLONING
3200
030
2.500
Tploste
2400
202
KROP
CLONING
DATA
10350
18500
23000
11003
10030
250030
1001030 200000000000000000000000000
2300
400
2000
0003
10000 53000
200050 600000 1006000
(c) Performance RLU RWRL Walker Dataset
(d) Performance D4RL Adroit Door Dataset
Figure 1: Performance MBO? on various RLU and D4RL datasets. For each of the above tasks have sub-sampled subsets of the original dataset obtain the desired number of data points. The subsets are the same throughout the paper. The box plots describe the first quartile of the dataset, with the whiskers extending to the full distribution, with outliers plotted individually, using the standard Seaborn (more info here).
Section demonstrates how the combination of these elements makes our planning algorithm capa- ble generating improved trajectories over the behavior trajectories from D, especially in low-data regimes. In higher-data regimes, variants of MBOP without the BC prior also be used for goal & constraint-based control. Further work will consider the addition of goal-conditioned fo and fa to allow for more data-efficient goal and constraint-baser control.
EXPERIMENTAL RESULTS
We look at two operating scenarios to demonstrate MBOP performance and flexibility. First we consider the standard offline settings where the evaluation environment and task identical to the behavior policy's. s. We show that MBOP is able to perform well with very little data. We then look at M3OP's 's ability to provide controllers that can naturally transfer to novel tasks with the same system dynamics. We use both goal-conditioned tasks (that ignore the original reward function) and constrained tasks (that require optimising for the original reward under some state constraint) to demonstrate the MBOP's 's transfer abilities. Accompanying videos are available here: https: 7/youta..e/nxxGaaen
4.1 METHODOLOGY
We use standard datasets from the RL Unplugged (RLU) GGulchhre et al. 2020 and D4RL (Fu al. (020)) papers. For both RLU and D4RL., policies are trained from offline datasets and then evaluated on the corresponding environment. For datasets with high variance in performance, we discard episodes that are below certain threshold for the training fo and fR. This only done on the Quadruped and Walker tasks from RLU, and only provides slight performance boost performance on unfiltered data for these two tasks can be found in the Appendix ss5.6) The unfiltered data is always used for training f.. We perform grid-search dataset, but for most tasks these parameters are mostly uniform. The full set of parameters for each experiment can be found in the Appendix Sec. 5.2 For experiments RLU, we generated
find optimal parameters for each

---

Published as a conference paper at ICLR 2021
Dataset type
Environment
BC (Ours)
MBOP (Ours)
MOPO
MBPO
random
halfcheetah hopper
0.0 - 0.0
6.3 4.0
31.9 12.8
30.7 6.0
random
9.0 = 0.2
10.8 0.3
13.3
random
walker2d
0.1 = 0.0
8.1 = 5.5
13.0 2.6
8.6
medium
halfcheetah hopper
35.0 + 2.5
44.6 0.8
40.2 2.7
28.3 22.7
medium
48.1 26.2
48.8 = 26.8
26.5 3.7
4,9 + 3.3
medium
walker2d
15.4 - 24.7
41.0 = 29.4
14.0 10.1
12.7 + 7.6
mixed
halfcheetah
0.0 + 0.0
42.3 0.9
54.0 + 2.6
47.3 + 12.6
mixed
hopper
9.5
12.4 5.8
92.5 - 6.3
49.8 + 30.4
mixed
walker2d
11.5 = 7.3
9.7
42.7 - 8.3
22.2 + 12.7
med-expert
halfcheetah
90.8 + 26.9
105.9 + 17.8
57.9 + 24.8
9.7 + 9.5
med-expent hopper
15 = 8.7
55.1 1 44.3
51.7 + 42.9
56.0 1 34.5
med-expert
walker2d
65.5 40.2
70.2 36.2
55.0
19.1
7.6 1 3.7
2020) and MBPO (Janner et al. 2019). with values taken from the MOPO paper YYu al. 2020 As Fu al. 2020). normalize the scores according to a converged SAC policy, reported their appendix. Scores are reported averaged over random seeds, with 20 episode runs per seed. is one standard deviation and represents variance due to seed and episode. We have inserted our BC prior as the BC haseline, and have set performance to 0.0 when it is negative. We include the performance of bchavior cloning (BC) from the batch data for comparison. We bold the highest mean.
Table I:
Results for MBOP on D4RL tasks compare to MOPO Yu et al.
additional smaller datasets to increase the difficulty of the problem. On all plots we also report the performance of the behavior policy used to generate the data (directly from the episode returns in the datasets) and label it as the DATA policy. All non-standard datasets will be available publicly.
For RLU the datasets are generated using a 70% performant MPO Abdolmaleki et al. 2018) policy
on the original task, and smaller versions of the datasets are fixed set of randomly sampled con- tiguous episodes (Dulac-Amold et al. 2020; Gulcehre et al. 2020). D4RL has 4 behavior policies, ranging from random behavior to expert demonstrations, and fully described in Fu et al. (2020). On all datasets, training is performed on 90% of data and 10% is used for validation.
4.2 PERFORMANCE ON RL-UNPLUGGED & D4RL
For experiments on RLU we consider the unperturbed RWRL cartpole-swingue walker and quadruped tasks (Tassa et al., 2018 Dulac-Arnold et al. 2020). For D4RI. we consider the hal fcheetah, hopper, wa ker 2d and Adroit tasks (Brockman et al. 2016 Rajeswaran et al., 2017). Results for the RLU tasks as well as Adroit t are presented in Figure remaining D4RL tasks, results are compared to those presented by MOPO Yu et al. (2200) in Table for four different data regimes (medi um, medi um expert, medi um replay, random). For all experiments we report MBOP performance as well as the performance a behavior cloning (BC) policy. The BC policy is simply the policy prior ft, with the control action the average ensemble output. We use this baseline to demonstrate the advantages brought about by planning beyond simple cloning.
On the
For the RLU datasets (Fig. D. we observe that MBCP is able to find a near-optimal policy on most dataset sizes in Cartpole and Quadruped with as little 5000 steps, which corresponds to 5 episodes, or approximately 50 seconds on Cartpole and 100 seconds on Quadruped. On the Walker datasets MBOP requires 23 episodes (approx. 10 minutes) before finds a reasonable policy, and with sufficient data converges to score of 900 which is near optimal. On most tasks, MBOP is able to generate policy significantly better than the behavior data as well as the the BC prior.
For the Adroit t. task, we show that MBCP is able to outperform the behavior policy after training on dataset of 50k data points generated by expert policy (Fig. [d).. For other D4RL datasets, we compare to the performance of MOPO (Yu et al. 2020). We show that on the medium and medi um- expert data regimes MBOP outperforms MOPO, sometimes significantly. However on variance datasets such as random and mixed MBOP is not as performant. This is likely condcy-conditioned priors, which we hope to render more flexible in future
due to the reliance work (for instance using modal stochastic models). There are nevertheless many tasks where human operator is running systems in relatively consistent yet sub-optimal manner, and one
7

---

Published as a conference paper at ICLR 2021
may want to either replicate improve upon the operator's control policy. In such scenarios, MBOP would likely be able to not only replicate but improve upon the operator's control strategy.
4.3 ZERO-SHOT TASK ADAPTATION
Loff-constrained
Right-constranin constrained
Unconstrained
(a) Visualized trajectories for constrained Cartpole.
MBOP trajectories on Curtpole with withost constraint.
RLU Quadruped Mean Heading on Goal Conditiondd Quadruped
Constraint Left
Constrain: Nore.
Constrainc Right
900
Fewert
Sal Poston
Heading Angle
(b) RLU Cartpole trajectories per constraint type
(c) RLU Goal-Directed Quadruped
Figure 2: The above figures describe performance of MBOP on constrained & goal-conditioned tasks. Fig. 2a illustrates sequences of frames from the RLU Cartpole task with constrained and unconstrained MBOP controllers. the constrained cases MBOF prevents the cart from crossing the middle the rail (dotted line) and contains it to one side. Fig. displays cart trajectories for constrained and unconstrained versions of the same controller. MBOF can maintain a performant policy (above 750) while respecting these constraints. Fig. displays goal-conditioned performance the RLU Quadruped. We ignore the original reward function and optimize directly for trajectories that maximize particular velocity vector. Although influence from fs and fR biases the controller maintain forward direction, we can still exert significant goal-directed influence on the policy.
One of the main advantages of using planning-base methods in the offline scenario is that they are easy to adapt to new objective functions. In the case of MBOP these would be novel objectives different from those optimized by the behavior policy that generates the offline data. We can easily take these new objectives into account by computing secondary objective return as follows: R E. foes (8t) where fotj is user- provided function that computes a scalar objective reward given a state. We can then adapt the trajectory update rule to take into account the secondary objective:
EN R. + may R', And Vt [1.
T. Σ R. tRa, R'.
To demonstrate this, we run MBOP on two types of modified objectives: goal-conditioned control, and constrained control. In goal-conditioned control, we ignore the original reward function (x 0) and define new goal (such as a velocity vector) and optimize trajectories relative to that goal. In constrained operation, we add a state-based constraint which we penalize during planning, while maintaining the original objective and find reasonable combination of к and Kebj-
We define three tasks: Dosition-constraiin encourage to stay either on the right or left side the track; heading nendiing-onditionne Quadruped, where provide target heading to the policy (Forward, Backwards, Right & Left); and finally height-constrained Walker: where we penalize the policy for bringing the torso height above certain threshold. Results Cartpole & Quadruped are presented in Figure
Cartpole, where penalize the cart's position to
We show that MBOP successfully integrates constraints that were not initially in the dataset and is able perform well objectives that are different from the objective of the behavior policy.
8

---

Published as a conference paper at ICLR 2021
Walker performs similarly, obtaining nearly 80% constraint satisfaction while maintaining ward of 730. More analysis is available in the Appendix Sec. 5.5
4.4 ALGORITHMIC INVESTIGATIONS
Ablations To better understand the benefits of MBOP's 's various elements, we perform three ablations: MBCP-NOPP which replaces fb with a Gaussian prior, -BOP-NOVE which removes fR's estimated returns, and PDDM which removes both, thus recovering the PDDM controller. We show performance of these four ablations on the Walker dataset in Fig. 3a A full set of ablations is available in the appendix Figures 4 & 5 Overall we see that the full combination of BC prior, value function and environment model are important for optimal performance. We also see that the PDDM approach is generally below either of the MBOP-NOPP and MBOP-NOVE ablations. Finally, note that the BC prior when used alone can perform well on certain environments, but on others it stagnates at behavior policy's performance.
Execution Speed A frequent concem with planning- -baedd---adddd-- methods their slower response time prohibiting practical use. We calculate the average control frequency of MBOP on the RLU Walker task using a single Intel(() Xeo W-2135 CPU 3. 3.70GHZ core and a Nvidia 1080TI and find that MBOP can operate at frequencies ranging from 106 Hz for h = 4 to 40 Hz for h 40, with BC operating at 362 Hz. Additional values are presented Appendix Sec. 5.4
Hyperparameter Stability
We perform grid sweep over the x (trajectory re-weighting) and H (planning horizon) on the three RLU environments and visualize the ef- fects on return in Fig. 3bb We observe that overall MBOP maintains consistent performance scores for wide ranges of hyperparameter val- ues, only really degrading near extreme values. Additional analysis is present in the Appendix's ' Section 5.5
FOOM
DATK
2300
42090
2000
L009000
210000
(a) MBOP ablations' performance on RLU Walker Dataset. We observe that MBOP is consistently more performant than its ablations.
CONCLUSION
Planning- based methods provide significantly more flexibility for external systems to interact with the learned controller. Bringing them into the offline data regime opens the door to their use on more real-world systems for which on- line training is not option.
(b) MBOP scnsitivity to Kappa (k) and Horizon (H).
MBCP provides an easy to implement, data- efficient, stable, and flexible algorithm for pol- icy generation.
is easy to implement because the learning components are simple supervised learners, it is data-efficient thanks to its use of multiple complementary estimators, and it is flexible due to its use of on-line planning which allows it to dynamically react to changing goals, costs and environmental constraints.
We show that MBOP can perform competitively in various data regimes, and can provide easily adaptable policies for more complex goal-conditioned or constrained tasks, even if the original data does not provide prior experience. Although MBOP's 's performance is degraded when offline data multi-modal or downright random, we believe there are large number of scenarios where the current operating policy (be it human automated) is reasonably consistent, but could benefit from being automated and improved upon. In these scenarios we believe that MBOP could be readily appli- cable. Future work intends to ameliorate performance by investigating the use of goal-conditioned policy priors and value estimates, as well as looking at effective ways to perform offline model se- lection and evaluation. We sincerely hope that MBOP can be useful as out-of-the-bon algorithm for learning stable and configurable control policies for real systems.

---

Published as a conference paper at ICLR 2021
REFERENCES
Abhas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv:1806.06%: 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754-44544 4765, 2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-I1), pP- 2011.
Gabriel Dulac-Amold, Nir Levine. Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learn- ing. arXiv preprint arXiv:2003. 1881, 2020.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, pp. 2052-2062, 2019.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI un- plugged: Benchmarks for offline reinforcement learning. arXiv preprint 1XXi::2006.188800 2020.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:
2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learming. pp.
2019b.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, pp. 12519- 12530, 2019.
Gregory Kahn, Picter Abbeel, and Sergey Levine. Badgr: An autonomous self-supervised learning- based navigation system. arXiv preprint 2rXiv:2002.05700, 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.05951, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pp. 6402-646 6411,,13222 2017.
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference Machine Learning, pp. 1-9, 2013.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.001653 2020.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: /509 050. 009.00999991.... 2015.
10

---

Published as a conference paper at ICLR 2021
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv:1811.01888, 2018.
Kevin Lu, Igor Mordatch, and Pieter Abbeel. Adaptive online planning for continual lifelong learn- ing. arXiv preprint arXiv:1912.01 188, 2019.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofrr Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv:2006.03647, 2020.
arXiv preprint
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333, 2015.
Ofir Nachum, Yinlam Chow, Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pp. 2318-2328, 2019.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference Robot Learning, pp. 1101-1112, 2020.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna. Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint vrxivv2007.09055: 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off- aff-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 100::100900077 2017.
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. arXiv preprint arXiv:2004.07804 2020.
J Rault, A Richalet, JL Testud, and J Papon. Model predictive heuristic control: application to industrial processes. Automatica, 14(5):1134428, 1978.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:7007.06347. 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiesser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844489 2016.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 1rxiv:1800.006990 2018.
Nolan Wagener, Ching-An Cheng, Jacob Sacks, and Byron Boots, An online learning approach to model predictive control. arXiv preprint
2019.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint arXiv: 1904996666069, 2019.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.

---

Published as a conference paper at ICLR 2021
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv: 149, 2015.
Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory parallel computation. Journal of Guidance, Control, and Dynamics, 40 (2):344-357, 2017a.
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 1714-1721. IEEE, 2017b.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. CoRR, abs/1911.11361, 2019. /tLLtttt///axx...orr////////////
1t111.11166 11361
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Tingnan Zhang, Jie Tan, and Vikas Sindhwani. Data efficient reinforcement learning for legged robots. In Conference on Robot Learning, pp. 1-10, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13339, 2020.
12

---

Published as a conference paper at ICLR 2021
APPENDIX
5.1
MBOP PERTINENCE TO ROBOTICS
MBOP provides general model-based approach for offline learning. We have considered only physics-bound tasks in this paper the underlying methods (MPC. MPPI) are known to work well |015] Kahn et al., 2020). Although this pa- does not implement MBOP on actual robots, this is upcoming work, and we believe that by hav- ing shown MBOP's performance over different environments (cartpole, walker, quadruped, Adroit, halfcheetah, hopper) involving under-actuated control, locomotion, and manipulation, MBOP' 's po- tential for applicability on a real systems is promising. More specifically, we believe MBOP provides couple key contributions specifically interesting to the roboties community:
on real systems (Nagabandi et al. 2020; Williams al. 2013
Ability to learn entirely offline without simulator.
Ability to constrain policy operation.
Ability to completely rephrase the policy's goal according to an arbitrary cost function.
These aspects make MBOP unique contribution that potentially opens a series of interesting re- search questions around zero-shot adaptation, leveraging behavior priors, using sub-optimal models, leveraging uncertainty, and more generally exploring the additional control opportunities provided by model-based methods that are much more difficult with model-free learnt controllers.
As mentioned above it is our intent to quickly try out MBOP on various robotic systems. If results are available by the time of CoRL 2020 they will be presented as well.
5.2 PERFORMANCE OF MBOP ABLATIONS AND ASSOCIATED HYPERRAARAEEERSS
We present mean evaluation performance and associated hyper parameters for runs MBOP and its ablations in set of tables. For RLU: Table 2 for Cartpole, 3 for Quadruped, 4 for Walker. For D4RL:
# Points
Policy
Horizon
Samples
Kappa
Sigmna
Heta
Mcan
1-STD
5000
CLONING
229.2
71.7
5000
MBOP
64
100
2.34
0.2
803.1
117.7
MBOP-NOPP
605.8
223.6
5000
POOM
125
100
0.7
0.8
0.2
726.6
131.8
25000
CLONING
350.7
108.2
25000
MBOP
64
0.2
792.0
90.4
MBDP-NOPP
25000
MBOP NOWE
128
100
0.7
0.8
0.2
776.5
128.7
2500
PDDM
125
100
0.7
10.4
0.2
720
69.3
CLONING
103000
MEOP-NOPP
64
100
1.4
0.2
8.2
832.1
51.1
103000
HBOP-NOVE
128
100
1.2
1.6
0.2
733.6
150.3
100000
FDUM
723.5
CLONING
200000
MBOF
64
100
0.5
1.6
0.2
840.7
7.5
200000
MBOP-NOPP
64
100
2.3
0.2
0.2
840.6
12.4
200000
MBDF-NOVF
300000
CLONING
612.0
63.9
503000
MBOP
64
100
1.4
1.6
0.2
845.7
6.7
500000
MBDF-NOPP
64
MBOP-NOVF
873.2
300000
POOM
125
100
1.2
0.2
0.2
781.9
06.6
Table 2: RLU Cartpole Performance
13

---

Published as a conference paper at ICLR 2021
Policy
Horizon
Samples
Koppo
Signa
Beta
5000
MBOP
1000
3.8
0.8
0.2
974.1
9.9
5000
MBOR-NOPP
16
1000
1.9
1.6
0.2
561.6
233.2
PDDM
569.8
25000
MBOP
1000
3.8
0.8
983.8
3.6
25000
MBOP.NOPP
16
1000
1.9
1.6
0.2
966.66
$7.4
25000
MBOP NOVF
16
1000
0.8
983.0
1.6
120.7
CLONNNG
0.8
100000
MBOP-NOO NOPP
16
1000
1.9
1.6
0.2
935.5
35.2
100000
MBOP-NOVE
16
1000
1.9
0.8
0.2
983.8
2.2
CLUNING
8.6
200000
MBOP NOPP
16
1000
1.9
1.6
984.5
12.1
200000
BUBE
16
1000
1.9
0.8
0.2
986.6
1.3
200000
PDDM
946.4
29.7
$00000
CLONING
973.1
5.6
MBOP
500000
MBOP NOVF
16
1000
1.9
0.8
0.2
984.2
2.2
500000
PDDM
32
1000
0.9
1.6
965.0
12.0
Table RLU-Quadruped Performance
14

---

Published as a conference paper at ICLR 2021
# Points
Policy
Horizon
Samples
Koppu
Sigma
Beta
Moun
I-STD
10000
CLONING
244.5
284.6
1900
MBOP
100
3.8
0.2
0.2
251.2
250.4
MBOP-NOPP
MBOR-NOVE
225.3
275.8
10000
POOM
100
18.8
0.8
0.2
37.0
16.8
14000
CLONING
402.4
263.0
14000
MBOP
100
37,5
0.1
0.2
489.7
250.3
14000
MBOP-NOP
32
100
0.2
53.1
24.0
37.5
14000
PDDM
32
0.2
57.4
23.2
CLONING
616.7
23000
MBOP
16
100
1.9
0.2
0.2
679.0
200.
2300
MBOP-NOPP
100
18.8
00.8
0.2
103.0
56.2
23000
0.2
41000
MBOP
100
3.8
00.2
0.2
752.0
118.0
41000
MBOP SOPP NOPP
100
11.3
1.6
0.2
160.9
80.5
41000
MBOP NOVF
32
0.2
700.5
91.7
CLONING
615.7
240.6
275.0
50000
MBOP-NOPP NOPP
100
22.5
1.6
0.2
87.4
78.2
50000
MBOP-NOVE
16
100
9.4
0.2
0.2
723.2
103
250000
CLONING
686.8
205.4
250000
MBOP NOPP
22.5
0.2
155.9
250000
MBOP-NOR
16
100
9,4
0..2
012
770.4
115.1
250000
32
231.3
112.2
1000000
CLONING
201.5
190.9
MBOP-NOPP
22.5
411.2
1000000
MBOP-NO NOVF
16
100
9.4
0.2
0.2
814.3
$8.4
1000000
PDDM
32
100
0.2
308.2
140.3
2000000
CLONING
743.6
35.6
2000000
872.4
22.5
2000000
MBOP NOVE
16
100
0.2
0.2
807.8
44.2
2000000
PDOM
32
100
2.8
0.2
46613
117.6
5000000
CLONING
759.9
48.2
5000000
MBOP
100
7.5
0.4
0.2
908
54.3
3000000
MBOP-NOVE
833.5
100.7
5000000
PDDM
32
100
2,8
1.6
0.2
620
08.7
Table 4: RLU-Walker Performance
15

---

Published as a conference paper at ICLR 2021
Points
Policy
Hurizon
Samples
Karpa
Sigma
Betu
Meun
I-STD
630
CLONING
352.2
942.3
22.7
400
MBOP-NOPP
200
0.05
-54.6
230
NOVF
4
200
0.01
0.05
0
202.3
779.7
400
PDDM
500
0.05
-52.9
2000
CLONING
2889
579.7
100
198 6
2000
MBOP-NOVF
16
1000
0.03
0.1
2903.7
531.2
2000
PDDM
500
0.01
0.05
0.2
-53.0
0.6
4000
CLONING
3019.1
180.4
4000
MBOP
200
0.01
0.05
3043.4
64.3
4000
MBOP-NOVP
200
0.03
0.05
2991.9
702.7
4000
PDUM
200
0.03
0.06
0.2
-52.
0.6
10000
CLONING
2980.3
335.3
10000
MBOP
100
0.3
0.05
3026 1
180.7
MBOP-NOPP
500
10000
100
10000
PODM
100
0.01
0.05
0.2
-53.1
0.7
50000
CLONING
2984.5
313.4
50000
MBOP
1000
0.03
0.2
3028 2
197.6
50000
MBOP-NOPP
100
0.01
0.05
-53.4
30000
PDDM
500
0.01
0.1
0.2
-53.0
3028.0
200000
MBOP
1000
0.03
0.2
29670 0
355.0
2000000
MROP-NOPP
4
500
0.01
0.05
-52.9
0.8
200000
MBOF-NDVF
0.3
3024-2
PDDM
500
0.3
0.2
-59.7
400000
MBOP
100
0.3
0.1
3000
388.4
400000
MBOP- NOPP
64
100
0.03
0.4
-61.4
2.1
400000
MBOP-NOVF
200
0.3
0.1
3019.5
128.9
1000000
CLONING
1000000
0.2
579.6
1000000
MBOP-NOO NOPP
64
1000
0.01
0.4
-60.7
2.0
1000000
MRRO-NOVE
200
0.3
0.1
3015.6
241.6
1000000
PODM
100
Table 5: D4RL Door Performance
Datsset
Pulicy
Horizon
Samples
Kappu
Sigmu
Deta
Mean
I.STD
med-espent
CLONING
T10138
3259.7
med-expert
MROP
100
0.2
12850.7
2160.7
expert
med-expert
MBOP-NOVF
7230.3
3450.9
med expert
PODM
100
0.2
-165.2
35.8
mixed
CLONING
.60
1.6
mixed
MBOP
100
0.2
5135.1
107.9
MRDP-NOPP
542.8
mixed
POUM
20
0.2
-273.6
38.9
medivm
CLONING
4242.4
301.5
medium
MBOP
100
0.2
$406.5
96.6
modium
MBOP-NOPP
100
0.2
-43700
79.9
medium
PODM
20
-331.9
CLONING
mindon
MROP
100
0.8
768.4
491.2
randons
MBOP-NOPP
100
0.31
254.D
567.8
MBOP-NOVE
493.6
534.7
-156.7
110.1
Table 6: D4RL HalfCheetah Performance
16

---

Published as a conference paper at ICLR 2021
Dataser
Policy
Horizon
a Samples
Kappa
Sigma
Beta
Mean
I-TTD
atd expert
CLONING
480L6
2824
med-expen
medeespent
30.1
med-expent
MBOP-NUVF
100.0
0.01
0.0
1055.7
1300.4
med-cxpent
PDDM
100.0
0.01
0.0
123.5
36.3
medium
CLONING
1556.1
846.7
medium
MВОP
100.0
0.3
0.01
0.0
1576.7
866,1
medium
MBOP-NOVE
100.0
0.3
0.01
1679.
770.0
medium
PDUM
100.0
0,3
0.01
0.0
104.7
7.8
mixel
CLONING
308.2
223.2
mixed
MROP
100.0
0.3
0.02
0.0
40055
189.1
46.4
mixed
PDDM
150
100.0
0.3
0.02
0.0
101.6
36.5
randoca
CLONING
289.5
randoma
MBOP
100.0
10
0.4
0.0
350.1
9.5
rundom
MBOP-NOPP
100.0
0.4
81.8
42.3
rundoca
PDDM
100.0
0.4
0.0
44.2
12.0
Table 7: D4RL Hopper Performance
Dataset
Policy
Honzon
Samples
Kappa
Sugma
Beta
Mean
I-TTD
asd expent
CLONING
3000L0
18443
med-expent
3222.8
1660.7
exper
med expert
MBOP NOVE
1000
0.05
2302.7
1981,2
med-cxpert
PDDM
1000
0.05
0.2
2004
113.1
mised
CLONING
$28.7
335.0
mixel
MROP
1000
0.02
447.1
243.8
mixed
MBOP-NOVE
228.8
mixed
PDDM
10
1000
0.02
246.0
5.6
medius
CLONING
7008
1134.
medium
MEOP
1000
0.1
0.2
18819
1350.7
moduum
MROP-NOPP
0.1
medias
medium
PODM
150
1000
0.1
0.2
.2.7
10.3
rundoea
CLONING
rundon
MBOP
1000
0.3
0.4
0
371.1
252.3
zundom
MBOP-NOFP
1000
0.3
0.4
0
484.5
268.9
zundom
PDDM
1000
0.3
0.4
498.9
463.0
Table 8: D4RL Walker2d Performance
17

---

Published as a conference paper at ICLR 2021
5.3
MBOP ABLATIONS
Full results for the various ablations of MBOP are visualized in Figures 4 and S
poUd Parformance
WallWalkerWalk
1000
1020
420
400
MTOP
MSCP-XOPP
200
FOOM
MIOO ROVE
230
FOOM CLORING
CLOMING
DATA
DATA
5000
25000
1000R0
200300
500030
13000
14000
23000
41000
50000
252000
1300000
2010000
5003000
(a) MBOP variants performance on RL- (b) MBOP variants performance on RI-Unplugged RWRL Walker
Unplugged RWRL Quadruped Dataset
Dataset
DARLAAdroooDeooeaeeet Performance
Sustaag Performance
3200
300
3000
700
850
2800
500
KBOP
KBCP-MOP
Ma3e NUP
KBOP NOVF
POOM
CLONING
3500
20/0300
DATA
2000
DATA
16620 53000
Dsta Points
200030
402000 1000000
(c) MBOP variants performance on RL- (d) MBOP variants performance on D4RL Adroit door-expert- Dataset
Unplugged RWRL Cartpole Dataset
Figure 4: Ablation results multi-sized datasets form RLU and D4RL..
18

---

Published as a conference paper at ICLR 2021
Houlfcheeaah Performance
4050
D4RL/Hopper Performance
14000
H60F
MOGP MBDIP-NEV CLININGG
MB0F-40P9
1550
MROF-NOV
FDOM
10000
CLONING
DATA
CO Returs
25:0
8000
2000
1550
1000
1000
medum
Date
(a) MBOP variants performance D4RL HalfChee- (h) MBOP variants performance D4RI. Hopper
tah tasks
tasks
D4RL/Walker2d Performance
10000
1000
4000
2000
sapert
(c) MBOP variants performance on D4RL
Walker
tasks
Figure Performance on D4RL tasks from MBOP
5.4 EXECUTION SPEED
Policy
Horizon
Frequency (Hz)
BC
N/A
362
MBOP
106
MBOP
8
71
MBOP
16
40
Table MROP maximum control frequencies (steps/second) including simulator time on an Tesla P100 using single core of a Xeon 2200 MHz equivalent processor.
Execution speeds on the RLU Walker task in represented in Table 9 We see that we can easily achieve control frequencies below 10Hz, but cannot currently attain 100Hz with longer horizons. For lower level control policies for which high-frequency is important, we would suggest distilling the controller into task-specific policy similar to MoREL (Kidambi al. 2020 or MOPO Yu et al. 2020).
5.5
MBOP PARAMETERS
All parameters were set as follows except for the D4RL Walker task where we use 15 ensemble networks.
FC Layers 2
Size FC Layers 500
19

---

Published as a conference paper at ICLR 2021
Ensemble Networks:
Learning Rate: : 0.001
Batch Size: 512
Epochs 40
CONTINUED ANALYSIS OF CONSTRAINED TASKS
We can see the height-constrainee Walker performance in Figure 6a MBOP is able to satisfy the height constraint 80% of the episode while maintaining reasonable performance. Over the various ablations have found that MBOP is better able maintain base task performance for similar constraint satisfaction rates.
perfreances
Cartpolo Constraint
1050
feetige EXICh perkreand
Left constraint
Right constraint.
1.0
Palcy
Policy
380
750
Poicy
530
(a) This figure describes the performance of MBOP RLU Walker when constrained stay below a height threshold. We see that MBOP is able to increase the rate of respect of the constraint com- pared to the behavior policy while maintaining similar episode turns.
XX0
Paicy
Policy
(b) Cartpole task constrained right left half track. can see that MBOP is able to respect the constraint while maintain- ing performance on both tasks.
Figure Effects of constraints on MBOP performance.
HYPERPARAMETER STABILITY
Figure 7 shows the sensitivity of MBOP and associated ablations to the Beta and Horizon parameters. Figure shows the effects of Sigma to MBOP and ablations on the RLU datasets. Figure 6b shows sensitivity to Horizon and Kappa in synchrony.
5.6 IMPACT FILTERING POOR EPISODES
As mentioned in the above part of the paper, for RLU / Quadruped and RLU / Walker we exclude the episodes with lowest returns before training the behavior cloning and value function models. In this section we report the performances on these environment with various filtering thresholds.
For each of these two environments, and each of the dataset sizes keep subset the initial dataset by filtering on the top episodes. We experiments with filters varying from the top- % to the 100-100% (i.e. the entire raw dataset).
20

---

Published as a conference paper at ICLR 2021
Initial # datapoints
Filtered Top Percent
Mean
1-STD
5000
902
92
5000
5
908
64
5000
10
897
108
5000
20
916
13
5000
40
961
30
5000
60
951
53
5000
80
955
94
5000
90
966
12
5000
100
960
43
25000
809
268
25000
850
229
25000
10
973
49
25000
20
965
16
25000
40
744
329
25000
60
365
289
25000
80
320
262
25000
90
221
178
25000
100
115
69
100000
976
67
100000
5
986
100000
10
987
100000
20
989
100000
40
985
23
100000
60
876
230
100000
80
896
221
10000
90
921
177
100000
100
547
353
200000
989
2
200000
5
990
200000
10
993
2
200000
20
994
200000
40
991
2
200000
60
990
200000
80
878
254
200000
90
889
259
200000
100
876
252
500000
991
500000
5
992
500000
10
995
500000
20
994
500000
40
991
500000
60
992
2
500000
80
986
50
500000
90
991
500000
100
991
2
Table 10: MBOP performance on RLU/ / Quadruped with various filtering thresholds for top episodes
21

---

Published as a conference paper at ICLR 2021
Initial # datapoints
Filtered Top Percent
Mean
1-STD
50000
49
63
50000
5
86
112
50000
10
195
252
50000
20
243
293
50000
40
636
269
50000
60
750
182
50000
80
772
119
50000
90
719
201
50000
100
770
124
250000
111
113
250000
397
397
250000
10
410
406
250000
20
810
171
250000
40
848
42
250000
60
842
45
250000
80
836
46
250000
90
848
45
250000
100
838
43
1000000
154
201
1000000
5
670
348
1000000
10
870
88
1000000
20
858
101
1000000
40
858
63
1000000
60
859
97
1000000
80
851
47
1000000
90
847
53
1000000
100
855
46
2000000
618
386
2000000
741
348
2000000
10
859
194
2000000
20
876
111
2000000
40
867
62
2000000
60
860
59
2000000
80
888
55
2000000
90
873
60
2000000
100
858
61
5000000
639
404
5000000
5
875
179
5000000
10
909
37
5000000
20
907
49
5000000
40
892
60
5000000
60
892
58
5000000
80
853
54
5000000
90
875
63
5000000
100
863
65
Table 11: MBOP performance RLU / Walker with various filtering thresholds for top episodes
22

---

Published as a conference paper at ICLR 2021
1656
RUISE 3000
POTEE JCR0
1809
POTOI A.RUU
6.5
636
210
276
132
031
136
021
12
94
19
RAR P014
MAP 2014
(a) Sensitivity to Beta parameter on RLU/ Quadruped.
(b) Sensitivity to Beta parameter on RLU Walker.
Datu Phaece 7606
Barhhrr 134005
LOBOR
300
PARE 20000
000 PUTS .000
1007
400
06
(d) Sensitivity to Horizon parameter on RLU
(c) Sensitivity to Beta parameter on RLU / Cartpole Quadruped.
400
200
700
100300
110
Ma00
4/0
300/005
10/00
232
130
(e) Sensitivity to Horizon parameter on RLU/ / Walker. (1) Sensitivity to Horizon parameter on RLU/ Cartpole.
Figure 7: MBOP sensitivity to Beta & Horizon on RLU datasets.
23

---

Published as a conference paper at ICLR 2021
1050
3340
PUTAA 21000
CLLAA PALIAA 236000
Jetn
forafoorr 200800
Daak Perto 5000M
100x
(a) Sensitivity Quadruped.
10 Sigma parameter
on RLU /
(b) Sensitivity to Sigma parameter RLU Walker.
Upr vi0
(c) Sensitivity to Sigma parameter on RLU / Cartpole.
Figure 8: MBCP sensitivity to Sigma on RLU datasets.
24

---

Published as a conference paper at ICLR 2021
MSD senitaity Harize MLY- 199%
Cartedey HEAP
Wallor Meee
Fasa Ousdriped: AODAN000
FaUR Weiber INBONHCI
6:5 s:0
6.3 600 M. 150 301
03) 9:0 81
150 3/4
681 830 K.1 03
180 30x
60
4k sio
Figure 9: Sensitivity to Horizon x Kappa RLU environments (full datasets). Legend represents average episode return.
25


Reviews:
[{"review_id": "cxmHYQ2xy1m", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper leverages model predictive control (MPI) and proposes an approach which utilizes a transition+reward model, expected return model, and behavior-cloned policy to plan in a fully offline setting. The authors conduct experiments and ablation studies, showing the benefit of their approach in small data regimes and transferability to secondary objectives (e.g., constraints).\n\nPositive feedback:\n\n1. MBOP is an interesting approach to utilizing environmental model together with a behaviour-cloned policy in an MPI framework. Its main novelties w.r.t PDDM include the use of the behavior-cloned policy and an expected return model. \n2. MBOP shows to be a strong baseline for offline reinforcement learning, especially in small data regimes, which is especially interesting. Their ablation studies show the importance of using behavior cloning.\n3. Unlike other methods for offline-RL which first learn models and then use them in to solve the underlying MDP, MDOP in its essence allows to learn secondary objectives without the need to retrain the policy on the new MDP. The constrained objectives are especially interesting with this respect.\n\nQuestions and concerns:\n\n1. It was unclear (even after reading the information in the appendix) how the models f_m, f_R and f_b are trained (e.g., when and if ensembles are used, etc.)\n\n2. The method is fundamentally different than MOPO and MBPO, and it not clear if the reason for improvements are due to fundamental algorithmic factors. Some examples:\n- The authors use the same model head for simulating the full trajectory. \n- The use the mean reward and not the maximum reward as in MOPO. MBOP also learns an expected return model in addition to the reward model in MOPO. \n- Are the same transition models used in MBOP as in MOPO and MBPO? \nSuch factors may greatly affect performance and overall results. If these are the factors that affected the boost in performance, it would greatly reduce the general interest of the proposed approach. It is thus currently unclear if the comparison is fair.\n\n3. The authors chose to discard episodes that are below a certain threshold for the training of f_b and f_R. This raises a question as to the ability of the algorithm to utilize non-optimal data, e.g., dirty data that wasn’t collected by a quasi-optimal policy, or a mixture of policies. \n\n4. As MBOP utilizes behavioral cloning at its core, it would be interesting to see how it compares to behaviour regularized offline reinforcement learning: https://arxiv.org/pdf/1911.11361.pdf\n\n5. While I understand this work is a generalization of previously developed methods (MPPI, PDDM), some of its underlying assumptions are unclear. In MBOP actions are linearly interpolated (beta mixtures and weighted trajectories). When and why do the authors assume such interpolation should work? \n\n6. There are two flaws that make the applicability of MBOP somewhat problematic:\n\n- Lots of hyper parameters - unclear how these should be chosen. This includes the many different models which are not fully discussed.\n\n- Uncertainty of the algorithm is not taken into account by MBOP. I believe this to be an essential requirement for an offline-RL algorithm to be applicable, as no new experience can be collected.\n\n7. Finally, the clarity of the paper could greatly be improved. Overall it feels like the paper is not self contained and hard to understand. Many notations are not explained, and some of the taken approaches are unclear. I feel that Algorithm 2 should be explained better. The authors should provide more detailed information on what each line signifies and why it is an important feature of their approach. \nMinor question: What does \"min\" signify in T_{min} in the Algorithm 2, line 11?\n\n\n\nTypos:\n\nSection 2.4:\n\n- “…is a truncated value function, which *provides* the expected”\n\nSection 4.3: \n\n- “*One* of the main…”\n- “We can *then* adapt the trajectory…”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VtHEePTGtTq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper offered a model-based offline planning by taking advantage of learned behavior-cloning  policy and learned value function in addition to learning the model of the environment. Model learning is adapted from a work from Nagbandi et al. As the method takes advantage of behavior-cloning  policy, its performance boosts when the prior policy is reasonably well (medium or expert level). Moreover, learning value function helps the method to achieve higher rewards compared to the behavior-cloning  policy. Having said that, at least from the algorithmic point of view, this work does not seem to have a huge contribution. Moreover, it fails to provide a good solution compared to other model based learning methods when the behavior-cloning policy is not good (although it performs better than the BC policy). In fact, it looks like it would be a suitable method only when behavior-cloning  policy is in medium level (For the expert level vanilla BC gives comparable result). While this could have potential applications, the current work does not offer examples of it. If the authors agree to my conclusion about the medium level BC, finding (or even mentioning) the possible scenarios and environments could improve the impact of their work (And if they don't agree I am eager to know why). Moreover, it looks like due to their model-learning algorithm, the application of the method is mainly limited to robotics. \nAnother point I want to raise is the conceptual similarity of this work with Value Iteration Networks (VINs). I think VINs are mainly for discrete state space tasks, but I am curious to know authors' opinion on this. Moreover, I think it is useful to mention it in the paper as a related work.      \nAlso, the authors mentioned that  \"Along with this high-level design, many implementation details such as consistent ensemble sampling during rollouts, or averaging returns over ensemble heads, appear to be important for a stable controller from our  experience.\" which I believe is in favor of their potential contribution. However, in the conclusion they  said \"MBOP is an easy to implement [...] algorithm\". These two seem contradicting to me.   \n\nUpdate: \nI thank the authors for their response and increase my score by 1. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "UmIzMWT7WK2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \nThe paper proposes an algorithm to improve on a 'semi-performant' policy on a real world system from logged data of this policy driving the system (offline setting).\nThe algorithm build a MPC based controller using as ingredients learnt models of the dynamics, reward and value function along with a clone of the semi performant policy.\n\nThe algorithm mixes in an original fashion the following ingredients\n- soft reward weighted trajectory averaging to select the next action from sampled trajectories (PDDM).\n- reuse of past MPC trajectories (linear mixture with next guiding samples)\n- guided shooting with a cloned policy (POPLIN) \n- learning of ensembles to capture a notion of learnt model uncertainty (PETS)\n\n\n##########################################################################\n\nReasons for score: \n \nThe combination of the aforementioned ideas to the problem of offline policy improvement is original.\nThe paper is well written, the related work section is thorough the results  convincing.\n \n\n##########################################################################\nRemarks:\n \n1. The model based approach to off-line learning is a very promising avenue. The explicit separate representation of reward and dynamics provides great flexibility as shown in the adaptation to new objectives or additional constraints.\n \n2. The paper is close to PDDM as acknowledged by the authors in the introduction sections. \nThe ablation study is interesting and shows that the behavior cloning is the key factor in the algorithm.\nThe reader is referred to Figure 2 and the results not discussed at all. \nI guess it is a space issue, but I'd like to read it. \n\n3. In the ablation section, the author say they recover PDDM when removing the policy and the value function.\nI would suggest you give a more central place to this statement.\nThe RL literature is (incredibly) messy and discussing relation (inclusion, generalization, special case) between methods \nwhen available should be highlighted more.\n\n#########################################################################\n\nSome typos: \n\n(1) sec 2.1 'aim to to'", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hd-XkBsWjGW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThis work studies the offline RL problem and proposes MBOP for the same. The proposed method learns ensembles of dynamics models, behavioral policies, and value functions using the offline dataset. Subsequently, the approach uses online MPC with a learned terminal value function. The paper demonstrates experimental results on standard benchmark tasks (RLU and D4RL) as well as zero-shot adaptation results.\n\nThe strengths of the paper are showing results with an MPC-based approach and zero-shot adaptation results. My main concerns with the paper are exaggerated novelty, improper attribution and missing related works, and narrowly applicable approach. In the current form/version, I do not feel comfortable recommending acceptance. However, I am open to considering again if the authors can address the concerns during the rebuttal stage (which I expect will require a major revision).\n\n----\n\n(1) **Exaggerated novelty:** This paper must be considered in the context of a number of recent model-based offline RL papers. In particular, after MOReL and MOPO (which have been on arXiv for ~5 months before ICLR deadline), the use of model-based methods for offline RL, by itself, can no longer be considered a novelty. What sets this apart from MOReL and MOPO is the use of MPC vs a dyna style policy learning algorithm. This distinction and difference is not surprising or new, since a number of papers have explored this theme in the standard RL setting. However, these papers have not be adequately cited or discussed (e.g. guided policy search, POLO [1], AOP [2] etc). The MBRL tutorial (https://sites.google.com/view/mbrl-tutorial) has additional references too. I would recommend the authors rewrite the paper, to better situate the contributions of this work relative to what is known in literature. Right now, the first mentions of the most relevant prior work (MOReL and MOPO) appear only in page 5.\n\n(2) **Missing related work:** Pointers to relevant works are missing. Notable omissions include Dyna based MBRL algorithms (e.g. Game-MBRL [3] and MBPO [4]). This also ties with the previous point about exaggerated claims of novelty. Furthermore, Game-MRBL also explores the adaptability of MBRL in detail. When the dynamics is the same but the reward function changes, then the approach considered in MBOP will work. However, when the dynamics is inconsistent or non-stationary (e.g. data for locomotion collected on the road but robot enters dirt on sidewalk), then the MBOP approach would fail rapidly.\n\nThe planning algorithm in PDDM (filtering + reward refinement) is based on the MPC algorithm used in POLO [1]. I would recommend the authors to cite both papers for proper scientific dissemination of information. Line 11 in algorithm 2 can be interpreted as a \"learning rate\" to shift a_t from current value towards a target value. This was proposed and analyzed in the work of Wagener et al. [4]. I recommend the authors to cite and discuss this work. The Adroit tasks were proposed in [5] but this paper is not cited.\n\nLine 6 in Alg2 is equivalent to running a for loop over the models in the ensemble and an inner for loop running for N/K trajectories per model. Can the authors clarify if this is true? If so, the consistent trajectory sampling approach has been utilized in a number of prior works including Game-MBRL.\n\n(3) **Narrowly applicable solution approach** : The paper relies on the use of a behavior cloning policy for regularization and assumes a single consistent policy is used to obtain the offline dataset. If multiple policies are used to construct the dataset, will the MBOP approach work? Do you propose to learn multi-modal generative models for the policies? Do you have experimental results in this regime?\n\n----\n\n**Update after author response**\n\nI appreciate the authors efforts in the revision. This has addressed my concerns about the related work and (partly) exaggerated novelty. As a result I have increased the score from a 4 to 5. \n\nIn my opinion, the distinction between MPC and policy optimization is minor -- policy optimization when given enough iterations can match an optimal planner, and similarly MPC when equipped with a good terminal value function can match a policy learner. The practical differences between MPC and policy learning have also been extensively explored in prior work (e.g. GPS, POLO, AOP etc). Thus, while MBOP provides an interesting case study in the use of MPC for offline RL, there is limited novel insight. Nevertheless, I appreciate the authors for the thoroughness of the case studies, and for updating the paper based on reviewer feedback.\n\n---\n\n**References**\n\n[1] Lowrey et al. Plan Online Learn Offline. ICLR 2019.\n\n[2] Lu et al. Adaptive Online Planning for Continual Lifelong Learning. 2019.\n\n[3] Rajeswaran et al. A Game Theoretic Framework for Model Based Reinforcement Learning. ICML 2020.\n\n[4] Janner et al. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS 2019.\n\n[5] Wagener et al. An Online Learning Approach to Model Predictive Control. RSS 2019.\n\n[6] Rajeswaran et al. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. RSS 2018.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Quality:
The experimental evaluation is thorough and well designed. 

# Clarity:
After rebuttal, the paper is well written and clear in its contributions.

# Originality:
The proposed approach builds over existing literature, as clearly indicated in the manuscript and acknowledged by the authors in the rebuttal, by mixing existing contributions in a novel fashion.

# Significance of this work: 
This work deal with an important and timely topic. The experimental results are convincing and demonstrate strong performance for the proposed approach.

# Overall:
This manuscript provides an incremental but solid contribution to the topic of model-based reinforcement learning.

# Personal comments:
- I disagree with Reviewer2, in that I feel that after the rebuttal the manuscript clearly and thoroughly states prior works and the novelty of the proposed approach. This is not a survey paper and should be hold to realistic standards.
Decision: Accept (Poster)