MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
Authors: Nicklas Hansen
Yixin Lin
Hao Su
Xiaolong Wang
Vikash Kumar
Aravind Rajeswaran
Venue: ICLR 2023 poster (2023)
Keywords: model-based reinforcement learning
visual reinforcement learning
learning from demonstrations

Abstract: Poor sample efficiency continues to be the primary challenge for deployment of deep Reinforcement Learning (RL) algorithms for real-world applications, and in particular for visuo-motor control. Model-based RL has the potential to be highly sample efficient by concurrently learning a world model and using synthetic rollouts for planning and policy improvement. However, in practice, sample-efficient learning with model-based RL is bottlenecked by the exploration challenge. In this work, we find that leveraging just a handful of demonstrations can dramatically improve the sample-efficiency of model-based RL. Simply appending demonstrations to the interaction dataset, however, does not suffice. We identify key ingredients for leveraging demonstrations in model learning -- policy pretraining, targeted exploration, and oversampling of demonstration data -- which forms the three phases of our model-based RL framework. We empirically study three complex visuo-motor control domains and find that our method is 160%-250% more successful in completing sparse reward tasks compared to prior approaches in the low data regime (100k interaction steps, 5 demonstrations). Code and videos are available at https://nicklashansen.github.io/modemrl.

Published as a conference paper at ICLR 2023
MODEM: ACCELERATING VISUAL MODEL-BASED REINFORCEMENT LEARNING WITH DEMONSTRATIONS
Nicklas Hansen'2 Yixin Lin', Hao Suf, Xiaolong Wang",
Vikash Kumar',
Aravind Rajeswaran
Meta AI University of California San Diego
(nihansen, haosu, xiw012] @ucsd. edu w aravra @meta.
ABSTRACT
Poor sample efficiency continues to be the primary challenge for deployment of deep Reinforcement Learning (RL) algorithms for real-world applications, and in particular for visuo-motor control. Model-based RL has the potential be highly sample efficient by concurrently learning world model and using synthetic rollouts for planning and policy improvement. However, practice, sample-efficien learning with model-based RL is bottlenecked by the exploration challenge. In this work, we find that leveraging just handful demonstrations can dramatically improve the sample-effiiency of model RI.. Simply appending demonstrations to the interaction dataset, however, does not suffice. We identify key ingredients for leveraging demonstrations in model learning policy pretraining, targeted exploration, and oversampling of demonstration data which forms the three phases of our model-based RL framework. We empirically study three complex visuo-motor control domains and find that our method is 160% 250% successful in completing sparse reward tasks compared prior approaches in the low data regime (100k interaction steps, 5 demonstrations) Code and videos are available at https::///// /eee://nniiii iashannsennnnn gá github. ic /nodemr...
INTRODUCTION
Reinforcement Learning (RL) provides a princi- pled and complete abstraction for training agents in unknown environments. However, poor sam- ple efficiency of existing algorithms prevent their applicability for real-world tasks like object manip- ulation with robots. This further exacerbated in visuo-motor control tasks which present both the challenges of visual representation learning as well motor control. Model-base based RL (MBRL) in principle (Brafman & Tennenholtz, 2002) improve the sample efficiency of RI. by concurrently learn- ing a world model and policy (Ha Schmidhuber, 2018; Ecoffet al., 2019; Schritwieser et al., 2020; Hafner et al., 2020; Hansen et al., 2022a). Use of imaginary rollouts from the learned model can re- duce the need for real environment interactions, and thus improve sample efficiency. However, series of practical challenges like the difficulty of exploration, the need for shaped rewards, and the need for a high-quality visual representation, prevent MBRL from realizing its full potential. In this work, seek to overcome these challenges from practical standpoint, and we propose to do so by using expert demonstrations to accelerate MBRL.
Adroit
Meta-World
53
81
31
15
16
3
3
BC+R3M (IL)
TD-MPC (MBRL) Ours (MBRL+IL)
FERM (MFRL+IL)
Figure Success rate (%) in sparse reward tasks. Given only 5 human demonstrations and limited amount of online interaction, our method significantly improves success rate 21 hard robotics tasks from pixels including dexterous manipulation, pick-and-place, and lo- comotion compared to strong baselines.
Expert demonstrations for visuo-motor control tasks can be collected using human teleoperation, kinesthetic teaching, or scripted policies. While demonstrations provide direct supervision for learning complex behaviors, they be costly to collect in large quantities (Baker et al., 2022). However, even a small number of demonstrations can significantly accelerate RL by circumventing challenges related to exploration. Prior works have studied this in the context of model-free RL. (MFRL)

---

Published as a conference paper at ICLR 2023
action
Vo
å
V
å
V2
environmentt
8
polley
latent state
buffer
h
encoder
h
50%
%
G2
50%
X
demos
X
X4 X2
(1) Pretrain policy with BC on demonstration data
(2) Seed buffer with rollouts
(3) Collect rollouts using model, learn model with data from all phases; oversample demonstration data
from pretrained policy
Figure 2. Our framework (MoDem) consists of three phases: (1) a policy pretraining phase where representation and policy is trained on a handful of demonstrations via BC, (2) seeding phase where the pretrained policy is used to generate rollouts for targeted model learning, and (3) an interactive learning phase where the model iteratively collects new rollouts and is trained with data from all three phases. Crucially, we aggressively oversample demonstration data for model learning, regularize the model using data augmentation, and reuse weights across phases, sg. stop-gres -ap-grrdient operator.
algorithms (Rajeswaran et al., 2018; Shah & Kumar, 2021; Zhan et al., 2020). In this work, we propose a new framework to accelerate model-based RL algorithms with demonstrations. On suite of challenging visuo-motor control tasks, we find that our method can train policies that are approx. 160% 250% more successful than prior state-of state-of-the-art (SOTA) baselines.
Off-Policy RL algorithms (Sutton Barto, 1998) - both model-based and model-free - can in principle admit any dataset in the replay buffer. Consequently, it is tempting to naïvely append demonstrations to the replay buffer of an agent. However, we show that this a poor choice (see Section 4), since the agent still starts with a random policy and must slowly incorporate the behavioral priors inherent in the demonstrations while learning the environment. Simply initializing the policy by behavior cloning (Pomerleau, 1988) the demonstrations is also insufficient. Any future learning of the policy is directly impacted by the quality of world model and/or critic training of which requires sufficiently exploratory datasets. To circumvent these challenges and enable stable and monotonic, yet sample-efficient learning, we propose Model-based Reinforcement Learning with Demonstrations (MoDem), three-phase framework for visual model-based RL using only handful of demonstrations. Our framework is summarized in Figure 2 and consists of:
Phase 1: Policy pretraining, where the visual representation and policy pretrained the demonstration dataset via behavior cloning (BC). While this pretraining by itself does not produce successful policies, it provides strong prior through initialization.
Phase 2: Seeding, where the pretrained policy, with added exploration, used to collect small dataset from the environment. This dataset is used to pretrain the world model and critic. Empirically, data collected by the pretrained policy far more useful for model and critic learning than random policies used in prior work, and is key to the success of our work as it ensures that the world model and critic benefit from the inductive biases provided by demonstrations. Without this phase, interactive learning quickly cause policy collapse after the first few iterations of training, consequently erasing the benefits of policy pretraining.
Phase 3: Finetuning with interactive learning, where we interleave policy learning using synthetic rollouts and world model learning using data from all three phases including fresh environment interactions. Crucially, we aggressively oversample demonstration data during world model learning, and regularize with data augmentation in all phases.
Our Contributions. Our primary contribution in this work is the development of MoDem, which we evaluate on 18 challenging visual manipulation tasks from Adroit (Rajeswaran et al., 2018) and Meta-World (Yu et al., 2019) suites with only sparse rewards, as well as locomotion tasks from DMControl (Tassa et al., 2018) that use dense rewards. Measured in terms of policy success after 100k interaction steps (and using just 5 demonstrations), MoDem achieves 160% 250% higher relative success and 38% 50% higher absolute success compared strong baselines. Through extensive empirical evaluations, we also elucidate the importance of each phase of MoDem, as well as the role of data augmentations and pre-trained visual representations.
2

---

Published as a conference paper at ICLR 2023
2 PRELIMINARIES
We start by formalizing our problem setting, and then introduce two algorithms, TD-MPC and Behav- ior Cloning (BC), that were previously proposed for online RL and imitation learning, respectively, and are both central to our work.
Problem Setting We model interaction between an agent and its environment as an infinite-horizon Markov Decision Process (MDP) defined as a tuple M == AS, T, R, where S E R", A R" are continuous state and action spaces, T: S x A + S is the environment transition function, R: S A + IR is a scalar reward function, and 7 E [0, 1) is the discount factor. T, R. are assumed to be unknown. The goal for the agent is to learn policy To: S - AA parameterized by A that achieves high long term performance, i.c. max, Ex Der= 7'r:]. while using as few interactions with the environment as possible referred to sample efficiency. In this work, we focus more specifically on high-dimensional MDPs with sparse rewards. This is motivated by applications in robotics and Embodied Intelligence where the state is not directly observable, but can be well-approximated through the combination: s = (x, q), where x denotes stacked RGB images observed by the agent's s camera, and q denotes proprioceptive sensory information, e.g., the joint pose of robot. Furthermore, shaped reward functions can be hard to script for real-world applications (Singh et al., 2019) or result in undesirable artifacts or behaviors (Amodei et al., 2016; Burda al., 2019). Thus, we desire to learn with simple sparse rewards that accurately capture task completion. The principal challenges with sample efficient learning in such MDPs are exploration due sparse rewards, and learning good representations of the high- high-dimensional state space (e.g. images). To help overcome these challenges, we assume access to a small number of successful demonstration trajectories Ddemo = (D1, D2...., Dv }. obtained through human teleoperation, kinesthetic teaching, or other means. Our goal is to learn a successful policy using minimal environment interactions, by accelerating the learning process using the provided demonstration trajectories.
Behavior Cloning (BC) is a simple and widely used framework for imitation learning (IL). This relies entirely on Deeme and presents an ideal scenario if successful policies can be trained since it incurs zero interaction sample complexity. BC trains a parameterized policy na : S ++ A to predict the demonstrated action from the corresponding observation (Pomerleau, 1988; Atkeson & Schaal, 1997). Behavior cloning with small dataset is known to be challenging due to covariate shift (Ross et al., 201 la; Rajeswaran et al., 2018) and the challenge of visual representation learning (Parisi et al., 2022; Nair et al., 2022). Further, collecting a large demonstration dataset is often infeasible due to the required human costs and expertise (Baker al., 2022). More importantly, BC cannot improve beyond the capabilities of the demonstrator since it lacks notion of task success. Together, these considerations motivate the need for combining demonstrations with sample-efficient RL.
TD-MPC (Hansen et al., 2022a) is model-based RL algorithm that combines Model Predictive Control (MPC), learned latent-space world model, and a terminal value function learned via temporal difference (TD) learning. Specifically, TD-MPC learns a representation z hg(s)) that maps the high-dimensional state (s) into a compact representation, and a dynamics model in this latent space z' = dg (z. a). In addition. TD-MPC also learns prediction heads, RR, Qe, To. for: (i) instantancous reward r = Rg (z, a). (ii) statc-action value Qe(z, a). and (iii) action a ~ Te (z)... The policy ma serves "guide" planning towards high-return trajectories, and is optimized weighted Q-values. The remaining components are jointly optimized to minimize TD-errors, reward prediction errors, and latent state prediction errors. The objective is given by
maximize temporally
LID-MPC (s. s')r,s))a+))) )) = Σ i=
[100 - (r: + 200ll/ (0
(I)
+ ||oo a.) - r ldd a.) - (s)
2, = ha(s), z.41 do a.).
where 0 an exponential moving average of 0, and s/ z/ are the (latent) states at time + 1. During environment interaction, TD-MPC uses sample-based planner (Williams et al., 2015) in conjunction with the learned latent world-model and value function (critic) for action selection. See Appendix B choice of visual MBRL algorithm
for additional background TD-MPC. We use TD-MPC due to its simplicity and strong empirical performance, but our framework, in principle, can be instantiated with any MBRL algorithm.
3

---

Published as a conference paper at ICLR 2023
3 MODEL-BASED REINFORCEMENT LEARNING WITH DEMONSTRATIONS
In this work, our goal is to accelerate the sample efficiency of (visual) model-based RL with a handful of demonstrations. To this end, propose Model-based Reinforcement Learning with Demonstrations (MoDem), simple and intuitive framework for visual RI. under a strict environment- interaction budget. Figure 2 provides overview of method. MoDem consists of three phases: (1) policy pretraining phase where the policy is trained on a handful of demonstrations via behavior cloning, (2) seeding phase where the pretrained policy is used collect small dataset for targeted world-model learning, and (3) an interactive learning phase where the agent iteratively collects new data and improves using data from all the phases, with special emphasis on the demonstration data. We describe each phase in more detail below.
Phase policy pretraining. We start by learning a policy from the demonstration dataset Delemo == [D1, D2...., DN} where each demonstration D. consists of (So, ag $1, "1,a..... 8T, ar). In general, the demonstrations may noisy or sub-optimal we do not explicitly make any optimality assumptions. Let ha: S R' denote the encoder and :h I'' + A denote the policy that maps from the latent state representation to the action space. In Phase 1, we pretrain both the policy and encoder using a behavior-cloning objective, given by
Cp1 (0) (s. (s.s)) lo log x (a
(2)
When To(-(2) is parameterized by an isotropic Gaussian distribution, as commonly used in practice, Eq. simplifies to the standard MSE loss. As outlined in Section 2, behavior cloning with small demonstration dataset is known to be difficult, especially from high-dimensional visual observa- tions (Duan et al., 2017; Jang al., 2021; Parisi al., 2022). In Section 4, indeed show that behavior cloning alone cannot train successful policies for the environments and datasets we study, even when using pre-trained visual representations (Parisi al., 2022; Nair et al., 2022). However, policy pretraining can provide strong inductive priors that facilitate sample-efficient adaptation in subsequent phases outlined below (Peters et al., 2010).
Algorithm 1 Model-Based Reinforcement Learning with Demonstrations (MoDem)
Require: O: randomly initialized parameters
D B: demonstrations, (empty) replay buffer
ne, Ne, llg policy, planning procedure
// Phase 1: policy pretraining
// Behavior cloning on demonstrations
1: for each policy update do
2: {st, )r Ddemo // Sample state-action pair from demos
3: Update he, Te by (p) // BC objective, Equation 2
// Phase seeding
// Pretrain model with rollouts from pretrained policy
4: for each seeding rollout do
5: T {st. at, rt, St+1 forr where ~ To(s;) // Act with pretrained policy and S,+1 1 ~ T(se, az), Ti ~ as) Deced f- Diced UT Add rollout to seeding dataset
6:
7: for each model update do {st, ag, re, St+1 I1:+ ~ (Ddemo D // Sample demos seeding, oversample demos Update hos dg. Ro, Qs. Fa by Lp2(f) // Model objective, Equation // Phase 3: interactive learning
8:
9:
// Collect rollouts by planning and finetune model
B Deoed
11: while interaction limit is not reached do
T + (st, a,, Ft, St+1 to:r where a, ~ [o(ss)) Planning with model and St+1 ~ T((s, az), ~ R(s az) B-BUT Add rollout to replay buffer
13:
14:
{st, Te, St+1 hest ~ (Ddemo UB) // Sample demos buffer, oversample demos
Update hep dg. Rg, Qo. To by L.3 // Model objective, Equation
16: Done! // Enjoy your visual latent dynamics model
4

---

Published as a conference paper at ICLR 2023
Phase 2: seeding. In the previous phase, only pretrained the policy. In Phase 2, our goal is to also pretrain the critic and world-model, which requires a "seeding" dataset with sufficient exploration. A random policy is conventionally used to collect such dataset in algorithms like TD-MPC. However, for visual RI. tasks with sparse rewards, a random policy is unlikely to yield successful trajectories or visit task-relevant parts of the state space. Thus,
collect small dataset with additive exploration using the policy from phase 1. Concretely, given Ty P1 and h from the first phase, we collect a dataset P1 P1 (ng (s)). To ensure sufficient variability in trajectories, we add Gaussian noise to actions (Hansen et al., 2022a). Let & (8,, a; ri. s)) be a generic trajectory snippet of length H. this phase, we learn wo, ho, de, Ro: Qo the policy, representation, dynamics, reward, and value models by minimizing the loss
DDeed {TI, 72, TK) J by rolling out To
(3) where is an "oversampling" rate that provides more weight to the demonstration dataset. In summary; the seeding phase plays the key role of initializing the world model, reward, and critic the task-relevant parts of the environment, both through data collection and demonstration oversampling. We find in Section 4 that the seeding phase is crucial for sample-efficient learning, without which the learning agent is unable to make best use of the inductive priors in the demonstrations.
(0) = = Bgg (CTD-MPO( E.) (1 11--) -R) EggeDieed [CTD- MPC-MMPC(0,C.) (0,
Phase 3: interactive learning. After initial pretraining of model and policy, we continue to improve the agent using fresh interactions with the environment. To do so, we initialize the replay buffer from the second phase, i.c. B < Ducedi A naïve approach to utilizing the demonstrations is to simply append them the replay buffer. However, find this to be ineffective in practice, since online interaction data quickly outnumbers demonstrations. In line with the seeding phase, we propose to aggressively oversample demonstration data throughout training, but progressively anneal away the oversampling through the course of training. Concretely, we minimize the loss
(ps
EDi [CTD-MPCl { + +11 (1-m)) x) B55,
(0, &)]
(4)
Finally, we find it highly effective to regularize the visual representation using data augmentation, which we apply in all phases. We detail our proposed algorithm in Algorithm 1.
RESULTS & DISCUSSION
Environments and Evaluation For perimental evaluation, we consider 21 challeng- ing visual control tasks. This includes 3 dex- terous hand manipulation tasks from the Adroit suite (Rajeswaran et al., 2018), 15 manipulation tasks from Meta- World, as well as 3 locomotion tasks involving high-dimensional embodiments from DMControl (Tassa et al., 2018). Figure 3
Table 1. Experimental setup. We consider a total of 21 visual RL tasks spanning 3 domains.
Domain
Tasks
Demos
Interactions
Reward
Adroit
3
5
100k
Sparse
Meta-World
15
5
100k
Sparse
DMControl
3
5
100k
Dense
provides illustrative frames from some of these tasks. For Adroit and Meta- Horld, we use sparse task completion rewards instead of comaan-shaped rewards. We use DM-Control illustrate that MoDem provides significant sample-efficient of Meta- Meta-World, study diverse collection of medium, hard, and very hard tasks as categorized by Seo et al. (2022). We put strong emphasis on sample-efficinccy and evaluate methods under an extremely constrained budget of only 5 demonstrations' and 100k online interactions, which translates to approximately one hour of real-robot time in Adroit. We summarize our experimental setup in Table All of our tasks, demonstrations, and an implementation of our method is available at ht bttps: / //nckki walckkasssaann Lasaansennnnn. gi thub. io //nodnnor//../// 1. Through experimental evaluation, we aim to study the following questions:
gains even for visual RL tasks with dense rewards. In the case
1. Can MoDem effectively accelerate model-based RL with demonstrations and lead to sample- efficient learning in complex visual RL tasks with sparse rewards?
2. What is the relative importance and contributions of each phase of MoDem?
3. How sensitive is MoDem to the source of demonstration data?
4. Does MoDem benefit from the use of pretrained visual representations?
Fach demonstration corresponds 50-500 online interaction steps, depending the task.
5

---

Published as a conference paper at ICLR 2023
Figure 3. Tasks. We evaluate methods total of 21 challenging image-based tasks spanning three domains (Adroit (Rajeswaran et al., 2018), Meta- World (Yu et al., 2019), DMControl (Tassa et al., 2018)). Observations contain rich visual features such as textures and shading, and require particularly fine- fine-grained control due to complex geometry. See Appendix D and for additional visualizations and full list of tasks.
raw high-resouution (224 x 224) RGB frames (pictured). Environments
Baselines for Comparison We consider set of strong baselines from prior work both visual II., model-free RL (MFRL) with demonstrations, and visual model-based RI. (MBRL). We list the most relevant prior work in Table 2 and select the strongest and most repre- sentative methods from each category. Specifically, we select the following baselines: (I) BC R3M that leverages the pretrained R3M visual representation (Nair et al., 2022) to train policy by behavior cloning the demonstration dataset. (2) state-based (oracle) DAPG (Rajeswaran et al., 2018) that regularizes a policy gradient method with demonstrations. (3) FERM (Zhan et al., 2020) com- bines model-free RL, contrastive rep- resentation learning, and imitation learning. It first performs contrastive pretraining demonstrations, then trains a model-free SAC (Haarnoja al., 2018) agent with online envi- ronment interactions while incorpo- rating random image augmentations. Finally, we also compare with (4) TD- MPC (Hansen et al., 2022a) instan- tiated both with and without demon- strations. Notably, we choose to com- pare to R3M it has been shown to improve over both end-to-end BC and other pretrained representations in sev- eral robotic domains, and we compare DAPG as it also serves as a state- based oracle for RRL (Shah & Kumar, 2021). We also compare to FERM which is conceptually closest to our method and has demonstrated great -finile-fficiency in real-robot manip- ulation tasks. Furthermore, it is also superset of SOTA model-free methods that do not leverage demonstrations (Srinivas et al., 2020; Laskin et al., 2020; Kostrikov al., 2020; Yarats al., 2021a). Lastly, our TD-MPC baseline appends demonstrations to the replay buffer at the start of training following Zhan et al. (2020) and can thus be interpreted as a model-based analogue of FERM (but without contrastive pretraining). We evaluate method (e.g., frame stacking, action repeat,
Table 2. Baselines. Given that our problem setting is at the intersection of IL and online RL, there are many poten- tial baselines. We categorize prior work into three distinct categories, and choose the best and most representative meth- ods from each category as our main points of comparison. Note that the listed model-based RI algorithms do not boot- strap from demonstrations in their original papers, but can be adapted by appending demonstrations to replay buffer. Selected baselines are in bold.
Category IL
Method
End-to-End BC (Atkeson & Schaal, 1997)
R3M (Nair al., 2022)
VINN (Pari
2022)
Model-free RL IL
DAPG Rajeswaran al. (2018)
FERM (Zhan al., 2020)
RRL (Shah & Kumar, 2021)
GAIL & Ermon, 2016)
AWAC (Nair al., 2020)
HER demos Nair
(2018)
VRL3 (Wang et al., 2022)
Model-based RL
TD-MPC (Hansen et al., 2022a)
Dreamer- V2 (Hafner al., 2020)
MWM (Seo al., 2022)
all baselines under the same experimental setup data augmentation, and access to robot state) for a fair comparison.
Benchmark Results Our main results are summarized in Figure 4. Our method achieves average success rate of 53% at 100k steps across Adroit tasks, whereas all baselines fail to achieve any non- trivial results under this sample budget. FERM solves a small subset of the Merl-World tasks, whereas our method solves all 15 tasks; see Figure 5 for individual task curves. We find that our TD-MPC and FERM baselines fare relatively better in we still observe that MoDem outperforms all baselines. Finally, we also find that behavior cloning with pretrained visual representations is ineffective with just 5 demonstrations.
which uses dense rewards. Nevertheless,

---

Published as a conference paper at ICLR 2023
Adroit (sparse)
Meta- Meta-World (sparse)
DMControl (dense)
80
100
100
seccess rate (N8
Norma loed retur
Success rate
60
40-
40
40
20
0:
0-
0
20
40
20
100
Interaction steps (H110))
Interaction steps (H10*)
Intoraction steps (H107)
BC+R3M
DAPG (oracle)
TD-MPC
MoDem (ours)
Figure 4. Main result. Success rate and episode return as a function of interaction steps for each of the three domains that we consider (Adroit, Meta- World, DMControl), aggregated across a total of 21 challenging, visual robotics tasks. Adroit and Meta- World use sparse rewards. Mean of 5 seeds; shaded area indicates 95% CIs. Our method is significantly more sample-efficien than prior methods.
Meta-World
Assembly
Basketball
Box close
Coffee push
Hand Insert
100
100
Peg Insert side
Pick place
Pick place wall
Push
Push wall
80
60
40-
20-
0-
Stick pull
Stick
Sweed
100
Sucoet
20
40
60
50
103.0
70
40
20
40
100
70
20
100
40
60
100
20
40
63
00
120
crepe (x100s
Intecctioon shepe (x11
Internation strpp Ox10
Interctioo stepe Extt
Intermction cheps DC10
BC+R3M
DAPG (oracle)
FERM
TD-MPC
MoDem (aurs)
Figure 5. Meta-
Meta-World. World. Success rate for our method and baselines on 15 difficult, sewrse-reward Meta-World tasks with image inputs. Mean of seeds; shaded area indicates 95% Cls.
Relative importance of each phase. We study the relative im- portance of phases by considering all three Adroit tasks, and ex- haustively evaluating all valid combinations of policy pretraining as opposed to random initialization; BC seeding as opposed to seeding with random interactions; and oversampling during interactive learning interaction data buffer. Results are shown in Figure 6. We find that
Phases
60
Success rate (%))
40-
opposed to adding demonstrations to the
20
each aspect of our framework policy pretraining, seeding, and
oversampling - greatly improve sample-efficiency, both individ- ually and in conjunction. However, naively adding demonstrations to TD-MPC has limited impact on its own. In addition, policy pre- training is the least effective on its own. We conjecture that this is
0-
Interaction steps (x10%)
TD-MPC (no demos)
Naive TD-MPC + demos
due to catastrophical forgetting of the inductive biases learned dur- ing pretraining when the model and policy are finetuned in phase two and three. This is a known challenge in RL with pretrained representations (Xiao et al., 2022; Wang et al., 2022), and could also explain the limited benefit of contrastive pretraining in FERM. Interestingly, we find that adding demonstrations to TD-MPC fol- lowing the procedure of FERM outperforms FERM at the 100k interaction mark. This result suggests that TD-MPC, by virtue of being model-based, can be more sample-efficient in difficult tasks like Adroit compared to its direct model-free counterpart.
Pretraining only
Oversampling only
Pretraining oversampling
Pretraining seeding
All (ours)
Figure 6. Relative contribution of each phase. Mean success across all Adroit tasks. 5 seeds, shaded areas are 95% Cls.
7

---

Published as a conference paper at ICLR 2023
demonstrations
BC seeding (% of total)
Augmentation
Oversampling ratio
Se anccessroteei (N))
60-
Satc rate 184
60-
Success roto (S4)
60-
rate (N)
60-
40-
40-
40-
40-
220
200
20
20
60
80
100
20
40
50
80
100
20
40
60
100
20
40
00
100
Interaction stops (x101)
Interaction steps (x10')
Interaction steps (x30")
Interaction steps (x10")
10
0%
5%
Enabled
Linear annealing Constant (25%)
20%
Disabled
Figure 7. Ablations. Success rate for various ablations of MoDem, aggregated across all Adroit tasks. Our ablations highlight the relative importance of each design choice. Mean of 5 seeds; shaded area indicates 95% Cls. See Figure 6 for an ablation of our three phases. Red is our default.
Ablations. We ablated our core design choices (each proposed phase) in the previous experiment. In this section, we aim to provide further insights on the quantitative behavior of our method. In Figure 7, we evaluate (from left right) the influence of following components: (I) number of human demonstrations provided, (2) percentage of total interaction data seeded from the pretrained policy, (3) regularization of the representation he with and without random image shift augmentation, and (4) linearly annealing the oversampling ratio from 75% to 25% during training vs. a constant ratio of 25%. Interestingly, we observe that - although more demonstrations are generally helpful- - our method still converges with only single demonstration, which strongly suggests that the primary use of demonstrations is to overcome the initial exploration bottleneck. We also find that although the success rate of the pretrained policy is generally low seeding model learning with BC rollouts rather than random exploration greatly speeds up convergence. However, our method is insensitive to the actual number of seeding episodes, which further shows that the main challenge is indeed to overcome exploration. Consistent with prior work on visual RL, we find data augmentation to be essential for learning (Laskin et al., 2020; Kostrikov al., 2020; Yarats et al., 2021a; Zhan al., 2020; Hansen et al., 2022a), whereas we find the oversampling ratio to be relatively less important.
Demonstration source. We evaluate the ef- ficacy of demonstrations depending on the source (human teleoperation vs. near-optimal policies) and report results for both our method and FERM in Figure 8 (left). Hu- man demonstrations as well as state-based ex- pert policies (trained with DAPG) are sourced from Rajeswaran et al. (2018). We find that demonstrations generated by a neural net- work policy lead marginally faster conver- gence for our method. We attribute this be- havior to human demonstrations being sub- optimal, multi-modal with respect to actions, and having Markovaon properties (Man- dlekar et al., 2021). However, more impor- tantly, we find that our method converges the same asymptotic performance regardless of the demonstration source. In comparison, FERM does not solve any of the Adroit tasks with either demonstration source, which suggests that algorithmic advances provided by MoDem are far more important than the source and quality of demonstrations.
Demonstration source
Visual representation
80
60
60
Se00es
40
40-
20
200
0
20
40
60
80
300
20
40
60
80
300
Interaction stops (x107)
Interaction stops (x100)
(ours)
End-to-End (aug)
Near-optimal lours)
End-to-End (no aug)
Human (FERM)
R3M (aug)
Near-optimal (FERM)
R3M (no aug)
Figure 8. We ablate (left) human vs. oracle policy demonstrations, and (right) visual representations for MoDem. Mean success rate across all Adroit tasks. 5 seeds, shaded areas are 95% Cls.
Pretrained visual representations. We consider a variant of our framework that additionally uses R3M (Nair et al., 2022) as a frozen visual representation, similar to the BC+R3M baseline. Results for this experiment are shown in Figure 8 (right). We find that learning the representation end-to- end using the BC and model learning objectives of our framework lead to more sample-efficient learning than using pretrained representation. Interestingly, we find that - while data augmentation is essential to the success of the end-to-end representation - data augmentation is not beneficial when using a pretrained representation. We conjecture that this is because the R3M representation already provides the same inductive biases (i.e., translational invariance) as the random image shift augmentation. Note, however, that all end-to-end image-based baselines throughout work also use
8

---

Published as a conference paper at ICLR 2023
data augmentation for fair comparison. Lastly, while number of interactions is usually the limiting factor in real-world applications, we remark that using a pretrained representation generally reduces the computational cost of training and can thus be considered a tradc-off.
5 RELATED WORK
Imitation Learning, and behavior cloning (Pomerleau, I 1988; Atkeson & Schaal, 1997; Ross et al., 201 1b)) in particular, has been successfully applied variety of roboties applications such as manipulation (Zhang et al., 2018;; Song et al., 2020; Young et al., 2020), locomotion (Nakanishi al., 2003), and autonomous driving (Bojarski et al., 2016), where demonstration collection is easy. Recent works have also explored extensions like third-perseen imitation learning (Torabi et al., 2018; Radosavovic al., 2021: Zakka al., 2021; Kumar al., 2022)) and multi-task learning (Duan et al., 2017; Sharma et al., 2018; Jang et al., 2021; Reed et al., 2022). Regardless, most prior work on IL require large datasets with hundreds of expert demonstrations to provide sufficient coverage of the state space. Instead, we require only few demonstrations and allow limited amount of interaction.
Sample-efficient RI. Algorithm design for RL. has large body work (Mnih et al., 2013; Lillicrap et al., 2016; Schrittwieser et al., 2020; Hansen et al., 2022a). However, direct world applications remain limited due to poor sample efficiency. Various methods have been proposed to improve sample-efficiency of RL (Levine et al., 2016; Zhang et al., 2018a; Ebert et al., 2018; Zhan et al., 2020). For example, Ebert al. (2018) learn a goal-conditioned world model via video prediction and use the learned model for planning, and other works improve sample-efficiees representations (Shah & Kumar, 2021; Pari et al., 2022; Parisi et al., 2022; Xiao et al., 2022; Xu et al., 2023), data augmentation (Yarats et al., 202]b;a; Hansen al., 2021; 2022b), and auxiliary objectives (Srinivas et al., 2020). Recently, Wu et al. (2022) have also demonstrated that modern model-based RL algorithms can solve locomotion tasks with only one hour of real-world training. Interestingly, they find that their choice of model-based RL algorithm Dreamer- V2 (Hafner et al., 2020) is consistently more sample-efficien than model-free alternatives across all tasks considered. Our work builds upon a state-of-the-e leveraging demonstrations can further yield large gains in sample- Our contributions are compatible with and largely orthogonal to most prior work model-based RL.
by using pretrained
model-bases method, TD-MPC (Hansen et al., 2022a), and shows that
RL with demonstrations. While II. and RL have been studied extensively in isolation, recent works have started to explore their intersection (Ho & Ermon, 2016; Rajeswaran et al., 2018; Nair ct al., 2018; 2020; Zhan et al., 2020; Shah & Kumar, 2021; Mandlckar et al., 2021; Wang et al., 2022; Baker al., 2022). Notably, Rajeswaran et al. (2018) augments policy gradient algorithm with demonstration data to solve dexterous manipulation tasks from states. By virtue of using policy gradient, their method achieves stable learning and high asymptotic performance, but poor sample efficiency. Shah & Kumar (2021) extended this to visual spaces using a pre-trained visual representation network, but inherit the same limitations. Zhan et al. (2020) accelerate Soft Actor- Critic (Haarnoja et al., 2018) agent with demonstrations and contrastive learning. Baker et al. (2022) first learn policy using imitation learning on large, crowd-soureed dataset and then finetune the policy using on-policy RL. In contrast to all prior work, we accelerate a model-based algorithm with demonstrations, which we find leads to significantly improved results compared to prior model-free alternatives.
6 CONCLUSION
In this work, we studied the acceleration of MBRL algorithms with expert demonstrations to improve the sample efficiency. We showed that naively appending demonstration data to an MBRL agent's replay buffer does not meaningfully accelerate the learning. We developed MoDem, a three phase framework to fully utilize demonstrations and accelerate MBRL. Through extensive experimental evaluation, we find that MoDem trains policies that are 250% - 350% more successful in sparse reward visual RL tasks in the low data regime (100k online interactions, 5 demonstrations). In this process, we also elucidated the importance of different phases in MoDem, the importance of data augmentation for visual MBRL, and the utility of pre-teei visual representations.

---

Published as a conference paper at ICLR 2023
Reproducibility statement. Experiments conducted with publicly available environments. We provide extensive implementation details in appendices, and have made our full implementation com available at htt.hha...c.//aaeccceeeeeeeeeeee mooeee.
Acknowledgements. Work was done at Meta Al. Nicklas Hansen, Hao Su, and Xiaolong Wang are additionally supported by gifts from Qualcomm Al and grants from NSF CCF-2112665 (TILOS).
REFERENCES
Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Francis Christiano, John Schulman, and Dandelion Mané. Concrete problems in ai safety. ArXiv, 100.0666,,,, 2016.
Christopher G. Atkeson and Stefan Schaal. Robot learning from demonstration. In ICML, 1997.
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. ArXiv, abs/2206.1 1795, 2022.
Mariusz Bojarski, David W. del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. ArXiv, abs/1604.00316, 2016.
Ronen I. Brafman and Moshe Tennenholtz. R-max a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res., 3:213-231, 2002.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. ArXiv, abs/1 (////810.1220 12894, 2019.
Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, P. Abbeel, and Wojciech Zaremba. One-shot imitation learning. ArXiv, abs//1003007321 2017.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex X. Lee, and Sergey Levine. Visual foresight: Model- based deep reinforcement learning for rision-based robotic control. ArXiv, abs/1812.00564 2018.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: new approach for hard-cxploration problems. ArXiv, 0bs//1901.1099) 2019.
Scott Fujimoto, H. V. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. ArXiv, abs//b02//2 102.099777 2018.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31, pp. 2451-2463. Curran Associates, Inc., 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, P. Abbeel, and Sergey Levine. Soft actor-critic algorithms and 2018.
applications. ArXiv,
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02199, 2020.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Conference on Neural Information Processing Systems, 2021.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In ICML, 2022a.
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning- www.--rrroaccccioo baseline, 2022b.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NIPS, 2016.
10

---

Published as a conference paper at ICLR 2023
Eric Jang. Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bo-z: Zero-shot task generalization with robotic imitation learning. In CoRL, 2021.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference Learning Representations, 2020.
Sateesh Kumar, Jonathan Zamora, Nicklas Hansen, Rishabh Jangir, and Xiaolong Wang. Graph inverse reinforcement learning from diverse videos. 2022.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1333 2016.
Lillicrap, J. Hunt, A. Pritzel, N. Heess, Erez, Y. Tassa, D. Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, 0eep/1599.09999 2016.
Ajay Mandlekar, Danfei Xu, J. Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Mart'in-Mart'in in. What matters in learning from offline human demonstrations for robot manipulation. ArXiv, 0bs/2108.032995 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Ricdmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2292-6299, 2018.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv, abs/2006.09359, 2020.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhi Gupta. R3m: A universal visual representation for robot manipulation. ArXiv, abs/2203. 12601, 2022.
Jun Nakanishi, Jun Morimoto, G. Endo, Gordon Cheng, Stefan Schaal, and Mitsuo Kawato. Learning from demonstration and adaptation of biped locomotion with dynamical movement primitives. Robotics and Autonomous Systems RaS, 2004, 01 2003. doi: 10. 1299//smmmmm.22044.322
Jyothish Pari, Nur Muhammad (Mahi) Shafiullah, Sridhar Pandian Arunachalam, and Lerrel Pinto.
The surprising effectiveness of representation learning for visual imitation. ArXiv, abs/2112.01511 2022.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Kumar Gupta. The unsurprising effectiveness of pre-trained vision models for control. In ICML, 2022.
Jan Peters, Katharina Mulling, and Yasemin Altun, Relative entropy policy search. Proceedings of the AAAI Conference Artificial Intelligence, 24(1), 2010.
Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. D. Touretzky (ed.), Advances in Neurul Information Processing Systems, volume 1. Borgan-Kaufmana 1988.
Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7865-7871, 2021.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations, In Proceedings of Robotics: Science and Systems (RSS), 2018
11

---

Published as a conference paper at ICLR 2023
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. ArXiv, abs/2205.06175. 2022.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTA7S, 2011 la.
Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 201
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, L. Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillierap. and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588 7839; 604-609, 2020.
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and P. Abbeel. Masked world models for visual control. ArXiv, 142444 2022.
Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. ArXiv, abs/2107.03380, 2021.
Pratyusha Sharma, Lekha Mohan, Lemrel Pinto, and Abhinav Kumar Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In CoRL, 2018.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. ArXiv, abs/1 190400785, 2019.
Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. Robotics and Automation Letters, 2020.
A. Srinivas, Michael Laskin, and Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In ICML, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. 1998.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, et al. Deepmind control suite. Technical report, DeepMind, 2018.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. ArXiv, abs/ Abb./1905.1194, 2018.
Che Wang, Xufang Luo, Keith W. Ross, and Dongsheng Li. Vrl3: A data-driven framework for visual deep reinforcement learning. ArXiv, 202/2202210224. 2022.
Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model predictive path integral control using covariance variable importance sampling. ArXiv, abs/15099011 149, 2015.
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and P. Abbeel. Daydreamer: World models for physical robot learning. ArXiv, abs/2206.14111 2022.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. ArXiv, 2ttp/2233.067... 2022.
Yifan Xu, Nicklas Hansen, Zirui Wang, Yung-Chieh Chan, Hao Su, and Zhuowen Tu. On the feasibility of cross-task transfer with model-based reinforcement learning. 2023.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control:
Improved data-augmented reinforcement learning. arXiv preprint arXiv:2 1nt.00007799955 2021a.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021b. URL https: httpp/s://pppeevieeeeee net forum? d= 6- TvGa.
net
12

---

Published as a conference paper at ICLR 2023
Door
100
60
Succss
20
0-
Inceraction staps (H10f)
Interaction steps 1H10P)
Interaction steps (x10f)
BC+R3M
DAPG (oracte)
FERM
TD-MPC
MoDemn (ours)
Figure 9. Adroit. Success rate across each of the three sparse-reward Adroit dexterous manipulation
tasks. Tasks are visualized Figure 13. Mean of seeds; shaded area 95% Cls.
Quadruped
Humanold walk
800
800
200
Epcode return
600
600
150
800
100
200
200
50
20
40
60
80
20
60
100
40
60
80
100
Interaction 1x10")
Interaction
(x100)
Interaction (x20")
BC+R3M
DAPG (oracle)
FERM
TD- MPC
MoDern (ours)
Figure 10. DMControl. Episode return across each of the three DMControl locomotion tasks. Quadruped Run and Humanoid Walk are visualized in Figure 13. See Tassa et al. (2018) for task details. Mean of 5 seeds; shaded area is 95% Cls.
Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Kumar Gupta, P. Abbeel, and Lerrel Pinto. Visual imitation made easy. CoRL, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning (CoRL), 2019.
Kevin Zakka, Andy Zeng, Peter R. Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In CoRL, 2021.
Albert Zhan, Philip Zhao, Lerrel Pinto, P. Abbeel, and Michael Laskin. A framework for efficient robotic manipulation. ArXiv, abs/2012.07975. 2020.
Marvin Zhang, Sharad Vikram, Laura Smith, P. Abbeel, Matthew J. Johnson, and Sergey Levine. Solar: Deep structured latent representations for model-based reinforcement learning. ArXiv, abs/1800 1ww.1808.00 881009905, 2018a.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Ken Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8, 2018b.
A ADDITIONAL RESULTS
Aggregate results for each of the three domains considered are shown in Figure 4. We additionally provide all individual task results for Adroit tasks in Figure for Meta-Wome World in Figure 5, and for DMControl in Figure 10. Note that Adroit and Meta- World tasks use sparse rewards. whereas DMControl tasks use dense rewards. We also provide additional comparisons to FERM (model-free method that uses demonstrations) and simpler instantiation of our framework that simply adds demonstrations to TD-MPC (model-based method) across all three domains; see Figure 12. We emphasize that the TD-MPC with demonstrations result is equivalent to the None ablation in Figure 6. We find that both aspects our framework (model learning, and leveraging demonstrations via each of our three phases) crucial to the performance of MoDem, both in sparse (Adroit, Meta- World) and dense (DMControl) reward domains. For completeness, we also include results for all methods with larger interaction budget; results shown in Figure 11.
13

---

Published as a conference paper at ICLR 2023
(sparse)
80
(50
20
50
100
150
200
250
steps (X10P)
BC+R3M
DAPG (oracle)
DAPG (oracle; 2M steps)
FERM
TD-MPC
MaDemn (ours)
Figure 11. Larger interaction budget. Success rate as a function of interaction steps, with larger interaction budget than in the remainder of our experiments. Mean of 5 seeds across all Adroit tasks; shaded area indicates 95% Cls. For completeness, we also include the success rate of DAPG at 2M interaction steps. We find that baselines continue improve beyond the 100k interaction budget, but do not close the performance gap.
B EXTENDED BACKGROUND: TD-MPC
Section provides high-level introduction to TD-MPC (Hansen et al., 2022a), the model-based RL algorithm that we choose to build upon. In an effort to make the paper more self-contained, we aim to provide further background the TD-MPC algorithm here.
Model components. TD-MPC consists learned components hgg de. Rg, Qs. Te: a representation (s)s) that maps nigh-dimensional states (s) into compact representation, latent dynam- ics model that predicts future latent states z' (2, a), instantaneous reward = Ro (2, a), a state-action value Qa(z, a), and a policy action å (z).. The aforementioned components summarized as follows:
Representation:
ha(st)
Latent dynamics:
2 (Zg at)
Reward:
F. Ro Zz, ai)
Value:
4. = Qo 2z, as)
Policy:
To (Z:)
where we generically refer learnable parameters of the model 9 (i.e., 9 is the combined parameter vector). All components are implemented as deterministic MLPs (except for he that additionally learns shallow ConvNet for image feature extraction); TD-MPC applies Gaussian noise to the policy outputs to make action sampling stochastic. TD-MPC selects actions by planning with the learned model, and the policy Ty thus not strictly needed. However, additionally learning a policy can greatly speed both learning (for computing TD-targets) and planning (warm-starting the sampling), and as we show this paper can be pretrained on demonstration data to learn a strong behavioral prior. Besides parameters 6, TD-MPC also leverages another set of parameters 8, which is defined as an exponential moving average of e.
Training. TD-MPC learns model from sequential data collected by interaction. Specifically, TD-MPC minimizes the objective given Equation 1, which consists of three terms: a TD-loss, a reward prediction loss, and latent state prediction loss. We first restate the objective from Equation 1, and then motivate each term in more detail. TD-MPC minimizes the objective
(ID-MPC(O:TT)
i-
(i((():) + C2Cn((:T) + CBC)
(6)
optimize on sequence
TD-loss
reward loss
Intent state lnss.
temporal weight
where 7 = as, and each of the three terms defined mean squared errors:
TD-loss:
LQ(!:T) ||Qe a,) (rx QQ((z no (z)2)))
Reward loss:
CR(0:T) = |Ro a.) -rillž
Latent state loss:
Ca(0;T) = [dd(z,, a.) ) hlp
14

---

Published as a conference paper at ICLR 2023
Here, the learned components hg, dg. Rø. Qo, To are defined in Equation 5, X is constant coefficient that weighs temporally near predictions higher (i.e., long-tern predictions are down- weighted), and C1:3 constant coefficients that balance the three mean squared errors. TD-MPC learns its representation and model by jointly optimizing the objective in Equation 6: the TD)-loss is used learn the state-action value function Qo. the reward loss is used to learn the reward predictor Rg, and the latent state consistency loss is empirically found to improve sample-efficient with sparse rewards (Hansen al., 2022a). To reduce compounding errors, TD-MPC recurrently predicts these quantities H steps into the future from predicted future latent states, and back propagate gradients through time.
in tasks
Finally, note that the TD-loss in Equation 7 requires estimating the quantity maxa, Qo (Z,, az). which can be prohibitively costly to compute using planning. Instead, TD-MPC optimizes a policy To to maximize the objective
++
(x:TT) = Σ X xO To (s(g())))
(8)
i=t
where sg denotes the stop-grad operator, and z, d a; Zo ho (so)) Equation 8 is optimized strictly wrt. the policy parameters. Intuitively. the policy objective can be interpreted as a generalization of the policy objective proposed in the model-free method DDPG (Lillicrap et al.. 2016). Generally, the learned policy is found to be inferior to planning, but can dramatically speed up learning (Hansen al., 2022a).
Planning (inference). TD-MPC adopts the Model Predictive Control (MPC) framework for planning using its learned model. Planning is used for action selection during inference and data collection, but not in learning. To differentiate between action selection via the learned policy wo and planning, we denote the planning procedure as HIg The planner IIg is sample-based planner based on MPPI (Williams et al., 2015), which iteratively fits indeinndeen Gaussian with diagonal covariance to the (importance weighted) return distribution over action sequences. This is achieved by sampling action sequences from prior, estimating their return by latent "imagination" using the model, and updating parameters of the return distribution using a weighted average over the sampled action sequences. Return estimation is combination of reward predictions using Rg, and value predictions using Qs. TD-MPC executes only the first planned action, and replan at each time step based on the most recent observation, i.e., a feedback policy is produced via receding horizon MPC. To speed up convergence of planning, fixed percentage (5% in practice) of sampled action sequences are sampled using the parameterized policy Ty (executed in the latent space of the model).
Readers are referred to Hansen et al. (2022a) for further background on the TD-MPC algorithm.
C WALL-TIME
While we are primarily concerned with sample-efficiency (i.e., num- ber of environment interactions required learn given task), we here break down the computational cost of each phase of our frame- work. Wall-times are shown in Table 3. We emphasize that our framework adds no significant overhead to phase 2 (seeding) and 3 (interactive learning), i.e., running our baseline TD-MPC takes equally much time for those two phases; the only overhead intro- duced by our framework is the 5 minute BC pretraining of phase 1. Lastly, we remark that wall-time can be reduced significantly by resizing image observations to a smaller resolution for applications that are sensitive to computational cost.
Table 3. Wall-time for each of
the three phases of MoDem.
Phase
Duration
1
5m
2
34m
3
6h3m
D EXTENDED EXPERIMENTAL SETUP
We evaluate methods extensively across three domains: Adroit (Rajeswaran et al., 2018), Morld (Yu et al., 2019), and DMControl (Tassa et al., 2018). See Figure 13 for task visualizations. In this section, we provide further details our experimental setup for each domain.
15

---

Published as a conference paper at ICLR 2023
Table 4. Meta-Worts tasks. We select 15 tasks from Meta- World based on task difficulty following the categorization of Seo et al. (2022). We experiment with all tasks from the medium, hard, and very hard categories that we are able to solve using MPC with ground-truth model and computational budget of 12 hours per demonstration. Note that the majority of Meta-World tasks are categorized as easy,
Difficulty
Tasks
easy
um
Basketball, Box Close, Coffee Push, Peg Insert Side, Push Wall, Soccer, Sweep, Sweep Into Assembly, Hand Insert, Pick Place, Push Stick Pull, Stick Push, Pick Place Wall
hard
very hard
D.1 ADROIT
We consider three tasks from Adroit: Door, Hammer, Pen. Our experimental setup for Adroit closely follows Nair et al. (2022); we use 224 x 224 RGB frames and proprioceptive information input, adopt their proposed view 1 viewpoint in all three tasks, and use episode lengths of 200 for Door, 250 for Hammer, and 100 for Pen. We use an action repeat of 2 for all tasks and methods, which we find improve sample-efficiency slightly across the board. Our evaluation constrains the sample budget to demonstrations and 100k interaction steps (equivalent to 200k environment steps), whereas prior work commonly 25-100 demonstrations (Parisi et al., 2022; Nair et al., 2022) and/or 4M environment steps (Rajeswaran et al., 2018; Shah & Kumar, 2021; Wang et al., 2022). To construct sparse reward signal for the Adroit tasks, we provide per-step reward of when the task is solved and 0 otherwise. For Pen we use the same success criterion as in Rajeswaran et al. (2018); for Door and Hammer we relax the success criteria the second-to-lase reward stage since we find that less than 5 of the human demonstrations achieve success within the given episode length using the stricter success criteria. We use these success criteria across all methods for a fair comparison.
D.2 META-WORLD
We consider a total of 15 tasks from Meta-World. Tasks are selected based on their difficulty according to Seo et al. (2022), which categorize tasks into easy, medium, hurd, and very hard categories; we discard easy tasks and select all tasks from the remaining 3 categories that we are able to generate demonstrations for using MPC with a ground-truth model and computational budget of 12 hours per demonstration. This procedure yields the task set shown in Table 4. We follow the experimental setup of Sco et al. (2022) and use the same camera across all tasks: modified corner_ camera where the position is adjusted with env. model. cam pos [2] = [0.75, 0.075, 0.7] as in prior work. We adopt the same action repeat (2) in all tasks, and an episode length of 200 that all of our considered tasks are solved within this time frame. Unlike Seo et al. (2022) that uses only RGB frames input, we also provide proprioceptive state information (end-effector position and gripper openness) since it is readily available and requires minimal architectural changes. We consider a variant of Hetta-World that uses a fixed goal (env._freee freeze rand vec True), but randomly select a goal for each seed such that we evaluate on total of 5 goals per task. To construct a sparse reward signal for the Meta- Meta-World tasks, we provide a per-step reward of when the task is solved according to the success criteria of Yu et al. (2019) and () otherwise.
find
D.3 DMCONTROL
We consider total of 3 locomotion tasks from DMControl: Walker Run, Quadruped Run, Humanoid Walk. We select tasks based on diversity in embodiments and task difficulty: Walker Run and Quadruped Run are categorized as medium difficulty tasks, and Humanoid Walk as hard difficulty according to Yarats et al. (2021a). We follow the experimental setup of Hansen et al. (2022a) for DMControl experiments and adopt both camera settings, hyperparameters, and their action repeat of 2 in all tasks. To be consistent across all three domains, observations include 224 x 224 RGB frames well proprioceptive state features provided by DMControl. Since rewards are only a function of the proprioceptive state in locomotion tasks, we evaluate DMControl tasks using the default, shaped rewards proposed by Tassa et al. (2018).
16

---

Published as a conference paper at ICLR 2023
Adroit (sparse)
Meta-World (sparse)
DMControl (dense)
80
100
100
ouccess rate (N5
Norma leed retur
Success rate
60
60
40-
40
40
20
0
20
100
40
Interaction steps (X110))
Interaction steps (HI0*)
Intoraction steps (H107)
FERM
TD-MPC demos
MoDem (ours)
Figure 12. Ours vs. appending demonstrations to buffer. Success rate and episode return function of interaction steps on all 21 tasks across each of the three domains that we consider (Adroit, Meta-Workes DMControl). Mean of 5 seeds; shaded area indicates 95% Cls. We find that both (i) using a model-based method, and (ii) leveraging demonstratinns via our three-phase framework vs. simply appending demonstrations to the interaction buffer is crucial the performance of MoDem.
E IMPLEMENTATION DETAILS
Environment and hyperparameters. Human demonstrations for Adroit are sourced from Ra- jeswaran et al. (2018) which recorded them via teleoperation. In lieu of human demonstrations for Woold and DMControl, we collect demonstrations for those tasks using MPC with a ground- truth model. We follow the experimental setup of Nair al. (2022) for Adroit, Seo al. (2022) for Meta-World, and Yarats et al. (2021a) for DMControl when applicable, but choose to use a unified multi-modal observation space across all domains. Observations are stack of the two most recent 224 x 224 RGB images from a third-person camera, and also include proprioceptive information (Adroit: finger joint positions, Poeta-World: end-effector position and gripper openness, DMControl: state features) as it can be assumed readily available even in real-worke robotics applications. Demon- strations are of the same length as episodes during interaction and include observations, actions, and rewards for each step. We consider only sparse reward variants of Adroit and Meta- World tasks since dense rewards are typically impractical to obtain for real-world manipulation tasks, and consider dense rewards in DMControl locomotion tasks where reward is only function of the robot state. We use an action repeat of in all tasks (i.e., 100k interactions = 200k environment steps). Adroit and DMControl environments are randomized, whereas we use fixed goals for Meta-Wan World. Following Hansen et al. (2022a) we apply image shift augmentation (Kostrikov et al., 2020) to all observations. As observations are 224 x 224 as opposed 84 x 84 as used in prior work, we shift images by +00 pixels to maintain the same ratio. Table lists all relevant hyperparameters. We closely follow the original hyperparameters of TD-MPC and emphasize that we use the same hyperparameters across nearly all tasks, but list them for completeness; hyperparameters specific to MoDem are highlighted.
Network architecture. We adopt the network architecture of TD-MPC but modify the encoder to accommodate high-resolution images and proprioceptive state information as input. Specifically, we modify the encoder h& to consist of three components: an image encoder, a proprioceptive state encoder, and a modality fusion module. We embed image and proprioceptive state into separate feature vectors, sum them element wise, and project them into the latent representation using 2-layer MLP. Total parameter count of model and policy 1.6M. We provide PyTorch-like overview of our architecture below. We here denote the latent state dimension as Z, the proprioceptive state dimension as Q, and the action dimension as A for simplicity. As in Hansen et al. (2022a), the Q function is implemented using clipped double Q-learning (Fujimoto et al., 2018).
Total paramaters: approx. 1M
(h1
(image): Sequential(
(3): Conv2d (keneelssee 17, st:ide=2)
(1): Rell!)
(2):
(kernel_sizem
scside=2)
(3): Re
(4): Conv2d (kernel_keeeeeeeeeeeeee 13,3), strice=21
(5): RelU!)
conv2d(kernee_siier 13,3)) stside=2)
(7): RelU!)
(8): (oneid/eerne_eszee133 13,3), scside=2]
(91: RelU!)
(101: (kennel [kernel_xxee __ste=
stride=2)
17

---

Published as a conference paper at ICLR 2023
(11): ReLU()
(12): (in_rriin_fattrro (
prrop_state): Sequential
(0): (i_eerr((__esstuueee))) oit_feetuuee==256
(1): (LU
(Inesritt_footurra=225 )
(fuse): Sequential1
[3): Linesrrkt__eorrrree= ost_festures=256)
(1): RTi
(2): [iness/ out__eeatu:ee-))
Sequentiel(
(01: Litear//i__eotturrrsHHt, eit_featuree=:12)
(21: (td((alpa)11.0)
oit__eettuee==112)
(31: Etd((appa=11.0)
(4): Sineartii__cedture==>31, oit_[eatures=-))
(R): Sequentiel{
(0): Lineariah_Festures**3a 11t_festtuee==12)
(11: sid((alpa==1.0)
idteaariin_"otturee=331, oit_features=512)
(31: ELU(alpha=1.0)
(4): Jinear/in__eaauuees:3l,, oat_festures-1))
(pi): Sequential(
(0): Vineer/lh__eaturese=, cu=_festures=5:2)
(1): (LU(sappa=1.01
(21: iinear/sh_foaturee-5111 oit_features=512)
(tU(alpha-1.0)
(4): [ineer oat_feeauure=-))
(11) Sequential
(0): Lineartiin_cedturee331.. (1t_(eatures=-12)
(1): (Sx2)))
(21: Tanh
(31: (inear(ti__oaturees-]] eit_featurre=5112)
BLU(alpha=1.0)
(5): Limear//h__ieatuzee-5133 fettuuestr)s-1))
(U2): Sequential
(0): Linesrrinn_Fetuure--11.. (1)__
(1): (Sy2)))
(21: Tanh
(31: iineartin_reatureeee eut__eatuues==122)
(aUphahaa)100)
F TASK VISUALIZATIONS
We visualize demonstration trajectories in Figure 13 for 8 of the tasks that consider. Each frame corresponds to raw 224 224 RGB image observations that our model takes input together with proprioceptive information. Adroit human demonstrations are visualized at key time steps, whereas Meta-World and DMControl demonstrations are shown at regular intervals of 20 interaction steps starting from (randomized) initial state.
Visualizations are shown on the following page +
18

---

Published as a conference paper at ICLR 2023
Table MoDem hyperparameters. We list all relevant hy- perparameters for our proposed method below. Highlighted rows are unique to MoDem, whereas the remainder are in- herited from TD-MPC but included for completeness.
Hyperparameter
Value
Discount factor (7)
0.99
Image resolution
224 x 224
Frame stack
Data augmentation
110 pixel image shifts
Action repeat
Seed steps
5, 000
Pretraining objective
Behavior cloning
Seeding policy
Behavior cloning
Number of demos
5
Demo sampling ratio
75%
25% (100k steps)
Replay buffer size Sampling technique
Unlimited
PER (a 0.6,5 3 0.4)
Planning horizon (IT)
Initial parameters (n) (
(0,2)
Population size Elite fraction Iterations
512
64
8 (Humanoid, Adroit)
(Meta-World)
6 (otherwise)
Policy fraction
5%
Number of particles
Momentum coefficient
Temperature (7)
0.5
MLP hidden size
512
MI. activation
ELU
Latent dimension
100 (Humanooi)
50 (otherwise)
Learning rate Optimizer (0)
3e-4
Adam (3, 0.9, 0.999)
Temporal coefficient (A)
0.5
Reward coefficient (c:)
0.5
Value loss coefficient (62)
0.1
Consistency loss coefficient (ca)
2
Exploration schedule (c)
0.1 + 0.05 (25k sleps)
Batch size
256
Momentum coefficient (()
0.99
Steps per gradient update
update frequency
19

---

Published as a conference paper at ICLR 2023
Door
Hammer
Pen
t=20
t=40
t=60
t=80
t=100
Bin Picking
Stick Pull
Quadruped Run Humanoid Walk Pick Place Wall
Figure 13. Task visualizations. We visualize demonstration trajectories for 8 of the total of 21 tasks that we consider. The raw 224 x 224 RGB image observations that our model takes as input together with proprioceptive information are shown; Adroit human demonstrations steps, whereas world and DMControl observations are visualized at equal time intervals of 20 interaction steps, starting at random initial state. Actual episode lengths 100 for Adroit Pen, 200 for Adroit Door, 250 for Adroit Hammer, 200 for Meta- World tasks, and 1000 for DMControl.
visualized at key time
20


Reviews:
[{"review_id": "spa-vPrGQb", "reviewer": "Reviewer_6dVv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper demonstrates that a small number of expert demonstrations could assist model-based RL to learn faster. Model-based RL algorithms suffer from low sample efficiency, especially in complex environments due to exploration challenges. The proposed method alleviates this problem by continually reusing demonstrations to mix in desired state-action distributions. The proposed framework consists of three phases: (1) policy pretraining on a handful of demonstrations, (2) seeding phase where pretrained policy, with added exploration, is used to collect a dataset from the environment, in order to train world model and critic, (3) finetuning policy interactively with data from all phases. Evaluation on visual manipulation tasks with sparse rewards and locomotion tasks with dense rewards indicate the effectiveness of the proposed method.", "summary_of_the_review": "The evaluation of the proposed method is convincing. However, the main idea is not novel enough.", "strength_and_weaknesses": "Strength:\n1. The motivation for improving sample efficiency is clearly explained.\n2. The paper presents experiments that indicated the effectiveness of the proposed method and provides extensive ablation studies on the importance of each designed phase.\n3. The paper is well-written and easy to follow.\nWeakness:\n1. The idea of combining expert demonstrations with RL lacks novelty, plenty of works have implemented this before.\n2. Under an unlimited sampling budget, will the proposed method and other baselines eventually converge at the same level?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S-iAh8aWG8", "reviewer": "Reviewer_FmiB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates a staged approach to model-based RL with behaviour demonstrations, exploring the effect of each of these stages. Initially, demonstration data is used to train a behaviour cloning policy. This policy is then used with some exploration to gather data for model learning. Finally model and policy fine tuning takes place using environment interaction. Data augmentation is applied, alongside demonstration sampling, which is used to ensure the policy does not deviate significantly from the initial BC mode. Results show that this approach (MoDem) outperforms a number of other acronyms on a number of simulated benchmarks (dm control, meta world, adroit), and that exploration with the BC model seems to be key to this success. ", "summary_of_the_review": "These results are potentially interesting to SOTA chasers in the environments tested and the paper describes a relatively clear and seemingly reproducible approach to do well in these settings. For this reason, I rate the paper as marginally above the acceptance threshold. I do not rate this higher, as I do not think the strength of contribution or novelty of this work warrants this.", "strength_and_weaknesses": "Strengths:\n\nThe paper is clearly written with decent ablations, and the 3 stage approach proposed provides clear improvements over a number of methods (seemingly strong baselines) on relatively challenging benchmarks.\n\nWeaknesses:\n\nWhile I appreciate the exhaustive empirical study of the approach, the findings are not unexpected, in light of prior work. I see nothing technically wrong with this work, but nothing groundbreaking either.\n\nI think it would be valuable to reflect on the relationship of the proposed work to Dagger [1], which at its core follows a similar idea, albeit without the model-based rl aspects.\n\nSome minor comments. \n\nFigure 1. \"solves 21 hard robotics tasks...\" Could you define solve? How can we solve manipulation, pick and place or locomotion? Does a 53% success rate constitute \"solve\"? Perhaps a better, more precise description is needed. \n\nEQ 1 - can you define H as the planning horizon here? it is mentioned much later, but nice to have it here already.\n\nFig 4 - why is the oracle model from state so poor?\n\nRefs:\n[1] Ross, Gordon and Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I77962xpyKa", "reviewer": "Reviewer_9fcC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes several techniques to improve the sample efficiency of the model-based RL. Specifically, demonstrations are leveraged in a more clever way: first training the policy, then training the world models and the critic with pre-trained policy, finally training the policy in a model-based manner. Experiments are conducted on three task suites with necessary ablation study.", "summary_of_the_review": "I think the paper proposes an effective method for improving the model-based RL, but the idea is not novel enough. Also, a lack of important experimental comparison is another major drawback. Details about the algorithm need to be modified and clarified.", "strength_and_weaknesses": "Strength:\n\nThe paper has clear writing with necessary explanations about the motivations. A good summary of previous related works is provided. The experiments are solid and thorough in terms of the number of tasks. Ablation studies are also conducted to prove the effectiveness of the proposed techniques.\n\nWeakness:\n\nFig. 2(c) is unclear. The caption says the step (c) collects data using the model, however, the action $a$ is sent back to the environment. I assume this means interaction with the real environment also happens to collect data. So is the model only used for value estimation in TD-learning? If so, since TD-learning only requires the value of the next state, why are three steps plotted here? Please clearly indicate whether each $r$, $a$ and $z$ are from the prediction model or the rollout trajectories.\n\nWhat is the policy gradient? It seems a majority of the model updates in phase 2 and 3 are using TD-MPC method, however, the policy gradient as an important component in TD-MPC is not clearly explained. Only in Section 2 the paper briefly mentions that ‘The policy … and is optimized to maximize temporally weighted Q-values’. How specifically is the temporally weighted Q-values implemented and what’s the explicit form of the policy gradient?\n\nWhy $\\pi$ at the seeding phase and $\\Pi$ at the third phase? It’s not clear to me why the policies use different notations in the second and third phase. Please explain it in the paragraph.\n\nLack of experiments. For experiments, the most important baseline methods to compare for the proposed algorithm should be those falling in the category of model-based RL+IL setting as in Table 2. However, only the TD-MPC method is compared in the main results. The MWM and Dreamer-v2 should also be compared in the experiments across three task suites. The paper only reports partial results in Fig. 12. Moreover, given that the results in one (Meta-World) of the two environments show similar performances across three methods, it does not prove the strong performance gain of the proposed method. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4Z2zQ9DcRM", "reviewer": "Reviewer_Ycvo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper identified three key ingredients to speed up the learning process for MBRL models. Specifically, the authors, building on TD-MPC, proposed a framework with policy pretraining, seeding, and finetuning with interactive learning, and showed great sample-efficiency improvement on a set of challenging visual controlling tasks.", "summary_of_the_review": "This paper identified three key ingredients to speed up the learning process for MBRL models. Experiments have shown significant improvement on a set of challenging control tasks.", "strength_and_weaknesses": "### Strength\n- The authors have provided a thorough comparison with prior work and have shown significant improvement on visuo-motor control tasks. Ablation studies also confirm the importance of each proposed training phase.\n\n### Weakness\n- minor\n    - Could the authors please elaborate more on your motivation? How is the expert demonstration helping the exploration bottleneck of model-based RL?\n    - TD-MPC uses real data for policy learning and conducts decision time planning. Meanwhile, other MBRL like Dreamer-V2 learns the policy in imagination rollouts, do you think the proposed framework can also benefit Dreamer-V2?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster