Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning
Authors: Jialong Wu
Haoyu Ma
Chaoyi Deng
Mingsheng Long
Venue: NeurIPS 2023 poster (2023)
Keywords: Model-based reinforcement learning
world model
pre-training
TL;DR: This paper presents Contextualized World Models (ContextWM), which utilize pre-training on in-the-wild videos to enable sample-efficient model-based reinforcement learning of visual control tasks in various domains.

Abstract: Unsupervised pre-training methods utilizing large and diverse datasets have achieved tremendous success across a range of domains. Recent work has investigated such unsupervised pre-training methods for model-based reinforcement learning (MBRL) but is limited to domain-specific or simulated data. In this paper, we study the problem of pre-training world models with abundant in-the-wild videos for efficient learning of downstream visual control tasks. However, in-the-wild videos are complicated with various contextual factors, such as intricate backgrounds and textured appearance, which precludes a world model from extracting shared world knowledge to generalize better. To tackle this issue, we introduce Contextualized World Models (ContextWM) that explicitly separate context and dynamics modeling to overcome the complexity and diversity of in-the-wild videos and facilitate knowledge transfer between distinct scenes. Specifically, a contextualized extension of the latent dynamics model is elaborately realized by incorporating a context encoder to retain contextual information and empower the image decoder, which encourages the latent dynamics model to concentrate on essential temporal variations. Our experiments show that in-the-wild video pre-training equipped with ContextWM can significantly improve the sample efficiency of MBRL in various domains, including robotic manipulation, locomotion, and autonomous driving. Code is available at this repository: https://github.com/thuml/ContextWM.

Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning
Jialong Wu," Haoyu Ma, Chaoyi Deng, Mingsheng Long
School of Software, BNRist, Tsinghua University, China
wujiallogg2220gmai.. com, {mhy22, dengey23)d wtt.://nnhaatta.aaaaaa.a edu.
mingshengätsinggua. www.gghuaaauuuu..... cn
Abstract
Unsupervised pre-training methods utilizing large and diverse datasets have achieved tremendous success across a range of domains. Recent work has in- vestigated such unsupervised pre-training methods for model-based reinforcement learning (MBRL) but is limited to domain-specific or simulated data. In this paper, we study the problem of pre-training world models with abundant in-the-wild videos for efficient learning of downstream visual control tasks. However, in- the-wild videos are complicated with various contextual factors, such as intricate backgrounds and textured appearance, which precludes a world model from extract- ing shared world knowledge to generalize better. To tackle this issue, we introduce Contextualized World Models (ContextWM) that explicitly separate contexr and dynamics modeling to overcome the complexity and diversity of in-the-wild videos and facilitate knowledge transfer between distinct scenes. Specifically, contextual- ized extension of the latent dynamics model is elaborately realized by incorporating context encoder to retain contextual information and empower the image decoder, which encourages the latent dynamics model to concentrate on essential temporal variations. Our experiments show that in-the-wild video pre-training equipped with ContextWM can significantly improve the sample efficiency of MBRL in various domains, including robotic manipulation, locomotion, and autonomous driving.
Code is available at this repository: https:/ ttttss///githen gighhu..b... wwt.:cccm//tttteeeee taoom/huml///e wtwptthhnnnnnttttll.
1 Introduction
Model-based reinforcement learning (MBRL) holds the promise of sample-efficient learning for visual control. Typically, a world model [20 40 is learned to approximate state transitions and control signals of the environment to generate imaginary trajectories for planning [22 behavior learning [71] In the wake of revolutionary advances in deep learning, world models have been realized as
action-condittinal video prediction models 34 or latent dynamics models 22 21 However, given the expressiveness and complexity of deep neural networks, the sample efficiency of MBRL can still be limited by the failure to learn an accurate and generalizable world model efficiently.
Pre-training and fine-tuning paradigm has been highly successful in computer vision natural language processing [57 13 fast adapt pre-trained representations for downstream tasks, while learning tabula rasa is still dominant in MBRL. Recent work has taken the first step towards pre- pre-training world model, named Action-free Pre-training from Videos (APV) [68] However, has been conducted by pre-training on domain-specinfe and carefully simulated video datasets, rather than abundant in-the-wild video data [17
82 26 and
18 30 {3). Prior attempts of leveraging real-world video
data result in either underfitting for video pre-training negligible benefits for downstream visual control tasks [68]. Against this backdrop, we naturally ask the following question:
Can world models pre-trained on diverse in-the-wild videos benefi sumple-efficient learning of downstream visual control tasks?
"Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

thethewwid Video Data
Visual Control Tacks
Robotic Manipulation
Human Object
Interaction
Robotic Lecomotion
Human
Molian
Vidlen
Pre-traniing
World
Fine Tinentuiig
Model
Autonomous Driving
Real-gorld
Derving
Videos
Illustration of In-the-wild Pre-training from Videos (IPV). In various domains, we pre-train world models with wn-the-wild videos by action-free video prediction (left) and then fine-tune the pre-trained one on downstream visual control tasks with model-based reinforcement learning (right).
Figure I:
It opens up a way to utilize the vast amount of videos available on the Internet and thereby fully release the potential of this methodology by applying scale. Large and diverse data spanning various scenes and tasks provide world knowledge that is widely generalizable and applicable to a variety of downstream tasks. For instance, as depicted in Figure [ world models for robotic manipulation tasks can probably benefit not only from videos of humans interacting with target objects diverse ways but also from motion videos that embody rigorous physical rules. Analogous to APV [68], we brand our paradigm as In tn-the- wild Pre-training from Videos (IPV).
However, in-the-wild videos are inherent high complexity with various factors, such intricate backgrounds, appearance, and shapes, as well complicated dynamics. Coarse and entangled modeling of context and dynamics can waste a significant amount of model capacity on modeling low-level visual details of what is there and prevent world models from capturing essential shared knowledge of what is happening. Biological studies reveal that in natural visual systems, ~-80% retinal ganglion cells are P-type operating spatial detail and color, while ~00 are M-type operating on temporal changes [8 (4). Partially inspired by this, we suggest that leveraging IPV for world models calls for not only appropriate data and model scaling (35 but more essentially, dedicated design for context and dynamics modeling.
In this paper, we present the Contextualized World Model (ContextWM)), latent dynamics model with visual observations that explicitly separates context and dynamics modeling to facilitate knowl- edge transfer of semantically similar dynamics between visually diverse scenes. Concretely, contextualized latent dynamics model derived to learn with variational lower bound 33 38 and
elaborately realized by incorporating context encoder to retain contextual information and parsimo- nious latent dynamics model to concentrate on essential temporal variations. Moreover, dual reward predictors are introduced to enable simultaneously enhancing task-relevant representation learning as well as learning an exploratory behavior. The main contributions of this work are three-fold:
From data-centric view, systematically study the paradigm of pre-training world models with in-the-wild videos for sample-efficient learning of downstream tasks, which is much more accessible and thus makes it possible build general-purpose world model.
We propose Contextualized World Models (ContextWMD), which explicitly model both the context and dynamics to handle complicated in-the-wer wild videos during pre- pre-training and also encourage task-relevant representation learning when fine-tuning with MBRL.
Our experiments show that equipped with ContextWM. an-the-wild video pre-training can significantly improve the sample efficiency of MBRL on various domains.
2 Related Work
Pre-training in RL. Three categories of pre-training exist in RL: unsupervised online pre-training, offline pre-training, and visual representation pre-training- The first two are both domain-specifie, which learn initial behaviors, primitive skills, or domain-specific representations by either online environment interaction [53 39 offline demonstrations 41 3). Recent work has explored general-

---

purpose visual pre- tre-training using in-the-wild datasets [69] 36 77 53 86 49 45 to accelerate policy learning in model-free manners. Plan2Explore [66], as a model- based method. pre-trains its world model and exploratory policy simultaneously by online interaction. Ye ef al. [8] and Xu et al. [78] build upon EfficientZero [80] and pre-train their world models using offline experience datasets across Atari games. To the best of our knowledge, APV [68) is the first to pre-train world models
with datasets from different domains, though the datasets are manually simulated demonstrations by
scripted policies across tasks from RLBench [32], which still lacks diversity and scale.
Visual control with in-the-wild videos. While several existing works leverage demonstration
videos for learning visual control [43 84 62]. rare efforts have been made to utilize off-the-shed
video datasets from the Internet. Shao et al. [70 and Chen et al. [9] learn reward functions using
videos from the
dataset [17. R3M [9] leverages diverse Ego4D dataset [18]
to learn reusable visual representations by time contrastive learning. Other applications with such
videos include video games 7 6), visual navigation [8, and autonomous driving [8
We instead
pre-train world models by video prediction on in-the-wild video datasets.
World models for visual RI. World models that explicitly model the state transitions and reward signals are widely utilized to boost sample efficiency visual RL. A straightforward method is to learn an action-conen conditiondd video prediction model [50 34] to generate imaginary trajectories. Ha and Schmidhuber [20] propose first use a variational autoencoder to compress visual observation into latent vectors and then use a recurrent neural network to predict transition on compressed representations. Dreamer [22 23), a series of such latent dynamics models learning via image reconstruction, have demonstrated their effectiveness for both video games and visual robot control. There also have been other methods that learn latent representations by contrastive learning [5T
or value prediction 63
25
Recent works also use Transformers [7] as visual encoders 667 or
dynamics models [T] 47 58
that make world models much more scalable and can be complementary with our effort to pre-train world models with vast amount of diverse video data.
Concurrent to our work, SWIM (46) also pre-trains a world model from large-scale human videos and then transfers it to robotic manipulation tasks in the real world. However, it designs a structured action space via visual affordances to connect human videos with robot tasks, incorporating a strong inductive bias and thus remarkably limiting the range of pre-training videos and downstream tasks.
3 Background
Problem formulation. We formulate a visual control task as a partially observable Markov decision process (POMDP), defined as a tuple (O, p. r, 7). O is the observation space, AA is the action space, p(o, Oee, a.ce) is the transition dynamics, r(Oge, aege) is the reward function, and y is the discount factor. The goal of MBRL is to learn agent that maximizes the expected cumulative rewards E. E "/=1>~--ppp (=_1ra), with a learned world model (f, approximating the unknown environment. In our pre-training and fine-tuning paradigm, we also have access to an in-the-wild video dataset D {(o)TT without actions and rewards to pre-train the world model.
Dreamer. Dreamer
is a visual model-based RL method where the world model is formulated as latent dynamics model 22 with the following four components:
21
23
24
Representation model:
2t ~ (z( žt-1, (1-11,0111,)
Image decoder: ôr ~ ~ 2x)
(I)
Transition model:
2, ~ pa 2.-1,
Reward predictor: P. pa (fr 2e)
The representation model, also known as posterior of 2t. approximates latent state 2, from previous state 2t-1, previous action 0:-1 and current observation or, while the transition model, also known as prior of zt, predicts directly from 2t-1 and 01-1- The overall models are jointly learned by
minimizing the negative variational lower bound (ELBO) 33 38):
C(0)= = Eg (aur (
Σ
(- In 2x) In
(2)
+B, KL (ze 2-1, |
p0 1 ((-1))
Behavior learning of Dreamer can be conducted by actor-critic learning purely on imaginary latent trajectories, for which we refer readers to Hafner et al. 21 for details.
3

---

Action-free Pre-training from Videos (APV). To leverage action-free video data, APV [68 first pre-trains an action-free -ree variant of the latent dynamics model (with representation qo (z, 2 0z). transition po( 1x-1) and image decoder (ó z<) as video prediction model, which drops action conditions and the reward predictor from Eq. ( and Eq. (2). When fine-tuned for MBRL,
APV stacks an action-conen octionccnnittioall model (with representation Is (s 81-1, 0 2t) and transition (5r St-1, a;-1)) on lop the action-free one. Action-free dynamics models and image decoders are initialized with pre-trained weights 0 from video pre-training. Furthermore, utilize pre-trained representations for better exploration, APV introduces video-based intrinsic bonus ri measuring the diversity of visited trajectories by random projection and nearest neighbors. The overall models during fine-tuning are optimized by minimizing the following objective:
L(0, 0) = (e (0:r) (21r2 lourr)
Σ
In pa (o In Po (re + Ar( image log loss
(3)
reward log loss.
+3² KL (2x -1, 0+) pa( (2- action-lree
+B. (8x((188-11 ar-1 2x) action-conditional KL. loss
Contextualized World Models
In this section, we present Contextualized World Models (ContextWM), a framework for both action- free video prediction and visual model-based RL with explicit modeling context and dynamics from observations. Our method introduces: (i) contextualized extension of the latent dynamics model (see Section 4.1). (ii) a concrete implementation of contextualized latent dynamies models tailored for visual control (see Section 4.2). We provide the overview and pseudo-code of our ContextWM in Figure and Appendix respectively.
4.1 Contextualized Latent Dynamics Models
While latent dynamics models have been successfully ap- plied for simple and synthetic scenes, in-the-wild images and videos are naturalistic and complex. Our intuition is that there two groups of information in the sequence of observations, namely
Q1
Q2
1
Z3
2T
context and time- dependent dynamics [76] I2 42 [5]. The context encodes
Q1
Q3
aT
static information about objects and concepts in the scene, such texture, shapes, and colors, while the dynamies encode the temporal transitions of the concepts, such as positions, layouts, and motions. To overcome the com- plexity of in-the-wild videos, it is necessary to explicitly represent the complicated contextual information, thus shared knowledge with respect to dynamics can be learned and generalized across distinct scenes.
13
FT
Figure 2: Probabilistic graphical model of a contextualized latent dynamics model, assuming time-invariant context c and time-dependent dynamics ž,. Solid lines show the generative model and dot- ted lines show variational inference.
We thus propose a contextualized extension of latent dy- namies models [16 [22), extending Eq. (T) As shown in Figure 2 the probabilistic model generates the observation o, conditioned not only on the current latent dynamics z, but also on context variable that include rich information (solid lines in Figure[). while the variational inference model approximates the posterior of the latent dynamics 2, conditioned on observations on and previous states žt-1, without referring to the context (dotted lines in Figure (). The proposed design allows contextualized image decoders to reconstruct diverse and complex observations using rich contextual information, expanding their ability beyond the expressiveness of latent dynamics variables On the other hand, through the variational information bottleneck [5] latent dynamics variables 2; are encouraged to selectively exclude contextual information and only capture essential temporal variations that are not included in the context c. Furthermore, the design of context-unaware latent dynamics inference encourages representation and transition models to learn this temporal information within high-level semantic space, in contrast to merely differentiating with the context frame in the low-level visual space, facilitating the acquisition of representations more transferable and robust to intricate contexts.

---

02
Action
curdiauned
81
01
#2
02
8g
03
21 22
23
Muliiaail
Mulisasle
Actico-faoz
O2 03
(a) Action-free video pre-training
(b) Action-comditioned fine-tuning with MBRL
Figure 3: Architecture of Contextualized World Models. Building upon the stacked latent model from APV [68]. we empower the image decoder by incorporating a context encoder that operates in parallel with the latent dynamics model. The context encoder captures rich contextual information from a randomly selected context frame and enhances the decoder features each timestep with multi-scale cross-attentics mechanism that enables flexible information shorteuts across spatial positions. This design encourages the latent dynamics model to focus only on essential temporal variations, while also allowing the decoder reconstruct complex observations more effectively.
As derived in Appendix B. the overall model can be learned with ELBO of conditional log probability In po(or: Fir 01:T; C)2 without the need to model the complex distribution of contexts p(c):
((0)= =
E (ELT (1T.%L))
in (o (o: 22, c) - In po 22)
(4)
contextunlized imago loss
+3, KL [g0 2t- ar.1
po 21-1, 11-1,,+-11]]
a1-1
4.2
Contextualized World Model Architectures
We then introduce a concrete implementation of contextualized latent dynamics models, specifically tailored for visual control. Our approach builds upon Dreamer (21), which utilizes a Recurrent State Space Model (RSSM) 22 To enable action-free pre-training from videos and action-conditioned fine-tuning for MBRL, we also incorporate the stacked latent model from APV (68) into our design. Notably, our work distinguishes itself from previous research through the elaborate design of contex- tualized components and dual reward predictor structure that enhances task-relevan representation learning. An overview the overall architecture is illustrated in Figure
Context formulation. There exist various choices of contextual information to be conditioned on, including text descriptions, pre-trained representations, semantic maps, or more sophisticated structured data [59]. As our focus is on end-to-end visual control, we choose the simplest one solely based on visual observations: we consider the context e as a single frame of observation randomly sampled from the trajectory segment 01:7: = oū, i ~ Uniform {1, 2, ...,, T). It is assumed that contextual information lies equally in each frame, and by randomly selecting a context frame, our context encoder should learn be robust temporal variations.
Multi-scale cross-attention conditioning. Given the context in the form of a frame of the same size as the reconstruction o,, a U-Net [60] architecture would be one of the most appropriate choices to empower a contextualized image decoder Po žt, c). as its multi-scale shortcuts help to propagate the information directly from context to reconstruction. Moreover, in our case, the image decoder learns by also conditioning on latent dynamics ž₁, as shown in Figure 3 While a conventional U-Net architecture incorporates shortcut features directly by concatenation or summation and thus forces spatial alignment between them, temporal variations such motions deformations cannot be neglected in our case. Inspired by recent advances generative models (59 we augment the decoder feature X gexhxw with the context feature Z xxxxxx by cross-attention mechanism
5

---

15 shown as follows (we refer to Appendix C.3 for details):
Flatten: Q Reshape (X) E R hwxe K V Reshape (2) E Rhwxe
Cross-Attention: R = Attention (QW?, kWk vWV) E 12 hxwx
(5)
Residual-Connection X ReLU (X BatchNorm (Reshapo (R))) Rexaxw
Dual reward predictors. For simplicity of behavior learning, we only feed the latent variable into the actor (t) and critic v(ss), thus introducing no extra computational costs when determining actions. However, this raises the question of whether the latent variable 8; contains sufficient task- relevant information for visual control and avoids taking shortcuts through the U-Net skip connections. For example, in robotics manipulation tasks, the positions of static target objects may not be captured by the time-dependent 8t, but by the time-invariant context c. We remark that reward predictors (re((s))) play crucial role in compelling the latent variable to encode task-relevant information, as the reward itself defines the task to be accomplished. Nevertheless, for hard-exploration tasks such as Meta-world [85], the video-based intrinsic bonus [68] may distort the exact reward regressed for bchavioral learning and constantly drift during the training process [6]], making it difficult for the latent dynamics model to capture task-relevant signals. Therefore, propose dual reward predictor structure comprising a behavioral reward predictor that regresses the exploratory reward r. + Argns for behavior learning, and additionally, representative reward predictor that regresses the pure reward ry to enhance task-relevant representation learning [31.
Overall objective.
The model parameters of ContextWM can be jointly optimized as follows:
ORL L
(0, sp, 0) = (e1 (0::7 (.0 lour)
Σ
- In po (8r c) contextualized image Loss
inference
In Inp + Art"t st) - In p.p (rels:)
+3g KL
o.) ll po (2)12111)
(6)
behavioral reward representative reward
action-frce
+G KL [g 1,-1, 2x) (p a.-11) [1)]
cation-conditional KT. loss
When pre-trained from in-the-wild videos, M can be optimized by minimizing objective that drops retiin-connitionel dynamics and reward predictors from Eq. (6), similar to APV [68] For behavior learning, adopt the actor-critic learning scheme of DreamerV2 (23). For complete description of pre-training and fine-tuning of ContextWM for MBRI,, we refer to Appendix A
Experiments
We conduct our experiments on various domains evaluate In-the-wild Pre-training from Videos (IPV) with Contextualized World Models (ContextWM)), in contrast to plain world models (WM) used by DreamerV2 and APV Our experiments investigate the following questions:
Can IPV improve the sample efficiency of MBRL?
How does ContextWM compare plain WM quantitatively and qualitatively?
What is the contribution of each of the proposed techniques in C?
How do videos from different domains or of different amounts affect IPV with ContextWM?
5.1 Experimental Setup
Visual control tasks. As shown in Figure we formulate our experiments on various visual control domains. Meta-world 885 is benchmark of 50 distinct robotic manipulation tasks and
the same six tasks as APV [68] DMC Remastered [19] is a challenging extension of the widely used robotic Iocomotion benchmark, DeepMind Control Suite 773,, by expanding a complicated graphical variety. We also conduct an autonomous driving task using the CARLA simulator [14), where an agent needs drive far as possible along Town04's highway without collision in 1000 timesteps.
See Appendix C.2 for details on visual control tasks.
"In our experiments, we rebrand the APV baseline [68 with the new name 'IPV w/ Plain WM' to note that it pre-trained on wiithe-wild data rather than only Action-free data.

---

Figure 4: Example image observations of our visual control benchmark tasks: Meta-world, DMCon- trol Remastered, and CARLA (left to right).
Pre-training datasets. We utilize multiple in-the-wild video datasets which can potentially benefit visual control.
(SSv2) [7] dataset contains 193K videos of people inter- acting with objects. Human3.6M [30] dataset contains videos of human poses over 3M frames under 4 different viewpoints. You'uube Driving [8] dataset collects 134 real-world driving videos, over 120 hours, with various weather conditions and regions. We also merge the three datasets to construct an assembled dataset for the purpose of pre-training general world model. See Figure for example video frames and Appendix C.T for details on data preprocessing.
Implementation details. For the newly introduced hyperparameter, we use 8, = 1.0 in Eq. (6) for all tasks. To ensure a sufficient capacity for both plain WM and ContextWM pre-trained on diverse in-the-wild videos, we implement visual encoders and decoders as 13-layer ResNets 227. Unless otherwise specified, we use the same hyperparameters with APV. See Appendix C.3 for more details.
Evaluation protocols. Following Agarwal et al. and APV, we conduct individual runs for each task and report interquartile mean with bootstrap confidence interval (CI) for individual tasks and with stratified bootstrap CI for aggregate results.
5.2 Meta-world Experiments
Figure 5a shows the learning curves on six robotic manipulation tasks from Meta-world. We observe that in-the-wild pre-training on videos from SSv2 dataset consistently improves the sample efficiency and final performance upon the DreamerV2 baseline. Moreover, proposed ContextWM surpasses its plain counterpart in terms of sample efficiency on five of the six tasks. Notably, the dial-turn task proves challenging, neither method is able solve it. These results demonstrate that our method of separating context and dynamics modeling during pre-training and fine-tuning can facilitate knowledge transfer from to-the-wild videos to downstream tasks.
SSv2 pre-trained results.
Ablation study. We first investigate the contribution of pre-training in our framework and report the aggregate performance with and without pre-training at the top of Figure [b] Our results show that plain WM seldom benefits from pre-training, indicating that the performance gain of IPV with plain WM is primarily due to the intrinsic exploration bonus. This supports our motivation that the complex contexts of in-the-wild videos can hinder knowledge transfer. In contrast, ContextWM significantly improves its performance with the aid of video pre-training. Additionally, we evaluate the contribution of the proposed techniques in ContextWM, shown the bottom of Figure (5b We experiment with replacing the cross-altention mechanism (Eq. (5)) with simple concatenation or removing the dual reward predictor structure. Our results demonstrate that all these techniques contribute to the performance of ContextWM, as both variants outperform the plain WM.
Effects of dataset size. To investigate the effects of pre-training dataset size, we subsample 1.5k and 15k videos from SSv2 dataset pre-train ContextWM. Figure 7a illustrates the performance of ContextWM with varying pre-training dataset sizes. We find that pre-training with only a small subset of in-the-wild videos can almost match the performance of pre-training with the full data. This is probably because, despite the diversity of contexts, SSv2 dataset still lacks diversity in dynamics patterns as there are only 174 classes of human- object interaction scenarios, which can be learned by proper models with only a small amount of data. It would be interesting for future work to explore whether there is a favorable scaling property w.r.t. pre-training dataset size when pre-training with more sophisticated video datasets with our in-the-wild video pre-training paradigm.
Effects of dataset domain. In Figure 7b we assess pre-training on various video datasets, including RLBench 32 videos curated by APV 68 and our assembled data of three in-the-wild datasets. We observe that while pre-training always helps, pre-training from more similar domain, RLBench, outperforms that from SSv2. Nevertheless, simulated videos lack diversity and scale and make difficult to learn world models that are broadly applicable. Fortunately, wild video pre-training
7

---

Lever Pull
Drawer Open
Door Lock
Effects Pre traning
103
Succos Ra5s
CorkedtM
CorkudtM phe,
an Welt (Pre
Pan WAll Pre N)
25
23 Envirnment Stops (xtif)) Archtechir
12
Emii cemerne Sieee (xio")
Envronment Stppe 16101
Emn comment Stese (x127)
Button Press Topdown Wall
Reach
Del Turn
101
Su000s5 Rda (50)
(50
Corkodiett
5
CorkoutiM Cros AMl CortodieM tebn Dual Rood
IPW Paan PV GeaudWW
Piie VM
T0
26
20
23 Envirommnt Steps (xlid)
12
Endronment Steos (xte')
Environmnn Stepe (x121
En rommnt Stats (x12))
(a) Learning curves
(b) Ablation study
Figure 5: Meta-world results. (a) Learning curves with in-the-wild pre-training on SSv2 dataset, as measured on the success rate, aggregated across eight runs. (b) Performance of ContextWM and plain WM with or without pre-training (lop) and performance of ContextWM and its variants that replace the cross-attention mechanism with naive concatenation or remove the dual reward predictor structure (botlom). We report aggregate results across a total of 48 runs over six tasks.
can continuously improve its performance with a more diverse assembled dataset, suggesting a promising scalable alternative for domain-specific pre-training.
5.3 DMControl Remastered Experiments
In the top row of Figure 6a we present the learning curves of IPV from SSv2 dataset and DreamerV2 on the DMC Remastered Tocomotion tasks. Remarkably, we observe that pre-training with the SSv2 dataset can significantly enhance the performance of ContextWM, even with a large domain gap between pre-training and fine-tuning, This finding suggests that ContextWM effectively transfers shared knowledge of separately modeling context and dynamics. contrast, a plain WM also benefits from pre-training, but it still struggles to solve certain tasks, such hopper stand. These results also suggest that our CM could be a valuable practice for situations where visual generalization is critical, ContextWM trained from scratch also presents competitive performance in Figure 6b
Effects of dataset domain. Figure 7b demonstrates the performance of pre-training on human motion videos from the Human3.8m 6MM dataset. However, we observe a negative transfer, as pre-training from Human3.6X 3.6 leads to inferior performance compared to training from scratch. We argue that the Human3. 6M dataset collected in the laboratory environment, rather than truly in-the-wild. These results support motivation that pre-training data lacking in diversity can hardly help learn world models that are generally beneficial, Additionally, we experiment with pre- training on the assembled data of three video datasets but find significant improvement over SSv2 pre-tree pre-training.
5.4 CARLA Experiments
The bottom row of Figure ba displays the learning curves of IPV from the SSv2 dataset and Dream- erV2 on the CARLA driving task under different weather and sunlight conditions. Similar to the DMC Remastered tasks, we find that MBRL benefits from pre-training on the SSv2 dataset, despite a significant domain gap. We observe that in almost all weather conditions, IPV with ContextWM learns faster than plain WM at the early stages of training and can also outperform plain WM in terms of the final performance. Minor superiority of ContextWM over plain WM also suggests that in autonomous driving scenarios, single frame may not contain sufficient contextual information, and more sophisticated formulations contextual information may further enhance performance.
Effects of dataset domain. We then assess the performance of ContextWM on CARLA by pre- training it from alternative domains. As shown in Figure 7b observe that pre-training from
8

---

Cheelan Run
Hopper
Remastared
Dreamery2
M
Pr Pice WM
Contadifi (Pr.
Pass Iel dhe 19
Play (eV. 30
300
207
180
197
Enveroment Stepe (c1o")
Envinnment Stepe (4117)
Dnvtromment Stope (x12")
Ersvarment Stepe (xtof)
CARLA C Noon
CARLA Wer Sunset
CARLA Dynamic Weather
CARLA
Esiood Notumr
12:3-
234-
Nommatond
Oreamer122 PV Comedinn
-00
Peir WMl pin
W0l Jrne
30
20
30
100
120
160
60
25
30
25
120
13.3
150
30
A0
70 160
135
25 54 75
180 125
159
Erviurmen Stape (x(of))
Endrrnmmnt (x (x10f)
Endrrsment Staps (xcef)
Envirommnnt Stapo (stor)
(a) Learning curves
(b) Effects of pre-training
Figure 6: DMC Remastered (top) and CARLA (bottom) results. (a) Learning curves with pre-training on SSv2 dataset, measured on the episode return, aggregated across eight runs. (b) Performance of Co M and plain WM with or without pre-training, aggregated across 24 runs over three tasks. Episode returns of each task in CARLA normalized to comparable ranges.
Mataooarid
Melwooorld
DMC Remaorrened
CARLA
Normsted Erpoudte Deeum
Suzzae Doo- (1)
Pre
5077
50.5 adoce
18 Binm boboor
of Duterhinng Copembiiee wess Previst
122 05.0 v693)
ND Pre-2um
Enionmmetttlaa (x/1f)
Enwonment Bleas (r10))
Enuteement Stepe Stepe./rtfPP (e1ts))
Enviommrrt Blass
Effects of datasct size
Effects of dataset domain
Figure 7: Analysis pre-training datasets. We report aggregated results over tasks. (a) Performance of ContextWM on Meta-world tasks, with varying pre-training dataset sizes. (b) Performance of Contex(WM with pre-training videos from various domains.
the YouTube Driving dataset or combination of three in-the-wild datasets can both improve upon learning from scratch. However, neither can significantly surpass the performance achieved by pre-training from SSv2, despite narrower domain gap between YouTube Driving and CARLA. We conjecture that this could be attributed to the higher complexity the YouTube Driving dataset in comparison to SSv2. It would be promising to explore the potential of scaling ContextWM further to capture valuable knowledge from more complex videos.
5.5
Qualitative Analysis
Video prediction. We visually investigate the future frames predicted by a plain WM and Contex- tWM on SSv2 in Figure Sal We find that our model effectively captures the shape and motion of the object, while the plain one fails. Moreover, we also observe that cross- aotssnattentions in our model (Eq. (5) successfully attend varying spatial positions of the context to facilitate the reconstruction of visual details. This shows our model works with better modeling of context and dynamics.
Video representations. We sample video clips of length 25 with two distinct labels (push something from right to left and from left to right) and visualize the averaged model states of the sampled videos using t-SNE 74 in Figure 8b Note that we do not utilize any labels of the videos pre-training. Video representations of plain WM may entangle with extra contextual information and thus are not sufficiently discriminative to object motions. However, ContextWM which separately models context and dynamics can provide representations well distributed according to different directions of motion.

---

Plai WMd Pie Tolne
Conest Prediced Groun.
woth
Predicted
Paiiing tiom
(a) Vidco prediction
(b) Video representations
Context
Reconstruction
Open-loop Production
Ground Inath
Concst IPV wf Onica/M
Shilt
IPV we! Plaia WM
(c) Compositional decoding
Figure 8: Qualitative analysis. (a) Future frames predicted SSv2. Predictions from model well capture the shape and motion of the water cup. Morcover, our context cross-attentions from the left upper corner (red box) of different frames successfully attend varying spatial positions of the context frame. (b) t-SNE visualization of average pooled representations of SSv2 videos with two distinct labels, from pre-trained plain WM and ContextWM, respectively. (c) Compositional decoding analysis on the DMCR domain, where fine-tuned ContextWM is able to combine the new context with dynamics information from the original trajectory.
Compisitional decoding. We conduct a compositional decoding analysis by sampling a random frame from another trajectory to replace the original context and leaving dynamics the same. As shown in Figure [c ContextWM correctly combines the new context with the original dynamics information. These results show that our model, fine-tuned on the DMCR domain, has successfully learned disentangled representations contexts and dynamics. In contrast, the plain WM suffers from learning entangled representations and thus makes poor predictions about the transitions.
6 Discussion
This paper presents Contextualized World Models (ContextWM), a framework for both action-free video prediction and visual mosel-based RL. We apply ContextWM to the paradigm of In in-the-wild Pre-training from Videos (IPV), followed by fine-tuning on downstream tasks to boost learning efficiency. Experiments demonstrate the effectiveness of our method in solving variety of visual control tasks from Meta-world, DMC Remastered, and CARLA. Our work highlights not only the benefits of leveraging abundant in-the-wild videos but also the importance of innovative world model design that facilitates knowledge transfer and scalable learning.
I. Limitations and future work. One limitation of our current method is that a randomly selected single frame may not sufficiently capture complete contextual information of scenes in real-world applications, such as autonomous driving. Consequently, selecting and incorporating multiple context frames as well as multimodal information [59] for better context modeling need further investigation. Our work is also limited by medium-scale sizes in terms of both world models and pre-training data, which hinders learning more broadly applicable knowledge. An important direction to systematically examine the scalability our method by leveraging scalable architectures like Transformers [47] 67] and massive-scale video datasets [18 48). Lastly, our work focuses on pre- training world models generative objectives, which utilize the model capacity inefficiently on image reconstruction to overcome intricate contexts. Exploring alternative pre-training objectives, such as contrastive learning [T] or self-prediction [65 8D) could further release the potential of
IPV by eliminating heavy components context modeling and focusing dynamics modeling.
10

---

Acknowledgments
We would like to thank many colleagues, in particular Haixu Wu, Baixu Chen, and Jincheng Zhong, for their valuable discussion. This work was supported by the National Key Research and Development Plan (2020AAA0109201), the National Natural Science Foundation of China (62022050 and 62021002), and the Beijing Nova Program (22011000662004))
References
[1] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):1447-1980 2018.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In NeurIPS, 2021.
[3] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Offline primitive discovery for accelerating offline reinforcement learning. In ICLR, 2021.
[4] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
[5] Alexander Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
[6] Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando De Freitas. Playing hard exploration games by watching youtube. In NeurlPS, 2018.
[7] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning by watching unlabeled online videos. In NeurlPS, 2022.
[8] Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos. In NeurIPS, 2020.
[9] Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from "in-the-wild" human videos. In RSS, 2021.
[10] Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. TransDreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.09481, 2022.
[11] Fei Deng, Ingook Jang, and Sungjin Ahn. DreamerPro: Recomstruction-fre model-based reinforcement learning with prototypical representations. In ICMI,, 2022.
[12] Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In NeurlPS, 2017.
[13] Jacob Devlin, Ming-Weo Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[14] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In CoRL, 2017.
[15) Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In ICML, 2020.
[16] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deep- MDP: Learning continuous latent space models for representation learning. In ICML., 2019.
[17] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. al. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
11

---

[18] Kristen Grauman, Andrew Westbury, Eugene Byme. Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[19] Jake Grigsby and Yanjun Qi. Measuring visual generalization in continuous control from pixels. arXiv preprint arXiv:2010.06740, 2020.
[20] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurIPS, 2018
[21] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
[22] Danijar Hafner, Timothy Lillicrap. Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, 2019.
[23] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR, 2021.
[24] Danijar Hafner, Jurgis Pasukomis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.044 30.00004044.......00000.0100...0 2023.
[25] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In ICML, 2022.
[26] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[28] David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of neurophysiology. 28(2):229-29) 1965.
[29] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[30] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6M1 Large scale datasets and predictive methods for 3d human sensing in natural environments. TPMAI, 2013.
[31] Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017.
[32] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. RLBench: The robot learning benchmark & learning environment, RA-L, 2020.
[33] Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduc- tion to variational methods for graphical models. Machine learning, 37:183-233, 1999.
[34] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model- based reinforcement learning for atari. In ICLR, 2020.
[35] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[36] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. In CVPR, 2022.
[37] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv.1412.6980, 2014.
12

---

[38] Diederik P Kingma and Max Welling. Auto-ennoding variational bayes. In ICLR, 2014.
[39] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. In NeurIPS, 2021.
[40] Yann LeCun. A path towards autonomous machine intelligence, preprint posted openreview, 2022.
[41] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005001643 2020.
[42] Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. In ICML, 2018.
[43] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In ICRA, 2018.
[44] Margaret Livingstone and David Hubel. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science, 240(4853);740-749, 1988.
[45] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, et al. Where we the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303. /8240, 2023.
[46] Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. In RSS, 2023.
[47] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. In ICLR, 2023.
[48] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo 100:: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
[49] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In CoRL, 2022.
[50] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action- conditional video prediction using deep networks in atari games. In NewrIPS, 2015.
[51] Masashi Okada and Tadahiro Taniguchi. Dreaming: Model-based reinforcement learning by latent imagination without reconstruction. In ICRA, 2021.
[52] Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. Iso-Dream: Isolating and leveraging noncontrollable visual dynamics in world models. In NeurlPS, 2022.
[53] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurpris- ing effectiveness of pre-trained vision models for control. In ICML., 2022.
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high performance deep learning library. In NeurIPS, 2019.
[55] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-drivee by self-supervised prediction. In ICML, 2017.
[56] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Daniilidis. Coarse-to- fine volumetric prediction for single-image 3d human pose. In CVPR, 2017.
[57] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[58] Jan Robine, Mare Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world models are happy with 100k interactions. In ICLR, 2023.
13

---

[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[60] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[61] Lukas Schäfer, Filippos Christianos, Josiah Hanna, and Stefano V Albrecht. Decoupled reinforcement learning to stabilise intrinsiaally exploration. In AAMAS, 2022.
[62] Karl Schmeckpeper, Annie Xie, Olch Rybkin, Stephen Tian, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Learning predictive models from observation and interaction. In ECCV, 2020.
[63] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
[64] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
[65] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self- welf-predictive representations. In ICLR. 2021.
[66] Ramanan Sekar, Olch Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning explore via self-supervised world models. In ICML, 2020.
[67] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In CoRL, 2022.
[68] Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In JCML, 2022.
[69] Rutav Shah and Vikash Kumar. RRL: Resnet as representation for reinforcement learning, In ICML, 2021.
[70] Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2Robot: Learning manipulation concepts from instructions and human demonstrations. IJRR, 2021.
[71] Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990.
[72] Richard Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT press, 2018.
[73] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancy, et al. Deepmind control suite. arXiv preprint arXiv: (18811000900 2018
[74] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 911),, 2008
[75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[76] Ruben Villegas, Jimei Yang, Scunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In ICLR, 2017.
[77] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203. (ttp:22233.06733.. 2022.
[78] Yifan Xu, Nicklas Hansen, Zirui Wang, Yung-Chiel Chan, Hao Su, and Zhuowen Tu. On the feasibility of cross-task transfer with model-based reinforcement learning. In ICLR, 2023.
14

---

[79] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In ICLR, 2022.
[80] Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. In NeurIPS, 2021.
[81] Weirui Ye, Yunsheng Zhang, Pieter Abbeel, and Yang Gao. Become proficient player with limited data through watching pure videos. In ICLR, 2023.
[82] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurIPS, 2014.
[83] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDDI00K: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
[84] Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain- meta-learning. In RSS, 2018.
[85] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2020.
[86] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. In NeurIPS, 2022.
[87] Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine Learning invariant representations for reinforcement learning without reconstruction. In ICLR, 2021.
[88] Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. In ECCV, 2022.
[89] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, 2020.
15

---

A
Pseudocode
For clarity, we first elaborate on the optimization objective for contextualized world models during action-free pre-training, which drops fine-tuning objective (Eq. (
dynamics and reward predictors from the
CCM--P Pt (8)
=
(xge (ourr)
Σ
In (os c) contextualized image loss
comeet-unaware lalent inference
+Bg KL (90(2] oz) po (e(
action free KL. loss
For behavior learning, we adopt the same actor-critic learning scheme based purely on imaginary
latent trajectories 8-, å,, frr with horizon H, as DreamerV2 [2] 23 The critic (g( is learned by
regressing the X-target 772 64):
++ H
Ceritio (E) Epp.
Σ
(ve($=) - sg(V>\))
(8)
2
V = fr +7 (1 ()ug(S++1) VV (1c(6++1)
if <
(9)
i+ H,
where sg is stop gradient function. And the actor (als) is learned by maximizing the imagined return by back-prop pok-propaggttin the gradients through the learned world model, with an entropy regularizer:
Lactor (1b) = ST
Σ (-V - H (mr((1()))))
(10)
Overall, a complete description of pre-training and fine-tuning of ContextWM for MBRL presented in Algorithm|
B Derivations
The variational bound for contextualized latent dynamics models p(o1.,, Firr 01.r, c) and variational posterior q(=1.T (a.r; 01:)) follows from importance weighting and Jensen's 's Inequality as shown
In zla,c)
In Inp(o, c) 2 Byy(sla,o)
((2la,0)
In p(poorr2,,, [Ll(a (0)o==(a
Imp(orle.. +mp(rel)z)) KL [q(z)a, o)lp(())a)],
where
KL [g(2|a, ()ll(aa) = (222a, o) In 11 dz
p( p( ((-11)
T
a( 02-1,0)) p( ا(ū a=-1)
d al ((
dæ glalkr-n
t=1
d g(2 o)
( 0
KL
For brevity, denote 21r7, a1.T, OU:7, FLT 2, 0, respectively.
16

---

Algorithm Contextualized World Models with In-the-wild Pre-training from Videos
1: // Action-free pre-training with in-the-wild videos
2: Initialize parameters 0 of action-free dynamics model, image encoder, and decoder randomly
3: Load in-the-wild video dataset D
4: for each training step do
5: // WORLD MODEL PRE-TRAINING
6: Sample random minibatch {our)) D
7: Get context & Oj: i ~ Uniform {1
8: Update world model by minimizing Cc in Eq.
9: end for
// Action-conditioned fine-tuning with MBRL
11: Load pre-trained parameters 0 of action-free free dynamics model, image encoder, and decoder
12: Initialize parameters é, sp of action-conditioned dynamics model and reward predictors randomly
13: Initialize parameters 15, & of actor Tp((a) (als) and critic (s)
Initialize replay buffer with random seed episodes 15: for each timestep / do
16:
// COLLECT TRANSITIONS
17:
Get representation 2; (--11 St (1--)))
18: Get action a, ~ The ( (Ag(st)
19:
Add transition {(o1, de, d r=)] to replay buffer B
20:
// WORLD MODEL FINE-TUNING
21:
Sample random minibatch {(O.T, (I:T,FI:T) ~ B
22:
Get context c f oz. f Uniform T) Update world model by minimizing LCWM (b. 90, in Eq. (6
23:
I/ BEHAVIOR LEARNING
25:
Imagine future rollouts {š,, f, f, + Artnt using world model and actor
Update actor and critic by minimizing objectives in Eq. and Eq. (TO)
27:
end for
Overall, have lower bound the log-likelihood of the data:
In imp(o,rla,c) c) 2 (x
Σ (Ip(o))2,)) + inp(re)2)) KL
We can also calculate that the tightness of the bound KL [g(2la.o)|l(22la...rr.)
C Experimental Details
C.1 Data Preprocessing
We use three common video datasets for in-the-wild video pre-training. All inputs are resized to x4x64 pixels.
The Something- Something--2 datase shows footage of humans interacting with common everyday objects. We use all videos in the training set but filter out videos less than 25 frames long, resulting in total of 162K videos for pre-training.
Somnething-Something
Human3.6M. The Human3.66 6M datase (30 contains videos of various activities performed by 11 human subjects. We use the particular dataset processed by Pavlakos et al. [56] which includes total of 840 videos with 210 scenarios and different viewpoints for each scenario. Each video has around 500 frames. To make sure the images used for pre-training better fit our downstream tasks, we place the subject at the center of each frame by cutting a bounding box calculated from the 3D joint positions provided by the dataset. In addition, padding of 200 pixels is performed around
httpe://developer: http://wwwpppaaaoooooco/coocota/
something
http://vision: wtw..iaaa....aaa...
17

---

the bounding box so that the subject is properly positioned. The image clipped using the padded bounding box is then resized 64 64 pixels.
YouTube Driving. The YouTube Driving datase
88 consists of real-world driving videos with
different road and weather conditions. As videos in this dataset usually have short opening, the first 100 frames of each video are skipped. The original video is downsampled 5 times. We employ all 134 videos in our pre-traie ure-training process, with each video containing roughly 10000 to 20000 frames.
Assembled Dataset. We compose the three datasets mentioned above into an assembled dataset. On each sample, we uniformly choose dataset from the three choices and sample videos from that dataset accordingly.
C.2 Benchmark Environments
Meta-world. Meta-world [85] a benchmark of 50 distinct robotic manipulation tasks. Following APV [68]. we use a subset of six tasks, namely Lever Pull. Drawer Open, Door Lock, Button Press Topdown Wall, Reach, and Dial Turn. In all tasks, the episode length is 500 steps without any action repeat. The action dimension is 4, and the reward ranges from 0 to 10. All methods are trained over 250K environment steps, which is consistent with the setting of APV 68
DMC Remastered. The DMC Remastered (DMCR) Suite [19 is variant of the DeepMind Control Suite [73] with randomly generated graphics emphasizing visual diversity. On initialization of each episode for both training and evaluation, the environment makes a random sample for 7 factors affecting visual conditions, including floor texture, background, robot body color, target color, reflectance, camera position, and lighting. Our agents trained and evaluated on three tasks: Cheetah Run, Hopper Stand, and Walker Run. All variation factors are used except camera position in the Hopper Stand task, where we find it too difficult for the agent to learn when the camera is randomly positioned and rotated. Following the common setup of DeepMind Control Suite [21] 79) we set the episode length 1000 steps with an action repeat of 2, and the reward ranges from 0 1. All methods are trained over IM environment steps.
CARLA. CARLA [4] is an autonomous driving simulator. We conduct a task using a similar setting as Zhang et al. 187, where the agent's 's goal is to progress along a highway in 1000 time-steps and avoid collision with 20 other vehicles moving along. The maximum episode length is set to 1000 without action repeat. Possible reasons to end an episode early include deviation from the planned route, driving off the road, and being stuck in traffic due to collisions. Each action in the CARLA environment is made up of steering dimension and accelerating dimension (or braking for negative acceleration). In addition to the reward function in Zhang et al., we add a aene-ng reward to prevent our agent from driving on the shoulder. Our reward function is follows:
Vego ih Al |center| A impluse X2 |steer]
The first term represents the effective distance traveled on the highway, where Uego is the velocity vector projected onto the highway's unit vector â, multiplied by a discretized time-step At = 0.05 and a lane-keeping term |center| penalize driving on the shoulder. The impulsc term represents collisions during driving, while the steer E [-1, term prevents excessive steering. The hyperparameters λ and A2 are set 10-3 and respectively. The central camera of the agent, observed as 64 x 64 pixels, is used as observation. We use three different weather and sunlight conditions offered by CARLA for training and evaluation: ClearNoon, WetSunset, and dynamic weather. In the dynamic weather setting, a random weather and sunlight condition is sampled at the start of each episode and changes realistically during the episode. To aggregate per- per-weather scores, normalize raw episode returns of cach weather to comparable ranges. Namely, linearly rescale episode returns from [-400, 1200] under ClearNoon, [-400, 800) under WetSunset, and [-400, 600] under dynamic weather to unified range of [0, 1]. All methods are trained over 150K environment steps.
httpps///gttab..cc///ttttttvvvvt
18

---

stage name
output size
image encoder
conv in
32x32
3x3, 48, stride 2
xx3, 3x3, 48
x2
stagel
16x16
average pool 2x2, stride 2
3x3, 96
x2
stage2
8x8
3x3, 96
average pool 2x2, stride 2
3 x3, 192
x2
stage3
4x4
3 x3, 192
average pool 2x2, stride 2
Table 1: Architecture for image encoders in world models. Building residual blocks 27 are shown brackets, with the number of blocks stacked. Numbers inside each bracket indicate the kernel size and output channel number of the corresponding building block, e.g. 3 3x3, 192 means of the convolutions the block uses a x kernel and the number of output channels 192. We use Batch Normalization 29 and ReLU activation function.
C.3 Model Details
Overall model. We adopt the stack latent model introduced by APV 68 and extend it into Contextualized World Model (ContextWM) with the following components:
Action-free
Representation model:
24 90(21 22-1
Transition model:
2 ~ (2 22-1)
Action-conditional
Representation model:
9t (0 14-1,
Transition model:
8, ~ (8 lel $t-1,a,-1
Dual reward predictors
Representative reward predictor:
6, ~ Pfffel
Behavioral reward predictor:
F: + Ar. ~ po (fr + Ari (st)
Contextualized image decoder:
6, ~ (0 ( c).
(11)
Specifically, the action-fres representation model utilizes ResNet-style image encoder process observation input o,. Moreover, the contextualized image decoder also incorporates a similar context encoder to extract contextual information from the context frame and augment the decoder features with cross-attention mechanism. We elaborate on the details below.
ensure a sufficient capacity for both plain WM and ContextWM pre-trained on diverse in-the-wild videos, we implement image encoders and decoders 13-layer ResNets (27]. For the sake of simplicity. we only show the architecture of image encoder in Table as the architecture of the context encoder is exactly the same, and the architecture of the image decoder is symmetric to the encoders. In the decoder, we use nearest-neighhor upsampling for unpooling layers. The outputs of the last residual block of two stages in the context encoder (stage2 and stage3) before average pooling (thus in the shape of 16 16 and 8 respectively) are passed the corresponding residual block of the image decoder and used to augment the incoming decoder features with cross-attention.
Image encoders and decoder.
The input features E Rexhxe of residual blocks in the image decoder are augmented with shorteut features Z Rexhxx of the same shape from the context encoder, by cross-attention mechanism as Eq. (5). Here we elaborate on the details. After Z is flattened into random
Context cross-attention.
sequence of tokens hl to mitigate the quadratic complexity of cross-attention
19

---

mask 75% of the tokens, resulting tokens of the length h.w... Moreover, we use learned position embedding Rhexe initialized with zeros for both Q and K, V, respectively.
Context augmentation. We randomly sample a frame of observation from the trajectory segment as the input to the context encoder. To prevent the image decoder from taking shortcut by trivially copying the context frame and thus impeding the training process, we apply random Cutout [89 as image augmentation on the context frame for all Meta-worlb tasks (except drawer open), as Meta-World observations have a static background and fixed camera position. Note that we do not utilize any augmentation for DMCR and CARLA tasks. Our comparison with plain WM is fair since we do not introduce any augmentation into the image encoder of the latent dynamics models, thus the agent determines its actions solely based on the original observation.
Latent dynamics model. The stacked latent model from APV [68] is adopted to enable action-free pre-training and action-conditionel fine-tuning, where both the action-free and action-conditioned models are built upon the discrete latent dynamics model introduced in DreamerV2 [23] where the latent state consists of deterministic part and a discrete stochastic part. Inputs to the latent dynamics model are representations from the image encoder. Following APV [68], we use 1024 as the hidden size of dense layers and the deterministic model state dimension.
Reward predictors, actor, and critic. Both of the dual reward predictors. as well as actor and critic, are all implemented as 4-layer MLPs with the hidden size 400 and ELU activation function, following DreamerV2 [23] and APV [68.
C.4 Hyperparameters
For the newly introduced hyperparameter, we use 3, = 1.0 for all tasks. We set 3 = 1.0 for pre-training, and $__ 0.0, 3, = 1.0 for fine-tuning, following APV [68] Despite adopt deeper architecture for both image encoders and decoders, we find that hyperparameters are relatively robust architectures, and the new deeper architecture under the same hyperparameters can match even slightly improve the performance originally reported by APV. Thus, we use the same hyperparameters with APV. For completeness, we report important hyperparameters Table 2
C.5 Qualitative Analysis Details
For video prediction in Figure Sa we sample a video clip of length 25 and let the model observe the first 15 frames and then predict the future 10 frames open-loop by the latent dynamics model. The context frame is selected as the last of the observed 15 frames. To visualize context attention, we compute the attss-nttention weight across the 16 x 16 feature maps without random masking and plot heatmap of the weights averaged over attention heads and over attention targets of small region (for example, 3 x 3) on the decoder feature map. For video representations Figure [bb] we sample video clips of length 25 and compute the averaged model states avg( (21:2) of dimension 2048 for each video clip, which are then visualized by t-SNE [74].
We explain the visualization scheme for Figure X
C.6 Computational Resources
We implement all the methods based on PyTorch (54 and train with automatic mixed precision. In terms of parameter counts, ContextWM consists of 26M and 47M parameters for video pre-training and MBRL,, respectively, while a plain WM consists of 24M and 44M parameters. In terms of training time, takes <377 hours for pre-training of ContextWM over 600K iterations, and fine- tuning of ContextWM with MBRL requires ~4 hours for each run of Meta-world experiments over 250K environment steps, ~3 hours for each run of DMC Remastered experiments over IM environment steps and ~6 hours for each run of CARLA experiments over 1 150K environment steps, respectively. Although ContextWM introduces new components of the model, we find it does not significantly impact the training time, since the context encoder only needs to operate on one frame of observations, and the cross-attention between features also accounts for a relatively small amount of computation. In terms of memory usage, Meta- world experiments require ~ ~8GB GPU memory, and DMC Remastered and CARLA experiments require ~7GB GPU memory, thus the experiments can be done using typical 24GB and 12GB GPUs, respectively.
20

---

Table Hyperparameters in our experiments. We use the same hyperparameters APV 668
Hyperparameter
Value
Image size
64 64 x
Image preprocess
Linearly rescale from [0,255] to [-0.5, 0.5]
Video segment length
25
KI weight 8* Optimizer
1.0
Pre-training from Videos
Adam 37
Learning rate
10
Batch size Training iterations
6 x 10 for SSv2 Human / YouTubeDrivvng
1.2 x 10 for Assembled dataset
Observation size
64 x 64 x
Observation preprocess
Linearly rescale from [0, 255] [-0.5,
Trajectory segment length T
50
Random exploration
5000 environment steps for Meta-world
1000 environment steps for DMCR and CARLA
Replay buffer capacity
10
Training frequency
Every 5 environment steps
Action-free KL weight 3=
0.0
Action-conditional KL weight 3.
1.0
Representative reward predictor weight 3.
1.0
Intrinsic reward weight
1.0 for Meta-world
0.1 for DMCR and CARLA
Fine-tuning with MBRL
Imagination horizon
Discount Y
0.99
X-target discount
0.95
Entropy regularization
x 10-4
Batch size
for Meta-world
16 for DMCR and CARLA
World model optimizer
Adam
World model learning rate
10
Actor optimizer
Adam
Actor learning
10
Critic optimizer
Adam
Critic learning
10-5
Evaluation episodes
10 for Meta-world and DMCR
for CARLA
Additional Experimental Results
D.1
Comparison with Additional Baselines
We compare our approach with state-of-the-art model-free RL method DrO-v2 77 and model- F
based RL method Iso-Dream [52] We adapt their official implementation to all the tasks in our experiments and present the results Figure As shown, our method consistently outperforms state- of-the-art baselines, which demonstrates the merit of pre-training from in in-the-wild videos. Model-free methods such as DrQ-v2 have been shown to perform worse than model-based methods (68 79]. and our results are consistent with this. We also remark that our method differs from Iso-Dream (52 since we separate modeling of context and dynamics at the semantic level while Iso-Dream isolates controllable and noncontrollable parts at the pixel level. Thus, contextual information such as body color has to be modeled by the dynamics branch in Iso-dream, which probably results in worse decoupling of context and dynamics well as inferior performance for challenging tasks. Moreover, compared to Iso-Dream, our model makes a simpler adjustment Dreamer architecture.
21

---

Meta-wordd
DMC Remastered
GARLA
Success Fale (S)
Esladde Rebure
533-
DKG+22
Ino-Dresm
IPV Contextrild
0ut-
62
CortesVVM
10
20
0.
255
63
10.6
150
Environment Steps (x11*)
Environment Steps (x110)
Environment Steps (x10*)
(a) Aggregate results
Lever Pull
Drawer Open
Door Lock
Success Faate (%)
Suczoes Flale (NK)
Sucoess
92
Environment Steps (x10*))
Enviroment Steps (x1(F))
Environmen Steps (ctof)
Button Press Topdown Well
Reach
Dial Turn
20
Succees Rate (5)
Succasss Rate (%5)
Sucocss (%)
W
Cantes/iM
Envtromment Steps px10*)
Envtronmnn Stops (xtof)
Envrorment Steps (xt0")
(b) Meta-world
DMCR Cheetah Run
DMCR Hopper Stand
DMCR Walker Run
002
700
DrOwy
0:3
ContextAM
E,000c Retum
Epicosi Return
400
600
422
100
+20
300
220
100
0.0
02
0.8
1.
0.
0.2
044
0.0
0.
02
0.4
0.5
1.0
Environment Steps Sleps (x101)
Environmant Stops (xildf))
Environment Steps Ox10f)
(c) DMC Remastered
CARLA Clear Noon
CARLA Wet Sureet
CARLA Dynamic Weather
1220
400
1950
Eslopde Retum
500
620
600
200
x20
200
-250
-200
ComceeNN
440
60
25
5.0
7.5
10.0
13.5
15.0
-100
00
25
5.8
75
12.0
-2.5
100
-400
60
255
53
75
10.0
150
Environment Steps (x11*)
Environment Steps (xIDf)
Emirroneent (x10f)
(d) CARLA
Figure 9: Comparison with additional baselines. DrQ-v2 (79 and Iso-Dream [52 (a) Aggregate performance different methods across all tasks on each domain. (b) Learning curves of different methods each task from Meta-world, DMC Remastered, and CARLA, respectively. We report aggregate results across eight runs for each task.
22

---

Lever Pull
Drawer Open
Door Lock
Succeess Palls (%)
Succiss Raie (%)
60
SucSess
40-
42
23
10
16
20
10
28
10
Environment Steps (xtof)
Environment Steps (x10")
Environment Sleps (xtorr)
Button Press Top Wall
Reach
Dial Tum
100
120
Success Rate (66
80
Successs Ratie (86)
40
Succcss Raie (5%)
02
02
IPV (Ens Onyy
20 Environment Stops (x10f)
15
10 13 20
25
23
25
Environment Steps (410")
Environment Steps (xcror)
Figure 10: Learning curves of IPV with ContextWM on Meta-world manipulation tasks when only the pre-trained parameters of the visual encoders and decoder are transferred, i.e., without transferring dynamics knowledge obtained by pre-training (denoted by Enc / Dec Only). We report success rates, aggregated across eight runs for each task.
D.2 Effects of Dynamics Knowledge
To investigate the importance of dynamics knowledge captured by pre-trained ContextWM, we report the performance of IPV with ContextWM when only the pre-trained parameters of the visual encoders and decoder are transferred. Figure indiccaas that only transferring visual encoders and decoder performs worse on four of six tasks. These results align with the observation of APV [68] and highlight that pre-trained world models are capable to capture essential dynamics knowledge beyond learning visual representations solely 49 [3), further boosting visual control learning.
E
Additional Discussions
Particularly strong performance gains on DMCR tasks. Our method obtains considerable performance gains on DMCR tasks. The main reason is that DMCR is a purposefully designed benchmark, which measures visual generalization and requires the agent extract task-relevant information as well as ignore visual distractors [9]. As demonstrated in Figure &c] our ContextWM has the advantage of separately modeling contexts (task-irrelevan) in DMCR) and dynamics (task- relevant in DMCR), which avoids wasting the capacity of dynamics models in modeling low-level visual details. Furthermore, pre-training with in-the-wild wild videos enables our models to eliminate diverse distractors and capture shared motions, which is essential for visual generalization in RI.. In contrast, a plain WM needs to model complicated contexts and dynamics an entangled manner, which adds difficulty to dynamics learning and behavior learning these features. We also emphasize that, motivated by separating contexts and enhancing temporal dynamics modeling, our proposed IPV w/ ContextWM is a seneral-purposs framework and, as shown in our experiments, can obtain adequate performance gains on various benchmarks that have more complicated entangling of contexts and dynamics beyond DMCR.
Discussion with Transformer-bar based world models. While a cross-attention mechanism is lever- aged for context information conditioning, we still use an RNN-based latent dynamics model from Dreamer 21 to model history observations. More powerful sequential backbones such as Transform- ers are orthogonal to our primary technical contribution of explicit context modeling and conditioning. Nevertheless, exploring Transformer based world models [10 47 58) is valuable, not only because of its favorable scalability but also of its flexibility incorporate pre-training on action-free videos by simply changing conditioning variables and to condition the contextualized decoder in a more native way. Overall, the combination of our pre-training framework with a Transformer architecture holds great potential, and we will delve deeper into this aspect in future work.
23

---

Additional Visualizations
Video prediction. We present additional showcases video prediction by plain WM and Contex- IWM, respectively, three video datasets, in Figure 11 On the challenging Something-Somen SSmething-v22 and YouTube Driving datasets, our model provides better prediction quality in comparison to plain WM and well captures dynamics information (e.g., how objects moving) with concentrated cross-attention. However, on the Human3.66 3.6 dataset, although our model better predicts human motions, we find its context cross-attention relatively divergent. This result supports our conjecture that our model pre-trained on the Human3.66 6M dataset lacking in diversity may suffer from overfitting, which results in insufficient modeling of context and dynamics well as inferior performance when fine-tuned with MBRL.
Predieted Ground- truth
(Comae
Context Attention
Predicted
(Phin WM
(a) Something Something-Ve V2
Ground- truth
Predicted (Comas
Context Altention
Predicted
(Pain WM)
(b) Human3.66
Ground- truth
Predicted Context Predicted orim WXD
Attention (ComerrWAi)
(c) YouTube Driving
Figure 11: Future frames predicted by plain WM and ContextWM, respectively, on three tetthe-widd video datasets. We also visualize context attention weight from target region of prediction (red box) to the context frame. We refer to Appendix C.5 for details.

---

Video representation. We present in Figure 12 additional t-SNE visualization similar to Figure g Our CM learns well-distributed representations that clearly separate motion directions while plain WM fails.
Compositional decoding. DMCR Walker Run task.
Figure 13
includes additional compositional decoding analysis on the
Plain WM Pre-trained
ContextWM Pre-trained
Moxing Aeay Iro Suneeting with Your Comeec Approcting Soreething with Way Cameia
Figure 12: Additional t-SNE visualization of average pooled representations of SSv2 videos with two distinct labels, from pre-trained plain WM and ContextWM, respectively.
Coneext
Reconstrction
Open-loop
Groudd
Cheled IPC ue Pluzs WMM Shill CoteocM bsi
IPV wel
Figure 13: Additional compositional decoding analysis on the DMCR Walker Run task.
G Broader Impact
Advancements in world models and training them with in-the-wild videos can raise concerns about the generation of deepfakes, realistic synthetic videos that mimic real events individuals. Additionally, advancements in MBRL can also increase automation domains like robotic manipulation and autonomous driving, bringing efficiency and safety benefits but also leading to job displacement and sociocconomic consequences. However, stages of development and our model is only research prototype, the aforementioned negative social impacts are not expected to manifest the short term.
important to note that as the field still in its early
25


Reviews:
[{"review_id": "px2gqq2Npd", "reviewer": "Reviewer_kJR6", "summary": "This submission presents the contextualized world model (ContextWM), a framework for leveraging in-the-wild videos for pre-training of a world model to be used in model-based reinforcement learning. Following the work from Seo et al. (2022), the authors pre-train an action-free version of the recurrent state-space model (RSSM) with two important modifications: 1) a context encoder processes a randomly sampled frame of the input video to provide context features, which the decoder can directly access to better reconstruct static visual details, enabling the dynamics model, on which the decoder is also conditioned, to focus on temporally varying information. 2) the authors also opt for using a dual-reward predictor during fine-tuning, which predicts the pure task reward in addition to the combined task and video-based intrinsic novelty reward proposed by Seo et al. (2022). This facilitates task-relevant representation learning.\nThe ContextWM is evaluated on Meta-world, the remastered DeepMind Control Suite, and a task with varying weather conditions in the CARLA driving simulator. In most benchmarks, ContextWM shows significant improvements in terms of sample efficiency or final performance. An ablation study and other analytical experiments further show the effectiveness of ContextWM and its design decisions.", "questions": "2. Why did you choose BatchNorm instead of LayerNorm? Did you experiment with both?\n1. maybe use a check mark in the legend of Figure 6b instead of the circle to indicate \"with pretraining\".\n1. It'd be very interesting to see more examples with other contrastive labels for the Video representations experiment in Section 5.5.\n\n## Acknowledgement of rebuttal\nI have read the rebuttal, other reviews. My relatively minor concerns have been addressed, given that the authors have provided some requested clarifications, a discussion of limitations, and additional insightful experiments. I strongly believe this paper should be accepted.", "limitations": "The discussion of limitations is very *limited*. The discussion section mostly mentions future work on scaling and exploring other pre-training objectives.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The proposed contextualized world model is novel and the design decisions are well motivated.\nFor the most part the description of the method and experimental setup is very clear. The performance improvements are significant and in some cases very impressive. The qualitative analysis provides some interesting insights, for instance the clear separation of video representations for two videos with contrastive labels in Figure 8b.", "weaknesses": "1. As the main contribution the cross-attention to the context should be explained in a bit more detail. Since the authors mention U-Nets, I wonder whether the decoder attends to context features at each of the corresponding three stages shown in Table 1 in the supplementary material or only at the decoder input level?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer kJR6 for providing a detailed review, valuable suggestions, and a positive evaluation of our paper.\n\n**Q1**: Details of cross-attentions in the decoder\n\nIn $\\underline{\\text{Appendix C.3 in the supplementary material}}$, we have elaborated the details of how our multi-scale context features are connected to the decoder in a U-Net style:\n\n> The outputs of the last residual block of two stages in the context encoder (stage2 and stage3) before average pooling (thus in the shape of 16 × 16 and 8 × 8, respectively) are passed to the corresponding residual block of the image decoder and used to augment the incoming decoder features with cross-attention.\n\nWe do not use the 32 × 32 features from stage 1 due to the quadratic memory complexity of cross-attention. We will clarify these details in the main text in a future revision.\n\n**Q2**: BatchNorm vs LayerNorm\n\nWe chose BatchNorm as it is the dominant normalization technique in CNNs. Note that our technical contributions are orthogonal to the choice of visual backbones. Exploring transformer-based backbones (e.g., ViTs), which are usually equipped with LayerNorm, is left for future work. We have experimented LayerNorm replacing BatchNorm in our architecture. Results in $\\underline{\\text{Fig. 5 of the global response attachment}}$ indicate that our architecture is robust to the choices.\n\n**Q3**: Additional visualization of video representations \n\nWe have provided additional examples in $\\underline{\\text{Fig. 2 of the global response attachment}}$. Given videos with two distinct labels, '*moving away from something with your camera*' and '*approaching something with your camera*', our ContextWM provides a clear separation of video representations while vanilla WM fails.\n\n**Q4**:  Suggestion on the legends\n\nWe appreciate this suggestion and will use a checkmark instead of the O mark to indicate \"with pre-training\" in a future revision.\n\n**Q5**: Limitations\n\nWe apologize for the limited discussion of limitations. We will add a detailed discussion in a future revision. Please see $\\underline{\\text{Q1 in the global response}}$ for the revised discussion on limitations.\n"}, {"review_id": "nt3t2h77NK", "reviewer": "Reviewer_JHvL", "summary": "This paper proposes a Contexturelized World Model with In-the-wild Video Pretraining, which extends recently proposed action-free pre-training from videos (APV) to the case of contextualized video-prediction models. Specifically, they propose to sample a randomly chosen frame and use it as \"contextualized information\" for the prediction model. The \"contextualized\" information is combined into the prediction model via multi-scale cross-attention mechanisms. Besides, during the model-based RL phase, they propose to predict both pure task reward and the sum of the task reward plus a weighted intrinsic reward. Empirical validations on several well-known benchmarks (Meta-world, DMC, and CARLA) show the superior performance of the proposed method.", "questions": "1. Did you try methods other than random sample selection to compute contextualized information? \n\n2. Did you try a comparison with transformer-based architecture? \n\n3. The logic behind several sentences is hard to capture.  Could you please explain more about these sentences?\n\n- In line 182, \"Nevertheless, ..... Therefore, we propose a dual reward predictor\" => Why does the dual predictor resolve the issue? Why not just balance lambda depending on the task? \n- In line 234, \"indicating that the performance gain of IPV with vanilla WM is primarily due to the intrinsic exploration bonus. \" => The logic is unclear. \n\n[Minor]\n- In eq. 6, How do you compute KL when t=1? \n- In Fig. 5-b, Pre: O should be Pre: ✓ or with Pre or something like that. \n- If I correctly understood, Fig. 3 is a bit misleading as a context variable $c$ is directly input into each frame, rather then recurrent prediction.", "limitations": "A limitation section should be added.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well-written and easy to follow. \n2. While the proposed method is simple, experimental results show a stable performance gain by the proposed methods. \n3. Besides the main contribution, the paper also provides an interesting analysis of the choice of pre-training datasets.", "weaknesses": "1. Technical novelty of the proposed method is not high. The main proposal is to add contextualized information to facilitate a better prediction. Besides, the paper does not discuss how the proposed approach (random sample selection) incorporates contextualized information well. \n\nEspecially, I'm not sure we can call the random variable $c$ a contextualized vector since it is computed via only single frames rather than the full context of the trajectories. In this sense, I'm wondering what happens if we predict/reconstruct observations using the same multi-scale cross-attention architecture but with the image from the same time step for each frame. In the setup, the model does not use contextualized information but has similar architecture to the proposed method, and thus more appropriate for the baseline. \n\n2. Lack of discussion with transformer-based world models. As shortly discussed, several studies incorporate transformer architecture in the prediction model. Since the transformer architecture predicts the future via autoregressive fashion, I think it is more natural to handle the contextualized information. However, the current manuscripts lack discussion on this point and comparison with transformer-based world models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer JHvL for providing an insightful review and valuable comments.\n\n**Q1**: **How we incorporate contextual information**\n\n**Clarification**: We apologize for using the ambiguous term 'context' without elaborate clarification. Videos and visual control trajectories are **spatiotemporal** data. They have contexts in the temporal dimension (namely the history) and contexts in the spatial dimension (namely visual details, e.g., colors, shapes, and layouts of objects). 'Contexts' in our paper stands only for **spatial contexts**. A context frame is selected to extract (static) contextual information in a multi-scale manner, which encourages the latent dynamics model to focus on temporal dynamics instead of wasting model capacity on capturing low-level visual details. It is prevalent in the literature to utilize a reference image to condition generative models of videos [1, 2]. For **temporal context** modeling, we use standard RSSMs from Dreamer to model history observations. More powerful sequential backbones such as Transformers can be explored, but it is orthogonal to our technical contribution of explicitly modeling (spatial) contexts.\n\n**Novelty**: Reviewer kJR6 and wPqn both recognize our methods as innovative and well-motivated. To the best of our knowledge, we are the first to separately model contextual information for world models to handle complicated spatiotemporal data.\n\n**Empirical evidence**: We have shown that our model successfully unleashes the power of in-the-wild video pre-training and obtains significant gains on downstream tasks. Qualitative evaluation in $\\underline{\\text{Fig. 8 of main paper}}$ and $\\underline{\\text{Fig. 1 of the global response attachment}}$ also supports that our model can separate contexts and dynamics.\n\n**Reviewer-proposed baseline**: If understood correctly, the proposal from the reviewer, which uses the image from the same step as the context for each frame, can hardly learn useful representations for MBRL. Note that we extract multi-scale shortcuts from the context frame in a U-Net manner. Learning to reconstruct each frame conditioned on one fixed context encourages the model to learn temporal variations. However, when conditioned on the context frame from the same step for each frame, the model can learn to trivially copy low-level feature shortcuts for reconstruction. Our experiments of this baseline support our justification, where **the norm of RSSM features rapidly shrinks to zero during training**.\n\n[1] Singer et al. Make-a-video: Text-to-video generation without text-video data.\n\n[2] Esser et al. Structure and content-guided video synthesis with diffusion models.\n\n**Q2**: Other context frame selection methods.\n\nAs shown in $\\underline{\\text{Fig. 3 of the global response attachment}}$, we have experimented with the first or the last frame as the context but found no significant performance difference. We have also tried randomly selecting three frames as the context and still obtained no performance gain, which indicates that a single frame is adequate for context modeling in our experimental benchmarks.\n\n**Q3**: Discussion and comparison with transformer-based architecture\n\nAs discussed in Q1, transformer-based architecture is immature and **orthogonal** to our technical contribution. A combination of powerful transformer architecture and our framework has the potential to improve performance further, which is left for future work. \n\nNevertheless, we have conducted preliminary comparisons to a transformer-based method, TransDreamer [3], without pre-training. Results are presented in $\\underline{\\text{Fig. 4 of the global response attachment}}$. We observe that even equipped with transformers, TransDreamer performs similarly to Dreamer and is inferior to our method. We do not compare with recently published work (IRIS, NeurIPS 2022 and TWM, ICLR 2023) since they only support discrete control (Atari) and utilize different actor-critic learning schemes, which cannot be directly compared with our work on continuous control.\n\n[3] Chen et al. TransDreamer: Reinforcement learning with transformer world models. 2022.\n\n**Q4**: Clarification on sentences\n\n- Line 234: Our 'IPV w/ vanilla WM' baseline mainly has two major differences with DreamerV2: in-the-wild video pre-training and video-based intrinsic bonus. Although 'IPV w/ vanilla WM' outperforms DreamerV2, our ablation study shows that this baseline can only benefit from the intrinsic bonus, but not in-the-wild pre-training.\n- Line 182: As shown, the intrinsic bonus is essential for learning efficiency, but it is computed using an ever-changing replay buffer during training. An additional predictor for pure task reward can force the dynamics model to encode task-relevant information, regardless of the intrinsic reward drifts, which helps representation learning. We do not downweight intrinsic reward since it needs extra hyperparameter tuning and, more importantly, may hurt exploration and learning efficiency.\n\n**Q5**: Minor questions\n\n- KL term: When t=1, the terms should be $\\text{KL}[q(z_1|o_1)\\|p(\\hat{z}_1)]$ and $\\text{KL}[q(s_1|z_1)\\|p(\\hat{s}_1)]$. Following the implementation of Dreamer, priors and posteriors of $z_1$ and $s_1$ are predicted with dummy previous states $z_0, s_0$ (all-zero initial states of RSSM) and actions $a_0$ (all-zero too), which unifies the implementations for t=1 and t>1.\n- Legends and Fig. 3: We appreciate the suggestions and will use check marks for the legend and revise Fig. 3 to make it clearer in a future revision.\n\n**Q6**: Limitations\n\nWe apologize for the limited discussion of limitations. We will add a detailed discussion in a future revision. Please see $\\underline{\\text{Q1 in the global response}}$ for the revised discussion on limitations.\n\n"}, {"review_id": "il2VCmAVvJ", "reviewer": "Reviewer_ajER", "summary": "This paper studies whether large-scale in-the-wild datasets can be used to pre-train world models for efficient downstream reinforcement learning. Specifically, they introduce Contextualized World Models (ContextWM), an architecture specifically designed to learn to separate context and dynamics modeling. Their experimentation reveals that the proposed methodology outperforms DreamerV2 on a variety of downstream tasks.", "questions": "- Is it possible to qualitatively demonstrate that ContextWM is able to separate context and dynamics modeling?\n- For some tasks, pre-training and choice of data to pre-train with makes a huge difference, e.g. DMC Re-mastered. For others, not so much, e.g. CARLA. Is there a sense of why this is the case?", "limitations": "There is no discussion on the limitations of the work.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- Well-written. The paper is well-written, and the figures aid in the understanding of the methodology.\n- Variety of experiments. The authors conduct a variety of experiments, comparing not just to DreamerV2, but also analyzing the effects of pre-training, architecture choices, dataset domain, etc. Furthermore, both quantitative and qualitative comparisons are included.\n- Strong experimental. Relative to DreamerV2, the proposed methodology achieves a strong performance -- the gap seems to be particularly large for the DMC Remastered tasks.", "weaknesses": "- Missing comparison to prior work. APV [49] is the most similar prior work which this work builds off of (and seems to be the SOTA in this space), and yet the proposed method is not compared to APV. Many of the tasks used, hyper-parameters, and evaluation protocols adopted are from APV, allowing for a comparison, yet somehow, this comparison is omitted. Looking at the results plots in the APV paper, visually, the performance of ContextWM seems similar to that of APV (and in some cases clearly worse, e.g. in the dial turn task). While one may claim that a comparison to APV may be unfair because both the data and the model would be different, this would still reveal whether the ability to use a larger amount of in-the-wild data as well as the changes to the model architecture are actually beneficial. Furthermore, training APV on this large-scale data and seeing whether ContextWM outperforms it would be an experiment which would reveal whether the architectural changes proposed are significant. Finally, APV primarily compares to DreamerV2 as this was the SOTA when APV was published -- now, DreamerV3 seems to be the SOTA, so a comparison to DreamerV3 should be conducted.\n- Incomprehensive ablations. It is not clear which one environment the ablation study is conducted on. It would be a lot more convincing if the ablation study was done across tasks, and if the trends held true across tasks as this would alleviate concerns of cherry-picking a task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer ajER for providing a thorough review and valuable questions. \n\n**Q1**: **Comparison with APV**\n\nWe **respectfully disagree with the comments that we do not compare with APV**. We apologize for not clarifying that **our 'IPV w/ vanilla WM' in $\\underline{\\text{Fig. 5 and 6 of main paper}}$ is the APV baseline trained on the same data**, equipped with the stacked latent model and intrinsic bonus proposed by APV. We brand it with the new name to emphasize that it is pre-trained on **I**n-the-wild data rather than only **A**ction-free data. Furthermore, since neither the one-layer RSSM in Dreamer nor the stacked RSSM in APV has a contextualized component as ContextWM, we named them vanilla WM at the beginning of our experimental section. We will change to more proper names in a future revision. In a word, **we have already trained APV on large-scale data and demonstrated that the architectural changes proposed are significant**.\n\nWe also note that the performance gap with the original APV comes from different pre-training datasets. **Using the same curated RLBench dataset, our ContextWM (see $\\underline{\\text{Fig. 7b of main paper}}$) can outperform originally reported APV results.** (Interestingly, we also find that the dial turn task can only benefit from RLBench data, regardless of the architecture.) It is unsurprising that pre-training data from a similar domain can further benefit downstream tasks. Nevertheless, **our methodology contribution unleashes the power of diverse video datasets instead of curated domain-specific ones to enable general-capable world model pre-training**. While the SSv2 dataset does not help downstream tasks in the original APV, our ContextWM pre-trained with it has been shown to benefit various control tasks. \n\n**Q2**: Comparison with DreamerV3\n\nSince our ContextWM is built upon DreamerV2, it is natural to compare it with DreamerV2 to reveal the significance of in-the-wild pre-training and the proposed architecture. **Our technical contributions are orthogonal to specific model-based RL methods** and can also combine with DreamerV3 to further improve performance, which is left for future work. Nevertheless, we conduct preliminary comparisons to DreamerV3 without pre-training. Results are presented in $\\underline{\\text{Fig. 4 of the global response attachment}}$. We conclude that even with several improved training techniques, DreamerV3 is still inferior to our method, showing the significance of in-the-wild video pre-training and explicit context modeling.\n\n**Q3**: **Clarification on ablation study**\n\nFollowing the protocol of Agarwal et al. [1] and APV,  **we conducted the ablation study on all the tasks and reported aggregated results**. Explanations of our ablation study results can be found in  $\\underline{\\text{Sec. 5.1}}$ and the captions of $\\underline{\\text{Fig. 5, 6, 7}}$. We will clarify it further in a future revision.\n\n[1] Agarwal et al. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 2021.\n\n**Q4**: **Qualitative evaluation**\n\nWhile it is challenging to learn fully separated representation, we have provided qualitative evaluation in $\\underline{\\text{Sec. 5.5 and Fig. 8 of main paper}}$ to demonstrate the ability of ContextWM to separate contexts and dynamics. We also provide additional demonstrations in **$\\underline{\\text{Fig. 1 of the global response attachment}}$** to show that our model finetuned on the DMCR domain successfully learned disentangled representations of contexts and dynamics. For further details, please refer to the qualitative analysis part of $\\underline{\\text{Q2 in the global response}}$.\n\n**Q5**: Particularly significant performance gains on DMCR tasks\n\nThanks for your insightful question. Please refer to $\\underline{\\text{Q2 in the global response}}$ for the detailed response.\n\n**Q6**: Limitations\n\nWe apologize for the limited discussion of limitations. We will add a detailed discussion in a future revision. Please see $\\underline{\\text{Q1 in the global response}}$ for the revised discussion on limitations.\n"}, {"review_id": "RzxiMVs1tE", "reviewer": "Reviewer_wPqn", "summary": "Learning a world-model that can generalize to different domains and tasks is difficult. The authors enhanced an existing framework for pre-training world models using in-the-wild videos, which can be fine-tuned on downstream tasks. In particular, the authors introduce a contextual encoder which helps in disentangling temporal dynamics from static contexts. Additionally, they include a cross-attention mechanism and dual reward predictors to improve the learning of task-relevant representations.", "questions": "●\tWhat does the context encode? The motivation and the qualitative analysis are convincing, but it would also be useful to test this further. This could be done via a decoding analysis. In particular, it would be very interesting to see the difference between the context and the dynamic in the case of the DMC remastered task, where the contextWM provides a significant advantage. \n\n●\tWhy do you think is the gap in performance to prior methods particularly strong for DMCR?\n\n●\tLooking at figure 6b (bottom), the performance of the pre-trained ContextWM on the SSv2 dataset for the CARLA driving task seems to outperform the one that is not-pretrained. However, the effects of the dataset domain of figure 7b (right) shows that there is almost no difference between the one pre-trained on SSv2 and the one without pre-training. What am I missing?\n\n●\tDoes the contextWM help to achieve better generalization? Could one, e.g., purposefully change the color, size or shape of the object for the meta-world and achieve better performance than the WM? \n", "limitations": "Fine", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The manuscript is very well written and structured. \n\n●\tThe idea of using a context to encode static information is novel, well-motivated and it shows good results in the DMC remastered task\n\n●\tLearning world models on in the wild videos is hard (and so far does not help downstream tasks, as shown by Seo et al. [49]) - The extension of the Action-Free Pre-training from Videos approach to in-the-wild videos is convincing and the context modulation via a U-Net for reconstruction is innovative. \n\n●\tThe authors perform several relevant ablations to illustrate the role of the different proposed components. \n", "weaknesses": "Overall, the context is only used together with the latent dynamic to decode the image. Is there a clear reason why the context is not taken into account for predicting the reward (in Fig. 3b the arrow of the context that goes to the reward seems to be misleading)? As the authors stated, the context might implicitly encode some important information about the task, e.g., the static position of an object, which can be helpful during the fine-tuning phase. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer wPqn for providing a detailed review and insightful questions. \n\n**Q1**: Utilizing contexts for predicting the reward\n\nWe agree that, in general, contextual and dynamics information are both important for task-relevant predictors (reward predictor, actor, and critic). We can also design cross-attention mechanisms for these MLP predictors. However, since contextual information in our architecture has a complicated structure (multi-scale CxHxW feature maps), this may bring extra design and implementation efforts, additional hyperparameters, and computation costs. To maintain a simple architecture and fair comparisons, we opt to predict rewards with only dynamics features and utilize our proposed dual reward predictor structure to encourage completely task-relevant feature encoding, which has been shown to work well in experimental benchmarks. Incorporating contexts and dynamics information for reward prediction and behavior learning in more complicated tasks is a promising future direction.\n\nFor the misleading Fig. 3b, we will revise it to make it clearer in a future revision. Thanks for your valuable suggestion.\n\n**Q2**: Visualization in the case of the DMCR tasks\n\nThanks for the valuable suggestion of a decoding analysis on the DMCR domain. To demonstrate the difference between context and dynamics, we conduct a **compositional decoding analysis** by sampling a random frame from another trajectory to replace the original context and leave dynamics unchanged. Our ContextWM shows excellent compositionality as it can correctly combine the new contextual information with the original dynamics information. We conclude that in this domain, the context encodes static visual factors such as the agent's body color, the background's texture, etc. For further details, please refer to the qualitative analysis part of $\\underline{\\text{Q2 in the global response}}$. \n\n\n**Q3**: Particularly strong performance on DMCR tasks\n\nThanks for your insightful question. Please refer to $\\underline{\\text{Q2 in the global response}}$ for the detailed response.\n\n**Q4**: Effects of pre-training with SSv2 on CARLA\n\nWe apologize for a mistake of plotting: **the 'w/o Pre-train' curve in Fig. 7 (CARLA) should be the same as the 'ContextWM (Pre: X)' curve in Fig. 6 (bottom)**, but was incorrectly plotted as the 'vanilla WM (Pre: O)' curve in Fig. 6 by mistake. We have carefully checked the figures to ensure there are no other mistakes and will correct this in a future revision—many thanks for pointing it out.\n\n**Q5**: ContextWM promotes generalization\n\nWe believe that our ContextWM can promote generalization due to better design of context and dynamics modeling. While changing the color, size, or shapes in Meta-world is difficult, we have conducted similar experiments in another benchmark, DMC Remastered, which randomly resets all visual factors on the initialization of each training and evaluation episode. As shown in $\\underline{\\text{Fig. 6 of main paper}}$, ContextWM outperforms vanilla WM significantly on DMCR, both with or without pre-training. These results demonstrate that **ContextWM can achieve better generalization and performance in unseen visual environments**. Measuring out-of-distribution generalization ability (e.g., training on standard DMC/Meta-world and testing on visually modified ones) of ContextWM and vanilla WM is an interesting future direction."}, {"review_id": "SZMiwu40Jf", "reviewer": "Reviewer_MYvK", "summary": "This paper presents a method for learning world models from in-the-wild videos. By utilizing a context encoder to capture contextual information, the proposed method explicitly models both the context and dynamics to facilitate knowledge transfer across scenes. Experiments are performed on various simulation benchmarks such as Meta-world, DMC Remastered, DeepMind Control Suite, and CARLA. ", "questions": "The authors propose to randomly select a frame for predicting the context feature, have the authors tried to use a fixed frame (e.g., the first frame or the last frame of the video clip)? Would the context feature extracted from the different frames be consistent with each other? And how would choosing different frames affect the final performance of the RL training? \n\nIt is assumed that the learned context feature captures the static properties of objects (e.g., colors, shapes), would modifying the context feature allow us to generate diverse video frames (e.g., changing the color of the object)? It would be interesting to show more temporal consistent video frames by editing the context feature. ", "limitations": "There is no discussion of limitations and failure cases. ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The motivation for learning better world models by disentangling context information and dynamics is clear and seems reasonable to me; \n\nThe proposed method for learning context information is simple and straightforward; \n\nThe paper is well-organized and easily read. ", "weaknesses": "The proposed framework is based on the assumption that the context information lies equally in each frame, however, it is very likely that some moving objects might be occluded at some time. There are no specific designs for handling these situations; \n\nThe experiments mainly validate the sample efficiency of the RL training process, there is no sufficient ablation study on the learned context information; \n\nCompared to the vanilla WM baseline, the performance of RL training on the Meta-world benchmark seems not very impressive; \n\nThe predicted video frames seem not very promising; \n\nThere is no discussion of limitations and failure cases. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer MYvK for providing insightful comments and questions. \n\n**Q1**: Discussion and ablation study on context frame selection\n\n**Single vs. multiple**: We agree with the reviewer that, in general, a single context frame cannot provide perfect contextual information, and it is challenging to learn fully disentangled contextual and dynamics representations. But our intuition is that **propagating most of the contextual information through a separate encoder can facilitate shared dynamics knowledge transfer between visually distinct pre-training and downstream domains**. Important dynamics information, such as occluded moving objects, can also be captured by the dynamics branch, as RSSMs can handle partial observability. As a first step towards broadly applicable world model pre-training, our experiments in various domains support that a single context frame can overcome complicated context distribution to unlock positive transfer. We have discussed the limitations and future work to utilize multiple frames in more complicated tasks (see $\\underline{\\text{Q1 in the global response}}$).\n\n**Feature consistency**: Our random selection strategy can encourage consistency between context features extracted from different frames. We note that they are not required to be strictly consistent, as our cross-attention mechanism allows contextual information to propagate adaptively between different frames (see $\\underline{\\text{Fig.8 of main paper}}$ for example). \n\n**Experiment results**: To support the discussion, we have experimented using a fixed frame (the first and last one) and multiple frames (randomly sampled three frames concatenated as inputs of the contextual encoder). Results are presented in **$\\underline{\\text{Fig. 3 of the global response attachment}}$**. We conclude that **different context frame selection schemes do not significantly affect performance**. Utilizing multiple frames does not provide benefits, probably because the experimental environment is simple or the way incorporating multiple frames is crude.\n\n**Q2**: Predicted video frames seem not very promising\n\nWe note that SSv2 and YoutubeDriving datasets are large-scale in-the-wild datasets. Generating high-fidelity videos on these domains is especially difficult, and little literature has successfully achieved this. Developing generative models of videos is still a rapidly evolving and immature field.\n\nHowever, as pointed out at the beginning of the paper, through large-scale video pre-training of a world model, **our work aims to boost sample efficiency of downstream model-based RL, rather than fidelity of video prediction**. In order to learn beneficial representation for MBRL, we believe world models should focus on essential dynamics information, such as object positions and motions, instead of low-level visual details. This naturally motivates our decision-oriented design facilitating separate contexts and dynamics modeling. We present predicted video frames in $\\underline{\\text{Fig. 8 of main paper}}$ in order to illustrate that ContextWM can learn important dynamics features from large-scale pre-training, which benefits downstream MBRL. On simpler datasets, e.g., Human3.6M, our model can make much better predictions. However, as shown in $\\underline{\\text{Fig. 7b of main paper}}$, pre-training on this dataset lacking in diversity can even hurt our ultimate goal: sample efficiency of visual control learning. Advances in generative models can help develop stronger backbones for world models but are not the focus of this work and are orthogonal to our contributions.\n\n**Q3**: Generating diverse video frames\n\nThanks for the insightful question. We have experimented with modifying the context feature to generate novel videos. We use the DMCR domain as our testbed as it supports modifying visual factors. While it is difficult to manipulate context features directly, we have done a workaround by sampling a frame from another trajectory to extract contextual information. In this way, we can modify the agent's color and the background's texture. As shown in $\\underline{\\text{Fig. 1 of the global response attachment}}$, ContextWM can make temporal consistent video predictions by correctly combining the new contextual information with the original dynamics information. For further details, see the qualitative analysis part of $\\underline{\\text{Q2 in the global response}}$. \n\n**Q4**: Performance gain on Meta-world\n\nFollowing prior work, Meta-world performance is measured by the success rate of 10 episodes, which naturally has a high variance. We have made a great effort to solidify our results, including massive repeated experiment runs. In $\\underline{\\text{Fig. 5b of main paper}}$, we have demonstrated the **statistical significance of our improvement over vanilla WM with a clear margin, aggregated over 48 runs on six Meta-world tasks**, following the protocols of APV and Agarwal et al. [1]. For particular tasks, e.g., Drawer Open, our method can learn with only half of the environment interactions, compared to the baseline.\n\n[1] Agarwal et al. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 2021.\n\n**Q5**: Limitations and failure cases\n\nWe apologize for the limited discussion of limitations. We will add a detailed discussion in a future revision. Please see $\\underline{\\text{Q1 in the global response}}$ for the revised discussion on limitations.\n\nFor failure cases, we have observed that it does not always provide significant gains when pre-training our model on video datasets from different domains or of different amounts (e.g., Human3.6M). We have discussed possible reasons in corresponding paragraphs in the experimental section."}]

Author rebuttal:
## Global Response to All Reviewers

We would like to thank the reviewers for their detailed comments. This paper aims to pre-train a broadly generalizable world model from in-the-wild videos to boost sample-efficient learning of downstream visual control tasks. Extensive experiments on large-scale video datasets and various visual control domains have demonstrated the effectiveness of our proposed In-the-wild Pre-training from Videos (IPV) with ContextWM.

We have made every effort to address all the reviewers' concerns and responded to the individual reviews below. We have also answered common questions raised by the reviewers in this global response. 

Note that **all the new figures supplementary to all responses are included in the PDF attachment of this global response**. We only present results on part of the tasks due to limited time and computational resources.

**Q1**: Revise the discussion on limitations

We apologize for the limited discussion of limitations. We will add the following expanded discussion on limitations in a future revision:

> **Limitations and future work.** One limitation of our current method is that a randomly selected single context frame may not be sufficient to capture complete contextual information of scenes in the real world. Consequently, selecting and incorporating multiple context frames as well as multimodal information [47] for better context modeling need further investigation. Our work is also limited by medium-scale sizes in terms of both world models and pre-training data, which may hinder learning broadly applicable knowledge. Given that, an important direction is to systematically examine the scalability of our method by leveraging scalable architectures like Transformers [36, 54] and massive-scale video datasets [12, 37]. Lastly, our work focuses on pre-training world models via generative objectives, which use massive parameters inefficiently on image reconstruction to model intricate contexts. Exploring alternative pre-training objectives, such as contrastive learning [40, 7] or self-prediction [52], could further release the potential of IPV by eliminating heavy components on context modeling and focusing on dynamics modeling.

**Q2**: Particularly significant performance gains on DMCR tasks

Our method obtains considerable performance gains on DMCR tasks. The main reason is that DMCR is a purposefully designed benchmark, which measures visual generalization and requires the agent to extract task-relevant information as well as ignore visual distractors. Our ContextWM has the advantage of separately modeling contexts (task-irrelevant in DMCR) and dynamics (task-relevant in DMCR), which avoids wasting the capacity of dynamics models in modeling low-level visual details. Furthermore, pre-training with in-the-wild videos enables our models to eliminate diverse distractors and capture shared motions, which is essential for visual generalization in RL. In contrast, vanilla WM needs to model complicated contexts and dynamics in an entangled manner,  which adds difficulty to dynamics learning and behavior learning on these features.

**Qualitative analysis**: To demonstrate the ability of ContextWM to separate context and dynamics modeling, we provide additional video prediction results in **$\underline{\text{Fig. 1 of the global response attachment}}$**. In the _Context Shift_ row of the figure, we sample a random frame from another trajectory to replace the original context and leave dynamics the same as the _IPV w/ ContextWM_ row to conduct a **compositional decoding analysis**. We can see that after shifting the context, ContextWM correctly combines contextual information from the new context with dynamics information from the original trajectory. These results show that our model finetuned on the DMCR domain has successfully learned disentangled representations of contexts and dynamics. The vanilla WM, on the other hand, suffers from learning entangled features and, as a result, makes poor predictions about the environment transitions.

**General-purpose framework**: We also emphasize that, motivated by separating contexts and enhancing temporal dynamics modeling, our proposed IPV w/ ContextWM is a general-purpose framework and, as shown, can obtain adequate performance gain on various benchmarks that have more complicated entangling of contexts and dynamics beyond DMCR.

**Q3**: Suggestion on the legends and Fig. 3 of main paper

Thanks for the valuable suggestions. We will use check marks for the legends to indicate 'with pre-training' and revise Fig. 3 to make the architecture clearer in a future revision.

Meta-review: The paper investigates the pre-training of world models using in-the-wild videos to enhance the sample efficiency in model-based reinforcement learning (MBRL) for tasks such as robotic manipulation and autonomous driving. To tackle the complexities found in these videos, it introduces Contextualized World Models (ContextWM), which explicitly model both the context and dynamics to facilitate knowledge transfer across distinct scenes.

Overall, reviewers have praised the paper for its clarity, methodological soundness, and compelling experimental results. The novel idea of contextual encoding is well-received, and the experiments showcase significant performance improvements in multiple domains. Some reviewers had concerns regarding technical details and comparisons to prior work, but the rebuttal has addressed these issues to some extent. There are still some points that could be clarified further, such as the role of cross-attention in context modeling and a deeper discussion with transformer-based world models. Yet, the consensus leans towards acceptance, particularly given the additional clarifications and experiments presented in the rebuttal. Therefore, the paper is recommended for acceptance with the understanding that the authors will continue to refine it by addressing the remaining weaknesses.
Decision: Accept (poster)