iVideoGPT: Interactive VideoGPTs are Scalable World Models
Authors: Jialong Wu
Shaofeng Yin
Ningya Feng
Xu He
Dong Li
Jianye HAO
Mingsheng Long
Venue: NeurIPS 2024 poster (2024)
Keywords: world model
model-based reinforcement learning
video prediction
visual planning
Primary area: reinforcement_learning
TL;DR: We propose iVideoGPT, an autoregressive transformer architecture for scalable world models, pre-train it on millions of trajectories and adapt it to a wide range of tasks, including video prediction, visual planning, and model-based RL.

Abstract: World models empower model-based agents to interactively explore, reason, and plan within imagined environments for real-world decision-making. However, the high demand for interactivity poses challenges in harnessing recent advancements in video generative models for developing world models at scale. This work introduces Interactive VideoGPT (iVideoGPT), a scalable autoregressive transformer framework that integrates multimodal signals—visual observations, actions, and rewards—into a sequence of tokens, facilitating an interactive experience of agents via next-token prediction. iVideoGPT features a novel compressive tokenization technique that efficiently discretizes high-dimensional visual observations. Leveraging its scalable architecture, we are able to pre-train iVideoGPT on millions of human and robotic manipulation trajectories, establishing a versatile foundation that is adaptable to serve as interactive world models for a wide range of downstream tasks. These include action-conditioned video prediction, visual planning, and model-based reinforcement learning, where iVideoGPT achieves competitive performance compared with state-of-the-art methods. Our work advances the development of interactive general world models, bridging the gap between generative video models and practical model-based reinforcement learning applications. Code and pre-trained models are available at https://thuml.github.io/iVideoGPT.

iVideoGPT: Interactive VideoGPTss are Scalable World Models
Jialong Wu W 12; Ningya Feng', Xu He Dong Li Jianye Hao Mingsheng Long 18
School of Software, BNRist, Tsinghua University. ZZhili College, Tsinghua University *Huawei Noah's 's Ark Lab, *College of Intelligence and Computing, Tianjin University wujiallong0229gmai.. com, ysf228mails. t edu. cn, mingshengetsingguu. .aaaaaeedduooo
Abstract
World models empower model-basee agents to interactively explore, reason, and plan within imagined environments for real-world decision-making. However, the high demand for interactivity poses challenges in harnessing recent ad advancements in video generative models for developing world models scale. This work intro- duces Interactive VideoGPT (iVideoGPT), scalable autoregressive transformer framework that integrates multimodal signals visuul observations, actions, and re- wards-into into sequence of tokens, facilitating an interactive experience of agents via next-token prediction. iVideoGPT features novel compressive tokenization tech- nique that efficiently discretizes high-dimensional visual observations. Leveraging scalable architecture, we are able to pre-train iVideoGPT on millions of human and robotic manipulation trajectories, establishing a versatile foundation that is adaptable to serve as interactive world models for a wide range of downstream tasks. These include action-conditions video prediction, visual planning, and model- based reinforcement learning, where iVideoGPT achieves competitive performance compared with state-oftthe-a methods. Our work advances the development interactive general world models, bridging the gap between generative video models and practical model- basel-based reinforcement learning applications. Code and pre-trained models are available at https: ttttps:/ttuume //thmmtlll... camuuml.gttuutt github. io/ iVideoGPT.
Introduction
Recent years have witnessed remarkable advancements in generative models of multimodal contents, including text (1, audio 19) and images (22). with video generation
emerging
frontier [1]. particularly significant application of these generative video models, learned in an unsupervised way diverse Internet-scale data, is to construct predictive world models [53 28 at scale. These world models are expected to accumulate commonsense knowledge about how the world works, enabling the prediction of potential future outcomes (e.g., visual observations and reward signals) based on the actions of agents. By leveraging these world models, agents employing model-based reinforcement learning (RL) can imagine, reason, and plan inside world models [20] 29 thus acquiring new skills more safely and efficiently with handful of trials in the real world.
Despite the fundamental connection, significant gaps remain between generative models for video generation and visual world models for agent learning. One primary challenge achieving the best of both interactivity and scalability. In model-based RL, world models predominantly utilize recurrent network architecture. This design naturally allows the transition of observations or latent states conditioned actions in each step, facilitating interactive behavior learning (29) 80 34). However, these recurrent models mostly focus on games or simulated environments with simple
'Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Human Manipulation
Video
Prediction
Trajectories
IVideoGPT
Visual Planning
Robot Manipulation Trajectories
Visual
Model-based RL
Figure 1: Practical applications of iVideoGPT, which is designed scale, allowing pre-training on millions of human and robotic manipulation trajectories. This results in single, versatile foundation of interactive world models, adaptable to wide range downstream tasks.
On the other hand, Internet-scale video generative models 33 [ can synthesize realistic long videos that are controllable via text descriptions [109] or future action sequences [TOI] at the beginning of generation. Although suitable for high-level planning [19, their trajectory-level interactivity does not provide sufficient granularity needed by agents to intervene step-by ste- during the simulation to learn precise basic skills efficiently. This dilemma naturally raises the question:
visuals and have limited capability to model complex, in-the-wild data at scale 448] 8
How can we leverage the advancements in scalable video generative models for developing interactive visual world models?
In this work, we explore world models that are both interactive and scalable within GPT-like autoregressive transformer framework [90 [90] Pioneering efforts have been made recently through diffusion models [102] and masked generative models [12]. Nevertheless, utilizing autoregressive transformers offers distinct advantages such as seamless integration with the established Large Lan- guage Model (LLM) ecosystem (TTO and greater flexibility in handling diverse conditions without the need for specific architectural modifications like adapter modules [77 [07]. We present Inter- active VideoGPT (iVideoGPT). a scalable world model architecture that incorporates multimodal signals, including visual observations, actions, and rewards, in an interactively autoregressive manner. Unlike multimodal LLMs that discretize visual observations into tokens frame-by-frame using image tokenizers [55] key innovation of iVideoGPT for enhancing scalability is to learn compressive tokenization for each observation conditioned on rich contextual observations, achieving an asymp- totic 16x reduction in token sequence length. We highlight that more compact video tokenization could not only facilitate more efficient training and generation but also enhance video quality. This is achieved by decoupling context from dynamics, allowing the model to focus on predicting the motion of objects while maintaining temporal consistency within the scene [99].
We demonstrate series of practical applications of i iVideoGPT for visual robotic manipulation, as illustrated Figure [ Mirroring the two-phase approach popularized by LLMs, our method involves pre-training followed by domain-specific adaptation. During pre-training, iVideeGGT is scalable for action-free video prediction across a mixture of over one million robotic and human manipulation trajectories [70 25] The pre-trained iVideoGPT serves as single, adaptable foundation of interactive world models for various downstream tasks, such as action-conditioned vidco prediction [6],6, visual planning (86). and visual model-based RL [105). Additionally, showcase the pre-trained transformer's preliminary zero-shot video generation capability without fine-tuning, requiring only tokenizer adaptation for unseen domains. We further explore variant of iVidcoGPT for goal-conditiones video prediction, underscoring the flexibility of sequence modeling.
The main contributions of this work can be summarized as follows:
We introduce Interactive VideoGPT (iVideoGPT), an autoregressive transformer architecture for scalable world models, which features compressive tokenization for visual observations.
We pre-train iVideoGPT manipulation trajectories and adapt it to nomain-speciffi tasks. The pre-trained models have been publicly available to encourage further research.
large-scale dataset comprising millions of robotic and human
Extensive experiments covering video prediction, visual planning, and visual model-basee RL demonstrate that iVideoGPT can simulate accurate and realistic experiences and provide competitive performance compared with statc-of-ther the-art methods.

---

OUT
02
03
0r
O4
Recurrent
model
Transformer or Diffusion model
Autoregressive Transformer
ApT-1
Q2
dy-1
tokens / latent features
01
01
Recurrent world models
(b) Video generation models
(c) Interactive video prediction models
Figure 2: Conceptual comparison among architectures, illustrated using single context frame
TTo = 1) for simplicity. (a) Recurrent architectures for world models like Dreamer [29 and MuZero
80 provide step-level interactivity but limited scalability. (b) Recent video generation advancements
like VideoGPT [01]] and Stable Video Diffusion g 7 use non-causal temporal modules that can
(c) Our model utilizes an autoregressive transformer that separately maps each step into a sequence of tokens, achieving both scalability and interactivity.
only offer trajectory-Ievel interactivity.
2 Problem Formulation
world model is an internal model learned by the agent to simulate the environment. This environment is typically modeled partially observable Markov decision process (POMDP) (8,0, o, A. p,r,,, r.)). At each step, St represents the underlying state of the environment, and Or ((s;) is the observation received by the agent, only providing incomplete information of st- After taking an action a₂ E A. p($++1(11 (5t, az) defines the transition probability from state &t to $141- The agent also receives immediate rewards Tz+1 r(s,, a,), and aim to learn policy such that a ~ x()aa) maximizing the 7-discounted accumulated rewards Ep.= 1
While world models can be learned from many types of data, video is one modality that is task- agnostic, widely available, and embeds broad knowledge that can be learned in a sule-vuper way. Thus, we formulate learning world models for visual control as interactive video prediction problem [102 [2] where
f W x3 is the space of video frames Concretely, given a short history visual observations of To frames 01:Ta at each step To ........... T 1, the agent takes an action 0g based on its policy and previous imagined observations, and then the world model need to approximate and sample the transition p(or+1,T2H T2+1 01:t, @1g:t) to feedback the agent.
As depicted in Figure a majority of advanced video generation models [01 8 104], including VideoGPT, can not deal with the interactive video prediction problem because they design non-causal modules fusing information along the temporal dimension, lacking the ability for causal, intermediate action control during generation (see extended discussion in Appendix C.2) Existing world models
in the literature of MBRL 22 80], such as Dreamer, utilize recurrent architecture but lack scalability.
Interactive VideoGPT
In this section, introduce Interactive VideoGPT, scalable world model architecture with great flexibility to integrate multimodal signals, including visual observations, actions, rewards, and other potential sensory inputs. At its core, iVideoGGT consists of compressive tokenizer to discretize video frames and an autoregressive transformer predicting subsequent tokens (Section 3.T). This model can acquire common knowledge of motions and interactions in various scenes through pre- training on diverse human and robotic manipulation videos (Section 3.2) and then effectively transfer to downstream tasks incorporating additional modalities (Section
3.1 Architecture
Compressive tokenization. Transformers particularly excel in operating over sequences of discrete tokens. VQGAN (22) is commonly used visual tokenizer that converts from pixels to discrete tokens. Instead of using an image tokenizer to discretize each frame independently [55] 63 27], leading to rapidly increasing sequence lengths, or using 3D tokenizer that compresses videos
Duue this connection. the terms "video frame" and "visual observation" interchangeably.
3

---

6g
Detakenize
150
15
130
Decoder
Decodor
iVideoGPT
Encoder
Encoder
Takenize
Attion
Action embed.
Adtion embed
embed.
0.44
dg
0g
Qg
(a) Compressive tokenization
(b) Interactive prediction with Transformers
Figure 3: Architecture of iVideoGPT, simplified to show only a single context frame (To = 1). (a) Compressive tokenization utilizes conditional VQGAN that discretizes future frames conditioned on context frames to handle temporal redundancy, significantly reducing the number of video tokens. (b) An autoregressive transformer integrates multimodal signals-visus observations, actions, and rewards- innrdds--nttooo a sequence of tokens, enabling interactive agent experiences through next-token prediction. Actions and rewards are optional and not included in action-free video pre-training.
we propose tokenize videos with novel conditional VQGAN consisting of dual encoders and decoders {(F,, De), (E.. Dp)). As illustrated in Figure a. initial context frames 01: rich in contextual information, are independently tokenized and reconstructed through N tokens: _(::N) (ot),, o = De(=)) for l,..., 1,... To. In contrast, due the temporal redundancy between context and future frames, only essential dynamies information, such as the position and pose of moving objects, needs to be encoded. This is achieved using a conditional encoder and decoder, which require far smaller number of tokens (n < N):
spatiotemporally the expense of interactivity [101 104],
(1:a) (or ), à, Pp(2zoo1:3) for To + 1,...,
(1)
We implement this conditioning mechanism using cross cross-attention between multi- feature maps (see details in Appendix A.I. Overall, the proposed tokenizer is trained with the following objective:
To LvogAN(Or,Ed Dc() +
Liokenizer
Σ (=To+1
LvogAN(or (,(lorr,,, Dp(-lor.s))
where LVQUAN (o; E, D) is a combination of L1 reconstruction loss, a commitment loss 89 perceptual loss 44] and optionally adversarial loss 22
There are primarily two benefits of the proposed tokenization. First, it significantly reduces the sequence length of tokenized videos, which grows linearly with the number of frames but at much smaller rate n. In this work, we set = 16 x 16 and n = 4 x 4, resulting in an asymptotic reduction of 16x, facilitating faster rollouts for model-based planning and reinforcement learning. Second, by conditional encoding, transformers predicting subsequent tokens can maintain temporal consistency of the context much easier and focus on modeling essential dynamics information (99] We discuss the assumptions and limitations of our tokenization in Section
After tokenization, the video flattened into se- (N) quence of tokens: x = =x ..... 2/ (N) [S], 2 ..... 22 ..... [S], (1) 2 TTo+1 with a length of L = (N 1)T, +n+ To) 1. Special slot tokens [S] are inserted to delineate frame boundaries and facilitate the integration of extra dimensmnnsi modalities such as actions (see Section 3.33 for details). As Figure 3p, GPT-like autoregressive transformer is utilized for interactive video prediction through next-token generation frame-by-frame brame... In this work, we take the model size of GPT-2 (76) but adopt the LLaMA architecture [8]] in order to embrace the latest innovations for LLM architecture, applying pre-normalization using RMSNorm 106), SwiGLU activation function 883. and rotary positional embeddings (85).
Interactive prediction with Transformers.
(a)

---

3.2 Pre-Training
Large language models can gain extensive knowledge from Internet text in a self-supervised way via next-word prediction. Similarly, the action-free video pre-training paradigm for world models 88 99 62] involves video prediction as a pre-training objective, providing Internetsscale supervision with physical world knowledge absent in LLMs. We pre-train iVideoGPT on this generic objective, applying a cross-cntropy loss to predict subsequent video tokens:
Cpre-trin E' (N+11)776++ log (()))a))))
where L is the total sequence length and (N + 1)TH + 1 marks the first token index of the frames to be predicted. Notably, we do not train iVideoGPT to generate context frames, making its capacity focus on dynamies information, previously discussed.
Pre-training data. While there are numerous videos available the Internet, due to computational limitations, we specifically pre-train VideoGGPT for the robotic manipulation domain. We leverage mixture of 35 datasets from the Open X- X-Embodiment (OXE) dataset [70] and the Something- Something v2 (SSv2) dataset [25]. totaling 1.4 million trajectories (see Appendix A.2 for details). OXE is diverse collection of robot learning datasets from a variety of robot embodiments, scenes, and tasks. These datasets are highly heterogeneous but can be easily unified in the action-free video prediction task. To further enhance the diversity, we also include SSv2, dataset of human- object interaction videos, as previous work has demonstrated knowledge transfer from these human manipulation videos for learning a world model for robotic manipulation tasks 99 62
Flexibility of sequence modeling. A sequence of tokens provides flexible way specify tasks, inputs, and outputs [60] (66.. To preliminarily showcase this flexibility, we introduce a variant of iVideoGPT for goal-conditioned video prediction: 1:00++1:100::011111 or), where the model predicts a video sequence reaching a specified goal observation Or. This is simply achieved by rearranging the frame sequence 01.7 (oT,on, 01, 02,-..,0r7 oT-1) while keeping the architecture and training procedure consistent as above (see details in Appendix A.2). Qualitative results of goal-conditioned prediction are shown in Figure 4. with further exploration left for future work
3.3 Fine-Tuning
Action conditioning & reward prediction. Our architecture is also designed to flexibly incorporate additional modalities for learning interactive world models, as illustrated in Figure 33.. Actions are integrated by linear projection and adding to the slot token embeddings. For reward prediction, instead of learning independent reward predictors, we add a linear head to the last token's 's hidden state of each observation. This multi-task learning approach can enhance the model's focus on task-relevant information, thereby improving prediction accuracy for control tasks [57]. We use a mean-squared error loss for reward prediction addition to the cross-entropy loss in Eq.
Tokenizer adaptation. We choose update the full model, including the tokenizer, for downstream tasks, finding this strategy more effective than parameter-efficient fine-tuning methods (39]. This is likely due the limited diversity of our pre-treined data compared to Internet-scale images, which, while extensive, may also not adequately cover specific real-world applications like robotics. Minimal literature explores adapting VQGAN tokenizer to cemain--ppccffi data. As our tokenization is designed for decoupling dynamics information from context conditions, we hypothesize that while our model may encounter unseen objects like different robot types downstream tasks, the fundamental knowledge of physics-such as motions and interactions-leamed by the transformer from diverse scenes commonly shared. This hypothesis supported by our experiments transferring iVideoGPT from mixed pre-training data to the unseen BAIR dataset [21], where the pre-trained transformer can zero-shot generalize to predict natural motions, requiring only the tokenizer to be fine-tuned for unseen robot grippers (see Figure B). This property is particularly important for scaling GPT- like transformers to large sizes, enabling lightweight alignment across domains while keeping the transformer intact. We leave an in-depth analysis of tokenizer adaptation for future work.
3 Unless otherwise specified, action- and goal-free video prediction is used as the default pre-trainn trainiiinng objective obtain pre- pre-trained models for all experiments.
5

---

Open x-Eibbodment (action-tres)
1-0
1-2
15
1-2
1-4
1-8
1-9
1-12
(contest)
(cotect)
Predictdd
RoboNet
1-C
1-3
1-7
1-0
1.2
1-:5
1-9
11
(conteet)
(pontese)
Predicted
VP2
1=3
L-22
1
11
Ground Iuth
(contaxt)
(context)
Prodicted
Open X Emboddieent (geal-condiioned))
Ground tuuh
[tontwst)
(goel)
Prediced
Figure 4: Qualitative evaluation: video prediction results of iVideoGPT on Open X- X-Emboddment,, RoboNet, and Vp2, Zoom in for details. Extended examples can be found in Appendix B.1
4 Experiments
In this section, we evaluate iVideoGPT in three different control-relevant settings and compare its performance with prior state-of-the-art methods. We demonstrate that iVideoGPT versatile to provide competitive performance across range of tasks (Section 4.1 4.2 and 4.3) and conduct in-depth analysis to understand the tokenization and prediction ability, data efficiency, model scaling, and computational efficiency (Section 4.4). Experimental details can be found in Appendix A
4.1 Video Prediction
Setup.
consists of 43k training and 256 videos, where we predict 15 frames from single initial frame, a standard protocol of prior works. The RoboNet dataset contains 162k videos across 7 robotic arms. Following prior works, we use 256 videos for testing. predicting 10 frames from two frames. Notably, RoboNet overlaps with our pre-training data OXE, from which we have carefully filtered test videos. We compare against a variety of video prediction models, including variational [91 98 4) diffusion (93), masked [104] (27, and autoregressive models
The BAIR robot pushing dataset 21
16
101, across four metrics: FVD 88 PSNR 40) SSIM 997, and LPIPS [08].
As shown in Table 1 iVideoGPT provides competitive performance compared to state- of-the-art methods, MAGVIT 104 for BAIR and FitVid (4) for RoboNet, while achieving both interactivity and scalability in its architecture. Initially pre-trained action-free, our model flexibly allows for action-comee conditiion,,,,, which notably improves FVD for BAIR by almost 20%. Although primary experiments are at low resolution of 64 x 64, iVideoGPT can be casily extended to 256 x 256 for RoboNet. We highlight that Mask VIT, prior method leveraging per-frame tokenization, suffers from temporal inconsistency and flicker artifacts in VQGAN reconstructions. Our model, which employs compressive tokenization conditioned on consistent contextual information, improves this and significantly outperforms Mask VIT. For qualitative results, refer Figure 4
Results.
4.2 Visual Planning
Setup. Vp is control-centric benchmark [86] that evaluates video prediction models for visual prediipre control (MPC) [24 20 across four Robosuite [117] and seven RoboDesk tasks [47].

---

Table 1: Video prediction results on the BAIR robot pushing and RoboNet datasets. We report the mean and standard deviation for each metric calculated over three runs. marks that the value is not reported in the original papers. LPIPS and SSIM scores are scaled by 100 for convenient display.
BAIR
FVD4
PSNRT SSIMI LPIPS
RoboNet
16
FVDJ
PSNRT
SSIM LPIPS.
action-free 64x64 resolution
action-conditionel & 64x 64 resolution
VidenGPT 101
103.3
MaskVIT 27
133.5
80.5
4.2
MaskViT
27
93.7
SVG [9]
123.2
23.9
87.8
6.0
FicVid 4
93.6
GHVAE 98
95.2
24.7
89.1
3.6
MCVD
93
89.5
16.9
78.0
FitVid
62.5
28.2
89.3
2.4
MAGVIT 104
62.0
19.3
78.7
12.3
iVideoGPT (ours)
63.2.
27.8
10.6
4.9,000
VidenGGPT (ours)
10:0
20.4
82.3 008 9.5 nm
action- conditionndi
256x 256 resolution
action-conditioned & 64x64 resolution
MaskVIT 27
211.7
20.4
67.1
17.0
VaskVIT 27
70.5
iVideoGPT (ours)
197.9
23.8
80.
14.7
IVideoGPT (ouns)
60.8
24.5
90.2.aas n.0
MideoGPT Jouns]
Success Rale
0.6
02
Rabosuita push
Flat block
Opan drawor
Open slido
Blue button
Grean button
Red button
Upright block
Aggregata
Figure 5: Visual MPC results on the Vp² benchmark. We report the mean and min/max performance of iViieoGPT over 3 control runs. On the right, we show the mean scores averaged across all tasks except flat block due to low simulator performance, normalized by the performance of the simulator.
Each environment's training dataset includes noisy scripted interaction trajectories. Following the protocol from the original benchmark paper, we trained iVideoGPT on 5k trajectories for Robosuite and 35k for RoboDesk, comparing our models with established baselines.
presents the success rates of iVideoGPT compared to baseline models. While Tian presents al. (86] observed that excellent perceptual metrics do not always correlate with effective control performance, iVideoGPT outperforms all baselines in two RoboDesk tasks with a large margin and achieves comparable average performance to the strongest model, SVG' (91]. In Appendix C.3 we analyze i /VideoGPT's suboptimal performance on the open slide task, which is attributed to both limitations of discretization in our model and imperfect built-in reward design of the benchmark.
Results.
4.3 Visual Model-based Reinforcement Learning
backoropagate
Setup. We conduct experiments on six robotic ma- nipulation tasks of varying difficulty from Meta- World [105]. Leveraging iVideoGPT as interactive world models, we have developed a model-based RL method adapted from MBPO [42], which augments the replay buffer with synthetic rollouts train stan- dard actor-critic RL algorithm (see Appendix[a AA.5 for the pseudo-code). Our implementation builds upon DrQ-v2 [03]. a state-of-the-arr visual model-free RL method. We also compare against state-of-the-arr madel-based RL algorithm, DreamerV3 [32], with and without world model pre-training (8I)
Dreamer
π
π
Env
D
Latent Imagination
MBPO (ours)
Env
iVideoGPT
World model rollouts
Standard model-free RL
Figure 6: Powerful iVideoGPTs enable sim- ple yet performant MBRL algorithm, decou- pling model rollouts and policy learning.
7

---

Stacowes Patt (86)
%
MetaWoord Aggnegaled
8 R
Succids Rain (1%)
60-
0
Endrrommnt Steps (wrift
Enveceeent Stepe (410f)
Envinrment Strpp (4110t)
40
Door Lock
Handle Pull Bile
Coffee Push
Dreamert (er77)
Susssee Daaa
14 &
0.0
02
D.
Normalized Environment Steps
(strd")
Stapa
Figure 7: Visual model-based RL on Meta-world. (Left) Aggregated results report interquartile mean and 95% confidence interval (CT) [2] across a total of 30 runs over six tasks. (Right) Individual results for each task. report mean and 95% CI across five runs, measuring success rates over 20 evaluation episodes. PT denotes pre-training.
Results. Figure 7 shows that our model-based algorithm not only remarkably improves the sample efficiency over its model-free counterpart but also matches or exceeds the performance of Dreamer V3. To our knowledge, this reports the first successful application of MBPO to visual continuous control tasks. These results highlight the opportunity, with powerful world models, eliminate the need for latent imagination-22 common strategy used in advanced MBRL systems to train policies on rollouts of latent states within world models 229 [0] (see comparison in Figure 6). Our development of
performant MBRL algorithms decouples model and policy learning, where VVideoGPT simply serves as drop-in in replacement of the environment. This can substantially simplify the design space, thereby greatly enhancing the practicality and effectiveness of MBRL algorithms in real-world applications.
Comparison to recurrent world models. We argue that recurrent world models lack the capacity for large-scale pre-training on real-world data-a crucial capability for modern foundation models. To validate this, pre-train DreamerV3 XL (200M parameters, comparable to iVidcoGPT) on the same dataset. As shown in Figure ii in the Appendix, DreamerV3 fails to capture natural robot dynamics, yielding low-quality, blurred predictions. Further evaluation the Meta- World benchmark in Figure reveals that DreamerV3 cannot benefit from such ineffective pre-training.
4.4 Model Analysis
Zero-shot prediction. We first analyze the zero-shot video prediction ability of large-scale pre- trained IVideoGPT on the unseen BAIR dataset. Interestingly, we observe in the second row of Figure [ that iVideoGPT, without fine-tuning, predicts a natural movement of a robot gripper-albein different one from our pre-training dataset. This indicates that while, due to insufficient diversity of pre-training data, our model has a limited ability of zero-shot generalization to completely unseen robots, effectively separates scene context from motion dynamics. In contrast, with an adapted tokenizer, the transformer that is not fine-tuned itself successfully transfers the pre-trained knowledge and predicts movements for the new robot type in the third row, providing a similar perceptual quality as the fully fine-tuned transformer in the fourth row. Quantitative results can be found in Figure 9a
Few-shot adaptation. Large-scale pre-trained models have proven effective, especially in data- scarce scenarios. Figure shows iVideoGPT's performance when fine-tuned with various sizes of action-free BAIR trajectories. We observe that pre-training offers minimal benefits when full downstream data is available, yet the advantages become significant under data scarcity (with 100 or 1,000 trajectories). We also adapt iVideoGPT using 1,000 eetion-conditioned BAIR trajectories, achieving an FVD of 82.3. The fast adaptation ability with handful of data is particularly crucial in model-based RL, As shown in Figure world models trained from scratch may generate inaccurate predictions, thereby degenerating the sample efficiency that is vital for model-based agents.
Model scaling. All previous experiments are conducted using an iVideoGPT with 12 transformer layers and disenimmnsi hidden states (138M parameters). To initially investigate the scaling
8

---

Breel
ruth
Prodided Jobo Ene-tuning!
yokweree (ree.red)
Prddided
2u fune)
Figure 8: Zero-shot prediction by pre-trained transformer in iVdeeoGPT. Without fine-tuning, the transformer predicts natural movements of different robot gripper using the pre-trained tokenizer (second row) but accurately predicts for the correct gripper type with an adapted tokenizer (third row).
From Scratch
WideoCPPT-139M MidsoGPT-43MM
Reconstruction LPIPS
0.20
Training GPU Memory
No Fine-luning
Validation Loss
300
Tokanizer Fine sunad
0.15
EVD
Full Fine-tuned
2.0
0.10
1.8-
Compresteer bakan (Durs)
0.05
7
LPRSS 12 Time
100
0.00
0 1,000
Full
0
10
20
30
40
50
60
70
Size Trajectories)
Pre-training Steps (x110))
Generation Time (s)
(a) Few-shot adaptation
(b) Model scaling
(c) Tokenization efficiency
Figure 9: Model analysis. (a) Video prediction results with various fine-tuning strategies and data sizes on BAIR. (b) Validation losses for the 138M and 436M transformer models on the pre-training dataset. (c) Computational efficiency and reconstruction quality of different tokenizers.
behavior of our model, we trained a larger iVideoGPT with 24 layers and 1024-dimensional hidden states (436M parameters). Figure 99 illustrates the validation loss curves on the pre-trained dataset. It shows that (1) the validation loss (perplexity) continues to decrease regardless of model size, and (2) increasing the model size accelerates the loss decrease. These results align with our expectation that larger model sizes and increased computation [88 can build more powerful iVideoGPTss.
Tokenization efficiency. We evaluate the effectiveness of our compressive tokenization by com- paring it against standard VQGAN tokenizers that independently convert each frame into 16 x 16 and 4 4 tokens. We train three tokenizers from scratch on RoboNet for the same number of steps. As Figure 9c the tokenizer with x 4 tokens suffers from low reconstruction quality due to its insufficient capacity. Our proposed tokenization method slightly compromises reconstruction quality compared to the standard 16 x 16 tokenizer but can provide more consistent contextual information, which is beneficial for video prediction tasks. More importantly, significantly enhances computa- tional efficiency with a significantly fewer amount of tokens, which greatly saves time and memory, allowing us scale the model size with fewer costs (see quantitative results in Appendix B.5).
Context-dy dynamiccs decoupling. Our tokenizer is designed with a bottleneck of much fewer tokens, focusing only on capturing necessary dynamics information for future frames while sharing contextual information with initial frames to reconstruct raw pixels. To explicitly visualize this decoupling of context and dynamics information, drop cross-attention blocks to context frames in the decoder when reconstructing future frames. The results Figure 10 show that the decoder can still reproduce the movement trajectories accurately but with minimal contextual information. This visualization supports the explanation of our model's generalization capability shown in Figure g
Goal-conditioned prediction. In Figure we also showcase video prediction generated by goal- conditioned iVideoGPT, pre-trained on massive human and robotic videos (Section 3.2). Unlike action-free prediction, which often results in trajectories diverging from the ground truth, the goal- conditioned model produces more accurate paths to reach specified goals. We believe this highlights the potential of leveraging the flexibility of unified sequence modeling paradigm.

---

1+0g 1+2
1-4
Led
1-5
1 10
1+ 12
54
1-0
L-2
1-4
1-5
1-0
L+ 10
1+
Reamt stuccion
Reomenuttin
In decsder)
Figure 10: Context-dynamien decoupling in our compressive tokenization. By removing cross- attention from future frames to context frames, the decoder can still reconstruct a trajectory that moves in the same way the original, but the visual context is almost entirely missing.
5 Related Work
World models for visual control. Learning general world models in visual domains remains a significant challenge in model-based reinforeement learning. A straightforward method involves learning condittondd video prediction models [69] (45.. Advanced model-based RL algorithms
34 33 utilize latent imagination for more efficient and accurate rollouts but compli- cate themsel ves by tightly coupling model and policy learning. We show that this complexity can be reduced with powerful world models that have accumulated generalizable knowledge beyond specific tasks. Recent efforts to facilitate this include leveraging scalable architectures like transformers [63
29
32
80
and pre-training from large-scale data [99 622. Of particular relevance to our work are UniSim [02
and Genie [12). which have developed extensively trained world models with diffusion and masked models, respectively, though neither is publicly available. Our work distinguishes itself by utilizing generic autoregressive transformer framework, advancing the flexibility of scalable world models.
Video generation and prediction. Recent developments in Internet-scale video generation models now enable the synthesis of realistic videos conditioned on class labels, text descriptions, and initial the last one also known as the video prediction problem. Various models have been developed, including deterministic RNNs [84] (66.. variational autoencoders
[T8] 30 diffusion
38 MII masked [104 277, and autoregressive models [101 [50 55). However, most recent works do
not treat video prediction as dynamics modeling problem and perform spatiotemporal compression
[01 8), thus providing limited interactivity to serve world models. We achieve both compressive tokenization and interactivity by context-awand representation, employing cross-attention mechanisms with minimal inductive bias. This method diverges from previous techniques that rely motion
vectors 43 or optical flows [22 and offers a more generic form of video tokenization.
6 Discussion
We introduced Interactive VideoGPT (iVideoGPT), a generic and efficient world model architecture that leverages a scalable autoregressive transformer to integrate multimodal signals into a sequence of tokens, providing interactive agent experience via next-token prediction. iVideoGPT has been pre-trained on millions human and robotic manipulation trajectories and adapted to a wide range of downstream tasks. As powerful foundation of world models, it enables accurate and generalizable video prediction as well as simplified yet performant model-based planning or reinforcement learning.
Limitations and future work. While iVideoGPT marks significant progress, there is substantial room for improvement. We found limited diversity in publicly available robotic data, including the large-scale Open X-Fmbodiment dataset, and initiated efforts to transfer knowledge from human videos (25]. We believe iVideoGPT should be pre-trained on more extensive data (26) to bridge knowledge between humans and robots. This also requires iVideoGPT to incorporate more modal- ities, such as multi-view observations, proprioceptive robot states, and actions, within the unified formulation beyond action-free video prediction. Specifically, to process high-dimensional visual observations, our compressive tokenization assumes that initial frames provide sufficient contexts for future frames, which works for low-level control tasks as model-basee agents often foresee tens of steps, but may falter scenarios with long videos and significant camera motion. This issue can be mitigated by keyframe extraction [51 but leaves an important future avenue of exploration. Finally, extending to more complex real-robot tasks is essential, as the benefits of model scaling to even larger sizes remain unobserved in this work within visually simple simulation for downstream control tasks.
10

---

Acknowledgements
We would like thank many colleagues, in particular Yuchen Zhang, Lanxiang Xing, and Haixu Wu, for their valuable discussion. This work was supported by the National Natural Science Foundation of China (U2342217 and 62021002), the BNRist Project, the Huawei Innovation Fund, and the National Engineering Research Center for Big Data Software.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. In NeurlPS, 2021.
[3] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Camphell, and Sergey Levine. Stochastic variational video prediction. In ICLR, 2018.
[4] Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint 13X1v2210..3315, 2021.
[5] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as versatile representation for robotics. In CVPR, 2023.
[6] Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Hydra: Hybrid robot actions for imitation learning. In CoRL, 2023.
[7] Andreas Blattmann, Tim Dockhorn Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:211..55127 2023.
[8] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[9] 7alán Borsos, Raphaël Marinier, Damien Vincent, Fugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier. Marco Tagliasacchi, al. AudioLM: language modeling approach audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[10) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-wood world control at scale. In RSS, 2023.
[11] Tim Brooks, Bill Peebles, Connor Holmes, Will DePuc, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[12] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In ICMI, 2024.
[13] Lili Chen, Shikhar Bahl, and Deepak Pathak. Playfusion: Skill acquisition via diffusion from language- annotated play. In CoRL, 2023.
[14] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In RSS, 2023.
[15] Zichen Jeff Cui. Yibin Wang, Nur Muhammad Mahi Shafiullah. and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. In ICLR, 2023.
[16] Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher. Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. RoboNet: Large-scale multi robot learning. In CoRL, 2019.
11

---

[17] Shivin Dass, Jullian Yapeter, Jesse Zhang, Jiahui Zhang, Karl Pertsch, Stefanos Nikolaidis, and Joseph J. Lim. CLVR jaco play dataset, 2023.
[18]
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In JCMI,, 2018.
[19] Yilun Du, Mengjiao Yang, Pete Florence. Fei Xia. Ayzaan Wahid, Brian Ichter. Pierre Sermanet, Tianhe
Yu, Pieter Abbecl, Joshua B Tenenbaum, al. Video language planning. In ICLR, 2024.
[20] Frederik Ehert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint 1r200:1812.00568 2018.
[21] Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. CoRL, 2017.
[22] Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[23] Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Finetuning offline world models in the real world. In CoRL, 2023.
[24] Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In ICRA, 2017.
[25] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilo,, Moritz Mueller- Auueiter-Freitag, et al. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
[26] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[27] Agrim Gupta. Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martin-Martin and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. In ICLR, 2023.
[28] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NewrIPS, 2018.
[29] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
[30] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, 2019.
[31] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR, 2021.
[32] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillierap. Mastering diverse domains through world models. arXiv preprint veXiv:201.004144 2023.
[33] Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. ICLR, 2024.
[34] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. ICMI,, 2022.
[35] Minho Hco, Youngwoon Lee, Doohyun Lee, and Joseph Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In RSS, 2023.
[36] Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In AAAI, 2018.
[37] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0233
2022.
[38] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Flect. Video diffuusion models. In NeurIPS, 2022.
[39] Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
12

---

[40] Quan Huyah-Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics letters, 44(13):800-80) 2008.
[41] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot shot task generalization with robotic imitation learning. In CoRL, 2021.
[42] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Modcl-based policy optimization. In NewrIPS, 2019.
[43] Yang Jin, Zhicheng Sun, Kun Xu, Kun Xu, Liwei Chen, Hao Jiang. Quzhe Huang. Chengru Song, Yuliang Liu, Di Zhang, Yang Song, Kun Gai, and Yadong Mu. Video-lavit: Unified video-language pre-training with decoupled isstallmoioona tokenization. In ICML, 2024.
[44] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[45] Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. In ICLR, 2020.
[46] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. In CoRL, 2018.
[47] Harini Kannan. Danijar Hafner, Chelsea Finn, and Dumitru Erhan. Robodesk: A multi-task reinforcement learning benchmark. https://githn:
resproocooeresssrcccrrooodess 2021.
[48] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.088 000301,...61..000 2020.
[49] Minchan Kim, Junhyek Han, Jachyung Kim, and Beomjoon Kim. Pre-and post-contact policy decomposi- tion for non- prepereeeeslle manipulation with zero-shot sim-to-real transfer. In IROS, 2023.
[50] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Homunn, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. In ICML, 2024.
[51] Didier Le Gall. Mpeg: A video compression standard for multimedia applications. Communications of the ACM, 34(4):46-58, 1991.
[52] Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: Context-aware controllable video synthesis. In NeurlPS, 2021.
[53] Yann LeCun. A path towards autonomous machine intelligence. preprint posted on openreview, 2022.
[54] Michelle A Lee, Yuke Zhu Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg. and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In ICRA, 2019.
[55] Hao Liu, Wilson Yan, Matei Zaharia, and Picter Abbecl. World model on million-length vidco and language with blockwise ringattention. arXiv preprint arXiv:2402.08226 2024.
[56] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
[57] Haoyu Ma, Jialong Wa, Ningya Fong, Chenjun Xiao, Dong Li, Jianye Hao, Jianmin Wang, and Mingsheng Long. Harmonydream: Task harmonization inside world models. In ICML, 2024.
[58] Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. In IROS, 2019.
[59] Tatsuya Matsushima, Hiroki Furuta, Yusuke Iwasawa, and Yutaka Matsuo. Weblab xArm Dataset, 2023.
[60] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning question answering. arXiv preprint 18870:1000.08730, 2018.
13

---

[61] Oier Mees, Jessica Borja-Diaz. and Wolfram Burgard. Grounding language with visual affordances over unstructured data. In ICRA, 2023.
[62] Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. In RSS, 2023.
[63] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. In ICLR, 2023.
[64] Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee. Unsupervised learning object structure and dynamics from videos. In NeurlPS, 2019.
[65] Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for imitation learning. In CoRL, 2022.
[66] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CVPR, 2015.
[67] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In RSS, 2024.
[68] Jihoon Oh, Naoaki Kanazawa, and Kento Kawaharazuka. X X-Embodiment U-Tokyo PR2 Datasets, 2023.
[69] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Aetion-conditional video prediction using deep networks atari games. NeurlPS, 2015.
[70] Abhishek Padalkar, Acom Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In ICRA, 2024.
[71] Abhishek Padalkar, Gabriel Quere, Antonin Raflin, João Silvério, and Freek Stulp. A guided reinforcement learning approach using shared control templates for learning manipulation skills in the real world. Research square preprint rs-3289569/11, 2023.
[72] Abhishek Padalkar, Gabriel Quere, Franz Steinmetz, Antonin Raftin, Matthias Nieuwenhuisen, João Silvério, and Freek Stulp. Guiding reinforcement learning with shared control templates. In ICRA, 2023.
[73] Suraj Patil, William Berman, Robin Rombach, and Patrick von Platen. amused: An open muse reproduc- tion. arXiv preprint arXiv:2401.01880 2024.
[74] Gabriel Quere, Annette Hagengruber, Maged Iskandar, Samuel Bustamante, Daniel I Leidner, Freek Stulp, and Joern Vogel. Shared Control Templates for Assistive Robotics. In ICRA, 2020.
[75] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understand- ing by generative pre- pre-training. 2018
[76] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[77] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[78] Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task agnostic offline reinforcement learning. In CoRL, 2022.
[79] Amrita Sawhney, Steven Lee, Kevin Zhang, Manuela Veloso, and Oliver Kroemer. Playing with food: Learning food item representations through interactive exploration. In Experimental Robotics: The 17th International Symposium, pages 309-322. Springer, 2021.
[80] Julian Schritwwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604-609, 2020.
[81] Younggyo Sco, Kimin Lec, Stephen L James, and Pieter Abbeel. Reinforeement learning with action-free pre-training from videos. In ICML, 2022.
[82] Rutav Shah, Roberto Martin,Martn,,..m and Yuke Zhu. MUTEX: Learning unified policies from multimodal task specifications. In CoRL, 2023.
14

---

[83] Noam Shazeer. Glu variants improve transformer. arXiv preprini arXi::2002.05202, 2020.
[84] Xingjian Shi, Zhourong Chen, Hao Wang, Dan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convo- lutional Istm network: A machine learning approach for precipitation nowcasting. In NeurlPS, 2015.
[85] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 567063, 2024.
[86] Stephen Tian. Chelsea Finn, and Jiajun Wu. A control-centric benchmark for video prediction. ICLR, 2023.
[87] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wXXv:2307.00288, 2023.
[88] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski. and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 1rXiv:1812.01117, 2018.
[89] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurlPS, 2017.
[90] Ashish Vaswani, Noam Shazeer, Niki Parnar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
[91] Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan. Quoe Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In NeurlPS, 2019.
[92] Jörn Vogel, Annette Hagengruber, Maged Iskandar, Gabriel Quere, Ulrike Leipscher, Samuel Bustamante, Alexander Dietrich, Hannes Hoeppner, Daniel Leidner, and Alin Aibu-Schfffer. Edan an emg-controlled daily assistant to help people with physical disabilities. In IROS, 2020.
[93] Vikram Voleti, Alexia JolicoeurMaartineuu, Martineau, and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. In NewrlPS, 2022.
[94] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruchh Quan Vuong, Andre He, Vivek Myers. Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In CoRL, 2023.
[95] Yixuan Wang, Zhuoran Li, Mingtong Zhang, Katherine Driggs-Campbent Jiajun Wu, Li Fei-Fei, and Yunzhu Li. D3fields: Dynamic 3d descriptor fields for zero-shot generalizable robotic manipulation. arXiv preprint arXiv:2309.1611 2023.
[96] Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, S Yu Philip, and Mingsheng Long. Predrn: A recurrent neural network for spatiotemporal predictive learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(2):2208-22 2205,, 2022.
[97] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 1 13())6000612, 2004.
[98] Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsca Finn. Greedy hicrarchical variational autoencoders for large-scale vidco prediction. In CVPR, 2021.
[99] Jialong Wu, Haoyu Ma. Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with in-the-wild videos for reinforcement learning. In NeurlPS, 2023.
[100] Ge Yan, Kris Wu, and Xiaolong Wang. UCSD Kitchens Dataset. August 2023.
[101] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint 1:Xi::2104.10157, 2021.
[102] Mengjiao Yang, Yilun Du Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Picter Abbeel. Learning interactive real-world simulators. In /CLR, 2024.
[103] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforeement learning. In ICLR, 2022.
[104] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang. Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Hsuan Yang, Yuan Hao, Irfan Essa, et al. MAGVIT: Masked generative video transformer. In CVPR, 2023.
15

---

[105] Tianhe Yu, Deirdre Quillen, Zhanpeng He. Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Mela-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2020.
[106] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In NeurlPS, 2019.
[107] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[108] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features a perceptual metric. CVPR, 2018.
[109] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free controllable text-to-video generation. In ICLR, 2024.
[110] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[111] Gaoyue Zhou, Victoria Dean, Mohan Kumar Srirama, Aravind Rajeswaran, Jyothish Pari, Kyle Hatch, Aryan Jain, Tianhe Yu. Pieter Abbeel, Lerrel Pinto, Chelsea Finn, and Abhinav Gupta. Train offline, test online: A real robot learning benchmark. In ICRA, 2023.
[112] Yifan Zhou, Shubham Sonawani, Mariano Phielipp. Heni Ben Amor, and Simon Stepputtis. Learning modular language-condiionee robot policies through attention. Autonomous Robots, pages 1-21, 2023.
[113] Yifan Zhou, Shubham Sonawani, Mariano Phielipp. Simon Stepputtis. and Heni Amor. Modularity through attention: Efficient training and transfer of language-comditioned policies for robot manipulation. In CoRL, 2023.
[114] Xinghao Zhu, Ran Tian, Chenfeng Xu, Mingyu Ding, Wei Zhan, and Masayoshi Tomizuka. Fanuc manipulation: A dataset for staming-basee manipulation with fanuc mate 200id robot. 2023.
[115] Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. In CoRL, 2022.
[116] Yifeng Zhu, Peter Stone, and Yuke Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters. 7(2):4126-4133, 2022.
[117] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roherto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning, 2022.
16

---

A Implementation and Experimental Details
The main hyperparameters of our experiment are detailed Tables 2 and 5 In this section, we provide comprehensive explanation of all experimental details.
A.1 Architecture
Table 2: Hyperparameters of VVdeoGGPT architectures.
VQGAN
Low-resolution
High-resolution
Parameters
114M
310M
Resolution
64 x 64
256 x 256
Down blocks
3
Down layers per block
2
2
Down channels
[128, 256, 512]
[128, 256, 256, 512, 768]
Mid block attention
False
False
Up blocks
3
Up layers per block
3
3
Up channels
[512, 256, 128]
[768, 512, 256, 256, 128]
Embedding dim
64
64
Codebook size
8192
8192
Norm
GroupNorm
GroupNorm
Norm group
32
32
Activation
SiLU
SiLU 32
Max cross-att. resolution
16
Transformer
Small
Medium
Parameters
138M
436M
Layers
12
24
Heads
12
16
Hidden dim
768
1024
Feedforward dim
3072
4096
Dropout
0.1
0.1
Activation
SiLU
SiLU
conditional VQGAN for compressive tokenization. This comprises two encoder-decoder pairs: (Fer D.) for context frames (referred to as the context encoder-decoder) and (EEpD,,) for future frames (referred to the prediction encoder-decoder) Both pairs share the same architecture (detailed in Table 2 but the prediction encoder-decoder has tighter bottleneck, focusing solely on encoding dynamic information. Specifically, uses 4 x 4 convolution to downsample 16 x 16 embeddings into 4 x 4 before looking up the codebook. Consequently, the prediction encoder-decoder needs be conditioned on the features of the context -ecoder-decoder to incorporate rich contextual information. This conditioning is implemented via multi-scale rross-attention mechanism, similar ContextWM 99).
Tokenizier. As illustrated in Figure
The intuition behind the multi-scale cross-attention across feature maps as follows: the context encoder extracts contextual features varying levels of abstraction, while the prediction encoder uses cross-attention to adaptively filter out contextual information and distill dynamics information. During retrieve contextual information
decoding, the prediction decoder blocks employ cross-attention corresponding levels, facilitating the gradual reconstruction of the scene. This framework enhances the model's ability understand and manipulate complex scenes by focusing on dynamic changes, rather than being overwhelmed by irrelevant visual details.
Specifically, at the end of each encoder block, let F!. Rexhxw be the feature map of context frame, and F). [xexhxw be the feature map of future frame. Before being processed by the next
17

---

Table Hyperparameters of iVideoGPT training and evaluation.
Low-resolution (64 64)
High-resolution (250 x 256)
VQGAN
Pre-train
BAIR
RoboNet
Vp2
Pre-train
RoboNet
GPU days
17
2
16
9 1.5 x 10
Training steps
1 106
2 10a
6 10a
2 x 10
2.5 10
Disc. start
1 x 10
Batch size
64
64
64
32
Sequence length
16
16
12
12
16
12
Context frames
2
Sampled future frames
6
7
6
6
6
Learning rate
x
10
1
x
10 -4
1 x 10 44
x x 10
10
X 10 -4
LR Schedule
Constant
Weight decay
0.0
Grad clip
1.0
Warmup steps
500
Loss balancing
Equal weights
Optimizer
AdamW
Mixed precision
bfl6
Transformer
Pre-train
BAIR
RoboNet
Vp2
Pre-train
RoboNet
GPU days
19
1.5
10
9
26
Training steps
7 x 10
10
x 10
x 10
3.5 x 10
5 x 105
Batch size
64
64
64
64
16
32
Sequence length
16
12
12
16
12
Context frames
2
1
2
2
Learning rate
X 10
LR Schedule
Cosine
Weight decay
0.01
Grad clip
1.0
Warmup steps
5000
Loss balancing
N/A or equal weights
Optimizer
AdamW W bf16
Mixed precision
Sampling temperature
1.0
Sampling top-k
100
block, Fb is augmented with F follows:
F 1++ (ncBlockk++
(4)
(Augment(F)). F1+1 "Annott F!))
This is achieved by performing cross-attention between the 2hw positions of the feature maps:
Flatten: Q Norm (Reshape (F)) + PosEmb e kkwxe
= Norm (Reshape (Fr)) + KV kuuxx
(5)
(QW9,KW* Resicual--Connecion:: Augment(PP F.) SiLU (F) + Reshape (R)) Rexhxw
Cross-Attention: R = Attention (QW4, KWK VWV) E phuxe
To reduce memory usage, apply the cross- cross-attention mechanism only when the feature map size below a certain threshold (16 x 16 for a 64 x 64 original resolution and 32 x 32 for a 256 x 256 resolution). This mechanism is symmetrically performed across the context and prediction decoder.
Since attention mechanisms can flexibly handle varied input lengths, the conditioning mechanism can be easily extended to accommodate different numbers of context frames. Each context frame is
18

---

independently processed by the context encoder and decoder, and their feature maps are concatenated serve as imputs for cross-attention in the prediction encoder and decoder.
Our VQGAN for 256 x 256 resolution is initialized from the pretrained model from aMUSEC We do not use discriminators for 64 x 64 resolution, effectively converting the VQGAN into vanilla VQVAE with an additional perceptual loss.
Transformer.
We flatten video into sequence of tokens:
x x=
(N)
[S1]] 22 Z2
(1) (N)
(1) [S2], TTo
(n) 1
(6)
where we use two types of slot tokens [S1] and [S2] before the start of context frames and future frames, respectively. Context and future frames do not share token IDs, resulting in transformer vocabulary of 16,386 tokens: the first 8,192 for context frames, the next 8,192 for future frames, and the last two for slot tokens. We adopt the autoregressive transformer architecture from LLaMA (87), but instantiate it to smaller models matching the size of GPT-2. We considered two model sizes, listed in Figure 2 Most of our experiments utilize a 138M parameter transformer, while preliminary scaling analysis conducted using a 436M model.
A.2 Action-free Video Pre-training
Data mixture. We pre-train iVideoGPT using 35 datasets from the Open X-Embodiment Dataset (OXE) 770 and mameethinn-Somethinn---- (SSv2) (25). To construct our training dataset from OXE, we implement filtering and weighting process similar to Octo [67]. Initially, we exclude datasets lacking image streams and those derived from mobile robots. Subsequently, datasets exhibiting excessive repetition or possessing low image resolutions are eliminated. The remaining datasets were categorized either "large" or "small," " and each was assigned weight based its size and diversity. We select 1% of samples from each subset as validation data and use the rest for training. For SSv2, we manually select classes with clear motion trends from the original 174 video classes as our pre-training data with weighting of 15% We use the official splits of SSv2 for training and validation. For a comprehensive breakdown of the mixture, refer Table
Training details. During training, we sample sequences of frames by first randomly selecting training video and then uniformly sampling a segment of specified length and step size, i.e., neighboring frames in the segment are spaced a certain number of steps apart in the original video. We observe that datasets are collected at different frequencies. To maintain consistency, we adjust sampling with varied step sizes, aligning each with its respective dataset frequency, as listed in Table For tokenizer training, the initial frames of the segment are used as context frames, and from the remaining frames, we randomly sample a subset as future frames to reduce memory requirements and increase batch size. For transformer training, we use the full segment of frames. The number of frames in minibatches for each dataset is detailed in Table 3 We use a mixture of OXE and SSv2 for training the tokenizer to ensure visual diversity, while only OXE is used for training the transformer. For data augmentation, we apply random resized crop and color jitter, ensuring consistency across the sequence. During both tokenizer and transformer training, we blend different losses with equal weights. Unless specified otherwise, we follow the same implementation details when fine-tuning iVideoGPT on downstream tasks.
Goal-conditioned prediction. To train a goal-conditioned variant of iVideoGPT on the same dataset, we first fine-tune the previously obtained tokenizer using two randomly sampled frames as context for 550k training steps. Then, we train transformer from scratch with the rearranged frame segment 01:T = (OT, 01, . or-1) ) for I million steps. The architecture and training procedures remain consistent with the above setup.
License. The Open X-Embodiment dataset follows the Apache license. RoboNet is licensed under Creative Commons Attribution 4.0, while BAIR follows Creative Commons BY 4.0. The V2 dataset is subject to the Data License Agreement
http::://gtabbbooo///ggggggggggs under openrail++ license
19

---

Table iVieeoGPT pre-training data mixture from the Open X-Embodiment 70 and Something- datasets.
Something V2
25
Dataset
Num of trajectories
Step size
Sampling weight
Fractal (RT-1) 10
87,212
12.8%
Bridge 94
28,935
2
12.8%
BC-Z
43,264
3
12.8%
RoboNet [16
82,649
12.8%
Kuka [46]
580,392
3
8.5%
Language Table 566
442,226
3
4.2%
Stanford Mask ViT 27
9,200
4.2%
UIUC D3Field
768
2.2%
Taco Play 78 61
3,603
5
0.5%
Jaco Play 17
1,085
3
0.5%
Roboturk 158
1,995
3
0.5%
Viola 115
150
0.5%
Toto 11
1,003
10
0.5%
Columbia Cairlab Pusht Real [4
136
3
0.5%
Stanford Kuka Multimodal Dataset 54
3,000
7
0.5%
Stanford Hydra Dataset (6]
570
3
0.5%
Austin Buds Dataset [116)
50
7
0.5%
NYU Franka Play Dataset 15
456
0.5%
Furniture Bench Dataset [35
5,100
3
0.5%
UCSD Kitchen Dataset 100
150
0.5%
UCSD Pick and Place Dataset
23
1,355
0.5%
Austin Sailor Dataset (65
240
7
0.5%
UTokyo PR2 Tabletop Manipulation
68
240
3
0.5%
UTokyo Xarm Pick and Place [59
102
3
0.5%
UTokyo Xarm Bimanual
(59
70
3
0.5%
KAIST Nonprehensile
49
201
3
0.5%
DLR SARA Pour (72)
100
3
0.5%
DLR SARA Grid 71
107
3
0.5%
DLR EDAN Shared Control
92
104
3
0.5%
ASU Table Top 113 12
110
0.5% 4
0.5%
UTAustin Mutex 82
1,500
Berkeley Fanuc Manipulation
114
415
3
0.5%
CMU Playing with Food
174
3
0.5%
CMU Play Fusion 13
576
2
0.5%
CMU Stretch
5
62
135
3
0.5%
Something-Sometting
120,581
15.0%
Total
1,417,954
100.0%
20

---

Algorithm Model-Based Policy Optimization (MBPO), adapted from 42
1: Initialize actor-critic Td, Up, world model pe
2: Initialize real replay buffer Deet with random policy
3: Initially train model Pe on Dreal
4: Initialize imagined replay buffer Pimag with random rollouts using pa
5: for N steps do
6: // Training
7: if model update step then
Update world model po on mini-batch from Dreal 9: end if
8:
10:
Update actor-critic Tæ, Ui with model-free objectives on a mini-batch from Dimag U Dreat
11:
// Data collection
12:
if model rollout step then
13:
Sample mini-batch of or uniformly from Deal Perform s-st model rollout starting from or using policy Toi add to Pinag
14:
15: end if
16: Take action environment according to Tei add to Dreal
17:
end for
A.3 Video Prediction
Evaluation metrics. We evaluate model across four different metrics Structural Similarity Index Mcasure (SSIM) 197), Peak Signal-to-noise Ratio (PSNR) [40], Learned Perceptual Image Patch Similarity (LPIPS) [108] and Fréchet Video Distance (FVD) [88]. Following prior works 91 4 [04],, we account for the stochastic nature of video prediction by sampling 100 future trajectories per test video and selecting the best one for the final PSNR, SSIM, and LPIPS scores. For FVD, we use all 100 samples.
A.4 Visual Planning
We use the official repository to evaluate our model on the vp2 benchmark. The reported baseline results are provided by the authors of the benchmark. For the Robosuite tasks, cost below 0.05 is considered a success.
A.5 Visual Model-based RL
Environments. Meta-worke [105]. following MIT License, is benchmark of 50 robotic manip- ulation tasks. We select six tasks for our experiments: Button Press Topdown Wall, Plate Slide, Hammer, Door Lock, Handle Pull Side, and Coffee Push. We the maximum episode length 200 environment steps with an action repeat of and a frame stack of across these tasks and adjust the number of training steps to match the varying difficulty levels. During experiments, we observed that high rewards do not consistently correlate with high success rates in the original Meta-world implementation. This discrepancy presents challenge the learning stability of agents. To address this, we introduce an additional bonus for task success Tboous = 10.0 alongside the original task reward miss:
Frask + Thonus
Brack success-
Moreover, Meta-world features hard-cxploration tasks, resulting in significant variance in the learning curves, which poses challenges to the accurate evaluation of RI. algorithm performance. To mitigate this issue, we initialize the replay buffer of all compared methods, with 5 successful demonstration trajectories for all tasks except Door Lock. This strategy, commonly used to accelerate reinforcement learning [36), helps stabilize the training process and provides a more reliable evaluation.
'We use public implementations of metrics: license for SSIM and PSNR, htttp:://iithbbb http:///ttttttcocccccccaaaaaaata Clause license for LPIPS, and htt.:///ss/ggttt.......///ooe/ea FVD.
www.tttitttttetoooooooooooaoaatt under MIT under BSD-2-
under NVidia license for
htttps://gttub..com/s-taaat
21

---

Table Hyperparameters of model-based RL with VideoGPT.
Model-based RI
Hyperparameter
Value
Init rollout batch size Interval
640
200 env. steps 32
Model rollout
Batch size
Horizon
Init training steps
1000 40 env. steps 10 env. steps
Tokenizer training interval
Transformer training interval Batch size
16
Sequence length
12
Model training
Context frames
Sampled future frames (tokenizer)
Learning rate
x 10
Weight decay
0 Adam
Optimizer
Model-based RL
Real data ratio
0.5
1-0
1-2
Cod
L-0
1-6
1-10
1-12
0=00
1-2
1-4
E-0
8-8
10
12
(MSe:PPT (2225)
Deamw (2056))
Figure 1I: Qualitative evaluation on the Open X-Embodiment dataset for Dreamerv pre-trained on the dataset ours.
Implementation Details. We have developed simple model-based RL algorithm using iVideoGPT as world model within the MBPO [42] framework, with DrQ-v2 [103] as the base actor-critic RL algorithm. Please refer to Algorithm for the pseudo-code. Our implementation is based on the official DrQ-v2 code using the same hyperparameters and architecture for actor-critic learning. Hyperparameters specific model-based based RL are listed in Table We use symlog transformation 32] for reward prediction in iVideoGPT.
Baselines. To compare our method with DreamerV3, which lacks native pre-training support, we APV 81 method enabling action-free pre-tran pre-training DreamerV2 baseline, modified to incorporate DreamerV3 features. We pre-train this model on the same dataset as iVideoGPT.
B Extended Experimental Results
B.1
Qualitative Evaluation
We present additional examples of video predictions by iVideoGPT on various datasets Figures [2]
13 15
I6
17 18, and 19. We also include an additional showcase of zero-shot predictions by the
pre-trained transformer iVideoGPT in Figure [4 supplementing Figure of the main text.
Additionally, we showcase prediction examples from the large-scale pre-trained DreamerV3-XX on the Open X-Embodiment dataset in Figures
B.2 Human Study
Numerical metrics like FVD don't always align with human-judged visual quality. To address this, we conduct human user study on the prediction results of various models. Due the lack of official
httte////tthbbc////oooeeeee//r/s under MIT License
22

---

(actiondree)
Ground huch
Joanted
Powdideed
Ground
(oanted)
Prodicted
Eiound buth
(oontest
Predicted
Ground
Joomood
Prodicted
Ground Iuth
Ground Iuth
Figure 12: Additional qualitative evaluation on the Open X- E-boddmenn dataset for action-free free video prediction.
pretrained models for most baselines, we are only able to compare iVideoGPT with VideoGPT 101 and MCVD (93] on the action-free BAIR dataset. We generate videos using these three models from the test set and ask users to label preferences between two randomly sampled videos based on the physical naturalness and feasibility of robot-object interactions. total of 386 annotations are collected from 9 participants. The results in Figure 20 demonstrate that iVideoGPT is preferred by human annotators more.
B.3
Visual Planning
Quantitative results on the Vp² benchmark are reported in Table
B.4 Visual Model-based RL
Comparison to FitVid-based world models. Although FitVid [4] is originally designed for the video prediction task and has not been used as world models for MBRL, we have implemented a baseline using Vid by replacing iVideoGPT in our implementation. To predict rewards, we add an MLP head on top of FitVid's 's latent states, parallel to the image decoder. As shown in Figure 2T MBPO with i iVideoGPT outperforms FitVid 5 out of 6 tasks and performs comparably remaining one. We also qualitatively observe that Fit Vid's imagined trajectories are blurrier compared to ours, which hinders its ability to simulate real environments accurately and may hurt MBRL performance.
the
B.5 Computational Efficiency
We report training and inference time and memory usage with various tokenizers in Table 7 and g respectively. Our proposed compressive tokenization provides significant memory savings during training and faster rollouts during generation. We note that although we use more complicated
23

---

Open (geak-condtionee))
L-0
Ground
(Conkeat)
Predited
buth
Prediced
Ground
buc
Ground bus
Ground buth
Ground
(corlact)
Predited
Figure 13: Additional qualitative evaluation the Open X-Embodiment dataset for goal-conditioned video prediction.
BAIR (ection-trae)
1=2
1+5
1+7
END
1H
IN
Ground immh
Predided Boverizrr the taned)
Protiied
Figure 14: Additional zero-shot prediction by the pre-trained transformer in iVideoGPT, supplement- ing Figure & of the main text.
vuth
Predited
Ground
Figure 15: Additional qualitative evaluation on the BAIR dataset, given future actions.
24

---

RoboNet (action-conditooned)
Ground
fruth
(context)
Predicted
Ground
fruth
(context)
Predicted
Ground
(context)
Prodicted
truth
(context)
Predicted
Figure 16: Additional qualitative evaluation the RoboNet dataset, highlighting accurate movements of the pushed objects.
RoboNet (action-ooddltonnd,, 256x256 resolution)
Ground truth
(context)
Ground
truth
(oontext)
Ground
truth
(contaxt)
Predicted
Figure 17: Additional qualitative evaluation on the RoboNet dataset, in high resolution (256 x 256).
25

---

VP2 (RaboDesk)
1=0
1=2
1=33
1=4
1=5
1-6
1=7
1=8
1=9
1= 1D
Ground
truth
(context)
Predicted
Ground truth
(context)
Predicted
(RoboSulte)
Ground truth
(context)
Predicted
Ground
(context)
Predicted
Figure 18: Additional qualitative evaluation on the Vp benchmark.
Meta-world [action-condittooeed & reward prediction)
1=0
1=1
1=2
1=3
1=4
1=5
1=
1=7
1=8
t=
1=
1= 11
Ground
Iruth
(contex))
Prodicted
Ground
Iruth
(context)
Predicted
21.22
16.03
16,06
16,89
21,72
20,50
26.49
20,58
20,75
26,73
truth
(conlext)
21.38
17.02
1612
17,08
GOLRS
GSUI
26,00
26,00
Predicted
4.2
SPI
5.99
5,9
Ground truth
(context)
500
Predicted
Figure 19: Additional qualitative evaluation Meta-world tasks. True and predicted rewards labeled at the top left corner. Zoom in for details.
26

---

MVdeoGPT win
Drew 24.6%
VideoGPT win 24.7%
MideoGPT va. VideoGPT
50.7%
MidcoGPT win
Draw
MGVD win
MideoGPT MCVD
7%
MCVD vs. VideoGPT
46.3%
29.7%
24.0%
Figure 20: Human study. Videos generated by three models, VideoGPT, MCVD, and iVideoGPPT, on the action-free BAIR dataset are presented to human users, who label their preferences based on the physical naturalness and feasibility of robot-object interactions.
Table 6: Quantitative results on the Vp² benchmark, reporting mean, min, and max performance over various control runs.
Success rate
iVideoGPT (ours)
Struct- VRNN
Tasks
FitVid
SVG
MCVD
MaskVIT
Simulator
Robosuite
mean
0.7833
0.6760
0.7980
0.7733
0.8260
0.5540
0.9350
0.7950
0.7900
0.8400
0.7900
0.8500
0.6000
0.9500
push
0.7750
0.6400
0.7600
0.7400
0.7900
0.5000
0,9200
Flat
mean
0.0333
0.0917
0.0200
0.0500
0.0400
0.0467
0.1333
0.0417
0.1333
0.0333
0.0667
0.1000
0.1333
0.1333
block
min
0,0250
0.0667
0.0000
0.0333
0.0000
0.0000
0.1333
0.3750
0.2533
0.1667
0.1167
0.0400
0.0267
0.7667
Open
0.3917
0.3333
0.2667
0.1333
0.1000
0. 1000
0.7667
drawer
min
0,3500
0.1333
0.0667
1.1000
0.0000
0.0000
0.7667
0.1611
0.3533
0.5733
0.1833
0.0867
0.1267
0.7167
Open
slide
max
0.1917
0,4000
0.7333
0.2000
0.1667
0.2333
0.7333
min
0.1250
0.2667
0.4667
0.1667
0.0333
0.0667
0.7000
0,9556
0,9400
0.9733
0.9500
0.9467
0.8667
1.0000
Blue
button
max
0,9833
0,9667
1.0000
1.0000
0.9667
0.9000
1.0000
min
0,9333
0.8667
0,9333
0.9000
0.9333
0.8000
1.0000
0,8250
0.8400
0.8133
0.8333
0.6400
0.6800
0.9667
Green
button
max
0,8667
0.9000
0,9000
0.8333
0.7000
0.8000
0,9667
min
0.7833
0.7667
0.7667
0.8333
0.6000
0.5667
0.9667
0,9222
0.5867
0.7600
0.7333
0.2400
0.3067
0,9000
Red
button
max
0,9333
0.6333
0.8667
0.7333
0.3333
0.3333
0,9000
min
0,9000
0,5000
0.6333
0.7333
0.1333
0,2333
0,9000
0.4472
0.5133
0.4867
0.5667
0.6200
0.3333
0.9000
Upright
max 0,4667
0.5667
0.6667
0.6000
0.7333
0.3667
0,9000
block
min 0,4250
0,5000
0,4000
0.5333
0.5000
0,3000
0,9000
tokenizer design, it is not the bottleneck of generation time. Additionally, although we use more tokens for context frames compared to the x tokenizer, generation time is primarily influenced by the number of forward passes of the autoregressive transformer, which remains the same.
Table 7:
Training efficiency of the transformer with various tokenizers, measured 40G A100 GPUs with per-device batch size of 16.
Tokenizer
Speed (#iters/secc
Memory (GB)
4 x 4
3.10
10.6
16 x 16
N/A
00M
Ours
2.62
22.3
27

---

Table Generation efficiency with various tokenizers, measured on an RTX 4090 GPU with a batch
size of 1.
Tokenizer
Tokenize (sec)
Generation (sec)
Detokenize (sec)
Memory (GB)
4
0.27
1.13
0.05
1.98
16 x 16
0.26
22.5
0.04
1.90
Ours
0.29
1.11
0.06
2.33
Tapdown
Slide
100
100
103
A44
voro Toobl (riee
23-
Environment Sleps (x10')
Environment Stepe (x10*)
Environment Stepe (x10")
Coffee
Success 100
100
102
20
20
Environment Sleps (x10*)
Envirommnt Stepe (x10f)
Environment Stepe (x110))
Figure 21: Visual model-based RL on Meta-world, comparing to an additional baseline using FitVid as world models.
C Extended Discussions
C.1 Differences with IRIS
Discrete tokenization and autoregressive transformers are prevalent in contemporary deep learning due to their simplicity and generality. iVideoGPT generally shares this architecture with IRIS [63]. but possesses distinguishing features, summarized as follows:
Pre-training and fine-tuning paradigm: iVideoGPT is designed for a paradigm that involves pre-training on large-scale videos and fine-tuning various downstream tasks. In contrast, IRIS focuses solely on MBRL with Transformer-based world models trained from scratch in the Atari domain.
Efficient tokenization: iVideoGPT proposes novel compressive tokenization to significantly reduce the number of tokens, saving time and memory (see Table 7 and 8). while IRIS uses per-frame tokenization.
Flexible action-condittoning design: iVideoGPT employs slot tokens with optional additive action embeddings support both action-free pre-training and action-condittoned fine- tuning, while IRIS strictly treats discrete Atari actions as tokens.
Off-policy MBRL implementation: iVideoGPT uses an off pol-cy RL algorithm while IRIS performs on-policy learning. On-policy learning needs large number of model rollouts, which, combined with inefficient tokenization, results in 7 days for wwok--nvirommen--stt training. In comparison, |VideoGPT only needs <4 hours.
C.2 Differences with VideoGPT
We elaborate on the the difference between the tokenizer in VideoGPT 101 and ours, and how they impact interactivity.
28

---

VideoGPT uses a VQVAE for video that relies on series of 3D convolutions to downsample across space and time. For example, downsamples original pixels from 16 64 x 64 to discrete tokens of 8 x 32 x 32 or 4 x 16 x 16, depending the downsampling ratio. The key issue is that this non-causal downsampling over the temporal dimension results in each token containing information from window of frames. As result, the entire video of a fixed length can only be reconstructed after VideoGPT generates all tokens. As shown in Figure 2 VideoGPT only allows the input of future action sequences at the beginning of prediction, preventing agent from interactively determining its actions based on predicted observations. In contrast, our tokenizer discretizes video frames separately, using conditional mechanism handle temporal redundancy, enabling frame-by-frame -rame video generation and allowing for intermediate action intervention.
Moreover, our tokenizer's 's novel design, with its cross-attention mechanism, is more efficient in handling temporal redundancy, converting videos into significantly fewer tokens (T. = 511 with N = 256, n = 16, 7 = 16, To as stated in Section B.I.. In contrast, VideoGPT finds that using a larger downsampling ratio than a token size x 32 32, results worse performance.
C.3 Failure Case Analysis for Visual Planning
Our model performs sub-optimally on the RoboDesk open slide task from the Vp² benchmark. In this section, we investigate the underlying causes through case studies, attributing the performance issues to limitations in both our model and the benchmark.
Inaccurate model prediction. Despite achieving excellent overall video prediction metrics, such as mean square error and perceptual loss, on the validation set for the open slide task, our model predicts wrong outcomes on a few trajectories. We visualize these trajectories in Figure and find that while the observation is limited to 64 x 64 resolution, the task of opening the slide requires the model to capture subtle changes, particularly whether the robot's gripper has made contact with the slide handle. Actually, even humans struggle to discriminate this detail with low-resolution inputs. Due to this uncertainty, the model may incorrectly predict a sequence of imprecise actions as successful. This overconfidence [66] can be exacerbated in the process of model predictive control, which samples a large number of action candidates and selects the "best" one according to the model. Our analysis provides explanation to the observation by Tian et al. [86] that overall excellent perceptual metrics do not always correlate with effective control performance, as the worst-case scenarios are critical in podel-predictive control.
Furthermore, we hypothesize that our two-stage architecture of tokenization and prediction can exacerbate the aforementioned uncertainty, as discrete tokenization inevitably results in some loss of information from the observations. This hypothesis is supported by the fact that end-to-end models like SVG' [91] and Vid [4] perform significantly better than two-stage models, including ours and others like Mask ViT (27. which uses a visual tokenizer, and Struet- VRNN 64 which employs a keypoint-based representation.
We anticipate that training and evaluating our model at higher resolution, such as 256 x 256, could mitigate these issues and enhance control performance. However, we currently conduct experiments lower resolution to ensure fair comparison with other models.
Imperfect built-in reward design. We observe that no current model in the Vp² benchmark consistently outperforms other models across all tasks, and iVideoGPT is no exception. Beyond models' inaccuracies in prediction for severely out-of-distriiuttin (OOD) actions, our analysis of this inconsistent performance also reveals flaws in the benchmark's 's built-in reward design.
In VP², scores for sampled actions are estimated mainly by a learned classifier that assesses task success based on model-predicted frames. This classifier, trained by the VP2 authors, appears to lack robustness and is easily fooled by OOD inputs, assigning high rewards to low-quality or unlikely-to- succeed predicted trajectories (see examples in Figure 23). Such an imperfect reward function likely contributes to the mixed results observed on this benchmark, with iVideoGPT even outperforming the oracle simulator one task. Addressing visual planning with imperfect rewards is an independent research problem and beyond the scope of this paper.
29

---

VP2: RoboDesk (Fallure Cases)
1-0
1-2
1-3
1-4
1-5
10
Ground truth
(context)
Predicted
Ground truth
(context)
Ground truth
(context)
Predicted
Figure 22: Failure case analysis on the RoboDesk open slide task from the vp benchmark, where, likely due to the low resolution of observations, model fails to discriminate between subtle changes, particularly whether the robot's s gripper has made contact with the slide handle.
= 0
1=2
t 4
t=6
t
1 10
t= 11
Fallure trajectory with Higher reward (-7179.46)
Reasonable trajectory with Lower reward (+7308.77)
Figure 23: Imperfect built-in reward in vp² benchmark. learned reward model can assign high rewards to predicted transitions that less likely succeed, which can mislead optimizers in model-predictive control.
D Computational Resources
We implement /VideoGPT PyTorch, using the diffusers and trans orrerr libraries. Our models are trained and evaluated
A100 and RTX 3090 GPU cluster. Each experiment utilizes GPUs in parallel, with 16 data loader workers per device. GPU days required for training are reported in Table 2 Experiments 64 64 resolution can be conducted with 24 GB of GPU memory per device, while 256 x 256 resolution requires 40 GB. The Open X-Embodiment dataset is particularly large, occupying about 5TB of disk space.
E Broader Impact
World models advance the development of autonomous machine intelligence, particularly through the valuable visual insights offered by videos. However, their full potential remains untapped without scalable and interactive architectures capable of distilling vast amounts of commonsense knowledge from multimodal data. This paper, we believe, takes an important step by introducing a flexible framework with specific focus on the robotic manipulation domain. Our results may pave the way for higher-quality world models applicable across diverse domains, enhancing performance in control tasks of embodied intelligence. Despite the benefits, designing and training these models is challenging, requiring substantial computational power and increasing the carbon footprint. Using
under Apache License
httppp///gg..a.oooo///gggggagoro under Apache License
30

---

underdeveloped. inaccurate world models for autonomous control could lead to risky actions and potential physical damage, which can be mitigated by developing uncertainty-awant models to prevent uncertain actions. Conversely, the advancement of these models could lead to job displacement in sectors relying on manual control tasks. Additionally, the underlying techniques for world models can be misused to generate synthetic videos that mimic real events or people. However, since our model merely a research prototype, trained only with robotic and human manipulation data and relatively small in scale, we do not anticipate immediate negative societal impacts such deepfakes or job displacement.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our experiments in Section 4 well support our claims and contributions made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed limitation in Section 6 and provided failure case analysis in Appendix C.3
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer:
[NA]
Justification:
We do not contribute theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have described clearly and fully the architecture and experiments in the main text and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
33

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The data used in this paper are all publicly available, detailed in Appendix We have released our code at
com/thuml/VVideoG7
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pbbll. pttppppullllleeeee Sute:////uudde//eeeuuubbbii (CooeSubmisssi. wwtplubmmsssioooo ssiiiissiccyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: See Table
Figure 5 and Figure 7
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////aaareecccccccccccccceaa
Answer: [Yes]
Justification: Our research does not involve human subjects, and all datasets are utilized in compliance with licensing requirements. Data representative evaluation practice is done as described in Appendix A.2] The research poses no harm to society. Details of the datasets and models have been thoroughly discussed in both the main text and appendix. No sensitive data is used in this study. Models will be released with licenses for reproducibility, after publication.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
35

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our models are only trained with robot and human manipulation data, without the risk of misuse, such deepfakes of important events or figures.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited all assets used in the paper. In Appendix A we explicitly mentioned and respected the license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
36

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have released our pre-trained models, and inference examples are available at Inttps://githut: httpeitthh....tttteeeeeeeeeeeeee com//hhumm//VideoGr"
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This study does not involve crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study does not involve such participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
37

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "zHtxsMb69O", "reviewer": "Reviewer_HkK9", "summary": "The authors present a paper that attempts to utilize actionless and action conditioned trajectories to learn a large scale interactive world model. This model is subsequently adapted for robot manipulation tasks. It is evaluated on video prediction, visual planning, model based RL. The model training is tested primarily on Metaworld in the embodied setup. The experiments demonstrate the generalization capabilities of the model.", "questions": "1. Is the dreamer baseline in the paper trained on OpenX data also? If not then the comparison is not fair.\n2. How do you ensure that the $z^(1:n)_t$ contain all the information about future frames and that there is no leakage between which tokens contain what information?", "limitations": "1. The evaluation suite is currently very small should be expanded\n2. Its unclear how much the human videos from Something-something-v2 help the method. If they do not help the method significantly, then this diminishes the contributions of the paper significantly as this would make it a study on the application of transformer based world modeling on large robot datasets like OXE. \n\nInspite of these limitations and weaknesses, I think the work makes interesting contributions and I would inclined to increase my score if the authors are able to address my questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written, clear and easy to understand\n2. The idea is well motivated. The question of utilizing internet scale knowledge for embodied intelligence is an open area of research.\n3. The experiment results are statistically significant with multiple runs reported accompanied by error bars\n4. The authors present a novel tokenize scheme which can benefit other video foundation models", "weaknesses": "1. **Performance on Video prediction:** The performance for the visual planning is rather unsatisfactory. Do the authors have any intuition about why their method only outperforms the baselines in 2 of the setups?\n2. **No analysis on amount of action data needed:** During the motivation of the method, the authors discuss coming up with methods that are able to learn from freely available videos. Robot data on the other hand is expensive. Thus, ablations on how much robot data is needed are necessary to understand the dependence of the model on robot specific data and whether or not the method is indeed benefitting from freely available human videos.\n3. **Lack of robot experiments:** Does the model extend to any real robot setups? Currently, the only low level control robot experiments available are on 6 in which the model matches the performance in 3. \n4. **No human user studies:** the paper only uses numerically metrics like Frechet Video Distance to judge the quality of generation. This metric does not always align with quality. The results of such a study would help bolster the quality of the work.\n5. **Some missing Baselines**: FitVid does indeed have an action conditioned model and this model could be used for all the MBRL experiments. This currently missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely appreciate Reviewer HkK9 for the comprehensive review and insightful questions. We have **addressed common questions in the global response and indivual questions below**.\n\n### Q1: Amount of action data needed\n\nAs described in $\\underline{\\text{Sec 3.2}}$, we **do not use any action data during pre-training**. Leveraging more accessible, action-free video data reduces the cost of data collection. Although action-free robot videos are still more expensive than Internet videos, there has been significant progress in data sharing (e.g., BridgeData, RH20T, OXE), making robot data increasingly affordable. **What we truly need are unified frameworks like iVideoGPT to better utilize these**.\n\n**Expensive action-conditioned data are only used for downstream tasks**. In $\\underline{\\text{Sec 4.4}}$, we have demonstrated the few-shot adaptation capabilities for both (action-free) video prediction (Fig 7, 8c) and MBRL (Fig 6). In addition, we have also adapted iVideoGPT with 1,000 action-conditioned BAIR trajectories, which achieves 82.3 FVD.\n\n### Q2: Contribution of human videos\n\nAs detailed in Appendix A.2, we used a mixture of OXE and SthSth for pre-training the tokenizer, as visual diversity is the key to learning context and dynamics information separation. The transformer is pre-trained on OXE only, as we found many SthSth videos with random camera motions are hard to predict. These choices were based on our experience and a comprehensive analysis is expensive and left for future work.\n\nTo assess the contribution of human videos, we pretrain iVideoGPT with pure OXE data (taking almost a week) and evaluate on OXE validation data. We observe human videos indeed help pre-training.\n\n| Pretrain data | FVD      | PSNR     | SSIM     | LPIPS   |\n| ------------- | -------- | -------- | -------- | ------- |\n| OXE only      | 48.3     | 24.3     | 85.5     | 9.1     |\n| OXE+SthSth    | **40.5** | **24.8** | **86.3** | **8.5** |\n\n\n### Q3: MBRL baseline with FitVid\n\nAlthough FitVid is originally designed for video prediction and has not been used as world model for MBRL, we have implemented a baseline using FitVid, by replacing iVideoGPT in our implementation. To predict rewards, we add an MLP head on top of FitVid's latent state, parallel to the image decoder.\n\nAs shown in $\\underline{\\text{Fig 5 of attached PDF}}$,  MBPO with iVideoGPT outperforms FitVid on 5/6 tasks and performs comparably on the remaining one. We also qualitatively observe that FitVid's imagined trajectories are blurrier (as shown in $\\underline{\\text{Fig 4 of attached PDF}}$) compared to ours, which hinders its ability to accurately simulate real environments and may hurt MBRL performance. \n\n### Q4: Dreamer baseline with pre-training\n\nAs elaborated in $\\underline{\\text{Q1 of global response}}$, Dreamer, with its RNN architecture, lacks the scalability to benefit from pre-training on real-world videos [1]. To empirically validate this, we have used a pre-training algorithm, APV [1], to pre-train DreamerV3-XL (200M parameters, comparable to iVideoGPT) on the same pre-training data as ours, and then finetune it on Metaworld tasks.\n\nThe results in $\\underline{\\text{Fig 5 of attached PDF}}$, indicate that Dreamer almost does not benefit from pre-training. We also visualize DreamerV3's predictions after pre-training in $\\underline{\\text{Fig 2 of attached PDF}}$, which is of greatly lower quality compared to iVideoGPT, validating the limited capacity of Dreamer architecture.\n\n[1] Reinforcement Learning with Action-Free Pre-Training. ICML 2022.\n\n### Q5: Human study\n\nAs official pretrained models are not released for most baselines, we are only able to compare iVideoGPT with VideoGPT and MCVD on the action-free BAIR dataset. We generate videos using three models from the test set and ask users to label preferences between two randomly sampled videos, based on the physical naturalness and feasibility of robot-object interactions.\n\nWe collect 386 annotations from 9 persons. The results in $\\underline{\\text{Fig 6 of attached PDF}}$ demonstrate that iVideoGPT is more preferred by human annotators.\n\n### Q6: Lack of real-robot experiments\n\nWe would like to clarify that, instead of being 'tested primarily on Metaworld in the embodied setup', our experiments in Sec. 4.2 (visual planning) and Sec. 4.3 (visual MBRL), although conducted in simulation, are **both embodied low-level control settings**.\n\nWe apologize that the limited time of rebuttal is insufficient for us to set up a real-robot experiment. We do not claim a contribution that our method is instantly ready for real-robot applications, which is left for future work, but we believe our method can help advance model-based methods in robotics research.\n\n### Q7: Visual planning performance\n\nWe find that inconsistent performance in the VP2 benchmark can be primarily attributed to inaccurate built-in reward design of this benchmark. Please refer to Q4 in global response for details.\n\n### Q8: \"Small\" evaluation suite\n\nWe respectfully disagree with \"The evaluation suite is currently very small\". In fact, **no baselines cover all three problem settings in our paper**. We hope additional results provided in this rebuttal can solidify our evaluations.\n\n### Q9: Context-dynamics decomposition in tokenization\n\nWhile we cannot guarantee in neural networks perfect decomposition and no information leakage, we aim to achieve this through a careful tokenizer design: a bottleneck with much fewer tokens compels to only capture necessary dynamics information for future frames and share contextual information with initial frames, to reconstruct raw pixels.\n\nWe visualize the decomposition property in $\\underline{\\text{Fig 1 in attached PDF}}$. The showcases illustrate that, by removing cross-attention from context frames into future frames, the decoder can still reconstruct a trajectory **moving the same way** as the normally reconstructed one but the **context is almost missing**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eaPj0JxhdR", "reviewer": "Reviewer_zSnp", "summary": "This paper introduces a new architecture for an action-conditioned video (and reward) prediction model.\nFirst, a VQGAN converts context frames individually into tokens.\nSecond, a conditional VQGAN converts future frames individually into tokens, conditioned on the context frames (using intermediate representations from the context encoder).\nThe idea is that the future encoder can focus on encoding changes in the scene, since the remaining information was already extracted from the context, allowing for a lower number of tokens.\nThen, an autoregressive transformer is trained to predict the next (future) token and is optionally conditioned on actions and can optionally predict the rewards in a reinforcement learning setting.\nThe model is pre-trained on action-free robotic manipulation videos and then fine-tuned and evaluated on video prediction, visual planning, and model-based reinforcement learning tasks.", "questions": "- Q1: Is the world model also conditioned on the actions from the context? In line 87 the condition only includes $a_{T_0:t}$, but in the sequence of tokens $x$, the context also includes slot tokens. This can also not be recognized in Figure 3(b).", "limitations": "The authors addressed all limitations adequately.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- S1: Encoding the dynamics information conditioned on context information is an intriguing idea for world models.\n- S2: The paper is clearly written and the visualizations are informative and appealing.", "weaknesses": "- W1: The paper is missing quantitative comparisons of the computational efficiency. For instance, it would be interesting to see the differences in training/inference time when using the different tokenizers (Figure 8(c)).\n      Furthermore, the authors state at the end of Section 2 that recurrent world models like Dreamer are not scalable, but I'm wondering how significant the difference is, considering the autoregressive next token prediction and use of multiple tokens per frame.\n- W2: The authors argue that iVideoGPT eliminates the need for latent imagination in model-based RL (Section 4.3 and Figure 5).\n      However, there are existing world models (e.g. IRIS [1]) that learn behaviors using reconstructed frames.\n      Moreover, Dreamer could also learn behaviors this way by using the reconstructions from the decoder (but I understand that this would be a different method).\n      In short, I don't think that this is a feature that is novel or specific to iVideoGPT.\n- W3: The proposed model has a lot in common with IRIS [1], which learns a discrete autoencoder (VQVAE) and uses an autoregressive transformer for next token prediction.\n      IRIS is briefly mentioned in the related work section, but I think the authors could emphasize the differences in more detail.\n\nSome typos:\n- 79: \"aims\"\n- 80: \"maximize\"?\n- 85: \"history of $T_0$ video frames\"?\n- 86: \"needs to\"\n- 107: $D_c(z_t)$ should be $D_c(z_t^{(1:N)})$?\n- Also in Eq. (1) it should be $D_p(z_t^{(1:n)}|o_{1:T_0})$?\n- 687: Table 2 -> Table 3\n\nI am willing to increase my scores after the listed weaknesses have been addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer zSnp for the thorough review and valuable questions. We also appreciate the pointed-out typos, which we will correct in a future revision.\n\n### W1: Computational efficiency\n\nWe apologize for missing a quantitative efficiency analysis. We have reported training/inference time and memory usages with various tokenizers, in $\\underline{\\text{Q2 of global response}}$. Our proposed compressive tokenization provides **significant memory savings during training and faster rollouts during generation**.\n\n#### Discussion on Dreamer\n\nWhile we have enhanced the scalability of Transformer-based world models through compact tokenization and resource savings, Dreamer with an RNN architecture may still be more efficient computationally. However, as demonstrated in $\\underline{\\text{Fig 2 of attached PDF}}$, when pre-training Dreamer XL (200M parameters, comparable to iVideoGPT) on the same dataset as us, we find that it has **insufficient capacity to support large-scale pre-training**, which is crucial for the success of modern foundation models. The results in $\\underline{\\text{Fig 5 of attached PDF}}$ further prove that Dreamer is **unable to benefit from such ineffective pre-training**.\n\n### W2: Eliminating latent imagination\n\nWe do not claim that eliminating latent imagination is a unique feature of iVideoGPT. However, since latent imagination is currently the dominant practice in MBRL (as seen in Dreamer, MuZero, etc.), we believe it is an advantage for iVideoGPT to simply serve as a plug-in replacement of the environment. As discussed in Sec 4.3, this can simplify the design space, reduce implementation complexity, and enhance the practicality of MBRL.\n\nThe key to this advantage is a powerful world model. Previous action-conditioned video prediction models like FitVid, without sufficient capacity and pretrained knowledge, can also function similarly to iVideoGPT but can **produce more blurring predictions** ($\\underline{\\text{Fig 4 of attached PDF}}$), thus **hinders MBRL directly on top of these inaccurate frames** ($\\underline{\\text{Fig 5 of attached PDF}}$). These results may explain why Dreamer employs latent imagination in MBRL. While IRIS shares a similar architecture with iVideoGPT, it lacks efficient tokenization and large-scale pre-training (as discussed below).\n\n### W3: Difference with IRIS\n\nWe summarize four key differences between our approach and IRIS. Please refer to $\\underline{\\text{Q3 of global response}}$.\n\n### Q1: Conditioning on the context actions\n\nIn our implementation, iVideoGPT is not conditioned on context actions. However, it can easily be extended to support this by adding the embedding of context actions to the slot tokens between context frames. We hypothesize that this would not significantly impact performance, as context actions can likely be inferred implicitly from context frames.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9cu6ZzIbdY", "reviewer": "Reviewer_yh7R", "summary": "This paper introduces Interactive VideoGPT (iVideoGPT), which builds world model based on VideoGPT architecture. iVideoGPT proposes compressive tokenization, and the model is trained using millions of human or robot videos (i.e., Open X embodiment dataset). The effectiveness of iVideoGPT is demonstrated in video prediction, visual planning, and visual model-based reinforcement learning.", "questions": "- What is main difference between iVideoGPT and  IRIS [1] for visual model-based reinforcement learning of visual planning?\n\n[1] Micheli, V., Alonso, E., & Fleuret, F. (2022). Transformers are sample-efficient world models. arXiv preprint arXiv:2209.00588.", "limitations": "See Weakness & questions.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The empirical validation is extensive including video prediction, visual planning and visual model-based reinforcement learning.\n- The scalability of iVideoGPT is impressive.", "weaknesses": "- Technical novelty is somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer yh7R for providing a thorough review, valuable questions, and a positive evaluation of our paper.\n\n### Q1: Technical novelty\n\nDiscrete tokenization and autoregressive transformer are prevelant in contemporary deep learning, due to their simplicity and generality. iVideoGPT generally shares this architecture with IRIS, but possesses distinguishing features (discussed below in Q2).\n\nDespite adopting a widely used architecture, we believe the most important contribution of our paper lies in **advancing a paradigm of pre-training a scalable world model architecture on large-scale data and adapting it into various downstream tasks**. This paradigm is well-established in many other fields of deep learning (e.g., languages and the closely related field of video generation) but is not yet commonly applied in MBRL (see $\\underline{\\text{Q1 of global response}}$ for extended discussion).\n\n### Q2: Difference between iVideoGPT and IRIS\n    \nWe summarize four key features of iVideoGPT that are different from IRIS:\n\n1. Pre-training and fine-tuning paradigm enhancing sample efficiency; \n2. Novel and efficient tokenization enabling **faster rollouts** (**~24x**, see Q2 in global response); \n3. Flexible action-conditioning design; \n4. Off-policy MBRL implementation. \n\nAll of these features are highly relevant to visual planning and visual MBRL. Please refer to $\\underline{\\text{Q3 of global response}}$ for a more detailed discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xW9QiCZysE", "reviewer": "Reviewer_6ajb", "summary": "This work studies the setting of planning and prediction from video world models. It proposes to use a GPT-style transformer world model that incorporates action and reward information in it's context and prediction pipeline. The model is further equipped with a novel tokenization technique based on VQGAN. Finally, the model is evaluated on three downstream applications, namely video prediction, planning and model-based RL. Ablations on the models and tokenizers scalability are conducted.", "questions": "Q1: What are the training times of your algorithm training and how do they compare to the runtimes of Dreamer?\nQ2: Can you elaborate on the term interactive and why other models are not interactive?\nQ3: What are the key differences between the proposed model and the MaskVIT model and to which of the difference do you attributed the improved action-conditioned prediction performance in section 4.1?", "limitations": "The paper contains an explicit section outlining limitations of the work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "Problem Statement\n* The problem of studying dyna-style video algorithms with foundational models is interesting and warrants investigation.\n\nClarity\n* The paper is very well written and the language is clear\n* The paper follows a very concrete thread and is easy to follow\n* All plots and figures are well-formatted and easy to understand\n\nRelated Work\n* The treatment of related work is done very nicely. The paper cites a total of 120 other works which is not often seen and thus formidable.\n* The work highlights older as well as recent work and positions itself well within the literature\n\nExperiments\n* The experiments contain a large number of baselines and comparisons that are useful to understand the capabilities of the model for downstream return maximization.", "weaknesses": "Clarity\n* In some cases, the writing makes very strong statements that are very broad and hard to be supported by evidence. E.g.\n  * L97 \"This model can acquire broad world knowledge\". This statement is not supported and it is unclear to me what \"broad world knowledge\" means. I believe concise language about the capabilities of our algorithms is important.\n\nMotivation\n* One issue with the paper is that some of the motivation is not quite clear to me. The paper argues that the provided world model has two benefits: It is interactive and scalable. \n  * First, it is not clear to me what it means for this model to be interactive. If the condition for a model is to be action-conditioned, then the dreamer world-model should also be interactive. In fact, even much older world-models would be interactive [1, 2]. \n  * Second, it is not clear to me that other models would not be scalable and the paper provides no evidence that they are not scalable. The text states in line 91 that dreamer lacks scability, however not citation or evidence for this is provided. For instance, dreamer uses state-space models (SSMs) and recent advances have built scalable SSMs [1].\n\nExperiments and Claims\n* For the tokenizer, the text claims that it \"decouples dynamics information from context conditions\" L161. This seems like a strong statement that would warrant evidence to be supported. I believe qualitatively looking at a small number of image sequence is insufficient to support such a strong claim about composition.\n* Overall, the experimental results seem a little weak. \n  * The model is outperformed on most tasks in the video-prediction setting. It is unclear why the one of the weakest baselines was used for the for action-conditioning comparison in section 4.1.\n  * In the visual planning experiment, the performance exceeds baselines in only 2/7 tasks.\n  * In the model-based RL experiments, the performance is mostly comparable to dreamer. There are cases where the paper argues that their suggested algorithm outperforms dreamer but given that the results are only reported over $5$ random trials, it seems that the results are within variance making this claim too vague. (It’s a little hard to tell because shaded regions are very transparent)\n* The benefits of scalability for the downstream control tasks have not been demonstrated. \n  * The work argues that high perceptual metrics don't necessarily correlate with good control performance (App B.2) but the main benefit of scale seems to be improved performance on those metrics.\n  * While the work shows in section 4.4 that scaling leads to lower validation loss, whether or not this is correlated with downstream performance is not demonstrated. I believe this point is important because we have to ask what is the marginal benefit of increasing computational complexity to train larger and larger models, if the application to downstream tasks does not benefit much from scale. Showing that larger networks and lower error lead to improved downstream performance is an experiment that might validate the need for scale.\n* Similarly, the need for specialized tokenization has not been argued for downstream applications. It seems a little detached from the goal. The main goal of the paper is to argue for scalability and that this tokenization technique is helpful. However, for the downstream applications, neither the planning nor RL sections demonstrate the importance of scaling or tokenizing. One experiment I can think of is to demonstrate that 4x4 tokenization is insufficient due to reconstruction quality this claim. The relationship between scaling and the novel tokenization is not analyzed either.\nOverall, I think the motivation of the work is not quite clear enough, the contributed algorithm provides minor improvements in the control settings and the claims for interactivity and scalability could be strengthened.\n* The text claims that the tokenization that is presented is more efficient but there is no experiment validating that this is true. A complex encoder structure is used and it is unclear whether this is actually more efficient. An experiment to validate this could measure total wallclock times on the same hardware.\n\n[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Albert Gu, Tri Dao. arXiv preprint arXiv:2312.00752.  \n[2] Action-Conditional Video Prediction using Deep Networks in Atari Games. Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, Satinder Singh. Advances in Neural Information Processing Systems 28 (NIPS 2015).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 6ajb for providing a detailed review and insightful questions. We have **responded to common questions in the global response and individual questions below**.\n\n### 1. Motivation of iVideoGPT: Best of both interactivity and scalability\n\nIn $\\underline{\\text{Q1 of global response}}$, we have clarified that iVideoGPT is motivated by two lines of research and aims to achieve both interactivity and scalability.\n\nTo specifically answer Reviewer 6ajb's question:\n\n- Interactivity: We define the term \"interactive\" as that a model can predict future steps incrementally, conditioned on intermediate actions (as defined in Sec 2 of paper). While **Dreamer and many classic models do have interactivity**, we remark that **many advanced video generation models do not**. These include well-known Sora and action-conditioned autoregressive models like VideoGPT as shown in Fig 2b of paper.\n\n- Scalability: \n    - Dreamer uses an RNN architecture which has weaker scalability (**defined and discussed with evidence in global response**). \n    - Why scalability is important? Pre-training foundation models with large-scale data is a well-established paradigm in many other fields of deep learning (e.g. languages and the closely related field of video generation), providing impressive success, but is **under-explored and not yet commonly applied in model-based control**.\n    - While the reviewer mentioned Mamba, we argue that Mamba's macro architecture is highly akin to Transformers, with interleaved temporal modules (SSM) and temporal-independent MLPs, which is far from Dreamer. To our knowledge, **Mamba has not been used in world models and no such Dreamer variants exist in the literature**.\n\n### 2. Tokenization\n\n***Context-dynamics decomposition***\n\nTo support the decomposition property of the tokenization, we visualized the decoder's reconstruction in $\\underline{\\text{Fig 1 of attached PDF}}$ by removing cross-attention for future frames. We observed that, without cross-attention, the decoder still reconstructs a trajectory **moving in the same way** as the original, but the **context is almost entirely missing**.\n\n***Computational efficiency***\n\nWe measure time and memory usage on the same hardware, which is reported in $\\underline{\\text{Q2 of global response}}$. Despite the complexity of our encoder-decoder structure, it only takes up a little more resources than a vanilla 16x16 per-frame tokenizer and saves a lot during training and generation of the transformer.\n    \n***Relationship with scaling***\n\nEfficient tokenization greatly saves computational comsuption for training and generation (rollouts), allowing us to **scale up the model size with fewer costs**. \n\n### 3. Experiments\n\nOur experiments aim to demonstrate that **one architecture and pre-trained model can be adapted into various downstream tasks to achieve competitive performance**. Although we do not significantly outperform state-of-the-art methods on all tasks, we believe it is a valuable step.\n\n***Video prediction baselines***\n\nWe only compare with MaskViT on action-conditioned BAIR and high-resolution RoboNet settings, since all other models do not report these settings and many strong baselines such as FitVid or MAGVIT do not release official training codes or instructions.\n\n***Visual planning performance***\n\nWe find that mixed results across models in the VP2 benchmark can be primarily attributed to its imperfect built-in reward design. Please refer to Q4 in global response for details.\n\n***Significance of MBRL experiments***\n\nTo better show the statistical significance of our superior performance, we follow the protocol from [1] and report aggregated performance using Inter-Quantile Mean across all 30 runs in $\\underline{\\text{Fig 5 of attached PDF}}$. It shows that **our method significantly outperforms DreamerV3**, a well-tuned strong baseline for this setting.\n\n[1] Deep reinforcement learning at the edge of the statistical precipice\n\n***Control benefits of scalability***\n\nThere are two kinds of scaling: **model scaling and data scaling**. In our experiments, we currently do not find that further model scaling can significantly improve performance on Metaworld tasks (iVideoGPT-436M performs on par with 138M), likely because Metaworld is a visually simple simulation environment. However, as shown in Fig. 6 of the paper, **data scaling does provide benefits**---pretraining iVideoGPT enhances MBRL performance. In contrast, Dreamer's lack of scalability limits its potential to benefit from pre-training (see Figs 2 \\& 5 in attached PDF).\n\n### 4. Others \n    \n***MBRL training time***\n\nOur method in PyTorch takes ~4 hours on 4090 to train for 100k environment steps. Official DreamerV3 in JAX takes ~1 hour. When fairly compared under the same framework, a DreamerV3 implementation in PyTorch takes ~3.5 hours for 100k steps, comparable to ours.\n\n***Difference with MaskViT***\n\niVideoGPT and MaskViT are **fundamentally different types of generative models**. MaskViT is a masked model that generates all video frames simultaneously through masked reconstruction, typically trained for fixed-length videos. In contrast, iVideoGPT is an autoregressive model, allowing flexible video generation of various lengths. Additionally, as discussed in $\\underline{\\text{Sec 4.1 of paper}}$, MaskViT uses per-frame tokenization and suffers from temporal inconsistencies, while iVideoGPT employs a novel **tokenization conditioned on consistent contextual information**, contributing to its improved performance.\n\n***Clarification on claims***\n\nWe apologize for the strong statements and will revise them for accuracy:\n\n- **\"broad world knowledge\"** => \"common knowledge of motions and interactions in various scenes through pre-training on diverse human and robotic manipulation videos.\"\n- **\"decouples dynamics information from context conditions\"** => \"designed to encourage the decoupling of dynamics information from context conditions.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their constructive feedback. We have made every effort to address all concerns and have responded to individual reviews. We have also **answered common questions in this global response**.

Please note that **all the new figures for all responses are included in the PDF attachment**.

### Q1: Motivation of iVideoGPT

As discussed in Sec. 1 of the paper, iVideoGPT is **motivated by two distinct lines of research**:

 - **World models** in RL serves as an **interactive** simulation of the real environments by modeling transitions. However, the dominant architectures are still based on RNNs (like Dreamer and MuZero), which have **limited scalability** compared to Transformers (see Fig 7 in [1]; here **scalability means an ability to increase capacity effectively with additional parameters**). Thus, current world models rarely do pre-training on large-scale real-world videos and struggle to benefit from it (see Fig 8c in [2] and Figs 2 \& 5 in attached PDF), leading to insufficient sample efficiency in model-based control.

 - **Video generation models** (e.g. Sora) are recently advanced by **scalable** architecture and large-scale training. They share advantages with foundation models in other fields (like LLMs): strong performance with sufficient task data, fast adaptation with few-shot downstream samples, and zero-shot generalization on unseen domains. However, these models typically generate all video frames simultaneously, allowing text/action conditions only at the start, thus **lacking the ability for interaction** during generation.

To bridge both sides, our work is a framework where **scalable and interactive architecture can be pre-trained with large-scale data and then adapted into various downstream tasks**. Across three control-relevant settings, iVideoGPT demonstrates favorable properties, including **competitive task performance** (Tab 1, Figs 4\&6 in paper), **few-shot adaptation** (Fig 8a in paper), and **zero-shot generalization** (Fig 7 in paper).

While many future directions remain to be explored, we believe that our work as a valuable prototype towards large world models, will contribute to the community.
    
[1] Scaling laws for neural language models. 2020.
[2] Reinforcement Learning with Action-Free Pre-Training. ICML 2022.

### Q2: Tokenization Efficiency

In addition to scalable Transformer architecture, we introduce a new compressive tokenization. By exploiting temporal redundancy, a significantly fewer amount of tokens can **greatly save time and memory, allowing us to scale the model size with fewer costs**.

To prove the efficiency, we report the time and memory consumption of iVideoGPT with various tokenizers, showing that our method achieves a good balance between efficiency and quality:

- Pre-training the transformer (A100 with per-device batch size 16)

|Tokenizer|Speed (iters/sec)|Memory (GB)|
|-|-|-|
|4x4|3.10|10.6|
|16x16|N/A|**OOM**|
|Ours|2.62|22.3|

- Inference (RTX 4090 with batch size 1)

|Tokenizer|Tokenize (sec)|Generation (sec)|Detokenize (sec)| Memory (GB)|
|-|-|-|-|-|
|4x4|0.27|1.13|0.05|1.98|
|16x16|0.26|**22.5**|0.04|1.90|
|Ours|0.29|1.11|0.06|2.33|

### Q3: Difference with IRIS

We can summarize the following differences with IRIS:

1. **Pre-training and fine-tuning paradigm**: iVideoGPT is designed for a paradigm that involves pre-training on large-scale videos and fine-tuning on various downstream tasks (see Q1 above). In contrast, IRIS focuses solely on MBRL with Transformer-based world models trained from scratch in the Atari domain.
2. **Efficient tokenization**: iVideoGPT proposes novel compressive tokenization to significantly reduce the number of tokens, saving time and memory (see Q2 above), while IRIS uses per-frame tokenization.
3. **Flexible action-conditioning design**: iVideoGPT employs slot tokens with optional additive action embeddings to support both action-free pre-training and action-conditioned fine-tuning, while IRIS strictly treats discrete Atari actions as tokens.
4. **Off-policy MBRL implementation**: iVideoGPT uses an off-policy RL algorithm while IRIS performs on-policy learning. On-policy learning needs a large number of model rollouts, which, combined with inefficient tokenization, results in ~7 days for 100k-environment-step training. In comparison, iVideoGPT only need takes ~4 hours.

We will include this extended discussion in a future revision.

### Q4: Visual planning performance

We note that **no current model in the VP2 benchmark consistently outperforms other models across all tasks**, and iVideoGPT is no exception. We conducted these visual planning experiments, aiming to show that our model can effectively handle various settings with competitive performance.

When analyzing the inconsistent performance on this benchmark, we primarily attribute it to **imperfect built-in reward designs of VP2**. In this benchmark, scores for sampled actions are mainly estimated by a learned classifier of task success based on model-predicted frames. This classifier, trained by VP2 authors, appears to lack robustness and can be easily fooled by out-of-distribution (OOD) inputs, assigning high rewards to trajectories that are less likely to succeed (see examples in Fig 3 of attached PDF). This imperfect reward function likely contributes to the mixed results observed on this benchmark (with iVideoGPT even outperforming the oracle simulator in one task). Addressing visual planning with imperfect rewards is another independent research problem, out of this paper's scope. We also find our model can produce incorrect predictions for severely OOD actions, likely due to narrow training data. We will include this discussion in the limitation section.

Meta-review: The paper "iVideoGPT: Interactive VideoGPTs are Scalable World Models" presents an approach to world modeling in reinforcement learning using interactive and scalable VideoGPTs. The authors propose a framework where a Transformer-based architecture is pre-trained on large-scale video data and then adapted to various downstream tasks, demonstrating competitive performance, few-shot adaptation, and zero-shot generalization. The reviewers agree that the paper makes a contribution to the field and recommend its acceptance.

The key strengths of the paper lie in its innovative combination of interactivity and scalability in world modeling. By leveraging the advantages of both reinforcement learning world models and advanced video generation models, iVideoGPT bridges the gap between these two domains. The authors introduce a compressive tokenization method that exploits temporal redundancy, which enables the model to scale efficiently while maintaining the ability to generate high-quality videos.

The reviewers raised several important questions and concerns, which the authors addressed comprehensively in their rebuttal. They clarified the motivation behind iVideoGPT, which highlights its potential to benefit from large-scale pre-training and its advantages over existing architectures like RNNs. The authors provided additional experiments and analyses to support their claims, including the efficiency of their tokenization method and the contribution of human videos to pre-training. The authors also addressed the differences between iVideoGPT and related works, such as IRIS and Dreamer, which are two important previous works. They highlighted iVideoGPT's unique features, including its pre-training and fine-tuning paradigm, efficient tokenization, flexible action-conditioning design, and off-policy MBRL implementation. These differences contribute to iVideoGPT's superior performance and sample efficiency compared to existing methods.

While the reviewers acknowledged that iVideoGPT does not significantly outperform state-of-the-art methods on all tasks, they recognized the value of the proposed framework as a step towards large-scale world models. The authors provided additional analyses to explain the mixed results in the visual planning benchmark, attributing them to imperfect built-in reward designs. The authors also responded to concerns about the technical novelty of iVideoGPT. They clarified the differences between iVideoGPT and related works, such as MaskViT, and provided evidence for the effectiveness of their approach in terms of computational efficiency and performance. The reviewers' concerns have been adequately addressed in the rebuttal, and the authors have shown a commitment to improving the paper based on the feedback received.  

As a result, I recommend accepting this paper.
Decision: Accept (poster)