iVideoGPT: Interactive VideoGPTs are Scalable World Models

Authors: jialong  wu
shaofeng  yin
ningya  feng
xu  he
dong  li
jianye  hao
mingsheng  long
"  𝑎 #$!
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-30 08:42:38

Abstract: World models empower model-based agents to interactively explore, reason, and plan within imagined environments for real-world decision-making. However, the high demand for interactivity poses challenges in harnessing recent advancements in video generative models for developing world models at scale. This work introduces Interactive VideoGPT (iVideoGPT), a scalable autoregressive transformer framework that integrates multimodal signals—visual observations, actions, and rewards—into a sequence of tokens, facilitating an interactive experience of agents via next-token prediction. iVideoGPT features a novel compressive tokenization technique that efficiently discretizes high-dimensional visual observations. Leveraging its scalable architecture, we are able to pre-train iVideoGPT on millions of human and robotic manipulation trajectories, establishing a versatile foundation that is adaptable to serve as interactive world models for a wide range of downstream tasks. These include action-conditioned video prediction, visual planning, and model-based reinforcement learning, where iVideoGPT achieves competitive performance compared with state-of-the-art methods. Our work advances the development of interactive general world models, bridging the gap between generative video models and practical model-based reinforcement learning applications. Code and pre-trained models are available at https://thuml.github.io/iVideoGPT.

Main paper: 
 introduction: 
 Recent years have witnessed remarkable advancements in generative models of multimodal contents, including text [1] , audio [9] , and images [22] , with video generation now emerging as a new frontier [11] . A particularly significant application of these generative video models, learned in an unsupervised way on diverse Internet-scale data, is to construct predictive world models [53, 28] at scale. These world models are expected to accumulate commonsense knowledge about how the world works, enabling the prediction of potential future outcomes (e.g., visual observations and reward signals) based on the actions of agents. By leveraging these world models, agents employing model-based reinforcement learning (RL) can imagine, reason, and plan inside world models [20, 29] , thus acquiring new skills more safely and efficiently with a handful of trials in the real world.Despite the fundamental connection, significant gaps remain between generative models for video generation and visual world models for agent learning. One primary challenge is achieving the best of both interactivity and scalability. In model-based RL, world models predominantly utilize recurrent network architecture. This design naturally allows the transition of observations or latent states conditioned on actions in each step, facilitating interactive behavior learning [29, 80, 34] . However, these recurrent models mostly focus on games or simulated environments with simple visuals and have limited capability to model complex, in-the-wild data at scale [48, 81] . On the other hand, Internet-scale video generative models [37, 7, 11] can synthesize realistic long videos that are controllable via text descriptions [109] or future action sequences [101] at the beginning of generation. Although suitable for high-level planning [19] , their trajectory-level interactivity does not provide sufficient granularity needed by agents to intervene step-by-step during the simulation to learn precise basic skills efficiently. This dilemma naturally raises the question:How can we leverage the advancements in scalable video generative models for developing interactive visual world models?In this work, we explore world models that are both interactive and scalable within a GPT-like autoregressive transformer framework [90, 75] . Pioneering efforts have been made recently through diffusion models [102] and masked generative models [12] . Nevertheless, utilizing autoregressive transformers offers distinct advantages such as seamless integration with the established Large Language Model (LLM) ecosystem [110] and greater flexibility in handling diverse conditions without the need for specific architectural modifications like adapter modules [77, 107] . We present Interactive VideoGPT (iVideoGPT), a scalable world model architecture that incorporates multimodal signals, including visual observations, actions, and rewards, in an interactively autoregressive manner. Unlike multimodal LLMs that discretize visual observations into tokens frame-by-frame using image tokenizers [55] , a key innovation of iVideoGPT for enhancing scalability is to learn compressive tokenization for each observation conditioned on rich contextual observations, achieving an asymptotic 16× reduction in token sequence length. We highlight that more compact video tokenization could not only facilitate more efficient training and generation but also enhance video quality. This is achieved by decoupling context from dynamics, allowing the model to focus on predicting the motion of objects while maintaining temporal consistency within the scene [99] .We demonstrate a series of practical applications of iVideoGPT for visual robotic manipulation, as illustrated in Figure 1 . Mirroring the two-phase approach popularized by LLMs, our method involves pre-training followed by domain-specific adaptation. During pre-training, iVideoGPT is scalable for action-free video prediction across a mixture of over one million robotic and human manipulation trajectories [70, 25] . The pre-trained iVideoGPT serves as a single, adaptable foundation of interactive world models for various downstream tasks, such as action-conditioned video prediction [21, 16] , visual planning [86] , and visual model-based RL [105] . Additionally, we showcase the pre-trained transformer's preliminary zero-shot video generation capability without fine-tuning, requiring only tokenizer adaptation for unseen domains. We further explore a variant of iVideoGPT for goal-conditioned video prediction, underscoring the flexibility of sequence modeling.The main contributions of this work can be summarized as follows:• We introduce Interactive VideoGPT (iVideoGPT), an autoregressive transformer architecture for scalable world models, which features compressive tokenization for visual observations. • We pre-train iVideoGPT on a large-scale dataset comprising millions of robotic and human manipulation trajectories and adapt it to domain-specific tasks. The pre-trained models have been publicly available to encourage further research. • Extensive experiments covering video prediction, visual planning, and visual model-based RL demonstrate that iVideoGPT can simulate accurate and realistic experiences and provide competitive performance compared with state-of-the-art methods. [29] and MuZero [80] provide step-level interactivity but limited scalability. (b) Recent video generation advancements like VideoGPT [101] and Stable Video Diffusion [8, 7] use non-causal temporal modules that can only offer trajectory-level interactivity. (c) Our model utilizes an autoregressive transformer that separately maps each step into a sequence of tokens, achieving both scalability and interactivity. 
problem formulation: 
 A world model is an internal model learned by the agent to simulate the environment. This environment is typically modeled as a partially observable Markov decision process (POMDP) (S, O, ϕ, A, p, r, γ). At each step, s t ∈ S represents the underlying state of the environment, and o t = ϕ(s t ) is the observation received by the agent, only providing incomplete information of s t .After taking an action a t ∈ A, p(s t+1 |s t , a t ) defines the transition probability from state s t to s t+1 . The agent also receives immediate rewards r t+1 = r(s t , a t ), and aim to learn a policy π such thata t ∼ π(o 1:t ) maximizing the γ-discounted accumulated rewards E p,π [ t γ t-1 r t ].While world models can be learned from many types of data, video is one modality that is taskagnostic, widely available, and embeds broad knowledge that can be learned in a self-supervised way. Thus, we formulate learning world models for visual control as an interactive video prediction problem [102, 12] where O = R H×W ×3 is the space of video frames2 . Concretely, given a short history visual observations of T 0 frames o 1:T0 , at each step t = T 0 , . . . , T -1, the agent takes an action a t based on its policy and previous imagined observations, and then the world model need to approximate and sample the transition p(o t+1 , r t+1 | o 1:t , a T0:t ) to feedback the agent.As depicted in Figure 2 , a majority of advanced video generation models [101, 8, 104] , including VideoGPT, can not deal with the interactive video prediction problem because they design non-causal modules fusing information along the temporal dimension, lacking the ability for causal, intermediate action control during generation (see extended discussion in Appendix C.2). Existing world models in the literature of MBRL [29, 80] , such as Dreamer, utilize recurrent architecture but lack scalability. 
interactive videogpt: 
 In this section, we introduce Interactive VideoGPT, a scalable world model architecture with great flexibility to integrate multimodal signals, including visual observations, actions, rewards, and other potential sensory inputs. At its core, iVideoGPT consists of a compressive tokenizer to discretize video frames and an autoregressive transformer predicting subsequent tokens (Section 3.1). This model can acquire common knowledge of motions and interactions in various scenes through pretraining on diverse human and robotic manipulation videos (Section 3.2) and then effectively transfer to downstream tasks incorporating additional modalities (Section 3.3).Compressive tokenization. Transformers particularly excel in operating over sequences of discrete tokens. VQGAN [22] is a commonly used visual tokenizer that converts from raw pixels to discrete tokens. Instead of using an image tokenizer to discretize each frame independently [55, 63, 27] , leading to rapidly increasing sequence lengths, or using a 3D tokenizer that compresses videos $ 𝑎 ! 𝑎 % 𝑜 $ 𝑜 ! 𝑜 % 𝑜 # [S] [S] [S]Reward headReward head spatiotemporally at the expense of interactivity [101, 104] , we propose to tokenize videos with a novel conditional VQGAN consisting of dual encoders and decoders {(E c , D c ), (E p , D p )}. As illustrated in Figure 3a , initial context frames o 1:T0 , rich in contextual information, are independently tokenized and reconstructed through N tokens: zEQUATION(1:N ) t = E c (o t ), ôt = D c (z t ) for t = 1, .. . , T 0 . In contrast, due to the temporal redundancy between context and future frames, only essential dynamics information, such as the position and pose of moving objects, needs to be encoded. This is achieved using a conditional encoder and decoder, which require a far smaller number of n tokens (n ≪ N ):EQUATIONWe implement this conditioning mechanism using cross-attention between multi-scale feature maps (see details in Appendix A.1). Overall, the proposed tokenizer is trained with the following objective:EQUATIONwhere L VQGAN (o; E, D) is a combination of a L 1 reconstruction loss, a commitment loss [89] , a perceptual loss [44] , and optionally an adversarial loss [22] .There are primarily two benefits of the proposed tokenization. First, it significantly reduces the sequence length of tokenized videos, which grows linearly with the number of frames but at a much smaller rate n. In this work, we set N = 16 × 16 and n = 4 × 4, resulting in an asymptotic reduction of 16×, facilitating faster rollouts for model-based planning and reinforcement learning. Second, by conditional encoding, transformers predicting subsequent tokens can maintain temporal consistency of the context much easier and focus on modeling essential dynamics information [99] . We discuss the assumptions and limitations of our tokenization in Section 6.Interactive prediction with Transformers. After tokenization, the video is flattened into a sequence of tokens: x = (zEQUATION2 , . . . , zEQUATIONT0+1 , . . . , zT0+1 , . . . ) with a length of L = (N + 1)T 0 + (n + 1)(T -T 0 ) -1. Special slot tokens [S] are inserted to delineate frame boundaries and facilitate the integration of extra low-dimensional modalities such as actions (see Section 3.3 for details). As Figure 3b , a GPT-like autoregressive transformer is utilized for interactive video prediction through next-token generation frame-by-frame. In this work, we take the model size of GPT-2 [76] but adopt the LLaMA architecture [87] in order to embrace the latest innovations for LLM architecture, applying pre-normalization using RMSNorm [106] , SwiGLU activation function [83] , and rotary positional embeddings [85] .Large language models can gain extensive knowledge from Internet text in a self-supervised way via next-word prediction. Similarly, the action-free video pre-training paradigm for world models [81, 99, 62] involves video prediction as a pre-training objective, providing Internet-scale supervision with physical world knowledge absent in LLMs. We pre-train iVideoGPT on this generic objective, applying a cross-entropy loss to predict subsequent video tokens:EQUATIONwhere L is the total sequence length and (N + 1)T 0 + 1 marks the first token index of the frames to be predicted. Notably, we do not train iVideoGPT to generate context frames, making its capacity focus on dynamics information, as previously discussed.Pre-training data. While there are numerous videos available on the Internet, due to computational limitations, we specifically pre-train iVideoGPT for the robotic manipulation domain. We leverage a mixture of 35 datasets from the Open X-Embodiment (OXE) dataset [70] and the Something-Something v2 (SSv2) dataset [25] , totaling 1.4 million trajectories (see Appendix A.2 for details). OXE is a diverse collection of robot learning datasets from a variety of robot embodiments, scenes, and tasks. These datasets are highly heterogeneous but can be easily unified in the action-free video prediction task. To further enhance the diversity, we also include SSv2, a dataset of humanobject interaction videos, as previous work has demonstrated knowledge transfer from these human manipulation videos for learning a world model for robotic manipulation tasks [99, 62] .Flexibility of sequence modeling. A sequence of tokens provides a flexible way to specify tasks, inputs, and outputs [60, 76] . To preliminarily showcase this flexibility, we introduce a variant of iVideoGPT for goal-conditioned video prediction: p(o 4 , with further exploration left for future work 3 .Action conditioning & reward prediction. Our architecture is also designed to flexibly incorporate additional modalities for learning interactive world models, as illustrated in Figure 3b . Actions are integrated by linear projection and adding to the slot token embeddings. For reward prediction, instead of learning independent reward predictors, we add a linear head to the last token's hidden state of each observation. This multi-task learning approach can enhance the model's focus on task-relevant information, thereby improving prediction accuracy for control tasks [57] . We use a mean-squared error loss for reward prediction in addition to the cross-entropy loss in Eq. (3).Tokenizer adaptation. We choose to update the full model, including the tokenizer, for downstream tasks, finding this strategy more effective than parameter-efficient fine-tuning methods [39] . This is likely due to the limited diversity of our pre-trained data compared to Internet-scale images, which, while extensive, may also not adequately cover specific real-world applications like robotics. Minimal literature explores adapting a VQGAN tokenizer to domain-specific data. As our tokenization is designed for decoupling dynamics information from context conditions, we hypothesize that while our model may encounter unseen objects like different robot types in downstream tasks, the fundamental knowledge of physics-such as motions and interactions-learned by the transformer from diverse scenes is commonly shared. This hypothesis is supported by our experiments transferring iVideoGPT from mixed pre-training data to the unseen BAIR dataset [21] , where the pre-trained transformer can zero-shot generalize to predict natural motions, requiring only the tokenizer to be fine-tuned for unseen robot grippers (see Figure 8 ). This property is particularly important for scaling GPTlike transformers to large sizes, enabling lightweight alignment across domains while keeping the transformer intact. We leave an in-depth analysis of tokenizer adaptation for future work.  
experiments: 
 In this section, we evaluate iVideoGPT in three different control-relevant settings and compare its performance with prior state-of-the-art methods. We demonstrate that iVideoGPT is versatile to provide competitive performance across a range of tasks (Section 4.1, 4.2, and 4.3) and conduct in-depth analysis to understand the tokenization and prediction ability, data efficiency, model scaling, and computational efficiency (Section 4.4). Experimental details can be found in Appendix A.Setup. The BAIR robot pushing dataset [21] consists of 43k training and 256 test videos, where we predict 15 frames from a single initial frame, a standard protocol of prior works. The RoboNet dataset [16] contains 162k videos across 7 robotic arms. Following prior works, we use 256 videos for testing, predicting 10 frames from two frames. Notably, RoboNet overlaps with our pre-training data OXE, from which we have carefully filtered test videos. We compare against a variety of video prediction models, including variational [91, 98, 4] , diffusion [93] , masked [104, 27] , and autoregressive models [101] , across four metrics: FVD [88] , PSNR [40] , SSIM [97] , and LPIPS [108] .Results. As shown in Table 1 , iVideoGPT provides competitive performance compared to stateof-the-art methods, MAGVIT [104] for BAIR and FitVid [4] for RoboNet, while achieving both interactivity and scalability in its architecture. Initially pre-trained action-free, our model flexibly allows for action-conditioning, which notably improves FVD for BAIR by almost 20%. Although primary experiments are at a low resolution of 64×64, iVideoGPT can be easily extended to 256×256 for RoboNet. We highlight that MaskViT, a prior method leveraging per-frame tokenization, suffers from temporal inconsistency and flicker artifacts in VQGAN reconstructions. Our model, which employs compressive tokenization conditioned on consistent contextual information, improves this and significantly outperforms MaskViT. For qualitative results, refer to Figure 4 .Setup. VP 2 is a control-centric benchmark [86] that evaluates video prediction models for visual model-predictive control (MPC) [24, 20] across four Robosuite [117] and seven RoboDesk tasks [47] .Table 1 : Video prediction results on the BAIR robot pushing and RoboNet datasets. We report the mean and standard deviation for each metric calculated over three runs. "-" marks that the value is not reported in the original papers. LPIPS and SSIM scores are scaled by 100 for convenient display. Each environment's training dataset includes noisy scripted interaction trajectories. Following the protocol from the original benchmark paper, we trained iVideoGPT on 5k trajectories for Robosuite and 35k for RoboDesk, comparing our models with established baselines.Results. Figure 5 presents the success rates of iVideoGPT compared to baseline models. While Tian et al. [86] observed that excellent perceptual metrics do not always correlate with effective control performance, iVideoGPT outperforms all baselines in two RoboDesk tasks with a large margin and achieves comparable average performance to the strongest model, SVG ′ [91] . In Appendix C.3, we analyze iVideoGPT's suboptimal performance on the open slide task, which is attributed to both limitations of discretization in our model and imperfect built-in reward design of the benchmark. Setup. We conduct experiments on six robotic manipulation tasks of varying difficulty from Meta-World [105] . Leveraging iVideoGPT as interactive world models, we have developed a model-based RL method adapted from MBPO [42] , which augments the replay buffer with synthetic rollouts to train a standard actor-critic RL algorithm (see Appendix A.5 for the pseudo-code). Our implementation builds upon DrQ-v2 [103] , a state-of-the-art visual model-free RL method. We also compare against a state-of-the-art model-based RL algorithm, DreamerV3 [32] , with and without world model pre-training [81] . Results. Figure 7 shows that our model-based algorithm not only remarkably improves the sample efficiency over its model-free counterpart but also matches or exceeds the performance of DreamerV3.To our knowledge, this reports the first successful application of MBPO to visual continuous control tasks. These results highlight the opportunity, with powerful world models, to eliminate the need for latent imagination-a common strategy used in advanced MBRL systems to train policies on rollouts of latent states within world models [29, 80] (see comparison in Figure 6 ). Our development of performant MBRL algorithms decouples model and policy learning, where iVideoGPT simply serves as a drop-in replacement of the environment. This can substantially simplify the design space, thereby greatly enhancing the practicality and effectiveness of MBRL algorithms in real-world applications.Comparison to recurrent world models. We argue that recurrent world models lack the capacity for large-scale pre-training on real-world data-a crucial capability for modern foundation models.To validate this, we pre-train DreamerV3 XL (200M parameters, comparable to iVideoGPT) on the same dataset. As shown in Figure 11 in the Appendix, DreamerV3 fails to capture natural robot dynamics, yielding low-quality, blurred predictions. Further evaluation on the Meta-World benchmark in Figure 7 reveals that DreamerV3 cannot benefit from such ineffective pre-training.Zero-shot prediction. We first analyze the zero-shot video prediction ability of large-scale pretrained iVideoGPT on the unseen BAIR dataset. Interestingly, we observe in the second row of Figure 8 that iVideoGPT, without fine-tuning, predicts a natural movement of a robot gripper-albeit a different one from our pre-training dataset. This indicates that while, due to insufficient diversity of pre-training data, our model has a limited ability of zero-shot generalization to completely unseen robots, it effectively separates scene context from motion dynamics. In contrast, with an adapted tokenizer, the transformer that is not fine-tuned itself successfully transfers the pre-trained knowledge and predicts movements for the new robot type in the third row, providing a similar perceptual quality as the fully fine-tuned transformer in the fourth row. Quantitative results can be found in Figure 9a .Few-shot adaptation. Large-scale pre-trained models have proven effective, especially in datascarce scenarios. Figure 9a shows iVideoGPT's performance when fine-tuned with various sizes of action-free BAIR trajectories. We observe that pre-training offers minimal benefits when full downstream data is available, yet the advantages become significant under data scarcity (with 100 or 1,000 trajectories). We also adapt iVideoGPT using 1,000 action-conditioned BAIR trajectories, achieving an FVD of 82.3. The fast adaptation ability with a handful of data is particularly crucial in model-based RL. As shown in Figure 7 , world models trained from scratch may generate inaccurate predictions, thereby degenerating the sample efficiency that is vital for model-based agents.Model scaling. All previous experiments are conducted using an iVideoGPT with 12 transformer layers and 768-dimensional hidden states (138M parameters). To initially investigate the scaling behavior of our model, we trained a larger iVideoGPT with 24 layers and 1024-dimensional hidden states (436M parameters). Figure 9b illustrates the validation loss curves on the pre-trained dataset. It shows that (1) the validation loss (perplexity) continues to decrease regardless of model size, and(2) increasing the model size accelerates the loss decrease. These results align with our expectation that larger model sizes and increased computation [48] can build more powerful iVideoGPTs.Tokenization efficiency. We evaluate the effectiveness of our compressive tokenization by comparing it against standard VQGAN tokenizers that independently convert each frame into 16 × 16 and 4 × 4 tokens. We train three tokenizers from scratch on RoboNet for the same number of steps. As Figure 9c , the tokenizer with 4 × 4 tokens suffers from low reconstruction quality due to its insufficient capacity. Our proposed tokenization method slightly compromises reconstruction quality compared to the standard 16 × 16 tokenizer but can provide more consistent contextual information, which is beneficial for video prediction tasks. More importantly, it significantly enhances computational efficiency with a significantly fewer amount of tokens, which greatly saves time and memory, allowing us to scale the model size with fewer costs (see quantitative results in Appendix B.5).Context-dynamics decoupling. Our tokenizer is designed with a bottleneck of much fewer tokens, focusing only on capturing necessary dynamics information for future frames while sharing contextual information with initial frames to reconstruct raw pixels. To explicitly visualize this decoupling of context and dynamics information, we drop cross-attention blocks to context frames in the decoder when reconstructing future frames. The results in Figure 10 show that the decoder can still reproduce the movement trajectories accurately but with minimal contextual information. This visualization supports the explanation of our model's generalization capability shown in Figure 8 .Goal-conditioned prediction. In Figure 4 , we also showcase video prediction generated by goalconditioned iVideoGPT, pre-trained on massive human and robotic videos (Section 3.2). Unlike action-free prediction, which often results in trajectories diverging from the ground truth, the goalconditioned model produces more accurate paths to reach specified goals. We believe this highlights the potential of leveraging the flexibility of a unified sequence modeling paradigm.  
related work: 
 World models for visual control. Learning general world models in visual domains remains a significant challenge in model-based reinforcement learning. A straightforward method involves learning action-conditioned video prediction models [69, 45] . Advanced model-based RL algorithms [29, 31, 32, 80, 34, 33] utilize latent imagination for more efficient and accurate rollouts but complicate themselves by tightly coupling model and policy learning. We show that this complexity can be reduced with powerful world models that have accumulated generalizable knowledge beyond specific tasks. Recent efforts to facilitate this include leveraging scalable architectures like transformers [63] and pre-training from large-scale data [99, 62] . Of particular relevance to our work are UniSim [102] and Genie [12] , which have developed extensively trained world models with diffusion and masked models, respectively, though neither is publicly available. Our work distinguishes itself by utilizing a generic autoregressive transformer framework, advancing the flexibility of scalable world models.Video generation and prediction. Recent developments in Internet-scale video generation models now enable the synthesis of realistic videos conditioned on class labels, text descriptions, and initial frames-the last one also known as the video prediction problem. Various models have been developed, including deterministic RNNs [84, 96] , variational autoencoders [18, 3, 30, 4] , diffusion [38, 11] , masked [104, 27] , and autoregressive models [101, 50, 55] . However, most recent works do not treat video prediction as a dynamics modeling problem and perform spatiotemporal compression [101, 8] , thus providing limited interactivity to serve as world models. We achieve both compressive tokenization and interactivity by context-aware representation, employing cross-attention mechanisms with minimal inductive bias. This method diverges from previous techniques that rely on motion vectors [43] or optical flows [52] and offers a more generic form of video tokenization. 
discussion: 
 We introduced Interactive VideoGPT (iVideoGPT), a generic and efficient world model architecture that leverages a scalable autoregressive transformer to integrate multimodal signals into a sequence of tokens, providing an interactive agent experience via next-token prediction. iVideoGPT has been pre-trained on millions of human and robotic manipulation trajectories and adapted to a wide range of downstream tasks. As a powerful foundation of world models, it enables accurate and generalizable video prediction as well as simplified yet performant model-based planning or reinforcement learning.Limitations and future work. While iVideoGPT marks significant progress, there is substantial room for improvement. We found limited diversity in publicly available robotic data, including the large-scale Open X-Embodiment dataset, and initiated efforts to transfer knowledge from human videos [25] . We believe iVideoGPT should be pre-trained on more extensive data [26] to bridge knowledge between humans and robots. This also requires iVideoGPT to incorporate more modalities, such as multi-view observations, proprioceptive robot states, and actions, within the unified formulation beyond action-free video prediction. Specifically, to process high-dimensional visual observations, our compressive tokenization assumes that initial frames provide sufficient contexts for future frames, which works for low-level control tasks as model-based agents often foresee tens of steps, but may falter in scenarios with long videos and significant camera motion. This issue can be mitigated by keyframe extraction [51] but leaves an important future avenue of exploration. Finally, extending to more complex real-robot tasks is essential, as the benefits of model scaling to even larger sizes remain unobserved in this work within visually simple simulation for downstream control tasks.The main hyperparameters of our experiment are detailed in Tables 2, 3, and 5. In this section, we provide a comprehensive explanation of all experimental details.A.1 Architecture Tokenizier. As illustrated in Figure 3 , we use a conditional VQGAN for compressive tokenization. This comprises two encoder-decoder pairs: (E c , D c ) for context frames (referred to as the context encoder-decoder) and (E p , D p ) for future frames (referred to as the prediction encoder-decoder).Both pairs share the same architecture (detailed in Table 2 ), but the prediction encoder-decoder has a tighter bottleneck, focusing solely on encoding dynamic information. Specifically, it uses a 4 × 4 convolution to downsample 16 × 16 embeddings into 4 × 4 before looking up the codebook. Consequently, the prediction encoder-decoder needs to be conditioned on the features of the context encoder-decoder to incorporate rich contextual information. This conditioning is implemented via a multi-scale cross-attention mechanism, similar to ContextWM [99] .The intuition behind the multi-scale cross-attention across feature maps is as follows: the context encoder extracts contextual features at varying levels of abstraction, while the prediction encoder uses cross-attention to adaptively filter out contextual information and distill dynamics information. During decoding, the prediction decoder blocks employ cross-attention to retrieve contextual information at corresponding levels, facilitating the gradual reconstruction of the scene. This framework enhances the model's ability to understand and manipulate complex scenes by focusing on dynamic changes, rather than being overwhelmed by irrelevant visual details.Specifically, at the end of each encoder block, let F l c ∈ R c×h×w be the feature map of a context frame, and F l p ∈ R c×h×w be the feature map of a future frame. Before being processed by the next independently processed by the context encoder and decoder, and their feature maps are concatenated to serve as inputs for cross-attention in the prediction encoder and decoder.Our VQGAN for 256 × 256 resolution is initialized from the pretrained model from aMUSEd4 [73] . We do not use discriminators for 64 × 64 resolution, effectively converting the VQGAN into a vanilla VQVAE with an additional perceptual loss.Transformer. We flatten a video into a sequence of tokens:EQUATION1 , . . . , zEQUATION2 , . . . , zEQUATIONT0+1 , . . . , zEQUATIONwhere we use two types of slot tokens [S1] and [S2] before the start of context frames and future frames, respectively. Context and future frames do not share token IDs, resulting in a transformer vocabulary of 16,386 tokens: the first 8,192 for context frames, the next 8,192 for future frames, and the last two for slot tokens. We adopt the autoregressive transformer architecture from LLaMA [87] , but instantiate it to smaller models matching the size of GPT-2. We considered two model sizes, listed in Figure 2 . Most of our experiments utilize a 138M parameter transformer, while preliminary scaling analysis is conducted using a 436M model.Data mixture. We pre-train iVideoGPT using 35 datasets from the Open X-Embodiment Dataset (OXE) [70] and Something-Something-v2 (SSv2) [25] . To construct our training dataset from OXE, we implement a filtering and weighting process similar to Octo [67] . Initially, we exclude datasets lacking image streams and those derived from mobile robots. Subsequently, datasets exhibiting excessive repetition or possessing low image resolutions are eliminated. The remaining datasets were categorized as either "large" or "small," and each was assigned a weight based on its size and diversity. We select 1% of samples from each subset as validation data and use the rest for training. For SSv2, we manually select 95 classes with clear motion trends from the original 174 video classes as our pre-training data with a weighting of 15%. We use the official splits of SSv2 for training and validation. For a comprehensive breakdown of the mixture, refer to Table 4 .During training, we sample sequences of frames by first randomly selecting a training video and then uniformly sampling a segment of a specified length and step size, i.e., neighboring frames in the segment are spaced a certain number of steps apart in the original video. We observe that datasets are collected at different frequencies. To maintain consistency, we adjust sampling with varied step sizes, aligning each with its respective dataset frequency, as listed in Table 4 . For tokenizer training, the initial frames of the segment are used as context frames, and from the remaining frames, we randomly sample a subset as future frames to reduce memory requirements and increase batch size. For transformer training, we use the full segment of frames. The number of frames in minibatches for each dataset is detailed in Table 3 . We use a mixture of OXE and SSv2 for training the tokenizer to ensure visual diversity, while only OXE is used for training the transformer.For data augmentation, we apply random resized crop and color jitter, ensuring consistency across the sequence. During both tokenizer and transformer training, we blend different losses with equal weights. Unless specified otherwise, we follow the same implementation details when fine-tuning iVideoGPT on downstream tasks.Goal-conditioned prediction. To train a goal-conditioned variant of iVideoGPT on the same dataset, we first fine-tune the previously obtained tokenizer using two randomly sampled frames as context for 550k training steps. Then, we train a transformer from scratch with the rearranged frame segment õ1:T = (o T , o 1 , o 2 , . . . , o T -1 ) for 1 million steps. The architecture and training procedures remain consistent with the above setup.License. The Open X-Embodiment dataset follows the Apache license. RoboNet is licensed under Creative Commons Attribution 4.0, while BAIR follows Creative Commons BY 4.0. The Something-Something-V2 dataset is subject to the Data License Agreement. Take action in environment according to π ϕ ; add to D real 17: end for A.3 Video Prediction Evaluation metrics. We evaluate our model across four different metrics5 : Structural Similarity Index Measure (SSIM) [97] , Peak Signal-to-noise Ratio (PSNR) [40] , Learned Perceptual Image Patch Similarity (LPIPS) [108] and Fréchet Video Distance (FVD) [88] . Following prior works [3, 91, 4, 104] , we account for the stochastic nature of video prediction by sampling 100 future trajectories per test video and selecting the best one for the final PSNR, SSIM, and LPIPS scores. For FVD, we use all 100 samples.We use the official repository 6 to evaluate our model on the VP 2 benchmark. The reported baseline results are provided by the authors of the benchmark. For the Robosuite tasks, a cost below 0.05 is considered a success.Environments. Meta-world [105] , following MIT License, is a benchmark of 50 robotic manipulation tasks. We select six tasks for our experiments: Button Press Topdown Wall, Plate Slide, Hammer, Door Lock, Handle Pull Side, and Coffee Push. We set the maximum episode length to 200 environment steps with an action repeat of 2 and a frame stack of 3 across these tasks and adjust the number of training steps to match the varying difficulty levels. During experiments, we observed that high rewards do not consistently correlate with high success rates in the original Meta-world implementation. This discrepancy presents a challenge to the learning stability of agents. To address this, we introduce an additional bonus for task success r bonus = 10.0 alongside the original task reward r task :EQUATIONMoreover, Meta-world features hard-exploration tasks, resulting in significant variance in the learning curves, which poses challenges to the accurate evaluation of RL algorithm performance. To mitigate this issue, we initialize the replay buffer of all compared methods, with 5 successful demonstration trajectories for all tasks except Door Lock. This strategy, commonly used to accelerate reinforcement learning [36] , helps stabilize the training process and provides a more reliable evaluation. Implementation Details. We have developed a simple model-based RL algorithm using iVideoGPT as a world model within the MBPO [42] framework, with DrQ-v2 [103] as the base actor-critic RL algorithm. Please refer to Algorithm 1 for the pseudo-code. Our implementation is based on the official DrQ-v2 code 7 , using the same hyperparameters and architecture for actor-critic learning. Hyperparameters specific to model-based RL are listed in Table 5 . We use a symlog transformation [32] for reward prediction in iVideoGPT.Baselines. To compare our method with DreamerV3, which lacks native pre-training support, we use APV [81] -a method enabling action-free pre-training on DreamerV2-as a baseline, modified to incorporate DreamerV3 features. We pre-train this model on the same dataset as iVideoGPT. Figures 12, 13, 15, 16, 17, 18, and 19 . We also include an additional showcase of zero-shot predictions by the pre-trained transformer in iVideoGPT in Figure 14 , supplementing Figure 8 Numerical metrics like FVD don't always align with human-judged visual quality. To address this, we conduct a human user study on the prediction results of various models. Due to the lack of official pretrained models for most baselines, we are only able to compare iVideoGPT with VideoGPT [101] and MCVD [93] on the action-free BAIR dataset. We generate videos using these three models from the test set and ask users to label preferences between two randomly sampled videos based on the physical naturalness and feasibility of robot-object interactions. A total of 386 annotations are collected from 9 participants. The results in Figure 20 demonstrate that iVideoGPT is preferred by human annotators more.Quantitative results on the VP 2 benchmark are reported in Table 6 .Comparison to FitVid-based world models. Although FitVid [4] is originally designed for the video prediction task and has not been used as world models for MBRL, we have implemented a baseline using FitVid by replacing iVideoGPT in our implementation. To predict rewards, we add an MLP head on top of FitVid's latent states, parallel to the image decoder. As shown in Figure 21 , MBPO with iVideoGPT outperforms FitVid on 5 out of 6 tasks and performs comparably on the remaining one. We also qualitatively observe that FitVid's imagined trajectories are blurrier compared to ours, which hinders its ability to simulate real environments accurately and may hurt MBRL performance.We report training and inference time and memory usage with various tokenizers in Table 7 and 8, respectively. Our proposed compressive tokenization provides significant memory savings during training and faster rollouts during generation. We note that although we use a more complicated C.1 Differences with IRIS Discrete tokenization and autoregressive transformers are prevalent in contemporary deep learning due to their simplicity and generality. iVideoGPT generally shares this architecture with IRIS [63] , but possesses distinguishing features, summarized as follows:• Pre-training and fine-tuning paradigm: iVideoGPT is designed for a paradigm that involves pre-training on large-scale videos and fine-tuning on various downstream tasks. In contrast, IRIS focuses solely on MBRL with Transformer-based world models trained from scratch in the Atari domain. • Efficient tokenization: iVideoGPT proposes novel compressive tokenization to significantly reduce the number of tokens, saving time and memory (see Table 7 and 8 ), while IRIS uses per-frame tokenization. • Flexible action-conditioning design: iVideoGPT employs slot tokens with optional additive action embeddings to support both action-free pre-training and action-conditioned finetuning, while IRIS strictly treats discrete Atari actions as tokens. • Off-policy MBRL implementation: iVideoGPT uses an off-policy RL algorithm while IRIS performs on-policy learning. On-policy learning needs a large number of model rollouts, which, combined with inefficient tokenization, results in 7 days for 100k-environment-step training. In comparison, iVideoGPT only needs ∼4 hours.We elaborate on the the difference between the tokenizer in VideoGPT [101] and ours, and how they impact interactivity.VideoGPT uses a VQVAE for video that relies on a series of 3D convolutions to downsample across space and time. For example, it downsamples original pixels from 16 × 64 × 64 to discrete tokens of 8 × 32 × 32 or 4 × 16 × 16, depending on the downsampling ratio. The key issue is that this non-causal downsampling over the temporal dimension results in each token containing information from a window of frames. As a result, the entire video of a fixed length can only be reconstructed after VideoGPT generates all tokens. As shown in Figure 2 , VideoGPT only allows the input of future action sequences at the beginning of prediction, preventing an agent from interactively determining its actions based on predicted observations. In contrast, our tokenizer discretizes video frames separately, using a conditional mechanism to handle temporal redundancy, enabling frame-by-frame video generation and allowing for intermediate action intervention.Moreover, our tokenizer's novel design, with its cross-attention mechanism, is more efficient in handling temporal redundancy, converting videos into significantly fewer tokens (L = 511 with N = 256, n = 16, T = 16, T 0 = 1 as stated in Section 3.1). In contrast, VideoGPT finds that using a larger downsampling ratio than a token size 8 × 32 × 32, results in worse performance.Our model performs sub-optimally on the RoboDesk open slide task from the VP 2 benchmark. In this section, we investigate the underlying causes through case studies, attributing the performance issues to limitations in both our model and the benchmark.Inaccurate model prediction. Despite achieving excellent overall video prediction metrics, such as mean square error and perceptual loss, on the validation set for the open slide task, our model predicts wrong outcomes on a few trajectories. We visualize these trajectories in Figure 22 and find that while the observation is limited to 64 × 64 resolution, the task of opening the slide requires the model to capture subtle changes, particularly whether the robot's gripper has made contact with the slide handle. Actually, even humans struggle to discriminate this detail with low-resolution inputs. Due to this uncertainty, the model may incorrectly predict a sequence of imprecise actions as successful. This overconfidence [66] can be exacerbated in the process of model predictive control, which samples a large number of action candidates and selects the "best" one according to the model. Our analysis provides an explanation to the observation by Tian et al. [86] that overall excellent perceptual metrics do not always correlate with effective control performance, as the worst-case scenarios are critical in model-predictive control.Furthermore, we hypothesize that our two-stage architecture of tokenization and prediction can exacerbate the aforementioned uncertainty, as discrete tokenization inevitably results in some loss of information from the observations. This hypothesis is supported by the fact that end-to-end models like SVG ′ [91] and FitVid [4] perform significantly better than two-stage models, including ours and others like MaskViT [27] , which uses a visual tokenizer, and Struct-VRNN [64] , which employs a keypoint-based representation.We anticipate that training and evaluating our model at a higher resolution, such as 256 × 256, could mitigate these issues and enhance control performance. However, we currently conduct experiments at a lower resolution to ensure a fair comparison with other models.Imperfect built-in reward design. We observe that no current model in the VP 2 benchmark consistently outperforms other models across all tasks, and iVideoGPT is no exception. Beyond models' inaccuracies in prediction for severely out-of-distribution (OOD) actions, our analysis of this inconsistent performance also reveals flaws in the benchmark's built-in reward design.In VP 2 , scores for sampled actions are estimated mainly by a learned classifier that assesses task success based on model-predicted frames. This classifier, trained by the VP 2 authors, appears to lack robustness and is easily fooled by OOD inputs, assigning high rewards to low-quality or unlikely-tosucceed predicted trajectories (see examples in Figure 23 ). Such an imperfect reward function likely contributes to the mixed results observed on this benchmark, with iVideoGPT even outperforming the oracle simulator in one task. Addressing visual planning with imperfect rewards is an independent research problem and beyond the scope of this paper. We implement iVideoGPT in PyTorch, using the diffusers8 and transformers9 libraries. Our models are trained and evaluated on an A100 and RTX 3090 GPU cluster. Each experiment utilizes 4 GPUs in parallel, with 16 data loader workers per device. GPU days required for training are reported in Table 2 . Experiments at 64 × 64 resolution can be conducted with 24 GB of GPU memory per device, while 256 × 256 resolution requires 40 GB. The Open X-Embodiment dataset is particularly large, occupying about 5TB of disk space.World models advance the development of autonomous machine intelligence, particularly through the valuable visual insights offered by videos. However, their full potential remains untapped without scalable and interactive architectures capable of distilling vast amounts of commonsense knowledge from multimodal data. This paper, we believe, takes an important step by introducing a flexible framework with a specific focus on the robotic manipulation domain. Our results may pave the way for higher-quality world models applicable across diverse domains, enhancing performance in control tasks of embodied intelligence. Despite the benefits, designing and training these models is challenging, requiring substantial computational power and increasing the carbon footprint. Using underdeveloped, inaccurate world models for autonomous control could lead to risky actions and potential physical damage, which can be mitigated by developing uncertainty-aware models to prevent uncertain actions. Conversely, the advancement of these models could lead to job displacement in sectors relying on manual control tasks. Additionally, the underlying techniques for world models can be misused to generate synthetic videos that mimic real events or people. However, since our model is merely a research prototype, trained only with robotic and human manipulation data and relatively small in scale, we do not anticipate immediate negative societal impacts such as deepfakes or job displacement.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The data used in this paper are all publicly available, detailed in Appendix A.We have released our code at https://github.com/thuml/iVideoGPT.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: See Table 1 , Figure 5 , and Figure 7 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our models are only trained with robot and human manipulation data, without the risk of misuse, such as deepfakes of important events or figures.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cited all assets used in the paper. In Appendix A, we explicitly mentioned and respected the license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Due to this connection, we use the terms "video frame" and "visual observation" interchangeably.Unless otherwise specified, action-and goal-free video prediction is used as the default pre-training objective to obtain pre-trained models for all experiments.https://github.com/huggingface/amused under openrail++ licenseWe use public implementations of metrics: https://github.com/francois-rozet/piqa under MIT license for SSIM and PSNR, https://github.com/richzhang/PerceptualSimilarity under BSD-2-Clause license for LPIPS, and https://github.com/universome/stylegan-v under NVidia license for FVD.6 https://github.com/s-tian/vp2https://github.com/facebookresearch/drqv2 under MIT Licensehttps://github.com/huggingface/diffusers under Apache Licensehttps://github.com/huggingface/transformers under Apache License 
 
 Appendix:  


Decision: The paper "iVideoGPT: Interactive VideoGPTs are Scalable World Models" presents an approach to world modeling in reinforcement learning using interactive and scalable VideoGPTs. The authors propose a framework where a Transformer-based architecture is pre-trained on large-scale video data and then adapted to various downstream tasks, demonstrating competitive performance, few-shot adaptation, and zero-shot generalization. The reviewers agree that the paper makes a contribution to the field and recommend its acceptance.

The key strengths of the paper lie in its innovative combination of interactivity and scalability in world modeling. By leveraging the advantages of both reinforcement learning world models and advanced video generation models, iVideoGPT bridges the gap between these two domains. The authors introduce a compressive tokenization method that exploits temporal redundancy, which enables the model to scale efficiently while maintaining the ability to generate high-quality videos.

The reviewers raised several important questions and concerns, which the authors addressed comprehensively in their rebuttal. They clarified the motivation behind iVideoGPT, which highlights its potential to benefit from large-scale pre-training and its advantages over existing architectures like RNNs. The authors provided additional experiments and analyses to support their claims, including the efficiency of their tokenization method and the contribution of human videos to pre-training. The authors also addressed the differences between iVideoGPT and related works, such as IRIS and Dreamer, which are two important previous works. They highlighted iVideoGPT's unique features, including its pre-training and fine-tuning paradigm, efficient tokenization, flexible action-conditioning design, and off-policy MBRL implementation. These differences contribute to iVideoGPT's superior performance and sample efficiency compared to existing methods.

While the reviewers acknowledged that iVideoGPT does not significantly outperform state-of-the-art methods on all tasks, they recognized the value of the proposed framework as a step towards large-scale world models. The authors provided additional analyses to explain the mixed results in the visual planning benchmark, attributing them to imperfect built-in reward designs. The authors also responded to concerns about the technical novelty of iVideoGPT. They clarified the differences between iVideoGPT and related works, such as MaskViT, and provided evidence for the effectiveness of their approach in terms of computational efficiency and performance. The reviewers' concerns have been adequately addressed in the rebuttal, and the authors have shown a commitment to improving the paper based on the feedback received.  

As a result, I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: The authors present a paper that attempts to utilize actionless and action conditioned trajectories to learn a large scale interactive world model. This model is subsequently adapted for robot manipulation tasks. It is evaluated on video prediction, visual planning, model based RL. The model training is tested primarily on Metaworld in the embodied setup. The experiments demonstrate the generalization capabilities of the model.
strength_weakness: strength_weakness: 1. **Performance on Video prediction:** The performance for the visual planning is rather unsatisfactory. Do the authors have any intuition about why their method only outperforms the baselines in 2 of the setups?
2. **No analysis on amount of action data needed:** During the motivation of the method, the authors discuss coming up with methods that are able to learn from freely available videos. Robot data on the other hand is expensive. Thus, ablations on how much robot data is needed are necessary to understand the dependence of the model on robot specific data and whether or not the method is indeed benefitting from freely available human videos.
3. **Lack of robot experiments:** Does the model extend to any real robot setups? Currently, the only low level control robot experiments available are on 6 in which the model matches the performance in 3. 
4. **No human user studies:** the paper only uses numerically metrics like Frechet Video Distance to judge the quality of generation. This metric does not always align with quality. The results of such a study would help bolster the quality of the work.
5. **Some missing Baselines**: FitVid does indeed have an action conditioned model and this model could be used for all the MBRL experiments. This currently missing.
questions: questions: 1. Is the dreamer baseline in the paper trained on OpenX data also? If not then the comparison is not fair.
2. How do you ensure that the $z^(1:n)_t$ contain all the information about future frames and that there is no leakage between which tokens contain what information?
limitations: limitations: 1. The evaluation suite is currently very small should be expanded
2. Its unclear how much the human videos from Something-something-v2 help the method. If they do not help the method significantly, then this diminishes the contributions of the paper significantly as this would make it a study on the application of transformer based world modeling on large robot datasets like OXE. 

Inspite of these limitations and weaknesses, I think the work makes interesting contributions and I would inclined to increase my score if the authors are able to address my questions.
paper_summary: paper_summary: This paper introduces a new architecture for an action-conditioned video (and reward) prediction model.
First, a VQGAN converts context frames individually into tokens.
Second, a conditional VQGAN converts future frames individually into tokens, conditioned on the context frames (using intermediate representations from the context encoder).
The idea is that the future encoder can focus on encoding changes in the scene, since the remaining information was already extracted from the context, allowing for a lower number of tokens.
Then, an autoregressive transformer is trained to predict the next (future) token and is optionally conditioned on actions and can optionally predict the rewards in a reinforcement learning setting.
The model is pre-trained on action-free robotic manipulation videos and then fine-tuned and evaluated on video prediction, visual planning, and model-based reinforcement learning tasks.
strength_weakness: strength_weakness: - W1: The paper is missing quantitative comparisons of the computational efficiency. For instance, it would be interesting to see the differences in training/inference time when using the different tokenizers (Figure 8(c)).
      Furthermore, the authors state at the end of Section 2 that recurrent world models like Dreamer are not scalable, but I'm wondering how significant the difference is, considering the autoregressive next token prediction and use of multiple tokens per frame.
- W2: The authors argue that iVideoGPT eliminates the need for latent imagination in model-based RL (Section 4.3 and Figure 5).
      However, there are existing world models (e.g. IRIS [1]) that learn behaviors using reconstructed frames.
      Moreover, Dreamer could also learn behaviors this way by using the reconstructions from the decoder (but I understand that this would be a different method).
      In short, I don't think that this is a feature that is novel or specific to iVideoGPT.
- W3: The proposed model has a lot in common with IRIS [1], which learns a discrete autoencoder (VQVAE) and uses an autoregressive transformer for next token prediction.
      IRIS is briefly mentioned in the related work section, but I think the authors could emphasize the differences in more detail.

Some typos:
- 79: "aims"
- 80: "maximize"?
- 85: "history of $T_0$ video frames"?
- 86: "needs to"
- 107: $D_c(z_t)$ should be $D_c(z_t^{(1:N)})$?
- Also in Eq. (1) it should be $D_p(z_t^{(1:n)}|o_{1:T_0})$?
- 687: Table 2 -> Table 3

I am willing to increase my scores after the listed weaknesses have been addressed.
questions: questions: - Q1: Is the world model also conditioned on the actions from the context? In line 87 the condition only includes $a_{T_0:t}$, but in the sequence of tokens $x$, the context also includes slot tokens. This can also not be recognized in Figure 3(b).
limitations: limitations: The authors addressed all limitations adequately.
paper_summary: paper_summary: This paper introduces Interactive VideoGPT (iVideoGPT), which builds world model based on VideoGPT architecture. iVideoGPT proposes compressive tokenization, and the model is trained using millions of human or robot videos (i.e., Open X embodiment dataset). The effectiveness of iVideoGPT is demonstrated in video prediction, visual planning, and visual model-based reinforcement learning.
strength_weakness: strength_weakness: - Technical novelty is somewhat limited.
questions: questions: - What is main difference between iVideoGPT and  IRIS [1] for visual model-based reinforcement learning of visual planning?

[1] Micheli, V., Alonso, E., & Fleuret, F. (2022). Transformers are sample-efficient world models. arXiv preprint arXiv:2209.00588.
limitations: limitations: See Weakness & questions.
paper_summary: paper_summary: This work studies the setting of planning and prediction from video world models. It proposes to use a GPT-style transformer world model that incorporates action and reward information in it's context and prediction pipeline. The model is further equipped with a novel tokenization technique based on VQGAN. Finally, the model is evaluated on three downstream applications, namely video prediction, planning and model-based RL. Ablations on the models and tokenizers scalability are conducted.
strength_weakness: strength_weakness: Clarity
* In some cases, the writing makes very strong statements that are very broad and hard to be supported by evidence. E.g.
  * L97 "This model can acquire broad world knowledge". This statement is not supported and it is unclear to me what "broad world knowledge" means. I believe concise language about the capabilities of our algorithms is important.

Motivation
* One issue with the paper is that some of the motivation is not quite clear to me. The paper argues that the provided world model has two benefits: It is interactive and scalable. 
  * First, it is not clear to me what it means for this model to be interactive. If the condition for a model is to be action-conditioned, then the dreamer world-model should also be interactive. In fact, even much older world-models would be interactive [1, 2]. 
  * Second, it is not clear to me that other models would not be scalable and the paper provides no evidence that they are not scalable. The text states in line 91 that dreamer lacks scability, however not citation or evidence for this is provided. For instance, dreamer uses state-space models (SSMs) and recent advances have built scalable SSMs [1].

Experiments and Claims
* For the tokenizer, the text claims that it "decouples dynamics information from context conditions" L161. This seems like a strong statement that would warrant evidence to be supported. I believe qualitatively looking at a small number of image sequence is insufficient to support such a strong claim about composition.
* Overall, the experimental results seem a little weak. 
  * The model is outperformed on most tasks in the video-prediction setting. It is unclear why the one of the weakest baselines was used for the for action-conditioning comparison in section 4.1.
  * In the visual planning experiment, the performance exceeds baselines in only 2/7 tasks.
  * In the model-based RL experiments, the performance is mostly comparable to dreamer. There are cases where the paper argues that their suggested algorithm outperforms dreamer but given that the results are only reported over $5$ random trials, it seems that the results are within variance making this claim too vague. (It’s a little hard to tell because shaded regions are very transparent)
* The benefits of scalability for the downstream control tasks have not been demonstrated. 
  * The work argues that high perceptual metrics don't necessarily correlate with good control performance (App B.2) but the main benefit of scale seems to be improved performance on those metrics.
  * While the work shows in section 4.4 that scaling leads to lower validation loss, whether or not this is correlated with downstream performance is not demonstrated. I believe this point is important because we have to ask what is the marginal benefit of increasing computational complexity to train larger and larger models, if the application to downstream tasks does not benefit much from scale. Showing that larger networks and lower error lead to improved downstream performance is an experiment that might validate the need for scale.
* Similarly, the need for specialized tokenization has not been argued for downstream applications. It seems a little detached from the goal. The main goal of the paper is to argue for scalability and that this tokenization technique is helpful. However, for the downstream applications, neither the planning nor RL sections demonstrate the importance of scaling or tokenizing. One experiment I can think of is to demonstrate that 4x4 tokenization is insufficient due to reconstruction quality this claim. The relationship between scaling and the novel tokenization is not analyzed either.
Overall, I think the motivation of the work is not quite clear enough, the contributed algorithm provides minor improvements in the control settings and the claims for interactivity and scalability could be strengthened.
* The text claims that the tokenization that is presented is more efficient but there is no experiment validating that this is true. A complex encoder structure is used and it is unclear whether this is actually more efficient. An experiment to validate this could measure total wallclock times on the same hardware.

[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Albert Gu, Tri Dao. arXiv preprint arXiv:2312.00752.  
[2] Action-Conditional Video Prediction using Deep Networks in Atari Games. Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, Satinder Singh. Advances in Neural Information Processing Systems 28 (NIPS 2015).
questions: questions: Q1: What are the training times of your algorithm training and how do they compare to the runtimes of Dreamer?
Q2: Can you elaborate on the term interactive and why other models are not interactive?
Q3: What are the key differences between the proposed model and the MaskVIT model and to which of the difference do you attributed the improved action-conditioned prediction performance in section 4.1?
limitations: limitations: The paper contains an explicit section outlining limitations of the work.

Comments: title: Deep Appreciation for Your Constructive Feedback
Appreciation for Your Support
Response to Post-rebuttal Feedback by Reviewer 6ajb
Required Action: Please Respond to the Author Rebuttal
Required Action: Please Respond to the Author Rebuttal
Required Action: Please Respond to the Author Rebuttal
Required Action: Please Respond to the Author Rebuttal
comment: Dear Reviewer 6ajb,

We deeply appreciate your time and effort in providing such detailed and constructive feedback. Your recognition of our contributions is greatly encouraging. Your insights have been invaluable in helping us strengthen the clarity and focus of our paper. We are committed to incorporating your feedback and revising the relevant parts on motivation, tokenizers, and experiments to ensure the paper meets your expectations and reaches its full potential.

Thank you again for your support and for recommending our work for acceptance.

Best regards,

Authors
Dear authors, thank you for the follow-up response. I will use this comment to summarize my position now that I believe I have a better understanding of the manuscript.

**Motivation**  
First, I appreciate you rephrasing the hyperbolic language. It makes the claims sound much more precise and fosters clarity of what is provided. 

That being said, I think that the scalability and interactivity arguments are not fleshed our sufficiently. For instance, [1 mine] also have an interactive world model that is scalable which is probably worth citing. What differentiates the presented work here is that the world-model operates in pixel-observation space. This in itself is interesting because I am not aware of any work that has done this on this complexity. The second argument that the rebuttal makes is that **one** model for downstream fine-tuning is what is being provided. However, this point does currently not play a central role in the writing.

Overall, I think the paper would benefit strongly from re-focusing the main story on pixel-observation spaces and a single-model for downstream learning rather than the interactive aspect. I believe the interactive and scalable aspects are not necessarily what makes this work stand out (I personally would adjust the title to include pixel-spaces/observation spaces. Note that that does not mean the name of the model needs to change since interactive video model is still true.).  A single, pixel observation-based world model for easy downstream finetuning would fit the story of this work better.

**Tokenizer**
I appreciate all the clarifications on this. Now that I understand better, I believe the text again would benefit from re-writing, focusing on the need of this tokenizer to enable embedding actions seemlessly while also providing an efficient encoding similar to what is needed for video prediction in iVideoGPT rather than things like context-lengths. This would tie the introduction of the tokenizer much better to the objective of building a world model which is currently missing.

**Experiments**
Thank you for running the additional experiments. I think the experiment section of this paper is going to be quite extensive which is great. Overall, I believe my point about the strengths of the experiments remains. Yet, given my previous explanation we have to put them in a different light. A world model in pixel observation space that is equally good as latent space models is a decent contribution. Tying the language of the claims to the experiments is what remains.

One crucial addition are the additional dreamer experiments to strengthen the claim that previous models are not scalable. However, as I pointed out, Dreamer is not the model that claims to be scalable, but rather [1] does. I understand that there is no time in the rebuttal to consider a comparison here but I do think that comparison would make this paper a stronger submission. Since there is no time for this now, I will not include this into my recommendation to the author's disadvantage but rather provide this as feedback for the next iteration of the paper.

"However, as shown in Fig. 6 of the paper, data scaling does provide benefits---pretraining iVideoGPT enhances MBRL performance."  
Figure 6 only demonstrates that pretraining is useful which is something we have known from offline RL, not that we need a lot of fine-tuning data. There are no experiments with varying amounts of data for **down-stream** performance. In Figure 8b the paper demonstrates that validation loss of pre-training goes down with larger models but in the rebuttal it's stated that that has no impact on the down-stream performance on meta-world. Thus, I believe there is still a disconnect between analyzing the scalability (both model and data) and down-stream performance of the model.

Thank you also for clarifying training times, I underestimated the reduction in computational cost from your tokenizer.

**Summary**
Overall, I think this paper makes a decent contribution and others might benefit from it being published. However, I think there are various sections that I personally would recommend re-writing. If this was a journal paper, it would be an accept with revisions. If we had more time, I would recommend borderline acceptance and raise my score to 5. Unfortunately, we don't have this option here which puts us at an impasse. Thus I am going to err on the side of optimism and trust that the authors incorporate large parts of my feedback and I will recommend acceptance. Most of the issues that I have with the paper can be fixed by (in some parts minor) re-writing. I am changing my scores as follows:

Presentation: 3 -> 2  
Contribution: 2 -> 4  
Overall score: 3 -> 6  

[1] TD-MPC2: Scalable, Robust World Models for Continuous Control. Nicklas Hansen, Hao Su, Xiaolong Wang. ICLR 2024.
Dear Reviewer zSnp,

We sincerely appreciate your dedicated re-evaluation of our paper and the subsequent positive rating. Your feedback has significantly improved our work.

Best regards,

Authors
Thank you for the detailed response. I have updated my score accordingly.
Dear Reviewer 6ajb,

Thank you again for your time and effort in reviewing our paper. We appreciate your careful review of our rebuttal materials and your recognition of our efforts in addressing your initial concerns.
    
Regarding your follow-up question: What is the difference between the tokenizer in VideoGPT and ours, and how do they impact interactivity?

VideoGPT uses a VQVAE for video that relies on **a series of 3D convolutions to downsample across space and time**. For example, it downsamples original pixels from $16 \times 64 \times 64$ to discrete tokens of $8 \times 32 \times 32$ or $4 \times 16 \times 16$ (depending on the downsampling ratio). The key issue is that this non-causal downsampling over the temporal dimension results in each token containing information from a window of frames. As a result, **the entire video can only be reconstructed after VideoGPT generates all tokens**. As shown in Fig. 2b of our paper, **VideoGPT only allows the input of future action sequences at the beginning of prediction, preventing an agent from interactively determining its actions based on predicted observations**. In contrast, our tokenizer discretizes video frames separately, using a conditional mechanism to handle temporal redundancy, **enabling frame-by-frame video generation and allowing for intermediate action intervention**.

Moreover, our tokenizer’s novel design, with its cross-attention mechanism, is more efficient in handling temporal redundancy, converting videos into significantly fewer tokens ($L=511$ with $N=256, n=16, T=16, T_0=1$ as stated in Line 124). In contrast, VideoGPT finds that using a larger downsampling ratio than a token size $8 \times 32 \times 32$, results in worse performance.

We hope this response addresses your remaining concerns, and we remain open to further discussion. If our response is satisfactory, we kindly ask you to consider re-evaluating your rating of our work based on the clarified understanding.

Best regards,
    
Authors
Dear authors, thank you for the thorough response. My initial questions have been answered satisfactory. I also appreciate all the efforts you put into additional experiments.

I have one follow-up question since I'm not too familiar with vision tokenizers for video prediction.

Q5: In the paper, the text states "Instead of [...]  using a 3D tokenizer that compresses videos spatiotemporally at the expense of interactivity, we propose to tokenize videos with a novel conditional VQGAN"  
Could you elaborate on why we cannot simply use the VQVAE encoder from the original VideoGPT and why that loses interactivity? What exactly distinguishes this tokenizer from VQVAE?
Dear Reviewer HkK9,


As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.


Best regards,

Area Chair, NeurIPS 2024
Dear Reviewer zSnp,


As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.


Best regards,

Area Chair, NeurIPS 2024
Dear Reviewer 6ajb,

As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.

Best regards,

Area Chair, NeurIPS 2024
Dear Reviewer yh7R,


As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.


Best regards,

Area Chair, NeurIPS 2024