The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
Authors: Ruili Feng
Han Zhang
Zhilei Shu
Zhantao Yang
Longxiang Tang
Zhicai Wang
Andy Zheng
Jie Xiao
Zhiheng Liu
Ruihang Chu
Yukun Huang
Yu Liu
Hongyang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: World Models
Video Generation Models
Primary area: deep_learning
TL;DR: This paper introduces The Matrix, a foundational realistic world simulator capable of generating infinitely long 720p high-fidelity real-scene video streams with real-time, responsive control.

Abstract: We present The Matrix, a foundational realistic world simulator capable of generating infinitely long 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives. Trained on limited supervised data from video games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains—deserts, grasslands, water bodies, and urban landscapes—in continuous, uncut hour-long sequences. With speeds of up to 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting—an environment present in neither gaming data nor real-world sources. This approach showcases the potential of game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.

The Matrix: Real-Time Moving Control
World Generation with
Ruili Fengl. Han Zhang! Zhilei Shu Zhhntao Yangl. Longxiang Tang!.
Zhicai Wang',, Andy Zheng Jie Xiao Zhiheng Liu 15, Ruihang Chu', Yukun Huang*5, Yu Liu Hongyang Zhang
Tongyi Lab, Uniiversity of Hong Kong, UUiieersity of Waterloo, *Vector Insititute, "Matrix Team
Equal Contritation "Engnnerr Advince, Project Leader
Abstract
We present The Matrix, a foundational realistic world simulator capable of gen- erating infinitely long 720p high-fidelits real-scene video streams with real-time, responsive control in both first- and third-person perspectives. Trained on limited data from video games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, grasslands, water bodies, and urban landscapes-in continuous, uncut hour-long sequences. With speeds of up to 16 FPS, the system supports real-time interactivity and demon- strates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data often infeasible. For ex- ample, The Matrix can simulate BMW X3 driving through an office setting an environment present in neither gaming data nor real-world sources. This approach showcases the potential of game data to advance robust world models, bridg- ing the gap between simulations and real-world applications in scenarios with http://gituub.coo//Ma limited data. See https://itthb.. com/Matris Team-Al/matrix Al/matrix, https://matritteam-
The Matrix allows users to traverse diverse terrains-deste
ai www.ggitthhhppoee hattubbio/pages/TheMaris for code data and project page.
Figure 1: The Matrix is foundational realistic world simulator capable of generating infinitely long 720p high-fidelity real-scene video streams with real-time, precise moving control. Due to size limitation, we recommend you to to the website provided in the abstract for videos.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Neural-interactive simulation, a concept popularized by The Matrix (1999), envisions a world fully constructed by AI to replicate 20th-century human society. This paper takes an initial step toward realizing this vision by developing a world model that enables neural networks to 'dream' visually authentic environments. The result an simulation that supports wall-time (16 Frame Per Second. FPS) interactive exploration across diverse landscapes, including deserts, grasslands, water terrains, and urban settings. Responding to real-time control signals, the world model predicts future frames in these environments in a streaming and auto-regressive fashion.
high-resolution (1280x 720 pixels, 720p)
World models offer a promising solution to the overwhelming costs of video game development, which can easily run into tens or even hundreds of millions of dollars. Traditional game creation depends on engines such as Unity 3D, Unreal Engine, and Blender, each requiring substantial expertise, intensive asset preparation, and meticulous hyperparameter tuning. Furthermore, games built with these engines are often limited in reusability, as each new title demands comprehensive redesign. In contrast, data-driven world models tackle these issues by minimizing the need for manual configuration, simplifying development workflows, and boosting scalability across projects.
Despite extensive research in world models [1] key challenges remain. First, prior studies have predominantly focused on simpler or less realistic video games, such as Atari [ B 4. Mario [5], Minecraft [6] 7 []. CS:GO [4) and DOOM 9. which fall short in replicating real-world fidelity. Second, current video generation techniques, like Sora [10], are constrained to short sequences of about 1 minute, forcing existing world models assemble independently generated clips with noticeable transitions. Finally, achieving real-time generation remains a major hurdle. For example, sfut-oo--th-aas game generators such as Genie [2] operate at speeds of FPS, while Nvidia's Cosmos [13] is also incapable of generating real-time worlds. This paper addresses these limitations by introducing scalable, high-fidelity world model in real time that enhances simulation realism and bridges the gap between virtual environments and reality. Notably, our world model is with strong domain generalization and real-time control. For example, our foundation model allows us to control BMW X3 driving through an indoor setting or in the sea- an environment present in neither gaming data nor real-world sources.
Our Contributions. Our contributions are as follows:
We introduce The Matrix, foundational simulator for realistic environments that generates infinitely long, high-fidelity 720p real-world video streams with real-time interactive control and strong domain generalization. Despite its capabilities, the model is lightweight, with only 2.7 billion parameters, and achieves a generation speed of 16 FPS on 8xA100 GPUs.
At the core of The Matrix is novel diffusion technique, the Shift- Window Denoising Process Model (Swin-DPM), enabling pre- pre-trained DiT models [14] to extrapolate seamlessly for smooth, continuous, and infinitely extendable video creation. This technique holds potential for broader applications in long-form video generation.
Additionally, we introduce GameData, platform that autonomously captures paired in- game
from CPU memory-alongsitn corresponding vidco frames, sig- nificantly reducing labeling costs and complexity. This platform produces Source, new training dataset for world models with action-fram paired data.
Technical Advantages of The Matrix. Our work advances the state-of-the-aa in the following aspects: 1) Infinite Video Generation: The Matrix generates consistent, infinitely long videos using strcaming, auto-regressive approach. 2) High-Quality Rendering: The Matrix delivers realistic rendering at a resolution 1280 x 720. 3) Real-Time, Frame-Level Control: The Matrix operates in 16 FPS, providing real-time, frame-level control for interactive applications. 4) Domain Generalization: Trained with small amounts of supervised game data and large amounts of unsupervised internet videos, The Matrix achieves strong domain generalization to real- wolld settings.
While we do not claim The Matrix to be foundational world model-since it is currently trained only on Forza Horizon 5, Cyberpunk 2077, and humanoid control tasks-we regard it as proof of concept demonstrating that the aforementioned four aspects can be simultaneously achieved. With more diverse data, The Matrix has the potential to evolve into a generalized world model.

---

Video DIT Backbone
Shift window Denoise Process Model ISwin-DPMM
Time Axis
DTT Backhone
CTTL Noss
KITK Nati
ZUTT Note
TITE Neta
Interactive Module
Frome
X
Warm-up
DIT w/ Interactive Module denoloe Ste0
Backbone
Drop First T Tokens Warm-up
Aellens
Interdttve Madules
Abrrr To
Blocks
20TTE Notso
Nost
+ Stream Consistency Model
X
Non-real-tmee Genevation
Di my Interactivn Medale dononie Stop
SCM Model
Red- time G Ceneration
10 Sporcip
WeIn-DPM after First Tokens
Figure 2: The training process of The Matrix begins with pretrained video DiT backbone. First, the Interactive Module is warmed up using Synthesized Observations of Unreal Rendered Contextual Environments data with unsupervised LoRA to make subsequent training focus movement, not visuals. Then, we train the Interactive Module for precise frame-level control. Swin-DPM enables infinite-length generation, and Stream Consistency Model is introduced to accelerate sampling to real-time speeds.
Related Work
World Simulation. Distinct from world models designed for agent learning, world simulation focuses on human interaction with neural networks through high-quality rendering, robust control, and strong domain generalization real-world scenarios. This research explores two types of control: video-level and frame-level. In video-level control, control signal is given at the start, and the model generates a responsive video sequence; notable examples include UniSim [15], Pandora [16), GameGen-X 17, MicroVGG [5). GAIA-1 (8), and Cosmos (3). To approximate continuous control, this approach often stitches together independently generated clips, which may result in visible transitions. In contrast, frame-level control provides fine-graines adjustments every few frames, enabling more precise, responsive interactions similar to gameplay. as seen in examples like Genie [1I] Genie 2 [12 DIAMOND [4) GameNGen [9], MineWorld [8), and Oasis [6]. Prior work in world simulation has typically focused on one of three aspects
sspeect-videe -vddo length, high resolution, or domain wettrraiiiaiiowwwhhhhoe addressing all three simultaneousll. The Matrix uniquely stands as foundation model capable of generating infinitely long, high-quality videos with high resolution, frame-level real-time control, and strong generalization to real-world contexts.
Methods
Achieving granular control is notoriously challenging, as labeling actions the frame level is typically cost-prohibitive. To address this, we develop the GameDDaa platform, which autonomously captures paired data of in-game states (extracted directly from CPU memory) alongside corresponding video frames, significantly reducing labeling costs and complexity. Additionally, The Matrix incorporates an advanced Interactive Module that Iearns and generalizes game movement interactions from a limited amount of labeled data combined with extensive unlabeled data from both games and real-world environments. This enables The Matrix to deliver exceptional accuracy across diverse scenarios, while maintaining robust performance in the gaming domain.
Generating figh-quality, real-time, and generalizable video simulations for infinite sequences presents additional technical challenges, often forcing previous simulators to compromise on one or more essential aspects. The Matrix overcomes these limitations by adapting the world model from pre-trained video Diffusion Transformer (DiT) model [4), leveraging its extensive pre-existing knowledge and generation quality. To enable infinite-length generation, The Matrix introduces novel diffusion approach, the
Denoising Process Model (Swin- DDM), which allows

---

Control
Linsar
TimeAxs
Control Signals (kays))
Layer Norm
Video Tokens (quertes)
Causal Cross-A tta
DET Block
Interactive Module
with
Interactiee
2x
DiT Block
Module
x 16
Embed Bik
DfT Block
Embeedi
Video Toksens
Test Frebed
Embed Rilk
Patchify
Linear
3D-VAF
Yuvk
Layee
Croe.
Sell-Am
Lawer Num
Linser
T5-Enc
C)
Video
Video
K Keyboard
Prompt
Inpui
Figure 3: The Interactive Module: After every two DiT blocks, the module merges the keyboard inputs into the video token feature through a Causal Cross-Attention Layer, where each keyboard input is limited to influence only the current and subsequent tokens. Here, every p frames are condensed into a single token.
the DiT model extrapolate for smooth, continuous, and indefinitely long video creation. Finally, we fine-tune a Stream Consistency Model (SCM), accelerating inference to real-time.
Video DiT Backbone. As preliminary, introduce the video DiT backbone, adapted from the publicly available DiT models [19]. It employs a 3D Variational
(VAE) to encode T' x p video frames into 7 video tokens. The backbone consists of 32 attention blocks, followed by a linear output head with LayerNorm [20]. Each attention block includes a self-attention layer operating on network features, cross- aatentattntionn layer linking conditions with self-attention outputs, and an FFN layer composed of two linear layers with GELU activation [2] in between. See Appendix Section A./ for further details.
3.1 Model Components
The Matrix comprises three main components: a) an Interactive Module that interprets user inten- tions (e.g., keyboard inputs) and integrates them into video token generation; b) Shift-Wiis Window Denoising Process Model (Swin-DPMI that enables infinite-length video generation; and c) a Stream Consistency Model (SCM) that accelerates sampling achieve real-time performance. As shown in Figure the model is fine-tuned from a pre-trained video DiT model through a three-stage process: first, we fix the DiT model parameters and train the Interactive Module; next, we train the Interactive Module and the DiT together following the Swin-DPM; finally, we optimize an SCM to accelerate inference real time. The first two stages leverage both labeled gaming and unlabeled video data to enhance generalization, while the final SCM training focuses on labeled gaming data to reduce optimization complexity.
Interactive Module. The Interactive Module consists of an Embedding block (see Figure 3) and a cross-attention layer. Its primary function translate keyboard inputs into natural language that guides video generation. For example, pressing "W' is interpreted as "The car is driving forward" in the Forza Horizon 5 scenario, as "The man is moving forward and looking up" when combined with an upward mouse movement in Cyberpunk 2077. For unlabeled real or game data, we apply default description: "The camera moving in an unknown way." To enhance robustness, we randomly replace labeled keyboard inputs with this default sentence during training with probability q 0. To prepare for training, we first warmup the base DiT model for few epochs using collected game and real-world data, fine-tuning LoRA weight [2 This process ensures that the Interactive Module focuses on learning interactions and movement patterns rather than simply fitting the video. Once translated, these natural language descriptions are processed by T5 encoder (23 and transformed into a vector embedding through two linear layers and a SiLU layer 24 between them. This vector embedding is then concatenated with its corresponding video token and the next video tokens, where w is a pre-defined causal relation range, typically set to w 4, as shown in Figure 3 We perform this cross-attention operation each time the DiT model completes an odd num-neree

---

AimeAxis
Frame p
3p
Tp
+lp
O/T Nosse
kTB Noise
2k/Tk
TK/Tx Noise
Add New Naise
XT
Wi
Cache
Video Tokeuss (N, T, W,
Dequeue
Denoise
DIT w/ Interactive M eddule denorise Stepa
x
[11,
Cache
Video Tokeme (N, T. H.
Dequeue
+1+1-th
Stmmk
Denoise
Figure 4: Shift-Window Denoising Process Model: The Swin-DPM transforms the traditional diffusion process into streaming one, where T video tokens with different noise levels are denoised simultaneously. After each token is fully denoised and dequeued for decoding, new token of pure noise is added to the queue. The dequeued token is then copied to the cache, allowing it to continue participating in attention computations until the next token is dequeued.
self-attention step, enabling effective information exchange across frames and achieving precise, frame-level control for video generation.
Shift-Window Denoising Process Model. Typical DiT models are limited to generating only few seconds of video, even when substantial spatial and temporal compression is applied via VAEs. This limitation is largely due to the high computational cost and memory demands attention mechanisms extended time durations. To address this, it becomes crucial to assume that temporal dependencies are confined within a limited time window, beyond which attention computations unnecessary. Building this idea, propose the Window Denoising Process Model (Swin-DPM), which leverages a sliding temporal window to manage dependencies effectively and As
enables the generation of long even infinite videos by producing tokens with stride of is shown in Figure within each window, a queue of video tokens undergoes denoising at various noise levels. After denoising steps (where leftmost, lowest-noisy token is dequeued into a cache. To maintain the queue length, a new token with Gaussian noise will be then added the rightmost position. Each cached token re-appedded to the window's 's token queue at noise level O until the next token is cached, allowing it to continue participating in denoising and ensuring continuity between different windows. The network of Swin-DPM is fine-tuned from pre-trained DiT model. During training, we sample 2w video tokens, where w the window size. We usually set w = T. The first w tokens are used solely for warming up SWin-DPM and do not participate in backpropagation; loss is computed only the last w tokens. At inference time, we follow the same setup: the first w tokens are for warmup and are discarded, with the generated video starting from the (w 1)-th token.
T is the number of diffusion solver steps), the
Stream Consistency Model. After extending the DiT model Swin-DPM, further address the need for achieving real-time rendering of the simulated world. A promising approach is to combine Swin-DPM with Consistency Models 255 26), leading method for accelerating diffusion. We use the Stream Consistency Model (SCM) 277, which distills the original diffusion process and its class-free guidance into four-step consistency model while incorporating the denoising window design from Swin-DPM. The training procedure is illustrated in Figure 2 This integration results in 10- 20x acceleration in inference speed, reaching rendering rate 16 FPS.
3.2 Construction of the Source Dataset
To train The Matrix model, we construct the Synthesized Observations of Unreal Rendered Contextual Environments (Source) dataset, which consists of two components: synthetic game data from Unreal Engine and real-world, unlabeled footage. The synthetic game data, collected using the GameData
5

---

Figure The results demonstrate frame-level precise control achieved by the Interactive Mod- uleacross diverse scenes, weather conditions, and movement modes. Due to size limitation, we recommend you to go to the website provided the abstract for videos.
Platform, serves as supervised training data for precise motion control, while the real-world footage improves the model's visual quality and generalization real-world scenarios.
After collection, the data is segmented into 6-second clips of continuous scenes and captioned using GPT-4o (28), resulting in dataset of 750k labeled samples and 1.2 million unlabeled samples. all with 60 FPS. The labeled game data is further refined to ensure a balanced distribution of all possible game states. For more details the dataset, Appendix Section B.2.
The GameData Platform. The GameData Platform is built on open-source tools: Cheat Engine software (29). the Reshade plugin [30] for DirectX, and OBS Recording software [1 Cheat Engine is used to capture in-game world status data, such as character (æ,, y; z) positions and camera movements. This status data is aligned with recorded video frames to create per-frame action-video pairs and also used to check if the character or camera stuck and requires reboot. We employ the Reshade plugin to remove all game UIs and HUDs and to standardize shading styles, providing more consistent, fow-complexity data source. Data for Forza Horizon 5 collected using autonomous scripts with random walking algorithms, while Cyberpunk 2077 data is gathered manually with human operators running the GameData Platform. See Appendix Section B.l I for more details on the GameData Platform.
4 Experiments
Since existing world models are not trained on Forza Horizon 5 or Cyberpunk 2077, unfair to compare The Matrix against prior world models both qualitatively and quantitatively. Therefore, we focus on experiments only testing the properties of The Matrix. The Matrix is trained on 32x A.00 GPUs in week. All inference processes are conducted 8x A100 GPUs.
Training Details. We train The Matrix on the Source dataset, using a pre-trained 2.3B parameter DiT model as the backbone, which generates 4 video tokens per second, each decoded into 4 frames by the VAE decoder [32. To match this generation rate, we downsample the videos and keyboard inputs in the Source dataset accordingly. For all training cases, we first warm up the base DiT model on unlabeled Source data for 20,000 steps with a batch size of 32. Following this, we train the Interactive Module on labeled Source data for additional 20,000 steps with the same batch size, introducing another 0.4B parameter. Next, we fine-tune The Matrix model using Swin-DPM over 60,000 steps, also with a batch size of 32. For the final Consistency Model distillation, use the Swin-DPM

---

Dromge "In desclate dosert, white SUV is driving scress the terrain, On an arrial shot, the whiicle novigating throughh rugged lendsoape, surrenneed by parchal vegetation and sparse truss The camura follows the car's movemant, capturting the thre traces it laave beline in chu sand. In the disanaa some buildings mountaino can be whit the siy filled with clead, with sunfight straaiigg thrrugh and casting of fight
(a) Long 1-minute video generated by The Matrix.
Prongt: On desslate dawee, white STrV is driving acress tarrain. On an aerial shoe, whicll navigating through rugged (andicapan, surrounded by parched vegeration and sparse trees. The camena follows the car's yeavement, capturing the tire tracks it lasves bokind in the sand. 3n the distance some fuildings and mosnttaies can whille the sky filled with cloudd, with sunlight streatinng through and canting light.
The car then driving intte the ocean
The then driving Seach baside forest
f19
drtving a forust
car is then driwing into river in a forest
(b) continuous 2.5-minute video generated by The Matrix, spanning multiple diverse scenes controlled through DiT text prompts.
Figure 6: Long world generation results by The Matrix.
checkpoints as teacher model and train the student network for 10,000 steps with a batch size of 32. More details can be found in Appendix Section A.2.
Metrics. We evaluate performance using metrics for both general visual quality and movement control precision. For general visual quality, use Fréchet Inception Distance (FID) (3), Fréchet Video Distance (FVD) (34), and CLIP Score (35) to assess text alignment. All metrics are evaluated on 2,048 seconds of randomly generated videos. To evaluate movement control precision, we generate 2,048 seconds of video based on keyboard inputs and text prompts from a fixed test set, then measure the Peak Signal-to-Noit Ratio (Sove-PSNR) (36) and Learned Perceptual Image Patch Similarity (Move-LPIPS) (37 between the generated videos and real videos with ground truth movements.
4.1 Precise Frame-Level Interactions
In this section, we evaluate the effectiveness of the Interactive Module by testing its performance in three distinct scenarios: the Forza Horizon 5 car driving scenario, the Cyberpunk 2077 city walking scenario, and a robotic arm task from the DROID dataset [38] We select 50,000 6-second clips from the DROID dataset, along with per-frame action labels of joint angles for seven joints, to form the training dataset. More details can be found Appendix Section B.3. The third scenario specifically designed to assess the effecttiveness of The Matrix in embodied AI tasks. For all scenarios, we follow
7

---

Prompt: The driving indeors.
Pronpe:: The driving indoor corridor.
"Pronge & woman inside car, warring oversized sunglasses and dressed in black.
Prompe: Garila girl ssing vucusess cleaner to mave acrios the wooden floor at fooe.
(a) The Matrix generalize its precise indoors or making people move as instructed. Go the wehsite for videos
control unlabeled scenes and objects, such driving
Prompe: Ih the center the picture, large yacht marred at the piar, and other small boats are scatterad around. The harbor is nurrounded unded by an ancient stona are decorated with domas tog. The whole duck
wall with fiidgee consocttngg banks. the distance, there are dense urban buildings dky light blue orange.
(b) The Matrix can also generate long, general vidcos by disabling the Interactive Module, acting as a powerful vidco generator. Figure Generalization ability of The Matrix on unseen scenes and objects.
the same training strategy: starting with a pre-trained DiT model, we first perform warm-up using unlabeled data, followed by fine-tuning the Interactive Module with labeled data.
Qualitative Results. Figure illlstrates examples of The Matrix's generated outputs across all sce- narios. The Matrix demonstrates the ability create vivid and dynamic worlds, accurately reflecting user interactions and intentions. It also models the physical behaviors within these environments, such dust being kicked up when drives through dry desert, water splashing when it travels through a river. Additional examples The Matrix's generation capabilities are provided in Appendix Section C.I.
The last two columns Table present the quantitative evaluation of interaction precision, using LPIPS and PSNR metrics. The results demonstrate that Interactive Module significantly improves control precision, and this enhancement is maintained throughout the subsequent Swin-DPP DPM and SCM processes.
Quantitative Results.
4.2
Mnfinit-HHoizon World Generation
Traditional world simulators focused on precise control often rely on small, auto-regressive generators trained from scratch minimize the significant memory and time costs associated with pre-trained DiT models. However, this approach compromises visual quality and limits the full potential of world simulators. In this work, we introduce world simulator leveraging pre-trained video diffusion models, enabling infinite-length world generation with real-time rendering capabilities. In this section, present evaluation of these advancements.
Generating Infinitely Long Videos. Figure showcases examples of generating 1-minute long worlds across diverse scenarios, including desert, river, grassland, snow, and day-to-night transitions.

---

Table 1: Ablation study the components of The Matrix. Note that there is trade-off between inference speed, control precision, and rendering quality. Move-LPIPS and Move-PSNR are computed between the generated videos and test videos with ground truth movements.
Component
Scene
Size
Speed
FVD.
FIDJ
CLIPT
Move-LP
Move-PSNRT
DiT Backbone
2.3B
1.41 FPS
1016.30
318.10
0.30
Cyberpunk
2.3B
1.88
1429.45
183.24
0.28
0.125
27.80
Warmup
DROID
2.3B
1.41 FPS
1133.16
224.98
0.29
0.191
27.72
Horizon
2.3B
1.41 FPS
1891.67
141.11
0.31
0.128
26.89
Cyberpunk
2.78
0.87 HPS
11T2.49
173.31
0.28
0.129
28.24
Interactive Module
DROID
2.7B
0.87 FPS
1200.82
237.66
0.30
0.180
27.90
Horizon
2.7B
0.87 FPS
1211.30
119.20
0.27
0.125
28.98
Swin-DPM
Horizon
2.7B
0.8 FPS
1651.50
163.27
0.24
0.113
29,90
+CCM
Horizon 3
2.78
FPS
1936.79
153.80
0.23
0.109
29.73
During generation, we switch the DiT prompt to adapt the environment, as shown in Figure 6b The Matrix's capability extends beyond this; it can generate truly infinite-length videos, with additional 15 15-minute examples available in Supplementary Videos. Table reepots the video quality and control precision of The Matrix after training with Swin-DPM, While some visual quality is sacrificed, control precision remains strong, and the visual quality still surpasses previous world simulators.
Real-Time Rendering. We further investigate integrating SCM with The Matrix. As reported in Table this integration highlights The Matrix's 's real-time rendering capability, with a slight trade-off in visual quality and minimal loss in control precision, while significantly improving rendering speed from 0.8 FPS to 16 FPS.
4.3 Generalization to Out-of-Distribution Worlds
In addition to superior visual quality, key advantage of using pre-trained video DiTs is their inherent ability to generalize across diverse scenes. We observe impressive generalization in The Matrix, showcasing the potential of future research into building world simulators with pre-trained DiTs.
control car in previously unseen scenes by describing the scenario in the prompt. The first two rows of Figure 77 demonstrate this capability, where the car driven through indoor environments, which were not part of the Source dataset.
Generating Unseen Scenes. With The Matrix,
Interacting with Unseen Objects. A remarkable feature is The Matrix's s ability to generalize interaction with real-world objects. As shown in the last two rows of Figure 7a by specifying human as the center object in the prompt, we move the person in response to keyboard input.
Generating Long Videos without Moving Control. Though The Matrix is trained on the Source dataset, can also funetion as a general long video generator. By disabling the Interactive Mod- uleand using only the DiT backbone trained after Swin-DPM, The Matrix can generate long videos corresponding to ordinary prompts. Figure 7 shows such an example, further proving The Matrix's strength as realistic world simulator.
5 Conclusion
We introduce The Matrix, a world simulator capable of generating infinitely long, high-fidelity video streams with precise real-time control. Trained on a blend of game data and real-world footage. The Matrix supports immersive exploration of dynamic environments, with zero-shot generalization to unseen scenarios. Operating at 16 FPS, it enables continuous, interactive simulations across diverse terrains, bridging the gap between virtual and real-world applications. This work highlights the potential of using game data to build robust world models with minimal supervision, and showcases the power of pre-trained video DiTs in enabling realistic, large-scale simulations.
Limitations.
The Matrix requires 8 GPUs for deployment, making it essource-intensive While it achieves a generation speed of 16 FPS, this is still slower than the originl frame rates. Moreover, the synthesized videos may occasionally exhibit physical implausibilities Currently, The Matrix is trained on only two labeled games as a proof of concept. We leave the exploration of these limitations future work. See more discussions in Appendix Section E.I.
temporal inconsistencies.
Broader Impacts. We envision The Matrix as foundational approach for Al-driven game generation, with the potential to transform the future of the gaming industry. The generated content blends real- world and synthetic game assets, enabling novel and immersive experiences.

---

Acknowledgments
We would like to express our gratitude to the Alibaba Cloud Elastic Computing-Heterogenn Hypervisor & Instance team for providing the Remote Gaming Console (RGC), which facilitates our data collection process. Special thanks Bo Li, Min He, and their colleagues for their invaluable support and assistance. We also extend our appreciation to Microsoft and CD Projekt Red for developing their remarkable games, which played key role in our data collection. A special acknowledgment goes to the Night City and Mr. V. who provided a peaceful environment for our research, We thank Shangwen Zhu from Shanghai Jiao Tong University, who contributed to the experiment design. We thank Deli Zhao from Alibaba DAMO Academy for discussions and consultant on the technique trend. We have special thank to Bryce Schmidtchen and Alberto Taiuti for valucable discussions. Finally, we thank Dr. Ping Luo of the University of Hong Kong for his insightful guidance on the collection robotics data.
10

---

References
[1] Jürgen Schmidhuber. On learning think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models. arXiv preprint urXiv:1511.09249, 2015.
[2] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, chess and shogi by planning with learned model. Nature, 588(78339:6044-669) 2020.
[3] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
[4] Eloi Alonso, Adam Jelley. Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter Atari, arXiv preprint (tt.:::200.129999 2024.
[5] Virtuals Protocol. Vidco game generation: A practical study using Mario. Preprint, 2024.
[6] Decart, Julian Quevedo, Quinn Mcintyre, Spruce Campbell, Xinlei Chen, and Robert Wachen. Oasis: A universe in a Transformer. 2024.
[7] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 04104, 2023.
[8] Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: real-time and open-source interactive world model minecraft. arXiv preprint arXiv:2504.88388 2025.
[9] Dani Valevkki, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusson models are real-time game engines, arXiv preprint arXiv:2408. 14837, 2024.
[10] OpenAL. Sora: Creating video from text. Preprint, 2024.
[11] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richic Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In International Conference on Machine Learning, 2024.
[12] Jack Parker-Holder, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Christos Kaplanis, Alexandre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung. Michael Dennis, Sultan Kenjeyev. Shangbang Long. Vlad Mnih, Harris Chan, Maxime Gazeau, Bonnie Li, Fabio Pardo, Luyu Wang. Lei Zhang, Frederic Besse, Tim Harley, Anna Mitenkova, Jane Wang. Jeff Clune, Demis Hassabis, Raia Hadsell, Adrian Bolton, Satinder Singh, and Tim Rocktäschel. Genie 2: A large-scale foundation world model. 2024.
[13] Niket Agarwal, Arslan Ali, Macicj Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical AL. arXiv preprint arXiv:2501.08 03575, 2025.
[14] William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEF/VV International Conference on Computer Vision, pages 4195-4205, 2023.
[15] Mengjiao Yang. Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real- world simulators. arXiv preprint arXiv:2310.06114, 2023.
[16] Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning. Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and vidco states. arXiv preprint wxXiv2240600455 2024.
[17] Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. GameGen-X: Interactive open-world game video generation. arXiv preprint arXiv:2411.00% 2024.
[18] Anthony Hu, Lloyd Russell, Hudson Yeo. Zak Murez, George Fedoseev. Alex Kendall, Jamie Shotton, and Gianluca Corrado. GAIA-I: A generative world model for autonomous driving, arXiv preprint arXiv:2309. 1rXiv:2309.17080 170800 2023.
[19] Zangwei Zheng. Xiangyu Peng. Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all. https://github com/uppcattecch/ppen-Sooor 2024. Software

---

[20] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06455 2016.
[21] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv:1606.08..0 2016.
[22] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation large language models. arXiv preprint arXiv22106009683 2021.
[23] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21((40):1-67 2020.
[24] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid weighted linear units for neural network function approximation reinforcement learning. Neural newworks, 107:3 11, 2018.
[25] Yang Song, Prafulla Dhariwal, Mark Chen, and Nyya Sutskever. Consisteney models. arXiv preprint arXiv:2303.01 2023.
[26] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In International Conference on Learning Representations, 2024.
[27] Akio Kodaira, Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, and Kurt Keutzer. Streandiffusion: A pipeline-level solution for real-time interactive generation. arXiv preprint arXiv: /2491, 2023.
[28] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. GPT-4o system card. arXiv preprint 21112:2410.21276. 2024.
[29] Cheat Engine. Cheat engine. wttp::///www..hhhttiiinng org/ 2024. Software
[30] Reshade, Reshade. https://reshade. me/ 2024. Software.
[31] OBS Recording, Obs studio. /ttps::/obsprojjct. com/ 2024. Software.
[32] Diederik Kingma. Auto-encoding variational Bayes arXiv preprint arXiv:1312.6114 2013.
[33] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berharr Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local nash equilihrium, Advances in neural information processing systems, 30, 2017.
[34] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: 17X2:18112.01717, 2018
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[36] Milan Sonka, Vaclav Hlavac, and Roger Boyle Image processing, analysis and machine vision. Springer, 2013.
[37] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2018.
[38] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. DROID: large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403. 12945, 2024.
[39] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018
[40] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01600 2019.
12

---

[41] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04953 2022.
[42] Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalahle, robust world models for continuous control. arXiv preprint 2xxi:23300.18266 2023.
[43] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. DayDreamer: World models for physical robot learning. In Conference on robot learning, pages 2226-2240. PMLR, 2023.
[44] Weidong Huang, Jiaming Ji, Borong Zhang, Chunhe Xia, and Yaodong Yang. SafeDreamer: Safe reinforcement learning with world models. arXiv preprint arXiv:2307.07176, 2023.
[45] Maxime Burchi and Radu Timofte. MuDreamer: Learning predictive world models without reconstruction.
arXiv preprint arXiv:2405. 15083, 2024.
[46] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. DeepSpeed Ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309. 14509, 2023.
[47] Simian Luo, Yiqin Tan, Longho Huang, Jian Li, and Hang Zhao. Latent consisteney models: Synthesizing high-resolution images with few-step inference, 2023.
[48] FFmpeg Developers. Ffmpeg, 2023. Software.
[49] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lui, et al. Intervvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. Proceedings of the IEEE conference on computer vision and pattern recognition. 2024.
[50] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 30599999...99.... 2023.
13

---

A More Related Works
World Model for Agent Learning. Developing world models for training agents has been long- standing research focus, aimed at enhancing policy learning within simulated environments rather than solely achieving high-fidelity reconstructions of observations. This research involves two primary stages: 1) modeling the training environment by reconstructing observations, rewards, and continuation signals, often through recurrent state-space space model; and 2) utilizing this model to predict future states, enabling reinforeement learning to optimize robust policy functions. Studies indicate that this method provides sample efficiency gain of over 1000% compared to directly learning policies from real environments, shows resilience across diverse domains, and can outperform fine- tuned expert agents on a range of benchmarks and data budgets 77 Key contributions in this area include Recurrent World Models [39], Dreamer (v1 40 v2 3) and v3 7 TD-MPC (v1 [1 and v2 422 DayDreamer [33], SafeDreamer [44]. and MuDreamer [45] Notably, MuZero 2 runs the self-play
of Monte Carlo tree search build world models for Atari, Go, chess and shogi, without external data.
B
Details in Experiments
B.1
DiT Backbone
The DiT backbone is adapted from the publicly available DiT models [9]. It consists of patch embedding module, a caption embedding module, a timestep embedding module, 32 DiT blocks, followed by a linear output head with LayerNorm [20]. The followings provide details of each module within the DiT backbone.
The Patch Embedding Module. The patch embedding module employs 3D convolution with kernel size of 1 x x 2, followed by reshape operation. Thus, the convolution can effectively process the video latent from the VAE encoder, and the reshape operation can further transform the feature into a sequence of tokens with 2,048 feature size. By using a 3D convolution, the module captures both spatial and temporal features, ensuring that the token sequence retains essential information from the video data.
Caption Embedding Module. The caption embedding module takes the caption token sequence encoded by the T5 model and further processes it through a two-layer FFN. Both the hidden feature size and the output feature size are set to 2,048, allowing the module generate rich and high- dimensional representations of the caption data.
Timestep Embedding Module. The timestep embedding module is implemented sinusoidal embedding module followed by two-layer FFN. Both the hidden feature size and the output feature size of this FFN are 2,048.
DiT Block. Each DiT block includes a self-attention layer operating on network features, a cross- attention layer linking conditions with self-attention outputs, and FFN layer composed of two linear layers with a GELU activation 22 in between.
B.2 Training Details
Upon obtaining the base DiT model, the training process consists of four distinct stages: (1) warm-up on unlabeled Source, (2) training of the Interactive Module, (3) fine-tuning using Swin-DPM, and (4) Stream Consistency Model distillation. Below, we first outline the common training configurations utilized across all stages, followed by detailed description of each individual phase.
Common Settings, All training procedures were executed with an overall batch size of 32 and learning rate of x 10-5, Mixed precision training was employed using bfloat16 to enhance computational efficiency. During preprocessing,
Mixed-precision
video inputs were resized to a resolution of 1280 x 720 pixels and set to 16 FPS. For sequences exceeding 25,200 frames in length, we used the Deepspeed Ulysses sequence parallelism strategy [46), distributing the sequence across GPUs to manage memory and computational demands effectively.
14

---

Warm-Up on Unlabeled Source Dataset. In the initial warm-up stage, we fine-tuned all linear layers of the base DiT model using Low-Rank Adaptation (LoRA) tailor the model to the source data distribution [22]. The LoRA rank was set to 128, and the model was trained for 20,000 steps. This adaptation ensures that the model parameters are suitably adjusted to the characteristics of the unlabeled source dataset before advancing to subsequent training phases.
Training of Interactive Module. The second stage focuses on training the Interactive Module, each of which is integrated after every two consecutive DiT blocks, totaling 16 Interactive Module. During this phase, the parameters of the base DiT model were frozen to concentrate the training solely on the Interactive Module. This stage was conducted over 20,000 training steps, enabling the Interactive Module to effectively interface with the base model without altering its foundational parameters.
Fine-Tuning Using Swin-DPM. The third stage involves comprehensive fine-tuning of all model parameters, including both the base DiT model and the Interactive Module, utilizing the Swin- DPMapproach. This extensive fine-tuning was carried out over 60,000 steps, allowing for the refinement and optimization of the entire model to better capture data intricacies and enhance overall performance.
Consistency Model Distillation. In the final stage, consistency model distillation was performed using the model from the preceding fine-tuning phase as the teacher model. The student model was initialized with the teacher's 's weights to facilitate knowledge transfer. During distillation, we employed a one-stage guided distillation technique (47), incorporating Classifier-Free Guidance (CFG) into the student model. For the Ordinary Differential Equation (ODE) solver within the consistency distillation framework, we utilized the Euler solver with a single-step size of 25/10000 This distillation process was conducted over 10,000 training steps.
C
The Source Dataset
C.1 The GameData Platform
We build framework, GameData Platform, for data collection. The framework consists of three components: Controlling, Simulation, and Observation.
Controling. In most games, we need to control a character to go to different scenes and make interactions. Intrinsically, the collected data can be reconstructed with initial states of game worlds and series of control signal. In order to make the collected data clean and meaningful, instead of being stuck in one corner, we designed two different control systems, namely the automatic one and the manual one. For the automatic control system, we use Cheat Engine for pivotal data access, such as XYZ coordinates in games. These data can be used to determine whether the game has been stuck for some time. We detect the coordinates of past period of time and determine whether they covered in a circle of a given size. the game is detected as stuck, we will reset the game state and restart the recording. Generally, the automatically generated control signals will move randomly, change direction, and change perspective. This is good enough for games that move on a 2DDllike surface. However, for games moving in 3D space, random signals will struggle with generating meaningful content, so we have to change to the manual system. Since our game running and captured cloud servers, human data collectors will observe the game through a low-defiittion streaming and control manually. Signals (from keyboards, mice, and gamepads) are translated and delivered through the socket server, and cloud servers will generate keyboard events through the virtual keyboard. Here, the latency between the control signals and the OBS screen recording is crucial. We eventually found that the control signals recorded on the cloud sever and the actual action responses in the recorded videos were generally no more than three frames apart. and in general, this delay is stable and can be subtracted directly from the timeline.
The game runs directly on the cloud servers. we can directly copy the server images to get a large number of running instances. The recorded videos and control signals will be uploaded to the data center. We set up series of video quality checks to filter out samples of low quality (still or overly noisy videos, and some undefined scenes). All games run at the highest quality while ensure the OBS screen recording does not get stuck. In order to avoid overly complicated situations,
Simulation.
15

---

removed the NPCs and running vehicles in the game. We use the Reshade to adjust the game scenes to make more reality-
Observation. We use OBS as the screen recorder. One can use scripts to control OBS for automatic recording. We recorded the game at native resolution of 2560 x 1600 (higher resolutions may cause the game and recording to lag). For the reality of the recorded videos, we removed GUIs and texts in the game through Reshade plugin, namely SeshaarrEffeettShddooggglee rendering of GUI related shaders in the game while left the native video untouched.
It can turn off the
Forza Horizon 5. In Forza Horizon 5, telemetry mechanism be used for game status retrieving. We can access the real-time game data through socket after checking on the telemetry option in settings. An example script for data listening can be found here We can access XYZ coordinates, velocities and accelerations. We use these data for stuck detection and sample filtering. Since Forza Horizon 5 is game that mainly takes on 2D area, we apply automatic pipeline that randomly walking on different game scenes (like dessert, grassland, the watery and the snowy areas). Control signals are simplified to going forward, turning left and turning right. During the data collection stage, if XYZ of is still for several seconds, the controller will try to move back. And if the 40 position points collected during the last 40 seconds can be covered with circle with radius of 80 meters, the controller will try to teleport the car to a random position. After raw sample collection, we apply some strategies to filter out samples of low quality. We use the acceleration data to detect if the car has collided with anything, and drop these video clips with collision. Sometimes the car is moving backward while the controlling input is moving forward, this because the direction of movement in the game is to provide acceleration. We filter out data with a large angle between acceleration and velocity. Due to some problems in the game itself, the video often changes suddenly at some time. We filter out video clips with large average error between any two adjacent frames.
Cyberpunk 2077. Cyberpunk 2077 is a game that offers realistic visuals and lighting effects. Due to the complexity of the game terrain, we have to choose the manual pipeline. For simplicity, the actions in game are reduced to two separated inputs. The first one makes the character move forward or stop. And the second one makes the direction of the character's 's sight move up, down, left and right. We disable the NPCs and moving vehicles with game mod. During the data collection, players observe the game through low-definition OBS streaming and send control signals. The signals are then mapped into "W" (moving forward) / "U" "D" "L." "R" (up, down, left and right) on the cloud servers. We access and record the XYZ coordinates of player through Cheat Engine. These coordinate sequences are then used for filtering out video clips where collisions occur between the character and the game scene.
C.2 The Source Dataset
We present the Source dataset from three perspectives: basic information, the annotation method used to convert the original data from GameData Platform to our desired format, and the filtering method applied to remove undesirable data.
C.2.1 Basic Information
The Source comprises data from both Forza Horizon 5 and Cyberpunk 2077. For Forza Horizon 5, we collected approximately 1,200,000 pairs of video and control signals, while for Cyberpunk 2077, gathered around 1,000,000 such pairs. All collected videos have duration of approximately 6 seconds, recorded at 60 FPS. For Forza Horizon 5, we specifically collected data from multiple scenes, including deserts, oceans, water bodies, grasslands, and fields. The videos from different scenes are illustrated in Figure AT along with the distribution of data volume for each scene Figure A2 (a). For Cyberpunk 2077, we focused on gathering data from urban environments that feature a significant number of tall buildings.
In Forza Horizon 5, the dataset includes only three distinct control signals: "moving forward" (denoted by "D"), "moving forward and turning left" (denoted by "DL.), and "moving forward and turning right" (denoted by "DR"). In contrast, the data for Cyberpunk 2077 encompasses five different
listener
16

---

woods
grass
desert
5e0
field
dusk
night
river
Figure Al: Examples of Horizon5 across different scenarios.
0.25
Cyberpunk2077
0.4
HorizonS
percenlage
percentage
grass
field
(a) statistics across different scenarios Horizon5
(b) statisties different control signals
Figure A2: (a) The statisties for Horizon5 across different scenarios include woods, grass, desert, sea, fields, rivers, and others. The results indicate that the quantity of data across these scenarios is relatively balanced. (b) The statistics of different control signals for Cyberpunk 2077 and Horizon5. In Cyberpunk 2077, the percentage of the "move forward" signal is relatively high, while other steering control signals are distributed more evenly. In Horizon5, all three control signals are evenly distributed.
control signals: "moving forward" (denoted by "W"), "turning left" (denoted by "L"), "turning right" (denoted by "R"), "looking upward" (denoted by "U"), and "looking downward" (denoted by "D").
C.2.2 Annotation Methods
The original data from GameData Platform typically has a duration of around 10 minutes, which is excessively long for training The Matrix. Therefore, we use FFmpeg [48] into 6-second clips. Next, we extract the corresponding control signals from the complete set of signals. We then InternVI 49 generate captions based on 12 uniformly extracted key frames from the videos. After the captioning process with Intern VL., we perform manual corrections on the generated captions eliminate obvious errors.
segment these videos
C.2.3 Filtering Methods
After the annotation step. a significant amount of undesired data remains, which could disrupt the training of The Matrix. To address this, we employ five filtering methods to eliminate these problematic data points, which we introduce as follows. Note that for Cyberpunk 2077, since we utilize human data collection rather than automatic methods, many of the following issues do not exist.
Balance Control Signals. Balancing the number of different control signals is beneficial for the training of The Matrix. The process of balancing control signals consists of three steps: First, we analyze the distribution of control signals for each 6-second video and record the results. 2) Next, we assess the overall distribution of control signals across the entire dataset to identify the most frequently occurring control signal. 3) Finally, we remove some data points that contain the highest proportion of this predominant control signal. repeatedly implement the second and third steps until the distribution is relatively balanced. The distribution results for Forza Horizon 5 and Cyberpunk 2077 are reported Figure A2 (b). We provide the pseudocode for the algorithm in Algorithm
17

---

Algorithm Control Signal Balancing Algorithm
Require: Dataset D containing control signals from 6-second videos
Ensure: Balanced Dataset B
1: Initialize B as an empty set
2: for each video in D do
3: Analyze the distribution of control signals in e
4: Record the results for
5: end for
6: while not isBalanced( B) do
7: overallDistribution + Assess the overall distribution of control signals in D
8: mostFrrqquenSignal + Identify the most frequently occurring control signal from overallDis-
tribution
9: +- Remove data points from D that contain mostErequenttSigall
10: end while
11: Set B 4- D
12: return B
Detect and Remove the Data with Collisions. In Forza Horizon 5, randomly generated control signals often cause the car collide with walls or rocks. Additionally, the car may be struck by other vehicles. These collisions can severely disrupt the training process, making it essential to identify and remove collision- affected data. Our analysis revealed that collisions consistently result in abrupt changes in acceleration over very short time. Thus, we use significant variations in acceleration as a reliable indicator of collision events and discard any corresponding data to maintain the integrity of the training process.
Detect and Remove Stuck Data. In Forza Horizon 5, after colliding with walls or rocks, the car often gets stuck; even when the "D" key is pressed, the car fails move. This stuck situation complicates the training data and negatively impacts the performance of The Matrix. Therefore, we need to detect and remove such instances. Detecting when the car stuck is relatively straightforward wwe simply calculate the distance the car has traveled within the video. If this distance falls below a certain threshold, we conclude that the car is stuck and discard the corresponding data.
Detect and Remove the Data with Mismatched Motion and Control. As introduced in ap- pendix C.I to quickly resolve a stuck situation, the car will move backward when stuck. As result, still move backward slower speed even when the "DI," "DR"
is possible for the keys are pressed. Similar situations may arise when "DL/DR" is pressed for a long period and then switched to "DR/DL." Although the acceleration directed to the right/left, the car may continue to move in the opposite direction for a brief period. We refer to this as mismatched motion and control, which complicates the training process. To address this issue, we calculate the directions of both the acceleration and the car's 's movement. If the angle between these two directions is too large, we discard the corresponding data.
Detect and Remove Artifacts. In Forza Horizon visual artifacts can occur when a car collides with obstacles like trees, introducing distortions into the generated videos. To filter out such corrupted data, we detect variations in pixel values across consecutive frames. Our analysis shows that applying a high threshold effectively identifies all videos containing these artifacts, enabling their removal.
C.3 The DROID dataset
C.3.1 Basic Information
DROID is a large, diverse robot manipulation dataset containing 76k demonstration trajectories (350 hours of interaction) collected across 564 scenes and 86 tasks over 12 months by 50 collectors worldwide. aims to improve the performance, robustness, and generalization of robotic manipulation policies. DROID uses the same hardware setup across all 13 institutions to streamline data collection while maximizing portability and flexibility. The setup consists of Franka Panda 7DoF robot arm, two adjustable Zed 2 stereo cameras, wristmounted Zed Mini stereo camera, and an Oculus Quest 2
18

---

headset with controllers for teleoperation. Everything is mounted on portable, height-adjustable desk for quick scene changes.
C.3.2 Filtering Methods
Remove Overly Complex Scenes and Balance Different Scenes. DROID is a collaborative effort involving multiple laboratories and contains data from 11 different environments, including domestic scenes like kitchens and bedrooms, as well as industrial settings such as factories and laboratories. Due to the complexity of these scenes, which poses challenges for subsequent captioning and video learning, we first classify the data based on scene labels. For each category, we manually select 50 less complex scenes. We then use DINO to encode and extract semantic features to calculate the mean, removing outliers within each scene based on this semantic mean. To balance the number of training samples across scenes, we ensure that the final dataset contains an approximately equal number of samples from each scene after outlier removal.
Filtering Frames Without Arm Presence and Removing Failed Executions. Since some frames in the videos do not contain the robotic arm or have only small visible area, we use Grounding DINOv2 [50] remove such frames. If more than 20% of the frames in a video meet this condition, the entire video is discarded. Additionally, to ensure accuracy in control, we remove data where the robotic arm fails follow the intended trajectory successfully, Finally, we use the spatial position of the robot gripper as a condition for each frame in the training.
D
More Examples
D.1 Generalization
We provide more results on the generalization ability of The Matrix in Figure A3
D.2
Long Video Generation
We provide several long video generation demos in the Supplementary Video files. Please check them after Unzip. All videos are heavily compressed to satisty the supplementary file size limit.
E
Discussions
E.1 Limitations
While The Matrix shows strong potential, there are several areas that remain challenging and should be the focus of future research and development:
Physical Implausibilities and Temporal Inconsistencies: While the model generates visually impressive videos, there are occasional physical implausibilities and temporal inconsisten- cies. These aspects highlight opportunities for enhancing realism in future iterations.
Limited Training Scope: Currently, The Matrix has been trained on only two labeled games (Forza Horizon and BMW). While these initial results show great promise, broadening the model's training across wider variety of games and environments will be crucial for improving generalization and real-world transferability.
Handling of Dynamic Objects: The Matrix performs well with static objects, such as the BMW car, but handling dynamic objects that exhibit complex behaviors remains an ongoing challenge. Addressing this limitation will be essential for expanding the model's capabilities in more dynamic, real-world scenarios.
These limitations reflect important areas for future study and refinement. While The Matrix is strong proof of concept, overcoming these challenges will enhance its applicability and performance diverse environments and more complex scenarios.
19

---

Prongt rad car driving the moustain.
Prompt: & driiving in village
Pronge: R man looking for place office.
Prompt: dröving
Pronee:: The swincesing the
Prompt: St man is riding try oar in the office.
Prompt: black dreving indsor corridor, with sewaal persons persofeerroes
Prompt: black driving enjoy with sevvral persons bestde
Figure A3: More generalization results of The Matrix unseen scenes and objects.
20

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Please Section
21

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please see Section !
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification:
We will release the data and code in the camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pblic/guideeee http://oeeubiiiiiooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
23

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: Please Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
25

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The sources of data and models have been explicitly acknowledged and cited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of license (e.g., CC-BY 4.0) should included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssiiipssiiiiaea... phcerssithcode thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/moddll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
26

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non- non-standard components.
Please refer to our L.M policy http:p//earrrppeecc//oofeeerrree for what should or should be described.
27


Reviews:
[{"review_id": "DU2d1lmi5T", "reviewer": "Reviewer_TvGJ", "summary": "The authors present a diffusion-based interactive video generation / simulation model (one in which the generated visuals reflect user text prompts and respond to user game inputs) while incorporating a novel “Shift-Window Denoising Process Model (Swin-DPM)” to enable infinite length video generation. They also introduce the “GameData” platform (used to generate the “Source” dataset) which captures paired game state and video frames. The authors boast that their method has improved rendering resolution and speeds versus existing methods (lines 58-64) though the experimental quantitative proof of this is somewhat lacking.\n\nThe Matrix diffusion model pipeline is represented in Figure 2 and is composed of: \n\n1) A pre-trained Video DiT backbone [14] (which the authors note in section 4.3 makes their method more robust to out-of-distribution scenarios), \n2) An Interactive Module responsible for responsible for “translating keyboard inputs into natural language that guides video generation” (described on page 4), \n3) A novel “Shift-Window Denoising Process Model (Swin-DPM)” (described on page 5 and Figure 4) where a shifting time sequence of video tokens is denoised simultaneously. After each token starting the queue is fully denoised the time-series advances with a new noisy input being added.\n4) A Stream Consistency Model (based on [27]) used to accelerate denoising although is sparingly described aside from a reference to existing work being given.\n\nThe GameData platform to autonomously collect data of paired video and game state is described on lines 172-181 where the authors use a number of open-source tools to capture game state, remove user interface (UI) elements, and capture paired video frames.\n\n__Novelty and Contributions:__\n- Diffusion interactive video generation model with improved visual fidelity and rendering speeds versus existing methods.\n- Propose a novel \"Shift-Window Denoising Process Model (Swin-DPM)\" for infinite video generation.\n- Describe their GameData platform for autonomous collection of training data.\n\n__Experiments:__\n\nExperiments display qualitative results in terms of long-form video generation with movement controls (Figure 6), generation of unlabeled scenes with movement control (Figure 7 a), and pure video generation without user interactivity (Figure 7 b). Quantitative results for an ablation study of The Matrix’s components are shown in Table 1 where the authors note a general trade-off between rendering speed, rendering quality and control precision.", "questions": "- Can the authors elaborate on why retraining an existing baseline method with their dataset may not be feasible (lines 183-184) to provide a more quantitative comparison to benchmarks?", "limitations": "Yes. Limitations are listed by the authors on lines 264-268.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "__Strengths:__\n\n- __State of the art (significance):__ The authors boast state of the art virtual world generation accommodating textual prompts and user inputs which the authors assert possess greater visual fidelity and frame rate compared to previous methods. They also assert that their method has better long term video generation capabilities. However, it would have been nice to see a more quantitative comparison to baselines (see weaknesses section).\n- __Novel Method (originality):__ Approach using diffusion models with a novel Shift-Window Denoising Process Model (Swin-DPM) to facilitate infinite video generation.\n- __Manuscript Clarity:__ I feel that clarity, significance and originality were well done though experimental quality related to benchmarks could be somewhat improved (see weaknesses below).\n\n__Weaknesses:__\n\n- __Quantitative experimental analysis (quality):__ The authors note that “Since existing world models are not trained on Forza Horizon 5 or Cyberpunk 2077, it is unfair to compare The Matrix against prior world models both qualitatively and quantitatively.” (lines 183-184). Is it not possible to retrain one/some of these models on your data to obtain a more quantitative comparison to baselines? I currently feel that this work is weakened without a substantive comparison to baselines (experimental quality could be improved). The authors claim advances in rendering speeds and resolution compared to existing methods, though some quantitative experimental results to support this claim would of been appreciated.\n\n- While the method can supposedly generate “infinite-length videos”, I would of found it insightful if there were some metrics to quantify/show the degradation (or lack of) of quality between the start to end frames of the generated video sequence.\n\n- __Computational limitations (significance):__ As the authors note in the Limitations section (lines 264-268), the model currently runs on 8xA100 GPUs which makes its usage prohibitive.\n\n__Current Assessment and Suggestions for Improvement:__\n\nI have currently marked the work as borderline reject. I feel that the submission currently relies too heavily on qualitative comparisons to existing visual generation methods instead of quantitative testing. The authors assert that quantitative testing versus existing methods is not possible due to these models being trained on different data. Can further justification be given by the authors as to why retraining with the existing dataset is not possible (see questions section below).\n\n__Minor Points:__\n- In case these have not yet been resolved, there are several typos related to missing spaces: 1) In Figure 1: “Matrixis”, 2) In Figure 2: “Modulefor” and “Modelis”, 3) on line 221 “Modulesignificantly”\n\n- I find the wording on line 65 to be somewhat odd “While we do not claim the Matrix to be a foundational world model” when the approach is described as “foundational” several times earlier (line 1, Figure 1, line 46).\n\n- My PDF viewer (Adobe Acrobat version 2025.001.20531) failed to be able to view the embedded videos and reported the error “A 3D data parsing error has occurred.”", "was_revised": true, "final_justification": "Following the authors' rebuttal, I have increased my score to accept. Most of my criticisms have been addressed in the rebuttal: additional comparison to Diamond baseline and further elaboration into runtime compute efficiency of model. Although, in the future it would be good to have a more extensive comparison to Diamond (or other method) in the multiple environments presented in this paper. Please see my comments to the author's rebuttal and original strengths and weaknesses section of my review for further details.", "rebuttal": "**Dear Reviewer TvGJ:** \n\nThank you for your valuable feedback and suggestions! We greatly appreciate your recognition of the novelty, significance, and originality of our work. Below, we address your concerns in detail.\n\n---\n\n### **Concern 1: Why is quantitative analysis against previous methods difficult?**\n\nWe understand your point about the importance of including quantitative comparisons with previous methods. While we agree that such comparisons would strengthen the paper, we faced several challenges that prevented us from including them in the main paper. However, during this rebuttal period, we have conducted a comparison with Diamond. We hope this addition will help clarify the challenges we encountered.\n\n- **Reason 1: Limited Open-Source Options:**\n\nAmong the competitors listed in Table 2, only Diamond and Oasis offer limited open-source code, both of which were primarily trained on the Minecraft dataset. Other methods do not provide their code or training pipeline, making it difficult to reproduce their results, even using their own datasets.\n\n- **Reason 2: Dataset and Architecture Differences:**\n\nAnother challenge we encountered is the significant difference between the Minecraft dataset (used by Diamond and Oasis) and the datasets we used, such as Forza Horizon and Cyberpunk. This mismatch in data labeling and network architecture made it challenging to apply their models directly to our data without significant modifications. This is why we opted not to include a comparison in the main paper.\n\nFinally, we conducted a comparison between our method and the Diamond world model in the Forza Horizon 5 setting. To ensure a fair comparison, both models were trained from scratch using 50,000 data clips over 4 full epochs. We used the default CSGO settings for Diamond. While Diamond requires additional labels like mouse clicks, weapons, etc., we randomly selected one example from the Diamond training set and inherited all training labels from it, except for the WASD direction label. While this approach might introduce some unintended biases, it was the most practical choice given the constraints.\n\nThe results are provided below. Please note that the results may not align exactly with the ones reported in the original paper. Additionally, the User-Study Accuracy is based on a manual inspection of the generated videos, where we counted how many frames we considered correctly aligned with the control inputs.\n\n**Tab. Comparason with Diamond on Forza-Horizon 5 Dataset**\n| **Method**          | **FVD ↓** | **Moving-PSNR ↑** | **Moving-LPIPS ↓** | **User-Study Acc ↑** |\n|-----------------------------|-----------|------------|-------------|-----------|\n| Diamond                    | 2111      | 9.44      | 0.703       | 0.51      |\n| The-Matrix                   | 1314      | 10.45      | 0.588       | 0.92      |\n\n### **Concern 2: Degradation of visual quality vs. time** \n\nThank you for your insightful suggestion! We are pleased to provide an FVD vs. time table below to demonstrate the effectiveness of our method in generating long-duration videos. While the initial 4 seconds show significant improvement, the remaining time durations do not exhibit noticeable degradation in quality.\n\n**Table: Evaluation Metrics Across Different Time Durations of The-Matrix**\n\n| Time Duration | FVD ↓ | Moving-PSNR ↑ | Moving-LPIPS ↓ |\n|---------------|-------|--------|---------|\n| 0-4s          | 1448  | 10.6   | 0.61    |\n| 4-8s          | 1952  | 10.0   | 0.64    |\n| 8-12s         | 2016  | 9.8    | 0.66    |\n| 12-16s        | 2070  | 9.4    | 0.67    |\n| 0-10s         | 1232  | 10.2   | 0.63    |\n| 10-20s        | 1952  | 10.0   | 0.64    |\n| 20-30s        | 2016  | 9.8    | 0.66    |\n| 30-40s        | 2070  | 9.4    | 0.67    |\n| 40-50s        | 2016  | 9.8    | 0.66    |\n| 50-60s        | 2070  | 9.4    | 0.67    |\n\nWe also further evaluate the performance over time using the **imaging quality**metric from the VBench. We sample 100 prompts from the in-distribution case (Forza-Horizon Scene) and out-of-distribution case (Real-World Scene), and generate videos of 96 seconds with them. We then measure the **imaging quality** metric of those videos at different times as follows:\n\n**Table: Performance across different time windows and environments**\n\n|   Time             | 6s | 12s | 24s | 48s | 96s |\n|----------------|---------|----------|----------|-----------|-----------|\n| OOD Cases         | 0.676   | 0.645    | 0.606    | 0.595     | 0.583     |\n| In-Distribution  Cases        | 0.640   | 0.619    | 0.607    | 0.609     | 0.603     |\n\nWhile OOD cases witness a slight degradation of quality, in-distribution cases are much stable and barely lose quality as time grows. Both of the cases can still maintain no-collapse visual quality as time grows.\n\n\n### **Concern 3: High computation cost (8 A100 GPUs)** \n\nWe are glad to report that our method now runs smoothly on 8 or even 4 L40 GPUs, utilizing a PCIe 4.0 connection with 64GB/s throughput—no need for NCCL. The rendering speed is over 32 FPS, which significantly reduces the computational cost to approximately $0.5 per hour for users. This represents a giant leap in both cost and speed compared to other world models based on diffusion methods!\n\nFurthermore, the entire codebase (which utilizes VNC to broadcast video frames from a remote server) has been open-sourced on GitHub for several months. It has already supported various startups working on world model products. The documentation includes detailed tutorials on how to use the code, and it is easy to set up. Unfortunately, due to NeurIPS policy restrictions, we cannot provide direct links to the code and supported products in this rebuttal, but we encourage you to explore these resources if you're interested. Attached is the GPU running time report of the Matrix DiT backbone under different settings.\n\n| GPU      | GPU Numbers | Communication | Latency for 1 Video Token | FPS |\n|----------|-------------|---------------|--------------------------|-----|\n| RTX4090 (L40s)  | 8           | PCIE 4.0      | 0.127s                   | 32  |\n| RTX4090 (L40s)  | 4           | PCIE 4.0      | 0.220s                   | 18  |\n| A100     | 8           | NVLink 4      | 0.101s                   | 40  |\n| A100     | 4           | NVLink 4      | 0.133s                   | 30  |\n| H100     | 8           | NVLink 4      | 0.077s                   | 51  |\n\n\n### **Concern 4: Typos**\n\nThank you for your careful proofreading! We will address these typos in the revised version of the paper.\n\n### **Concern 5: Use of the word \"foundational\"**\n\nThank you for pointing that out! We will replace \"foundational\" with \"novel\" to improve the coherence of the context.\n\n### **Concern 6: Document Viewing Issues**\n\nWe apologize for the inconvenience you experienced. This may have been due to a version conflict in Adobe Acrobat. We will ensure this issue is resolved in the revision.\n\n\n---\nWe sincerely hope these clarifications and additions address your concerns. Thank you again for your thoughtful feedback, and we appreciate your consideration of our work."}, {"review_id": "voEpl7ry9b", "reviewer": "Reviewer_jKm1", "summary": "*The Matrix* is a real-time world simulator that generates infinitely long, high-fidelity 720p video streams with frame-level control at 16 FPS. Trained on a mix of labeled game data and unlabeled real-world footage, it generalizes to unseen environments with zero-shot capability. Key innovations include the Swin-DPM for infinite video generation, an Interactive Module for user input integration, and SCM for real-time performance. A new dataset, Source, supports training with paired action-video data. The system demonstrates strong visual quality, precise control, and scalability, paving the way for AI-driven simulations in gaming, robotics, and virtual environments.", "questions": "- *The Matrix* shows impressive long-form generation (e.g., the 15-minute desert video). However, will the quality degrade over time in more complex scenarios like urban scenes? Quantitative evaluations would help assess long-term fidelity.\n- Could you provide details on the human effort involved in data annotation—such as the number of annotators and total time spent? This would help assess the cost of transferring the system to new games.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Generates high-quality videos at real-time speed.\n- Substantial effort is demonstrated, including platform development, data collection, and novel model design and training.\n- The module design is intuitive and effective.\n- The paper is well-written.\n\n**Weaknesses**\n\n- The core task is questionable. The system uses significant resources to generate immersive virtual worlds that could be more efficiently produced using traditional 3D game engines.\n- The visual diversity of the generated worlds appears limited by the training data.\n- There is little attention to world consistency or coherence—generated environments tend to be forgotten after just a few seconds.", "was_revised": true, "final_justification": "I appreciate the authors' response and continue to hold a positive view of the submission.", "rebuttal": "**Dear Reviewer jKm1:**\n\nThank you so much for your positive feedback and for recognizing our contributions in systematic framework design, dataset collection, and data pipeline establishment, as well as the overall performance of our approach. Below, we address your concerns in detail.\n\n---\n\n### **Concern 1: Core task is generating virtual worlds, which could be easily produced using traditional 3D engines.**\n\nWe appreciate your insightful assessment of the core task and the challenge of generalization in world models. Before addressing the generalization ability of our approach, we’d like to first defend the usage of world models from a few perspectives.\n\n- **Why world models are useful, even if they only do what traditional 3D engines can also do:**\n\n**Perspective 1: Scaling up 3D engines with data is difficult.**\nImagine you are tasked with creating a sim2real simulation for a company that has built a new factory in a new location. To simulate this scenario, you might need to:\n\n1. Collect numerous videos of the factory, as we did in the DROID dataset, and use them to train a world model, which requires minimal human labor.\n\n2. Alternatively, you could recruit several CG engineers to collect 3D mesh data and develop an entire new 3D environment, which would require substantial time and debugging.\nWhile current 3D GS methods can help accelerate the second approach, they still require heavy intervention from experienced CG engineers, which is both inconvenient and expensive compared to the first method. While CG-based methods dominate today, we believe that world models can offer a more efficient, data-driven alternative for creating sim2real simulations.\n\n**Perspective 2: 3D engines cannot conveniently provide gradients for downstream tasks.**\nAlthough techniques like differentiable 3D rendering exist, traditional 3D game engines generally struggle to provide gradient information for downstream tasks. As a result, neural network tasks often require methods like reinforcement learning or Q-learning to handle the absence of gradients. World models, however, can provide gradients directly, making them much easier to use in such scenarios. We discuss this aspect in detail in Section A of the supplementary materials.\n\n- **Generalization ability of our method (differentiating it from traditional 3D engines):**\nAs reported in Section 4.3 and shown in Figures 7a and A3 in the appendix (if you encounter any issues viewing the appendix, we suggest using Chrome or Preview on Mac, as we will address this Adobe version conflict in the revision), our model demonstrates strong out-of-distribution generalization. By pretraining on a large amount of realistic unlabeled data, our model can generalize to unseen scenarios, which is a key advantage over traditional 3D game engines.\n\n\n### **Concern 2: Visual diversity of the generated world.**\n\nFor the visual diversity aspect, we refer you to our earlier response regarding the generalization ability of our model. The diversity is a direct result of the pretraining on a wide range of real-world data, which enables our model to generate highly varied virtual worlds.\n\n\n### **Concern 3: Generated world coherence over time.**\n\nThank you for raising this critical point! Currently, we sacrifice long-term coherence for faster rendering speed. Specifically, we removed long-duration self-attention and use a small history cache. By increasing the range of the self-attention module or expanding the history cache, we can significantly improve consistency, though this will come at the cost of slower processing speed. Currently, we set a stride parameter for the Swin-DPM to trade off between speed and coherence. Stride=1 is used for scenarios pursuing extreme real-time experience ,while larger strides take care of coherence over longer time. We’ve conducted numerical experiments to validate this trade-off, and the results show that increasing the attention range through increasing the Swin-DPM stride size does indeed improve coherence.\n\n**Table: Effect of different strides to improve coherence (VBench Metrics)**\n\n| VBench Metrics         | Stride 1 | Stride 2 | Stride 3 | Stride 4 |\n|------------------------|----------|----------|----------|----------|\n| subject_consistency    | 0.862    | 0.873    | 0.882    | 0.885    |\n| background_consistency | 0.900    | 0.911    | 0.914    | 0.917    |\n| imaging_quality        | 0.662    | 0.675    | 0.689    | 0.695    |\n| motion_smoothness      | 0.976    | 0.976    | 0.978    | 0.977    |\n| dynamic_degree         | 0.99     | 1.00     | 1.00     | 1.00     |\n| aesthetic_quality      | 0.527    | 0.533    | 0.530    | 0.528    |\n\nTo collect those metrics, we select 100 prompts randomly sampled from the testing dataset, and generate 768 frames (48s) each from those prompts. We use the custom_input mode in VBench to evaluate those results under the above seven metrics. \n\n### **Concern 4: Will the quality degrade over time?**\n\nTo evaluate long-term performance, we measured the FVD score over time, and the results are provided below. Our model demonstrates consistent and stable performance across long durations, with no significant degradation in quality. Although the first 4 seconds show substantial improvement, the quality remains stable throughout the remaining duration, with no noticeable decline.\n\n**Table: Evaluation Metrics Across Different Time Durations of The-Matrix**\n\n| Time Duration | FVD ↓ | Moving-PSNR ↑ | Moving-LPIPS ↓ |\n|---------------|-------|--------|---------|\n| 0-4s          | 1448  | 10.6   | 0.61    |\n| 4-8s          | 1952  | 10.0   | 0.64    |\n| 8-12s         | 2016  | 9.8    | 0.66    |\n| 12-16s        | 2070  | 9.4    | 0.67    |\n| 0-10s         | 1232  | 10.2   | 0.63    |\n| 10-20s        | 1952  | 10.0   | 0.64    |\n| 20-30s        | 2016  | 9.8    | 0.66    |\n| 30-40s        | 2070  | 9.4    | 0.67    |\n| 40-50s        | 2016  | 9.8    | 0.66    |\n| 50-60s        | 2070  | 9.4    | 0.67    |\n\nWe also further evaluate the performance over time using the **imaging quality**metric from the VBench. We sample 100 prompts from the in-distribution case (Forza-Horizon Scene) and out-of-distribution case (Real-World Scene), and generate videos of 96 seconds with them. We then measure the **imaging quality** metric of those videos at different times as follows:\n\n**Table: Performance across different time windows and environments**\n\n|   Time             | 6s | 12s | 24s | 48s | 96s |\n|----------------|---------|----------|----------|-----------|-----------|\n| OOD Cases         | 0.676   | 0.645    | 0.606    | 0.595     | 0.583     |\n| In-Distribution  Cases        | 0.640   | 0.619    | 0.607    | 0.609     | 0.603     |\n\nWhile OOD cases witness a slight degradation of quality, in-distribution cases are much stable and barely lose quality as time grows. Both of the cases can still maintain no-collapse visual quality as time grows.\n\n### **Concern 5: Human labor used in data annotation.**\n\nAlmost the entire data collection process, except for the Cyberpunk dataset, was automated, as detailed in Section B of the appendix. However, some limited human labor was involved to maintain the realism of the training data.\n\n- For non-Cyberpunk data (e.g., DROID), we employed 10 annotators to review video clips and ensure they contained valid motion scenes (e.g., walking, running, juggling in the wild). This process took approximately 10 days to complete.\n\n- For Cyberpunk 2077 data, we recruited 20 human annotators to play the game and collect data. The collection process lasted 24 days (6 hours per day), during which approximately 2,880 hours of data was collected. This data was split into around 1.8 million 6-second video clips paired with control signals. We developed a visualization tool to assist annotators in making their keyboard inputs more uniformly distributed and used algorithms to filter out low-quality data (e.g., data where the player was stuck or made incorrect actions). The final dataset consisted of about 1 million valid data pairs.\n\nWe hope this clarifies the data annotation process. Please let us know if you need further information.\n\n---\n\nWe sincerely appreciate your thoughtful feedback and hope that our revisions address your concerns. Thank you again for your consideration."}, {"review_id": "DHVKmSg6sM", "reviewer": "Reviewer_m9ww", "summary": "This paper introduces \"The Matrix,\" a world simulator that generates infinitely long, high-fidelity (720p) video streams with real-time, interactive control. The model is trained on a combination of supervised data from video games and unsupervised real-world footage. Key contributions include the GameData platform for automated data collection, the novel Swin-DPM technique for infinite video generation, and the demonstration of zero-shot generalization to new scenarios.", "questions": "Questions：\n1. In line126, how much does the value of the probability q affect the performance of the model?\n\n2. Have you tried different window stride besides s=1?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The introduction of the GameData platform is a major strength. It addresses a critical bottleneck in training world models: the prohibitive cost and complexity of acquiring large-scale, action-labeled data.\n\n2. The core technical contribution, the Shift-Window Denoising Process Model (Swin-DPM), presents a novel and well-reasoned approach to extending pre-trained diffusion models for infinite-horizon video generation.\n\n3. The 1-minute video demo and other visual results are highly effective.\n\nWeaknesses\n\n1. The paper presents SCM as a key component in Figure 2, which could be interpreted as a novel part of the proposed architecture. However, the text clarifies that SCM is an adaptation of a prior method used to achieve real-time speeds. This is not a flaw in the method, but a point of potential confusion regarding the paper's specific innovations. I recommend that you clarify the role of SCM in Figure 2 and the main text. For example, you could relabel the component in the figure as \"Acceleration via SCM\" to explicitly attribute the technique and frame it as a crucial optimization step rather than a core generative model contribution. This would improve the clarity of the paper's originality.\n\n2. The paper claims real-time generation (16 FPS), but this performance is contingent on the SCM. The core generative model, Swin-DPM, appears to have significant efficiency limitations due to its auto-regressive nature. Based on the evidence you provide in Table 1, The + Swin-DPM model runs at only 0.8 FPS. The jump to 16 FPS only occurs after applying SCM, which also degrades visual quality (FVD increases from 1651.50 to 1936.79). Please acknowledge that the core Swin-DPM model is not real-time by itself and that the real-time capability is achieved through a distillation process (SCM) that trades some visual fidelity for speed. This would provide a more complete and transparent assessment of the method's performance.\n\n3. Hyperparameter Sensitivity (Line 126): During training, you replace labeled inputs with a default description with probability q=0.1. How sensitive is the model's control precision (e.g., Move-PSNR) to this value? Providing a brief ablation study or sensitivity analysis for q would help justify this choice and improve the work's reproducibility.\n\n4. Stride Choice in Swin-DPM (Line 143): The paper states that Swin-DPM generates tokens with a stride of s=1. Have you experimented with larger strides (e.g., s > 1)? While a larger stride might risk temporal inconsistency, it could also improve the baseline efficiency of the model before SCM. A brief discussion on why s=1 was chosen would be valuable.\n\n5. The manuscript would benefit from a thorough proofread to correct several small typographical errors (e.g., in the caption of Figure 2, \"Swin-DPMenables\" and \"Modelis\").", "was_revised": true, "final_justification": "Based on the rebuttal and discussion, I will keep my score.", "rebuttal": "**Dear Reviewer m9ww:**\n\nThank you for your positive feedback. We are especially encouraged by your appreciation of the GameData platform, Swin-DPM, and the visual results of our methods. Below, we address your concerns in detail.\n\n---\n\n### **Concern 1: Contribution of SCM and Swin-DPM to real-time efficiency**\n\nWe apologize for any potential misunderstanding regarding the contribution of SCM and Swin-DPM to real-time efficiency. While SCM uses consistency modeling as its distillation method, this technique alone would not be sufficient to achieve the final level of efficiency.\n\nSwin-DPM does not directly improve the inference speed of the videos from a global level (for example, a 4-second video still takes around 4 minutes to produce). However, it drastically decreases the latency. Previously, you would have had to wait 4 minutes to see just the first frame. Now, the 4 minutes are evenly distributed across each video frame, allowing you to see 4 frames (1 second of video, which contains 16 frames) every 25 seconds. This means the consistency model provides you with a near real-time experience, rather than the long wait that would be required to view several seconds of video in non-real-time settings.\n\nFurthermore, we have significantly optimized the SCM framework. In the initial setting, using only the consistency model and Swin-DPM would result in an inference speed of less than **4 fps**, due to the following reasons:\n\n- **GPU Utilization:** Swin-DPM does not fully utilize the GPU, which limits the parallel attention computation, preventing linear acceleration across multiple GPUs.\n\n- **Inefficient CPU Operations:** There are several inefficient CPU operations within the DiT pipeline (e.g., computing rope and other constants) that are typically not a concern in non-real-time scenarios but drastically limit real-time efficiency.\n\n- **NCCL and PCIe Communication Bandwidth:** In non-real-time scenarios, communication happens at a relatively low frequency, so bandwidth limitations are not significant. However, in our real-time setting, bandwidth becomes a critical issue.\n\nWe have addressed all of these challenges, resulting in significant acceleration for real-time inference. All the code has been open-sourced for several months, and you can try it yourself if you're interested! However, due to NeurIPS' policy, we cannot provide direct links here. You can easily find the code by searching for it on Google.\n\n\n### **Concern 2: Hyperparameter sensitivity**\n\nThank you for raising this point! We conducted a numerical ablation study on the probability of the \"replace label\" (q), and the results are shown below. Additionally, the User-Study Accuracy is based on a manual inspection of the generated videos, where we counted how many frames we considered correctly aligned with the control inputs.\n\n**Table: Effect of Replace Rate on Evaluation Metrics**\n\n| Replace Rate      | FVD ↓ | Moving-PSNR ↑ | Moving-LPIPS ↓ | User Study ACC ↑ |\n|-------------------|-------|---------------|----------------|------------------|\n| 0.00              | 1344  | 10.42         | 0.600          | 0.89             |\n| 0.05              | 1314  | 10.45         | 0.594          | 0.87             |\n| 0.10              | 1311  | 10.44         | 0.594          | 0.88             |\n| 0.15              | 1318  | 10.44         | 0.594          | 0.86             |\n\nWe believe the results demonstrate that while the hyperparameter is sensitive, we have identified an optimal range that provides the best trade-off between accuracy and robustness. We will include more details in the revised manuscript to clarify this.\n\n### **Concern 3: Stride choices**\n\nYes, we did experiment with larger stride sizes, and we observed that they can improve consistency and coherence in generation. However, this comes at the cost of increased computational burden. Larger strides result in larger feature sizes that need to be stored on the GPU, and the square rate of self-attention operations increases accordingly. Given these trade-offs, we chose a stride of 1 for the best efficiency, balancing computational load and performance. Attached is a numerical evaluation of stride size vs. video coherence using VBench (time cost increasing could be viewed as linear growth to the stride size). To collect those metrics, we select 100 prompts randomly sampled from the testing dataset, and generate 768 frames (48s) each from those prompts. We use the custom_input mode in VBench to evaluate those results under the above seven metrics.\n\n\n**Table: Effect of different strides to improve coherence (VBench Metrics)**\n\n| VBench Metrics         | Stride 1 | Stride 2 | Stride 3 | Stride 4 |\n|------------------------|----------|----------|----------|----------|\n| subject_consistency    | 0.862    | 0.873    | 0.882    | 0.885    |\n| background_consistency | 0.900    | 0.911    | 0.914    | 0.917    |\n| imaging_quality        | 0.662    | 0.675    | 0.689    | 0.695    |\n| motion_smoothness      | 0.976    | 0.976    | 0.978    | 0.977    |\n| dynamic_degree         | 0.99     | 1.00     | 1.00     | 1.00     |\n| aesthetic_quality      | 0.527    | 0.533    | 0.530    | 0.528    |\n\n### **Concern 4: Improving manuscript writing and proofreading**\n\nThank you for pointing this out! We will carefully polish the paper during the revision process to improve clarity and readability.\n\n---\n\nWe sincerely hope that these clarifications address your concerns. Thank you again for your thoughtful feedback and for considering our work."}, {"review_id": "MoAm5zn7pC", "reviewer": "Reviewer_7MCB", "summary": "This paper proposes a novel video generation model for generating infinitely long videos efficiently with action control. It also proposes a new dataset consisting of labeled game data for training.", "questions": "1. Why does the interactive module convert action labels into text prompts instead of directly conditioning on the action for more precise control?\n2. How is the unlabelled real-world/synthetic data from the source dataset constructed? Can the authors provide some examples of these unlabeled data and explain how they differ from the video data used to train the base video model?\n3. Most qualitative results focus on the Forza Horizon variant of the model, with only movement control with \"WASD\". Can the authors provide more results trained on Cyberpunk 2077 and the DROID dataset on camera control/robot joint control? The 4th and 5th row in Fig. 5 are too vague to demonstrate the model's controlability with the Cybepunk data and the 6th row in in Fig 5. doesn't show the control space for the robot grasping data. Also, are these model variants also have strong generatibilty to unseen environments like the  Forza Horizon varaint?\n4. Why does the action space of the Cybepunk data only encode forward motion and the four directions of the camera motion? Will it not work if both the 4 direction camera motion and the 4 direction agent motion are encoded?\n5. Does the model support joint training on all the game data with a uniform action representation, rather than fine-tuning a different variant to accommodate the different action spaces of the different subsets?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is easy to follow, and the infinite-long world generation task is well motivated with high application potential.\n2. The proposed source dataset from video games is a promising solution for obtaining high-quality action-labeled video data. I appreciate the engineering effort of the authors to create such a dataset.\n3. The qualitative results are strong and promising, showing the generalizability and various application aspects of the model.\n4. The whole pipeline is sound with a good combination of existing modules and an extensive ablation study.\n\nWeaknesses:\n1. Some technical contributions are a bit overclaimed. The claimed high-quality rendering is the feature of the base video model, and the real-time generation is achieved with a stream consistency model, which is not the contribution of this work.\n2. The current model still lacks some important features for world generation, such as realistic physics, interaction with other objects/characters, a memory mechanism for consistency, and precise movement/camera control. I would still call it a long-horizon video generation model with some vague controlability instead of a world model.\n3. Missing technical details. 1). The warm-up stage in Fig. 2 is not explained in the method section. It seems from the caption that it is trained from \"Synthesized Observations of Unreal Rendered Contextual Environments data\", but later in the training details, the authors \"warm up the base DiT model on unlabeled Source data\", which is \"real-world unlabelled footage\". Which datasource is actually being used for warmup and how is the model trained in this step? 2). In Sec. 4.1, it says more details about the training datasets can be found in Appendix B.3, but this section is missing in the supplementary. I'm particularly curious about how the DROID robotics dataset is used in training.\n4. The action space of the model is oversimplified. The authors map action to very high-level text prompts, sacrificing precise controllability.", "was_revised": true, "final_justification": "After reading the author's rebuttal and the other reviewer's comments, I drop my rating to weak accept as I still find most of the technical contributions are engineering implementations without much research insights in video generation. Some of the claims also lack supportive quantitative numbers and video examples and I suggest the authors include them in the final manuscript. However, I'm still in favor of accepting this work as a strong open-sourced baseline for further research in world generation.", "rebuttal": "**Dear Reviewer 7MCB:**\n\nThank you for your positive feedback and for appreciating our contributions to dataset development, pipeline design, and the qualitative results of our method. Below, we address your concerns in detail.\n\n---\n\n### **Concern 1: High-quality rendering as a feature of the base video model**\n\nWe agree with your assessment that the base model contributes significantly to the rendering quality. However, we have made substantial efforts to ensure that the introduction of Swin-DPM, SCM, and control labels does not degrade visual quality. Generating long videos with stable visual quality remains a significant challenge in the video generation domain, and it is important to note that the base model alone does not possess this capability.\n\n\n### **Concern 2: Real-time generation achieved by the consistency model**\n\nWe apologize for any potential misunderstanding. While SCM utilizes the consistency model as its distillation method, this technique alone would not have achieved the real-time efficiency we observed.\n\nAlthough Swin-DPM does not improve inference speed at a global level (for instance, a 4-second video still takes approximately 4 minutes to produce), it dramatically reduces latency. Previously, you would have to wait 4 minutes just to see the first frame; now, the 4 minutes are evenly distributed across each video frame, meaning you can see 4 frames (1 second, containing 16 frames) every 25 seconds. This gives the consistency model a near real-time experience, whereas, without this optimization, you would have to wait for several seconds to view a short video.\n\nAdditionally, we optimized the SCM framework significantly. In the initial setup, using just the consistency model + Swin-DPM would yield an inference speed of less than **4 fps**, due to several factors:\n\n- **GPU Utilization:** Swin-DPM does not fully utilize the GPU, preventing linear acceleration across multiple GPUs.\n\n- **Inefficient CPU Operations:** Certain operations within the DiT pipeline (e.g., calculating rope and other constants) are inefficient in real-time scenarios, limiting performance.\n\n- **NCCL and PCIe Communication Bandwidth:** In non-real-time scenarios, communication occurs at lower frequencies, so bandwidth limitations are less critical. However, in real-time settings, these limitations become significant.\n\nWe have addressed these challenges, resulting in significant improvements to real-time inference speed. The code has been open-sourced for several months, and you are welcome to try it yourself. However, due to NeurIPS' rebuttal policy, we cannot provide direct links here, but you can search for it online.\n\n\n### **Concern 3: Lack of important features for world generation**\n\nWe acknowledge that Matrix is still an early exploration in the field of world models. However, we believe it demonstrates important capabilities that suggest the potential for future world-generation models. Specifically:\n\n- **Generalization Ability:** As reported in Section 4.3 and shown in Figures 7a and A3 in the appendix (if you experience issues viewing the appendix, we recommend using Chrome or Preview on Mac, as we will address this Adobe version conflict in the revision), Matrix allows for rich out-of-distribution generalization. This is due to its pretraining on a large amount of realistic unlabeled data, showing the potential for building general foundational world models through scaling up training and data size.\n\n- **Interaction with Objects:** Although we are not able to provide visual examples at this time, a small number of object interactions do exist in the Matrix. For example, when a car collides with a tree or brushwood, the tree shows breakage effects; when the car hits a building, it stops; and when transitioning from land to water, the interaction with the ground changes. While we acknowledge that this is not yet a fully generalized interaction across all objects, these examples suggest that the research trend is moving towards the capability of object interaction. We will add this discussion to the paper.\n\n\n### **Concern 4: Oversimplification of the action space**\n\nWe agree that simplifying the action space may sacrifice the precision of movement. However, this trade-off allows the model to generalize better to real-world settings. As explained earlier, generalization of control in out-of-distribution scenarios is achieved by altering text prompts. The model is pretrained from the DiT model and inherits its understanding of text-based commands. By randomly dropping control signals during training, the model learns to manipulate the movements of real-world objects.\n\nWe hope to expand this further in future work by collecting additional types of interactions and using language as a unified protocol to represent them, enabling the model to create novel interactions. However, due to the high workload required, we did not implement this idea in the current work. This future direction also contributes to our decision to convert action labels into text.\n\nWe further conducted an experiment to validate the effectiveness of using transfer action labels in the form of one-hot labels instead of text prompts. Due to time constraints during the rebuttal period, the model may not have been fully trained (note that the numerical results may not be aligned with the paper), but the preliminary results suggest that text prompt-style actions outperform one-hot labels across all metrics. Specifically, here User-Study Acc. suggesst we mannully look into generated videos and count how many percentage of frames we think is correctly aligned with controls.\n\nWe hypothesize the following reasons for this result:\n\n- **Text Prompts Provide Semantically Rich Representations:** Text prompts offer distinct and semantically meaningful representations for actions. In contrast, one-hot labels are less expressive. For example, concepts like \"press W for 1 second, then press S for 1 second\" or \"moving irregularly\" are difficult to capture with one-hot labels but can be easily represented with text-based prompts.\n\n- **Base DiT Model Familiarity:** The base DiT model has been pretrained with text prompt embeddings, and it cooperates well with these embeddings. Since the model is already familiar with text-based representations, it is more effective when using them compared to one-hot labels.\n\n\n**Table: Comparison of Action Spaces on Evaluation Metrics**\n\n| Action Space | FVD ↓  | Moving-PSNR ↑ | Moving-LPIPS ↓ | User Study ACC ↑ |\n|--------------|--------|---------------|----------------|------------------|\n| one-hot      | 1311   | 10.42         | 0.590          | 0.72             |\n| text         | 1288   | 10.52         | 0.600           | 0.90             |\n\n### **Concern 5: Construction of real-world data**\n\nDue to the rebuttal policy, we are unable to provide specific examples of the real-world data. However, we are happy to explain how this data is constructed and how it differs from video data used to train the base model. You can get an idea of what the data looks like by searching for keywords like \"4K HUD walking in Tokyo, Shibuya Street.\" These videos share similar movement patterns to the game data, typically recorded from first- or third-person perspectives, with uniform movement speeds and high video quality. Unlike the base model, which uses a wide variety of videos from the internet, our real-world data focuses on walking, driving, and other movement scenarios captured from first- or third-person perspectives.\n\n### **Concern 6: More cases in Cyberpunk and DROID and their generalization ability**\n\nWhile we are unable to provide additional video cases due to the rebuttal policy, we can address your concerns. In terms of generalization ability, the answer is yes for the Cyberpunk cases, but no for DROID. The DROID scenario is more specialized and narrow, making generalization to other domains more difficult. However, the Cyberpunk dataset generalizes well, as the base model is already familiar with urban scenes.\n\n### **Concern 7: Why is the action space of Cyberpunk limited to four directions of camera movement and forward motion?**\n\nThe main reason for limiting the action space is to reduce the complexity of the data. Encoding all possible walking directions and camera movements would result in over 18 control signals, increasing the amount of data required for training. This would place a heavy burden on data collection, filtering, and balancing. We experimented with this approach but abandoned it due to the extreme workload. While the current action space can logically produce similar movement effects to the 18 possible control signals, we decided to keep the model simple for this work.\n\n### **Concern 8: Does the model support joint training on all game data with a uniform action representation?**\n\nYes, the model does support joint training with a uniform action representation across different games. However, since we only collected data for two games, we did not implement this feature in the current work. The action space design, where all actions are first converted into text, was developed with this capability in mind, as previously explained.\n\nBelow, we provide the training results for first training the Forza Horizon 5 scenario alone, followed by joint training on both datasets. The results demonstrate similar performance in terms of visual quality and movement fidelity.\n\n**Table: Evaluation Metrics for Joint Training on Different Scenarios**\n\n| Scenario                          | FVD ↓ | Moving-PSNR ↑ | Moving-LPIPS ↓ |\n|------------------------------------|-------|---------------|----------------|\n| Forza-Horizon 5                    | 1288  | 10.52         | 0.600          |\n| Forza-Horizon 5 + Cyberpunk 2077   | 1282  | 10.37         | 0.599          |\n---\n\nWe hope these clarifications address your concerns. Thank you once again for your thoughtful feedback and for considering our work."}, {"review_id": "w5VHrVyDd5", "reviewer": "Reviewer_6pVJ", "summary": "The work proposes an interactive video generator capable of: generating videos in real-time, generate long and high-resolution videos, condition generation on actions specified on a frame-by-frame basis, transfer knowledge learned from synthetic video game environments to real-world scenarios.\nTo realize such model, the authors consider a pretrained DiT video generator, introduce \"Interactive Module\" layers to a accept the control signals, Shift-Window Denoising Process Model to generate long videos auto regressively, and Stream Consistency Model to accelerate inference and achieve real-time generation.\nSome quantitative ablations and qualitative evaluations are shown to support the paper claims.", "questions": "The paper is explained clearly and my doubts regarding the work stem could be clarified with more evaluation. Several technical design choices or components of the training recipes as highlighted in previous review points do not find an ablation. I would reserve a higher score for the work if it presented experiments evaluating:\n- Data mixture: How important it is to use real data? How much data is needed? Are multiple videogame data needed to achieve real-world controllability?\n- Action representation: Is it necessary to translate keyboard and mouse actions to textual descriptions?\n- Swin-DPM and SCM: What is the effect of their main hyper parameters on final quality of the model?\n- How each component affects real-world transfer performance: present metrics that link each technical contribution to the amount of real-world transfer performance they are responsible for", "limitations": "The paper presents a limitation section, but does not describe method limitations deeply. The section should mention the issues reported in the review regarding lack of memory, limited environment understanding, very limited ability to perform real-world knowledge transfer and difficulties in handling dynamic objects besides the BMW car.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "QUALITY\n- The generated driving results are compelling, showing 16fps generation of high resolution videos with fine grained control for lengths up to 15 minutes. This is a difficult result to achieve. Driving results however show limitations in the situations the model can simulate. The model does not seem able to handle properly jumps, with the car appearing to always drive on mostly flat surfaces despite the presence of dunes in the desert environment. The model has little memory for previously generated part of the environment. When the car turns back to a region observed few seconds ago, a new scene is generated instead.\n- I appreciate the attempt to transfer the learned interaction to real world scenes, but quality of results in this task is limited and not accompanied by evaluation. Non-car object such as people do not appear to be well controllable and car driving results in unseen environments have rather limited quality. The relationship between each proposed framework components and real-world transfer performance is not shown.\n- Evaluating interactive video generators is challenging. For this reason the paper contains no comparisons to previous baselines. The ablations section however does not present such challenges and is not very extensive. The reader could have been provided with better insights into the operated choices for the Shift-Window Denoising Process Model and Stream Consistency Model hyper parameters, more designs for the Interactive Module (To cite one, can we use discrete actions rather than converting them to text?) and deeper ablations into data (To cite one, how much synthetic data is needed? What is the impact of jointly training with real data? How much does using different video games help real-world transfer?). The significance of the paper is diminished by the lack of deeper quantitative analysis.\n\n\nCLARITY\nThe paper describes the proposed system and data collection pipeline with good clarity. \n\n\nSIGNIFICANCE\n- The paper positions itself under the category of works aiming to produce interactive world simulators. This is a growing area with several previous works as \"GameGAN\", \"Playable Video Generation\", \"Genie\", \"DIAMOND, \"GameNGen\", \"Oasis\" and many concurrent works such as \"Matrix-Game\", \"WorldMem\", \"DeepVerse\". This area of work is likely to become of broader interest in the context of large scale video generation models.\n- The paper presents a recipe for creating an interactive video generation model starting from a pretrained one which can be of interest for the community\n- The paper presents a meaningful procedure for data collection that can be re-used by the community to perform data collection on synthetic video game environments\n- The learned actions focus mostly on camera manipulation (moving the car, egocentric view or the subject). The work focuses on simpler scenarios, e.g. by removing pedestrians and cars when acquiring Cyberpunk 2077 data. This poses a limit on the capabilities the model can develop and reduces significance. This is a challenging aspect across multiple concurrent works.\n\nORIGINALITY\n- Originality of the paper is fair. The problem of building interactive video generator is established. The employed techniques for action injection consist in a cross attention layer with causality, Shift-Window Denoising Process Model adopts the established idea (see \"Diffusion Forcing\") of adopting progressively decreasing time steps per successive frames or blocks of frames, Consistency Models have been employed widely to speedup video generators.\n- The most original component of the paper lies in the attempt of generalizing results to real-world videos. Quality of results is limited in this task and evaluation not present.", "was_revised": true, "final_justification": "During their rebuttal the authors addressed my outstanding concerns through additional experiments, so I update my score to lean towards acceptance.", "rebuttal": "**Dear Reviewer 6pVJ:**\n\nThank you for your insightful feedback! We are encouraged by your appreciation of the significance and quality of our work. Below, we address your concerns in detail.\n\n---\n\n### **Concern 1: Swin-DPM is similar to Diffusion-Forcing**\n\nThank you for raising this point! While Diffusion-Forcing is indeed an impressive approach, there are notable differences between it and our method, Swin-DPM:\n\n- **Length of Generated Videos:** Diffusion-Forcing struggles with the accumulation error of auto-regressive models, making it challenging to generate videos longer than 10 minutes. In contrast, Swin-DPM can generate stable-quality videos over extended periods, such as half an hour, far beyond the capabilities of Diffusion-Forcing.\n\n- **Real-Time Capabilities:** Diffusion-Forcing is not designed for real-time applications and does not focus on optimizing latency, video frame production, and consistency within limited memory, computational resources, and time constraints. Swin-DPM, on the other hand, incorporates techniques to significantly reduce latency and improve real-time video generation.\n\n### **Concern 2: Importance of Data Mixture**\n\nWe agree that data mixture is important for enhancing the generalization ability of the control model. However, we believe that multiple game datasets are not strictly necessary to achieve this. Our approach shows that a well-designed dataset and careful training can lead to strong generalization capabilities, even with a single game dataset. Below we report the results of two different settings, trained with game data only and trained with real-world data fusing. The results demonstrate that training fusing real-world data can improve generalization on real-world scenes.\n\n**Table: Performance Comparison of Training Strategies**\n\n| Training Strategy           | Game Scene Moving-PSNR ↑ | Game Scene Moving-LPIPS ↓ | Real-World Scene Moving-PSNR ↑ | Real-World Scene Moving-LPIPS ↓ |\n|----------------------------|-------------------|--------------------|-------------------------|--------------------------|\n| Game Data Only             | 29.92             | 0.112              | 26.14                   | 0.199                    |\n| Fusing with Real-World Data| 29.90             | 0.113              | 27.79                   | 0.172                    |\n\nIn addition, we report the results of varying the rate at which the action label is replaced with NULL, where the User-Study Accuracy is based on a manual inspection of the generated videos, where we counted how many frames we considered correctly aligned with the control inputs. Please note that these results were reproduced from training on a small subset of the full dataset and may not align with the results presented in the paper. \n\n**Table: Effect of Replace Rate on Evaluation Metrics**\n\n| Replace Rate      | FVD ↓ | Moving-PSNR ↑ | Moving-LPIPS ↓ | User Study ACC ↑ |\n|-------------------|-------|---------------|----------------|------------------|\n| 0.00              | 1344  | 10.42         | 0.600          | 0.89             |\n| 0.05              | 1314  | 10.45         | 0.594          | 0.87             |\n| 0.10              | 1311  | 10.44         | 0.594          | 0.88             |\n| 0.15              | 1318  | 10.44         | 0.594          | 0.86             |\n\n### **Concern 3: Action Representation**\n\nFor the main task, it is not strictly necessary to transfer keyboard inputs into text representations. However, we chose to do so primarily for generalization reasons. The generalization of control in out-of-distribution (OOD) scenarios is achieved by altering the text prompts. The model is pretrained using the DiT model and inherits its understanding of text-based commands. By randomly dropping control signals during training, the model learns to manipulate the movements of real-world objects.\n\nWe hope to expand this in future work by collecting a broader range of interaction types and using language as a unified protocol to understand and merge them into the model. This will enable the model to generate novel interactions independently. We did not implement this idea in the current work due to the substantial workload involved, but we see it as an important avenue for future research. This future direction is also the reason we converted action labels to text.\n\nWe further conducted an experiment to validate the effectiveness of using transfer action labels in the form of one-hot labels instead of text prompts. Due to time constraints during the rebuttal period, the model may not have been fully trained (note that the numerical results may not be aligned with the paper), but the preliminary results suggest that text prompt-style actions outperform one-hot labels across all metrics. Specifically, here User-Study Acc. suggests we manually look into generated videos and count how many percentage of frames we think is correctly aligned with controls.\n\nWe hypothesize the following reasons for this result:\n\n- **Text Prompts Provide Semantically Rich Representations:** Text prompts offer distinct and semantically meaningful representations for actions. In contrast, one-hot labels are less expressive. For example, concepts like \"press W for 1 second, then press S for 1 second\" or \"moving irregularly\" are difficult to capture with one-hot labels but can be easily represented with text-based prompts.\n\n- **Base DiT Model Familiarity:** The base DiT model has been pretrained with text prompt embeddings, and it cooperates well with these embeddings. Since the model is already familiar with text-based representations, it is more effective when using them compared to one-hot labels.\n\n**Table: Comparison of Action Spaces on Evaluation Metrics**\n\n| Action Space | FVD ↓  | Moving-PSNR ↑ | Moving-LPIPS ↓ | User Study ACC ↑ |\n|--------------|--------|---------------|----------------|------------------|\n| one-hot      | 1311   | 10.42         | 0.590          | 0.72             |\n| text         | 1288   | 10.52         | 0.600           | 0.90             |\n\n\n### **Concern 4: Swin-DPM and SCM Parameter Effectiveness**\n\nRegarding Swin-DPM, the main parameter we focus on is the stride size. We have evaluated the effect of this parameter through ablation experiments, which we have detailed below. For SCM, there are relatively few parameters to tune. It uses a default setting of 4 diffusion steps with no conditional guidance (CFG). Could you please clarify which specific parameters you are referring to? This would help us provide more targeted insights on the parameter effectiveness.\n\n**Table: Effect of different strides over VBench Metrics**\n\n| VBench Metrics         | Stride 1 | Stride 2 | Stride 3 | Stride 4 |\n|------------------------|----------|----------|----------|----------|\n| subject_consistency    | 0.862    | 0.873    | 0.882    | 0.885    |\n| background_consistency | 0.900    | 0.911    | 0.914    | 0.917    |\n| imaging_quality        | 0.662    | 0.675    | 0.689    | 0.695    |\n| motion_smoothness      | 0.976    | 0.976    | 0.978    | 0.977    |\n| dynamic_degree         | 0.99     | 1.00     | 1.00     | 1.00     |\n| aesthetic_quality      | 0.527    | 0.533    | 0.530    | 0.528    |\n\n### **Concern 5: How Each Component Affects the Performance of Real-World Transfer**\n\nWe recognize that real-world transfer is an important aspect of our work, and each component of our method contributes in different ways to improving its effectiveness in this regard. We are currently working on additional experiments to further analyze the influence of each component on the real-world transferability. We will include these findings in the revised version of the paper and the rebuttal as soon as possible to provide a more detailed breakdown of how each component affects the performance.\n\n---\n\nWe hope these clarifications address your concerns. Thank you again for your thoughtful feedback and for considering our work."}]

Author rebuttal:

Meta-review: The paper introduces a video generation model capable of producing long, high-fidelity 720p video streams with interactive control at 18-50 fps. The model relies on a novel diffusion technique, coined Shift-Window Denoising Process (Swin-DPM) to generate long high-quality videos. In addition, developing the model required building a video game data collection tool, with which the authors created a dataset from Forza Horizon 5 and Cyberpunk 2077, which the authors plan to release.

The reviewers agreed on the impressive quality and length of the generated videos and appreciated the engineering effort, including data collection, and the commitment to open-source data and parts of their code. But they also had some concerns about lack of evidence of some claims, especially that Swin-DPM might not be key for fast image generation, lack of comparison to baselines, a simplified action space, quality degradation with time, or missing technical details. 
The authors put effort in the rebuttal, performing more experiments, more notably the comparison with Diamond (a relevant baseline) and ablations to understand the impact of Swin-DPM and SCM on inference time, and addressed most of the concerns, but not all. Still all reviewers agreed the merits outperform the limitations and support acceptance.

I agree with the reviewers on the significance, engineering depth, and potential impact as a strong baselines for future research in world models. However, my recommendation is conditional on two factors:
1. The authors must improve the final version with the insights from the discussion during the rebuttal period, especially around the controversy about the claim that Swin-DPM is the key element that contributes to real-time performance as opposed to the optimization of SCM. 
2. The chairs agree there are not ethical concerns on the data collection (see my comments to SAC below). 

Regarding my confidence level: 
* The controversy about the role of Swin-DPM makes me not recommend spotlight, but if the authors address it properly in the final version this could be bumped up. 
* The ethical concern is the main reason for my confidence level on whether the paper should be bumped down to reject.
Decision: Accept (poster)