Title: The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
Venue: NeurIPS 2025
Accepted: yes
Keywords: World Models

Video Generation Models
Abstract: We present The Matrix, a foundational realistic world simulator capable of generating infinitely long 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives. Trained on limited supervised data from video games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains—deserts, grasslands, water bodies, and urban landscapes—in continuous, uncut hour-long sequences. With speeds of up to 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting—an environment present in neither gaming data nor real-world sources. This approach showcases the potential of game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes an interactive video generator capable of: generating videos in real-time, generate long and high-resolution videos, condition generation on actions specified on a frame-by-frame basis, transfer knowledge learned from synthetic video game environments to real-world scenarios.\nTo realize such model, the authors consider a pretrained DiT video generator, introduce \"Interactive Module\" layers to a accept the control signals, Shift-Window Denoising Process Model to generate long videos auto regressively, and Stream Consistency Model to accelerate inference and achieve real-time generation.\nSome quantitative ablations and qualitative evaluations are shown to support the paper claims.\n\nSTRENGTHS AND WEAKNESSES:\nQUALITY\n- The generated driving results are compelling, showing 16fps generation of high resolution videos with fine grained control for lengths up to 15 minutes. This is a difficult result to achieve. Driving results however show limitations in the situations the model can simulate. The model does not seem able to handle properly jumps, with the car appearing to always drive on mostly flat surfaces despite the presence of dunes in the desert environment. The model has little memory for previously generated part of the environment. When the car turns back to a region observed few seconds ago, a new scene is generated instead.\n- I appreciate the attempt to transfer the learned interaction to real world scenes, but quality of results in this task is limited and not accompanied by evaluation. Non-car object such as people do not appear to be well controllable and car driving results in unseen environments have rather limited quality. The relationship between each proposed framework components and real-world transfer performance is not shown.\n- Evaluating interactive video generators is challenging. For this reason the paper contains no comparisons to previous baselines. The ablations section however does not present such challenges and is not very extensive. The reader could have been provided with better insights into the operated choices for the Shift-Window Denoising Process Model and Stream Consistency Model hyper parameters, more designs for the Interactive Module (To cite one, can we use discrete actions rather than converting them to text?) and deeper ablations into data (To cite one, how much synthetic data is needed? What is the impact of jointly training with real data? How much does using different video games help real-world transfer?). The significance of the paper is diminished by the lack of deeper quantitative analysis.\n\n\nCLARITY\nThe paper describes the proposed system and data collection pipeline with good clarity. \n\n\nSIGNIFICANCE\n- The paper positions itself under the category of works aiming to produce interactive world simulators. This is a growing area with several previous works as \"GameGAN\", \"Playable Video Generation\", \"Genie\", \"DIAMOND, \"GameNGen\", \"Oasis\" and many concurrent works such as \"Matrix-Game\", \"WorldMem\", \"DeepVerse\". This area of work is likely to become of broader interest in the context of large scale video generation models.\n- The paper presents a recipe for creating an interactive video generation model starting from a pretrained one which can be of interest for the community\n- The paper presents a meaningful procedure for data collection that can be re-used by the community to perform data collection on synthetic video game environments\n- The learned actions focus mostly on camera manipulation (moving the car, egocentric view or the subject). The work focuses on simpler scenarios, e.g. by removing pedestrians and cars when acquiring Cyberpunk 2077 data. This poses a limit on the capabilities the model can develop and reduces significance. This is a challenging aspect across multiple concurrent works.\n\nORIGINALITY\n- Originality of the paper is fair. The problem of building interactive video generator is established. The employed techniques for action injection consist in a cross attention layer with causality, Shift-Window Denoising Process Model adopts the established idea (see \"Diffusion Forcing\") of adopting progressively decreasing time steps per successive frames or blocks of frames, Consistency Models have been employed widely to speedup video generators.\n- The most original component of the paper lies in the attempt of generalizing results to real-world videos. Quality of results is limited in this task and evaluation not present.\n\nQUESTIONS:\nThe paper is explained clearly and my doubts regarding the work stem could be clarified with more evaluation. Several technical design choices or components of the training recipes as highlighted in previous review points do not find an ablation. I would reserve a higher score for the work if it presented experiments evaluating:\n- Data mixture: How important it is to use real data? How much data is needed? Are multiple videogame data needed to achieve real-world controllability?\n- Action representation: Is it necessary to translate keyboard and mouse actions to textual descriptions?\n- Swin-DPM and SCM: What is the effect of their main hyper parameters on final quality of the model?\n- How each component affects real-world transfer performance: present metrics that link each technical contribution to the amount of real-world transfer performance they are responsible for\n\nLIMITATIONS:\nThe paper presents a limitation section, but does not describe method limitations deeply. The section should mention the issues reported in the review regarding lack of memory, limited environment understanding, very limited ability to perform real-world knowledge transfer and difficulties in handling dynamic objects besides the BMW car.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel video generation model for generating infinitely long videos efficiently with action control. It also proposes a new dataset consisting of labeled game data for training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is easy to follow, and the infinite-long world generation task is well motivated with high application potential.\n2. The proposed source dataset from video games is a promising solution for obtaining high-quality action-labeled video data. I appreciate the engineering effort of the authors to create such a dataset.\n3. The qualitative results are strong and promising, showing the generalizability and various application aspects of the model.\n4. The whole pipeline is sound with a good combination of existing modules and an extensive ablation study.\n\nWeaknesses:\n1. Some technical contributions are a bit overclaimed. The claimed high-quality rendering is the feature of the base video model, and the real-time generation is achieved with a stream consistency model, which is not the contribution of this work.\n2. The current model still lacks some important features for world generation, such as realistic physics, interaction with other objects/characters, a memory mechanism for consistency, and precise movement/camera control. I would still call it a long-horizon video generation model with some vague controlability instead of a world model.\n3. Missing technical details. 1). The warm-up stage in Fig. 2 is not explained in the method section. It seems from the caption that it is trained from \"Synthesized Observations of Unreal Rendered Contextual Environments data\", but later in the training details, the authors \"warm up the base DiT model on unlabeled Source data\", which is \"real-world unlabelled footage\". Which datasource is actually being used for warmup and how is the model trained in this step? 2). In Sec. 4.1, it says more details about the training datasets can be found in Appendix B.3, but this section is missing in the supplementary. I'm particularly curious about how the DROID robotics dataset is used in training.\n4. The action space of the model is oversimplified. The authors map action to very high-level text prompts, sacrificing precise controllability.\n\nQUESTIONS:\n1. Why does the interactive module convert action labels into text prompts instead of directly conditioning on the action for more precise control?\n2. How is the unlabelled real-world/synthetic data from the source dataset constructed? Can the authors provide some examples of these unlabeled data and explain how they differ from the video data used to train the base video model?\n3. Most qualitative results focus on the Forza Horizon variant of the model, with only movement control with \"WASD\". Can the authors provide more results trained on Cyberpunk 2077 and the DROID dataset on camera control/robot joint control? The 4th and 5th row in Fig. 5 are too vague to demonstrate the model's controlability with the Cybepunk data and the 6th row in in Fig 5. doesn't show the control space for the robot grasping data. Also, are these model variants also have strong generatibilty to unseen environments like the  Forza Horizon varaint?\n4. Why does the action space of the Cybepunk data only encode forward motion and the four directions of the camera motion? Will it not work if both the 4 direction camera motion and the 4 direction agent motion are encoded?\n5. Does the model support joint training on all the game data with a uniform action representation, rather than fine-tuning a different variant to accommodate the different action spaces of the different subsets?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"The Matrix,\" a world simulator that generates infinitely long, high-fidelity (720p) video streams with real-time, interactive control. The model is trained on a combination of supervised data from video games and unsupervised real-world footage. Key contributions include the GameData platform for automated data collection, the novel Swin-DPM technique for infinite video generation, and the demonstration of zero-shot generalization to new scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The introduction of the GameData platform is a major strength. It addresses a critical bottleneck in training world models: the prohibitive cost and complexity of acquiring large-scale, action-labeled data.\n\n2. The core technical contribution, the Shift-Window Denoising Process Model (Swin-DPM), presents a novel and well-reasoned approach to extending pre-trained diffusion models for infinite-horizon video generation.\n\n3. The 1-minute video demo and other visual results are highly effective.\n\nWeaknesses\n\n1. The paper presents SCM as a key component in Figure 2, which could be interpreted as a novel part of the proposed architecture. However, the text clarifies that SCM is an adaptation of a prior method used to achieve real-time speeds. This is not a flaw in the method, but a point of potential confusion regarding the paper's specific innovations. I recommend that you clarify the role of SCM in Figure 2 and the main text. For example, you could relabel the component in the figure as \"Acceleration via SCM\" to explicitly attribute the technique and frame it as a crucial optimization step rather than a core generative model contribution. This would improve the clarity of the paper's originality.\n\n2. The paper claims real-time generation (16 FPS), but this performance is contingent on the SCM. The core generative model, Swin-DPM, appears to have significant efficiency limitations due to its auto-regressive nature. Based on the evidence you provide in Table 1, The + Swin-DPM model runs at only 0.8 FPS. The jump to 16 FPS only occurs after applying SCM, which also degrades visual quality (FVD increases from 1651.50 to 1936.79). Please acknowledge that the core Swin-DPM model is not real-time by itself and that the real-time capability is achieved through a distillation process (SCM) that trades some visual fidelity for speed. This would provide a more complete and transparent assessment of the method's performance.\n\n3. Hyperparameter Sensitivity (Line 126): During training, you replace labeled inputs with a default description with probability q=0.1. How sensitive is the model's control precision (e.g., Move-PSNR) to this value? Providing a brief ablation study or sensitivity analysis for q would help justify this choice and improve the work's reproducibility.\n\n4. Stride Choice in Swin-DPM (Line 143): The paper states that Swin-DPM generates tokens with a stride of s=1. Have you experimented with larger strides (e.g., s > 1)? While a larger stride might risk temporal inconsistency, it could also improve the baseline efficiency of the model before SCM. A brief discussion on why s=1 was chosen would be valuable.\n\n5. The manuscript would benefit from a thorough proofread to correct several small typographical errors (e.g., in the caption of Figure 2, \"Swin-DPMenables\" and \"Modelis\").\n\nQUESTIONS:\nQuestions：\n1. In line126, how much does the value of the probability q affect the performance of the model?\n\n2. Have you tried different window stride besides s=1?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n*The Matrix* is a real-time world simulator that generates infinitely long, high-fidelity 720p video streams with frame-level control at 16 FPS. Trained on a mix of labeled game data and unlabeled real-world footage, it generalizes to unseen environments with zero-shot capability. Key innovations include the Swin-DPM for infinite video generation, an Interactive Module for user input integration, and SCM for real-time performance. A new dataset, Source, supports training with paired action-video data. The system demonstrates strong visual quality, precise control, and scalability, paving the way for AI-driven simulations in gaming, robotics, and virtual environments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Generates high-quality videos at real-time speed.\n- Substantial effort is demonstrated, including platform development, data collection, and novel model design and training.\n- The module design is intuitive and effective.\n- The paper is well-written.\n\n**Weaknesses**\n\n- The core task is questionable. The system uses significant resources to generate immersive virtual worlds that could be more efficiently produced using traditional 3D game engines.\n- The visual diversity of the generated worlds appears limited by the training data.\n- There is little attention to world consistency or coherence—generated environments tend to be forgotten after just a few seconds.\n\nQUESTIONS:\n- *The Matrix* shows impressive long-form generation (e.g., the 15-minute desert video). However, will the quality degrade over time in more complex scenarios like urban scenes? Quantitative evaluations would help assess long-term fidelity.\n- Could you provide details on the human effort involved in data annotation—such as the number of annotators and total time spent? This would help assess the cost of transferring the system to new games.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a diffusion-based interactive video generation / simulation model (one in which the generated visuals reflect user text prompts and respond to user game inputs) while incorporating a novel “Shift-Window Denoising Process Model (Swin-DPM)” to enable infinite length video generation. They also introduce the “GameData” platform (used to generate the “Source” dataset) which captures paired game state and video frames. The authors boast that their method has improved rendering resolution and speeds versus existing methods (lines 58-64) though the experimental quantitative proof of this is somewhat lacking.\n\nThe Matrix diffusion model pipeline is represented in Figure 2 and is composed of: \n\n1) A pre-trained Video DiT backbone [14] (which the authors note in section 4.3 makes their method more robust to out-of-distribution scenarios), \n2) An Interactive Module responsible for responsible for “translating keyboard inputs into natural language that guides video generation” (described on page 4), \n3) A novel “Shift-Window Denoising Process Model (Swin-DPM)” (described on page 5 and Figure 4) where a shifting time sequence of video tokens is denoised simultaneously. After each token starting the queue is fully denoised the time-series advances with a new noisy input being added.\n4) A Stream Consistency Model (based on [27]) used to accelerate denoising although is sparingly described aside from a reference to existing work being given.\n\nThe GameData platform to autonomously collect data of paired video and game state is described on lines 172-181 where the authors use a number of open-source tools to capture game state, remove user interface (UI) elements, and capture paired video frames.\n\n__Novelty and Contributions:__\n- Diffusion interactive video generation model with improved visual fidelity and rendering speeds versus existing methods.\n- Propose a novel \"Shift-Window Denoising Process Model (Swin-DPM)\" for infinite video generation.\n- Describe their GameData platform for autonomous collection of training data.\n\n__Experiments:__\n\nExperiments display qualitative results in terms of long-form video generation with movement controls (Figure 6), generation of unlabeled scenes with movement control (Figure 7 a), and pure video generation without user interactivity (Figure 7 b). Quantitative results for an ablation study of The Matrix’s components are shown in Table 1 where the authors note a general trade-off between rendering speed, rendering quality and control precision.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths:__\n\n- __State of the art (significance):__ The authors boast state of the art virtual world generation accommodating textual prompts and user inputs which the authors assert possess greater visual fidelity and frame rate compared to previous methods. They also assert that their method has better long term video generation capabilities. However, it would have been nice to see a more quantitative comparison to baselines (see weaknesses section).\n- __Novel Method (originality):__ Approach using diffusion models with a novel Shift-Window Denoising Process Model (Swin-DPM) to facilitate infinite video generation.\n- __Manuscript Clarity:__ I feel that clarity, significance and originality were well done though experimental quality related to benchmarks could be somewhat improved (see weaknesses below).\n\n__Weaknesses:__\n\n- __Quantitative experimental analysis (quality):__ The authors note that “Since existing world models are not trained on Forza Horizon 5 or Cyberpunk 2077, it is unfair to compare The Matrix against prior world models both qualitatively and quantitatively.” (lines 183-184). Is it not possible to retrain one/some of these models on your data to obtain a more quantitative comparison to baselines? I currently feel that this work is weakened without a substantive comparison to baselines (experimental quality could be improved). The authors claim advances in rendering speeds and resolution compared to existing methods, though some quantitative experimental results to support this claim would of been appreciated.\n\n- While the method can supposedly generate “infinite-length videos”, I would of found it insightful if there were some metrics to quantify/show the degradation (or lack of) of quality between the start to end frames of the generated video sequence.\n\n- __Computational limitations (significance):__ As the authors note in the Limitations section (lines 264-268), the model currently runs on 8xA100 GPUs which makes its usage prohibitive.\n\n__Current Assessment and Suggestions for Improvement:__\n\nI have currently marked the work as borderline reject. I feel that the submission currently relies too heavily on qualitative comparisons to existing visual generation methods instead of quantitative testing. The authors assert that quantitative testing versus existing methods is not possible due to these models being trained on different data. Can further justification be given by the authors as to why retraining with the existing dataset is not possible (see questions section below).\n\n__Minor Points:__\n- In case these have not yet been resolved, there are several typos related to missing spaces: 1) In Figure 1: “Matrixis”, 2) In Figure 2: “Modulefor” and “Modelis”, 3) on line 221 “Modulesignificantly”\n\n- I find the wording on line 65 to be somewhat odd “While we do not claim the Matrix to be a foundational world model” when the approach is described as “foundational” several times earlier (line 1, Figure 1, line 46).\n\n- My PDF viewer (Adobe Acrobat version 2025.001.20531) failed to be able to view the embedded videos and reported the error “A 3D data parsing error has occurred.”\n\nQUESTIONS:\n- Can the authors elaborate on why retraining an existing baseline method with their dataset may not be feasible (lines 183-184) to provide a more quantitative comparison to benchmarks?\n\nLIMITATIONS:\nYes. Limitations are listed by the authors on lines 264-268.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}