GameGen-X: Interactive Open-world Game Video Generation
Authors: Haoxuan Che
Xuanhua He
Quande Liu
Cheng Jin
Hao Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Open-world Game Video Generation
Interactive Control
Diffusion Transformers
Primary area: generative models
TL;DR: We introduce GameGen-$\mathbb{X}$, the first diffusion transformer model tailored for the generation and controllable interaction of open-world game videos, unifying multi-modal game-related control signals.

Abstract: We introduce GameGen-$\mathbb{X}$, the first diffusion transformer model specifically designed for both generating and interactively controlling open-world game videos. 
    This model facilitates high-quality, open-domain generation by approximating various game elements, such as innovative characters, dynamic environments, complex actions, and diverse events. 
    Additionally, it provides interactive controllability, predicting and altering future content based on the current clip, thus allowing for gameplay simulation.
    To realize this vision, we first collected and built an Open-World Video Game Dataset (OGameData) from scratch. 
    It is the first and largest dataset for open-world game video generation and control, which comprises over one million diverse gameplay video clips with informative captions.
    GameGen-$\mathbb{X}$ undergoes a two-stage training process, consisting of pre-training and instruction tuning. 
    Firstly, the model was pre-trained via text-to-video generation and video continuation, enabling long-sequence open-domain game video generation with improved fidelity and coherence.
    Further, to achieve interactive controllability, we designed InstructNet to incorporate game-related multi-modal control signal experts.
    This allows the model to adjust latent representations based on user inputs, advancing the integration of character interaction and scene content control in video generation.
    During instruction tuning, only the InstructNet is updated while the pre-trained foundation model is frozen, enabling the integration of interactive controllability without loss of diversity and quality of generated content. 
    GameGen-$\mathbb{X}$ contributes to advancements in open-world game design using generative models. 
    It demonstrates the potential of generative models to serve as auxiliary tools to traditional rendering techniques, demonstrating the potential for merging creative generation with interactive capabilities.
    The project will be available at https://github.com/GameGen-X/GameGen-X.

Published as a conference paper at ICLR 2025
GAMEGEN-X X: INTERACTIVE OPEN-WORLD GAME VIDEO GENERATION
Haoxuan Che Xuanhua He2.3+ Quande Liu Cheng Jin', Hao Chen 'The Hong Kong University of Science and Technology
University of Science and Technology of China
Hefei Institute of Physical Science, Chinese Academy of Sciences "The Chinese University of Hong Kong (hche, cjinag, jhc]0cse. ust hexuanhua @ma 11, ustc. edu. cn qdliu022 60 geauul226@@mmi.... com
ABSTRACT
We introduce GameGen-X, the first diffusion transformer model specifically de- signed for both generating and interactively controlling open-world game videos. This model facilitates high-quality, open-domain generation by approximating various game elements, such innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive control- lability, predicting and altering future content based the current clip, thus al- lowing for gameplay simulation. To realize this vision, we first collected and built Open-World Video Game Dataset (OGameData) from scratch. is the first and largest dataset for open-world game video generation and control, which com- prises over one million diverse gameplay video clips with informative captions. GameGen-X undergoes a two-stage training process, consisting of pre-training and instruction tuning. Firstly, the model was pre-trained via text-to-video gen- eration and video continuation, enabling long-sequence open-domain game video generation with improved fidelity and coherence. Further, to achieve interactive controllability, designed InstructNet incorporate game-related multi-modal control signal experts. This allows the model to adjust latent representations based user inputs, advancing the integration of character interaction and scene con- tent control in video generation. During instruction tuning, only the InstructNet is updated while the pre-trained foundation model is frozen, enabling the integra- tion of interactive controllability without loss of diversity and quality of generated content. GameGen-X XX contributes advancements open- woold game design us- ing generative models. It demonstrates the potential of generative models to serve as auxiliary tools to traditional rendering techniques, demonstrating the potential for merging creative generation with interactive capabilities. The project will be available / cow/GameGen X/ X/GameGen-
INTRODUCTION
Generative models (Croitoru et al. (2033): Ramesh et al. (2022); Tim Brooks & Ramesh (2024); Rombach et al. (2022b)) have made remarkable progress in generating images or videos conditioned on multi-modal inputs such as text, images, and videos. These advancements have benefited content creation in design, advertising, animation, and film by reducing costs and effort. Inspired by the success generative models in these creative fields, modern game industry. This exploration is particularly important because developing open-world video game prototypes is a resource resource-intension intensive and costly endeavor, requiring substantial investment in concept design, asset creation, programming, and preliminary testing (Anastasial (2023)). Even early development stages of games still involved months of intensive work by small teams to build functional prototypes showcasing the game's potential (Wikipedia (2023)).
natural to explore their application in the
Equaal contribution; "Co-corresponding authors.

---

Published as a conference paper at ICLR 2025
GameGen-X can generate novel open-world video games and enable interactive control to simulate game playing. Best view with Acrobat Reader and click the image to play the interactive control demo videos.
Figure 1:
Several pioneering works, such as World Model (Ha & Schmidhuber (018)), GameGAN (Kim eet al. )2000), R2PLAY (Jin et al. (2024)), Genie (Bruce al. (2024)). and GameNGen (Valevski cct al. 2024)). have explored the potential of neural models to simulate or play video games. They have primarily focused on 2D games like "Pac-Man", "Super Mario", and early 3D games such as "DOOM (1993)". Impressively, they demonstrated the feasibility of simulating interactive game en- vironments. However, the generation of novel, complex open-word open-world game content remains open problem. A key difficulty lies in generating novel and coherent next-gen game content. Open-world games feature intricate environments, dynamic events, diverse characters, and complex actions that are far more challenging to generate (Eberly (2006)). Further, ensuring interactive controllability, where the generated content responds meaningfully to user inputs, remains a formidable challenge. Addressing these challenges is crucial for advancing the use of generative models in game content design and development. Moreover, successfully simulating and generating these games would also be meaningful for generative models, as they strive for highly realistic environments and interac- tions, which in turn may approach real-world simulation (Zhu et al. (2024). ).
In this work, provide an initial answer to the question: Can diffusioon model generate and con- trol high-quality, complex open-world video game content? Specifically, we introduce GameGen-2 XX, the first diffusion transformer model capable of both generating and simulating open-world video games with interactive control. GameGen-XX sets a new benchmark by excelling at generating diverse and creative game content, including dynamic environments, varied characters, engaging events, and complex actions. Moreover, GameGie-XX enables interactive control within generative models, al- lowing users influence the generated content and unifying character interaction and scene content control for the first time. It initially generates a video clip to set up the environment and charac- ters. Subsequently, produces video clips that dynamically respond to user inputs by leveraging the current video clip and multimodal user control signals. This process can be seen as simulating game-like experience where both the environment and characters evolve dynamically.
GameGen-X X undergoes two-stage training: foundation model pre-training and instruction tuning. In the first stage, the foundation model is pre-trained on OGameData using text-to-video generation and video continuation tasks. This enables the model to learn broad range of open-world game dynamics and generate high-quality game content. In the second stage, InstructNet is designed to enable multi-modal interactive controllability. The foundation model is frozen, and InstructNet is trained to map user inputs-suu as structured text instructions for game environment dynamics and keyboard controls for character movements and actions onto the generated game content. This al- lows GameGen-XX to generate coherent and controllable video clips that evolve based on the player's inputs, simulating an interactive gaming experience. To facilitate this development, we constructed
2

---

Published as a conference paper at ICLR 2025
Video-level Filtering Y
Scene
Collection
TranaNetv2
(32,000 vidoo covering 1504 non game conten)
(Non-paayblle contents)
Structured Annotation
(Vico clipe containing cantein sowne]
Summarrp
Annotating
Meta>
<Character>
Score 2
<Amossperee>
Threshold
Colrecherd
Hango
<lavo
Score Threshold
<Trant>
<Light>
cAst>
2.2
aborter
VideoCUP
CLI-WA
Figure 2: The OGameData collection and processing pipeline with human-in -nt-iin-ttthoo....
Open-World Video Game Dataset (OGameData), the first large-scale dataset for game video gen- eration and control. This dataset contained videos from over 150 next-generation games and was built by using human-in-the-tt in-the-loop proprietary data pipeline that involves scoring, filtering, sorting, and structural captioning. OGameData contains one million video clips from two subsets including OGameData-GiBe and OGameDDta-INNS, providing the foundation for training generative models capable of producing realistic game content and achieving interactive control, respectively.
In summary, GameGen-X offers a novel approach for interactive open-world game video generation, where complex game content is generated and controlled interactively. While challenges for prac- tical application remain, it explores the potential for generative models to complement traditional game design methods by offering new approach to scalable content generation. Our main contri- butions summarized follows: 1) We developed OGameData, large-scale dataset curated for open-world game video generation and interactive control, containing one million video-text pairs. It collected from 150+ next-gen games, and empowered by vision language models. 2) We in- troduced GameGin-XX, the first generative model for open-world video game content, combining a foundation model with the InstructNet. GameGen-X utilizes two-stage training strategy, with the foundation model and InstructNet trained separately to ensure stable, high-quality content genera- tion and control. InstructNet provides multi-modal interactive controllability, allowing players to influence the continuation of generated content, simulating gameplay. 3) We conducted extensive experiments comparing our model's s generative and interactive control abilities to other open-source and commercial models. Results show that our approach excels in high-quality game content gener- ation and offers superior control over the environment and character. Experimental results indicate that approach achieves strong performance in game content generation and provides enhanced control over the environment and character compared to other models.
2 OGAMEDATA: LARGE-SCALE FINE-GRAINED GAME DOMAIN DATASET
OGameData is the first dataset designed open-world game video generation and interactive control. As shown in Table OGameData excels in fine-grained annotations, offering structural caption with high text density for video clips per minute.
meticulously designed for game video by offering game-specific knowledge and incorporating clements such game names, player perspectives, and character details. It comprises two parts: the generation dataset (OGameData- GEN) and the instruction dataset (OGameData-INS The resulting OGameData-GEB is tailored for training the generative foundation model, while OGameData- is optimized for instruction tuning and interactive control tasks. The details and analysis are in Appendix
2.1 DATASET CONSTRUCTION PIPELINE
As illustrated Figure 2 we developed a robust data processing pipeline encompassing collec- tion, cleaning, segmentation, filtering, and structured caption annotation. This process integrates
3

---

Published as a conference paper at ICLR 2025
both AI and human expertise, automated techniques alone are insufficient due to domain specific intricacies present in various games.
Data Collection and Filtering, We gathered video from the Internet, local game engines, and exist- ing dataset (Chen et al. (2024); Du et al. (2024)), which contain more than 150 next-gen games and game engine direct outputs. These data specifically focus gameplay footage that mini- mizes UI elements. Despite the rigorous collection, some low-quality videos were included, and these videos lacked essential metadata like game name, genre, and player perspective. Low-quality videos were manually filtered out, with human experts ensuring the integrity of the metadata, such as game genre and player perspective. To prepare videos for clip segmentation, we used PyScene and Vet2 (Souček & Lokoë] (2020)) to detect scene changes, discarding clips shorter than seconds and splitting longer clips into 16-second segments. To filter and annotate clips, we se- quentially employed models: CLIP-AVA (Schuhmann (2023) for aesthetic scoring, UniMatch (Xu al. (2023)) for motion filtering, VideoCLIIP (Xu et (2021) for content similarity, and CoTrack- v2 KKraev et al. (2023)) for camera motion.
Structured Text Captioning The OGameData supports the training of two key functionalities: text-to-video generation and interactive control. These tasks require distinct captioning strategies. For OGameData-GENN, detailed captions are crafted to describe the game metadata, scene context, and key characters, ensuring comprehensive textual descriptions for the generative model founda- tion training. In contrast, OGameData-INS focuses on describing the changes in game scenes for interactive generation, using concise instruction- -based captions that highlight differences between initial and subsequent frames. This structured captioning approach enables precise and fine-grained generation and control, allowing the model to modify specific elements while preserving the scene.
Table 1: Comparison of OGameData and previous large-scale text-video paired datasets.
Dateset
Domain
Tedovidoo pairs
Caption density
Caprioner
Reselation
Parpose
Total video Ion.
Activiyyee Caba Heilhron et al. 2015
Actica
KSK
23 words'mnn
Manual
Understanding
845h:
DiDaMe A Femtings e
Flickr
45k
70 worddmmin
Monual
Temporal el localisation
87t.
YouCwik Zo el 2015)
Cooking
32k
26 wordiemin
Manual
Understanding
176h
How2 Sunderss 2009
Instruct
191k
207 wordstman
Manual
Understanding
305B
Miralai pira
Open
350k
264 wordsmmin
GPT-4V
720P
Gemention
16000h
OtiumeData
Game
1000k
607
words/min
GPT144
720P-48
Generation & Coatrol
2.2 DATASET SUMMARY
As depicted in Table ! OGameData comprises 1 million high-resolution video clips, derived from sources spanning minutes hours. Compared to other domain domain-specific datasets (Caba Heilbron iet al. (2015); Zhou et al. (2018); (Sanabria et al. (2018); Anne Hendricks et al.) (2017). OGame Data stands out for its scale, diversity, and richness of text-video pairs. Unlike the latest open-domain generation dataset Miradata (Ju et (024)). OGameData focuses on the game domain and provides more fine-graing greigrra annotations, featuring extensive captions per unit of time. This is crucial for training interactive controllability. This dataset features several key characteristics: OGa features highly fine-grained text and boasts a large number of trainable video-text pairs, enhancing text-video alignment in model training. Additionally, it comprises two subsets generation and control supporting both types of training tasks. The dataset's high quality is ensured by meticulous curation from over 10 human experts. Each video clip is accompanied by captions generated using vision language models, maintaining clarity and coherence and ensuring the dataset remains free of UI and visual artifacts. Critical to its design, OGameData is tailored specifically for the gaming domain. It effectively excludes -ongpaayylay scenes, incorporating a diverse array of game styles while preserving authentic in-game camera perspectives. This specialization ensures the dataset represents real gaming experiences, maintaining high Nomain-specifie relevance.
[Sanaraaanabria
3 GAMEGEN-X
GameGen-X: X is a novel generative diffusion model that learns to generate open-world game videos and interactively control the environments and characters in them. The overall framework is illus- trated in Fig 3 In section 3.1 we introduce the problem formulation. In section 3.2, we discuss the design and training of the foundation model, which facilitates both initial game content genera- tion and video continuation. In section 3.3] we delve into the design of InstructNet and explain the process of instruction tuning, which enables clip-level interactive control over generated content.
4

---

Published as a conference paper at ICLR 2025
Foundation Model Pretralning: Pretraining foundotion Model with OGomerDoto-GFN Dotaset
Protrain Curation
Protraining
For
MAde Age
Opp
Nooie
holes
loss
Notre
Hiniy
Text Condition
Vack Foundation
Aischeed
Tee
Model
OSancDotp GEN Betsee
Instruction Cip Autoregression Finetuning DGomeDeto INS
Multi modal Instruction Formulation
Instruction Turing
F
DGameDuto-iNW5
Video Clips
Kitto Prompts
Ciips
Structured Instructions
Kryboard Bindings
Foundation
Model
Rtybeet Stactiood
Erender
InstructNet.
Maider Viaties
Provetts
Figure 3: An overview of our two-stage training framework. In the first stage, we train the foundation model via OGameData-GGEN In the second stage, InstructNet is trained via OGameData INS.
3.1 GAME VIDEO GENERATION AND INTERACTION
The primary objective of GameCie-XX X is to generate dynamic game content where both the virtual environment and characters are synthesized from textual descriptions, and users can further influence the generated content through interactive controls. Given textual description I' that specifies the initial game scene-including characters, environments, and corresponding actions and events-we aim to generate a video sequence V {Vi) that brings this scene to life. We model the condi- tional distribution: p(V1. T, CI:N), where C1:N represents the sequence of multi-modal control inputs provided by the user over time. These control inputs allow users to manipulate character movements and scene dynamics, simulating an interactive gaming experience.
It generates initial video clip based on T, capturing the specified game elements including characters and environments. 2) InstructNet: It enables the controllable continuation of the video clip by incorporating user- provided control inputs. By unifying text-to-video generation with interactive controllable video continuation, our approach synthesizes game-like video sequences where the content evolves in response to user interactions. Users can influence the generated video at each generation step by providing control inputs, allowing for manipulation of the narrative and visual aspects of the content.
Our approach integrates
main components: 1) Foundation Model:
3.2 FOUNDATION MODEL TRAINING FOR GENERATION
Video Clip Compression. To address the redundancy in temporal and spatial information (Lab & etc. (2024)), introduce 3D Spatio-Temporee Variational Autoencoder (3D-VAB) to compress video clips into latent representations. This compression enables efficient training on high-resolution videos with longer frame sequences, Let V E Fxxxxx denote video clip, where F is the num- ber of frames, H and W are the height and width of each frame, and C' is the number of channels. The encoder E' compresses V into a latent representation = E(V) E FB "0'xC"xXa x xw" where F' F/sj. H' H/SK, W' W/sg. and C' is the number of latent channels. Here, St, 8h, and Sw are the temporal and spatial downsampling factors. Specifically, 3D-VAE first performs the spatial downsampling to obtain frame-level latent features. Further. it conducts temporal com- pression to capture temporal dependencies and reduce redundancy over frame effectively, inspired by Yu et al. (2023a). By processing the video clip through the 3D-VAF, we can obtain latent tensor of spatial-temporalll informative and reduced dimensions. Such æ can support long video and high-resolution model training, which meets the requirements of game content generation.
Masked Spatial-Temporal Diffusion Transformer. GameGen-X introduces a Masked Spatial- Temporal Diffusion Transformer (MSDiT). Specifically, MSDiT combines spatial attention, tempo- ral attention, and cross-attention mechanisms (Vaswani (2017)) ) to effectively generate game videos guided by text prompts. For each time step f, the model processes latent features 2, that capture frame details. Spatial attention enhances intra-frame relationships by applying self-attention
5

---

Published as a conference paper at ICLR 2025
spatial dimensions (H', W"). Temporal attention ensures coherence across frames by operating over the time dimension F", capturing inter-frame dependencies. Cross-attention integrates guid- ance of external text features f obtained via T5 (Raffel et al.] (2020a)), aligning video generation with the semantic information from text prompts. As shown in Fig. 4 we adopt the design of stack- ing paired spatial and temporal blocks, where each block is equipped with cross-attention and one of spatial or temporal attention. Such design allows the model to capture spatial details, tempo- ral dynamics, and textual guidance simultancously, enabling GameGen-X to generate high-fidelity, temporally consistent videos that are closely aligned with the provided text prompts.
Additionally, we introduce a masking mechanism that excludes certain frames from noise addition and denoising during diffusion processing. A masking function (i) over frame indices i E defined as: (i) 1 if i > x. and (i) O if S x. where T is the number of context frames provided for video continuation. The noisy latent representation at time step t is computed (TT) where 6 ~ (O0 I) Gaussian noise of the same dimensions as and @ denotes element- wise multiplication. Such masking strategy provides the support of training both text-to-video and video continuation into one foundation model.
as: & = (1-
Unified Video Generation and Continuation. By integrating the texi-to-video diffusion train- ing logic with the masking mechanism, GameGen-X effectively handles both video generation and continuation tasks within unified framework. This strategy aims to enhance the simulation experi- ence by enabling temporal continuity, catering to an extended and immersive gameplay experience. Specifically, for rext-to-video generation, where initial frames are provided, we set = 0, and the masking function becomes (i) = 1 for all frames i. The model learns the conditional dis- tribution pVV [T), where T is the text prompt. The diffusion process is applied to all frames, and the model generates video content solely based on the text prompt. For video continuation, initial frames V1. are provided as context. The masking mechanism ensures that these frames remain un- changed during the diffusion process, as (i) 0 for i S x. The model focuses on generating the subsequent frames 1n++1:N by learning the conditional distribution p(U3+1:N |M:.: 7). This allows the model to produce video continuations that are consistent with both the preceding context and the text prompt. Additionally, during the diffusion training (Song et al. (2020a b); Ho et al. (2020); Rombach et al. (2022a)) we incorporated the bucket training (Zheng et al. (0024b), classifier- free diffusion guidance (Ho & Salimans (2021) ) and rectified flow (Liu et al. (023b)) for better genera- tion performance. Overall, this unified training approach enhances the ability to generate complex, contextually relevant open open-world game videos while ensuring smooth transitions and continuations.
3.3 INSTRUCTION TUNING FOR INTERACTIVE CONTROL
InstructNet Design. To enable interactive controllability in video generation, we propose Instruct- Net, designed to guide the foundation model's predictions based on user inputs, allowing for control of the generated content. The core concept is that the generation capability is provided by the foundation model, with InstructNet subtly adjusting the predicted content using user input signals. Given the high requirement for visual continuity in-game content, our approach aims to minimize abrupt changes, ensuring a seamless experience. Specifically, the primary purpose of InstructNet is to modify future predictions based instructions. When no user input signal is given, the video extends naturally. Therefore, we keep the parameters of the pre-trained foundation model frozen, which preserves its inherent generation and continuation abilities. Meanwhile, the additional train- able InstructNet is introduced to handle control signals. As shown in Fig. 4 InstructNet modifies the generation process by incorporating control signals via the operation fusion expert layer and instruction fusion expert layer. This component comprises N InstructNet blocks, each utilizing a specialized Operation Fusion Expert Layer and an Instruct Fusion Expert Layer to integrate differ- ent conditions. The output features are injected into the foundation model to fuse the original latent, modulating the latent representations based on user inputs and effectively aligning the output with user intent. This enables users to influence character movements and scene dynamics. InstructNet is primarily trained through video continuation to simulate the control and feedback mechanism in gameplay. Further, Gaussian noise is subtly added to initial frames to mitigate error accumulation.
Multi-modal Experts. Our approach leverages multi-modal experts to handle diverse controls, which is crucial for several reasons. Intuitively, each structured text, keyboard binding, and video prompt-unigu impacts the video prediction process, requiring specialized handling to fully cap- ture their distinct characteristics. By employing multi-modal experts, can effectively integrate
6

---

Published as a conference paper at ICLR 2025
Timestep/
MLP
Video Clip
1pood jelodta
|aindung
spoia
(URN
JaNeT
Video DiT Foundation Model
Generate
Latent Modification
Video
InstructNet
Video Prompt
Latent
Block
InstructNet
InstructNct Block
Fined
notse
Block
75
Mutti-endii
Expert
Godi ig Mood oniom
Lewer
Spuraal
Puseer
Fusiio
FFN Layer
+
Terporsl Sod atts.
InstructNet
Figure 4:
The architecture of GameGen-X, including the foundation model and InstructNet. It enables the latent modification based on user inputs, mainly instruction and operation prompts, allowing for interactive control over the video generation process.
these varied inputs, ensuring that each control signal well utilized. Let fi and Jo be structured instruction embedding and keyboard imput embedding, respectively. fo is used to modulate the la- tent features via operation fusion expert as follows: & = r(fo @ 3ー + 3(fo), where and a are the mean and standard deviation of 2, (Joo and ((fo) scale and shift parameters predicted by a neural network conditioned on c, where e includes both structured text instructions and keyboard inputs. and @ denotes element-wise multiplication. The keyboard signal primarily influences video motion direction and character control, exerting minimal impact on scene content. Consequently, lightweight feature scaling and shifting approach is sufficient effectively process this informa- tion. The instruction text is primarily responsible for controlling complex scene elements such as the environment and lighting. To incorporate this text information into InstructNet. we utilize an instruction fusion expert, which integrates fi into the model through cross-attention. Video prompts Vp. such as canny edges, motion vectors, or pose sequences, are introduced to provide auxiliary in- formation. These prompts are processed through the 3D-VAE encoder to extract features ep. which are then incorporated into the InstructNet via addition with z. It's worth clarifying that, during the inference, these video prompts or video editing.
necessary, except to execute the complex action generation
Interactive Control. Interactive control is achieved through autoregressive generation process. Based on a sequence past frames V1.2. the model generates future frames +x+ while adhering control signals. The overall objective is model the conditional distribution: p(U++1:N U1:r, c)- During generation, the foundation model predicts future latent representations, and InstructNet mod- ifies these predictions according the control signals. Thus, users can influence the video's pro- gression by providing structured text commands or keyboard inputs, enabling a high degree of con- trollability in the open-world game environment. Furthermore, the incorporation of video prompts Vp provides additional guidance, making it possible to edit or stylize videos quickly, which is par- ticularly useful for specific motion patterns.
EXPERIMENTS
4.1 QUANTITATIVE RESULTS
Metrics. To comprehensively evaluate the performance of GameGie-XX, we utilize a suite of metrics that capture various aspects of video generation quality and interactive control, follow- ing Huang et al. 2024b) and Yang et al. (0024). These metrics include Fréchet Inception Distance (FID), Fréchet Video Distance (FVD), Text- Video Alignment (TVA), User Preference (UP), Motion Smoothness (MS), Dynamic Degrees (DD), Subject Consistency (SC), and Imaging Quality (IQ). It's worth noting that the TVA and UP subjective scores that indicate whether the generation meets the requirements humans, following Yang et al. (2024). By employing this comprehensive set of metries, we aim thoroughly evaluate model capabilities in generating high-quality, realistic,
7

---

Published as a conference paper at ICLR 2025
and interactively controllable video game content. Readers can find experimental settings and metric introductions in Appendix D.2
Table 2: Generation Performance Evaluation (* denotes key metric for generation ability)
Method
Resolution
Frumes
FID
FVD*
TVA*1
UPo1 [
MS
DD1
SC
1
Mira (hang et al
2023)
480p
60
360.9
2254,2
0.27
0.25
0.98
0.62
0.94
0.63
OpenSara-Plan Cah
2024
720p
102
407.0
1940,9
0.38
0.43
0.99
0.42
0.92
0.39
Cog VigVideoX555 5D Yang el a
4024
480p
49
316.9
1310.2
0.49
0.37
0.99
0.94
0.92
0.53
OpenSoral. 2 Zhogg et
720p
102
318.1
1016.3
0.50
0.37
0.98
0.90
0.87
0.52
GameGen-X X (Ours)
720p
102
252.1
759.8
0.87
0.82
0.99
0.80
0.94
0.50
Table 3: Control Performance Evaluation (* denotes key metric for control ability)
Method
Resolution
Frames
SR-C*
SR-E*" 1
UP 1
MS 1
DD1 1
SC
1Q 1
OpenSora- Plann Lab etc. 2024
720p
102
26.6%
31.7%
0.46
0.99
0.72
0.90
0.51
CogVideoX SB Yang el A724
480p
49
23.0%
30.3%
0.45
0.98
0.63
0.85
0.55
OpenSoral Zheng el 20246
720p
102
21.6%
14.2%
0.17
0.99
0.97
0.45
GameGen-X (Ours)
720p
102
63.0%
56.8%
0.71
0.99
0.88
0.88
0.44
Generation and Control Ability Evaluation. As shown in Table 2 we compared GameGen-
X against four well-known open-source models, i.e., Mira (Zhang et al.] (2023)), OpenSora- (Yang et al. Plan1.2 1.2 (Lab & etc. (2024)), OpenSoral.2 (Zheng et al. (2024b)) and CogVideoX
(2024) to evaluate its generation capabilities, Notably, both Mira and OpenSoral .2 explicitly men-
tion training on game data, while the other two models, although not specifically designed for this purpose, can still fulfill certain generation needs within similar contexts. Our evaluation showed that GameGen-X performed well on metrics such as FID, FVD, TVA, MS, and SC. It implies GameGen- X's strengths in generating high-quality and coherent video game content while maintaining com- petitive visual and technical quality. Further, we investigated the control ability of these models, except Mira, which does not support video continuation, as shown in Table 3 We used conditioned video clips and dense prompts to evaluate the model generation response. For GameGen-X: we em- ployed instruct prompts to generate video clips. Beyond the aforementioned metrics, we introduced the Success Rate (SR) to measure how often the models respond accurately to control signals. This is evaluated by both human experts and PLLa VA (Xu et al. (2024).. The SR metric is divided into two parts: SR for Character Actions (SR-C), which assesses the model's responsiveness to charac- ter movements, and SR for Environment Events (SR-E), which evaluates the model's s handling of changes in weather, lighting, and objects. As demonstrated, GameGen-2 X exhibits superior control ability compared to other models, highlighting its effectiveness in generating contextually appropri- ate and interactive game content. Since IQ metrics favor models trained on natural scene datasets, such models score higher. In generation performance, CogVideo's Video's 's 8fps videos and OpenSora 1.2's frequent scene changes result in higher DD.
Table Ablation Study for Generation Ability
Table 5: Ablation Study for Control Ability.
Method
FID
FVD
TVA
UP
MS 1 SC
Method
SR-C7
SR-E
UP?
M8
SC
MiraData
0.48
0.99
0.94
Instruct Caption
0.99
0.87
w/ Short Caption
303.8
1167.7
0.53
0.49
0:99
0.94
wio Decocoposirion
32.7%
23.3%
0.41
0.99
0.88
w/ Progression
294.2
1169.8
0.68
0.53
0.99
0.93
w/o InstruciNet
12.3%
17.5%
0.16
0.98
0.86
Baseline
289.5
1181.3
0.67
0.99
Baseline
45.0%
0.50
0.99
0.90
Ablation Study. As shown in Table 4 we investigated the influence of various data strategies, including leveraging MiraData (Ju et al. (2024)) short captions (Chen el al. (2024)), and progression training (Lab & etc. (2024)). The results indicated that our data strategy outperforms the others, particularly in terms of semantic consistency, distribution alignment, and user preference. The visual quality metrics are comparable across all strategies. This consistency implies that visual quality metrics may be less sensitive to these strategies or that they might be limited in evaluating game domain generation. Further, as shown in T we explored the effects of our design on interactive control ability through ablation studies. This experiment involved evaluating the impact of removing key components such as InstructNet, Instruct Captions, or the decomposition process. The results demonstrate that the absence of InstructNet significantly reduces the SR and UP, highlighting its crucial role in user ussr-preeeeence interactive controllability. Similarly, the removal of Instruct Captions and the decomposition process also negatively impacts control metrics, although to a lesser extent. These findings underscore the importance of each component in enhancing the model's 's ability to generate and control game content interactively.
8

---

Published as a conference paper at ICLR 2025
Character: Assassin
Character: Mage
Action: Fly the flight
Action: Drive the carriage
Environment: Sakura forest
Environment: Rainforest
Snowstorm
Event: Heavy rain
Figure 5: The generation showcases of characters, environments, actions, and events.
Generated open-domain dlip
Key "D"
Key "A'
6g: Fire the sky
Gy: Show Figure 6: The qualitative results of different control signals, given the same open-domain clip.
night
fog
Get sky
4.2 QUALITATIVE RESULTS
Generation Functionality. Fig. Fig.BSillustraate the basic generation capabilities of our model in gener- ating variety of characters, environments, actions, and events. The examples show that the model can create characters such as assassins and mages, simulate environments such as Sakura forests and rainforests, execute complex actions like flying and driving, and reproduce environmental events like snowstorms and heavy rain. This demonstrates the model's ability to generate and control diverse scenarios, highlighting its potential application in generating open open-world game videos.
Interactive Control Ability. As shown in Fig. 6 our model demonstrates the capability to control both environmental events and character actions based on textual instructions and keyboard inputs. In the example provided, the model effectively manipulates various aspects of the scene, such as lighting conditions atmospheric effects, highlighting its ability to simulate different times of day and weather conditions. Additionally, the character's movements, primarily involving naviga- tion through the environment, are precisely controlled through input keyboard signals. This inter- active control mechanism enables the simulation of dynamic gameplay experience. By adjusting environmental factors like lighting and atmosphere, the model provides a realistic and immersive setting. Simultaneously, the ability manage character movements ensures that the generated tent responds intuitively to user interactions. Through these capabilities, our model showcases its potential to enhance the realism and engagement of open-world video game simulations.
Open-domale Generation, Gameplay Simulation and Further Analysis. As shown in Fig. we presented initial qualitative experiment results, where GameGen-X. generates novel domain game video clips and interactively controls them, which can be seen as game simulation. Further, we compared GameGen-X with other open-source models the open-domain generation ability as shown in Fig. 7 All the open-source models can generate some game-like content, implying their training involves corresponding game source data. As expected, the GameGien-X can better meet the game content requirements in character details, visual environments, and camera logic, owing to the strict dataset collection and building of OGameData. Further, we compared GameGen-X with other commercial products including Kling, Pika, Runway, Luma, and Tongyi, as shown in Fig In the left part, i.e., the initially generated video clip, only Pika, Kling1.5, and GameGen- X correctly followed the text description. Other models either failed display the character depicted them entering the cave instead of exiting. the right part, both GameGen-X and Kling 1.5 successfully guided the character out of the cave. GameGen-X achieved high-quality control response well as

---

Published as a conference paper at ICLR 2025
CogVideok
OpenSora- OpenSSor-Plaan
OpenSora
GameGGnx
Figure Qualitative comparison with open-source models in the open-domain generation.
RumWWay
Pika
Tore'l
Luma
KüneLs
GameGen-
Impeetive poreaions juenngg the cooele esieenee
Figure 8: Qualitative comparison with commercial models in the interactive control ability.
maintaining consistent camera logic, obeying the game-like experience the same time. This is owing to the design of holistic training framework and InstructNet.
5 CONCLUSION
We have presented GameGen-X, the first diffusion transformer model with multi-modal interac- tive control capabilities, specifically designed for generating open-world game videos. By simu- lating key elements such as dynamic environments, complex characters, and interactive gameplay, GameGen-X advances the state of generative models by demonstrating their potential in both gen- erating and controlling game content. The development of the OGameData provided crucial foun- dation for our model's training, enabling capture the diverse and intricate nature open-worll world games. Through a two-stage training process, GameGen-X X integrates content generation with in- teractive control, improving coherence and user adaptability in simulated gameplay. Beyond its technical contributions, GameGen-X presents new possibilities for game content design. It suggests a shift towards more automated, data-driven methods that could help streamline early-stage game content creation. By leveraging models to create immersive worlds and interactive gameplay, may move closer to a future where game engines are more attuned to creative, user- guideg experi- ences. While challenges remain (Appendix E). GameGen-X represents early step toward novel paradigm in game design. It lays the groundwork for future research into the role of generative models in creating interactive digital worlds.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. This work was supported by the Hong Kong Innovation and Technology Com- mission (Project No. MHP/002/22 and PTCPD/17 ITCPD//1-9). Additionally, we extend our sincere gratitude for the valuable discussions, comments, and help provided by Dr. Guangyi Liu, Mr. Wei Lin and Mr. Jingran Su (listed in alphabetical order). We also appreciate the HKUST SuperPOI) for computation support.
REFERENCES
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. arXiv preprint arXiv:2405.12333 2024.
Anastasia.
The rising costs of aaa game development, 2023, URL https: htt////ww.n...... net/ opmentt/ Accessed: 2024-6-15.
the-
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments video with natural language. Pruceedings of the IEEE international conference on computer vision, pp. 5803-5812, 2017.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holdes Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative inter- active environments. In Forty-first International Conference Machine Learning, 2024.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuaa Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13320- 13331, 2024.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9); 10850-10869, 2023.
Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang- Animateanything: Fine-grained open domain image animation with motion guidance. arXiv e- prints, pp. arXiv-2311, 2023.
Decart. Oasis: A universe a transformer, 2024. URL https: /wwwwwww.. decart aii/ articles/oasis- iatassssinttraactivee ai-ivveeeiiviiooo vieoo---v-----ei game model Accessed: 2024-10-31.
David Eberly. 3D game engine design: practical approach to real-time computer graphics. CRC Press, 2006.
BAAI Emu3 Team. Emu3: Next-token prediction is all you need, 2024. com/ com/baaivision: (amioon//mm3)) Emu3
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang Yaohui Wang, Yu Qiao, Mancesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image dif- fusion models without specific tuning. The Twelfth International Conference on Learning Representations, 2023.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint
2018.
Jingwen He, Tianfan Xue, Dongyang Liu, Xinqi Lin, Peng Gao, Dahua Lin, Yu Qiao. Wanli Ouyang, and Ziwei Liu. Venhancer: Generative space-time enhancement for video generation. arXiv preprint arXiv:2407.07666 2024a.
11

---

Published as a conference paper at ICLR 2025
Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, Man Zhou, and Jie Zhang. Id-animator: Zero-shot
human video generation. arXiv preprint
24%25:2404.15275, 2024b.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint arXiv:2010.0422 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jiancheng Huang, Mingfu Yan, Songyan Chen, Yi Huang, and Shifeng Chen. Magicfight: Person- alized martial arts combat video generation. In ACM Multimedia 2024, 2024a.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vhench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024b.
Yonggang Jin, Ge Zhang, Hao Zhao, Tianyu Zheng, Jiawei Guo, Liuyu Xiang, Shawn Yue, Stephen W Huang, Wenhu Chen, Zhaofeng He, et al. Read to play (r2-play): Decision trans- former with multimodal game instruction. arXiv preprint wxXhv2242..04154 2024.
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: large-scale video dataset with long durations and structured captions, 2024. UtLhttps: /ttttp:://axx..oo///bbb..
2407 06358
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimküüler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph. 42(4):139 2023.
Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to simulate dynamic environments with gamegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1231-1244 2020.
PKU- Yuan Lab and Tuzhan Al etc. Open-sora-plan, April 2024. htthsssp:://oo..orr 5281/zenodo. 10948109
Guangyi Liu, Zeyu Feng, Yuan Gao, Zichao Yang, Xiaodan Liang, Junwei Bao, Xiaodong He, Shuguang Cui, Zhen Li, and Zhiting Hu. Composable text controls in latent space with odes, 2023a. URL //tp///aavvv////////a//eee/aaaav
htsp:/bbs/222.2/ 00638
Guangyi Liu, Yu Wang. Zeyu Feng, Qiyu Wu, Liping Tang, Yuan Gao, Zhen Li, Shuguang Cui, Julian McAuley, Zichao Yang, Eric P. Xing, and Zhiting Hu. Unified generation, reconstruction, and representation: Generalized diffusion with adaptive latent
2024. URL
lhttps://arxiss org. abs/ www.////2...2 2402 19009
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference Learning Repre- sentations (ICLR), 2023b.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401. 0030448 2024.
12

---

Published as a conference paper at ICLR 2025
Willi Menapace, Stephane Lathuiliere, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci. Playable video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10061-10070, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140)1167, 2020a. URL http:
//jml wwtp://ppppppppppeeee /ttpeers/v1// http///112000000 074 html
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020b.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint arXiv:2006. 10704, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp.
2022b.
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. How2: a large-scale dataset for multimodal language understanding, arXiv preprint
2018.
Christoph Schuhmann. Improved aesthetic predictor.
https://githin com/
2023. Accessed:
2023-10-04.
Inkyu Shin, Qihang Yu, Xiaohui Shen, So Kweon, Kuk-Jin Yoon, and Liang- Chieh Chen. En- hancing temporal consistency in video editing by reconstructing videos with 3d gaussian splatting. arXiv preprint N.X::22406.02541, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Tomáš Souček and Jakub Lokoč. Transnet v2: An effective deep network architecture for fast shot transition detection. arXiv preprint arXiv:2008.0448 2020.
Stability AI. sd-vae-ft-mse hugging face, 2024. URL htttp:: co/
stabilityaissd-vee-Ee ft--------------eeeeeee------- Accessed: 2024-11-21.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputingg 568:127063. 2024.
Connor Holmes Will DePue Yufei Guo Li Jing David Schnurr Joe Taylor Troy Luhman Eric Luhman Clarence Ng Ricky Wang Tim Brooks, Bill Peebles and Aditya Ramesh. Video generation models as world simulators, 2024. URL tecttt::/oppnaeee com/ hommreseearch/ wor d- imm lators Accessed: 2024-6-15.
wrideo-generatiinn
13

---

Published as a conference paper at ICLR 2025
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. 14837, 2024.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint wrXiv:2308.06571, 2023a.
Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang. Changxin Gao, and Nong Sang. Videolem: Video latent consistency model. arXiv preprint arXiv:2312.09109, 2023b.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et Internvid: A large-scale video-text dataset for multimodal understand- ing and generation. arXiv preprint M.Xiv:3307006442 2023c.
Zhao Wang, Aoxue Li, Enze Xie, Lingting Zhu, Yong Guo, Qi Dou, and Zhenguo Li. Customvideo: Customizing text-to-video generation with multiple subjects. arXiv preprint arXiv:2401.0990 2024.
Wikipedia. Development of the last of us part ii, 2023. URL https://en. wwt.pedwwi.ppii....mmmm. org w wiki/Devee www.e/eeeeeooeeet..oemmmmmmm of The _last_oft Laces_oo___________e_e___ee Pas__art____ Accessed: 2024-09-16.
Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint (4ti::4006009555. 2024.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang. Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310. /2/00, 2023.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2 /4084, 2021.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free free llava extension from images to videos for video dense captioning. arXiv preprint DrXiv:2404.1699, 2024.
Ziming Liu Haotian Zhou Qianli Ma Xuanlei Zhao, Zhongkai Zhao and Yang You. Opendit: An casy, fast and memory-efficcee system for dit training and inference, 2024. URL https:// github. com NUS AI- waw.aaaaaaa/.essss/reereeeeeeeee
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians for modeling dynamic urban scenes. arXiv preprint arXiv:2401.013:
2024.
Mengjiao Yang. Yilun Du Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbecl. Learning interactive world simulators. arXiv preprint arXiv:2310.06114, 2023.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint 24v::240.006777 2024.
Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos. arXiv preprint arXiv:2501.0833 2025.
Lijun Yu, José Lezama. Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.057::1 2023a.
14

---

Published as a conference paper at ICLR 2025
Lijun Yu, José Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David C. Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming- Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tok- enizer key to visual generation. 2023b. URL https:/ /tas:///pp... http::://app.smmanni..o.mm .rrrgol.ooooooo/oooo.o.oo.oo
CorpusSD::25B30733
Zhaoyang Zhang. Ziyang Yuan, Xuan Ju, Yiming Gao, Xintao Wang, Chun Yuan, and Ying Shan. Mira: A mini-step towards sora-like long video generation. https://github. com/
mi sra-space/Miia ra 2023. ARC Lab, Tencent PCG.
Xuanlei Zhao, Shenggan Cheng, Chang Chen, Zangwei Zheng, Ziming Liu, Zheming Yang, and
Yang You. Dsp: Dynamic sequence parallelism for multi-dimennional transformers, 2024a. URL httpps:/aara
2400. 10266
Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast, 2024b. (ttts::
wttp////////...a// 12588
Tianyu Zheng. Ge Zhang, Xingwei Qu, Ming Kuang, Stephen W Huang, and Zhaofeng He. More- 3s: Multimodal-base offline reinforcement learning with shared semantic spaces. arXiv preprint arXiv:2402.12844 2024a.
Zangwei Zheng. Xiangyu Peng, Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024b.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.1 1039, 2024.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Zheng Zhu, Xiaofeng Wang, Wangho Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, et al. Is sora a world simulator? a comprehensive survey on general world models and beyond. arXiv preprint 2r55::2405.03520, 2024.
15

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
A.1 VIDEO DIFFUSION MODELS
The advent of diffusion models, particularly latent diffusion models, has significantly advanced im- age generation, inspiring researchers to extend their applicability to video generation (Liu et al.) (2023a 2024)). This field can be broadly categorized into two approaches: image-to-video and text- to-video generation. The former involves transforming static image into dynamic video, while the latter generates videos based solely on textual descriptions, without any input images. Pioneer- ing methods in this domain include AnimateDiff (Guo et al. (2023)). Dynamicrafter (Xing et al. (2023)), Modelscope (Wang et al. (2023a) ) AnimateAnything (Dai et al. (2023)), and Stable Video Diffusion (Rombach et al. (2022b)) These techniques typically leverage pre-trained text-to-image models, integrating them with various temporal mixing layers to handle the temporal dimension inherent in video data. However, the traditional U-Net based framework encounters scalability is- sues, limiting its ability to produce high-quality videos. The success of transformers in the natural language processing community and their scalability has prompted researchers adapt this architec- ture for diffusion models, resulting in the development of DiTs (Peebles & Xie (2023). Subsequent work, such as Sora (Tim Brooks & Ramesh (2024)) has demonstrated the powerful capabilities of DiTs in video generation tasks. Open-source implementations like Latte (Ma et al. (2024)), Open- sora (Zheng et al. (2024b), and Opensora-
(Lab & etc. (2024)) have further validated the su- perior performance of DiT-based models over traditional U-Net structures in both text-to-vider and image-to- video generation. Despite these advancements, the exploration of gaming video generation and its interactive controllability remains under-explored.
A.2 GAME SIMULATION AND INTERACTION
Several pioneering works have attempted to train models for game simulation with action inputs. For example, UniSim (Yang et al.) (2023) and Pandora (Xiang et al.) (2024) built a diverse dataset of real-world and simulated videos and could predict continuation video given previous video segment and an action prompt via a supervised learning paradigm, while PVG (Menapace et al. (202I) and Genie (Bruce al. (224 ) focused on unsupervised learning of actions from videos. Similar to our work, GameGAN (Kim et al. (2020)), GameNGen (Valevski et al. (2024)) and DIA- MOND (Alonso al. )2224) focused the playable simulation of early games such as Atari and DOOM, and demonstrates its combination with a gaming agent for interaction (Zheng et al. (2024a). Recently, Oasis (Decart (2024) simulated Minecraft at real-time level, including both the footage and game system via the diffusion model while GameFactory (Yu et al. (2255) brought the game domain manipulation real-world prior. However, they didn't explore the potential of generative models in simulating the complex environments of next- next-generation games. Instead, GameGen-X X can create intricate environments, dynamic events, diverse characters, and complex actions with a high degree of realism and variety. Additionally, GameGen-X allows the model to generate subse- quent frames based on the current video segment and player-provided multi-modal control signals. This approach ensures that the generated content is not only visually compelling but also contextu- ally appropriate and responsive to player actions, bridging the gap between simple game simulations and the sophisticated requirements of gexeerenetttoon open- world games.
B DATASET
B. DATA AVAILABILITY STATEMENT AND CLARIFICATION
We are committed to maintaining transparency and compliance in our data collection and sharing methods. Please note the following:
Publicly Available Data:
The data utilized in studies is publicly available. We do not use any exclusive or private data sources.
Data Sharing Policy: Our data sharing policy aligns with precedents set by prior works, such as Intern Vid (Wang et al. (2023c)), Panda-70M (Chen et al. (2024)), and Miradata (Ju jet al. (2024)). Rather than providing the original raw data, we only supply the YouTube video IDs necessary for downloading the respective content.
16

---

Published as a conference paper at ICLR 2025
The data released is intended exclusively for research purposes. Any po- tential commercial usage is not sanctioned under this agreement.
Usage Rights:
Compliance with YouTube Policies: Our data collection and release practices strictly adhere to YouTube's ss data privacy policies and fair of use policies. We ensure that no user data or privacy rights are violated during the process.
Data License: The dataset is made available under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
Moreover, the OGameData dataset is only available for informational purposes only. The copyright remains with the original owners of the video. All videos of the OGame datasets are obtained from the Internet which is not the property of our institutions. Our institution is not responsible for the content or the meaning of these videos. Related to the future open-soureing version, the researchers should agree not reproduce, duplicate, copy, sell, trade, resell, or exploit for any commercial purposes, any portion of the videos, and any portion of derived data, and not to further copy, publish, distribute any portion of the OGameData dataset.
B.2 CONSTRUCTION DETAILS
Data Collection. Following sites and local game engines as one of our primary video sources. Prior research predominantly focused on collecting game cutscenes and gameplay videos containing UI elements. Such videos are not ideal for training a game video generation model due to the presence of UI elements and non-playable content. In contrast, our method adheres to the following principles: 1) We exclu- sively collect videos showcasing playable content, as our goal is to generate actual gameplay videos rather than cutscenes or CG animations. 2) We ensure that the videos high-quality and devoid of any UI elements. To achieve this, we only include high-quality games released post-2015 and capture some game footage directly from game engines enhance diversity. Following the Inter- net data collection stage, we collected 32,000 videos from YouTube, which cover more than 150 next-generation video games. Additionally, we recorded the gameplay videos locally, to collect the keyboard control signals. We purchased games on the Steam platform to conduct our instruction data collection. To accurately simulate the in in-game lighting and weather effects, we parsed the game's console functions and configured the weather and lighting change events to occur randomly every 5-10 seconds. To emulate player input, we developed a virtual keyboard that randomly controls the character's ' movements within the game scenes, Our data collection spanned multiple distinct game areas, resulting in nearly 100 hours of recorded data. The program meticulously logged the output signals from the virtual keyboard, and we utilized Game Bar to capture the corresponding gameplay footage. This setup allowed us to synchronize the keyboard signals with frame-level data, ensuring precise alignment between the input actions and the visual output.
Ju et al. (2024) and Chen et al. (2024), we selected online video web-
Video-level Selection and Annotation. Despite our rigorous data collection process, some low- quality videos inevitably collected into our dataset. Additionally, the collected videos lack essential metadata such as game name, genre, and player perspective. This metadata is challenging to annotate using Al alone. Therefore, we employed human game experts filter and annotate the videos. In this stage, human experts manually review each video, removing those with UI elements or non- playable content. For the remaining usable videos, they annotate critical metadata, including game name, genre (c.g., ACT, FPS, RPG), and player perspective (First-person, Third-person). After this filtering and annotation phase, we curated a dataset of 15,000 high-quality videos complete with game metadata.
Scene Detection and Segmentation. The collected videos, ranging from several minutes to hours, are unsuitable for model training due to their extended duration and numerous scene changes. We employed TransNetV2 (Souček & Lokoë (2020) and PyScene for scene segmentation, which can adaptively identify scene change timestamps within videos. Upon obtaining these timestamps, discard video clips shorter than 4 seconds, considering them too brief. For clips longer than 16 seconds, we divide them into multiple 16-second segments, discarding any remainder shorter than 4 seconds. Following this scene segmentation stage, we obtained around 1,000,000 video clips, each containing 4-16 seconds of content 24 frames per second.
Clips-level Filtering and Annotation. Some clips contain game menus, maps, black screens, low- quality scenes, or nearly static scenes, necessitating further data cleaning. Given the vast number
17

---

Published as a conference paper at ICLR 2025
of clips, manual inspection is impractical. Instead, sequentially employed an aesthetic scoring model, a flow scoring model, the video CLIP model, and a camera motion model for filtering and annotation. First, we used the CLIP-AVA model (Schubmann (023)) score each clip aesthetically. We then randomly sampled 100 clips to manually determine a threshold, filtering out clips with aesthetic scores below this threshold. Next, applied the UniMatch model (Xu et al. (2023)) to filter out clips with either excessive or minimal motion. To address redundancy, we used the video- CLIP (Xu et al. (202I) model to calculate content similarity within clips from the same game, removing overly similar clips. Finally, we utilized CoTrackerV2 (Karaev et al. (2233) to annotate clips with camera motion information, such as "pan-left" or "zoom-in."
Structural Caption. We propose Structural captioning approach for generating captions for OGameData--GEN and OGaSeData-INS. To achieve this, we uniformly sample 8 frames from each video and stack them into single image. Using this image as representation of the video's content, we designed two specific prompts to instruct GPT-4o to generate captions. For OGameData- GEN, we have GPT-4o describe the video across five dimensions: Summary of the video, Game Meta information, Character details, Frame Descriptions, and Game Atmosphere. This Structural infor- mation enables the model to learn mappings between text and visual information during training and allows us to independently modify one dimension's s information while keeping the others un- changed during the inference stage. For OGameData-INNS, five perspectives, with each perspective described in a short sentence. The Environment Basic di- mension describes the fundamental environment information, while the Transition dimension cap- tures changes in the environment. The Light and Act dimensions describe the lighting conditions and character actions, respectively. Lastly, the MISC dimension includes meta-information about the video, such as keyboard operations or camera motion. This Structural captioning approach al- lows the model to focus entirely on content changes, thereby enhancing control over the generated video. By enabling independent modification of specific dimensions during inference, we achieve fine-grained generation and control, ensuring the model effectively captures both static and dynamic aspects of the game world.
decompose the video changes into
Prompt Design. In our collection of 32,000 videos, we identified two distinct categories. The first category comprises free-camera videos, which primarily focus showcasing environmental and scenic elements. The second category consists of gameplay videos, characterized by the player's perspective during gameplay, including both first-person and third-person views. We believe that free-camera videos can help the model better align with engine-specific features, such as textures and physical properties, while gameplay videos can directly guide the model's behavior: To leverage these two types of videos effectively, we designed different sets of prompts for each category. Each set includes a summary prompt and a dense prompt. The core purpose of the summary prompt is to succinctly describe all the scene elements in single sentence, whereas the dense prompt provides structural, fine- fine-grained guidance. Additionally, structural instruction prompts. These prompts describe the differences between the initial frame and subsequent frames across various dimensions, simulating how instructions can guide the generation of follow-up video content.
achieve interactive controllability, we designed
Frryou are ChatGFT, a Jazge Janquage medel croined by Openti, based cn the GPI-4 sichileeluzee
Knokledgo cutosE; 2023-10. Cursent dates 2024 15.
Inage input capaballt lini Snabled, Porsonality: 72.
Character
You aze a video gane envisoraent capcioning assiscant that genesates concise descript lons gans
cnvireanent.
Skills
Analyzing sequence of inages that zepresent gane envizonaent
Ident flying key envi rnvirnntet features and acrospheric elensacs
11
Geneeat ing x brimf, oohweent caplisa that captares the main, elements of the gatw world Constraiaca
12
nept fon should
20
long
The son aust. desuzibe the main mnvirommtta Ivatuzws viaible
15
The caption aust Include the overall ataoaphaze or nood of the aetting
16
Use present teose describe the enviroment
Iuput: [8 sequential franus of the gau anvirumenn, accanged Output: IA ocnciss, English capcion describing the nain features and atacsphore of the qaae envi romeent]
of 4 lnages macht]
19
Exauple: misly Tarwal suzzounda ancient zuins, with Lowering Lowes and crmbbling wtone strnctared ing mysterion atmoephere.
18

---

single, donse coption 2-4 scntcnccs covering catiro envizonnent shown Published as a conference paper at ICLR 2025
Listing Summary prompt for free-camera videos
prongt surnry You ChatGPT, large language model crained OpenAT, based the GPT-4 architacturo.
Knowledge cutott: 2023-10,
Current date: 2024-05-15,
input capsollities: Snabied. Personality:
Charactor
You are a highly skilled vidco gane carirenment captioning AI assistant, Your task is to generate anvironment based sequential trames provided caption should comprchensively describe kay eismonts gana world and sotting. Skills
detailen,
Identitying the style and genre ot the vieeo gane
enviromaental features landscapes Detailing atmosphere, lighting,
Recomieing deseribing
everail at setting Moting key architectural eienents, structures, or natural tormstions Describing any notable weather effects cavironmental conaicions cohesive doscription Using and preciso lanquage to paint detailed picture for reader Constrsints
13
14
Synthesizing
world
16
17
The input will be single inage containing 8 crames ot the game onri anvironnent, arranged in two zows franes each
The oucpuc
Background
20
This video is from GAM ID.
21
caption
aain unvironeental festures that are focua the framu
23
The overall styie or gonre of the game world (e.a. fantasy, sci-fi, post post-apccalyotio)
24
Rey details about the laadscape, vegeration, and torrain
notable
settlementa visible
day, and weathez conditions Usc concise yet acseriptive language to capture the essential elements
general atncschozo,
27
28
The change of envizontent in these franes
Avoid speculating about
in the francstrr
Listing 2: Dense prompt for free-camera videos
prongt Sorpgt_suurryy You are ChatGPT, a large language model trained by OpenAT, based on the GPZ-4
architecture,
Knokledge octoffr 2023 10.
Cursent dater 2024-05-15.
Image ioput ospeet leeees Snabled.
Porsonality: 72.
Character
capt toning
generates descript fone
plips.
+ Skills
Anslyzing sequence ot images that represent short video ciip
10
it ix chi
view, identi key charactery their actions, else, Ident key objects
cnntentents..
11
Senerating orlet, conerent caption that captures the main elenents of the vides
12
Constrsiats
13
caption should
20
ception
characteríe)
15
The caption must describe the anvizonrent shown in the video
16
Use present tense to descrise the actions
17
there nuitiple aistinct sons, focus the prominent
images each) Output: MA cenciso, Englteh capcion describing the main character and action (t) in the video] Example: There ia person walking on path surrounded treea and ruins an ancient city, fer
Input: 18 sequential zeamos of the vidco, arranged
20
Listing 3: Summary prompt for gameplay videos
prungt_aunnzy Yrr CbatOPT, Jazge Janyuage modil Lewined by OpenAI, based the GPT-4 azchiteciuze.
Knowledge outoff: 2023 10.
Curseot dates 2024-00 15. Image input ospent litees Personality: v2. Charactez
You highty iled video coptioning AT arsistant, Your task generate decel led, dense bept Jon should conpcehansivell dsscribe key viumenca
provided input. capt clip based aequent video.
Beills
Telent 3 fylin the style and of the video footage
Recognising naming the nain ooject or character in tocus
biig the beckground anvi ronment and secting
19

---

Published as a conference paper at ICLR 2025
Moting key caseza angles, sovanents, and ahot types
13
Syathesizing the a francs lnto a cehesive description of the video action
14
Using vivid and precise languago to paint actailed picture for the reader
Ihe input wiil be a single inage containing 2 franco of the video, arranged two zows francs
cach, in sequential order
17
The oucpuc should be single, danse caption of 2-6 sencences covering the entiro 8-frane video 200 words long
18
The caption should Backgeuund!
20
Zhis video is from GAKS ID.
21
st The caption must montica:
22
The juet or
that ia the Fucus of the video
it third-ourson view, includo the nans of the nain character, the aspearance, clothing, and anything related to the character generation guidance.
24
The style or genze (e.g. ficat parson/Litzd pezson, ahooces, oped sacing, sto.)
25
Kay details about the backeround environaent
setting
Any notabie conera angles, movermats, or shot types
27
Use concise yet deseriptive lanquage Lo captuze the wssential elenenta
28
Avold spoculating about pazta of the video not represeaced od in the franceer
Listing 4: Dense prompt for gameplay videos
Yrr ChatoPT, Jazge lanquage aedel croined by opendi, based on the GPI-4 architesture.
Knoxledoe cutoff; 2023-10.
Curzent date: 2024-05-15.
Image input capsollities: Snabled. Porsonality: v2. + Character
You are a highly skilled AI accistant speciellzing in dotecting and describing changes in vigco Your
sequentiai franes from video and stractural coption focusing on the action and changes that occur after the first franc. This Structural caption will be use io Lewin video goneration nodel to czeate controllable video sequences based on coxtual comnands.
is
concise
Skills
Cazefully observing the fleat frans to vatablish a baseline, comparing subaequuni content to the
first frane
10
Picase describe the input video in the following i dincnsions, providing single, concise sentence for
1. Enviromenttal Basics: Describo what the whole scona looks like.
12
2. Main Charactar: Dirwet the protagunist' is ast ions and novemwoss.
13
3. Environeencal Changes: Cousand how the sosne should change over Liae.
14.
Sky/Lighcing: Instruct Constraints
adjust sky conditions and lighting effects.
16.
The input will be a siegle image containiog 2 fransa of the video, assaeged in two zows of Transa each, in sequential order
17
solely changez
after the first frane
Do not describe elementa that renain constant throughout the sequence
19
Use plear, precise languege to deser the changes Frame each, dimension as cleat, aet innallle insiruction, Keep instruction worde,
20
21
sentence only,
sentence should be concise and than
22
Use inperalive langoage acitable for dizeeting video yenezation ion mudel,
23
If informaticn for a particuiar diaenslon is not available, provide a general instruction 12ke dinension. Do not include nanbers or bullen pointa before vach suniuace in output.
'Kaintain current state'
25
Please use simpie worde, That ruct lone
26
27
first feane carufully as bazeline
18
Analyse the suesequent cortent as cont inuous video sequence
25
Avaid using tarae like *frane, "inage, or "figure" in your deset ipt ion Deseribe the sequince as 1f it wezw cunt inocus video, not aupacale Temnas Background
30
31
This video from GAMB TD. Pocus on devcribing the changes in aot ion, environment, or charastez ioning cathes chan iduntifying apeci fie gane wlenenia.
33
Output
34
Your output should 3lst, with eash number currespording the diawnsion as liyted above, For example:
35
Enviromentall Reeicss (Your rnst ian for Prviroments! Revics).
36
Main Characters (Your Insiroction for Hain characiar]- Environeenaal Changesz (Your Instruction
ronnental Changezi. Sky/liight ings (Your feut ruct fon for Sky/tiight ing).
39
40
Please procese the input and provide the Strusturai, instructional ourpuez"rr
Listing 5: Instruction prompt for interactive control
B.3 DATASET SHOWCASES
We provide a visualization of the video clips along with their corresponding captions. We sampled four cases from the GENN dataset and the OGameData-INS dataset, respectively. Both
20

---

Published as a conference paper at ICLR 2025
A sample from OGameData-GEN Caption: An empty futuristic city street seen under a large overpass with neon lights and tall buildings. In this sequence from Cyberpunk 2077, the scene unfolds in a sprawling urban environment marked by towering skyscrapers and elevated highways. The video clip showcases first-person perspective that gradually moves forward along an empty street framed by futuristic neon-lit buildings on the right and industrial structures on the left. The atmospheric lighting casts dramatic shadows across the pavement, enhancing the gritty cyberpunk aesthetic of Night City. As the camera progresses smoothly towards a distant structure adorned with holographic advertisements, captures key details like overhead cables and a monorail track above, highlighting both verticality and depth in this open-world dystopian setting devoid of any charac- ters or vehicles this moment. The scene emphasizes a gritty, dystopian cyberpunk atmosphere, characterized by neon-lit buildings, dramatic shadows, and sense of desolate futurism devoid of characters or vehicles."
Figure 9:
Figure 10: A sample from OGameData-GEil Caption: A person in a white hat walks along a forested riverbank sunset. In the dim twilight of a picturesque, wooded lakeshore in Red Dead Redemption 2, Arthur Morgan, dressed in his iconic red coat and wide-brimmed white hat, strides purposefully along the water's edge. The eight sequential frames capture him closely from behind at an over- soer-hhe-sholddrr camera angle as he walks towards the dense tree line under a dramatic evening sky tinged with pink and purple hues. Each step takes place against a tranquil backdrop featur- ing rippling water reflecting dying sunlight and silhouetted trees that deepen the serene yet subtly ominous atmosphere typical of this open-world action-adventur game. Dust particles float visibly through the air Arthur's 's movement stirs up small puff's from the soil beneath his boots, adding to the immersive realism of this richly detailed environment. The scene captures a tranquil yet subtly ominous atmosphere.
types of captions are Structural, offering multidimensional annotations of the videos. This Structural approach ensures a comprehensive and nuanced representation of the video content.
Structural Captions in OGameData-GEN It is evident from Fig. and Fig. 10 that the captions in "GameData-Gul densely capture the overall information and intricate details of the vidcos, fol- lowing the sequential set of "Summary", 'Game Meta Information', 'Character Information', 'Frame Description', and 'Atmosphrre..
Structural Instructions in OGameData-INS, In contrast, the instructions in OGameData-INe which are instruction-orientte and often use imperative sentences, effectively capture the changes in subsequent frames relative to the initial frame, shown in Fig. and Fig. T2 It has five decou-
21

---

Published as a conference paper at ICLR 2025
Figure 11: sample from OGameData- Caption: Environmental Basics: Maintain the dense forest scenery with mountains in the distant background. Main Character: Move forward along the path while maintaining steady pace, Environmental Changes: Gradually clear some trees and bushes to reveal more of the landscape ahead. Sky/Lighting: Keep consistent daylight conditions with scattered clouds. aesthetic score: 5.02, motion score: 27.37, camera motion: camera size: full shot.
Undetermined.
Figure 12: sample from OGameData-INS, Caption: Environmental Basics: Show scenic outdoor environment with trees, grass, and clear water body in the foreground. Main Character: Move the protagonist slowly forward towards the right along the water's edge. Environmental Changes: Maintain current state without significant changes to background elements. Sky/Lighting: Keep sky conditions bright and lighting consistent throughout. aesthetic score: 5.36, motion score: 9.37, camera motion: zoom in. camera size: full shot".
pled dimensions following the sequential of 'Environment Basic', 'Character Action', 'Environment Change', 'Lighting and Sky', and 'Mise".
B.4 QUANTITATIVE ANALYSIS
To demonstrate the intricacies of our proposed dataset, conducted comprehensive analysis compassing several key aspects. Specifically, we examined the distribution of game types, game genres, player viewpoints, motion scores, aesthetic scores, caption lengths, and caption feature dis- tributions. Our analysis spans both the OGameData-GEN dataset and the OGameData-INS dataset, providing detailed insights into their respective characteristics.
Game-related Data Analysis. Our dataset encompasses a diverse collection of 150 games, with a primary focus on next-generation open-world titles. For the OGameData GEN dataset, as depicted view- points. Furthermore, includes wide array of game genres, including RPG, Action, Simulation, and FPS, thereby showcasing the richness and variety of the dataset. In contrast, the OGameData- INS dataset, as shown Fig. 14 is composed of five meticulously selected high-quality open-world games, each characterized by detailed and dynamic character motion. Approximately half of the videos feature the main character walking forward (zooming in), while others depict lateral move- ments such moving right left. These motion patterns enable us effectively train instructive network. To ensure the model's 's attention remains on the main character, we exclusively selected third-person perspectives.
in Fig. 13 player perspectives are evenly distributed between first-person and
22

---

Published as a conference paper at ICLR 2025
Perspective Distribution
Came Type Distribution
Third person
52.7%
40
Purconbos (%)
23.00%
30
47.3%
20
First person
10
3.31%
1.2%%
1.20:
RET
Nimdaatoo
AbV
Oltrrs
Sumbel Harms
Game
Figure 13: Statistical analysis of the OGameData- dataset. The left pie chart illustrates the distribution of player perspectives, with 52.7% of the games featuring a third-person perspective and 47.3% featuring a first-person perspective. The right bar chart presents the distribution of game types, demonstrating predominance of RPG (55.23%) and ACT (33.08%) genres, followed by Simulation (3.35%) and FPS (3. 17%), among others.
Percentage Distribution Motion Scores
Percentage Distribution of Aesthetic Scores
3.0
Percentage (%)
Percentage (96)
2.5
3
1.0
0.5
0.0
40
160
4.50
4.75
5.00
5.25
5.50
5.75
6.00
6.25
6.50
Motion Score
Aesthetic Score
Percentage Distribution of Game Count
Percentage Distribution Camera Motion
Percentage (%))
Percentage (%)
20
20
The Witcher
Red Datd Retermaten2 2
Balen Riag
Asseniin' Creen Drginn Digine
Zoan
Biusg
Undesermeed
Parx loft
Par 1attt
Blttc
tst domnn
Zostt nut
Game Name
Camera Motion
Figure 14: Comprehensive analysis of the OGameData-IN INS dataset. The top-left histogram shows the distribution of motion scores, with most scores ranging from 100. The top-right histogram illustrates the distribution of aesthetic scores, following Gaussian distribution with the majority of scores between 4.5 and 6.5. The bottom-left bar chart presents the game count statistics, highlighting the most frequently occurring games. The bottom-right bar chart displays the camera motion statis- tics, with significant portion of the clips featuring zoom-in motions, followed by various other camera movements.
23

---

Published as a conference paper at ICLR 2025
Parcertage Distribtton of Motion Soores
Parcentage Dutrribtion of Assthetic Sicoess
Percentage Datribution of Word Counts
2.6
20
Parcentege
14-
Persentagg
Purcentuuge (60
2.0
1.5
76
1.0
20
75 100 150
175
4.5
5.0
0.5
5.0
0.3
50
100
130
200
250
100
Figure 15: Clip-related data analysis for the OGameData--Gie dataset. The left histogram shows the distribution of motion scores, with most scores ranging from 0 to 75. The middle histogram displays the distribution of aesthetic scores, following a Gaussian distribution with the majority of scores between 4.5 and 6. The right histogram illustrates the distribution of word counts in captions, predominantly ranging between 100 and 200 words. This detailed analysis highlights the rich and varied nature of the clips and their annotations, providing comprehensive information for model training.
Clips-related Data Analysis. Apart from the game-related data analysis, we also conducted clip- related data analysis, encompassing metrics such as motion score, aesthetic score, and caption dis- tribution. This analysis provides clear insights into the quality of our proposed dataset. For the OGameData-GENN dataset, as illustrated in Fig. [5] most motion scores range from 0 to 75, while the aesthetic scores follow Gaussian distribution, with the majority of scores falling between 4.5 and 6. Furthermore, this dataset features dense captions, with most captions containing between 100 to 200 words, providing the model with comprehensive game-related information. For the OGameData- PNN dataset, shown in Fig. 14 the aesthetic and motion scores are consistent with those of the DGameDData-GH:: dataset. However, the captions in OGameDataaINS are significantly shorter, enabling the model to focus more the instructional content itself. This design choice ensures that the model prioritizes the instructional elements, thereby enhancing its effectiveness in understanding and executing tasks based on the provided instructions.
C IMPLEMENTATION AND DESIGN DETAILS
C.I TRAINING STRATEGY
We adopted a two-phase training strategy to build our model. In the first phase, our goal was to train foundation model capable of both video continuation and generation. To achieve this, we allocated 75% of the training probability to text-to-video generation tasks and 25% to video extension tasks. This approach allowed the model to develop strong generative abilities while also building solid foundation for video extension.
To enhance the model's ability to handle diverse scenarios, we implemented bucket based sampling strategy. Videos were sampled across a range of resolutions (480p, 512x512, 720p. and 1024x1024) and durations (from single frames to 480 frames 24 fps), as shown in Table 6 For example, 1024x 1024 videos with 102 frames had an 8.00% sampling probability, while 480p videos with 408 frames were sampled with an 18.00% probability. This approach ensured the model was ex- posed to both short and long videos with different resolutions, preparing it for wide variety of tasks. For longer videos, we extracted random segments for training. All videos were resized and center-cropped to meet resolution requirements before being processed through a 3D VAE, which compressed spatial dimensions by 8x and temporal dimensions by 4x, reducing computational costs significantly.
We employed several techniques to optimize training and improve output quality. Rectified flow (Liu et al. (2023b)) was used to accelerate training and enhance generation accuracy. The Adam optimizer with a fixed learning rate of 5e-4 was applied for 20 epochs. Additionally, we followed common practices in diffusion models by randomly dropping text inputs with 25% probability to strengthen the model's 's generative capabilities Ho & Salimans (2021).
After completing the first training phase, we froze the base model and shifted our focus to training an additional branch, InstructNet, in the second phase. This phase concentrated entirely on the video extension task, with 100% probability assigned to this task. Unlike the first phase, we abandoned
24

---

Published as a conference paper at ICLR 2025
Table 6: Video Sampling Probabilities by Resolution and Frame Count
Resolution
Number of Frames
Sampling Probability (%)
1024
102
8.00
1024x1024
1.80
1024x1024
2.00)
480p
204
6,48
480p
408
18,00
480p
89
6.48
720p
102
54.00
512x512
51
3.24
the bucket-based sampling strategy and instead used videos with a fixed resolution of 720p and a duration of 4 seconds. To enhance control over the video extension process, we introduced addi- tional conditions through InstructNet. In 20% of the samples, no control conditions were applied, allowing the model to generate results freely. For the remaining 80% of the samples, control condi- tions are included with the following probabilities: 30% of the time, both text and keyboard signals provided control; 30% of the time, only text is provided; and for another 30%, both text and video prompt are used as control. In the remaining 10% of cases, all three control conditions-texx keyboard signals, and video prompts- porated, we sample from set of different prompt types with equal probability, including canny-edge videos, motion vector videos, and pose sequence videos. In both phases of training, during video extension tasks, we retain the first frame of latent as reference for the model.
applied simultaneously. When video prompts are incor-
C.2 MODEL ARCHITECTURE
Regarding the model architecture, our framework comprises four primary components: 3D VAE for video compression, T5 model for text encoding, the base model, and InstructNet.
We extended the 2D VAE architecture from Stable Diffusion Stability Al (2024) by incor- porating additional temporal layers to compress temporal information. Multiple layers of Causal 3D CNN Yu et al. (2023b) were implemented to compress inter-frame information. T he VAE decoder maintains architectural symmetry with the encoder. Our 3D VAF effectively compresses videos in both spatial and temporal dimensions, specifically reducing spatial dimensions by a factor of 8 and temporal dimensions by a factor of 4.
3D VAE.
Text Encoder. We employed the T5 model Raffel al. (2020b) with a maximum sequence length of 300 tokens to accommodate our long-form textual inputs.
Our MSDIT is composed of stacked Spatial Transformer Blocks and Temporal Transformer Blocks, along with an initial embedding layer and final layer that reorganizes the serialized tokens back into 2D features. Overall, our MSDiT consists of 28 layers, with each layer containing both spatial and temporal transformer block, in addition to the embedding and final layers. Starting with the embedding layer, this layer first compresses the input features further, specifically performing a 2x downsampling along the height and width dimensions to transform the spatial features into tokens suitable for transformer processing, The resulting latent representation 2, is augmented with various meta-information such the video's aspect ratio, frame count, timesteps, and frames per second (fps). These metadata are projected into the same channel dimension the latent feature via MLP layers and directly added to z, resulting in z'. Next, z' is processed through the stack of Spatial Transformer Blocks and Temporal Transformer Blocks, after which is decoded back into spatial features. Throughout this process, the latent channel dimension is set to 1152. For the transformer blocks, we use 16 attention heads and apply several techniques such as query-key normalization (QK norm) (Henry et al. (2020) and rotary position embeddings (RoPE) (Su et al. (2024)) to enhance the model's performance. Additionally, we leverage masking techniques to enable the model to support both text-to- video generation and video extension tasks. Specifically, we unmask the frames that the model should condition during video extension tasks. In the forward pass the base model, unmasked frames are assigned a timestep value of 0, while the remaining frames retain their original timesteps. The pseudo-codes
Masked Temporal Diffusion Transformer.
25

---

Published as a conference paper at ICLR 2025
of our feature processing pipeline and the Masked Temporal Transformer block shown in the following.
class BaseModel:
initialize (config))
# Step 1: Set base configurations
set piea_sigma, in_channels, out_channels, and model depth based on config
initialize hidden size and positional embedding parameters
Define embedding layers create patch embedder for input create timestep embedder for temporal information create caption embedder for auxiliary input create positional embeddings for spatial and temporal contexts
# Step 2:
12
Define processing blocks create spatial blocks for frame-level operations temporal blocks for sequence- operations
# Stop 3:
# Step 4: Define final output layer
initialize the final transformation layer to reconstruct output
19
30
function forward (x, timestep, y, mask-None, x_mask=None, fps-Ncne, height-None, width-Nene): # Step 1: Frepare inputs
preprocess x, timestep, and y for model input
# Step 2: Compute positional embeddings
derive positional enbeddings based on input size and dynamic dimensions
# Step 3: Compute timestep and auxiliary embeddings encode timestep information encode auxiliary input (e.g., captions) if provided
Step 4:
Enbed input video
apply spatial and temporal embeddings to video input
Step 5; Process through spatial and temporal blocks for each spatial temporal block pair: apply spatial block to refine frame-level features apply temporal block to model dependencies across frames
36
Step 6: Finalize output
transform processed features to reconstruct the output
return final output
class TemporalTrans formerBlock: TorporraltrammfoomeeBBllooo
initialize tiidee__size, num_heads): set hidden_sia
create Temporalattention with hicden_sion and heads
create LayerNorm with hidden_size
function t_mask_select (x_mask, masked_x, T, S): reshape x to [B, т, S, C]
IB, CI apply mask: where x mask True, keep values from x; otherwise, use masked
reshape masked
26

---

Published as a conference paper at ICLR 2025
reshape result back [B, T return result
CJ
13
function forward (x, X_mask-None, T-None, S-None) set x_m to (modulated input)
13
if x_mask is not None:
18
create masked version of x with zeros
repiace x with masked using t _amass__selee
apply attention to x_m
23
if x_mask is not None: reapply mask to output using t m_mask_seeee,
add residual connection (x + x_m)
23
apply layer normalization return final output
InstructNet Our InstructNet consists of 28 InstructNet Blocks, alternating between Spatial and Tem- consists poral Attention mechanisms, with each type accounting for half of the total blocks. The attention mechanisms and dimensionality
InstructNet Blocks maintain consistency with the base model. The InstructNet Block incorporates textual instruction information through an Instruction Fusion Expert utilizing cross-attention, while keyboard operations are integrated via an Operation Fusion Expert through feature modulation. Keyboard inputs initially projected into one-hot encodings, and then transformed through an MLP to match the latent feature dimensionality. The resulting key- board features are processed through an additional MLP to predict affine transformation parameters, which are subsequently applied to modify the latent features. Vidco prompts are incorporated into InstructNet through additive fusion the embedding layer.
C.3 COMPUTATION RESOURCES AND COSTS
Regarding computational resources, our training infrastructure consisted of 24 NVIDIA H800 GPUs distributed across three servers, with each server hosting 8 GPUs equipped with 80GB of memory per unit. We implemented distributed training across both machines and GPUs, leveraging Zero- 2 optimization to reduce computational overhead. The training process was structured into two phases: the base model training, which took approximately 25 days, and the InstructNet training phase, completed in 7 days. For storage, we utilized approximately 50TB to accommodate the dataset and model checkpoints.
D EXPERIMENT DETAILS AND FURTHER ANALYSIS
D.1 FAIRNESS STATEMENT AND CONTRIBUTION DECOMPOSITION
In our experiments, we compared four models (OpenSora-Plann, OpenSora, MiraDiT, and Cog Video- X) and five commercial models (Gen-2, Kling 1.5, Tongyi, Pika, and Luma). OpenSora-Plan, Open- Sora, and MiraDiT explicitly state that their training datasets (Panda-70M, MiraData) include significant amount of 3D game//ngine- rendered scenes. This makes them suitable baselines for evaluating game content generation. Additionally, while Cog Video-X and commercial models do disclose training data, their outputs suggest familiarity with similar visual domains. Therefore, the comparisons are fair in the context of assessing game content generation capabilities. To address concerns about potential overlap between training and test data, we ensured that the test set included only content types not explicitly present in the training set.
Additionally, to disentangle the effects of data and framework design, we sampled 10K subsets from both MiraData (which contain high-quality game video data) and OGameData and conducted set of ablation experiments with OpenSora (a state-of-the-e open-sourced video generation framework). The results follows:
27

---

Published as a conference paper at ICLR 2025
Table 7: The decomposition of contributions from OGameData and model design
Model
FID
FVD
TVA
UP
MS
DD
SC
IQ
Ours w/ OGameData
289.5
1181.3
0.83
0.67
0.99
0,64
0.95
0.49
OpenSora w/ OGameData
295.0
1186.0
0.70
0.48
0.99
0.84
0.93
0.50
Ours w/ MiraData
303.7
1423.6
0.57
0.30
0.98
0.96
0.91
0.53
As shown in the table above, we supplemented a comparison with OpenSora on MiraData. In comparing Domain Alignment Metries(averaged FID and FVD scores) and Visual Quality Metrics (averaged TVA, UP, MS, DD, SC, and IQ scores), our framework and dataset demonstrate clear advantages. Aligning the dataset (row and row 2), it can observed that our framework (735.4, 0.76) outperforms the OpenSora framework (740.5, 0.74), indicating the advantage of our architec- ture design. Additionally, fixing the framework, the model training on the OGameData (735.4, 0.76) surpasses the model training on MiraData (863.65, 0.71), highlighting our dataset's superiority in the gaming domain. These results confirm the efficacy of our framework and the significant advantages of our dataset.
D.2 EXPERIMENTAL SETTINGS
In this section, we delve into the details of our experiments, covering the calculation of metries, implementation details, evaluation datasets, and the details of our ablation study.
Evaluation Benchmark. To evaluate the performance of our methods and other benchmark methods, we constructed two evaluation datasets: OGameEval-Gen and OGlmeEvvl-Inss. The OGl dataset contains 50 text-video pairs sampled from the OGameData-GENN dataset, ensuring that these samples were not used during training. For fair comparison, the captions were generated using GPT-4o. For the OGameEval-Ins dataset, we sampled the last frame of ten videos from the OGameData--Ny INN eval dataset, which were also unused during training. We generated two types of instructional captions for each video: character control (e.g., move-left, move-right) and environment control (e.g., turn to rainy, turn sunny, turn to foggy, and create river in front of the main character). Consequently, we have 60 text-video pairs for evaluating control ability. To ensure fair comparison, for each instruction, we utilized GPT-4o to generate two types of captions: Structural instructions to evaluate our methods and dense captions to evaluate other methods.
Metric Details. To comprehensively evaluate the performance of GameGen-X, we utilize suite of metrics that capture various aspects of video generation quality and interactive control. This implementation is based on VBench (Huang et al. (2024b)) and CogVideo3. (Yang al.] (2024). By employing this set of metrics, we aim to provide a comprehensive evaluation of GameGen- - capabilities in generating high-quality, realistic, and interactively controllable video game content. The details are following:
FID (Fréchet Inception Distance) (Heusel et al. (2077)): Measures the visual quality of generated frames by comparing their distribution to real frames. Lower scores indicate better quality.
FVD (Fréchet Video Distance) (Rakhimov el al. (2020): Assesses the temporal coherence and over- all quality of generated videos. Lower scores signify more realistic and coherent video sequences.
UP (User Preference) (Yang et al. (2024)): In alignment with the methods of Cog we im- plemented single-blind study to evaluate video quality. The final quality score for each video is the average of evaluations from all ten experts. The details are shown Table S
TVA (Text-Video Alignment) (Yang el al. (2024)): Following the evaluation criteria established by VogVideoX, we conducted a bingle-blind study to assess tixt-video aligmment. The final quality score for each video is the average of evaluations from all ten experts. The details are shown in Table 9
SR (Success Rate): We assess the model's control capability through collaboration between hu- mans and AI, calculating a success rate. The final score is the average, and higher scores reflect models with greater control precision.
MS (Motion Smoothness) (Huang et al. (20246)) Measures the fluidity of motion in the generated videos. Higher scores reflect smoother transitions between frames.
28

---

Published as a conference paper at ICLR 2025
DD (Dynamic Degrees) (Huang er al. (2024b Assesses the diversity and complexity of dynamic elements in the video. Higher scores indicate richer and more varied content.
SC (Subject Consistency) (Huang et al.
2024b)): Measures the consistency of subjects (c.g., char- acters, objects) throughout the video. Higher scores indicate better consistency.
2024b) Measures the technical quality of the generated frames, including sharpness and resolution. Higher scores indicate clearer and more detailed images.
IQ (Imaging Quality) (Huang et al.
Table User Preference Evaluation Criteria.
Score
Evaluation Criteria
High video quality: 1. The appearance and morphological features of objects in the video completely consistent 2. High picture stability, maintaining high resolution consistently Overall compostion/colorrbuunaaiic match reality 4. The picture is visually appealing
0.5 Average video quality: 1. The appearance and morphological features objects in the video are at least 80% consistent 2. Moderate picture stability, with only 50% of the frames maintaining high resolution 3. Overall compostioou/olor/buuadree match reality by at least 70% 4. The picture has some visual appeal
0 Poor video quality: large inconsistencies in appearance and morphology, low video resolution, and compossttion/lyyout not matching reality
Table Text-video Alignment Evaluation Criteria.
Score
Evaluation Criteria
100% follow the text instruction requirements, including but not limited to: elements completely correct, quantity requirements consistent, elements complete, features curate, etc.
0.5 100% follow the text instruction requirements, but the implementation has minor flaws such distorted main subjects inaccurate features.
0 Does not 100% follow the text instruction requirements, with any of the following is- sues: Generated elements are inaccurate 2. Quantity incorrect 3. Elements are incomplete 4. Features are inaccurate
Ablation Experiment Design Details. We evaluate our proposed methods from two perspectives: generation and control ability. Consequently, we design comprehensive ablation study. Due to the heavy cost of training on our OGameData-GEN, dataset, we follow the approach of Pixart- alpha (Chen et al.] (2023)) and sample smaller subset for the ablation study. Specifically, we sample 20k samples from OGameData-GEN to train the generation ability and 10k samples from Srameeata-GEN-AAe and OGameData Data---aa-INN--bbl The generation process is trained for 3 epochs, while the control pro- cess trained for epochs. All experiments are conducted on H800 GPUs, utilizing the PyTorch framework. Here, we provide a detailed description of our ablation studies:
OGameData INS to train the control ability. This resulted in two datasets,
The baseline's setting is aligned with our model. Only utilizing a smaller dataset and training 3 epochs for generation and 2 epochs for instruction tuning with InstructNet.
Baseline:
w/ MiraData: To demonstrate the quality of our proposed datasets, we sampled the same video hours sample from MiraData. These videos are also from the game domain. Only utilizing this dataset, we train a model for 3 epochs.
w/ Short Caption: To demonstrate the effectiveness of our captioning methods, re-caption the OGameData DaameData-Gen- -eb dataset using simple and short captions. We train the model's generation ability for 3 epochs and use the rewritten short OGameEval-Gers captions to evaluate this variant.
w/ Progressive Training: To demonstrate the effectiveness our mixed-scale and mixed-tempont training, adopt different training method. Initially, we train the model using 480p resolution and 102 frames for 2 epochs, followed by training with 720p resolution and 102 frames for an additional epoch.
29

---

Published as a conference paper at ICLR 2025
w/o Instruct Caption: We recaption the OGameData-INS-Ab dataset our ablation utilizing dense caption. Based on this dataset and the baseline model, we train the model with InstructNet for 2 epochs to evaluate the effectiveness of our proposed structural caption methods.
w/o Decomposition: The decoupling of generation and control tasks is essential in our approach. In this variant, we combine these two tasks. We trained the model on the merged DaaeneDDta-Ge--AAA dataset for 5 epochs, splitting the training equally: 50% for generation and 50% for instruction tuning.
and OGameDDta-INS
w/o InstructNet: To evaluate the effectiveness of our InstructNet, we utilized OGameDData-NN-AAbl to continue training the baseline model for control tasks for epochs.
D.3 HUMAN EVALUATION DETAILS
Overview. We recruited 10 volunteers through online application process, specifically select- ing individuals with both gaming domain expertise and AIGC community experience. Prior to the evaluation, all participants provided informed consent. The evaluation framework was designed to assess three key metrics: user preference, video-text alignment, and control success rate. We im- plemented blind evaluation protocol where videos and corresponding texts were presented without model attribution. Evaluators were not informed about which model generated each video, ensuring unbiased assessment.
User Preference. To assess the overall quality of generated videos, we evaluate them across several dimensions, such as motion consistency, aesthetic appeal, and temporal coherence. This evaluation focuses specifically on the visual qualities of the content, independent of textual prompts or control signals. By isolating the visual assessment, we can better measure the model's 's ability to generate high-quality, visually compelling, and temporally consistent videos. To ensure unbiased evalua- tion, volunteers were shown the generated videos without any accompanying textual prompts. This approach allows to focus solely visual quality metrics, such as temporal consistency, composi- tion, object coherence, and overall quality. The evaluation criteria in Table & consisst of three distinct quality tiers, ranging from high-quality outputs that demonstrate full consistency and visual appeal to low-quality outputs that exhibit significant inconsistencies in appearance and composition.
Video Alignment. The text-video alignment evaluation aims to assess how well the model can follow textual instructions to generate visual content, with a particular focus on gaming-style aesthetics. This metric looks at both semantic accuracy (how well the text elements are represented) and stylistic consistency (how well the vidco matches the specific gaming style), providing a mea- sure of the model's ability to faithfully interpret textual descriptions within the context of gaming- Evaluators were shown paired video outputs along with their corresponding textual prompts. The evaluation framework focuses on two main aspects: (I) the accuracy of the implementation of in- structional elements, such as object presence, quantity, and feature details, and (2) how well the video incorporates gaming-specine visual aesthetics. The evaluation criteria in Table Juse a three- tier scoring system: score of 1 for perfect alignment with complete adherence instructions, 0.5 for partial success with minor flaws, and 0 for significant deviations from the specified requirements. This approach provides clear, quantitative way to assess how well the model follows instructions, while also considering the unique demands of generating game-style content.
The purpose of the control success rate evaluation is to assess the model's ability to accurately follow control instructions provided in the prompt. This evaluation focuses on how well the generated videos follow the specified control signals while maintaining natural transitions and avoiding any abrupt changes or visual inconsistencies. By combining human judgment with Al-assisted analysis, this evaluation aims to provide a robust measure of the model's performance in responding to user controls. We implemented a hybrid evaluation approach, combining feedback from human evaluators and Al-generated analysis. Volunteers were given questionnaires where they watched the generated videos and assessed whether the control instructions had been successfully followed. For each prompt, we generated three distinct videos using different random seeds to en- sure diverse outputs. The evaluators scored each video: score of was given if the control was successfully implemented, and O if it was not. The criteria for successful control included strict adherence to the textual instructions and smooth, natural transitions between scenes without abrupt changes or visual discontinuities. In addition to human evaluations, we used PLLa (Xu et al.) (0244) to generate captions for each video, which were provided to the evaluators as supplemen-
Success Rate.
30

---

Published as a conference paper at ICLR 2025
tary tool for assessing control success. Evaluators examined the captions for the presence of key control-related elements from the prompt, such as specific keywords or semantic information (e.g., "turn left," "rainy," or "jump"). This allowed for secondary validation of control success, ensuring that the wener--eneerated content matched the intended instructions both visually and semantically. For each prompt, we computed the success rate for each model by averaging the scores from the human evaluation and the Al-based caption analysis. This dual-verification process provided com- prehensive assessment of the model's control performance. Higher scores indicate better control precision, reflecting the model's 's ability to accurately follow the given instructions.
D.4 ANALYSIS OF GENERATION SPEED AND CORRESPONDING PERFORMANCE
In this subsection, we supplement our work with experiments and analyses related to generation speed and performance. Specifically, we conducted 30 open-domain generation inferences on single A800 and single H800 GPU, with the CUDA environment set to 12.1. We recorded the time and corresponding FPS, and reported the VBench metrics, including SC, background consistency (BC), DD, aesthetic quality (AQ), IQ, and averaged score of them (overall).
Generation Speed. The Table I0 reported the generation speed and corresponding FPS. In terms of generation speed, higher resolutions and more sampling steps result in increased time consumption. Similar to the conclusions found in GameNGen (Valevski et al. (2024)) the model generates videos with acceptable imaging quality and relatively high FPS at lower resolutions and fewer sampling steps (e.g., 320x256, 10 sampling steps).
Table 10: Performance comparison between A800 and H800
Resolution
Frames
Sampling Steps
Time (A800)
FPS (A800)
Time (H800)
FPS (H800)
320 x 256
102
10
~7.5s/sample
13.6
~5. Is/smple
20.0
848 x 480
102
10
~60s/sample
1.7
~20. ~00 Is/sapple
5.07
848 x 480
30
1366//smmle
0.75
<44.1 Is/samlee
2.31
848 x
50
196s/sumple
0.52
1.47
1280 x 720
10
1600/sample
0.64
2.66
1280 720
102
30
3315/sample
0.32
~57.55/samplcc
1.77
1280 x 720
102
50
4555s/sampl
0.23
~~160. 1-16o.is/sample
0.64
Performance Analysis. From Table [1] we can observe that increasing the number of sampling steps generally improves visual quality at the same resolution, reflected in the improvement of the Overall score. For example, at resolutions of 848x480 and 1280x720, increasing the sampling steps from 10 50 significantly improved the Overall score, from 0.737 to 0.800 and from 0.655 to 0.812, respectively. This suggests that higher resolutions typically require more sampling steps achieve optimal visual quality. On the other hand, we qualitatively studied the generated videos. We observed that at resolution of 320p, our model can produce visually coherent and texture- rich results with only 10 sampling steps. As shown Fig. [6) details such as road surfaces, cloud textures, and building edges are generated clearly. At this resolution and number of sampling steps, the model can achieve FPS on a single H800 GPU. We also observed the impact of sampling steps the generation quality at 480p/720p resolutions, shown in Fig. [7] At 10 sampling steps, we observed a significant enhancement in high-frequence details. Sampling with 30 and 50 steps not only further enriched the textures but also increased the diversity, coherence, and overall richness of the generated content, with more dynamic effects such as cape movements and ion effects. This aligns with the quantitative analysis metrics.
Table 11: Performance metrics for different resolutions and sampling steps
Resolution
Frames
Sampling Steps
SC
BC
DD
AQ
IQ
Average
320 256
102
10
0.944
0.962
0.4
0.563
0.335
0.641
848 x 480
102
0.947
0.954
0.8
0.598
0.389
0.737
848 x 480
102
30
0.964
0.960
0.9
0.645
0.573
0,808
848 480
102
50
0.955
0.961
0.9
0.615
0.570
0.800
1280 720
102
0.957
0.963
0.3
0.600
0.453
0.655
1280 x 720
102
30
0.954
0.956
0.7
0.617
0.558
0.757
1280 720
102
50
0.959
0.959
0.8
0.657
0.584
0.812
31

---

Published as a conference paper at ICLR 2025
Figure 16: Generated scenes with resolution of 320x256 and 10 sampling steps. Despite the lower resolution, the model effectively captures key scene elements.
Sampling Steps
10 Sampling Steps
30 Sampling Steps
50 Sampling Steps
Figure 17: Generated scenes at a resolution of 848x480 with varying sampling steps: 5, 10, 30, and 50. As the number of sampling steps increases, the visual quality of the generated scenes improves significantly.
32

---

Published as a conference paper at ICLR 2025
Figure 18: Character Generation Diversity. The model demonstrates its capability to generate a wide range of characters. The first three rows depict characters from existing games, showcasing detailed and realistic designs. The last two rows present open-domain character generation, illus- trating the model's 's versatility in creating unique and imaginative characters.
D.5 FURTHER QUALITATIVE EXPERIMENTS
Basic Functionality. Our model is designed to generate high-quality game videos with creative content, as illustrated in Fig. 18 Fig. 19 Fig. 20 and Fig 21 It demonstrates a strong capability for diverse scene generation, including the creation of main characters from over 150 existing games well novel, out-of-domain characters. This versatility extends to simulating wide arra of actions such as flying, driving, and biking, providing a wide variety of gameplay experiences. In addition, our model adeptly constructs environments that transition naturally across different seasons, from spring to winter. It can depict a range of weather conditions, including dense fog, snowfall, heavy rain, and ocean waves, thereby enhancing the ambiance and immersion of the game. By introducing diverse and dynamic scenarios, the model adds depth and variety to generated game content, offering glimpse into potential engine-like features from generative models.
Open-domain Generation Comparison. To evaluate the open-domain content creation capabilities of our method compared other open-source models, we utilized GPT-4o randomly generate captions. These captions were used to create open-domain game video demos. We selected three distinct caption types: Structural captions aligned with our dataset, short captions, and dense and general captions that follow human style. The results for Structural captions are illustrated in Fig. 23 Fig. 22
Fig. 24 Fig. 25 Fig. 26) and Fig. 27 The outcomes for short captions are depicted in Fig. 28 and Fig. 29 while the results for dense captions are visualized in Fig. 30 For each caption type,
24 we generated a scene depicting warrior walking through stormy wasteland. The results show that CogVideox lacks scene consistency due to dramatic light changes. In contrast, Opensora-Plan fails to accurately follow the user's instructions by missing realistic lighting effects. Additionally, Opensora's output lacks dynamic motion, as the main character appears to glide rather than walk. Our method achieves superior results compared these approaches. providing a more coherent and accurate depiction. We selected the scene visualized in Fig. 29Jas an example of short caption generation. As depicted, the results from Cog VideoX fail to fully capture the textual description, particularly missing the ice-crystal hair of the fur-clad wanderer. Additionally, Opensora-Plan lacks the auroras in the sky, and Opensora's 's output also misses the ice-crystal hair feature. These shortcomings highlight the robustness of our method, which effectively interprets and depicts details even with concise captions. The dense caption results are visualized in Fig. 30 Our method effectively captures the text details, including the golden armor and the character standing atop cliff. In contrast, other methods fail to accurately depict the golden armor and the cliff, demonstrating the superior capability of our approach in representing detailed information.
we selected one example for detailed analysis. As illustrated in Fig.
33

---

Published as a conference paper at ICLR 2025
Figure 19: Action Variety Scene Generation. The model effectively demonstrates diverse action
scenarios. From top bottom: piloting helicopter, flying through canyon, third-person person driving, first-person motorcycle riding, and third-person motorcycle riding. Each row showcases the model's dynamic range in generating realistic and varied action sequences.
Figure 20: Environmental Variation in Scene Generation. The model illustrates its capability to produce diverse environments. From top to bottom: a summer scene with an approaching hurricane, snow-covered winter village, summer thunderstorm, lavender fields in summer, and snow- covered winter landscape. These examples highlight the model's 's ability to capture different seasonal and weather conditions vividly.
34

---

Published as a conference paper at ICLR 2025
Figure 21: Event Diversity in Scene Generation. The model showeases its ability to depict range of dynamic events. From top to bottom: dense fog. raging wildfire, heavy rain, and powerful ocean waves. Each scenario highlights the model's capability to generate realistic and intense atmospheric conditions.
CogVideeX
OpenSera OpenSora-Pin
GameGen
Figure 22: Structural Prompt: A spectral mage explores haunted mansion filled with ghostly apparitions. In "Phantom Manor," the protagonist, a mysterious figure shrouded in ethereal robes, glides through the dark, decaying halls of ancient mansion. The walls are lined with faded portraits and cobweb-covered furniture. Ghostly apparitions flicker and out of existence, their mournful wails echoing through the corridors. The mage's staff glows with faint, blue light, illuminating the path ahead and revealing hidden secrets. The air is thick with an cerie, supernatural presence, creating chilling, immersive atmosphere. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.
35

---

Published as a conference paper at ICLR 2025
GameGen. Open OpenSora CogVideok
Figure 23: Structural Prompt: A robotic explorer traverses canyon filled with ancient, alien ruins. In "Mechanized Odyssey," the main character, sleek, humanoid robot with glowing core, navi- gates through vast, rocky canyon. The canyon walls adorned with mysterious, ancient carvings and partially buried alien structures. The robot's sensors emit soft, blue light, illuminating the path ahead and revealing hidden details in the environment. The sky is deep, twilight purple, with distant stars beginning to appear, adding to the sense of exploration and discovery. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.
CogVideox
OpenSora OpenSora
GameGen-X
Figure 24: Structural Prompt: A lone warrior walks through stormy wasteland, the sky filled with lightning and dark clouds. "Stormbringer"; the protagonist, clad weathered armor with glowing amulet, strides through barren, rocky landscape. The ground is cracked and dry, and the air is thick with the smell of ozone. Jagged rocks and twisted metal structures dot the horizon, while bolts of lightning illuminate the scene intermittently. The warrior's path is lit by the occasional flash, creating dramatic and foreboding atmosphere. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.
36

---

Published as a conference paper at ICLR 2025
OpenSon-ffan CogVideok
OpenSora
GameGon
Figure 25: Structural Prompt:A cybernetic detective walks down a neon-lit alley in a bustling city. In "Neon Shadows," the protagonist wears a trench coat with glowing circuitry, navigating through a narrow alley filled with flickering holographic advertisements, Rain pours down, causing puddles on the ground to reflect the vibrant city lights. The buildings loom overhead, casting long shadows that create a sense of depth and intrigue. The detective's 's steps are steady, their eyes scanning the surroundings for clues in this cyberpunk mystery. aesthetic score: 6.55, motion score: 12.69, per- spective: Third person.'
Opersorra CogVideoX
OpenSora
G
Figure 26: Structural Prompt: A spectral knight walks through a haunted forest under a blood- red moon. In "Phantom Crusade," " the protagonist, a translucent, ethereal figure clad in spectral armor, moves silently through dark, misty forest. The trees are twisted and gnarled, their branches reaching out like skeletal hands. The blood-red moon casts an eerie light, illuminating the path with sinister glow. Ghostly wisps float through the air, adding to the chilling atmosphere. The knight's armor shimmers faintly, reflecting the moonlight and creating a hauntingly beautiful scene. aesthetic score; 6.55, motion score: 12.69, perspective: Third person.
37

---

Published as a conference paper at ICLR 2025
CogVideo
OpenSora Paan
OpenSore
GameGen
Figure 27: Structural Prompt: A cybernetic monk walks through high-tech temple under serene sky. In "Digital Zen," the protagonist, a serene figure with cybernetic enhancements integrated into their traditional monk robes, walks through temple that blends ancient architecture with ad technology. Soft, ambient lighting and the gentle hum of technology create peaceful atmosphere. The temple's walls are adorned with holographic screens displaying calming patterns and mantras. The monk's 's cybernetic components emit faint, soothing glow, symbolizing the fusion spirituality and technology in this tranquil sanctuary. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.
OpenSera CogVideox
OpenSora
GameSen.
Figure 28: Short Prompt: "Echoes of the Void": A figure cloaked in darkness with eyes like stars walks through a valley where echoes of past battles appear as ghostly figures. The ground littered with ancient, rusted weapons, and the sky is an endless void with single, massive planet looming close, its rings casting cerie shadows.
38

---

Published as a conference paper at ICLR 2025
OpenSora- CogVidooX
Opentorr
GameGen
Figure 29: Short Prompt: "Glacier Wanderer": A fur-clad wanderer with ice-crystal hair treks across a glacier under sky painted with auroras. Giant ice sculptures of mythical creatures line his path, each breathing out cold mist. The horizon shows mountains that pierce the sky, glowing with an inner light.
OpenSora OpenSora-PPo Cogvideox
GameGon.X
Figure 30: Dense Prompt: lone Tarnished warrior, clad in tattered golden armor that glows with inner fire, stands atop cliff overlooking vast, blighted landscape. The sky burns with an other- worldly amber light, casting long shadows across the desolate terrain. Massive, twisted trees with bark-like blackened iron stretch towards the heavens, their branches intertwining to form grotesque arches. In the distance, colossal ring structure hovers on the horizon. its edges shimmering with arcane energy. The air thick with ash and embers, swirling around the warrior in mesmerizing patterns. Below, a sea of mist conceals untold horrors, occasionally parting to reveal glimpses of ancient ruins and fallen titans. The warrior raises curved sword that pulses with crimson runes, preparing to descend into the nightmarish realm below. The scene exudes a sense of epic scale and foreboding beauty, capturing the essence of world the brink of cosmic change.
39

---

Published as a conference paper at ICLR 2025
RunWoy
Pica
TongWi
Gameeenn- KLnnd.
Figure 31: Comparison results of GameGen- X with commercial models. This figure contrasts our approach with several commercial models. The left side displays results from rext-generated videos, while the right side shows text-based continuation of videos. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Kling1.55, and GameCin-XX. Luma, Kling and GameGen-X: effectively followed the caption in the first part, including capturing the fiery red sky, while Gen2, Pika, and Tongyi did not. the second part, our method successfully directed the character to turn right, control other methods struggled achieve.
Interactive Control Ability Comparison. To comprehensively assess the controllability our model, we compared it with several commercial models, including Runway Gen2, Pika, Luma, Tongyi, and KLing 1.5. Initially, we generated a scene using the same caption across all models. Subsequently, extended the video incorporating text instructions related environmental changes and character direction. The results are presented in Fig. 31 Fig. 32 Fig. 33 Fig. 34 and Fig. 35 Our findings reveal that while commercial models produce high-quality outputs, Runway, Pika, and Luma fall short of meeting game demo creation needs due to their free camera perspectives, which lack the dynamic style typical games. Although Tongyi and KLing can gen- erate videos with a game-like style, they lack adequate control capabilities; Tongyi fails to respond environmental changes and character direction, while KLing struggles with character direction adjustments.
Video Prompt. In addition text and keyboard inputs, our model accepts video prompts, such as edge sequences motion vectors, as inputs. This capability allows for more customized video generation. The generated results are visualized in Fig. an Fig.
40

---

Published as a conference paper at ICLR 2025
RunWay
Pila
Yoney
Lurra
King1.5
GarmeGenxx
Japo
keyond the bool
Figure 32: Comparison results GameGen-X with commercial models. This figure presents com- parison between our approach and several commercial models. The left side depicts text-generated video results, while the right side shows text-based video continuation. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Kling1.5, and GameGen-X. In the initial seg- ment, Luma, Kling .5, and GameGin-X effectively adhered to the caption by accurately depicting the dense fog and path, while other models lacked these elements. In the continuation, only Klingl ..5 and our approach successfully transformed the environment by clearing the fog, whereas other meth- ods failed to follow the text instructions.
41

---

Published as a conference paper at ICLR 2025
RunWayy
Pika
TongYi
Luma
GameCen-X Klng1.5
SAT frees foveing covveny nemver etics
Figure 33: Comparison results of GameGen-X with commercial models. This figure compares our approach with several commercial models. The left side displays text-generated video results, while the right side shows text-based video continuation. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Kling1.5, and our method. In the initial segment, all methods effectively followed the caption. However, in the continuation segment, only our model successfully controlled the character to turn left.
42

---

Published as a conference paper at ICLR 2025
RunWay
Tonehi
KLinl1
GameGGe--
pATTIE SIr OO OUSTE of CONNVNNT doCNTO WOODD
Figure 34: Comparison results of GameGen-X with commercial models. This figure presents comparison between our approach and several commercial models. The left side showcases text- generated video results, while the right side illustrates video continuation using text. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Klingl.5,, and our method. In first segment, only Pika, K.ingl.5, and our method correctly followed the text description. Other models either failed to display the character or depicted them entering cave instead of exiting. In the continuation segment, both method and Kling1.5 successfully guided the character out of the cave. Our approach maintains a consistent camera perspective, enhancing the game- like experience compared to K.ing1.55.
43

---

Published as a conference paper at ICLR 2025
RunWoyy
PiKa
Toneli
Esmma
Klinsts
GameGeena
Jeaneg warting bertiaattyy
Figure 35: Comparison results GameGen-X with commercial models. This figure presents com- parison between approach and several commercial models. The left side shows text- eext-generated video results, while the right side illustrates video continuation using text. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Kling1.5, and our method. In the initial seg- ment, all methods successfully followed the text description. However, in the continuation segment, only our method effectively altered the environment by darkening the sky and revealing the stars.
Figure 36: Video Generation with Motion Vector Input. This figure demonstrates how given motion vectors enable the generation of videos that follow specific movements. Different environments were created using various text descriptions, all adhering to the same motion pattern.
44

---

Published as a conference paper at ICLR 2025
Figure 37: Video Scene Generation with Canny Sequence Input. Using the same canny sequence, different text inputs can generate video scenes that match specific content requirements.
45

---

Published as a conference paper at ICLR 2025
E DISCUSSION
E. LIMITATIONS
Despite the advancements made by GameGen-X, several key challenges remain:
Real-Time Generation and Interaction: In the realm of gameplay, real-time interaction is crucial, and there is significant appeal in developing a video generation model that enables such interactivity. However, the computational demands of diffusion models, particularly concerning the sampling process and the complexity spatial and temporal self-attention mechanisms, present formidable challenges.
Consistency Auto Retro-eegeesivve Generation: Auto-regressive generation often leads accumu- lated errors, which can affect both character consistency and scene coherence over long se- quences (Valevski et al. (2024). This issue becomes particularly problematic when revisiting pre- viously generated environments, as the model may struggle to maintain cohesive and logical pro- gression.
Complex Action Generation: The model struggles with fast and complex actions, such as combat sequences, where rapid motion exceeds its current capacity (Huang et al. (2024a)). these scenar- ios, video prompts are required to guide the generation, thereby limiting the model's autonomy and its ability to independently generate realistic, high-moo motion content.
Generation: GameGen-X is not yet capable of generating ultra-high-resolutiee content (e.g., 2K/4K) due to memory and processing constraints (He et al. (2024a)). The current hardware limitations prevent the model from producing the detailed and high-resolution visuals that are often required for next-gen AAA games, thereby restricting its applicability in high-end game development.
Long-Term Consistency in Video Generation: In gameplay, maintaining scene consistency is crucial, especially players transition between and return to scenes. However, model currently exhibits a limitation in temporal coherence due to its short-term memory capacity of just 1-108 frames. This constraint results in significant scene alterations upon revisiting, highlighting the need enhance our model's memory window for better long-term scene retention. Expanding this capability is essential for achieving more stable and immersive video generation experiences.
Physics Simulation and Realism: While our methods achieve high visual fidelity, the inherent con- straints of generative models limit their ability consistently adhere to physical laws. This includes realistic light reflections and accurate interactions between characters and their environments. These limitations highlight the challenge of integrating visually compelling content with the physical real- ism required for experience.
Multi- Ciir--hharcrrr Generation: The distribution our current dataset limits our model's ability to generate and manage interactions among multiple characters. This constraint is particularly evident in scenarios requiring coordinated combat or cooperative tasks.
Integration with Existing Game Engines: Presently, the outputs of our model are not directly com- patible with existing game engines. Converting video outputs into 3D models may offer a feasible pathway to bridge this gap, enabling more practical applications in game development workflows.
In summary, while GameGen-X marks a significant step forward in open-world game generation, addressing these limitations is erucial for its future development and practical application in real- time, high-resolution, and complex game scenarios.
E.2 POTENTIAL FUTURE WORKS
Potential future works may benefit from the following aspects:
Real-Time Optimization: One of the primary limitations of current diffusion models, including GameGen-X X, is the high computational cost that hinders real-time generation. Future research can focus optimizing the model for real-time performance (Zhao et al. (2024b a); Xuanlei Zhao & You (2024), essential for interactive gaming applications. This could involve the design of lightweight diffusion models that retain generative power while reducing the inference time. Addi-
46

---

Published as a conference paper at ICLR 2025
tionally, hybrid approaches that blend autoregressive methods with non-autoregressive mechanisms may strike a balance between generation speed and content quality (Zhou et al. (2024)). Techniques like model distillation multi-stage refinement might further reduce the computational overhead, allowing for more efficient generation processes (Wang et al. (2023b)). Such advances will be cru- cial for applications where instantaneous feedback and dynamic responsiveness are required, such as real-time gameplay and interactive simulations.
Improving Consistency: Maintaining consistency over long sequences remains significant chal- lenge, particularly in autoregressive generation, where small errors can accumulate over time and result in noticeable artifacts. To improve both spatial and temporal coherence, future works may in- corporate map-based constraints that impose global structural rules on the generated scenes, ensur- ing the continuity of environments even over extended interactions YYan et al. (2024). For character consistency, the introduction of character-conditione the visual and behavioral fidelity of in-game characters across scenes and actions (He et al., (2024b); Wang et al. )2024). This can be achieved by integrating embeddings that track identity, pose, and interaction history, helping the model to better account for long-term dependencies and minimize discrepancies in character actions appearances over time. These approaches could further enhance the realism and narrative flow in game scenarios by preventing visual drift.
embeddings could help the model maintain
Handling of Complex Actions: Currently, GameGen-X struggles with highly dynamic and complex actions, such as fast combat sequences or large-scale motion changes, due to limitations in capturing rapid transitions. Future research could focus enhancing the model's ability to generate realis- tic motion by integrating motion-aware components, such as temporal convolutional networks or recurrent structures, that better capture fast-changing dynamics (Huang al. (2024a)). Moreover, training on high-frame-rate datasets would provide the model with more granular ar temporal informa- tion, improving its ability to handle quick motion transitions and intricate interactions. Beyond data, incorporating external guidance, such as motion vectors or pose estimation prompts, can serve as ad- ditional control signals to enhance the generation of fast-paced scenes. These improvements would reduce the model's dependency on video prompts, enabling it to autonomously generate complex and fast moving actions in real-time, increasing the depth and variety of in-game interactions.
Advanced Model Architectures: Future advancements in model architecture will likely move to- wards full 3D representations to better capture the spatial complexity of open-world games. The current 2D+1D approach, while effective, limits the model's ' ability to fully understand and replicate 3D spatial relationships. Transitioning from 2D+1D attention-basee video generation to more so- phisticated 3D attention architectures offers exciting direction for improving the coherence and realism of generated game environments ( (Yang et al. (2024); Lab & etc. 2024)). Such a framework could better grasp the temporal dynamics and spatial structures within video sequences, improv- ing the fidelity of generated environments and actions. On the other dimension, instead of only focusing on the generation task, future models could integrate more unified framework that si- multaneously learns both video generation and video understanding. By unifying generation and understanding, the model could ensure consistent layouts, character movements, and environmental interactions across time, thus producing more cohesive and immersive content (Emu3 Team (2024)) This approach could significantly enhance the ability of generative models to capture complex video dynamics, advancing the state of video-based game simulation technology.
Scaling with Larger Datasets: While OGameData provides a comprehensive foundation for training GameGen-X, further improvements in model generalization could be achieved by scaling the dataset to include more diverse examples of game environments, actions, and interactions (Ju et al. (2024); Wang et al. (2023c)) Expanding the dataset with additional games, including those from wider range of genres, art styles, and gameplay mechanics, would expose the model to a broader set of scenarios. This would enhance the model's ability to generalize across different gaming contexts, al- lowing it to generate more diverse and adaptable content. Furthermore, incorporating user-generated content, modding tools, or procedurally generated worlds could enrich the dataset, offering a more varied set of training examples. This scalability would also improve robustness, reducing overfitting and enhancing the model's 's capacity to handle novel game mechanics and environments, thereby improving performance across a wider array of use cases.
Integration of 3D Techniques: A key opportunity for future development lies in integrating advanced 3D modeling with 3D Gaussian Splatting (3DGS) techniques (Kerbl et al. (2023)). Moving beyond 2D video-based approaches, incorporating 3DGS allows the model to generate complex spatial inter-
47

---

Published as a conference paper at ICLR 2025
actions with realistic object dynamies. 3DGS facilitates efficient rendering of intricate environments and characters, capturing fine details such as lighting, object manipulation, and collision detection. This integration would result in richer, more immersive gameplay, enabling players to experience highly interactive and dynamic game worlds (Shin et al. (024)).
Virtual Reality: A compelling avenue for future research is the potential to adapt these generative techniques beyond gaming into real-world applications. If generative models can accurately sim- ulate highly realistic game environments, it opens the possibility of applying similar techniques to real-world simulations in areas such as autonomous vehicle testing, virtual training environments, augmented reality (AR), and scenario planning. The ability create interactive, realistic, and con- trollable simulations could have profound implications in fields such as robotics, urban planning, and education, where virtual environments are used to test and train systems under realistic but controlled conditions. Bridging the gap between virtual and real-world simulations would not only extend the utility of generative models but also demonstrate their capacity to model complex, dy- namic systems in wide range of practical applications.
In summary, addressing these key areas future work has the potential to significantly advance the capabilities of generative models in game development and beyond. Enhancing real-time generation, improving consistency, and incorporating advanced 3D techniques will lead to more immersive and interactive gaming experiences, while the expansion into real-world applications underscores the broader impact these models can have.
48


Reviews:
[{"review_id": "HsuQ8QPAqT", "reviewer": "Reviewer_c3SK", "summary": "In this paper the authors present a new dataset of modern AAA games for the purpose of world model training, which they call the Open-World Video Game Dataset (OGameData). Then then present their model, GameGen-X, a diffusion transformer for generating and controlling game video. GameGen-X is similar to other video generation models with the addition of InstructNet, which modifies the latents of GameGen-X for controllability. The authors present comparisons with a number of open-source video models. Finding that they generally produce more game-like video and may be better at control, according to some metrics.", "questions": "1. Where did the dataset come from?\n2. Why splice up the data by scene?\n3. What was the methodology with the human expert evaluators? \n4. Did the authors receive ethics approval?\n5. The authors indicate the dataset is split nearly evening between first and third person video, but primarily show results for third person video, why is this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The primary strength of the paper without question is the authors' new dataset. There is no dataset even close to this in terms of quality or size, it's a really exciting potential addition to this research area. This is primarily a strength in terms of originality, quality, and significance. I say primarily since the authors do not include access to the dataset at the review stage, though they do not have some metrics. \n\nThe authors' GameGen-X and InstructNet are also strengths, but I have concerns with them limiting them as strengths, as I'll get to below.", "weaknesses": "The paper is relatively free of weaknesses in terms of originality, thanks in large part due to OGameData. However, the authors' work has some weaknesses in terms of the quality, clarity, and significance. This primarily comes down to (1) the authors' stated motivations and how this aligns with their work, (2) the way the authors overview their system, and (3) the experiments\n\n### Motivations and the Dataset\n\nThe authors motivate in two primary ways: (1) imagining this as a prototyping or early development tool for open world game developers and (2) imagining this as leading to future interactive experiences with greater user control. These are fine as motivations, but the authors' choice of processing the dataset runs counter to them, somewhat. Specifically, the authors have broken apart their video clips into distinct scenes, meaning their model or other models trained on this dataset will not observe scene transitions. This is somewhat of an oddity for either of the authors' stated motivations and there's no justification for this choice given in the paper. Similarly, the authors do not actually have control input for any of the collected game data. This is again a bit of an oddity given the authors' stated purposes. I would guess that the authors collected this data from some sort of web scrape of gameplay video rather than collecting the video themselves through playing games such that they could capture actual control inputs. However, this isn't specified in the paper. This is a potential concern, especially if the authors did scrape an online repository of videos when such scraping went against the terms and service of the site in question. This should be clarified. \n\n### System Overview\n\nSimply, the authors do not describe any of their system implementation in sufficient detail for replication. The authors state that code will be made available but do not make such code available for review. As such, there's no detail on the system architecture in terms of parameters or hyper parameters. The authors also not disclose the computation required to train their model or the training split used from their dataset. All of this would be required (potentially in appendices or in an external code repo) to ensure that the work is replicable. \n\n### Experiments\n\nI have a number of concerns with the current setup of the experiments. The authors only compare against open video models, which are not attempting the same task and are not trained on the same dataset. As such, it's unclear the extent to which this is just that the training dataset for GameGen-X is more similar to the test dataset. While the authors do specify that the experiments are over test data, given the distribution of games in the dataset, its highly likely that GameGen-X had already trained on the same game that the test data used in the experiments came from. As such, this seems much closer to testing on the training set. \n\nThe authors also have several metrics that require human expert raters, but who these experts were or what information they had is not specified. Further, the authors say they only use a single-blind setup, which may suggest the experts knew who they were. As such, there's a clear risk of bias here in terms of the experts feeling social pressure to more positively rate the more game-like videos if they knew that was the goal of this research. Clarity around the methodology and whether the authors have ethics approval would be necessary for readers to trust any of the human participant-based results. Relatedly, the authors state that the SR metric is \"evaluated by both human experts and PLLaVA\". However, the authors only present a single number. As such, it's not clear how the human expert and PLLaVA evaluations were combined. This throws doubt upon the SR metrics. \n\nThe authors repeatedly bold values from their own work, indicating it is the best, when there is equivalent work from prior models. This may mislead readers in terms of understanding when the performance. \n\nThe ablation study is helpful, as it demonstrates the value of OGameData, and several of the authors' components. However, since the authors do not train any other models on their dataset outside of these ablations, it's difficult to determine the exact value of the different components of their work. \n\nOverall, I'd say that the experiments are currently the largest weakness of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dbctPE2iDD", "reviewer": "Reviewer_bSLg", "summary": "This paper proposes GameGen-X, a diffusion based model for open-world game generation. Specifically, this paper proposes two detailed crafted datasets: OGameData-Gen and OGameData-Ins. OGameData-Gen is used to pre-train the diffusion model to understand and generate continuous open-world game-style videos, where OGameData-Ins is used to instruct tune the model to understand special inputs (e.g., keyboard inputs) to better control the continuation of the game generation based on some input frames. The dataset is well-curated to have 1M videos, with multiple filtering metrics and human-in-loop filtering to maintain the high quality. Then, this paper trains a video diffusion model with two-stage training on the two datasets for open-world game generation. Specifically, an instruct net is designed to take in different special inputs. Empirically, on their provided evaluation dataset, GameGen-X achieves superior performance than other state-of-the-art video diffusion models (e.g., kling).", "questions": "Please refer to weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A large well-curated dataset for open-world video games. The curation of the dataset contains filtering on different aspects (e.g., semantic alignment, motion), which results in a high-quality large-scale dataset. \n2. The idea to build a video diffusion model for open-world video games is essentially interesting, and the results and demo videos are impressive. Besides, quantitatively, the proposed approach also achieves better performance than other SoTA diffusion models. \n3. Detailed ablation studies demonstrate the effectiveness of the proposed component (i.e., two-stage training strategy and the design of the instructnet).\n4. This paper is well-written and easy to follow.", "weaknesses": "1. One main concern is the proposed GameGen-X is specially fine-tuned/designed for open-world video games, while other diffusion models compared (e.g., kling) are trained for a general text-to-video generation, which makes the comparison somehow unfair to other models. \n2. The qualitative examples in the website demo for game generation (e.g., under generation comparison) don't seem to look much better than other models (e.g., cogvideoX).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2TjAlZNpxt", "reviewer": "Reviewer_WQcm", "summary": "In this paper, the authors introduce a diffusion transformer model aimed at generating and controlling video game sequences in challenging 3D open-domain game worlds. The authors also present the gameplay dataset they collected to train the model, OGameData. The dataset has 1 million video clips from across 150 videos, annotated with text descriptions using GPT4o. Both the model and the dataset have 2 components. One for text-to-video generation (OGameData-GEN and the pretrained foundation model) and one for instruction tuning (OGameData-INS and InstructNet).", "questions": "Clarification Questions:\n\n-\tIs it the correct understanding that the OGameData-GEN dataset comprises of data from 150 video games, whilst the OGameData-INS dataset contains only a subset of 5 game titles? Without checking Appendix B for clarification, it is difficult for the reader to grasp these details from the main body of the paper.\n-\tFor video clip compression (Section 3.2) it would be good to add more details about the size of the latent representation z, as well as the resolution of the video clips used in training.\n-\tHow were the spatial and temporal downsampling factors determined (s_t, s_h, s_w)?\n-\tIn section 3.2, under unified video generation and continuation, you mention incorporating bucket training, classifier-free diffusion guidance and rectified flow for better generalization performance – did you run any ablation studies to understand better the impact of introducing these 3 components?\n-\tWhat are the values x for context length that you considered for video continuation?\n-\tIn the InstructNet design, what were the considerations for choosing N (the number of InstructNet blocks)? Did you experiment with different values?\n-\tFor the multi-modal experts introduced in Section 3.3, what are the sizes considered for the instruction embeddings and keyboard input embeddings (f_I and f_O)?\n-\tUnder Interactive control, you mention the incorporation of video prompts V_p enhances the model’s ability to generate motion-consistent frames – did you conduct any experiments or ablations to measure the observed improvement?\n-\tIs there a mention on the computational resources required to store and stream the data for training, as well as for training the foundation model and InstructNet? It would be a useful proxy for people planning to reproduce the work.\n-\tSimilarly, is there any information presented on the inference times of GameGen-X?\n-\tIn evaluating the control ability you mention using both human experts and PLLaVa. What is the ratio between the 2 evaluation modalities?\n-\tOn qualitative results for generation, apart from the discussion on diversity, would it be possible to elaborate on the length and consistency of the videos generated by GameGen-X? From the demo videos included, most are under <8-10 seconds.\n-\tIn the ablation studies (Tables 4 and 5), there seem to be no DD and IQ metrics – what is the reason for that?\n\nMinor comments/Suggestions:\n\n-\tIn Section 2.2, it would be good to specify the human experts’ level of familiarity with the titles and elaborate on how the GPT-4o text annotations were checked for quality and accuracy.\n-\tIn Section 3.3, you introduce the c condition under the Interactive Control subsection, but it is mentioned beforehand in Multi-modal experts. It would be clearer to the reader to introduce the structure of c under the Multi-modal experts’ subsection, where it appears for the first time.\n-\tFor readability, it would be good to illustrate z, the latent variable in Figure 4.\n-\tIt would be useful to include a more detailed explanation on the choice of baselines in the experiments Section. For example, Mira is not included under the results for control ability, is it because it does not have support for it? It would be good to clarify that.\n-\tIt would be good to link to Appendix D (Discussion) when mentioning remaining challenges in the conclusion.\n-\tI know this appeared after the submission deadline, but it would be worth adding to the related work section as a referece: https://www.decart.ai/articles/oasis-interactive-ai-video-game-model", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The work is original in the sense that is the first main contribution to the field in terms of interactive video game generation in large scale, complex, open worlds\n\n- It is great to see such examples of tackling complex research environments at scale, with potential direct benefits to the game development process. \n\n- The author(s) introduce a complex system, both in terms of the dataset it required for training (including a resource intensive collection and curation process), as well as in terms of the pretrained foundation model and the interactive control network, allowing users to control the output via either text or mouse and keyboard inputs", "weaknesses": "- There is one strong concern I have regarding the data collection process for the OGameData dataset. My score highly depends on evidence that data collection will pass the ethics review and there is evidence provided on the consent given by the humans that produced the data. There should be understanding and agreement for it to be used for research purposes and open sourced. Please elaborate on how the data for OGameData has been collected? In Appendix B.1. you mention selecting online video websites as one of the primary sources. It would be good to know:\n   - The exact sources of the video data\n   - Any agreements or permissions obtained from video creators and game studios\n   - The ethical review process they followed, if any\n   - How you plan to address potential copyright or licensing issues\n\n- It is unclear why all UI elements have been removed from the dataset, it would be great to gain further clarity on that from the author(s). In a lot of open-world gameplay , the player relies on UI element understanding, such as health levels, navigation information via mini maps, affordance of actions to take, inventory etc. \n   - How does this decision impact the model's ability to generate realistic gameplay experiences?\n   - Do you plan to incorporate UI elements in future iterations of the model?\n\n- Please correct me if I missed this, but the main body of the paper does not clearly indicate that all the data and the generation is within the constraints of a single agent. What is the model’s ability to model other dynamic environment elements (NPCs, other players, moving vehicles etc.)? It would be good to:\n   - Explicitly state whether the model is limited to single-agent scenarios\n   - If so, discuss the implications of this limitation on the model's applicability\n   - If not, provide details on how the model handles multiple dynamic elements in the environment\n\n- The paper is dense, so it took a while to disambiguate if the main body of the paper provides sufficient detail for capturing the core contributions of the paper or if a lot of essential details were included in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DiahPF6aVi", "reviewer": "Reviewer_1M8F", "summary": "This work focuses on generating high-quality, controllable open-world game videos that feature game engine traits. It emphasizes interactive controllability to simulate gameplay effectively. Notably, the authors collected a large-scale Open-World Video Game Dataset (OGameData), which consists of over one million diverse gameplay video clips from more than 150 games, along with informative captions generated by GPT-4o. Methodologically, they introduce a diffusion transformer model as the foundation model and a specially designed network called InstructNet for interactive control. The model is trained on the large-scale OGameData dataset using a two-stage process involving pre-training of the foundation model and instruction tuning for InstructNet.", "questions": "1.  Please provide data on the time required to generate a video segment at different resolutions or for different types of content. A section to analyze the trade-offs between generation quality and speed would be better.\n\n2.  The training details of InstructNet lack specificity regarding the acquisition of video data corresponding to keyboard bindings. It would be beneficial to include more comprehensive information on the data collection process and the training methodology employed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work collects a substantial number of open-world game videos from over 150 games, ultimately constructing more than 1,000,000 text-video pairs with highly detailed annotations. Its scale and diversity of annotations make it stand out, and the release of this dataset is expected to advance the field of game video generation.\n2. It produces high-quality, more general realistic game video content. Previous works on game video generation often focused on specific game types, primarily 2D games or limited early 3D games. This work offers a more diverse and high-definition range of scene types for game video generation.", "weaknesses": "1. This work attempts to address the interactive control of open-world game video generation for gameplay simulation. However, to fully tackle the interactive issue, the generation speed needs to be considered,  as interactive experiences demand stringent timing requirements, which poses significant challenges. For instance, Google’s [1] achieves real-time rendering, even making it a viable game engine. While this work focuses on higher-resolution video generation, exploring the relationship between speed and performance would be beneficial, along with providing data on rendering time and speed. \n\n[1] Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024.\n\n2. The paper claims to simulate game engine features like diverse events, yet the examples provided offer quite limited dynamic event simulation, primarily addressing environmental changes like weather and lighting. There remains a gap to true gameplay simulation, such as incorporating NPC interactions or triggering more game-like special events.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)