GameGen-$\mathbb{X}$: Interactive Open-world Game Video Generation

## Abstract

Abstract We introduce GameGen- 𝕏 𝕏 \mathbb{X} blackboard_X , the first diffusion transformer model specifically designed for both generating and interactively controlling open-world game videos.
This model facilitates high-quality, open-domain generation by simulating an extensive array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events.
Additionally, it provides interactive controllability, predicting and altering future content based on the current clip, thus allowing for gameplay simulation.
To realize this vision, we first collected and built an Open-World Video Game Dataset (OGameData) from scratch.
It is the first and largest dataset for open-world game video generation and control, which comprises over one million diverse gameplay video clips sampling from over 150 games with informative captions from GPT-4o.
GameGen- 𝕏 𝕏 \mathbb{X} blackboard_X undergoes a two-stage training process, consisting of foundation model pre-training and instruction tuning.
Firstly, the model was pre-trained via text-to-video generation and video continuation, endowing it with the capability for long-sequence, high-quality open-domain game video generation.
Further, to achieve interactive controllability, we designed InstructNet to incorporate game-related multi-modal control signal experts.
This allows the model to adjust latent representations based on user inputs, unifying character interaction and scene content control for the first time in video generation. During instruction tuning, only the InstructNet is updated while the pre-trained foundation model is frozen, enabling the integration of interactive controllability without loss of diversity and quality of generated video content.
GameGen- 𝕏 𝕏 \mathbb{X} blackboard_X represents a significant leap forward in open-world video game design using generative models.
It demonstrates the potential of generative models to serve as auxiliary tools to traditional rendering techniques, effectively merging creative generation with interactive capabilities.
The project will be available at https://github.com/GameGen-X/GameGen-X .

## 1 Introduction

Generative models (Croitoru et al. (); Ramesh et al. (); Tim Brooks & Ramesh (); Rombach et al. ()) have made remarkable progress in generating images or videos conditioned on multi-modal inputs such as text, images, and videos.
These advancements have benefited content creation in design, advertising, animation, and film by reducing costs and effort.
Inspired by the success of generative models in these creative fields, it is natural to explore their application in the modern game industry.
This exploration is particularly important because developing open-world video game prototypes is a resource-intensive and costly endeavor, requiring substantial investment in concept design, asset creation, programming, and preliminary testing (Anastasia ()).
Even early development stages of games still involved months of intensive work by small teams to build functional prototypes showcasing the game’s potential (Wikipedia ()).

Several pioneering works, such as World Model (Ha & Schmidhuber ()), GameGAN (Kim et al. ()), R2PLAY (Jin et al. ()), Genie (Bruce et al. ()), and GameNGen (Valevski et al. ()), have explored the potential of neural models to simulate or play video games.
They have primarily focused on 2D games like “Pac-Man”, “Super Mario”, and early 3D games such as “DOOM (1993)”.
Impressively, they demonstrated the feasibility of simulating interactive game environments.
However, the generation of novel, complex open-world game content remains an open problem.
A key difficulty lies in generating novel and coherent next-gen game content.
Open-world games feature intricate environments, dynamic events, diverse characters, and complex actions that are far more challenging to generate (Eberly ()).
Further, ensuring interactive controllability, where the generated content responds meaningfully to user inputs, remains a formidable challenge.
Addressing these challenges is crucial for advancing the use of generative models in game content design and development.
Moreover, successfully simulating and generating these games would also be meaningful for generative models, as they strive for highly realistic environments and interactions, which in turn may approach real-world simulation (Zhu et al. ()).

In this work, we provide an initial answer to the question: Can a diffusion model generate and control high-quality, complex open-world video game content?
Specifically, we introduce GameGen-$\mathbb{X}$, the first diffusion transformer model capable of both generating and simulating open-world video games with interactive control.
GameGen-$\mathbb{X}$ sets a new benchmark by excelling at generating diverse and creative game content, including dynamic environments, varied characters, engaging events, and complex actions.
Moreover, GameGen-$\mathbb{X}$ enables interactive control within generative models, allowing users to influence the generated content and unifying character interaction and scene content control for the first time.
It initially generates a video clip to set up the environment and characters.
Subsequently, it produces video clips that dynamically respond to user inputs by leveraging the current video clip and multimodal user control signals.
This process can be seen as simulating a game-like experience where both the environment and characters evolve dynamically.

GameGen-$\mathbb{X}$ undergoes a two-stage training: foundation model pre-training and instruction tuning.
In the first stage, the foundation model is pre-trained on OGameData using text-to-video generation and video continuation tasks.
This enables the model to learn a broad range of open-world game dynamics and generate high-quality game content.
In the second stage, InstructNet is designed to enable multi-modal interactive controllability.
The foundation model is frozen, and InstructNet is trained to map user inputs—such as structured text instructions for game environment dynamics and keyboard controls for character movements and actions—onto the generated game content.
This allows GameGen-$\mathbb{X}$ to generate coherent and controllable video clips that evolve based on the player’s inputs, simulating an interactive gaming experience.
To facilitate this development, we constructed Open-World Video Game Dataset (OGameData), the first large-scale dataset for game video generation and control.
This dataset was collected from over 150 next-generation games and was built by using a human-in-the-loop proprietary data pipeline that involves scoring, filtering, sorting, and structural captioning.
OGameData contains one million video clips from two subsets including OGameData-GEN and OGameData-INS, providing the foundation for training generative models capable of producing realistic game content and achieving interactive control, respectively.

In summary, GameGen-$\mathbb{X}$ offers a novel approach for interactive open-world game video generation, where complex game content is generated and controlled interactively.
It lays the foundation for a new potential paradigm in game content design and development.
While challenges for practical application remain, GameGen-$\mathbb{X}$ demonstrates the potential for generative models to serve as a scalable and efficient auxiliary tool to traditional game design methods.
Our main contributions are summarized as follows:
1) We developed OGameData, the first comprehensive dataset specifically curated for open-world game video generation and interactive control, which contains one million video-text pairs. It is collected from 150+ next-gen games, and empowered by GPT-4o.
2) We introduced GameGen-$\mathbb{X}$, the first generative model for open-world video game content, combining a foundation model with the InstructNet.
GameGen-$\mathbb{X}$ utilizes a two-stage training strategy, with the foundation model and InstructNet trained separately to ensure stable, high-quality content generation and control.
InstructNet provides multi-modal interactive controllability, allowing players to influence the continuation of generated content, simulating gameplay.
3) We conducted extensive experiments comparing our model’s generative and interactive control abilities to other open-source and commercial models.
Results show that our approach excels in high-quality game content generation and offers superior control over the environment and character.

## 2 OGameData: large-scale fine-grained game domain dataset

Figure: Figure 2: The OGameData collection and processing pipeline with human-in-the-loop.
Refer to caption: x2.png

OGameData is the first dataset designed for open-world game video generation and interactive control.
As shown in Table 1, OGameData excels in fine-grained annotations, offering a structural caption with high text density for video clips per minute.
It is meticulously designed for game video by offering game-specific knowledge and incorporating elements such as game names, player perspectives, and character details.
It comprises two parts: the generation dataset (OGameData-GEN) and the instruction dataset (OGameData-INS).
The resulting OGameData-GEN is tailored for training the generative foundation model, while OGameData-INS is optimized for instruction tuning and interactive control tasks.
The construction details, dataset analysis, and captioning prompt designs are in Appendix B.

### 2.1 Dataset Construction Pipeline

As illustrated in Figure 2, we developed a robust data processing pipeline encompassing collection, cleaning, segmentation, filtering, and structured caption annotation.
This process integrates both AI and human expertise, as automated techniques alone are insufficient due to domain-specific intricacies present in various games.

Data Collection and Filtering.
We gathered content from more than 150 next-gen games and game engine direct outputs to ensure diversity.
These data are from the internet and local game engines, specifically focusing on gameplay footage that minimizes UI elements.
Despite the rigorous collection, some low-quality videos were included, and these videos lacked essential metadata like game name, genre, and player perspective.
Low-quality videos were manually filtered out, with human experts ensuring the integrity of the metadata, such as game genre and player perspective.
To prepare videos for clip segmentation, we used PyScene and TransNetV2 (Souček & Lokoč ()) to detect scene changes, discarding clips shorter than 4 seconds and splitting longer clips into 16-second segments.
To filter and annotate clips, we sequentially employed models: CLIP-AVA (Schuhmann ()) for aesthetic scoring, UniMatch (Xu et al. ()) for motion filtering, VideoCLIP (Xu et al. ()) for content similarity, and CoTrackerV2 (Karaev et al. ()) for camera motion.

Structured Text Captioning.
The OGameData supports the training of two key functionalities: text-to-video generation and interactive control.
These tasks require distinct captioning strategies.
For OGameData-GEN, detailed captions are crafted to describe the game metadata, scene context, and key characters, ensuring comprehensive textual descriptions for the generative model foundation training.
In contrast, OGameData-INS focuses on describing the changes in game scenes for interactive generation, using concise instruction-based captions that highlight differences between initial and subsequent frames.
This structured captioning approach enables precise and fine-grained generation and control, allowing the model to modify specific elements while preserving the scene.

**Table 1: Comparison of OGameData and previous large-scale text-video paired datasets.**
| Dataset | Domain | Text-video pairs | Caption density | Captioner | Resolution | Purpose | Total video len. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ActivityNet (Caba Heilbron et al. ()) | Action | 85K | 23 words/min | Manual | - | Understanding | 849h |
| DiDeMo (Anne Hendricks et al. ()) | Flickr | 45k | 70 words/min | Manual | - | Temporal localization | 87h |
| YouCook2 (Zhou et al. ()) | Cooking | 32k | 26 words/min | Manual | - | Understanding | 176h |
| How2 (Sanabria et al. ()) | Instruct | 191k | 207 words/min | Manual | - | Understanding | 308h |
| MiraData (Ju et al. ()) | Open | 330k | 264 words/min | GPT-4V | 720P | Generation | 16000h |
| OGameData (Ours) | Game | 1000k | 607 words/min | GPT-4o | 720P-4k | Generation & Control | 4000h |

### 2.2 Dataset Summary

As depicted in Table 1, OGameData comprises 1 million high-resolution video clips, derived from sources spanning minutes to hours.
Compared to other domain-specific datasets (Caba Heilbron et al. (); Zhou et al. (); Sanabria et al. (); Anne Hendricks et al. ()), OGameData stands out for its scale, diversity, and richness of text-video pairs.
Even compared with the latest open-domain generation dataset Miradata (Ju et al. ()), our dataset still has the advantage of providing more fine-grained annotations, which feature the most extensive captions per unit of time.
This dataset features several key characteristics: OGameData features highly fine-grained text and boasts a large number of trainable video-text pairs, enhancing text-video alignment in model training.
Additionally, it comprises two subsets—generation and control—supporting both types of training tasks.
The dataset’s high quality is ensured by meticulous curation from over 10 human experts.
Each video clip is accompanied by captions generated using GPT-4o, maintaining clarity and coherence and ensuring the dataset remains free of UI and visual artifacts.
Critical to its design, OGameData is tailored specifically for the gaming domain.
It effectively excludes non-gameplay scenes, incorporating a diverse array of game styles while preserving authentic in-game camera perspectives.
This specialization ensures the dataset accurately represents real gaming experiences, maintaining high domain-specific relevance.

## 3 GameGen- 𝕏 𝕏 \mathbb{X} blackboard_X

GameGen-$\mathbb{X}$ is the first generative diffusion model that learns to generate open-world game videos and interactively control the environments and characters in them.
The overall framework is illustrated in Fig 3.
In section 3.1, we introduce the problem formulation.
In section 3.2, we discuss the design and training of the foundation model, which facilitates both initial game content generation and video continuation.
In section 3.3, we delve into the design of InstructNet and explain the process of instruction tuning, which enables clip-level interactive control over generated content.

Figure: Figure 3: An overview of our two-stage training framework. In the first stage, we train the foundation model via OGameData-GEN. In the second stage, InstructNet is trained via OGameData-INS.
Refer to caption: x3.png

### 3.1 Game Video Generation and Interaction

The primary objective of GameGen-$\mathbb{X}$ is to generate dynamic game content where both the virtual environment and characters are synthesized from textual descriptions, and users can further influence the generated content through interactive controls.
Given a textual description $T$ that specifies the initial game scene—including characters, environments, and corresponding actions and events—we aim to generate a video sequence $\mathbb{V}=\{V_{t}\}_{t=1}^{N}$ that brings this scene to life.
We model the conditional distribution: $p(V_{1:N}\,|\,T,C_{1:N})$, where $C_{1:N}$ represents the sequence of multi-modal control inputs provided by the user over time.
These control inputs allow users to manipulate character movements and scene dynamics, simulating an interactive gaming experience.

Our approach integrates two main components:
1) Foundation Model: It generates an initial video clip based on $T$, capturing the specified game elements including characters and environments.
2) InstructNet: It enables the controllable continuation of the video clip by incorporating user-provided control inputs.
By unifying text-to-video generation with interactive controllable video continuation, our approach synthesizes game-like video sequences where the content evolves in response to user interactions.
Users can influence the generated video at each generation step by providing control inputs, allowing for manipulation of the narrative and visual aspects of the content.

### 3.2 Foundation Model Training for Generation

Video Clip Compression.
To address the redundancy in temporal and spatial information (Lab & etc. ()), we introduce a 3D Spatio-Temporal Variational Autoencoder (3D-VAE) to compress video clips into latent representations.
This compression enables efficient training on high-resolution videos with longer frame sequences.
Let $V\in\mathbb{R}^{F\times C\times H\times W}$ denote a video clip, where $F$ is the number of frames, $H$ and $W$ are the height and width of each frame, and $C$ is the number of channels.
The encoder $E$ compresses $V$ into a latent representation $z=E(V)\in\mathbb{R}^{F^{\prime}\times C^{\prime}\times H^{\prime}\times W^{
\prime}}$, where $F^{\prime}=F/s_{f}$, $H^{\prime}=H/s_{h}$, $W^{\prime}=W/s_{w}$, and $C^{\prime}$ is the number of latent channels.
Here, $s_{t}$, $s_{h}$, and $s_{w}$ are the temporal and spatial downsampling factors.
Specifically, 3D-VAE first performs the spatial downsampling to obtain frame-level latent features.
Further, it conducts temporal compression to capture temporal dependencies and reduce redundancy over frame effectively, inspired by Yu et al. ().
By processing the video clip through the 3D-VAE, we can obtain a latent tensor $z$ of spatial-temporally informative and reduced dimensions.
Such $z$ can support long video and high-resolution model training, which meets the requirements of game content generation.

Masked Spatial-Temporal Diffusion Transformer.
GameGen-$\mathbb{X}$ introduces a Masked Spatial-Temporal Diffusion Transformer (MSDiT).
Specifically, MSDiT combines spatial attention, temporal attention, and cross-attention mechanisms (Vaswani ()) to effectively generate game videos guided by text prompts.
For each time step $t$, the model processes latent features $z_{t}$ that capture frame details.
Spatial attention enhances intra-frame relationships by applying self-attention over spatial dimensions $(H^{\prime},W^{\prime})$.
Temporal attention ensures coherence across frames by operating over the time dimension $F^{\prime}$, capturing inter-frame dependencies.
Cross-attention integrates guidance of external text features $f$ obtained via T5 (Raffel et al. ()), aligning video generation with the semantic information from text prompts.
As shown in Fig. 4, we adopt the design of stacking paired spatial and temporal blocks, where each block is equipped with cross-attention and one of spatial or temporal attention.
Such design allows the model to capture spatial details, temporal dynamics, and textual guidance simultaneously, enabling GameGen-$\mathbb{X}$ to generate high-fidelity, temporally consistent videos that are closely aligned with the provided text prompts.

Additionally, we introduce a masking mechanism that excludes certain frames from noise addition and denoising during diffusion processing.
A masking function $M(i)$ over frame indices $i\in I$ is defined as:
$M(i)=1$ if $i>x$, and $M(i)=0$ if $i\leq x$, where $x$ is the number of context frames provided for video continuation.
The noisy latent representation at time step $t$ is computed as:
$\tilde{z}_{t}=(1-M(I))\odot z+M(I)\odot\epsilon_{t},$ where $\epsilon_{t}\sim\mathcal{N}(0,\mathbf{I})$ is Gaussian noise of the same dimensions as $z$, and $\odot$ denotes element-wise multiplication.
Such a masking strategy provides the support of training both text-to-video and video continuation into one foundation model.

Unified Video Generation and Continuation.
By integrating the text-to-video diffusion training logic with the masking mechanism, GameGen-$\mathbb{X}$ effectively handles both video generation and continuation tasks within a unified framework.
This strategy aims to enhance the simulation experience by enabling temporal continuity, catering to an extended and immersive gameplay experience.
Specifically, for *text-to-video generation*, where no initial frames are provided, we set $x=0$, and the masking function becomes $M(i)=1$ for all frames $i$.
The model learns the conditional distribution $p(V\,|\,T)$, where $T$ is the text prompt.
The diffusion process is applied to all frames, and the model generates video content solely based on the text prompt.
For *video continuation*, initial frames $v_{1:x}$ are provided as context.
The masking mechanism ensures that these frames remain unchanged during the diffusion process, as $M(i)=0$ for $i\leq x$.
The model focuses on generating the subsequent frames $v_{x+1:N}$ by learning the conditional distribution $p(v_{x+1:N}\,|\,v_{1:x},T)$.
This allows the model to produce video continuations that are consistent with both the preceding context and the text prompt.
Additionally, during the diffusion training (Song et al. (; ); Ho et al. (); Rombach et al. ()), we incorporated the bucket training (Zheng et al. (), classifier-free diffusion guidance (Ho & Salimans ()) and rectified flow (Liu et al. ()) for better generation performance.
Overall, this unified training approach enhances the ability to generate complex, contextually relevant open-world game videos while ensuring smooth transitions and continuations.

### 3.3 Instruction Tuning for Interactive Control

InstructNet Design.
To enable interactive controllability in video generation, we propose *InstructNet*, designed to guide the foundation model’s predictions based on user inputs, allowing for control of the generated content.
The core concept is that the generation capability is provided by the foundation model, with InstructNet subtly adjusting the predicted content using user input signals.
Given the high requirement for visual continuity in-game content, our approach aims to minimize abrupt changes, ensuring a seamless experience.
Specifically, the primary purpose of InstructNet is to modify future predictions based on instructions.
When no user input signal is given, the video extends naturally.
Therefore, we keep the parameters of the pre-trained foundation model frozen, which preserves its inherent generation and continuation abilities.
Meanwhile, the additional trainable InstructNet is introduced to handle control signals.
As shown in Fig. 4, InstructNet modifies the generation process by incorporating control signals via the operation fusion expert layer and instruction fusion expert layer.
This component comprises $N$ InstructNet blocks, each utilizing a specialized Operation Fusion Expert Layer and an Instruct Fusion Expert Layer to integrate different conditions.
The output features are injected into the foundation model to fuse the original latent, modulating the latent representations based on user inputs and effectively aligning the output with user intent.
This enables users to influence character movements and scene dynamics.
InstructNet is primarily trained through video continuation to simulate the control and feedback mechanism in gameplay.
Further, Gaussian noise is subtly added to initial frames to mitigate error accumulation.

Figure: Figure 4: The architecture of GameGen-$\mathbb{X}$, including the foundation model and InstructNet. It enables the latent modification based on user inputs, mainly instruction and operation prompts, allowing for interactive control over the video generation process.
Refer to caption: x4.png

Multi-modal Experts.
Our approach leverages multi-modal experts to handle diverse controls, which is crucial for several reasons.
Intuitively, each structured text, keyboard binding, and video prompt—uniquely impacts the video prediction process, requiring specialized handling to fully capture their distinct characteristics.
By employing multi-modal experts, we can effectively integrate these varied inputs, ensuring that each control signal is well utilized.
Let $f_{I}$ and $f_{O}$ be structured instruction embedding and keyboard input embedding, respectively.
$f_{O}$ is used to modulate the latent features via operation fusion expert as follows:
$\hat{z}=\gamma(f_{O})\odot\frac{z-\mu}{\sigma}+\beta(f_{O}),$
where $\mu$ and $\sigma$ are the mean and standard deviation of $z$, $\gamma(f_{O})$ and $\beta(f_{O})$ are scale and shift parameters predicted by a neural network conditioned on $c$, and $\odot$ denotes element-wise multiplication.
The keyboard signal primarily influences video motion direction and character control, exerting minimal impact on scene content. Consequently, a lightweight feature scaling and shifting approach is sufficient to effectively process this information.
The instruction text is primarily responsible for controlling complex scene elements such as the environment and lighting.
To incorporate this text information into InstructNet, we utilize an instruction fusion expert, which integrates $f_{I}$ into the model through cross-attention.
Video prompts $V_{p}$, such as canny edges, motion vectors, or pose sequences, are introduced to provide auxiliary information, especially for clips with significant motion dynamics.
These prompts are processed through the 3D-VAE encoder to extract features $e_{p}$, which are then incorporated into the InstructNet via addition with $z$.
It’s worth clarifying that, during the inference, these video prompts are not necessary, except to execute the complex action generation or video editing.

Interactive Control.
Interactive control is achieved through an autoregressive generation process.
Based on a sequence of past frames $v_{1:x}$, the model generates future frames $v_{x+1:N}$ while adhering to control signals.
The overall objective is to model the conditional distribution:
$p(v_{x+1:N}\,|\,v_{1:x},c),$
where $c$ includes both structured text instructions and keyboard inputs.
During generation, the foundation model predicts future latent representations, and InstructNet modifies these predictions according to the control signals.
Thus, users can influence the video’s progression by providing structured text commands or keyboard inputs, enabling a high degree of controllability in the open-world game environment.
Furthermore, the incorporation of video prompts $V_{p}$ enhances the model’s ability to generate motion-consistent frames.
The prompts provide additional guidance, making it possible to edit or stylize videos quickly, which is particularly useful for specific motion patterns.

## 4 Experiments

### 4.1 Quantitative Results

Metrics.
To comprehensively evaluate the performance of GameGen-$\mathbb{X}$, we utilize a suite of metrics that capture various aspects of video generation quality and interactive control, following Huang et al. () and Yang et al. ().
These metrics include Fréchet Inception Distance (FID), Fréchet Video Distance (FVD), Text-Video Alignment (TVA), User Preference (UP), Motion Smoothness (MS), Dynamic Degrees (DD), Subject Consistency (SC), and Imaging Quality (IQ).
It’s worth noting that the TVA and UP are subjective scores that indicate whether the generation meets the requirements of humans, following Yang et al. ().
By employing this comprehensive set of metrics, we aim to thoroughly evaluate model capabilities in generating high-quality, realistic, and interactively controllable video game content.
Readers can find experimental settings and metric introductions in Appendix C.1.

**Table 2: Generation Performance Evaluation (* denotes key metric for generation ability)**
| Method | Resolution | Frames | FID* $\downarrow$ | FVD* $\downarrow$ | TVA* $\uparrow$ | UP* $\uparrow$ | MS $\uparrow$ | DD $\uparrow$ | SC $\uparrow$ | IQ $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mira (Zhang et al. () | 480p | 60 | 360.9 | 2254.2 | 0.27 | 0.25 | 0.98 | 0.62 | 0.94 | 0.63 |
| OpenSora-Plan1.2 (Lab & etc. () | 720p | 102 | 407.0 | 1940.9 | 0.38 | 0.43 | 0.99 | 0.42 | 0.92 | 0.39 |
| CogVideoX-5B (Yang et al. () | 480p | 49 | 316.9 | 1310.2 | 0.49 | 0.37 | 0.99 | 0.94 | 0.92 | 0.53 |
| OpenSora1.2 (Zheng et al. () | 720p | 102 | 318.1 | 1016.3 | 0.50 | 0.37 | 0.98 | 0.90 | 0.87 | 0.52 |
| GameGen-$\mathbb{X}$ (Ours) | 720p | 102 | 252.1 | 759.8 | 0.87 | 0.82 | 0.99 | 0.80 | 0.94 | 0.50 |

**Table 3: Control Performance Evaluation (* denotes key metric for control ability)**
| Method | Resolution | Frames | SR-C* $\uparrow$ | SR-E* $\uparrow$ | UP $\uparrow$ | MS $\uparrow$ | DD $\uparrow$ | SC $\uparrow$ | IQ $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OpenSora-Plan1.2 (Lab & etc. () | 720p | 102 | 26.6% | 31.7% | 0.46 | 0.99 | 0.72 | 0.90 | 0.51 |
| CogVideoX-5B (Yang et al. () | 480p | 49 | 23.0% | 30.3% | 0.45 | 0.98 | 0.63 | 0.85 | 0.55 |
| OpenSora1.2 (Zheng et al. () | 720p | 102 | 21.6% | 14.2% | 0.17 | 0.99 | 0.97 | 0.84 | 0.45 |
| GameGen-$\mathbb{X}$ (Ours) | 720p | 102 | 63.0% | 56.8% | 0.71 | 0.99 | 0.88 | 0.88 | 0.44 |

Generation and Control Ability Evaluation.
As shown in Table 2, we compared GameGen-$\mathbb{X}$ against four well-known open-source models, i.e., Mira (Zhang et al. ()), OpenSora-Plan1.2 (Lab & etc. ()), OpenSora1.2 (Zheng et al. ()) and CogVideoX-5B (Yang et al. ()) to evaluate its generation capabilities.
Notably, both Mira and OpenSora1.2 explicitly mention training on game data, while the other two models, although not specifically designed for this purpose, can still fulfill certain generation needs within similar contexts.
Our evaluation showed that GameGen-$\mathbb{X}$ performed well on metrics such as FID, FVD, TVA, MS, and SC.
It implies GameGen-$\mathbb{X}$’s strengths in generating high-quality and coherent video game content while maintaining competitive visual and technical quality.
Further, we investigated the control ability of these models, as shown in Table 3.
We used conditioned video clips and dense prompts to evaluate the model generation response.
For GameGen-$\mathbb{X}$, we employed instruct prompts to generate video clips.
Beyond the aforementioned metrics, we introduced the Success Rate (SR) to measure how often the models respond accurately to control signals.
This is evaluated by both human experts and PLLaVA (Xu et al. ()).
The SR metric is divided into two parts: SR for Character Actions (SR-C), which assesses the model’s responsiveness to character movements, and SR for Environment Events (SR-E), which evaluates the model’s handling of changes in weather, lighting, and objects.
As demonstrated, GameGen-$\mathbb{X}$ exhibits superior control ability compared to other models, highlighting its effectiveness in generating contextually appropriate and interactive game content.
Since IQ metrics favor models trained on natural scene datasets, such models score higher.
In generation performance, CogVideo’s 8fps videos and OpenSora 1.2’s frequent scene changes result in higher DD.

**Table 4: Ablation Study for Generation Ability**
| Table 4: Ablation Study for Generation Ability<br>Method<br>FID $\downarrow$<br>FVD $\downarrow$<br>TVA $\uparrow$<br>UP $\uparrow$<br>MS $\uparrow$<br>SC $\uparrow$<br>w/ MiraData<br>303.7<br>1423.6<br>0.70<br>0.48<br>0.99<br>0.94<br>w/ Short Caption<br>303.8<br>1167.7<br>0.53<br>0.49<br>0.99<br>0.94<br>w/ Progression<br>294.2<br>1169.8<br>0.68<br>0.53<br>0.99<br>0.93<br>Baseline<br>289.5<br>1181.3<br>0.83<br>0.67<br>0.99<br>0.95 | Table 5: Ablation Study for Control Ability.<br>Method<br>SR-C $\uparrow$<br>SR-E $\uparrow$<br>UP $\uparrow$<br>MS $\uparrow$<br>SC $\uparrow$<br>w/o Instruct Caption<br>31.6%<br>20.0%<br>0.34<br>0.99<br>0.87<br>w/o Decomposition<br>32.7%<br>23.3%<br>0.41<br>0.99<br>0.88<br>w/o InstructNet<br>12.3%<br>17.5%<br>0.16<br>0.98<br>0.86<br>Baseline<br>45.6%<br>45.0%<br>0.50<br>0.99<br>0.90 |
| --- | --- |

Ablation Study.
As shown in Table 5, we investigated the influence of various data strategies, including leveraging MiraData (Ju et al. ()), short captions (Chen et al. ()), and progression training (Lab & etc. ()).
The results indicated that our data strategy outperforms the others, particularly in terms of semantic consistency, distribution alignment, and user preference.
The visual quality metrics are comparable across all strategies.
This consistency implies that visual quality metrics may be less sensitive to these strategies or that they might be limited in evaluating game domain generation.
Further, as shown in Table 5, we explored the effects of our design on interactive control ability through ablation studies.
This experiment involved evaluating the impact of removing key components such as InstructNet, Instruct Captions, or the decomposition process.
The results demonstrate that the absence of InstructNet significantly reduces the SR and UP, highlighting its crucial role in user-preference interactive controllability.
Similarly, the removal of Instruct Captions and the decomposition process also negatively impacts control metrics, although to a lesser extent.
These findings underscore the importance of each component in enhancing the model’s ability to generate and control game content interactively.

Figure: Figure 5: The generation showcases of characters, environments, actions, and events.
Refer to caption: x5.png

### 4.2 Qualitative Results

Generation Functionality.
Fig. 5 illustrates the basic generation capabilities of our model in generating a variety of characters, environments, actions, and events.
The examples show that the model can create characters such as assassins and mages, simulate environments such as Sakura forests and rainforests, execute complex actions like flying and driving, and reproduce environmental events like snowstorms and heavy rain.
This demonstrates the model’s ability to generate and control diverse scenarios, highlighting its potential application in generating open-world game videos.

Figure: Figure 6: The qualitative results of different control signals, given the same open-domain clip.
Refer to caption: x6.png

Interactive Control Ability.
As shown in Fig. 6, our model demonstrates the capability to control both environmental events and character actions based on textual instructions and keyboard inputs.
In the example provided, the model effectively manipulates various aspects of the scene, such as lighting conditions and atmospheric effects, highlighting its ability to simulate different times of day and weather conditions.
Additionally, the character’s movements, primarily involving navigation through the environment, are precisely controlled through input keyboard signals.
This interactive control mechanism enables the simulation of a dynamic gameplay experience.
By adjusting environmental factors like lighting and atmosphere, the model provides a realistic and immersive setting.
Simultaneously, the ability to manage character movements ensures that the generated content responds intuitively to user interactions.
Through these capabilities, our model showcases its potential to enhance the realism and engagement of open-world video game simulations.

Figure: Figure 7: Qualitative comparison with open-source models in the open-domain generation.
Refer to caption: x7.png

Open-domain Generation, Gameplay Simulation and Further Analysis.
As shown in Fig. 1, we presented initial qualitative experiment results, where GameGen-$\mathbb{X}$ generates novel domain game video clips and interactively controls them, which can be seen as a game simulation.
Further, we compared GameGen-$\mathbb{X}$ with other open-source models in the open-domain generation ability as shown in Fig. 7.
All the open-source models can generate some game-like content, implying their training involves corresponding game source data.
As expected, the GameGen-$\mathbb{X}$ can better meet the game content requirements in character details, visual environments, and camera logic, owing to the strict dataset collection and building of OGameData.
Further, we compared GameGen-$\mathbb{X}$ with other commercial products including Kling, Pika, Runway, Luma, and Tongyi, as shown in Fig. 8.
In the left part, i.e., the initially generated video clip, only Pika, Kling1.5, and GameGen-$\mathbb{X}$ correctly followed the text description.
Other models either failed to display the character or depicted them entering the cave instead of exiting.
In the right part, both GameGen-$\mathbb{X}$ and Kling1.5 successfully guided the character out of the cave.
GameGen-$\mathbb{X}$ achieved high-quality control response as well as maintaining a consistent camera logic, obeying the game-like experience at the same time.
This is owing to the design of a holistic training framework and InstructNet.

Readers can find more qualitative results and comparisons in Appendix C.2, and click [https://3A2077.github.io](https://3A2077.github.io) to watch more demo videos.
Additionally, we provide related works in Appendix A, dataset and construction details in Appendix B, and a discussion of limitations and potential future work in Appendix D.

Figure: Figure 8: Qualitative comparison with commercial models in the interactive control ability.
Refer to caption: x8.png

## 5 Conclusion

We have presented GameGen-$\mathbb{X}$, the first diffusion transformer model with multi-modal interactive control capabilities, specifically designed for generating open-world game videos.
By simulating key elements such as dynamic environments, complex characters, and interactive gameplay, GameGen-$\mathbb{X}$ sets a new benchmark in the field, demonstrating the potential of generative models in both generating and controlling game content.
The development of the OGameData provided a crucial foundation for our model’s training, enabling it to capture the diverse and intricate nature of open-world games.
Through a two-stage training process, GameGen-$\mathbb{X}$ achieved a mutual enhancement between content generation and interactive control, allowing for a rich and immersive simulation experience.
Beyond its technical contributions, GameGen-$\mathbb{X}$ opens up new horizons for the future of game content design.
It suggests a potential shift towards more automated, data-driven methods that significantly reduce the manual effort required in early-stage game content creation.
By leveraging models to create immersive worlds and interactive gameplay, we may move closer to a future where game engines are more attuned to creative, user-guided experiences.
While challenges remain, GameGen-$\mathbb{X}$ represents an initial yet significant leap forward toward a novel paradigm in game design.
It lays the groundwork for future research and development, paving the way for generative models to become integral tools in creating the next generation of interactive digital worlds.

Acknowledgments.
We extend our sincere gratitude for the valuable discussions, comments, and help provided by Mr. Xiaotian He, Mr. Chengjie Jiang, Mr. Wei Lin, Mr. Jingran Su, Mr. Yunlong Zhao, Mr. Guanhua Zhao and Mr. Lingting Zhu (listed in alphabetical order).

## Appendix A Related Works

### A.1 Video Diffusion Models

The advent of diffusion models, particularly latent diffusion models, has significantly advanced image generation, inspiring researchers to extend their applicability to video generation (Liu et al. (; )).
This field can be broadly categorized into two approaches: image-to-video and text-to-video generation.
The former involves transforming a static image into a dynamic video, while the latter generates videos based solely on textual descriptions, without any input images.
Pioneering methods in this domain include AnimateDiff (Guo et al. ()), Dynamicrafter (Xing et al. ()), Modelscope (Wang et al. ()), AnimateAnything (Dai et al. ()), and Stable Video Diffusion (Rombach et al. ()).
These techniques typically leverage pre-trained text-to-image models, integrating them with various temporal mixing layers to handle the temporal dimension inherent in video data.
However, the traditional U-Net based framework encounters scalability issues, limiting its ability to produce high-quality videos.
The success of transformers in the natural language processing community and their scalability has prompted researchers to adapt this architecture for diffusion models, resulting in the development of DiTs (Peebles & Xie ().
Subsequent work, such as Sora (Tim Brooks & Ramesh ()), has demonstrated the powerful capabilities of DiTs in video generation tasks.
Open-source implementations like Latte (Ma et al. ()), Opensora (Zheng et al. (), and Opensora-Plan (Lab & etc. ()) have further validated the superior performance of DiT-based models over traditional U-Net structures in both text-to-video and image-to-video generation.
Despite these advancements, the exploration of gaming video generation and its interactive controllability remains under-explored.

### A.2 Game Simulation and Interaction

Several pioneering works have attempted to train models for game simulation with action inputs.
For example, UniSim (Yang et al. ()) and Pandora (Xiang et al. ()) built a diverse dataset of real-world and simulated videos and could predict a continuation video given a previous video segment and an action prompt via a supervised learning paradigm, while PVG (Menapace et al. ()) and Genie (Bruce et al. ()) focused on unsupervised learning of actions from videos.
Similar to our work, GameGAN (Kim et al. ()), GameNGen (Valevski et al. ()), and DIAMOND (Alonso et al. ()) focused on the playable simulation of early games such as Atari and DOOM, and demonstrates its combination with a gaming agent for interaction (Zheng et al. ()).
However, they didn’t explore the potential of generative models in simulating the complex environments of next-generation games.
Instead, GameGen-$\mathbb{X}$ can create intricate environments, dynamic events, diverse characters, and complex actions with a high degree of realism and variety.
Additionally, GameGen-$\mathbb{X}$ allows the model to generate subsequent frames based on the current video segment and player-provided multi-modal control signals.
This approach ensures that the generated content is not only visually compelling but also contextually appropriate and responsive to player actions, bridging the gap between simple game simulations and the sophisticated requirements of next-generation open-world games.

## Appendix B Dataset

### B.1 Construction Details

Data Collection.
Following Ju et al. (), we selected online video websites and local game engines as one of our primary video sources.
Prior research predominantly focused on collecting game cutscenes and gameplay videos containing UI elements.
Such videos are not ideal for training a game video generation model due to the presence of UI elements and non-playable content.
In contrast, our method adheres to the following principles: 1) We exclusively collect videos showcasing playable content, as our goal is to generate actual gameplay videos rather than cutscenes or CG animations.
2) We ensure that the videos are high-quality and devoid of any UI elements.
To achieve this, we only include high-quality games released post-2015 and capture some game footage directly from game engines to enhance diversity.
Additionally, we record the gameplay videos locally, to collect the keyboard control signals.
Following the data collection stage, we collected 32,000 videos from more than 150 next-generation video games.

Video-level Selection and Annotation.
Despite our rigorous data collection process, some low-quality videos inevitably collected into our dataset.
Additionally, the collected videos lack essential metadata such as game name, genre, and player perspective.
This metadata is challenging to annotate using AI alone.
Therefore, we employed human game experts to filter and annotate the videos.
In this stage, human experts manually review each video, removing those with UI elements or non-playable content.
For the remaining usable videos, they annotate critical metadata, including game name, genre (e.g., ACT, FPS, RPG), and player perspective (First-person, Third-person).
After this filtering and annotation phase, we curated a dataset of 15,000 high-quality videos complete with game metadata.

Scene Detection and Segmentation.
The collected videos, ranging from several minutes to hours, are unsuitable for model training due to their extended duration and numerous scene changes.
We employed TransNetV2 (Souček & Lokoč ()) and PyScene for scene segmentation, which can adaptively identify scene change timestamps within videos.
Upon obtaining these timestamps, we discard video clips shorter than 4 seconds, considering them too brief.
For clips longer than 16 seconds, we divide them into multiple 16-second segments, discarding any remainder shorter than 4 seconds.
Following this scene segmentation stage, we obtained around 1,000,000 video clips, each containing 4-16 seconds of content at 24 frames per second.

Clips-level Filtering and Annotation.
Some clips contain game menus, maps, black screens, low-quality scenes, or nearly static scenes, necessitating further data cleaning.
Given the vast number of clips, manual inspection is impractical.
Instead, we sequentially employed an aesthetic scoring model, a flow scoring model, the video CLIP model, and a camera motion model for filtering and annotation.
First, we used the CLIP-AVA model (Schuhmann ()) to score each clip aesthetically.
We then randomly sampled 100 clips to manually determine a threshold, filtering out clips with aesthetic scores below this threshold.
Next, we applied the UniMatch model (Xu et al. ()) to filter out clips with either excessive or minimal motion.
To address redundancy, we used the video-CLIP (Xu et al. ()) model to calculate content similarity within clips from the same game, removing overly similar clips.
Finally, we utilized CoTrackerV2 (Karaev et al. ()) to annotate clips with camera motion information, such as ”pan-left” or ”zoom-in.”

Structural Caption.
We propose a Structural captioning approach for generating captions for OGameData-GEN and OGameData-INS.
To achieve this, we uniformly sample 8 frames from each video and stack them into a single image.
Using this image as a representation of the video’s content, we designed two specific prompts to instruct GPT-4o to generate captions.
For OGameData-GEN, we have GPT-4o describe the video across five dimensions: Summary of the video, Game Meta information, Character details, Frame Descriptions, and Game Atmosphere.
This Structural information enables the model to learn mappings between text and visual information during training and allows us to independently modify one dimension’s information while keeping the others unchanged during the inference stage.
For OGameData-INS, we decompose the video changes into five perspectives, with each perspective described in a short sentence.
The Environment Basic dimension describes the fundamental environment information, while the Transition dimension captures changes in the environment.
The Light and Act dimensions describe the lighting conditions and character actions, respectively.
Lastly, the MISC dimension includes meta-information about the video, such as keyboard operations or camera motion.
This Structural captioning approach allows the model to focus entirely on content changes, thereby enhancing control over the generated video.
By enabling independent modification of specific dimensions during inference, we achieve fine-grained generation and control, ensuring the model effectively captures both static and dynamic aspects of the game world.

Prompt Design.
In our collection of 32,000 videos, we identified two distinct categories.
The first category comprises free-camera videos, which primarily focus on showcasing environmental and scenic elements.
The second category consists of gameplay videos, characterized by the player’s perspective during gameplay, including both first-person and third-person views.
We believe that free-camera videos can help the model better align with engine-specific features, such as textures and physical properties, while gameplay videos can directly guide the model’s behavior.
To leverage these two types of videos effectively, we designed different sets of prompts for each category.
Each set includes a summary prompt and a dense prompt.
The core purpose of the summary prompt is to succinctly describe all the scene elements in a single sentence, whereas the dense prompt provides structural, fine-grained guidance.
Additionally, to achieve interactive controllability, we designed structural instruction prompts.
These prompts describe the differences between the initial frame and subsequent frames across various dimensions, simulating how instructions can guide the generation of follow-up video content.

Figure: Listing 1: Summary prompt for free-camera videos

Figure: Listing 2: Dense prompt for free-camera videos

Figure: Listing 3: Summary prompt for gameplay videos

Figure: Listing 4: Dense prompt for gameplay videos

Figure: Listing 5: Instruction prompt for interactive control

### B.2 Dataset Showcases

We provide a visualization of the video clips along with their corresponding captions.
We sampled four cases from the OGameData-GEN dataset and the OGameData-INS dataset, respectively.
Both types of captions are Structural, offering multidimensional annotations of the videos.
This Structural approach ensures a comprehensive and nuanced representation of the video content.

Structural Captions in OGameData-GEN.
It is evident from Fig. 9 and Fig. 10 that the captions in OGameData-GEN densely capture the overall information and intricate details of the videos, following the sequential set of ‘Summary’, ‘Game Meta Information’, ‘Character Information’, ‘Frame Description’, and ‘Atmosphere’.

Figure: Figure 9: A sample from OGameData-GEN. Caption: An empty futuristic city street is seen under a large overpass with neon lights and tall buildings. In this sequence from Cyberpunk 2077, the scene unfolds in a sprawling urban environment marked by towering skyscrapers and elevated highways. The video clip showcases a first-person perspective that gradually moves forward along an empty street framed by futuristic neon-lit buildings on the right and industrial structures on the left. The atmospheric lighting casts dramatic shadows across the pavement, enhancing the gritty cyberpunk aesthetic of Night City. As the camera progresses smoothly towards a distant structure adorned with holographic advertisements, it captures key details like overhead cables and a monorail track above, highlighting both verticality and depth in this open-world dystopian setting devoid of any characters or vehicles at this moment. The scene emphasizes a gritty, dystopian cyberpunk atmosphere, characterized by neon-lit buildings, dramatic shadows, and a sense of desolate futurism devoid of characters or vehicles.‘
Refer to caption: x9.png

Figure: Figure 10: A sample from OGameData-GEN. Caption: A person in a white hat walks along a forested riverbank at sunset. In the dim twilight of a picturesque, wooded lakeshore in Red Dead Redemption 2, Arthur Morgan, dressed in his iconic red coat and wide-brimmed white hat, strides purposefully along the water’s edge. The eight sequential frames capture him closely from behind at an over-the-shoulder camera angle as he walks towards the dense tree line under a dramatic evening sky tinged with pink and purple hues. Each step takes place against a tranquil backdrop featuring rippling water reflecting dying sunlight and silhouetted trees that deepen the serene yet subtly ominous atmosphere typical of this open-world action-adventure game. Dust particles float visibly through the air as Arthur’s movement stirs up small puffs from the soil beneath his boots, adding to the immersive realism of this richly detailed environment. The scene captures a tranquil yet subtly ominous atmosphere.
Refer to caption: x10.png

Structural Instructions in OGameData-INS.
In contrast, the instructions in OGameData-INS, which are instruction-oriented and often use imperative sentences, effectively capture the changes in subsequent frames relative to the initial frame, as shown in Fig. 11 and Fig. 12.
It has five decoupled dimensions following the sequential of ‘Environment Basic’, ‘Character Action’, ‘Environment Change’, ‘Lighting and Sky’, and ‘Misc’.

Figure: Figure 11: A sample from OGameData-INS. Caption: Environmental Basics: Maintain the dense forest scenery with mountains in the distant background. Main Character: Move forward along the path while maintaining a steady pace. Environmental Changes: Gradually clear some trees and bushes to reveal more of the landscape ahead. Sky/Lighting: Keep consistent daylight conditions with scattered clouds. aesthetic score: 5.02, motion score: 27.37, camera motion: Undetermined. camera size: full shot.
Refer to caption: x11.png

Figure: Figure 12: A sample from OGameData-INS. Caption: Environmental Basics: Show a scenic outdoor environment with trees, grass, and a clear water body in the foreground. Main Character: Move the protagonist slowly forward towards the right along the water’s edge. Environmental Changes: Maintain current state without significant changes to background elements. Sky/Lighting: Keep sky conditions bright and lighting consistent throughout. aesthetic score: 5.36, motion score: 9.37, camera motion: zoom in. camera size: full shot”.
Refer to caption: x12.png

### B.3 Quantitative Analysis

To demonstrate the intricacies of our proposed dataset, we conducted a comprehensive analysis encompassing several key aspects.
Specifically, we examined the distribution of game types, game genres, player viewpoints, motion scores, aesthetic scores, caption lengths, and caption feature distributions.
Our analysis spans both the OGameData-GEN dataset and the OGameData-INS dataset, providing detailed insights into their respective characteristics.

Game-related Data Analysis.
Our dataset encompasses a diverse collection of 150 games, with a primary focus on next-generation open-world titles.
For the OGameData-GEN dataset, as depicted in Fig. 13, player perspectives are evenly distributed between first-person and third-person viewpoints.
Furthermore, it includes a wide array of game genres, including RPG, Action, Simulation, and FPS, thereby showcasing the richness and variety of the dataset.
In contrast, the OGameData-INS dataset, as shown in Fig. 14, is composed of five meticulously selected high-quality open-world games, each characterized by detailed and dynamic character motion.
Approximately half of the videos feature the main character walking forward (zooming in), while others depict lateral movements such as moving right or left.
These motion patterns enable us to effectively train an instructive network.
To ensure the model’s attention remains on the main character, we exclusively selected third-person perspectives.

Figure: Figure 13: Statistical analysis of the OGameData-GEN dataset. The left pie chart illustrates the distribution of player perspectives, with 52.7% of the games featuring a third-person perspective and 47.3% featuring a first-person perspective. The right bar chart presents the distribution of game types, demonstrating a predominance of RPG (55.23%) and ACT (33.08%) genres, followed by Simulation (3.35%) and FPS (3.17%), among others.
Refer to caption: extracted/5968319/figures/game_distribution_nature_style.png

Figure: Figure 14: Comprehensive analysis of the OGameData-INS dataset. The top-left histogram shows the distribution of motion scores, with most scores ranging from 0 to 100. The top-right histogram illustrates the distribution of aesthetic scores, following a Gaussian distribution with the majority of scores between 4.5 and 6.5. The bottom-left bar chart presents the game count statistics, highlighting the most frequently occurring games. The bottom-right bar chart displays the camera motion statistics, with a significant portion of the clips featuring zoom-in motions, followed by various other camera movements.
Refer to caption: extracted/5968319/figures/game_ins_1.png

Figure: Figure 15: Clip-related data analysis for the OGameData-GEN dataset. The left histogram shows the distribution of motion scores, with most scores ranging from 0 to 75. The middle histogram displays the distribution of aesthetic scores, following a Gaussian distribution with the majority of scores between 4.5 and 6. The right histogram illustrates the distribution of word counts in captions, predominantly ranging between 100 and 200 words. This detailed analysis highlights the rich and varied nature of the clips and their annotations, providing comprehensive information for model training.
Refer to caption: extracted/5968319/figures/game_gen_game_3.png

Clips-related Data Analysis.
Apart from the game-related data analysis, we also conducted clip-related data analysis, encompassing metrics such as motion score, aesthetic score, and caption distribution.
This analysis provides clear insights into the quality of our proposed dataset.
For the OGameData-GEN dataset, as illustrated in Fig. 15, most motion scores range from 0 to 75, while the aesthetic scores follow a Gaussian distribution, with the majority of scores falling between 4.5 and 6.
Furthermore, this dataset features dense captions, with most captions containing between 100 to 200 words, providing the model with comprehensive game-related information.
For the OGameData-INS dataset, as shown in Fig. 14, the aesthetic and motion scores are consistent with those of the OGameData-GEN dataset.
However, the captions in OGameData-INS are significantly shorter, enabling the model to focus more on the instructional content itself.
This design choice ensures that the model prioritizes the instructional elements, thereby enhancing its effectiveness in understanding and executing tasks based on the provided instructions.

## Appendix C Experiment Details and Further Analysis

### C.1 Experimental Settings

In this section, we delve into the details of our experiments, covering the calculation of metrics, implementation details, evaluation datasets, and the details of our ablation study.

Evaluation Benchmark.
To evaluate the performance of our methods and other benchmark methods, we constructed two evaluation datasets: OGameEval-Gen and OGameEval-Ins.
The OGameEval-Gen dataset contains 50 text-video pairs sampled from the OGameData-GEN dataset, ensuring that these samples were not used during training.
For a fair comparison, the captions were generated using GPT-4o.
For the OGameEval-Ins dataset, we sampled the last frame of ten videos from the OGameData-INS eval dataset, which were also unused during training.
We generated two types of instructional captions for each video: character control (e.g., move-left, move-right) and environment control (e.g., turn to rainy, turn to sunny, turn to foggy, and create a river in front of the main character).
Consequently, we have 60 text-video pairs for evaluating control ability.
To ensure a fair comparison, for each instruction, we utilized GPT-4o to generate two types of captions: Structural instructions to evaluate our methods and dense captions to evaluate other methods.

Metric Details.
To comprehensively evaluate the performance of GameGen-$\mathbb{X}$, we utilize a suite of metrics that capture various aspects of video generation quality and interactive control.
This implementation is based on VBench (Huang et al. ()) and CogVideoX (Yang et al. ()).
By employing this set of metrics, we aim to provide a comprehensive evaluation of GameGen-$\mathbb{X}$’s capabilities in generating high-quality, realistic, and interactively controllable video game content.
The details are following:

FID (Fréchet Inception Distance) (Heusel et al. ()): Measures the visual quality of generated frames by comparing their distribution to real frames. Lower scores indicate better quality.

FVD (Fréchet Video Distance) (Rakhimov et al. ()): Assesses the temporal coherence and overall quality of generated videos. Lower scores signify more realistic and coherent video sequences.

TVA (Text-Video Alignment) (Yang et al. ()):
Following the evaluation criteria established by CogVideoX, we conducted a single-blind study to assess text-video alignment.
To ensure fairness, we enlisted ten human experts who independently scored the alignment for each video sample.
These samples were randomly distributed among the evaluators, and importantly, the origin of the samples, specifically which models generated them, was not disclosed to the experts.
The final alignment score for each video is the average of these evaluations, with higher scores indicating superior alignment.
The score details are shown in Table 6.

SR (Success Rate):
We assess the model’s control capability through a collaboration between humans and AI, calculating a success rate.
On the one hand, the ten experts directly evaluate whether the control is successful.
On the other hand, we leverage a large language model, Pllava, to generate dense captions for each video, ensuring fairness in evaluation.
We then examine whether keywords from the instruction text, like ”turn left,” ”turn right,” or ”rainy,” are present in the generated captions.
If a keyword appears, the control is deemed successful.
The final score is the average, and higher scores reflect models with greater control precision.

UP (User Preference) (Yang et al. ()):
In alignment with the methods of CogVideoX, we implemented a single-blind study to evaluate video quality. To ensure fairness, we randomly distributed the samples to ten human experts who assessed the videos based on quality attributes such as realism and immersion, without considering text-video alignment. The experts were unaware of which models produced the samples. The final quality score for each video is the average of evaluations from all ten experts. The details are shown in Table 7.

MS (Motion Smoothness) (Huang et al. ()): Measures the fluidity of motion in the generated videos. Higher scores reflect smoother transitions between frames.

DD (Dynamic Degrees) (Huang et al. ()): Assesses the diversity and complexity of dynamic elements in the video. Higher scores indicate richer and more varied content.

SC (Subject Consistency) (Huang et al. ()): Measures the consistency of subjects (e.g., characters, objects) throughout the video. Higher scores indicate better consistency.

IQ (Imaging Quality) (Huang et al. ()): Measures the technical quality of the generated frames, including sharpness and resolution. Higher scores indicate clearer and more detailed images.

**Table 6: Text-video Alignment Evaluation Criteria.**
| Score | Evaluation Criteria |
| --- | --- |
| 1 | 100% follow the text instruction requirements, including but not limited to: elements completely correct, quantity requirements consistent, elements complete, features accurate, etc. |
| 0.5 | 100% follow the text instruction requirements, but the implementation has minor flaws such as distorted main subjects or inaccurate features. |
| 0 | Does not 100% follow the text instruction requirements, with any of the following issues: 1. Generated elements are inaccurate 2. Quantity is incorrect 3. Elements are incomplete 4. Features are inaccurate |

**Table 7: User Preference Evaluation Criteria.**
| Score | Evaluation Criteria |
| --- | --- |
| 1 | High video quality: 1. The appearance and morphological features of objects in the video are completely consistent 2. High picture stability, maintaining high resolution consistently 3. Overall composition/color/boundaries match reality 4. The picture is visually appealing |
| 0.5 | Average video quality: 1. The appearance and morphological features of objects in the video are at least 80% consistent 2. Moderate picture stability, with only 50% of the frames maintaining high resolution 3. Overall composition/color/boundaries match reality by at least 70% 4. The picture has some visual appeal |
| 0 | Poor video quality: large inconsistencies in appearance and morphology, low video resolution, and composition/layout not matching reality |

Ablation Experiment Design Details.
We evaluate our proposed methods from two perspectives: generation and control ability.
Consequently, we design a comprehensive ablation study.
Due to the heavy cost of training on our OGameData-GEN dataset, we follow the approach of Pixart-alpha (Chen et al. ()) and sample a smaller subset for the ablation study.
Specifically, we sample 20k samples from OGameData-GEN to train the generation ability and 10k samples from OGameData-INS to train the control ability.
This resulted in two datasets, OGameData-GEN-Abl and OGameData-INS-Abl.
The generation process is trained for 3 epochs, while the control process is trained for 2 epochs.
All experiments are conducted on 8 H100 GPUs, utilizing the PyTorch framework.
Here, we provide a detailed description of our ablation studies:

Baseline: The baseline’s setting is aligned with our model.
Only utilizing a smaller dataset and training 3 epochs for generation and 2 epochs for instruction tuning with InstructNet.

w/ MiraData: To demonstrate the quality of our proposed datasets, we sampled the same video hours sample from MiraData. These videos are also from the game domain. Only utilizing this dataset, we train a model for 3 epochs.

w/ Short Caption: To demonstrate the effectiveness of our captioning methods, we re-caption the OGameData-Gen-Abl dataset using simple and short captions. We train the model’s generation ability for 3 epochs and use the rewritten short OGameEval-Gen captions to evaluate this variant.

w/ Progressive Training: To demonstrate the effectiveness of our mixed-scale and mixed-temporal training, we adopt a different training method. Initially, we train the model using 480p resolution and 102 frames for 2 epochs, followed by training with 720p resolution and 102 frames for an additional epoch.

w/o Instruct Caption: We recaption the OGameData-INS-Abl dataset on our ablation utilizing a dense caption. Based on this dataset and the baseline model, we train the model with InstructNet for 2 epochs to evaluate the effectiveness of our proposed structural caption methods.

w/o Decomposition: The decoupling of generation and control tasks is essential in our approach. In this variant, we combine these two tasks. We trained the model on the merged OGameData-Gen-Abl and OGameData-INS-Abl dataset for 5 epochs, splitting the training equally: 50% for generation and 50% for instruction tuning.

w/o InstructNet: To evaluate the effectiveness of our InstructNet, we utilized OGameData-INS-Abl to continue training the baseline model for control tasks for 2 epochs.

### C.2 Further Qualitative Experiments

Basic Functionality.
Our model is designed to generate high-quality game videos with creative content, as illustrated in Fig. 16, Fig. 17, Fig. 18, and Fig. 19.
It demonstrates a strong capability for diverse scene generation, including the creation of main characters from over 150 existing games as well as novel, out-of-domain characters.
This versatility extends to simulating a wide array of actions such as flying, driving, and biking, providing a wide variety of gameplay experiences.
In addition, our model adeptly constructs environments that transition naturally across different seasons, from spring to winter.
It can depict a range of weather conditions, including dense fog, snowfall, heavy rain, and ocean waves, thereby enhancing the ambiance and immersion of the game.
By introducing diverse and dynamic scenarios, the model adds depth and variety to generated game content, offering a glimpse into potential engine-like features from generative models.

Figure: Figure 16: Character Generation Diversity. The model demonstrates its capability to generate a wide range of characters. The first three rows depict characters from existing games, showcasing detailed and realistic designs. The last two rows present open-domain character generation, illustrating the model’s versatility in creating unique and imaginative characters.
Refer to caption: x13.png

Figure: Figure 17: Action Variety in Scene Generation. The model effectively demonstrates diverse action scenarios. From top to bottom: piloting a helicopter, flying through a canyon, third-person driving, first-person motorcycle riding, and third-person motorcycle riding. Each row showcases the model’s dynamic range in generating realistic and varied action sequences.
Refer to caption: x14.png

Figure: Figure 18: Environmental Variation in Scene Generation. The model illustrates its capability to produce diverse environments. From top to bottom: a summer scene with an approaching hurricane, a snow-covered winter village, a summer thunderstorm, lavender fields in summer, and a snow-covered winter landscape. These examples highlight the model’s ability to capture different seasonal and weather conditions vividly.
Refer to caption: x15.png

Figure: Figure 19: Event Diversity in Scene Generation. The model showcases its ability to depict a range of dynamic events. From top to bottom: dense fog, a raging wildfire, heavy rain, and powerful ocean waves. Each scenario highlights the model’s capability to generate realistic and intense atmospheric conditions.
Refer to caption: x16.png

Open-domain Generation Comparison.
To evaluate the open-domain content creation capabilities of our method compared to other open-source models, we utilized GPT-4o to randomly generate captions.
These captions were used to create open-domain game video demos.
We selected three distinct caption types: Structural captions aligned with our dataset, short captions, and dense and general captions that follow human style.
The results for Structural captions are illustrated in Fig. 21, Fig. 20, Fig. 22, Fig. 23, Fig. 24, and Fig. 25.
The outcomes for short captions are depicted in Fig. 26 and Fig. 27, while the results for dense captions are visualized in Fig. 28.
For each caption type, we selected one example for detailed analysis.
As illustrated in Fig. 22, we generated a scene depicting a warrior walking through a stormy wasteland.
The results show that CogVideoX lacks scene consistency due to dramatic light changes.
In contrast, Opensora-Plan fails to accurately follow the user’s instructions by missing realistic lighting effects.
Additionally, Opensora’s output lacks dynamic motion, as the main character appears to glide rather than walk.
Our method achieves superior results compared to these approaches, providing a more coherent and accurate depiction.
We selected the scene visualized in Fig. 27 as an example of short caption generation.
As depicted, the results from CogVideoX fail to fully capture the textual description, particularly missing the ice-crystal hair of the fur-clad wanderer.
Additionally, Opensora-Plan lacks the auroras in the sky, and Opensora’s output also misses the ice-crystal hair feature.
These shortcomings highlight the robustness of our method, which effectively interprets and depicts details even with concise captions.
The dense caption results are visualized in Fig. 28.
Our method effectively captures the text details, including the golden armor and the character standing atop a cliff.
In contrast, other methods fail to accurately depict the golden armor and the cliff, demonstrating the superior capability of our approach in representing detailed information.

Figure: Figure 20: Structural Prompt: A spectral mage explores a haunted mansion filled with ghostly apparitions. In “Phantom Manor,” the protagonist, a mysterious figure shrouded in ethereal robes, glides through the dark, decaying halls of an ancient mansion. The walls are lined with faded portraits and cobweb-covered furniture. Ghostly apparitions flicker in and out of existence, their mournful wails echoing through the corridors. The mage’s staff glows with a faint, blue light, illuminating the path ahead and revealing hidden secrets. The air is thick with an eerie, supernatural presence, creating a chilling, immersive atmosphere. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.
Refer to caption: x17.png

Figure: Figure 21: Structural Prompt: A robotic explorer traverses a canyon filled with ancient, alien ruins. In “Mechanized Odyssey,” the main character, a sleek, humanoid robot with a glowing core, navigates through a vast, rocky canyon. The canyon walls are adorned with mysterious, ancient carvings and partially buried alien structures. The robot’s sensors emit a soft, blue light, illuminating the path ahead and revealing hidden details in the environment. The sky is a deep, twilight purple, with distant stars beginning to appear, adding to the sense of exploration and discovery. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.
Refer to caption: x18.png

Figure: Figure 22: Structural Prompt: A lone warrior walks through a stormy wasteland, the sky filled with lightning and dark clouds. In “Stormbringer“, the protagonist, clad in weathered armor with a glowing amulet, strides through a barren, rocky landscape. The ground is cracked and dry, and the air is thick with the smell of ozone. Jagged rocks and twisted metal structures dot the horizon, while bolts of lightning illuminate the scene intermittently. The warrior’s path is lit by the occasional flash, creating a dramatic and foreboding atmosphere. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.
Refer to caption: x19.png

Figure: Figure 23: Structural Prompt:A cybernetic detective walks down a neon-lit alley in a bustling city. In “Neon Shadows,” the protagonist wears a trench coat with glowing circuitry, navigating through a narrow alley filled with flickering holographic advertisements. Rain pours down, causing puddles on the ground to reflect the vibrant city lights. The buildings loom overhead, casting long shadows that create a sense of depth and intrigue. The detective’s steps are steady, their eyes scanning the surroundings for clues in this cyberpunk mystery. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.‘
Refer to caption: x20.png

Figure: Figure 24: Structural Prompt: A spectral knight walks through a haunted forest under a blood-red moon. In “Phantom Crusade,” the protagonist, a translucent, ethereal figure clad in spectral armor, moves silently through a dark, misty forest. The trees are twisted and gnarled, their branches reaching out like skeletal hands. The blood-red moon casts an eerie light, illuminating the path with a sinister glow. Ghostly wisps float through the air, adding to the chilling atmosphere. The knight’s armor shimmers faintly, reflecting the moonlight and creating a hauntingly beautiful scene. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.
Refer to caption: x21.png

Figure: Figure 25: Structural Prompt: A cybernetic monk walks through a high-tech temple under a serene sky. In “Digital Zen,” the protagonist, a serene figure with cybernetic enhancements integrated into their traditional monk robes, walks through a temple that blends ancient architecture with advanced technology. Soft, ambient lighting and the gentle hum of technology create a peaceful atmosphere. The temple’s walls are adorned with holographic screens displaying calming patterns and mantras. The monk’s cybernetic components emit a faint, soothing glow, symbolizing the fusion of spirituality and technology in this tranquil sanctuary. aesthetic score: 6.55, motion score: 12.69, perspective: Third person.
Refer to caption: x22.png

Figure: Figure 26: Short Prompt: “Echoes of the Void”: A figure cloaked in darkness with eyes like stars walks through a valley where echoes of past battles appear as ghostly figures. The ground is littered with ancient, rusted weapons, and the sky is an endless void with a single, massive planet looming close, its rings casting eerie shadows.
Refer to caption: x23.png

Figure: Figure 27: Short Prompt: “Glacier Wanderer”: A fur-clad wanderer with ice-crystal hair treks across a glacier under a sky painted with auroras. Giant ice sculptures of mythical creatures line his path, each breathing out cold mist. The horizon shows mountains that pierce the sky, glowing with an inner light.
Refer to caption: x24.png

Figure: Figure 28: Dense Prompt: A lone Tarnished warrior, clad in tattered golden armor that glows with inner fire, stands atop a cliff overlooking a vast, blighted landscape. The sky burns with an otherworldly amber light, casting long shadows across the desolate terrain. Massive, twisted trees with bark-like blackened iron stretch towards the heavens, their branches intertwining to form grotesque arches. In the distance, a colossal ring structure hovers on the horizon, its edges shimmering with arcane energy. The air is thick with ash and embers, swirling around the warrior in mesmerizing patterns. Below, a sea of mist conceals untold horrors, occasionally parting to reveal glimpses of ancient ruins and fallen titans. The warrior raises a curved sword that pulses with crimson runes, preparing to descend into the nightmarish realm below. The scene exudes a sense of epic scale and foreboding beauty, capturing the essence of a world on the brink of cosmic change.
Refer to caption: x25.png

Interactive Control Ability Comparison.
To comprehensively assess the controllability of our model, we compared it with several commercial models, including Runway Gen2, Pika, Luma, Tongyi, and KLing 1.5. Initially, we generated a scene using the same caption across all models.
Subsequently, we extended the video by incorporating text instructions related to environmental changes and character direction. The results are presented in Fig. 29, Fig. 30, Fig. 31, Fig. 32, and Fig. 33.
Our findings reveal that while commercial models can produce high-quality outputs, Runway, Pika, and Luma fall short of meeting game demo creation needs due to their free camera perspectives, which lack the dynamic style typical of games.
Although Tongyi and KLing can generate videos with a game-like style, they lack adequate control capabilities; Tongyi fails to respond to environmental changes and character direction, while KLing struggles with character direction adjustments.

Figure: Figure 29: Comparison results of GameGen-$\mathbb{X}$ with commercial models. This figure contrasts our approach with several commercial models. The left side displays results from text-generated videos, while the right side shows text-based continuation of videos. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Kling1.5, and GameGen-$\mathbb{X}$. Luma, Kling1.5, and GameGen-$\mathbb{X}$ effectively followed the caption in the first part, including capturing the fiery red sky, while Gen2, Pika, and Tongyi did not. In the second part, our method successfully directed the character to turn right, a control other methods struggled to achieve.
Refer to caption: x26.png

Figure: Figure 30: Comparison results of GameGen-$\mathbb{X}$ with commercial models. This figure presents a comparison between our approach and several commercial models. The left side depicts text-generated video results, while the right side shows text-based video continuation. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Kling1.5, and GameGen-$\mathbb{X}$. In the initial segment, Luma, Kling1.5, and GameGen-$\mathbb{X}$ effectively adhered to the caption by accurately depicting the dense fog and path, while other models lacked these elements. In the continuation, only Kling1.5 and our approach successfully transformed the environment by clearing the fog, whereas other methods failed to follow the text instructions.
Refer to caption: x27.png

Figure: Figure 31: Comparison results of GameGen-$\mathbb{X}$ with commercial models. This figure compares our approach with several commercial models. The left side displays text-generated video results, while the right side shows text-based video continuation. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Kling1.5, and our method. In the initial segment, all methods effectively followed the caption. However, in the continuation segment, only our model successfully controlled the character to turn left.
Refer to caption: x28.png

Figure: Figure 32: Comparison results of GameGen-$\mathbb{X}$ with commercial models. This figure presents a comparison between our approach and several commercial models. The left side showcases text-generated video results, while the right side illustrates video continuation using text. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Kling1.5, and our method. In the first segment, only Pika, Kling1.5, and our method correctly followed the text description. Other models either failed to display the character or depicted them entering the cave instead of exiting. In the continuation segment, both our method and Kling1.5 successfully guided the character out of the cave. Our approach maintains a consistent camera perspective, enhancing the game-like experience compared to Kling1.5.
Refer to caption: x29.png

Figure: Figure 33: Comparison results of GameGen-$\mathbb{X}$ with commercial models. This figure presents a comparison between our approach and several commercial models. The left side shows text-generated video results, while the right side illustrates video continuation using text. From top to bottom, the models include Runway Gen2, Pika, Tongyi, Luma, Kling1.5, and our method. In the initial segment, all methods successfully followed the text description. However, in the continuation segment, only our method effectively altered the environment by darkening the sky and revealing the stars.
Refer to caption: x30.png

Video Prompt.
In addition to text and keyboard inputs, our model accepts video prompts, such as edge sequences or motion vectors, as inputs.
This capability allows for more customized video generation.
The generated results are visualized in Fig. 34 and Fig. 35.

Figure: Figure 34: Video Generation with Motion Vector Input. This figure demonstrates how given motion vectors enable the generation of videos that follow specific movements. Different environments were created using various text descriptions, all adhering to the same motion pattern.
Refer to caption: x31.png

Figure: Figure 35: Video Scene Generation with Canny Sequence Input. Using the same canny sequence, different text inputs can generate video scenes that match specific content requirements.
Refer to caption: x32.png

## Appendix D Discussion

### D.1 Limitations

Despite the advancements made by GameGen-$\mathbb{X}$, several key challenges remain:

Real-Time Generation and Interaction:
In the realm of gameplay, real-time interaction is crucial, and there is a significant appeal in developing a video generation model that enables such interactivity.
However, the computational demands of diffusion models, particularly concerning the sampling process and the complexity of spatial and temporal self-attention mechanisms, present formidable challenges.

Consistency in Auto-Regressive Generation:
Auto-regressive generation often leads to accumulated errors, which can affect both character consistency and scene coherence over long sequences (Valevski et al. ()).
This issue becomes particularly problematic when revisiting previously generated environments, as the model may struggle to maintain a cohesive and logical progression.

Complex Action Generation:
The model struggles with fast and complex actions, such as combat sequences, where rapid motion exceeds its current capacity (Huang et al. ()).
In these scenarios, video prompts are required to guide the generation, thereby limiting the model’s autonomy and its ability to independently generate realistic, high-motion content.

High-Resolution Generation:
GameGen-$\mathbb{X}$ is not yet capable of generating ultra-high-resolution content (e.g., 2K/4K) due to memory and processing constraints (He et al. ()).
The current hardware limitations prevent the model from producing the detailed and high-resolution visuals that are often required for next-gen AAA games, thereby restricting its applicability in high-end game development.

Long-Term Consistency in Video Generation:
In gameplay, maintaining scene consistency is crucial, especially as players transition between and return to scenes. However, our model currently exhibits a limitation in temporal coherence due to its short-term memory capacity of just 1-108 frames. This constraint results in significant scene alterations upon revisiting, highlighting the need to enhance our model’s memory window for better long-term scene retention. Expanding this capability is essential for achieving more stable and immersive video generation experiences.

Physics Simulation and Realism:
While our methods achieve high visual fidelity, the inherent constraints of generative models limit their ability to consistently adhere to physical laws. This includes realistic light reflections and accurate interactions between characters and their environments. These limitations highlight the challenge of integrating visually compelling content with the physical realism required for experience.

Multi-Character Generation:
The distribution of our current dataset limits our model’s ability to generate and manage interactions among multiple characters. This constraint is particularly evident in scenarios requiring coordinated combat or cooperative tasks.

Integration with Existing Game Engines:
Presently, the outputs of our model are not directly compatible with existing game engines. Converting video outputs into 3D models may offer a feasible pathway to bridge this gap, enabling more practical applications in game development workflows.

In summary, while GameGen-$\mathbb{X}$ marks a significant step forward in open-world game generation, addressing these limitations is crucial for its future development and practical application in real-time, high-resolution, and complex game scenarios.

### D.2 Potential Future Works

Potential future works may benefit from the following aspects:

Real-Time Optimization:
One of the primary limitations of current diffusion models, including GameGen-$\mathbb{X}$, is the high computational cost that hinders real-time generation.
Future research can focus on optimizing the model for real-time performance (Zhao et al. (; ); Xuanlei Zhao & You (), essential for interactive gaming applications.
This could involve the design of lightweight diffusion models that retain generative power while reducing the inference time.
Additionally, hybrid approaches that blend autoregressive methods with non-autoregressive mechanisms may strike a balance between generation speed and content quality (Zhou et al. ()).
Techniques like model distillation or multi-stage refinement might further reduce the computational overhead, allowing for more efficient generation processes (Wang et al. ()).
Such advances will be crucial for applications where instantaneous feedback and dynamic responsiveness are required, such as real-time gameplay and interactive simulations.

Improving Consistency:
Maintaining consistency over long sequences remains a significant challenge, particularly in autoregressive generation, where small errors can accumulate over time and result in noticeable artifacts.
To improve both spatial and temporal coherence, future works may incorporate map-based constraints that impose global structural rules on the generated scenes, ensuring the continuity of environments even over extended interactions (Yan et al. ()).
For character consistency, the introduction of character-conditioned embeddings could help the model maintain the visual and behavioral fidelity of in-game characters across scenes and actions (He et al. (); Wang et al. ().
This can be achieved by integrating embeddings that track identity, pose, and interaction history, helping the model to better account for long-term dependencies and minimize discrepancies in character actions or appearances over time.
These approaches could further enhance the realism and narrative flow in game scenarios by preventing visual drift.

Handling of Complex Actions:
Currently, GameGen-$\mathbb{X}$ struggles with highly dynamic and complex actions, such as fast combat sequences or large-scale motion changes, due to limitations in capturing rapid transitions.
Future research could focus on enhancing the model’s ability to generate realistic motion by integrating motion-aware components, such as temporal convolutional networks or recurrent structures, that better capture fast-changing dynamics (Huang et al. ()).
Moreover, training on high-frame-rate datasets would provide the model with more granular temporal information, improving its ability to handle quick motion transitions and intricate interactions.
Beyond data, incorporating external guidance, such as motion vectors or pose estimation prompts, can serve as additional control signals to enhance the generation of fast-paced scenes.
These improvements would reduce the model’s dependency on video prompts, enabling it to autonomously generate complex and fast-moving actions in real-time, increasing the depth and variety of in-game interactions.

Advanced Model Architectures:
Future advancements in model architecture will likely move towards full 3D representations to better capture the spatial complexity of open-world games.
The current 2D+1D approach, while effective, limits the model’s ability to fully understand and replicate 3D spatial relationships.
Transitioning from 2D+1D attention-based video generation to more sophisticated 3D attention architectures offers an exciting direction for improving the coherence and realism of generated game environments (Yang et al. (); Lab & etc. ()).
Such a framework could better grasp the temporal dynamics and spatial structures within video sequences, improving the fidelity of generated environments and actions.
On the other dimension, instead of only focusing on the generation task, future models could integrate a more unified framework that simultaneously learns both video generation and video understanding.
By unifying generation and understanding, the model could ensure consistent layouts, character movements, and environmental interactions across time, thus producing more cohesive and immersive content (Emu3 Team ()).
This approach could significantly enhance the ability of generative models to capture complex video dynamics, advancing the state of video-based game simulation technology.

Scaling with Larger Datasets:
While OGameData provides a comprehensive foundation for training GameGen-$\mathbb{X}$, further improvements in model generalization could be achieved by scaling the dataset to include more diverse examples of game environments, actions, and interactions (Ju et al. (); Wang et al. ()).
Expanding the dataset with additional games, including those from a wider range of genres, art styles, and gameplay mechanics, would expose the model to a broader set of scenarios.
This would enhance the model’s ability to generalize across different gaming contexts, allowing it to generate more diverse and adaptable content.
Furthermore, incorporating user-generated content, modding tools, or procedurally generated worlds could enrich the dataset, offering a more varied set of training examples.
This scalability would also improve robustness, reducing overfitting and enhancing the model’s capacity to handle novel game mechanics and environments, thereby improving performance across a wider array of use cases.

Integration of 3D Techniques:
A key opportunity for future development lies in integrating advanced 3D modeling with 3D Gaussian Splatting (3DGS) techniques (Kerbl et al. ()).
Moving beyond 2D video-based approaches, incorporating 3DGS allows the model to generate complex spatial interactions with realistic object dynamics.
3DGS facilitates efficient rendering of intricate environments and characters, capturing fine details such as lighting, object manipulation, and collision detection.
This integration would result in richer, more immersive gameplay, enabling players to experience highly interactive and dynamic game worlds (Shin et al. ()).

Virtual to Reality:
A compelling avenue for future research is the potential to adapt these generative techniques beyond gaming into real-world applications.
If generative models can accurately simulate highly realistic game environments, it opens the possibility of applying similar techniques to real-world simulations in areas such as autonomous vehicle testing, virtual training environments, augmented reality (AR), and scenario planning.
The ability to create interactive, realistic, and controllable simulations could have profound implications in fields such as robotics, urban planning, and education, where virtual environments are used to test and train systems under realistic but controlled conditions.
Bridging the gap between virtual and real-world simulations would not only extend the utility of generative models but also demonstrate their capacity to model complex, dynamic systems in a wide range of practical applications.

In summary, addressing these key areas of future work has the potential to significantly advance the capabilities of generative models in game development and beyond. Enhancing real-time generation, improving consistency, and incorporating advanced 3D techniques will lead to more immersive and interactive gaming experiences, while the expansion into real-world applications underscores the broader impact these models can have.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Where did the dataset come from?
2. Why splice up the data by scene?
3. What was the methodology with the human expert evaluators? 
4. Did the authors receive ethics approval?
5. The authors indicate the dataset is split nearly evening between first and third person video, but primarily show results for third person video, why is this?
soundness: 3
strengths: The primary strength of the paper without question is the authors' new dataset. There is no dataset even close to this in terms of quality or size, it's a really exciting potential addition to this research area. This is primarily a strength in terms of originality, quality, and significance. I say primarily since the authors do not include access to the dataset at the review stage, though they do not have some metrics. 

The authors' GameGen-X and InstructNet are also strengths, but I have concerns with them limiting them as strengths, as I'll get to below.
summary: In this paper the authors present a new dataset of modern AAA games for the purpose of world model training, which they call the Open-World Video Game Dataset (OGameData). Then then present their model, GameGen-X, a diffusion transformer for generating and controlling game video. GameGen-X is similar to other video generation models with the addition of InstructNet, which modifies the latents of GameGen-X for controllability. The authors present comparisons with a number of open-source video models. Finding that they generally produce more game-like video and may be better at control, according to some metrics.
weaknesses: The paper is relatively free of weaknesses in terms of originality, thanks in large part due to OGameData. However, the authors' work has some weaknesses in terms of the quality, clarity, and significance. This primarily comes down to (1) the authors' stated motivations and how this aligns with their work, (2) the way the authors overview their system, and (3) the experiments

### Motivations and the Dataset

The authors motivate in two primary ways: (1) imagining this as a prototyping or early development tool for open world game developers and (2) imagining this as leading to future interactive experiences with greater user control. These are fine as motivations, but the authors' choice of processing the dataset runs counter to them, somewhat. Specifically, the authors have broken apart their video clips into distinct scenes, meaning their model or other models trained on this dataset will not observe scene transitions. This is somewhat of an oddity for either of the authors' stated motivations and there's no justification for this choice given in the paper. Similarly, the authors do not actually have control input for any of the collected game data. This is again a bit of an oddity given the authors' stated purposes. I would guess that the authors collected this data from some sort of web scrape of gameplay video rather than collecting the video themselves through playing games such that they could capture actual control inputs. However, this isn't specified in the paper. This is a potential concern, especially if the authors did scrape an online repository of videos when such scraping went against the terms and service of the site in question. This should be clarified. 

### System Overview

Simply, the authors do not describe any of their system implementation in sufficient detail for replication. The authors state that code will be made available but do not make such code available for review. As such, there's no detail on the system architecture in terms of parameters or hyper parameters. The authors also not disclose the computation required to train their model or the training split used from their dataset. All of this would be required (potentially in appendices or in an external code repo) to ensure that the work is replicable. 

### Experiments

I have a number of concerns with the current setup of the experiments. The authors only compare against open video models, which are not attempting the same task and are not trained on the same dataset. As such, it's unclear the extent to which this is just that the training dataset for GameGen-X is more similar to the test dataset. While the authors do specify that the experiments are over test data, given the distribution of games in the dataset, its highly likely that GameGen-X had already trained on the same game that the test data used in the experiments came from. As such, this seems much closer to testing on the training set. 

The authors also have several metrics that require human expert raters, but who these experts were or what information they had is not specified. Further, the authors say they only use a single-blind setup, which may suggest the experts knew who they were. As such, there's a clear risk of bias here in terms of the experts feeling social pressure to more positively rate the more game-like videos if they knew that was the goal of this research. Clarity around the methodology and whether the authors have ethics approval would be necessary for readers to trust any of the human participant-based results. Relatedly, the authors state that the SR metric is "evaluated by both human experts and PLLaVA". However, the authors only present a single number. As such, it's not clear how the human expert and PLLaVA evaluations were combined. This throws doubt upon the SR metrics. 

The authors repeatedly bold values from their own work, indicating it is the best, when there is equivalent work from prior models. This may mislead readers in terms of understanding when the performance. 

The ablation study is helpful, as it demonstrates the value of OGameData, and several of the authors' components. However, since the authors do not train any other models on their dataset outside of these ablations, it's difficult to determine the exact value of the different components of their work. 

Overall, I'd say that the experiments are currently the largest weakness of this paper.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to weakness.
soundness: 3
strengths: 1. A large well-curated dataset for open-world video games. The curation of the dataset contains filtering on different aspects (e.g., semantic alignment, motion), which results in a high-quality large-scale dataset. 
2. The idea to build a video diffusion model for open-world video games is essentially interesting, and the results and demo videos are impressive. Besides, quantitatively, the proposed approach also achieves better performance than other SoTA diffusion models. 
3. Detailed ablation studies demonstrate the effectiveness of the proposed component (i.e., two-stage training strategy and the design of the instructnet).
4. This paper is well-written and easy to follow.
summary: This paper proposes GameGen-X, a diffusion based model for open-world game generation. Specifically, this paper proposes two detailed crafted datasets: OGameData-Gen and OGameData-Ins. OGameData-Gen is used to pre-train the diffusion model to understand and generate continuous open-world game-style videos, where OGameData-Ins is used to instruct tune the model to understand special inputs (e.g., keyboard inputs) to better control the continuation of the game generation based on some input frames. The dataset is well-curated to have 1M videos, with multiple filtering metrics and human-in-loop filtering to maintain the high quality. Then, this paper trains a video diffusion model with two-stage training on the two datasets for open-world game generation. Specifically, an instruct net is designed to take in different special inputs. Empirically, on their provided evaluation dataset, GameGen-X achieves superior performance than other state-of-the-art video diffusion models (e.g., kling).
weaknesses: 1. One main concern is the proposed GameGen-X is specially fine-tuned/designed for open-world video games, while other diffusion models compared (e.g., kling) are trained for a general text-to-video generation, which makes the comparison somehow unfair to other models. 
2. The qualitative examples in the website demo for game generation (e.g., under generation comparison) don't seem to look much better than other models (e.g., cogvideoX).

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 1
questions: Clarification Questions:

-	Is it the correct understanding that the OGameData-GEN dataset comprises of data from 150 video games, whilst the OGameData-INS dataset contains only a subset of 5 game titles? Without checking Appendix B for clarification, it is difficult for the reader to grasp these details from the main body of the paper.
-	For video clip compression (Section 3.2) it would be good to add more details about the size of the latent representation z, as well as the resolution of the video clips used in training.
-	How were the spatial and temporal downsampling factors determined (s_t, s_h, s_w)?
-	In section 3.2, under unified video generation and continuation, you mention incorporating bucket training, classifier-free diffusion guidance and rectified flow for better generalization performance – did you run any ablation studies to understand better the impact of introducing these 3 components?
-	What are the values x for context length that you considered for video continuation?
-	In the InstructNet design, what were the considerations for choosing N (the number of InstructNet blocks)? Did you experiment with different values?
-	For the multi-modal experts introduced in Section 3.3, what are the sizes considered for the instruction embeddings and keyboard input embeddings (f_I and f_O)?
-	Under Interactive control, you mention the incorporation of video prompts V_p enhances the model’s ability to generate motion-consistent frames – did you conduct any experiments or ablations to measure the observed improvement?
-	Is there a mention on the computational resources required to store and stream the data for training, as well as for training the foundation model and InstructNet? It would be a useful proxy for people planning to reproduce the work.
-	Similarly, is there any information presented on the inference times of GameGen-X?
-	In evaluating the control ability you mention using both human experts and PLLaVa. What is the ratio between the 2 evaluation modalities?
-	On qualitative results for generation, apart from the discussion on diversity, would it be possible to elaborate on the length and consistency of the videos generated by GameGen-X? From the demo videos included, most are under <8-10 seconds.
-	In the ablation studies (Tables 4 and 5), there seem to be no DD and IQ metrics – what is the reason for that?

Minor comments/Suggestions:

-	In Section 2.2, it would be good to specify the human experts’ level of familiarity with the titles and elaborate on how the GPT-4o text annotations were checked for quality and accuracy.
-	In Section 3.3, you introduce the c condition under the Interactive Control subsection, but it is mentioned beforehand in Multi-modal experts. It would be clearer to the reader to introduce the structure of c under the Multi-modal experts’ subsection, where it appears for the first time.
-	For readability, it would be good to illustrate z, the latent variable in Figure 4.
-	It would be useful to include a more detailed explanation on the choice of baselines in the experiments Section. For example, Mira is not included under the results for control ability, is it because it does not have support for it? It would be good to clarify that.
-	It would be good to link to Appendix D (Discussion) when mentioning remaining challenges in the conclusion.
-	I know this appeared after the submission deadline, but it would be worth adding to the related work section as a referece: https://www.decart.ai/articles/oasis-interactive-ai-video-game-model
soundness: 3
strengths: - The work is original in the sense that is the first main contribution to the field in terms of interactive video game generation in large scale, complex, open worlds

- It is great to see such examples of tackling complex research environments at scale, with potential direct benefits to the game development process. 

- The author(s) introduce a complex system, both in terms of the dataset it required for training (including a resource intensive collection and curation process), as well as in terms of the pretrained foundation model and the interactive control network, allowing users to control the output via either text or mouse and keyboard inputs
summary: In this paper, the authors introduce a diffusion transformer model aimed at generating and controlling video game sequences in challenging 3D open-domain game worlds. The authors also present the gameplay dataset they collected to train the model, OGameData. The dataset has 1 million video clips from across 150 videos, annotated with text descriptions using GPT4o. Both the model and the dataset have 2 components. One for text-to-video generation (OGameData-GEN and the pretrained foundation model) and one for instruction tuning (OGameData-INS and InstructNet).
weaknesses: - There is one strong concern I have regarding the data collection process for the OGameData dataset. My score highly depends on evidence that data collection will pass the ethics review and there is evidence provided on the consent given by the humans that produced the data. There should be understanding and agreement for it to be used for research purposes and open sourced. Please elaborate on how the data for OGameData has been collected? In Appendix B.1. you mention selecting online video websites as one of the primary sources. It would be good to know:
   - The exact sources of the video data
   - Any agreements or permissions obtained from video creators and game studios
   - The ethical review process they followed, if any
   - How you plan to address potential copyright or licensing issues

- It is unclear why all UI elements have been removed from the dataset, it would be great to gain further clarity on that from the author(s). In a lot of open-world gameplay , the player relies on UI element understanding, such as health levels, navigation information via mini maps, affordance of actions to take, inventory etc. 
   - How does this decision impact the model's ability to generate realistic gameplay experiences?
   - Do you plan to incorporate UI elements in future iterations of the model?

- Please correct me if I missed this, but the main body of the paper does not clearly indicate that all the data and the generation is within the constraints of a single agent. What is the model’s ability to model other dynamic environment elements (NPCs, other players, moving vehicles etc.)? It would be good to:
   - Explicitly state whether the model is limited to single-agent scenarios
   - If so, discuss the implications of this limitation on the model's applicability
   - If not, provide details on how the model handles multiple dynamic elements in the environment

- The paper is dense, so it took a while to disambiguate if the main body of the paper provides sufficient detail for capturing the core contributions of the paper or if a lot of essential details were included in the appendix.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.  Please provide data on the time required to generate a video segment at different resolutions or for different types of content. A section to analyze the trade-offs between generation quality and speed would be better.

2.  The training details of InstructNet lack specificity regarding the acquisition of video data corresponding to keyboard bindings. It would be beneficial to include more comprehensive information on the data collection process and the training methodology employed.
soundness: 3
strengths: 1. This work collects a substantial number of open-world game videos from over 150 games, ultimately constructing more than 1,000,000 text-video pairs with highly detailed annotations. Its scale and diversity of annotations make it stand out, and the release of this dataset is expected to advance the field of game video generation.
2. It produces high-quality, more general realistic game video content. Previous works on game video generation often focused on specific game types, primarily 2D games or limited early 3D games. This work offers a more diverse and high-definition range of scene types for game video generation.
summary: This work focuses on generating high-quality, controllable open-world game videos that feature game engine traits. It emphasizes interactive controllability to simulate gameplay effectively. Notably, the authors collected a large-scale Open-World Video Game Dataset (OGameData), which consists of over one million diverse gameplay video clips from more than 150 games, along with informative captions generated by GPT-4o. Methodologically, they introduce a diffusion transformer model as the foundation model and a specially designed network called InstructNet for interactive control. The model is trained on the large-scale OGameData dataset using a two-stage process involving pre-training of the foundation model and instruction tuning for InstructNet.
weaknesses: 1. This work attempts to address the interactive control of open-world game video generation for gameplay simulation. However, to fully tackle the interactive issue, the generation speed needs to be considered,  as interactive experiences demand stringent timing requirements, which poses significant challenges. For instance, Google’s [1] achieves real-time rendering, even making it a viable game engine. While this work focuses on higher-resolution video generation, exploring the relationship between speed and performance would be beneficial, along with providing data on rendering time and speed. 

[1] Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024.

2. The paper claims to simulate game engine features like diverse events, yet the examples provided offer quite limited dynamic event simulation, primarily addressing environmental changes like weather and lighting. There remains a gap to true gameplay simulation, such as incorporating NPC interactions or triggering more game-like special events.

Meta-review: This is a nice paper, introducing an excellent dataset for generating game videos that will likely be of much use, and a diffusion-based method for controllable game video generation. The dataset is perhaps the biggest contribution, but the method, while not super novel, is also a contribution. Potential concerns includes that the paper is a little short on technical details (but that has improved, and the code will anyway be open-sourced) and that the authors seem to overselling their contribution ("unique", "pioneering"... just stick to the facts and people will take you more seriously). The concerns are not serious enough to prevent acceptance of the paper.

Decision: Accept (Poster)