GameGen-X: Interactive Open-world Game Video Generation

Authors: Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, Hao Chen
Primary area: generative models
Keywords: open-world game video generation, interactive control, diffusion transformers

Abstract:
We introduce GameGen-$\mathbb{X}$, the first diffusion transformer model specifically designed for both generating and interactively controlling open-world game videos. 
    This model facilitates high-quality, open-domain generation by approximating various game elements, such as innovative characters, dynamic environments, complex actions, and diverse events. 
    Additionally, it provides interactive controllability, predicting and altering future content based on the current clip, thus allowing for gameplay simulation.
    To realize this vision, we first collected and built an Open-World Video Game Dataset (OGameData) from scratch. 
    It is the first and largest dataset for open-world game video generation and control, which comprises over one million diverse gameplay video clips with informative captions.
    GameGen-$\mathbb{X}$ undergoes a two-stage training process, consisting of pre-training and instruction tuning. 
    Firstly, the model was pre-trained via text-to-video generation and video continuation, enabling long-sequence open-domain game video generation with improved fidelity and coherence.
    Further, to achieve interactive controllability, we designed InstructNet to incorporate game-related multi-modal control signal experts.
    This allows the model to adjust latent representations based on user inputs, advancing the integration of character interaction and scene content control in video generation.
    During instruction tuning, only the InstructNet is updated while the pre-trained foundation model is frozen, enabling the integration of interactive controllability without loss of diversity and quality of generated content. 
    GameGen-$\mathbb{X}$ contributes to advancements in open-world game design using generative models. 
    It demonstrates the potential of generative models to serve as auxiliary tools to traditional rendering techniques, demonstrating the potential for merging creative generation with interactive capabilities.
    The project will be available at https://github.com/GameGen-X/GameGen-X.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730170832100

thread_type:
official_review

<thread_object>
creator: reviewer_1m8f

<content>
summary:
This work focuses on generating high-quality, controllable open-world game videos that feature game engine traits. It emphasizes interactive controllability to simulate gameplay effectively. Notably, the authors collected a large-scale Open-World Video Game Dataset (OGameData), which consists of over one million diverse gameplay video clips from more than 150 games, along with informative captions generated by GPT-4o. Methodologically, they introduce a diffusion transformer model as the foundation model and a specially designed network called InstructNet for interactive control. The model is trained on the large-scale OGameData dataset using a two-stage process involving pre-training of the foundation model and instruction tuning for InstructNet.

strengths:
1. This work collects a substantial number of open-world game videos from over 150 games, ultimately constructing more than 1,000,000 text-video pairs with highly detailed annotations. Its scale and diversity of annotations make it stand out, and the release of this dataset is expected to advance the field of game video generation.
2. It produces high-quality, more general realistic game video content. Previous works on game video generation often focused on specific game types, primarily 2D games or limited early 3D games. This work offers a more diverse and high-definition range of scene types for game video generation.

weaknesses:
1. This work attempts to address the interactive control of open-world game video generation for gameplay simulation. However, to fully tackle the interactive issue, the generation speed needs to be considered,  as interactive experiences demand stringent timing requirements, which poses significant challenges. For instance, Google’s [1] achieves real-time rendering, even making it a viable game engine. While this work focuses on higher-resolution video generation, exploring the relationship between speed and performance would be beneficial, along with providing data on rendering time and speed. 

[1] Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024.

2. The paper claims to simulate game engine features like diverse events, yet the examples provided offer quite limited dynamic event simulation, primarily addressing environmental changes like weather and lighting. There remains a gap to true gameplay simulation, such as incorporating NPC interactions or triggering more game-like special events.

questions:
1.  Please provide data on the time required to generate a video segment at different resolutions or for different types of content. A section to analyze the trade-offs between generation quality and speed would be better.

2.  The training details of InstructNet lack specificity regarding the acquisition of video data corresponding to keyboard bindings. It would be beneficial to include more comprehensive information on the data collection process and the training methodology employed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730486512881

thread_type:
official_review

<thread_object>
creator: reviewer_wqcm

<content>
summary:
In this paper, the authors introduce a diffusion transformer model aimed at generating and controlling video game sequences in challenging 3D open-domain game worlds. The authors also present the gameplay dataset they collected to train the model, OGameData. The dataset has 1 million video clips from across 150 videos, annotated with text descriptions using GPT4o. Both the model and the dataset have 2 components. One for text-to-video generation (OGameData-GEN and the pretrained foundation model) and one for instruction tuning (OGameData-INS and InstructNet).

strengths:
- The work is original in the sense that is the first main contribution to the field in terms of interactive video game generation in large scale, complex, open worlds

- It is great to see such examples of tackling complex research environments at scale, with potential direct benefits to the game development process. 

- The author(s) introduce a complex system, both in terms of the dataset it required for training (including a resource intensive collection and curation process), as well as in terms of the pretrained foundation model and the interactive control network, allowing users to control the output via either text or mouse and keyboard inputs

weaknesses:
- There is one strong concern I have regarding the data collection process for the OGameData dataset. My score highly depends on evidence that data collection will pass the ethics review and there is evidence provided on the consent given by the humans that produced the data. There should be understanding and agreement for it to be used for research purposes and open sourced. Please elaborate on how the data for OGameData has been collected? In Appendix B.1. you mention selecting online video websites as one of the primary sources. It would be good to know:
   - The exact sources of the video data
   - Any agreements or permissions obtained from video creators and game studios
   - The ethical review process they followed, if any
   - How you plan to address potential copyright or licensing issues

- It is unclear why all UI elements have been removed from the dataset, it would be great to gain further clarity on that from the author(s). In a lot of open-world gameplay , the player relies on UI element understanding, such as health levels, navigation information via mini maps, affordance of actions to take, inventory etc. 
   - How does this decision impact the model's ability to generate realistic gameplay experiences?
   - Do you plan to incorporate UI elements in future iterations of the model?

- Please correct me if I missed this, but the main body of the paper does not clearly indicate that all the data and the generation is within the constraints of a single agent. What is the model’s ability to model other dynamic environment elements (NPCs, other players, moving vehicles etc.)? It would be good to:
   - Explicitly state whether the model is limited to single-agent scenarios
   - If so, discuss the implications of this limitation on the model's applicability
   - If not, provide details on how the model handles multiple dynamic elements in the environment

- The paper is dense, so it took a while to disambiguate if the main body of the paper provides sufficient detail for capturing the core contributions of the paper or if a lot of essential details were included in the appendix.

questions:
Clarification Questions:

-	Is it the correct understanding that the OGameData-GEN dataset comprises of data from 150 video games, whilst the OGameData-INS dataset contains only a subset of 5 game titles? Without checking Appendix B for clarification, it is difficult for the reader to grasp these details from the main body of the paper.
-	For video clip compression (Section 3.2) it would be good to add more details about the size of the latent representation z, as well as the resolution of the video clips used in training.
-	How were the spatial and temporal downsampling factors determined (s_t, s_h, s_w)?
-	In section 3.2, under unified video generation and continuation, you mention incorporating bucket training, classifier-free diffusion guidance and rectified flow for better generalization performance – did you run any ablation studies to understand better the impact of introducing these 3 components?
-	What are the values x for context length that you considered for video continuation?
-	In the InstructNet design, what were the considerations for choosing N (the number of InstructNet blocks)? Did you experiment with different values?
-	For the multi-modal experts introduced in Section 3.3, what are the sizes considered for the instruction embeddings and keyboard input embeddings (f_I and f_O)?
-	Under Interactive control, you mention the incorporation of video prompts V_p enhances the model’s ability to generate motion-consistent frames – did you conduct any experiments or ablations to measure the observed improvement?
-	Is there a mention on the computational resources required to store and stream the data for training, as well as for training the foundation model and InstructNet? It would be a useful proxy for people planning to reproduce the work.
-	Similarly, is there any information presented on the inference times of GameGen-X?
-	In evaluating the control ability you mention using both human experts and PLLaVa. What is the ratio between the 2 evaluation modalities?
-	On qualitative results for generation, apart from the discussion on diversity, would it be possible to elaborate on the length and consistency of the videos generated by GameGen-X? From the demo videos included, most are under <8-10 seconds.
-	In the ablation studies (Tables 4 and 5), there seem to be no DD and IQ metrics – what is the reason for that?

Minor comments/Suggestions:

-	In Section 2.2, it would be good to specify the human experts’ level of familiarity with the titles and elaborate on how the GPT-4o text annotations were checked for quality and accuracy.
-	In Section 3.3, you introduce the c condition under the Interactive Control subsection, but it is mentioned beforehand in Multi-modal experts. It would be clearer to the reader to introduce the structure of c under the Multi-modal experts’ subsection, where it appears for the first time.
-	For readability, it would be good to illustrate z, the latent variable in Figure 4.
-	It would be useful to include a more detailed explanation on the choice of baselines in the experiments Section. For example, Mira is not included under the results for control ability, is it because it does not have support for it? It would be good to clarify that.
-	It would be good to link to Appendix D (Discussion) when mentioning remaining challenges in the conclusion.
-	I know this appeared after the submission deadline, but it would be worth adding to the related work section as a referece: https://www.decart.ai/articles/oasis-interactive-ai-video-game-model
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685948397

thread_type:
official_review

<thread_object>
creator: reviewer_bslg

<content>
summary:
This paper proposes GameGen-X, a diffusion based model for open-world game generation. Specifically, this paper proposes two detailed crafted datasets: OGameData-Gen and OGameData-Ins. OGameData-Gen is used to pre-train the diffusion model to understand and generate continuous open-world game-style videos, where OGameData-Ins is used to instruct tune the model to understand special inputs (e.g., keyboard inputs) to better control the continuation of the game generation based on some input frames. The dataset is well-curated to have 1M videos, with multiple filtering metrics and human-in-loop filtering to maintain the high quality. Then, this paper trains a video diffusion model with two-stage training on the two datasets for open-world game generation. Specifically, an instruct net is designed to take in different special inputs. Empirically, on their provided evaluation dataset, GameGen-X achieves superior performance than other state-of-the-art video diffusion models (e.g., kling).

strengths:
1. A large well-curated dataset for open-world video games. The curation of the dataset contains filtering on different aspects (e.g., semantic alignment, motion), which results in a high-quality large-scale dataset. 
2. The idea to build a video diffusion model for open-world video games is essentially interesting, and the results and demo videos are impressive. Besides, quantitatively, the proposed approach also achieves better performance than other SoTA diffusion models. 
3. Detailed ablation studies demonstrate the effectiveness of the proposed component (i.e., two-stage training strategy and the design of the instructnet).
4. This paper is well-written and easy to follow.

weaknesses:
1. One main concern is the proposed GameGen-X is specially fine-tuned/designed for open-world video games, while other diffusion models compared (e.g., kling) are trained for a general text-to-video generation, which makes the comparison somehow unfair to other models. 
2. The qualitative examples in the website demo for game generation (e.g., under generation comparison) don't seem to look much better than other models (e.g., cogvideoX).

questions:
Please refer to weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686390614

thread_type:
official_review

<thread_object>
creator: reviewer_c3sk

<content>
summary:
In this paper the authors present a new dataset of modern AAA games for the purpose of world model training, which they call the Open-World Video Game Dataset (OGameData). Then then present their model, GameGen-X, a diffusion transformer for generating and controlling game video. GameGen-X is similar to other video generation models with the addition of InstructNet, which modifies the latents of GameGen-X for controllability. The authors present comparisons with a number of open-source video models. Finding that they generally produce more game-like video and may be better at control, according to some metrics.

strengths:
The primary strength of the paper without question is the authors' new dataset. There is no dataset even close to this in terms of quality or size, it's a really exciting potential addition to this research area. This is primarily a strength in terms of originality, quality, and significance. I say primarily since the authors do not include access to the dataset at the review stage, though they do not have some metrics. 

The authors' GameGen-X and InstructNet are also strengths, but I have concerns with them limiting them as strengths, as I'll get to below.

weaknesses:
The paper is relatively free of weaknesses in terms of originality, thanks in large part due to OGameData. However, the authors' work has some weaknesses in terms of the quality, clarity, and significance. This primarily comes down to (1) the authors' stated motivations and how this aligns with their work, (2) the way the authors overview their system, and (3) the experiments

### Motivations and the Dataset

The authors motivate in two primary ways: (1) imagining this as a prototyping or early development tool for open world game developers and (2) imagining this as leading to future interactive experiences with greater user control. These are fine as motivations, but the authors' choice of processing the dataset runs counter to them, somewhat. Specifically, the authors have broken apart their video clips into distinct scenes, meaning their model or other models trained on this dataset will not observe scene transitions. This is somewhat of an oddity for either of the authors' stated motivations and there's no justification for this choice given in the paper. Similarly, the authors do not actually have control input for any of the collected game data. This is again a bit of an oddity given the authors' stated purposes. I would guess that the authors collected this data from some sort of web scrape of gameplay video rather than collecting the video themselves through playing games such that they could capture actual control inputs. However, this isn't specified in the paper. This is a potential concern, especially if the authors did scrape an online repository of videos when such scraping went against the terms and service of the site in question. This should be clarified. 

### System Overview

Simply, the authors do not describe any of their system implementation in sufficient detail for replication. The authors state that code will be made available but do not make such code available for review. As such, there's no detail on the system architecture in terms of parameters or hyper parameters. The authors also not disclose the computation required to train their model or the training split used from their dataset. All of this would be required (potentially in appendices or in an external code repo) to ensure that the work is replicable. 

### Experiments

I have a number of concerns with the current setup of the experiments. The authors only compare against open video models, which are not attempting the same task and are not trained on the same dataset. As such, it's unclear the extent to which this is just that the training dataset for GameGen-X is more similar to the test dataset. While the authors do specify that the experiments are over test data, given the distribution of games in the dataset, its highly likely that GameGen-X had already trained on the same game that the test data used in the experiments came from. As such, this seems much closer to testing on the training set. 

The authors also have several metrics that require human expert raters, but who these experts were or what information they had is not specified. Further, the authors say they only use a single-blind setup, which may suggest the experts knew who they were. As such, there's a clear risk of bias here in terms of the experts feeling social pressure to more positively rate the more game-like videos if they knew that was the goal of this research. Clarity around the methodology and whether the authors have ethics approval would be necessary for readers to trust any of the human participant-based results. Relatedly, the authors state that the SR metric is "evaluated by both human experts and PLLaVA". However, the authors only present a single number. As such, it's not clear how the human expert and PLLaVA evaluations were combined. This throws doubt upon the SR metrics. 

The authors repeatedly bold values from their own work, indicating it is the best, when there is equivalent work from prior models. This may mislead readers in terms of understanding when the performance. 

The ablation study is helpful, as it demonstrates the value of OGameData, and several of the authors' components. However, since the authors do not train any other models on their dataset outside of these ablations, it's difficult to determine the exact value of the different components of their work. 

Overall, I'd say that the experiments are currently the largest weakness of this paper.

questions:
1. Where did the dataset come from?
2. Why splice up the data by scene?
3. What was the methodology with the human expert evaluators? 
4. Did the authors receive ethics approval?
5. The authors indicate the dataset is split nearly evening between first and third person video, but primarily show results for third person video, why is this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732205265293

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update on Acknowledgment for Comments, Anonymous Website, Paper Revision, and Project Release

comment:
The author team of GameGen-X sincerely appreciates your contributions in handling this submission and assisting us in refining the quality of this work. We are pleased to see that the reviewers acknowledge the following aspects of our work.
 1. **The unique contribution of OGameData, which focuses on the game video domain and has a well-curated filter, design, and annotation** (Reviewer c3SK, 1M8F, bSLg).
 2. **The first major contribution to large-scale, complex, open-world interactive video game generation** (Reviewer WQcm, 1M8F).
 3. **Technical contributions of GameGen-X, including better performance, complex system design, and interactive control** (Reviewer bSLg, WQcm).
 4. **Detailed experimental design and results** (Reviewer 1M8F, bSLg)

&nbsp; 
As this is a pioneering exploration into **generating open-domain game video content and interacting with them**, we have made efforts to address the reviewers' concerns in the relevant sections.

1. **Updates on the Anonymous Website and Streaming Demo Video**

We updated several groups of qualitative comparison videos on our website (https://3a2077.github.io/). 

Additionally, to answer specific problems from reviewers supplementarily, we updated videos to support:
  - **Long video generation and consistency**: a 400-second streaming-generated video with 10 times acceleration, where it creates a man horsing in a forest and we use around 100 sets of control signal sequences to control it. https://drive.google.com/file/d/1vZE4SKzLDqfErBV0B5MAbVHUizVZysdS/view?usp=sharing.
  - **The reason for minimizing UI elements**: sample videos in an early model without minimizing the UI elements in data collection, https://drive.google.com/file/d/1Te95mJf5tdHpmUOqCrwdfDCMmhJD8168/view?usp=sharing.
  - **Ability for generating multiple objects or first perspective videos**: sample videos with multiple objects or the first perspective, https://drive.google.com/file/d/1-PLP8sohLyI5Wsn_gnOPbDFjD-_c5Ppq/view?usp=sharing.
-  **Visualization of results across various resolutions and sampling steps**: sample videos with various resolutions and sampling steps, our model can achieve 20 FPS under 320p/10 sampling steps with acceptable visual quality (https://drive.google.com/file/d/16ibysz0LpdmPvew2elD4OcWu3GLooZok/view?usp=sharing).

2. **Paper Revision**
  Based on the reviewers' valuable comments, we revised our paper from the following perspectives:
  - Added Appendix B.1 Data Availability Statement and Clarification, and revised the data collection part in Appendix B.2 Construction Details: discuss and ensure the data compliance and supplement more details for the data collection pipeline including data source and keyboard data collection method.
  - Added Appendix C Implementation and Design Details: to improve the reproduction ability, we supplement more information related to the training strategy, the core design of the model architecture, and computation resources.
  - Added Appendix D.1 Fairness Statement and Contribution Decomposition: discuss the fairness of comparison with open-sourced models from the dataset and ability perspective and the decoupling of the contribution of OGameData and GameGen-X.
  - Added Appendix D.3 Human Evaluation Details: provide more information related to the evaluation pipeline and details.
  - Added Appendix D.4 Analysis of Generation Speed and Corresponding Performance: provide more information related to the generation speed and visual quality.
  - Beyond the significant parts mentioned above, we also included other valuable suggestions from reviews in our paper. Feel free to ask for any further advice and comments.
  
3. **Dataset & Code Release**

Reviewer c3SK raises a question that we believe will be a concern for all reviewers. 
Below is our response:

**Yes**. We do have the plan to release our dataset and our code to support the research community once the paper is accepted, or possibly even earlier if the review scores are favorable. To demonstrate our commitment, we have provided a subset of small datasets on our anonymous website (around 10K annotations). Additionally, all reviewers are welcome to request extra video samples with specific prompts or functionality, and we will generate those videos and make them available on the website.
</content>
</thread_object>

</thread>
<thread>
creation date:
1739018427843

thread_type:
public_comment

<thread_object>
creator: ~haoxuan_che1

<content>
title:
Acknowledge and Revision Illustration

comment:
We appreciate the area chairs’ and reviewers' constructive feedback in reviewing our paper. Based on these comments, we have made improvements to the manuscript. We particularly appreciate the reviewers' recognition of our work, including motivations, experimental design, system architecture and the construction of the dataset.

In response to the review comments, we have made the following improvements in the camera-ready version:

- Added Appendices B.1 and B.2, providing detailed data availability statements and data collection process details.
- Added Appendix C, offering implementation details including training strategies, model architecture design, and computational resources.
- Added Appendices D.1, D.3, and D.4, supplementing fairness statements, contribution breakdown, human evaluation procedures, and generation speed and performance analysis.
- Fixed typos including the collected data modality, used computational resources, and the tense in the paper.
- Updated the comparison videos on the project website and added demonstration videos for long video generation, UI element minimization, multi-object generation, and effects at different resolutions.
- Updated acknowledgment, references, and citations to include previously omitted sources.
- Revised several key sections to clarify the methodology and contributions, ensuring a more objective and rigorous presentation of our findings.

We once again thank the area chairs and reviewers for their valuable suggestions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This is a nice paper, introducing an excellent dataset for generating game videos that will likely be of much use, and a diffusion-based method for controllable game video generation. The dataset is perhaps the biggest contribution, but the method, while not super novel, is also a contribution. Potential concerns includes that the paper is a little short on technical details (but that has improved, and the code will anyway be open-sourced) and that the authors seem to overselling their contribution ("unique", "pioneering"... just stick to the facts and people will take you more seriously). The concerns are not serious enough to prevent acceptance of the paper.