Unbounded: A Generative Infinite Game of Character Life Simulation

Authors: Jialu Li, Yuanzhen Li, Neal Wadhwa, Yael Pritch, David E. Jacobs, Michael Rubinstein, Mohit Bansal, Nataniel Ruiz
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Text-to-Image Generation, Interactive Image Generation

Abstract:
We introduce the concept of a generative infinite game, a video game that transcends the traditional boundaries of finite, hard-coded systems by using generative models. Inspired by James P. Carse's distinction between finite and infinite games, we leverage recent advances in generative AI to create Unbounded: a game of character life simulation that is fully encapsulated in generative models. Specifically, Unbounded draws inspiration from sandbox life simulations and allows you to interact with your autonomous virtual character in a virtual world by feeding, playing with and guiding it - with open-ended mechanics generated by an LLM, some of which can be emergent. In order to develop Unbounded, we propose technical innovations in both the LLM and visual generation domains. Specifically, we present: (1) a specialized, distilled large language model (LLM) that dynamically generates game mechanics, narratives, and character interactions in real-time, and (2) a new dynamic regional image prompt Adapter (IP-Adapter) for vision models that ensures consistent yet flexible visual generation of a character across multiple environments. We evaluate our system through both qualitative and quantitative analysis, showing significant improvements in character life simulation, user instruction following, narrative coherence, and visual consistency for both characters and the environments compared to traditional related approaches.

Decision: accept (poster)
Reviewer scores: 6, 6

Reviews and discussion:
<thread>
creation date:
1730676844815

thread_type:
official_review

<thread_object>
creator: reviewer_nnkj

<content>
summary:
This work combines various aspects of Generative AI to develop a LLM-based game engine. The design principles refer to open-world, continuous interaction games with limited simultaneous characters but significant opportunities for situation generation and interaction. Capitalizing upon recent research in the field, the system introduces a number of technical innovations both in terms of architecture (in the form of a dynamic regional prompt adapter) and in term of implementation (distillation of Small LLM). With an existing implementation, the paper includes elements of comparative evaluation,  both quantitative and qualitative.

strengths:
This is an ambitious piece of work, tackling difficult technical issues in balancing generation and interaction, as well as various training and fine-tuning protocols. It appears to have innovated on a number of aspects, for instance the dynamic mask to balance character and environment generation. It gives some interesting, detailed, insights such as those of Figure 4. The improvement over IP-Adapter thus appears more than incremental. Another strong points is the Small LLM distillation framework.
The comparative LLM evaluation has staged an appropriate number of LLM, capturing size and diversity. 
Finally, the paper demonstrates a good command of recent relevant work and compares its approach to really recent alternative methods (2023, 2024), which is welcome.
Having considered the authors' rebuttal and additional data, I have raised my score for contribution to 3 ('good').

weaknesses:
From a fundamental perspective, there is a lack of awareness of some game design issues. The introduction is relatively naïve considering how these issues are addressed in game/play theory (Caillois, Huizinga). With the fine line between simulation and entertainment, and notwithstanding the reference to The Sims and Tamagotchi, contemporary audiences tend to have gameplay expectations that can only be met with sustained and meaningful (in the narrative sense) interactions. This feeds into the overall perspective of the evaluation methods which in the paper are primarily focused individual ‘event’ generation. While sustained UX evaluation may be at odds with the system’s current level of maturity, some of this could have been addressed through even slightly longer exchanges or chains of events construed as minimal narrative experience (which is claimed in the abstract but does not really materialize in the examples). The small examples given, including on Figure 1, do not convey a very strong sense of gameplay. 

There is no evidence that the number of user-simulator examples is at all sufficient for the intended purpose of multi-topic data collection. Nor is there any discussion of some hierarchical organization of topics that would come close to scenarios, game genres or elementary forms of plot backbones.

Despite the statement on replicability, the level of implementation details is unclear. Implementation details are rather cursory and uneven with some in-depth details but no systematic discussion, and minor duplications from other sections.

Evaluation has both positive (see above) and negative aspects.
Unless I am mistaken, Table I shows a couple of inconsistencies for results.
On Environment Consistency, how can 0.322 be the best score when it is in-between 0.381 and 0.257? How can 0.675 be the best score in-between 0.595 and 0.832?
It is difficult to find the Qualitative component of evaluation very convincing in the absence of a more systematic approach for generating test situations. From a similar perspective, the example Prompts do not contain advanced mechanisms for controlling response, or specific examples. One would thus wonder how LLM response can be kept consistent across runs, and what type of statistical sampling has been associated to LLM use?

questions:
To which extent are results from Figure 6 consistent with results from Table I?
How have you ensured the quality and fairness (for evaluation) of the text prompts sample?
How can the current approach maintain narrative consistency over a number of interactions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690662044

thread_type:
official_review

<thread_object>
creator: reviewer_4ebp

<content>
summary:
The authors present "UNBOUNDED" a virtual pet game based on LLMs. The authors present two novel technical elements to support this game: a regional IP-Adapter for character-environment consistency and a domain-specific distillation approach for finetuning a smaller LLM to allow for real time gameplay. The authors then present experiments comparing their approach to baselines in terms of visual quality and consistency, and overall game quality as scored by GPT-4o.

strengths:
The main strengths of the paper are (1) the originality of entirely basing a virtual pet game on LLMs, (2) the significance of the two primary technical contributions to figure coherent image generation and real time LLM application uses, and (3) the quality of the image portion of the evaluation/experiments. The paper is also overall well-written, especially in terms of the technical aspects. These primarily justify the positive aspects of my above scores.

weaknesses:
The paper has two major weaknesses in its present draft, the way it positions itself in terms of prior work and the evaluation. 

### Prior Work

The authors do not appear to have engaged with the field of technical games research at all. This is a shame, as their game can be understood as an AI-based game [1] or specifically a NN game [2]. More broadly, the authors' work in terms of generation would fit into the Procedural Content Generation paradigm [3]. There is a 3+ decades long history of work in this space that is relevant to the authors' work [4], with significant recent work at the intersection of LLMs and game generation [5], and it is crucial that the authors update the introduction and related work to situate their work within this field.

### Evaluation

The authors primarily evaluate their work in terms of image quality and I have no major concern with these results (though the DreamSim results seem to incorrectly attribute the best score to their system). However, I have major concerns with the "Evaluation of LLM Generations", which is functionally the closest thing presented to a holistic evaluation of UNBOUNDED. Firstly, the norm within the area would be to make use of a user study for evaluation purposes [6,7]. Particularly since the authors make such an effort to allow for real time control, this would seem to follow naturally. Secondly, the evaluation is somewhat concerning due to the training process. Specifically, since the distillation approach makes use of GPT-4 to simulate user interaction data and the authors sample for uniqueness/variety, its fairly likely that the newly sampled user interaction data approximations will be very similar to these. Further, the largest model the authors compare to is again a GPT-4 variant. Given that GPT-4 is being used for both generation of data, as the system, and as the evaluation, there's a clear risk for bias. This could be addressed by any number of ways, including automated metrics for measuring game quality developed in prior game generation-like work [8,9]. But at present, it is difficult as an outside reader to get a sense of the actual quality of UNBOUNDED as a game. Failing this, it might be beneficial to include longer examples of gameplay in the supplementary materials at least.

### Figures

While these are not of major concern, I wanted to note two aspects of figures that I felt could be improved. First, the authors make use of a large number of fonts in their figures, and their primary font is a bit difficult to read. I'd suggest updating the figures to use a single, more legible font. Second, Figure 4 is largely superfluous. If the authors wish to retain it, it might be a better fit for an appendix.

1. Treanor, M., Zook, A., Eladhari, M. P., Togelius, J., Smith, G., Cook, M., ... & Smith, A. (2015). AI-based game design patterns.
2. Zhu, J., Villareale, J., Javvaji, N., Risi, S., Löwe, M., Weigelt, R., & Harteveld, C. (2021, May). Player-AI interaction: What neural network games reveal about AI as play. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-17).
3. Shaker, N., Togelius, J., & Nelson, M. J. (2016). Procedural content generation in games.
4. Pell, B. (1992). METAGAME: A new challenge for games and learning.
5. Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., Togelius, J., & Yannakakis, G. N. (2024). Large language models and games: A survey and roadmap. arXiv preprint arXiv:2402.18659.
6. Anjum, A., Li, Y., Law, N., Charity, M., & Togelius, J. (2024, May). The Ink Splotch Effect: A case study on ChatGPT as a co-creative game designer. In Proceedings of the 19th International Conference on the Foundations of Digital Games (pp. 1-15).
7. Guzdial, M., & Riedl, M. O. (2021). Conceptual game expansion. IEEE Transactions on Games, 14(1), 93-106.
8. Khalifa, A., Green, M. C., Perez-Liebana, D., & Togelius, J. (2017, August). General video game rule generation. In 2017 IEEE Conference on Computational Intelligence and Games (CIG) (pp. 170-177). IEEE.
9. Guzdial, M., & Riedl, M. (2018, September). Automated game design via conceptual expansion. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (Vol. 14, No. 1, pp. 31-37).

questions:
1. How do the authors position their work in the field of technical game design? 
2. What do longer gameplay examples look like? 
3. How do the authors respond to my concerns around their "Evaluation of LLM Generations"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732248416528

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all the reviewers for their thoughtful feedback. We’re glad that they acknowledge our originality of entirely basing a virtual pet game on LLMs (reviewer 4EBp), and acknowledge our two main technical contributions in coherent image generation and real-time LLM application uses (reviewer 4EBp, Nnkj), and recognize that our paper has good image generation evaluations (reviewer 4EBp) and compares to really recent alternative methods (reviewer Nnkj). We address their questions below and have updated the submission paper accordingly, and hope for a re-evaluation of the paper score if their concerns have been addressed.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an interesting experiment in creating a generative game where the core "game engine" is based on LLMs and diffusion models. While obviously there is a long way to go, there are interesting things to learn from the paper. The "core technical" contribution of the paper is perhaps the regional IP-adapter used to ensure that placement of characters in images work, but the bigger contribution is the system as a whole. It would have been nice with a proper user study to understand how people play with the system, but... next paper?
The first version of this paper was seriously lacking in its related work section, essentially ignoring the most relevant research community, but this has been much improved in the updated version.