Unbounded: A Generative Infinite Game of Character Life Simulation
Authors: Jialu Li
Yuanzhen Li
Neal Wadhwa
Yael Pritch
David E. Jacobs
Michael Rubinstein
Mohit Bansal
Nataniel Ruiz
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-Image Generation
Interactive Image Generation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We introduce the concept of a generative infinite game, a video game that transcends the traditional boundaries of finite, hard-coded systems by using generative models. Inspired by James P. Carse's distinction between finite and infinite games, we leverage recent advances in generative AI to create Unbounded: a game of character life simulation that is fully encapsulated in generative models. Specifically, Unbounded draws inspiration from sandbox life simulations and allows you to interact with your autonomous virtual character in a virtual world by feeding, playing with and guiding it - with open-ended mechanics generated by an LLM, some of which can be emergent. In order to develop Unbounded, we propose technical innovations in both the LLM and visual generation domains. Specifically, we present: (1) a specialized, distilled large language model (LLM) that dynamically generates game mechanics, narratives, and character interactions in real-time, and (2) a new dynamic regional image prompt Adapter (IP-Adapter) for vision models that ensures consistent yet flexible visual generation of a character across multiple environments. We evaluate our system through both qualitative and quantitative analysis, showing significant improvements in character life simulation, user instruction following, narrative coherence, and visual consistency for both characters and the environments compared to traditional related approaches.

Published as a conference paper at ICLR 2025
UNBOUNDED: A GENERATIVE INFINITE GAME OF CHARACTER LIFE SIMULATION
Jialu Li
Yuanzhen Li' Neal Wadhwa' Yael Pritch
David E. Jacobs'
Michael Rubinstein' Mohit Bansal Nataniel Ruiz
'Google 2he University of North Carolina at Chapel Hill
Archibus the Wizerd gives foecback students, pointing out thair strangit's and areas for improvement. He getting hungry.
Archibus eots plate of pears sitting on the flowery meadows. He contant and full but feels sigghtly bored.
Book to the cleesroom Archibus the Wizard opens up dusty book. causingg the room to hum with arcane energy. What next?
go the flowery meedows and forage for peers.
Archibus, let's heed back to the clessroom to studty arcane magic.
Archibus let's go sleep. it's gatting lato.
An example of UNBOUNDED. We follow the life of Archibus, the user's custom wizard character. The user can interact with the generative game using natural language, and Archibus' hunger, energy and fun meters update accordingly. A spontaneous and unconstrained story unfolds while the user playing, and the character can explore new environments with myriad of possible actions and unexpected interactions. The game runs in interactive speeds, refreshing every second.
Figure 1:
ABSTRACT
We introduce the concept generative infinite game, video game that scends the traditional boundaries of finite, hard-coded systems by using genera- tive models. Inspired by James Carse's distinction between finite and infinite games (Carse 1986), we leverage recent advances in generative Al to create UN- BOUNDED: a game of character life simulation that is fully encapsulated in gen- erative models. Specifically, UNBOUNDED draws inspiration from sandbox life simulations and allows you to interact with your autonomous virtual character virtual world by feeding, playing with and guiding it with open-ended mechan- ics generated by an LLM, some of which can be emergent. In order to develop UNBOUNDED, propose technical innovations in both the L.LM and visual gen- eration domains. Specifically, we present: (1) a specialized, distilled large lan- guage model (LLM) that dynamically generates game mechanics, narratives. and character interactions in real-time, and (2) a new dynamic regional image prompt Adapter (IP-Adapter) for vision models that ensures consistent yet flexible visual generation of a character across multiple environments. We evaluate our system through both qualitative and quantitative analysis, showing significant improve- ments in character life simulation, user instruction following, narrative coherence, and visual consistency for both characters and the environments compared to tra- ditional related approaches.
Work done during internship Google

---

Published as a conference paper at ICLR 2025
INTRODUCTION
In his work "Finite and Infinite Games: A Vision of Life as Play and Possibility" Carse 1986,, James P. Carse introduces distinction between two types of games. Carse defines finite games as those "played for the purpose of winning," with boundaries, fixed rules, and definitive endpoint. In contrast, infinite games are "played for the purpose of continuing the play." with no fixed bound- aries and evolving rules. Traditional video games are, inherently, finite games due to the limitations of computer programming and computer graphics. For example, all game mechanics have to be fully pre-defined in the programming language and all graphics assets have to be pre-designed (modulo procedural generation which still grapples with structural limitations). This allows for only finite, and sometimes predefined, set of actions and paths that can be taken. They also feature predefined rules, boundaries, and win conditions.
Recent advances in generative models have been impressive. We hypothesize that these develop- ments have finally opened up the possibility of creating the first generative infinite video game. Two advancements have made this achievable: (1) large language models (LLMs) that can encode persistent video game mechanics (e.g. interactions with the game environment or characters, char- acter state tracking, object permanence), generate interactive stories, and produce spontaneous, and sometimes emergent, behaviors; and (2) visual generative models capable of producing high-quality images that follow prompts. In this work, we present UNBOUNDED, what we believe to be the first interactive generative infinite game, where game behaviors and outputs are generated by AI models, transcending the constraints of hard-coded systems. UNBOUNDED draws inspiration from sandbox life simulations and digital pet games such as Little Computer People, The Sims and Tamagotchi. incorporates elements from tabletop roleplaying games like Dungeons & Dragons, which offer unconstrained storytelling experiences that have been unattainable in video games.
Previous work in game generation includes fluidic games merging gameplay with design Gaudl et al. 20188), automated game design (Cook al.. 2016), and Al-focused games (Treanor al. (0155). These emphasize exploration within predefined game parameters, unlike our vision of generative infinite games, which evolve beyond fixed structures with real-time content and mechanics genera- tion. Other efforts, like Agarwal (2024). (Sun et al. (2023), and (Liapis et al.) 2019),, generate parts of games, but not all components. In contrast, in our work, all of the game mechanics, characters, environments, narrative, and graphics are fully produced by generative models. This is more similar in spirit recent work Genie (Bruce et al.) 2024) and GameNGen (Valevski et al., 2024), although in contrast to these works that mainly generate platformers with diffuse mechanics or regenerate be- haviors of one pre-existing game, our work proposes an open-ended narrative experience with stable game mechanics enabled by an LLM-based game engine.
UNBOUNDED offers a gameplay loop centered around character simulation and open-ended interac- tion (as shown in Figure II in Appendix). Players can insert their characters into the game, defining their appearance and personality. The game generates a world where these characters can explore environments, interact with objects, and engage in conversations. The game generates scenar- ios, stories, and challenges based on the player's actions and choices, creating a personalized and infinite gaming experience.
Specifically, UNBOUNDED has the following capabilities: (1) Character Personalization: players can insert their characters into the game, defining their appearance and personality. (2) Game En- vironment Generation: UNBOUNDED generates a persistent world that the characters can explore and interact. (3) Open-Ended Interaction: Players can interact with the character using natural lan- guage instructions, and there are no pre-defined rules to constraint the interaction. (4) Real-Time Generation: we pay special attention the speed of the game and achieve 5-10x speedups over a naive implementation, serving each new scene with a latency of about one second.
Our approach introduces technical innovations in both LLM and vision generation domains. On the language side, we developed a LLM based game engine capable of maintaining consistent game mechanics, generating coherent narratives, and producing contextual character responses in real-time. Our distilled specialized model is fine-tuned on data automatically generated with two collaborative strong LLM agents, without the need for human annotation in the loop. Our distilled LLM model handles the dynamic generation of game rules and scenarios, adapting to player input and game state. In visual generation, we introduce a new regional Ad-ater which allows for the consistent generation of characters and environments while maintaining visual coherence across
2

---

Published as a conference paper at ICLR 2025
multiple images. Specifically, regional IP-Adapter conditions the image generation the game environment and character appearance encoded modulated by a dynamic mask obtained from at- tention outputs in cross-attention layers. This is in order to mitigate the interference between the environment and character, in order to have both reliably appear in the scene. This approach enables real-time image generation that reflects the game state and player actions.
The contributions of this work are conceptual and technical. We introduce the notion of gener- ative infinite game, demonstrating its feasibility and potential impact on the future of interactive entertainment. We present new paradigm for game design where the game logic and content are encapsulated within generative models. Our main technical contributions include the specialized distilled LLM for game logic and narrative generation and the regional AP-Adepeer for consistent visual generation. We demonstrate the effectiveness of our regional IP-Adapter through both quan- titative and qualitative evaluations, surpassing state-of-the-art in both character and environment consistency. Furthermore, we show that our distilled L.LM performs comparably to very large LLM while having interactive speed. These advancements enable the creation of UNBOUNDED and lay the groundwork for future research and development in the field of Al-driven interactive experiences.
RELATED WORK
Controllable Text-to-Image Generation. Controllable text-to-image generation becomes a key rescarch direction in diffusion model applications, enabling diverse ways to guide the generation process. For instance, ControlNet (Zhang et al., (023) introduces conditioning mechanisms that utilize control signals such depth maps, poses, edges, and segmentation maps to guide image generation. Other works focus on layout control using bounding boxes to control object place- ment within the generated images (Li et al. 2023 SShin et al.] (022). Beyond these control signals, another major area of research involves personalization, where the goal is to generate consistent characters (Ruiz al. 2023; Gal al. 2022] Kumari al., (023) consistent face identities (Li iet al. 2024b) Wang et al. 2024d] Yan et al. [2023] Ruiz et al. 2024 Jeong et al. 2023) across mul- tiple generations. However, most existing approaches lack support for conditioning both characters
and environments separately (Tewel et al. 2024. Gal et al. 2024| Avrahami et al. [2024] Wei et al.,
2023 Liu et al., 2024|] Tao et al. [0255] He et al. 2024, Zhou et al. [024d]] Wang et al. 2024e C
Shentu et al. 2024; Wang et al.
2024d). Others may require predetined masks or layouts for gen-
erating characters, with the environment remaining identical to the input image (Chen et al. 2024 Lugmayr et al. 2022 Yang et al. (023). This limitation makes it difficult to seamlessly integrate characters into different environments while ensuring both consistency and alignment with the input prompts. IP-Atapter Ye et al.) (203)) tackles this task by conditioning the generation the envi- ronment and character images. However, IP-Adapter tends to over reconstruct the conditions, which causes interference between them. In this paper, build our approach on IP-Adapter and propose an improved regional IP-Adapter with block drop, separating character and environment generation enhance consistency.
Game Generation. In the field of Procedural Content Generation (PCG), diverse approaches have been explored to create dynamic game content (Summerville et al. 2018 Shaker et al.) 2016) or
game rules Khalifa et al. (2017). including concept maps (Treanor et al. 2012). conceptual ex- pansion (Guzdial & RiedI, 2021] 2018), Markov Chains (Snodgrass & Ontañón 2014). Bayes Nets (Guzdial & Riedl 2016). and LSTMs (Summerville & Mateas 2016). Recent research has advanced to using GANs for generating game levels or dynamic environments (Volz et al.) 2018 Kumaran al. 2019; Schubert al. 202T; Kim et al., 2020), diffusion models for game genera- tion (Zhou et al. 2024b] Sun et al. 2023), 1,23.11 L.LMs to design and generate the game environments or
mechanics (Sudhakaran et al. 2024 Todd et al. [023] Nasir & Togelius 2023 Hu et al. 2024 Zala
et al. 2024 Anjum et al. 2024]] Chung & Kreminski 2024 2024: Chung al. 2024). While Al typically aids in generating individual game components in previous works, our work aims to fully generate all game behaviors, including graphics, characters, environments, and narrative, using generative
models.
Bruce al. (202)) synthesize interactive games using video diffusion model based previous frames and user actions, but their approach doesn't 't involve infinite games. We argue that generating compelling infinite games by modeling pixels alone is challenging, so we use language models for
3

---

Published as a conference paper at ICLR 2025
open-ended game mechanics. Valevski et al. (2024) train a diffusion model to be a game engine running Doom, but it focuses on a single, finite game that already exists.
Besides generating the game content, there has been exploration in letting Al take different roles
(c.g., competitor, designer, or teammate) in games (Treanor et al.) 2015a Zhu et al. 2021] Gallotta
et al.
2024 Pell 1992 Agarwal et al.) 2023), and system for automated game design
grids Another line focuses on generating narrative text games or using Al tools to generate
[Cook] 2022),
or act games (Ammanabrolu et al. 2020 Li al., 20244a] Latitude Inc. 2023 Wang et al. 2023a
Zhu et al., 2023, Kreminski et al. 2020 Cui et al., 2023 Zhou et al., 2023 Dambekodi et al.
2020). Our work differs by (I) generating all game mechanics, graphics, characters, environments,
and narrative, (2) adding personalization with custom characters and story ares, and (3) achieving
real-time interactivity through innovations in vision and language.
Large Language Models in Image Generation. Large language models have demonstrated strong in-context learning capability (Brown) 2020), which enables them to solve diverse customized tasks based on human instructions and in-context examples. In the field of image generation, large language models have been employed for various tasks, such as image layout generator based on prompts (Cheng al. 2 b). interactive multi-turn image generation (Zeqiang et al. 2023] Huang et al. 2024 Gong et al. 2023 Wang et al., (023b), and interleaved text and image gen- erator (Team 2024 Zhou al. 2024a). Unlike these applications, work focuses distilling specialized LLM, serving as game engine, responsible for generating game mechanics, narratives and character interactions.
METHOD
We introduce UNBOUNDED, an interactive generative infinite game powered by text-to-image gen- eration models and large language models. UNBOUNDED offers: (1) Custom Character Person- alization: users create unique characters with customizable appearances and personalities; (2) Dy- namic World Creation: the system generates a persistent, interactive game world for exploration; (3) Open-Ended Interaction and Gameplay: players interact with their characters via natural lan- guage, with the game dynamically generating new scenarios and storylines based on player actions; and (4) Generation in Interactive Speed: the game runs with near real-time interactivity, achieving a refresh rate close to one second. We detail the methods enabling these capabilities in this section.
3.1 PERSONALIZATION OF LATENT CONSISTENCY MODELS FOR CHARACTER CONSISTENCY
A key feature of UNBOUNDED is its ability to serve fully generative model-based game with real- time interaction. This is made possible through the use of latent consistency models (LCM) (Luo) ct al. 2023) which allow for high-resolution image generation with as few two diffusion steps. By utilizing LCMs, UNBOUNDED achieves real-time text-to-image (T2I) generation, critical for delivering interactive gaming experience with refresh rate close to second.
To support the use of custom characters in the game, we incorporate DreamBooth (Ruiz et al., 2023) into the T2I model. Given set of character images, fine-tune the diffusion model using LoRA modules (Hu et al.; 2021). During fine-tuning, we append a unique identifier, "[V]", which has a weak prior in the model, denote the subject. DreamBooth personalization performed on the base diffusion model, and we merge the subject-specific LoRA with the LCM LoRA trained for few-step diffusion. This simple arithmetic LoRA merging works surprisingly well in maintaining both inference speed and subject preservation. Other alternatives for personalization exist, and many do not need setup time, yet we find that they often fail in strongly preserving the character's 's features, which is a critical component in order to have a satisfactory experience in this type of game.
3.2 REGIONAL IP-ADAPTER WITH BLOCK DROP FOR ENVIRONMENT CONSISTENCY
Another key feature of UNBOUNDED is generating the character in pre-defined environments per- forming different actions based on user instructions. Thus, maintaining both character and environ- ment consistency is essential. While character consistency is handled as discussed in Sec. Sec. two additional challenges arise: ensuring the environment consistency across different generations and
4

---

Published as a conference paper at ICLR 2025
(a) Overall Framework
(b) Dynamic Mask
LoRA
Dreamboth LoRA
[V] witch
Regional IP-Adapter
K
Rank and Mask
Esnaath the decort rky, [V] witch coRnec..
Regional IP-Atteer
Character IP-Adapter
Block Drop
Text Cross Attention
Down biocks: Drop
Middle bincks: Keep
Up blocks:
IP.Adapter
Beneath the decort oby. [V] witck couoes...
Dynamic Mask
Figure 2: (a) Our overall image generation method. We achieve real-time image generation with LCM LoRA, maintain character consistency with DreamBooth LoRAs, and introduce a regional IP-Adapter (shown in (c)) for improved environment and character consistency. (b) Our proposed dynamic mask genreation separating the environment and character conditioning, preventing inter- ference between the two.
accurately placing the character within the environment, without losing alignment with text prompts. We find that existing method fails to consistently perform well for all requirements in interactive speed. As one of our main technical contributions we propose novel regional IP-Adapter order to consistently implant character in pre-defined environments following text prompt.
3.2.1 REGIONAL ID-ATEREER
We propose an improved version of IP-Adapters (Ye et al. 2023) that enables dual-conditioning on both subjects and environments, allowing for the generation of pre-defined character in a user- specified environment. Unlike the original IP-Adapters, which focus on single-image conditioning, our approach introduces condiilnniii and dynamic regional injection mechanism to represent both concepts simultancously the generated images.
Let us start with an example. As shown in Figure 2 given the text prompt "Beneath the desert sky, [V] witch coaxes the cacti blossom with vibrant. glowing flowers" and the desert environment image, the model needs to know that the character should be generated beside cacti and within the desert environment. This requires the model to correctly (1) preserve the environment (2) preserve the character (3) follow the prompt. Utilizing IP-Adapter to encode the environment greatly harms both (2) and (3) (Figure 4 Our regional IP-Adepter solves this by implementing a novel atten- tion separation mechanism for generating the two elements. Specifically, mask-based approach that leverages cross-attentions between the character text embedding and the hidden states each layer of the model. As shown in Figure our approach applies the adapter to the regions corresponding to the environment and character separately, preventing the environment conditioning to interfere with the character's appearance, and vice versa. At each cross-altention layer, we calculate the attention map between a pre-defined character text embedding Ke and the output hidden states from the text cross- cross-attention layer O₃:
introduce dynamic
4, WgOr W
Vd
where W,, and We are projection weights adopted from the text cross- attennin layers. The dynamic mask for the regional IP-Adapter is defined

---

Published as a conference paper at ICLR 2025
{
A threshold A. threshold
M,
(2)
We rank the attention scores Ag and set the threshold at top r%, dynamically updating the mask at each attention layer throughout the diffusion process, The output of the cross-attention blocks is calculated as:
O = O, + ae Me Og ac(1 Mc) O.
where O₂ and O₂ represent the outputs of the IP-Adapter image rross-attention layers of environ- ment and character, and Og and o, are the IP-Adapter scales to adjust the strength of the environment and character conditioning respectively. Our regional IP-Adapter with dynamic masks allows the model generate the character without the interference from the environment conditioning, greatly enhancing character consistency preservation (Figure 4 Besides, preserves environment consis- tency by generating the background around the character conditioning on the environment image, while maintaining the implicit layout information in the hidden states from the text cross-attention layers O₁, ensuring the character is placed accurately following the text prompt.
3.2.2 BLOCK- WISE DROP ON ENVIRONMENT CONDITIONING
dynamic mask from the cross-attention between character text and hidden states. This mask's quality is key for separating character and environment generation. Figure 12 Appendix shows attention maps between character embeddings and hidden states in cross-attention layers of down sample blocks. We observe that the attention doesn't 'tt focus on the character but spreads across the full image for these blocks. We take this as strong indication that the diffusion model doesn't 't separate character and environment generation in these layers and instead focuses on overall image structure based on text prompts. This aligns with Wang et al. (2024))'s 's finding that down sample blocks capture spatial layouts more, while up sample blocks capture style. We drop the regional IP- IP-Ateteer in down sample blocks, using it only in mid and up sample blocks. This allows for better spatial layout generation between character and environment. Additionally, we find that adding dynamic regional Adaperss in up sample blocks more strongly aligns the generated background with the conditioning environment by extracting relevant semantic information while preserving seuracter-speciffe details like appearance and poses.
For our regional Adapter,
3.3 LANGUAGE MODEL GAME ENGINE WITH OPEN ENDED INTERACTIONS AND INTEGRATED GAME MECHANICS
UNBOUNDED simulates character actions in pre-defined environments with images generated based on scene text descriptions while monitoring the character's state and providing the user with natural language interaction with the character and game environment. For example, the user can take the character to different environments, interact with the character (c.g. "I pet pet the character on its head"), essentially take any open ended action e.g. pet pet the character and then take them for a rocket ride at the space station.". Since the game is ultimately built on a language model, these expansive capabilities present several challenges: (1) Environment Binding: the model needs place the character in the correct environment based on the natural language instructions. (2) Coherent Story Generation: the model generates coherent narrative descriptions and character responses that align with user-specifted character traits. (3) Game Mechanics Generation: the model needs to monitor the state of the character (i.e., hunger, energy, fun, hygiene). and update them based on user interactions and story events. (4) Prompi Rewriting: the model needs to rewrite the narratives for the diffusion model (i.e., append special token "[V]" for the character, align environment descriptions to the pre-generated environments for better environment consistency).
Surprisingly, we find that a very large language model (e.g., GPT-4, GPT-4o) with detailed instruc- tions and using in-context learning (Brown, 2020) can exhibit these capabilities. Nevertheless, using such large models as game engines is not directly feasible due to the large latency (e.g.. 5 seconds response). Given this, we propose distill these capabilities from
for a 7B model to give very large model into a smaller model based on Gemma-28 (Team et al., 2024) for game logic and narrative generation that supports real-time interaction. In this section propose two key technical

---

Published as a conference paper at ICLR 2025
contributions (1) design for a character life simulation game using two very large language models that control for world modeling and user interaction respectively (2) a framework for distilling this knowledge into smaller language model that is fast enough to achieve interactive speeds.
3.3.1 CHARACTER LIFE SIMULATION WITH MULTI-LLM COLLABORATION
We build a character life simulation game with two LLMs agents. One agent serves the world simulation model, responsible for setting up game environments, generating narratives and image descriptions, tracking character states and simulating the character's behavior. The second agent functions as a user model, simulating the player's 's interactions with the world simulation model. has three types of interactions: continuing the story within the current environment, moving the character into different environments, or interacting with the character to maintain the healthy state of the character. In each interaction category, the user has the option to provide personality details of the character or guide the character actions that, in turn, guide the simulator's narrative genera- tion. This interaction between the world simulation LLM and the user LLM allows for dynamic character life simulation game with virtually unlimited interaction possibilities and narrative paths.
3.3.2 FRAMEWORK FOR SMALL LLM DISTILLATION
We propose a framework for distilling the capabilities of larger LLMs into the smaller, more efficient model using synthetic data generated by the multiple stronger LLMs. Our framework contains two steps: (1) automated data collection and (2) small model distillation.
Automated Data Collection Our goal is to build general-purpose character life simulator pable of understanding a wide range of character traits and generating games across diverse topics. To achieve this, the first step is to gather a diverse dataset of topics and characters. We prompt large LLM to generate pairs of topics and corresponding main characters. To ensure data diversity, we retain only the generated pairs whose ROUGE-L similarity to existing data is below 0.7, follow- ing ( (Wang et al. 2022). which demonstrated the importance of diverse data for enhancing an LLM's ability to follow instructions. This process results in 5,000 unique topie-character pairs, which serve as the basis for user-simulator interaction data.
In the second step, we collect multi-round interaction data between the world simulation LLM and the user LLM. The process begins with the world simulation LLM setting up the game environment and initiating a character action based on a randomly sampled topic-character pair from the dataset. The user LLM is then prompted to provide interaction inputs, while the world simulation LLM generates updated character actions, states, and responses. This iterative process continues for five interaction rounds per session, resulting in total of 5,000 user-simulator interaction examples. All the prompt templates are Appendix.
Distillation Once the interaction data has been collected, we fine-tune the smaller Gemma-2B model using the 5,000 synthetic user-simulator interaction samples. During supervised fine-tuning, we mask out the loss on user input data, focusing the optimization on learning the world simulation model's 's behavior based on multi-round interaction history and current user input. This approach enables Gemma-2B to replicate the capabilities of larger L.MMs as a game engine while supporting real-time interaction. Our distilled 2B demonstrates performance comparable to GPT-4o, effectively following user input and supporting unbounded interactions.
EXPERIMENTAL SETUP
4.1 EVALUATION BENCHMARKS
Evaluation of Image Generations evaluate our image generation approach, we collect an eval- uation dataset consisting of 5,000 (character image, environment description, text prompt) triplets with GPT4o (OpenAl 2023). It includes 5 characters (dog, cat, panda, witch, and wizard), 100 diverse environments, and 1,000 text prompts (10 per environment). We evaluate the image genera- tion performance under three criteria: environment consistency, character consistency and semantic alignment with the text prompt. To measure similarity between images, we employ CLIP-I (Rad-
ford et al. 2021), DINO (Caron et al.] 202T). and DreamSim (Fu et al. 2023). We denote the
7

---

for Published as a conference paper at ICLR 2025
Table 1: Comparison of UNBOUNDED and other methods for maintaining environment consistency and character consistency. UNBOUNDED achieves the best performance in maintaining consistency, while maintaining comparable semantic alignment with the text prompt. Best scores are in bold.
Methods
Environment Comiisency
Character Consistency
Semantic Alignment
CLIP- > DINO
DreamSim"
CLIP-1
DINO f
DreamSim"
CLI--T
IP.Adapter al
0.168
IP-Adapter- hearacc
Ron RS 2004
0.334
0.151
0.832
0.246
0.124
0.872
0.098
StoryDiffision ahober at DUSAN
0.528
0.257
0.733
0.629
0.464
0.545
0.242
Ours
0.563
0.322
0.675
0.676
0.470
0.488
0.242
Table 2: Ablations of the effectiveness of dynamic regional IP-Adapter and block drop in our con-
sistent image generation approach.
No.
Block
Regional
Scale
Environment Consistency
Character Consistency
Aligument CLIP-T t
Drop
IP-Adapter
CLIP-1
DINO#
DreamSim
CLIP-I
DINO
DreumSim
1.
x
1.0
0.123
0.111
0.885
0.073
0.024
0.973
0.034
2.
x
1.0
0.414
0,331
0.647
0.337
0.147
0,832
0.149
3.
1.0
0.676
0.470
0.488
4.
x
x
0.5
0.470
0,381
0.595
0.366
0.139
0,832
0.168
5.
0.5
0.332
0.627
0:574
0.575
0.252
6.
0.5
0.549
0.263
0.726
0.705
0.514
0.450
0.216
similarity between environment reference image and the generated images CLIP-I DINO DreamSim and the similarity between the character reference image and the generated images as CLIP-IS, DINO DreamSim Additionally, we use CLIP-T (Radford et al. 2021) to evaluate the semantic alignment with the text prompt. Given that UNBOUNDED is a character life simulation game, ensuring the presence of the character the image is important. Therefore, we further utilize Doindinng-INOOO
Grounding- DINO (Liu et al., (023) to detect the presence of the character in the generated images. We set similarity scores to 0 and distance scores if there is no character the generated image.
Evaluation of LLM Generations We collect an additional evaluation dataset with 100 user- simulator interaction samples using the pipeline in Sec. 3.3 Each user-simulator interaction sample contains five rounds of interaction between the user and the world model. We use GPT-4 (OpenAI, 2023) as a judge, scoring the response between two models (baseline model vs. our model) in over- all score, and then in four aspects: accuracy of character state update, environment relevance, story coherence, and user input instruction following. Scores range from 0 to 10.
4.2 IMPLEMENTATION DETAILS
Our image generator is built on SDXL (Podell al. 2023). We train DreamBooth LoRA of rank 16 with batch size 1 and a constant learning rate le-4 for 500 steps on a single A100, which takes approximately 30 mins. The special token append before the character is "sks". During inference, we merge the LCM-LORA with DreamBooth LoRA with seale 1.0 for each. We use -Adapter-plus-sdx-vii for encoding the environment, and encoding the character. The dynamic mask ratio 7% in set to be 60%.
Our LLM is built on Gemma-2B (Team et al. 2024). We distill the LLM using 5,000 user-simulator interaction samples collected from GPT-4o. We train the LLM for 6,500 steps, with batch size 8, distributed across 4 A000s, and learning rate Ic-4. The learning rate scheduler is set to be cosine annealing (Loshchilov & Hutter 2016), and the warmup steps ratio is 0.03. In evaluation, we use an L.M to generate responses with sampling. The sampling hyperparameters are set be default.
5
RESULTS AND ANALYSIS
5.1 COMPARISON WITH DIFFERENT APPROACHES FOR MAINTAINING ENVIRONMENT CONSISTENCY AND CHARACTER CONSISTENCY
Quantitative Results We compare regional IP-Adapter with block drop with previous ap- proaches in maintaining environment consistency and character consistency. For all the approaches, merge the character LoRA and LCM LoRA with the model support fast inference and im-
8

---

Published as a conference paper at ICLR 2025
Character
Environment
Story Diffusion IP--dopter Ade-ter--IIsseeee
[V] witch raised hends and the twisted trunks unwoundd. their branches
sky, glowing sperke
Environment Consistency
X
Character Consistency
Semantic Alignment
[V] wizerd kneels casting water's surface reflecting myriad lurinescent the cleering.
Environment Consistency
X X3
Character Consistency
Semantic Alignment
Amidst the strange rock formetions. [V] pande finds hidden grove filld with
glowing, otherworidly flora.
Environment Consistency
EXX K
Character Consistency
Semantic Alignment
Comparison with other approaches for generating environment and character consistent images based on text prompts. We observe that our method strongly outperfoms related work.
Figure 3:
prove character consistency and provide apples-to-apples comparison. As shown in Table 1 our approach consistently outperforms previous approach in maintaining environment consistency and character consistency, while achieving comparable performance in maintaining semantic align- ment. Specifically, our approach significantly overtakes Story Diffusion (Zhou et al. 2024c) by 0.047 in CLIP-IC, and 0.057 in DreamSim for character consistency, and 0.035 in CLIP-I 0.065 in D and 0.058 DreamSim F. for environment consistency, demonstrating the effectiveness of our approach. Besides, approach also achieves comparable performance in maintaining mantic alignment, suggesting strong text following capabilities.
Qualitative Results We present qualitative comparison with other approaches in Figure Our regional IP-Adapter with block drop consistently generates images with high character consistency, whereas other methods may fail to include the character generate characters with inconsistent appearances (see Example & 2). Furthermore, we show that our approach balances environment consistency and character consistency well, while other approaches might generate environments that differ from the condition environment (e.g., StoryDiffusion in Example & 3).
5.2 EFFECTIVENESS OF DYNAMIC REGIONAL IP-ADAPTER WITH BLOCK DROP
Quantitative Results We demonstrate that our regional IP-Adapter with block drop is essential for placing the character in the environments following the text prompt, while maintaining both environment and character consistency with ablation studies. As shown in Table [ adding block drop improves both environment and character consistency compared with multi-IP-Adapter (No. 2. vs. No. I.), with an increase of 0.291 in CLIP-I and 0.264 in CLIP-I alongside better alignment between the text prompt and the generated image. Furthermore, character consistency and text alignment while maintaining comparable performance in environment consistency (No. 3 No. 2). We also explore the effect of the environment IP-Adapter scale, and
regional IP-Adapter enhances

---

Published as a conference paper at ICLR 2025
Regional IP Adapter
Character
Environment
No Condition
IP- AdoAdeeerr
+ Block Drop
[V] dog playfully chased tal under the sway of honging lanterns, cobblestone paths sightly vibrating with unseen robot.
[V] cautiously ascends the creaky wooden each groening louder climbs the nerrow; winding
haunted
Figure 4: Comparison between our regional IP-Adapter approach and baseline approaches. Table 3: Comparison of UNBOUNDED and different LLMs on serving as game engines for open ended interactions and integrated game mechanics. We use GPT-4 to provide pairwise scores be- tween our model and other LLMs.
State Update
Environment Relevance
Story Coherence
Instruction Following
Model
Overall
Base
Ours
Base
Ours
Base
Ours
Base
Ours
Base
Ours
Gemma-2R TTeam al.
2024
6.22
7.44
5.60
7.47
6.12
7.94
6.34
7.57
6.43
7.67
Gemma- 7B Teamn al
A124
6.80
7.39
6.29
7.43
7.07
7.91
6.90)
7.48
6.89
7.53
Llama3.2 2-38 MMC
2021
7.21
7.50
6.86
7.38
7.63
7.93
7.36
7.56
7.31
7.67
Ours-
7.68
7.82
7,50
7.74
8.10
7.78
7.93
7,82
7,97
GPT-4o OpenA] 2023
7.76
7.68
7.69
7.66
8.20
8.10
7.95
7.82
7.85
7.82
our findings indicate that using a smaller scale (e.g., 0.5) generally improves character consistency, though it slightly compromises environment consistency (No. 6 vs. No. 3).
Qualitative Results As shown in Figure conditioning on the environment using IP-Adapter achieves good environment reconstruction, but the character consistency is influenced by the en- vironment style. Introducing block drop improves adherence the text prompt, resulting in images with the correct spatial layout for both the character and the environment. However, the character's appearance remains influenced by the surrounding environment. By incorporating our proposed re- gional injection mechanism with our proposed dynamic mask scheme, the generated images achieve strong character consistency while maintaining effective conditioning on the environment.
5.3 EFFECTIVENESS OF DISTILLING SPECIALIZED LARGE LANGUAGE MODEL
We show that diverse user-simulator interaction data effectively distills Gemma-2B into ca- pable game engine. As shown in Table zero-shot inference with small LLMs (i.c., Gemma-2B, Llama3.2-3B), slightly larger LLM (1.e., Gemma-7B) results in lower performance compared to ours, highlighting the importance of distillation from a stronger LLM for game world and character action simulation. Furthermore, we show that our model achieves performance comparable GPT- 4o, validating the effectiveness of our approach. We also investigate the impact of distillation data size on performance by comparing a Gemma-2B model distilled with 1K data and 5K data. Results show that using larger dataset consistently improves performance across all aspects, highlighting the potential for further enhancements with more data to fully match GPT-4o's performance.
CONCLUSION
We introduce UNBOUNDED, an interactive generative infinite game based on generative models. UNBOUNDED is built two main components, specialized, distilled LLM for real-time inter- action, and fast diffusion model with our proposed regional IP-Adapter for consistent generation across multiple scenes. We show that proposed approach allows for interactive game sub- sumed in generative models, with consistent characters, environments and story and an expansive gameplay characteristic of infinite game.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Shiran Zada, Peyman Milanfar, Shlomi Fruchter, Michael Goin and Matthew Guzdial for the thoughtful feedback and discussion.
REFERENCES
Neal Agarwal. Infinite craft.
http://eaal.fuuuiniiiiiitt Saatnnitterraafteeeeeee 2024. URL https: Accessed: 2024-10-29.
//neal. www.innfiineeerrrannioaaoooo
Rohan Agarwal, Zhiyu Lin, and Mark Riedl. A controllable co-creative agent for game system design. arXiv preprint arXiv:2308.0233 02317, 2023.
Prithviraj Ammanabrolu, Wesley Cheung, Dan Tu, William Broniec, and Mark Riedl. Bringing stories alive: Generating interactive fiction worlds. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pp- 3-9, 2020.
Asad Anjum, Yuting Li, Noelle Law, Megan Charity, and Julian Togelius. The ink splotch effect: A case study chatgpt co-creative game designer. In Proceedings of the 19th International Conference on the Foundations of Digital Games, pp. 1-15, 2024.
Omri Avrahami, Amir Hertz, Yael Vinker, Moab Arar, Shlomi Fruchter, Ohad Fried, Daniel Cohen- Or, and Dani Lischinski. The chosen one: Consistent characters in text-to-image diffusion models. In ACM SIGGRAPH 2024 conference papers, pp. 1-12, 2024.
Tom B Brown. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165. 2020.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative inter- active environments. In Forty-first International Conference on Machine Learning, 2024.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
James P. Carse. Finite and Infinite Games: A Vision of Life Play and Possibility. Free Press, 1986.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero- shot object-level image customization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6593-6602, 2024.
Junhao Cheng, Xi Lu, Hanhui Li, Khun Loun Zai, Baiqiao Yin, Yuhao Cheng, Yiqiang Yan, and Xi- aodan Liang. Autostudio: Crafting consistent subjects in multi-turn interactive image generation. arXiv preprint arXiv:2406. 388, 2024a.
Junhao Cheng, Baiqiao Yin, Kaixin Cai, Minbin Huang, Hanhui Li, Yuxin He, Xi Lu, Yue Li, Yifei Li, Yuhao Cheng, et al. Theatergen: Character management with Ilm for consistent multi-turn image generation. arXiv preprint WrXi:22404.1819, 2024b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, March 2023. URL https:
//1 //lmsys.org///
John Joon Young Chung and Max Kreminski. Patchview: LIm-powered worldbuilding with gener- ative dust and magnet visualization. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pp. 1-19, 2024.
John Joon Young Chung, Melissa Roemmele, and Max Kreminski. Toyteller: Toy-playing with character symbols for ai- powpeee visual storytelling. Adjunct Proceedings of the 37th Annual ACM Symposium User Interface Sofware and Technology, pp. 1-5, 2024.
11

---

Published as a conference paper at ICLR 2025
Michael Cook. Puck: slow and personal automated game designer. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 18, pp. 232- 239, 2022.
Michael Cook, Simon Colton, and Jeremy Gow. The angelina videogame design system part IEEE Transactions Computational Intelligence and Al in Games, 9(2):192-200 2016.
Christopher Cui, Xiangyu Peng, and Mark Riedl. Thespian: Multi-character text role- pole-playin game
agents. arXiv preprint arXiv:2308.011 01882, 2023.
Sahith Dambekodi, Spencer Frazier, Prithviraj Ammanabrolu, and Mark O Riedl. Playing text-based games with common sense. arXiv preprint urXiv:2012.0275/ 2020.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.09344 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0100 01611,, 2022.
Rinon Gal, Or Lichter, Elad Richardson, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Lem-lookahead for encoder-based text-to-image personalization. arXiv preprint arXiv:2404.03620, 2024.
Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius, and Georgios N Yannakakis. Large language models and games: A survey and roadmap. arXiv preprint arXiv:2402.18652 2024.
Swen E Gaudl, Mark J Nelson, Simon Colton, Rob Saunders, Edward J Powley, Peter Ivey, Blanca Pérez Ferrer, and Michael Cook. Exploring novel game spaces with fluidic games, arXiv preprint arXiv: 00003 2018.
Yuan Gong, Youxin Pang. Xiaodong Cun, Menghan Xia, Yingqing He, Haoxin Chen, Longyue Wang, Yong Zhang, Xintao Wang, Ying Shan, et al. Talecrafter: Interactive story visualization with multiple characters. arXiv preprint arXiv:2305.1822 2023.
Matthew Guzdial and Mark Riedl. Game level generation from gameplay videos. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 12, pp. 44-50, 2016.
Matthew Guzdial and Mark Riedl. Automated game design via conceptual expansion. In Pro- ceedings of the AAAI Conference Artificial Intelligence and Interactive Digital Entertainment, volume 14, pp. 31-37, 2018.
Matthew Guzdial and Mark O Riedl. Conceptual game expansion. IEEE Transactions on Games, 14(1):93-106, 2021.
Huiguo He, Huan Yang. Zixi Tuo, Yuan Zhou, Qiuyue Wang, Yuhang Zhang, Zeyu Liu, Wenhao Huang, Hongyang Chao, and Jian Yin. Dreamstory: Open-domain story visualization by 11m- guided multi-subject consistent diffusion. arXiv preprint arXiv:2407.1289/ 2024.
Chengpeng Hu, Yunlong Zhao, and Jialin Liu. Game generation via large language models. In 2024 IEEE Conference Games (CoG). pp. 1-4, IEEE, 2024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhte Zhu,, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0940 006685 2021.
Minbin Huang. Yanxin Long, Xinchi Deng, Ruihang Chu, Jiangfeng Xiong. Xiaodan Liang, Hong Cheng, Qinglin Lu, and Wei Liu. Dialoggen: Multi-modal interactive dialogue system for multi- turn text-to-image generation. arXiv preprint arXiv:2403.088. 088557 2024.
12

---

Published as a conference paper at ICLR 2025
Hyeonho Jeong. Gihyun Kwon, and Jong Chul Ye. Zero-shot generation of coherent storybook from plain text story using diffusion models. arXiv preprint arXiv:2302.039000 2023.
Ahmed Khalifa, Michael Cerny Green, Diego Perez- and Julian Togelius. General video game rule generation. In 2017 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1770-77. IEEE, 2017.
Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to simulate dynamic environments with gamegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1231-1240, 2020.
Max Kreminski, Melanie Dickinson, Michael Mateas, and Noah Wardrip-Fruin. Why are we like this?: The ai architecture of co-creative storytelling game. In Proceedings of the 1 15th Interna- tional Conference on the Foundations of Digital Games, pp. 2020.
Vikram Kumaran, Bradford Mott, and James Lester. Generating game levels for multiple distinct games with a common latent space. Proceedings of the AAAI Conference on Artificial Intelli- gence and Interactive Digital Entertainment, volume 15, pp. 102-108, 2019.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-conceptt customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pullern Recognition, pp. 1931-1941, 2023.
Latitude Inc. Ai dungeon, 2023. URL wwtph:tpp////m http:///ddungnnnccoo com/ Accessed: 2024-10-29.
Yoko Li, Paul Phu, Danilowiez Alex, and AP. Ai tamago. https://github.comm///a
AI AI-tamago 2024a.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22511-22521, 2023.
2023.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Pho- tomaker: Customizing realistic human photos via stacked id embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024b.
Antonios Liapis, Georgios N. Yannakakis, Mark J. Nelson, Mike Preuss, and Rafael Bidarra. Or- chestrating game generation. IEEE Transactions Games, 11(1):48 68, 2019. doi: 10.1109/ TTG.2018.277076.
Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, Yanfeng Wang, and Weidi Xic. Intelli- gent grimm-open-ended visual storytelling via latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6190-6200, 2024.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.054: 054499. 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 03983,, 2016.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.
Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEÉE/CVF conference on computer vision and pattern recognition, pp. 11461-11471, 2022.
Simian Luo, Yiqin Tan, Longho Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:231000433777 2023.
Meta.
attps://ai.meta.com Ilama-2 3- connect 2024 1o024-vision--o edge wwwpdeeeeeeeeleeeeeeeeeeeeeeeeee mobile delee-devcess..
Llama3.2,
2024.
URL
Muhammad U Nasir and Julian Togelius. Practical peg through large language models. 2023
IEEE Conference Games (CoG), pp. 1-4. IEEE, 2023.
13

---

Published as a conference paper at ICLR 2025
OpenAl. Openai models, 2023. URL httpttttpppsttooorrrt ope.m.opppaai..
Barney Pell. Metagame: A new challenge for games and learning. 1992.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint wrXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR. 2021.
Ciara Rowles, Shimon Vainer, Dante De Nigris, Slava Elizarov, Konstantin Kutsy, and Simon Donné. Ipadapter-instruet, Resolving ambiguity in image-based conditioning using instruct prompts. arXiv preprint 2XXx:22408003209 2024.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to- text-to-image diffusion models for subject- bobject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500- 22510, 2023.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personaliza- tion of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Viston and Pattern Recognition (CVPR), pp. 6536,, June 2024.
Frederik Schubert, Maren Awiszus, and Bodo Rosenhahn. Toad-gan: a flexible framework for few- shot level generation in token-based games. IEEE Transactions Games, 14(2):284-293, 2021.
Noor Shaker, Julian Togelius, and Mark Nelson. Procedural content generation in games. 2016.
Junjie Shentu, Matthew Watson, and Noura Al Moubayed. Attencraft: Attention-guided disentan- glement of multiple concepts for text-to-image customization. arXiv preprint arXiv:2405. 17965, 2024.
Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot trans- fer. Advances in Neural Information Processing Systems, 35:33754-33767, 2022.
Sam Snodgrass and Santiago Ontañón. Experiments in map generation using markov chains. In FDG, 2014.
Shyam Sudhakaran, Miguel Gonzälez-Duque, Matthias Freiberger, Claire Glanois, Elias Najarro, and Sebastian Risi. Mariogpt: Open-ended text2level generation through large language models. Advances in Neural Information Processing Systems, 36, 2024.
Adam Summerville and Michael Mateas. Super mario as string: Platformer level generation via
Istms, arXiv preprint arXiv:1603. 1003000930, 2016.
Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmgård, Amy K Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. Procedural content generation via machine learning (pegml). IEEE Transactions on Games, 10(3):257-270, 2018.
Yuqian Sun, Zhouyi Li, Ke Fang, Chang Hee Lee, and Ali Asadipour. Language as reality: co- creative storytelling game experience 1001 nights using generative ai. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 19, pp. 425 2023.
Ming Tao, Bing-Kun Kun Bao, Hao Tang, Yaowei Wang, and Changsheng Xu. Storyimager: unified and efficient framework for coherent story visualization and completion. In European Conference on Computer Vision, pp. 479-495. Springer, 2025.
Chameleon Team. Chameleon: Mixed-modaa early-fusion foundation models. arXiv preprint 00588880 2024.
14

---

Published as a conference paper at ICLR 2025
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.08225, 2024.
Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free consistent text-to-image generation. ACM Transactions on Graphics (TOG), 43(4); 1-18, 2024.
Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. In Proceedings of the 18th International Con- ference on the Foundations of Digital Games, pp. 1-8, 2023.
Mike Treanor, Bryan Blackford, Michael Mateas, and Ian Bogost. Game-o-matic: Generating videogames that represent ideas. In Proceedings of the The third workshop on Procedural Content Generation Games, pp. 1-8, 2012.
Mike Treanor, Alexander Zook, Mirjam P Eladhari, Julian Togelius, Gillian Smith, Michael Cook, Tommy Thompson, Brian Magerko, John Levine, and Adam Smith. Ai-based game design pat- terns. 2015a.
Mike Treanor, Alexander Zook, Mirjam Palosaari Eladhari, Julian Togelius, Gillian Smith, Michael Cook, Tommy Thompson, John Levine, and Adam Smith. Ai-based game design patterns. 06 2015b.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint a4Xiv:2408. 14837, 2024.
Vanessa Volz, Jacob Schrum, Jialin Liu, Simon M Lucas, Adam Smith, and Sebastian Risi. Evolv- ing mario levels in the latent space of deep convolutional generative adversarial network. In Proceedings of the genetic and evolutionary computation conference, pp. 221-2221222 2018.
Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.0273. 2024a.
Jiahao Wang, Caixia Yan, Haonan Lin, Weizhan Zhang, Mengmeng Wang, Tieliang Gong, Guang Dai, and Hao Sun. Oneactor: Consistent character generation via cluster-conditioned guidance. arXiv preprint arXhv:2404.10267, 2024b.
Kuan-Chieh Wang, Daniil Ostashev, Yuwei Fang, Sergey Tulyakov, and Kfir Aberman. Moa: Mfxture-of-attention for cootect-context disentanglement personalized image generation. arXiv preprint arXiv:2404.1110
2024c.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprini arXiv:2401.075 2024d.
Ruoyao Wang, Graham Todd, Eric Yuan, Ziang Xiao, Mare-Alexandre Côté, and Peter Jansen. Bytesized32: A corpus and challenge task for generating task-specific world models expressed as text games. arXiv preprint arXiv:2305.1 14879, 2023a.
Wen Wang, Canyu Zhao, Hao Chen, Zhekai Chen, Kecheng Zheng, and Chunhua Shen. Au- tostory: Generating diverse storytelling images with minimal human effort. arXiv preprint arXiv:2311.11243, 2023b.
Xiyu Wang, Yufei Wang, Satoshi Tsutsui, Weisi Lin, Bihan Wen, and Alex Kot. Evolving sto- rytelling: benchmarks and methods for new character customization with diffusion models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3751-3760, 2024e.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint ar 221. 2022.
Yuxiang Wei, Yabo Zhang, Zhilong Ji. Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 15943-15953 2023.
15

---

Published as a conference paper at ICLR 2025
Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, and Bin Fu. Facestudio: Put your face everywhere in seconds. arXiv preprint arXiv:2312.02663, 2023.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18381- 18391, 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308..677 06721,,,,, 2023.
Abhay Zala, Jaemin Cho, Han Lin, Jachong Yoon, and Mohit Bansal. Envgen: Generating and adapting environments via ilms for training embodied agents. arXiv preprint 2XXv:2443.12014 2024.
Lai Zeqiang, Zhu Xizhou, Dai Jifeng. Qiao Yu, and Wang Wenhai. Mini-dalle3; Interactive text to image by prompting large language models. arXiv preprint arXiv:2310.07653, 20233
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image the IEEF/CVE International Conference on Computer Vision,
diffusion models. In Proceedings the
pp. 3836-3847, 2023.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprini arXiv:2408. /039, 2024a.
Hongwei Zhou, Jichen Zhu, Michael Mateas, and Noah Wardrip-Fruin The eyes, the hands and the brain: What can text-to-image models offer for game design and visual creativity? In Proceedings of the 19th International Conference on the Foundations of Digital Games, pp. 1-13, 2024b.
Wei Zhou, Xiangyu Peng, and Mark Riedl. Dialogue shaping: Empowering agents through npc interaction. arXiv preprint vrXh::2307.15833, 2023.
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffu- sion: Consistent self-attention for long-range image and video generation. arXiv preprint arXiv:2405.01434, 2024c.
Zhengguang Zhou, Jing Li, Huaxia Li, Nemo Chen, and Xu Tang. Storymaker: Towards holistic consistent characters in text-to-image generation. arXiv preprint arXiv:2409. 12576, 2024d.
Andrew Zhu, Lara Martin, Andrew Head, and Chris Callison- Burcch. Calypso: Lims as dungeon master's s assistants. In Proceedings of the AAAI Conference on Artificial Intelligence and Interac- five Digital Entertainment, volume 19, pp. 380-390, 2023.
Jichen Zhu, Jennifer Villareale, Nithesh Javvaji, Sebastian Risi, Mathias Löwe, Rush Weigelt, and Casper Harteveld. Player-ai interaction: What neural network games reveal about ai as play. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-17, 2021.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
In this appendix, we present the following:
A qualitative example of our UNBOUNDED in Sec.
Human evaluation results for LLM as game engine Sec.
Human evaluation results for dynamic regional IP-Adapter in Sec.
Evaluation of LLMM as game engine with multiple samples in Sec. D
A pipeline figure for our UNBOUNDED in Figure 11 and the attention figure in UNet when
extracting dynamic mask in Figure I2
Prompts for seer-simulation data collection in Sec. E
Evaluation prompts we use for querying GPT-4 as a judge for LLM evaluation in Sec.
Reproducibility statement in Sec. G
A QUALITATIVE EXAMPLE OF UNBOUNDED
In this section, we include one qualitative example of UNBOUNDED, The example was collected us- ing GPT-4o as the user interacting with our game engine, with no interference or cherry-picking of favorable outcomes. As shown in Figures 8 TO given an input game topic, UNBOUNDED gen- erates relevant and diverse environments aligned with the theme. Besides, when the user interacts with the character, the character responds naturally (e.g., the witch's eyes light up with excitement as she says, "Let's see what this puzzle holds!"). Furthermore, the character's state updates dynam- ically throughout the game. As the game progresses, the character's states-Such hunger, energy, fun, and hygiene- decline, reflecting the character growing hungry, tired, less entertained, or dirty. Users can freely engage with the character to restore these states by feeding them, playing, letting them rest, or encouraging them to bathe (c.g., the user says. "I show the witch a celestial puzzle to solve for extra fun," and the character's fun level increases). Lastly, during gameplay. For instance, when the user stated, "The witch is adventurous and courageous, eager to uncover secrets and delve into the unknown realms that ancient wisdom holds. Please continue the generation accordingly." the game engine autonomously took the witch to explore new environment, interpreting it as the destination connected the ancient tome's hidden passage. This example provides an overview of how the game operates and demonstrates how unbounded interactions can lead to unique and dynamic game outcomes.
observe emergent behaviors
B HUMAN EVALUATION FOR LLM AS GAME ENGINE
In this section, we present hu- man evaluation to assess the perfor- mance of our distilled LLM as game engine. Specifically, we sam- ple 10 user-simulator interaction amples using the pipeline described in Sec. 33 Each interaction con- sists of five rounds of exchanges be- tween the user and the world model. Responses from the world model are generated using both GPT-4o and our distilled small LLM. We compare the responses from the two models across a total of paired examples based on the dialogue history and ask four human annotators to evaluate them across five categories: overall score, state update accuracy, environment relevance, story coherence, and adherence to user input instructions. The user interface employed for collecting human annotations is shown in Figure 5 To eliminate bias, the response order between the two models is randomized, ensuring that annotators are unaware of which model generated given response. Table 4 details the annotators' votes for win, lose, and tie outcomes for UNBOUNDED and GPT-4o. Our distilled small LLM demonstrates performance comparable GPT-4oo across all evaluated categories.
Table 4: Human Evaluation on 50 responses from LLM on game topics. We show the detailed win/lose/tic counts on each evaluation category.
Criteria
Win ( (%)
Lose ( (%)
Tie (%)
Overall Score State Update Environment Relevance
26.5
27.0
46.5
20.5
24.0
55.5
16.0
15.5
68.5
Story Coherence Instruction Following
15.0
18.0
67.0
14.5
14.5
71.0
17

---

Published as a conference paper at ICLR 2025
or dilepee
Enkkatng 1/90
plemel who in uent withens
Cha, doweinng burenbeteen behavir and condering 1e compleeity
and
2-A peopis Jay asookingghle
bigur
y Lowrling
cooeeed wnduttig E377
Oralgue
Cralegue
2
Di 2
Dalogan
Figure 5: Example user interface for human evaluation for LLM game engine.
C HUMAN EVALUATION FOR IMAGE GENERATION
In this section, we present hu- man evaluation to assess the per- formance of our dynamic regional IP-Adapter in maintaining environ- ment consistency, character consis- tency, and semantic alignment with the text prompt. Specifically, we sample 50 character-environmmett text prompt triplets from our image evaluation benchmark and generate corresponding images using our ap- proach and StoryDiffusion Zhou al.. (2024)). Four human annotators were tasked with evaluating the images across three dimensions: character consistency, environment consistency, and semantic aligument. Given that UNBOUNDED functions character life simulation game, ensuring the presence of the character in the image critical. To emphasize this, annotators were instructed to disfavor images where the character is absent. In such cases, the image is also rated lower in terms of environment consistency and seman- tie alignment. The user interface used for collecting human annotations is shown in Figure 6 To minimize bias, the order of images generated by the models randomized, ensuring that an- notators were unaware of the model responsible for generating each image. 3 Jummarizees the annotators' votes for win, lose, and tie outcomes for the dynamic regional IP-Adapter and StoryDif- fusion. Our evaluation demonstrates that the dynamic regional IP-Adapter significantly outperforms Story Diffusinno in maintaining character consistency, environment consistency, and semantic align- ment with the text prompt.
Table 5: Human Evaluation on 50 generated images from our dynamic regional IP-Adapter and StoryDiffusion. We show the detailed win/lose/tie counts on each evaluation cat- egory.
Criteria
Win (%)
Lose (%)
Tie (%)
Character Consistency 60.0
19.0
21.0
Environment Consistency
54.0
21.0
25.0
Semantic Alignment
57.5
23.0
19.5
D EVALUATION OF LLM AS GAME ENGINE WITH MULTIPLE SAMPLES
We evaluated our LLM using two decoding strategies- -regddy decoding and sampling to obtain more robust comparisons. Greedy decoding generates consistent output for identical inputs, ducing randomness in evaluation. The results are presented in Table We observed that greedy decoding outperforms GPT-4o when evaluated using GPT-4 as the judge. Additionally,
18

---

Published as a conference paper at ICLR 2025
Character
Environment
Image1
Image2
Evaluating: 11 /0
The sks wizard pointed his staff to the sky, and the grasses grew taller, allowing more butterflies flutter gently among them.
Character Consistency
Image1
Image2
Environment Consistency
Image1
Image2
Semantic Alignment
Image1
Image2
Figure 6: Example user interface for human evaluation for our dynamic regional IP-Adapter.
ducted three more runs using sampling (samples 2, 3, and 4) and included the results here. Despite the variations in responses across different samples, all outputs maintain high quality under the evaluation criteria.
Table 6: Evaluation results comparing our model with GPT-4o under greedy and sampled decoding. Scores are judged by GPT-4 on five metrics: Overall, State Update, Environment Relevance, Story Coherence, and Instruction Following.
Sample
Overall
State Update
Environment Relevance
Story Coherence
Instruction Following
Ours
GPT-4o
Ours
GPT-4o
Ours
GPT-4o
Ours
GPT-40
Ours
GPT-4o
Greedy
7.93
7.44
7,83
7.20
8.27
7.62
8,04
7.62
8.05
7.60
Samplel
7.68
7.76
7.66
8.10
7.82
7.95
7.82
Sample2
7.50
6.94
7.40
7.76
7.08
7.60
7.08
7.64
7,04
Sample3
7.87
7.38
7.89
7.12
8.24
7.68
7.92
7.52
8.01
7.51
Sample4
7.61
7.16
7.56
6.90
7.90
7.47
7.69
7.28
7.72
7.30
E PROMPT FOR SYNTHETIC COLLECTION
INTERACTION DATA
In this section, we provide the prompt templates we use collect the ussu-simulation data. Specif- ically, we query GPT-3.5 to generate diverse topics and character descriptions using the template shown in Figure 13 The prompt template for guiding the potential user interactions for user LLM To ensure user interactions align with the dialogue history, we include the interaction history as input to the user LLM. The world simulation LLM prompt template, shown in Figure 15 also takes in the dialogue history and generates the next character actions, states and narratives. We constraint the world simulation LLM to generate one storyline at a time, allowing users to choose how continue the story.
is shown in Figure 14
19

---

Published as a conference paper at ICLR 2025
A game of the witch: Step into starry nightscape at celestial observatory perched high above the clouds. Use astrological magic to manipulate the stars and planets, crafting spells that can influence time itself. Solve cosmic puzzles and unlock ancient knowledge hidden among the constellations.
Environments
witch stands the observatory's telescope, gazing at the interconnected stars.
show the witch celestial puzzle to solve for extra fun.
witch examines the celestial puzzle, its stars glimmering with potential solutions. [Character Response] The witch's eyes light up with excitement and says: "Let's see what this puzzle holds!"
+
bring the witch to the ancient library within nebula.
+
witch steps into the ancient library, marveling at the mystical tomes and celestial artifacts.
Figure 7: Qualitative Example of Unbounded (Split 1). The example is collected with GPT-4o as the user interact with our game.
F EVALUATION PROMPT
We include the prompt template we compare the outputs from two LLMSs in Figure T6 The prompt adapted from Vicuna (Chiang et al., 2023), and has been validated as an effective tool for comparing the performance of two LLMs on given task.
G REPRODUCIBILITY STATEMENT
First, include the implementation details of UNBOUNDED in Sec. 4.2 covering the training hyperparameters for Dreambooth fine-tuning, and LLM distillation, and the hyperparameters we use during inference for both LLM and image generation. Second, provide detailed description of the user-simulation data we collect for training in Sec. S.c.3.3 and further include the prompt template used query GPT models Appendix Sec. E Lastly, for the LLM-based evaluation, the prompt template for querying GPT-4 to compare two LLM outputs is provided in Appendix Sec. F
20

---

Published as a conference paper at ICLR 2025
The witch is curious and intellectual, always drawn to the mysteries and knowledge hidden in ancient texts. Please continue the generation accordingly.
witch carefully pulls out a dusty, ancient tome, its cover glowing with celestial energy.
The witch has strong affinity for celestial magic, often feeling invigorated by the cosmic forces within mystical artifacts. Please continue the generation accordingly.
witch concentrates, casting spells from the mystical tome, feeling the cosmic forces invigorate.
offer the witch magical snack infused with cosmic energy to satisfy hunger.
delve deeper into the tome. witch enjoys the magical snack, feeling rejuvenated and ready to
[Character Response] The witch gratefully accepts the snack, saying: "Thank you, this give me the strength to continue."
guide the witch through playful, magical exercise boost her fun.
witch dances with the celestial artifacts, feeling a surge of playful, magical energy.
+
[Character Response] The witch laughs and says: "This is delightful! Let's what rewards these new skills!"
Figure Qualitative Example of Unbounded (Split 2).
21

---

Published as a conference paper at ICLR 2025
The witch has a lively and imaginative spirit, thriving on the energy of playful magic and celestial exploration. Please continue the generation accordingly.
witch discovers a hidden passage in the ancient tome.
The witch is adventurous and courageous, eager to uncover secrets and delve into the unknown realms that ancient wisdom holds. Please continue the generation accordingly.
witch steps through the hidden passage, finding herself in swirling portal to celestial realm.
encourage the witch to rest and regain energy before venturing further.
witch finds serene spot in the celestial realm to rest, feeling the cosmic energies revive.
[Character Response] The witch nods and says: "A rest will refresh my mind. Thank you for the suggestion."
bring the witch to the floating island with magical gardens.
witch arrives on the floating island, marveling at the magical gardens and sparkling waterfalls light.
Figure 9: Qualitative Example of Unbounded (Split 3).
22

---

Published as a conference paper at ICLR 2025
offer the witch a refreshing bath beneath the waterfalls light rejuvenate and improve hygiene.
witch splashes in the refreshing waters, feeling rejuvenated and her hygiene improved.
[Character Response] The witch smiles gratefully and says: "A bath sounds wonderful. Thank you!"
bring the witch to the vast, star-studded sky with cosmic puzzles.
witch steps into the vast, star-studded sky, feeling the ancient cosmic puzzles calling her.
provide the witch with nourishing meal to alleviate her hunger and restore energy.
witch enjoys the nourishing meal, feeling her energy and spirit revive.
[Character Response]
The witch eagerly eats the meal and says: "This is exactly what needed. Thank you!"
engage the witch with delightful cosmic game to boost her fun.
fun levels skyrocket. witch joyfully participates in the delightful cosmic game, feeling her
[Character Response] The witch laughs joyfully and says: "This is much fun! feel completely engaged."
Figure 10: Qualitative Example of Unbounded (Split 4).
23

---

Published as a conference paper at ICLR 2025
Game Setup
choices with instructions
User instruction input
Hunt the dragon!
Character Action
Go buy pation.
Step1: User input character and game topic.
Entening the Porect...
Exploring
Encounter the Dragen!
Character: young witch. Topic: Dragon Hunting.
Brafssh
Blratch 2
Bradch 7t
Hunt the dragori
Go the village buy magic potions.
Step2: Game Environment Generation
Fighting with magic
Dragon retesat
Gring vilaor..
Chopping..
Rest
EExplore pond.
Find pation shop.
Rest have food.
Forect
pand
Rest and ent
to sond
Buy potion
Hove feod
willgee
in
Continued...
Figure 11: Example of UNBOUNDED. Based on initial user input, UNBOUNDED sets up game simulation environments, and generates character actions in the environments. Users can interact with the character with natural language instructions, exploring the game with unlimited options.
M witch coaxes the caoti bloccom with vibrant, glowing flowers
Down Block
Mid Block
Up Blocks
Figure 12: Attention map between character embedding and hidden states in cross-attention layers in different blocks. The character embedding we use is "A [V] witch".
Topic Character Collection
You're that responsible for generating the charactor and story topics for the charactor. Hore/re several instructions you need to follow:
story topic, you pair together.
The character should be diverse enough cover human, animal, rohot.
The topic should be very high-level (e.g. space adventure, daiy life, adventure the candy garden).
The generate
topic should diverse poesible. etory topic one language
description story topic
Inees.
6. Please generate 20 more characters and story topice.
Figure 13: Prompt template used to collect diverse topic and character data.
24

---

Published as a conference paper at ICLR 2025
User LLM
slory. give character state. several instructions you need follow:
agent
story
1. each round, you follow the instructions from ("Continue Goneration", "Describe Character Details and Continue Gonoration", "Interact with Character", "Move the Character to New Environments")
"Cantinue Genoration", you only inputt "contnue generation".
3.In "Dascribe Character Details and Contruee Generation', you need to provide character dotails l-gg. parsonality the main character, which environment the story happens).
4n "Interact with Cheracler", interact with cheracler
cheracler state healthy specily
with the cheracter, feed the state you update, but only provide
character, ask character rest, and
character bath). interaction action you will take. Please provide interaction to update only one state at time.
In "Move the Character to New Environments", you will lake the characler to explore another environment that have been generated under [Environment], You don't need to mention the environment number (e (ENV2), only mention the environment name, le bringg the character the sea).
You should try generate diverze characteristics and explore diverse interactions. The interaction ehould be reesonable based on the given environment and previous generated stories, and try to make the character state healthy. 7. You only need generate the characteristic or the interaction, you don't need generate the story about The interaction should be direct. natural easy understand. The output contain loss than wards.
Hone's one exampie:
Input: [Environment]
1] 2. plush amchair sis invitinly before crocking firpplace. JENV 2]
grassy
[Original Story)
ska cal explores the new smells, [ENV 1]
[Character State]
Hunger: 0%
Energy:
Fun: 50%
Hygiene: 100%
Output Example 1: Continue Generation.
Output Example My cat is energetic and loves to play with water. Please continue the generation accordingly.
Output Example my say gift..
Outpul Example feed with food.
Outpul Example bring my cat the firaplace.
Figure 14: Prompt template used query user LLM.
25

---

Published as a conference paper at ICLR 2025
World Simulation LLM
You're agent that acts world madel character defined human instructions, You first generate potential onvironments for world, then generate stories
sound game, leave space for the users' interaction. When generating the stories, you should generate them one by one, where each story split simple so that it can be captured visual image. Human can internupt and give additional information of the character, and you need to ganerate the charactors reaction and the following stories based
character living world. stories
given informaion. Here're
instructions you need to follow:
1. Please first describe the environments that will appear the story, the formal "Environment description) [ENV NT. The total environment number should not exceed 5, Try to make the description focus on the environment, and avoid mentioning any character or human animal in environment.
generaling each story splil, please slways follow the formet '[pliilled story part] Nj" JENV indicales story happerts in which of the previously generated environment. You should generate story split at lime, and continue generalion besed on the new instruction glven by the human.
3. Please generatee the story the same envirorment logether, inslead of generaling one story each environment Only move the next instruction.
environment when expicilly asked by
The langth each splitted story should exceed 50 words. The content should be able to included one visual Image. You should ganorate story split at time.
5. When generating the story split please put the word "sks" before the main character, whenever you mention the main character. The main Max"). and avoid reforring the
character is specified
instruction. Always refer character with its (n.g. "sks dog"
character with "he" or "she".
6. Please do not put the word "sks" before characters other than the main character:
7. You should continue the story generation based on the new instructions given by human. You should always output a story split in the format
[Splitedd story part] [ENV NJ. You can add new environments needed following the formal (Environment descr (otio)) [ENV Nj. You should alwayy
generate [Orginal Story) as
(Splitted story DaV NJ, [Envirorment] 8. You should ty to generate the environment and story as diverse as poesible.
before (Environment description (ENV N)
9. Besides the story generation, please indude the character's 's response after [Response) when needed. For example, when the user said '1
play with
" The response from dog can 'The dog wags his tall and said: I'm happy can play with me". Only generate the response there's human interaction given by the user. And you need generate story split after reeponze.
Laslly, you need to maintain the state of the characler. The character state indludes aspects: hunger, energy. fun and hygiene. The statle is 100% for hyg bygenne and energy. 50% for fun and 0% for hunger You need to auto-update the state based on the story.
11. If the user simply continue the story generation, hunger will up and others will go down based on the story generated. When the user provide somo intoraction, you should update state based the user feading the character decrease hunger, user esks the character to rest will increase energy. the user cleens the character will increase hygiene).
interaction je.g. The playing with the character will increase fun,
12. You should update the state by at loast 15% each time. You should update all the four states when continue goneration, and update partil instructions.
states when the provide
energy hygiene and the (Character State] to ask the user to interact with the character to rocover the character state continue simulation.
The characler simulstion
below or hunger above 100%. You should output [Action Needed]
14. You don't need provide additionll instructions like "You can give further instructions to continue the story or interact with the character. at the end of the response.
Here's one example:
Humen: (Instruction)
Please describe the day my cat Sunny. Main character:
Model:
[Environmant]
1. Large grassy area enclosed by a blue fence. JENV 1]
2. A plush amchair sits inittingl before crackling fireplace. (ENV
[Originel Story]
A sks explores all the new smells. JENV
(Charaatee State]
Hunger: 0%
Energy: 100%
Fun: 50%
Hygiene: 100%
(optional) [Reaponse] cat your
Figure 15: Prompt template used query world simulation LLM.
26

---

Published as a conference paper at ICLR 2025
Evaluation
You are a helpful and precise assistant for checking the qualty of the answer. We would like to request your fandback on the performance of two Al input history dialogue displayed below:
assistants in response to
TTask Prompt]: You're an agent that acts as world model on a character defined by human instructions You need to first generate the potertial environments for the world, and then genterate stories of the character living the world. The stories should sound like a game, and leeve spece for the usere' should generate them one one, where each story spiit simple that It can be captured in visual generate the character's maction the following
interaction. When generating the stories, image. Human internupt give additional information the character, stories based on the given information. on. Harc're several instructions you need to follow:
you
Please first describe all the environments that will appear the story. in the format "Environment description) [ENV NT. The lotal environment number should not exceed Try make the description focus on the environment, avoid mentioning character human or aniall ervironment.
When generating each story split, please always follow the format "Splitted story part] [ENV NJ: (ENV NJ indicates the story happens which of the previouely generated environment You should generale story spit al time, and continue generation based on the new instruction given the human.
generate the story onvironment when explictly asked by the human instruction.
environment togather, instead of generating story
anvironment. Only
the
4. The length of each splited story should not exceed 50 words. The content should be able to be included in one visual image. You should
generate 1 story spiit at time.
5. When generaling the story spil, please
word "sks" before the main cheracter, whenever you mention the main character. The main instrution. Always
character specifid in
"oka dog" Max"). avoid referring to
character with "he" "she".
6. Please do not put the word "skss before characters other than the main character.
7. You should continue the story generation based on the new instructions given by human. You should always output story spiit the environments needed following the format (Environment description) [ENV Ng. You should
format (Splitted part) always generate (Onginal Story) as the title before [Splitedd story part NJ [Environment] the before [Environment description] N]
NJ. You
B. You should try generate the environment and story diverse possible.
Besides the story generation, please indudd the characler's resporse after (Response] when needed. For example, when the user said Only generate response
play with dog The resporse from dog when thera's human interaction glven the usar. And you nead generata story spitt after response.
"The dog wags his tall
I'm happy you can play
10. Lastly you need to maintain the state the character. The character state includes 4 aspocts: hunger, energy, fun and hygicne. The state is 100% for hygiene and energy, 50% for fun and 0% for hunger. You need to auto update the state based on the story.
up and others will down based the generated. When the interaction g. user playing with character will incresse the user feeding the character will decrease hunger, the user asks the character to rest will increase energy. the user cleans the character will increase hygiene).
11. user simply continue the generation, hunger
provide some interaction, you should updale state based
12. You should updale the stale by least 15% each time.
life simulstion end energy hygiene fun below hunger ls 100%, You should output [Action Neadad] and the [Character State] to ask the user interact with character to recover the character state to continue simulitionn
[Diaiogua History): (dialogue)
[The Start of Assistant 1's Answer):
(enswert)
(The End Assistant I's Anewer]:
[The Start of Assistant 2% Answer] (enewer2)
Please Assistant Answer]
character state update accuracy. environment relevance, story coharenoe, users instruction following Each assistant receives an
overall score on scale of to 10, where a highor scom indicates better overal performance. Basides, each assistant mocaves scores on charactor state update accuracy, environment relevance, story coherence, users' instruction following soparately, with score on scale 1 to 10.
for Assistant and 2, respectively. The two in following lines, outpul ecore different aspedt in compreherisl eiveesiiveeeee explanation of your eveluslion, avoiding any polential bies and ensuring that the order in which the responses were presented does not affect your judgment.
Please first outout single line containing only two values indicating the
separated
space.
format, Then, the subeequent please provide
Figure 16: Prompt template used to query GPT4 to compare the performance between two LLM outputs. The prompt is adapted from Vicuna (Chiang et al. 2023).
27


Reviews:
[{"review_id": "B5fG2n5Q8V", "reviewer": "Reviewer_4EBp", "summary": "The authors present \"UNBOUNDED\" a virtual pet game based on LLMs. The authors present two novel technical elements to support this game: a regional IP-Adapter for character-environment consistency and a domain-specific distillation approach for finetuning a smaller LLM to allow for real time gameplay. The authors then present experiments comparing their approach to baselines in terms of visual quality and consistency, and overall game quality as scored by GPT-4o.", "questions": "1. How do the authors position their work in the field of technical game design? \n2. What do longer gameplay examples look like? \n3. How do the authors respond to my concerns around their \"Evaluation of LLM Generations\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The main strengths of the paper are (1) the originality of entirely basing a virtual pet game on LLMs, (2) the significance of the two primary technical contributions to figure coherent image generation and real time LLM application uses, and (3) the quality of the image portion of the evaluation/experiments. The paper is also overall well-written, especially in terms of the technical aspects. These primarily justify the positive aspects of my above scores.", "weaknesses": "The paper has two major weaknesses in its present draft, the way it positions itself in terms of prior work and the evaluation. \n\n### Prior Work\n\nThe authors do not appear to have engaged with the field of technical games research at all. This is a shame, as their game can be understood as an AI-based game [1] or specifically a NN game [2]. More broadly, the authors' work in terms of generation would fit into the Procedural Content Generation paradigm [3]. There is a 3+ decades long history of work in this space that is relevant to the authors' work [4], with significant recent work at the intersection of LLMs and game generation [5], and it is crucial that the authors update the introduction and related work to situate their work within this field.\n\n### Evaluation\n\nThe authors primarily evaluate their work in terms of image quality and I have no major concern with these results (though the DreamSim results seem to incorrectly attribute the best score to their system). However, I have major concerns with the \"Evaluation of LLM Generations\", which is functionally the closest thing presented to a holistic evaluation of UNBOUNDED. Firstly, the norm within the area would be to make use of a user study for evaluation purposes [6,7]. Particularly since the authors make such an effort to allow for real time control, this would seem to follow naturally. Secondly, the evaluation is somewhat concerning due to the training process. Specifically, since the distillation approach makes use of GPT-4 to simulate user interaction data and the authors sample for uniqueness/variety, its fairly likely that the newly sampled user interaction data approximations will be very similar to these. Further, the largest model the authors compare to is again a GPT-4 variant. Given that GPT-4 is being used for both generation of data, as the system, and as the evaluation, there's a clear risk for bias. This could be addressed by any number of ways, including automated metrics for measuring game quality developed in prior game generation-like work [8,9]. But at present, it is difficult as an outside reader to get a sense of the actual quality of UNBOUNDED as a game. Failing this, it might be beneficial to include longer examples of gameplay in the supplementary materials at least.\n\n### Figures\n\nWhile these are not of major concern, I wanted to note two aspects of figures that I felt could be improved. First, the authors make use of a large number of fonts in their figures, and their primary font is a bit difficult to read. I'd suggest updating the figures to use a single, more legible font. Second, Figure 4 is largely superfluous. If the authors wish to retain it, it might be a better fit for an appendix.\n\n1. Treanor, M., Zook, A., Eladhari, M. P., Togelius, J., Smith, G., Cook, M., ... & Smith, A. (2015). AI-based game design patterns.\n2. Zhu, J., Villareale, J., Javvaji, N., Risi, S., Löwe, M., Weigelt, R., & Harteveld, C. (2021, May). Player-AI interaction: What neural network games reveal about AI as play. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-17).\n3. Shaker, N., Togelius, J., & Nelson, M. J. (2016). Procedural content generation in games.\n4. Pell, B. (1992). METAGAME: A new challenge for games and learning.\n5. Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., Togelius, J., & Yannakakis, G. N. (2024). Large language models and games: A survey and roadmap. arXiv preprint arXiv:2402.18659.\n6. Anjum, A., Li, Y., Law, N., Charity, M., & Togelius, J. (2024, May). The Ink Splotch Effect: A case study on ChatGPT as a co-creative game designer. In Proceedings of the 19th International Conference on the Foundations of Digital Games (pp. 1-15).\n7. Guzdial, M., & Riedl, M. O. (2021). Conceptual game expansion. IEEE Transactions on Games, 14(1), 93-106.\n8. Khalifa, A., Green, M. C., Perez-Liebana, D., & Togelius, J. (2017, August). General video game rule generation. In 2017 IEEE Conference on Computational Intelligence and Games (CIG) (pp. 170-177). IEEE.\n9. Guzdial, M., & Riedl, M. (2018, September). Automated game design via conceptual expansion. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (Vol. 14, No. 1, pp. 31-37).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmOOPEEhwh", "reviewer": "Reviewer_Nnkj", "summary": "This work combines various aspects of Generative AI to develop a LLM-based game engine. The design principles refer to open-world, continuous interaction games with limited simultaneous characters but significant opportunities for situation generation and interaction. Capitalizing upon recent research in the field, the system introduces a number of technical innovations both in terms of architecture (in the form of a dynamic regional prompt adapter) and in term of implementation (distillation of Small LLM). With an existing implementation, the paper includes elements of comparative evaluation,  both quantitative and qualitative.", "questions": "To which extent are results from Figure 6 consistent with results from Table I?\nHow have you ensured the quality and fairness (for evaluation) of the text prompts sample?\nHow can the current approach maintain narrative consistency over a number of interactions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is an ambitious piece of work, tackling difficult technical issues in balancing generation and interaction, as well as various training and fine-tuning protocols. It appears to have innovated on a number of aspects, for instance the dynamic mask to balance character and environment generation. It gives some interesting, detailed, insights such as those of Figure 4. The improvement over IP-Adapter thus appears more than incremental. Another strong points is the Small LLM distillation framework.\nThe comparative LLM evaluation has staged an appropriate number of LLM, capturing size and diversity. \nFinally, the paper demonstrates a good command of recent relevant work and compares its approach to really recent alternative methods (2023, 2024), which is welcome.\nHaving considered the authors' rebuttal and additional data, I have raised my score for contribution to 3 ('good').", "weaknesses": "From a fundamental perspective, there is a lack of awareness of some game design issues. The introduction is relatively naïve considering how these issues are addressed in game/play theory (Caillois, Huizinga). With the fine line between simulation and entertainment, and notwithstanding the reference to The Sims and Tamagotchi, contemporary audiences tend to have gameplay expectations that can only be met with sustained and meaningful (in the narrative sense) interactions. This feeds into the overall perspective of the evaluation methods which in the paper are primarily focused individual ‘event’ generation. While sustained UX evaluation may be at odds with the system’s current level of maturity, some of this could have been addressed through even slightly longer exchanges or chains of events construed as minimal narrative experience (which is claimed in the abstract but does not really materialize in the examples). The small examples given, including on Figure 1, do not convey a very strong sense of gameplay. \n\nThere is no evidence that the number of user-simulator examples is at all sufficient for the intended purpose of multi-topic data collection. Nor is there any discussion of some hierarchical organization of topics that would come close to scenarios, game genres or elementary forms of plot backbones.\n\nDespite the statement on replicability, the level of implementation details is unclear. Implementation details are rather cursory and uneven with some in-depth details but no systematic discussion, and minor duplications from other sections.\n\nEvaluation has both positive (see above) and negative aspects.\nUnless I am mistaken, Table I shows a couple of inconsistencies for results.\nOn Environment Consistency, how can 0.322 be the best score when it is in-between 0.381 and 0.257? How can 0.675 be the best score in-between 0.595 and 0.832?\nIt is difficult to find the Qualitative component of evaluation very convincing in the absence of a more systematic approach for generating test situations. From a similar perspective, the example Prompts do not contain advanced mechanisms for controlling response, or specific examples. One would thus wonder how LLM response can be kept consistent across runs, and what type of statistical sampling has been associated to LLM use?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)