MarioGPT: Open-Ended Text2Level Generation through Large Language Models
Authors: Shyam Sudhakaran
Miguel González-Duque
Matthias Freiberger
Claire Glanois
Elias Najarro
Sebastian Risi
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Models
Procedural Content Generation
Open-endedness
Novelty Search
TL;DR: We introduce a Text2Level system that is able to take text prompts and generate Mario levels. The language model can also be used to mutate existing levels in an open-ended fashion

Abstract: Procedural Content Generation (PCG) is a technique to generate complex and diverse environments in an automated way. However, while generating content with PCG methods is often straightforward, generating meaningful content that reflects specific  intentions and constraints remains challenging. Furthermore, many PCG algorithms lack the ability to generate content in an open-ended manner. Recently, Large Language Models (LLMs) have  shown to be incredibly effective in many diverse domains. These trained LLMs can be fine-tuned, re-using information and accelerating training for new tasks. Here, we introduce MarioGPT, a fine-tuned GPT2 model trained to generate tile-based game levels, in our case Super Mario Bros levels. MarioGPT can not only generate diverse levels, but can be  text-prompted for controllable level generation, addressing one of the key challenges of current PCG techniques.  As far as we know, MarioGPT is the first text-to-level model and combined with novelty search it enables the  generation of diverse levels with varying play-style dynamics (i.e. player paths) and the open-ended discovery of an increasingly diverse range of content. 
Code available at https://github.com/shyamsn97/mario-gpt.

MarioGPT: Open-Ended Text2Level Generation through Large Language Models
Shyam Sudhakaran', Miguel González-Duque' Matthias Freiberger" Claire Glanois', Elias Najarro', Sebastian Risi IT University of Copenhagen, moddlai Copenhagen abyamsnaair@rrooommal com, sebrâitu.c
(a) "many pipes, many enemies, little blocks, low (h) "no pipes, some enemies, many blocks, high clevation"
elevation"
(c) "many pipes, many enemics"
(d) "no pipes, no enemies, many blocks"
Prompt that does exist in the dataset: "many pipes, enemics, many blocks"
(I) Failure case: "Many pipes. enemies, blocks"
MarioGPT is able to successfully generate levels that follow the text prompt (a e). Failure cases rarely happen: for example in (f) the model manages to generate many pipes and some blocks, but it still generates enemies even though it was prompted with "no enemies".
Figure 1:
Abstract
Procedural Content Generation (PCG) is a technique to generate complex and diverse environments in an automated way. However, while generating content with PCG methods is often straightforward, generating meaningful content that reflects specific intentions and constraints remains challenging. Furthermore, many PCG algorithms lack the ability to generate content in an open- ended manner. Recently, Large Language Models (LLMs) have shown to incredibly effective in many diverse domains. These trained LLMs can be fine-tuned, re-using infor- mation and accelerating training for new tasks. Here, we introduce MarioGPT, fine-tuned GPT2 model trained generate tile-based game levels, in case Super Mario Bros levels. MarioGPT can not only generate diverse levels, but can be text-prompted for controllable level generation, addressing one of the key challenges of current PCG techniques. As far as we know, MarioGPT is the first text-to-level model and combined with novelty search it enables the generation of diverse levels with varying play-style dynamics (i.e. player paths) and the open- ended discovery of an increasingly diverse range of content. Code available at
Introduction
Procedural Content Generation (PCG) refers to techniques that automatically create game content, such as levels, maps, or characters [36]. Some of the benefits of PCG are an increase in the replayability of game and reduced production costs.
TTese authors contributed equally to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Initial Tokenized Irved 2,34,66 15.1
Geverated level
GPT Layers
Prompt
Jipes, I cnienty sone blocks, Lou rirvation
Prozen Text Encoder
Cross
Prediction 16,3314,55
Allstiinn
MarioGIT
Figure 2: MarioGPT prediction pipeline. Our MarioGPT model is a finetuned version of the distilled GPT2 language model. Like GPT2, MarioGPT is trained predict next token sequences, Levels are represented as strings, which are tokenized by a Byte-Pair Encoding, similar to the original GPT2 model. The level split by columns and flattened into single vector (or batch of vectors for multiple levels). To incorporate prompt information, utilize frozen text encoder in the form of pretrained bidirectional LLM (BART), and output the average hidden states of the model's forward pass. This average hidden state is then used in the cross attention layers the GPT2 architecture combination with actual level sequence being passed into the model.
Recently, developments in PCG and machine learning have started to influence each other in different ways [30]. PCG researchers are now incorporating machine saarning-basee approaches into their systems and models such as Generative Adversarial Networks (GANs) [13] can be trained to generate levels for games as diverse as Doom [10] or Super Mario Bros, training on levels from the Video Game Level Corpus [45]. However, current approaches in this field of Procedural Content Generation via Machine Learning (PCGML) [39] often rely on costly searching inside of the latent space of the underlying neural networks. It would be more desirable to being able to directly condition a generator to create levels with certain properties, ideally in natural language.
To address these challenges, we propose MarioGPT (Figure 2). a fine-tuned GPT-2 model trained to generate Mario levels. Our model demonstrates how LLMs can be combined with PCG techniques, enabling the effective creation of new and diverse levels through natural language prompts (Figure 1). Large language models (LLMs) trained on a diverse corpus such as the GPT-n family model [29], capture the statistical correlations of the human experience in the form of language correlations. Through this process, GPT acquires knowledge of how represent and predict intricate sequences. We utilize this knowledge to provide our model with the ability to generate levels that incorporate simple artefacts as well as more complex relational properties. Surprisingly, a high percentage (88%) of MarioGPT generated levels are in fact playable.
Furthermore, we combine MarioGPT with novelty search [22], a diversity-secking algorithm, to continually generate diverse levels in an open-ended manner. The combination of LLMs with algorithms such as novelty search opens up many interesting new directions for future research. We hope our work opens the door to more flexible and controllable PCG methods that can generate infinite content that is complex, diverse, and functional. To facilitate this, the code run the experiments in this paper is publicly available at: https: : h/t//ggttttt githhub. com/ shyamsn97 /marraarrra//aaa//eaammaaa gpt.
2 Background and Related Work
Procedural Content Generation. Procedural Content Generation (PCG) algorithms [36] deal with the automatic creation of game content (e.g. for level design, character generation, environment modeling, etc.). As reviewed in [36, 46), earlier works often focused on evolutionary computation [4], solver-based methods [37] or constructive generation methods (such as cellular automata, grammar- based methods, etc). More recently, deep learning for PCG [27, 39] has emerged as a promising approach to learning to generate high-quality game content in data-driven manner, which is not only aesthetically pleasing but also functional and challenging. However, the diversity, originality and playability of the generated content in addition to the controllability of its generation, remain major challenges [39]. Our work aims to show how conditioned language models, paired with novelty-drived approaches content generation [26, 2], could help tackle these shortcomings.
Neural Network-based Level Generation. Recent works the space of video game level generation, particularly for Super Mario [2, 8, 45, 33, 35, 34], also leveraged neural network architectures to create levels. Beukman et al. [2] evolved neural networks in order to generate levels, while others [8, 45, 12, 34] performed evolution / search in the latent space of trained generative model. These

---

works showed that guided sampling of the latent space of the learned generative model could result in diverse set of levels.
Previous works that utilize a trained generative model [8, 45, 34, 12] also explored the abilities to control characteristics in generated levels. However, to do so these methods relied on searching the latent space (e.g. through quality diversity algorithms [28, 9] or evolutionary strategies [14]) for levels with specific target characteristics (e.g. level with many pipes). This is a significant limitation because even though the generative models may represent a rich set of content, one has to search through its latent space to try to find the content that actually satisfies specific characteristics. MarioGPT is able to improve upon this limitation by incorporating text prompts into the actual generative process, allowing for easily controllable level generation. In other words, instead of searching for level with specific characteristics, MarioGPT allows us to just ask for it. Concurrently to our work, Todd et al. [42] showed that L.LMs can also be used to generate levels for other games such as Sokoban but their model did not allow for any text-prompting.
Open-Endedness and Genetic Algorithms. The open-endedness paradigm focuses on algorithms that can produce infinite innovation [24]. These open-ended algorithms are popular in the field of PCG, where designers and players both can benefit from diverse and never-ending content. However, PCG must balance the hard task of generating content with diversity as well as playability. Genetic algorithms (GA). a family of optimization algorithms that are inspired by the principles of natural selection, are commonly used as the backbone for more open-ended search methods. Because GAs allow the integration of multiple objectives, they are particularly suitable for achieving a balance between fitness and diversity.
In that regard, novelty search approaches [23] aim finding the most novel solutions at each generation, in comparison to what has been seen (i.e. an archive of previously discovered highly- novel individuals). What makes novelty-search powerful, and motivated its use in this paper, that it guides the generation towards increasingly diverse solutions in an open-ended fashion. Novelty search keeps track of solutions in an archive and measures diversity by the distance between their behavior characteristics (BCs) compared to that of their & closest neighbors. This makes novelty search very flexible, allowing for the use of many different behavior characteristic types.
Sequence Modelling and Transformers. Classic approaches to sequence modelling using recurrent neural networks (RNNs) [31] and Long Short Term Memory (LSTM) networks [15] have traditionally been constrained by the fading memory of the network's state vector, as well as limited scalability due to the temporal interdependency of the operations. Transformers [44] address both challenges by applying associative attention [1] to learned reprojections of the windowed input sequence, which is commonly referred as self-attemtion. These architectural innovations have enabled Large Language Models (LLMs) to learn from massive datasets. Additionally, such models have also shown to be effective in accelerated learning of down-stream tasks. Fine-tuning LLMs [7] involves using pre-trained model weights as weight initialization for new tasks.
One particularly relevant use of pretrained/ / fine-tuned LLMs comes from the method Evolution through Large Models (ELM), proposed in Lehman et al. [21]. ELM utilizes an LLM diff model (3), which is trained on code diffs obtained by Github data, giving the model the ability to modify code snippet based on a particular commit message. This diff model is used as a "mutation operator". for a GA that evolves population of programs. The wide generative capabilities of the LLM produce diverse mutations, resulting in novel individuals that vary increasingly over the course of the GA.
3 Open-Ended Level Generation through LLMs
Here we present our complete approach to open-ended level generation through LLMs, which is composed of two parts. First, we introduce our prompt-conditioned model MarioGPT (Figure 2) in Section 3. which generates levels -encoded as text- given natural-language prompt. Second, we detail how MarioGPT can be used in a novelty-search evolutionary loop (Figure 3) in Section 3.2, allowing the approach to produce a continual stream of diverse levels.
Level Representation. Mario levels are represented similarly to previous works [45, 8, 3,, 33, 34, 12), using the levels provided in the Video Game Level Corpus (VGLC) [40]. We utilize relatively small set of path- path-annotated levels, taken from Super Mario Bros. and Super Mario Bros.: The Lost Levels (in total 37 levels). For more details specific tiles present, see Section 6. I in the Appendix. These
3

---

Sampling
Mutation
Antivee movi level
Level L
Monatd Level I,
Novelty Selection
E BCU, theshhdd
MMutation process
MaticIPT
Mansseett
Transorri randon dilce by MustoaPP umpling
Trandier barder MarisBat ixpanting
E4
Level Li
Musated Level L,
Figure 3: Novelty search setup and MarioGPT mutation operators. level is sampled from a set of top clites the archive, mutated, and, if novel enough, added to the archive. The mutation process involves two main steps: (1) Pick random slice from the level and replace it with new MarioGPT sample, using random prompt. (2) Inpaint the border region with MarioBert preserve path consistency.
levels are stitched together, to essentially make one giant level, allowing us to sample freely without worrying about the ends of the levels. Each tile is represented
string. The string representation and characters are tokenized into discrete values using a Byte Pair Encoding tokenizer used in the original GPT2 model [29]. The tokenizer learns mapping that maps each tile to its unique token. One limitation from the dataset the simplified representation of enemies. Even though levels
contain many different enemies, each with different behaviors and features, the dataset represents them all as the same token.
3.1 MarioGPT Model
Our model, MarioGPT, a prompt-conditioned unidirectional language model, optimized for long sequence level prediction. More precisely, MarioGPT's architecture relies distilled, lightweight version of GPT2 [29] transformer architecture called DistilGPT2 [32] Encoding slices of Mario levels strings, similar to the approach taken in Summerville and Mateas [38],
fine-tune this distilled version of GPT2 on predicting next tokens in Mario levels. To generate levels, we concatenate window of previous 50 columns into single vector and feed them into MarioGPT.
Architecture:
MarioGPT's architecture the same as the DistilGPT2 architecture, except the cross attention weights are utilized for prompting. Even though DistilGPT2 supports context lengths up to size 1024, we limit our context lengths to 700, as we found increasing it did little to increase performance. In total, MarioGPT has 96 million parameters (86 million of the original DistilGPT2 parameters and 10 million from cross attention weights). We train MarioGPT for 50,000 steps, sampling random slices of levels at each iteration and optimize the model using the Adam optimizer [20]. In total, MarioGPT sees 200,000 training samples. Because the model relatively small, it can be trained using single Nvidia GeForce RTX 2080 Ti GPU.
Prompting details: In order to incorporate prompt information, we fine-tune the attention layers' cross attention weights, as illustrated in Figure 2. Prompts are encoded through BART [25], frozen pre-trained language model. Prompts are passed through the frozen language model and the hidden states from the forward pass are averaged into single vector. This average hidden state is then used the cross attention layers of the GPT2 architecture in combination with the actual level sequence being passed into the model. We represent our prompts as combinations of specific features along with keywords that correspond to quantiles (e.g. none, little, some, many). This allows us to easily generate level/prompt pairs by counting corresponding tile values. For more details on the prompts, see Section 6.2 in the Appendix.
'We utilize the open transformers library, specifically https: http::///auggigaacccoooitttitttt

---

In addition, it possible to use synonyms for words. For example, changing "many" to "a lot" or "a ton". produces similar results because the BART encoder can generalize well.
3.2 Open-Ended Mario Level Generation with Novelty Search
In the realm of PCG, it is important to not only generate levels with diverse physical features, but also levels that elicit a wide range of player behavior. When comes to creating Mario levels, the focus is on the different paths a player can take to complete the level. This is often a challenge for many algorithms (such as [45, 8)) and requires the use of external agent for evaluation. However, with MarioGPT, it is possible to generate diverse and controllable levels that approximate realistic player path, reducing the need for an external agent and producing levels that are directly playable. To encourage diversity in generated levels, we integrate MarioGPT within novelty search augmented genetic algorithm NNS-MarioGPT), where language-modee play the role of mutation operators. As illustrated in Figure 3, NSTMarioGPT iteratively samples and mutates elite levels from archive of generated levels.
Novelty Search: Mutated levels are only stored in the archive if they achieve higher novelty score compared to the previous elites. The novelty score is measured the mean distance between the behavioral characteristic vector of the levels and the behavioral characteristic vector of the K closest elements from the archive (K-means). Our goal in level generation to create paths that result in diverse player behavior, we use predicted player paths as our basis for these behavior characteristics. More specifically, we are interested in the relative patterns of predicted paths. For instance, if player character moves in straight line on high elevated blocks, we want the path's s representation to be close in behavior space to a path that moves straight in lower elevation. To achieve this, we represent the behavior characteristic as the normalized average of the predicted path's 's coordinates, allowing a smooth representation of paths (Figure 4). Thus the significance of single block difference is reduced, making it harder for mutated levels to be added to the archive. This is desired because we don't 't want the archive to fill up with levels that only vary slightly from the existing levels in the archive. For all our novelty search experiments, we
small neighhorhood of size 4, which results in a behavioral characteristic of dimension 100. We initialize our archive with a small number of levels (30), as we found mutations are significant enough to generate diverse set of levels without a big starting population.
Figure 4: Novelty search behavior characteristie. Left: level, Right: smoothed moving average of generated path.
The LLM-based mutation operation introduced in this paper (Figure 3) transforms a randomly picked slice of a level (a slice between 40 80 columns) with new MarioGPT prediction, guided by a random prompt. By itself, MarioGPT is able, through mutations, to produce a variety of levels with varying agent paths. However, because MarioGPT is unidirectional model, we cannot guarantee that the new generated path is consistent with the rest the level. To further improve path consistency, incorporate a fine-tuned mask prediction model (which call MarioBert), based on the Bert architecture. The BERT language model [7] is a bidirectional LLM that shows impressive performance in the task of mask prediction, which is analogous to image in in-painting. This ability is ideal for our use case, where MarioBert is used to inpaint its border region after the newly sampled slice, smoothly joining the mutated slice and the rest of level. This be observed in the second step of the "Mutation process" part of Figure 3.
Mutations:
4 Experiments and Results
4.1 Tile Prediction Accuracy
To measure how proficient MarioGPT is in generating levels and because the majority of tiles in these levels are air tiles, we focus on comparing non-air tile prediction accuracy. We compare to basclines: LSTM, as proposed in Summerville and Mateas [38] and MarioGPT that is trained
5

---

Table 1: Training Reconstruction Accuracy Validation Set
Model
Tile Ace.
Path Acc.
Promptable?
LSTM
46%
39%
NO
from-scratch- MariioGTT
31%
23%
YES
21%
11%
YES
MarioGPT
93%
91%
YES
from scratch (without using pretrained GPT2 weights), with results reported in Table 1. For all our baselines, we train for the same amount (200,000 samples). The results show that MarioGPT (using pretrained GPT2 model) outperforms all other baselines with regards to tile prediction. In addition, training MarioGPT from scratch and training adapter layer (a small multi layer network top of the original prediction layer) results in models that performs worse than even the LSTM baseline (given the 200,000 training samples). These models were trained with minimal hyperparameter search, so their performance can likely be improved. However, as a tangential point, this shows a major benefit of fine-tuning pretrained models, which seem to require much less effort in regards to hyperparameters.
4.2 Measuring Playability of Levels
To test for playability, we deploy Robin Baumgarten's A agent [43, 19] in 250 generated levels'. The reason for choosing Robin Baumgarten's 's A agent for measuring playability comes from its performance the 2009 Mario AI competition, where it beat handcrafted controllers and even simple evolved neural networks on getting the furthest in an infinite-level setting, as well as solving a corpus of levels [43]. We find that 88.4% of all MarioGPT--genrraee levels can be completed by the agent, and are therefore considered playable (compared to the best haseline, the LSTM,, which achieves around 31 % solvable levels). Moreover,
find that only one of the successful levels needed a retry with the A# agent. We further test whether the path generated by the model matches that of the A* agent to assess their feasibility. Table 2 shows the mean absolute error (MAE) between suggested and actual agent path for playable and not playable levels respectively. We see that for playable levels, the MAE between the path generated by the model and the actually taken path by the agent is 1.15 tiles. i.e. paths are on average about 1 tile apart. For the non non-playable levels, this average difference of taken paths is significantly higher with 4.56 tiles. Thus, we can conclude that in playable levels, the agent mostly takes a similar path as the one generated by the model. The significantly higher MAE of 4.56 in non- oon-plaabble levels on the other hand indicates that the path generated by the models in these cases may not be feasible for the agent.
Table 2: Mean average error (MAH) between paths suggested by model and Baumgarten's 's A agent. Results are averaged over 5 runs per level to account for minor stochastic variation in agent simulation. MAES are computed between y coordinates of path trajectories for every point on the x axis (which goes across the level) both trajectories have visited.
Playable
Not Playable
All
1.15
4.56
1.56
Considering the MAE of 1.56 tiles for paths in all levels, we can conclude that in the majority of the cases, the path generated by the model is similar to the path taken by an actual agent, and having the model generate a path through the level jointly with the level an effective approach to obtain high-quality levels in terms of playability.
To investigate the quality of the generated paths further, we visualize the paths with the most, least and median overlap (i.e. the levels corresponding to the maximum, minimum and median values for the mean absolute error in height) as well as two interesting handpicked examples in Figure 5.
'Since the agent's performance depends on the available compute, we test for playability by running each level times.

---

Figures 5d and 5e show that paths generated by MarioGPT tend to have more airtime than Baum- garten's 's agent in the sense that they only weakly take into account "gravity"" This result may be attributed to the nature of the path annotations in the models training set. In Summerville et al. [40], the authors use an A path solver find a path through the level, while an actual agent, such as the one we used for comparison here, is more strongly bound by game physics (especially "gravity") and has to avoid enemies in the level. A second reason for non-playable levels can seen Figure Sc: Baumgarten's S agent is spawned in tight space from which it can not escape, while the model has generated a path that traverses beyond the actual level, again path that would likely be suggested by solver. We argue that these issues in part be attributed the paths in the training data stemming from a solver rather than an actual agent, and could be alleviated in future work by annotating the training data with the trajectories of actual agents.
Level with minimum MAE between (b) Level with median MAE between paths
paths
(c) Level with maximum MAE between paths
(d) Playable level with interesting path trajectories
(e) Non-playable level with interesting path trajectories
Figure 5: A vs. MarioGPT generated paths. Levels with (a) minimum (0.02). (a) median (0.89) and (a) maximum (11.0) mean absolute eтror (MAE) between trajectory of actual A agent (denoted as A), and model suggestion (denoted as P), as well as interesting sand-picked examples. Positions where both trajectories overlap are marked with * Paths suggested by the model generally tend to have more airtime than the A* agent (d. e), likely due game physics not being accounted for the original path annotations of the training data.
4.3 Is MarioGPT memorizing?
(a) Sample with temp 1.0
(b) Closest datasct sample to (a)
(c) Sample with temp 1.7
(d) Closest dataset sample to (c)
(e) Sample with temp 2.8
Closest dataset sample (e)
Figure 6: Generated levels vs closest in dataset. Temperature of 1.0 ends up spitting out almost exactly what in the dataset, while increasing temperature improves sample diversity.
Memorization dynamics in LLMs remain an open problem when training transformer architectures [41, 5, 16]. While LLMs are incredibly powerful, they can sometimes overfit extremely and end up regurgitating training data. One popular way to alleviate this issue is to add some randomness in predictions in the form a tunable "temperature" parameter [17]. To evaluate whether MarioGPT generating levels that are identical to the training set, we sample with different temperature parameters
7

---

and compare them the closest level in the training dataset. From Figure 6, we can see that increasing temperature results in samples that are more diverse, but lack quality, In our case, when generating levels we use a temperature of 2.4-2.7, as it can generate diverse samples while still retaining some quality. There are many possible improvements to explore in the future. One common way is to simply increase the richness of the dataset. The more samples the model has access to, the less likely it is to overfit. We could also improve MarioGPT's sampling abilities by introducing different search methods other than sampling with temperature, such as constrained beam search [6] and dataset augmented search [16], to increase diversity while preserving more quality.
4.4 Guided Level Generation via Prompting
Through simple prompting, we are able to guide MarioGPT towards controllable and diverse level generation. We empirically evaluate the prompting ability of MarioGPT by generating 1,000 samples with various combinations of prompts, and check how accurate the generated levels are to the prompt descriptions. The results suggest that MarioGPT can generate levels that match their given prompts most of the time (Table 3). MarioGPT is the most accurate with blocks and the least accurate with enemies. This is expected because there are fewer total tiles of enemies, while there are many more block tiles observed during training.
Table 3: Prompt actual description accuracy
pipes
enemies
blocks
elevation
81%
68%
92%
76%
We visually evaluated the system, displaying selected pronpt-conditioned generations in Figure 1. In addition, we evaluate the importance of the keywords in the prompt by comparing the distribution of the number of pipes between levels generated with random prompts without pipes-related commands (e.g. "some enemies, some blocks, high elevation") versus random prompts with pipes-related commands (e.g. "little pipes, some enemies, some blocks, high elevation"). The distribution without pipe prompts is scattered, while the ones with pipe prompts result in distributions with peaks, indicating that the keywords actually have an effect on the level generated (see Figure 11 in the Appendix).
MarioGPT is also able to generate levels from text descriptions that are not represented in the dataset. For instance, Figure le shows a successful approximation of the prompt, "many pipes, no enemies, many blocks", with a slight inaccuracy in that it has 1 less pipe (5 pipes is considered "many", while 4 are present). However, this is not always the case, as can be seen in Figure If, where the model, prompted by "many pipes, no enemies, some blocks", generates a level with the correct number of pipes and blocks but generates too many enemies. In future work, we hope to explore more ways to incorporate prompt importance, such as editing levels with tiles to create more samples or prompt tuning [18].
4.5
Generating Diverse Levels with Novelty Search
Through the combination of an LLM (Section 3.1) and novelty search (Section 3.2), we are able to continuously generate diverse levels in an open-ended fashion. Specifically, NS- Mariaaii is able to generate a collection of levels with a diverse set of predicted agent paths. We project the archive as a set of 2D embeddings in Figure 7 and darken the embedding points that are added later in the process. We can see that the levels are increasingly filling up empty spots in the embedding space. We also compare the distribution of levels generated by novelty search to levels generated by random prompts in Figure Sa. Visually, we can see that the levels generated by novelty search are more spread out in t-SNE space and the sampled ones, indicating that they are more diverse. Finally, we have also evaluated the playability of levels generated by novelty search, and find that the majority are solvable and non-playane plaayabb levels not clustered but rather scattered across t-SNE space. This indicates that there is no trade-off between path diversity and the ability to generate solvable levels. Figure 8b shows the corresponding results.
Figure displays the overlayed predicted paths (in a level grid) as more and more levels get added to the archive during novelty search. Similar as in Figure 7, we can see that over time, the space
8

---

سددة
20
10.
0
18
-10-
-20
-20
-10
10
20
Figure 7: t-SNE of the levels in the archive. t-SNE embeddings are computed from the behavioral characteristic. Darker points indicate more recently added elements. Although novelty search is using the behavioral characteristics of the player paths, the levels also demonstrate visual novelty.
novelty-search random-sampling
20
solvable
10.
unsolvable
10
5
-5
-10
-15
-20
(a) Novelty search vs. random sampling
(b) Playable vs. non-playable levels
Figure 8: Comparing exploration for novelty search vs. random sampling and playable vs. non- playable levels. (a) t-SNE of both the embeddings of novelty-seane search levels and levels generated with random prompts. The visualization suggests that novelty search enables a much wider exploration of the space of levels. (b) Unsolvable levels not clustered together but instead scattered across the t-SNE space. This distribution indicates that there is no correlation between the diversity of levels and their solvability.
of possible predicted agent paths gets filled, as increasingly diverse levels are mutated and added to the archive. As more levels are added to the archive, more and more of the tiles empty space in the grid are being filled up, indicating that NS-MarioGPT is discovering variety of levels that produce diverse paths. Concretely, we found that after 300 levels are added to the archive, around 78% of the possible coordinates are filled up. However, there still many overlapping paths in the archive, meaning that similar paths are still being added to the archive. This is an issue that could be improved by using
related time series distance metrics that account for patterns in path [11].
Levels with the highest level with the lowest novelty, shown in Figure 10a, has path that is much more common in the archive, which can be seen by its almost identical look compared to the 2nd lowest in Figure 10c. The levels with higher novelty, Figure 10b and Figure 10d, have more unique patterns, but share similar
lowest novelty score from the archive are also shown in Figure The

---

(a) Archive after 10 levels
(b) 50 levels
(c) 150 levels
(d) 300 levels
Figure 9: Generated path populations during novelty search, Each line is a path through a level. Over time, NS- MS-MarioGPT fills more and more of the space of all possible paths.
iatt (a) Level with lowest novelty score
(b) Level with highest novelty score
(c) Level with 2nd lowest novelty score
(d) Level with 2nd highest novelty score
Figure 10: Comparison of most and least novel levels in the archive. The two least novel levels are very similar to each other, while the most novel levels have more distinct path patterns.
pattern towards the end. This indicates that one was created by mutating the other. We also found that the diversity starts to plateau after around 350-400 generations. However, this is very sensitive the behavior characteristic (the smoothed predicted path of the level), so it may be different for other behavior characteristics.
While NS-MarioGPT's still able to discover many diverse levels through its simple mutation process, more complex functions could also be explored. For instance, crossover, common mutation utilized in many genetic algorithms, would increase mutation diversity which can lead to more diverse levels.
5 Conclusion
Here we introduced MarioGPT, a fine-tuned GPT2 LLM that can not only generate diverse levels, but guide its generation via a language prompt. This ability is useful in the field of Procedural Content Generation, where balancing controllable and diverse generation is difficult task. We showed that MarioGPT is also able to (1) predict player interaction in generated levels, (2) generate diverse and playable environments, and (3) reduce the need for expensive external agent interactions (MarioGPT generate playable levels approximately 88% of the time). Additionally, when combined with diversity- driven algorithm like novelty search, MarioGPT can generate open-ended and functional content. While MarioGPT can generate diverse content, it is still limited. It can sometimes follow prompt instruction and it is also subject to memorizing the dataset. We hope to improve these limitations by introducing richer data the system.
One major benefit of re-using an existing LLM is that we can take advantage of all the research and improvements that have gone into these architectures. We plan to leverage the scalability of LLMs and train MarioGPT on bigger, more detailed annotated levels. Also, we are particularly excited about incorporating human feedback into the level generation process through reinforcement learning from human feedback (RLHF) [47]. The ability to fine-tune these models on human feedback allows users to continually tune their generated levels towards desired characteristies. Ultimately, hope that MarioGPT opens the door to more controllable and diverse PCG systems.
Acknowledgements
This project was supported by a DFF-Research Projectl grant (9131- 00042B) and a European Research Council (ERC) grant (GA no. 101045094, project "GROW-Ar").
10

---

References
[1] Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. Ncural machine translation by jointly learning to align and translate. arXiv preprint urXiv:1409.0473. 2014.
[2] Michael Beukman, Christopher W Cleghom, and Steven James. Procedural content generation using neuroevolution and novelty search for diverse video game levels. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '22, page 1028-1037, New York, NY, USA, 2022. Association for Computing Machinery.
[3] Herbie Bradley, Honglu Fan, Harry Saini, Reshinth Adithyan, Shivanshu Purohit, and Joel Lehman. Diff models
way edit code. CarperAl Blog, Jan 2023.
[4] Cameron Browne and Frederic Maire. Evolutionary game design. IEEE Transactions on Computational Intelligence and Al in Games, 2(1):1-16, 2010.
[5] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2022.
[6] Katsuki Chousa and Makoto Morishita. Input augmentation improves constrained beam search for neural machine translation: Nut at wat 2021, 2021.
[7] Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
[8] Matthew C. Fontaine, Ruilin Liu, Ahmed Khalifa, Jignesh Modi, Julian Togelius, Amy K. Hoover, and Stefanos Nikolaidis. Illuminating Mario scenes in the latent space of generative adversarial network, 2020.
[9] Matthew C. Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K. Hoover. Covariance matrix adaptation for the rapid illumination of behavior space. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference. ACM, jun 2020.
[10] Edoardo Giacomello, Pier Luca Lanzi, and Daniele Loiacono. Doom level generation using generative adversarial networks. In 2018 IEEE Games, Entertainment, Media Conference (GEM), pages 316-323. IEEE, 2018.
[11] Omer Gold and Micha Sharir. Dynamic time warping and geometric edit distance: Breaking the quadratic barrier, 2016.
[12] Miguel González-Duque, Rasmus Berg Palm, Søren Hauberg, and Sebastian Risi. Mario plays on manifold: Generating functional content in latent space through differential geometry, 2022.
[13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):119):44444) 144, 2020.
[14] Nikolaus Hansen. The CMA evolution strategy: A tutorial, 2016. hal-01297037v2f.
[15] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(5:1133511780 1997.
[16] Daphne Ippolito. Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski. Katherine Lee, Christopher A. Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives false sense of privacy, 2022.
[17] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-sofee softmax. 2016.
[18] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know?, 2019.
[19] Ahmed. Khalifa. The mario Al framework. https://githin wom/amidosii Mario- AI-Framework, 2009.

---

[20] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
[21] Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. Evolution through large models, 2022.
[22] Joel Lehman and Kenneth Stanley. Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary Computation, 19(2):189-223 06 2011.
[23] Joel Lehman, Kenneth O Stanley, et al. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pages 329-336, 2008
[24] Joel Lehman Kenneth O. Stanley, and Lisa Soros. Open-endedness: The last grand challenge you've ve never heard of. open- endedness- theeeesss-theeeee aate-llat-gradd---
hwtp://lwww.reill....o.. com/radar/ heard-o--- Accessed:
2017-12-19.
[25] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyamov, and Luke Zettlemoyer. Bart: Denoising sequence-to-See to-sequence pre-training for natural language generation, translation, and comprehension, 2019.
[26] Antonios Liapis, Georgios N Yannakakis, and Julian Togelius. Constrained novelty search: A study on game content generation. Evolutionary computation, 23(1):101-129 2015.
[27] Jialin Liu, Sam Snodgrass, Ahmed Khalifa, Sebastian Risi, Georgios N Yannakakis, and Julian Togelius. Deep learning for procedural content generation. Neural Computing and Applications, 33(1):19-37, 2021.
[28] Jean-Bac Baan-Baptstte Mouret and Jeff Clune. Illuminating search spaces by mapping elites, 2015.
[29] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188):9, 2019.
[30] Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence, 2(8):428-45) 436, 2020.
[31] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal represen- tations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
[32] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter, 2019.
[33] Anurag Sarkar and Seth Cooper. Dungeon and platformer level blending and generation using conditional vaes. In Proceedings of the IEEE Conference Games (CoG), 2021.
[34] Anurag Sarkar and Seth Cooper. Generating and blending game levels via quality- -iverryyy in the latent space of a variational autoencoder. In Proceedings of the Foundations of Digital Games, 2021.
[35] Anurag Sarkar, Zhihan Yang, and Seth Cooper. Conditional level generation and game blending. In Proceedings of the Experimental Al in Games (EXAG) Workshop at AIDDE, 2020.
[36] Noor Shaker, Julian Togelius, and Mark J Nelson. Procedural content generation in games. Springer, 2016
[37] Adam M Smith and Michael Mateas. Answer set programming for procedural content genera- tion: A design space approach. IEEE Transactions on Computational Intelligence and Al in Games, 3(3):187 8700 2011.
[38] Adam Summerville and Michael Mateas. Super Mario as a string: Platformer level generation via Istms, 2016.
[39] Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmgård, Amy K Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. Procedural content generation via machine learning (PCGML). IEEE Transactions Games, 10(3):257-270, 2018.
12

---

[40] Adam James Summerville, Sam Snodgrass, Michael Mateas, and Santiago Ontañón. The VGLC: The video game level corpus, 2016.
[41]
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memoriza- tion without overfitting: Analyzing the training dynamics of large language models, 2022.
[42] Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cemy Green, and Julian Togelius. Level generation through large language models. In Proceedings of the 18th International Conference the Foundations of Digital Games, pages 1 1-8, 2023.
[43] Julian Togelius, Sergey Karakovskiy, and Robin Baumgarten. The 2009 Mario Al competition. In IEEE Congress on Evolutionary Computation, pages 1-8, 2010.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[45] Vanessa Volz, Jacob Schrum, Jialin Liu, Simon M. Lucas, Adam Smith, and Sebastian Risi. Evolving Mario levels in the latent space of deep convolutional generative adversarial network, 2018.
[46] Georgios N Yannakakis and Julian Togelius. Artificial intelligence and games, volume 2. Springer, 2018.
[47] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019.
13

---

Appendix
6.1 Dataset Details
Table 4: Unique Mario tiles
Tile Type
Symbol
Visualization
Empty
Unbreakable
Breakable
Question Block
? Q
Coin
Enemy
Left pipe top
Right pipe top
CILILLLII
>
Left pipe lower
Right pipe lower
TIB
Cannon Top
Cannon Body
b
Path
6.2 Constructing Prompts
Prompts are represented as combinations of specific features (e.g. pipes, enemies, blocks, elevation) alongside quantitative keywords:
no, little, some, many, [0-1000]} pipes
no, little, some, many, [0-1000] enemies
little, some, many, [0-1000]] blocks
low, high} elevation
As an example, "no pipes, many enemies, low elevation" or "many pipes, many enemies, many blocks" both possible prompts. The keywords "no", "little", "some". "many" are calculated from quantiles of the corresponding count within 50 column window (Table 5). The "low" and "high" elevation are determined from the height of the highest unbreakable blocks segment of the level.
Table 5: Prompt Quantiles and corresponding counts within 50 column window
tile
little
some
many
pipes
2
enemies
0
1
3
blocks
50
75
176
14

---

(a) "Random prompts without pipe prompt"
(b) "little pipes random prompt"
(e) "some pipes random prompt"
(d) "many pipes random prompt"
Effect of prompt conditioning. Comparison of the distribution of the number of pipes between levels generated with random prompts without pipes-related commands (e.g. "some enemies, some blocks, high elevation") versus random prompts with pipes-related commands (e.g. "little pipes, some enemies, some blocks, high elevation") )) The distribution without pipe prompts is scattered, while the ones with pipe prompts result in distributions with peaks.
Figure 11:
15


Reviews:
[{"review_id": "Zfyvdi6jbW", "reviewer": "Reviewer_6gGD", "summary": "This submission proposes MarioGPT, which aims to generate diverse-and-playable Mario levels with LLM (GPT-2) through language prompts. The input and output of MarioGPT are level representations, not natural language. Human prompts are involved by incorporating the cross-attention layer into the LLM. A Novelty Search algorithm is further proposed to increase the diversity of generated levels.", "questions": "1. In the field of PCG, is there any game other than Super Mario? What is the generalization ability of the proposed method to other games (if there is any)?", "limitations": "Same as in the Questions part.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ Leveraging LLM into Procedural Content Generation (PCG) is a natural idea. This submission successfully accomplishes this, and the proposed MarioGPT could be really usable.\n+ This submission conducts a comprehensive study of various aspects of the proposed method, including the tile/path prediction accuracy, the playability of generated levels, whether MarioGPT is memorizing the training dataset, guidability by prompts. The overall experimental parts are convincing. The authors also analyze some current limitations and provide feasible solutions to them.\n+ The proposed Novelty Search algorithm successfully improves the diversity of agent paths.", "weaknesses": "- Overall, I think Procedural Content Generation (PCG) with LLM is 'quite' an obvious idea, making me doubt this submission's 'technical' contribution to the community. I did not see any other obvious weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and useful comments!\nIn terms of PCG applied to other games, while Super Mario Bros. is often used as a benchmark, there are certainly a variety of other games our approach could be applied to in the future. Another common PCG-benchmark are dungeon-like games such as the classic video game “The Legend of Zelda” (e.g. Khalifa et al. [1]). Our approach should generalize to such dungeon-like games as well because MarioGPT processes levels by flattening tile-based levels into long vectors, which is agnostic to content that can be represented as a string. Additionally, concurrently to our work, Todd et al. [2] showed that LLMs can also be used to generate levels for other games such as Sokoban, which indicates the generality of the approach. \n\nFuture work could also focus on other types of content, usually represented as graphs or images (e.g. Doom levels [3] or StarCraft maps [4]). However, these non-string representation would require more substantial changes to the MarioGPT architecture.\n\n[1] Khalifa, A., Perez-Liebana, D., Lucas, S. M., & Togelius, J. (2016, July). General video game level generation. In Proceedings of the Genetic and Evolutionary Computation Conference 2016 (pp. 253-259).\n\n[2] Todd, G., Earle, S., Nasir, M. U., Green, M. C., & Togelius, J. (2023, April). Level Generation Through Large Language Models. In Proceedings of the 18th International Conference on the Foundations of Digital Games (pp. 1-8).\n[3] E. Giacomello, P. L. Lanzi, and D. Loiacono, “DOOM Level Generation using Generative Adversarial Networks.” arXiv, Apr. 24, 2018. [Online]. Available: http://arxiv.org/abs/1804.09154\n[4] Togelius, J., Preuss, M., Beume, N., Wessing, S., Hagelbäck, J., & Yannakakis, G. N. (2010). Multiobjective Exploration of the StarCraft Map Space. 2010 IEEE Symposium on Computational Intelligence and Games (CIG), 265–272. https://doi.org/10.1109/ITW.2010.5593346\n"}, {"review_id": "o7u1EWvDDp", "reviewer": "Reviewer_wPh9", "summary": "In this work the authors fine tune a DistilGPT2 model to produce diverse, largely playable Mario levels. They incorporate a novelty search to favor diversity in levels, and also explore conditioning level generation on natural language user prompts. Their novelty search proceeds as follows: Given an initial set of levels, a level is sampled from the set and randomly mutated. The mutation involves deleting a random contiguous segment of the level and sampling a new one in its place from their model, then using a fine tuned BERT model to do inpainting on the border regions to smooth the transition between this and the rest of the level. The mutated level is accepted if it is sufficiently different from the other levels, using a distance metric comparing the player path through the level.", "questions": "* Relating to the weakness point above, in Table 1 I'm a little surprised that everything except MarioGPT does so poorly (LSTM does okay at least though). It'd be helpful to have a little intuition for why this is so hard?\n- The concurrent work cited by this paper (Todd et al 2023) finds that pretraining (vs from-scratch) GPT2 has little to no impact on a range of metrics. That is of course a somewhat different task and setup, but I'm curious if you have thoughts on why you seem to find the opposite result in this domain?\n- What embedding is used for the tSNE? Is it something like the last layer of MarioGPT?\n- Do you guarantee that each tile is parsed as a separate token? Without this it seems like the network could potentially get confused about reasoning about lengths/distances, like if one token represented two blocks while another represented just one. Seems like it works either way, but I'm curious if this is something you're doing or might look at.\n- The language conditioning is cool! Of course, a natural extension of this would be moving towards level generation from more unstructured natural language. For example, flexibility with synonyms to \"many\" or more complex things like \"a series of narrow pillars with an enemy on top of each one\". That sounds like enough for a separate paper but I'm personally curious if you've done any preliminary work in that direction or what you think of it. To be clear, I don't think this is needed in this submission I just think that the work is cool and am curious!", "limitations": "I think that this paper does an adequate job of presenting its limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The use of LLMs for game level generation certainly makes sense, and I think it's valuable to publish work in this area for the NeurIPS community to see and build on. The technical approach taken here is original, to my knowledge.\n- The combination of genetic algorithms with LLMs is interesting and novel, and I found the use of BERT for inpainting to smooth out the transition between the mutated segment and the rest of the level to be a particular clever approach.\n- The paper also does a great job of visualizing the levels produced by the model which is very helpful in getting a feel for the sorts of outputs that their model produces. The playability and temperature analyses were quite interesting as well.", "weaknesses": "- Section 4.5: It's hard to tell that the novelty search is actually helping, without a comparison to an ablated version that doesn't do the novelty search. In particular it's hard to tell if the darker points in Figure 9 seem to be filling in the space in between because of the novelty search, or if it's just that there are more sampled points so they happen to fill in some of the space in between the earlier points. Likewise with the rightmost graphs in figure 7 looking denser than the leftmost ones – taking more samples will always add some density, so in order to really conclude something you need a comparison to an ablation. Showing something like 300 levels from the novelty search plotted on the same tSNE as 300 levels from a version without novelty search (eg as a different color) would make it much clearer that there's actually an improvement in diversity (and likewise some analogous experiment for the other figure).\n* In a similar vein, in Table 3 it's hard to know how much the prompt is helping without seeing the scores without prompting (for an idea of what scores a fairly random approach would achieve); though I think is a less major issue.\n* After seeing the LSTM from prior work used in the Table 1 comparison, I was surprised that LSTM comparisons weren't given for any of the future evaluations, which would be helpful for understanding how much better this LLM setup is than prior work. For example, we learn that 88% of levels are solvable, but how does this compare to the LSTM approach? Likewise in the diversity analysis, a naive LSTM baseline would be interesting (but no need for all the novelty search machinery). This would just help me get a sense of where this work stands compared to prior work on these metrics that are being measured. That said, I wouldn't reject this paper on that basis – 88% playable is a great statistic regardless of what you compare it to, and I don't actually expect that the LSTM would make particularly good levels, but having the full comparison would strengthen the evaluation.\n- As a last, minor point, It's surprising (in a way that makes me feel that an explanation is needed) that such poor performance is obtained by from-scratch-MarioGPT on the tile prediction task given it was trained on 200,000 datapoints. The \"tile accuracy\" is presumably a measure of how well the system does next-tile prediction (analogous to next-token prediction in language modeling, and presumably this is the objective that all these models are trained on). I'd imagine something as simple as \"always predict an air tile\" since most levels look like they're more than half air, or \"predict the same tile that you saw to the left\" would do quite well, and the inability to pick up on even simple trends like this after 200,000 datapoints is surprising to me. Perhaps there is additional complexity I'm missing?\n\nOverall, this is an interesting paper that I end up borderline reject on because of the evaluation points mentioned above, but I don't have a huge amount of prior experience in this area or what typical evaluations are for PCG, so I am open to revising through discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First off, we thank the reviewer for the great and thoughtful feedback!\n\n**To answer the reviewer’s open questions:**\n\n > 1. \"in Table 1 I'm a little surprised that everything except MarioGPT does so poorly (LSTM does okay at least though). It'd be helpful to have a little intuition for why this is so hard?”\n\nWe hypothesize that the parameter space of MarioGPT-from-scratch is quite large and makes it harder to optimize. That being said, we simply utilized the same hyperparameters presented in the DistilGPT2 work: https://github.com/huggingface/transformers/blob/main/examples/research_projects/distillation/train.py, so there may be some improvements to be made with a more optimized hyperparameter search. We also hypothesize that extending the training process may also help. The LSTM seems to have an easier time, possibly because of the smaller parameter space.\n\n> 2. “The concurrent work cited by this paper (Todd et al 2023) finds that pretraining (vs from-scratch) GPT2 has little to no impact on a range of metrics. That is of course a somewhat different task and setup, but I'm curious if you have thoughts on why you seem to find the opposite result in this domain?”\n\nWe hypothesize that this is because of some notable differences to our work and the approach in the work by Todd et al (2023). In Todd et al. the text prompts and the generation of the level are done together (as opposed to using a frozen text encoder like our approach), which may have had an effect on performance. In addition, generating Sokoban levels could pose other challenges compared generating Super Mario Bros levels, which we will also explore in more detail in future work.\n\n> 3. “What embedding is used for the tSNE? Is it something like the last layer of MarioGPT?”\n\nThe embedding space is actually the smoothed moving average of the predicted path, which is essentially a time series (can be seen in Figure 4). We will make this clearer in the paper. \n\n> 4. “Do you guarantee that each tile is parsed as a separate token?”\n\nThis is a great point! We verified this and each tile is indeed parsed as a separate token. We will make sure to add this detail in the paper.\n\n> 5. ...a natural extension of this would be moving towards level generation from more unstructured natural language. For example, flexibility with synonyms to \"many\" or more complex things like \"a series of narrow pillars with an enemy on top of each one\". \n\nWe did some preliminary experiments that show it is possible to use synonyms for words. For example, changing “many” to “a lot” or “a ton”, produces similar results. For more complex ones like the one mentioned by the reviewer (\"a series of narrow pillars with an enemy on top of each one”), we wouldn’t expect the currently fined-tuned MarioGPT to work well. However, the approach should be able to generalize to such more complex prompts by keeping the MarioGPT architecture the same but adding diversity the annotated training set. Currently there’s no easy way to add “visual” descriptions of levels yet, but there are many exciting approaches that we hope to explore in future work, like utilizing visual language understanding models.\n\n**Addressing the weaknesses pointed out by the reviewer:**\n\n> “Section 4.5: It's hard to tell that the novelty search is actually helping, without a comparison to an ablated version that doesn't do the novelty search. ”\n\nThis was a very valuable comment and as suggested, we now created an archive by generating levels using randomly sampled prompts (as shown in green in Figure 2; see PDF attached to the rebuttal summary at the top). We combined the novelty search archive and the sampled archive and ran t-SNE on the combined archive. Visually, we can see that random sampling actually results in a decently diverse set of levels (indicated by the spread of points in t-SNE space). However, our novelty search approach clearly results in a much more diverse set of levels, as can be seen by the spread of points shown in orange.\n\n> “... in Table 3 it's hard to know how much the prompt is helping without seeing the scores without prompting”\n\nThis was a very valid point and we have now investigated the effect of prompt conditioning in more detail, which is shown in Figure 1 in the PDF attached to the main rebuttal summary. In this figure, we compared the distribution of the number of pipes between levels generated with random prompts without pipes-related commands to random prompts with pipes-related commands The results demonstrate that the prompt-conditioning has a significant effect on the tile distribution. We see similar results for the other tile types.\n\n> “... we learn that 88% of levels are solvable, but how does this compare to the LSTM approach?”\n\nRunning the same evaluation setup as MarioGPT (over 250 randomly sampled prompts), we found that the LSTM generates playable levels ~31% of the time.\n\n> “As a last, minor point, It's surprising (in a way that makes me feel that an explanation is needed) that such poor performance is obtained by from-scratch-MarioGPT on the tile prediction task given it was trained on 200,000 datapoints. ”\n\nTo clarify, tile accuracy here refers to non-air tiles. As pointed out by the reviewer, a majority of tiles are air tiles, and each model ends up heavily predicting air tiles, so we focus primarily on their ability to predict non-air tiles. However, there may be better metrics to measure time predictions here, namely those that can deal with in-balanced class datasets.\n\nAs mentioned in the response to the 1st question above, MarioGPT-from-scratch performance can probably be improved with more time and optimized hyperparameter search. However, as a tangential point, this shows a major benefit of fine-tuning pretrained models, which seem to require much less effort in regards to hyperparameters. We will clarify these points in the in the paper. \n\n\n\n\n"}, {"review_id": "7rgnvWh4fi", "reviewer": "Reviewer_aEvk", "summary": "This paper proposes a novel idea of using LLMs to perform Procedural Content Generation. It addresses several challenges including the diversity of the generated environments, the existence of a feasible solution (playability) and generating with language guidance. It proposes two key algorithms.\n\n1. MarioGPT - Fine-tuning on DistilGPT2 so that it can predict the next slice of the game. It also allows prompting - the text conditioning is done by using BART to encode text prompts then feeding the average hidden states into the cross attention.\n\n2. Novelty search for open-endedness - Using MarioGPT as a mutation operator in the novelty search evolutionary loop, prompting the MarioGPT with random prompts and generate candidates, and retain the candidates that pass the novelty criteria to add to the archive. Then do masked token prediction (impainting) to ensure that the level generated has a feasible path solution.\n\nThis paper conducts experiments that demonstrate (1) the high token prediction accuracy of MarioGPT, (2) good playability - i.e. most generated levels have feasible path solutions, (3) using a suitable temperature to control the trade-off between diversity and quality, (4) the accuracy in following the prompt and (5) the diversity of generated levels (in terms of the predicted path) achieved by novelty search.", "questions": "1. In section 4.4 \"Guided Level Generation via Prompting\", under what sampling temperature are these results measured? I am curious how / whether this randomness affects how accurately the model follows the prompts as well. The authors mentioned that randomness helps with diversity - does this sacrifice accuracy of following prompts?\n\n2. On a relevant note, in section 4.3 the authors mentioned that the temperature controls the tradeoff between diversity and quality. Could you elaborate what the definition of \"quality\" is?\n\n3. In section 4.5, at what point does the “diversity” start to plateau? Are there baseline studies regarding the diversity of generated paths?\n\n4. In section 4.2, are there quantitative baselines regarding the playability of generated environments? Could you share intuition regarding the trade-offs between playability versus diversity of the generated environments?", "limitations": "As mentioned above, (1) the measurement of capability to understand the instructions semantically and (2) the generalizability of this approach beyond this type of games remains to be further studied.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper proposes a novel framework for text-guided PCG: having a text-conditioning, generative model that predicts new levels, then having a novelty search algorithm reject the levels that are too similar to existing ones.\n\n1. This paper is written clearly. The methods are well motivated. The experiments are well-designed with detailed analysis and discussion.", "weaknesses": "1. Experiments are lacking in measuring “controllability”. The prompts are limited to a small set. The capability of following instructions on a more semantic level are not measured, such as the scene layout. and the spatial, functional, and semantic relationships between objects. (e.g. how tall the pipes are, “there is an enemy to the right of a pipe”).\n\n2. The technique of first auto-regressively generating a scene, then using “inpainting” to ensure a feasible solution seems to be pretty tied to this specific type of games where the scene moves linearly in 1 dimension (i.e. the character can only move forward, not backward), and hard to generalize to other PGC scenarios. For example, in order to generalize to the cases where the character can move both forward and backward, non-trivial changes are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the great feedback!\n\n**Addressing the questions from the reviewer**\n\n> In section 4.4 \"Guided Level Generation via Prompting\", under what sampling temperature are these results measured? ... The authors mentioned that randomness helps with diversity - does this sacrifice accuracy of following prompts?\nOn a relevant note, in section 4.3 the authors mentioned that the temperature controls the tradeoff between diversity and quality. Could you elaborate what the definition of \"quality\" is?\n\nThese are some great points brought up by the reviewer. Large Language Models, especially those that are trained on smaller datasets, like the set of levels we used in our work, suffer from overfitting and even “memorization”. This can be seen in Figure 7, where a temperature of 1.0 effectively results in the model spitting out a level (or slice of a level) that exists in the dataset. To alleviate this, temperature can be used as a toggle-able hyperparameter to control the degree of randomness or uncertainty in predictions. This effectively changes the skewed distribution of token predictions (such as a very large logit value for a single token), to a more smoothed distribution. This has an important tradeoff – more randomness introduces more diversity in tokens at the cost of “tile accuracy”, which means that increasing temperature can reduce the impact of the prompt and the effect of previous contexts. This is what we mean by “quality diversity” tradeoff, where \"quality\" here indicates how \"natural\" the level is, or how similar are its characteristics to the original levels. In section 4.4, we utilize a common temperature value of 1.4. We will make sure to clarify this in the paper.\n\n> In section 4.5, at what point does the “diversity” start to plateau? \n\nWe found that the diversity starts to plateau after around 350-400 generations. However, this is very sensitive to the behavior characteristic (the smoothed predicted path of the level), so it may be different for other behavior characteristics.\n\n> Are there baseline studies regarding the diversity of generated paths?\n\nWith regards to baseline studies of diversity of generated paths, we include a t-SNE plot of levels generated by novelty search along with levels that are generated from randomly sampled prompts (Figure 2 in rebuttal pdf). The spread of levels generated by novelty search is much larger than the levels generated by sampling, indicating that novelty search results in a more diverse set of generated levels.\n\n> In section 4.2, are there quantitative baselines regarding the playability of generated environments? Could you share intuition regarding the trade-offs between playability versus diversity of the generated environments?\n\nWe found that the best baseline, the LSTM, achieves around ~31% solvable levels, suggesting that MarioGPT is much better at generating solvable levels (achieves ~88%). With regards to the trade-off between diversity of generated levels (from novelty search) and playability, we plotted TSNE embeddings (in Figure 3 of the rebuttal pdf) and labeled “green” for solvable and “red” for unsolvable. We can see that the unsolvable levels are spread across the space without a discernible pattern, indicating that there isn’t really a relation between diversity of levels and playability. This is expected, because our mutations consist of just resampling portions of the level (with a random prompt) and stitching it together with the inpainting model. These models both learn how to generate valid and playable levels, so the mutated portions follow the same.\n\n**Addressing the weaknesses pointed out by the reviewer**\n> Experiments are lacking in measuring “controllability”.  ... The capability of following instructions on a more semantic level are not measured, such as the scene layout. and the spatial, functional, and semantic relationships between objects. (e.g. how tall the pipes are, “there is an enemy to the right of a pipe”).\n\nThis is a great point! For evaluating controllability, we included a new experiment (detailed in Figure 1 of the rebuttal pdf), where we compare random prompts excluding pipe descriptions, to those with pipe descriptions. The distribution of number of pipes for levels where pipes are excluded in the prompts is very spread out, while the other ones have peaks corresponding to the description. This indicates that prompting does have an effect, and it's possible to control the generation.\n\nOur prompts are currently simple (only dealing with counts of objects), but we still see that they can guide the model towards generating a diverse set of levels. Additionally, given how language models are able to scale well with richer data, we believe that our system can scale to many more detailed prompts like the ones mentioned by the reviewer. \nFor future work, we not only want to explore more of these semantic descriptions mentioned by the reviewer, but also visual descriptions of the level. This is an exciting direction and we hope to explore more in the future!\n\n> The technique of first auto-regressively generating a scene, then using “inpainting” to ensure a feasible solution seems to be pretty tied to this specific type of games where the scene moves linearly in 1 dimension (i.e. the character can only move forward, not backward), and hard to generalize to other PGC scenarios...\n\nIn regards to the reviewer’s concern about the generality of the inpainting method: Because both the tile prediction model and the inpainting model work on character level predictions, they are actually robust to any orientation. For instance, for a game like Sokoban, where movements can be in any direction, the level is still composed of tiles that can be passed into the LLM one by one. Figuring out the \"borders\" after resampling a portion of the level and inpainting the surrounding tiles also does not require much change in the system.\n\n\n"}, {"review_id": "baxFj7fCwx", "reviewer": "Reviewer_an9Y", "summary": "This paper proposes MarioGPT with novelty search, a method that can generate new Super Mario Bros levels. MarioGPT is finetuned from GPT-2 to generate levels. The novelty search is used to get novel levels by randomly selecting a generated level, mutating it, and then filtering it based on a novelty criterion: the mean distance between the mutated level and the K closest elements from the existing levels (K-means). Experimental results show that MarioGPT has a better level of reconstruction accuracy than LSTM, and 88.33% of the generated levels can be played by Robin Baumgarten’s A* agent.", "questions": "1. How to control detailed level settings like the initial moving direction of the enemy?\n\n2. Can MarioGPT be tuned on Super Mario Makers which consists of thousands of novel levels contributed by the gaming community?\n\n3. Why do from-scratch-MarioGPT and adapter-MarioGPT perform worse than LSTM?\n\n4. Mario GPT generates trajectories as well as game contexts. What about generating trajectories first, then selecting a trajectory and keeping it unchanged, and then generating, mutating, and filtering the game contexts? Will this improve the “playable rate” of the generated games?\n", "limitations": "See above questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of generating Super Mario Bros levels is interesting and might be helpful for both the reinforcement learning community and the game community.\n\n2. The result shows that the finetuned GPT has the ability to generate decent Super Mario Bros levels, which is insightful.\n", "weaknesses": "1. The evaluation criteria do not seem to be appropriate. For example, \"playable by an A* agent\" is not a convincing indicator that can measure the quality of generated levels. Actually, \"playable by an A* agent\" primarily measures the simplicity of the generated levels rather than the quality of the levels. Moreover, in section 4.4, \"Guided Level Generation via Prompting\", the \"empirical evaluation\" is not objective, and lacks baseline (like LSTM) comparisons. Since this paper claims that the generated levels are text-controllable, it would be better to provide a more explicit demonstration of controllability.\n\n2. The baseline (LSTM) in section 4.1 is weak. Table 1 shows that the pre-trained GPT-2 is very important in the generation process. Then, how does a GPT-2 without finetune perform? How does GPT-3/3.5/4 perform? Also, I doubt the conclusion that LSTM is not promptable since it can be integrated with a text encoder.\n\n3. Although the idea is interesting, the generated levels are simple. 88.33% of the generated levels can be played by an A* agent, and only horizontal movements are included (Mario cannot crawl into the pipes and there is no hidden brick, making the generated levels even simpler than traditional Super Mario Level 1-1). This greatly reduces the significance of this work, both in the fields of artificial intelligence and gaming industry.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback! \n\n**Addressing the qs from the reviewer**\n\n> How to control detailed level settings like the initial moving direction of the enemy?\n\nBecause of the limitations of the current dataset and its labels, we do not know the initial direction of the enemy. However, our system can scale with more diverse prompts and we plan to explore that front in future work, touching upon more semantic and visual descriptions of levels.\n\n> Can MarioGPT be tuned on Super Mario Makers which consists of thousands of novel levels contributed by the gaming community?\n\nGPT2 has been shown to be a very scalable architecture and can digest large amounts of diverse data. Given this, we believe that our architecture can scale with increasingly diverse levels (such as those generated by Super Mario Maker). We hope to explore this more in future work.\n\n> Why do from-scratch-MarioGPT and adapter-MarioGPT perform worse than LSTM?\n\nWe hypothesize that the parameter space of MarioGPT-from-scratch is quite large and makes it harder to optimize. That being said, we simply utilized the same hyperparameters presented in the DistilGPT2 work: https://tinyurl.com/distillcode, so there may be some improvements to be made with a more optimized hyperparameter search and longer training. Tangentially, this highlights one benefit of finetuning, as there was little to no hyper parameter tuning.\n\n> What about generating trajectories first...?\n\nI’s definitely possible to mask out all other parts of the level and keep the path tiles, allowing MarioGPT to inpaint the level around the desired path. To use this, we would probably need to train a separate network to just generate “playable” paths, which could perform the same (or worse) than the current system. That being said, this inpainting ability can still be very useful for path customization; we plan to look more into this in future work.\n\n**Addressing the weaknesses pointed out by the reviewer**\n\n> ...\"playable by an A* agent\" is not a convincing indicator that can measure the quality of generated levels...\n\nFor procedural content generation of Super Mario Levels (and many other tile-based games), A* is in fact the standard when it comes to measuring playability (see references [1-5])\n\nThe reason for choosing Robin Baumgarten’s A* agent for measuring playability comes from its performance on the 2009 Mario AI competition, where it beat handcrafted controllers and even simple evolved neural networks on getting the furthest in an infinite-level setting, as well as solving a corpus of levels [6]. To summarize, A* is able to solve difficult Mario levels beyond human performance (we refer the interested reader to a video of the A* agent in question: https://tinyurl.com/astaragent). \n\n[1] Volz, Vanessa, et al. \"Evolving mario levels in the latent space of a deep convolutional generative adversarial network.\" Proceedings of the genetic and evolutionary computation conference. 2018.\n\n[2] Awiszus, Maren, Frederik Schubert, and Bodo Rosenhahn. \"TOAD-GAN: Coherent style level generation from a single example.\" \n\n[3] Schrum, Jacob, Vanessa Volz, and Sebastian Risi. \"Cppn2gan: Combining compositional pattern producing networks and gans for large-scale pattern generation.\"\n\n[4] Volz, Vanessa, et al. \"Tools for Landscape Analysis of Optimisation Problems in Procedural Content Generation for Games.\" \n\n[5] Edwards, Maria, Ming Jiang, and Julian Togelius. \"Search-based exploration and diagnosis of TOAD-GAN.\"\n\n[6] Togelius, Julian, Sergey Karakovskiy, and Robin Baumgarten. \"The 2009 mario ai competition.\"\n\n> ...It would be better to provide a more explicit demonstration of controllability\n\nWe compared levels that are prompted without a “pipes” description to those that specify either “little”, “some”, “many” pipes (see Figure 1 of rebuttal pdf). We found that without specifying pipes, the distribution of the number of pipes that MarioGPT generates is scattered, while specifying keyword descriptions results in distributions that match. This shows that prompt-conditioning has a significant effect on level generation. We saw similar behavior for the other tile categories as well.\n\n> ...how does a GPT-2 without finetune and how does GPT-3/3.5/4 perform?\n\nGPT2 and GPT3.5 cannot generate legitimate mario levels when prompted, even when giving few shot examples and tile descriptions. This is because GPT2 turns out to be not powerful enough for complex in-context learning, and GPT3.5 seems to hallucinate and output random tiles that do not correspond to any prompt the user has given. \n\nThe reviewer pointed out a great point about the LSTM: The LSTM is not quite promptable in the same way MarioGPT is, (there’s no real attention mechanism), but we have included a simple way to prompt (by appending the prompt embeddings to input). Even with this, the LSTM ends up only generating ~31% playable levels. \n\n> Although the idea is interesting, the generated levels are simple..\n\nWith regards to the simplicity of levels (reviewer pointed out that there are no pipe interactions + hidden bricks): Some of the generated levels are still quite difficult for the A* agent, which only clears about 87% of the original Super Mario Bros and Super Mario Bros 2. The generated levels are also quite complex for human players, some requiring finely timed jumps and sharp increases / decreases in elevation.\n\nThe generated levels, especially when combined with Novelty Search, are in fact often quite complex (see Figure 8). Hidden bricks and similar elements could be added easily by adding denoting characters to the training levels, upon which the model would learn an appropriate token, which can then also be implemented in the simulator parsing the MarioGPT output. In general, as long as complex mechanisms can be converted to tokens and reflected in simulation, MarioGPT should be capable of generating them.\n\n"}]

Author rebuttal:
We would like to thank all the reviewers for their insightful comments, which helped us to significantly improve our paper. The most important new additions are: (1) a comparison to an ablated version without novelty search, which shows that novelty search is indeed crucial in allowing the discovery of a larger space of levels than random sampling; (2) we now also compare our approach to a prompt-able LSTM-baseline in terms of level playability and find that an LSTM-based approach generates playable levels around 31% of the time, which is significantly lower than the 88% for MarioGPT; (3) we clarify that A* is the current standard to evaluate procedurally generated levels and that solvable by A* does not adequate to levels being simple; (4) we added a controllability test to examine how much the text influences the generated levels, which shows that MarioGPT is indeed steerable through text prompts; and (5) We evaluated the playability of levels generated by novelty search, showing that the majority are solvable and there is no discernible tradeoff between path diversity and the ability to generate solvable levels. The new figures related to these additions can be found in the attached PDF.

Meta-review: This paper proposed an LLM based approach for Procedural Content Generation, specifically for Super Mario Bros levels. 

There has been concerns around the lack of technical contribution, the applicability to other content generation and the evaluation against baselines, while the authors have done a good job in convincing the reviewers during the rebuttal stages.

Based on the interesting application and also the potential interests from a broader community, I would recommend accepting this paper, while also encouraging the authors to incorporate the new figures/results obtained during the rebuttal stage to the camera ready version.
Decision: Accept (poster)