Title: MarioGPT: Open-Ended Text2Level Generation through Large Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Models

Procedural Content Generation

Open-endedness

Novelty Search
Abstract: Procedural Content Generation (PCG) is a technique to generate complex and diverse environments in an automated way. However, while generating content with PCG methods is often straightforward, generating meaningful content that reflects specific  intentions and constraints remains challenging. Furthermore, many PCG algorithms lack the ability to generate content in an open-ended manner. Recently, Large Language Models (LLMs) have  shown to be incredibly effective in many diverse domains. These trained LLMs can be fine-tuned, re-using information and accelerating training for new tasks. Here, we introduce MarioGPT, a fine-tuned GPT2 model trained to generate tile-based game levels, in our case Super Mario Bros levels. MarioGPT can not only generate diverse levels, but can be  text-prompted for controllable level generation, addressing one of the key challenges of current PCG techniques.  As far as we know, MarioGPT is the first text-to-level model and combined with novelty search it enables the  generation of diverse levels with varying play-style dynamics (i.e. player paths) and the open-ended discovery of an increasingly diverse range of content. 
Code available at https://github.com/shyamsn97/mario-gpt.

MarioGPT: Open-Ended Text2Level Generation through Large Language Models

Abstract

Procedural Content Generation (PCG) is a technique to generate complex and diverse environments in an automated way. However, while generating content with PCG methods is often straightforward, generating meaningful content that reflects specific intentions and constraints remains challenging. Furthermore, many PCG algorithms lack the ability to generate content in an open-ended manner. Recently, Large Language Models (LLMs) have shown to be incredibly effective in many diverse domains. These trained LLMs can be fine-tuned, re-using information and accelerating training for new tasks. Here, we introduce MarioGPT, a fine-tuned GPT2 model trained to generate tile-based game levels, in our case Super Mario Bros levels. MarioGPT can not only generate diverse levels, but can be text-prompted for controllable level generation, addressing one of the key challenges of current PCG techniques. As far as we know, MarioGPT is the first text-to-level model and combined with novelty search it enables the generation of diverse levels with varying play-style dynamics (i.e. player paths) and the openended discovery of an increasingly diverse range of content. Code available at https://github.com/shyamsn97/mario-gpt.

1 Introduction

Procedural Content Generation (PCG) refers to techniques that can automatically create game content, such as levels, maps, or characters [36]. Some of the benefits of PCG are an increase in the replayability of a game and reduced production costs.

Prediction

56, 13, 14, 88, ...

level

2, 14, 26, 33, 13, 88, ...

Prompt

3 pipes, 1 enemy,

some blocks,

low elevation

Encoder

Generated level

Cross

attention

GPT Layers

MarioGPT

Recently, developments in PCG and machine learning have started to influence each other in different ways [30]. PCG researchers are now incorporating machine learning-based approaches into their systems and models such as Generative Adversarial Networks (GANs) [13] can be trained to generate levels for games as diverse as Doom [10] or Super Mario Bros, training on levels from the Video Game Level Corpus [45]. However, current approaches in this field of Procedural Content Generation via Machine Learning (PCGML) [39] often rely on costly searching inside of the latent space of the underlying neural networks. It would be more desirable to being able to directly condition a generator to create levels with certain properties, ideally in natural language.

To address these challenges, we propose MarioGPT (Figure 2), a fine-tuned GPT-2 model trained to generate Mario levels. Our model demonstrates how LLMs can be combined with PCG techniques, enabling the effective creation of new and diverse levels through natural language prompts (Figure 1). Large language models (LLMs) trained on a diverse corpus such as the GPT-n family model [29], capture the statistical correlations of the human experience in the form of language correlations. Through this process, GPT acquires knowledge of how to represent and predict intricate sequences. We utilize this knowledge to provide our model with the ability to generate levels that incorporate simple artefacts as well as more complex relational properties. Surprisingly, a high percentage (88%) of MarioGPT generated levels are in fact playable.

Furthermore, we combine MarioGPT with novelty search [22], a diversity-seeking algorithm, to continually generate diverse levels in an open-ended manner. The combination of LLMs with algorithms such as novelty search opens up many interesting new directions for future research. We hope our work opens the door to more flexible and controllable PCG methods that can generate infinite content that is complex, diverse, and functional. To facilitate this, the code to run the experiments in this paper is publicly available at: https://github.com/shyamsn97/mario-gpt .

2 Background and Related Work

Procedural Content Generation. Procedural Content Generation (PCG) algorithms [36] deal with the automatic creation of game content (e.g. for level design, character generation, environment modeling, etc.). As reviewed in [36, 46], earlier works often focused on evolutionary computation [4], solver-based methods [37] or constructive generation methods (such as cellular automata, grammarbased methods, etc). More recently, deep learning for PCG [27, 39] has emerged as a promising approach to learning to generate high-quality game content in a data-driven manner, which is not only aesthetically pleasing but also functional and challenging. However, the diversity, originality and playability of the generated content in addition to the controllability of its generation, remain major challenges [39]. Our work aims to show how conditioned language models, paired with novelty-driven approaches to content generation [26, 2], could help tackle these shortcomings.

Neural Network-based Level Generation. Recent works in the space of video game level generation, particularly for Super Mario [2, 8, 45, 33, 35, 34], also leveraged neural network architectures to create levels. Beukman et al. [2] evolved neural networks in order to generate levels, while others [8, 45, 12, 34] performed evolution / search in the latent space of a trained generative model. These

works showed that guided sampling of the latent space of the learned generative model could result in a diverse set of levels.

Previous works that utilize a trained generative model [8, 45, 34, 12] also explored the abilities to control characteristics in generated levels. However, to do so these methods relied on searching the latent space (e.g. through quality diversity algorithms [28, 9] or evolutionary strategies [14]) for levels with specific target characteristics (e.g. a level with many pipes). This is a significant limitation because even though the generative models may represent a rich set of content, one has to search through its latent space to try to find the content that actually satisfies specific characteristics. MarioGPT is able to improve upon this limitation by incorporating text prompts into the actual generative process, allowing for easily controllable level generation. In other words, instead of searching for a level with specific characteristics, MarioGPT allows us to just ask for it. Concurrently to our work, Todd et al. [42] showed that LLMs can also be used to generate levels for other games such as Sokoban but their model did not allow for any text-prompting.

Open-Endedness and Genetic Algorithms. The open-endedness paradigm focuses on algorithms that can produce infinite innovation [24]. These open-ended algorithms are popular in the field of PCG, where designers and players both can benefit from diverse and never-ending content. However, PCG must balance the hard task of generating content with diversity as well as playability. Genetic algorithms (GA), a family of optimization algorithms that are inspired by the principles of natural selection, are commonly used as the backbone for more open-ended search methods. Because GAs allow the integration of multiple objectives, they are particularly suitable for achieving a balance between fitness and diversity.

In that regard, novelty search approaches [23] aim at finding the most novel solutions at each generation, in comparison to what has been seen (i.e. an archive of previously discovered highlynovel individuals). What makes novelty-search powerful, and motivated its use in this paper, is that it guides the generation towards increasingly diverse solutions in an open-ended fashion. Novelty search keeps track of solutions in an archive and measures diversity by the distance between their behavior characteristics (BCs) compared to that of their k closest neighbors. This makes novelty search very flexible, allowing for the use of many different behavior characteristic types.

Sequence Modelling and Transformers. Classic approaches to sequence modelling using recurrent neural networks (RNNs) [31] and Long Short Term Memory (LSTM) networks [15] have traditionally been constrained by the fading memory of the network's state vector, as well as limited scalability due to the temporal interdependency of the operations. Transformers [44] address both challenges by applying associative attention [1] to learned reprojections of the windowed input sequence, which is commonly referred to as self-attention . These architectural innovations have enabled Large Language Models (LLMs) to learn from massive datasets. Additionally, such models have also shown to be effective in accelerated learning of down-stream tasks. Fine-tuning LLMs [7] involves using pre-trained model weights as a weight initialization for new tasks.

One particularly relevant use of pretrained / fine-tuned LLMs comes from the method Evolution through Large Models (ELM), proposed in Lehman et al. [21]. ELM utilizes an LLM diff model [3], which is trained on code diffs obtained by Github data, giving the model the ability to modify a code snippet based on a particular commit message. This diff model is used as a "mutation operator", for a GA that evolves a population of programs. The wide generative capabilities of the LLM produce diverse mutations, resulting in novel individuals that vary increasingly over the course of the GA.

3 Open-Ended Level Generation through LLMs

Here we present our complete approach to open-ended level generation through LLMs, which is composed of two parts. First, we introduce our prompt-conditioned model MarioGPT (Figure 2) in Section 3.1, which generates levels -encoded as text- given a natural-language prompt. Second, we detail how MarioGPT can be used in a novelty-search evolutionary loop (Figure 3) in Section 3.2, allowing the approach to produce a continual stream of diverse levels.

Level Representation. Mario levels are represented similarly to previous works [45, 8, 35, 33, 34, 12], using the levels provided in the Video Game Level Corpus (VGLC) [40]. We utilize a relatively small set of path-annotated levels, taken from Super Mario Bros. and Super Mario Bros.: The Lost Levels (in total 37 levels). For more details on specific tiles present, see Section 6.1 in the Appendix. These

Sampling

Mutation*

Archive of

Level L

Mutated Level L

novel levels

Novelty Selection

Novelty criteria: ↓≥k BC(L; Lk)> threshold

*Mutationprocess

MarioGPT

MarioBert

Transform a random slice

Transform border regions

by MarioGPT sampling

by MarioBert inpainting

Mutated Level L

levels are stitched together, to essentially make one giant level, allowing us to sample freely without worrying about the ends of the levels. Each tile is represented as a string. The string representation and characters are tokenized into discrete values using a Byte Pair Encoding tokenizer used in the original GPT2 model [29]. The tokenizer learns a mapping that maps each tile to its own unique token. One limitation from the dataset is the simplified representation of enemies. Even though levels contain many different enemies, each with different behaviors and features, the dataset represents them all as the same token.

3.1 MarioGPT Model

Our model, MarioGPT, is a prompt-conditioned unidirectional language model, optimized for long sequence level prediction. More precisely, MarioGPT's architecture relies on a distilled, lightweight version of GPT2 [29] transformer architecture called DistilGPT2 [32] * . Encoding slices of Mario levels as strings, similar to the approach taken in Summerville and Mateas [38], we can fine-tune this distilled version of GPT2 on predicting next tokens in Mario levels. To generate levels, we concatenate a window of previous 50 columns into a single vector and feed them into MarioGPT.

Architecture : MarioGPT's architecture is the same as the DistilGPT2 architecture, except the cross attention weights are utilized for prompting. Even though DistilGPT2 supports context lengths up to size 1024, we limit our context lengths to 700, as we found increasing it did little to increase performance. In total, MarioGPT has 96 million parameters (86 million of the original DistilGPT2 parameters and 10 million from cross attention weights). We train MarioGPT for 50,000 steps, sampling 4 random slices of levels at each iteration and optimize the model using the Adam optimizer [20]. In total, MarioGPT sees 200,000 training samples. Because the model is relatively small, it can be trained using a single Nvidia GeForce RTX 2080 Ti GPU.

Prompting details : In order to incorporate prompt information, we fine-tune the attention layers' cross attention weights, as illustrated in Figure 2. Prompts are encoded through BART [25], a frozen pre-trained language model. Prompts are passed through the frozen language model and the hidden states from the forward pass are averaged into a single vector. This average hidden state is then used in the cross attention layers of the GPT2 architecture in combination with the actual level sequence being passed into the model. We represent our prompts as combinations of specific features along with keywords that correspond to quantiles (e.g. none, little, some, many). This allows us to easily generate level/prompt pairs by counting corresponding tile values. For more details on the prompts, see Section 6.2 in the Appendix.

In addition, it is possible to use synonyms for words. For example, changing 'many' to 'a lot' or 'a ton', produces similar results because the BART encoder can generalize well.

3.2 Open-Ended Mario Level Generation with Novelty Search

In the realm of PCG, it is important to not only generate levels with diverse physical features, but also levels that elicit a wide range of player behavior . When it comes to creating Mario levels, the focus is on the different paths a player can take to complete the level. This is often a challenge for many algorithms (such as [45, 8]) and requires the use of an external agent for evaluation. However, with MarioGPT, it is possible to generate diverse and controllable levels that approximate a realistic player path, reducing the need for an external agent and producing levels that are directly playable. To encourage diversity in generated levels, we integrate MarioGPT within a novelty search augmented genetic algorithm (NS-MarioGPT), where language-models play the role of mutation operators. As illustrated in Figure 3, NS-MarioGPT iteratively samples and mutates elite levels from an archive of generated levels.

Novelty Search : Mutated levels are only stored in the archive if they achieve a higher novelty score compared to the previous elites. The novelty score is measured as the mean distance between the behavioral characteristic vector of the levels and the behavioral characteristic vector of the K closest elements from the archive ( K -means). Our goal in level generation is to create paths that result in diverse player behavior, so we use predicted player paths as our basis for these behavior characteristics. More specifically, we are interested in the relative patterns of predicted paths. For instance, if a player character moves in a straight line on high elevated blocks, we want the path's representation to be close in behavior space to a path that moves straight in lower elevation. To achieve this, we represent the behavior characteristic as the normalized average of the predicted path's coordinates, allowing a smooth representation of paths (Figure 4). Thus the significance of a single block difference is reduced, making it harder for mutated levels to be added to the archive. This is desired because we don't want the archive to fill up with levels that only vary slightly from the existing levels in the archive. For all our novelty search experiments, we use a small neighborhood of size 4 , which results in a behavioral characteristic of dimension 100 . We initialize our archive with a small number of levels (30), as we found mutations are significant enough to generate a diverse set of levels without a big starting population.

0

20

40

60

80

100

0.00

0.25

0.50

0.75

1.00

Mutations: The LLM-based mutation operation introduced in this paper (Figure 3) transforms a randomly picked slice of a level (a slice between 40 -80 columns) with a new MarioGPT prediction, guided by a random prompt. By itself, MarioGPT is able, through mutations, to produce a variety of levels with varying agent paths. However, because MarioGPT is a unidirectional model, we cannot guarantee that the new generated path is consistent with the rest of the level. To further improve path consistency, we incorporate a fine-tuned mask prediction model (which we call MarioBert), based on the Bert architecture. The BERT language model [7] is a bidirectional LLM that shows impressive performance in the task of mask prediction, which is analogous to image in-painting. This ability is ideal for our use case, where MarioBert is used to inpaint its border region after the newly sampled slice, smoothly joining the mutated slice and the rest of level. This can be observed in the second step of the "Mutation process" part of Figure 3.

4 Experiments and Results

4.1 Tile Prediction Accuracy

To measure how proficient MarioGPT is in generating levels and because the majority of tiles in these levels are air tiles, we focus on comparing non-air tile prediction accuracy. We compare to baselines: LSTM, as proposed in Summerville and Mateas [38] and MarioGPT that is trained

from scratch (without using pretrained GPT2 weights), with results reported in Table 1. For all our baselines, we train for the same amount (200,000 samples). The results show that MarioGPT (using a pretrained GPT2 model) outperforms all other baselines with regards to tile prediction. In addition, training MarioGPT from scratch and training an adapter layer (a small multi layer network on top of the original prediction layer) results in models that performs worse than even the LSTM baseline (given the 200,000 training samples). These models were trained with minimal hyperparameter search, so their performance can likely be improved. However, as a tangential point, this shows a major benefit of fine-tuning pretrained models, which seem to require much less effort in regards to hyperparameters.

4.2 Measuring Playability of Levels

To test for playability, we deploy Robin Baumgarten's A* agent [43, 19] in 250 generated levels * . The reason for choosing Robin Baumgarten's A* agent for measuring playability comes from its performance on the 2009 Mario AI competition, where it beat handcrafted controllers and even simple evolved neural networks on getting the furthest in an infinite-level setting, as well as solving a corpus of levels [43]. We find that 88.4% of all MarioGPT-generated levels can be completed by the agent, and are therefore considered playable (compared to the best baseline, the LSTM, which achieves around 31% solvable levels). Moreover, we find that only one of the successful levels needed a retry with the A* agent. We further test whether the path generated by the model matches that of the A* agent to assess their feasibility. Table 2 shows the mean absolute error (MAE) between suggested and actual agent path for playable and not playable levels respectively. We see that for playable levels, the MAE between the path generated by the model and the actually taken path by the agent is 1 . 15 tiles, i.e. paths are on average about 1 tile apart. For the non-playable levels, this average difference of taken paths is significantly higher with 4 . 56 tiles. Thus, we can conclude that in playable levels, the agent mostly takes a similar path as the one generated by the model. The significantly higher MAE of 4 . 56 in non-playable levels on the other hand indicates that the path generated by the models in these cases may not be feasible for the agent.

Table 2: Mean average error (MAE) between paths suggested by model and Baumgarten's A* agent. Results are averaged over 5 runs per level to account for minor stochastic variation in agent simulation. MAEs are computed between y coordinates of path trajectories for every point on the x axis (which goes across the level) both trajectories have visited.

Considering the MAE of 1 . 56 tiles for paths in all levels, we can conclude that in the majority of the cases, the path generated by the model is similar to the path taken by an actual agent, and having the model generate a path through the level jointly with the level is an effective approach to obtain high-quality levels in terms of playability.

To investigate the quality of the generated paths further, we visualize the paths with the most, least and median overlap (i.e. the levels corresponding to the maximum, minimum and median values for the mean absolute error in height) as well as two interesting handpicked examples in Figure 5.

Figures 5d and 5e show that paths generated by MarioGPT tend to have more airtime than Baumgarten's agent in the sense that they only weakly take into account "gravity". This result may be attributed to the nature of the path annotations in the models training set. In Summerville et al. [40], the authors use an A* path solver to find a path through the level, while an actual agent, such as the one we used for comparison here, is more strongly bound by game physics (especially "gravity") and has to avoid enemies in the level. A second reason for non-playable levels can be seen in Figure 5c: Baumgarten's agent is spawned in a tight space from which it can not escape, while the model has generated a path that traverses beyond the actual level, again a path that would likely be suggested by a solver. We argue that these issues can in part be attributed to the paths in the training data stemming from a solver rather than an actual agent, and could be alleviated in future work by annotating the training data with the trajectories of actual agents.

(a) Level with minimum MAE between

paths

(b) Level with median MAE between

paths

(c) Level with maximum MAE

between paths

(d) Playable level with interesting path

trajectories

(e) Non-playable level with interesting

path trajectories

P

AAAA

A

A

AA

A

A

A

PPPPP

AAAA

A

P

AAA

P

A

AAAAAAAAAAA

中

AA

AA

AA

A

AAAAAAAA

H

点点

AA

号

BAAAA

AAA

AAA

4.3 Is MarioGPT memorizing?

(a) Sample with temp 1.0

(b) Closest dataset sample to (a)

(c) Sample with temp 1.7

(d) Closest dataset sample to (c)

(e) Sample with temp 2.8

(f) Closest dataset sample to (e)

Memorization dynamics in LLMs remain an open problem when training transformer architectures [41, 5, 16]. While LLMs are incredibly powerful, they can sometimes overfit extremely and end up regurgitating training data. One popular way to alleviate this issue is to add some randomness in predictions in the form a tunable "temperature" parameter [17]. To evaluate whether MarioGPT is generating levels that are identical to the training set, we sample with different temperature parameters

and compare them the closest level in the training dataset. From Figure 6, we can see that increasing temperature results in samples that are more diverse, but lack quality. In our case, when generating levels we use a temperature of 2.4-2.7, as it can generate diverse samples while still retaining some quality. There are many possible improvements to explore in the future. One common way is to simply increase the richness of the dataset. The more samples the model has access to, the less likely it is to overfit. We could also improve MarioGPT's sampling abilities by introducing different search methods other than sampling with temperature, such as constrained beam search [6] and dataset augmented search [16], to increase diversity while preserving more quality.

4.4 Guided Level Generation via Prompting

Through simple prompting, we are able to guide MarioGPT towards controllable and diverse level generation. We empirically evaluate the prompting ability of MarioGPT by generating 1,000 samples with various combinations of prompts, and check how accurate the generated levels are to the prompt descriptions. The results suggest that MarioGPT can generate levels that match their given prompts most of the time (Table 3). MarioGPT is the most accurate with blocks and the least accurate with enemies. This is expected because there are fewer total tiles of enemies, while there are many more block tiles observed during training.

We visually evaluated the system, displaying selected prompt-conditioned generations in Figure 1. In addition, we evaluate the importance of the keywords in the prompt by comparing the distribution of the number of pipes between levels generated with random prompts without pipes-related commands (e.g. "some enemies, some blocks, high elevation") versus random prompts with pipes-related commands (e.g. "little pipes, some enemies, some blocks, high elevation"). The distribution without pipe prompts is scattered, while the ones with pipe prompts result in distributions with peaks, indicating that the keywords actually have an effect on the level generated (see Figure 11 in the Appendix).

MarioGPT is also able to generate levels from text descriptions that are not represented in the dataset. For instance, Figure 1e shows a successful approximation of the prompt, "many pipes, no enemies, many blocks", with a slight inaccuracy in that it has 1 less pipe (5 pipes is considered "many", while 4 are present). However, this is not always the case, as can be seen in Figure 1f, where the model, prompted by "many pipes, no enemies, some blocks", generates a level with the correct number of pipes and blocks but generates too many enemies. In future work, we hope to explore more ways to incorporate prompt importance, such as editing levels with tiles to create more samples or prompt tuning [18].

4.5 Generating Diverse Levels with Novelty Search

Through the combination of an LLM (Section 3.1) and novelty search (Section 3.2), we are able to continuously generate diverse levels in an open-ended fashion. Specifically, NS-MarioGPT is able to generate a collection of levels with a diverse set of predicted agent paths. We project the archive as a set of 2D embeddings in Figure 7 and darken the embedding points that are added later in the process. We can see that the levels are increasingly filling up empty spots in the embedding space. We also compare the distribution of levels generated by novelty search to levels generated by random prompts in Figure 8a. Visually, we can see that the levels generated by novelty search are more spread out in t-SNE space and the sampled ones, indicating that they are more diverse. Finally, we have also evaluated the playability of levels generated by novelty search, and find that the majority are solvable and non-playable levels are not clustered but rather scattered across t-SNE space. This indicates that there is no trade-off between path diversity and the ability to generate solvable levels. Figure 8b shows the corresponding results.

Figure 9 displays all the overlayed predicted paths (in a level grid) as more and more levels get added to the archive during novelty search. Similar as in Figure 7, we can see that over time, the space

20

10

0

-10

-20

-20

-10

0

10

20

(a) Novelty search vs. random sampling

(b) Playable vs. non-playable levels

novelty-search

20

O

solvable

10

random-sampling

15

O

unsolvable

10

5

5

0

0

-5

-10

-15

-10

-20

-10

-5

0

5

10

-20

-10

0

10

20

of possible predicted agent paths gets filled, as increasingly diverse levels are mutated and added to the archive. As more levels are added to the archive, more and more of the tiles / empty space in the grid are being filled up, indicating that NS-MarioGPT is discovering a variety of levels that produce diverse paths. Concretely, we found that after 300 levels are added to the archive, around 78% of the possible coordinates are filled up. However, there are still many overlapping paths in the archive, meaning that similar paths are still being added to the archive. This is an issue that could be improved by using more related time series distance metrics that account for patterns in a path [11].

Levels with the highest and lowest novelty score from the archive are also shown in Figure 10. The level with the lowest novelty, shown in Figure 10a, has a path that is much more common in the archive, which can be seen by its almost identical look compared to the 2nd lowest in Figure 10c. The levels with higher novelty, Figure 10b and Figure 10d, have more unique patterns, but share a similar

0

20

40

60

80

100

0

2

4

6

8

10

12

14

(a) Archive after 10 levels

0

20

40

60

80

100

0

2

4

6

8

10

12

14

(b) 50 levels

0

20

40

60

80

100

0

2

4

6

8

10

12

14

(c) 150 levels

0

20

40

60

80

100

0

2

4

6

8

10

12

14

(d) 300 levels

(a) Level with lowest novelty score

(b) Level with highest novelty score

(c) Level with 2nd lowest novelty score

(d) Level with 2nd highest novelty score

LLE

0.000

0.0.0.0

pattern towards the end. This indicates that one was created by mutating the other. We also found that the diversity starts to plateau after around 350-400 generations. However, this is very sensitive to the behavior characteristic (the smoothed predicted path of the level), so it may be different for other behavior characteristics.

While NS-MarioGPT is still able to discover many diverse levels through its simple mutation process, more complex functions could also be explored. For instance, crossover, a common mutation utilized in many genetic algorithms, would increase mutation diversity which can lead to more diverse levels.

5 Conclusion

Here we introduced MarioGPT, a fine-tuned GPT2 LLM that can not only generate diverse levels, but can guide its generation via a language prompt. This ability is useful in the field of Procedural Content Generation, where balancing controllable and diverse generation is a difficult task. We showed that MarioGPT is also able to (1) predict player interaction in generated levels, (2) generate diverse and playable environments, and (3) reduce the need for expensive external agent interactions (MarioGPT can generate playable levels approximately 88% of the time). Additionally, when combined with a diversity-driven algorithm like novelty search, MarioGPT can generate open-ended and functional content. While MarioGPT can generate diverse content, it is still limited. It can sometimes not follow prompt instruction and it is also subject to memorizing the dataset. We hope to improve these limitations by introducing richer data to the system.

One major benefit of re-using an existing LLM is that we can take advantage of all the research and improvements that have gone into these architectures. We plan to leverage the scalability of LLMs and train MarioGPT on bigger, more detailed annotated levels. Also, we are particularly excited about incorporating human feedback into the level generation process through reinforcement learning from human feedback (RLHF) [47]. The ability to fine-tune these models on human feedback allows users to continually tune their generated levels towards desired characteristics. Ultimately, we hope that MarioGPT opens the door to more controllable and diverse PCG systems.

References

Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014.

Michael Beukman, Christopher W Cleghorn, and Steven James. Procedural content generation using neuroevolution and novelty search for diverse video game levels. In Proceedings of the Genetic and Evolutionary Computation Conference , GECCO '22, page 1028-1037, New York, NY, USA, 2022. Association for Computing Machinery.

Herbie Bradley, Honglu Fan, Harry Saini, Reshinth Adithyan, Shivanshu Purohit, and Joel Lehman. Diff models - a new way to edit code. CarperAI Blog , Jan 2023.

Cameron Browne and Frederic Maire. Evolutionary game design. IEEE Transactions on Computational Intelligence and AI in Games , 2(1):1-16, 2010.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2022.

Katsuki Chousa and Makoto Morishita. Input augmentation improves constrained beam search for neural machine translation: Ntt at wat 2021, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.

Matthew C. Fontaine, Ruilin Liu, Ahmed Khalifa, Jignesh Modi, Julian Togelius, Amy K. Hoover, and Stefanos Nikolaidis. Illuminating Mario scenes in the latent space of a generative adversarial network, 2020.

Matthew C. Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K. Hoover. Covariance matrix adaptation for the rapid illumination of behavior space. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference . ACM, jun 2020.

Edoardo Giacomello, Pier Luca Lanzi, and Daniele Loiacono. Doom level generation using generative adversarial networks. In 2018 IEEE Games, Entertainment, Media Conference (GEM) , pages 316-323. IEEE, 2018.

Omer Gold and Micha Sharir. Dynamic time warping and geometric edit distance: Breaking the quadratic barrier, 2016.

Miguel González-Duque, Rasmus Berg Palm, Søren Hauberg, and Sebastian Risi. Mario plays on a manifold: Generating functional content in latent space through differential geometry, 2022.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Nikolaus Hansen. The CMA evolution strategy: A tutorial, 2016. hal-01297037v2f.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A. Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy, 2022.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2016.

Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know?, 2019.

Ahmed. Khalifa. The mario AI framework. https://github.com/amidos2006/ Mario-AI-Framework , 2009.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.

Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. Evolution through large models, 2022.

Joel Lehman and Kenneth O. Stanley. Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary Computation , 19(2):189-223, 06 2011.

Joel Lehman, Kenneth O Stanley, et al. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE , pages 329-336, 2008.

Joel Lehman, Kenneth O. Stanley, and Lisa Soros. Open-endedness: The last grand challenge you've never heard of. https://www.oreilly.com/radar/ open-endedness-the-last-grand-challenge-youve-never-heard-of/ . Accessed: 2017-12-19.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019.

Antonios Liapis, Georgios N Yannakakis, and Julian Togelius. Constrained novelty search: A study on game content generation. Evolutionary computation , 23(1):101-129, 2015.

Jialin Liu, Sam Snodgrass, Ahmed Khalifa, Sebastian Risi, Georgios N Yannakakis, and Julian Togelius. Deep learning for procedural content generation. Neural Computing and Applications , 33(1):19-37, 2021.

Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites, 2015.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence , 2(8):428-436, 2020.

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter, 2019.

Anurag Sarkar and Seth Cooper. Dungeon and platformer level blending and generation using conditional vaes. In Proceedings of the IEEE Conference on Games (CoG) , 2021.

Anurag Sarkar and Seth Cooper. Generating and blending game levels via quality-diversity in the latent space of a variational autoencoder. In Proceedings of the Foundations of Digital Games , 2021.

Anurag Sarkar, Zhihan Yang, and Seth Cooper. Conditional level generation and game blending. In Proceedings of the Experimental AI in Games (EXAG) Workshop at AIIDE , 2020.

Noor Shaker, Julian Togelius, and Mark J Nelson. Procedural content generation in games . Springer, 2016.

Adam M Smith and Michael Mateas. Answer set programming for procedural content generation: A design space approach. IEEE Transactions on Computational Intelligence and AI in Games , 3(3):187-200, 2011.

Adam Summerville and Michael Mateas. Super Mario as a string: Platformer level generation via lstms, 2016.

Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmgård, Amy K Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. Procedural content generation via machine learning (PCGML). IEEE Transactions on Games , 10(3):257-270, 2018.

Adam James Summerville, Sam Snodgrass, Michael Mateas, and Santiago Ontañón. The VGLC: The video game level corpus, 2016.

Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models, 2022.

Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. In Proceedings of the 18th International Conference on the Foundations of Digital Games , pages 1-8, 2023.

Julian Togelius, Sergey Karakovskiy, and Robin Baumgarten. The 2009 Mario AI competition. In IEEE Congress on Evolutionary Computation , pages 1-8, 2010.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Vanessa Volz, Jacob Schrum, Jialin Liu, Simon M. Lucas, Adam Smith, and Sebastian Risi. Evolving Mario levels in the latent space of a deep convolutional generative adversarial network, 2018.

Georgios N Yannakakis and Julian Togelius. Artificial intelligence and games , volume 2. Springer, 2018.

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019.

6 Appendix

6.1 Dataset Details

6.2 Constructing Prompts

Prompts are represented as combinations of specific features (e.g. pipes, enemies, blocks, elevation) alongside quantitative keywords:

{ no, little, some, many, [0-1000] } pipes

{ no, little, some, many, [0-1000] } enemies

{ little, some, many, [0-1000] } blocks

{ low, high } elevation

As an example, " no pipes, many enemies, low elevation " or " many pipes, many enemies, many blocks " are both possible prompts. The keywords "no", "little", "some", "many" are calculated from quantiles of the corresponding count within a 50 column window (Table 5). The "low" and "high" elevation are determined from the height of the highest unbreakable blocks in a segment of the level.

(a) "Random prompts

without pipe prompt"

(b) "little pipes +

random prompt"

(c) "some pipes +

random prompt"

(d) "many pipes +

random prompt"

0.25

0.5

0.4

0.40

0.35

0.20

0.4

0.30

0.15

0.3

0.3

0.25

0.20

0.10

0.2

0.2

0.15

0.05

0.1

0.1

0.10

0.05

0.00

0.0

0.D

0.00

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MarioGPT with novelty search, a method that can generate new Super Mario Bros levels. MarioGPT is finetuned from GPT-2 to generate levels. The novelty search is used to get novel levels by randomly selecting a generated level, mutating it, and then filtering it based on a novelty criterion: the mean distance between the mutated level and the K closest elements from the existing levels (K-means). Experimental results show that MarioGPT has a better level of reconstruction accuracy than LSTM, and 88.33% of the generated levels can be played by Robin Baumgarten’s A* agent.\n\nSTRENGTHS:\n1. The idea of generating Super Mario Bros levels is interesting and might be helpful for both the reinforcement learning community and the game community.\n\n2. The result shows that the finetuned GPT has the ability to generate decent Super Mario Bros levels, which is insightful.\n\nWEAKNESSES:\n1. The evaluation criteria do not seem to be appropriate. For example, \"playable by an A* agent\" is not a convincing indicator that can measure the quality of generated levels. Actually, \"playable by an A* agent\" primarily measures the simplicity of the generated levels rather than the quality of the levels. Moreover, in section 4.4, \"Guided Level Generation via Prompting\", the \"empirical evaluation\" is not objective, and lacks baseline (like LSTM) comparisons. Since this paper claims that the generated levels are text-controllable, it would be better to provide a more explicit demonstration of controllability.\n\n2. The baseline (LSTM) in section 4.1 is weak. Table 1 shows that the pre-trained GPT-2 is very important in the generation process. Then, how does a GPT-2 without finetune perform? How does GPT-3/3.5/4 perform? Also, I doubt the conclusion that LSTM is not promptable since it can be integrated with a text encoder.\n\n3. Although the idea is interesting, the generated levels are simple. 88.33% of the generated levels can be played by an A* agent, and only horizontal movements are included (Mario cannot crawl into the pipes and there is no hidden brick, making the generated levels even simpler than traditional Super Mario Level 1-1). This greatly reduces the significance of this work, both in the fields of artificial intelligence and gaming industry.\n\nQUESTIONS:\n1. How to control detailed level settings like the initial moving direction of the enemy?\n\n2. Can MarioGPT be tuned on Super Mario Makers which consists of thousands of novel levels contributed by the gaming community?\n\n3. Why do from-scratch-MarioGPT and adapter-MarioGPT perform worse than LSTM?\n\n4. Mario GPT generates trajectories as well as game contexts. What about generating trajectories first, then selecting a trajectory and keeping it unchanged, and then generating, mutating, and filtering the game contexts? Will this improve the “playable rate” of the generated games?\n\nLIMITATIONS:\nSee above questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel idea of using LLMs to perform Procedural Content Generation. It addresses several challenges including the diversity of the generated environments, the existence of a feasible solution (playability) and generating with language guidance. It proposes two key algorithms.\n\n1. MarioGPT - Fine-tuning on DistilGPT2 so that it can predict the next slice of the game. It also allows prompting - the text conditioning is done by using BART to encode text prompts then feeding the average hidden states into the cross attention.\n\n2. Novelty search for open-endedness - Using MarioGPT as a mutation operator in the novelty search evolutionary loop, prompting the MarioGPT with random prompts and generate candidates, and retain the candidates that pass the novelty criteria to add to the archive. Then do masked token prediction (impainting) to ensure that the level generated has a feasible path solution.\n\nThis paper conducts experiments that demonstrate (1) the high token prediction accuracy of MarioGPT, (2) good playability - i.e. most generated levels have feasible path solutions, (3) using a suitable temperature to control the trade-off between diversity and quality, (4) the accuracy in following the prompt and (5) the diversity of generated levels (in terms of the predicted path) achieved by novelty search.\n\nSTRENGTHS:\n1. This paper proposes a novel framework for text-guided PCG: having a text-conditioning, generative model that predicts new levels, then having a novelty search algorithm reject the levels that are too similar to existing ones.\n\n1. This paper is written clearly. The methods are well motivated. The experiments are well-designed with detailed analysis and discussion.\n\nWEAKNESSES:\n1. Experiments are lacking in measuring “controllability”. The prompts are limited to a small set. The capability of following instructions on a more semantic level are not measured, such as the scene layout. and the spatial, functional, and semantic relationships between objects. (e.g. how tall the pipes are, “there is an enemy to the right of a pipe”).\n\n2. The technique of first auto-regressively generating a scene, then using “inpainting” to ensure a feasible solution seems to be pretty tied to this specific type of games where the scene moves linearly in 1 dimension (i.e. the character can only move forward, not backward), and hard to generalize to other PGC scenarios. For example, in order to generalize to the cases where the character can move both forward and backward, non-trivial changes are needed.\n\nQUESTIONS:\n1. In section 4.4 \"Guided Level Generation via Prompting\", under what sampling temperature are these results measured? I am curious how / whether this randomness affects how accurately the model follows the prompts as well. The authors mentioned that randomness helps with diversity - does this sacrifice accuracy of following prompts?\n\n2. On a relevant note, in section 4.3 the authors mentioned that the temperature controls the tradeoff between diversity and quality. Could you elaborate what the definition of \"quality\" is?\n\n3. In section 4.5, at what point does the “diversity” start to plateau? Are there baseline studies regarding the diversity of generated paths?\n\n4. In section 4.2, are there quantitative baselines regarding the playability of generated environments? Could you share intuition regarding the trade-offs between playability versus diversity of the generated environments?\n\nLIMITATIONS:\nAs mentioned above, (1) the measurement of capability to understand the instructions semantically and (2) the generalizability of this approach beyond this type of games remains to be further studied.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work the authors fine tune a DistilGPT2 model to produce diverse, largely playable Mario levels. They incorporate a novelty search to favor diversity in levels, and also explore conditioning level generation on natural language user prompts. Their novelty search proceeds as follows: Given an initial set of levels, a level is sampled from the set and randomly mutated. The mutation involves deleting a random contiguous segment of the level and sampling a new one in its place from their model, then using a fine tuned BERT model to do inpainting on the border regions to smooth the transition between this and the rest of the level. The mutated level is accepted if it is sufficiently different from the other levels, using a distance metric comparing the player path through the level.\n\nSTRENGTHS:\n- The use of LLMs for game level generation certainly makes sense, and I think it's valuable to publish work in this area for the NeurIPS community to see and build on. The technical approach taken here is original, to my knowledge.\n- The combination of genetic algorithms with LLMs is interesting and novel, and I found the use of BERT for inpainting to smooth out the transition between the mutated segment and the rest of the level to be a particular clever approach.\n- The paper also does a great job of visualizing the levels produced by the model which is very helpful in getting a feel for the sorts of outputs that their model produces. The playability and temperature analyses were quite interesting as well.\n\nWEAKNESSES:\n- Section 4.5: It's hard to tell that the novelty search is actually helping, without a comparison to an ablated version that doesn't do the novelty search. In particular it's hard to tell if the darker points in Figure 9 seem to be filling in the space in between because of the novelty search, or if it's just that there are more sampled points so they happen to fill in some of the space in between the earlier points. Likewise with the rightmost graphs in figure 7 looking denser than the leftmost ones – taking more samples will always add some density, so in order to really conclude something you need a comparison to an ablation. Showing something like 300 levels from the novelty search plotted on the same tSNE as 300 levels from a version without novelty search (eg as a different color) would make it much clearer that there's actually an improvement in diversity (and likewise some analogous experiment for the other figure).\n* In a similar vein, in Table 3 it's hard to know how much the prompt is helping without seeing the scores without prompting (for an idea of what scores a fairly random approach would achieve); though I think is a less major issue.\n* After seeing the LSTM from prior work used in the Table 1 comparison, I was surprised that LSTM comparisons weren't given for any of the future evaluations, which would be helpful for understanding how much better this LLM setup is than prior work. For example, we learn that 88% of levels are solvable, but how does this compare to the LSTM approach? Likewise in the diversity analysis, a naive LSTM baseline would be interesting (but no need for all the novelty search machinery). This would just help me get a sense of where this work stands compared to prior work on these metrics that are being measured. That said, I wouldn't reject this paper on that basis – 88% playable is a great statistic regardless of what you compare it to, and I don't actually expect that the LSTM would make particularly good levels, but having the full comparison would strengthen the evaluation.\n- As a last, minor point, It's surprising (in a way that makes me feel that an explanation is needed) that such poor performance is obtained by from-scratch-MarioGPT on the tile prediction task given it was trained on 200,000 datapoints. The \"tile accuracy\" is presumably a measure of how well the system does next-tile prediction (analogous to next-token prediction in language modeling, and presumably this is the objective that all these models are trained on). I'd imagine something as simple as \"always predict an air tile\" since most levels look like they're more than half air, or \"predict the same tile that you saw to the left\" would do quite well, and the inability to pick up on even simple trends like this after 200,000 datapoints is surprising to me. Perhaps there is additional complexity I'm missing?\n\nOverall, this is an interesting paper that I end up borderline reject on because of the evaluation points mentioned above, but I don't have a huge amount of prior experience in this area or what typical evaluations are for PCG, so I am open to revising through discussion.\n\nQUESTIONS:\n* Relating to the weakness point above, in Table 1 I'm a little surprised that everything except MarioGPT does so poorly (LSTM does okay at least though). It'd be helpful to have a little intuition for why this is so hard?\n- The concurrent work cited by this paper (Todd et al 2023) finds that pretraining (vs from-scratch) GPT2 has little to no impact on a range of metrics. That is of course a somewhat different task and setup, but I'm curious if you have thoughts on why you seem to find the opposite result in this domain?\n- What embedding is used for the tSNE? Is it something like the last layer of MarioGPT?\n- Do you guarantee that each tile is parsed as a separate token? Without this it seems like the network could potentially get confused about reasoning about lengths/distances, like if one token represented two blocks while another represented just one. Seems like it works either way, but I'm curious if this is something you're doing or might look at.\n- The language conditioning is cool! Of course, a natural extension of this would be moving towards level generation from more unstructured natural language. For example, flexibility with synonyms to \"many\" or more complex things like \"a series of narrow pillars with an enemy on top of each one\". That sounds like enough for a separate paper but I'm personally curious if you've done any preliminary work in that direction or what you think of it. To be clear, I don't think this is needed in this submission I just think that the work is cool and am curious!\n\nLIMITATIONS:\nI think that this paper does an adequate job of presenting its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This submission proposes MarioGPT, which aims to generate diverse-and-playable Mario levels with LLM (GPT-2) through language prompts. The input and output of MarioGPT are level representations, not natural language. Human prompts are involved by incorporating the cross-attention layer into the LLM. A Novelty Search algorithm is further proposed to increase the diversity of generated levels.

STRENGTHS:
+ Leveraging LLM into Procedural Content Generation (PCG) is a natural idea. This submission successfully accomplishes this, and the proposed MarioGPT could be really usable.
+ This submission conducts a comprehensive study of various aspects of the proposed method, including the tile/path prediction accuracy, the playability of generated levels, whether MarioGPT is memorizing the training dataset, guidability by prompts. The overall experimental parts are convincing. The authors also analyze some current limitations and provide feasible solutions to them.
+ The proposed Novelty Search algorithm successfully improves the diversity of agent paths.

WEAKNESSES:
- Overall, I think Procedural Content Generation (PCG) with LLM is 'quite' an obvious idea, making me doubt this submission's 'technical' contribution to the community. I did not see any other obvious weaknesses.

QUESTIONS:
1. In the field of PCG, is there any game other than Super Mario? What is the generalization ability of the proposed method to other games (if there is any)?

LIMITATIONS:
Same as in the Questions part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07