SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: A* vs. MarioGPT generated paths. Levels with (a) minimum (0.02), (a) median (0.89) and (a) maximum (11.0) mean absolute error (MAE) between trajectory of actual A* agent (denoted as A), and model suggestion (denoted as P), as well as interesting hand-picked examples. Positions where both trajectories overlap are marked with *. Paths suggested by the model generally tend to have more airtime than the A* agent (d, e), likely due to game physics not being accounted for in the original path annotations of the training data.
(a) Sample with temp 1.0
(b) Closest dataset sample to (a)
(c) Sample with temp 1.7
(e) Sample with temp 2.8
Figure 6: Generated levels vs closest in dataset. Temperature of 1.0 ends up spitting out almost exactly what is in the dataset, while increasing temperature improves sample diversity.
(d) Closest dataset sample to (c)
(f) Closest dataset sample to (e)
Memorization dynamics in LLMs remain an open problem when training transformer architectures [41, 5, 16]. While LLMs are incredibly powerful, they can sometimes overfit extremely and end up regurgitating training data. One popular way to alleviate this issue is to add some randomness in predictions in the form a tunable "temperature" parameter [17]. To evaluate whether MarioGPT is generating levels that are identical to the training set, we sample with different temperature parameters
and compare them the closest level in the training dataset. From Figure 6, we can see that increasing temperature results in samples that are more diverse, but lack quality. In our case, when generating levels we use a temperature of 2.4-2.7, as it can generate diverse samples while still retaining some quality. There are many possible improvements to explore in the future. One common way is to simply increase the richness of the dataset. The more samples the model has access to, the less likely it is to overfit. We could also improve MarioGPT's sampling abilities by introducing different search methods other than sampling with temperature, such as constrained beam search [6] and dataset augmented search [16], to increase diversity while preserving more quality.
4.4 Guided Level Generation via Prompting
Through simple prompting, we are able to guide MarioGPT towards controllable and diverse level generation. We empirically evaluate the prompting ability of MarioGPT by generating 1,000 samples with various combinations of prompts, and check how accurate the generated levels are to the prompt descriptions. The results suggest that MarioGPT can generate levels that match their given prompts most of the time (Table 3). MarioGPT is the most accurate with blocks and the least accurate with enemies. This is expected because there are fewer total tiles of enemies, while there are many more block tiles observed during training.
Table 3: Prompt vs actual description accuracy
pipes	enemies	blocks	elevation
81%	68%	92%	76%
We visually evaluated the system, displaying selected prompt-conditioned generations in Figure 1. In addition, we evaluate the importance of the keywords in the prompt by comparing the distribution of the number of pipes between levels generated with random prompts without pipes-related commands (e.g. "some enemies, some blocks, high elevation") versus random prompts with pipes-related commands (e.g. "little pipes, some enemies, some blocks, high elevation"). The distribution without pipe prompts is scattered, while the ones with pipe prompts result in distributions with peaks, indicating that the keywords actually have an effect on the level generated (see Figure 11 in the Appendix).
MarioGPT is also able to generate levels from text descriptions that are not represented in the dataset. For instance, Figure 1e shows a successful approximation of the prompt, "many pipes, no enemies, many blocks", with a slight inaccuracy in that it has 1 less pipe (5 pipes is considered "many", while 4 are present). However, this is not always the case, as can be seen in Figure 1f, where the model, prompted by "many pipes, no enemies, some blocks", generates a level with the correct number of pipes and blocks but generates too many enemies. In future work, we hope to explore more ways to incorporate prompt importance, such as editing levels with tiles to create more samples or prompt tuning [18].
4.5 Generating Diverse Levels with Novelty Search
Through the combination of an LLM (Section 3.1) and novelty search (Section 3.2), we are able to continuously generate diverse levels in an open-ended fashion. Specifically, NS-MarioGPT is able to generate a collection of levels with a diverse set of predicted agent paths. We project the archive as a set of 2D embeddings in Figure 7 and darken the embedding points that are added later in the process. We can see that the levels are increasingly filling up empty spots in the embedding space. We also compare the distribution of levels generated by novelty search to levels generated by random prompts in Figure 8a. Visually, we can see that the levels generated by novelty search are more spread out in t-SNE space and the sampled ones, indicating that they are more diverse. Finally, we have also evaluated the playability of levels generated by novelty search, and find that the majority are solvable and non-playable levels are not clustered but rather scattered across t-SNE space. This indicates that there is no trade-off between path diversity and the ability to generate solvable levels. Figure 8b shows the corresponding results.
Figure 9 displays all the overlayed predicted paths (in a level grid) as more and more levels get added to the archive during novelty search. Similar as in Figure 7, we can see that over time, the space
Figure 7: t-SNE of the levels in the archive. t-SNE embeddings are computed from the behavioral characteristic. Darker points indicate more recently added elements. Although novelty search is using the behavioral characteristics of the player paths, the levels also demonstrate visual novelty.
(a) Novelty search vs. random sampling
(b) Playable vs. non-playable levels
Figure 8: Comparing exploration for novelty search vs. random sampling and playable vs. nonplayable levels. (a) t-SNE of both the embeddings of novelty-search levels and levels generated with random prompts. The visualization suggests that novelty search enables a much wider exploration of the space of levels. (b) Unsolvable levels are not clustered together but instead scattered across the t-SNE space. This distribution indicates that there is no correlation between the diversity of levels and their solvability.
of possible predicted agent paths gets filled, as increasingly diverse levels are mutated and added to the archive. As more levels are added to the archive, more and more of the tiles / empty space in the grid are being filled up, indicating that NS-MarioGPT is discovering a variety of levels that produce diverse paths. Concretely, we found that after 300 levels are added to the archive, around  $78\%$  of the possible coordinates are filled up. However, there are still many overlapping paths in the archive, meaning that similar paths are still being added to the archive. This is an issue that could be improved by using more related time series distance metrics that account for patterns in a path [11].
Levels with the highest and lowest novelty score from the archive are also shown in Figure 10. The level with the lowest novelty, shown in Figure 10a, has a path that is much more common in the archive, which can be seen by its almost identical look compared to the 2nd lowest in Figure 10c. The levels with higher novelty, Figure 10b and Figure 10d, have more unique patterns, but share a similar
(a) Archive after 10 levels
(b) 50 levels
(c) 150 levels
(d) 300 levels
Figure 9: Generated path populations during novelty search. Each line is a path through a level. Over time, NS-MarioGPT fills more and more of the space of all possible paths.
(b) Level with highest novelty score
(a) Level with lowest novelty score
(c) Level with 2nd lowest novelty score
Figure 10: Comparison of most and least novel levels in the archive. The two least novel levels are very similar to each other, while the most novel levels have more distinct path patterns.
(d) Level with 2nd highest novelty score
pattern towards the end. This indicates that one was created by mutating the other. We also found that the diversity starts to plateau after around 350-400 generations. However, this is very sensitive to the behavior characteristic (the smoothed predicted path of the level), so it may be different for other behavior characteristics.
While NS-MarioGPT is still able to discover many diverse levels through its simple mutation process, more complex functions could also be explored. For instance, crossover, a common mutation utilized in many genetic algorithms, would increase mutation diversity which can lead to more diverse levels.
Here we introduced MarioGPT, a fine-tuned GPT2 LLM that can not only generate diverse levels, but can guide its generation via a language prompt. This ability is useful in the field of Procedural Content Generation, where balancing controllable and diverse generation is a difficult task. We showed that MarioGPT is also able to (1) predict player interaction in generated levels, (2) generate diverse and playable environments, and (3) reduce the need for expensive external agent interactions (MarioGPT can generate playable levels approximately  $88\%$  of the time). Additionally, when combined with a diversity-driven algorithm like novelty search, MarioGPT can generate open-ended and functional content. While MarioGPT can generate diverse content, it is still limited. It can sometimes not follow prompt instruction and it is also subject to memorizing the dataset. We hope to improve these limitations by introducing richer data to the system.
One major benefit of re-using an existing LLM is that we can take advantage of all the research and improvements that have gone into these architectures. We plan to leverage the scalability of LLMs and train MarioGPT on bigger, more detailed annotated levels. Also, we are particularly excited about incorporating human feedback into the level generation process through reinforcement learning from human feedback (RLHF) [47]. The ability to fine-tune these models on human feedback allows users to continually tune their generated levels towards desired characteristics. Ultimately, we hope that MarioGPT opens the door to more controllable and diverse PCG systems.
This project was supported by a DFF-Research Project1 grant (9131- 00042B) and a European Research Council (ERC) grant (GA no. 101045094, project "GROW-AI").
[40] Adam James Summerville, Sam Snodgrass, Michael Mateas, and Santiago Ontañón. The VGLC: The video game level corpus, 2016.
[41] Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models, 2022.
[42] Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. In Proceedings of the 18th International Conference on the Foundations of Digital Games, pages 1-8, 2023.
[43] Julian Togelius, Sergey Karakovskiy, and Robin Baumgarten. The 2009 Mario AI competition. In IEEE Congress on Evolutionary Computation, pages 1-8, 2010.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[45] Vanessa Volz, Jacob Schrum, Jialin Liu, Simon M. Lucas, Adam Smith, and Sebastian Risi. Evolving Mario levels in the latent space of a deep convolutional generative adversarial network, 2018.
[46] Georgios N Yannakakis and Julian Togelius. Artificial intelligence and games, volume 2. Springer, 2018.
[47] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019.
6.1 Dataset Details
Table 4: Unique Mario tiles
Tile Type	Symbol	Visualization
Empty	-	■
Unbreakable	X	■
Breakable	S	■
Question Block	? / Q	■
Coin	o	■
Enemy	E	■
Left pipe top	<	■
Right pipe top	>	■
Left pipe lower	[	■
Right pipe lower	]	■
Cannon Top	B	■
Cannon Body	b	■
Path	x	■
6.2 Constructing Prompts
Prompts are represented as combinations of specific features (e.g. pipes, enemies, blocks, elevation) alongside quantitative keywords:
- { no, little, some, many, [0-1000]} pipes
- { no, little, some, many, [0-1000] } enemies
- { little, some, many, [0-1000]} blocks
-  $\{low, high\}$  elevation
As an example, "no pipes, many enemies, low elevation" or "many pipes, many enemies, many blocks" are both possible prompts. The keywords "no", "little", "some", "many" are calculated from quantiles of the corresponding count within a 50 column window (Table 5). The "low" and "high" elevation are determined from the height of the highest unbreakable blocks in a segment of the level.
Table 5: Prompt Quantiles and corresponding counts within a 50 column window
tile	no	little	some	many
pipes	0	1	2	5
enemies	0	1	3	7
blocks	0	50	75	176
(a) "Random prompts without pipe prompt"
Figure 11: Effect of prompt conditioning. Comparison of the distribution of the number of pipes between levels generated with random prompts without pipes-related commands (e.g. "some enemies, some blocks, high elevation") versus random prompts with pipes-related commands (e.g. "little pipes, some enemies, some blocks, high elevation"). The distribution without pipe prompts is scattered, while the ones with pipe prompts result in distributions with peaks.
(b) "little pipes + random prompt"
(c) "some pipes + random prompt"
(d) "many pipes + random prompt"