# Evaluating the Expressive Range of Super Mario Bros Level Generators
## 4 Experiments and Results
#### Expressive Range  
Heatmaps and box/whisker graphs were created to perform a visual reading of the generators. One of the most commonly studied concepts in the generation of video game levels is difficulty. To do this, in Figure 3b, one can see the heatmaps for the three generators and the SMB levels. The GAN produces mostly linear and less diverse levels, while the GA and MC produce semi-linear levels. Regarding lenience, the GAN does not create highly difficult levels in comparison with the GA, which, through the evolutionary method, can build challenging scenarios. The original SMB levels cover a larger area of the generative space with respect to these two metrics; this is very different from the behavior of the other generators, whose respective distributions have a lower variance. Visually, the GAN generator is the most dissimilar. These levels created through Genetic Algorithms and Markov Chains are those that come closest to the characteristics of the original levels. However, a more in-depth analysis must be performed to accurately make this conclusion. Figure 3b,c is also intended to show the degree of difficulty in the generated content. Having enemies too close together can make it difficult to solve the level since the player has a limited jumping height, and rows of enemies can kill them. In this, one can see that the MC generates various configurations on the Y axis, obtaining a wide expressive range regarding the compression of enemies. The GAN obtains poor performance, while the GA concentrates all of the values, giving rise to a low diversity of enemy distribution. \begin{table}
\begin{tabular}{l l} \hline \hline
**Hyperparameters** & **Value** \\ \hline Optimizer & RMSprop \\ Batch size & 32 \\ Learning rate & 0.00005 \\ Epochs & 5000 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Hyperparameters for the Generative Adversarial Network. Figure 2: Examples of the levels generated by each generator. Most levels present similarities, for example, in the absence of structures in the GAN generator or the lack of structural verticality in the GA generator. (**a**) Level generated by the GA generator. (**b**) Level generated by the GAN generator. (**c**) Level generated by the MC generator. The heatmaps in Figure 3a,d are related to the levels' design in appearance and navigation. Figure 3a shows how the GA and MC generators obtain a similar linearity. The GA and MC differ mainly in how the floor tiles are stacked, resulting in denser levels for the GA generator than for the MC. Regarding the GAN, the levels are highly linear with a low density, which results in SMB levels with a low number of columns and complex ground structures. Again, the SMB levels have a wide distribution, as seen on the Y axis, where the density displayed runs along the entire axis. Additionally, the heatmap in Figure 3d shows a limited number of interesting elements with the GA, which produces the greatest number of elements other than soil, but with a low variance in comparison with the MC generator. In this case, there is a similarity in behavior between the original SMB levels and the GAN and MC generators. Still, the MC generator exhibits greater monotony between this pair of variables, covering a larger area of the generative space where its projection is almost linear. Last, Figure 3e shows again how the SMB levels cover a much more uniform space than that of the other generators. This characteristic is desired since a high diversity is needed due to the expressiveness of the algorithms. The three generators distribute their data in a similar way, where the greatest variation with respect to the calculated metrics is given by the MC generator. Curiously, these levels escape the expressive range that the original levels have, since, despite their having been provided as training data, the Markov Chains manage to generate content that has not been seen with respect to the evaluated metrics. This may be caused by the number of columns that the MC considers to create the  
Figure 3: Expressive range of each generator.