abstract: In this paper we introduce new methods for the improved training of generative adversarial networks (GANs) for image synthesis. We construct a variant of GANs employing label conditioning that results in $128\times 128$ resolution image samples exhibiting global coherence. We expand on previous work for image quality assessment to provide two new analyses for assessing the discriminability and diversity of samples from class-conditional image synthesis models. These analyses demonstrate that high resolution samples provide class information not present in low resolution samples. Across 1000 ImageNet classes, $128\times 128$ samples are more than twice as discriminable as artificially resized $32\times 32$ samples. In addition, 84.7% of the classes have samples exhibiting diversity comparable to real ImageNet data.
bibliography: example_paper.bib

# Introduction

Characterizing the structure of natural images has been a rich research endeavor. Natural images obey intrinsic invariances and exhibit multi-scale statistical structures that have historically been difficult to quantify . Recent advances in machine learning offer an opportunity to substantially improve the quality of image models. Improved image models advance the state-of-the-art in image denoising , compression , in-painting , and super-resolution . Better models of natural images also improve performance in semi-supervised learning tasks and reinforcement learning problems .

One method for understanding natural image statistics is to build a system that synthesizes images *de novo*. There are several promising approaches for building image synthesis models. Variational autoencoders (VAEs) maximize a variational lower bound on the log-likelihood of the training data . VAEs are straightforward to train but introduce potentially restrictive assumptions about the approximate posterior distribution (but see ). Autoregressive models dispense with latent variables and directly model the conditional distribution over pixels . These models produce convincing samples but are costly to sample from and do not provide a latent representation. Invertible density estimators transform latent variables directly using a series of parameterized functions constrained to be invertible . This technique allows for exact log-likelihood computation and exact inference, but the invertibility constraint is restrictive.

Generative adversarial networks (GANs) offer a distinct and promising approach that focuses on a game-theoretic formulation for training an image synthesis model . Recent work has shown that GANs can produce convincing image samples on datasets with low variability and low resolution . However, GANs struggle to generate globally coherent, high resolution samples - particularly from datasets with high variability. Moreover, a theoretical understanding of GANs is an on-going research topic .

In this work we demonstrate that that adding more structure to the GAN latent space along with a specialized cost function results in higher quality samples. We exhibit $128\times 128$ pixel samples from all classes of the ImageNet dataset with increased global coherence (Figure ). Importantly, we demonstrate quantitatively that our high resolution samples are not just naive resizings of low resolution samples. In particular, downsampling our $128 \times 128$ samples to $32 \times 32$ leads to a 50% decrease in visual discriminability. We also introduce a new metric for assessing the variability across image samples and employ this metric to demonstrate that our synthesized images exhibit diversity comparable to training data for a large fraction (84.7%) of ImageNet classes. In more detail, this work is the first to:

-   Demonstrate an image synthesis model for all 1000 ImageNet classes at a 128x128 spatial resolution (or any spatial resolution - see Section ).

-   Measure how much an image synthesis model actually uses its output resolution (Section ).

-   Measure perceptual variability and 'collapsing' behavior in a GAN with a fast, easy-to-compute metric (Section ).

-   Highlight that a high number of classes is what makes ImageNet synthesis difficult for GANs and provide an explicit solution (Section ).

-   Demonstrate experimentally that GANs that perform well perceptually are not those that memorize a small number of examples (Section ).

-   Achieve state of the art on the Inception score metric when trained on CIFAR-10 without using any of the techniques from (Section ).

# Background

A generative adversarial network (GAN) consists of two neural networks trained in opposition to one another. The generator $G$ takes as input a random noise vector $z$ and outputs an image $X_{fake} = G(z)$. The discriminator $D$ receives as input either a training image or a synthesized image from the generator and outputs a probability distribution $P(S ~|~ X)=D(X)$ over possible image sources. The discriminator is trained to maximize the log-likelihood it assigns to the correct source: $$\begin{gathered}
  \label{eqn:eqlabel}
L = E[\log P(S = real ~~|~ X_{real} )] + \\ E[\log P(S = fake ~|~ X_{fake})] $$
\end{gathered}$$ The generator is trained to minimize the second term in Equation .

The basic GAN framework can be augmented using side information. One strategy is to supply both the generator and discriminator with class labels in order to produce class conditional samples . Class conditional synthesis can significantly improve the quality of generated samples . Richer side information such as image captions and bounding box localizations may improve sample quality further .

Instead of feeding side information to the discriminator, one can task the discriminator with reconstructing side information. This is done by modifying the discriminator to contain an auxiliary decoder network[^1] that outputs the class label for the training data or a subset of the latent variables from which the samples are generated . Forcing a model to perform additional tasks is known to improve performance on the original task (e.g. ). In addition, an auxiliary decoder could leverage pre-trained discriminators (e.g. image classifiers) for further improving the synthesized images . Motivated by these considerations, we introduce a model that combines both strategies for leveraging side information. That is, the model proposed below is class conditional, but with an auxiliary decoder that is tasked with reconstructing class labels.

# AC-GANs

We propose a variant of the GAN architecture which we call an auxiliary classifier GAN (or AC-GAN). In the AC-GAN, every generated sample has a corresponding class label, $c \sim p_c$ in addition to the noise $z$. $G$ uses both to generate images $X_{fake} = G(c,z)$. The discriminator gives both a probability distribution over sources and a probability distribution over the class labels, $P(S ~|~ X),~ P(C ~|~ X) = D(X)$. The objective function has two parts: the log-likelihood of the correct source, $L_S$, and the log-likelihood of the correct class, $L_C$.

$$\begin{gathered}
L_S = E[\log P(S = real ~~|~ X_{real} )] + \\ E[\log P(S = fake ~|~ X_{fake})]
\end{gathered}$$ $$\begin{gathered}
L_C = E[\log P(C = c ~~|~ X_{real} )] + \\ E[\log P(C = c ~|~ X_{fake})]
\end{gathered}$$

$D$ is trained to maximize $L_S + L_C$ while $G$ is trained to maximize $L_C - L_S$. AC-GANs learn a representation for $z$ that is independent of class label (e.g. ).

Structurally, this model is not tremendously different from existing models. However, this modification to the standard GAN formulation produces excellent results and appears to stabilize training. Moreover, we consider the AC-GAN model to be only part of the technical contributions of this work, along with our proposed methods for measuring the extent to which a model makes use of its given output resolution, methods for measuring perceptual variability of samples from the model, and a thorough experimental analyis of a generative model of images that creates $128 \times 128$ samples from all 1000 ImageNet classes.

Early experiments demonstrated that increasing the number of classes trained on while holding the model fixed decreased the quality of the model outputs. The structure of the AC-GAN model permits separating large datasets into subsets by class and training a generator and discriminator for each subset. All ImageNet experiments are conducted using an ensemble of 100 AC-GANs, each trained on a 10-class split.

# Results

We train several AC-GAN models on the ImageNet data set . Broadly speaking, the architecture of the generator $G$ is a series of 'deconvolution' layers that transform the noise $z$ and class $c$ into an image . We train two variants of the model architecture for generating images at $128\times 128$ and $64\times 64$ spatial resolutions. The discriminator $D$ is a deep convolutional neural network with a Leaky ReLU nonlinearity . As mentioned earlier, we find that reducing the variability introduced by all 1000 classes of ImageNet significantly improves the quality of training. We train 100 AC-GAN models – each on images from just 10 classes – for 50000 mini-batches of size 100.

Evaluating the quality of image synthesis models is challenging due to the variety of probabilistic criteria and the lack of a perceptually meaningful image similarity metric. Nonetheless, in later sections we attempt to measure the quality of the AC-GAN by building several *ad-hoc* measures for image sample discriminability and diversity. Our hope is that this work might provide quantitative measures that may be used to aid training and subsequent development of image synthesis models.

## Generating High Resolution Images Improves Discriminability

Building a class-conditional image synthesis model necessitates measuring the extent to which synthesized images appear to belong to the intended class. In particular, we would like to know that a high resolution sample is not just a naive resizing of a low resolution sample. Consider a simple experiment: pretend there exists a model that synthesizes $32\times 32$ images. One can trivially increase the resolution of synthesized images by performing bilinear interpolation. This would yield higher resolution images, but these images would just be blurry versions of the low resolution images that are not discriminable. Hence, the goal of an image synthesis model is not simply to produce high resolution images, but to produce high resolution images that are more discriminable than low resolution images.

To measure discriminability, we feed synthesized images to a pre-trained Inception network and report the fraction of the samples for which the Inception network assigned the correct label[^2]. We calculate this accuracy measure on a series of real and synthesized images which have had their spatial resolution artificially decreased by bilinear interpolation (Figure , top panels). Note that as the spatial resolution is decreased, the accuracy decreases - indicating that resulting images contain less class information (Figure , scores below top panels). We summarized this finding across all 1000 ImageNet classes for the ImageNet training data (black), a $128\times 128$ resolution AC-GAN (red) and a $64\times 64$ resolution AC-GAN (blue) in Figure (bottom, left). The black curve (clipped) provides an upper-bound on the discriminability of real images.

The goal of this analysis is to show that synthesizing higher resolution images leads to increased discriminability. The $128\times 128$ model achieves an accuracy of 10.1% $\pm$ 2.0% versus 7.0% $\pm$ 2.0% with samples resized to $64\times 64$ and 5.0% $\pm$ 2.0% with samples resized to $32\times 32$. In other words, downsizing the outputs of the AC-GAN to $32 \times 32$ and $64 \times 64$ decreases visual discriminability by 50% and 38% respectively. Furthermore, 84.4% of the ImageNet classes have higher accuracy at $128\times 128$ than at $32 \times 32$ (Figure , bottom left).

We performed the same analysis on an AC-GAN trained to $64 \times 64$ spatial resolution. This model achieved less discriminability than a $128\times 128$ AC-GAN model. Accuracies from the $64 \times 64$ model plateau at a $64 \times 64$ spatial resolution consistent with previous results. Finally, the $64\times 64$ resolution model achieves less discriminability at 64 spatial resolution than the $128\times 128$ model.

To the best of our knowledge, this work is the first that attempts to measure the extent to which an image synthesis model is 'making use of its given output resolution', and in fact is the first work to consider the issue at all. We consider this an important contribution, on par with proposing a model that synthesizes images from all 1000 ImageNet classes. We note that the proposed method can be applied to any image synthesis model for which a measure of 'sample quality' can be constructed. In fact, this method (broadly defined) can be applied to any type of synthesis model, as long as there is an easily computable notion of sample quality and some method for 'reducing resolution'. In particular, we expect that a similar procecure can be carried out for audio synthesis.

## Measuring the Diversity of Generated Images

An image synthesis model is not very interesting if it only outputs one image. Indeed, a well-known failure mode of GANs is that the generator will collapse and output a single prototype that maximally fools the discriminator . A class-conditional model of images is not very interesting if it only outputs one image per class. The Inception accuracy can not measure whether a model has collapsed. A model that simply memorized one example from each ImageNet class would do very well by this metric. Thus, we seek a complementary metric to explicitly evaluate the intra-class perceptual diversity of samples generated by the AC-GAN.

Several methods exist for quantitatively evaluating image similarity by attempting to predict human perceptual similarity judgements. The most successful of these is multi-scale structural similarity (MS-SSIM) . MS-SSIM is a multi-scale variant of a well-characterized perceptual similarity metric that attempts to discount aspects of an image that are not important for human perception . MS-SSIM values range between 0.0 and 1.0; higher MS-SSIM values correspond to perceptually more similar images.

As a proxy for image diversity, we measure the MS-SSIM scores between 100 randomly chosen pairs of images within a given class. Samples from classes that have higher diversity result in lower mean MS-SSIM scores (Figure , left columns); samples from classes with lower diversity have higher mean MS-SSIM scores (Figure , right columns). Training images from the ImageNet training data contain a variety of mean MS-SSIM scores across the classes indicating the variability of image diversity in ImageNet classes (Figure , x-axis). Note that the highest mean MS-SSIM score (indicating the least variability) is 0.25 for the training data.

We calculate the mean MS-SSIM score for all 1000 ImageNet classes generated by the AC-GAN model. We track this value during training to identify whether the generator has collapsed (Figure , red curve). We also employ this metric to compare the diversity of the training images to the samples from the GAN model after training has completed. Figure  plots the mean MS-SSIM values for image samples and training data broken up by class. The blue line is the line of equality. Out of the 1000 classes, we find that 847 have mean sample MS-SSIM scores below that of the maximum MS-SSIM for the training data. In other words, 84.7% of classes have sample variability that exceeds that of the least variable class from the ImageNet training data.

There are two points related to the MS-SSIM metric and our use of it that merit extra attention. The first point is that we are 'abusing' the metric: it was originally intended to be used for measuring the quality of image compression algorithms using a reference 'original image'. We instead use it on two potentially unrelated images. We believe that this is acceptable for the following reasons: First: visual inspection seems to indicate that the metric makes sense - pairs with higher MS-SSIM do seem more similar than pairs with lower MS-SSIM. Second: we restrict comparisons to images synthesized using the same class label. This restricts use of MS-SSIM to situations more similar to those in which it is typically used (it is not important which image is the reference). Third: the metric is not 'saturated' for our use-case. If most scores were around 0, then we would be more concerned about the applicability of MS-SSIM. Finally: The fact that training data achieves more variability by this metric (as expected) is itself evidence that the metric is working as intended.

The second point is that the MS-SSIM metric is not intended as a proxy for the entropy of the generator distribution in pixel space, but as a measure of perceptual diversity of the outputs. The entropy of the generator output distribution is hard to compute and pairwise MS-SSIM scores would not be a good proxy. Even if it were easy to compute, we argue that it would still be useful to have a separate measure of perceptual diversity. To see why, consider that the generator entropy will be sensitive to trivial changes in contrast as well as changes in the semantic content of the outputs. In many applications, we don't care about this contribution to the entropy, and it is useful to consider measures that attempt to ignore changes to an image that we consider 'perceptually meaningless', hence the use of MS-SSIM.

## Generated Images are both Diverse and Discriminable

We have presented quantitative metrics demonstrating that AC-GAN samples may be diverse and discriminable but we have yet to examine how these metrics interact. Figure shows the joint distribution of Inception accuracies and MS-SSIM scores across all classes. Inception accuracy and MS-SSIM are anti-correlated ($r^2 = -0.16$). In fact, 74% of the classes with low diversity (MS-SSIM $\geq 0.25$) contain Inception accuracies $\leq 1\%$. Conversely, 78% of classes with high diversity (MS-SSIM $< 0.25$) have Inception accuracies that exceed 1%. In comparison, the Inception-v3 model achieves 78.8% accuracy on average across all 1000 classes . A fraction of the classes AC-GAN samples reach this level of accuracy. This indicates opportunity for future image synthesis models.

These results suggest that GANs that drop modes are most likely to produce low quality images. This stands in contrast to a popular hypothesis about GANs, which is that they achieve high sample quality at the expense of variability. We hope that these findings can help structure further investigation into the reasons for differing sample quality between GANs and other image synthesis models.

## Comparison to Previous Results

Previous quantitative results for image synthesis models trained on ImageNet are reported in terms of log-likelihood . Log-likelihood is a coarse and potentially inaccurate measure of sample quality . Instead we compare with previous state-of-the-art results on CIFAR-10 using a lower spatial resolution ($32\times 32$). Following the procedure in , we compute the Inception score[^3] for 50000 samples from an AC-GAN with resolution ($32\times 32$), split into 10 groups at random. We also compute the Inception score for 25000 extra samples, split into 5 groups at random. We select the best model based on the first score and report the second score. Performing a grid search across 27 hyperparameter configurations, we are able to achieve a score of 8.25 $\pm$ 0.07 compared to state of the art 8.09 $\pm$ 0.07 . Moreover, we accomplish this without employing any of the new techniques introduced in that work (i.e. virtual batch normalization, minibatch discrimination, and label smoothing).

This provides additional evidence that AC-GANs are effective even without the benefit of class splitting. See Figure for a qualitative comparison of samples from an AC-GAN and samples from the model in .

## Searching for Signatures of Overfitting

One possibility that must be investigated is that the AC-GAN has overfit on the training data. As a first check that the network does not memorize the training data, we identify the nearest neighbors of image samples in the training data measured by L1 distance in pixel space (Figure ). The nearest neighbors from the training data do not resemble the corresponding samples. This provides evidence that the AC-GAN is not merely memorizing the training data.

A more sophisticated method for understanding the degree of overfitting in a model is to explore that model's latent space by interpolation. In an overfit model one might observe discrete transitions in the interpolated images and regions in latent space that do not correspond to meaningful images . Figure (Top) highlights interpolations in the latent space between several image samples. Notably, the generator learned that certain combinations of dimensions correspond to semantically meaningful features (e.g. size of the arch, length of a bird's beak) and there are no discrete transitions or 'holes' in the latent space.

A second method for exploring the latent space of the AC-GAN is to exploit the structure of the model. The AC-GAN factorizes its representation into class information and a class-independent latent representation $z$. Sampling the AC-GAN with $z$ fixed but altering the class label corresponds to generating samples with the same 'style' across multiple classes . Figure (Bottom) shows samples from 8 bird classes. Elements of the same row have the same $z$. Although the class changes for each column, elements of the global structure (e.g. position, layout, background) are preserved, indicating that AC-GAN can represent certain types of 'compositionality'.

## Measuring the Effect of Class Splits on Image Sample Quality.

Class conditional image synthesis affords the opportunity to divide up a dataset based on image label. In our final model we divide 1000 ImageNet classes across 100 AC-GAN models. In this section we describe experiments that highlight the benefit of cutting down the diversity of classes for training an AC-GAN. We employed an ordering of the labels and divided it into contiguous groups of 10. This ordering can be seen in the following section, where we display samples from all 1000 classes. Two aspects of the split merit discussion: the number of classes per split and the intra-split diversity. We find that training a fixed model on more classes harms the model's ability to produce compelling samples (Figure ). Performance on larger splits can be improved by giving the model more parameters. However, using a small split is not sufficient to achieve good performance. We were unable to train a GAN to converge reliably even for a split size of 1. This raises the question of whether it is easier to train a model on a diverse set of classes than on a similar set of classes: We were unable to find conclusive evidence that the selection of classes in a split significantly affects sample quality.

We don't have a hypothesis about what causes this sensitivity to class count that is well-supported experimentally. We can only note that, since the failure case that occurs when the class count is increased is 'generator collapse', it seems plausible that general methods for addressing 'generator collapse' could also address this sensitivity.

## Samples from all 1000 ImageNet Classes

We also generate 10 samples from each of the 1000 ImageNet classes, hosted [here](https://goo.gl/photos/8bgHBkCwDEVTXAPaA). As far as we are aware, no other image synthesis work has included a similar analysis.

# Discussion

This work introduced the AC-GAN architecture and demonstrated that AC-GANs can generate globally coherent ImageNet samples. We provided a new quantitative metric for image discriminability as a function of spatial resolution. Using this metric we demonstrated that our samples are more discriminable than those from a model that generates lower resolution images and performs a naive resize operation. We also analyzed the diversity of our samples with respect to the training data and provided some evidence that the image samples from the majority of classes are comparable in diversity to ImageNet data.

Several directions exist for building upon this work. Much work needs to be done to improve the visual discriminability of the $128\times 128$ resolution model. Although some synthesized image classes exhibit high Inception accuracies, the average Inception accuracy of the model ($10.1\% \pm 2.0\%$) is still far below real training data at 81%. One immediate opportunity for addressing this is to augment the discriminator with a pre-trained model to perform additional supervised tasks (e.g. image segmentation, ).

Improving the reliability of GAN training is an ongoing research topic. Only 84.7% of the ImageNet classes exhibited diversity comparable to real training data. Training stability was vastly aided by dividing up 1000 ImageNet classes across 100 AC-GAN models. Building a single model that could generate samples from all 1000 classes would be an important step forward.

Image synthesis models provide a unique opportunity for performing semi-supervised learning: these models build a rich prior over natural image statistics that can be leveraged by classifiers to improve predictions on datasets for which few labels exist. The AC-GAN model can perform semi-supervised learning by ignoring the component of the loss arising from class labels when a label is unavailable for a given training image. Interestingly, prior work suggests that achieving good sample quality might be independent of success in semi-supervised learning .

# Hyperparameters

We summarize hyperparameters used for the ImageNet model in Table and for the CIFAR-10 model in Table .

```latex
\begin{table*}[h]\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}rllllll@{}} \toprule
Operation              & Kernel       & Strides      & Feature maps & BN?          & Dropout & Nonlinearity \\ \midrule
$G_x(z)$ -- $110 \times 1 \times 1$ input                                                                 \\
Linear                 & N/A          & N/A          & $768$        & $\times$     & 0.0     & ReLU       \\
Transposed Convolution & $5 \times 5$ & $2 \times 2$ & $384$        & $\surd$      & 0.0     & ReLU       \\
Transposed Convolution & $5 \times 5$ & $2 \times 2$ & $256$        & $\surd$      & 0.0     & ReLU       \\
Transposed Convolution & $5 \times 5$ & $2 \times 2$ & $192$        & $\surd$      & 0.0     & ReLU       \\
Transposed Convolution & $5 \times 5$ & $2 \times 2$ & $3$          & $\times$     & 0.0     & Tanh       \\
$D(x)$ -- $128 \times 128 \times 3$ input                                                                   \\
Convolution            & $3 \times 3$ & $2 \times 2$ & $16$         & $\times$     & 0.5     & Leaky ReLU \\
Convolution            & $3 \times 3$ & $1 \times 1$ & $32$         & $\surd$      & 0.5     & Leaky ReLU \\
Convolution            & $3 \times 3$ & $2 \times 2$ & $64$         & $\surd$      & 0.5     & Leaky ReLU \\
Convolution            & $3 \times 3$ & $1 \times 1$ & $128$        & $\surd$      & 0.5     & Leaky ReLU \\
Convolution            & $3 \times 3$ & $2 \times 2$ & $256$        & $\surd$      & 0.5     & Leaky ReLU \\
Convolution            & $3 \times 3$ & $1 \times 1$ & $512$        & $\surd$      & 0.5     & Leaky ReLU \\
Linear                 & N/A          & N/A          & $11$         & $\times$     & 0.0     & Soft-Sigmoid\\
Optimizer              & \multicolumn{6}{@{}l@{}}{Adam ($\alpha = 0.0002$, $\beta_1 = 0.5$, $\beta_2 = 0.999$)}  \\
Batch size             & \multicolumn{6}{@{}l@{}}{100}                                                    \\
Iterations             & \multicolumn{6}{@{}l@{}}{50000}                                                  \\
Leaky ReLU slope       & \multicolumn{6}{@{}l@{}}{0.2}                                                   \\
Weight, bias initialization  & \multicolumn{6}{@{}l@{}}{Isotropic gaussian ($\mu = 0$, $\sigma = 0.02$), Constant($0$)} \\ \bottomrule
\end{tabular}}
\vspace{0.2cm}
\caption{\label{tab:model_description} ImageNet hyperparameters.
  A Soft-Sigmoid refers to an operation over $K+1$ output units where we apply a Softmax activation to $K$ of
  the units and a Sigmoid activation to the remaining unit.
We also use activation noise in the discriminator as suggested in \cite{IMPROVEDTECHNIQUES}.}
\end{table*}
```
```latex
\begin{table*}[h]\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}rllllll@{}} \toprule
Operation               & Kernel       & Strides      & Feature maps & BN?          & Dropout & Nonlinearity \\ \midrule
$G_x(z)$ -- $110 \times 1 \times 1$ input                                                                 \\
Linear                  & N/A          & N/A          & $384$        & $\times$     & 0.0     & ReLU       \\
Transposed Convolution  & $5 \times 5$ & $2 \times 2$ & $192$        & $\surd$      & 0.0     & ReLU       \\
Transposed Convolution  & $5 \times 5$ & $2 \times 2$ & $96$         & $\surd$      & 0.0     & ReLU       \\
Transposed Convolution  & $5 \times 5$ & $2 \times 2$ & $3$          & $\times$     & 0.0     & Tanh       \\
$D(x)$ -- $32 \times 32 \times 3$ input                                                                   \\
Convolution             & $3 \times 3$ & $2 \times 2$ & $16$         & $\times$     & 0.5     & Leaky ReLU \\
Convolution             & $3 \times 3$ & $1 \times 1$ & $32$         & $\surd$      & 0.5     & Leaky ReLU \\
Convolution             & $3 \times 3$ & $2 \times 2$ & $64$         & $\surd$      & 0.5     & Leaky ReLU \\
Convolution             & $3 \times 3$ & $1 \times 1$ & $128$        & $\surd$      & 0.5     & Leaky ReLU \\
Convolution             & $3 \times 3$ & $2 \times 2$ & $256$        & $\surd$      & 0.5     & Leaky ReLU \\
Convolution             & $3 \times 3$ & $1 \times 1$ & $512$        & $\surd$      & 0.5     & Leaky ReLU \\
Linear                  & N/A          & N/A          & $11$         & $\times$     & 0.0     & Soft-Sigmoid\\
Generator Optimizer     & \multicolumn{6}{@{}l@{}}{Adam ($\alpha = [0.0001, 0.0002, 0.0003]$, $\beta_1 = 0.5$, $\beta_2 = 0.999$)}  \\
Discriminator Optimizer & \multicolumn{6}{@{}l@{}}{Adam ($\alpha = [0.0001, 0.0002, 0.0003]$, $\beta_1 = 0.5$, $\beta_2 = 0.999$)}  \\
Batch size              & \multicolumn{6}{@{}l@{}}{100}                                                   \\
Iterations              & \multicolumn{6}{@{}l@{}}{50000}                                                 \\
Leaky ReLU slope        & \multicolumn{6}{@{}l@{}}{0.2}                                                   \\
Activation noise standard deviation & \multicolumn{6}{@{}l@{}}{$[0,0.1,0.2]$}                                                   \\
Weight, bias initialization  & \multicolumn{6}{@{}l@{}}{Isotropic gaussian ($\mu = 0$, $\sigma = 0.02$), Constant($0$)} \\ \bottomrule
\end{tabular}}
\vspace{0.2cm}
\caption{\label{tab:cifar_model_description} CIFAR-10 hyperparameters.
  When a list is given for a hyperparameter it means that we performed a grid search using the values in the list.
  For each set of hyperparameters, a single AC-GAN was trained on the whole CIFAR-10 dataset.
  For each AC-GAN that was trained, we split up the samples into groups so that we could give some sense of the variance of the Inception Score.
  To the best of our knowledge, this is identical to the analysis performed in \cite{IMPROVEDTECHNIQUES}.
}
\end{table*}
```

[^1]: Alternatively, one can force the discriminator to work with the joint distribution $(X,z)$ and train a separate inference network that computes $q(z|X)$ .

[^2]: One could also use the Inception score , but our method has several advantages: accuracy figures are easier to interpret than exponentiated KL-divergences; accuracy may be assessed for individual classes; accuracy measures whether a class-conditional model generated samples from the intended class. To compute the Inception accuracy, we modified a version of Inception-v3 supplied in `https://github.com/openai/improved-gan/`.

[^3]: The Inception score is given by $\exp\left( E_x [D_{\small KL}(p(y|x) \; || \; p(y))] \right)$ where $x$ is a particular image, $p(y|x)$ is the conditional output distribution over the classes in a pre-trained Inception network given $x$, and $p(y)$ is the marginal distribution over the classes.


author: Neil D. Lawrence
bibliography: ../other.bib; ../lawrence.bib; ../zbooks.bib
title: Data Science and Digital Systems: The 3Ds of Machine Learning Systems Design

# Introduction

There is a lot of talk about the fourth industrial revolution centered around AI. If we are at the start of the fourth industrial we also have the unusual honour of being the first to name our revolution before it's occurred.

The technology that has driven the revolution in AI is machine learning. And when it comes to capitalising on the new generation of deployed machine learning solutions there are practical difficulties we must address.

In 1987 the economist Robert Solow quipped "You can see the computer age everywehere but in the productivity statistics". Thirty years later, we could equally apply that quip to the era of artificial intelligence.

From my perspective, the current era is merely the continuation of the information revolution. A revolution that was triggered by the wide availability of the silicon chip. But whether we are in the midst of a new revolution, or this is just the continuation of an existing revolution, it feels important to characterize the challenges of deploying our innovation and consider what the solutions may be.

There is no doubt that new technologies based around machine learning have opened opportunities to create new businesses. When home computers were introduced there were also new opportunities in software publishing, computer games and a magazine industry around it. The Solow paradox arose because despite this visible activity these innovations take time to percolate through to *existing* businesses.

## Brownfield and Greenfield Innovation

Understanding how to make the best use of new technology takes time. I call this approach, *brownfield innovation*. In the construction industry, a brownfield site is land with pre-existing infrastructure on it, whereas a *greenfield* site is where construction can start afresh.

The term brownfield innovation arises by analogy. Brownfield innovation is when you are innovating in a space where there is pre-existing infrastructure. This is the challenge of introducing artificial intelligence to existing businesses. Greenfield innovation, is innovating in areas where no pre-existing infrastructure exists.

One way we can make it easier to benefit from machine learning in both greenfield and brownfield innovation is to better characterise the steps of machine learning systems design. Just as software systems design required new thinking, so does machine learning systems design.

In this paper we characterise the process for machine learning systems design, convering some of the issues we face, with the 3D process: Decomposition[^1], Data and Deployment.

We will consider each component in turn, although there is interplace between components. Particularly between the task decomposition and the data availability. We will first outline the *decomposition* challenge.

One of the most succesful machine learning approaches has been supervised learning. So we will mainly focus on *supervised learning* because this is also, arguably, the technology that is best understood within machine learning.

# Decomposition

Machine learning is not magical pixie dust, we cannot simply automate all decisions through data. We are constrained by our data (see below) and the models we use. Machine learning models are relatively simple function mappings that include characteristics such as smoothness. With some famous exceptions, e.g. speech and image data, inputs are constrained in the form of vectors and the model consists of a mathematically well behaved function. This means that some careful thought has to be put in to the right sub-process to automate with machine learning. This is the challenge of *decomposition* of the machine learning system.

Any repetitive task is a candidate for automation, but many of the repetitive tasks we perform as humans are more complex than any individual algorithm can replace. The selection of which task to automate becomes critical and has downstream effects on our overall system design.

## Pigeonholing

The machine learning systems design process calls for separating a complex task into decomposable separate entities. A process we can think of as pigeonholing.

Some aspects to take into account are

1.  Can we refine the decision we need to a set of repetitive tasks where input information and output decision/value is well defined?

2.  Can we represent each sub-task we've defined with a mathematical mapping?

The representation necessary for the second aspect may involve massaging of the problem: feature selection or adaptation. It may also involve filtering out exception cases (perhaps through a pre-classification).

All else being equal, we'd like to keep our models simple and interpretable. If we can convert a complex mapping to a linear mapping through clever selection of sub-tasks and features this is a big win.

For example, Facebook have *feature engineers*, individuals whose main role is to design features they think might be useful for one of their tasks (e.g. newsfeed ranking, or ad matching). Facebook have a training/testing pipeline called [FBLearner](https://www.facebook.com/Engineering/posts/fblearner-flow-is-a-machine-learning-platform-capable-of-easily-reusing-algorith/10154077833317200/). Facebook have predefined the sub-tasks they are interested in, and they are tightly connected to their business model.

It is easier for Facebook to do this because their business model is heavily focused on user interaction. A challenge for companies that have a more diversified portfolio of activities driving their business is the identification of the most appropriate sub-task. A potential solution to feature and model selection is known as *AutoML* . Or we can think of it as using Machine Learning to assist Machine Learning. It's also called meta-learning. Learning about learning. The input to the ML algorithm is a machine learning task, the output is a proposed model to solve the task.

One trap that is easy to fall in is too much emphasis on the type of model we have deployed rather than the appropriateness of the task decomposition we have chosen.

**Recommendation**: Conditioned on task decomposition, we should automate the process of model improvement. Model updates should not be discussed in management meetings, they should be deployed and updated as a matter of course. Further details below on model deployment, but model updating needs to be considered at design time. This is the domain of AutoML.

The answer to the question which comes first, the chicken or the egg is simple, they co-evolve . Similarly, when we place components together in a complex machine learning system, they will tend to co-evolve and compensate for one another.

To form modern decision making systems, many components are interlinked. We decompose our complex decision making into individual tasks, but the performance of each component is dependent on those upstream of it.

This naturally leads to co-evolution of systems, upstream errors can be compensated by downstream corrections.

To embrace this characteristic, end-to-end training could be considered. Why produce the best forecast by metrics when we can just produce the best forecast for our systems? End to end training can lead to improvements in performance, but it would also damage our systems decomposability and its interpretability, and perhaps its adaptability.

The less human interpretable our systems are, the harder they are to adapt to different circumstances or diagnose when there's a challenge. The trade-off between interpretability and performance is a constant tension which we should always retain in our minds when performing our system design.

# Data

It is difficult to overstate the importance of data. It is half of the equation for machine learning, but is often utterly neglected. We can speculate that there are two reasons for this. Firstly, data cleaning is perceived as tedious. It doesn't seem to consist of the same intellectual challenges that are inherent in constructing complex mathematical models and implementing them in code. Secondly, data cleaning is highly complex, it requires a deep understanding of how machine learning systems operate and good intuitions about the data itself, the domain from which data is drawn (e.g. Supply Chain) and what downstream problems might be caused by poor data quality.

A consequence of these two reasons, data cleaning seems difficult to formulate into a readily teachable set of principles. As a result it is heavily neglected in courses on machine learning and data science. Despite data being half the equation, most University courses spend little to no time on its challenges.

## The Data Crisis

Anecdotally, talking to data modelling scientists. Most say they spend 80% of their time acquiring and cleaning data. This is precipitating what I refer to as the "data crisis". This is an analogy with software. The "software crisis" was the phenomenon of inability to deliver software solutions due to increasing complexity of implementation. There was no single shot solution for the software crisis, it involved better practice (scrum, test orientated development, sprints, code review), improved programming paradigms (object orientated, functional) and better tools (CVS, then SVN, then git).

However, these challenges aren't new, they are merely taking a different form. From the computer's perspective software *is* data. The first wave of the data crisis was known as the *software crisis*.

### The Software Crisis

In the late sixties early software programmers made note of the increasing costs of software development and termed the challenges associated with it as the "[Software Crisis](https://en.wikipedia.org/wiki/Software_crisis)". Edsger Dijkstra referred to the crisis in his 1972 Turing Award winner's address .

> The major cause of the software crisis is that the machines have become several orders of magnitude more powerful! To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem.
>
> Edsger Dijkstra, The Humble Programmer

We can update Dijkstra's quote for the modern era.

> The major cause of *the data crisis* is that machines have become more interconnected than ever before. Data access is therefore cheap, but data quality is often poor. What we need is cheap high quality data. That implies that we develop processes for improving and verifying data quality that are efficient.
>
> There would seem to be two ways for improving efficiency. Firstly, we should not duplicate work. Secondly, where possible we should automate work.

What I term "The Data Crisis" is the modern equivalent of this problem. The quantity of modern data, and the lack of attention paid to data as it is initially "laid down" and the costs of data cleaning are bringing about a crisis in data-driven decision making. This crisis is at the core of the challenge of *technical debt* in machine learning .

Just as with software, the crisis is most correctly addressed by 'scaling' the manner in which we process our data. Duplication of work occurs because the value of data cleaning is not correctly recognised in management decision making processes. Automation of work is increasingly possible through techniques in artificial intelligence, but this will also require better management of the data science pipeline so that data about data science (meta-data science) can be correctly assimilated and processed. The Alan Turing institute has a program focussed on this area, [AI for Data Analytics](https://www.turing.ac.uk/research_projects/artificial-intelligence-data-analytics/).

Data is the new software, and the data crisis is already upon us. It is driven by the cost of cleaning data, the paucity of tools for monitoring and maintaining our deployments, the provenance of our models (e.g. with respect to the data they're trained on).

Three principal changes need to occur in response. They are cultural and infrastructural.

## The Data First Paradigm

First of all, to excel in data driven decision making we need to move from a *software first* paradigm to a *data first* paradigm. That means refocusing on data as the product. Software is the intermediary to producing the data, and its quality standards must be maintained, but not at the expense of the data we are producing. Data cleaning and maintenance need to be prized as highly as software debugging and maintenance. Instead of *software* as a service, we should refocus around *data* as a service. This first change is a cultural change in which our teams think about their outputs in terms of data. Instead of decomposing our systems around the software components, we need to decompose them around the data generating and consuming components.[^2] Software first is only an intermediate step on the way to be coming *data first*. It is a necessary, but not a sufficient condition for efficient machine learning systems design and deployment. We must move from *software orientated architecture* to a *data orientated architecture*.

## Data Quality

Secondly, we need to improve our language around data quality. We cannot assess the costs of improving data quality unless we generate a language around what data quality means.

[Data Readiness Levels](http://inverseprobability.com/2017/01/12/data-readiness-levels) are an attempt to develop a language around data quality that can bridge the gap between technical solutions and decision makers such as managers and project planners. The are inspired by Technology Readiness Levels which attempt to quantify the readiness of technologies for deployment. They contain three grades of data readinees. Grade C data is *hearsay* data. Data that is purported to exist, but has not been electronically loaded into a computer system (e.g. an analysis package such as R or made available via a REST API). Moving data from Grade C to Grade B also involves legal and ownership issues. Once data can be loaded in it becomes Grade B. Grade B is available electronically, but undergoes a process of validation. Aspects like missing values, outlier representation, duplicate records are dealt with in Grade B. Grade B also has some of the characteristics of *exploratory data analysis* . Grade A is then data in context. Finally, at Grade A we consider the appropriateness of data to answer a particular question. In historical statistics Grade A data might be data that is ready for confirmatory data analysis. Many statisticians and machine learning researchers are used to only dealing with data at Grade A. Either because they work mainly with benchmark data or because data was actively collected with a particular question in mind. The major change for the era of data science is that so much data is available by *happenstance*.

## Move Beyond Software Engineering to Data Engineering

Thirdly, we need to improve our mental model of the separation of data science from applied science. A common trap in current thinking around data is to see data science (and data engineering, data preparation) as a sub-set of the software engineer's or applied scientist's skill set. As a result we recruit and deploy the wrong type of resource. Data preparation and question formulation is superficially similar to both because of the need for programming skills, but the day to day problems faced are very different.

**Recommendation**: Build a shared understanding of the language of data readiness levels for use in planning documents, the costing of data cleaning, and the benefits of reusing cleaned data.

## Combining Data and Systems Design

One analogy I find helpful for understanding the depth of change we need is the following. Imagine as an engineer, you find a USB stick on the ground. And for some reason you *know* that on that USB stick is a particular API call that will enable you to make a significant positive difference on a business problem. However, you also know on that USB stick there is potentially malicious code. The most secure thing to do would be to *not* introduce this code into your production system. But what if your manager told you to do so, how would you go about incorporating this code base?

The answer is *very* carefully. You would have to engage in a process more akin to debugging than regular software engineering. As you understood the code base, for your work to be reproducible, you should be documenting it, not just what you discovered, but how you discovered it. In the end, you typically find a single API call that is the one that most benefits your system. But more thought has been placed into this line of code than any line of code you have written before.

Even then, when your API code is introduced into your production system, it needs to be deployed in an environment that monitors it. We cannot rely on an individual's decision making to ensure the quality of all our systems. We need to create an environment that includes quality controls, checks and bounds, tests, all designed to ensure that assumptions made about this foreign code base are remaining valid.

This situation is akin to what we are doing when we incorporate data in our production systems. When we are consuming data from others, we cannot assume that it has been produced in alignment with our goals for our own systems. Worst case, it may have been adversarialy produced. A further challenge is that data is dynamic. So, in effect, the code on the USB stick is evolving over time.

Anecdotally, resolving a machine learning challenge requires 80% of the resource to be focused on the data and perhaps 20% to be focused on the model. But many companies are too keen to employ machine learning engineers who focus on the models, not the data.

A reservoir of data has more value if the data is consumable. The data crisis can only be addressed if we focus on outputs rather than inputs.

For a data first architecture we need to clean our data at source, rather than individually cleaning data for each task. This involves a shift of focus from our inputs to our outputs. We should provide data streams that are consumable by many teams without purification.

**Recommendation**: We need to share best practice around data deployment across our teams. We should make best use of our processes where applicable, but we need to develop them to become *data first* organizations. Data needs to be cleaned at *output* not at *input*.

# Deployment

## Continuous Deployment

Once the decomposition is understood, the data is sourced and the models are created, the model code needs to be deployed.

To extend our USB stick analogy further, how would we deploy that code if we thought it was likely to evolve in production? This is what datadoes. We cannot assume that the conditions under which we trained our model will be retained as we move forward, indeed the only constant we have is change.

This means that when any data dependent model is deployed into production, it requires *continuous monitoring* to ensure the assumptions of design have not been invalidated. Software changes are qualified through testing, in particular a regression test ensures that existing functionality is not broken by change. Since data is continually evolving, machine learning systems require 'continual regression testing': oversight by systems that ensure their existing functionality has not been broken as the world evolves around them. An approach we refer to as *progression testing* . Unfortunately, standards around ML model deployment yet been developed. The modern world of continuous deployment does rely on testing, but it does not recognize the continuous evolution of the world around us.

**Recommendation**: We establish best practice around model deployment. We need to shift our culture from standing up a software service, to standing up a *data as a service*. Data as a Service would involve continual monitoring of our deployed models in production. This would be regulated by 'hypervisor' systems[^3] that understand the context in which models are deployed and recognize when circumstance has changed and models need retraining or restructuring.

**Recommendation**: We should consider a major re-architecting of systems around our services. In particular we should scope the use of a *streaming architecture* (such as Apache Kafka) that ensures data persistence and enables asynchronous operation of our systems.[^4] This would enable the provision of QC streams, and real time dash boards as well as hypervisors.

Importantly a streaming architecture implies the services we build are *stateless*, internal state is deployed on streams alongside external state. This allows for rapid assessment of other services' data.

# Conclusion

Machine learning has risen to prominence as an approach to *scaling* our activities. For us to continue to automate in the manner we have over the last two decades, we need to make more use of computer-based automation. Machine learning is allowing us to automate processes that were out of reach before.

We operate in a technologically evolving environment. Machine learning is becoming a key component in our decision making capabilities. But the evolving nature of data driven systems means that new approaches to model deployment are necessary. We have characterized three parts of the machine learning systems design process. *Decomposition* of the problem into separate tasks that are addressable with a machine learning solution. Collection and curation of appropriate *data*. Verificaction of data quality through data readiness levels. Using *progression* testing in our *deployments*. Continuously updating models as appropriate to ensure performance and quality is maintained.

[^1]: In earlier versions of the Three D process I've referred to this as the design stage, but decomposition feels more appropriate for what the stage involves and that preserves the word design for the overall process of machine learning systems design.

[^2]: This is related to challenges of machine learning and technical debt , although we are trying to frame the solution here rather than the problem.

[^3]: Emulation, or surrogate modelling, is one very promising approach to forming such a hypervisor. Emulators are models we fit to other models, often simulations, but the could also be other machine learning modles. These models operate at the meta-level, not on the systems directly. This means they can be used to model how the sub-systems interact. As well as emulators we shoulc consider real time dash boards, anomaly detection, mutlivariate analysis, data visualization and classical statistical approaches for hypervision of our deployed systems.

[^4]: These approaches are one area of focus for my own team's reasearch. A data first architecture is a prerequisite for efficient deployment of machine learning systems.


abstract: > Maximalism in art refers to drawing on and combining multiple different sources for art creation, embracing the resulting collisions and heterogeneity. This paper discusses the use of maximalism in game design and particularly in data games, which are games that are generated partly based on open data. Using Data Adventures, a series of generators that create adventure games from data sources such as Wikipedia and OpenStreetMap, as a lens we explore several tradeoffs and issues in maximalist game design. This includes the tension between transformation and fidelity, between decorative and functional content, and legal and ethical issues resulting from this type of generativity. This paper sketches out the design space of maximalist data-driven games, a design space that is mostly unexplored.
author: Gabriella A. B. Barros$^{1}$, Michael Cerny Green$^{1}$, Antonios Liapis$^{2}$ and Julian Togelius$^{1}$
1: Tandon School of Engineering, New York University, New York, USA
2: Institute of Digital Games, University of Malta, Msida, Malta
firstname.lastname@example.com, firstname.lastname@example.com, email@example.com, firstname.lastname@example.com
bibliography: maximalism.bib
title: Data-driven Design: A Case for Maximalist Game Design
  Paper type: Position paper

# Introduction

The unprecedented availability of digital data impacts most human endeavors, including game design. In particular, freely available data can be combined with procedural content generation (PCG) and computation creativity to create systems that can generate games (or game content) based on open data. We have previously identified such games as "data games" .

This paper explores some of the aesthetic challenges, particularities and concerns associated with games that are created from data. We start from the idea that the use of data games is in many ways similar to notions in art such as collage, sampling, and remixing. We draw on content from many different sources, causing creative collisions between them. This lets us apply some of the same conceptual apparatus to study data games as has been applied to these types of art. We also start from a series of game generators we have created, collectively referred to as "Data Adventures". These generators create adventure games, such as murder mysteries, from open data from e.g. Wikipedia and OpenStreetMap. Our ongoing struggle with getting these generators to produce playable and interesting content from something as varied and occasionally unreliable as Wikipedia has illuminated both possibilities and pitfalls of this approach.

This paper is an attempt to explore the design space of maximalist data-driven games (and other data games) in order to form an initial understanding of it. It is also an attempt to systematize reflections from our own and others' attempts at creating such games. We address the following questions:

-   What does it mean for games designed from/for data to be maximalist?

-   What is the tradeoff between transforming data and staying true to the source in terms of generating games?

-   What are the characteristics of game content that can be generated from data?

-   For what purposes can data-driven maximalist games be designed and how does that affect their character?

-   What new legal and ethical issues, including copyright issues and the potential for generating offensive, misinforming and biased content, are raised by this type of game design?

# Data-driven design and data games

This age of data sharing (whether sharing is free or not) has certainly been advantageous to research in computational creativity. While computational creativity does not necessarily need to emulate human creativity , freely available human-annotated data can be exploited as an inspiring set to any creative software. In natural language generation, Google N-grams have been exploited to identify analogies and similes , corpora of phonetic information for all words have been exploited to generate jokes , and books of a specific author have been used to generate stories typical of the genre . In visual generation, crowdsourced annotations of data were used to create image filters , while object recognition models based on deep learning of Google images was used to choose how generated 3D shapes would represent an object . Similarly, deep learning from massive musical corpora was used to create new music .

In the creative domain of games, on the other hand, similar approaches have been used to create different game components. Google's autocomplete function (which uses a form of N-grams) was used to discover names for enemies and abilities of a game character whose name was provided by the player . In the same game, Google image search used discovered names to select images for these enemies' sprites. In other work, () used Youtube playthroughs to find associations in the placement of level elements (e.g. platforms, enemies) which were used to generate levels for *Super Mario Bros* (Nintendo 1985). Patterns in *Starcraft II* maps (Blizzard Entertainment 2010) were learned through deep learning ; these encodings were used to change the frequency of minerals in the map without the usual exploratory process of e.g. an evolutionary algorithm. To better coordinate the learning process of level patterns, a corpus of diverse games has been collected .

While using existing game data —often annotated with human notions of quality— has been explored in computational game creativity , most efforts perform minor adjustments to existing games. Game generators such as Angelina , A Rogue Dream , and Game-O-Matic use data outside the game domain, enhancing their outcomes with human-provided associations (and content such as images). Even so, the core gameplay loop is simple: in Angelina, for example, the player performs the basic actions of a platformer game (e.g. jump, run); in A Rogue Dream the player moves along 4 directions and perhaps uses one more action. Gameplay in all these games is mechanics-heavy, relying on fast reactions to immediate threats rather than on high-level planning or cognitive ability. Many *data games* take an existing game mechanic and generate new content for that game from open data . In some cases, such as the game *Bar Chart Ball*, a new game mechanic is added to an existing data visualization . To play even simple data games, the player must have some understanding of the underlying data. Playing data games requires some mental effort, deduction or memory; not only dexterity.

While most data-driven game generation software focus on a simple and tight gameplay loop, there is considerable potential in using and re-using information outside of games to create more complex game systems and more involved experiences. We argue that data-driven game generation can allow for a new gameplay experience. Using the Data Adventures series of game generators as a concrete example, we articulate the tenets of maximalism in game design inspired by the art movement of the same name. Moreover, we discuss two possible dimensions of maximalist game design, and how it can start from the raw data on one end or from the gameplay experience on the other. Finally, we envision the potential uses and issues of maximalist game design.

# Maximalism in data-driven design

We are inspired by the notion of maximalism in the arts, rather than in the game design sphere. In music, for example, maximalism "embraces heterogeneity and allows for complex systems of juxtapositions and collisions, in which all outside influences are viewed as potential raw material" . We similarly embrace the use of heterogeneous data sources as notes (i.e. the individual components) and melody (i.e. the overarching game or narrative structure) to produce a game as an orchestration of dissimilar instruments . In that sense, maximalism in data-driven design is likened with mixed media in art, where more than one medium is used. De facto, the heterogeneity of the data, its sources, and the people who contribute to its creation and curation will insert juxtapositions and collisions. This may not always be desired, and several catastrophic, inconsequential or seemingly random associations should be redacted. However, the "grain" of data-driven design is built on the collision and absurdity of different elements that find their way into the game.

It should be noted that maximalism in the artistic sphere refers to materials or identities of elements within an image, song, or novel. We refer to maximalist game design in that sense, focusing on how game elements originating from different data sources (or transformed in different ways) are visualized, combined and made to interact together, thus not directly opposed to minimalist game design. 's () minimalist game design encourages removing the unnecessary parts of the design, highlighting the important bits. 's approach () refers to the game loop; minimalist games have a simple core game loop which is largely unchanged throughout the game. Sicart uses *Minecraft* (Mojang, 2011) as an example where the simple core loop gather$\rightarrow$craft$\rightarrow$build that remains relevant and unchanged (except from the specific materials worked) throughout the game.

A data-driven game, maximalist in the artistic sense, can also be minimal in the gameplay loop sense. Data Adventures has a simple core gameplay loop of traveling to a new location, talking to a non-player character in that location, learning the clue for the next location. Games that we would define as maximalist on the design sense, on the other hand, have the broadest mechanics of options for solving a problem — e.g. killing a dragon with stealth, magic, followers, swords, fists, poison, etc. in *Skyrim* (Bethesda, 2011) — or subsystems that are so elaborate or numerous that the player becomes unable to distinguish a core game loop — e.g. the diverse driving, shooting, spraying, running, etc. minigames in *Saints Row IV* (Deep Silver, 2013) which are the main ways to progress in the game. While certainly data-driven design can offer the latter form of maximalism, e.g. with individual minigames where different forms or sources or data are presented and interacted with in each, not all data-driven games need to have maximalist game loops.

# Case Study: Data Adventure Games

The Data Adventures series of game generators exemplify the use of a high volume of data to procedurally generate content . The generated adventure games use information gathered from Wikipedia, DBpedia, Wikimedia Commons and OpenStreetMap (OSM) to automatically create an adventure, complete with plot, characters, items and in-game locations. The series consists of three games: Data Adventures , WikiMystery and DATA Agent . Each evolved from the previous one, with DATA Agent being the most recent, complex and powerful. Most of the gameplay, however, is the same: a point-and-click interface inspired by "Where In The World is Carmen Sandiego?" (Brøderbund Software 1985).

The series' first installment is Data Adventures, an exploration game created from the connections between two Wikipedia articles about specific people. Two Non-Playable Characters (NPCs) are generated representing each of these people. The player receives a quest from the first NPC, asking them to find the second one. To do so, the player has to travel through cities, talking to other generated NPCs and reading books. All information is created from a path linking one article (of the starting NPC) to the other article (of the goal NPC). Figures  and show a map screen generated using OSM and a location showing a NPC and a book.

The second game, WikiMystery, plays differently from Data Adventures. On one hand, the game has an arguably more interesting plot, where the player is a detective trying to solve a murder. Additionally, it is generated using only one input: the victim's name. The system finds people related to the victim, forming a pool of possible suspects, and evolves a small list of suspects that are somehow related to each other. It also provides evidence of innocence to any suspect that is, as the name implies, innocent. The player's goal is to find the one suspect which has no evidence of innocence, and arrest him or her. It thus requires that all four pieces of evidence (one per innocent NPC) are collected before the game can be completed. Figures  and show a location screen and the accusation screen, where the player identifies the culprit and provides evidences of innocence.

In DATA Agent, the player acts as a time-traveler in charge of finding a murder suspect, who went back in time and killed an important person. The game provides a list of suspects, and the player must travel through locations and uncover clues by talking to NPCs or interacting with items, in order to identify which among the suspects is the culprit. Similar to WkiMystery, DATA Agent's generator is capable of creating a full adventure when given a real person's name. This person becomes the center of the story, as the victim of a murder. Using artificial intelligence techniques over Wikipedia and DBpedia content, the system finds articles related to the person's article, and fleshes out links between suspects and the victim. Every in-game NPC, object, location, dialog or image is created from real information. Unlike WikiMystery, there is no evidence of innocence. The game finishes when an a suspect NPC is interrogated by the player and answers wrongly on personal information; the player must have collected the real information during gameplay. NPCs in the game have a much more involved dialog system, and can give information about suspects or about themselves, such as their birth day and occupation, or the reason they were chosen as suspects by the system. Figures  and show a dialog screen and an in-game location.

# Designing Games for Maximalism

A major challenge of maximalist game design is deciding what to prioritize. One can shape data in order to fit the game, or modify the game design to better showcase the original data. One can also have data ingrained in the game mechanics, directly affecting gameplay, or show the data in a decorative manner. Maintaining a balance between data transformation to fit other data and the game itself, or staying faithful to the original data while providing an engaging experience is challenging. This section describes two dimensions of maximalist game design: *Data Transformation versus Data Fidelity* and *Functionality versus Decoration*.

## Data Transformation versus Data Fidelity

The tension between data fidelity and data transformation is rooted in the priorities of a maximalist designer: the original game design or the original data. When using open data, designers may wish to adapt that data to the game, or to keep the data as it is and mold the game around it. Extensive data transformations may improve the game experience, but are also susceptible to loss of information or inaccuracies.

Transforming data gives designers more freedom and might be preferred if they have an inflexible idea, or if the data itself is malleable. DATA Agent is such an example of data transformation. In the game, the engine transforms individual facts about separate people into a murder mystery. The facts are also transformed into dialog lines, used by NPCs when prompted by the player. Some facts are altered purposely, in order to "lie": the culprit's dialog differs from reality in order to point to the time-agent (and thus, the player) that he or she is guilty. WikiMystery, on the other hand, uses proof of innocence in a similar manner but never misrepresents the actual data: all proofs given by NPCs are true, and the player must memorize them in order to use them in the game's accusation sequence.

On the other hand, designers may instead wish to stay faithful to the original data, molding the game to the data instead. This way, information present in the data is more likely to be clearly presented within game content. The rigidity to data restricts what kind of game elements can be used, or forces designers to be creative in their implementations. While less time might be spent cleaning and translating data, more time will likely be spent on raw game and mechanic design. An example can be found in Data Adventures, where data instantiated in the game is sourced from OSM and Wikipedia articles about people, places, and concepts. The designers built a game that could involve all four of those elements: a game where the player travels around the world searching for links to the goal NPC. It introduces some alterations from the original material, but most of it remains unchanged in the game. However, the game lacks a convincing narrative and theme, such as a murder mystery in later installments of the data adventures series. Another example is WikiRace[^1], where the game uses Wikipedia to navigate the game.

## Functional versus Decorative

Another dimension pertaining to maximalist game design is functionality versus decoration. We define data being *functional* when it has a strong impact on gameplay. If the player does not have to interact with the data, or the data does not impact gameplay in a significant way, then it is *decorative*.

In order to be functional, data can be incorporated in a variety of ways. In DATA Agent, dialog and character names heavily rely on open data. To progress in the game, you must interact with these characters and talk to them. The data is functional, as remembering which NPC (by name) has a certain fact is necessary to identify the culprit. In OpenTrumps , the raw mechanics of the game come from open data, as the cards themselves are created from it. The maps in *FreeCiv* generated by () are based on real-world terrain data, and impact gameplay as terrain affects players' city production.

On the other hand, any data that does not serve a functional purpose is decorative. Data can serve a decorative purpose in many ways. A Rogue Dream uses open data to name player abilities, however the in-game effects of the abilities are not affected by their names or the underlying data. In DATA Agent, city maps and NPC profile images are used as visual stimuli and play no mechanical role in game. *World of Warcraft* (Blizzard, 2004) uses real-world time to create an aesthetic day-night cycle in-game, which has no affect on actual gameplay.

## Instances of Data-driven Design

Figure  shows the two dimensions described above. The X-axis represents Data Transformation versus Data Fidelity, while the Y-axis represents Functional versus Decorative. Games where the goal is to preserve the original data, adapting the game to do so, are on the leftmost side of the figure: WikiRace and OpenTrumps exemplify this. Data in these games is also extremely functional: all mechanics in these games rely on a direct interaction with and understanding of the data (in WikiRace through reading the articles, and OpenTrumps through the values that affect deck superiority). Less faithful to the data, but similarly functional, is the FreeCiv map generation where geographic data is used as-is; resource placement is based on the original data but also adapted (i.e. transformed) for playability.

Moving upwards along the Y-axis, we find Open Data Monopoly and WikiMystery. Both use data in a decorative manner, the former as names for lots on the board and the latter as images, but some of the original data is also functionally translated (e.g. lots prices in Open Data Monopoly and proof of evidence in WikiMystery). On the far end of the Y-axis we have ANGELINA , which uses visuals as framing devices but without affecting the core platformer gameplay. The visuals are based on text of a newspaper article: the source data is transformed via natural language processing, tone extraction, and image queries.

# Purposes of Maximalist Games

While we attempt to highlight the principles and directions for designing maximalist game experiences, it is important to consider the purpose of such a game. Maximalist game design is desirable for many different reasons, highlighted in this section. Depending on the purpose, moreover, the design priorities could shift in the spectrum of data fidelity or decoration versus function.

## Learning

Modern-day practice sees students of all levels refer to Wikipedia for definitions, historical information, and tutorials. When browsing such knowledge repositories, it is common to access linked articles that may not have been part of the topic of inquiry. Maximalist game design that exploits open sources of knowledge, such as Wikipedia, can be used as a tool for learning in a playful context. One strength found in games is their ability to motivate a player for long periods of time. They also allow several ways to engage players, which can vary based on decisions and learning goals of game designers. Furthermore, games present failure as a necessary event for learning , causing players to explore and experiment more, since failing in game is less consequential than in the real world . Studies have also shown that when games immerse the player in a digital environment, they enhance the player's education of the content within the game . Games, unlike raw open data, are adaptive to players' skill level and are the most fun and engaging when they operate on the edge a player's *zone of proximal development* . Thus, we believe players can learn facts within open data during gameplay.

Data-driven maximalist games intended for learning can highlight and allow the player to interact with the data playfully. DATA Agent, to a degree, builds on this concept by creating NPCs out of articles about people, whether they are historical or fictional. These NPCs can then answer questions about their birth date or life's work. More relevant to the game progression, each NPC leads to an object (NPC, item or location) about another article, which can be interacted with and is associated to the current NPC somehow. The Data Adventures series was not designed with the explicit purpose of learning in mind; possibly, alternative design priorities could build on more "educational" principles. It would be possible, for example, to check for understanding by asking the player questions relating to the data in a diegetic manner — not unlike DATA agent, where the player must interrogate suspects and cross-check with their own obtained knowledge to detect falsehoods.

Information used to instantiate data should be fairly transparent to the player-learner. Therefore, transformation of data should not be convoluted, and perhaps even textual elements from original articles can be used as "flavor text". The veneer between encyclopedic content and game content does not need to be thick, in order to ensure that the right information is provided. In terms of function versus decoration, maximalist games for learning tend to edge closer towards data that influences the outcome of a game session in order to motivate learners to understand and remember the data. Such checks for understanding, however gamified they may be, will have an impact on the success or failure of the game.

## Data exploration

Data transformed into interactive game content, forming a consistent whole that goes far beyond the sum of its parts, can allow human users to explore the data in a more engaging way. Data visualization has been used extensively with a broad variety of purposes — far beyond the ones listed here — to take advantage of how most humans can more easily think through diagrams . In that vein, gameplay content originating from data can act as a form of highly interactive data visualization. The fact that data from different sources is combined together based on associations imagined by an automated game designer allows players to reflect on the data and make new discoveries or associations of their own. Due to the potential of emotional engagement that games have beyond mere 2D bar plots, the potential for lateral thinking either through visual, semantic, gameplay or emotional associations is extraordinary. In order for a game to offer an understanding of the data that is used to instantiate it and allow for that data to be re-imagined, the transformation into game content should be minimal. Examples of data games which already perform such a highly interactive data visualization are BarChartBall or OpenTrumps . However, a more maximalist approach could benefit games like the above by providing a more consistent storyline and progression, as well as a stronger emotional investment in the data.

## Contemporaneity

Automated game design has been always motivated, to a degree, by the desire to create perpetually fresh content. With data-driven design, this can be taken a step further by generating a new game every day. Such a game could be contextually relevant based on the day itself, e.g. building around historical events which happened on this day (such an extensive list can be found on <a href="onthisday.com" class="uri">onthisday.com</a> and Wikipedia) or people who had important personal events on that day (e.g. date of birth, date of death, graduation day). Moreover, the social context can be mined and used to drive the automated design process by including for instance trending topics of Twitter or headlines of today's newspapers. Early examples of such data-driven process have been explored for example by ANGELINA which used titles and articles from The Guardian website and connected them with relevant visuals and the appropriate mood. It is expected that a more maximalist data-driven design process would strengthen the feeling of contemporaneity by including more data sources (i.e. more data to transform) or stronger gameplay implications (i.e. broader transformations and functional impact).

Contemporaneity can make games generated on a specific day appealing to people who wish to get a "feel" for current issues but not necessarily dig deeply. On the other hand, the plethora of games (30 games per month alone) and the fact that each game is relevant to that day only could make each game itself less relevant. Contemporaneity and the fleeting nature of daily events could be emphasized if each game was playable only during the day that it was produced, deleting all its files when the next game is generated. This would enhance the perceived value of each game, similarly to *permadeath* in rogue-like games as it enhances nostalgia and the feeling of loss when a favorite gameworld is forever lost.

Any maximalist game could satisfy a contemporaneity goal, but such games can be more amenable to data transformation. For example, data could be transformed to more closely fit the theme of the day, e.g. query only female NPCs on International Women's Day. Contemporaneous data can be functional (to more strongly raise awareness of issues) but can also easily be decorative, e.g. giving a snowy appearance to locations during the Christmas holidays.

## Personalization

When game content is generated from data, it is possible to highlight certain bits of information. When the game takes player input as part of the data selection process, it personalizes their experience. If player information is available in the form of interests, important personal dates such as birthdays, or even social networks, the potential data sources that can be selected to form the game can be narrowed down. Presenting game content which is personally relevant (e.g. adventures with NPCs based on people living before Christ for an archeology student), or contextually relevant (such as solving the murder of an NPC born on the player's birthday) could contribute to a more engaging experience. It might also be possible to tailor the game's source repositories based on such personal interests. There are numerous online wikis, most of which follow a common format; therefore a user can implicitly (via personal interests) or explicitly (by providing a direct URL) switch search queries of a data-driven maximalist game to a specific wiki of choice.

## Opinion & Critique

Often designers want to make a statement through their games. For instance, Game-o-matic creates games from manually defined associations (as *micro-rhetorics*). *September 12th: A Toy World* (Newsgaming 2003) makes a political statement about the futility of America's War on Terror. Open data could similarly be used in a game to critique some aspect of culture by adding a weight of relevance and realism. For instance, a game such as *September 12th* could use the real map or skyline of Baghdad, or data on daily deaths in Iraq, to instantiate the challenge of the game. Similarly, if designers wish to critique the unprofessional use of social media in the White House, one could use real tweets to form dialog lines rather than generating them as in DATA Agent .

## Entertainment

Ostensibly, all games have entertainment as a (primary or secondary) purpose. This includes maximalist games, even if they have an additional purpose as listed in this paper. It is meaningful therefore to investigate what data-driven maximalist design has to offer to the entertainment dimension of any such game. Since maximalism —as we define it— does not necessarily apply to the mechanics of a game, a more relevant dimension is the end-user aesthetic that such games facilitate, following the mechanics-dynamics-aesthetics framework of (). Data-driven maximalist games primarily enhance the aesthetic of *discovery*, similarly to data exploration via such a game, and *expression* if it can be personalized to a user based on provided input such as birthday, hometown or interests. In many ways, data-driven games can enhance the aesthetic of *fantasy* by using and transforming real-world information. DATA agent, for example, describes an alternate history setting where a famous historical figure has been murdered (often by colleagues). The fantasy aesthetic is further enhanced by having a player take the role of a detective traveling through time and space to interrogate suspects. Other possible aesthetics that can be enhanced through data are *sensation* if the data comes from sources of high quality video, audio, or visuals (e.g. paintings of the National Gallery of London), or *fellowship* if the data comes from other users (e.g. anonymous users' trending tweets or social media postings of the player's friends). Evidently, games geared primarily towards entertainment can be fairly flexible in terms of data transformation, and can adapt the data to the intended game mechanics and game flow. While data can act as a decoration in such games (if intended to enhance the sensation aesthetic), in general games intended primarily for entertainment are fairly focused in the mechanics and feedback loops, and thus data would primarily be transformed into functional elements.

## Human Computation

Presenting hand-picked results from a vast database in an engaging, playful way is not only relevant for humans to consume. The human-computer interaction loop can be closed if human users provide feedback on the quality of the data itself. This human feedback can be used internally by the game, adapting its criteria in order to avoid unwanted data repositories, queries, associations or transformations made to the data. For instance, a future DATA agent version could re-compute the set of suspects for the next games (removing one or more suspects from the pool of possible suspects) if a player provides negative feedback explicitly (e.g. via a 'report' button) or implicitly (e.g. by being unable to solve the mystery). More ambitiously, the positive or negative feedback of players engaging with the playable —transformed— data can be fed back to the source repositories which instantiated the game. This can allow for instance misinformation found in Wikipedia to be flagged, alerting moderators that either a human error (e.g. a wrong date or a misquote) or malformed data (e.g. unreadable titles) exists and must be corrected. Whether these corrections should be made by an expert human curator, or directly based on player interactions with the game could be a direction for future research.

# Issues with Data-Driven Game Design

Accomplishing *good* data-driven maximalist game design is a challenge. While the previous sections presented ways of doing so, there are still many implementation- or game-specific details which affect the design process. Beyond the core challenge of a good game design, there are several peripheral challenges to the design task itself which however spring from the practice of data-driven design. We elaborate on those peripheral challenges here.

## Legal & Ethical Issues

Any software which relies on external data that it cannot control may be prone to legal or ethical violations. Privacy of personal information may be a concern for a game generated from the social media profile of a user, especially if that game can then be played by a broader set of people. Using results from Google Images may lead to direct infringements of copyrights; using results from models built from text mining, on the other hand, may or may not result in such copyright infringements depending on whether the model returns actual copyrighted material. The issue of copyright becomes more complex when the data is transformed: relevant to data mining, a judge has ruled for fair use for Google Books as "Google Books is also transformative in the sense that it has transformed book text into data for purposes of substantive research, including data mining and text mining in new areas" . One can only assume that transformations of data into game content, depending on the fidelity to the original data and the purpose (e.g. data exploration and education), would make for a clearer case of fair use.

Game content built on fair use or open data combined into an interactive experience may lead to unexpected issues. This is especially true in cases where the player has sufficient agency to interpret or act upon content of high fidelity with the original data in an open-ended fashion: consider, for example, a violent shooter game where opponents' visual depictions (3D models or faces) are those of Hollywood celebrities. Even in Data Adventures, where player interaction is fairly "curated", a generated game featured solving the murder of Justin Bieber . Apart from the fictional narrative of a popular celebrity's death, the game identifies another celebrity as the murderer: both of these decisions may cause concern to highly visible people (be they depicted murdered, murderers, or suspects). A disclaimer that the game characters are fictional can only alleviate that much of the ethical responsibility of game designers for such data-driven games.

## Misinformation & Bias

Connected to the concerns of misrepresenting contemporary or historical celebrities are the inherent issues of error in the source data. Before data is transformed into game content, open repositories that can be edited by anyone can be saturated by personal opinion and perhaps deliberate misinformation. As noted previously, not all data provided by different stakeholders in the information age are factual; this may be more pronounced in certain repositories than others. Beyond deliberate misinformation, an inherent bias is also present even in "objective" data. For example, algorithms for Google query results or image results are based on machine learned models that may favor stereotypes (based on what most people think of a topic). Even though WikiMystery uses what we arguably consider "objective" repositories such as Wikipedia, the 8 most popular locations in 100 generated games were in North America , pointing to a bias of the articles or the DBpedia entries chosen to be digitized. Other cases where misinformation may arise is when different content is combined inaccurately: examples from the Data Adventures series include cases where an image search for a character named Margaret Thatcher resulted in an image of Aung San Suu Kyi . When data-driven design uses social network data such as trending topics on Twitter, then the potential for sensitive or provocative topics to be paired with inappropriate content or combined in an insensitive way becomes a real possibility. If data-driven maximalist games are intended towards critique or opinion, the misinformation or misappropriation could be deliberately inserted by a designer (by pairing different repositories) or accidentally introduce a message that runs contrary to the intended one.

# Outlook

Maximalist game design encourages creation through reuse and combination. If one imagines its most powerful form, it would likely involve taking any mixture of information, pouring it into any game content cast, and reveling in its results. It would provide a freedom to interact with any data in the best, most personalized way possible.

Current PCG techniques allow for unlimited playability for a large variety of games. However, they can lack a level of contemporaneity and relevance that could be provided by open data. Additionally, research has suggested that concepts can be effectively learned through gameplay . Using games as a method of interacting with open data may create a novel way for learning about the data in a fun way. Rather than use Wikipedia to learn about specific people and places for the first time, players could play games where they can talk to these people and visit these places.

Open data is available to all, to create as well as consume. Sometimes the data is inaccurate. The idea of visualizing this information in any form can provide means to "debug" the original data, in a more engaging way than just browsing Wikipedia or poring through a massive database.

# Conclusion

This paper discussed an approach to game design inspired by the notion of maximalism in the arts. It encourages the reuse and combination of heterogeneous data sources in the creative design process. Maximalist game design embraces the generation of game content using different data sources, re-mixing them in order to achieve something new.

We drew from our experience with the Data Adventures series to propose a mapping of the maximalist game design space along two dimensions, *data transformation versus data fidelity* and *functionality versus decoration*. The former focuses on the extent that the data is transformed from its original form, while the latter refers to the actual role of the data in the game. Additionally, we described how maximalist game design can serve different purposes in the design process and which tradeoffs emerge from each purpose. Finally, we highlight issues and ethical concerns that may arise from and in maximalist games.

# Acknowledgements

Gabriella Barros acknowledges financial support from CAPES and Science Without Borders program, BEX 1372713-3. Antonios Liapis has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 693150.

[^1]: http://2pages.net/wikirace.php