Title: CONDITIONAL IMAGE SYNTHESIS WITH AUXILIARY CLASSIFIER GANS

Abstract: Synthesizing high resolution photorealistic images has been a long-standing challenge in machine learning. In this paper we introduce new methods for the improved training of generative adversarial networks (GANs) for image synthesis. We construct a variant of GANs employing label conditioning that results in 128 × 128 resolution image samples exhibiting global coherence. We expand on previous work for image quality assessment to provide two new analyses for assessing the discriminability and diversity of samples from class-conditional image synthesis models. These analyses demonstrate that high resolution samples provide class information not present in low resolution samples. Across 1000 ImageNet classes, 128 × 128 samples are more than twice as discriminable as artificially resized 32 × 32 samples. In addition, 84.7% of the classes have samples exhibiting diversity comparable to real ImageNet data.

1. INTRODUCTION:

Characterizing the structure of natural images has been a rich research endeavor. Natural images obey intrinsic invariances and exhibit multi-scale statistical structures that have historically been difficult to quantify (Simoncelli & Olshausen, 2001) . Recent advances in machine learning offer an opportunity to substantially improve the quality of image models. Improved image models advance the state-of-the-art in image denoising (Ballé et al., 2015) , compression (Toderici et al., 2016) , in-painting (van den Oord et al., 2016a) , and super-resolution (Ledig et al., 2016) . Better models of natural images also improve performance in semi-supervised learning tasks (Kingma et al., 2014; Springenberg, 2015; Odena, 2016; Salimans et al., 2016) and reinforcement learning problems (Blundell et al., 2016) .

One method for understanding natural image statistics is to build a system that synthesizes images de novo. There are several promising approaches for building image synthesis models. Variational autoencoders (VAEs) maximize a variational lower bound on the log-likelihood of the training data (Kingma & Welling, 2013; Rezende et al., 2014) . VAEs are straightforward to train but introduce potentially restrictive assumptions about the approximate posterior distribution (but see Rezende & Mohamed (2015) ; Kingma et al. (2016) ). Autoregressive models dispense with latent variables and directly model the conditional distribution over pixels (van den Oord et al., 2016a; b) . These models produce convincing samples but are costly to sample from and do not provide a latent representation. Invertible density estimators transform latent variables directly using a series of parameterized functions constrained to be invertible (Dinh et al., 2016) . This technique allows for exact log-likelihood computation and exact inference, but the invertibility constraint is restrictive.

Generative adversarial networks (GANs) offer a distinct and promising approach that focuses on a game-theoretic formulation for training an image synthesis model (Goodfellow et al., 2014) . Recent work has shown that GANs can produce convincing image samples on datasets with low variability and low resolution (Denton et al., 2015; Radford et al., 2015) . However, GANs struggle to generate globally coherent, high resolution samples -particularly from datasets with high variability. Moreover, a theoretical understanding of GANs is an on-going research topic (Uehara et al., 2016; Mohamed & Lakshminarayanan, 2016) . In this work we demonstrate that that adding more structure to the GAN latent space along with a specialized cost function results in higher quality samples. We exhibit 128 × 128 pixel samples from all classes of the ImageNet dataset (Russakovsky et al., 2015) with increased global coherence (Figure 1 ). Importantly, we demonstrate quantitatively that our high resolution samples are not just naive resizings of low resolution samples. In particular, downsampling our 128 × 128 samples to 32 × 32 leads to a 50% decrease in visual discriminability. We also introduce a new metric for assessing the variability across image samples and employ this metric to demonstrate that our synthesized images exhibit diversity comparable to training data for a large fraction (84.7%) of ImageNet classes.

2. BACKGROUND:

A generative adversarial network (GAN) consists of two neural networks trained in opposition to one another. The generator G takes as input a random noise vector z and outputs an image X f ake = G(z). The discriminator D receives as input either a training image or a synthesized image from the generator and outputs a probability distribution P (S | X) = D(X) over possible image sources. The discriminator is trained to maximize the log-likelihood it assigns to the correct source:

L = E[log P (S = real | X real )] + E[log P (S = f ake | X f ake )]

The generator is trained to minimize that same quantity.

The basic GAN framework can be augmented using side information. One strategy is to supply both the generator and discriminator with class labels in order to produce class conditional samples (Mirza & Osindero, 2014) . Class conditional synthesis can significantly improve the quality of generated samples (van den Oord et al., 2016b) . Richer side information such as image captions and bounding box localizations may improve sample quality further (Reed et al., 2016a; b) .

Instead of feeding side information to the discriminator, one can task the discriminator with reconstructing side information. This is done by modifying the discriminator to contain an auxiliary decoder network 1 that outputs the class label for the training data (Odena, 2016; Salimans et al., 2016) or a subset of the latent variables from which the samples are generated (Chen et al., 2016) . Forcing a model to perform additional tasks is known to improve performance on the original task (e.g. Sutskever et al. (2014) ; Szegedy et al. (2014) ; Ramsundar et al. (2016) ). In addition, an auxiliary decoder could leverage pre-trained discriminators (e.g. image classifiers) for further improving the synthesized images (Nguyen et al., 2016) . Motivated by these considerations, we introduce a model that combines both strategies for leveraging side information. That is, the model proposed below is class conditional, but with an auxiliary decoder that is tasked with reconstructing class labels.

(noise) (latent) (data) InfoGAN (Chen, et al., 2016) . . .

(noise) (class) (data) AC-GAN (Present Work) (noise) (class) (data)

Conditional GAN (Mirza & Osindero, 2014) (noise) . . .

(class) (data):

Semi-Supervised GAN (Odena, 2016; Salimans, et al., 2016) Figure 2 : A comparison of several GAN architectures with the proposed AC-GAN architecture.

3. AC-GANS:

We propose a variant of the GAN architecture which we call an auxiliary classifier GAN (or AC-GAN -see Figure 2 ). In the AC-GAN, every generated sample has a corresponding class label, c ∼ p c in addition to the noise z. G uses both to generate images X f ake = G(c, z). The discriminator gives both a probability distribution over sources and a probability distribution over the class labels, P (S | X), P (C | X) = D(X). The objective function has two parts: the log-likelihood of the correct source, L S , and the log-likelihood of the correct class, L C .

L S = E[log P (S = real | X real )] + E[log P (S = f ake | X f ake )] L C = E[log P (C = c | X real )] + E[log P (C = c | X f ake )] D is trained to maximize L S + L C while G is trained to maximize L C -L S .

AC-GANs learn a representation for z that is independent of class label (e.g. Kingma et al. (2014) ).

Early experiments demonstrated that increasing the number of classes trained on while holding the model fixed decreased the quality of the model outputs (Appendix B). The structure of the AC-GAN model permits separating large datasets into subsets by class and training a generator and discriminator for each subset. We exploit this property in our experiments to train across the entire ImageNet data set.

4. RESULTS:

We train several AC-GAN models on the ImageNet data set (Russakovsky et al., 2015) . Broadly speaking, the architecture of the generator G is a series of 'deconvolution' layers that transform the noise z and class c into an image (Odena et al., 2016) . We train two variants of the model architecture for generating images at 128 × 128 and 64 × 64 spatial resolutions. The discriminator D is a deep convolutional neural network with a Leaky ReLU nonlinearity (Maas et al., 2013) . See Appendix A for more details. As mentioned earlier, we find that reducing the variability introduced by all 1000 classes of ImageNet significantly improves the quality of training. We train 100 AC-GAN modelseach on images from just 10 classes -for 50000 mini-batches of size 100.

Evaluating the quality of image synthesis models is challenging due to the variety of probabilistic criteria (Theis et al., 2015) and the lack of a perceptually meaningful image similarity metric. Nonetheless, in subsequent sections we attempt to measure the quality of the AC-GAN by building several ad-hoc measures for image sample discriminability and diversity. Our hope is that this work might provide quantitative measures that may be used to aid training and subsequent development of image synthesis models. 

4.1. GENERATING HIGH RESOLUTION IMAGES IMPROVES DISCRIMINABILITY:

Building a class-conditional image synthesis model necessitates measuring the extent to which synthesized images appear to belong to the intended class. In particular, we would like to know that a high resolution sample is not just a naive resizing of a low resolution sample. Consider a simple experiment: pretend there exists a model that synthesizes 32 × 32 images. One can trivially increase the resolution of synthesized images by performing bilinear interpolation. This would yield higher resolution images, but these images would just be blurry versions of the low resolution images that are not discriminable. Hence, the goal of an image synthesis model is not simply to produce high resolution images, but to produce high resolution images that are more discriminable than low resolution images.

To measure discriminability, we feed synthesized images to a pre-trained Inception network (Szegedy et al., 2015) and report the fraction of the samples for which the Inception network assigned the correct label2 . We calculate this accuracy measure on a series of real and synthesized images which have had their spatial resolution artificially decreased by bilinear interpolation (Figure 3 , top panels). Note that as the spatial resolution is decreased, the accuracy decreases -indicating that resulting images contain less class information (Figure 3 , scores below top panels). We summarized this finding across all 1000 ImageNet classes for the ImageNet training data (black), a 128 × 128 resolution AC-GAN (red) and a 64 × 64 resolution AC-GAN (blue) in Figure 3 (bottom, left). The black curve (clipped) provides an upper-bound on the discriminability of real images.

The goal of this analysis is to show that synthesizing higher resolution images leads to increased discriminability. The 128 × 128 model achieves an accuracy of 10.1% ± 2.0% versus 7.0% ± 2.0% with samples resized to 64 × 64 and 5.0% ± 2.0% with samples resized to 32 × 32. In other words, downsizing the outputs of the AC-GAN to 32 × 32 and 64 × 64 decreases visual discriminability by 50% and 38% respectively. Furthermore, 84.4% of the ImageNet classes have higher accuracy at 128 × 128 than at 32 × 32 (Figure 3 , bottom left).

We performed the same analysis on an AC-GAN trained to 64 × 64 spatial resolution. This model achieved less discriminability than a 128×128 AC-GAN model. Accuracies from the 64×64 model plateau at a 64×64 spatial resolution consistent with previous results. Finally, the 64×64 resolution model achieves less discriminability at 64 spatial resolution than the 128 × 128 model.

4.2. MEASURING THE DIVERSITY OF GENERATED IMAGES:

An image synthesis model is not very interesting if it only outputs one image. Indeed, a well-known failure mode of GANs is that the generator will collapse and output a single prototype that maximally fools the discriminator (Goodfellow et al., 2014; Salimans et al., 2016) . A class-conditional model of images is not very interesting if it only outputs one image per class. The Inception accuracy can not measure whether a model has collapsed. A model that simply memorized one example from each ImageNet class would do very well by this metric. Thus, we seek a complementary metric to explicitly evaluate the intra-class diversity of samples generated by the AC-GAN.

Several methods exist for quantitatively evaluating image similarity by attempting to predict human perceptual similarity judgements. The most successful of these is multi-scale structural similarity (MS-SSIM) (Wang et al., 2004b; Ma et al., 2016) . MS-SSIM is a multi-scale variant of a wellcharacterized perceptual similarity metric that attempts to discount aspects of an image that are not important for human perception (Wang et al., 2004a) . MS-SSIM values range between 0.0 and 1.0; higher MS-SSIM values correspond to perceptually more similar images. As a proxy for image diversity, we measure the MS-SSIM scores between randomly chosen pairs of images within a given class. Samples from classes that have higher diversity result in lower mean MS-SSIM scores (Figure 4 

4.3. GENERATED IMAGES ARE BOTH DIVERSE AND DISCRIMINABLE:

We have presented quantitative metrics demonstrating that AC-GAN samples may be diverse and discriminable but we have yet to examine how these metrics interact. Figure 6 shows the joint distribution of Inception accuracies and MS-SSIM scores across all classes. Inception accuracy and MS-SSIM are anti-correlated (r 2 = -0.16). In fact, 74% of the classes with low diversity (MS-SSIM ≥ 0.25) contain Inception accuracies ≤ 1%. These results suggest that GANs that drop modes the intended class. To compute the Inception accuracy, we modified a version of Inception-v3 supplied in https://github.com/openai/improved-gan/. are most likely to produce low quality images. Conversely, 78% of classes with high diversity (MS-SSIM < 0.25) have Inception accuracies that exceed 1%. In comparison, the Inception-v3 model achieves 78.8% accuracy on average across all 1000 classes (Szegedy et al., 2015) . A fraction of the classes AC-GAN samples reach this level of accuracy. This indicates opportunity for future image synthesis models.

4.4. COMPARISON TO PREVIOUS RESULTS:

Previous quantitative results for image synthesis models trained on ImageNet are reported in terms of log-likelihood (van den Oord et al., 2016a; b) . Log-likelihood is a coarse and potentially inaccurate measure of sample quality (Theis et al., 2015) . Addditionally, log-likelihood is intractable to compute for GANs. Instead we compare with previous state-of-the-art results on CIFAR-10 using a lower spatial resolution (32 × 32). Following the procedure in Salimans et al. (2016) , we compute the Inception score3 for 50000 samples from an AC-GAN with resolution (32 × 32), split into 10 groups at random. We also compute the Inception score for 25000 extra samples, split into 5 groups at random. We select the best model based on the first score and report the second score. Performing a grid search across 27 hyperparameter configurations, we are able to achieve a score of 8.25 ± 0.07 compared to state of the art 8.09 ± 0.07 (Salimans et al., 2016) . Moreover, we accomplish this without employing any of the new techniques introduced in that work (i.e. virtual batch normalization, minibatch discrimination, and label smoothing). This provides additional evidence that AC-GANs are effective even without the benefit of class splitting (Appendix B).

4.5. SEARCHING FOR SIGNATURES OF OVERFITTING:

One possibility that must be investigated is that the AC-GAN has overfit on the training data. As a first check that the network does not memorize the training data, we identify the nearest neighbors of image samples in the training data measured by L1 distance in pixel space (Figure 7 ). The nearest neighbors from the training data do not resemble the corresponding samples. This provides evidence that the AC-GAN is not merely memorizing the training data. A more sophisticated method for understanding the degree of overfitting in a model is to explore that model's latent space by interpolation. In an overfit model one might observe discrete transitions in the interpolated images and regions in latent space that do not correspond to meaningful images (Bengio et al., 2012; Radford et al., 2015; Dinh et al., 2016) . Figure 8 (left) highlights interpolations in the latent space between several image samples. Notably, the generator learned that certain combinations of dimensions correspond to semantically meaningful features (e.g. size of the arch, length of a bird's beak) and there are no discrete transitions or 'holes' in the latent space. A second method for exploring the latent space of the AC-GAN is to exploit the structure of the model. The AC-GAN factorizes its representation into class information and a class-independent latent representation z.

Sampling the AC-GAN with z fixed but altering the class label corresponds to generating samples with the same 'style' across multiple classes (Kingma et al., 2014) . 

5. DISCUSSION:

This work introduced the AC-GAN architecture and demonstrated that AC-GANs can generate globally coherent ImageNet samples. We provided a new quantitative metric for image discriminability as a function of spatial resolution. Using this metric we demonstrated that our samples are more discriminable than those from a model that generates lower resolution images and performs a naive resize operation. We also analyzed the diversity of our samples with respect to the training data and provided some evidence that the image samples from the majority of classes are comparable in diversity to ImageNet training data. We hope that these metrics might provide quantitative measures of sample quality for evaluating and improving future image synthesis models. Image synthesis models provide a unique opportunity for performing semi-supervised learning. Namely, these models build a rich prior over natural image statistics that can be leveraged by classifiers to improve predictions on datasets for which few labels exist. The AC-GAN model can perform semi-supervised learning by simply ignoring the component of the loss arising from class labels when a label is unavailable for a given training image. Interestingly, prior work suggests that achieving good sample quality might be independent of success in semi-supervised learning (Salimans et al., 2016) .

A HYPERPARAMETERS:

Operation Kernel Strides Feature maps BN? Dropout Nonlinearity

G x (z) -110 × 1 × 1 input Linear N/A N/A 768 × 0.0 ReLU Transposed Convolution 5 × 5 2 × 2 384 √ 0.0 ReLU Transposed Convolution 5 × 5 2 × 2 256 √ 0.0 ReLU Transposed Convolution 5 × 5 2 × 2 192 √ 0.0 ReLU Transposed Convolution 5 × 5 2 × 2 3 × 0.0 Tanh D(x) -128 × 3 × 3 input Convolution 3 × 3 2 × 2 16 × 0.5 Leaky ReLU Convolution 3 × 3 1 × 1 32 √ 0.5 Leaky ReLU Convolution 3 × 3 2 × 2 64 √ 0.5 Leaky ReLU Convolution 3 × 3 1 × 1 128 √ 0.5 Leaky ReLU Convolution 3 × 3 2 × 2 256 √ 0.5 Leaky ReLU Convolution 3 × 3 1 × 1 512 √ 0.5 Leaky ReLU Linear N/A N/A 11 × 0.0

Soft-Sigmoid Optimizer Adam (α = 0.0002, β 1 = 0.5, β 2 = 10 -3 ) Batch size 100 Iterations 50000 Leaky ReLU slope 0.2 Weight, bias initialization Isotropic gaussian (µ = 0, σ = 0.02), Constant(0)

Table 1: Model hyperparameters. A Soft-Sigmoid refers to an operation over K + 1 output units where we apply a Softmax activation to K of the units and a Sigmoid activation to the remaining unit. We also use activation noise in the discriminator as suggested in Salimans et al. (2016) .

B MEASURING THE EFFECT OF CLASS SPLITS ON IMAGE SAMPLE QUALITY.:

Class conditional image synthesis affords the opportunity to divide up a dataset based on image label. In our final model we divide 1000 ImageNet classes across 100 AC-GAN models. In this section we describe early experiments that highlight the benefit of cutting down the diversity of classes for training an AC-GAN. We employed an ordering of the labels and divided it into contiguous groups of 10. This ordering can be seen in the following section, where we display samples from all 1000 classes. Two aspects of the split merit discussion: the number of classes per split and the intra-split diversity.

We find that training a fixed model on more classes harms the model's ability to produce compelling samples (Figure 9 ). Performance on larger splits can be improved by giving the model more parameters. However, using a small split is not sufficient to achieve good performance. We were unable to train a GAN (Goodfellow et al., 2014) to converge reliably even for a split size of 1. This raises the question of whether it is easier to train a model on a diverse set of classes than on a similar set of classes. We were unable to find conclusive evidence that the selection of classes in a split significantly affects sample quality.

:

Alternatively, one can force the discriminator to work with the joint distribution (X, z) and train a separate inference network that computes q(z|X)(Dumoulin et al.,

2016;Donahue et al., 2016).

One could also use the Inception score(Salimans et al., 2016), but our method has several advantages: accuracy figures are easier to interpret than exponentiated KL-divergences; accuracy may be assessed for individual classes; accuracy measures whether a class-conditional model generated samples from

The Inception score is given by exp(Ex[DKL(p(y|x) || p(y))]) where x is a particular image, p(y|x) is the conditional output distribution over the classes in a pre-trained Inception network(Szegedy et al., 2014) given x, and p(y) is the marginal distribution over the classes.

Figures and Tables:

Figure 1: 128×128 resolution samples from 5 classes taken from an AC-GAN trained on the ImageNet dataset. Note that the classes shown have been selected to highlight the success of the model and are not representative. Samples from all ImageNet classes are in the Appendix.

Figure 3: Generating high resolution images improves discriminability. Top: Training data and synthesized images from the zebra class resized to a lower spatial resolution (indicated above) and subsequently artificially resized to the original resolution. Inception accuracy is shown below the corresponding images. Bottom Left: Summary of accuracies across varying spatial resolutions for training data and image samples from 64 × 64 and 128 × 128 models. Error bar measures standard deviation across 10 subsets of images. Dashed lines highlight the accuracy at the output spatial resolution of the model. The training data (clipped) achieves accuracies of 24%, 54%, 81% and 81% at resolutions of 32, 64, 128, and 256 respectively. Bottom Right: Comparison of accuracy scores at 128 × 128 and 32 × 32 spatial resolutions (x and y axis, respectively). Each point represents an ImageNet class. 84.4% of the classes are below the line of equality. The green dot corresponds to the zebra class.

, left columns); samples from classes with lower diversity have higher mean MS-SSIM scores (Figure 4, right columns). Training images from the ImageNet training data contain a variety of mean MS-SSIM scores across the classes indicating the variability of image diversity in ImageNet classes (Figure 5, left panel, x-axis). Note that the highest mean MS-SSIM score (indicating the least variability) is 0.25 for the training data. We calculate the mean MS-SSIM score for all 1000 ImageNet classes generated by the AC-GAN model. We track this value during training to identify whether the generator has collapsed (Figure 5, right panel, red curve). We also employ this metric to compare the diversity of the training images to the samples from the GAN model after training has completed. Figure 5 (left) plots the mean MS-SSIM values for image samples and training data broken up by class. The blue line is the line of equality. Out of the 1000 classes, we find that 847 have mean sample MS-SSIM scores below that of the maximum MS-SSIM for the training data. In other words, 84.7% of classes have sample variability that exceeds that of the least variable class from the ImageNet training data.

Figure 4: Examples of different MS-SSIM scores. The top and bottom rows contain AC-GAN samples and training data, respectively.

Figure 5: (Left) Comparison of the mean MS-SSIM scores between pairs of images within a given class for ImageNet training data and samples from the GAN (blue line is equality). The horizontal red line marks the maximum MS-SSIM value across all ImageNet classes. Each point is an individual class. The mean standard deviation of scores across the training data and the samples was 0.06 and 0.08 respectively. Scores below the red line (84.7% of classes) arise from classes where GAN training largely succeeded. (Right) Intra-class MS-SSIM for selected ImageNet classes throughout a training run. Classes that successfully train tend to have decreasing mean MS-SSIM scores, to a point.

Figure 6: Inception accuracy vs MS-SSIM for all 1000 ImageNet classes (r 2 = -0.16). Samples from AC-GAN models do not achieve variability at the expense of discriminability.

Figure 7: Nearest neighbor analysis. (Left) Samples from a single ImageNet class. (Right) Corresponding nearest neighbor (L1 distance) in training data for each sample.

Figure 8 (right) shows samples from 8 bird classes. Elements of the same row have the same z. Although the class changes for each column, elements of the global structure (e.g. position, layout, background) are preserved, indicating that AC-GAN can represent certain types of 'compositionality'.

Figure 8: (Left) Latent space interpolations for selected ImageNet classes. Left-most and right-columns show three pairs of image samples -each pair from a distinct class. Intermediate columns highlight linear interpolations in the latent space between these three pairs of images. (Right) Class-independent information contains global structure about the synthesized image. Each column is a distinct bird class while each row corresponds to a fixed latent code z.

Figure 9: Mean pairwise MS-SSIM values for 10 ImageNet classes plotted against the number of ImageNet classes used during training. We fix everything except the number of classes trained on, using values from 10 to 100. We only report the MS-SSIM values for the first 10 classes to keep the scores comparable. MS-SSIM quickly goes above 0.25 (the red line) as the class count increases. These scores were computed using 9 random restarts per class count, using the same number of training steps for each model.

Several directions exist for building upon this work. Much work needs to be done to improve the visual discriminability of the 128×128 resolution model. Although some synthesized image classes exhibit high Inception accuracies, the average Inception accuracy of the model (10.1% ± 2.0%) is still far below real training data at 81%. One immediate opportunity for addressing this is to augment the discriminator with a pre-trained model to perform additional supervised tasks (e.g. image segmentation,Ronneberger et al. (2015)). Such techniques might allow for the synthesis of even higher resolution images with global coherence and meaningful visual content.Improving the robustness and reliability of training a GAN is an ongoing research topic. Only 84.7% of the ImageNet classes avoided mode dropping and exhibited a diversity comparable to real training data. Training stability was vastly aided by dividing up 1000 ImageNet classes across 100 AC-GAN models. Building a single unified model that could generate diverse samples from all 1000 classes would be an important step forward.

Table content: <table/>

Peer review (iclr2017):
Apologies for the late review. This submission proposes method for class-conditional generative image modeling using auxiliary classifiers. Compared to normal GANs the generator also receives a randomly sampled class label c from the class distribution. The discriminator has two outputs and two corresponding objectives: determine whether a sample is real or generated, and independently to predict the (real or sampled) class label corresponding to the sample. Figure 2. nicely illustrates related methods - this particular method bears similarities to InfoGANs and Semi-supervised GANs. Compared to infogans, this method also encourages correspondence between the latent c and the real class labels for the real examples (whereas infogans are presented as fully unsupervised). The authors attempt at evaluating the method quantitatively by looking at the discriminability and diversity of samples. It is found - not surprisingly - that higher resolution improves discriminability (because more information is present). Discriminability: Figure 3 doesn’t have legends so it is a bit hard to understand what is going on. Furthermore, my understanding is that when evaluating discriminability the authors downsample and then bicubically upsample the image, which is much more like a blurring, very different from retraining all the models to work on low resolution in the first place. Diversity: The authors try to quantitatively evaluate diversity of samples by measuring the average MS-SSIM between randomly selected pairs of points within each class. I think this method is significantly flawed and limited, for reasons mentioned in (Theis et al, 2015, A note on the evaluation…). In its behaviour, MS-SSIM is not that dissimilar from Euclidean distance - although it is nonlinear and is bounded between -1 and 1. Evaluating diversity/entropy of samples in high dimensions is very hard, especially if the distributions involved are non-trivial for example concentrated around manifolds. Consider for example a generative model which randomly samples just two images. Assuming that the MSSSIM between these two images is -1, this generative model can easily achieve an average MSSSIM score of 0, implying a conclusion that this model has more diversity than the training data itself. Conversely, SSIM is designed not to be sensitive to contrast and average pixel intensity, so if a model is diverse in this sense, that will be ignored by this measure. Overall, the paper proposes a new way to incorporate class labels into training GAN-type models. As far as I know the particular algorithm is novel, but I consider it incremental compared to what has been done before. I think the proposed evaluation metrics are flawed, especially when evaluating the diversity of the samples for the aforementioned reasons.