Title: THE GAN LANDSCAPE: LOSSES, ARCHITECTURES, REGULARIZATION, AND NORMALIZATION

Abstract: Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of "tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.
Under review as a conference paper at ICLR 2019 useful in the context of high-capacity architectures. Finally, we discuss some common pitfalls, reproducibility issues, and practical considerations. We provide reference implementations, including training and evaluation code on Github 1 and provide pre-trained models on TensorFlow Hub. 2 2 THE GAN LANDSCAPE 2.1 LOSS FUNCTIONS Let P denote the target (true) distribution and Q the model distribution. Goodfellow et al. (2014) suggest two loss functions: the minimax GAN and the non-saturating (NS) GAN. In the former the discriminator minimizes the negative log-likelihood for the binary classification task. In the latter the generator maximizes the probability of generated samples being real. In this work we consider the non-saturating loss as it is known to outperform the minimax variant. The corresponding loss functions are
In Wasserstein GAN (WGAN) (Arjovsky et al., 2017) the authors propose to consider the Wasserstein divergence instead of the original Jensen-Shannon (JS). In particular, under the optimal discriminator, minimizing the proposed value function with respect to the generator minimizes the Wasserstein distance between P and Q. The drawback is that one has to ensure a 1-Lipschitz discriminator due to exploited Kantorovich-Rubenstein duality. The corresponding loss functions are
Finally, we consider the least-squares loss (LS) which corresponds to minimizing the Pearson χ 2 divergence between P and Q (Mao et al., 2016) . The intuition is that this loss function is smooth and saturates slower than the sigmoid cross-entropy loss of the JS formulation. The corresponding loss functions are
REGULARIZATION AND NORMALIZATION OF THE DISCRIMINATOR

1. INTRODUCTION:

Deep generative models are a powerful class of unsupervised machine learning models. The power of these models was recently harnessed in a variety of applications, including image generation, learned compression, and domain transfer (Isola et al., 2017; Radford et al., 2016; Agustsson et al., 2018; Tschannen et al., 2018) . Generative adversarial networks (Goodfellow et al., 2014) are one of the main approaches to learning such models in a fully unsupervised fashion. The GAN framework can be viewed as a two-player game where the first player, the generator, is learning to transform some simple input distribution (usually a standard multivariate Normal or uniform) to a distribution on the space of images, such that the second player, the discriminator, cannot tell whether the samples belong to the true distribution or were synthesized. Both players aim to minimize their own loss and the solution to the game is the Nash equilibrium where neither player can improve their loss unilaterally. This powerful framework can also be derived by minimizing a divergence between the model distribution and the true distribution (Nowozin et al., 2016; Arjovsky et al., 2017) .

Training GANs involves solving a minimax problem over the parameters of the generator and the discriminator which are usually parameterized as deep convolutional neural networks. Consequently, this minimax problem is notoriously hard to solve in practice. As a result, a plethora of loss functions, regularization and normalization schemes, coupled with neural architecture choices, have been proposed (Goodfellow et al., 2014; Salimans et al., 2016; Miyato et al., 2018; Gulrajani et al., 2017; Arjovsky et al., 2017; Mao et al., 2016) .

Our contributions. In this work we provide a thorough empirical analysis of these competing approaches, and help the researchers and practitioners navigate this space. We first define the GAN landscape -the set of loss functions, normalization and regularization schemes, and the most commonly used architectures. We explore this search space on several modern large-scale data sets by means of hyperparameter optimization, considering both "good" sets of hyperparameters reported in the literature, as well as ones obtained by Gaussian Process regression. By analyzing the impact of the loss function, we conclude that the non-saturating loss is sufficiently stable across data sets, architectures and hyperparameters. We then proceed to decompose the effect of various normalization and regularization schemes, as well as varying architectures. We show that both gradient penalty (Gulrajani et al., 2017) as well as spectral normalization (Miyato et al., 2018) are Gradient norm penalty. In the context of Wasserstein GANs this penalty can be interpreted as a soft penalty for the violation of 1-Lipschitzness (WGAN GP) (Gulrajani et al., 2017) . Hereby, the gradient is evaluated on a linear interpolation between training points and generated samples as a proxy to the optimal coupling. The gradient penalty can also be evaluated around the data manifold which encourages the discriminator to be piece-wise linear in that region (Dragan) (Kodali et al., 2017) . However, the gradient norm penalty can be considered purely as a regularizer for the discriminator and it was shown that it can improve the performance for other losses (Fedus et al., 2018) . Furthermore, the penalty can be scaled by the "confidence" of the discriminator in the context of f-divergences (Roth et al., 2017) . A drawback of gradient penalty (GP) regularization scheme is that it can depend on the model distribution Q which changes during training. One drawback of Dragan is that it is unclear to which extent the Gaussian assumption for the manifold holds. Finally, computing the gradient norms implies a non-trivial running time penalty -essentially doubling the running time. We also investigate the impact of a regularizer ubiquitous in supervised learning -the L 2 penalty on all the weights of the network.

Discriminator normalization. Normalizing the discriminator can be useful from both the optimization perspective (more efficient gradient flow, a more stable optimization), as well as from the representation perspective -the representation richness of the layers in a neural network depends on the spectral structure of the corresponding weight matrices (Miyato et al., 2018) .

From the optimization point of view, several techniques have found their way into the GAN literature, namely batch normalization (BN) (Ioffe and Szegedy, 2015) and layer normalization (LN) (Ba et al., 2016) . Batch normalization in the context of GANs was suggested by Denton et al. (2015) and further popularized by Radford et al. (2016) . It normalizes the pre-activations of nodes in a layer to mean β and standard deviation γ, where both β and γ are parameters learned for each node in the layer. The normalization is done on the batch level and for each node separately. In contrast, with Layer normalization, all the hidden units in a layer share the same normalization terms β and γ, but different samples are normalized differently (Ba et al., 2016) . Layer normalization was first applied in the context of GANs in Gulrajani et al. (2017) .

From the representation point of view, one has to consider the neural network as a composition of (possibly non-linear) mappings and analyze their spectral properties. In particular, for the discriminator to be a bounded linear operator it suffices to control the maximum singular value. This approach is followed in Miyato et al. (2018) where the authors suggest dividing each weight matrix, including the matrices representing convolutional kernels, by their spectral norm. Furthermore, the authors argue that a key advantage of spectral normalization over competing approaches is that it results in discriminators of higher rank.

2.3. GENERATOR AND DISCRIMINATOR ARCHITECTURE:

We explore two classes of architectures in this study: deep convolutional generative adversarial networks (DCGAN) (Radford et al., 2016) and residual networks (ResNet) (He et al., 2016) , both of which are ubiquitous in GAN research. Recently, Miyato et al. (2018) defined a variation of DCGAN, so called SNDCGAN. Apart from minor updates (cf. Section 4) the main difference to DCGAN is the use of an eight-layer discriminator network. The details of both networks are summarized in Table 3 . The other architecture, ResNet19, is an architecture with five ResNet blocks in the generator and six ResNet blocks in the discriminator, that can operate on 128 × 128 images. We follow the ResNet setup from Miyato et al. (2018) , with the small difference that we simplified the design of the discriminator. The detailed parameters of discriminator and generator are summarized in Table 4a and Table 4b . With this setup we were able to reproduce the current state of the art results. An ablation study on various ResNet modifications is available in the Appendix.

2.4. EVALUATION METRICS:

We focus on several recently proposed metrics well suited to the image domain. For an in-depth overview of quantitative metrics we refer the reader to (Borji, 2018) .

Inception Score (IS). Proposed by Salimans et al. (2016) , IS offers a way to quantitatively evaluate the quality of generated samples. Intuitively, the conditional label distribution of samples containing meaningful objects should have low entropy, and the variability of the samples should be high. which can be expressed as IS = exp(E x∼Q [d KL (p(y | x), p(y))]). The authors found that this score is well-correlated with scores from human annotators. Drawbacks include insensitivity to the prior distribution over labels and not being a proper distance.

As an alternative Heusel et al. (2017) proposed the Frechet Inception Distance (FID). Samples from P and Q are first embedded into a feature space (a specific layer of InceptionNet). Then, assuming that the embedded data follows a multivariate Gaussian distribution, the mean and covariance are estimated. Finally, the Fréchet distance between these two Gaussians is computed, i.e.

FID = ||µ x -µ y || 2 2 + Tr(Σ x + Σ y -2(Σ x Σ y ) 1 2 ),

where (µ x , Σ x ), and (µ y , Σ y ) are the mean and covariance of the embedded samples from P and Q, respectively. The authors argue that FID is consistent with human judgment and more robust to noise than IS. Furthermore, the score is sensitive to the visual quality of generated samplesintroducing noise or artifacts in the generated samples will reduce the FID. In contrast to IS, FID can detect intra-class mode dropping, i.e. a model that generates only one image per class can score a perfect IS, but will suffer from have a high FID (Lucic et al., 2018) . Bińkowski et al. (2018) argued that FID has no unbiased estimator and suggest Kernel Inception distance (KID) instead.

In Appendix B we empirically compare KID to FID and observe that both metrics are very strongly correlated (Spearman rank-order correlation coefficient of 0.994 for LSUN-BEDROOM and 0.995 for CELEBA-HQ-128 datasets). As a result we focus on FID as it is likely to result in the same ranking.

Multi-scale Structural Similarity for Image Quality (MS-SSIM) and Diversity. A critical issue in GANs are mode collapse and mode-dropping -failing to capture a mode, or low-diversity of generated samples from a given mode. The MS-SSIM score (Wang et al., 2003) is used for measuring the similarity of two images where higher MS-SSIM score indicates more similar images. Several recent works suggest using the average pairwise MS-SSIM score within a given class as a proxy for the diversity of generated samples (Odena et al., 2017; Fedus et al., 2018) . The drawback of this (β1, β2, n dis ) {(0.5, 0.900, 5), (0.5, 0.999, 1), (0.5, 0.999, 5), (0.9, 0.999, 5)}

(b) Gaussian Process regression ranges PARAMETER RANGE LOG Learning rate α [10 -5 , 10 -2 ] Yes λ for L2 [10 -4 , 10 1 ] Yes λ for non-L2 [10 -1 , 10 2 ] Yes β1 × β2 [0, 1] × [0, 1] No

approach is that we do not know the class corresponding to the generated sample, so it is usually applied on one-class data sets, such as CELEBA-HQ-128. In this work we use the same setup as in Fedus et al. (2018) . In particular, given a batch size b, we compute the average pairwise MS-SSIM score on 5 batches, of 5 × b × (b -1)/2 image pairs in total. We stress that the diversity should only be taken into account together with the FID and IS metrics.

2.5. DATA SETS:

We consider three data sets, namely CIFAR10, CELEBA-HQ-128, and LSUN-BEDROOM. The LSUN-BEDROOM data set (Yu et al., 2015) contains slightly more than 3 million images3 . We randomly partition the images into a train and test set whereby we use 30588 images as the test set. Secondly, we use the CELEBA-HQ data set of 30k images (Karras et al., 2018) . We use the 128 × 128 × 3 version obtained by running the code provided by the authors. 4 We use 3000 examples as the test set and the remaining examples as the training set. Finally, we also include the CIFAR10 data set which contains 70K images (32x32x3), partitioned into 60000 training instances and 10000 testing instances. The baseline FID scores are 12.6 for CELEBA-HQ-128, 3.8 for LSUN-BEDROOM, and 5.19 for CIFAR10. Details on FID computation are presented in Section 4.

2.6. EXPLORING THE GAN LANDSCAPE:

The search space for GANs is prohibitively expensive: exploring all combinations of all losses, normalization and regularization schemes, and architectures is outside of the practical realm. Instead, in this study we analyze several slices of this tensor for each data set. In particular, to ensure that we can reproduce existing results, we perform a study over the subset of this tensor on CIFAR10. We then proceed to analyze the performance of these models across CELEBA-HQ-128 and LSUN-BEDROOM.

In Section 3.1 we fix everything but the loss. In Section 3.2 we fix everything but the regularization and normalization scheme. Finally, in Section 3.3 we fix everything but the architecture. This allows us to decouple some of these design choices and provide some insight on what matters most.

As noted in Lucic et al. (2018) , one major issue preventing further progress is the hyperparameter tuning -currently, the community has converged to a small set of parameter values which work on some data sets, and may completely fail on others. In this study we combine the best hyperparameter settings found in the literature (Miyato et al., 2018) , and perform Gaussian Process regression in the bandit setting (Srinivas et al., 2010) to possibly uncover better hyperparameter settings. We then consider the top performing models and discuss the impact of the computational budget.

We summarize the fixed hyperparameter settings in Table 1a which contains the "good" parameters reported in recent publications (Fedus et al., 2018; Miyato et al., 2018; Gulrajani et al., 2017) . In particular, we consider the cross product of these parameters to obtain 24 hyperparameter settings to reduce the bias. Finally, to provide a fair comparison, we perform Gaussian Process optimization in the bandit setting (Srinivas et al., 2010) on the parameter ranges provided in Table 1b . We run 12 rounds (i.e. we communicate with the oracle 12 times) of the optimization, each with a batch of 10 hyperparameter sets selected based on the FID scores from the results of the previous iterations. loss is stable over both data sets. Gradient penalty and spectral normalization improve the sample quality. From the computational budget perspective (i.e. how many models one needs to train to reach a certain FID), both spectral normalization and gradient penalty perform better than the baseline, but the former is more efficient.

As we explore the number of discriminator updates per generator update (1 or 5), this leads to an additional 240 hyperparameter settings which in some cases outperform the previously known hyperparameter settings. Batch size is set to 64 for all the experiments. We use a fixed the number of discriminator update steps of 100K for LSUN-BEDROOM data set and CELEBA-HQ-128 data set, and 200K for CIFAR10 data set. We apply the Adam optimizer (Kingma and Ba, 2015) .

3. RESULTS AND DISCUSSION:

Given that there are 4 major components (loss, architecture, regularization, normalization) to analyze for each data set, it is infeasible to explore the whole landscape. Hence, we opt for a more pragmatic solution -we keep some dimensions fixed, and vary the others. For each experiment we highlight three aspects: (1) FID distribution of the top 5% of the trained models, (2) the corresponding sample diversity score, and (3) the tradeoff between the computational budget (i.e. number of models to train) and model quality in terms of FID. Each model was retrained 5 times with a different random seed and we report the median score. The variance for models obtained by Gaussian Process regression is handled implicitly so we train each model once.

3.1. IMPACT OF THE LOSS FUNCTION:

Here the loss is either the non-saturating loss (NS) (Goodfellow et al., 2014) , the least-squares loss (LS) (Mao et al., 2016) , or the Wasserstein loss (WGAN) (Arjovsky et al., 2017) . We use the ResNet19 with generator and discriminator architectures detailed in Table 4a . We consider the most prominent normalization and regularization approaches: gradient penalty (Gulrajani et al., 2017) , and spectral normalization (Miyato et al., 2018) . Both studies were performed on CELEBA-HQ-128 and LSUN-BEDROOM with hyperparameter settings shown in Table 1a .

The results are presented in Figure 1 . We observe that the non-saturating loss is stable over both data sets. Spectral normalization improves the quality of the model on both data sets. Similarly, the gradient penalty can help improve the quality of the model, but finding a good regularization tradeoff is non-trivial and requires a high computational budget. Models using the GP penalty benefit from 5:1 ratio of discriminator to generator updates as suggested by (Gulrajani et al., 2017) . We also performed a study on hinge loss (Miyato et al., 2018) and present it in the Appendix. 

3.2. IMPACT OF REGULARIZATION AND NORMALIZATION:

The goal of this study is to compare the relative performance of various regularization and normalization methods presented in the literature. To this end, and based on the loss study, we fix the loss to non-saturating loss (Goodfellow et al., 2014) . We use the ResNet19 with generator and discriminator architectures described in Table 4a . Finally, we consider batch normalization (BN) (Ioffe and Szegedy, 2015) , layer normalization (LN) (Ba et al., 2016) , spectral normalization (SN), gradient penalty (GP) (Gulrajani et al., 2017) , dragan penalty (DR) (Kodali et al., 2017) , or L 2 regularization. We consider both CELEBA-HQ-128 and LSUN-BEDROOM with the hyperparameter settings shown in Table 1a and Table 1b .

The results are presented in Figure 2 . We observe that adding batch norm to the discriminator hurts the performance. Secondly, gradient penalty can help, but it doesn't stabilize the training. In fact, it is non-trivial to strike a balance of the loss and regularization strength. Spectral normalization helps improve the model quality and is more computationally efficient than gradient penalty. This is consistent with recent results in Zhang et al. (2018) . Similarly to the loss study, models using GP penalty benefit from 5:1 ratio of discriminator to generator updates. Furthermore, in a separate ablation study we observed that running the optimization procedure for an additional 100K steps is likely to increase the performance of the models with GP penalty.

Impact of Simultaneous Regularization and Normalization. Given the folklore that the Lipschitz constant of the discriminator is critical for the performance, one may expect simultaneous regularization and normalization could improve model quality. To quantify this effect, we fix the loss to non-saturating loss (Goodfellow et al., 2014) , use the Resnet19 architecture (as above), and combine several normalization and regularization schemes, with hyperparameter settings shown in Table 1a coupled with 24 randomly selected parameters. The results are presented in Figure 3 . We observe that one may benefit from additional regularization and normalization. However, a lot of computational effort has to be invested for somewhat marginal gains in FID. Nevertheless, given enough computational budget we advocate simultaneous regularization and normalization -spectral normalization and layer normalization seem to perform well in practice.

3.3. IMPACT OF GENERATOR AND DISCRIMINATOR ARCHITECTURES:

An interesting practical question is whether our findings also hold for a different model capacity.

To this end, we also perform a study on SNDCGAN from Miyato et al. (2018) . We consider the non-saturating GAN loss, gradient penalty and spectral normalization. While for smaller architectures regularization is not essential (Lucic et al., 2018) , the regularization and normalization effects might become more relevant due to deeper architectures and optimization considerations. The results are presented in Figure 4 . We observe that both architectures achieve comparable results and benefit from regularization and normalization. Spectral normalization strongly outperforms the baseline for both architectures.

4. COMMON PITFALLS:

In this section we focus on several pitfalls we encountered while trying to reproduce existing results and provide a fairly and accurate comparison.

Metrics. There already seems to be a divergence in how the FID score is computed: (1) Some authors report the score on training data, yielding a FID between 50k training and 50k generated samples (Unterthiner et al., 2018) . Some opt to report the FID based on 10k test samples and 5k generated samples and use a custom implementation (Miyato et al., 2018) . Finally, Lucic et al. (2018) report the score with respect to the test data, in particular FID between 10k test samples, and 10k generated samples. The subtle differences will result in a mismatch between the reported FIDs, in some cases of more than 10%. We argue that FID should be computed with respect to the test data set as and use 10k test samples and 10k generated samples on CIFAR10 and LSUN-BEDROOM, and 3k vs 3k on CELEBA-HQ-128 as in in Lucic et al. (2018) . Similarly, there are several ways to compute a diversity score using MS-SSIM and we follow the approach from Fedus et al. (2018) . We provide the implementation details in Section G of the Appendix.

Details of neural architectures. Even in popular architectures, like ResNet, there is still a number of design decision one needs to make, that are often omitted from the reported results. Those include the exact design of the ResNet cell (order of layers, when is ReLu applied, when to upsample and downsample, how many filters to use). Some of these differences might lead to potentially unfair comparison. As a result, we suggest to use the architectures presented within this work as a solid baseline. An ablation study on various ResNet modifications is available in the Appendix.

Data sets. A common issue is related to data set processing -does LSUN-BEDROOM always correspond to the same data set? In most cases the precise algorithm for upscaling or cropping is not clear which introduces inconsistencies between results on the "same" data set.

Implementation details and non-determinism. One major issue is the mismatch between the algorithm presented in a paper and the code provided online. We are aware that there is an embarrassingly large gap between a good implementation and a bad implementation of a given model. Hence, when no code is available, one is forced to guess which modifications were done. Another particularly tricky issue is removing randomness from the training process. After one fixes the data ordering and the initial weights, obtaining the same score by training the same model twice is non-trivial due to randomness present in certain GPU operations (Chetlur et al., 2014) . Disabling the optimizations causing the non-determinism often results in an order of magnitude running time penalty.

While each of these issues taken in isolation seems minor, they compound to create a mist which introduces friction in practical applications and the research process (Sculley et al., 2018) .

5. RELATED WORK:

A recent large-scale study on GANs and Variational Autoencoders was presented in Lucic et al. (2018) . The authors consider several loss functions and regularizers, and study the effect of the loss function on the FID score, with low-to-medium complexity data sets (MNIST, CIFAR10, CELEBA), and a single (InfoGAN style) architecture. In this limited setting, the authors found that there is no statistically significant difference between recently introduced models and the original non-saturating GAN. A study of the effects of gradient-norm regularization in GANs was recently presented in Fedus et al. (2018) . The authors posit that the gradient penalty can also be applied to the non-saturating GAN, and that, to a limited extent, it reduces the sensitivity to hyperparameter selection. In a recent work on spectral normalization, the authors perform a small study of the competing regularization and normalization approaches (Miyato et al., 2018) . We are happy to report that we could reproduce these results and we present them in the Appendix.

Inspired by these works and building on the available open-source code from Lucic et al. (2018) , we take one additional step in all dimensions considered therein: more complex neural architectures, more complex data sets, and more involved regularization and normalization schemes.

6. CONCLUSION:

In this work we study the GAN landscape: losses, regularization and normalization schemes, and neural architectures, and their impact on the on the quality of generated samples which we assess by recently introduced quantitative metrics. Our fair and thorough empirical evaluation suggests that one should consider non-saturating GAN loss and spectral normalization as default choices when applying GANs to a new data set. Given additional computational budget, we suggest adding the gradient penalty from Gulrajani et al. (2017) and train the model until convergence. Furthermore, additional marginal gains can be obtained by combining normalization and regularization empirically confirming the importance of the Lipschitz constant of the discriminator. Furthermore, both types of architectures proposed up-to this point perform reasonably well. A separate ablation study uncovered that most of the tricks applied in the ResNet style architectures lead to marginal changes in the quality and should be avoided due to the high computational cost. As a result of this large-scale study we identify the common pitfalls standing in the way of accurate and fair comparison and propose concrete actions to demystify the future results -issues with metrics, data set preprocessing, non-determinism, and missing implementation details are particularly striking. We hope that this work, together with the open-sourced reference implementations and trained models, will serve as a solid baseline for future GAN research.

A FID AND INCEPTION SCORES ON CIFAR10:

We present an empirical study with SNDCGAN and ResNet CIFAR architectures on CIFAR10 in figure 5 and figure 6 . In addition to Section 3.1, we evaluate one more kind of loss on CIFAR10. Here HG, NS and WGAN stand for hinge loss, non saturating loss and Wasserstein loss respectively. We observe that hinge loss performs very similar to non-saturating loss. (Miyato et al., 2018) .

B COMPARISON OF FID AND KID METRICS:

The KID metric introduced by Bińkowski et al. ( 2018) is an alternative to FID. We use models from our Regularization and Normalization study (see Section 3.2) to compare both metrics. Here, by model we denote everything that needs to be specified for the training -including all hyper-parameters, like learning rate, λ, Adam's β, etc. The Spearman rank-order correlation coefficient between KID and FID scores is approximately 0.994 for LSUN-BEDROOM and 0.995 for CELEBA-HQ-128 datasets.

To evaluate a practical setting of selecting several best models, we compare the intersection between the set of "best K models by FID" and the set of "best K models by KID" for K ∈ 5, 10, 20, 50, 100. The results are summarized in Table 2 .

This experiment suggests that FID and KID metrics are very strongly correlated, and for the practical applications one can choose either of them. Also, the conclusions from our studies based on FID should transfer to studies based on KID. We used the same architecture as Miyato et al. (2018) , with the parameters copied from the GitHub page5 . In Table 3a and Table 3b , we describe the operations in layer column with order. Kernel size is described in format [f ilter h, f ilter w, stride], input shape is h × w and output shape is h × w × channels. The slopes of all lReLU functions are set to 0.1. The input shape h × w is 128 × 128 for CELEBA-HQ-128 and LSUN-BEDROOM, 32 × 32 for CIFAR10. ResBlock is defined in Table 5 . The addition layer merges two paths by adding them. The first path is a shortcut layer with exactly one convolution operation, while the second path consists of two convolution operations.

The downscale layer and upscale layer are marked in Table 5 . We used average pool with kernel [2, 2, 2] for downscale, after the convolution operation. We used unpool from https://github.com/tensorflow/ tensorflow/issues/2169 for upscale, before convolution operation. h and w are the input shape to the ResNet block, output shape depends on the RS parameter. ci and co are the input channels and output channels for a ResNet block. 

EQUATION

Linear - - 4 × 4 × 512 ResBlock [3, 3, 1] U 8 × 8 × 512 ResBlock [3, 3, 1] U 16 × 16 × 256 ResBlock [3, 3, 1] U 32 × 32 × 256 ResBlock [3, 3, 1] U 64 × 64 × 128 ResBlock [3, 3, 1] U 128 × 128 × 64 BN, ReLU - - 128 × 128 × 64 Conv [3, 3, 1] - 128 × 128 × 3 Sigmoid - - 128 × 128 × 3 Table 5: ResNet block definition. (a) ResBlock discriminator LAYER KERNEL RS OUTPUT Shortcut [3, 3, 1] D h/2 × w/2 × co BN, ReLU - - h × w × ci Conv [3, 3, 1] - h × w × co BN, ReLU - - h × w × co Conv [3, 3, 1] D h/2 × w/2 × co Addition - - h/2 × w/2 × co (b) ResBlock generator LAYER KERNEL RS OUTPUT Shortcut [3, 3, 1] U 2h × 2w × co BN, ReLU - - h × w × ci Conv [3, 3, 1] U 2h × 2w × co BN, ReLU - - 2h × 2w × co Conv [3, 3, 1] - 2h × 2w × co Addition - - 2h × 2w × co

EQUATION

EQUATION

Linear - - 4 × 4 × 256 ResBlock [3, 3, 1] U 8 × 8 × 256 ResBlock [3, 3, 1] U 16 × 16 × 256 ResBlock [3, 3, 1] U 32 × 32 × 256 BN, ReLU - - 32 × 32 × 256 Conv [3, 3, 1] - 32 × 32 × 3 Sigmoid - - 32 × 32 × 3 D RESNET ARCHITECTURE ABLATION STUDY

We have noticed six minor differences on Resnet architecture comparing to implementation from https: //github.com/pfnet-research/chainer-gan-lib/blob/master/common/net.py (Miyato et al., 2018) . We did ablation study to verify the impact of these differences. Figure 7 shows the impact of the ablation study, with details described as following.

• DEFAULT: ResNet CIFAR architecture with spectral normalization and non-saturating GAN loss.

• SKIP: Use input as output for the shortcut connection in the discriminator ResBlock. By default it was a conv layer with 3x3 kernel.

• CIN: Use ci for the discriminator ResBlock hidden layer output channels. By default it was co in our setup, while Miyato et al. (2018) used co for first ResBlock and ci for the rest.

• OPT: Use an optimized setup for the first discriminator ResBlock, which includes: (1) no ReLU, (2) a conv layer for the shortcut connections, (3) use co instead of ci in ResBlock.

• CIN OPT: Use CIN and OPT together. It means the first ResBlock is optimized while the remaining ResBlocks use ci for the hidden output channels.

• SUM: Use reduce sum for the discriminator output. By default it was reduce mean.

• TAN: Use tanh for the generator output, as well as range [-1, 1] for discriminator input. By default it was sigmoid and discriminator input range [0, 1].

• EPS: Use a bigger epsilon 2e -5 for generator batch normalization. By default it was 1e -5 in TensorFlow.

• ALL: Apply all the above differences together.

In the ablation study, the CIN experiment obtained the worst FID score. Combining with OPT, the CIN results were improved to the same level as the others which is reasonable because the first block has three input channels, which becomes a bottleneck for the optimization. Hence, using OPT and CIN together performs as well as the others. Overall, the impact of these differences are minor according to the study on CIFAR10. To visualize the FID score on each data set, Figure 8 , Figure 9 and Figure 10 show the generated examples by GANs. We select the examples from the best FID run, and then increase the FID score for two more plots. 

:

Link removed to preserve anonymity.

Link removed to preserve anonymity.

The images are preprocessed to 128 × 128 × 3 using TensorFlow resize image with crop or pad.

Available online at https://github.com/tkarras/progressive_growing_of_gans.

https://github.com/pfnet-research/chainer-gan-lib

Figures and Tables:

Figure1: Impact of the loss function: FID distribution for top 5% models. The non-saturating (NS) loss is stable over both data sets. Gradient penalty and spectral normalization improve the sample quality. From the computational budget perspective (i.e. how many models one needs to train to reach a certain FID), both spectral normalization and gradient penalty perform better than the baseline, but the former is more efficient.

Figure2: Impact of regularization and normalization: FID distribution for top 5% models. Both gradient penalty (GP) and spectral normalization (SN) outperform the baseline and should be considered, while former being more computationally expensive. Unfortunately none fully address the stability issues.

Figure 3: Impact of simultaneous normalization and regularization: FID distribution for top 5% models. Gradient penalty coupled with spectral normalization (SN) or layer normalization (LN) strongly improves the performance over the baseline.

Figure 4: Impact of the neural architectures: FID distribution for top 5% models. Both spectral normalization and gradient penalty can help improve upon the non-regularized baseline.

Figure5: An empirical study with SNDCGAN and ResNet cifar architectures on CIFAR10. We recover the state of the art results recently reported inMiyato et al. (2018).

Figure6: We show the Inception Score for each model within our study which corresponds to recently reported results(Miyato et al., 2018).

Figure 7: Ablation study of ResNet architecture differences. The experiment codes are described in Section D.

Figure 8: Examples generated by GANs on CELEBA-HQ-128 data set.

Figure 9: Examples generated by GANs on LSUN-BEDROOM data set.

Hyperparameter ranges used in this study. The Cartesian product of the fixed values suffices to uncover the existing results. Gaussian Process optimization in the bandit setting(Srinivas et al., 2010) is used to select good hyperparameter settings from the specified ranges.

Table content: <table><tr><td/><td>(a) Fixed values</td></tr><tr><td>PARAMETER</td><td>DISCRETE VALUE</td></tr><tr><td colspan="2">Learning rate α {0.0002, 0.0001, 0.001}</td></tr><tr><td colspan="2">Reg. strength λ {1, 10}</td></tr></table>

Intersection between set of top K experiments selected by FID and KID metrics.

Table content: <table><tr><td/><td>LSUN-BEDROOM</td><td>CELEBA-HQ-128</td></tr><tr><td>K = 5</td><td>4/5</td><td>2/5</td></tr><tr><td>K = 10</td><td>9/10</td><td>8/10</td></tr><tr><td>K = 20</td><td>18/20</td><td>15/20</td></tr><tr><td>K = 50</td><td>49/50</td><td>46/50</td></tr><tr><td>K = 100</td><td>95/100</td><td>98/100</td></tr></table>

SNDCGAN architecture.

Table content: <table><tr><td/><td colspan="2">(a) SNDCGAN discriminator</td><td colspan="3">(b) SNDCGAN generator</td></tr><tr><td>LAYER</td><td colspan="2">KERNEL OUTPUT</td><td>LAYER</td><td colspan="2">KERNEL OUTPUT</td></tr><tr><td colspan="2">Conv, lReLU [3, 3, 1]</td><td>h × w × 64</td><td>z</td><td>-</td><td>128</td></tr><tr><td colspan="2">Conv, lReLU [4, 4, 2]</td><td>h/2 × w/2 × 128</td><td>Linear, BN, ReLU</td><td>-</td><td>h/8 × w/8 × 512</td></tr><tr><td colspan="2">Conv, lReLU [3, 3, 1]</td><td>h/2 × w/2 × 128</td><td colspan="2">Deconv, BN, ReLU [4, 4, 2]</td><td>h/4 × w/4 × 256</td></tr><tr><td colspan="2">Conv, lReLU [4, 4, 2]</td><td>h/4 × w/4 × 256</td><td colspan="2">Deconv, BN, ReLU [4, 4, 2]</td><td>h/2 × w/2 × 128</td></tr><tr><td colspan="2">Conv, lReLU [3, 3, 1]</td><td>h/4 × w/4 × 256</td><td colspan="2">Deconv, BN, ReLU [4, 4, 2]</td><td>h × w × 64</td></tr><tr><td colspan="2">Conv, lReLU [4, 4, 2]</td><td>h/8 × w/8 × 512</td><td>Deconv, Tanh</td><td>[3, 3, 1]</td><td>h × w × 3</td></tr><tr><td colspan="2">Conv, lReLU [3, 3, 1]</td><td>h/8 × w/8 × 512</td><td/><td/></tr><tr><td>Linear</td><td>-</td><td>1</td><td/><td/></tr><tr><td colspan="3">C.2 RESNET ARCHITECTURE</td><td/><td/></tr><tr><td colspan="6">The ResNet19 architecture is described in Table 4. RS column stands for the resample of the residual block,</td></tr><tr><td colspan="6">with downscale(D)/upscale(U)/none(-) setting. MP stands for mean pooling and BN for batch normalization.</td></tr></table>

Table 6 described the ResNet CIFAR architecture we used in Figure5for reproducing the existing results. Note that RS is set to none for third ResBlock and fourth ResBlock in discriminator. In this case, we used the same ResNet block defined in Table5without resampling.

Table content: <table/>

ResNet 19 architecture corresponding to "resnet small" in https://github.com/ pfnet-research/sngan_projection.

Table content: <table><tr><td/><td colspan="3">(a) ResNet19 discriminator</td></tr><tr><td>LAYER</td><td colspan="3">KERNEL RS OUTPUT</td></tr><tr><td>ResBlock</td><td>[3, 3, 1]</td><td>D</td><td>64 × 64 × 64</td></tr><tr><td>ResBlock</td><td>[3, 3, 1]</td><td>D</td><td>32 × 32 × 128</td></tr><tr><td>ResBlock</td><td>[3, 3, 1]</td><td>D</td><td>16 × 16 × 256</td></tr><tr><td>ResBlock</td><td/><td/><td/></tr></table>

ResNet CIFAR architecture.

Table content: <table><tr><td colspan="4">(a) ResNet CIFAR discriminator</td></tr><tr><td>LAYER</td><td colspan="3">KERNEL RS OUTPUT</td></tr><tr><td>ResBlock</td><td>[3, 3, 1]</td><td>D</td><td>16 × 16 × 128</td></tr><tr><td>ResBlock</td><td/><td/><td/></tr></table>

Table 8 and Table 9 summarize the top 2 parameters for SNDCGAN architecture, ResNet19 architecture and ResNet CIFAR architecture, respectively. Models are ranked according to the median FID score of five different random seeds with fixed hyper-parameters in Table1a. Note that ranking models according to the best FID score of different seeds will achieve better but unstable result. Gaussian Process optimization hyper-parameters are not included in this table. For ResNet19 architecture with at most two regularizers, we have run it only once due to computational overhead. To show the model stability, we listed the best FID score out of five seeds from the same parameters in column best. Spectral normalization is clearly outperforms the other normalizers on SNDCGAN and ResNet CIFAR architectures, while on ResNet19 both layer normalization and spectral normalization work well.

Table content: <table/>

ResNet CIFAR parameters DATA SET MEDIAN BEST LR(×10 -3 ) β 1 β 2 n disc λ NORM

Table content: <table><tr><td>CIFAR10</td><td>31.40</td><td>28.12 0.200</td><td>0.500 0.999 5</td><td>--</td></tr><tr><td>CIFAR10</td><td>33.79</td><td>30.08 0.100</td><td>0.500 0.999 5</td><td>--</td></tr><tr><td>CIFAR10</td><td>23.57</td><td>22.91 0.200</td><td>0.500 0.999 5</td><td>-SN</td></tr><tr><td>CIFAR10</td><td>25.50</td><td>24.21 0.100</td><td>0.500 0.999 5</td><td>-SN</td></tr><tr><td>CIFAR10</td><td>22.98</td><td>22.73 0.200</td><td>0.500 0.999 1</td><td>1 SN+GP</td></tr><tr><td>CIFAR10</td><td>23.57</td><td>22.91 0.200</td><td>0.500 0.999 5</td><td>-SN</td></tr></table>

Peer review (iclr2019):
 This paper seems to be an exposition on the primary performance affecting aspects of generative adversarial networks (GANs). This can possibly affect our understanding of GANs, helping practitioners get the most in their applications, and perhaps leading to innovations that positively affect GAN performance. Normally, expositions such as this I find difficult to recommend for publication. In these times, one can find "best practices" with a reasonable amount of rigor on data science blogs and such. An exposition that I would recommend for publication, would need to exhibit a high sense of depth and rigor for me to deem it publication worthy. This paper, for me, achieves this level of quality. The authors start off by giving a precise, constrained list of hyperparameters and architectural components that they would explore. This is listed in the title and explained in detail in the beginning of the paper. The authors are right in explaining that they could not cover all hyperparameters and chose what I feel are quite salient ones. My one ask would have been a survey of how activations might affect performance. I sense that everyone has settled upon LeakyReLUs for internal layers, but a survey of that work and experimentation within the authors* framework would have been nice. The authors then explain the metrics for evaluation and datasets. The datasets offered a healthy variety for typical image recognition tasks. It would be interesting to see what these metrics would reveal when applied to other types of data (e.g. scientific images). The authors explain, with graphs, the results of the loss, normalization, and architectures. I feel the discussion on loss was rushed, and I gained no insight on what the authors thought was a prominent difference between the three losses studied. Perhaps the authors had no salient observations for loss, but explicitly stating such would be useful to the reader. The only observation I gained as far as this is that non-saturating loss would possibly be stable across various datasets. Regularization and normalization are discussed in much more detail, and I think the authors made helpful and interesting observations, such as the benefits of spectral normalization and the fact that batch normalization in the discriminator might be a harmful thing. These are good takeaways that could be useful to a vast number of GANs researchers. For architectures to be a main pillar of the paper, I feel that this area could have been explored in greater detail. I feel that this discussion devolved into a discussion, again, about normalization rather than the architectural differences in performance. Unless I am misunderstanding something, it seems that the authors simply tested one more architecture, for the express purpose of testing whether their observations about normalization would hold. As a bonus, the authors bring up some problems they had in making comparisons and reproducing results. I think this is an extremely important discussion to have, and I am glad that the authors detailed the obstacles in their journey. Hopefully this will inspire other researchers to avoid adding to the complications in this field. The graphs were difficult to parse. I was able to make them out, but perhaps separating the top row (FID and diversity graphs) into separate figures, separate lines, or something would have reduced some confusion. In addition, different charts presenting only one loss function, with their spectral normalization and gradient penalty variants, would have made the effects of the normalization more obvious on the FID distribution graphs. If this can be changed before publication, I would strongly suggest it. I appreciate that the authors provided source code via GitHub. However, in the future, the authors should be careful to provide an anonymous repository for review purposes. I had to be careful not to allow myself to focus on the author names which are prominent in the repository readme, and one of whom has his/her name in the GitHub URL itself. I didn*t immediately recognize the names and thus it was easy for me not to retain them or focus on them. However, if it had been otherwise, it might have risked biasing the review. In all, I think this is a good and useful paper from which I have learned and to which I will refer in the future as I continue my research into GANs and VAEs. I would suggest changing the title to be more appropriate and accurate (the researchers are primarily focused on showing the positive and negative effects of normalization across various loss functions and architectures). But altogether, I believe this is a paper worth publishing at ICLR.