user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Progressive Growing of GANs for Improved Quality, Stability, and Variation

Paper content:
We describe a new training methodology for generative adversarial networks.
The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses.
This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2.
We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10.
Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator.
Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation.
As an additional contribution, we construct a higher-quality version of the CelebA dataset.
Generative methods that produce novel samples from high-dimensional data distributions, such as images, are finding widespread use, for example in speech synthesis (van den Oord et al., 2016a) , image-to-image translation (Zhu et al., 2017; Liu et al., 2017; Wang et al., 2017) , and image inpainting (Iizuka et al., 2017) .
Currently the most prominent approaches are autoregressive models (van den Oord et al., 2016b; , variational autoencoders (VAE) (Kingma & Welling, 2014) , and generative adversarial networks (GAN) BID14 .
Currently they all have significant strengths and weaknesses.
Autoregressive models -such as PixelCNN -produce sharp images but are slow to evaluate and do not have a latent representation as they directly model the conditional distribution over pixels, potentially limiting their applicability.
VAEs are easy to train but tend to produce blurry results due to restrictions in the model, although recent work is improving this (Kingma et al., 2016) .
GANs produce sharp images, albeit only in fairly small resolutions and with somewhat limited variation, and the training continues to be unstable despite recent progress (Salimans et al., 2016; BID16 BID5 Kodali et al., 2017) .
Hybrid methods combine various strengths of the three, but so far lag behind GANs in image quality (Makhzani & Frey, 2017; Ulyanov et al., 2017; BID10 .Typically
, a GAN consists of two networks: generator and discriminator (aka critic). The generator
produces a sample, e.g., an image, from a latent code, and the distribution of these images should ideally be indistinguishable from the training distribution. Since it is generally
infeasible to engineer a function that tells whether that is the case, a discriminator network is trained to do the assessment, and since networks are differentiable, we also get a gradient we can use to steer both networks to the right direction. Typically, the generator
is of main interest -the discriminator is an adaptive loss function that gets discarded once the generator has been trained.There are multiple potential problems with this formulation. When we measure the distance
between the training distribution and the generated distribution, the gradients can point to more or less random directions if the distributions do not have substantial overlap, i.e., are too easy to tell apart . Originally, Jensen-Shannon divergence
was used as a distance metric BID14 , and recently that formulation has been improved (Hjelm et al., 2017 ) and a number of more stable alternatives have been proposed, including least squares (Mao et al., 2016b) , absolute deviation with margin (Zhao et al., 2017) , and Wasserstein distance BID16 . Our contributions are largely orthogonal
to this ongoing discussion, and we primarily use the improved Wasserstein loss, but also experiment with least-squares loss.The generation of high-resolution images is difficult because higher resolution makes it easier to tell the generated images apart from training images (Odena et al., 2017) , thus drastically amplifying the gradient problem. Large resolutions also necessitate using
smaller minibatches due to memory constraints, further compromising training stability. Our key insight is that we can grow both
the generator and discriminator progressively, starting from easier low-resolution images, and add new layers that introduce higher-resolution details as the training progresses. This greatly speeds up training and improves
stability in high resolutions, as we will discuss in Section 2.The GAN formulation does not explicitly require the entire training data distribution to be represented by the resulting generative model. The conventional wisdom has been that there
is a tradeoff between image quality and variation, but that view has been recently challenged (Odena et al., 2017) . The degree of preserved variation is currently
receiving attention and various methods have been suggested for measuring it, including inception score (Salimans et al., 2016) , multi-scale structural similarity (MS-SSIM) (Odena et al., 2017; Wang et al., 2003) , birthday paradox BID2 , and explicit tests for the number of discrete modes discovered (Metz et al., 2016) . We will describe our method for encouraging variation
in Section 3, and propose a new metric for evaluating the quality and variation in Section 5. Section 4.1 discusses a subtle modification to the initialization
of networks, leading to a more balanced learning speed for different layers. Furthermore, we observe that mode collapses traditionally plaguing
GANs tend to happen very quickly, over the course of a dozen minibatches. Commonly they start when the discriminator overshoots, leading to
exaggerated gradients, and an unhealthy competition follows where the signal magnitudes escalate in both networks. We propose a mechanism to stop the generator from participating in
such escalation, overcoming the issue (Section 4.2).We evaluate our contributions using the CELEBA, LSUN, CIFAR10 datasets
. We improve the best published inception score for CIFAR10. Since the
datasets commonly used in benchmarking generative methods
are limited to a fairly low resolution, we have also created a higher quality version of the CELEBA dataset that allows experimentation with output resolutions up to 1024 × 1024 pixels. This dataset and our full implementation are available at https://github.com/tkarras/progressive_growing_of_gans
, trained networks can be found at https://drive.google.com/open?id=0B4qLcYyJmiz0NHFULTdYc05lX0U along with result images, and a supplementary video illustrating the datasets, additional
results, and latent space interpolations is at https://youtu.be/G06dEcZ-QTg.
While the quality of our results is generally high compared to earlier work on GANs, and the training is stable in large resolutions, there is a long way to true photorealism.
Semantic sensibility and understanding dataset-dependent constraints, such as certain objects being straight rather than curved, leaves a lot to be desired.
There is also room for improvement in the micro-structure of the images.
That said, we feel that convincing realism may now be within reach, especially in CELEBA-HQ.
TAB4 shows network architectures of the full-resolution generator and discriminator that we use with the CELEBA-HQ dataset.
Both networks consist mainly of replicated 3-layer blocks that we introduce one by one during the course of the training.
The last Conv 1 × 1 layer of the generator corresponds to the toRGB block in Figure 2 , and the first Conv 1 × 1 layer of the discriminator similarly corresponds to fromRGB.
We start with 4 × 4 resolution and train the networks until we have shown the discriminator 800k real images in total.
We then alternate between two phases: fade in the first 3-layer block during the next 800k images, stabilize the networks for 800k images, fade in the next 3-layer block during 800k images, etc.
DISPLAYFORM0 256 × 64 × 64 1.2M Conv 3 × 3 LReLU 256 × 64 × 64 590k Upsample -256 × 128 × 128 -Conv 3 × 3 LReLU 128 × 128 × 128 295k Conv 3 × 3 LReLU 128 × 128 × 128 148k Upsample -128 × 256 × 256 -Conv 3 × 3 LReLU 64 × 256 × 256 74k Conv 3 × 3 LReLU 64 × 256 × 256 37k Upsample -64 × 512 × 512 -Conv 3 × 3 LReLU 32 × 512 × 512 18k Conv 3 × 3 LReLU 32 × 512 × 512 9.2k Upsample -32 × 1024 × 1024 -Conv 3 × 3 LReLU 16 × 1024 × 1024 4.6k Conv 3 × 3 LReLU 16 × 1024 × 1024 2.3k Conv 1 × 1 linear 3 × 1024 ×64 × 512 × 512 18k Downsample -64 × 256 × 256 -Conv 3 × 3 LReLU 64 × 256 × 256 37k Conv 3 × 3 LReLU 128 × 256 × 256 74k Downsample -128 × 128 × 128 -Conv 3 × 3 LReLU 128 × 128 × 128 148k Conv 3 × 3 LReLU 256 × 128 × 128 295k Downsample -256 × 64 × 64 -Conv 3 × 3 LReLU 256 × 64 × 64 590k Conv 3 × 3 LReLU 512 × 64 × 64 1.2M Downsample -512 × 32 × 32 -Conv 3 × 3 LReLU 512 × 32 × 32 2.4M Conv 3 × 3 LReLU 512 × 32 × 32 2.4M Downsample -512 × 16 × 16 -Conv 3 × 3 LReLU 512 × 16 × 16 2.4M Conv 3 × 3 LReLU 512 × 16 × 16 2Our latent vectors correspond to random points on a 512-dimensional hypersphere, and we represent training and generated images in [-1,1] .
We use leaky ReLU with leakiness 0.2 in all layers of both networks, except for the last layer that uses linear activation.
We do not employ batch normalization, layer normalization, or weight normalization in either network, but we perform pixelwise normalization of the feature vectors after each Conv 3 × 3 layer in the generator as described in Section 4.2.
We initialize all bias parameters to zero and all weights according to the normal distribution with unit variance.
However, we scale the weights with a layer-specific constant at runtime as described in Section 4.1.
We inject the across-minibatch standard deviation as an additional feature map at 4 × 4 resolution toward the end of the discriminator as described in Section 3.
The upsampling and downsampling operations in TAB4 correspond to 2 × 2 element replication and average pooling, respectively.We train the networks using Adam (Kingma & Ba, 2015) with α = 0.001, β 1 = 0, β 2 = 0.99, and = 10 −8 .
We do not use any learning rate decay or rampdown, but for visualizing generator output at any given point during the training, we use an exponential running average for the weights of the generator with decay 0.999.
We use a minibatch size 16 for resolutions 4 2 -128 2 and then gradually decrease the size according to 256 2 → 14, 512 2 → 6, and 1024 2 → 3 to avoid exceeding the available memory budget.
We use the WGAN-GP loss, but unlike BID16 , we alternate between optimizing the generator and discriminator on a per-minibatch basis, i.e., we set n critic = 1.
Additionally, we introduce a fourth term into the discriminator loss with an extremely small weight to keep the discriminator output from drifting too far away from zero.
To be precise, DISPLAYFORM1 2 ], where drift = 0.001.

assistant: text: We train generative adversarial networks in a progressive fashion, enabling us to generate high-resolution images with high quality.