Title: LARGE SCALE GAN TRAINING FOR HIGH FIDELITY NATURAL IMAGE SYNTHESIS

Abstract: Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128×128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.5 and Fréchet Inception Distance (FID) of 7.4, improving over the previous best IS of 52.52 and FID of 18.65. 

1. INTRODUCTION:

Figure 1 : Class-conditional samples generated by our model.

The state of generative image modeling has advanced dramatically in recent years, with Generative Adversarial Networks (GANs, Goodfellow et al. (2014) ) at the forefront of efforts to generate highfidelity, diverse images with models learned directly from data. GAN training is dynamic, and sensitive to nearly every aspect of its setup (from optimization parameters to model architecture), but a torrent of research has yielded empirical and theoretical insights enabling stable training in a variety of settings. Despite this progress, the current state of the art in conditional ImageNet modeling (Zhang et al., 2018) achieves an Inception Score (Salimans et al., 2016) of 52.5, compared to 233 for real data.

In this work, we set out to close the gap in fidelity and variety between images generated by GANs and real-world images from the ImageNet dataset. We make the following three contributions towards this goal:

• We demonstrate that GANs benefit dramatically from scaling, and train models with two to four times as many parameters and eight times the batch size compared to prior art. We introduce two simple, general architectural changes that improve scalability, and modify a regularization scheme to improve conditioning, demonstrably boosting performance.

• As a side effect of our modifications, our models become amenable to the "truncation trick," a simple sampling technique that allows explicit, fine-grained control of the tradeoff between sample variety and fidelity.

• We discover instabilities specific to large scale GANs, and characterize them empirically.

Leveraging insights from this analysis, we demonstrate that a combination of novel and existing techniques can reduce these instabilities, but complete training stability can only be achieved at a dramatic cost to performance.

Our modifications substantially improve class-conditional GANs. When trained on ImageNet at 128×128 resolution, our models (BigGANs) improve the state-of-the-art Inception Score (IS) and Fréchet Inception Distance (FID) from 52.52 and 18.65 to 166.5 and 7.4 respectively. We also successfully train BigGANs on ImageNet at 256×256 and 512×512 resolution, and achieve IS and FID of 232.5 and 8.1 at 256×256 and IS and FID of 241.5 and 11.5 at 512×512. Finally, we train our models on an even larger dataset -JFT-300M -and demonstrate that our design choices transfer well from ImageNet. Code and weights for our pretrained generators are publicly available1 .

2. BACKGROUND:

A Generative Adversarial Network (GAN) involves Generator (G) and Discriminator (D) networks whose purpose, respectively, is to map random noise to samples and discriminate real and generated samples. Formally, the GAN objective, in its original form (Goodfellow et al., 2014) involves finding a Nash equilibrium to the following two player min-max problem:

EQUATION

where z ∈ R dz is a latent variable drawn from distribution p(z) such as N (0, I) or U [-1, 1] .

When applied to images, G and D are usually convolutional neural networks (Radford et al., 2016) . Without auxiliary stabilization techniques, this training procedure is notoriously brittle, requiring finely-tuned hyperparameters and architectural choices to work at all.

Much recent research has accordingly focused on modifications to the vanilla GAN procedure to impart stability, drawing on a growing body of empirical and theoretical insights (Nowozin et al., 2016; Sønderby et al., 2017; Fedus et al., 2018) . One line of work is focused on changing the objective function (Arjovsky et al., 2017; Mao et al., 2016; Lim & Ye, 2017; Bellemare et al., 2017; Salimans et al., 2018) to encourage convergence. Another line is focused on constraining D through gradient penalties (Gulrajani et al., 2017; Kodali et al., 2017; Mescheder et al., 2018) or normalization (Miyato et al., 2018) , both to counteract the use of unbounded loss functions and ensure D provides gradients everywhere to G.

Of particular relevance to our work is Spectral Normalization (Miyato et al., 2018) , which enforces Lipschitz continuity on D by normalizing its parameters with running estimates of their first singular values, inducing backwards dynamics that adaptively regularize the top singular direction. Relatedly Odena et al. (2018) analyze the condition number of the Jacobian of G and find that performance is dependent on G's conditioning. Zhang et al. (2018) find that employing Spectral Normalization in G improves stability, allowing for fewer D steps per iteration. We extend on these analyses to gain further insight into the pathology of GAN training.

Other works focus on the choice of architecture, such as SA-GAN (Zhang et al., 2018) which adds the self-attention block from (Wang et al., 2018) to improve the ability of both G and D to model global structure. ProGAN (Karras et al., 2018) trains high-resolution GANs in the single-class setting by training a single model across a sequence of increasing resolutions.

In conditional GANs (Mirza & Osindero, 2014) class information can be fed into the model in various ways. In (Odena et al., 2017) it is provided to G by concatenating a 1-hot class vector to the noise vector, and the objective is modified to encourage conditional samples to maximize the corresponding class probability predicted by an auxiliary classifier. (2018) , D is conditioned by using the cosine similarity between its features and a set of learned class embeddings as additional evidence for distinguishing real and generated samples, effectively encouraging generation of samples whose features match a learned class prototype.

Objectively evaluating implicit generative models is difficult (Theis et al., 2015) . A variety of works have proposed heuristics for measuring the sample quality of models without tractable likelihoods (Salimans et al., 2016; Heusel et al., 2017; Bińkowski et al., 2018; Wu et al., 2017) . Of these, the Inception Score (IS, Salimans et al. (2016) ) and Fréchet Inception Distance (FID, Heusel et al. (2017) ) have become popular despite their notable flaws (Barratt & Sharma, 2018) . We employ them as approximate measures of sample quality, and to enable comparison against previous work.

3. SCALING UP GANS:

In this section, we explore methods for scaling up GAN training to reap the performance benefits of larger models and larger batches. As a baseline, we employ the SA-GAN architecture of Zhang et al. (2018) , which uses the hinge loss (Lim & Ye, 2017; Tran et al., 2017) GAN objective. We provide class information to G with class-conditional BatchNorm (Dumoulin et al., 2017; de Vries et al., 2017) and to D with projection (Miyato & Koyama, 2018) . The optimization settings follow Zhang et al. (2018) (notably employing Spectral Norm in G) with the modification that we halve the learning rates and take two D steps per G step. For evaluation, we employ moving averages of G's weights following Karras et al. (2018) ; Mescheder et al. (2018) ; Yazc et al. (2018) , with a decay of 0.9999. We use Orthogonal Initialization (Saxe et al., 2014) , whereas previous works used N (0, 0.02I) (Radford et al., 2016) or Xavier initialization (Glorot & Bengio, 2010) . Each model is trained on 128 to 512 cores of a Google TPUv3 Pod (Google, 2018), and computes BatchNorm statistics in G across all devices, rather than per-device as is typical. We find progressive growing (Karras et al., 2018) unnecessary even for our 512×512 models. Additional details are in Appendix C.

We begin by increasing the batch size for the baseline model, and immediately find tremendous benefits in doing so. Rows 1-4 of Table 1 show that simply increasing the batch size by a factor of 8 improves the state-of-the-art IS by 46%. We conjecture that this is a result of each batch covering more modes, providing better gradients for both networks. One notable side effect of this scaling is that our models reach better final performance in fewer iterations, but become unstable and undergo complete training collapse. We discuss the causes and ramifications of this in Section 4. For these experiments, we report scores from checkpoints saved just before collapse.

We then increase the width (number of channels) in each layer by 50%, approximately doubling the number of parameters in both models. This leads to a further IS improvement of 21%, which we posit is due to the increased capacity of the model relative to the complexity of the dataset. Doubling We note that class embeddings c used for the conditional BatchNorm layers in G contain a large number of weights. Instead of having a separate layer for each embedding (Miyato et al., 2018; Zhang et al., 2018) , we opt to use a shared embedding, which is linearly projected to each layer's gains and biases (Perez et al., 2018) . This reduces computation and memory costs, and improves training speed (in number of iterations required to reach a given performance) by 37%. Next, we add direct skip connections (skip-z) from the noise vector z to multiple layers of G rather than just the initial layer. The intuition behind this design is to allow G to use the latent space to directly influence features at different resolutions and levels of hierarchy. In BigGAN, this is accomplished by splitting z into one chunk per resolution, and concatenating each chunk to the conditional vector c which gets projected to the BatchNorm gains and biases. In BigGAN-deep, we use an even simpler design, concatenating the entire z with the conditional vector without splitting it into chunks. Previous works (Goodfellow et al., 2014; Denton et al., 2015) have considered variants of this concept; our implementation is a minor modification of this design. Skip-z provides a modest performance improvement of around 4%, and improves training speed by a further 18%.

3.1. TRADING OFF VARIETY AND FIDELITY WITH THE TRUNCATION TRICK:

Unlike models which need to backpropagate through their latents, GANs can employ an arbitrary prior p(z), yet the vast majority of previous works have chosen to draw z from either N (0, I) or U[-1, 1]. We question the optimality of this choice and explore alternatives in Appendix E.

Remarkably, our best results come from using a different latent distribution for sampling than was used in training. Taking a model trained with z ∼ N (0, I) and sampling z from a truncated normal (where values which fall outside a range are resampled to fall inside that range) immediately provides a boost to IS and FID. We call this the Truncation Trick: truncating a z vector by resampling the values with magnitude above a chosen threshold leads to improvement in individual sample quality at the cost of reduction in overall sample variety. Figure 2 (a) demonstrates this: as the threshold is reduced, and elements of z are truncated towards zero (the mode of the latent distribution), individual samples approach the mode of G's output distribution. Related observations about this trade-off were made in (Marchesi, 2016; Pieters & Wiering, 2014) .

This technique allows fine-grained, post-hoc selection of the trade-off between sample quality and variety for a given G. Notably, we can compute FID and IS for a range of thresholds, obtaining the variety-fidelity curve reminiscent of the precision-recall curve (Figure 17 ). As IS does not penalize lack of variety in class-conditional models, reducing the truncation threshold leads to a direct increase in IS (analogous to precision). FID penalizes lack of variety (analogous to recall) but also rewards precision, so we initially see a moderate improvement in FID, but as truncation approaches zero and variety diminishes, the FID sharply drops. The distribution shift caused by sampling with different latents than those seen in training is problematic for many models. Some of our larger models are not amenable to truncation, producing saturation artifacts (Figure 2 (b)) when fed truncated noise. To counteract this, we seek to enforce amenability to truncation by conditioning G to be smooth, so that the full space of z will map to good output samples. For this, we turn to Orthogonal Regularization (Brock et al., 2017) , which directly enforces the orthogonality condition:

EQUATION

where W is a weight matrix and β a hyperparameter. This regularization is known to often be too limiting (Miyato et al., 2018) , so we explore several variants designed to relax the constraint while still imparting the desired smoothness to our models. The version we find to work best removes the diagonal terms from the regularization, and aims to minimize the pairwise cosine similarity between filters but does not constrain their norm:

EQUATION

where 1 denotes a matrix with all elements set to 1. We sweep β values and select 10 -4 , finding this small added penalty sufficient to improve the likelihood that our models will be amenable to truncation. Across runs in Table 1 , we observe that without Orthogonal Regularization, only 16% of models are amenable to truncation, compared to 60% when trained with Orthogonal Regularization.

3.2. SUMMARY:

We find that current GAN techniques are sufficient to enable scaling to large models and distributed, large-batch training. We find that we can dramatically improve the state of the art and train models up to 512×512 resolution without need for explicit multiscale methods like Karras et al. (2018) . Despite these improvements, our models undergo training collapse, necessitating early stopping in practice. In the next two sections we investigate why settings which were stable in previous works become unstable when applied at scale. 

4. ANALYSIS:



4.1. CHARACTERIZING INSTABILITY: THE GENERATOR:

Much previous work has investigated GAN stability from a variety of analytical angles and on toy problems, but the instabilities we observe occur for settings which are stable at small scale, necessitating direct analysis at large scale. We monitor a range of weight, gradient, and loss statistics during training, in search of a metric which might presage the onset of training collapse, similar to (Odena et al., 2018) . We found the top three singular values σ 0 , σ 1 , σ 2 of each weight matrix to be the most informative. They can be efficiently computed using the Alrnoldi iteration method (Golub & der Vorst, 2000) , which extends the power iteration method, used in Miyato et al. (2018) , to estimation of additional singular vectors and values. A clear pattern emerges, as can be seen in Figure 3 (a) and Appendix F: most G layers have well-behaved spectral norms, but some layers (typically the first layer in G, which is over-complete and not convolutional) are ill-behaved, with spectral norms that grow throughout training and explode at collapse.

To ascertain if this pathology is a cause of collapse or merely a symptom, we study the effects of imposing additional conditioning on G to explicitly counteract spectral explosion. First, we directly regularize the top singular values σ 0 of each weight, either towards a fixed value σ reg or towards some ratio r of the second singular value, r • sg(σ 1 ) (with sg the stop-gradient operation to prevent the regularization from increasing σ 1 ). Alternatively, we employ a partial singular value decomposition to instead clamp σ 0 . Given a weight W , its first singular vectors u 0 and v 0 , and σ clamp the value to which the σ 0 will be clamped, our weights become:

EQUATION

where σ clamp is set to either σ reg or r • sg(σ 1 ). We observe that both with and without Spectral Normalization these techniques have the effect of preventing the gradual increase and explosion of either σ 0 or σ0 σ1 , but even though in some cases they mildly improve performance, no combination prevents training collapse. This evidence suggests that while conditioning G might improve stability, it is insufficient to ensure stability. We accordingly turn our attention to D.

4.2. CHARACTERIZING INSTABILITY: THE DISCRIMINATOR:

As with G, we analyze the spectra of D's weights to gain insight into its behavior, then seek to stabilize training by imposing additional constraints. Figure 3 (b) displays a typical plot of σ 0 for D (with further plots in Appendix F). Unlike G, we see that the spectra are noisy, σ0 σ1 is well-behaved, and the singular values grow throughout training but only jump at collapse, instead of exploding.

The spikes in D's spectra might suggest that it periodically receives very large gradients, but we observe that the Frobenius norms are smooth (Appendix F), suggesting that this effect is primarily concentrated on the top few singular directions. We posit that this noise is a result of optimization through the adversarial training process, where G periodically produces batches which strongly perturb D . If this spectral noise is causally related to instability, a natural counter is to employ gradient penalties, which explicitly regularize changes in D's Jacobian. We explore the R 1 zero-centered gradient penalty from Mescheder et al. (2018) :

EQUATION

With the default suggested γ strength of 10, training becomes stable and improves the smoothness and boundedness of spectra in both G and D, but performance severely degrades, resulting in a 45% reduction in IS. Reducing the penalty partially alleviates this degradation, but results in increasingly ill-behaved spectra; even with the penalty strength reduced to 1 (the lowest strength for which sudden collapse does not occur) the IS is reduced by 20%. Repeating this experiment with various strengths of Orthogonal Regularization, DropOut (Srivastava et al., 2014) , and L2 (See Appendix I for details), reveals similar behaviors for these regularization strategies: with high enough penalties on D, training stability can be achieved, but at a substantial cost to performance.

We also observe that D's loss approaches zero during training, but undergoes a sharp upward jump at collapse (Appendix F). One possible explanation for this behavior is that D is overfitting to the train- 

5.1. EVALUATION ON IMAGENET:

We evaluate our models on ImageNet ILSVRC 2012 (Russakovsky et al., 2015) at 128×128, 256×256, and 512×512 resolutions, employing the settings from Table 1 , row 8. The samples generated by our models are presented in Figure 4 , with additional samples in Appendix A, and online2 . We report IS and FID in Table 2 . As our models are able to trade sample variety for quality, it is unclear how best to compare against prior art; we accordingly report values at three settings, with complete curves in Appendix D. First, we report the FID/IS values at the truncation setting which attains the best FID. Second, we report the FID at the truncation setting for which our model's IS is the same as that attained by the real validation data, reasoning that this is a passable measure of maximum sample variety achieved while still achieving a good level of "objectness." Third, we report FID at the maximum IS achieved by each model, to demonstrate how much variety must be traded off to maximize quality. In all three cases, our models outperform the previous state-of-the-art IS and FID scores achieved by Miyato et al. (2018) and Zhang et al. (2018) .

In addition to the BigGAN model introduced in the first version of the paper and used in the majority of experiments (unless otherwise stated), we also present a 4x deeper model (BigGAN-deep) which uses a different configuration of residual blocks. As can be seen from We note that some failure modes of our partially-trained models are distinct from those previously observed. Most previous failures involve local artifacts (Odena et al., 2016) , images consisting of texture blobs instead of objects (Salimans et al., 2016) , or the canonical mode collapse. We observe class leakage, where images from one class contain properties of another, as exemplified by Figure 4 (d). We also find that many classes on ImageNet are more difficult than others for our model; our model is more successful at generating dogs (which make up a large portion of the dataset, and are mostly distinguished by their texture) than crowds (which comprise a small portion of the dataset and have more large-scale structure). Further discussion is available in Appendix A.

5.2. ADDITIONAL EVALUATION ON JFT-300M:

To confirm that our design choices are effective for even larger and more complex and diverse datasets, we also present results of our system on a subset of JFT-300M (Sun et al., 2017) . The full JFT-300M dataset contains 300M real-world images labeled with 18K categories. Since the category distribution is heavily long-tailed, we subsample the dataset to keep only images with the 8.5K most common labels. The resulting dataset contains 292M images -two orders of magnitude larger than ImageNet. For images with multiple labels, we sample a single label randomly and independently whenever an image is sampled. To compute IS and FID for the GANs trained on this dataset, we use an Inception v2 classifier (Szegedy et al., 2016) trained on this dataset. Quantitative results are presented in Table 3 . All models are trained with batch size 2048. We compare an ablated version of our model -comparable to SA-GAN (Zhang et al., 2018) but with the larger batch size -against a "full" BigGAN model that makes uses of all of the techniques applied to obtain the best results on ImageNet (shared embedding, skip-z, and orthogonal regularization). Our results show that these techniques substantially improve performance even in the setting of this much larger dataset at the same model capacity (64 base channels). We further show that for a dataset of this scale, we see significant additional improvements from expanding the capacity of our models to 128 base channels, while for ImageNet GANs that additional capacity was not beneficial.

In Figure 19 (Appendix D), we present truncation plots for models trained on this dataset. Unlike for ImageNet, where truncation limits of σ ≈ 0 tend to produce the highest fidelity scores, IS is typically maximized for our JFT-300M models when the truncation value σ ranges from 0.5 to 1. We suspect that this is at least partially due to the intra-class variability of JFT-300M labels, as well as the relative complexity of the image distribution, which includes images with multiple objects at a variety of scales. Interestingly, unlike models trained on ImageNet, where training tends to collapse without heavy regularization (Section 4), the models trained on JFT-300M remain stable over many hundreds of thousands of iterations. This suggests that moving beyond ImageNet to larger datasets may partially alleviate GAN stability issues.

The improvement over the baseline GAN model that we achieve on this dataset without changes to the underlying models or training and regularization techniques (beyond expanded capacity) demonstrates that our findings extend from ImageNet to datasets with scale and complexity thus far unprecedented for generative models of images.

6. CONCLUSION:

We have demonstrated that Generative Adversarial Networks trained to model natural images of multiple categories highly benefit from scaling up, both in terms of fidelity and variety of the generated samples. As a result, our models set a new level of performance among ImageNet GAN models, improving on the state of the art by a large margin. We have also presented an analysis of the training behavior of large scale GANs, characterized their stability in terms of the singular values of their weights, and discussed the interplay between stability and performance. 

APPENDIX B ARCHITECTURAL DETAILS:

In the BigGAN model (Figure 15 ), we use the ResNet (He et al., 2016) GAN architecture of (Zhang et al., 2018) , which is identical to that used by (Miyato et al., 2018) , but with the channel pattern in D modified so that the number of filters in the first convolutional layer of each block is equal to the number of output filters (rather than the number of input filters, as in Miyato et al. (2018) ; Gulrajani et al. (2017) ). We use a single shared class embedding in G, and skip connections for the latent vector z (skip-z). In particular, we employ hierarchical latent spaces, so that the latent vector z is split along its channel dimension into chunks of equal size (20-D in our case), and each chunk is concatenated to the shared class embedding and passed to a corresponding residual block as a conditioning vector. The conditioning of each block is linearly projected to produce per-sample gains and biases for the BatchNorm layers of the block. The bias projections are zero-centered, while the gain projections are centered at 1. Since the number of residual blocks depends on the image resolution, the full dimensionality of z is 120 for 128 × 128, 140 for 256 × 256, and 160 for 512 × 512 images.

The BigGAN-deep model (Figure 16 ) differs from BigGAN in several aspects. It uses a simpler variant of skip-z conditioning: instead of first splitting z into chunks, we concatenate the entire z with the class embedding, and pass the resulting vector to each residual block through skip connections.

BigGAN-deep is based on residual blocks with bottlenecks (He et al., 2016) , which incorporate two additional 1 × 1 convolutions: the first reduces the number of channels by a factor of 4 before the more expensive 3 × 3 convolutions; the second produces the required number of output channels. While BigGAN relies on 1 × 1 convolutions in the skip connections whenever the number of channels needs to change, in BigGAN-deep we use a different strategy aimed at preserving identity throughout the skip connections. In G, where the number of channels needs to be reduced, we simply retain the first group of channels and drop the rest to produce the required number of channels.

In D, where the number of channels should be increased, we pass the input channels unperturbed, and concatenate them with the remaining channels produced by a 1 × 1 convolution. As far as the network configuration is concerned, the discriminator is an exact reflection of the generator. There are two blocks at each resolution (BigGAN uses one), and as a result BigGAN-deep is four times deeper than BigGAN. Despite their increased depth, the BigGAN-deep models have significantly fewer parameters mainly due to the bottleneck structure of their residual blocks. For example, the Figure 19 : JFT-300M IS vs. FID at 256×256. We show truncation values from σ = 0 to σ = 2 (top) and from σ = 0.5 to σ = 1.5 (bottom). Each curve corresponds to a row in Table 3 . The curve labeled with baseline corresponds to the first row (with orthogonal regularization and other techniques disabled), while the rest correspond to rows 2-4 -the same architecture at different capacities (Ch).

APPENDIX E CHOOSING LATENT SPACES:

While most previous work has employed N (0, I) or U[-1, 1] as the prior for z (the noise input to G), we are free to choose any latent distribution from which we can sample. We explore the choice of latents by considering an array of possible designs, described below. For each latent, we provide the intuition behind its design and briefly describe how it performs when used as a drop-in replacement for z ∼ N (0, I) in an SA-GAN baseline. As the Truncation Trick proved more beneficial than switching to any of these latents, we do not perform a full ablation study, and employ z ∼ N (0, I) for our main results to take full advantage of truncation. The two latents which we find to work best without truncation are Bernoulli {0, 1} and Censored Normal max (N (0, I), 0), both of which improve speed of training and lightly improve final performance, but are less amenable to truncation.

We also ablate the choice of latent space dimensonality (which by default is z ∈ R 128 ), finding that we are able to successfully train with latent dimensions as low as z ∈ R 8 , and that with z ∈ R 32 we see a minimal drop in performance. While this is substantially smaller than many previous works, direct comparison to single-class networks (such as those in Karras et al. (2018) , which employ a z ∈ R 512 latent space on a highly constrained dataset with 30,000 images) is improper, as our networks have additional class information provided as input.

LATENTS:

• N (0, I). A standard choice of the latent space which we use in the main experiments.

• U[-1, 1]. Another standard choice; we find that it performs similarly to N (0, I).

• Bernoulli {0, 1}. A discrete latent might reflect our prior that underlying factors of variation in natural images are not continuous, but discrete (one feature is present, another is not). This latent outperforms N (0, I) (in terms of IS) by 8% and requires 60% fewer iterations. • max (N (0, I), 0), also called Censored Normal. This latent is designed to introduce sparsity in the latent space (reflecting our prior that certain latent features are sometimes present and sometimes not), but also allow those latents to vary continuously, expressing different degrees of intensity for latents which are active. This latent outperforms N (0, I) (in terms of IS) by 15-20% and tends to require fewer iterations. • Bernoulli {-1, 1}. This latent is designed to be discrete, but not sparse (as the network can learn to activate in response to negative inputs). This latent performs near-identically to N (0, I). • Independent Categorical in {-1, 0, 1}, with equal probability. This distribution is chosen to be discrete and have sparsity, but also to allow latents to take on both positive and negative values. This latent performs near-identically to N (0, I). • N (0, I) multiplied by Bernoulli {0, 1}. This distribution is chosen to have continuous latent factors which are also sparse (with a peak at zero), similar to Censored Normal but not constrained to be positive. This latent performs near-identically to N (0, I). • Concatenating N (0, I) and Bernoulli {0, 1}, each taking half of the latent dimensions. This is inspired by Chen et al. (2016) , and is chosen to allow some factors of variation to be discrete, while others are continuous. This latent outperforms N (0, I) by around 5%. • Variance annealing: we sample from N (0, σI), where σ is allowed to vary over training.

We compared a variety of piecewise schedules and found that starting with σ = 2 and annealing towards σ = 1 over the course of training mildly improved performance. The space of possible variance schedules is large, and we did not explore it in depth -we suspect that a more principled or better-tuned schedule could more strongly impact performance. • Per-sample variable variance: N (0, σ i I), where σ i ∼ U[σ l , σ h ] independently for each sample i in a batch, and (σ l , σ h ) are hyperparameters. This distribution was chosen to try and improve amenability to the Truncation Trick by feeding the network noise samples with non-constant variance. This did not appear to affect performance, but we did not explore it in depth. One might also consider scheduling (σ l , σ h ), similar to variance annealing. 29 ), the noise spikes resemble an impulse response: at each spike, the spectra jump upwards, then slowly decrease, with some oscillation.

APPENDIX F MONITORED TRAINING STATISTICS:

(a) G σ0 (b) G σ 0 σ 1 (c) G σ1 (d) G σ2 (e) D σ0 (f) D σ 0 σ 1 (g) D σ1 (h) D σ2

One possible explanation is that this behavior is a consequence of D memorizing the training data, as suggested by experiments in Section 4.2. As it approaches perfect memorization, it receives less and less signal from real data, as both the original GAN loss and the hinge loss provide zero gradients when D outputs a confident and correct prediction for a given example. If the gradient signal from real data attenuates to zero, this can result in D eventually becoming biased due to exclusively received gradients that encourage its outputs to be negative. If this bias passes a certain threshold, D will eventually misclassify a large number of real examples and receive a large gradient encouraging positive outputs, resulting in the observed impulse responses.

This argument suggests several fixes. First, one might consider an unbounded loss (such as the Wasserstein loss (Arjovsky et al., 2017) ) which would not suffer this gradient attentuation. We found that even with gradient penalties and brief re-tuning of optimizer hyperparameters, our models did not stably train for more than a few thousand iterations with this loss. We instead explored changing the margin of the hinge loss as a partial compromise: for a given model and minibatch of data, increasing the margin will result in more examples falling within the margin, and thus contributing to the loss. 3 . Training with a smaller margin (by a factor of 2) measurably reduces performance, but training with a larger margin (by up to a factor of 3) does not prevent collapse or reduce the noise in D's spectra. Increasing the margin beyond 3 results in unstable training similar to using the Wasserstein loss. Finally, the memorization argument might suggest that using a smaller D or using dropout in D would improve training by reducing its capacity to memorize, but in practice this degrades training.

:

https://tfhub.dev/s?q=biggan

https://drive.google.com/drive/folders/1lWC6XEPD0LT5KUnPXeve_kWeY-FxH002

Unconstrained models could easily learn a different output scale to account for this margin, but the use of Spectral Normalization constrains our models and makes the specific selection of the margin meaningful.

Figures and Tables:

Figure 2: (a) The effects of increasing truncation. From left to right, the threshold is set to 2, 1, 0.5, 0.04. (b) Saturation artifacts from applying truncation to a poorly conditioned model.

Figure 3: A typical plot of the first singular value σ 0 in the layers of G (a) and D (b) before Spectral Normalization. Most layers in G have well-behaved spectra, but without constraints a small subset grow throughout training and explode at collapse. D's spectra are noisier but otherwise betterbehaved. Colors from red to violet indicate increasing depth.

Figure 4: Samples from our BigGAN model with truncation threshold 0.5 (a-c) and an example of class leakage in a partially trained model (d).

Figure 7: Comparing easy classes (a) with difficult classes (b) at 512×512. Classes such as dogs which are largely textural, and common in the dataset, are far easier to model than classes involving unaligned human faces or crowds. Such classes are more dynamic and structured, and often have details to which human observers are more sensitive. The difficulty of modeling global structure is further exacerbated when producing high-resolution images, even with non-local blocks.

Figure 8: Interpolations between z, c pairs.

Figure 9: Interpolations between c with z held constant. Pose semantics are frequently maintained between endpoints (particularly in the final row). Row 2 demonstrates that grayscale is encoded in the joint z, c space, rather than in z.

Figure 10: Nearest neighbors in VGG-16-fc7 (Simonyan & Zisserman, 2015) feature space. The generated image is in the top left.

Figure 11: Nearest neighbors in ResNet-50-avgpool (He et al., 2016) feature space. The generated image is in the top left.

Figure 12: Nearest neighbors in pixel space. The generated image is in the top left.

Figure 13: Nearest neighbors in VGG-16-fc7 (Simonyan & Zisserman, 2015) feature space. The generated image is in the top left.

Figure 14: Nearest neighbors in ResNet-50-avgpool (He et al., 2016) feature space. The generated image is in the top left.

Figure 15: (a) A typical architectural layout for BigGAN's G; details are in the following tables. (b) A Residual Block (ResBlock up) in BigGAN's G. (c) A Residual Block (ResBlock down) in BigGAN's D.

Figure 20: Training statistics for a typical model without special modifications. Collapse occurs after 200000 iterations.

Figure 21: G training statistics with σ 0 in G regularized towards 1. Collapse occurs after 125000 iterations.

Figure 22: D training statistics with σ 0 in G regularized towards 1. Collapse occurs after 125000 iterations.

Figure 23: G training statistics with an R1 Gradient Penalty of strength 10 on D. This model does not collapse, but only reaches a maximum IS of 55.

Figure 24: D training statistics with an R1 Gradient Penalty of strength 10 on D. This model does not collapse, but only reaches a maximum IS of 55.

Figure 25: G training statistics with Dropout (keep probability 0.8) applied to the last feature layer of D. This model does not collapse, but only reaches a maximum IS of 70.

Figure 26: D training statistics with Dropout (keep probability 0.8) applied to the last feature layer of D. This model does not collapse, but only reaches a maximum IS of 70.

Figure 27: Additional training statistics for a typical model without special modifications. Collapse occurs after 200000 iterations.

Figure 28: Additional training statistics with an R1 Gradient Penalty of strength 10 on D. This model does not collapse, but only reaches a maximum IS of 55.

Fréchet Inception Distance (FID, lower is better) and Inception Score (IS, higher is better) for ablations of our proposed modifications. Batch is batch size, Param is total number of parameters, Ch. is the channel multiplier representing the number of units in each layer, Shared is using shared embeddings, Skip-z is using skip connections from the latent to multiple layers, Ortho. is Orthogonal Regularization, and Itr indicates if the setting is stable to 10 6 iterations, or it collapses at the given iteration. Other than rows 1-4, results are computed across 8 random initializations.

Table content: <table/>

ing set, memorizing training examples rather than learning some meaningful boundary between real and generated images. As a simple test for D's memorization (related toGulrajani et al. (2017)), we evaluate uncollapsed discriminators on the ImageNet training and validation sets, and measure what percentage of samples are classified as real or generated. While the training accuracy is consistently above 98%, the validation accuracy falls in the range of 50-55%, no better than random guessing (regardless of regularization strategy). This confirms that D is indeed memorizing the training set; we deem this in line with D's role, which is not explicitly to generalize, but to distill the training data and provide a useful learning signal for G. Additional experiments and discussion are provided in Appendix G. Evaluation of models at different resolutions. We report scores without truncation (Column 3), scores at the best FID (Column 4), scores at the IS of validation data (Column 5), and scores at the max IS (Column 6). Standard deviations are computed over at least three random initializations.

Table content: <table><tr><td>Model</td><td>Res.</td><td>FID/IS</td><td>(min FID) / IS</td><td>FID / (valid IS)</td><td>FID / (max IS)</td></tr><tr><td>SN-GAN</td><td>128</td><td>27.62/36.80</td><td>N/A</td><td>N/A</td><td>N/A</td></tr><tr><td>SA-GAN</td><td>128</td><td>18.65/52.52</td><td>N/A</td><td>N/A</td><td>N/A</td></tr><tr><td>BigGAN</td><td>128</td><td>8.7 ± .6/98.8 ± 3</td><td colspan="2">7.7 ± .2/126.5 ± 0 9.6 ± .4/166.3 ± 1</td><td>25 ± 2/206 ±</td></tr><tr><td>BigGAN</td><td colspan="4">256 8.7 ± .1/142.3 ± 2 7.7 ± .1/178.0 ± 5 9.3 ± .3/233.1 ± 1</td><td>25 ± 5/291 ±</td></tr><tr><td>BigGAN</td><td>512</td><td>8.1/144.2</td><td>7.6/170.3</td><td>11.8/241.4</td><td>27.0/275</td></tr><tr><td colspan="6">BigGAN-deep 128 5.7 ± .3/124.5 ± 2 6.3 ± .3/148.1 ± 4 7.4 ± .6/166.5 ± 1 25 ± 2/253 ± 11</td></tr><tr><td colspan="5">BigGAN-deep 256 6.9 ± .2/171.4 ± 2 7.0 ± .1/202.6 ± 2 8.1 ± .1/232.5 ± 2</td><td>27 ± 8/317 ±</td></tr><tr><td colspan="2">BigGAN-deep 512</td><td>7.5/152.8</td><td>7.7/181.4</td><td>11.5/241.5</td><td>39.7/298</td></tr></table>

BigGAN-deep substantially outperforms BigGAN across all resolutions and metrics. This confirms that our findings

Table content: <table><tr><td colspan="2">Ch. Param (M) Shared Skip-z Ortho.</td><td>FID</td><td>IS</td><td colspan="2">(min FID) / IS FID / (max IS)</td></tr><tr><td>64</td><td>317.1</td><td colspan="2">48.38 23.27</td><td>48.6/23.1</td><td>49.1/23.9</td></tr><tr><td>64</td><td>99.4</td><td colspan="2">23.48 24.78</td><td>22.4/21.0</td><td>60.9/35.8</td></tr><tr><td>96</td><td>207.9</td><td colspan="2">18.84 27.86</td><td>17.1/23.3</td><td>51.6/38.1</td></tr><tr><td>128</td><td>355.7</td><td colspan="2">13.75 30.61</td><td>13.0/28.0</td><td>46.2/47.8</td></tr></table>

BigGAN results on JFT-300M at 256×256 resolution. The FID and IS columns report these scores given by the JFT-300M-trained Inception v2 classifier with noise distributed as z ∼ N (0, I) (non-truncated). The (min FID) / IS and FID / (max IS) columns report scores at the best FID and IS from a sweep across truncated noise distributions ranging from σ = 0 to σ = 2. Images from the JFT-300M validation set have an IS of 50.88 and FID of 1.94. extend to other architectures, and that increased depth leads to improvement in sample quality. Both BigGAN and BigGAN-deep architectures are described in Appendix B.Our observation that D overfits to the training set, coupled with our model's sample quality, raises the obvious question of whether or not G simply memorizes training points. To test this, we perform class-wise nearest neighbors analysis in pixel space and the feature space of pre-trained classifier networks (Appendix A). In addition, we present both interpolations between samples and class-wise interpolations (where z is held constant) in Figures8 and 9. Our model convincingly interpolates between disparate samples, and the nearest neighbors for its samples are visually distinct, suggesting that our model does not simply memorize training data.

Table content: <table/>

BigGAN architecture for 128 × 128 images. ch represents the channel width multiplier in each network from Table 1.

Table content: <table><tr><td>z ∈ R 120 ∼ N (0, I) Embed(y) ∈ R 128</td><td>RGB image x ∈ R 128×128×3</td></tr><tr><td>Linear (20 + 128) → 4 × 4 × 16ch</td><td>ResBlock down ch → 2ch</td></tr><tr><td>ResBlock up 16ch → 16ch</td><td>Non-Local Block (64 × 64)</td></tr><tr><td>ResBlock up 16ch → 8ch</td><td>ResBlock down 2ch → 4ch</td></tr><tr><td>ResBlock up 8ch → 4ch</td><td>ResBlock down 4ch → 8ch</td></tr><tr><td>ResBlock up 4ch → 2ch</td><td>ResBlock down 8ch → 16ch</td></tr><tr><td>Non-Local Block (64 × 64)</td><td>ResBlock down 16ch → 16ch</td></tr><tr><td>ResBlock up 2ch → ch</td><td>ResBlock 16ch → 16ch</td></tr><tr><td>BN, ReLU, 3 × 3 Conv ch → 3</td><td>ReLU, Global sum pooling</td></tr><tr><td>Tanh</td><td>Embed(y)•h + (linear → 1)</td></tr><tr><td>(a) Generator</td><td>(b) Discriminator</td></tr></table>

BigGAN architecture for 256 × 256 images. Relative to the 128 × 128 architecture, we add an additional ResBlock in each network at 16×16 resolution, and move the non-local block in G to 128 × 128 resolution. Memory constraints prevent us from moving the non-local block in D.

Table content: <table><tr><td/><td>RGB image x ∈ R 256×256×3</td></tr><tr><td/><td>ResBlock down ch → 2ch</td></tr><tr><td>ResBlock up 16ch → 16ch</td><td>ResBlock down 2ch → 4ch</td></tr><tr><td>ResBlock up 16ch → 8ch</td><td>Non-Local Block (64 × 64)</td></tr><tr><td>ResBlock up 8ch → 8ch</td><td>ResBlock down 4ch → 8ch</td></tr><tr><td>ResBlock up 8ch → 4ch</td><td>ResBlock down 8ch → 8ch</td></tr><tr><td>ResBlock up 4ch → 2ch</td><td>ResBlock down 8ch → 16ch</td></tr><tr><td>Non-Local Block (128 × 128)</td><td>ResBlock down 16ch → 16ch</td></tr><tr><td>ResBlock up 2ch → ch</td><td>ResBlock 16ch → 16ch</td></tr><tr><td>BN, ReLU, 3 × 3 Conv ch → 3</td><td>ReLU, Global sum pooling</td></tr><tr><td>Tanh</td><td>Embed(y)•h + (linear → 1)</td></tr><tr><td>(a) Generator</td><td>(b) Discriminator</td></tr></table>

BigGAN architecture for 512 × 512 images. Relative to the 256 × 256 architecture, we add an additional ResBlock at the 512 × 512 resolution. Memory constraints force us to move the non-local block in both networks back to 64 × 64 resolution as in the 128 × 128 pixel setting.

Table content: <table><tr><td>z ∈ R 160 ∼ N (0, I) Embed(y) ∈ R 128</td><td>RGB image x ∈ R 512×512×3</td></tr><tr><td>Linear (20 + 128) → 4 × 4 × 16ch</td><td>ResBlock down ch → ch</td></tr><tr><td>ResBlock up 16ch → 16ch</td><td>ResBlock down ch → 2ch</td></tr><tr><td>ResBlock up 16ch → 8ch</td><td>ResBlock down 2ch → 4ch</td></tr><tr><td>ResBlock up 8ch → 8ch</td><td>Non-Local Block (64 × 64)</td></tr><tr><td>ResBlock up 8ch → 4ch</td><td>ResBlock down 4ch → 8ch</td></tr><tr><td>Non-Local Block (64 × 64)</td><td>ResBlock down 8ch → 8ch</td></tr><tr><td>ResBlock up 4ch → 2ch</td><td>ResBlock down 8ch → 16ch</td></tr><tr><td>ResBlock up 2ch → ch</td><td>ResBlock down 16ch → 16ch</td></tr><tr><td>ResBlock up ch → ch</td><td>ResBlock 16ch → 16ch</td></tr><tr><td>BN, ReLU, 3 × 3 Conv ch → 3</td><td>ReLU, Global sum pooling</td></tr><tr><td>Tanh</td><td>Embed(y)•h + (linear → 1)</td></tr><tr><td>(a) Generator</td><td>(b) Discriminator</td></tr></table>

BigGAN-deep architecture for 128 × 128 images.

Table content: <table><tr><td>z ∈ R 128 ∼ N (0, I) Embed(y) ∈ R 128</td><td>RGB image x ∈ R 128×128×3</td></tr><tr><td>Linear (128 + 128) → 4 × 4 × 16ch</td><td>3 × 3 Conv 3 → ch</td></tr><tr><td>ResBlock 16ch → 16ch</td><td>ResBlock down ch → 2ch</td></tr><tr><td>ResBlock up 16ch → 16ch</td><td>ResBlock 2ch → 2ch</td></tr><tr><td>ResBlock 16ch → 16ch</td><td>Non-Local Block (64 × 64)</td></tr><tr><td>ResBlock up 16ch → 8ch</td><td>ResBlock down 2ch → 4ch</td></tr><tr><td>ResBlock 8ch → 8ch</td><td>ResBlock 4ch → 4ch</td></tr><tr><td>ResBlock up 8ch → 4ch</td><td>ResBlock down 4ch → 8ch</td></tr><tr><td>ResBlock 4ch → 4ch</td><td>ResBlock 8ch → 8ch</td></tr><tr><td>ResBlock up 4ch → 2ch</td><td>ResBlock down 8ch → 16ch</td></tr><tr><td>Non-Local Block (64 × 64)</td><td>ResBlock 16ch → 16ch</td></tr><tr><td>ResBlock 2ch → 2ch</td><td>ResBlock down 16ch → 16ch</td></tr><tr><td>ResBlock up 2ch → ch</td><td>ResBlock 16ch → 16ch</td></tr><tr><td>BN, ReLU, 3 × 3 Conv ch → 3</td><td>ReLU, Global sum pooling</td></tr><tr><td>Tanh</td><td>Embed(y)•h + (linear → 1)</td></tr><tr><td>(a) Generator</td><td>(b) Discriminator</td></tr></table>

BigGAN-deep architecture for 512 × 512 images.

Table content: <table><tr><td>z ∈ R 128 ∼ N (0, I) Embed(y) ∈ R 128</td><td>RGB image x ∈ R 512×512×3</td></tr><tr><td>Linear (128 + 128) → 4 × 4 × 16ch</td><td>3 × 3 Conv 3 → ch</td></tr><tr><td>ResBlock 16ch → 16ch</td><td>ResBlock down ch → ch</td></tr><tr><td>ResBlock up 16ch → 16ch</td><td>ResBlock ch → ch</td></tr><tr><td>ResBlock 16ch → 16ch</td><td>ResBlock down ch → 2ch</td></tr><tr><td>ResBlock up 16ch → 8ch</td><td>ResBlock 2ch → 2ch</td></tr><tr><td>ResBlock 8ch → 8ch</td><td>ResBlock down 2ch → 4ch</td></tr><tr><td>ResBlock up 8ch → 8ch</td><td>ResBlock 4ch → 4ch</td></tr><tr><td>ResBlock 8ch → 8ch</td><td>Non-Local Block (64 × 64)</td></tr><tr><td>ResBlock up 8ch → 4ch</td><td>ResBlock down 4ch → 8ch</td></tr><tr><td>Non-Local Block (64 × 64)</td><td>ResBlock 8ch → 8ch</td></tr><tr><td>ResBlock 4ch → 4ch</td><td>ResBlock down 8ch → 8ch</td></tr><tr><td>ResBlock up 4ch → 2ch</td><td>ResBlock 8ch → 8ch</td></tr><tr><td>ResBlock 2ch → 2ch</td><td>ResBlock down 8ch → 16ch</td></tr><tr><td>ResBlock up 2ch → ch</td><td>ResBlock 16ch → 16ch</td></tr><tr><td>ResBlock ch → ch</td><td>ResBlock down 16ch → 16ch</td></tr><tr><td>ResBlock up ch → ch</td><td>ResBlock 16ch → 16ch</td></tr><tr><td>BN, ReLU, 3 × 3 Conv ch → 3</td><td>ReLU, Global sum pooling</td></tr><tr><td>Tanh</td><td>Embed(y)•h + (linear → 1)</td></tr><tr><td>(a) Generator</td><td>(b) Discriminator</td></tr></table>

Peer review (iclr2019):
 Summary: This paper proposes a suite of tricks for training large-scale GANs, and obtaining state-of-the-art results for high-resolution images. The paper starts from a self-attention GAN baseline (Zhang 2018), and proposes: - Increasing batch size (8x) and model size (2x) - Splitting noise z in multiple chunks, and injecting it in multiple layers of the generator - Sampling from truncated normal distribution, where samples with norms that exceed a specific threshold are resampled. This seems to be used only at test-time and is used to control variety-fidelity tradeoff. The generator is encouraged to be smooth using an orthogonal regularization term. In addition, the paper proposes practical recipes for characterizing collapse in GANs. In the generator, the exploding of the top 3 singular values of each weight matrix seem to indicate collapse. In the discriminator, the sudden increase of the ratio of first/second singular value of weight matrices indicate collapse in GANs. Interestingly, the paper suggests that various regularization methods which can improve stability in GAN training, do not necessarily correspond to improvement in performance. Strengths: - Proposed techniques are intuitive and very well motivated - One of the big pluses of this work is that authors try to "quantify" each proposed technique with training speed and/or performance improvement. This is really a good practice. - Detailed analysis for detecting collapse and improving stability in large-scale GAN - Probably no need to mention that, but results are quite impressive Weaknesses: - Computational budget required is massive. The paper mentions model use from 128-256 TPUs, which severely limits reproducibility of results. Comments/Questions: - Can you elaborate more on why BatchNorm statistics are computed across all devices as opposed to per-device? Was this crucial for best performance? - It is not clear if provided analysis for large-scale GANs apply for small-medium sized GANs. Providing such analysis would be also helpful for the community. - How do you see the impact of the suggested techniques on tackling harder data-modalities for GANs, e.g. text or sequential data in general? Overall recommendation: The paper is well written, ideas are well motivated/justified and results are very compelling. This is a good paper and I higly recommend acceptance.