Title: Large Scale GAN Training for High Fidelity Natural Image Synthesis
Authors: Andrew Brock
Jeff Donahue
Karen Simonyan
Venue: ICLR 2019
Keywords: GANs
Generative Models
Large Scale Training
Deep Learning

Abstract: Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick", allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128x128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.3 and Frechet Inception Distance (FID) of 9.6, improving over the previous best IS of 52.52 and FID of 18.65.

TL;DR: GANs benefit from scaling up.

Review discussion:
This paper present extensions of the Self-Attention Generative Adversarial Network approach SAGAN, leading to impressive images generations conditioned on imagenet classes.
The key components of the approach are :
- increasing the batch size by a factor 8
- augmenting the width of the networks by 50%
These first two elements result in an Inception score (IS) boost from 52 to 93.
- the use of shared embeddings for the class conditioned batch norm layers, orthonormal regularization and hierarchical latent space bring an additional boost of IS 99.
The core novel element of the paper is the truncation trick: At train time, the input z is sampled from a normal distribution but at test time, a truncated normal distribution is used: when the magnitude of elements of z are above a certain threshold, they are re-sampled.
Variations of this threshold lead to variations in FD and IS, as shown in insightful experiments. The comments that more data helps (internal dataset experiments) is also informative.
Very nice to have included negative results and detailed parameter sweeps.
This is a very nice work with impressive results, a great progress achievement in the field of image generation.
Very well written.
Suggestions/questions:
- it would be nice to also propose unconditioned experiments.
It would be good to give an idea in the text of TPU-GPU equivalence in terms of feasibility of a standard GPU implementation - computation time it would involve.
- I understand that no data augmentation was used during training?
- clarification of the truncation trick: if the elements of z are re-sampled and are still above the threshold, are they re-sampled again and again until they are all below the given threshold?
- A sentence could be added to explain the truncation trick in the abstract directly since it is simple to understand and is key to the quality of the results.
- A reference to Appendix C could be given at the beginning of the Experiments section to help the reader find these details more easily.
- It would be nice to display more Nearest neighbors for the dog image.
- It would be nice to add a figure of random generations.
- make the bib uniform: remove unnecessary doi - url - cvpr page numbers
We would like to thank Reviewer 3 for the review and constructive suggestions. Our responses inline:
>it would be nice to also propose unconditioned experiments.
-We agree; this was simply not within the scope of the work we conducted.
>I understand that no data augmentation was used during training?
-This is correct, and consistent with previous works (Spectral Normalization and WGAN-GP). We briefly experimented with data augmentation (random crops and horizontal flips) but did not notice any measurable performance difference.
>clarification of the truncation trick: if the elements of z are re-sampled and are still above the threshold, are they re-sampled again and again until they are all below the given threshold?
-Yes, this can effectively be seen as modifying the PDF of z to have no mass outside of the truncation threshold. TensorFlow offers a built-in implementation with tf.random.truncated_normal.
>A sentence could be added to explain the truncation trick in the abstract directly since it is simple to understand and is key to the quality of the results.
-We have revised the abstract to explain the truncation trick as controlling the tradeoff between fidelity and diversity by reducing the variance of the Generator’s input.
>A reference to Appendix C could be given at the beginning of the Experiments section to help the reader find these details more easily.
-Thanks for the pointer! We have added this reference.
>It would be nice to add a figure of random generations.
-In the caption of Figure 5, we include a link to an anonymous drive folder with sample sheets at different resolutions and truncation values, with 12 random images per class.
>make the bib uniform: remove unnecessary doi - url - cvpr page numbers
-Thanks, we have fixed this.
We've recently been made aware of two prior works that observe a correlation between the variance of the latent noise and the variety/quality of the Generator outputs. We will be adding references accordingly.
[1] Marco Marchesi. Megapixel Size Image Creation using Generative Adversarial Networks. arXiv preprint arXiv:1706.00082.
[2] Mathijs Pieters and Marco Wiering. Comparing Generative Adversarial Network Techniques for Image Creation and Modification. arXiv preprint arXiv:1803.09093.
Summary:
The authors present a empirical investigation of methods for scaling GANs to complex datasets, such as ImageNet, for class-conditioned image generation. They first build and describe a strong baseline based on recently proposed techniques for GANs and push the performance on large datasets with several modifications presented sequentially, to obtain strong state-of-the-art IS/FID scores, as well as impressive visual results. The authors propose a simple truncation trick to control the fidelity/variance which is interesting on its own but cannot always scale with the architecture. The authors further propose a orthogonalization-based regularization to mitigate this problem. An investigation of training collapse at large scale is also performed; the authors investigate some regularization schemes based on gathered empirical evidence.
As a result, they explore and discard Spectral Normalization of the generator as a way to prevent collapse and show that a severe tradeoff between stability and quality can be controlled when using zero-centered gradient penalties in the Discriminator.
In the end, no solution that can ensure quality and stability is found, except having prohibitively large amounts of data (~300M images). Models are evaluated on the ImageNet and on this internal, bigger dataset.
Pros:
- This investigation gives a significant amount of insights on GAN stability and performance at large scales, which should be useful for anyone working with GANs on complex datasets (and that have access to great computational resources).
- Even though commonly used evaluations metrics for GANs are still not fully adequate, the authors obtain quantitative performance significantly beyond previous work, which seems indeed correlated with remarkable visual results.
- The baseline and added modifications are well presented and clearly explained. The Appendices also have great value in that regard.
Cons:
- Discussions sometimes lack depth or are absent.
For example, it is unclear to me why some larger models are not amenable to truncation.
Besides visible artifacts, what does it mean? Why does a smoother G reduces those artifacts? Were samples from those networks better without using truncation? Why would this be?
Authors report how wider networks perform best, and how deeper networks degrade performance. Again, discussions are lacking, and it doesn’t seem the authors tried to understand why such behaviors were shown.
Even though this is mostly an empirical investigation, I think some more efforts should be put in understanding and explaining why some of those behaviors are shown, as I think it can bootstrap future work more easily.
- In Section 3.1 : “Across runs in Table 1, we observe that without Orthogonal Regularization, only 16% of models are amenable to truncation compared to 60% with Orthogonal Regularization.” For me, this is not particularly clear. Is this something the reader should understand from Table 1?
- I question the choice of sections chosen to be in the main paper/appendices. I greatly appreciated the negative results reported in the main text as well as in the appendices and this has significant value.
However, as this is to me mostly a detailed empirical investigation and presentation of high-performance GANs on large scales, I would be likely to share this with colleagues who want to tackle similar problems. In this case, if future readers limit themselves to the main text, I think it can have more value to present some content form Appendix B and C than to have more than a full page on stability investigations and attempted tricks that turned out not to be used to reach maximal performance.
However I do not want to discourage publishing of negative results, and I definitely wish to see this investigation in the paper, but I merely question the positioning of such information. With regard to my first negative point above about the lack of discussions, it seems the analysis of Section 4 is disproportionate compared to other places.
Suggestions/Comments:
- Regarding the diversity/fidelity tradeoff using different truncation thresholds, I think constraining the norm of the sampled noise vectors to the exact threshold value (by projecting the samples on the 0-centered hyper-sphere of radius = threshold) could yield even more interesting or more informative Figures, as obtained scores or samples on the edge of that hyper-sphere might provide information on the ‘guaranteed’ (not proven) quality/fidelity of samples mapped from inside that hyper-sphere.
- In Appendix D, the Figures could be slightly clarified by using a colored heatmap to color the curve, with colors corresponding to the threshold values. Similar curves could also be produced with the hyper-sphere projection proposed above to have a slightly clearer idea of the behavior on the limit of that hyper-sphere.
- In Section 4.2, in the second paragraph, you refer to Appendix F and describe “sharp upward jump at collapse” in D’s loss.
However, it seems the only Figure showing D’s loss when unconstrained is Figure 26, in which it is hard to notice any significant jump in the loss.
- In Appendix F, Figure 20 d), the title seems wrong. It seems to report sigma^2 values, but the title says “losses”.
This investigation of GAN scalability is successful results-wise even though the inability to stabilize training without sacrificing great performance on ImageNet is disappointing. The improvement over previous SOTA is definitely significant. This work thus shows a modern GAN architecture for complex datasets that could be a strong basis for future work.
However, I think the paper could and should be improved with some more detailed analysis and discussions of exhibited behaviors in order to further guide and encourage future work. It could also be clarified on some aspects, and potentially re-structured a bit to be better align with its probable impact directions.  I would also be curious to see the proposed techniques applied on simpler datasets. Can this be useful for someone having less compute power and working on something similar to CelebA?
We would like to thank Reviewer 2 for their review and constructive suggestions. Our responses inline:
>Discussions sometimes lack depth or are absent.
-We have added an additional section (Appendix G) expanding on our discussion and providing additional insight into the observed instabilities.
>For example, it is unclear to me why some larger models are not amenable to truncation.
Besides visible artifacts, what does it mean? Why does a smoother G reduces those artifacts?
-Truncation introduces a train-test disparity in G’s inputs--at sampling time, G is given a distribution it has effectively never seen in training. The observation that imposing orthogonality constraints improves amenability to truncation is empirical. Our suspicion is that if G is not encouraged to be “smooth” in some sense, then it is likely that G will only properly generate images given points from the untruncated distribution. We hypothesize that models which are not amenable end up learning mappings which, when given truncated noise, either attenuate or amplify certain activation pathways, leading to extreme output values (hence the observed saturation artifacts). We speculate that encouraging G’s filters to have minimum pairwise cosine similarity means that, when exposed to distribution shift, the network’s features are less correlated and less likely to align and amplify an activation path it would otherwise have learned to scale properly.
>Were samples from those networks better without using truncation? Why would this be?
-Samples from those networks without truncation do not have measurably different quality, and their training metrics (losses, singular values) show no differences. Aside from empirically testing each network individually for amenability to truncation, we found no other way to check for that amenability.
> Authors report how wider networks perform best, and how deeper networks degrade performance. Again, discussions are lacking, and it doesn’t seem the authors tried to understand why such behaviors were shown.
Even though this is mostly an empirical investigation, I think some more efforts should be put in understanding and explaining why some of those behaviors are shown, as I think it can bootstrap future work more easily.
-We are wary of explanations for which we do not have evidence. For each of the modifications introduced in Section 3, we offer a succinct conjecture as to why that change improves performance, but we are not aware of any existing reliable, informative metric which we could employ to understand or trace the source of each observed behavior, particularly with respect to GAN stability or performance.
Regarding depth vs width: This paper is empirical, and we only briefly experimented with increasing depth analogously to increasing width.
While increasing width provided an immediate measurable benefit, increasing depth did not. We felt that it was better to report the results of this brief investigation than to omit it for a lack of investigatory depth.
> In Section 3.1 : “Across runs in Table 1, we observe that without Orthogonal Regularization, only 16% of models are amenable to truncation compared to 60% with Orthogonal Regularization.” For me, this is not particularly clear. Is this something the reader should understand from Table 1?
-This means that of all the models we trained for the study presented in Table 1 which did not use Orthogonal Regularization, only 16% were amenable to truncation. Of all the models which we trained for the study presented in Table 2 which did use Orthogonal Regularization, 60% were amenable to truncation. This is not reflected in Table 1, which is merely a presentation of how the introduced modifications impact performance.
>I question the choice of sections chosen to be in the main paper/appendices. I greatly appreciated the negative results reported in the main text as well as in the appendices and this has significant value.
However, as this is to me mostly a detailed empirical investigation and presentation of high-performance GANs on large scales, I would be likely to share this with colleagues who want to tackle similar problems. In this case, if future readers limit themselves to the main text, I think it can have more value to present some content form Appendix B and C than to have more than a full page on stability investigations and attempted tricks that turned out not to be used to reach maximal performance.
However I do not want to discourage publishing of negative results, and I definitely wish to see this investigation in the paper, but I merely question the positioning of such information. With regard to my first negative point above about the lack of discussions, it seems the analysis of Section 4 is disproportionate compared to other places.
-We appreciate this suggestion.
While we recognize that this paper generally has a strong focus on implementation details, we felt that this instability was one of the most salient behaviors we observed, and that future work would be best served by presenting our investigations and attempts to understand its source, even if these methods did not improve performance.  The information in Appendix B and C is intended to be of interest to those who want to reproduce our experiments, so it largely comprises hyperparameters and architectural details that we felt were not necessary to understand the main results of the paper.
>In Appendix F, Figure 20 d), the title seems wrong. It seems to report sigma^2 values, but the title says “losses”.
-Thanks! This was indeed an error, which we’ve corrected in the updated draft.
>I would also be curious to see the proposed techniques applied on simpler datasets. Can this be useful for someone having less compute power and working on something similar to CelebA?
-The goal of this work is to explore GANs at large scale; the exploration of small or medium scale models would indeed be interesting for another study.
Having said that, we do evaluate BigGAN on conditional CIFAR-10 (mentioned briefly in Appendix C.2) and obtain an IS of 9.22 and an FID of 14.73 without truncation, which to our knowledge are better than any published results.
Summary:
This paper proposes a suite of tricks for training large-scale GANs, and obtaining state-of-the-art results for high-resolution images. The paper starts from a self-attention GAN baseline (Zhang 2018), and proposes:
-	Increasing batch size (8x) and model size (2x)
-	Splitting noise z in multiple chunks, and injecting it in multiple layers of the generator
-	Sampling from truncated normal distribution, where samples with norms that exceed a specific threshold are resampled. This seems to be used only at test-time and is used to control variety-fidelity tradeoff. The generator is encouraged to be smooth using an orthogonal regularization term.
In addition, the paper proposes practical recipes for characterizing collapse in GANs. In the generator, the exploding of the top 3 singular values of each weight matrix seem to indicate collapse. In the discriminator, the sudden increase of the ratio of first/second singular value of weight matrices indicate collapse in GANs. Interestingly, the paper suggests that various regularization methods which can improve stability in GAN training, do not necessarily correspond to improvement in performance.
Strengths:
-	Proposed techniques are intuitive and very well motivated
-	One of the big pluses of this work is that authors try to "quantify" each proposed technique with training speed and/or performance improvement. This is really a good practice.
-	Detailed analysis for detecting collapse and improving stability in large-scale GAN
-	Probably no need to mention that, but results are quite impressive
Weaknesses:
-	Computational budget required is massive. The paper mentions model use from 128-256 TPUs, which severely limits reproducibility of results.
Comments/Questions:
-	Can you elaborate more on why BatchNorm statistics are computed across all devices as opposed to per-device? Was this crucial for best performance?
-	It is not clear if provided analysis for large-scale GANs apply for small-medium sized GANs. Providing such analysis would be also helpful for the community.
-	How do you see the impact of the suggested techniques on tackling harder data-modalities for GANs, e.g. text or sequential data in general?
Overall recommendation:
The paper is well written, ideas are well motivated/justified and results are very compelling. This is a good paper and I higly recommend acceptance.
We would like to thank Reviewer 1 for their review and constructive suggestions. Our responses inline:
>Can you elaborate more on why BatchNorm statistics are computed across all devices as opposed to per-device? Was this crucial for best performance?
-The primary reason is to ensure that training is invariant to the per-device batch size.
When scaling from resolution 128x128 to 256x256, we increase the number of devices but maintain the same overall batch size, reducing the per-device batch size. Cross-replica BatchNorm ensures that the smaller per-device batch size does not affect training. Switching to per-device BatchNorm at 128x128 results in a performance drop, albeit not a crippling one: for a model which would otherwise get an IS of 92 and an FID of 9.5, switching to per-device BatchNorm results in an IS of 78 and FID of 13.
>It is not clear if provided analysis for large-scale GANs apply for small-medium sized GANs. Providing such analysis would be also helpful for the community.
-The goal of this work is to explore GANs at large scale; the exploration of small or medium scale models would indeed be interesting for another study.
Having said that, we do evaluate BigGAN on conditional CIFAR-10 (mentioned briefly in Appendix C.2) and obtain an IS of 9.22 and an FID of 14.73 without truncation, which to our knowledge are better than any published results.
>How do you see the impact of the suggested techniques on tackling harder data-modalities for GANs, e.g. text or sequential data in general?
-Any of the proposed techniques could be applied to standard GANs for text or other sequential data in principle, but we have not experimented with these applications ourselves.
We would like to thank all reviewers for their reviews. We have uploaded a revised draft incorporating this feedback.
Specifically:
-We have added reference to the two papers mentioned in an earlier comment, as well as “The Unusual Effectiveness of Averaging in GAN Training,” Yazici et al., arXiv:1806.04498.
-Added Appendix G expanding on our discussion, and referenced this appendix at the end of section 4.2
-Fixed typos in captions
-Added a brief section on pitfalls of negative results in negative results appendix

Meta-review:
role: AC
data: metareview: The paper proposes a set of tricks leading to a new SOTA for sampling high resolution images. It is clearly written and the presented contribution will be of high interest for practitioners.

Decision: Accept (Oral)