Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper has presented TransGAN: a purely Transformer-based architecture for Generator and Discriminator in GANs. To solve the huge computation complexity of self-attention in high-resolution, this paper introduces grid-self-attention. Also, to solve the unstable training process of GANs, this paper introduces several techniques such as multi-scale Discriminator, and modified normalization. The experiments are conducted on various datasets.
main_review: Strength:
1. This paper tries to build a pure Transformer-based architecture for the Generator and Discriminator in GANs. previous works have demonstrated the effectiveness of Transformer-based architecture in classification, object detection, and segmentation tasks. So the idea of Transformer-based models in GANs is well motivated and intuitive.

2. Building Transformer-based architectures for Generator and Discriminator in GANs may have many challenges, like unstable training process, the huge computation complexity of self-attention in high resolutions. This paper proposes several methods to solve these problems.

Weakness:
1. The experimental results of this paper are weak and some comparisons are unfair obviously. For example, the comparison in Table 1 is unfair since the FID of StyleGAN v2 and StyleGAN v2+DiffAug on CIFAR-10 are calculated in 10k samples in the original paper[68]. However, in this paper, the FID is calculated with 50k samples. Applying different numbers of samples for calculating FID can yield different FID scores, more samples can yield better performance. Also, the original result of StyleGAN-v2 on LSUN-Church(256$\times$256) is $\textbf{3.86}$ for FID score, while TransGAN achieves 8.94, indicating the current results of TransGAN are significantly worse than StyleGAN2.

2. The strength of applying the self-attention module for Generator and Discriminator in GANs is the capability of capturing long-range dependencies, However, the grid self-attention module is also a local operator, which may show limitations in capturing long-range dependencies. So I am wondering whether the idea of applying the grid self-attention module may restrict the capability of the Generator?
limitations_and_societal_impact: The authors have addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work successfully takes an important step in bringing transformer architecture to the family of GAN models, so far dominated by convolutional architectures. Another contribution is the localised attention ('Grid Self-Attention') that addresses the problem of increased memory requirements of attention at higher-resolution feature map. The empirical evaluation shows competitive results on several benchmarks, including state-of-the-art scores on STL-10; however, it is missing an important benchmark - ImageNet. 

Despite such limitation and missing references and comparisons to some relevant prior work (which hopefully could be addressed in rebuttal), this work seems to meet the quality and novelty requirements for NeurIPS. I am willing to improve the rating further if my concerns are addressed.
main_review: The paper is generally clearly written, tackles a significant problem and proposes architectural novelties that will likely influence future works in generative modelling. Authors also made their code open, which will help reproducibility. I enjoyed reading a paper and think this work has potential to be among the most important ones submitted to NeurIPS this year; however, several issues should be addressed.

1. The Multi-Scale discriminator part is a bit obscure. 
- 'multi-scale discriminator is designed to take varying size of patches' - how do they vary, what's the distribution of patch size?
- 'split the input images into three different sequences' - how is it split?
Sequence sizes and Figure 2 do not reflect either of these statements, my best guess is that the whole image was resized and input at each of the stages, but it's neither varying nor split. If I am correct, values of P are not reported, so it is very difficult to understand what really happens with the input. 

2. In addition, Section 3.2 is missing relevant references: multi-scale discriminators have been in use for quite a while, both in image [4, 5, 6] and audio domain [7, 8], both with constant and random patch selection. Even though authors cite [5], they claim multi-scale discriminators to be their contribution 77: 'we also improve the discriminator with a multi-scale structure'. Number of stages and their composition is also not ablated.

3. The dataset choice seems somewhat peculiar given the claims that (i) the model design was focused on high resolutions and (ii) transformers are data-hungry, as compared to CNNs. Firstly, 'church' is neither most commonly used class out of LSUN categories, nor the one that contains most examples (relevance of 'smaller categories' notwithstanding, it's 126k vs 3M of popular 'bedrooms'). Secondly, the important ImageNet baseline is not addressed (which was studied both in unconditional [1] and conditional case [2, 3]). Authors explain the absence of conditional setup with simplicity argument, yet this is a significant limitation.

4. Overall, BigGAN [2] is almost not mentioned, even though it also applied self-attention.

5. Table 3. / Section 4.4
Authors discuss the issues with attention modules at higher-resolution maps, but do not provide FLOPs of various models, which would certainly help the reader have a clearer picture of those differences, and allow better comparison with CNN-based benchmarks. Parameter counts are also not provided.
The memory issues caused by some attention variants could potentially be solved e.g. by smaller batch sizes (or with micro-batching for fair comparison), or with ZeRO optimiser [9] (especially as all-experiments were carried in a multi-GPU setup).

Typos, wording, etc.:
- 63: 'more ambitious' - it's subjective and somewhat combative statement, suggest prior works were less ambitious. Maybe 'our goal required addressing additional new challenges'
- 203: non-overlapped->non-overlapping
- 248: describe->described, also e-8 -> e^{-8}
- 253: train->trains
- 320: fail-> fails

References:

[1] Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, Sylvain Gelly, 'High-Fidelity Image Generation With Fewer Labels', ICML, 2019

[2] Andrew Brock, Jeff Donahue, Karen Simonyan, 'Large Scale GAN Training for High Fidelity Natural Image Synthesis', 2018

[3] Prafulla Dhariwal, Alex Nichol, 'Diffusion Models Beat GANs on Image Synthesis', 2021

[4]  Emily L Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. 'Deep generative image models using a Laplacian pyramid of adversarial networks' NeurIPS, 2015.

[5] Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, 'Progressive growing of GANs for improved quality, stability, and variation' ICLR, 2018

[6] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris N Metaxas. Stackgan, 'Text to photo-realistic image synthesis with stacked generative adversarial networks' ICCV, 2017

[7] Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, Karen Simonyan 'High Fidelity Speech Synthesis with Adversarial Networks', 2019

[8] Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, Aaron Courville 'MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis', 2019

[9] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He, 'ZeRO: Memory Optimizations Toward Training Trillion Parameter Models'

limitations_and_societal_impact: The main limitation of this work is addressing only the unconditional generation problem, which has lower potential for applications. Conditional generation is important both inside and outside image domain and one of the major benchmark datasets, ImageNet, is not considered.

In broader impact, authors could acknowledge potential harmful uses of GANs, not only the positive ones.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
This work proposes TransGAN by replacing the CNN-based structure of both the generator and the discriminator in GANs with Transformer-based structure. To reduce the computing load, grid self-attention is proposed. Meanwhile, a relative position encoding scheme is introduced to the attention module to make the model better aware of the position information. Experimental results suggest that TransGAN achieves comparable performance to state-of-the-art CNN-based GANs.
main_review: 
- Studying the GAN architecture is encouraging.
- Grid self-attention makes it possible to apply transformer to *large-scale* GAN training.
limitations_and_societal_impact: - Why not reporting FID on CelebA-HQ 256 and Church 256?
- Since the improvement of TransGAN over StyleGAN2 is marginal (or even worse), what is the advantage of using Transformer instead of CNN? The authors claim that "convolutions have strong inductive bias" (Line 46), but do not show that why transformer can solve this problem?
- The interpolation is not that smooth as claimed by the authors. Please see the hair region of Fig. 1(b).
- An important ablation is missing. The title is "two" transformers make one strong GAN. What if we only use one transformer? The authors only compare TransGAN (G-trans, D-trans) with state-of-the-art GANs (G-CNN, D-CNN), without the comparison to (G-CNN, D-trans) and (G-trans and D-CNN). In this way, I am not sure on which part the transformer-based structure is helpful.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes to build a GAN model using only transformer blocks, and completely free of convolutions. Several architecture designs are proposed and a unique training recipe is also introduced for stably training the proposed TransGAN. TransGAN  can scale up to high resolution (e.g. 256 × 256), and delivers state-of-the-art results on multiple benchmarks. 
main_review: Strengths:
+ TransGAN is the first attempt to use only transformers to build a strong GAN. While the backbone’s building blocks is not such novel, the target task and the way to tackle it is very new and deserves to be encouraged.
+ The authors have extensively introduce the architectures of the discriminator, the self-attention mechanism and the generator. They have revealed useful design insights for transformer-based image generation. 
+ The authors investigate many techniques to stabilize the network training and improve the performance, including data augmentation, relative position encoding, and simpler normalization borrowed from [73, 16]. 
+ The proposed method shows strong empirical results as well as scalability. TransGAN achieves SOTA results on STL-10 and CelebA-128, is close to SOTA and outperforms StyleGAN-v2 (even with DiffAug) on CIFAR-10, and remains very competitive on CelebA-256 and LSUN-Church 256. 
+ The visual quality of generated results at 256 level is impressive and can rival the best CNN GANs. This is one big step forward towards justifying transformer-based generation.
+ Ablation experiments on each architectures or training algorithm components are very thorough and convincing.
+ Figure 5 shows intermediate generation results to compare TransGAN behavior with CNN-based GAN, as well as interpolation on latent space. They looks very interesting: TransGAN learns to generate images with global awareness, totally different from the local refinement regime of CNN GANs; meanwhile, the latent space seems the same smooth.
+ The paper is overall quite well-written and easy to follow.

Weaknesses:
- Corresponding to the first strength, I have to point out that no building block in TransGAN is particularly novel. However, I actually feel this fine and acceptable, as it demonstrates the value of putting each moving part together in one right pipeline.
- The description of Multi-scale Discriminator is complicated and I’m not sure if I followed well. Is this anything similar to token-to-token ViT, or Pyramid ViT? Since the idea of progressively changing tokens has been exploited by several ViT works, the authors need to describe their context better.
- Also for the grid self-attention, please discuss its relationship to Swin Transformer and others window-based ViTs.
- Why WGAN-GP loss is chosen? Is gradient penalty your necessary training component? Can other losses (e.g., Hinge, least-square) work with TransGAN?
- Please report TransGAN model size and/or memory usage.
- Minor comments: This paper makes big progress to high resolution generation task, but there is still a gap behind the highest-resolution that CNN GANs can now scale up to, e.g. 1024. While it can reasonably go to the authors’ future work, some discussions to shed light on how they plan to do so would be welcomed.

limitations_and_societal_impact: The social impact has been discussed by the authors at the end of the paper.
review_title: null

Meta-review: This submission demonstrates that GANs with transformer-based architectures can learn to generate high-quality and high-resolution images, achieving strong results competitive with (or in some cases better than) convolution-based GANs on a variety of benchmark datasets.

Reviewers were generally in agreement that the paper is well-executed and does a nice job of tackling the scale-related challenges that arise from applying transformers to high resolution images. Although not all results are state-of-the-art, the paper makes use of very non-traditional architectures and as such doesn’t benefit as much from the tricks of the trade that have been developed and refined over several years of progress in convolutional GAN models, so this should not preclude acceptance. Another criticism that came up in review was the lack of results in the class-conditional setting (e.g. ImageNet). I would agree with the authors that it’s fair to save this for future work especially given the thoroughness of the evaluation in unconditional settings. Reviewers pointed out certain aspects of the method that weren’t clear and/or well-ablated in the original submission (e.g. what happens if transformers are used for only G and/or only D?) and the authors provided thorough responses including additional results/ablations. These results and clarifications should of course be included in the camera-ready version of the paper.

Given its strong image generation results using non-traditional architectures of current interest to much of the NeurIPS audience (with findings/ideas potentially useful beyond image generation), I recommend accepting the submission.
Decision: Accept (Poster)