Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a transformer based generative model which shows competitive results on multiple image generation benchmarks. It combines existing techniques of blocked and axial attention to reduce the computational complexity of self-attention mechanism. To further reduce the complexity, the proposed attention mechanism is only used at lower resolution blocks. The paper also employs technique similar to self-modulation to condition image generation process on the input latent code at all levels. 

main_review: The paper combines two existing techniques of blocked and axial attention in a novel way to create a convolutional free generator. The final proposed architecture shows competitive or better image generation compared to existing state of the art techniques on multiple datasets according to the FID metric.  

**Clarity**: The paper is easy to follow and well written. 

**Additional comments and questions**:  
* The paper proposes cross-attention which is similar to self-modulation technique. It would be good to see an ablation experiment that shows the performance of using self-modulation instead of cross-attention and its added advantage.   
* Will the proposed architecture also provide benefit in case of conditional image generation?   
* In Table 5, the paper only shows the result with cross-attention+blocked axial attention, It would be interesting to see the benefit when only blocked axial attention is employed without using cross-attention. 
* In Table 6, does having blocked axial attention for all layers (i.e. configuration [5,0]) leads to high memory requirement as it is not shown? 
* Though the paper proposes convolutional free generator and therefore has MLPs at the higher resolution blocks, but will having convolutional blocks instead of MLPs at later blocks provide any benefit? 
Does the ConvNetR1 baseline includes self-attention layer at one the generator blocks similar to self-attention GANs? 

**Related work missing**: One of the relevant related work "Self-supervised gans via auxiliary rotation loss." is missing and would be great to see a fair performance comparison with this as the baseline. This above work shows 23.6 FID metric on unconditional image generation on ImageNet 128x128 dataset which is better than the score reported by the proposed technique.

####################
Update: 

Thanks for addressing all the comments in the rebuttal. I am adjusting my score based on the answers. 

Additional comments: Are the results on ImageNet dataset with HiT-S architecture? If possible, it would be interesting to compare HiT-L (with similar #params as BigGANs) with BigGAN for ImageNet dataset in the final draft of paper.  
After reading other reviewer’s comment, I agree about including attention visualization in the paper. 


limitations_and_societal_impact: Yes, the paper touched upon the limitations and societal impact of the work in the conclusion.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: To address the quadratic complexity of the self-attention operation, this paper proposes a new Transformer-based generator for high-resolution image generation based on GANs, denoted as HiT. In the low-resolution stage, the authors propose a multi-axis blocked self-attention. In the high-resolution stage, they keep multi-layer perceptrons reminiscent of the implicit neural function. Extensive experiments demonstrate the effectiveness of the proposed method. 
main_review: 1. Some motivations should be highlighted. What are the issues of directly using ViT into GAN? In this paper, the authors do not directly apply ViT in GAN. However, there are some papers that propose ViT-based GAN methods. In the low-resolution stage, why replace standard global self-attention. In the high-resolution stage, why drop self-attention in the proposed method?

2. Some related work should be discussed in the paper. There are some papers [1, 2] that propose ViT-based GAN methods. Could you please discuss the difference between them? In addition, one recent paper [3] proposes new attention to generate high-resolution images. Could you please discuss it in the related work?

    [1] TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up, arxiv 2021

    [2] ViTGAN: Training GANs with Vision Transformers, arxiv 2021

    [3] Video Super-Resolution Transformer, arxiv 2021

3. The novelty of the proposed method should be highlighted.  The proposed method seems to combine the Nested Transformer and the Axial attention. The authors should highlight that such a combination is non-trivial. 

4. Some technical details are not clear. In Figure 1, what is block $B_i$? Are the two $B_0$ the same in the figure? In Line 154, why $b \times b \sim H/b \times W/b$? In Lines 157 and 174, could you please give a more detailed analysis of complexity?

5. The experiments can be improved. In Table 3, it would be better to compare with BigGANs [4].  In the ablation study,  the authors should compare with the Nested Transformer.

[4] Large Scale GAN Training for High Fidelity Natural Image Synthesis, ICLR 2019

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a new transformer-based generator for high-resolution image generation. It addresses the quadratic scaling problem of the attention operator with multi-axis blocked self-attention which considering the within blocks and across blocks attention. The proposed cross-attention for input and intermediate features is also novel. The evaluation is done in ImageNet and FFHQ datasets.  Promising results are provided. 
main_review: This is a well-written paper with carefully designed experiments. The motivation for utilizing the multi-axis blocked self-attention is reasonable. The proposed cross-attention for input and intermediate features is novel and interesting. The authors conduct various experiments to show the effectiveness of the proposed results, both quantitatively and qualitatively.

Line 12: "Hit has a linear computational complexity" is a bit overclaim. Since the full HiT model uses balancing attention between axes and its computational complexity is N*sqrt(N).

Why not apply bCR at 1024x1024 resolution? Is not using bCR the reason why HIT is worse than StyleGAN2 at the 1024x1024 resolution?
limitations_and_societal_impact: I think the authors have addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper has presented HiT, a Transformer-based generator for high-resolution image synthesis based on GANs. Instead of using full attention, this paper has proposed the multi-axis blocked self-attention, which captures local and global dependencies within non-overlapping image blocks in parallel. In addition, HiT introduces the cross-attention mechanism to integrate noise information into multi-stages of the generator. Extensive experiments demonstrate the superiority of the proposed method.
main_review: Originality:  This paper presents a new Transformer-based architecture for the generator in GANs. The authors introduce several new techniques to make it work for the Transformer-based generative model.

Quality: This submission is technically sound and most claims are well supported.

Clarity: This paper is clearly written and well organized. 

Significance: The experiment results are very important for exploring Transformer-based architecture for generative models.

Weakness:
1. I am curious about what is a good noise injection method in the Transformer-based generative models. Previous well-known methods StyleGAN and StyleGAN2 apply the AdaIN and modulated convolution for noise injection. HiT proposes the cross-attention for noise injection, the authors should conduct experiments to compare cross-attention with these methods, maybe AdaIN and modulated convolution also work for the Transformer-based generative model.

2. Since the Transformer-based architectures are more data-hungry, maybe the authors can conduct experiments on large-scale datasets: LSUN-cats and LSUN-cars and compare the performance with StyleGAN2 without using bCR. 

3. Transformer-based architectures have shown effectiveness in classification models, one of the most important reasons is the capability of capturing long-range dependencies. But in HiT, the self-attention module is only applied in low-resolution stages, make the generator fails to capture long-range dependencies in high-resolution stages. The authors may give some explanations for self-attention modules in low-resolution stages are enough in generative high-quality images.
limitations_and_societal_impact: The authors have addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper describes a purely attention-based GAN generator architecture where low-resolution deeper layers feature larger-scale (more distant) communication and higher-resolution layers synthesize locally using MLPs,  but still conditioned (indirectly) on the initial latent code.  State-of-the-art results are claimed for 128^2 unconditional ImageNet synthesis (but I believe this claim to be false, see below). In addition, results are shown for CelebA-HQ and FFHQ. While the architecture have some similarities to the near-concurrent work of Hudson and Zitnick (ICML 2021), lack of comparisons should not be held against this paper.

I remain unconvinced by the argumentation and how the results support them. Visually high-quality results are only shown for the simple face image datasets, for which attention mechanisms do not appear to be as necessary as for more complex ones that feature scene compositions. I feel I do not walk away feeling that I learned something.
main_review: 
The paper argues for an all-attention GAN generator, where the quadratic cost of self-attention is mitigated by restricting explicit long-range dependency modeling to coarse, deep layers.

My largest problem with the paper is that the quest for getting rid of convolutions is not really motivated. We all know metrics like FID and IS only tell a very coarse story, but the submission does not really analyze the properties of the resulting architecture. How does it build the pictures? How is its functionality different from existing main types of model architectures. For instance, compare to the attention map visualizations and composiotionality studies of Hudson and Zitnick that I found very illuminating.

The paper claims a record FID=31.87 and IS=21.32 for unconditional 128x128 ImageNet synthesis. I believe this to be incorrect: the BigBiGAN paper (Donahue and Simonyan, Large Scale Adversarial Representation Learning, NeurIPS 2019) presents an unconditional 128x128 model ("no \mathcal{E} (GAN)" in Table 1) with FID=30.91 and IS=23.56. The claim of "outperforming previous ConvNet-based models with a notable margin" (L235) does not appear to be justified.

Overall, I find image synthesis architectures fascinating, and that the quest for how to best do it is definitely unsolved. While it is interesting that the problem can be solved using all-attention architectures, I am not convinced by arguments that rely solely on metric numbers without analysis of the model's properties ("what makes it tick"). Moreover, the best results here are presented on aligned face datasets that are known to function well on existing architectures. I would be more convinced seeing high-quality results on more complex datasets, supported also by analysis other than FID and IS.

Smaller comments:

The dimensions have been chosen quite poorly in Fig. 2; I find it impossible to understand without consulting the text. The choice of b=2 does not match well with the choice of 4x4 inputs, as dimension=2 is now overloaded.

The authors do not present visualizations of the attention maps and how they relate to the synthesis results.

All synthesis results in Fig. 3 show serious aliasing artifacts. I suspect that the ImageNet dataset has been preprocessed incorrectly when downsizing to 128^2, a common error due to the poor design choices in current deep learning frameworks. (Note that e.g. TORCH.NN.FUNCTIONAL.INTERPOLATE does *not* perform correct filtering when downsampling; comparing its output to e.g. PIL.Image.resize with resample=PIL.Image.BICUBIC or Matlab's imresize will reveal clear differences.)
limitations_and_societal_impact: The required impact statement (end of Sec. 5) does not appear like a serious attempt.
review_title: null

Meta-review: The paper proposes a transformer-based generative model that shows better results on multiple standard benchmarks. The model includes multi-axis blocked self-attention at early stages and uses MLP for late stages. The reviewers appreciate the idea of a transformer-based generator, clear writing, and experimental results. There are also concerns about the STOA results on ImageNet (compared to BigBiGAN) and the motivation and advantage of transformer-based architecture compared to ConvNet. The rebuttal addressed some of the concerns. Overall, the significance of the work outweighs the limitations. I recommend accepting the work. The authors are encouraged to carefully examine the downsampling functions for 128x128 Imagenet (use antialias=True in TF, for example) and discuss the latest results (e.g., BigBiGAN) in ImageNet experiments. 
Decision: Accept (Poster)