Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to adopt the StyleGAN architecture (i.e. individual styles that affect image generation) to a transformer based model for image generation. The model generates individual image patches which are modulated by style tokens through an attention mechanism. The model is evaluated on the FFHQ and LSUN Church datasets and performs similarly well or better than StyleGAN2 based on FID, precision, and recall.
main_review: The paper proposes to model images as a sequence of image patches. These image patches start out as learned constants (similarly to StyleGAN) and are then modulated via style tokens. The interaction between content and style tokens is modeled via an attention mechanism (between content and style) such that this approach also scales to high-resolution images (1024px).

It would be helpful to be a bit more clear about the dimensions of the individual tokens, i.e. what values do you use for mn, n, p, and d in Fig 2. Also, how many layers do you use? Since you compare to StyleGAN2 it would also be helpful to mention training time and other requirements (e.g. memory) compared to StyleGAN2, as well as how long it takes to generate a new image at different resolutions.

Since you also use stlye mixing I would like to see some qualitative results of that, i.e. what do generated images look like that are generated from two different styles? Can we get similar behavior as in StyleGAN where we can control coarse layout and texture details at different layers? Overall I would be happy to see a more detailed analysis of the "style space" in this approach, given that this is arguably one of the biggest advantages of the original StyleGAN architecture.

Since you use different styles to directly modulate individual content tokens (i.e. image patches): can you get localized behaviors for the style tokens, e.g. mix two styles "spatially" where e.g. one half of the image is controlled by one style and the other half by another style? Or do reference based image translation, e.g. replace the hair in a given image with the hair of another image?
limitations_and_societal_impact: Limitations and societal impact are not discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes TokenGAN for unconditional image synthesis. The token-based generator takes the learned content tokens and style tokens from the latent space as input, and output a series of style-modulated content tokens, which are then concatenated and reshaped to get the output image. Cross-attention is used to assign the styles to different content tokens. The model adopts the style-based generator idea from StyleGAN and introduces the recent token-based structures and transformers to the traditional style-based generator. Experiments are conducted on FFHQ and LSUN CHURCH datasets and the quantitative results are slightly better than StyleGAN2.
main_review: Strengths:
1. The proposed approach is well-motivated. It takes advantage of the latent code mapping and style-based modulation of StyleGAN, and further improves it by replacing the convolution-based architecture with token-based cross-attention.
2. By using a series of content tokens and style tokens, each patch on the image can adaptively attend to content-aware styles. So the structure can control the styles of different patches, in contrast to a single global style code in StyleGAN.

Weakness:
1. The quantitative results (FID, precision, and recall) are only slightly better than StyleGAN2. The qualitative results only show the generated images of the proposed TokenGAN, but does not show the comparison with StyleGAN2. From the generated images shown on the paper, I did not observe very significant improvements from StyleGAN2.
2. Some ablation studies are missing: (1) the number of content token (2) Mixing different style tokens (used in this paper) v.s. mixing styles at different layers (used in StyleGAN) for style mixing. (3) Style modulation by amplifying each channel (in this paper) v.s. by adaptive normalization (in StyleGAN).
3. For the discussion on image inversion, evaluation metrics such as LPIPS distance should be reported and compared with StyleGAN2.
4. I am curious about the complexity and efficiency of the proposed model compared with StyleGAN2.
5. It would be good to visualize how the style tokens can impact different regions (impact tokens) of the images. The authors tried to visualize the attention maps in Fig. 3, but I also expect other visualizations or evaluations illustrating how the multiple style tokens and cross-attention benefit the generated image quality and attribute disentanglement. For example, a visualization of style mixing might indicate how changing one style token (and keeping other style tokens unchanged) can affect the synthesized images.
6. Lines 174 - 178 on implementation details are not clear to me. Could the authors explain more on how to incorporate the skip connections and how to do upsampling in the generator? Are there interactions (such as self-attention) between the content tokens? If not, how does the generator deal with global consistency and context information?
7. This paper lacks discussion and comparison with a similar work [21] which also uses transformer for GANs. 
limitations_and_societal_impact: The authors do not address the limitations and potential negative societal impact of their work. It would be good to add some discussions about  the potential limitations and failure cases of this work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, a transformer-based modification of the generator in the ubiquitous StyleGAN model is proposed (named _TokenGAN_). More specifically, the entire generator is replaced by a convolution-free architecture where style-content modulation is now modeled using cross-attention layers instead of AdaIN. Similar to the original StyleGAN, this should allow for decoupled control of style and content at different granularities. The paper shows that the proposed approach achieves FID scores comparable to or better than StyleGAN2 for unconditional image synthesis on FFHQ and LSUN-Churches.
main_review: __Strengths:__

The proposed approach is a simple, yet working modification of the generator of StyleGAN. The use of cross-attention to model style-content interactions is a good, novel idea. The attention maps depicted in Figure 3 nicely demonstrate that the generator trained on FFHQ attends to human-interpretable concepts. The paper thoroughly reports error bars / standard deviations where applicable.

__Weaknesses:__

Although the paper states the "fine-grained controllability of visual tokens" (l.60), no experiments are made to explicitly verify this claim or how controllability is increased over the StyleGAN baseline.
Furthermore, the abstract states that the approach is "able to synthesize high-fidelity images [...], dispensing with convolutions entirely" (l.15-16). However, as the discriminator is the same as for StyleGAN2, this statement does only hold for the architecture of the generator, but not the complete model. Following on from this, it would be interesting to see if it is also possible to replace the discriminator with a pure Transformer architecture.

Given that the proposed architecture is mostly transformer-based, are there any notable differences in applicability to other datasets? For example, how does the model handle (very) small datasets or large-scale, complex datasets? Is overfitting a problem? If so, how is this being adressed? Here, plotting nearest neighbors (e.g. in VGG space) would be an interesting analysis option.

Finally, the paper does not mention on which patch-size $p$ the models are trained. How do different $p$ affect the results? Furthermore, how do different $m$ affect the model? I can only find a study regarding $n$. Also: Is $\mathbf{K}$ shared across different style modulation layers?

__Clarity:__
The presentation of the cross-attention layer and style/content tokens is clear; however, the "upsampling procedure" (l. 20-21 in the supplementary) remains unclear to me and should probably be included in a Figure (e.g. Fig. 2)

__Significance/Originality:__
Replacing the convolutions in the generator of StyleGAN has the potential to allow applications to more complex datasets than the often used FFHQ or single-scene LSUN variants. Unfortunately, this remains unexplored in this work, but might be of interest to the community.

__Justification of Rating:__
I think the use of cross-attention as a modulation level for style content is an interesting idea and is sufficiently justified in the paper. However, apart from the visualization of attention weights in Figure 3, the paper does not show how the proposed method differs in applicability compared to StyleGAN(2) and what the impact is of switching to a transfomer architecture. For example, how do the suggested modifications alter convergence speed? What's the effect when trained on smaller/larger datasets? Is overfitting a larger/smaller problem than in the underlying work? Including the weaknesses mentioned above, I currently think that the paper (in its current state) would be a good fit for a workshop, but needs to be expanded (see the comments above) to be published as a full conference paper. However, I am curious about the authors' response and the opinions of the other reviewers, and I am happy to be proven wrong.

_Minor_:
- l. 53: i -> in
- l. 63: decompose -> decomposes
- l. 69: missing space after "TokenGAN"
- l. 135: missing space after "TokenGAN"
- l. 186: adopts -> adopted
- l. 268: view -> viewing
- l. 278: demonstrate -> demonstrated
- Fig. 2: "concate" -> "concatenate"/"concat"



__[UPDATE] Post-Rebuttal__

After re-reading the paper and rebuttal, I think my original rating was too harsh, and I am therefore raising my score to 5. However, I am still not convinced that the proposed approach offers many advantages over the convolution-based StyleGAN(2) architecture, i.e., the gains presented here are rather marginal.  I think the work would benefit from a more fundamental analysis of the differences (training dynamics, applicability) between TokenGAN and StyleGAN, where then an improvement in the typical metrics (e.g. FID) would be nice but not mandatory.
limitations_and_societal_impact: Not discussed in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates the application of transformer for image generation by replacing the convolutional layer with MLP(including attention). The motivation provided is the limitation of the style-based generator (e.g.,  StyleGAN), which leverages the single style over multiple image regions, allowing a less entangled representation learning in W and visible artifacts in generated images.  To combat this shortcoming, authors introduce transformer-liked architecture, and successfully generate high-quality images (1024*1024) without touching any convolutional layers in generator.
main_review: Pros:
Authors firstly exploit the image generation with transformer to synthesize high-fidelity images. The transformer-liked architecture has been used a series of computer vision tasks,  in this paper authors attempt to introduce it into image generation tasks. To achieve better performance, authors devise the model based on the StyleGANv2, aiming to use the well-devised framework and techniques.

One of the interesting parts is to generate the high-quality image without using any convolutional layers, which exhibits the power of the transferor-liked architecture.  Comparing StyleGANv2, the proposed method is able to obtain better performance in some metrics.

The proposed method can be further used for other tasks (image editing), which almost keep the property of the StyleGAN.
limitations_and_societal_impact: Cons:
The proposed method maybe not corresponding to the provided motivation(the limitation of the stylegan), since the proposed method is based on the styleganv2, and leverages all the techniques of StyleGANv2. I am wondering the motivation of the proposed method. In fact, I guess the motivation is to use firstly transformer to generate high-fidelity image.

I am hoping that more analysis or insights could be exhibited in this paper, since the transformer is firstly used in generation model. However, all shown and reported results are little boring, although it looks better than styleganv2. In fact, this paper is based styleganv2.

The transformer is only exploited in generator instead of discriminator. The related work has already shown that the transformer obtains outstanding performance in classification task, why in this paper authors do not use it in discriminator.

Since the learnable position encoding is intruded in content tokens, I am wondering why the position encoding do not be injected in style token? Besides, I fail to find the code presenting the position encoding when checking the provided code. Please correct me if I am wrong.

The detail for the value of m and k are loss, even in supplementary material.  Is the model performance influenced by these selected values?
review_title: null

Meta-review: The paper proposed a transformer-based generator architecture for GAN and showed it achieved comparable image synthesis results with the StyleGAN2 baseline. It initially received a mixed rating, with two reviewers rated it above the bar and two below the bar. The rebuttal addressed some of the concerns. Both of the reviewers that were originally leaning toward a rejection recommendation upgraded the score. Overall, the meta-reviewer agreed that the proposed method has its merit. While, given the success of transformers for various vision tasks, it was kind of expected that one could achieve comparable results to CNNs with a transformer-based architecture for a GAN generator, it is still nice to see such a design being revealed and shown comparable performance. While it is unfortunate that the discriminator is still based on CNNs and the additional capability achieved by the architecture is underwhelming, the meta-reviewer thought the paper still presents a good first step. The authors are encouraged to incorporate the reviewer feedback in the final version.
Decision: Accept (Poster)