Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers
Authors: Yanhong Zeng
Huan Yang
Hongyang Chao
Jianbo Wang
Jianlong Fu
Venue: NeurIPS 2021 Poster (2021)
Keywords: GAN
Transformer
Image Synthesis
Visual Token

Abstract: We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. Different from existing paradigms that directly synthesize a full image from a single input (e.g., a latent code), the new formulation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis. Specifically, it takes as input a sequence of latent tokens to predict the visual tokens for synthesizing an image. Under this perspective, we propose a token-based generator (i.e., TokenGAN). Particularly, the TokenGAN inputs two semantically different visual tokens, i.e., the learned constant content tokens and the style tokens from the latent space. Given a sequence of style tokens, the TokenGAN is able to control the image synthesis by assigning the styles to the content tokens by attention mechanism with a Transformer. We conduct extensive experiments and show that the proposed TokenGAN has achieved state-of-the-art results on several widely-used image synthesis benchmarks, including FFHQ and LSUN CHURCH with different resolutions. In particular, the generator is able to synthesize high-fidelity images with (1024x1024) size, dispensing with convolutions entirely.

Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers
Yanhong Zeng" 1.2 Huan Yang² Hongvang Chao Jianbo Wang* Jianlong Fu? Schooo of Computer Science and Engineering, Sun Yat-sen University 2 Key Laboratory of Machine Intelligence and Advanced Computing. Ministry of Education, China MMicrosoft Research Asia *University of Tokyo
Abstract
We present new perspective of achieving image synthesis by viewing this task as a visual token generation problem. Different from existing paradigms that directly synthesize full image from a single input (e.g., latent code), the new formulation enables flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis. Specifically, it takes as input sequence of latent tokens to predict the visual tokens for synthesizing an image. Under this perspective, we propose a token-based generator TokeGGANN) Particularly. the TokenGAN inputs two semantically different visual tokens, i.e., the learned constant content tokens and the style tokens from the latent space. Given sequence of style tokens, the TokenGAN is able to control the image synthesis by assigning the styles to the content tokens by attention mechanism with Transformer. We conduct extensive experiments and show that the proposed TokenGAN has achieved state-of-the-an results on several widely- used image synthesis benchmarks, including FFHQ and LSUN CHURCH with different resolutions. In particular, the generator is able to synthesize high-fidelity images with 1024 x 1024 size, dispensing with convolutions entirely.
1
Introduction
Unconditional image synthesis generates images from latent codes by adversarial training 10 16
19
25
27 33 50
Recent advances have been achieved by a style-based generator architecture terms
of both the visual quality and resolution of generated images (26 27 28 37 52). In particular, the
style-based generator has been widely used many other generative tasks, including facial editing
99 40), style transfer [ 38) image super-resolution
17 1 and image inpainting [2] (2)..
The key to the success of the style-based generator lies the learning of the style control based on the intermediate latent space W [27] [28]. Instead of feeding the input latent code E Z through the input layer only a), the style generator maps the input z to an intermediate latent space w E W, which then controls the "style" of the image at each layer via adaptive instance normalization (AdaIN (2I) (Figure b). It has been demonstrated that such design allows a less entangled representation learning in W. leading to better generative image modeling [2 23 27 40)
Despite the promising results, the style-based generator suffer from the style control via AdaIN operation 228 [2).. Specifically, the style control is content-independer original information of features by normalization and assigns new styles decided by the latent codes regardless of the image/feature content. Besides, the style code w affects the entire image by scaling and biasing complete feature maps with single value via AdaIN operation 35| 54] Such an
It "washes away" the
imposed single style over multiple image regions can inevitably result in entangled representation of
"hii work was done while Yanhong Zeng was research intern at Microsoft Research Asia.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

M
w
Z
M
G
G
CG Cm
(
(a) Traditional Generator
(b) Style- bayle-based Generator
(c) Token- bassen-baeeddeeee Generator
Overview different generators. (a) A traditional generator [16] feeds single latent z as input to control the image synthesis. (b) A style- generator [27] maps to an intermediate latent space w E W to control the styles the content e via AdaIN 21.. (c) Our token-based generator starts from sequence of content tokens {c1,"
Figure 1:
) and controls each content token with set of style tokens {si,..., {n} E S by attention mechanism with a visual Transformer. M denotes the
mapping network and G denotes the generator network.
different image variations (e.g., hairstyle, facial expression) 28 44 52 modeling
These limitations for image lead to visible artifacts in the synthesized results (e.g., droplet artifacts [27
To get rid of the issues caused by StyleGAN's style modeling, we introduce new perspective visual token generation problem. The visual token is popular
that views image synthesis representation of an image patch with a predefined size and position [6 [3); and has shown an impressive superiority in various tasks with the development of Transformer models 6
46).
Inspired by the appealing property of the token-based representation, we propose to achieve image synthesis by visual token generation. Specifically, it takes input sequence of latent tokens to predict the visual tokens of an image. Such a token-basee representation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis.
Under this new paradigm, we design token-based generator, i.e., TokenGAN, for the visual token generation problem. Specifically, we consider two different types of input tokens in the generator, i... the content tokens and the style tokens. The content tokens are learned as the constant input in the generator network and the style tokens are projected from a learned intermediate latent space (Fi Given sequence of style tokens, the TokenGAN learns to control the visual token generation by rendering each content token with related style tokens according to their semantics. In particular, since the Transformer has been verified to be effective in sequence modeling in broad range tasks 4 we adopt a generator network architecture from a visual Transformer to model the relations between the content tokens and the style tokens. Through such content-dependent style modeling by the attention mechanism Transformer, the TokenGAN able to achieve content-aware and fine-graine mne-ggaieed style learning for image synthesis.
We conduct both guantitative comparisons and qualitative analysis on several unconditional image generation benchmarks. The results show that the token-based generator has achieved comparable results to the state-of ofr in image synthesis. We summarize our contributions as below:
We propose new perspective of achieving image synthesis by visual token generation. Such a token- based representation enables flexible local manipulation for different image regions, leading
better image modeling.
We propose a token-based generator (i.e., TokenGAN) for the visual token generation. Specifically, the TokenGAA introduces the style tokens and the content tokens. It adopts Transformer-basse network for content-aware style modeling.
We show extensive experiments (quantitative and qualitative comparisons, study style editing, image inversion and image interpolation) to verify the effectiveness the token- based generator. Specifically, the token-based generator is able to synthesize high-fidelity 1024 x 1024 images without any convolutions in the generator.
2 Related works
2.1
Style-based generator
The distinguishing feature of the style-based generator its unconventional generator architecture
26 27 28 52]. Typically, the style-based generator consists of mapping network ff and synthesis

---

network The mapping network is used to transform the input latent code to an intermediate latent code E W for learning a less entangled latent space. The synthesis network follows a progressive growing design that is able to first output low-resolution images that are not affected significantly by high-resolution layers 25 27 Such an architecture design can lead to an automatic, unsupervised separation of high-level attributes in different layers. For example, when trained on human faces, the low-resolution layers control coarse styles (e.g., pose, identity) of images and high-resolution layers control fine styles (e.g., micro-structuree color scheme) [25]
To control the attributes of the image, the style-based generator produces styles from the intermediate latent code w by affine transforms, which then control each layer of the synthesis network via adaptive instance normalization (AdaIN) [2] 35).
The AdaIN operation removes styles from previous layers by normalizing each feature map to zero mean and umit deviation, and it assigns the new styles by scaling and biasing the complete normalized feature maps. However, it has been demonstrated such an AdalN operation can destroy the magnitude information of features and thus results in the well-known droplet artifacts
StyleGAN can be limited due to the complex semantic components in each feature map [28 (2). In this paper, explore new architecture by flattening the image as a sequence of tokens, which enables fine-grained control of the image by assigning token-wil wise semantic-aware styles based on the attention mechanism.
28 Besides, the spale-specific style disentanglement
2.2 Transformer in vision
The Transformer typically takes input sequence of vectors, called tokens 4 The Transformer is built for sequence modeling solely on attention mechanisms over the tokens, dispensing with recurrence and convolutions entirely (5]
Due to its great success in the field of natural language processing, an increasing number of works attempt to extend Transformer for computer vision tasks 6 tion by pre-training a sequence Transformer with the task of auto-regressive next pixel prediction and masked pixel prediction [8]. Dosoyvitskiy et al. propose split an image into patches and feed these patches into a standard Transformer (ViT) for image classification [13] They show that a ViT with large-scale training can trump CNNs equipped with inductive bias. DETR a seminal work that views object detection as a direct set prediction problem, eliminating the need for hand-crafted components and achieving impressive performance by Transformer (6. To mitigate the issue of high computation complexity associated with long sequences caused by high-resolution images, a branch of works explores lightweight Transformer (e.g.., Deformable DETR) for vision tasks [55]
43 30)
g 36 55
For example, iGPT shows promising results in image classifica-
36 Parmar al. cast image generation an autoregressive sequence generation problem and propose Image Transformer with a local self-attention mechanism [36]. However, Image Transformer can suffer from its quadratic computation cost and its long inference time due to auto-regressive prediction. Most existing works adopt a hybrid CNN-Transformer architecture, which consists of CNN head for feature extraction, a Transformer encoder-decoder backbone, and a CNN tail for feature decoding (77 46, 48). For example, IPT fully utilizes the Transformer architecture by large-scale pre- pre-training and achieves promising results in several image restoration tasks 77. In parallel work, Hudson et al. introduce bipartite structure to maintain computation of linear efficiency for long-range interactions across the image based on convolution backbone [22] Our lokenbbaser generator inherits the network architecture from Transformer without any convolution layers and is able to yield surprising promising results for high-resolution image synthesis.
Transformer is also attracting increasing attention in low-level vision tasks 14
3 Approach
In this section, we introduce the details of the proposed token-based generator (TokenGAN). As depicted in Figure the TokenGAN takes as input two kinds of tokens to generate the visual tokens of images by a Transformer. We introduce the input tokens in Section 3.1 and the token-based generator architecture in Section 3.2 following the overall optimization in Section 3.3
3.1 Visual tokens
It's intuitive that the visual tokens of an image contain the information of both content (i.e., semantics) and related styles. In our token-based generator, choose to separate these two kinds of information
3

---

Content Tokens
(OOOO
(0i...,Com
Style Modulation
Normi
Gm. Learnable K
000
Style Normalization
Style Modulation
(OOD
Normalization
mix (st
x
COO
Modulation
Position Encoding
(OOD
(+) 1
Token to Image
P Pa)
Style Normalization
Style Tokens
Concat
Style Modulation
Reshape
Mapping
25
Network
Token Image
(a) Visual Tokens
(b) Token- based Generalor with T Transformers
Figure 2: The overview of the TokenGAN for the visual token generation task for image synthesis. The TokenGAN takes as input two kinds of visual tokens, i.e., the style tokens and the content tokens, generate visual tokens of an image. Specifically, TokenGAN learns to render each content token by attended style tokens with Transformer, leading to content-aware and fin-grained style control.
by a sequence of content tokens and sequence of style tokens, so that the interactions between them can be modeled and controlled for the image synthesis. We introduce the details of each as below.
Content tokens. As depicted in Figure a the TokenGAN starts from the sequence learned
constant content tokens с E (pmxpp2xx xd) through the input layer. Specifically, d is the dimension in terms of channels, p X p the patch size of the content token, and m is the length of the content token sequence. To maintain the position information of each token, we add position encodings
Rmxx(p² xd) to each content token following a commonly used paradigm [ 13 For simplicity, we rewrite the notation for the content tokens as (ci, Cm where C₁ RR
Style tokens. Given latent code from the input latent space 2, the mapping network adopts several MLPs to map the input z to set of different style tokens (si..... Sn E S, where s; E Rd As shown in Figure b. the style tokens are paired with set of learnable semantic embedding key-value structure each style modulation layer of the TokenGAN. With the Transformer modeling, all the content tokens will match with the semantic embedding and then fetch the new style from the style tokens based on the matching results. The fetched new styles are used to control the values of the content tokens, which are finally decoded images. Such token-wise style control enables content-aware and fine-grained style learning for image synthesis.
3.2 Token- based generator
As shown in Figure 2 the TokenGAN consists multiple layers of style blocks and each style block consists of style normalization layer and style modulation layer. Specifically, the style normalization layer removes the styles from previous layers by relieving the dependence on the original statistics of input token features. At the same time, the style modulation layer assigns new styles from the current layer to the content tokens. particular, the new styles are calculated based on the pairwise interactions between the style tokens and the content tokens by cross-atted cross-attention mechanism. We introduce more details about the style normalization, content-aware style modeling, and style modulation in our generator as below.
Style normalization.
We apply style normalization on the input content tokens to remove the styles from previous layers, which is denoted as:
Norm(c;) (c. p(c.)) o(c.), i= 1, ......
(I)

---

nose
eyes mouth
jaw
hair
background
Visualization of the attention maps obtained by Eq. (2). Each column highlights the image regions that are affected most by the corresponding content-aware style token. It shows that TokenGAN is able to attend to meaningful semantics for different persons in an unsupervised way. The tags are associated by human based on the attention responses for clear presentation.
Figure 3:
where each c E Rd denotes a content token, denotes the dimension of each content token, and m is the length of the token sequence. Specifically, the style-based generator typically removes the styles from previous layers by Instance Normalization [41], which may destroy the information conveyed by the magnitude of the feature map relative each other [27 28 41). To avoid the above issue, we adopt LayerNorm relative magnitude following previous works 22 29 43
[43] to relive the style dependence while maintaining the information of the
Content-aware style modeling. After style normalization, the TokenGAN calculates new styles to control the input content tokens. As shown in the style modulation in Figure 2b, the style tokens paired with set learnable semantic embedding as a key-value structure. Given the content tokens {c Cm } from the input and the style tokens from the mapping network (s S, the token-wise new styles are calculated by attention mechanism:
CK
S' Attention K, softmar
Vd SS.
(2)
where S' E mmxd denotes new styles for all the content tokens, the input content tokens are packed together into a matrix C md similarly with the style tokens as matrix E RR and the matrix K Rxx for the learnable semantic embedding in where each row vector indicates learned semantic. We omit the linear projection in the standard attention calculation for simplicity [43].
Through such a cross-attention mechanism, the new token-wise styles are fetched for style modulation based on the matching results. Particularly, tokens with similar semantics will have similar styles (e.g., the same color for the two eyes). We visualize the attended regions for different key-value pairs (i.e., semantic-style embedding) in Figure 3 and we can find that learned embedding K is able to attend to meaningful image regions.
Style modulation. Different from the traditional Transformer that adds the attention results back to the input features as residual features, our attention results are used as new styles by amplifying each channel of the content tokens:
C' =
(3)
where each row vector in C E md indicates content token, and indicates clement-wise multiplication. Such a style modulation affects the operation subsequent embedding layers (which implemented by fully-connected layers) and thus control the style of generated images.
Implementation details. We translate the generated visual tokens the image by concatenation and reshaping. In practice, we follow StyleGAN2 and adopt ssip-generator architecture in Token- GAN. Such a skip generator generates images each layer at different resolutions (e.g., from 42 to g2) and progressively upsamples and sums the images from the previous layer by skip connections the next layer 28 25). After progressive summing, the generator takes the output in the last layer as the final results.
5

---

3.3 Overall optimization
Style mixing. To further encourage the styles to localize, we employ the mixing regularization technique during training [27]. To be specific, given percentage of images are generated using two random latent codes. We run two latent codes 21, Ze through the mapping network, and have the corresponding {s}. ... s.). ... ). We randomly choose an inject point to mix the styles, so that a part of the final style tokens from 21 and another part from 22. This regularization technique prevents the network from assuming that set of style tokens are correlated. A similar strategy is
adopted in terms of different layers following the style-based generator [77]
Optimization objectives.
We denote the generated image output by the token-based generator as:
G(z) (f( C),
(4)
where z E 2 ~ (0 1) indicates the input latent code, C = [CI,-... Cm denotes the sequence of learned constant content tokens through the input layer, and f. 9. G indicate the mapping network, the generator network and the token-based generator, respectively. Our token-based generator follows the same optimization objectives used by the style-based generator [27 28), i.c., an non non-saturating logistic adversarial loss with RI regularization. To be specific, the generator loss is:
LG (o.1)) + (=p((D(G(Z)))))
(5)
where D is the discriminator and the discriminator loss is
LD EE- Pallog(1 exp(-1((x)) + Ee- + exp(D(((())) + Rr
(6)
where x ~ Pa denotes images from the real data, and R1 (v) is the regularization term calculated by:
R1 (vb)
7 Fx 2
[ [VD
Specifically, the RS regularization term
32 is computed less frequently than the main loss function
following the commonly used lazy regularization strategy, thus greatly diminishing the computational
cost and the overall memory usage 27 28 52
4 Experiments
4.1 Experiment Setup
Dataset To evaluate the token-based generator and make fair comparisons with the SOTA approach (i.e.. the style-based generator (28)) ). we conduct experiments on the most commonly-used public research datasets, i.c., FFickr-Faces-HH (FFHQ) (27] and Large-scale Scene Understanding (LSUN) [47]. Specifically, FFHQ consists of 70,000 high-quality images of human face, which is able to evaluate the model's s ability synthesize high-frequency details. We conduct experiments FFHQ with both 256 256 and 1024 x 1024 image size following previous works [15] 222. Besides, we choose the church set of LSUN 47 to evaluate the synthesis in terms of complex scenes.
Evaluation. In this section, we conduct both quantitative and qualitative experiments on the token- based generator. Specifically, we report quantitative results by three commonly-used metrics, i.c., Fréchet Inception Distance (FID) [1), Precision and Recall [29 39]. We use FID as it has been widely used in many generation tasks the image quality assessment, we use the metrics of Precision and Recall to evaluate the distribution learned by GAN. Specifically, the Precision metric intuitively measures the quality of samples from the generated images, while recall measures the proportion of real images that is covered by the learned distribution by GAN 39
effective perceptual metric 4 26 27). In addition to
We use 8 NVIDIA V10 GPUs for training each model. We build upon the public Pytorch implementation of StyleGAN? 28). For fair comparisons, we inherit all of the training details and parameter settings from the configuration setup. All models are trained and report the best results under the same training iterations. For each iteration, we the size of mini-batch as 32. We use the Adam solver 3 with the same momentum parameters 31 = 0, B2 = 0.99 to train both the generator and discriminator. We apply R₁ regularization for every 16 iterations.
Training details

---

Figure 4: Uncurated results for FFHQ- 1024 (27 and LSUN CHURCH 47 The results show that token-based generator is able to synthesize both high-frequency details (e.g., hair and beard on the human face) and plausible structures for complex scenes (e.g., the church).
Table 1:
Quantitative comparisons with the state-of-the-ar model (i.e., StyleGAN2 (28) FFHQ- 256 (27), FFHQQ1024(27] and LSUN CHURCH [47]. We compute each metric 10 times with different random seeds and report their average. The result shows that the token-based generator achieves comparable performance with the SOTA. 1 the higher, the better. J the lower, the better.
Model
Metrics
FFHQ-256 (27
FFHQ-1024 .77
LSUN CHURCH 47
FID4
6.09 = 0.051
5.20 + 0.049
5.88 + 0.043
StyleGAN2 27
Precesion
0.643 + 0.002
0.653 = 0.002
0.587 = .002
Recall f
0.401 + 0.002
0.411 = 0.002
0.358 + 0.002
FID
5.41 + 0.050
5.21 + 0.032
5.56 + 0.037
TokenGAN(Ours)
Precesion
0.660 + 0.002
0.651 0.001
0.577 + 0.002
Recallf
0.447 + 0.003
0.442 = 0.004
0.376 + 0.002
4.2 Unconditional image synthesis
Quantitative comparisons. We report quantitative comparisons with StyleGAN2 28 due to its state-of-the-art performance. For fair comparisons, we report the results with the same training iterations FFHQ-256 (27). FFHQ-1024 (27) and LSUN CHURCH (47 by three objective metrics, i.e, FID, Precision and Recall Table [ The results show that the token-based generator has achieved the state-of-the-ar results terms of both perceived quality and distribution modeling.
Qualitative results. To demonstrate the image quality of the synthesis results for the token-based generator, we randomly sample noise z E ~N(00, 1) to generate images for high-resolution images (i.e., with resolution 1024 1024) of human face and the images of complex church scenes. The uncurated results can found in Figure 4 It shows that the token-based generator able to synthesize both high-frequency details of the human face and plausible structures for complex scenes.
Style editing. We study and report the results of style editing by the TokenGAN in Figure 5 All the results obtained by editing the style token of interest and then re-synthesizing images using edited style tokens. The result shows several interesting properties of the TokenGAN, First, the TokenGAN has similar behavior as StyleGAN2, i.e., controlling coarse/middle/fne styles at different layers (e.g., pose/hairstyle/color scheme in 82 8²/162 642 /1282 256 layers). Besides, TokenGAN shows localized behaviors for the style tokens. For example, the first row, editing the style token for the pose "globally" changes the pose while remaining the styles of other regions (e.g., hairstyle, background). In the second row, editing the style token for hair length, ToKenGAN "locally" turns the hair longer while remaining other styles.
7

---

changing pose
coarse style
editing hair length
editing hairstyle
middle style
adding beards
adding smiling
changing color scheme
fine style
Figure 5: Results of style editing. All the results obtained by editing the style token of in- terest and then re-synthesizing images using edited style tokens. shows that TokenGAN con- trols coarse/middle/fie styles different resolution layers (e.g., pose/hairstyle/coll scheme in 82 6²/162 - 642 /422/122 256 layers). Besides, the TokenGAN is able to get localized behavior by editing specific style while remaining other styles the same time. For example, in the second row, the TokenGAN turns the hair length longer without changing other styles.
Image inversion. To better apply well-trained GANs real-world applications, the technique of GAN inversion has attracted an increasing attention [40] (33 [ (2. Such an inversion technique enables real image editing by searching the accurate latent code the learned latent space to recover the real image [33] It has been demonstrated that the model with better inversion results tend learn better image modeling of the real data (28). Specifically, we randomly sample several real images from TEHQ-256 (28) and adopt the optimization based inversion technique used by most works 27
For fair comparisons, we use the same setting following StyleGAN2 [27]. The inversion results by the style-based generator and the token-based generator are shown in Figure shows that the token-based generator able to reconstruct real images better. For example, StyleGAN2 tends to reconstruct the lips and the headscarf in the fifth case with the same red color,

---

Real Image
StyleGAN2
Ours
Figure 6: Visual results of image inversion by StyleGAN2 28 and the token based generator. We adopt the same inversion technique following the common paradigm O 222 27). It shows that the token-based generator able to reconstruct fine- aeeegrained details with the dense style control over image regions (e.g., the facial expressions in the third case).
Table 2:
Quantitative comparison of the reconstructed images by image inversion in terms of mean average error (MAE, range=(0,255)) and LPIPS distance [5]].
Model
MAEI
LPIPS |
Model
MAE]
LPIPS
StyleGAN2 28
16.45
0.1539
TokenGAN
13.43
0.1238
while the -baed generator is able to reconstruct accurate colors for the lips and the headscarf respectively. This is because TokenGAN renders different regions by using different style tokens via attention mechanism, while StyleGAN2 renders them by a single style vector.
For better quantitative comparison for image inversion, randomly sample 1,000 real images from FFHQ-256 and report the mean absolute error (MAF, range=(0,255]) and the LPIPS distance [51] of the inversion results by StyleGAN2 and TokenGAN. The results in Table show that TokenGAN is able to reconstruct significantly more accurate results with much lower MAE and shorter LPIPS distance, which is in line with the visual results in Figure
Image interpolation. To further explore the property of the learned latent space by the token-based generator, we perform image interpolation by a linear interpolation space S. In practice, we randomly sample two noises 21, Z2 and run them through the mapping
the learned intermediate latent
..,,). , x2 = {s} S network have their corresponding sequence of style tokens s = {s} s + -a) S a (0, After that we perform linear interpolation for each token by s²
synthesize a new image. As shown Figure 7 we show the sampled styles s² s2 in the first column and the last column, and the interpolated styles between them. The results show that the token-based generator able to perform perceptually smooth transition between two different styles (e.g., glasses, age, hair length, coloring, etc.)
and use the new style tokens 3 := {s] .., s
4.3 Ablation study
To study the effectiveness of each component of the token-based generator, this section presents abla- tion studies on FFHQ-256 27 after 0.4 million training iterations terms of FID [19]. Specifically, calculate the metric 10 times with different random seed and report the average results.

---

Figure 7: Results of image interpolation by the token- based generator. In practice, sample two latent codes (the 1st and the 7th column) and perform linear interpolation them the learned intermediate latent space (interpolated results are from the 2th the 6th column). The token-based generator shows smooth transition of high-level attributes in results (e.g., glasses in the second row).
Table Quantitative ablation study on the number of style tokens.
style tokens
8
16
32
64
FID
7.66 + 0.060 7.02 + 0.060 6.81 + 0.044 7.60 + 0.071
The number of style tokens. To study the effectiveness of the style tokens, we conduct ablation studies by using different number of style tokens. As shown in Table 3 with the growing number of the style tokens, there are more style tokens to model the distribution of styles for different semantics and the quality of images improved. However, when the number of style tokens grows 64, the learning will be difficult and performance would drop.
Table Comparisons on the content tokens.
Table 5: Comparisons on the style normalization.
(Mea, M128, M256)
FID
Model
FID
(162, 162,,2"
18.54 0.087
InstanceNorm 41
13.7 + 0.051
(322, 322, 642) (642, 642, 128²)
15.69 0.067
PixelNorm 25
6.96 + 0.050
6.81 0.044
LayerNorm
43
6.81 = 0.044
The number of content tokens. We empirically set the number of content tokens m at different layers according the image size for the best performance and training efficiency. We conduct ablation study from layer 64 to 256 and denote the number of content tokens in these layers as (m61, m12s, m256). The results in Table 4 show that more content tokens could provide more fine-grained control over the whole images, leading to better results in image generation.
Style normalization layers. We compare different style normalization layers in Table The results show that, PixelNorm (25] and LayerNorm [43] significantly outperforms the style normalizatioin by instance normalization in the token-based generator [41]. We choose LayerNorm the style normalization layer in the token-based generator.
5 Conclusion
In this paper, we propose novel TokenGAN showing that the boken-based representation image features and styles could enable content-aware and fine-grad ineeggrined style learning for image synthesis. Specifically, we use Transformer to model the interaction between content tokens and style tokens, which facilitates the perceived quality of generated images and shows promising properties in terms of style editing. In the future, we will study to extend the generative transformers to extensive applications, e.g., style transfer [20], animation generation [45], image inpainting [49], etc.
10

---

Acknowledgments and Disclosure of Funding
Funding in direct support of this work: NSF China under Grant 61672548 and U161 1461, GPUs provided by Microsoft Research Asia. Additional revenues related to this work: internship at Microsoft Research Asia.
References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In ICCV, pages 4432 4441, 2019.
[2] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegant:: How to edit the embedded images? In CVPR, pages 9296-8305, 2020.
[3] Diederik P. Kingma andJimmy Ba. Adam: A method for stochastic optimization. In ICLR. 2015.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2018.
[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint vxXiv:2005.1410 2020.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[7] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, pages 12299-123100 2021.
[8] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. ICML, pages 1691-1703, 2020.
[9] Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncovering the local semantics of gans. In CVPR, pages 5771-5780, 2020.
[10] Emily Denton, Soumith Chintala, Arthur Szlam, and Robert Fergus. Deep generative image models using laplacian pyramid of adversarial networks. In NeurlPS, pages 1486-1494, 2015.
[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI-HIIT, pages 4171-4186, 2019.
Ming-Wei
[12] Chris Donahue, Zachary C Lipton, Akshay Balsubramani, and Julian McAuley. Semantically decomposing the latent spaces of generative udversarial networks. In ICLR, 2018.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[14] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873- 12883, 2021.
[15] Rinon Gal, Dana Cohen, Amit Bermano, and Daniel Cohen-Or. Swagan: A style-based wavelet-driven generative model. arXiv preprint arXiv:2102.066 2021.
[16] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In NewrIPS, 2014.
[17] Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior, In CVPR, pages 3012-3021, 2020.
2020.

---

[18] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, al. A survey on visual transformer. arXiv preprint arXiv:2012. /2556, 2020.
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two scale update rule converge to local nash equilibrium. In NewrIPS, pages
2017.
[20] Zhiyuan Hu, Jia Jia, Bei Liu, Yaohua Bu, and Jianlong Fu. Aesthetic-aware image style transfer. In ACM MM, pages 3320-3329, 2020.
[21] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, pages 1501-1510, 2017.
[22] Drew A Hudson and C. Lawrence Zitnick. Generative adversarial transformers. JCML, 2021.
[23] Ali Jahanian, Lucy Chai, and Phillip Isola. On the "stcerability" of generative adversarial networks. In ICLR, 2019.
[24] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. In NeurlPS, 2021.
[25] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2018.
[26] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NeurIPS, 2020.
[27] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410, 2019.
[28] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, pages 81 11-80119 1--881999 19, 2020.
[29] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. NeurIPS, 32:3927-3936, 2019.
[30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[31] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. In CVPR, pages 2437-2445, 2020.
[32] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICMI., pages 3481-3490, 2018
[33] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
[34] Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders. In NeurIPS, pages 4797-4805, 2016.
[35] Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun-Yar Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, pages 2337-2346, 2019.
[36] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, pages 4055 4064, 2018.
[37] Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoen- coders. In CVPR, pages 14104-14113, 2020.
[38] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. In CVPR, pages 2287-2296, 2021.
12

---

[39] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assess- ing generative models via precision and recall. In NewrlPS, pages 5234-5243, 2018.
[40] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, pages 9243-9252, 2020.
[41] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maxi- mizing quality and diversity in feed-forward stylization and texture synthesis. In CVPR, pages 6924 6932, 2017.
[42] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, pages 1747-1756, 2016.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, pages 5998-600 6008, 2017.
[44] Yi Wang, Ying-Cong Chen, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Attentive normalization for conditional image generation. In CVPR, pages 5094-5103, 2020.
[45] Hongwei Xue, Bei Liu, Huan Yang. Jianlong Fu, Houqiang Li, and Jiebo Luo. Learning fine-grained motion embedding for landscape animation. In ACM MM, pages 291-299, 2021.
[46] Fuzhi Yang. Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture trans- former network for image super-resolution. In CVPR, pages 5791-5800, 2020.
[47] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1506.03365 2015.
[48] Yanhong Zeng, Jianlong Fu, and Hongyang Chao. Learning joint spatial-temport transforma- tions for video inpainting. In ECCV, pages 528-543, 2020.
[49] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-context encoder network for high-quality image inpainting. In CVPR, pages 1486-1494, 2019.
[50] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, pages 7354-7363, 2019.
[51] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The un- reasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595 2018.
[52] Heliang Zheng, Jianlong Fu, Yanhong Zeng, Jiebo Luo, and Zheng-Jun Zha. Learning semantic- aware normalization for generative adversarial networks. NeurIPS, 33:21853-2113 2020.
[53] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In ECCV, pages 192-608,72 2020.
[54] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region- adgativ normalization. In CVPR, pages 3104-5113, 2020.
[55] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021.
13


Reviews:
[{"review_id": "TmPxNBgq98V", "reviewer": "Reviewer_cPNP", "summary": "This paper proposes TokenGAN for unconditional image synthesis. The token-based generator takes the learned content tokens and style tokens from the latent space as input, and output a series of style-modulated content tokens, which are then concatenated and reshaped to get the output image. Cross-attention is used to assign the styles to different content tokens. The model adopts the style-based generator idea from StyleGAN and introduces the recent token-based structures and transformers to the traditional style-based generator. Experiments are conducted on FFHQ and LSUN CHURCH datasets and the quantitative results are slightly better than StyleGAN2.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed approach is well-motivated. It takes advantage of the latent code mapping and style-based modulation of StyleGAN, and further improves it by replacing the convolution-based architecture with token-based cross-attention.\n2. By using a series of content tokens and style tokens, each patch on the image can adaptively attend to content-aware styles. So the structure can control the styles of different patches, in contrast to a single global style code in StyleGAN.\n\nWeakness:\n1. The quantitative results (FID, precision, and recall) are only slightly better than StyleGAN2. The qualitative results only show the generated images of the proposed TokenGAN, but does not show the comparison with StyleGAN2. From the generated images shown on the paper, I did not observe very significant improvements from StyleGAN2.\n2. Some ablation studies are missing: (1) the number of content token (2) Mixing different style tokens (used in this paper) v.s. mixing styles at different layers (used in StyleGAN) for style mixing. (3) Style modulation by amplifying each channel (in this paper) v.s. by adaptive normalization (in StyleGAN).\n3. For the discussion on image inversion, evaluation metrics such as LPIPS distance should be reported and compared with StyleGAN2.\n4. I am curious about the complexity and efficiency of the proposed model compared with StyleGAN2.\n5. It would be good to visualize how the style tokens can impact different regions (impact tokens) of the images. The authors tried to visualize the attention maps in Fig. 3, but I also expect other visualizations or evaluations illustrating how the multiple style tokens and cross-attention benefit the generated image quality and attribute disentanglement. For example, a visualization of style mixing might indicate how changing one style token (and keeping other style tokens unchanged) can affect the synthesized images.\n6. Lines 174 - 178 on implementation details are not clear to me. Could the authors explain more on how to incorporate the skip connections and how to do upsampling in the generator? Are there interactions (such as self-attention) between the content tokens? If not, how does the generator deal with global consistency and context information?\n7. This paper lacks discussion and comparison with a similar work [21] which also uses transformer for GANs. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not address the limitations and potential negative societal impact of their work. It would be good to add some discussions about  the potential limitations and failure cases of this work.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "PnZlOyqdHQp", "reviewer": "Reviewer_whGT", "summary": "This paper investigates the application of transformer for image generation by replacing the convolutional layer with MLP(including attention). The motivation provided is the limitation of the style-based generator (e.g.,  StyleGAN), which leverages the single style over multiple image regions, allowing a less entangled representation learning in W and visible artifacts in generated images.  To combat this shortcoming, authors introduce transformer-liked architecture, and successfully generate high-quality images (1024*1024) without touching any convolutional layers in generator.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nAuthors firstly exploit the image generation with transformer to synthesize high-fidelity images. The transformer-liked architecture has been used a series of computer vision tasks,  in this paper authors attempt to introduce it into image generation tasks. To achieve better performance, authors devise the model based on the StyleGANv2, aiming to use the well-devised framework and techniques.\n\nOne of the interesting parts is to generate the high-quality image without using any convolutional layers, which exhibits the power of the transferor-liked architecture.  Comparing StyleGANv2, the proposed method is able to obtain better performance in some metrics.\n\nThe proposed method can be further used for other tasks (image editing), which almost keep the property of the StyleGAN.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Cons:\nThe proposed method maybe not corresponding to the provided motivation(the limitation of the stylegan), since the proposed method is based on the styleganv2, and leverages all the techniques of StyleGANv2. I am wondering the motivation of the proposed method. In fact, I guess the motivation is to use firstly transformer to generate high-fidelity image.\n\nI am hoping that more analysis or insights could be exhibited in this paper, since the transformer is firstly used in generation model. However, all shown and reported results are little boring, although it looks better than styleganv2. In fact, this paper is based styleganv2.\n\nThe transformer is only exploited in generator instead of discriminator. The related work has already shown that the transformer obtains outstanding performance in classification task, why in this paper authors do not use it in discriminator.\n\nSince the learnable position encoding is intruded in content tokens, I am wondering why the position encoding do not be injected in style token? Besides, I fail to find the code presenting the position encoding when checking the provided code. Please correct me if I am wrong.\n\nThe detail for the value of m and k are loss, even in supplementary material.  Is the model performance influenced by these selected values?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "9SfqpylSup", "reviewer": "Reviewer_SrfN", "summary": "In this paper, a transformer-based modification of the generator in the ubiquitous StyleGAN model is proposed (named _TokenGAN_). More specifically, the entire generator is replaced by a convolution-free architecture where style-content modulation is now modeled using cross-attention layers instead of AdaIN. Similar to the original StyleGAN, this should allow for decoupled control of style and content at different granularities. The paper shows that the proposed approach achieves FID scores comparable to or better than StyleGAN2 for unconditional image synthesis on FFHQ and LSUN-Churches.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Strengths:__\n\nThe proposed approach is a simple, yet working modification of the generator of StyleGAN. The use of cross-attention to model style-content interactions is a good, novel idea. The attention maps depicted in Figure 3 nicely demonstrate that the generator trained on FFHQ attends to human-interpretable concepts. The paper thoroughly reports error bars / standard deviations where applicable.\n\n__Weaknesses:__\n\nAlthough the paper states the \"fine-grained controllability of visual tokens\" (l.60), no experiments are made to explicitly verify this claim or how controllability is increased over the StyleGAN baseline.\nFurthermore, the abstract states that the approach is \"able to synthesize high-fidelity images [...], dispensing with convolutions entirely\" (l.15-16). However, as the discriminator is the same as for StyleGAN2, this statement does only hold for the architecture of the generator, but not the complete model. Following on from this, it would be interesting to see if it is also possible to replace the discriminator with a pure Transformer architecture.\n\nGiven that the proposed architecture is mostly transformer-based, are there any notable differences in applicability to other datasets? For example, how does the model handle (very) small datasets or large-scale, complex datasets? Is overfitting a problem? If so, how is this being adressed? Here, plotting nearest neighbors (e.g. in VGG space) would be an interesting analysis option.\n\nFinally, the paper does not mention on which patch-size $p$ the models are trained. How do different $p$ affect the results? Furthermore, how do different $m$ affect the model? I can only find a study regarding $n$. Also: Is $\\mathbf{K}$ shared across different style modulation layers?\n\n__Clarity:__\nThe presentation of the cross-attention layer and style/content tokens is clear; however, the \"upsampling procedure\" (l. 20-21 in the supplementary) remains unclear to me and should probably be included in a Figure (e.g. Fig. 2)\n\n__Significance/Originality:__\nReplacing the convolutions in the generator of StyleGAN has the potential to allow applications to more complex datasets than the often used FFHQ or single-scene LSUN variants. Unfortunately, this remains unexplored in this work, but might be of interest to the community.\n\n__Justification of Rating:__\nI think the use of cross-attention as a modulation level for style content is an interesting idea and is sufficiently justified in the paper. However, apart from the visualization of attention weights in Figure 3, the paper does not show how the proposed method differs in applicability compared to StyleGAN(2) and what the impact is of switching to a transfomer architecture. For example, how do the suggested modifications alter convergence speed? What's the effect when trained on smaller/larger datasets? Is overfitting a larger/smaller problem than in the underlying work? Including the weaknesses mentioned above, I currently think that the paper (in its current state) would be a good fit for a workshop, but needs to be expanded (see the comments above) to be published as a full conference paper. However, I am curious about the authors' response and the opinions of the other reviewers, and I am happy to be proven wrong.\n\n_Minor_:\n- l. 53: i -> in\n- l. 63: decompose -> decomposes\n- l. 69: missing space after \"TokenGAN\"\n- l. 135: missing space after \"TokenGAN\"\n- l. 186: adopts -> adopted\n- l. 268: view -> viewing\n- l. 278: demonstrate -> demonstrated\n- Fig. 2: \"concate\" -> \"concatenate\"/\"concat\"\n\n\n\n__[UPDATE] Post-Rebuttal__\n\nAfter re-reading the paper and rebuttal, I think my original rating was too harsh, and I am therefore raising my score to 5. However, I am still not convinced that the proposed approach offers many advantages over the convolution-based StyleGAN(2) architecture, i.e., the gains presented here are rather marginal.  I think the work would benefit from a more fundamental analysis of the differences (training dynamics, applicability) between TokenGAN and StyleGAN, where then an improvement in the typical metrics (e.g. FID) would be nice but not mandatory.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not discussed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7.5"}, "extra_text": {}}, {"review_id": "1ojP9OhG4Dj", "reviewer": "Reviewer_skfS", "summary": "This paper proposes to adopt the StyleGAN architecture (i.e. individual styles that affect image generation) to a transformer based model for image generation. The model generates individual image patches which are modulated by style tokens through an attention mechanism. The model is evaluated on the FFHQ and LSUN Church datasets and performs similarly well or better than StyleGAN2 based on FID, precision, and recall.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes to model images as a sequence of image patches. These image patches start out as learned constants (similarly to StyleGAN) and are then modulated via style tokens. The interaction between content and style tokens is modeled via an attention mechanism (between content and style) such that this approach also scales to high-resolution images (1024px).\n\nIt would be helpful to be a bit more clear about the dimensions of the individual tokens, i.e. what values do you use for mn, n, p, and d in Fig 2. Also, how many layers do you use? Since you compare to StyleGAN2 it would also be helpful to mention training time and other requirements (e.g. memory) compared to StyleGAN2, as well as how long it takes to generate a new image at different resolutions.\n\nSince you also use stlye mixing I would like to see some qualitative results of that, i.e. what do generated images look like that are generated from two different styles? Can we get similar behavior as in StyleGAN where we can control coarse layout and texture details at different layers? Overall I would be happy to see a more detailed analysis of the \"style space\" in this approach, given that this is arguably one of the biggest advantages of the original StyleGAN architecture.\n\nSince you use different styles to directly modulate individual content tokens (i.e. image patches): can you get localized behaviors for the style tokens, e.g. mix two styles \"spatially\" where e.g. one half of the image is controlled by one style and the other half by another style? Or do reference based image translation, e.g. replace the hair in a given image with the hair of another image?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are not discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a transformer-based generator architecture for GAN and showed it achieved comparable image synthesis results with the StyleGAN2 baseline. It initially received a mixed rating, with two reviewers rated it above the bar and two below the bar. The rebuttal addressed some of the concerns. Both of the reviewers that were originally leaning toward a rejection recommendation upgraded the score. Overall, the meta-reviewer agreed that the proposed method has its merit. While, given the success of transformers for various vision tasks, it was kind of expected that one could achieve comparable results to CNNs with a transformer-based architecture for a GAN generator, it is still nice to see such a design being revealed and shown comparable performance. While it is unfortunate that the discriminator is still based on CNNs and the additional capability achieved by the architecture is underwhelming, the meta-reviewer thought the paper still presents a good first step. The authors are encouraged to incorporate the reviewer feedback in the final version.
Decision: Accept (Poster)