Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers
Year: 2022

Abstract:
We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. Different from existing paradigms that directly synthesize a full image from a single input (e.g., a latent code), the new formulation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis. Specifically, it takes as input a sequence of latent tokens to predict the visual tokens for synthesizing an image. Under this perspective, we propose a token-based generator (i.e.,TokenGAN). Particularly, the TokenGAN inputs two semantically different visual tokens, i.e., the learned constant content tokens and the style tokens from the latent space. Given a sequence of style tokens, the TokenGAN is able to control the image synthesis by assigning the styles to the content tokens by attention mechanism with a Transformer. We conduct extensive experiments and show that the proposed TokenGAN has achieved state-of-the-art results on several widelyused image synthesis benchmarks, including FFHQ and LSUN CHURCH with different resolutions. In particular, the generator is able to synthesize high-fidelity images with 1024× 1024 size, dispensing with convolutions entirely.

heading: 1 Introduction
text: Unconditional image synthesis generates images from latent codes by adversarial training [10, 16, 19, 25, 27, 33, 50]. Recent advances have been achieved by a style-based generator architecture in terms of both the visual quality and resolution of generated images [26, 27, 28, 37, 52]. In particular, the style-based generator has been widely used in many other generative tasks, including facial editing [9, 40], style transfer [1, 38], image super-resolution [17, 31], and image inpainting [2, 52].
The key to the success of the style-based generator lies in the learning of the style control based on the intermediate latent spaceW [27, 28]. Instead of feeding the input latent code z ∈ Z through the input layer only (Figure1-a), the style-based generator maps the input z to an intermediate latent space w ∈ W , which then controls the “style” of the image at each layer via adaptive instance normalization (AdaIN [21]) (Figure 1-b). It has been demonstrated that such a design allows a less entangled representation learning inW , leading to better generative image modeling [12, 23, 27, 40]. Despite the promising results, the style-based generator can suffer from the style control via AdaIN operation [28, 52]. Specifically, the style control is content-independent. It “washes away” the original information of features by normalization and assigns new styles decided by the latent codes regardless of the image/feature content. Besides, the style code w affects the entire image by scaling and biasing complete feature maps with a single value via AdaIN operation [35, 44, 54]. Such an imposed single style over multiple image regions can inevitably result in entangled representation of
∗This work was done while Yanhong Zeng was a research intern at Microsoft Research Asia.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
different image variations (e.g., hairstyle, facial expression) [28, 44, 52]. These limitations for image modeling can even lead to visible artifacts in the synthesized results (e.g., droplet artifacts [27]).
To get rid of the issues caused by StyleGAN’s style modeling, we introduce a new perspective that views image synthesis as a visual token generation problem. The visual token is a popular representation of an image patch with a predefined size and position [6, 7, 13]; and has shown an impressive superiority in various tasks with the development of Transformer models [6, 13, 43, 46]. Inspired by the appealing property of the token-based representation, we propose to achieve image synthesis by visual token generation. Specifically, it takes as input a sequence of latent tokens to predict the visual tokens of an image. Such a token-based representation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis.
Under this new paradigm, we design a token-based generator, i.e., TokenGAN, for the visual token generation problem. Specifically, we consider two different types of input tokens in the generator, i.e., the content tokens and the style tokens. The content tokens are learned as the constant input in the generator network and the style tokens are projected from a learned intermediate latent space (Figure1-c). Given a sequence of style tokens, the TokenGAN learns to control the visual token generation by rendering each content token with related style tokens according to their semantics. In particular, since the Transformer has been verified to be effective in sequence modeling in a broad range of tasks [6, 43], we adopt a generator network architecture from a visual Transformer to model the relations between the content tokens and the style tokens. Through such a content-dependent style modeling by the attention mechanism in Transformer, the TokenGAN is able to achieve content-aware and fine-grained style learning for image synthesis.
We conduct both quantitative comparisons and qualitative analysis on several unconditional image generation benchmarks. The results show that the token-based generator has achieved comparable results to the state-of-the-art in image synthesis. We summarize our contributions as below:
• We propose a new perspective of achieving image synthesis by visual token generation. Such a token-based representation enables flexible local manipulation for different image regions, leading to a better image modeling. • We propose a token-based generator (i.e., TokenGAN) for the visual token generation. Specifically, the TokenGAN introduces the style tokens and the content tokens. It adopts a Transformer-based network for content-aware style modeling. • We show extensive experiments (quantitative and qualitative comparisons, study on style editing, image inversion and image interpolation) to verify the effectiveness of the tokenbased generator. Specifically, the token-based generator is able to synthesize high-fidelity 1024× 1024 images without any convolutions in the generator.

heading: 2 Related works

heading: 2.1 Style-based generator
text: The distinguishing feature of the style-based generator is its unconventional generator architecture [26, 27, 28, 52]. Typically, the style-based generator consists of a mapping network f and a synthesis
network g. The mapping network is used to transform the input latent code z to an intermediate latent code w ∈ W for learning a less entangled latent space. The synthesis network follows a progressive growing design that is able to first output low-resolution images that are not affected significantly by high-resolution layers [25, 27]. Such an architecture design can lead to an automatic, unsupervised separation of high-level attributes in different layers. For example, when trained on human faces, the low-resolution layers control coarse styles (e.g., pose, identity) of images and high-resolution layers control fine styles (e.g., micro-structure, color scheme) [25].
To control the attributes of the image, the style-based generator produces styles from the intermediate latent code w by affine transforms, which then control each layer of the synthesis network via adaptive instance normalization (AdaIN) [21, 35]. The AdaIN operation removes styles from previous layers by normalizing each feature map to zero mean and unit deviation, and it assigns the new styles by scaling and biasing the complete normalized feature maps. However, it has been demonstrated such an AdaIN operation can destroy the magnitude information of features and thus results in the well-known droplet artifacts [28]. Besides, the scale-specific style disentanglement in StyleGAN can be limited due to the complex semantic components in each feature map [28, 52]. In this paper, we explore a new architecture by flattening the image as a sequence of tokens, which enables fine-grained control of the image by assigning token-wise semantic-aware styles based on the attention mechanism.

heading: 2.2 Transformer in vision
text: The Transformer typically takes as input a sequence of vectors, called tokens [43]. The Transformer is built for sequence modeling solely on attention mechanisms over the tokens, dispensing with recurrence and convolutions entirely [5, 11, 43, 30]. Due to its great success in the field of natural language processing, an increasing number of works attempt to extend Transformer for computer vision tasks [6, 7, 8, 13, 18, 36, 55]. For example, iGPT shows promising results in image classification by pre-training a sequence Transformer with the task of auto-regressive next pixel prediction and masked pixel prediction [8]. Dosoyvitskiy et al. propose to split an image into patches and feed these patches into a standard Transformer (ViT) for image classification [13]. They show that a ViT with large-scale training can trump CNNs equipped with inductive bias. DETR is a seminal work that views object detection as a direct set prediction problem, eliminating the need for hand-crafted components and achieving impressive performance by Transformer [6]. To mitigate the issue of high computation complexity associated with long sequences caused by high-resolution images, a branch of works explores lightweight Transformer (e.g.., Deformable DETR) for vision tasks [55].
Transformer is also attracting increasing attention in low-level vision tasks [14, 24, 34, 36, 42]. Parmar et al. cast image generation as an autoregressive sequence generation problem and propose Image Transformer with a local self-attention mechanism [36]. However, Image Transformer can suffer from its quadratic computation cost and its long inference time due to auto-regressive prediction. Most existing works adopt a hybrid CNN-Transformer architecture, which consists of a CNN head for feature extraction, a Transformer encoder-decoder backbone, and a CNN tail for feature decoding [7, 46, 48]. For example, IPT fully utilizes the Transformer architecture by a large-scale pre-training and achieves promising results in several image restoration tasks [7]. In parallel work, Hudson et al. introduce bipartite structure to maintain computation of linear efficiency for long-range interactions across the image based on a convolution backbone [22]. Our token-based generator inherits the network architecture from Transformer without any convolution layers and is able to yield surprising promising results for high-resolution image synthesis.

heading: 3 Approach
text: In this section, we introduce the details of the proposed token-based generator (TokenGAN). As depicted in Figure 2, the TokenGAN takes as input two kinds of tokens to generate the visual tokens of images by a Transformer. We introduce the input tokens in Section 3.1 and the token-based generator architecture in Section 3.2, following the overall optimization in Section 3.3.

heading: 3.1 Visual tokens
text: It’s intuitive that the visual tokens of an image contain the information of both content (i.e., semantics) and related styles. In our token-based generator, we choose to separate these two kinds of information
by a sequence of content tokens and a sequence of style tokens, so that the interactions between them can be modeled and controlled for the image synthesis. We introduce the details of each as below.
Content tokens. As depicted in Figure 2-a, the TokenGAN starts from the sequence of learned constant content tokens c ∈ Rm×(p2×d) through the input layer. Specifically, d is the dimension in terms of channels, p × p is the patch size of the content token, and m is the length of the content token sequence. To maintain the position information of each token, we add position encodings p ∈ Rm×(p2×d) to each content token following a commonly used paradigm [7, 13]. For simplicity, we rewrite the notation for the content tokens as {c1, · · · , cm}, where ci ∈ Rd.
Style tokens. Given a latent code z from the input latent space Z , the mapping network adopts several MLPs to map the input z to a set of different style tokens {s1, · · · , sn} ∈ S, where si ∈ Rd. As shown in Figure 2-b, the style tokens are paired with a set of learnable semantic embedding as a key-value structure in each style modulation layer of the TokenGAN. With the Transformer modeling, all the content tokens will match with the semantic embedding and then fetch the new style from the style tokens based on the matching results. The fetched new styles are used to control the values of the content tokens, which are finally decoded to images. Such a token-wise style control enables content-aware and fine-grained style learning for image synthesis.

heading: 3.2 Token-based generator
text: As shown in Figure 2, the TokenGAN consists of multiple layers of style blocks and each style block consists of a style normalization layer and a style modulation layer. Specifically, the style normalization layer removes the styles from previous layers by relieving the dependence on the original statistics of input token features. At the same time, the style modulation layer assigns new styles from the current layer to the content tokens. In particular, the new styles are calculated based on the pairwise interactions between the style tokens and the content tokens by a cross-attention mechanism. We introduce more details about the style normalization, content-aware style modeling, and style modulation in our generator as below.
Style normalization. We apply style normalization on the input content tokens to remove the styles from previous layers, which is denoted as:
Norm(ci) = (ci − µ(ci)) / σ(ci), i = 1, · · · ,m, (1)
where each ci ∈ Rd denotes a content token, d denotes the dimension of each content token, and m is the length of the token sequence. Specifically, the style-based generator typically removes the styles from previous layers by Instance Normalization [41], which may destroy the information conveyed by the magnitude of the feature map relative to each other [27, 28, 41]. To avoid the above issue, we adopt LayerNorm [43] to relive the style dependence while maintaining the information of the relative magnitude following previous works [28, 29, 43].
Content-aware style modeling. After style normalization, the TokenGAN calculates new styles to control the input content tokens. As shown in the style modulation in Figure 2-b, the style tokens are paired with a set of learnable semantic embedding as a key-value structure. Given the content tokens {c1, · · · , cm} from the input and the style tokens from the mapping network {s1, · · · , sn} ∈ S, the token-wise new styles are calculated by attention mechanism:
S′ = Attention(C,K,S) = softmax( CKT√
d )S, (2)
where S′ ∈ Rm×d denotes new styles for all the content tokens, the input content tokens are packed together into a matrix C ∈ Rm×d, similarly with the style tokens as matrix S ∈ Rn×d, and the matrix K ∈ Rn×d for the learnable semantic embedding in where each row vector indicates a learned semantic. We omit the linear projection in the standard attention calculation for simplicity [43].
Through such a cross-attention mechanism, the new token-wise styles are fetched for style modulation based on the matching results. Particularly, tokens with similar semantics will have similar styles (e.g., the same color for the two eyes). We visualize the attended regions for different key-value pairs (i.e., semantic-style embedding) in Figure 3, and we can find that the learned embedding K is able to attend to meaningful image regions.
Style modulation. Different from the traditional Transformer that adds the attention results back to the input features as residual features, our attention results are used as new styles by amplifying each channel of the content tokens: C′ = C S′, (3) where each row vector in C ∈ Rm×d indicates a content token, and indicates element-wise multiplication. Such a style modulation affects the operation of subsequent embedding layers (which we implemented by fully-connected layers) and thus control the style of generated images.
Implementation details. We translate the generated visual tokens to the image by concatenation and reshaping. In practice, we follow StyleGAN2 and adopt a skip-generator architecture in TokenGAN. Such a skip generator generates images in each layer at different resolutions (e.g., from 42 to 82 ) and progressively upsamples and sums the images from the previous layer by skip connections to the next layer [28, 25]. After progressive summing, the generator takes the output in the last layer as the final results.

heading: 3.3 Overall optimization
text: Style mixing. To further encourage the styles to localize, we employ the mixing regularization technique during training [27]. To be specific, a given percentage of images are generated using two random latent codes. We run two latent codes z1, z2 through the mapping network, and have the corresponding {s11, · · · , s1n}, {s21, · · · , s2n}. We randomly choose an inject point to mix the styles, so that a part of the final style tokens from z1 and another part from z2. This regularization technique prevents the network from assuming that a set of style tokens are correlated. A similar strategy is adopted in terms of different layers following the style-based generator [27].
Optimization objectives. We denote the generated image output by the token-based generator as: G(z) = g(f(z),C), (4)
where z ∈ Z ∼ N (0, 1) indicates the input latent code, C = {c1, · · · , cm} denotes the sequence of learned constant content tokens through the input layer, and f , g, G indicate the mapping network, the generator network and the token-based generator, respectively. Our token-based generator follows the same optimization objectives used by the style-based generator [27, 28], i.e., an non-saturating logistic adversarial loss with R1 regularization. To be specific, the generator loss is:
LG = Ez∼N (0,1)[log(1 + exp(−D(G(z))))], (5) where D is the discriminator and the discriminator loss is
LD = Ex∼Pd [log(1 + exp(−D(x)))] + Ez∼N (0,1)[log(1 + exp(D(G(z))))] +R1(ψ), (6) where x ∼ Pd denotes images from the real data, and R1(ψ) is the regularization term calculated by:
R1(ψ) = γ
2 Ex∼Pd
[ ‖∇Dψ(x)‖2 ] . (7)
Specifically, the R1 regularization term [32] is computed less frequently than the main loss function following the commonly used lazy regularization strategy, thus greatly diminishing the computational cost and the overall memory usage [27, 28, 52].

heading: 4 Experiments

heading: 4.1 Experiment Setup
text: Dataset To evaluate the token-based generator and make fair comparisons with the SOTA approach (i.e., the style-based generator [28]), we conduct experiments on the most commonly-used public research datasets, i.e., Flickr-Faces-HQ (FFHQ) [27] and Large-scale Scene Understanding (LSUN) [47]. Specifically, FFHQ consists of 70,000 high-quality images of human face, which is able to evaluate the model’s ability to synthesize high-frequency details. We conduct experiments on FFHQ with both 256× 256 and 1024× 1024 image size following previous works [15, 22]. Besides, we choose the church set of LSUN [47] to evaluate the synthesis in terms of complex scenes.
Evaluation. In this section, we conduct both quantitative and qualitative experiments on the tokenbased generator. Specifically, we report quantitative results by three commonly-used metrics, i.e., Fréchet Inception Distance (FID) [19], Precision and Recall [29, 39]. We use FID as it has been widely used in many generation tasks as an effective perceptual metric [4, 26, 27]. In addition to the image quality assessment, we use the metrics of Precision and Recall to evaluate the distribution learned by GAN. Specifically, the Precision metric intuitively measures the quality of samples from the generated images, while recall measures the proportion of real images that is covered by the learned distribution by GAN [39].
Training details We use 8 NVIDIA V100 GPUs for training each model. We build upon the public Pytorch implementation of StyleGAN22 [28]. For fair comparisons, we inherit all of the training details and parameter settings from the configuration E setup. All models are trained and report the best results under the same training iterations. For each iteration, we set the size of mini-batch as 32. We use the Adam solver [3] with the same momentum parameters β1 = 0, β2 = 0.99 to train both the generator and discriminator. We apply R1 regularization for every 16 iterations.
2https://github.com/rosinality/stylegan2-pytorch

heading: 4.2 Unconditional image synthesis
text: Quantitative comparisons. We report quantitative comparisons with StyleGAN2 [28] due to its state-of-the-art performance. For fair comparisons, we report the results with the same training iterations on FFHQ-256 [27], FFHQ-1024 [27] and LSUN CHURCH [47] by three objective metrics, i.e., FID, Precision and Recall in Table 1. The results show that the token-based generator has achieved the state-of-the-art results in terms of both perceived quality and distribution modeling.
Qualitative results. To demonstrate the image quality of the synthesis results for the token-based generator, we randomly sample noise z ∈ Z ∼ N (0, 1) to generate images for high-resolution images (i.e., with resolution 1024× 1024) of human face and the images of complex church scenes. The uncurated results can be found in Figure 4. It shows that the token-based generator is able to synthesize both high-frequency details of the human face and plausible structures for complex scenes.
Style editing. We study and report the results of style editing by the TokenGAN in Figure 5. All the results are obtained by editing the style token of interest and then re-synthesizing images using edited style tokens. The result shows several interesting properties of the TokenGAN. First, the TokenGAN has a similar behavior as in StyleGAN2, i.e., controlling coarse/middle/fine styles at different layers (e.g., pose/hairstyle/color scheme in 82/162 − 642/1282 − 2562 layers). Besides, the TokenGAN shows localized behaviors for the style tokens. For example, in the first row, editing the style token for the pose "globally" changes the pose while remaining the styles of other regions (e.g., hairstyle, background). In the second row, editing the style token for hair length, TokenGAN "locally" turns the hair longer while remaining other styles.
Image inversion. To better apply well-trained GANs to real-world applications, the technique of GAN inversion has attracted an increasing attention [40, 53, 1, 2]. Such an inversion technique enables real image editing by searching the most accurate latent code in the learned latent space to recover the real image [53]. It has been demonstrated that the model with better inversion results tend to learn a better image modeling of the real data [28]. Specifically, we randomly sample several real images from FFHQ-256 [28] and adopt the optimization-based inversion technique used by most works [27, 40, 53]. For fair comparisons, we use the same setting following StyleGAN2 [27]. The inversion results by the style-based generator and the token-based generator are shown in Figure 6. It shows that the token-based generator is able to reconstruct real images better. For example, StyleGAN2 tends to reconstruct the lips and the headscarf in the fifth case with the same red color,
while the token-based generator is able to reconstruct accurate colors for the lips and the headscarf respectively. This is because TokenGAN renders different regions by using different style tokens via attention mechanism, while StyleGAN2 renders them by a single style vector.
For a better quantitative comparison for image inversion, we randomly sample 1,000 real images from FFHQ-256 and report the mean absolute error (MAE, range=[0,255]) and the LPIPS distance [51] of the inversion results by StyleGAN2 and TokenGAN. The results in Table 2 show that TokenGAN is able to reconstruct significantly more accurate results with much lower MAE and shorter LPIPS distance, which is in line with the visual results in Figure 6.
Image interpolation. To further explore the property of the learned latent space by the token-based generator, we perform image interpolation by a linear interpolation in the learned intermediate latent space S. In practice, we randomly sample two noises z1, z2 and run them through the mapping network to have their corresponding sequence of style tokens s1 := {s11, · · · , s1n}, s2 := {s21, · · · , s2n}. After that we perform linear interpolation for each token by s3i = α× s1i + (1− α)× s2i , α ∈ (0, 1) and use the new style tokens s3 := {s31, · · · , s3n} to synthesize a new image. As shown in Figure 7, we show the sampled styles s1, s2 in the first column and the last column, and the interpolated styles between them. The results show that the token-based generator is able to perform perceptually smooth transition between two different styles (e.g., glasses, age, hair length, coloring, etc.)

heading: 4.3 Ablation study
text: To study the effectiveness of each component of the token-based generator, this section presents ablation studies on FFHQ-256 [27] after 0.4 million training iterations in terms of FID [19]. Specifically, we calculate the metric 10 times with different random seed and report the average results.
The number of style tokens. To study the effectiveness of the style tokens, we conduct ablation studies by using different number of style tokens. As shown in Table 3, with the growing number of the style tokens, there are more style tokens to model the distribution of styles for different semantics and the quality of images are improved. However, when the number of style tokens grows to 64, the learning will be difficult and performance would drop.
The number of content tokens. We empirically set the number of content tokens m at different layers according to the image size for the best performance and training efficiency. We conduct ablation study from layer 642 to 2562 and denote the number of content tokens in these layers as (m64,m128,m256). The results in Table 4 show that more content tokens could provide more fine-grained control over the whole images, leading to better results in image generation.
Style normalization layers. We compare different style normalization layers in Table 5. The results show that, PixelNorm [25] and LayerNorm [43] significantly outperforms the style normalizatioin by instance normalization in the token-based generator [41]. We choose LayerNorm as the style normalization layer in the token-based generator.

heading: 5 Conclusion
text: In this paper, we propose a novel TokenGAN showing that the token-based representation of image features and styles could enable content-aware and fine-grained style learning for image synthesis. Specifically, we use Transformer to model the interaction between content tokens and style tokens, which facilitates the perceived quality of generated images and shows promising properties in terms of style editing. In the future, we will study to extend the generative transformers to extensive applications, e.g., style transfer [20], animation generation [45], image inpainting [49], etc.
Acknowledgments and Disclosure of Funding
Funding in direct support of this work: NSF of China under Grant 61672548 and U1611461, GPUs provided by Microsoft Research Asia. Additional revenues related to this work: internship at Microsoft Research Asia.

Review summaries:
This paper proposes TokenGAN for unconditional image synthesis. The token-based generator takes the learned content tokens and style tokens from the latent space as input, and output a series of style-modulated content tokens, which are then concatenated and reshaped to get the output image. Cross-attention is used to assign the styles to different content tokens. The model adopts the style-based generator idea from StyleGAN and introduces the recent token-based structures and transformers to the traditional style-based generator. Experiments are conducted on FFHQ and LSUN CHURCH datasets and the quantitative results are slightly better than StyleGAN2.

In this paper, a transformer-based modification of the generator in the ubiquitous StyleGAN model is proposed (named _TokenGAN_). More specifically, the entire generator is replaced by a convolution-free architecture where style-content modulation is now modeled using cross-attention layers instead of AdaIN. Similar to the original StyleGAN, this should allow for decoupled control of style and content at different granularities. The paper shows that the proposed approach achieves FID scores comparable to or better than StyleGAN2 for unconditional image synthesis on FFHQ and LSUN-Churches.

This paper proposes to adopt the StyleGAN architecture (i.e. individual styles that affect image generation) to a transformer based model for image generation. The model generates individual image patches which are modulated by style tokens through an attention mechanism. The model is evaluated on the FFHQ and LSUN Church datasets and performs similarly well or better than StyleGAN2 based on FID, precision, and recall.

This paper investigates the application of transformer for image generation by replacing the convolutional layer with MLP(including attention). The motivation provided is the limitation of the style-based generator (e.g.,  StyleGAN), which leverages the single style over multiple image regions, allowing a less entangled representation learning in W and visible artifacts in generated images.  To combat this shortcoming, authors introduce transformer-liked architecture, and successfully generate high-quality images (1024*1024) without touching any convolutional layers in generator.