Improved Transformer for High-Resolution GANs
Authors: Long Zhao
Zizhao Zhang
Ting Chen
Dimitris N. Metaxas
Han Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Image generation
Transformers
Generative adversarial networks

Abstract: Attention-based models, exemplified by the Transformer, can effectively model long range dependency, but suffer from the quadratic complexity of self-attention operation, making them difficult to be adopted for high-resolution image generation based on Generative Adversarial Networks (GANs). In this paper, we introduce two key ingredients to Transformer to address this challenge. First, in low-resolution stages of the generative process, standard global self-attention is replaced with the proposed multi-axis blocked self-attention which allows efficient mixing of local and global attention. Second, in high-resolution stages, we drop self-attention while only keeping multi-layer perceptrons reminiscent of the implicit neural function. To further improve the performance, we introduce an additional self-modulation component based on cross-attention. The resulting model, denoted as HiT, has a nearly linear computational complexity with respect to the image size and thus directly scales to synthesizing high definition images. We show in the experiments that the proposed HiT achieves state-of-the-art FID scores of 30.83 and 2.95 on unconditional ImageNet $128 \times 128$ and FFHQ $256 \times 256$, respectively, with a reasonable throughput. We believe the proposed HiT is an important milestone for generators in GANs which are completely free of convolutions. Our code is made publicly available at https://github.com/google-research/hit-gan.

Improved Transformer for High-Resolution GANs
Long Zh Zizhao Zhang²
Ting Chen
Dimitris N. Metaxas'
Han Zhang?
Rutgers University
Google Cloud AI Google Research
Abstract
Attention-base models, exemplified by the Transformer, can effectively model long range dependency, but suffer from the quadratic complexity of self-attention operation, making them difficult to be adopted for high-resolution image generation based Generative Adversarial Networks (GANs). In this paper, we introduce key ingredients to Transformer to address this challenge. First, in low-resolution stages of the generative process, standard global self-attention is replaced with the proposed multi-axis blocked self-attention which allows efficient mixing of local and global attention. Second, in high-resolution stages, drop self-attention while only keeping multi-layer perceptrons reminiscent of the implicit neural function. To further improve the performance, we introduce an additional ceff-modulttion component based on cross-attention. The resulting model, denoted as HiT, has a nearly linear computational complexity with respect the image size and thus directly scales to synthesizing high definition images. We show in the experiments that the proposed HiT achieves tfate-of-the-ere FID scores of 30.83 and 2.95 on unconditional ImageNet 128 x 128 and FFHQ 256 x 256, respectively, with a reasonable throughput. We believe the proposed HiT is an important milestone for generators in GANs which are completely free of convolutions. Our code is made publicly available at
com/ reogllerreeercchhhttttt
1 Introduction
Attention-basee models demonstrate notable learning capabilities for both encoder-based and decoder- based architectures [66] 69) due their self-attention operations which can capture long-range depen- dencies in data. Recently, Vision Transformer [12), one of the most powerful attention-based models, has achieved great success on encoder-based vision tasks, specifically image classification [12] 60), segmentation [37] (64), and vision-language modeling [46). However, applying the Transformer to image generation based on Generative Adversarial Networks (GANs) is still an open problem.
The main challenge of adopting the Transformer as a decoder/generator lies in two aspects. On one hand, the quadratic scaling problem brought by the self-attention operation becomes generating pixel-level details for high-resolution images. For example, synthesizing a high definition image with the resolution of 1024 x 1024 leads to sequence containing around million pixels in the final stage, which unaffordable for the standard self-attention mechanism. On the other hand, generating images from noise inputs poses higher demand for spatial coherency in structure, color, and texture than discriminative tasks, and hence a more powerful yet efficient self-attention mechanism is desired for decoding feature representations from inputs.
worse when
In view of these two key challenges, we propose a novel Transformer based decoder/generator in GANs for high-resolution image generation, denoted HiT. HiT employs hierarchical structure of Transformers and divides the generative process into low-resolution and high-resolution stages, focusing feature decoding and pixel-level generating, respectively. Specifically, its low-resolutin
"This work was done while Long Zhao a student researcher at the Google Brain team. Correspondence to: Long Zhao (I23II @rutgers.edu) and Han Zhang (zhanghan google.com).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

stages follow the design of Nested Transformers [73 but enhanced by the proposed multi- axis blocked self-attention to better capture global information. Assuming that spatial features are well decoded after low-resolution stages, in high-resolution stages, we drop all self-attention operations in order to handle extremely long sequences for high definition image generation. The resulting high-resolution stages of HiT are built by multi-layer perceptrons (MLPs) which have linear complexity with respect to the sequence length. Note that this design aligns with the recent findings [58] [59] that pure MLPs manage to learn favorable features for images, but it simply reduces to implicit neural function [4 4 in the case of generative modeling. To further improve the performance, we
present an additional cross-attention module that acts as a form of self- ceff-modulation [8] In summary, this paper makes the following contributions:
We propose HiT, a Transformer- baased generator for high-fidelity image generation. Standard self-attention operations are removed in the high-resolution stages of HiT, reducing them to an implicit neural function. The resulting architecture easily scales to high definition image synthe- sis (with the resolution of 1024 x 1024) and has a comparable throughput to StyleGAN2 (28]
To tame the quadratic complexity and enhance the representation capability of self-attention operation in low-resolution stages, we present a new form of sparse self attention operation, namely multi-axis blocked self-attention. It captures local and global dependencies within non- overlapping image blocks in parallel by attending to a single axis of the input tensor at a time, each of which uses half of attention heads. The proposed multi-axis blocked self-attente is efficient, simple to implement, and yields better performance than other popular self-attention working on image blocks for generative tasks.
operations (37 62 73
In addition, we introduce a cross-attention module performing attention between the input and intermediate features. This module re-weights intermediate features of the model as a function of the input, playing role as self- eef-loouuttion g . and provides important global information to high-resolution stages where self-attention operations are absent.
The proposed HiT obtains competitive FID [6 scores of 30.83 and 2.95 unconditional ImageNet [49 128 x 128 and FFHQ (28 256 x 256, respectively, highly reducing the gap ones. We also show that HiT not only
between Conv beseb GANs and Transformer-basee works for GANs but also can serve as a general decoder for other models such as VQ- VAEV [61]. Moreover, observe that the proposed HiT can obtain more performance improvement from regularization than its ConvNet-based counterparts. To the best of our knowledge, these are the best reported scores for image generator that is completely free of convolutions, which is an important milestone towards adopting Transformers for high-resolution generative modeling.
2 Related work
Transformers for image generation. There are two main streams of image generation models built on Transformers (63] in the literature. One stream of them [14] [4) is inspired by auto-regressive models that Iearn the joint probability of the image pixels. The other stream focuses on designing Transformer-basee architecture for generative adversarial networks (GANs) [15]. This work follows the spirit of the second stream. GANs have made great progress in various image generation tasks, such as image-to- to-image translation [57 74 75 79 and text-to-image synthesis [70] 71], but most of
them depend on ConvNet-basse based backbones. Recent attempts 32) build a pure Transformer- based GAN by a careful design of attention syper--prammttes as well as upsampling layers. However, such a model only validated on small scale datasets (e.g., CIFAR-I0 31 and S0 [0] consisting of 32 x 32 images) and does not scale to complex real-world data. To our best knowledge, no existing work has successfully applicd a Transformer-baat architecture completely free of convolutions for high-resolution image generation in GANs.
GANformer 21 leverages the Transformer as plugin component to build the bipartite structure to allow long-range interactions during the generative process, but its main backbone is still ConvNet based on StyleGAN (28) GANformer and HiT are different in the goal of using attention modules. GANformer utilizes the attention mechanism to model the dependences of objects in a generated scene/image. Instead, HiT explores building efficient attention modules for synthesizing general objects. Our experiments demonstrate that even for simple face image datasets, incorporating the attention mechanism can still lead to performance improvement. We believe our work reconfirms the necessity of using attention modules for general image generation tasks, and more importantly,
2

---

Embodding
Repeat N
Position Encoding
UpSample
High-Resolution Stages: Implicit Functions
Cross Attention
MLP
Low-Resolution Stages: Multi-Axis Nested Transformer
Latent Embedding
Repeat x M
Output Image
Position
UpSample
Encoding
Be
B+
B
Cross Attention
MultiAAiss
Attertion
Linear Projection
MLP
MLB
Input Latent Code
Latent Embedding
Figure 1: HiT architecture. In each stage, input the feature is first organized into blocks (denoted as B,). HiT's low-resolution stages follow the decoder design Nested Transformer [73] which is then enhanced by the proposed multi-axis blocked self-attention. We drop self-attention operations in the high-resolution stages, resulting in implicit neural functions. The model further boosted by cross-attention modules which allow intermediate features to be modulated directly by the input latent code. The detailed algorithm can be found in the supplementary materials.
present efficient architecture for cnaantion-based high-resolution generators which might benefit the design of future attention-based GANs.
Attention models. Many efforts 2
17 65
have been made to reduce the quadratic complexity of propose to compute
self-attention operation. When handling vision tasks, some works [37 62 73
local self-attention in blocked images which takes advantage of the grid structure of images. [6] also presents local self-attention within image blocks but it does not perform self-attention across blocks
as in HiT. The most related works to the proposed multi-axis blocked self-attention are [17 65 where they also compute attention along axes. But our work differs notably that we compute different attentions within heads blocked images. Some other works (221222) avoid directly applying standard self-attention to the input pixels, and perform attention between the input and a small set of latent units. Our gross-attention module differs from them in that apply cross-attention to generative modeling designed for pure Transformer-basee architectures.
Implicit neural representations. The largest popularity of implicit neural represeetations/lunetion (INRs) is studied in 3D deep learning to represent 3D shape in cheap and continuous way [41 Recent studies
42 43
34 [54] explore the idea of using INRs for image generation, where they learn a hyper MLP network to predict an RGB pixel value given its coordinates on the image grid. Among them, [ 54 are closely related
[3]
high-resolution stages of the generative process. One remarkable difference is that our model is driven by the cross-attention module and features generated in previous stages instead of the hyper-network presented in 54].
Approach
3.1 Main Architecture
In the case of unconditional image generation, HiT takes latent code NoO, input and generates an image of the target resolution through a hicrarchical structure. The latent code first projected into initial feature with the spatial dimension of Ho x Wo and channel dimension of Cg- During the generation process, we gradually increase the spatial dimension of the feature map while reducing the channel dimension in multiple stages. We divide the generation stages into low-resolution stages and high-resolution stages to balance feature dependency range in decoding and computation efficiency. The overview of the proposed method is illustrated in Figure [
In low-resolution stages, we allow spatial mixing of information by efficient attention mechanism. We follow the decoder form of Nested Transformer [73 where in each stage, the input feature is first

---

[4, 4,722
(4, 4, C/
Attention Axis
Input C]
Blocked Input
Output
4,
Channel Splitting
Concat
Blocking
Unblocking
1/2 Heads 4, C/
[4, 4, C/2
Attention 2nd
Figure 2: Multi-axis self-attention architecture. The different stages of multi-axis self-attention for [4, 4, C] input with the block size of b 2. The input is first blocked into 2 non-overlapping [2, 2, C) patches. Then regional and dilated self-attention operations are computed along two different axes, respectively, each of which uses a half of attention heads. The attention operations run in parallel for each of the tokens and their corresponding attention regions, illustrated with different colors. The spatial dimensions after attention are the same the input image.
divided into non-overlapping blocks where each block can be considered a local patch. After being combined with learnable positional encoding, each block is processed independently via a shared attention module. We enhance the local celf-attention of Nested Transformer by the proposed multi- axis blocked self-attention that can produce a richer feature representation by explicitly considering local (within blocks) as well as global (across blocks) relations. We denote the overall architecture of these stages as multi-axis Nested Transformer.
Assuming that spatial dependency is well modeled in the low-resolution stages, the high-resolution stages can focus on synthesizing pixel-level image details purely based on the local features. Thus in the high-resolution stage, we remove all self-attention modules and only maintain MLPs which can further reduce computation complexity. The resulting architecture in this stage can be viewed as an implicit neural function conditioned on the given latent feature well as positional information.
We further enhance HiT by adding a cross-atted cross-attention module at the beginning of each stage which allows the network to directly condition the intermediate features the initial input latent code. This kind of self modulationn layer leads to improved generative performance, especially when self- attention modules descriptions for the two main architectural components of HiT: (i) multi-axis blocked self-attention module and (ii) cross-attention module, respectively.
absent in high-resolution stages. In the following sections, we provided detailed
3.2 Multi-Axis Blocked Self-Attention
Different from the blocked self-attention in Nested Transformer (73, the proposed multi-axis blocked self-attention performs attention on more than a single axis. The attentions performed in two axes correspond to two forms of sparse self-attention, namely regional attention and dilated attention. Regional attention follows the spirit of blocked local self-attention [37 62 where tokens attend to their neighbours within non-overlapped blocks. To remedy the loss of global attention. dilated attention captures long-range dependencies between tokens across blocks: it subsamples attention regions in manner similar dilated convolutions with fixed stride equal to the block size. Figure illustrates an example of these two attentions.
To be specific, given an input of image with size (H, W. C), it blocked into a tensor X of the shape (b x b, H x W. C) representing (b, b) oveelapalapping blocks each with the size of (# W Dilated attention mixes information along the first axis of X while keeping information along other axes independent; regional attention works in analogous manner over the second axis of X. They are straightffrward implement: attention over the i-th axis of can be implemented by einsum operation which is available in most deep learning libraries.

---

We mix regional and dilated attention in a single layer by computing them in parallel, each of which uses a half of attention heads. Our method can be easily extended to model more than two axes by performing attention on each of the axis in parallel. Axial attention 17 20 65 can be viewed as special case of our method, where the blocking operation before attention is removed. However, we find blocking is the key to achieve significant performance improvement in the experiment. This is because the blocked structure reveals a better inductive bias for images. Compared with [7 37 65 where different attention modules are interleaved in consecutive layers, our approach aggregates local and global information in a single round, which is not only more flexible for architecture design but also shown to yield better performance than interleaving in our experiment.
Balancing attention between axes. In each multi-axis blocked self-attention module, the input feature is blocked in a balanced way such that we have b x 6 = 4. x K. This ensures that regional and dilated attention is computed on an input sequence of a similar length, avoiding half of the attention heads are attended to a too sparse region. In general, performing dot-product attention between two input sequences of length N H W requires a total of ( computation. When computing the balanced multi-axis blocked attention
image with the block size of S, i.e. S == VN, we perform attention on S sequences of length S, which is a total of (SS-5²) = O(NVN) computation. This leads an O saving in computation over standard self-attention.
3.3 Cross-Attention for Sell-Modulation
To further improve the global information flow, we propose to let the intermediate features of the model directly attend to a small tensor projected from the input latent code. This is implemented via a cross-attention operation and can be viewed form of self-modulation [8. The proposed technique has two benefits. First, as shown in [8], self- modulladtntionn stabilizes the generator towards favorable conditioning values and also appears to improve mode coverage. Second, when self-attention modules are absent in high-resolution stages, attending to the input latent code provides an alternative way to capture global information when generating pixel-level details.
Formally, let X, be the first-layer feature representation of the I-th stage. The input latent code 2 is first projected into 2D spatial embedding Z with the resolution of Hz Wz and dimension of Cz by linear function. X₁ is then treated as the query and Z as the key and value. We compute their cross-attention following the update rule: X MHA (X,, Z + Pz). where MHA represents (HHA(X2,
the standard mulit-head self-attention, X is the output, and Pz is the learnable positional encoding having the same shape as Z. Note that 2 is shared across all stages. For an input feature with the sequence length of N, the embedding size is a pre-defined hyperparameter far less than N (i.c., H2* x Wz < N). Therefore, the resulting cross- cross-attention operation has linear complexity O(N).
In our initial experiments, we find that compared with cross-attention, using AdaIN 19 and modu- lated layers 28 for Transformer-basee generators requires much higher memory cost during model training, which usually leads to out-of-memory errors when the model is trained for generating high- resolution images. As result, related work like ViT-GAN [32), which uses AdaIN and modulated layers, can only produce images up to the resolution of 64 x 64. Hence, we believe cross-attention is a better choice for high-resolution generators based on Transformers.
4 Experiments
4.1 Experiment Setup
Datasets, We validate the proposed method on three datasets: ImageNet [49], CelebA-HQ (25). and FFHQ (28.. ImageNet (LSVRC2012) [49 contains roughly 1.2 million images with 1000 distinct categories and we down-sample the images to 128 x 128 and 256 x 256 by bicubic interpolation. We use random crop for training and center crop for testing. This dataset is challenging for image generation since it contains samples with diverse object categories and textures. We also adopt ImageNet as the main test bed during the ablation study.
CelebA-HQ 25 is high-quality version of the CelebA dataset 38 containing 30,000 of the facial images at 1024 x 1024 resolution. To align with (25, we use these images for both training and evaluation. FFHQ 2X includes vastly more variation than CelebA-HQ in terms of age, ethnicity and image background, and also has much better coverage of accessories such as eyeglasses, sunglasses,
5

---

and hats. This dataset consists of 70,000 qugh-tualtty images at 1024 x 1024 resolution, out of which we use 50,000 images for testing and train models with all images following [28]. We synthesize images on these two datasets with the resolutions of 256 x 256 and 1024 x 1024.
Evaluation metrics. We adopt the Inception Score (IS) 151 and the Fréchet Inception Distance (FID) [16) for quantitative evaluation. Both metrics are calculated based on a pre-trained Inception-v3 image classifier (56). Inception score computes KL-divergence between the real image distribution and the generated image distribution given the pre-trained classifier. Higher inception scores mean better image quality. FID is a more principled and comprehensive metric, and has been shown to be more consistent with human judgments of realism [16] (0]. Lower FID values indicate closer distances between synthetic and real data distributions. In our experiments, 50,000 samples are randomly generated for each model to calculate the inception score and FID on ImageNet and FFHQ, while 30,000 samples are produced for comparison on CelebA-HQ. Note that we follow [5T] to split the synthetic images into groups (5000 images per group) and report their averaged inception score.
4.2 Implementation Details
Architecture configuration. In our implementation, HiT starts from an initial feature of size 8 x 8 projected from the input latent code. We use pixel shuffle [53] for upsampling the output of each stage, as we find using nearest neighbors leads to model failure which aligns with the observation in Nested Transformer [73]. The number of low-resolution stages is fixed be 4 as a good trade-off between computational speed and generative performance. For generating images larger than the resolution of 128 x 128, we scale HiT to different model capacities - small, base, and large, denoted as HiT-(S, B, L). We refer to the supplementary materials for their detailed architectures.
It is worth emphasizing that the flexibility of the proposed multi-axis blocked self-attention makes it possible for us to build smaller models than (37]. This is because they interleave different types of attention operations and thus require the number of attention blocks in single stage to be even (at least 2). In contrast, our multi-axis blocked self-attention combines different attention outputs within heads which allows us to use an arbitrary number (e.g., 1) of attention blocks in model.
Training details. In all the experiments, we use a ResNet- based discriminator following the archi- tecture design of (28) Our model is trained with standard non-saturating logistic GAN loss with Ri gradient penalty 40 applied to the discriminator. R1 penalty penalizes the discriminator for deviating from the Nash-equilibrium by penalizing the gradient on real data alone. The gradient penalty weight is set to 10. Adam 29 is utilized for optimization with 31 = and 3/ = 0.99. The learning rate is 0.0001 for both the generator and discriminator. All the models are trained using TPU for one million iterations on ImageNet and 500,000 iterations on FFHQ and CelebA-HQ. for the image resolution of 128 x 128 and 256 x 256 while to 32 for the resolution of 1024 x 1024. To keep our setup simple, we do not employ any training tricks for GAN training, such as progressive growing, equalized learning rates, pixel normalization. noise injection, and mini-batch standard deviation that recent literature demonstrate to be cruicial for obtaining state-of-the-ar results 25 28 When training on FFHQ and HQQ. we utilize balanced consistency regularization (bCR) 72 77 for additional regularization where images are augmented by flipping, color, translation, and cutout as 76 bCR enforces that for both real and generated images, two sets of augmentations applied to the same input image should yield the same output. bCR is only added to the discriminator with the weight of 10. We also decrease the learning rate by half to stabilize the training process when bCR is used.
We set the mini-batch size 256
4.3 Results on ImageNet
Unconditional generation. We start by evaluating the proposed HiT on the ImageNet 128 x 128 dataset, targeting the unconditional image generation setting for simplicity. In addition to recently reported state-of-the-an art GANs, we implement ConvNet--asse generator following the widely-used architecture from [69] while using the exactly same training setup of the proposed HiT (e.g., losses and R1 gradient penalty) denoted as ConvNet- R3 for a fair comparison. The results are shown in T We can see that HiT outperforms the previous ConvNet- based methods by a notable margin in terms of both IS and FID scores. Note that as reported in [1I, BigGAN 4 achieves 30.91 FID on unconditional ImageNet, but its model is far larger (more than 70M parameters) than HiT (around 30M parameters). Therefore, the results are not directly comparable. Even though, HiT (30.83 FID)

---

Table 1: Comparison with the state-of-the-ase meth- ods on the ImageNet 128 128 dataset. t based on supervised pre-trained ImageNet classifier.
Table 2: Reconstruction FID on the Ima-
geNet 256 256 dataset. We note that VQ- VAE-2 utilizes a hierarchical organization of VQ- VAE and thus has two codebooks Z.
Method
FID L
IS 1
Vanilla GAN 15
54.17
14.01
Method
Embedding size and |
FID
PacGAN2 (35
57.51
13.50
MGAN [8]
58.88
13.22
VQ- VQ-VAE [61]
32, 1024
75.19 34.30
Logo-GAN-AE
50
50.90
14.44
DALL-E 47
32, 8192
Logo-GAN-RC
50
38.41
18.86
64, 512
SS-GAN (sBN)
9
43.87
VQ-VAE-2
48
10.00
32, 512
Self-Conditioned GAN 36
40.30
15.82
VQGAN 14
16, 1024
8.00
ConvNet-
37.18
19.55
HiT (Ours)
30.83
21.64
VQ-HiT (Ours)
16, 1024
6.37
Figure 3: Unconditional image generation results of HiT trained on ImageNet 128 x 128.
is slightly better than BigGAN and achieves the state of the art among models with a similar capacity as shown in Table [ We also note that 88 36) leverage auxiliary regularization techniques and our method is complementary to them. Examples generated by HiT on ImageNet are shown in Figure 3 from which we observe nature visual details and diversity.
Reconstruction. We are also interested the reconstruction ability of the proposed HiT and evaluate by employing
decoder for the vector quantised variational auto-encoder (VQ-VAE (61), state-of-the-aat approach for visual representation learning. In addition to the reconstruction loss and adversarial loss, our HiT-based VQ-VAE variant, namely VQ-HiT, also trained with the perceptual loss (24) following the setup of [4 78 Please refer to the supplementary materials for more details on the architecture design and model training. We evaluate the metric of reconstruction FID on ImageNet 256 x 256 and report the results in Table 2 Our VQ-HiT attains the best performance while providing significantly more compression (i.e., smaller embedding size and fewer number of codes in the codebook Z) than 47 48
4.4 Higher Resolution Generation
Baselines. To demonstrate the utility of our approach for high-fidelity images, we compare to state- of-the-art techniques for generating images on CelebA-HQ and FFHQ focusing on common resolutions of 256 x 256 and 1024 x 1024. The main competing method of the proposed HiT is StyleGAN 27 28 a hypernetwork- based ConvNet archieving the best performance on these two datasets. On the FFHQ dataset, apart from our ConvNettbaseed counterparts, we also compare to most recent INR-based methods including CIPS [1 and INR-GAN (54) These two models are closely related to HiT the high-resolution stages of our approach can be viewed as form of INR. Following the protocol of [ we report the results of StyleGAN2 28 trained without style-mixing and path regularization of the generator this dataset.
7

---

Table 3: Comparison with the state-of-the-ar methods on CelebA-HQ (left) and FFHQ (right) with
the resolutions of 256 256 and 1024 x 1024. bCR is not applied the 1024 1024 resolution.
FID (CelebA-HQ) x1024
FID J (FFHQ) X
Method
x256
Method
x1024
VAEBM
68
20.38
U-Net GAN 152
7.63
StyleALAE 45
19.21
StyleALAE
45
13.09
PG-GAN 25
8.03
VQGAN 14
11.40
COCO-GAN 33
9.49
INR-GAN [54]
9.57
16.32
VQGAN 14
10.70
CIPS II
4.38
10.07
StyleGAN 27
5.17
StyleGAN2 28
3.83
4.41
HiT-B (Ours)
3.39
8.83*
HiT-B (Ours)
2.95
6.37*
Table 4: Comparison with the main competing methods terms of number of network parameters, throughput, and FID on FFHQ 256 x 256. The throughput is measured single Tesla V100 GPU.
#params (million)
Throughput (images sec)
FID (FFHQ x
Architecture
Model
ConvNet
StyleGAN2 28
30.03
95.79
3.83
CIPS II
45.90
27.27
4.38
INR
INR-GAN
54
107.03
266.45
9.57
HiT-S
38.01
86.64
3.06
Transformer
HiT-B
46.22
52.09
2.95
HiT-L
97.46
20.67
2.58
Results. We report the results in Table Impressively, the proposed HiT obtains the best FID scores the resolution 256 256 and sets the new state the art on both datasets. Meanwhile, performance is also competitive at the resolution of 1024 1024 but only slightly shy of StyleGAN. This is due our finding that conventional regularization techniques such as [77] cannot be directly applied to Transformer-basee architectures for synthesizing ultra high- resolutin images. We believe this triggers a new research direction and will explore it as our future work. It is also worth mentioning that our method consistently outperforms INR-based models, which suggests the importance of involving the self-attention mechanism into image generation.
Table 4 provides a more detailed comparison to our main competing methods in terms of the number of parameters, throughput, and FID on FFHQ 256 x 256. We find that the proposed HiT-S has comparable runtime performance (throughput) with StyleGAN while yielding better generative results. More importantly, FID scores can be further improved when larger variants HiT are employed. Figure Jillustrates our synthetic face images on CelebA-HQ.
4.5 Ablation Study
We then evaluate the importance of different components our model by its ablation on ImageNet 128 x 128. A lighter-weight training configuration is employed in this study to reduce the training time. We start with a baseline architecture without any attention modules which reduces to a pure implicit neural function conditioned on the input latent code 3 30) We build upon this baseline by gradually incorporating cross- attention and self-attention modules. We compare the methods that perform attention without blocking (including standard self-attention (63) axial attention [17 655
and with blocking (including blocked local attention [22 73 )) We also compare a variant of our method where different types of attention are interleaved other than combined in attention heads.
The results are reported in Table As we can see, the latent code conditional INR baseline cannot fit training data favourably since it lacks an efficient way to capture the latent code during the generative process. Interestingly, incorporating the proposed cross-attention module improves the baseline and achieves the state of the art, thanks to its function of self- saff-modulation. More importantly, we find that blocking vital for attention: performing different types of attention after blocking can all improve the performance by notable margin while the axial attention cannot. This is due the fact that the

---

IV
UV
Figure 4: Synthetic face images by HiT-B trained on CelebA-HQ 1024 1024 and 256 x 256.
Table 5:
conditioned the input latent Ablation study. We start with the INR-based generator 30
code and gradually improve it with the proposed attention components and their variations. O/M denotes "out-of-memory" error: the model cannot be trained for the batch size of one.
#params (million)
Throughput (images sec)
Model configuration
FID +
IS
Latent-code conditioned INR decoder
42.68
110.39
56.33
16.19
Cross-attention for self- reff-modulation
61.55
82.67
35.94
19.42
All-to-all self-attention 63
67.60
O/M
O/M
+ one of
Axial attention
67.60
74.21
35.15
19.79
Blocked local attention
33.70
19.96
Interleaving blocked regional and dilated attention
67.60
75.54
32.96
20.75
Multi-axis blocked self-attention (Ours)
32,23
20.96
Balancing attention between axes (Full model)
67.60
75.33
31.87
21.32
image structure after blocking introduces better inductive bias for images. At last, we observe that the proposed multi-axis blocked self-attention yields better FID scores than interleaving different attention outputs, and it achieves the best performance after balancing the attention axes.
In T we investigate another form of our variations, where we incorporate attention in different number of stages across the generator. We can see that the more in the stack the attention is applied (lowrresolution stages), the better the model performance will be. This study provides a validation for the effectiveness of self-attention operation in generative modeling. However, for feature resolutions larger than 64 64, we not observe obvious performance gains brought by self-attention operations but a notable degradation in both training and inference time.
4.6 Effectiveness of Regularization
We further explore the effectiveness of regularization for the proposed Transformer-based generator. Note that different from the recent interest in training GANs in the few-shot scenarios [26] 76),, we study the influence regularization for ConvNet-bor -assdd and Transformer-base data regime. We use the whole training set of FFHQ 256 256, and compare HiT-based variants with StyleGAN2 (26). The regularization method bCR [77). As shown in Table 7 all variants of HiT achieve much larger margins of improvement than StyleGAN2. This confirms the finding 12 23 60 that Transformer- based architectures are much more data- hatt-h than ConvNets both classification and generative modeling settings, and thus strong data augmentation and regularization techniques are crucial for training Transformer based generators.
generators in the full-
Conclusion
present HiT, novel Transformer-based generator for high-resolution image generation based on GANs. To address the quadratic scaling problem of Transformers, we structure the low-resolution stages HiT following the design of Nested Transformer and enhance by the proposed multi-axis blocked self-attention. In order to handle extremely long inputs in the high-resolution stages,

---

Table 6: Performance as function of the number of self-attention stages on ImageNet 128 x 128. The attention configuration is defined using the protocol Ja. b), where a and refer to the number of stages in the low-resolution and high-resolution stages of the model, respectively.
Attention configuration
[0, 5]
[1,4] 4]
[2,3]
[3, 2]
[4, 1]
#params (million)
61,55
66.01
67.19
67.52
67.60
Throughput (images sec)
82.67
80.88
80.22
78.06
75.33
FID +
35.94
34.16
33.69
32.72
31.87
Table The effectiveness of bCR 77 on both StyleGAN2 and HiT-based variants. indicates the results of StyleGAN2 are obtained from [26] which uses a lighter-weight configuration of [28).
+ bCR
777
StyleGAN2 28
HiT-S
HiT-B
HiT-L
X
5.28
6.07
5.30
5.13
3.91
3.06
2.95
2.58
A FID
1.37
3.01
2.35
2.55
drop self-attention operations and reduce the model into implicit functions. We further improve the model performance by introducing cross-attention module which plays role of self-modulation Our experiments demonstrate that HiT achieves highly competitive performance for high-resolution image generation compared with its ConyNet-based counterparts. For future work, will investigate Transformer- based architectures for discriminators in GANs and efficient regularization techniques for Transformer-basee generators synthesize ultra high-resolution images.
Ethics statement. We note that although this paper does not uniquely raise any new ethical challenges, image generation is field with several ethical concerns worth acknowledging. For example, there are known issues around bias and fairness, either in the representation of generated images [39] or the implicit encoding of stereotypes 55 Additionally, such algorithms are vulnerable to malicious use (5), mainly through the development deepfakes and other generated media meant to deceive public [67],
image denoising technique that could be used for privacy-ivvading surveillance monitoring purposes. We also note that the synthetic image generation techniques have the potential to mitigate bias and privacy issues for data collection and annotation. However, such techniques could be misused to produce misleading information, and researchers should explore the techniques responsibly.
Acknowledgements
We thank Chitwan Saharia, Jing Yu Koh, and Kevin Murphy for their feedback to the paper. We also thank Ashish Vaswani, Mohammad Taghi Saffar, and the Google Brain team for research discussions and technical assistance. This research has been partially funded by the following grants ARO MURI 805491, NSF IIS- 1793883, NSF CNS-1747778. NSF IIS 1763523, DOD-ARO ACC-W91 NSF 0IA-2040638 to Dimitris N. Metaxas.
References
[1] Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. Image generators with conditionally inddeeedeet pixel synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[2] Iz Beltagy, Matthew E Peters. and Arman Cohan. Longformer: The long-document transformer. arXiv
preprint arXiv:2004.05) 05500, 150, 2020.
[3] Tristan Bepler, Ellen D Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger. Explicitly disentan- gling image content from translation and rotation with spatial-Van VAE. In Advances in Neural Information Processing Systems (NeurIPS), pages 15409 15419, 2019.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proceedings of the International Conference on Learning Representations (ICLR). 2019.
10

---

[5] Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigationn. arXiv preprini (rXivv8002.02226 2018.
[6] Jiezhang Cao, Yawei Li, Kai Zhang, and Luc Van Gool. Video Super-Resolution Transformer. arXiv
preprint 100 06847, 2021.
[7] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Vios Cross-attention multi-scale vision trans- former for image classification. arXiv preprint arXiv:2103. 00..11...49999 14899, 2021.
[8] Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[9] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised GANs via auxiliary rotation loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12154 12163, 2019.
[10] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature lcarning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 215-223, 2011. 2011
[11] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems (NewrIPS), 2019.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations (ICTR), 2021.
[13] Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of functions. arXiv preprint arXiv22102.04777 2021.
[14] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high- nigh-resolution image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[15] Tan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farlcy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems (NeurIPS), 2014.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhaar Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), pages 6629-6640, 2017.
[17] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint 1tti::1992.12/1819 2019.
[18] Quan Hoang, Tu Dinh Nguyen. Trung Le, and Dinh Phung. MGAN: Training generative adversarial nets with multiple generators. In Proceedings of the International Conference on Learning Representations (ICLR), 2018
[19] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1501-1510, 2017.
[20] Zilong Huang. Xinggang Wang, Lichao Huang, Chang Huang. Yunchao Wei, and Wenyu Liu. CCNet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 603-612, 2019.
[21] Drew A Hudson and C Lawrence Zitnick. Generative adversarial transformers. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
[22] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman. Oriol Vinyals. and Joao Carreira. Perceiver: General pereeption with iterative attention. arXiv preprint arXiv:2103.03206 2021.
[23] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. TransGAN: Two transformers can make one strong
GAN. arXiv preprint Xr:: 20..200000000 102.07074. 2021.
[24] Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Proceedings of the European Conference on Computer Vision (ECCV), pages 694-711, 2016.
[25] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[26] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
11

---

[27] Tero Karras, Samuli Laine. and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 4401-4410, 2019.
[28] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 8110-8108 8119, 2020.
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICIR), 2014.
[30] Marian Kleineberg, Matthias Fey, and Frank Weichert. Adversarial generation of continuous implicit shape representations. In Eurographics, 2020.
[31] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[32] Kwonjoon Lee, Huiwen Chang. Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. ViTGAN: Training GANs with Vision Transformers. arXiv preprint arXiv:2107 10Xiv:2107.04588 2021.
[33] Chich Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da- Da-Cheng Juan, Wei Wei, and Hwann-Tzong Tzong Chen. COCO-GAN: Generation by Parts via Conditional Coordinating. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4512 -521,, 2019.
[34] Chich Hubert Lin, Hsin- Hsin-Ying Lee, Yen-Chi Cheng, Sergey Tulyakov, and Ming- Hsuan Yang. InfinityGAN: Towards adfinie-resolutte image synthesis, arXiv preprint arXiv:2104. 0346333,,,, 2021.
[35] Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. PacGAN: The power of two samples in generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS). 2018.
(36] Steven Liu, Tongzhou Wang, David Bau, Jun- Yan Zhu, and Antonio Torralba. Diverse image generation via seff-conditioned GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14286-14295, 2020.
[37] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 140400 2021.
[38] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2015.
[39] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. PULSE: Self-supervised photo upsampling via latent space exploration of generative models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages
2020.
[40] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In Proceedings of the International Conference on Machine Learning (ICML), pages 348 133490, 2018.
[41] Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 4460-4470, 2019.
[42] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pages 405-421, 2020.
[43] Jeong Joon Park, Peter Florence. Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 165-174, 2019.
[44] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer. Alexander Ku, and Dustin Tran. Image transformer. In Proceedings of the International Conference Machine Learning (ICML), pages 4055 4064, 2018
[45] Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14104-14113 2020.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 00000000........000000002200 2021.
[47] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092. 2021.
[48] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high- fidelity images with VQ- VAE-2. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
12

---

[49] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)2211-25) 2015.
[50] Alexander Sage, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. Logo synthesis and manipulation with clustered generative adversarial networks. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5879-5888, 2018.
[51] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[52] Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. U-Net Based Discriminator for Generative Adversarial Networks. In Proceedings the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8207-8218 8216, 2020.
[53] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and vidco convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition (CVPR), pages 1874-1883, 2016.
using an efficient sub-pixel
[54] Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[55] Ryan Steed and Aylin Caliskan. Image representations leareed with unsupervised pre-training contain human-like biases. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, pages 701-713, 2021.
[56] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2818-2826, 2016.
[57] Yu Tian, Xi Peng. Long Zhao, Shaoting Zhang, and Dimitris N Metaxas. CR-GAN: Learning complete representations for multi-view generation. In Proceedings of the International Joint Conference on Artificial Intelligence (ICAII. pages 942-948, 2018
[58] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Daniel Keysers, Jakoh Uszkoreit, Mario Lucic, et al. MI.P-Mixer: An MLI-ML.P Architecture for Vision. arXiv preprint arXiv:2 105.01601, 2021.
[59] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. ResMLP: Feedforward networks for image classification with data data-efficient training. arXiv preprint arXiv: 200: 03404, 2021.
[60] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[61] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeuriPS), pages 5309-6318. 2017.
[62] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backhones. In Proceedings of the IEFF Conference on Computer Vision and Patiern Recognition (CVPR), 2021.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), pages 6000-6010 2017.
[64] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. MaX-DeepLah: End-to-end panoptic segmentation with mask transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[65] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chiel Chen. Axial- DeepLab: Stand-alone axial-attention for panoptic segmentation. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pages 108-126, 2020.
[66] Xiaolong Wang, Ross Girshick. Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7794- 7803, 2018
[67] Mika Westerlund. The emergence of deepfake technology: A review. Technology Innovation Management Review, 911). 2019.
[68] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. VAEBM: A Symbiosis between Variational Autoencoders and Energy- based Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
13

---

[69] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proceedings of the International Conference Machine Learning (ICML), pages 7354-7363, 2019.
[70] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for rext-to-image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[71] Han Zhang. Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolci Huang, and Dimitris N Mctaxas. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 5907-5915, 2017.
[72] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[73] Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, and Tomas Pfister. Aggregating nested transformers. arXiv preprint arXiv:2105, 12723, 2021.
[74] Long Zhao, Peng, Tian, Mubbasir Kapadia, and Dimitris Metaxas. Learning to forecast and refine residual motion for image-to-vides video generation. In Proceedings of the European Conference on Computer Vision (ECCV), pages
2018.
[75] Long Zhao. Xi Peng. Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas. Towards image-to to-video translation: A structure-aware approach via multi-stage generative adversarial networks. International 2020.
Journal of Computer Vision, 128( (0010002:4422555
[76] Shengyu Zhao, Zhijian Liu, Lin, Jun Zhu, and Song Han. Differentiable augmentation for data- efficient gan training. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[77] Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for GANs. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
[78] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolci Zhou. In-domain GAN inversion for real image editing. In Proceedings of the European Conference Computer Vision (ECCV), pages 592-608, 2020.
[79] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2223 2232, 2017.
14


Reviews:
[{"review_id": "nuhWYIg2Hhp", "reviewer": "Reviewer_QCfs", "summary": "To address the quadratic complexity of the self-attention operation, this paper proposes a new Transformer-based generator for high-resolution image generation based on GANs, denoted as HiT. In the low-resolution stage, the authors propose a multi-axis blocked self-attention. In the high-resolution stage, they keep multi-layer perceptrons reminiscent of the implicit neural function. Extensive experiments demonstrate the effectiveness of the proposed method. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Some motivations should be highlighted. What are the issues of directly using ViT into GAN? In this paper, the authors do not directly apply ViT in GAN. However, there are some papers that propose ViT-based GAN methods. In the low-resolution stage, why replace standard global self-attention. In the high-resolution stage, why drop self-attention in the proposed method?\n\n2. Some related work should be discussed in the paper. There are some papers [1, 2] that propose ViT-based GAN methods. Could you please discuss the difference between them? In addition, one recent paper [3] proposes new attention to generate high-resolution images. Could you please discuss it in the related work?\n\n    [1] TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up, arxiv 2021\n\n    [2] ViTGAN: Training GANs with Vision Transformers, arxiv 2021\n\n    [3] Video Super-Resolution Transformer, arxiv 2021\n\n3. The novelty of the proposed method should be highlighted.  The proposed method seems to combine the Nested Transformer and the Axial attention. The authors should highlight that such a combination is non-trivial. \n\n4. Some technical details are not clear. In Figure 1, what is block $B_i$? Are the two $B_0$ the same in the figure? In Line 154, why $b \\times b \\sim H/b \\times W/b$? In Lines 157 and 174, could you please give a more detailed analysis of complexity?\n\n5. The experiments can be improved. In Table 3, it would be better to compare with BigGANs [4].  In the ablation study,  the authors should compare with the Nested Transformer.\n\n[4] Large Scale GAN Training for High Fidelity Natural Image Synthesis, ICLR 2019\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "mzWzQ01gA5P", "reviewer": "Reviewer_TdeG", "summary": "This paper proposes a new transformer-based generator for high-resolution image generation. It addresses the quadratic scaling problem of the attention operator with multi-axis blocked self-attention which considering the within blocks and across blocks attention. The proposed cross-attention for input and intermediate features is also novel. The evaluation is done in ImageNet and FFHQ datasets.  Promising results are provided. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written paper with carefully designed experiments. The motivation for utilizing the multi-axis blocked self-attention is reasonable. The proposed cross-attention for input and intermediate features is novel and interesting. The authors conduct various experiments to show the effectiveness of the proposed results, both quantitatively and qualitatively.\n\nLine 12: \"Hit has a linear computational complexity\" is a bit overclaim. Since the full HiT model uses balancing attention between axes and its computational complexity is N*sqrt(N).\n\nWhy not apply bCR at 1024x1024 resolution? Is not using bCR the reason why HIT is worse than StyleGAN2 at the 1024x1024 resolution?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors have addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "l8GjxrFjBnz", "reviewer": "Reviewer_6oDm", "summary": "The paper proposes a transformer based generative model which shows competitive results on multiple image generation benchmarks. It combines existing techniques of blocked and axial attention to reduce the computational complexity of self-attention mechanism. To further reduce the complexity, the proposed attention mechanism is only used at lower resolution blocks. The paper also employs technique similar to self-modulation to condition image generation process on the input latent code at all levels. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper combines two existing techniques of blocked and axial attention in a novel way to create a convolutional free generator. The final proposed architecture shows competitive or better image generation compared to existing state of the art techniques on multiple datasets according to the FID metric.  \n\n**Clarity**: The paper is easy to follow and well written. \n\n**Additional comments and questions**:  \n* The paper proposes cross-attention which is similar to self-modulation technique. It would be good to see an ablation experiment that shows the performance of using self-modulation instead of cross-attention and its added advantage.   \n* Will the proposed architecture also provide benefit in case of conditional image generation?   \n* In Table 5, the paper only shows the result with cross-attention+blocked axial attention, It would be interesting to see the benefit when only blocked axial attention is employed without using cross-attention. \n* In Table 6, does having blocked axial attention for all layers (i.e. configuration [5,0]) leads to high memory requirement as it is not shown? \n* Though the paper proposes convolutional free generator and therefore has MLPs at the higher resolution blocks, but will having convolutional blocks instead of MLPs at later blocks provide any benefit? \nDoes the ConvNetR1 baseline includes self-attention layer at one the generator blocks similar to self-attention GANs? \n\n**Related work missing**: One of the relevant related work \"Self-supervised gans via auxiliary rotation loss.\" is missing and would be great to see a fair performance comparison with this as the baseline. This above work shows 23.6 FID metric on unconditional image generation on ImageNet 128x128 dataset which is better than the score reported by the proposed technique.\n\n####################\nUpdate: \n\nThanks for addressing all the comments in the rebuttal. I am adjusting my score based on the answers. \n\nAdditional comments: Are the results on ImageNet dataset with HiT-S architecture? If possible, it would be interesting to compare HiT-L (with similar #params as BigGANs) with BigGAN for ImageNet dataset in the final draft of paper.  \nAfter reading other reviewer’s comment, I agree about including attention visualization in the paper. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the paper touched upon the limitations and societal impact of the work in the conclusion. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6-7"}, "extra_text": {}}, {"review_id": "_aDLS3-PSF6", "reviewer": "Reviewer_Uc2m", "summary": "This paper has presented HiT, a Transformer-based generator for high-resolution image synthesis based on GANs. Instead of using full attention, this paper has proposed the multi-axis blocked self-attention, which captures local and global dependencies within non-overlapping image blocks in parallel. In addition, HiT introduces the cross-attention mechanism to integrate noise information into multi-stages of the generator. Extensive experiments demonstrate the superiority of the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:  This paper presents a new Transformer-based architecture for the generator in GANs. The authors introduce several new techniques to make it work for the Transformer-based generative model.\n\nQuality: This submission is technically sound and most claims are well supported.\n\nClarity: This paper is clearly written and well organized. \n\nSignificance: The experiment results are very important for exploring Transformer-based architecture for generative models.\n\nWeakness:\n1. I am curious about what is a good noise injection method in the Transformer-based generative models. Previous well-known methods StyleGAN and StyleGAN2 apply the AdaIN and modulated convolution for noise injection. HiT proposes the cross-attention for noise injection, the authors should conduct experiments to compare cross-attention with these methods, maybe AdaIN and modulated convolution also work for the Transformer-based generative model.\n\n2. Since the Transformer-based architectures are more data-hungry, maybe the authors can conduct experiments on large-scale datasets: LSUN-cats and LSUN-cars and compare the performance with StyleGAN2 without using bCR. \n\n3. Transformer-based architectures have shown effectiveness in classification models, one of the most important reasons is the capability of capturing long-range dependencies. But in HiT, the self-attention module is only applied in low-resolution stages, make the generator fails to capture long-range dependencies in high-resolution stages. The authors may give some explanations for self-attention modules in low-resolution stages are enough in generative high-quality images.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "_FGNADP7Pfu", "reviewer": "Reviewer_JC42", "summary": "The paper describes a purely attention-based GAN generator architecture where low-resolution deeper layers feature larger-scale (more distant) communication and higher-resolution layers synthesize locally using MLPs,  but still conditioned (indirectly) on the initial latent code.  State-of-the-art results are claimed for 128^2 unconditional ImageNet synthesis (but I believe this claim to be false, see below). In addition, results are shown for CelebA-HQ and FFHQ. While the architecture have some similarities to the near-concurrent work of Hudson and Zitnick (ICML 2021), lack of comparisons should not be held against this paper.\n\nI remain unconvinced by the argumentation and how the results support them. Visually high-quality results are only shown for the simple face image datasets, for which attention mechanisms do not appear to be as necessary as for more complex ones that feature scene compositions. I feel I do not walk away feeling that I learned something.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper argues for an all-attention GAN generator, where the quadratic cost of self-attention is mitigated by restricting explicit long-range dependency modeling to coarse, deep layers.\n\nMy largest problem with the paper is that the quest for getting rid of convolutions is not really motivated. We all know metrics like FID and IS only tell a very coarse story, but the submission does not really analyze the properties of the resulting architecture. How does it build the pictures? How is its functionality different from existing main types of model architectures. For instance, compare to the attention map visualizations and composiotionality studies of Hudson and Zitnick that I found very illuminating.\n\nThe paper claims a record FID=31.87 and IS=21.32 for unconditional 128x128 ImageNet synthesis. I believe this to be incorrect: the BigBiGAN paper (Donahue and Simonyan, Large Scale Adversarial Representation Learning, NeurIPS 2019) presents an unconditional 128x128 model (\"no \\mathcal{E} (GAN)\" in Table 1) with FID=30.91 and IS=23.56. The claim of \"outperforming previous ConvNet-based models with a notable margin\" (L235) does not appear to be justified.\n\nOverall, I find image synthesis architectures fascinating, and that the quest for how to best do it is definitely unsolved. While it is interesting that the problem can be solved using all-attention architectures, I am not convinced by arguments that rely solely on metric numbers without analysis of the model's properties (\"what makes it tick\"). Moreover, the best results here are presented on aligned face datasets that are known to function well on existing architectures. I would be more convinced seeing high-quality results on more complex datasets, supported also by analysis other than FID and IS.\n\nSmaller comments:\n\nThe dimensions have been chosen quite poorly in Fig. 2; I find it impossible to understand without consulting the text. The choice of b=2 does not match well with the choice of 4x4 inputs, as dimension=2 is now overloaded.\n\nThe authors do not present visualizations of the attention maps and how they relate to the synthesis results.\n\nAll synthesis results in Fig. 3 show serious aliasing artifacts. I suspect that the ImageNet dataset has been preprocessed incorrectly when downsizing to 128^2, a common error due to the poor design choices in current deep learning frameworks. (Note that e.g. TORCH.NN.FUNCTIONAL.INTERPOLATE does *not* perform correct filtering when downsampling; comparing its output to e.g. PIL.Image.resize with resample=PIL.Image.BICUBIC or Matlab's imresize will reveal clear differences.)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The required impact statement (end of Sec. 5) does not appear like a serious attempt.", "review_text": "", "extra_scores": {"ethical_concerns": "Nothing but the same things that apply to all image generation work, but the writeup in Sec. 5 is still just a stub.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a transformer-based generative model that shows better results on multiple standard benchmarks. The model includes multi-axis blocked self-attention at early stages and uses MLP for late stages. The reviewers appreciate the idea of a transformer-based generator, clear writing, and experimental results. There are also concerns about the STOA results on ImageNet (compared to BigBiGAN) and the motivation and advantage of transformer-based architecture compared to ConvNet. The rebuttal addressed some of the concerns. Overall, the significance of the work outweighs the limitations. I recommend accepting the work. The authors are encouraged to carefully examine the downsampling functions for 128x128 Imagenet (use antialias=True in TF, for example) and discuss the latest results (e.g., BigBiGAN) in ImageNet experiments. 
Decision: Accept (Poster)