ViTGAN: Training GANs with Vision Transformers
Authors: Kwonjoon Lee
Huiwen Chang
Lu Jiang
Han Zhang
Zhuowen Tu
Ce Liu
Venue: ICLR 2022 Spotlight (2022)

Abstract: Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases. In this paper, we investigate if such performance can be extended to image generation. To this end, we integrate the ViT architecture into generative adversarial networks (GANs). For ViT discriminators, we observe that existing regularization methods for GANs interact poorly with self-attention, causing serious instability during training. To resolve this issue, we introduce several novel regularization techniques for training GANs with ViTs. For ViT generators, we examine architectural choices for latent and pixel mapping layers to faciliate convergence. Empirically, our approach, named ViTGAN, achieves comparable performance to the leading CNN- based GAN models on three datasets: CIFAR-10, CelebA, and LSUN bedroom.

Published as a conference paper at ICLR 2022
VITGAN: TRAINING GANS WITH VISION TRANS- FORMERS
Kwonjoon Lee
Huiwen Chang²
Ce Liu' *Microsoft Azure Al [huiwenchang, lujiian, [aanghan]@googleee, com
Lu Jiang"
Han Zhang²
Zhuowen Tu
'CC San Diego Google Research
Honda Research Institute
kw10420eng ucsd. prssdesse edu
ztu@ucsd. edu ce we.lii@microooft..
ABSTRACT
Recently, Vision Transformers (ViTs) have shown competitive performance image recognition while requiring less vision-specific inductive biases. In this paper, we investigate if such performance can be extended image generation. To this end, we integrate the ViT architecture into generative adversarial networks (GANs). For ViT discriminators, we observe that existing regularization methods for GANs interact poorly with self-attention, causing serious instability during training. To resolve this issue, we introduce several novel regularization techniques for training GANs with ViTs. For ViT generators, we examine architectural choices for latent and pixel mapping layers to facilitate convergence. Empirically, our approach, named ViTGAN, achieves comparable performance to the leading CNN- based GAN models on three datasets: CIFAR-10 10, CelebA, and LSUN bedroom. Our code is available onlind"
INTRODUCTION
Convolutional neural networks (CNNs) (LeCun et al. 1989) are dominating computer vision today, thanks their powerful capability of convolution (weight-sharing and local-connectivity) and pooling (translation equivariance). Recently, however, Transformer architectures (Vaswani et al. 2017) have started to rival CNNs in many vision tasks.
In particular, Vision Transformers (ViTs) (Dosovitskiy et al., 2021), which interpret an image as a sequence of tokens (analogous to words in natural language), have been shown achieve comparable classification accuracy with smaller computational budgets (i.e., fewer FLOPs) on the ImageNet benchmark. Unlike CNNs, ViT's capture different inductive bias through self-attention where each patch is attended to all patches of the same image. ViTs, along with their variants (Touvron et al.) 2020 Tolstikhin et al. (022), though still in their infancy, have demonstrated advantages modeling non-local contextual dependencies (Ranfil et al. 2021 Sttudell et al. 2021) well as promising efficiency and scalability. Since their recent inception, ViT's have been used in various tasks such object detection (Beal et al. 2020), video recognition (Bertasius et al. 2021 Arnab et al. 20211). multitask pre- pre-training (Chen et al. 2020a),, etc.
In this paper, examine whether Vision Transformers perform the task of image generation without using convolution or pooling, and more specifically, whether ViTs can be used to train generative adversarial networks (GANs) with comparable quality to CNN-based GANs. While we can naively train GANs following the design of the standard ViT (Dosovitskiy et al.) 2021). find that GAN training becomes highly unstable when coupled with ViTs, and that adversarial training is frequently hindered by high-varianee gradients in the later stage of discriminator training. Furthermore, conventional regularization methods such as gradient penalty (Gulrajani al., 2017 Mescheder et al. 2018). spectral normalization (Miyato et al. 2018) cannot resolve the instability issue, even though they are proved to be effective for CNN-based GAN models (shown in Fig. 4 As unstable training is uncommon in the CNN-based GANs training with appropriate regularization, this presents a unique challenge to the design of ViT-based GANs.
https: thw.. http:///gtttuuu.oom...oo/./ wtwp:hhwwwwwwoo//coaaaaaaaaeeeee

---

Published as a conference paper at ICLR 2022
We propose several necessary modifications stabilize the training dynamics and facilitate the convergence of baT-b GANs. In the discriminator, we design an improved spectral normalization that enforces Lipschitz continuity for stabilizing the training dynamics. In the generator, propose two key modifications to the layer normalization and output mapping layers after studying several architecture designs. Our ablation experiments validate the necessity of the proposed techniques and their central role in achieving stable and superior image generation.
The experiments are conducted on three public image synthesis benchmarks: CIFAR-IO, CelebA, and LSUN bedroom. The results show that our model, named ViTGAN, yields comparable performance the leading CNN- based StyleGAN2 (Karras et al., 2020b Zhao al., 2020a) when trained under the same setting. Moreover, we are able to outperform StyleGAN2 by combining the StyleGAN2 discriminator with our ViTGAN generator.
Note that it is not our intention to claim ViTGAN is superior to the best pest-performin GAN models such as StyleGAN2 ADA (Karras al. (020a) which are equipped with highly-optimized hyper- parameters, architecture contigurations, and sophisticated data augmentation methods. Instead, our work aims to close the performance gap between the conventional CNN-based GAN architectures and the novel GAN architecture composed of vanilla ViT layers. Furthermore, the ablation in Table 4 shows the advantages of ViT's intrinsic capability (i.e., adaptive connection weight and global context) for image generation.
2 RELATED WORK
Generative Adversarial Networks Generative adversarial networks (GANs) (Goodfellow et al. 2014) model the target distribution using adversarial learning. It is typically formulated as a min- max optimization problem minimizing some distance between the real and generated data distri- butions, e.g., through various f-divergences (Nowozin et al. 2016) or integral probability metrics (IPMs) (Müller 1997 Song & Ermon 2020) such as the Wasserstein distance (Arjovsky et al.) 2017).
GAN models are notorious for unstable training dynamics. As result, numerous efforts have been proposed to stabilize training, thereby ensuring convergence. Common approaches include spectral normalization (Miyato et al. 2018), gradient penalty (Gulrajani et al. 2017 Mescheder et al. 2018
Kodali al. 2017). consistency regularization (Zhang al./ 2020 Zhao al.) 2021). and data
augmentation (Zhao et al., 2020a Karras et al. [020a]] Zhao et al. 2020b] Tran et al. (021). These
techniques are all designed inside convolutional neural networks (CNN) and have been only verified
in convolutional GAN models. However, we find that these methods are insufficient for stabilizing
the training of Transformer-batt different task of pretraining, This paper introduces several novel techniques to overcome the unstable adversarial training of Vision Transformers.
GANs. A similar finding was reported in (Chen et al./ (02I)) on
Vision Transformers Vision Transformer (ViT) (Dosovitskiy et al. 2021 a convolution-fre
Transformer that performs image classification over sequence of image patches. VIT demonstrates the superiority of the Transformer architecture over the classical CNNs by taking advantage of pretraining on large-scale datasets, Afterward, DeiT (Touvron et al.) 2020 improves ViTs' sample efficiency using knowledge distillation as well as regularization tricks. MLP-Mixer (Tolstikhin et al.. 2021) further drops self-attention and replaces it with MLP to mix the per-location feature. In parallel, VIT has been extended to various computer vision tasks such as object detection (Beal) al., 2202)), action recognition in video (Bertasius et al.] 202T) Arnab al. (202T), and multitask pretraining (Chen et al., 2020a). Our work is among the first to exploit Vision Transformers in the GAN model for image generation.
Generative Transformers in Vision Motivated by the success of GPT-3 (Brown et al. 2020) few pilot works study image generation using Transformer by autoregressive learning (Chen et al. 2020b Esser et al. (021)) or cross-modal learning between image and text (Ramesh et al. 2021 These methods are different from ours they model image generation as a autoregressive sequence learning problem. On the contrary, our work trains Vision Transformers in the generative adversarial training paradigm. Recent work of (Hudson & Zitnick| 2021), embeds (cross-)ate )attention module within the CNN backbone (Karras et al. 2020b) in a similar spirit to (Zhang et al., (2019). The closest work to ours is TransGAN (Jiang et al. 2021). presenting GAN model based on Swin Transformer
2

---

Published as a conference paper at ICLR 2022
Real / Fake
MUP
Fourier Embedding 1ou (x, y)
Generated Patches
Transformer Encoder
Transformer Encoder
Projection of Flattened Patches
Mapping Network
Overlapping Patches
Latent
Generator
Discriminator
Figure 1: Overview of the proposed VITGAN framework. Both the generator and the diseriminator are designed based on the Vision Transformer (ViT). Discriminator score is derived from the clas- sification embedding (denoted as [*] in the Figure). The generator generates pixels patch-by-pan based on patch embeddings.
backbone (Liu et al. 2021b). Our approach is complementary to theirs as we propose key techniques for training stability within the original ViT backbone (Dosovitskiy et al. 2021.
PRELIMINARIES: VISION TRANSFORMERS (VITs)
Vision Transformer (Dosovitskiy et al. 2021) is pure transformer architecture for image classifi- cation that operates upon a sequence of image patches. The 2D image x E R Hxx x is flattened into sequence of image patches, following the raster scan, denoted by Xp Nx( C) where HXW is the effective sequence length and P x P x C is the dimension of each image patch. N
p2
Following BERT (Devlin et al., 2019), a learnable classification embedding Xelass is prepended to the image sequence along with the added ID positional embeddings Epos to formulate the patch embedding hg- The architecture of ViT follows the Transformer architecture (Vaswani et al. 2017).
ho |eelassi E E
X E
Eposs
E R (P*.C)D
Epos (N+1)XD
h/ MSA(LN((((1)) he-1,
1....,,
MLP(LN(b()) +h/, y LN(h)))
Equation applies multi-headed self-attention (MSA). Given learnable matrices W We, W, corresponding to query, key, and value representations, single self-attention head is computed by:
QK
Attention, (X) softmax
V.
Vdn
where Q XW, K XW and V XW... Multi-headed self-attention aggregates infor- mation from // self-attention heads by means of concatenation and linear projection: MSA(X) concat #1 AAttention, (X) WW/W bb
METHOD
Fig. illustraaes the architecture of the proposed ViTGAN with VIT discriminator and a ViT-based generator. We find that directly using ViT as the discriminator makes the training volatile. We introduce techniques both generator and discriminator to stabilize the training dynamics and facilitate the convergence: (7) regularization on ViT discriminator and (2) new architecture for generator.
3

---

Published as a conference paper at ICLR 2022
4.1 REGULARIZING VIT BASED DISCRIMINATOR
Enforcing Lipschitzness of Transformer Discriminator Lipschitz continuity plays critical role in GAN discriminators. It was first brought to attention as a condition to approximate the Wasserstein distance in WGAN (Arjovsky al. 2017). and later was confirmed in other GAN settings (Fedus 2018 Miyato et al.] 2018: Zhang et al. 2019) beyond the Wasserstein loss. In particular, (Zhou
et al. et al. 2019 proves that Lipschitz discriminator guarantees the existence of the optimal discriminative
function as well as the existence of a unique Nash equilibrium. A very recent work (Kim et al., 20221. however, shows that Lipschitz constant of standard dot product self-attention (i.e., Equation 5) layer can be unbounded, rendering Lipschitz continuity violated in ViTs. To enforce Lipschitzness of our ViT discriminator, we adopt L2 attention proposed in (Kim et al. (02T). As shown in Equation 6 we replace the dot product similarity with Fuclidean distance and also tie the weights for the projection matrices for query and key in self-attention:
d(xw, q. XW dd
Attention, (X) = softmax
XWe. e, where Wg 9 = Was
(6)
W 9. Wk. k. and W. are the projection matrices for query, key, and value, respectively. d(::-) computes vectorized L2 distances between two sets of points. vd, is the feature dimension for each head. This modification improves the stability of Transformers when used for GAN discriminators.
Improved Spectral Normalization. To further strengthen the Lipschitz continuity, we also apply spectral normalization (SN) (Miyato et al. (0188) in the discriminator training. The standard SN uses power iterations to estimate spectral norm of the projection matrix for each layer in the neural network. Then it divides the weight matrix with the estimated spectral norm, so Lipschitz constant of the resulting projection matrix equals 1. We observe that making the spectral norm equal to stabilized the training but GANs seemed to be underfitting in such settings (c.f. Table 3b Similarly, we find RI gradient penalty cripples GAN training when ViT-based discriminators are used (c.f Figure 4 (Dong et al. 2021) suggests that the small Lipschitz constant of MI. block may cause the output of Transformer collapse
rank-1 matrix. Without the spectral normalization, our GAN model initially learns in a healthy manner but the training becomes unstable (later on) since we cannot guarantee Lipschitzness of Transformer discriminator.
We find that multiplying the normalized weight matrix of each layer with the spectral norm at initialization is sufficient to solve this problem. Concretely, we use the following update rule for our spectral normalization, where a computes the standard spectral norm of weight matrices:
WisN (isn(W)=:r == a (Winat) W/ W/o(W).
Overlapping Image Patches. VIT discriminators are prone to overfitting due to their exceeding learning capacity. Our discriminator and generator use the same image representation that partitions an image sequence of overllaalg patches according to a predefined grid P x P. These arbitrary grid partitions, not carefully tuned, may encourage the discriminator to memorize local cues and stop providing meaningful loss for the generator. We simple trick mitigate this issue by allowing some overlap between image patches (Liu et al. 2021b a). For each border edge of the patch, we extend it by pixels, making the effective patch size (P 20) x (P+20). 20).
This results in a sequence with the same length but less sensitivity to the predefined grids. It may also give the Transformer better sense of which ones are neighboring patches to the current patch, hence giving a better sense of locality.
Convolutional Projection. To allow Transformers to leverage local context as well as global context, we apply convolutions when computing Q K, V in Equation While variants of this idea were proposed in (Wu et al.) 1021 Guo et al. 2021), we find the following simple option works well: we apply 3 convolution after reshaping image token embeddings into a feature map of size и W. x D. We note that this formulation does not harm the expressiveness of the original P Transformer, as the original Q, K, V projection can be recovered by using the identity convolution kernel.
4

---

Published as a conference paper at ICLR 2022
Fourier Embrdding E._ y)
MLP
Norm
Transformer Encoder
Transformer Encoder
Transformer Encoder
Multi-Head Attention
WW
Norm
Mapping Network
Mapping Network
Mapping Network
Embedding
Latent z
Latent
Latent z
Self-Modulaten LayerNorm in Transformer Encoder
(A)
(B)
(C)
Figure Generator Architecture Variants. The diagram on the left shows three generator architectures we consider: (A) adding intermediate latent embedding w every positional embedding. (B) prepending w to the sequence, and (C) replacing normalization with self-modulated layernorm (SLN) computed by learned affine transform (denoted as A in the figure) from On the right, show the details of the self-mot moduulatutinn operation applied in the Transformer block.
4.2 GENERATOR DESIGN
Designing generator based on the VIT architecture is nontrivial task. A challenge is converting ViT from predicting set of class labels to generating pixels over a spatial region. Before introducing our model, we discuss two plausible baseline models, as shown in Fig. 12 [A)and) (B) Both models swap ViT's input and output to generate pixels from embeddings. specifically from the latent vector w derived from a Gaussian noise vector z by an MIP, Le.,
MI.P(z) (called mapping network (Karras et al. 2019) in Fig. 2). The two baseline generators differ in their input sequences. Fig, (A) takes as input a sequence of positional embeddings and adds the intermediate latent vector w to every positional embedding. Alternatively, Fig. 2 (B) prepends the sequence with the latent vector. This design is inspired by inverting ViT where w is used to replace the classification embedding h2 in Equation
To generate pixel values, a linear projection E (DP((P-OC) is learned in both models map D- dimensional output embedding to an image patch of shape P x P x C. The sequence of N HxW x image patches {x are finally reshaped to form an whole image x.
p2
These baseline transformers perform poorly compared the CNN-based generator. We propose novel generator following the design principle of VIT. Our VITGAN Generator, shown in Fig. (c). consists of two components (I) a Transformer block and (2) an output mapping layer.
ho Epos:
E Epos E NxD.
(8)
h': MSA(SLN( (he-hhhhh1 w)) + he-1,
... L, L.
(9)
h MLP(SLN((,) w)) h'',
f = L.
(10)
y SIN(h,, w) = [y' ..'.....""
y'....,y" ERD
(11)
[x]],
x = [fo y".,...., fo y)
x ERPx x HxWxC
(12)
The proposed generator incorporates two modifications facilitate the training.
Self-Modulated LayerNorm. Instead of sending the noise vector the input to VIT, we use to modulate the layernorm operation in Equation ( This is known as self-modulation (Chen al. 2019
since the modulation depends external information. The self-modulated layernorm (SLN) in Equation computed by:
h σ
SL, w) = SLN(h, MLP(z)) = Y
3x(w,,
(13)
where y and track the mean and the variance of the summed inputs within the layer, and n and 3 compute adaptive normalization parameters controlled by the latent vector derived from z. @ the plement-wise dot product.
5

---

Published as a conference paper at ICLR 2022
Table 1: Comparison to representative GAN architectures unconditional image generation benchmarks. "Results from the original papers. All other results are our replications and trained with DiffAug (Zhao et al.
2020a) bCR for fair comparison. means lower is better.
CIFAR 10
CelebA 64x64
LSUN 64x64
LSUN 128x128
Architecture
1
FID 4
IS
FID 4
IS
FID 4
IS
BigGAN+ DiffAug (Zhao et al.
2020a)
8.59*
9.25*
StyleGAN2 Karras
2020b)
5.60
9.41
3.39
3.43
2.33
2.44
3.26
2.26
TransGAN Jiang et al.
2021
9.02"
9.26"
Vinilla-ViT
12.7
8.40
20.2
2.57
218.1
2.20
ViTGAN (Ours)
4.92
9.69
3.74
3.21
2.40
2.28
2.48
2.26
StyleGAN2-D+ (Ours)
4.57
9.89
1.49
2.46
1.87
2.32
Implicit Neural Representation for Patch Generation.
We use implicit neural representa-
tion (Park et al. 2019 Mescheder et al.) 2019 Tancik et. 2020 Sitzmann et al. (020) to learn a
continuous mapping from a patch embedding y' E 12 to patch pixel values x; E ER'xC When coupled with Fourier features (Tancik et al., 2020) or sinusoidal activation functions (Sitzmann et al. 2020), implicit representations can constrain the space of generated samples to the space of varying natural signals. Concretely, similarly (Anokhin et al. 2021 x (E y")
where E fou E PR..D is Fourier encoding of P x P spatial locations and fo(:)) is 2-layer MLP. For details, please refer to Appendix D We find implicit representation to be particularly helpful for training GANs with ViT-based generators, c.f.Table 3a
EXPERIMENTS
We train and evaluate our model on various standard benchmarks for image generation, including CIFAR-10 (Krizhevsky et al. 2009). LSUN bedroom (Yu et al., 2013) and CelebA (Liu et al. (015). We consider three resolution settings: 32x32, 64x64, 128 x 128, and 256x256. We strive to use consistent setup across different resolutions and datasets, and as such, keep all key hyper-parameters except for the number of Transformer blocks and patch sizes, the same. For more details about the datasets and implementation, please refer to Appendix C and
5.1 MAIN RESULTS
Table shows the main results on three standard benchmarks for image synthesis. Our method compared with the following baseline architectures. TransGAN (Jiang et al. [022]) is the only existing convolution-free GAN that is entirely built on the Transformer architecture. Vanilla-Vi7 is ViT-based GAN that employs the generator illustrated in Fig. 2 (A) a vanilla ViT discriminator without any techniques discussed in Section 4.1 For a fair comparison, R1 penalty and bCR (Zhao ct al. 2021) + DiffAug (Zhao et al. 2020a) were used for this baseline. The architecture with the generator illustrated in Fig. (B) separately compared in Table 3a| In addition, BigGAN (Brock) ct al. 2019) and StyleGAN2 (Karras et al. 2020b) are also included as saate-of-the-ard CNN-based
GAN models.
Our ViTGAN model outperforms other Transformer-basee GAN models by a large margin. This stems from the improved stable GAN training on the Transformer architecture. As shown in Fig. our method overcomes the spikes in gradient magnitude and stabilizes the training dynamics. This enables ViTGAN to converge either lower or comparable FID on different datasets.
ViTGAN achieves comparable performance to the leading CNN-based models, i.e. BigGAN and StyleGAN2. Note that T focus on comparing the architectures, we use generic version of StyleGAN2. More comprehensive comparisons with StyleGiAN2 are included in Appendix A. As shown in Fig. the image fidelity of the best Transformer baseline (Middle Row) has been notably improved by the proposed VITGAN model (Last Row). Even compared with StyleGAN2, VITGAN generates images with comparable quality and diversity. Notice there appears to be a perceivable difference between the images generated by Transformers and CNNs, e.g.in the background of the CelebA images. Both the quantitative results and qualitative comparison substantiate the efficacy of the proposed VITGAN competitive Transformer-haar GAN model.

---

Published as a conference paper at ICLR 2022
CIFAR- 10 32 X 32
(a) StyleGAN2 (FID= = 5.60)
(b) Vanilla- ViT (FID = 12.7)
(c) ViTGAN (FID = 4.92)
KSUJJ
CelebA 64 x 64
(d) StyleGAN2 (FID 3.39)
(e) Vanilla-ViT (FID = 23.61)
(f) VITGAN (FID 3.74)
LSUN bedroom 64 x 64
(g) StyleGAN2 (FID 2.33)
(h) Vanilla-ViT (FID = 218.1)
(i VITGAN (FID = 2.40)
Figure 3: Qualitative Comparison. We compare our ViTGAN with StyleGAN2, and our best Transformer baseline, i.e., vanilla pair of ViT generator and discriminator described in Section 5.1 on the CIFAR-10 32 x 32, CelebA 64 x 64 and LSUN Bedroom 64 x 64 datasets. Results on LSUN Bedroom 128 x 128 and 256 x 256 can be found in the Appendix.
7

---

Published as a conference paper at ICLR 2022
SN
Ours
Gradient Norm
Ours
Ours
Gradient
Gradlent
139
50
109
Discniminttor dratnnng steps (x 1000)
Discreninator training steps ( 1000)
Glecrintntter training steps Ex 1909)
(a) Gradient L2 Norm (CIFAR)
Gradient L2 Norm (CelebA)
Gradient L2 Norm (LSUN)
R1
18
SN
SN
14
(.)
PD
18 iEE 3 aIC Diseriminator training stops 1000)
140
139
Disonmbrator craining steps
Discriminator traininng stops 1000)
(d) FID (CIFAR)
(e) FID (CelebA)
(f) FID (LSUN))
Figure 4: (a-c) Gradient magnitude (L2 norm over all parameters) of VIT discriminator and (d-f) FID score (lower the better) function of training iteration. Our ViTGAN are compared with two baselines of Vanilla ViT discriminators with R1 penalty and spectral norm (SN). The remaining architectures are the same for all methods. Our method overcomes the spikes of gradient magnitude, and achieve lower FIDs (on CIFAR and CelebA) comparable FID (on LSUN).
5.2 ABLATION STUDIES
We conduct ablation experiments on the CIFAR dataset to study the contributions of the key techniques and verify the design choices in our model.
Compatibility with CNN-based GANs In Ta- ble we mix and match the generator and discriminator of our ViTGAN and the leading CNN-based GAN: StyleGAN2. With the Style- GAN2 generator, our ViTGAN discriminator outperforms the vanilla ViT discriminator. Be- sides, our VITGAN generator still works to- gether with the StyleGAN2 discriminator. The results show the proposed techniques are patible with both Transformer-base and CNN- based generators and discriminators.
Table 2: VITGAN CIFAR-10 when paired
with CNN-based generators or discriminators.
Generator
Discriminator
FID J
IS
ViTGAN StyleGAN2
ViTGAN
4.92
9.69
StyleGAN2
5.60
9.41
StyleGAN2
Vanilla- VIT ViTGAN
17.3
8.57
StyleGAN2
8.02
9.15
ViTGAN
StyleGAN2
4.57
9.89
Generator architecture Tahhle[a GAN
performances under three generator architectures, as shown in Fig. 2 Fig. 2 (B) underperforms other architectures. We find that Fig. (A) works well but lags behind Fig. (C) due to its instability. Regarding the mapping between patch embedding and pixels, implicit neural representation (denoted as NeurRep in Table offers consistently better performance than linear mapping, suggesting the importance of implicit neural representation in the ViTGAN generators.
Discriminator regularization Table 3b validates the necessity of the techniques discussed in Section 4.1 First, we compare GAN performances under different regularization methods. Training GANs with VIT discriminator under R1 penalty (Mescheder et al. 2018) is highly unstable, as shown in Fig. sometimes resulting in complete training failure (indicated as IS=NaN Row I of Table 3bp. Spectral normalization (SN) is better than RI penalty. But SN still exhibits high-variance gradients and therefore suffers from low quality scores. Our +SN regularization improves the stability significantly (c.f .Fg. 4 and achieves the best IS and FID scores as consequence. On the other hand, the overlapping patch is a simple trick that yields further improvement over the L,+ISN method. However, the overlapping patch by itself does not work well (see comparison between
8

---

Published as a conference paper at ICLR 2022
Row 3 and 9). The above results validate the essential role of these techniques in achieving the final performance of the ViTGAN model.
Table 3: Ablation studies of ViTGAN on CIFAR-10.
Aug.
Reg.
Overlap
Proj.
FID
IS
X
RI
2c4
NaN
Embedding
Output Mapping
FID +
IS
X
R1
129
4.99
R1
13.1
8.71
Fig
2
(A)
Linear
14.3
8.60
Fig
(A
NeurRep
11.3
9.05
SN
121
4.28
Fig 2
Linear
328
1.01
SN
X
10.2
8.78
Fig (
NeurRep
285
2.46
L2+SN
X
X
168
2.36
Fig (
Linear
15.1
8.58
ISN
X
8.51
9.12
L2+ISN
8.36
9,02
Fig (C)
NeurRep
6.66
9.30
L2+ISN
6.66
9.30
L2+INN
4.92
9.69
(a) Ablation studies of generator architectures. NeurRep denotes implicit neural representation. ViT discriminator without convolutional projection used for this ablation.
(b) Ablation studies of discrminator regularization. 'Aug.', 'Reg'. 'Overlap" and 'Proj.' stand for Dif- fAug Zhao et al. (2020a) + bCR Zhao ct al. (2021), and regularization method, overlapping image patches, and convolutional projection, respectively.
Necessity of self-attention Two intrinsic advan- tages of ViTs over CNNs are (a) adaptive connection weights based on both content and position, and (b) globally contextualized representation. We show the importance of each of them by ablation analysis in
Table 4: Ablations studies of self-attention LSUN Bedroom 32x32.
Method
attention
local
global FID
StyleGAN2
2.59
Table
We conduct this ablation study on the LSUN Bedroom dataset considering that its large-scale (~33 million images) helps compare sufficiently trained vision transformers.
VIT
1.94
MLP-Mixer
3.23
ViT-local
x
1.79
ViTGAN
/
1.57
First, to see the importance of adapting connection weights, maintain the network topology (i.e., each token is connected to all other tokens) and use fixed weights between tokens as in CNNs. We adopt the recently proposed MLP-Mixer (Tolstikhin et al.) (202I) as instantiation of this idea. By comparing the 2nd and 3rd rows of the table, we see that is beneficial to adapt connection weight between tokens. The result of MLP-Mixer the ISUUN Bedroom dataset suggests that the deficiency of weight adaptation cannot be overcome by merely training on a large-scale dataset.
Second, see the effect of self-attention scope (global v.s. local), we adopt local self-attention of window 3ize=3x3 (Ramachandran et al. 2001).. The ViT based on local attention (4th row in Table 4 outperforms the ones with global attention. This shows the importance of considering local context (2nd row in T). The VIT with convolutional projection (last row in Table which considers both local and global contexts outperforms other methods.
For results on applying these techniques into either one of discriminator or generator, please refer to Table in the Appendix.
CONCLUSION AND LIMITATIONS
We have introduced ViTGAN, leveraging Vision Transformers (ViTs) in GANs, and proposed essential techniques to ensuring its training stability and improving convergence. Our experiments on standard benchmarks demonstrate that the presented model achieves comparable performance to leading CNN-based GANs. Regarding the limitations, VITGAN is a novel GAN architecture consisting solely of Transformer layers. This could be improved by incorporating advanced training (e.g., (Jeong Shin 2021 Schonfeld et al. 20200) or coarse-to to-fooofeeeee architectures (e.g., (Liu et al. have not verified ViTGAN on high-resolution
2021b) into the ViTGAN framework. Besides, images. Our paper establishes a concrete baseline of VIT on resolutions up to 256 256 and we hope that it can facilitate future research vision transformers for high-resolution image synthesis.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
We acknowledge that the image generation techniques have the risks to be misused to produce misleading information. Researchers should explore the techniques responsibly. Future research may have ethical and fairesss implication especially when generating images of identifiable faces. unclear whether the proposed new architecture would differentially impact groups of people of certain appearance (e.g., skin color), age (young aged), or with a physical disability. These concerns ought to be considered carefully. We did not undertake any subject studies or conduct any data-collection for this project. We are committed in our work abide by the eight General Ethical Principles listed at ICLR Code of Ethics (https: /ettps://iclr.oom http:ppbllccooooffthhhe.ccc
REPRODUCIBILITY STATEMENT
All of our experiments are conducted on public benchmarks. We describe the implementation details in the experiment section and in the Appendix [ and Appendix [ We will release our code and models to ensure reproducibility.
ACKNOWLEDGMENTS
This work was supported in part by Google Cloud Platform (GCP) Credit Award. We would also like to acknowledge Cloud TPU support from Google's TensorFlow Research Cloud (TRC) program. Also, we thank the anonymous reviewers for their helpful and constructive comments and suggestions.
REFERENCES
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. Image generators with conditionally-iddeeeneen pixel synthesis. In CVPR, 2021.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, 2017.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint arXiv:2103. /5691, 2021.
Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer-based object detection. arXiv preprint arXiv:2012.09958, 2020.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding? In ICML, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ICLR, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In ICML, 2020a.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020b.
Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. In ICLR, 2019.
10

---

Published as a conference paper at ICLR 2022
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. CoRR, abs/2104.02057, 2021.
Ming-Wei Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In ACL. 2019.
Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention not all you need: Pure attention loses rank doubly exponentially with depth. arXiv preprint arXiv:2103.03404 2021.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition scale. In ICLR, 2021.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M. Dai, Shakir Mohamed, and Ian J. Goodfellow. Many paths to equilibrium: Gans do not need to decrease divergence every step. In ICLR, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In NeurlPS, 2017.
Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Ychui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolutional neural networks meet vision transformers, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In NeurlPS, 2017.
Drew A Hudson and C. Lawrence Zitnick. preprint:2103.01229 2021.
Generative adversarial transformers. arXiv
Jongheon Jeong and Jinwoo Shin. Training [gan|s with stronger augmentations via contrastive dis- criminator. In ICLR, 2021. URLpptrp:://oeeeeeeeee wtt.://wwwweeeeeoooooeeeeeeeeeee
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers can make one strong gan. arXiv preprint arXiv:2102.07077 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In JCLR, 2018
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NeurIPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020b.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In ICML., 2021.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv: 17X.v:1705.07215. 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
11

---

Published as a conference paper at ICLR 2022
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541-551, 1989. doi: 10. 1..1162200 http://eco..199....444
Rui Liu, Hanming Deng. Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fuseformer: Fusing fine-grained information in transformers for video inpainting. In ICCV, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei. Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In ICCV, 2015.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Which training methods for gans do actually converge? In ICML, 2018.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, pp. 429 443, 1997.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In NeurIPS, 2016.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In NeurlPS, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12091 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. ArXiv preprint, 2021.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurlPS, 2016.
Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. A u-net based discriminator for generative adversarial networks. In CVPR, 2020.
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NeurIPS, 2020.
Jiaming Song and Stefano Ermon. Bridging the gap between f-gans and wasserstein gans. In ICML, 2020.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv preprint 105.:2205.00633. 2021.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, 2020.
12

---

Published as a conference paper at ICLR 2022
Ily Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. arXiv preprint arXlv:2105.01601, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. arXiv preprint arXiv:2012.12877, 2020.
Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Cheung. On data augmentation for gan training. IEEE Transactions on Image Processing, 30: 1882-1877, 2021.
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In CVPR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint 17X2 2021.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better, 2021.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint (56::::500003665. 2015.
Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention generative adversarial networks. ICML, 2019.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. generative adversarial networks. In ICLR, 2020.
Consistency regularization for
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In NeurIPS, 2020a.
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for GAN training. arXiv preprint:2006.000 02595, 2020b.
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for gans. In AAAI, 2021.
Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. Lipschitz generative adversarial nets. In ICML, 2019.
13

---

Published as a conference paper at ICLR 2022
A MORE RESULTS
A.1 EFFECTS OF DATA AUGMENTATION
Table presents the comparison of the Convolution-based GAN architectures (BigGAN and Style- GAN2) and our Transformer-based architecture (ViTGAN). This table complements the results in Table I of the main paper by closer examination the network architecture performance with and without using data augmentation. The differentiable data augmentation (DiffAug) (Zhao et al. 2020a)
used in this study.
As shown, data augmentation plays more critical role in VITGAN. This is not unexpected because discriminators built on Transformer architectures are more capable of over-fitting or memorizing the data. DiffAug increases the diversity of the training data, thereby mitigating the overfitting issue in adversarial training. Nevertheless, with DiffAug, ViTGAN performs comparably to the CNN-based GAN models: BigGAN and StyleGAN2.
In addition, Table includes the model performance without using the balanced consistency regular- ization (bCR) (Zhao al. 202T).
Table 5: Effectiveness of data augmentation and regularization the CIFAR-10 dataset. ViTGAN results here are based on the ViT backbone without convolutional projection.
Method
Data Augmentation
Conv
FID 4
IS
StyleGAN2
None
8.32
9.21
StyleGAN2
DiffAug
Y
5.60
9.41
ViTGAN
None
N
30.72
7.75
VITGAN
DiffAug
N
6.66
9.30
ViTGAN w/o. bCR
DiffAug
N
8.84
9.02
A.2 FID OF STYLEGAN2
We implemented StyleGAN2, Vanilla- and ViTGAN on the same codebase to allow for fair comparison between the methods. Our implementation follows StyleGAN2 + DiffAug (Zhao et al.) 2020a) and reproduces the StyleGAN2 FID reported in (Zhao et al. 2020a) which published in NeurlPS 2020: the FID our StyleGAN2 (+ DiffAug) re-implementation is 5.60 versus the 5.79 FID reported in Zhao et al. (2020a).
There is another contemporary work called StyleGAN2 (+ ADA) (Karras et al. 2020a). Both StyleGAN2 (+ DiffAug) (Zhao et al. 2020a) and StyleGAN2 (+ ADA) KKrrras et al. same StyleGAN2 architecture which is considered as the state-of-the-art CNN-based GAN. Due to their differences in data augmentation methods and hyperparameter settings, they report different FIDs on the CIFAR-10 dataset.
2020a) the
Note our goal is compare the architecture difference under the same training setting, we only compare StyleGAN2 (+ DiffAug) (Zhao et al. 2020a) and use the same data augmentation method DiffAug both StyleGAN2 and VITGAN.
A.3 NECESSITY OF SELF-ATTENTIO
In Table extend the result from Table by applying self-attention or local attention to either one of discriminator (D) or generator (G). The comparison between ViT vs. MLP shows that it is more important to adaptive weights the generator than the discriminator. The comparison between ViT and ViT-local shows that it is beneficial to have locality on both generator and discriminator.
A.4
SAMPLES
Figure 5 and Figure 6 show uncurated samples from our ViTGAN model trained on LSUN Bedroom 128 128 and L.SUN Bedroom 256 x 256, respectively. For 256 256 resolution, use the
14

---

Published as a conference paper at ICLR 2022
Table 6: Detailed ablations studies of self-attention on LSUN Bedroom 32x32.
Method
FID
ViT-D + VIT-G
1.94
MLP-D + VIT-G
2.33
ViT-D + MLP-G
2.86
MLP-D + MLP-G
3.23
ViT-local-D ViT-local-G
1.79
ViT-local-D ViT-G
2.03
ViT-D + ViT-local-G
2.04
same architecture as 128 x 128 resolution except for increasing the sequence length to 1024 for the generator. The FID of ViTGAN on LSUN Bedroom 256 x 256 is 4.67. Our method outperforms GANformer (Hudson & Zimick 2021), which achieved an FID of 6.51 on LSUN Bedroom 256 x 256.
P
Figure 5: Samples from VITGAN on 128 x 128 resolution. Our ViTGAN was trained on LSUN Bedroom 128 x 128 dataset. We achieved the FID of 2.48.
15

---

Published as a conference paper at ICLR 2022
Figure 6: Samples from VITGAN on 256 x 256 resolution. Our ViTGAN was trained on L.UUN Bedroom 256 x 256 dataset. We achieved the FID of 4.67.
16

---

Published as a conference paper at ICLR 2022
B COMPUTATION COST ANALYSIS
We would like to note that computational complexity is not the main challenge of using Transformers for high-resolution datasets. Due to various smart tokenization strategies such non-overlapping patches (Dosovitskiy et al. 2021), stack of strided convolutions (Xiao et al. 2021). etc, we are able to suppress the explosion of effective eeeueeee standard image resolutions. As a result, the sequence length stays within a region where a quadratic time complexity is not a big issue. Actually, as shown in Table 7 ViTs are much more compute efficient than CNN (StyleGAN2). For our ViT-based discriminator, owing to the approach from (Xiao al. 2021) we were able to scale up from x2x to 128x x 128 without the increase in sequence length (=64). For our ViT-based generator, since we did not employ any convolution stack, we had increase the sequence length as we increased the resolution. This issue, in principle, can be addressed by adding up-convolutional (deconvolutional) blocks or Swin Transformer (Liu et al.) 2021b) blocks after the initial low oow-resouution Transformer stage. Table(]reveal that we are able to suppress the increase in computation cost at higher-resolutions by using Swin Transformer blocks. For this hypothetical ViTGAN-Swa variant, we use Swin Transformer blocks starting from 128? resolution stage, and the number of channels is halved for every upsampling stage as done in Swin Transformer.
sequence length it is typically just a few hundred for
Please note that the main goal of this paper is not to design new Transformer block for general computer vision applications: we instead focus on addressing challenges unique to the combination of Transformers and GANs. Techniques presented in this paper still valid for numerous variants of ViTs such as Swin (Liu et al. (0211), CvT (Wu et al. 2021), CMT (Guo et al. 2021). etc.
The experiments in the current paper is not based Swin Transformers since using Swin Trans- formers resulted in inferior FID according to our experiments. However, we believe that with careful tuning of architectural configurations such as numbers of heads, channels, layers, etc., ViTGAN- may be able to achieve better FID score. We were unable to perform this due to our limited computational resources. We leave leave future work.
Table 7: Generator computation cost comparison among methods.
Method
# Params @4 FLOPs@64* FLOPs@ 128
FLOPs@255
StyleGAN2 (Karras al. 2020h)
24M
7.8B
11.5B
15.2B
ViTGAN
38M N/A
2.6B
11.8B
52.1B
ViTGAN-S Swin
N/A
3.1B
3.5B
C EXPERIMENT DETAILS
Datasets The CIFAR-10 10 dataset (Krizhevsky et al. 2009) is standard benchmark for image generation, containing 50K training images and 10K test images. Inception score (IS) (Salimans) et al. 2016) and Fréchet Inception Distance (FID) (Heusel et al. 2017) are computed over the 50K images. The LSUN bedroom dataset Yuu et al. 2015) is a large-scale image generation benchmark, consisting of ~3 million training images and 300 images for validation. On this dataset, FID is
computed against the training set due to the small validation set. The CelebA dataset (Liu et al. 2015) comprises 162,770 unlabeled face images and 19,962 test images. We use the aligned version of CelebA, which is different from cropped version used in prior literature (Radford et al. 2016 Jiang ct al. 202T).
Implementation Details. We consider three resolution settings: 32x32 on the CIFAR dataset, 64x64 on CelebA and LSUN bedroom, and 128 128 on LSUN bedroom. For x2x32 resolution, we use a 4-block ViT-based discriminator and a 4-block ViT-based generator. For 64x resolution, we increase the number of blocks to 6. Following ViT-Small (Dosovitskiy et al. 202T). the input/output feature dimension is 384 for all Transformer blocks, and the MLP hidden dimension is 1,536. Unlike (Dosovitskiy et al. 2021), we choose the number of attention heads to be 6. We find increasing the number of heads does not improve GAN training. For 32x32 resolution,
use patch size 4x4, yielding a sequence length of 64 patches. For 64 64x64 64 resolution, we simply increase the patch size to 8x8, keeping the same sequence length as in 32x32 resolution. For 128x 128 resolution
17

---

Published as a conference paper at ICLR 2022
generator, we use patch size 8 and 8 Transformer blocks. For 128x 128 resolution discriminator, we maintain the sequence length 64 and 4 Transformer blocks. Similarly to (Xiao et al. 2021) 3 x 3 convolutions with stride 2 were used until desired sequence length reached.
Translation, Color, Cutout, and Scaling data augmentations (Zhao et al.
2020a Karras et al. 2020a)
applied with probability 0.8. All baseline transformer- GANN models, including ours, use balanced consistency regularization (bCR) with Ayeal = Aake = 10.0. Other than bCR, we do not employ regularization methods typically used for training ViTs (Touvron al., 2020) such as Dropout, weight decay, or Stochastic Depth. We found that LeCam regularization (Tseng et al. 2021 similar bCR, improves the performance. But for clearer ablation, we do include the LeCam regularization. We train our models with Adam with 31 0.0, 32 0.99, and learning rate of 0.002 following the practice of (Karras et al. 2020b). In addition, we employ non-saturating logistic loss (Goodfellow et al. 2014), exponential moving average of generator weights (Karras et al. 2018 and equalized learning rate (Karras al. 2018). We use mini-batch size of 128.
Both ViTGAN and StyleGAN2 are based on Tensorflow 2 implementation and trained Google Cloud TPU v2-32 and v3-8.
D IMPLEMENTATION NOTES
Patch Extraction We use simple trick to mitigate the over-fitting of the ViT-based discriminator
by allowing some overlap between image patches. For each border edge of the patch, we extend it by o pixels, making the effective patch size (P + 20) x (P + +2). where o = P. Although this operation has connection to convolution operation with kernel (P 2o) x (P + 2o) and stride P x P, do not regard it as a convolution operator in our model because we do not use convolution in our implementation. Note that the extraction of (non-overlapping) patches in the Vanilla ViT (Dosovitskiy et al. 2021 also has connection to a convolution operation with kernel P x P and stride P x P.
Positional Embedding Each positional embedding of VIT networks is linear projection of patch position followed by a sine activation function. The patch positions are normalized to lie between -1. and 1.0.
Implicit Neural Representation for Patch Generation Each positional embedding is linear projection pixel coordinate followed by a sine activation function (hence the name Fourier encoding). The pixel coordinates for p2 pixels 2-layer ML. takes positional embedding E fou as its input, and is conditioned on patch embedding y" via weight modulation as in Karras et al. 2020b Anokhin et al. 2021
normalized lie between 11.0 and 1.0. The
18


Reviews:
[{"review_id": "rWKqDonp24", "reviewer": "Reviewer_vpzj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe authors give a comprehensive review and discussion to modern related works, including some contemporary approaches. \n\nThe whole paper is well organized and written, and easy to follow. \n\nThe direction of exploring transformer on image generation is novel, although several works have investigated Vision-Transformer and achieved competitive results on image recognition.  \n\nDetailed strategies are proposed for the components in transformer-based generators and discriminators. The work presents comprehensive ablation studies and promising results compared to CNN-based benchmarks.\n\nWeaknesses:\n\nThere are several questions about the insight and effect of some design:\n1. L2 attention is adopt in ViT-based discriminator duo to the consideration of training stability (i.e., Lipschitzness). How does it perform if using in the generator? \n2. The capacity of discriminator has shown to be important to guiding the training of a high-performing generators (e.g., StyleGANs and BigGAN). The results in Table 2 also show that using a powerful CNN-based discriminator performs better than Transformer-based discriminator. Could the authors provide more insight or discussion about it? Locality that CNNs intrinsically have is supposed to be very important for vision tasks and whether it is necessary to eliminate such inductive biases. \n3. The authors also point out the limitation of current method on high-resolution image generation.  The challenges of extending transformer to such a scenario may arise from several aspects, including computational overhead and effectiveness of using global context, as the low-level features in high-resolution is supposed to be naturally local.  It would be helpful if the authors could provide deeper insight or discussion about it.\n\nMinor suggestions:  L is used to denote patch length in the section of ViT definition. It is a bit confusing to use L as well in Eqn. 2-4, which is supposed to represent layer index. \n", "summary_of_the_paper": "The authors aim to investigate the effect of transformer, a convolution-free network architecture which has achieved impressive performance on NLP, on image generation. Vision Transformer works regard an image as a sequence of tokens The work presents several challenges of developing a vision-transformer encoder, including architecture designs for embedding latent vectors and patch generation in generators and some regularizations for stabilizing the training of VIT-based discriminators. The experiments show the proposed framework can achieve competitive performance with CNN-based GANs on several datasets with image resolutions up to 128x128.", "summary_of_the_review": "The motivation of investigating the effect of modern architecture Transformer, without using spatial convolution or pooling, on image generation is interesting and valuable. Transformer-like architectures have shown remarkable performance on a wide range of tasks, including image recognition, while exploring it on GANs for image generation is a novel and challenging direction. The current framework (without significant modification to the original ViT) has shown promising results compared to CNN benchmarks. On the other hand, the paper can to be improved by providing more discussion and analysis about the relation between locality that convolution intrinsically has and global context that self-attention captures. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "okMyeo-Yz0", "reviewer": "Reviewer_oHXw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The architecture of the proposed method is quite clean and simple. The design of the ViT architecture part don't change much, compared with the original one. This paper mainly focus on finding regularization techniques and minor changes, which works for ViT.\n2. Several useful techniques and tricks are proposed with detailed experiments, which could be referred and taken by other ViT+GAN works.\n3. The paper is well-organized. Also, it contains a section of preliminaries for Vision Transformer, which might help new ViT+X follower.\n\nCons:\n1. This paper might sound a bit lack of novelty to me, which looks like one of pursuers for ViT+X. It think the main tackle for making naive ViT not worth and meaningful to combine with GAN is the unacceptable and unaccessible computation cost and memory cost, when we're targeting on typical resolution 256x256, 512x512, even more.\n2. With Cons#1. Basically, the authors take a less meaningful resolution baseline of range from 32x32 to 64x64, and one 128x128 for LSUN. I suggest experiments of resolution 256 or 512 would be helpful to provide.\n3. Also, other computation cost related information could be a great plus, such as FLOPs or Latency(ms), which could make a more fair comparison with traditional CNN-based methods or other ViT+GAN methods.\n4. Some of proposed techniques are well-introducted and involved by recent works.\n1) Overlapping Image Patches:\n\nFuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting (ICCV 2021)\n\nSwin Transformer: Hierarchical Vision Transformer using Shifted Windows (ICCV 2021)\n\n2) Convolutional Projection:\n\nCvT: Introducing Convolutions to Vision Transformers", "summary_of_the_paper": "This paper introduces a novel new GAN design, named ViTGAN, which integrates the Vision Transformer(ViT) architecture into Generative Adversarial Networks(GAN), replace both discriminators and generators. Several novel regularization techniques are proposed for addressing the traditional regularization methods work poorly with self-attention by direct combination in discriminators. Also, a few of architectural choices are examined for latent and pixel mapping layers to faciliate convergence in generators. Combining all the empirical design and choices, the method ViTGAN achieves comparable performance to the SOTA CNN-based GAN models on mainstream metrics.", "summary_of_the_review": "I recommend the paper would be worthy with the rating 6.\nIf all my concerns could be well-addressed, a score of 8 is deserved. And this works would be one of best ViT+GAN proposal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "nSybb1CFO7R", "reviewer": "Reviewer_mS8C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is generally well written and easy to follow.\n2. The authors analyze the limitations of directly applying ViT models to training GANs and adopt several techniques to remedy such issues during the training.\n3. Comprehensive ablation study shows the benefits of each component.\n\nCons:\n1. During the evaluation, the authors only provide the FID numbers without providing the computation statistics (e.g., FLOPs) or model size. As we know, existing GAN models (e.g., StyleGAN2, StyleGAN3) can be scaled up to further improve the performance (especially when augmentation is used to prevent overfitting). Therefore, without providing the computation/model size of different models, it is hard to demonstrate if the proposed model actually has an advantage.\n2. The proposed architecture seems to still be applicable to low-resolution images. It would be better to investigate the feasibility of scaling up to high-resolution images (e.g., 1024x1024). For example, showing the results on a small-scale dataset and measuring the computation on large resolutions would be great.\n3. Many of the techniques like L2 attention to enforce Lipschitz constant (Kim et al., 2021), overlapping patches (Liu et al., 2021), convolutional projection (Wu et al., 2021; Guo et al., 2021), self-modulation (Chen et al., 2019), Fourier features (Tancik et al., 2020) are already proposed in existing work. The work seems more like a study combining existing techniques.\n4. In Table 5, the FID of StyleGAN2 is worse with DiffAug, which is contrary to the results from (Zhao et al., 2021). Is there any reason why it happens?", "summary_of_the_paper": "In this paper, the authors study employing ViTs for GAN-based image generation. For discriminators, they propose improved methods to enforce Lipschitz constraints and spectral norm regularization, and modified architecture; for generators, they adopted self-modulated LayerNorm and Fourier features to improve the performance. Experiments on CIFAR, LSUN bedroom, and CelebA show superior performance compared to existing methods.", "summary_of_the_review": "The paper is overall well written and presents a solid study of using ViT models to build image GANs. However, the lack of computation/model size statistics may lead to unfair comparison; the novelty might be limited since most of the techniques are already proposed. I would like to hear from the authors for the final decision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j_nlRxVP67d", "reviewer": "Reviewer_Wrur", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "***Strengths***:\n1. The paper is presented clearly, particularly the method. \n\n2. Most experiments can verify the effectiveness of the core components of the method.\n\n***Weaknesses***:\n\nConcern on the motivation:\n\n1. What is the advantage of transformer over CNN in generative modeling? The authors claim to design a new GAN structure, but it is not clear what the advantages of the new structure are.\n\nConcern on the experiments:\n\n2. The paper chooses StyleGAN2 as a strong baseline. But I cannot find any comparison between the FLOPs and the parameters between ViTGAN and SyleganGAN2.\n\n3. The last two rows of Table 1 demonstrate that the ViT-based discriminator is not very effective, so what is the advantage of the ViT-based discriminator? Besides, could the authors provide the result of StyleGAN2-G + ViTGAN-D? With the same discriminator, does ViTGAN-G work better than StyleGAN2-G?\n\n4. Section 4.1 has presented the local and global text fusion is necessary when designing the discriminator. Swin transformer works with the same mechanism. Why don't the authors compare your design with Swin transformer?\n\n5. The state-of-the-art GANs have advanced high-resolution image synthesis. Why not compare ViTGAN with these state-of-the-art GANs on high-resolution image synthesis？ Does the computational overhead make ViTGAN impossible to train on high-resolution images?\n", "summary_of_the_paper": "This paper has presented a new GAN architecture based on the ViT.  Trivially extending ViT for image generation brings huge performance drop as well as unstable training. The authors have found that high-variance gradients in the transformer-based discriminator cause unstable training.  The new regularizers are proposed to stabilize the training. To better introduce the stochastics and leverage the local context, modifications on the generator such as self-modulated LayerNorm and the implicit functions of the path are facilitated for the training. The authors have conducted ViTGAN on several low-resolution datasets, and ablation studies are conducted to support the effectiveness of the main components of ViTGAN.", "summary_of_the_review": "Concerns about the method and experiment results have been presented in Weaknesses. \nI think the authors need more important experiments to support the advantages of ViT-based GAN over CNN-based GAN. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6F625tyW4i", "reviewer": "Reviewer_2CzQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and structured. I like the connection of recent results in transformer literature regarding Lipschitzness of self-attention and its direct implication and application in the GAN setting. The ablation studies of the proposed changes are mainly convincing, the proposed changes all contribute to the final performance. \n\nThere are several major issues that the authors should address:\n- **Usefulness of ISN**: It is not clear why R1 penalty works so much worse for ViTs; I would expect the regularization to be independent of the network architecture. The authors write that “our method overcomes the spikes in gradient magnitude and stabilizes the training dynamics” - R1 seems to be the best at least regarding gradient magnitude; why does collapse happen then? Any intuitions or further investigation would be very interesting. Also, what is the gamma value for R1? Maybe I missed it, but I could not find it in the paper. This detail is crucial, as gamma is dependent on the dataset, and it is necessary to do sweeps for good values. A good gamma value might make one of the main contributions (ISN) less important. Of course, it is still good not having to do parameter sweeps, which would be an advantage of ISN.\n- **Baselines**: the code for TransGAN is available and it is the primary ViT-based baseline. Hence, it would be important to see how well TransGAN does on other datasets except for CiFAR when compared to ViTGAN. Furthermore, the StyleGAN2 baseline on CIFAR should include data augmentation, which would improve over ViTGAN (Stylegan2-ADA: FID=2.9). I don’t mind the numbers not being bold for the proposed approach; instead, numbers from the literature should be reported faithfully.\n-**Limitations**: This is not a technical issue, but it should clearly state why it is hard to scale ViTGAN up to higher resolutions. Being more upfront about the problem in the limitation section would be nice. The authors aim to close the gap to the current state-of-the-art GAN, but GANs can operate on >1MP resolution. The authors should expand more on the challenges of scaling up such models. On a side note, TransGAN did evaluate on 256x256, so it is not impossible to scale (but challenging).\n-**Contributions**: The main idea of using ViTs for GANs is not original itself, and has been done before (TransGAN). In terms of technical contributions, as mentioned above, I am still not convinced of the usefulness of ISN. Lastly, for an empirical study, the experiments are somewhat lacking and I would like to see more and stronger baselines and more datasets.\n\nMinor Issues:\n\n- **Quantitative samples**: If not stated otherwise, I assume the shown samples are cherrypicked? If yes, the authors should mention this in the figure caption. Also, normally such samples are done via truncation, is truncation possible with a ViTGAN?\n- **General Motivation**: It is still unclear what the advantage would be to use a ViT in a GAN instead of standard CNNs (other than it just being generally interesting). The experiments regarding self-attention and adaptive weights are not very convincing, especially since a CNN discriminator improves over a ViT discriminator\n- Last paragraph intro: “Stylegan2-ADA [...] is equipped with sophisticated data augmentation methods”, this is not an advantage of CNN GANs; the authors even run experiments with DiffAugment in the appendix.\n- The authors mention SWIN transformers as a possible way to improve performance. Why don’t they run this experiment? This experiment should be a simple drop-in and also has been done before (TransGAN).\n- Section 5 “necessity of self-attention”. The authors evaluate the usefulness of adaptive weights and global representations of ViTs. It is unclear if the experiments look at G, D, or both. If it is either one, it might be interesting to look a the other side as well.\n- The experiments show that spectral norm at initialization (ISN) improves results. Why does it make sense to use the initial spectral norm? Currently, the motivation or intuition why it makes sense is missing.\n- Does A correspond to style codes like the W space in a StyleGAN? This correspondence would be interesting to investigate. I suspect it will not be similar to style codes because of the non-hierarchical nature of a ViT and would be a minor downside of the current architecture.\n- SLN is used ins Eq (9,10) but introduced later in the text", "summary_of_the_paper": "This paper investigates the use of ViTs in GANs.  Currently, CNNs are the standard neural network architecture for GANs, and it is a consequential idea to apply the recent ViT architecture in the generative setting. A question the authors aim to answer is how standard regularization techniques like spectral normalization work for this different type of architecture since it is unclear if they are readily applicable. The authors propose an adaptation of spectral normalization that works better when using ViTs, and two modifications to the ViT generator.", "summary_of_the_review": "This work is an empirical study and of interest to the community. The novelty of the idea and the technical contributions are somewhat limited.  As for empirical insights, there are several interesting findings, but one of the main contributions (ISN) needs to be compared to a stronger baseline, ie., a sweep over gamma values. Further, the comparisons to other baselines are currently incomplete, and the limitations will need more discussion. I rate this paper at 5, but I am happy to move up the score once the raised issues are addressed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a GAN architecture with a ViT-based discriminator and a ViT-based generator. The paper initially received a mixed rating with two "slightly above the acceptance threshold" ratings and "three slightly below the acceptance threshold" ratings. Several concerns were raised in the reviews, including whether there are advantages of using a ViT-based GAN architecture over the CNN-based GAN and whether the proposed method can be extended to high-resolution image synthesis. These concerns are well-addressed in the rebuttal with most of the reviewers increasing their ratings to be above the bar. The meta-reviewer agrees with the reviewers' assessments and would like to recommend acceptance of the paper.
Decision: Accept (Spotlight)