Vector-quantized Image Modeling with Improved VQGAN
Authors: Jiahui Yu
Xin Li
Jing Yu Koh
Han Zhang
Ruoming Pang
James Qin
Alexander Ku
Yuanzhong Xu
Jason Baldridge
Yonghui Wu
Venue: ICLR 2022 Poster (2022)
Keywords: VQGAN
Vision Transformers
Vector-quantized Image Modeling

Abstract: Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively. The discrete image tokens are encoded from a learned Vision-Transformer-based VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. The improved ViT-VQGAN further improves vector-quantized image modeling tasks, including unconditional, class-conditioned image generation and unsupervised representation learning. When trained on ImageNet at 256x256 resolution, we achieve Inception Score (IS) of 175.1 and Fr'echet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN, which obtains 70.6 and 17.04 for IS and FID, respectively. Based on ViT-VQGAN and unsupervised pretraining, we further evaluate the pretrained Transformer by averaging intermediate features, similar to Image GPT (iGPT). This ImageNet-pretrained VIM-L significantly beats iGPT-L on linear-probe accuracy from 60.3% to 73.2% for a similar model size. ViM-L also outperforms iGPT-XL which is trained with extra web image data and larger model size.

Published as a conference paper at ICLR 2022
VECTOR-QUANTIZER IMAGE MODELING WITH IM- PROVED VQGAN
Jiahui Yu
Xin
Jing Yu Koh
Han Zhang
Ruoming Pang
James Qin
Alexander Ku
Yuanzhong Xu
Jason Baldridge
Yonghui Wu
Google Research j jiahuiyu@google
ABSTRACT
Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining Transformer to predict rasterized image tokens autoregres- sively. The discrete image tokens are encoded from a learned Vision- Transformer- based VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. The improved ViT-VQGAN further improves vector- quantized image modeling tasks, including unconditional, class-conditioned im- age generation and unsupervised representation learning. When trained on Im- ageNet at 256 x 256 resolution, we achieve Inception Score (IS) of 175.1 and Fréchet Inception Distance (FID) 4.17 a dramatic improvement over the vanilla VQGAN, which obtains 70.6 and 17.04 for IS and FID, respectively. Based on ViT-VQGAN and unsupervised pretraining, we further evaluate pretrained Transformer by averaging intermediate features, similar to Image GPT (iGPT). This SrangeNet-pretraing VIM-L significantly beats iGPT-L on linear- proeerr-roobeeooo ac- curacy from 60.3% 73.2% for similar model size. ViM-L also outperforms iGPT-XL which trained with extra web image data and larger model size.
INTRODUCTION
Natural language processing (NLP) has recently experienced dramatic improvements from learning general-purpose representations by pretraining language models unlabeled text corpora. This strategy has produced large performance gains for a wide range of natural language generation
(NLG) and natural language understanding (NLU) tasks (Dai & Le 2015) Radford et al., 2018 2019 Brown et al. 2020). Conceptually, generative pretraining models the data density P(X in tractable way, with the hope of also helping discriminative tasks of P(Y|X) (Lasserre al. 2006);
importantly, there are no limitations on whether the signals are from the language domain or others, such vision.
In computer vision, in contrast, most recent unsupervised or self-supervised learning research fo- cuses on applying different random augmentations to images, with the pretraining objective dis-
tinguish image instances (Chen et al. 2020b; He et al. 2020 Chen et al., 2020d; Grill et al. 2020)] Chen et al., |020cc Caron et al. 2021 II. The quality of learned representation relies on manually cho- sen augmentations, such as random brightness, cropping, blurring, and others. Chen et al. (2020a)
explored GPT-style (Radford al. 2018) generative pretraining on images by autoregressively pre- dicting pixels without incorporating knowledge of the 2D structure. Each pixel is represented as 9-bit value created by clustering (R, G. B) pixel values, using k-means with k=512. Unfortunately, this color encoding does not scale to typical image resolutions as it entails very long sequences to represent the image (e.g., e. 224 x 224 resolution leads to 50,176 tokens per image), and this demands much more memory and computation for training, compared to language models. As result, iGPT's maximum resolution is 64 x 64 for image recognition at scale scale-whide which severely limits its represen- tation capabilities.

---

Published as a conference paper at ICLR 2022
Transformer Encoder
Input
Codebooc
Reconstruction
Quantiottion
Autoregressive Transformer
Transforer Decoder
StyleGAN Dicrrminstor
Image Generation
Image
Understanding
Stage Quantiration
Stage Vector-quaantizrr Image Modeling
Figure 1: Overview of ViT-VQGAN (left) and Vector-quantized Image Modeling (right) for both image generation and image understanding.
Remarkable image generation results have been achieved by pre-quantizing images into discrete latent variables and modeling them autoregressively, including VQVAE (Oord et al.) 2017). DALL- E (Ramesh et al., (202)) and VQGAN (Esser et al., (02T). In these approaches, a convolution neu- ral network (CNN) is learned to auto-encode an image and a second stage CNN or Transformer is learned to model the density of encoded latent variables. These have been proved effective for image generation, but few studies have evaluated the learned representation discriminative tasks [Ramesh et al. 202T Esser et al. 2021
We explore an approach we refer to as Vector-quantizer Image Modeling (VIM) and apply it to both image generation and image understanding tasks. VIM follows two-stage approach:
Stage 1: Image Quantization. Given image of resolution 256x256, Vision- Transformer- based VQGAN encodes it into x22 discretized latent codes where the code- book size is 8192. We propose multiple improvements from architecture codebook Icarning-to VQGAN (Esser et al. 2021). The resulting ViT- VQGAN is more efficient and improves reconstruction fidelity in terms of pixel-wise reconstruction metrics, Inception Score (IS) and Fréchet Inception Distance (FID). ViT- is trained end-to-end on image-only data with combined objective functions logit-laplace loss, loss, adversar- ial loss and perceptual loss (Johnson et al., 20166) Zhang et al., 2018).
Stage 2: Vector-quantized Image Modeling. We train a Transformer model to predict rasterized 32x32- 1024 image tokens autoregressively, where image tokens are encoded by learned Stage 1 ViT- For unconditional image synthesis or unsupervised learning, we pretrain decoder-only Transformer model predict the next token. For class-conditioned image synthesis, class-id token is prepended before the image tokens. evaluate the quality of unsupervised learning, we average the intermediate Transformer features and leamn linear head to predict the logit of the classes (a.k.a., linear-probe),
We show that one key component for improving both image generation and image understanding with VIM is have better image quantizer with respect to both computational efficiency and reconstruction quality. An efficient quantizer can speed up Stage 2 training, where random augmen- tations are applied first to an image, followed by the encoder of image quantizer to obtain the input tokens. Moreover, an image quantizer with better reconstruction quality can reduce information loss compared with the original image in pixel space, which critical for image understanding tasks.
The evaluations of our proposed ViT-VQGAN and VIM are studied with three aspects. (1) We evaluate the image quantizer based on reconstruction quality metrics including É distance, &g dis- tance, log-laplace distance, well as Inception Score (IS) and Fréchet Inception Distance (FID) of reconstructed images. (2) We evaluate the capabilities of the learned quantizer for unconditional flass-conditioned image synthesis based FID and IS, and compare with other methods. (3) We rely on linear-probe accuracy to evaluate representations with the common intuition that good features should linearly separate the classes of downstream tasks.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
Image Synthesis. Image generation has received much attention with the progress of deep gener-
ative models, including Generative Adversarial Networks (GANs) (Goodfellow et al., 2014 Zhang
al. (0196), Variational Autoencoders (VAEs) (Kingma & Welling 2014] Vahdat & Kautz 2020),
Diffusion Models (Song & Ermon] 2019; Dhariwal & Nichol, 2021) and Autoregressive Models
van den Oord et 201]]261))aaaaa et al.] 2018). Unlike many autogressive methods which generate
sequence directly in pixel space, VQVAE (van den Oord et al., 2017], Razavi et al. (2119) decom-
poses the image generation process into two stages: the first stage trains a vector quantized autoen-
coder with image reconstruction objective to convert an image into shorter sequence of discrete
tokens. Then the second stage learns an autoregressive model, e.g., PixelSNAIL (Chen et al. 2018),
to model the underlying distribution of token sequences. Driven by the effectiveness of VQVAE and progress in sequence modeling (Vaswani et al.] 2017 Devlin et al., 2019). many approaches follow the two-stage paradigm. DALL-E (Ramesh et 20221) improves token prediction in second stage by using Transformers (Vaswani et al., 2017). resulting in a strong text-to-image synthesis model. VQGAN (Esser al. 2021] further uses adversarial loss and perceptual loss (Johnson al.,
2016 Zhang et al. 2018) to train better autoencoder in the first stage to synthesize greater detail in images.
Image Recognition with Generative Pretraining. Many image generation models (Goodfellow)
et al. 2014, Kingma & Welling 2014 Radford et al., 2016 Donahue et al., 2017 Higgins et al,,
2017 have been studied for their capabilition libilliiiesssiot in representation learning. However, their performance
is usually not superior to competing self-supervised approaches that solve auxiliary classification
tasks (Noroozi & Favaro 2016a Gidaris et al. 2018a; van den Oord et al. 2018). BigBiGAN
(Donahue & Simonyan 2019a) first demonstrated that a generation- based model can match other self-supervised methods in representation learning ImageNet. iGPT (Chen et al. 2020a) uses the autoregressive objective to learn a giant transformer that directly predicts pixel values, producing even more competitive results. Compared to iGPT, our method first tokenizes the original image into discrete image tokens and then trains a transformer to predict them. As a result, our approach obtains comparable results with smaller model and less data. Similar to our method in predicting image tokens, BEiT (Bao et al. (202I) follows pre- pre-training scheme of BERT Devlin et al. (2019) by learning to recover randomly masked image tokens with bidirectional transformer. Unlike BEÎT, we explore vector-quantized image modeling for image generation in addition to image recognition.
3
IMAGES WITH VIT-VQGAN
The Vector-quantized Variational AutoEncoder (VQVAE) (van den Oord et al. 2017) is a CNN- based auto-encoder whose latent space is matrix of discrete learnable variables, trained end-to-end via straight-througe estimation. [Esser et al. (2021) introduce VQGAN, a model which improves upon VQVAE by introducing adversarial loss produced by discriminator. Below, introduce further improvements to VQGAN that boost efficiency and enhance reconstruction quality.
3.1 VQGAN WITH VISION TRANSFORMERS
The core network architectures used by both VQVAE and VQGAN to encode and reconstruct im- ages are CNNs. VOGAN introduces transformer-like elements the form of non-local attention block (Zhang et al. 2019a), allowing it to capture distant interactions with fewer layers. We pro- pose taking this appreach one step further by replacing the CNN encoder and decoder with Vision Transformer (ViT) (Dosovitskiy et al. 2020), as shown in Figure Given sufficient data (for which unlabeled image data is plentiful) we find that ViT-VQGAN is less constrained by the inductive pri- imposed by convolutions. Furthermore, ViT-VQGAN yields better computational efficiency on accelerators, and produces higher quality reconstructions, as shown in Table I
The encoder of ViT-VQGAN first maps 8x8 non-overlapping image patches into image tokens, followed by Transformer blocks, encoding a 256x256 resolution image into a 32x32-1024 token sequence. The decoder performs the inverse operation, mapping each image token from latent vari- ables back to 8 x image patches and regrouping them into a 256 x 256 image (see Figure At the output of transformer blocks, we apply a two-layer ferd-forward network with tanh activation layer
3

---

Published as a conference paper at ICLR 2022
Architecture
Model Size
Throughput f (imgs/sec)
82 loss (Ic-2)
Logit-Laplace loss
FID
IS
(encoder-decoder)
VIT-VQGAN
Small-Small
1520
3.34
-2.44
1.99
184.4
CNN-VOGAN
Channels
946
3.81
-2.36
2.26
178.7
ViT-VQGAN
Base-Base
960
3.09
-2.54
1.55
190.2
CNN-VOGAN
Channels 2
400
3.44
-2.46
1.91
183.4
ViT-VQGAN
Small-Large
384
2.88
-2.58
1.28
192.3
Table 1: ViT-VQGAN achieves better speed-quality trade-off's compared with CNN-VQGAN.
This in turn further speeds up Stage 2 training. Throughputs are benchmarked with the same 128 CloudTPUv4 devices.
in the middle. No activation applied at the output of ViT ViT-NQGAN encoder or decoder. We find that this simple approach yields high quality reconstructions without any noticeable grid artifacts.
3.2 CODEBOOK LEARNING
Vanilla VQVAEs usually suffer from low codebook usage due to the poor initialization of the code- book. Therefore, during training significant portion of codes are rarely used, or dead. The re- duction in effective codebook size results in worse reconstructions in stage quantizer training and poor diversity in stage for image synthesis. As a result, VOGAN (Esser al., (02T) relies on top- k and top-p (nucleus) sampling heuristics (Holtzman et al. 2020) with default codebook size of 1024 to obtain best results for image synthesis. We propose two improvements that can significantly encourage the codebook usage even with larger codebook size of 8192. During image synthesis, we perform simple sampling with temperature of 1.0 without top-k and top-p heuristics.
The training objective of vector-quamtizatuon is defined follows:
Lvo [s][==(xx] el + (1lz sg[e]]l.
(I)
Here, sg(x) = x. d sg(x) = the stop- proggrtaient operator, /3 commitment loss hyperparameter set to 0.25 in all our experiments, and e is the codebook vector. The quantized codebook index is determined by looking up the codebook vector closest to the input features zx(x) in terms of the Euclidean distance, yielding argmin, ||
de
e
Factorized codes. We introduce a linear projection from the output of the encoder to low- dimensional latent variable space for code index lookup (e.g., reduced from 768-d vector a 32-d or 8-d vector per code) and find it has an immediate boost of codebook usage. The factorization can be viewed decoupling code lookup and code embedding: coded from input on lower-dimensional lookup space and then project the matched latent code to the digndiimensional embedding space. Our experiments show reducing dimension lookup space from 256-d d to 32-d consistently improves reconstruction quality. A detailed illustration is provided in the supplementary materials.
lookup the the closest variable
eg-ormmalied codes. We also apply l normalization on the encoded latent variables ze (x) and codebook latent variables The codebook variables are initialized from a normal distribution. By mapping all latent variables a sphere, the Euclidean distance of fg-normalized latent variables (()((=(())) Ez(e,) evolves the cosine similarity of two vectors between (x) and e, further improving training stability and reconstruction quality shown in our experiments.
3.3 VIT-VQGAN TRAINING LOSSES
We use a combination of logit-laplace loss, l loss, perceptual loss Johnson et al. 2016 Zhang ctt al., 2018) based on VGG network (Simonyan & Zisserman, 2014] and GAN loss with architec- ture of StyleGAN discriminator (Karras al. 2020). Loss balancing weights are configured with a
sweep to optimize image reconstruction quality, codebook usage, FID and In- ception Score. After the sweep. apply the same set of hyper-paramedt of training losses to all datasets including CelebA-HO. FFHQ, and ImageNet. Logit-Laplace loss can be viewed as normal- ized & loss which assumes the noise the pixel level is laplace-distributed while &z loss assumes
4

---

Published as a conference paper at ICLR 2022
Model
Size
#Params
#Blocks
#Heads
Model Dim
Hidden Dim
Dropout
#Tokens
ViT-VOGAN
Small
32M
8
8
512
2048
0.0
1024
ViT-VQGAN
Base
91M
12
12
768
3072
0.0
1024
ViT-VQGAN
Large
599M
32
16
1280
5120
0.0
1024
VIM
Base Large
650M
24
16
1536
6144
1024
VIM
1697M
36
32
2048
8192
0.1
1024
Table Transformer architectures of Stage I ViT-VQGAN and Stage 2 VIM.
noise of Gaussian distribution. We find logit-laplace loss contributes to codebook usage while &g loss and perceptual loss significantly contribute to FID. The final loss combination we used by default
LvQ + 0.1 LAdv + 0.1 Perreppual + 0.1 LLLogi-llpplace + 1.0L2.
One caveat on the VGGi-bsed perceptual loss is that the VGG network is pretrained with supervised classification loss, so the supervision might leak into Stage 2 for sinear-probe accuracy measurement. Thus, for all of our reported unsupervised learning results, exclude the perceptual loss during ViT-VOGAN training. For all unconditional and class-conditiones image synthesis, we use VIT- VQGAN quantizers trained with perceptual loss, it leads to figher-idellty reconstructions.
VECTOR-QUANTIR
IMAGE MODELING
With learned ViT-VQGAN, images are encoded into discrete latent code ids flattened in the raster order, similar to Image GPT (Chen et al.] 2020aa). A decoder-only Transformer model used to model the density of image data P(r) autoregressively as
P(x) II 2/ 22. Iz:1:0)
where A is learnable weights. The training objective is to minimize the negative log-lik log-likelihood of the data [eexx=log
Table summarizes the architecture configurations for the Transformers. We first embed discrete image token ids into a learnable embedding space each position, with an additive learnable 2D positional embedding. Both embedding dimensions are the same as model dimension. We apply stack of Transformer blocks to the inputs with causal attention over entire sequence, A dropout ratio of 0.1 is used in all residual, actívation and attention outputs. At the final layer of all Trans- former blocks, we apply an additional layer normalization.
4.1 IMAGE SYNTHESIS
With pretrained generative Transformer model, unconditional image generation is achieved by simply sampling token-by-token from the output softmax distribution. All samples used for both qualitative and quantitative results obtained without temperature reduction. The sampled tokens are then fed into the decoder of ViT-VQGAN to decode output images. Our default Stage I ViT- VQGAN encodes input images of resolution 256 256 into 32 32 latent codes with codebook size 8192, while Stage 2 Transformer takes the flattened image tokens with total a length of 1024.
Class-conditioned ImageNet generation is also widely used benchmark for measuring capabiltiy of models for image synthesis. We extend the unconditional generation to conditiond generation by prepending class-id token before the image tokens. Separate embedding layers are learned from scratch for class-id token and image tokens, with the embedding dimension the same as the Transformer model dimension. During sampling, class-id token is provided at the first position to decode the remaining image tokens autorcgressively.
4.2 UNSUPERVISED LEARNING
For the image understanding task, we feed all image tokens of the input into a pretrained Trans- former, and get sequence of 1024 token features. Similar to Image GPT (Chen al. 2020a
5

---

Published as a conference paper at ICLR 2022
Model
Dataset
Latent Size
dim 2
FID Validation
DALL-E dVAE
Web data
32 x 32
8192
32.00
VOGAN
ImageNet
16 x 16
1021
7.94
VOGAN
ImageNet
16 16
16384
4,98
VQGAN"
ImageNet
32
8192
1.49
VQGAN"
ImageNet
64 x 64 & 32 x 32
512
1.45
VIT- VOGAN (Ours)
ImageNet
32
8192
1.28
ViT-VQGAN (Ours)
CelebA-HQ
32 x 32
8192
4,66
ViT-VQGAN (Ours)
FFHQ
32 x 32
8192
3.13
Table 3: Fréchet Inception Distance (FID) between reconstructed validation split and original val- idation split on ImageNet, CelebA-HH and FFHQ. denotes models trained with Gumbel-Softmal reparameterization in Ramesh et (202I) denotes models trained with multi-scall hierarchi-
cal codebook as in Razavi et al. (2019)
take layer output specific block I over total blocks I., average the sequence of token features (frozen) and insert a softmax layer (learnable) projecting averaged feature to class logits. We only take one specific Transformer block output instead of concatenating different block outputs as in iGPT (Chen et al.] 2020a). We find that most discriminating feature for the linear-probe is typically near the middle of all Transformer blocks.
5
EXPERIMENTS
5.1 IMAGE QUANTIZATION
We train the proposed ViT-VQGAN on three datasets separately, CelebA-HQ KKrras et al. 2019) FFHQ (Karras al. 2019), and ImageNet (Krizhevsky et al. 2012). For CelebA-HQ and FFHQ. we follow the default train and validation split VQGAN (Esser et al. 2021 For Stage 1 image quantization, three different architecture sizes are experimented, Small, Base and Large for either encoder or decoder, as defined in Table 2 The smallest ViT-VQGAN-SS has a Small-size encoder and Small-size decoder, while ViT-VQGANT largest ViT-VQGAN-SL has asymmetric Small-size encoder and Large-size decoder, with the motivation that Stage 2 training only requires forward propagation of the encoder of ViT-VQGAN (in inference/decoding for image synthesis, the decoder of ViT-VQGAN is still required to decode images from codes predicted during Stage 2).
has a Base-size encoder and Base-size decoder. The
We train all ViT-VQGAN models with training batch size of 256 distributed across 128 CloudT- PUv4 for a total 500,000 000 training steps. For both ViT-VQGAN and StyleGAN discriminator, Adam optimizer KKingma & Ba 2014) used with 31 = 0.9 and 8n = 0.99 with the learning rate linearly warming up to a peak value of x 10 over 50,000 steps and then decaying to 5 x 10 over the remaining 450.000 steps with cosine schedule. We use a decoupled weight decay (Loshchilov) & Hutter 2017) of 1 x 10-1 for both ViT-VQGAN and StyleGAN discriminator. All models are trained with an input image resolution 256 x 256 on CloudTPUv4.
Table 3 shows FID between reconstructed images and original images in the validation split on ImageNet, CelebA-HQ and FFHQ datasets. Without multi-scale hierarchical codebook or gumbel- softmax, ViT-VQGAN is able to achieve better FID with large codebook size of 8192 compared with vanilla VQGAN.
Table Aabliidrovidss extensive ablations the proposed modifications, with empirical results on mean l distance, l2 distance. logit-laplace distance, Inception Score and FID on ImageNet. Among different model sizes, ViT-VQGAN-SS (small-encoder, small-decoder) performs worse than ViT- VQGAN-BB (base-encoder, base-decoder) and ViT- TT-VQGAN--LL (small-encoder, large-decoder), but achieves much better throughput. The CNN-based VQGAN architecture is worse in both quality and throughput compared with ViT-based VQGAN. The StyleGAN discriminator (arrras)
al. 2019 is more stable and yields better reconstruction quality than PatchGAN (Isola et al. 2017)

---

Published as a conference paper at ICLR 2022
Codebook Usage 4
LingttLLaplace J
Encoder Size
Decoder Size
Architecture
Discriminator
E2- G-normalized
Throughput
Latent dim
f, (1e-2)
f (1c-3)
HID 4
Ablation on
IS
Base
Base
VIT
StyleGAN
32
3.06
3.09
-2.54
190.2
1.55
96%
960
Small Small
Small
VIT
StyleGAN
32
3.22
3.34
-2.44
184.4
1.99
95%
1520
Model Size
Large
VIT
StyleGAN
32
2.93
2.88
-2.58
192.3
1.28
95%
384
CNN
StyleGAN
32
3,45
3.81
-2.36
178.7
2.26
63%
946
Architecture
Base
Base
VIT
PatchGAN
32
2.82
2.58
-2.62
165.6
3.88
89%
1227
Base
Base
VIT
StyleGAN
256
3.60
4.28
-2.38
160.1
3.68
4%
954
Base
Base
StyleGAN
128
3.41
3.93
-2.44
173.9
2.77
14%
960
Codebook
Base
Base
StyleGAN
3.18
3.37
-2.49
179.5
2.50
37%
960
Learning
Base
Base
StyleGAN
16
3.00
2.96
-2.54
191.2
1.50
95%
960
Base
Base
VIT
StyleGAN
2.98
2.92
-2.55
189.5
1.52
96%
960
Base
Base
VIT
StyleGAN
3.55
4.18
-2.37
143.8
3.68
96%
960
Base
Base
VIT
StyleGAN
32
4.13
5.41
-2.00
123.6
5.44
2%
960
Table 4: Ablation study VIT-VQGAN. The codebook usage is calculated the percentage of used codes given a batch 256 test images averaged over the entire test set.
CelebA-HQ 256 x 256
FFHQ 256 x 256
Method
FID 4
Method
FID 4
GLOW
Kingma& & Dhariwal 2018
69.0
VDVAE (1 = 0.7)
Child 2021
38.8
NVAE
Vahdat & Kautz
2020)
40.3
VDVAE (f = 1.00)
33.5
PIONEER
(Heljakka
2018
25.3
VDVAE (f 0.8)
29.8
NCPVAE (Aneja ct 2020)
24.8
VDVAE (f 0.9)
28.5
VAEBM
Xiao el al.
2021
20.4
VOGAN+P.SNAIL
21.9
Style ALAE (Pidhorskyr el al
2020
19.2
BigGAN
12.4
DC-VAE Parrarr et al. 202
15.8
U-Net GAN Schonfeld et al. 2020
10.9
PGGAN Karras
2018
8.0
StyleGAN2
Karras 2020
3.8
VQGAN W/ Top-E samplin)
10.2
VQGAN (w/ top-E sampling)
9.6
VIT-VQGAN (Ours)
7.0
VÊT-VQGAN (Ours)
5.3
Table 5: FID comparison with unconditional image synthesis on CelebA-HQ and FFHQ.
(which used for VQGAN). For codebook learning, factorized codes with low-dimensional latent variables consistently achieve better reconstruction quality when the latent dimension is reduced from 256 16 or 8. Moreover, removing by noooraaaiiotiion leads much worse results.
5.2 IMAGE SYNTHESIS
On top of the pre-learned ViT-VQGAN image quantizer, we train stage 2 transformer models for unconditional and class-conditioned image synthesis and compare with previous work. We default model size of ViT-VQGAN-SS (small-encoder, small-decoder) for stage and VIM-Large for stage 2 (model architectures are listed in Table as we find beneficial put more computation in stage 2 while keeping stage 1 transformers lightweight. We also present a model size ablation study and comparison with VQGAN in the Appendix. Models are trained with global training batch size of 1024 for total of 450,000 training steps. We use Adam optimizer (Kingma & Ba, 2014) with 3 = 0.9 and 32 = 0.96 with the learning rate linearly warming up to a peak constant value of 4.5 x 10-4 over the first 5000 steps and then exponentially decaying x 10-5 starting from 80,000 steps. To save memory, we use factorized version of Adam, Adafactor (Shazeer & Stern 2018), with the first moment quantized into Int8 and factorized second moments. No other techniques like mixed-preciste training, model sharding, or gradient compression is used. All models trained with an input image resolution 256 x 256 on CloudTPUv4.

---

Published as a conference paper at ICLR 2022
Model
Acceptance Rate
FID
IS
Validation data
1.0
1.62
235.0
DCTransformer (Nash et al.
2021
1.0
36.5
n/a
BigGAN (Brock
2019
1.0
7.53
168.6
BigGAN-deep
203.6
IDPM (Nichol & Dhariwal 2021
1.0
12.3
n/a
ADM-G, no guid. (Dhanwal & Nichol 202
1.0
10.94
101.0
ADM-G, 1.0 guid.
1,0
4.59
186.7
VQVAF-2 Raaavi al.
2019
1.0
~31
455
VQGAN Esser al.
2021
1,0
17.04
70.6
VQGAN
10.26
125.5
VQGAN
0.25
7.35
188.6
ViT-VQGAN (Ours)
1.0
4.17
175.1
ViT-VQGAN (Ours)
0.5
3.04
227.4
Table FID comparison for class-conditional image synthesis on ImageNet with resolution 256 256. Acceptance rate shows results based on ResNet-101 classifier-based rejection sampling.
Figure 2: Uncurated set of samples from class-conditioned image generation trained ImageNet. Top row shows sampled class ids while bottom row shows fine-grained dog species from class id 184 to 189. More samples shown in Appendix.
Our main results on unconditional image synthesis on CelebA-HQ and FFHQ are summarized in Table Without top-k and top-p (nucleus) sampling heuristics, achieve FID of 7.0 on CelebA- HQ and 5.3 on FFHQ, significantly better than VQGAN (Esser et al. 2021). Table 6 shows class- conditioned image synthesis models ImageNet, following Section 4.1 Based ViT-VQGAN- SS, we achieve IS of 175.1 and FID of 4.17, improving over the IS of 70.6 and FID of 17.04 with vanilla VQGAN. When applied with classifier-based rejection sampling, the ViT-VQGAN based model further achieves best FID of 3.04 and best Inception Score of 321.7. Qualitative results are sampled and shown in Figure (see the Appendix for more).
5.3 UNSUPERVISED LEARNING
After the generative pretraining autoregressively model the density of ViT-VQGAN quantized image tokens, we evaluate the learned representation under the common linear protocol on ImageNet classification. We follow the same training syypr--prameed as the unconditional image synthesis models on ImageNet, and use ViT-VOGAN-SS image quantizer for better training throughput. As discussed in Section 3.3 the VIT-VQGAN-SS image quantizer trained without perceptual loss for unsupervised learning (perceptual loss is based on supervised VGG network trained on ImageNet, which may make comparison unfair). We apply an average pooling over the token features at specific transformer block from totally L blocks. Similar to findings reported in iGPT (Chen al.) 2020a), the representations from the middle transformer blocks has better linear- linear-probe probe accuracy the Transformer block of index 15 (36 blocks in total) for VIM-Large and index (24 blocks in total) for VIM-Base (architecture configurations listed in Table
(more study be found in Appendix). Specifically,
8

---

Published as a conference paper at ICLR 2022
Method
#Tokens
Features
Params
Top- 1
Discriminative Pretraining
Jigsaw (Noroozi & Favaro
2016b)
4096
94M
44.6
RelativePosition (Doersch et al.
2015
4096
94M
51.4
Rotation (Gidaiis et
2018b
8192
86M
55.4
AMDIM [Bachman et al
2019
8192
626M
68.1
CPC v2 (Henaf 2020
4096
303M
71.5
MoCo (He et al.
2020
8192
375M
68.6
SimCLR (Chen et al. 2020b)
8192
375M
76.5
SWAV (Caron et al. 2020)
2048
93M
75.3
DINO CCron et al.
2021
2048
85M
75.3
BYOL Grill ct al.
2020
8192
375M
78.6
BiGAN (Donahue et al.
2016)
512
138M
31.0
BigBiGAN (Donahue & Simonyan 2019b
4096
86M
56.6
Generative Pretraining
BigBiGAN
16384
344M
61.3
iGPT-L Chen et al.
2020a)
32 x 32
1536
1362M
60.3
iGPT-L
48 x 48
1536
1362M
65.2
iGPT-XL (extra data)
64 x 64
3072
6801M
68.7
iGPT-XI. (extra data, feature ensemble)
x
5 x 3072
6801 M
72.0
VIM-Base (Ours)
32 x 32
1024
650M
65.1
VIM-Large (Ours)
32 x 32
2048
1697M
73.2
VIM-Base + DALL-E dVAE quantizer
32 x 32
1024
650M
63.8
(-1.3)
VIM-Base + CNN-VQGAN quantizer
32 x 32
1024
650M
61.8 (-3.3)
Table 7: Linear-probe accuracy with different unsupervised learning methods on ImageNet. DALL- dVAE (Ramesh al. 2021) image quantizer is trained with extra image data. VIM-Large is trained without dropout in transformers.
Table 7 shows the comparisons among different approaches divided into two categories: discrimina- tive pretraining methods to distinguish among cropped augmented image patches; and generative pretraining methods to generate image pixels or patches. The linear- linear-probe probe accuracy our proposed VIM with ViT-VQGAN are superior other generative pretraining approaches like iGPT, and com- petitive with discriminative pretraining methods like BYOL (Grill et al., 2020) and DINO (Caron VIM-L significaall improves over iGPT-L,, increas-
et al. (02T). Specifically, ImageNet-pretrainge ing lincar-probe accuracy from 60.3% to 73.2% for a similar model size. VIM-L also outperforms iGPT-XL, which is larger and trained with extra web image data. Moreover, we also compare differ- ent stage quantizers including CNN-based VQGAN and pretrained DALL-E dVAE (trained with extra web-scale image data P in Table these results are all worse than ViT-VQGAN quantizer, suggesting the importance of the multiple changes defined in Section 3
ETHICS
Tasks involving generation raise a number of issues that should be considered, such as possible biases in underlying models and data-lspy with respect capabilities for people with differ- ent demographic backgrounds. The three datasets used in this paper-ImageNet, CelebA-HQ. and FFHQ are all widely used in the literature, but it is worthwhile highlighting their unique natures and recent scholarship around them.
The FFHQ datase 70,000 images collected from Flickr, all of which have licenses appro- priate for sharing, and the data maintainers provide means for individuals to opt-out of inclusion in the dataset. FFHQ specifically collected spect to faces of people. This is confirmed in our models' generated examples, which cover broad range of perceived ages, genders and ethnicities. Nevertheless, Balakrishnan et al.) (2020) provide an extensive analysis of multiple forms of bias in datasets (including CelebA-HQ and FFHQ) and algorithms for face generation; not only do they find imbalances skin tone FFHQ, but also correlations between multiple attributes such as skin tone and hair length. Based on this and other
broad range of demographics with
https:///itthie wvw hww.:aaos/fgggdaaaasseeses

---

Published as a conference paper at ICLR 2022
factors such as privacy and copyright, they argue that synthetically-crett creetedd face datasets, for which multiple attributes can be controlled, is an important direction of investment and general inquiry.
The CelebA-HQ dataset covers celebrities, which brings consequent bias toward images of attrac- tive people who are mostly in age range of twenty to forty years old. Esser et al. (2020) discusses these biases in details, and they furthermore project images from the FFHQ dataset onto CelebA- HQ: the main effect of which is to produce images of younger people with features conforming more norms of celebrities popular in the United States of America. Our model's generations appear to have a similar bias as derived from training on CelebA-HQ. Neverethless, they do show broad coverage of different perceived genders and ethnicities, but with age skewed to the 20-40 year old range.
ImageNet is, of course, quite pervasive in computer vision. In this paper, we learn to generate images given ImageNet class labels; these labels mostly concern animals, plants and things. People are sometimes generated when conditioning on classes such sunglasse data images contain people wearing sunglasses, but the generated images contain few depictions of people overall. Nevertheless, it is important to recognize that ImageNet itself was created with biases in terms of image selection and label annotation as result of its process of creation (Denton eet al. 202I). Given this, results present on ImageNet cover a significant, but nonetheless biased, sample of the kinds of scenes and objects one might encounter across the entire world.
since the training
There are also potential problematic aspects of image generation models, as demonstrated with biases found in the PULSE model (Menon al. 2020) (see Section 6) and in model correlations with human biases found social psychology (Steed & Caliskan 2021), as well as with possible uses of such models create fake media (Westerlund 2019).
REFERENCES
Jyoti Aneja, Alexander G. Schwing, Jan Kautz, and Arash Vahdat. NCP-VAE: variational autoen- coders with noise contrastive priors. CoRR, abs/2010.02917, 2020. URL https: /ectps:/aariiv
prg/abs/2010, 02917
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:/906.009/0, 2019.
Guha Balakrishnan, Yuanjun Xiong, Wei Xia, and Pietro Perona. Towards causal benchmarking of bias in face analysis algorithms, 2020.
Hangbo Bao, Li Dong, and Furu Wei. Beit: BERT pre-training of image transformers. arXiv preprint arXiv:2/06.0822 08254, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 1415500055.11... 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 098882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691 1703. PMLR. 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1697-1607. PMLR, 2020b.
10

---

Published as a conference paper at ICLR 2022
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi cemi-supervised learners. arXiv preprint arXiv:2006.10023 2020c.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autore- gressive generative model. In Jennifer G. Dy and Andreas Krause (eds.), ICML, 2018.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020d.
Rewon Child. Very deep {vae}s generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021. URL https: /oopnneviiww..o wtt.:eeeeeeeoooomm..............
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. Advances in neural information processing systems, 28:3079-3000 2015.
Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, and Hilary Nicole. On the ge- nealogy of machine learning datasets: A critical history of imagenet. Big Data & Society, 8(2): 0053951721100 10. 10.117720535112210033
2021. doi: 10. 10.117/9053951/2110399999
1725519555. URL httlhttpp///o// org/
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.). NAACL-HLT, 2019.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2/05.05233, 2021.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), NeurIPS, 2019a.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation leamiig. arXiv preprint 10X70:490, 2019b.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 105.:1660509782.. 2016.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. ICLR, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Patrick Esser, Robin Rombach, and Björ Ommer. A note on data biases in generative models. In NeurlPS 2020 Workshop on Machine Learning for Creativity and Design, 2020. URL https:
//arxiv.org/aaa 2012/ 02516
Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018a.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07725 2018b.
Ian J. Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.

---

Published as a conference paper at ICLR 2022
lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733. 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Ari Heljakka, Arno Solin, and Juho Kannala. Pioneer networks: Progressively growing generative autoencoder. In Asia conference on computer vision, 2018.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning, pp. 1182-4192. PMLR, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious of neural text degeneration. In International Conference on Learning Representations, 2020. URL https: //openreview. net/ wtw.:neeeeoommmmmmd.d.ee.eeeee/
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexci A. Efros. Image-to-image translation with conditional adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967-5976, 2017. doi: 10.1109/CVPR.2017.633
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution In European conference on computer vision, pp. 194-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im-
proved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL http://///oeeeeeeeeeeeeeeeeeeeee
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69800 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ix1 convolu- tions. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N.
and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL
iggs www.inng.....neeeeeeeeeee cctp/apper/111///1eee.. Paper. .raaer.pdff pdf
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Julia A Lasserre, Christopher M Bishop, and Thomas P Minka. Principled hybrids of generative and discriminative models. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 006), volume 1, pp. 87-94. IEEE, 2006.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.055 2017.
12

---

Published as a conference paper at ICLR 2022
Sachit Menon, Alex Damian, McCourt Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self supf-suue photo upsampling via latent space exploration of generative models. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Charlie Nash, Jacob Menick, S. Dieleman, and P. Battaglia. Generating images with sparse repre- sentations. ICML, abs/2103.03841, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic mod- els. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1172-8171. PMLR, 18-24 Jul 2021. URL ni nichol 121..111.. html
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), ECCV, 2016a.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69-84. Springer, 2016b.
Aaron van den Oord, Oriol Vinyals. and Koray Kavukcuoglu. Neural discrete representation learn- ing. arXiv preprint arXiv: 1.0997,7070 2017.
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative au- toencoder. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recog- nition (CVPR), pp. 823-832, June 2021.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Jennifer G. Dy and Andreas Krause (eds.), ICML, 2018.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition (CVPR), 2020. [to appear].
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2016.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), ICML, 2021.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VAE-2. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), NeurlPS, 2019.
Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. A u-net based discriminator for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8207-8216, 2020.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp.
PMLR, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Aché--Bur Buc, Emily B. Fox, and Roman Garnett (eds.), NeurlPS, 2019.
13

---

Published as a conference paper at ICLR 2022
Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training con- tain human-likk biases. In The 2021 ACM Conference on Faimess, Accountabilliyy, and Trans- parency (ACM FAceT 2021), 2021.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin (eds.), NeurIPS, 2020.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), NeurIPS, 2016.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. Vishwanathan, and Roman Garnett (eds.), NeurIPS, 2017.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1007:v:8807.0330
abs/1807.0332 003748 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Mika Westerlund, The emergence of deepfake technology: A review. Technology Innovation
Management Review, 9:40-53, 53, 11/2019 2019. ISSN 1927-0321. doi: http://doi.org/10.222
timreview/12822. URL Eimreview.w..................... ca aataaaittlcleeeeee cettcceee1222 1282
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. [VAEBM]: symbiosis between
variational autoencoders and energy-based models. In International Conference on Learning Representations, 2021. URLhttps: /tttps://operrevieew
Han Zhang, Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference machine learning, pp. 7354-7363 PMLR, 2019a.
Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, 2019b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
A LINEAR-PROBE ON IMAGENET
We show linear-probe accuracy from different layers in a pretrained VIM-Base Transformer model in Figure 3 Similar to iGPT (Chen et al. 2020a), we also find the last few layers may not be the best layers for discriminative features, the generative pretraining objective is to cover the original image tokens. The linear- probe accuracy increases quickly from the first
IageNet Liner-grabe Accunity
transformer output, reaches its peak at middle layers, and finally decreases for the last few blocks. Interestingly, we find for both VIM- Base and VIM-Large, the middle transformer block has the near-best result. This observation connects the transformer model to an encoder- decoder model where the encoder encodes im- age tokens into high-level semantic features and the decoder takes feature information to gener- ate output image tokens. We leave for future study regrading the interpretability of pretrained VIM models.
Figure 3: Linear-probe accuracy from differ- ent layers in a pretrained VIM-Base Transformer model.
14

---

Published as a conference paper at ICLR 2022
B MODEL SIZES OF CLASS-CONDITIONES IMAGENET SYNTHESIS
We also present results different sizes of Stage 2 Transformers for conssconniiionee image syn-
thesis and compare with VQGAN (Esser et al. 2021 summarized in Table
Model
Stage-2 Transformer Size
#Tokens
FID
IS
Validation data
1.62
235.0
VQGAN (Esser et al. 2021
1.4B
16 x
16
17.04
70.6
VIT-VQGAN VIM -Bse
650M
16
16
11.20
97.2
VIT-VQGAN + VIM-Large
16
5.3
149.9
ViT-VQGAN + VIM-Base
650M
32 x 32
8.81
110.8
ViT-VQGAN + VIM-Large
1.7B
32
x 32
4.17
175.1
Table FID comparison for class-conditional image synthesis ImageNet with different Trans- former sizes in Stage 2. Results are reported without rejection sampling.
C IMPLEMENTATION DETAILS OF FACTORIZED CODEBOOK
As we introduced Section 3.2, we use a linear projection to reduce the encoded embedding to low-dimensional variable space for code lookup. A detailed illustration is shown in Figure 4
Transformer Encoder
Linsor projection
Input
Codebook
32d
Recorstruction
Ozamisation
8152 Cocks, Code
Trarsformer Decoder
StyleSA Discrininttor
Beal/Fa Fuke
Stage 1: Image
Quantization
Figure 4: Illustration of factorized codes and codebook details.
D MORE SAMPLES ON
IMAGENET SYNTHESIS
15

---

Published as a conference paper at ICLR 2022
House Finch
Great Grey Owl
Terrapin
Komodo Dragon
Night Snake
Peacock
Irish Terrier
Norfolk Terrier
Norwich Terrier
Yorkshire Terrier
Figure 5:
Uncurated set of samples from class-conditioned generation trained on ImageNet.
16

---

Published as a conference paper at ICLR 2022
Wood Rabbit
Crock Pot
Lumbermill
Scale
Strawberry
Grand Piano
Guenon Monkey
Anemone Fish
Jay
Photocopicr
Figure 6: Uncurated set of samples from class-conditioned generation trained on ImageNet.
17


Reviews:
[{"review_id": "kWUMqEWqWLU", "reviewer": "Reviewer_6eTH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper builds on top of VQGAN and DALL-E like models to achieve SOTA FID and IS scores for image modeling and unsupervised feature learning. This paper is well-written and mostly clear, however a few clarifications would be helpful (see below).\n\nStrengths:\nthis paper achieves very good empirical results by combining various minor innovations based on first quantize the image representation followed by training a transformer based LM. The description of the architecture as well as the presentation of the results are very clear.\n\nCons:\nWhile the paper is clearly written, there do needs to be a few classifications. \n- For example, 'perceptual loss' is mentioned in several places in the paper but no definition or comment is made about what exactly is being used. It is better to have a footnote for the completeness of the paper. \n- In the introduction, it isn't super clear what are the main novelty of the paper over VQGAN and DALL-E, which also uses quantization and transformers. It would be better if contributions are outlined more clearly, especially w.r.t. the architecture in DALL-E.\n- Training time, hardware used are missing in section 5.1. How long did it take to train 500K steps, and what are the 128 accelerators? \n\nQuestions:\n- for Eq. 1 in section 3.2, is it not equivalent to get rid of the stop-gradient operators and just have a single term in the loss function, but multiply /beta to the weights of the encoder during training?\n- Are there implementation or code available accompanying this paper?\n", "summary_of_the_paper": "This paper builds on top of VQGAN and DALL-E like models to achieve SOTA FID and IS scores for image modeling and unsupervised feature learning.", "summary_of_the_review": "This paper achieves very good empirical results and build on-top of several recent works on VQ + Transformers for image modeling. However the paper lacks some details and clarifications. Given the significant improvement of FID and IS, the paper could also do more to analyze which components of the proposed method was instrumental in achieving these gains.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "biSJQbjYMzs", "reviewer": "Reviewer_GySC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The performance gains, validated on different datasets, are very clear over state-of-the-art generative models, even compared with recent denoising diffusion methods. Meanwhile, the proposed modifications are simple and apparently effective. The paper is also well and clearly written.\n\nSome other questions:\n- Could you please give some explanations on ResNet-101 classifier-based rejection sampling? Is it the one used in VQVAE2 [1] and better than top-k sampling or nucleus sampling?\n- What is the intuitive understanding to explain why  $\\ell_{2}$ normalization is so vital?\n- Is the generation quality only bounded by reconstruction? If not, what are the underlying flaws of auto-regressive probability models?\n- Considering the fixed generation order of vector-quantized image modeling, I wonder if the new proposed reconstruction method  could combine with other transformer-based models like BERT, to further advance image inpainting tasks like [2]?\n\n[1] Generating Diverse High-Fidelity Images with VQ-VAE-2, NeurIPS 2019.\n\n[2] High-Fidelity Pluralistic Image Completion With Transformers, ICCV 2021. ", "summary_of_the_paper": "This paper mainly investigates how to further improve the image generation quality of previous work VQGAN, where several modifications are proposed to train a better quantized auto-encoder model. After modeling the discrete tokens with an auto-regressive transformer, we could observe that the generation results of the proposed ViT-VQGAN are really amazing and beat most of previous works in various benchmarks. Meanwhile, ViT-VQGAN also demonstrates its exceptional representation capabilities through unsupervised linear-probing.\n\nTo summarize, it seems there are several key factors to obtain a better reconstruction model:\n\n- Project the latent codes into lower dimension, which leads to a larger codebook size and less ‘dead’ codes.\n- StyleGAN Discriminator.\n- $\\ell_{2}$ normalization on latent variables.\n- Stronger backbone (ViT-like architecture or decoder with more parameters).\n\n", "summary_of_the_review": "Overall, the value of this paper is mostly experimental, or more like “bag of tricks”. The detailed analysis and explanation of why some components are essential are not stated very clearly, which may make the whole paper more interesting, but it indeed shines some light on the image generative model. Hence currently I tend to accept this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VFVpvB3Phvh", "reviewer": "Reviewer_zYPH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n* The manuscript is well written, and easy to follow. \n* Validating the discriminative power of autoregressive models on  ViT-VQGAN is a big plus. This shows the potential of generative models on classification tasks, just as in GPT. \n\n**Weaknesses**\n* In my opinion, comparing ViT-VQGAN of 32x32 latent dimension with VQGAN of 16x16 latent dimension in Table 6 is not fair at all, because the former is much slower than the latter in terms of sampling and training speed. This is my major concern in the experiments. \n* It seems that ViT-VQGAN marginally improves VQGAN (1.28 vs. 1.49) in terms of rFID. Can we say that this improvement is significant? In addition, I’m still struggling with the reason why ViT is better than CNN in VQ methods. Is capturing the global relationship between patches (as in the classification) significant in VQ methods? \n* It seems that the reproducibility is somewhat limited, because the detailed description on the computing environment is not presented. If the standard GPU machines having multiple V100 or A100 GPUs are used to train the 1.7B transformer with a sequence length of 1,024, it may not be possible to compute the gradient, even in the case of the single sample (batch size=1). I guess that the authors have tried many techniques, including mixed-precision training (obviously the standard technique), model sharding, gradient compression, storing Adam statistics in fp16 format, and so many other techniques. Unfortunately, I cannot find these details in the manuscript. \n\n**Additional comments**\n* The uncurated samples in Figure 2 and others in the supplementary are too smooth, which typically appears when we’re sampling with low top-k or top-p. Are these samples obtained by top-p=1.0 and top-k=8192?\n* The reference to Table 3 is missing. Considering the context, the sentence “Table 4 shows FID between reconstructed images and original images ...” should be revised to refer to Table 3.", "summary_of_the_paper": "This work replaces the CNN encoder/decoder into ViT to minimize the distortion caused by quantization as minimal as possible. Experiments show that transformers learned on the discrete representation learned by ViT-VQGAN can generate high-quality samples, and provide discriminative features better than iGPT. ", "summary_of_the_review": "* I’m seeing the potential and practical benefits of the proposed method, even though it is a fairly straightforward approach by replacing CNN into ViT in the VQ-GAN framework. In addition, including the linear probing experiments is also a plus.\n* However, I have many concerns on the experiments and reproducibility. Without resolving these issues, my recommendation is borderline reject. \n\n== Updates after the rebuttal ==\n\nMy major concerns have been addressed by the response, thereby raising my score from 5 to 6. I really appreciate the authors reporting these additional results in such a short time.\n\nThough the empirical results of ViT-VQGAN are quite impressive, I still have a concern about the technical novelty. In the response to Reviewer 6eTH, the authors have argued that ViT-VQGAN is the first approach to consider vector quantization image modelling on both generation and recognition, while iGPT didn’t consider the vector quantization. However, according to the original iGPT paper (http://proceedings.mlr.press/v119/chen20s/chen20s.pdf) published in ICML’20 proceedings, they have reported the results of linear probing of iGPT on the top of discrete codes from VQ-VAE. Please revise the response to Reviewer 6eTH. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BwGbrc7u23x", "reviewer": "Reviewer_mxmU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n========\n- VQVAE and VQGAN have shown impressive generative performances but they were still lagging behind CNN based GAN models such as styleGAN. This work reduces the gap between them and shows qualitatively good generations. \n- Although the model is a relatively straight-forward combination of ViT and VQGAN, its simplicity makes the model widely applicable\n- The tricks introduced to improve the model such as projection to low dimensional space for code index lookup and l2-normalization of codes are useful findings. \n- Strong quantitative results and ablation studies make the method convincing.\n- The paper is written clearly.\n\nweaknesses / questions\n==================\n- The claim for better codebook usage can be better supported by doing an analysis on their usage.\n- In Section 3.3 multiple losses are used for training ViT-VQGAN. It would be good to see some quantitative results for choosing such a weighting between different losses, and some intuition of why the logit-laplace loss contributes to better codebook usage. \n- For VQGAN results, is it using the same number of codebook entries as ViT-VQGAN? I could not find this information.\n- As ViT has shown better performance than CNN based encoder/decoder, a natural next question is using it for the discriminator also. Have you tried this, if not, why not?  \n- Regarding styleGAN discriminator, was the improvement in metrics due to the discriminator architecture or the training procedure of stylegan discriminator (e.g its regularization)?\n- Typo: the second last row in page 6, table 4 -> table 3", "summary_of_the_paper": "This paper proposes vision transformer based VQGAN whose encoder and decoder are implemented as transformers rather than standard CNNs. It provides few improvements that demonstrates quantitatively better results in multiple datasets in unconditional / conditional image generation. It also shows how the proposed model can perform well as an unsupervised representation learning framework.\n", "summary_of_the_review": "I believe this paper introduces a simple and effective method for improving VQGAN using vision transformers. The findings in this paper and the model would be useful to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper adopts ViT in the VQ-GAN framework replacing CNN, and achieves SOTA FID and IS scores. The empirical results are pretty impressive. It could benefit some practical applications. 

The technical novelty is limited, but the tricks such as l2-normalization of codes are interesting.
Decision: Accept (Poster)