Year: 2022

Abstract:
Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively. The discrete image tokens are encoded from a learned Vision-Transformerbased VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. The improved ViT-VQGAN further improves vectorquantized image modeling tasks, including unconditional, class-conditioned image generation and unsupervised representation learning. When trained on ImageNet at 256 × 256 resolution, we achieve Inception Score (IS) of 175.1 and Fréchet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN, which obtains 70.6 and 17.04 for IS and FID, respectively. Based on ViT-VQGAN and unsupervised pretraining, we further evaluate the pretrained Transformer by averaging intermediate features, similar to Image GPT (iGPT). This ImageNet-pretrained VIM-L significantly beats iGPT-L on linear-probe accuracy from 60.3% to 73.2% for a similar model size. ViM-L also outperforms iGPT-XL which is trained with extra web image data and larger model size.

text: Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively. The discrete image tokens are encoded from a learned Vision-Transformerbased VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. The improved ViT-VQGAN further improves vectorquantized image modeling tasks, including unconditional, class-conditioned image generation and unsupervised representation learning. When trained on ImageNet at 256 × 256 resolution, we achieve Inception Score (IS) of 175.1 and Fréchet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN, which obtains 70.6 and 17.04 for IS and FID, respectively. Based on ViT-VQGAN and unsupervised pretraining, we further evaluate the pretrained Transformer by averaging intermediate features, similar to Image GPT (iGPT). This ImageNet-pretrained VIM-L significantly beats iGPT-L on linear-probe accuracy from 60.3% to 73.2% for a similar model size. ViM-L also outperforms iGPT-XL which is trained with extra web image data and larger model size.

heading: 1 INTRODUCTION
text: Natural language processing (NLP) has recently experienced dramatic improvements from learning general-purpose representations by pretraining language models on unlabeled text corpora. This strategy has produced large performance gains for a wide range of natural language generation (NLG) and natural language understanding (NLU) tasks (Dai & Le, 2015; Radford et al., 2018; 2019; Brown et al., 2020). Conceptually, generative pretraining models the data density P (X) in a tractable way, with the hope of also helping discriminative tasks of P (Y |X) (Lasserre et al., 2006); importantly, there are no limitations on whether the signals are from the language domain or others, such as vision.
In computer vision, in contrast, most recent unsupervised or self-supervised learning research focuses on applying different random augmentations to images, with the pretraining objective to distinguish image instances (Chen et al., 2020b; He et al., 2020; Chen et al., 2020d; Grill et al., 2020; Chen et al., 2020c; Caron et al., 2021). The quality of learned representation relies on manually chosen augmentations, such as random brightness, cropping, blurring, and others. Chen et al. (2020a) explored GPT-style (Radford et al., 2018) generative pretraining on images by autoregressively predicting pixels without incorporating knowledge of the 2D structure. Each pixel is represented as a 9-bit value created by clustering (R, G, B) pixel values, using k-means with k=512. Unfortunately, this color encoding does not scale to typical image resolutions as it entails very long sequences to represent the image (e.g., 224× 224 resolution leads to 50,176 tokens per image), and this demands much more memory and computation for training, compared to language models. As a result, iGPT’s maximum resolution is 64 × 64 for image recognition at scale—which severely limits its representation capabilities.
Remarkable image generation results have been achieved by pre-quantizing images into discrete latent variables and modeling them autoregressively, including VQVAE (Oord et al., 2017), DALLE (Ramesh et al., 2021) and VQGAN (Esser et al., 2021). In these approaches, a convolution neural network (CNN) is learned to auto-encode an image and a second stage CNN or Transformer is learned to model the density of encoded latent variables. These have been proved effective for image generation, but few studies have evaluated the learned representation in discriminative tasks (Ramesh et al., 2021; Esser et al., 2021).
We explore an approach we refer to as Vector-quantized Image Modeling (VIM) and apply it to both image generation and image understanding tasks. VIM follows a two-stage approach:
• Stage 1: Image Quantization. Given an image of resolution 256×256, a VisionTransformer-based VQGAN encodes it into 32×32 discretized latent codes where the codebook size is 8192. We propose multiple improvements–from architecture to codebook learning–to VQGAN (Esser et al., 2021). The resulting ViT-VQGAN is more efficient and improves reconstruction fidelity in terms of pixel-wise reconstruction metrics, Inception Score (IS) and Fréchet Inception Distance (FID). ViT-VQGAN is trained end-to-end on image-only data with combined objective functions of logit-laplace loss, `2 loss, adversarial loss and perceptual loss (Johnson et al., 2016; Zhang et al., 2018).
• Stage 2: Vector-quantized Image Modeling. We train a Transformer model to predict rasterized 32×32 = 1024 image tokens autoregressively, where image tokens are encoded by a learned Stage 1 ViT-VQGAN. For unconditional image synthesis or unsupervised learning, we pretrain a decoder-only Transformer model to predict the next token. For class-conditioned image synthesis, a class-id token is prepended before the image tokens. To evaluate the quality of unsupervised learning, we average the intermediate Transformer features and learn a linear head to predict the logit of the classes (a.k.a., linear-probe).
We show that one key component for improving both image generation and image understanding with VIM is to have a better image quantizer with respect to both computational efficiency and reconstruction quality. An efficient quantizer can speed up Stage 2 training, where random augmentations are applied first to an image, followed by the encoder of image quantizer to obtain the input tokens. Moreover, an image quantizer with better reconstruction quality can reduce information loss compared with the original image in pixel space, which is critical for image understanding tasks.
The evaluations of our proposed ViT-VQGAN and VIM are studied with three aspects. (1) We evaluate the image quantizer based on reconstruction quality metrics including `1 distance, `2 distance, log-laplace distance, as well as Inception Score (IS) and Fréchet Inception Distance (FID) of reconstructed images. (2) We evaluate the capabilities of the learned quantizer for unconditional or class-conditioned image synthesis based on FID and IS, and compare with other methods. (3) We rely on linear-probe accuracy to evaluate representations with the common intuition that good features should linearly separate the classes of downstream tasks.

heading: 2 RELATED WORK
text: Image Synthesis. Image generation has received much attention with the progress of deep generative models, including Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Zhang et al., 2019b), Variational Autoencoders (VAEs) (Kingma & Welling, 2014; Vahdat & Kautz, 2020), Diffusion Models (Song & Ermon, 2019; Dhariwal & Nichol, 2021) and Autoregressive Models (van den Oord et al., 2016; Parmar et al., 2018). Unlike many autogressive methods which generate sequence directly in pixel space, VQVAE (van den Oord et al., 2017; Razavi et al., 2019) decomposes the image generation process into two stages: the first stage trains a vector quantized autoencoder with image reconstruction objective to convert an image into a shorter sequence of discrete tokens. Then the second stage learns an autoregressive model, e.g., PixelSNAIL (Chen et al., 2018), to model the underlying distribution of token sequences. Driven by the effectiveness of VQVAE and progress in sequence modeling (Vaswani et al., 2017; Devlin et al., 2019), many approaches follow the two-stage paradigm. DALL-E (Ramesh et al., 2021) improves token prediction in second stage by using Transformers (Vaswani et al., 2017), resulting in a strong text-to-image synthesis model. VQGAN (Esser et al., 2021) further uses adversarial loss and perceptual loss (Johnson et al., 2016; Zhang et al., 2018) to train a better autoencoder in the first stage to synthesize greater detail in images.
Image Recognition with Generative Pretraining. Many image generation models (Goodfellow et al., 2014; Kingma & Welling, 2014; Radford et al., 2016; Donahue et al., 2017; Higgins et al., 2017) have been studied for their capabilities in representation learning. However, their performance is usually not superior to competing self-supervised approaches that solve auxiliary classification tasks (Noroozi & Favaro, 2016a; Gidaris et al., 2018a; van den Oord et al., 2018). BigBiGAN (Donahue & Simonyan, 2019a) first demonstrated that a generation-based model can match other self-supervised methods in representation learning on ImageNet. iGPT (Chen et al., 2020a) uses the autoregressive objective to learn a giant transformer that directly predicts pixel values, producing even more competitive results. Compared to iGPT, our method first tokenizes the original image into discrete image tokens and then trains a transformer to predict them. As a result, our approach obtains comparable results with smaller model and less data. Similar to our method in predicting image tokens, BEiT (Bao et al., 2021) follows pre-training scheme of BERT Devlin et al. (2019) by learning to recover randomly masked image tokens with a bidirectional transformer. Unlike BEiT, we explore vector-quantized image modeling for image generation in addition to image recognition.

heading: 3 VECTOR-QUANTIZED IMAGES WITH VIT-VQGAN
text: The Vector-quantized Variational AutoEncoder (VQVAE) (van den Oord et al., 2017) is a CNNbased auto-encoder whose latent space is a matrix of discrete learnable variables, trained end-to-end via straight-through estimation. Esser et al. (2021) introduce VQGAN, a model which improves upon VQVAE by introducing an adversarial loss produced by a discriminator. Below, we introduce further improvements to VQGAN that boost efficiency and enhance reconstruction quality.

heading: 3.1 VQGAN WITH VISION TRANSFORMERS
text: The core network architectures used by both VQVAE and VQGAN to encode and reconstruct images are CNNs. VQGAN introduces transformer-like elements in the form of non-local attention block (Zhang et al., 2019a), allowing it to capture distant interactions with fewer layers. We propose taking this approach one step further by replacing the CNN encoder and decoder with Vision Transformer (ViT) (Dosovitskiy et al., 2020), as shown in Figure 1. Given sufficient data (for which unlabeled image data is plentiful) we find that ViT-VQGAN is less constrained by the inductive priors imposed by convolutions. Furthermore, ViT-VQGAN yields better computational efficiency on accelerators, and produces higher quality reconstructions, as shown in Table 1.
The encoder of ViT-VQGAN first maps 8×8 non-overlapping image patches into image tokens, followed by Transformer blocks, encoding a 256×256 resolution image into a 32×32=1024 token sequence. The decoder performs the inverse operation, mapping each image token from latent variables back to 8×8 image patches and regrouping them into a 256×256 image (see Figure 1). At the output of transformer blocks, we apply a two-layer feed-forward network with a tanh activation layer
in the middle. No activation is applied at the output of ViT-VQGAN encoder or decoder. We find that this simple approach yields high quality reconstructions without any noticeable grid artifacts.

heading: 3.2 CODEBOOK LEARNING
text: Vanilla VQVAEs usually suffer from low codebook usage due to the poor initialization of the codebook. Therefore, during training a significant portion of codes are rarely used, or dead. The reduction in effective codebook size results in worse reconstructions in stage 1 quantizer training and poor diversity in stage 2 for image synthesis. As a result, VQGAN (Esser et al., 2021) relies on topk and top-p (nucleus) sampling heuristics (Holtzman et al., 2020) with a default codebook size of 1024 to obtain best results for image synthesis. We propose two improvements that can significantly encourage the codebook usage even with a larger codebook size of 8192. During image synthesis, we perform simple sampling with temperature of 1.0 without top-k and top-p heuristics.
The training objective of vector-quantization is defined as follows:
LVQ = ‖sg[ze(x)]− e‖22 + β‖ze(x)− sg[e]‖22. (1)
Here, sg(x) ≡ x, ddx sg(x) ≡ 0 is the stop-gradient operator, β is a commitment loss hyperparameter set to 0.25 in all our experiments, and e is the codebook vector. The quantized codebook index is determined by looking up the codebook vector closest to the input features ze(x) in terms of the Euclidean distance, yielding i = argminj‖ze(x)− ej‖22. Factorized codes. We introduce a linear projection from the output of the encoder to a lowdimensional latent variable space for code index lookup (e.g., reduced from a 768-d vector to a 32-d or 8-d vector per code) and find it has an immediate boost of codebook usage. The factorization can be viewed as decoupling code lookup and code embedding: we lookup the the closest variable encoded from input on a lower-dimensional lookup space and then project the matched latent code to the high-dimensional embedding space. Our experiments show reducing dimension of lookup space from 256-d to 32-d consistently improves reconstruction quality. A detailed illustration is provided in the supplementary materials.
`2-normalized codes. We also apply `2 normalization on the encoded latent variables ze(x) and codebook latent variables e. The codebook variables are initialized from a normal distribution. By mapping all latent variables on a sphere, the Euclidean distance of `2-normalized latent variables ‖`2(ze(x)) − `2(ej)‖22 evolves to the cosine similarity of two vectors between ze(x) and e, further improving training stability and reconstruction quality shown in our experiments.

heading: 3.3 VIT-VQGAN TRAINING LOSSES
text: We use a combination of logit-laplace loss, `2 loss, perceptual loss (Johnson et al., 2016; Zhang et al., 2018) based on VGG network (Simonyan & Zisserman, 2014) and GAN loss with architecture of StyleGAN discriminator (Karras et al., 2020). Loss balancing weights are configured with a hyper-parameter sweep to optimize image reconstruction quality, codebook usage, FID and Inception Score. After the sweep, we apply the same set of hyper-parameters of training losses to all datasets including CelebA-HQ, FFHQ, and ImageNet. Logit-Laplace loss can be viewed as normalized `1 loss which assumes the noise at the pixel level is laplace-distributed while `2 loss assumes
the noise is of a Gaussian distribution. We find logit-laplace loss contributes to codebook usage while `2 loss and perceptual loss significantly contribute to FID. The final loss combination we used by default is L = LVQ + 0.1 LAdv + 0.1 LPerceptual + 0.1 LLogit-laplace + 1.0L2.
One caveat on the VGG-based perceptual loss is that the VGG network is pretrained with supervised classification loss, so the supervision might leak into Stage 2 for linear-probe accuracy measurement. Thus, for all of our reported unsupervised learning results, we exclude the perceptual loss during ViT-VQGAN training. For all unconditional and class-conditioned image synthesis, we use ViTVQGAN quantizers trained with perceptual loss, as it leads to higher-fidelity reconstructions.

heading: 4 VECTOR-QUANTIZED IMAGE MODELING
text: With a learned ViT-VQGAN, images are encoded into discrete latent code ids flattened in the raster order, similar to Image GPT (Chen et al., 2020a). A decoder-only Transformer model is used to model the density of image data P (x) autoregressively as
P (x) = n∏ i=1 P (xi|x1, x2, ..., xi−1; θ), (2)
where θ is learnable weights. The training objective is to minimize the negative log-likelihood of the data L = Ex∈X [−logP (x)]. Table 2 summarizes the architecture configurations for the Transformers. We first embed discrete image token ids into a learnable embedding space at each position, with an additive learnable 2D positional embedding. Both embedding dimensions are the same as model dimension. We apply a stack of Transformer blocks to the inputs with causal attention over the entire sequence. A dropout ratio of 0.1 is used in all residual, activation and attention outputs. At the final layer of all Transformer blocks, we apply an additional layer normalization.

heading: 4.1 IMAGE SYNTHESIS
text: With a pretrained generative Transformer model, unconditional image generation is achieved by simply sampling token-by-token from the output softmax distribution. All samples used for both qualitative and quantitative results are obtained without temperature reduction. The sampled tokens are then fed into the decoder of ViT-VQGAN to decode output images. Our default Stage 1 ViTVQGAN encodes input images of resolution 256 × 256 into 32 × 32 latent codes with a codebook size 8192, while Stage 2 Transformer takes the flattened image tokens with total a length of 1024.
Class-conditioned ImageNet generation is also a widely used benchmark for measuring capabiltiy of models for image synthesis. We extend the unconditional generation to class-conditioned generation by prepending a class-id token before the image tokens. Separate embedding layers are learned from scratch for class-id token and image tokens, with the embedding dimension the same as the Transformer model dimension. During sampling, a class-id token is provided at the first position to decode the remaining image tokens autoregressively.

heading: 4.2 UNSUPERVISED LEARNING
text: For the image understanding task, we feed all image tokens of the input into a pretrained Transformer, and get a sequence of 1024 token features. Similar to Image GPT (Chen et al., 2020a),
we take a layer output at a specific block l over total blocks L, average over the sequence of token features (frozen) and insert a softmax layer (learnable) projecting averaged feature to class logits. We only take one specific Transformer block output instead of concatenating different block outputs as in iGPT (Chen et al., 2020a). We find that most discriminating feature for the linear-probe is typically near the middle of all Transformer blocks.

heading: 5 EXPERIMENTS

heading: 5.1 IMAGE QUANTIZATION
text: We train the proposed ViT-VQGAN on three datasets separately, CelebA-HQ (Karras et al., 2019), FFHQ (Karras et al., 2019), and ImageNet (Krizhevsky et al., 2012). For CelebA-HQ and FFHQ, we follow the default train and validation split as VQGAN (Esser et al., 2021).1 For Stage 1 image quantization, three different architecture sizes are experimented, Small, Base and Large for either encoder or decoder, as defined in Table 2. The smallest ViT-VQGAN-SS has a Small-size encoder and Small-size decoder, while ViT-VQGAN-BB has a Base-size encoder and Base-size decoder. The largest ViT-VQGAN-SL has an asymmetric Small-size encoder and Large-size decoder, with the motivation that Stage 2 training only requires forward propagation of the encoder of ViT-VQGAN (in inference/decoding for image synthesis, the decoder of ViT-VQGAN is still required to decode images from codes predicted during Stage 2).
We train all ViT-VQGAN models with a training batch size of 256 distributed across 128 CloudTPUv4 for a total 500,000 training steps. For both ViT-VQGAN and StyleGAN discriminator, Adam optimizer (Kingma & Ba, 2014) is used with β1 = 0.9 and β2 = 0.99 with the learning rate linearly warming up to a peak value of 1× 10−4 over 50,000 steps and then decaying to 5× 10−5 over the remaining 450,000 steps with a cosine schedule. We use a decoupled weight decay (Loshchilov & Hutter, 2017) of 1× 10−4 for both ViT-VQGAN and StyleGAN discriminator. All models are trained with an input image resolution 256× 256 on CloudTPUv4. Table 3 shows FID between reconstructed images and original images in the validation split on ImageNet, CelebA-HQ and FFHQ datasets. Without multi-scale hierarchical codebook or gumbelsoftmax, ViT-VQGAN is able to achieve better FID with a large codebook size of 8192 compared with vanilla VQGAN.
Table 4 provides extensive ablations on the proposed modifications, with empirical results on mean `1 distance, `2 distance, logit-laplace distance, Inception Score and FID on ImageNet. Among different model sizes, ViT-VQGAN-SS (small-encoder, small-decoder) performs worse than ViTVQGAN-BB (base-encoder, base-decoder) and ViT-VQGAN-SL (small-encoder, large-decoder), but achieves much better throughput. The CNN-based VQGAN architecture is worse in both quality and throughput compared with ViT-based VQGAN. The StyleGAN-based discriminator (Karras et al., 2019) is more stable and yields better reconstruction quality than PatchGAN (Isola et al., 2017)
1https://github.com/CompVis/taming-transformers
(which was used for VQGAN). For codebook learning, factorized codes with low-dimensional latent variables consistently achieve better reconstruction quality when the latent dimension is reduced from 256 to 16 or 8. Moreover, removing `2-normalization leads to much worse results.

heading: 5.2 IMAGE SYNTHESIS
text: On top of the pre-learned ViT-VQGAN image quantizer, we train stage 2 transformer models for unconditional and class-conditioned image synthesis and compare with previous work. We use a default model size of ViT-VQGAN-SS (small-encoder, small-decoder) for stage 1 and VIM-Large for stage 2 (model architectures are listed in Table 2), as we find it beneficial to put more computation in stage 2 while keeping stage 1 transformers lightweight. We also present a model size ablation study and comparison with VQGAN in the Appendix. Models are trained with a global training batch size of 1024 for a total of 450,000 training steps. We use Adam optimizer (Kingma & Ba, 2014) with β1 = 0.9 and β2 = 0.96 with the learning rate linearly warming up to a peak constant value of 4.5× 10−4 over the first 5000 steps and then exponentially decaying to 1× 10−5 starting from 80,000 steps. To save memory, we use a factorized version of Adam, Adafactor (Shazeer & Stern, 2018), with the first moment quantized into Int8 and factorized second moments. No other techniques like mixed-precision training, model sharding, or gradient compression is used. All models are trained with an input image resolution 256× 256 on CloudTPUv4.
Our main results on unconditional image synthesis on CelebA-HQ and FFHQ are summarized in Table 5. Without top-k and top-p (nucleus) sampling heuristics, we achieve FID of 7.0 on CelebAHQ and 5.3 on FFHQ, significantly better than VQGAN (Esser et al., 2021). Table 6 shows classconditioned image synthesis models on ImageNet, following Section 4.1. Based on ViT-VQGANSS, we achieve IS of 175.1 and FID of 4.17, improving over the IS of 70.6 and FID of 17.04 with vanilla VQGAN. When applied with classifier-based rejection sampling, the ViT-VQGAN based model further achieves best FID of 3.04 and best Inception Score of 321.7. Qualitative results are sampled and shown in Figure 2 (see the Appendix for more).

heading: 5.3 UNSUPERVISED LEARNING
text: After the generative pretraining to autoregressively model the density of ViT-VQGAN quantized image tokens, we evaluate the learned representation under the common linear protocol on ImageNet classification. We follow the same training hyper-parameters as the unconditional image synthesis models on ImageNet, and use ViT-VQGAN-SS image quantizer for better training throughput. As discussed in Section 3.3, the ViT-VQGAN-SS image quantizer is trained without perceptual loss for unsupervised learning (perceptual loss is based on a supervised VGG network trained on ImageNet, which may make comparison unfair). We apply an average pooling over the token features at a specific transformer block l from totally L blocks. Similar to findings reported in iGPT (Chen et al., 2020a), the representations from the middle transformer blocks has better linear-probe accuracy (more study can be found in Appendix). Specifically, we use the Transformer block of index 15 (36 blocks in total) for VIM-Large and index 10 (24 blocks in total) for VIM-Base (architecture configurations are listed in Table 2).
Table 7 shows the comparisons among different approaches divided into two categories: discriminative pretraining methods to distinguish among cropped or augmented image patches; and generative pretraining methods to generate image pixels or patches. The linear-probe accuracy of our proposed VIM with ViT-VQGAN are superior to other generative pretraining approaches like iGPT, and competitive with discriminative pretraining methods like BYOL (Grill et al., 2020) and DINO (Caron et al., 2021). Specifically, ImageNet-pretrained VIM-L significantly improves over iGPT-L, increasing linear-probe accuracy from 60.3% to 73.2% for a similar model size. VIM-L also outperforms iGPT-XL, which is larger and trained with extra web image data. Moreover, we also compare different stage 1 quantizers including CNN-based VQGAN and pretrained DALL-E dVAE (trained with extra web-scale image data)2 in Table 7; these results are all worse than ViT-VQGAN quantizer, suggesting the importance of the multiple changes defined in Section 3.

heading: 6 ETHICS
text: Tasks involving generation raise a number of issues that should be considered, such as possible biases in underlying models and data—especially with respect to capabilities for people with different demographic backgrounds. The three datasets used in this paper–ImageNet, CelebA-HQ, and FFHQ–are all widely used in the literature, but it is worthwhile highlighting their unique natures and recent scholarship around them.
The FFHQ dataset3 contains 70,000 images collected from Flickr, all of which have licenses appropriate for sharing, and the data maintainers provide means for individuals to opt-out of inclusion in the dataset. FFHQ was specifically collected to cover a broad range of demographics with respect to faces of people. This is confirmed in our models’ generated examples, which cover a broad range of perceived ages, genders and ethnicities. Nevertheless, Balakrishnan et al. (2020) provide an extensive analysis of multiple forms of bias in datasets (including CelebA-HQ and FFHQ) and algorithms for face generation; not only do they find imbalances in skin tone in FFHQ, but also correlations between multiple attributes such as skin tone and hair length. Based on this and other
2https://github.com/openai/dall-e 3https://github.com/NVlabs/ffhq-dataset
factors such as privacy and copyright, they argue that synthetically-created face datasets, for which multiple attributes can be controlled, is an important direction of investment and general inquiry.
The CelebA-HQ dataset covers celebrities, which brings a consequent bias toward images of attractive people who are mostly in age range of twenty to forty years old. Esser et al. (2020) discusses these biases in details, and they furthermore project images from the FFHQ dataset onto CelebAHQ: the main effect of which is to produce images of younger people with features conforming more to norms of celebrities popular in the United States of America. Our model’s generations appear to have a similar bias as derived from training on CelebA-HQ. Neverethless, they do show broad coverage of different perceived genders and ethnicities, but with age skewed to the 20-40 year old range.
ImageNet is, of course, quite pervasive in computer vision. In this paper, we learn to generate images given ImageNet class labels; these labels mostly concern animals, plants and things. People are sometimes generated when conditioning on classes such as sunglasses since the training data images contain people wearing sunglasses, but the generated images contain few depictions of people overall. Nevertheless, it is important to recognize that ImageNet itself was created with biases in terms of image selection and label annotation as a result of its process of creation (Denton et al., 2021). Given this, results present on ImageNet cover a significant, but nonetheless biased, sample of the kinds of scenes and objects one might encounter across the entire world.
There are also potential problematic aspects of image generation models, as demonstrated with biases found in the PULSE model (Menon et al., 2020) (see Section 6) and in model correlations with human biases found in social psychology (Steed & Caliskan, 2021), as well as with possible uses of such models to create fake media (Westerlund, 2019).

heading: A LINEAR-PROBE ON IMAGENET
text: We show linear-probe accuracy from different layers in a pretrained VIM-Base Transformer model in Figure 3. Similar to iGPT (Chen et al., 2020a), we also find the last few layers may not be the best layers for discriminative features, as the generative pretraining objective is to recover the original image tokens. The linearprobe accuracy increases quickly from the first transformer output, reaches its peak at middle layers, and finally decreases for the last few blocks. Interestingly, we find for both VIMBase and VIM-Large, the middle transformer block has the near-best result. This observation connects the transformer model to an encoderdecoder model where the encoder encodes image tokens into high-level semantic features and the decoder takes feature information to gener-
ate output image tokens. We leave for future study regrading the interpretability of pretrained VIM models.

heading: B MODEL SIZES OF CLASS-CONDITIONED IMAGENET SYNTHESIS
text: We also present results of different sizes of Stage 2 Transformers for class-conditioned image synthesis and compare with VQGAN (Esser et al., 2021)4 summarized in Table 8.
C IMPLEMENTATION DETAILS OF FACTORIZED CODEBOOK
As we introduced in Section 3.2, we use a linear projection to reduce the encoded embedding to a low-dimensional variable space for code lookup. A detailed illustration is shown in Figure 4.

heading: D MORE SAMPLES ON CLASS-CONDITIONED IMAGENET SYNTHESIS
text: 4https://github.com/CompVis/taming-transformers

Review summaries:
This paper mainly investigates how to further improve the image generation quality of previous work VQGAN, where several modifications are proposed to train a better quantized auto-encoder model. After modeling the discrete tokens with an auto-regressive transformer, we could observe that the generation results of the proposed ViT-VQGAN are really amazing and beat most of previous works in various benchmarks. Meanwhile, ViT-VQGAN also demonstrates its exceptional representation capabilities through unsupervised linear-probing.  To summarize, it seems there are several key factors to obtain a better reconstruction model:  - Project the latent codes into lower dimension, which leads to a larger codebook size and less ‘dead’ codes. - StyleGAN Discriminator. - $\ell_{2}$ normalization on latent variables. - Stronger backbone (ViT-like architecture or decoder with more parameters).  

This paper proposes vision transformer based VQGAN whose encoder and decoder are implemented as transformers rather than standard CNNs. It provides few improvements that demonstrates quantitatively better results in multiple datasets in unconditional / conditional image generation. It also shows how the proposed model can perform well as an unsupervised representation learning framework.