Language Model Beats Diffusion - Tokenizer is key to visual generation
Authors: Lijun Yu
Jose Lezama
Nitesh Bharadwaj Gundavarapu
Luca Versari
Kihyuk Sohn
David Minnen
Yong Cheng
Agrim Gupta
Xiuye Gu
Alexander G Hauptmann
Boqing Gong
Ming-Hsuan Yang
Irfan Essa
David A Ross
Lu Jiang
Venue: ICLR 2024 poster (2024)
Keywords: language model
diffusion model
video generation
visual tokenization
Primary area: generative models
TL;DR: With a good visual tokenizer, language model style transformer outperforms diffusion models on image and video generation.

Abstract: While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation. To effectively use LLMs for visual generation, one crucial component is the visual tokenizer that maps pixel-space inputs to discrete tokens appropriate for LLM learning. In this paper, we introduce \modelname{}, a video tokenizer designed to generate concise and expressive tokens for both videos and images using a common token vocabulary. Equipped with this new tokenizer, we show that LLMs outperform diffusion models on standard image and video generation benchmarks including ImageNet and Kinetics. In addition, we demonstrate that our tokenizer surpasses the previously top-performing video tokenizer on two more tasks: (1) video compression comparable to the next-generation video codec (VCC) according to human evaluations, and (2) learning effective representations for action recognition tasks.

Published as a conference paper at ICLR 2024
LANGUAGE MODEL BEATS DIFFUSION TOKENIZER IS KEY TO VISUAL GENERATION
Lijun Yuut* José Lezamal Nitesh B. Gundavarapul Luca Versarit Kihyuk Sohn
David Minnen Yong Cheng"
Agrim Gupta' Xiuye Gu Alexander G. Hauptmann Irfan Essal
Boqing Gong? Ming-Hsuan Yang!
David A. Rosst Lu Jiang
"Goglle, *Carnegie Mellon University
ABSTRACT
While Large Language Models (LLMS) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation. To effectively use LLMs for visual generation. one crucial com- ponent is the visual tokenizer that maps pixel-space inputs to discrete tokens ap- propriate for LLM learning. In this paper, we introduce MAGVIT-v2, a video tokenizer designed to generate concise and expressive tokens for both videos and images using a common token vocabulary. Equipped with this new tokenizer, we show that LLMs outperform diffusion models on standard image and video gener- ation benchmarks including ImageNet and Kinetics. In addition, we demonstrate that our tokenizer surpasses the previously top- nop-performing video tokenizer two more tasks: (1) video compression comparable to the next- next-generation video codec (VVC) according human evaluations, and (2) learning effective representations for action recognition tasks.
INTRODUCTION
Large transformer-based language models, commonly referred to LMs or LLMs, are the de facto models for natural language generation (OpenAI, 2023; Google, 2023; Touvron et al., 2023). Over time, LMs have expanded their capabilities to generate content in various modalities, asserting their dominance in other domains like audio (Agostinelli et al., 2023), speech (Rubenstein et al., 2023), code generation (Li al., 2023), medical applications (Singhal et al., 2023) and robotics (Zitkovich et al., 2023).
LMS are capable of generating images and videos. To do so, the image pixels are mapped into sequence of discrete tokens by a visual tokenizer (c.f. Section 2). These tokens are then fed into the LM transformer, if they were lexical words, for generative modeling. Despite notable ad- vancements in employing LMs for visual generation (Esser et al., 2021; Chang et al., 2022), LMs still do not perform as well as diffusion models (Rombach et al., 2022). For instance, when evalu- ating on the ImageNet dataset, gold standard benchmark for image generation, the best language model (Lee et al., 2022) underperforms the diffusion model (Gao al., 2023) by substantial 48% margin (FID 3.41 1.79 when generating images at the 256x 256 resolution).
Why do language models lag behind diffusion models in visual generation? This paper suggests that primary reason is the lack of a good visual representation, resembling our natural language system, for effectively modeling the visual world. To substantiate this hypothesis, this paper shows that, when utilizing good visual tokenizer, the masked language model (Devlin et al., 2019; Chang et al., 2022; Yu et al., 2023a) surpasses the mate-of-theean diffusion models in terms of both generation fidelity and efficiency across image and video benchmarks, given the same training data, comparable model size, and training budget. To the best of our knowledge, this provides the first evidence that language models beat diffusion models on the hallmark ImageNet benchmark.
is worth emphasizing that our intention is not to assert whether the language model is superior to others, but promote the exploration of visual tokenization methods for L.MSs. A fundamental difference of LLMS from other models, such as diffusion models, is that LLMs utilize discrete latent format: tokens obtained from a visual tokenizer. We show that the values of these discrete visual tokens should not be overlooked considering their distinct advantages as follows. (1) Com- patibility with LLMs. The main advantage of a token representation is that it shares the same form
Work done during a research internship Google Research.

---

Published as a conference paper at ICLR 2024
as language tokens, making it straightforward to leverage the optimizations community has de- veloped over many years for LLMs. This includes faster training and inference speeds (Shazeer, 2019; Lester et al., 2021), advancements in model infrastructure (Dao et al., 2022: Du et al., 2022), learning recipes for model scaling (Brown et al., 2020; Chowdhery et al., 2022), and GPU/TPU op- timization, among other innovations. Unifying vision and language by the same token space could set the stage for a true multimodal LLM that can understand, generate, and reason within our visual environment. (2) Compressed representation. The discrete token may offer fresh perspective on video compression. The visual tokens can serve as new video compression format to reduce disk storage and bandwidth during internet transfers. Unlike compressed RGB pixels, these tokens can be fed directly into generative models, bypassing the conventional decompression and latent encoding steps. This allows for faster processing in generative video applications, especially bene- ficial in edge computing cases. (3) Visual understanding benefits. Prior research has shown that the discrete tokens are valuable as a pre-training target in self-supervised representation learning, as discussed in BEiT (Bao et al., 2021) and BEVT (Wang et al., 2022). Additionally, rescarch finds that using tokens as the model inputs improves the robustness and generalization (Mao et al., 2021).
In this paper, we introduce MAGVIT-v2, video tokenizer designed to map videos (and images) into compact discrete tokens. Our model is built on the state-of of-thh--rttttttt video tokenizer, MAGVIT (Yu al., 2023a), within the VQ-VAE framework (Van Den Oord et al., 2017). We propose two new techniques. First, a novel lookup-free quantization method enables the learning of a large vocabulary that is able improve generation quality of the language model. Second, through extensive empir- ical analyses, we have identified modifications to the tokenizer that not only enhance generation quality but also enable the tokenization of both images and videos using a shared vocabulary.
We empirically demonstrate that our model outperforms the previously lop-performing video tok- enizer, MAGVIT, in three key areas. First, our model significantly improves the generation quality of MAGVIT, establishing the state of the art on the common image and video benchmarks. Second, user studies indicate that its compression quality exceeds that of MAGVIT and the current video compression standard, HEVC (Sullivan et al., 2012). Moreover, it is on par with the next-generation video codec, VVC (Bross et al., 2021). Finally, we show that, compared to MAGVIT, our new tokens are stronger for video understanding tasks across two setups and three datasets. The main contributions of this work are:
A new video tokenizer that outperforms the previously best-performing video tokenizer in three areas: visual generation, video compression, and action recognition.
A novel lookup-free quantization approach that enables improving the visual generation quality of language models by learning a large vocabulary.
To the best of our knowledge, the first evidence suggesting that a language model can outperform diffusion models on ImageNet when provided with the same training data, an equivalent model size, and similar training budget.
A video compressor with better quality than HEVC and VVC, at similar bit rates, according to user studies. To our knowledge, this is the first successful attempt of a visual tokenizer designed for video generation to achieve comparable results to standard codecs.
2 BACKGROUND
Language Model (LM) for visual generation. LMs have been extended to generate images and videos. A visual tokenizer f is used to first map visual inputs into a sequence of discrete tokens. A video V E RTxHxWxs (or image when T = 1) is tokenized into a discrete representation X = f(V) E (1,2.... K}T* x xW', where K is the codebook (vocabulary) size of the visual tokenizer. X is flattened into 1D token sequence obtained using raster scan ordering and then fed into an LM transformer for generative modeling.
Two types of LMs are commonly used for visual generation. The Autoregressive LM (AR-LM) includes ImageGPT (Chen al., 2020), DALL-E (Ramesh et al., 2021). 1), Parti (Yu et al., 2022b), etc. An AR-LM predicts the next token given the previous tokens along with additional conditioning information с using a categorical distribution for pe Xxxi c). During inference, AR-LMs use the standard autoregressive decoding over the tokens. Finally, the tokens are converted back to pixels by a decoder associated with the visual tokenizer.
The Masked LM (MLM) is another type of language model for visual generation, such as: MaskGIT (Chang et al., 2022), MAGVIT (Yu et al., 2023a), Phenaki (Villegas et al., 2022), and MUSE (Chang al., 2023), among others. An MLM is trained using masked token objective (De-
2

---

Published as a conference paper at ICLR 2024
vlin et al., 2019), where some tokens in the sequence are randomly masked and need to be predicted
given the observed tokens. Let m {0,1}" be a random binary sequence where m mn1 E [0, n - 1]. The MLM learns Po(x; {xj : m, = 1, vÿ}; c) for all where m, 0. To generate video or image during inference, the MLM uses the non-autoregressive decoding algorithms for images and videos (Chang al., 2022; Yu et al., 2023a). The decoding starts with a fully masked sequence, which is iteratively filled by repeating two steps: (1) sample the whole sequence x(1) from po given the non-masked tokens from the previous step, (2) re-mask the [A(L) .nJ tokens in x(1) with the lowest probability. following a decreasing masking ratio schedule X(t), according to timestamp t. Denoising Diffusion Models (DDM). DDMs (Sohl-Dickstein et al.. 2015; Song & Ermon, 2019) are regarded as the statc-of-the-art in visual generation due to their high-quality image (Dhariwal & Nichol, 2021: Ho et al., 2022a) and video generation (Ho al., 2022c). For instance, DDPM (Ho et al., 2020) learns a denoising process parameterized as conditional Gaussian distributions over image pixels. Recently, diffusion models and language models have displayed a significant overlap. Recent DDMs diffuse over latents rather than raw pixels. These latents are obtained using models similar to the visual tokenizer used by LMs. In fact, the very first latent diffusion, proposed by Rombach et al. (2022), is derived from visual tokenizer. Additionally, the diffusion model's architecture has been shifting from the U-Net to the transformer architecture (Peebles & Xie, 2022). Consequently, the boundaries between diffusion and language models in visual generation have become less distinct. Yet, a fundamental difference between DDMs and LMs lies in the latent format, i.e., continuous vs. discrete. We have discussed the benefits of having discrete tokens in Section I and will show that the proposed tokenizer improves in these aspects.
Visual tokenization. Visual tokenization plays an essential role mapping pixels into a discrete representation suitable for generative modeling. VQ-VAE (Van Den Oord et al., 2017) is comnr-- stone work in image tokenization. A VQ- IQ-VAE model consists of a convolutional neural network (CNN) encoder, RT*HxW*x xxXXXx x3 the VQ-VAE's encoder E produces latent embeddings Z E(V) R x H' xW'xd
cector-quantization (VQ) bottleneck, and CNN decoder. Given video V
Each embedding vector E Rd in Z is then passed through the vector quantizer which assigns it to the closest entry e Rd in the learned codebook embedding Ce C R xd.
(z) = Ci2 where i arg min ||z c2l2. j=((122....K)
(1)
To get discrete tokens, we drop the embedding dimension and represent Z by its indices X xw', 21, .K) KKT'x xI''',,'' For decoding, embeddings of all image tokens given as input to the decoder D reconstruct the input V D(Z). Following VQ-VAE, VQGAN (Esser et al., 2021) introduces an adversarial loss and feature-level perceptual losses to enhance the image quality.
Video tokenization is more challenging and VQGAN has been adapted to meet this purpose (Ge et al., 2022: Villegas et al., 2022: Yu et al., 2023a). The state of the art in video tokenization MAGVIT (Yu al.. 2023a), which introduces a better 3D architecture, an inflation technique for initialization using image pre-training, and robust training losses. With MAGVIT, the LMs achieve leading generation quality across multiple video benchmarks. However, MAGVIT struggles to tokenize images and often results in noticeable flickering in longer videos.
3 METHOD
We introduce new video tokenizer designed to map the spatial-temporal dynamics from a visual scene into compact discrete tokens suitable for language models. Compared with image generation, video generation still faces substantial challenges in generating consistent and realistic motion. We are interested in exploring the capabilities of language models in tackling this unsolved challenge. Therefore, this paper focuses on video tokenizer that can effectively represent video for generative modeling. Our approach builds upon the state-of-the-a video tokenizer, MAGVIT, as detailed in Yu et al. (2023a). This section highlights two new designs: lookup- free quantizer and collection of enhancements to the tokenizer model.
3.1 LOOKUP-FREE QUANTIZER
Although the community has made great progress in developing VQ-VAEs, the relationship be- tween improvements in the reconstruction quality and subsequent generation quality is still not well understood. A common misconception is that improving reconstruction equates to improving the generation of the language model. For example, enlarging the vocabulary can improve reconstruc- tion quality. However, such improvement only extends to generation when the vocabulary size is small, and very large vocabulary can actually hurt the performance of the language model.
3

---

Published as a conference paper at ICLR 2024
As illustrated by the dashed curves in Fig. 1, the reconstruction FID, indicated by the right y-axis (where a lower value is better), improves as the vocabulary size (the z-axis) increases. The orange solid curve in Fig. 1 represents the LM's generation quality (the left y-axis). The generation FID initially improves but deteriorates for larger vocabulary. This may shed light on why the vocabulary size of most language models for visual generation is around 1-8k (Esser al., 2021; Villegas al., 2022), which is significantly smaller than the size of natural language vocabulary, i.e. over 200k.
A simple trick for training a larger codebook in- volves decreasing the code embedding dimen- sion when increasing the vocabulary size (Yu et al., 2022a). This trick captures the intuition of limiting the representational capacity of indi- vidual tokens, which in turn facilitates learning over the distribution of large vocabulary.
VQ Reconstruction
VQ Generation
L LFQ Reconstruction
LFQ Generation
3.4
Reconstrattion FID.
Generation FID
19.0
18.2
3.0
17.4
2.8
16.6
2.2
Lookup-Free Quantization (LFQ). Moti-
15.8
1.8
vated by the above observation, we reduce the
VQ-VAE codebook's 's embedding dimension to zero. Formally, the codebook C E R Rxxd xd is re- placed with an integer set C where |C] K. Recall that in VQ-VAE models. the quantizer must look up all K d-dimensional embeddings in the codebook, where d is typically 256, when computing the closest codebook entry to the en- coder output. This new design eliminates the need for such embedding lookup entirely hence call it lookup-free quantizaation (LFQ). We found that LFQ grow the vocabulary size in way benefiting the generation quality of language models. As shown by the blue curves in Fig. 1, both reconstruction and generation consistently improves as the vocabulary size increases property not observed in current VQ- VAE methods.
15.0
1.4
10
12
14 16
Vocabulary (2*k)
Figure 1: Reconstruction and generation qual- ity curves in FID on ImageNet when scaling the tokenizer's 's vocabulary size with Vector Quantiza- tion (VQ) and Lookup-Free Quantization (LFQ). Comparison is done at 128 x 128 resolution using an MLM with 306-372M parameters.
While various LFQ methods are available, this paper diseusses a straightforward variant that sumes independent codebook dimensions and binary latents. Specifically, the latent space of LFQ is decomposed as the Cartesian product of single- diieeesiienii variables, C X i-1 C.. Given feature vector z E Iloog, K each dimension of the quantized representation g(z) is obtained from:
log, K
q(2:) Cr.j: where j arg ming |z, Ci,ell,
where C., is the j-th value in C.. With C, = [,1,1), the arg min can be computed by the sign function as
((2i) sign(z;) = - ( & 0} + 11z, > 0}.
(3)
With LFQ, the token index for (z)) is given by:
Log, K
log, K
Inder(z)
Σ
arg ||z Ci,K
Call
2 2 0},
h-0 0
where |Col = 1 sets the virtual basis.
We add an entropy penalty during training to encourage codebook utilization:
(S) This penalty is inspired by a similar loss used in image VQGAN model (Chang et al., 2022), which is also found in entropy-basee clustering (Jansen et al., 2020). In LFFQ, given the independence among dimensions, we rewrite H (q(z)) = Silog K H(q(z))) The [[[(g(z)) term can be approximated with sub-groups of dimensions for K > 218 where direct estimation is memory bound.
Lentropy = B (q(z))] H[]])
We note that there are various other variants of L.FQ, e.g., opting for the multivariant over the binary codebook C₁ or employing other quantization techniques such Agustsson et al. (2019). As the first paper to introduce this concept, we focus on the simplest form with independent binary dimensions, which shows promising improvements. Other LFQ methods merit further research.
In addition to the entropy penalty (Eq, (5)), an LFQ-based tokenizer is trained using the standard combination of reconstruction, CAN, perceptual, and commitment losses (Esser et al., 2021), ex- cluding the inapplicable codebook loss. Following Yu et al. (2023a), we use LeCAM regulariza- tion (Tseng et al., 2021) for improved stability.
4

---

Published as a conference paper at ICLR 2024
%
rel
loate
Patch Emb
Patch Emb
Paten Emb
Caural conk. time
3D CNN
3D CNN
3D CNN
Cancaa CNN
Spatial Transformar
Causal Tranaformer
Cauaal Transforme
Quartizer
Quantiorr
Quantoer
Apa
(a) C-VIVIT
(b) C-VIVIT MAGVIT
(c) Causal 3D CNN
Figure 2: Causal tokenizer architecture comparison. The decoders, which omitted from the
figure, employ an architecture that is symmetric to the encoder. See detailed architecture diagram in the Appendis.
3.2 VISUAL. TOKENIZER MODEL IMPROVEMENT
Joint image-video tokenization. A desirable feature of visual tokenization is the capability to tokenize images and videos using a shared codebook. However, the MAGVIT tokenizer, which utilizes the 3D CNN, faces challenges in tokenizing images due to the temporal receptive field.
To build a joint image-video tokenizer, a new design is needed. We begin our discussion by revis- iting existing method C-ViVIT (Villegas et al., 2022). As depicted in Fig. 2a, C-ViViT employs full spatial transformer blocks combined with causal temporal transformer blocks. This approach performs reasonably well but has two drawbacks. First, unlike CNNs, the positional embeddings makes difficult to tokenize spatial resolutions that were not seen during training. Second, empiri- cally we found that 3D CNNS perform better than spatial transformer and produce tokens with better spatial causality of the corresponding patch.
To tackle these drawbacks, we explore two plausible designs. Fig. 2b combines C-ViViT and MAGVIT. Assuming a temporal compression ratio of 4, a 3D CNN processes blocks of 4 frames followed by a causal transformer. In Fig. 2c, we use the temporally causal 3D convolution to replace the regular 3D CNN. Specifically, the temporal padding scheme for regular 3D convolution layer with kernel size (ke, kn, kw) includes [سيي frames before and | frames after the input frames. In contrast, causal 3D convolution layer pads with k 1 frames before the input and nothing after, so that the output for each frame only depends on the previous frames. In consequence, the first frame is always independent of other frames, allowing the model to tokenize single images.
Temporal convolutional subsampling with stride s is sufficient for sx down-sampling by mapping 1+ x frames into + t. After a regular x up-sampling, we drop the first which maps 1 + ł frames into 1 + f. and allows for the tokenization of a single image. Tab. 5a empirically compares the designs Fig. 2, and we find that the causal 3D CNN performs the best. Architecture modifications.
resulting frames,
addition to using causal 3D CNN layers, we made several other architectural modifications to improve upon the MAGVIT model. First, we change the encoder downsamplers from average pooling into strided convolutions to leverage learned kernels, and re- place the decoder upsamplers from nearest resizing followed by convolution with depth-to- to-sceee operator. Second, we defer the temporal downsampling from the first few encoder blocks to the last ones. In addition, the downsampling layer in the discriminator now utilizes 3D blur pooling (Zhang, 2019) to encourage shift invariance. Finally, we add one adaptive group normalization layer before the residual blocks at each resolution in the decoder pass in the quantized latents as the control signal following StyleGAN (Karras et al., 2019). Tabs. 5b and 5c empirically verify these designs.
Token factorization for efficient prediction. The output tokens can be fed into language models to generate videos. To assist smaller transformers predicting in a large vocabulary, we can factorize the LFQ token's latent space into equal subspaces. For instance, rather than predicting using codebook of size 218, we can predict in two concatenated codebooks, each of size 29, We embed each subspace token separately and use their embedding summation as the token embedding for the transformer input. We find it beneficial to use weight tying (Press & Wolf, 2017), a common technique in language modeling. which involves sharing the weights between the embedding and softmax layers. For the output layer with a factorized vocabulary, we use the embedding matrix for each subspace to obtain logits with seperate prediction heads.
5

---

Published as a conference paper at ICLR 2024
1324x1328
LPIPS 0.1665
0.1167
0.1082
512x768 Original
LPIPS
0.1349
0.0788
0.0726 Ours (Web images)
VQGAN (ImageNet)
Ours (ImageNet)
Figure 3: Image reconstruction samples with different tokenizers. We compare the VQGAN used in MaskGIT (Chang et al., 2022) with two of our models trained ImageNet and web im- ages (Chen et al., 2022). Original images are by Eric TERRADE and Barth Bailey Unsplash.
4 EXPERIMENTS
This section empirically verifies the proposed tokenizer across three distinct tasks: video and image generation, video compression, and action recognition. Fig. 3 visually compares the re- construction quality of our tokenizer with prior works. More qualitative samples are shown at https://nagvis cs. cmu.edu/va
4.1 EXPERIMENTAL SETUPS
Datasets. We use Kinetics-600 (K600) (Carreira al., 2018) and UCF-101 (Soomro al., 2012) for video generation experiments, along with ImageNet (Deng et al., 2009) for image generaton. In addition, MCL-JCV (Wang al., 2016) used as the testbed for video compression, with Kinetics- 400 (K400) (Kay et al., 2017) 7) and SSv2 (Goyal et al., 2017) for video understanding.
Implementation details We follow the tokenizer training setting and hyperparameters in (Yu et al., 2023a), unless stated otherwise. LFQ is used, which eliminates the codebook embedding, increase the default codebook size K 218 The weight of Lentropy follows an annealing schedule with a 3x higher starting point and linearly decays to a fixed value of 0.1 within 2k steps. We defer details regarding the evaluation setup of each subsection the Appendix.
4.2 VISUAL GENERATION
The masked language model (MLM) (Devlin et al., 2019) used in image and video generation. To verify the tokenizer, We select the MLM due to its competitive performance on benchmark datasets (Yu et al., 2023a; Lezama al., 2023). In the Appendix,
employ the same MLM transformers in MAGVIT (Yu al., 2023a).
also show that autoregressive language model (AR- LM) coupled with the proposed tokenizer outperforms the prior work MAGVIT. As we use smaller MLM 300M parameters) with large codebook (2 2662K), the token factorization as discussed in Section 3.2 is applied using two heads with each predicting from codebook of size 29.
Video generation. We consider two standard video benchmarks, UCF-101 for class- conditionll generation and K600 for frame prediction with 5-frame condition. FVD (Unterthiner et al., 2018 8) is used our primary evaluation metric. Tab. shows that model surpasses all prior arts in both

---

Published as a conference paper at ICLR 2024
Table 1: Video generation results: frame prediction on Kinetics-600 6000 and class-conditional genera- tion on UCF-101. We adopt the evaluation protocol of MAGVIT.
Type
Method
K600 FVDJ UCF FVD] #Params #Steps
GAN
TrIVD-GAN-FP (Lue et al., 2020)
25.7
Diffusion
Video Diffusion (Ho al., 2022c)
16.2+0.3
1.1B
256
Diffusion
RIN (Jabri et al., 2023)
10.8
411M
1000
AR-LM + VQ
TATS (Ge et aI., 2022)
332118
321M
1024
MLM VQ
Phenaki (Villegas et al., 2022)
0.24
227M
48
MLM VQ
MAGVIT (Yu et al., 2023a)
9.9
7612
306M
12
MLM + LFQ
non-causal baseline
11,6
307M
12
5.2
12
MLM+ + LFQ
MAGVIT-v2 (this paper)
307M
4.3++0.1
58+3
24
Table 2: Image generation results: class-conditional generation on ImageNet 512 Guidance
indicates the classifier-fres diffusion guidance (Ho & Salimans, 2021). indicates usage of extra training data. We adopt the evaluation protocol and implementation of ADM.
w/o guidance IS
w/ guidance IS1
Type
Method
#Params #Steps
FID
FID
GAN
StyleGAN-XI XL (Sauer al.. 2022)
2.41
267.8
168M
1
Diff. VAE*
DiT-XL/2 (Peebles & Xie, 2022)
12.03
105.3
3.04
240.8
675M
250
Diffusion
ADM+Upsample (Dhariwal Nichol, 2021)
9.96
121.8
3.85
221.7
731M
2000
Diffusion
RIN (Jabri et al., 2023)
3.95
216.0
320M
1000
Diffusion
simple diffusion (Hoogeboom et al., 2023)
3.54
205.3
3.02
248.7
2B
512
Diffusion
VDM++ (Kingma & Gao, 2023)
2.99
232.2
2.65
278.1
2B
512
MLM + VQ
MaskGIT (Chang al., 2022)
7.32
156.0
227M
12
MLM + VQ
OPC+Ups (Lezama ct al., 2023)
3.62
249.4
619M
72
4.61
192.4
12
MLM + LFQ MAGVIT-v2 (this paper)
307M
3.07
213.1
1.91
324.3
64
benchmarks. Specifically, it outperforms the previous best model MAGVIT by a large margin, while using the same MLM transformer backbone. In addition, it significantly outperforms the non-causal baseline on frame prediction, highlighting the contribution of the causal tokenizer. These results demonstrate the essential role of good visual tokenizer in enabling LMs generate high-quality videos. Fig. 4 shows qualitative samples from the model.
Image generation on ImageNet. We evaluate MAGVIT-v2 on image generation under the stan- dard ImageNet class-conditional setting. We present results for resolution 512x512 Tab. 2 and refer to the Appendix for 256x 256 results. FID (Heusel et al., 2017) and Inception Score (IS) (Sali- mans et al., 2016) are used evaluation metries. Our model surpasses the best performing diffusion models both in sampling quality (FID and IS) and inference-tine efficiency (sampling steps).
is worth noting that all the models compared are trained using the same ImageNet training data, with a comparable model size and training budget. Therefore, the performance primarily evaluates the model's capabilities. The masked language model, equipped with our tokenizer, exhibits no- table improvement in FID over the best diffusion model baseline 512 512 (FID=1.91 vs. 2.65, 28%(). While this margin narrows at 256x resolution, the MLM uses a 50% reduced model size and needs much fewer decoding steps (e.g., 64 250) to get the image generation quality. Qualitative samples in comparison with other models are shown in Fig. 5.
4.3 VIDEO COMPRESSION
Ours
MAGVIT
VVC
HEVC
2200
We conduct a subjective rater study to assess the compression quality of MAGVIT-v2. The study is conducted on the 30 videos of the MCL-JCV dataset, resized to resolution of 640x Sixteen raters are engaged, each providing responses to an average of roughly 800 pairwise-preferecce questions.
2000
Elo soore
1800
1600
1400
0.08
bits pixel (bpp)
Figure 6: Video compression rater study.
7

---

Published as a conference paper at ICLR 2024
Condition Generation
Figure 4: Frame prediction samples on Kinetics-600.
DPC
ADM- Upsample
simple diffusion
VDM++
StylcGAN-XI.
Ours
Figure 5: Class-conditional generation samples ImageNet 512x We compare with each of the previous works with a random sample from the same image class.
We calculate Elo scores (Elo & Sloan, 2008) based on pairwise preferences to quantify the relative visual quality between the models. The study compares our model with MAGVIT as well as the current video compression standard HEVC (H.265) video codec (Sullivan et al., 2012) and the next- generation codec VVC (H.266) (Bross al., 2021). As shown in Fig. 6, raters prefer our model to the compared methods at multiple bit rates.
Table 3: Video compression metrics.
We also compare the compression quality using common distortion metrics (LPIPS, PSNR, and MS- SSIM). Tab. 3 compares 0.0384 bpp, the bit rate of MAGVIT, with full curves in the Appendix. The results show that our model outperforms MAGVIT on all metrics, and it outperforms all methods on LPIPS, metric which correlates more closely with subjective quality assessments. At equal bit rates, standard codecs may render local details more accurately than neural models but also introduce block artifacts, detrimental to perceptual quality yet not captured by PSNR and MS-SSIM (Agusts- son et al., 2019). Despite promising results with TPUs, further research is needed to adapt our model to run efficiently on CPUs like standard codecs.
Method
LPIPS|
PSNR1
MS-SSIM
HEVC (Sullivan et al., 2012)
0.199
30.10
0.943
VVC (Bross 2021)
0.153
32.65
0.966
MAGVIT (Yu al., 2023a)
0.144
23.70
0.846
MAGVIT (this paper)
Table Video action recognition performance (classification accuracyf x 100).
4.4 VIDEO UNDERSTANDING
In this subsection, assess the tokenizer's ca- pability to learn video understanding model for action recognition. Two setups are exam- ined: (1) using tokens prediction targets for the transformer's output, and (2) using tokens the input to the transformer. For the former setup, we use a similar architecture following the BEVT (Wang et al., 2022) pre- training. For the tokens as inputs, to work with the ViViT backbone (Amab et al., 2021), we detokenize the tokens to pixels before feeding them to frozen ViVIT transformers trained raw pixels.
Token as transformer's: Output SSv2
Input
Tokenizer
SSv2
K400
K600
3D VQ-VAE
64.13
41.27
44.44
45.67
MAGVIT (Yu et al., 2023a)
67.22
57.34
72.29
74.65
MAGVIT-12 (this paper)
67.38
62.40
75.34 77.93
Raw pixel
64.83
63.08
76.13
78.92
HoG descriptor (Wei et al., 2022)
65.86
n/a
n/a
n/a
Tab. 4 shows that MAGVIT-v2 outperforms the previous best MAGVIT these evaluations. Specif- ically, when using the decoded tokens input, the performance approaches that of the model trained
8

---

Published as a conference paper at ICLR 2024
Table 5: Ablation study verifying key design choices.
(a) Causal architectures on UCF-101. (b) Image tokenization on
(c) Video tokenization on UCF- 101,
ImageNet 128x 128.
FID is calculated on the first frame.
FVD1 LPIPS
FIDJ
LPIPS
#Params FID, FVD]
MAGVIT
24.55 0.0988
MAGVIT
39M
n/a
107.15
MAGVIT .LQQ
2.65
0.1292
+ LTR & large vocabulary
16.12
0,0694
MAGVIT-2:
C-VIVIT MAGVIT
C-VIVIT
Causal CNN
67M
90M
58M
28.02
13.52
7.06
437 54
96.33
+ uptlowmsampler
deeper model
large vocahulary
2,48
1.21
1.34
1.20
0.0686
0.0821
0.0790
3D blur pooling
deeper model
late temporal downsample
updowwsamplerr
11.11
15.37
8.62 0.0537
8,90 0,0542
0.0653
adaptive normalization
1.15
0,0685
with ground pixels using the same ViVIT backbone. While these numbers are still worse than the state-of-the-art in action recognition, they represent solid improvements credited the new tokenizer.
4.5 ABLATION STUDY
In Fig. 1, we have ablated LFQ vs. VQ and the vocabulary size. In Tab. 5, we validate the key designs proposed in Section 3.2. Specifically, Tab. 5a compares the architecture illustrated in Fig. 2; Tab. 5b and Tab. 5c verify the LFQ and other improvements on ImageNet and UCF-101, respectively.
5 RELATED WORK
Visual tokenization. Beyond the VQ-VAE models discussed in Section 2, additional models have been proposed. ViT-VQGAN (Yu et al., 2022a) introduces transformer blocks as a substitute for CNNs for image tokenization. C-ViVIT (Villegas et al., 2022) further extends this idea for video to- kenization. Early studies on video tokenization treat frames as independent images with no temporal compression (Wu et al., 2022; Gupta et al., 2022). Later research (Yan al., 2021; Ge et al., 2022; Yu et al., 2023a) integrates 3D CNNs to tokenize spatial-temporal volumes. Despite these advances in vector quantization (VQ), the codebook learned by previous VQ models is relatively small (e.g.. 8k) due to the difficulty in improving the generation quality with larger vocabularies. In contrast, our tokenizer can induce a large vocabulary (e.g., 262k) that can be effectively modeled by an LM, leading to enhanced image and video generation quality.
(ima-ae,- videa]. Text-to-image and text-to- video generation has gamnered significant rapid advancements using both language models (Yu et al., 2023b; Chang al., 2023) and diffusion models (Ho et al., 2022a; Blattmann et al., 2023; Singer et al., 2022; Ge et al., 2023; Ramesh al.. 2022). Although diffusion models, such as Midjourney. are considered the top performers in these tasks, it is unclear whether their advantage stems from the model, data, or some other unidentified factors. Indeed, is challenging to scientifically compare these text-to-image models as they are trained on varied datasets, with some even being proprietary data, under inconsistent training conditions. To facilitate a fairer comparison, this paper prioritizes using the ImageNet and Kinetics benchmarks.
Diffusion models. Exhibiting high quality sampling, pixel-space diffusion models (Sohl- Dickstein et al.. 2015: Song & Ermon, 2019; Ho al., 2020) raised the top of the generative modeling space for both image (Ho et al.. 2020; Dhariwal & Nichol, 2021; Saharia et al., 2022) and video (Ho al., 2022c;a; Singer et al., 2022) synthesis. The pixel-space denoising diffusion models (DDMs) are later refined by the latent-space DDM (Rombach et al., 2022), which conducts diffusion over the continuous latent embeddings derived from a pre-trained variational autoencoder (VAE). Binary latents for image modeling were used in Wang et al. (2023), where the diffusion process is parameterized with Bernoulli distributions. Recent studies have identified advantages in substituting the U-Net (Ronneberger al., 2015) denoising backbone with Transformer (Peebles & Xie, 2022; Jabri et al., 2023) or a hybrid of both (Hoogeboom et al., 2023), making the distinctions between diffusion and language models in visual generation more blurred, with a key distinction being their latent format - continuous for diffusion and discrete for language models.
6 CONCLUSION AND FUTURE WORK
We introduce MAGVIT-v2, a novel video tokenizer that exploits lookup-free quantization along with architectural advancements to tokenize images and video with a shared vocabulary. The ex- periments show that our tokenizer outperforms the previously leading video tokenizer across three areas: visual generation, video compression, and action recognition in videos. Our results suggest that good visual tokenizer is key for enabling language models to excel in image and video gener- ation. These results demonstrate the great capabilities of LMs in visual generation, and advocate for further exploration of advanced visual tokenization methods designed for L.MMs.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like to express our gratitude Yu-Chuan Su and Sergey loffe for their valuable com- ments our work, to Josh Dillon for discussions, and to Eirikur Agustsson for help in compression evaluation.
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicLM: Generating music from text. arXiv:2301.11325, 2023.
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Gener- ative adversarial networks for extreme learned image compression. In ICCV, 2019. 4, 8
Anurag Arnb,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.
ViVIT: A video vision transformer. ICCV, 2021. 8, 17
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEIT: BERT pre-training of image transform- ers. In ICLR, 2021. 2, 17
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion
models. In CVPR, 2023. 9
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2018. 19
Benjamin Bross, Ye-Kui Wang. Yan Ye, Shan Liu, Jianle Chen, Gary J. Sullivan, and Jens-Rainer Ohm. Overview of the versatile video coding (VVC) standard and its applications. IEEE Trans- actions on Circuits and Systems for Video Technology, 31(10):3736-376 3764, 2021. 2,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In NeurIPS, 2020. 2
Joao Carreira, Eric Noland, Andras note about Kinetics-600. arXiv:
Chloe Hillier, and Andrew Zisserman. A short 2018. 6
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. MaskGIT: Masked genera- tive image transformer. In CVPR, 2022. 1, 2, 3, 4, 6, 7,
Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, José Lezama, Lu Jiang. Ming-Hswe Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. ICML., 2023.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In ICML, 2020. 2
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. PaLI: jointly-scale multilingual language-image model. In ICLR, 2022. 6
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaI.: Scaling language modeling with pathways. arXiv:2204.02311, 2022.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, Flashattention: Fast and memory-
efficient exact attention with io-awareness. In NewrIPS, 2022. 2
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 6
10

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. 1,
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NeurlPS, 2021. 3, 9, 19
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaMs: Efficient scaling language models with mixture-of-experts. In ICML, 2022.
Arpad E. Elo and Sam Sloan. The rating of chessplayers past and present. Ishi Press International, 2008. 8, 17
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR. 2021. 1,3, 19
Gustav Theodor Fechner. Elemente der psychophysik, volume 2. Breitkopf u. Härtel, 1860. 17
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. arXiv:2303. 14389, 2023. 1, 15, 19
songwei Ge. Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic VQGAN and tme-sennstive transformer. In ECCV, 2022. 3, 7,79 9
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. arXiv preprint arXiv:2305. 10474, 2023. 9
Google. PaLM technical report. arXiv:2305. 2023.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski. Joanna Materzynska, Susanne West- phal, Hcuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martin, and Li Fei-Fei. MaskViT: Masked visual pre-training for video prediction. In ICLR, 2022. 9
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017. 7
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS Workshops, 2021. 7,
15, 19
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 3, 9
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv:2210.02300 2022a. 3,
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. JMLR, 23(1):2249-2811, 2022b. 19
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In ICLR Workshops, 2022c. 3, 7.9
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In ICML, 2023. 7, 9, 19
Allan Jabri, David J Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. In ICMI, 2023. 7, 9, 19
11

---

Published as a conference paper at ICLR 2024
Aren Jansen, Daniel PW Ellis, Shawn Hershey, Channing Moore, Manoj Plakal, Ashok C Popat, and Rif A Saurous. Coincidence, categorization, and consolidation: Learning to recognize sounds with minimal supervision. In ICASSP, 2020. 4
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 5
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv: / 770500695, 2017. 6
Diederik Kingma and Ruiqi Gao. Understanding the diffusion objective as a weighted integral of elbos. arXiv:2303.00885 2023. 7, 19
Doyup Lee, Chihcon Kim, Sachoon Kim, Minsu Cho, and WOOK SHIN HAN. Draft-and-revis:: Effective image generation with contextual rq rq-transformer. In NeurIPS, 2022. 1, 19
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021. 2
José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with Foken-Critie In ECCV. 2022. 19
José Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, and Irfan Essa. Discrete predictor-corrector diffusion models for image synthesis. In ICLR, 2023. 6, 7, 15, 19
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. StarCoder: may the source be with you! arXiv:2305.0616/1 2023. 1
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transfornation-basee adversarial video prediction on large-scale data. 2xXiv:2003.04035, 2020. 7
Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, and Irfan Essa Discrete representations strengthen vision transformer robustness. In ICLR, 2021. 2
OpenAl. GPT-4 technical report. arXiv:2303.08774 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv:22/2.09748, 2022. 3, 7, 9, 19
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In EACL, 2017.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. 2:X1v 202, 9
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 1, 9, 19
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomed- ical image segmentation. In MICCAI, 2015.
Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry. Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. AudioPal.M: A large language model that can speak and listen. arXiv:2306. / /2925, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NewrIPS, 2022.
12

---

Published as a conference paper at ICLR 2024
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NeurlPS, 2016.
Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling stylegan to large diverse datasets. In SIGGRAPII, 2022. 7, 19
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arX1v:1911.02150, 2019. 2
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv:2209.14777 2022. 9
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Colc-Lewis, Darlene Neal, et al. Towards expert-level medical question answering with large language models. arXiv:2305. 096/9,997700 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015. 3,9
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019. 3,9
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. 121i:: 2012. 6
Gary Sullivan, Jens-Raine Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions Circuits and Systems for Video Technology, 22(( 19111114616666 2012. 2,8
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302. 13971, 2023.
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming- Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In CVPR, 2021.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv:1812.01777., 2018. 2018. 6
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017. 2, 3
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Herman Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv:2210.02399, 2022. 2, 3, 4, 7.9
Haiqiang Wang. Weihao Gan, Sudeng Hu, Joe Yuchieh Lin, Lina Jin, Longguang Song, Ping Wang, Ioannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. MCL-JCV: a IND-based H. 264/AVC video quality assessment dataset. ICIP, 2016. 6, 17
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang. Luowei Zhou, and Lu Yuan. BEVT: BERT pretraining of video transformers. In CVPR, 2022. 2, 8, 17, 18
Ze Wang, Jiang Wang, Zicheng Liu, and Qiang Qiu. Binary latent diffusion. In CVPR, 2023. 9, 19
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yun Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022. 8
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. NÜWA: Visual synthesis pre-training for neural visual world creation. In ECCV, 2022. 9
13

---

Published as a conference paper at ICLR 2024
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using VQ-VAE and transformers. arXiv:2104.10152 2021.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQ- GAN. In ICLR, 2022a. 4, 9
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv:2206. 10789, 2022b. 2
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsua Yang. Yuan Hao, Irfan Essa, et al. MAGVIT: Masked generative video transformer. In CVPR, 2023a. 1, 3, 4, 7, 8, 9, 15, 19
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-moda models: Pretraining and instruction tuning. arXiv:2309.0259) 2023b. 9
Richard Zhang. Making convolutional networks shift-invariant again. ICMI,, 2019. 5
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar, Fast training of diffusion models with masked transformers. arXiv:2306.09300 2023. 19
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. RT-2: Vision-language-action models transfer web knowl- edge to robotic control. In CoRL, 2023.
14

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
A.1 IMAGE AND VIDEO GENERATION
We set up two image tokenizers to downsample by 16x and 32x, where they are used for generation at x 256 and 512x respectively. In both cases, an image represented as 16x 16 tokens. We train them on the ImageNet training for 270 epochs using batch size of 256, both with 256x256 images.
With this tokenizer train Masked Language Model following Yu al. (2023a), using the token factorization described in Section 3.2. We train for 1080 epochs in accordance with the prior best model MDT (Gao al., 2023), with batch size 1024 for better efficiency. For preprocessing and data augmentation, we randomly crop 80-100% of an image while keeping the aspect ratio, followed by random horizontal flipping. The class label is dropped for 10% of the training batches to enable classifier- freeeerrrreeeeeeee guidance (Ho & Salimans, 2021). For unguided generation, we use temperature 30 for 512 512 and 15 for 256 256 in the non-autoregressive decoding. For guided generation, we adopt the guidance schedule from Gao et al. (2023) with temperature scaling (Lezama et al., 2023), where we use guidance scale 25 with temperature 15.
We inflate an image tokenizer trained at 128 128 for video modeling. Different from the inflation Yu al. (2023a), we the temporally last slice to correspond to the causal padding scheme. In addition, we disable the inflation for the discriminator and train it from scratch for better stability. We train the causal video tokenizer on Kinetics-6f 660 training set for 190 epochs with batch size 256. This tokenizer is also used in subsequent evaluations of video compression and action recognition.
With the causal tokenizer producing x 16x 16 tokens for 17x 128x x 128 clip, the first 2 x 16x 16 tokens are provided as the condition of the first 5 frames, per the standard setup of Kinetics-60l frame prediction benchmark. We train the MLM transformer following Yu et al. (2023a) with token factorization for 360 epochs with batch size 256. The model is sampled with cosine schedule using temperature 32.
A.2 MODEL SETUP AND HYPERPARAMETERS
Fig. 7 illustrates the architecture of proposed MAGVIT-v2. We provide detailed training hyper- parameters for our models as listed below:
Video input: 17 frames, frame stride 128 128 resolution.
Base channels: 128,
VQVAE channel multipliers: 2, 4.
Discriminator channel multipliers: 2,4, 4, 4.4.
Number residual blocks: 4.
Latent shape: 5 16 x 16.
Vocabulary size: 218
Initialization: central inflation from 2D model trained on ImageNet with this setup.
Entropy loss weight: 0.1.
Entropy loss annealing steps: 2000.
Entropy loss annealing factor:
Reconstruction loss weight: 5.0.
Generator loss type: Non-saturating.
Generator adversarial loss weight: 0.1.
Discriminator gradient penalty: r1 with cost 10.
Perceptual loss weight: 0.1.
Commitment loss weight: 0.25.
LeCAM weight: 0.001.
Peak learning rate: 10-4
Learning rate schedule: linear warm up and cosine decay.
Optimizer: Adam with 31 0 and B2 0.99.
EMA model decay rate: 0.999.
Batch size: 256.
15

---

Published as a conference paper at ICLR 2024
VQVAE
Discriminator
Encoder
Decoder
Conv 3x3x3, 3->128 LeakyReLU
GroupNorm + Swish
T-Causal Conv 3x3x3, 3->128
T-Causal Conv 3x3x3, 128->3
ResBlockDown 128->256
3x
ResBlock 128
ResBlockDown 256->512
ResBlock 256->128
ResBlock 128
ResBlockDown 512->512
Adaptive GroupNorm
T-Causal Conv 3x3x3,
T-Causal 3x3x3,
ResBlockDown 512->512
128->128, stride 1x2x2
256->1024, depth space 1x2x2
ResBlockDown 512->512
ResBlock 128->256
4x
ResBlock 256
Conv 3x3x3, 512
LeakyReLU
3x
ResBlock 256
Adaptive GroupNorm
Linear 512 LeakyReLU
T-Causal Conv 3x3x3,
T-Causal Conv 3x3x3, 256->2048, depth to space 2x2x2
Linear 1
256->256, stride 2x2x2
3x
ResBlock 256
BesBlockDown
Conv 3x3x3, X-Y LeakkRReL
X->Y
ResBlock 512->256
Blur Pooling
4x
ResBlock 256
2x2x2
Adaptive GroupNorm
Xsy
T-Causal Conv 3x3x3,
T-Causal Conv 3x3x3,
Conv 3x3x3, LeaakyRelU
256->256, stride 2x2x2
512->4096, depth space 2x2x2
ResBlock
ResBlock 256=512
ResBlock 512
GroupNorm Swish
T-Causal Cony 3x3x3, X
3x
ResBlock 512
GroupNorm + Swish
Adaptive GroupNorm
4x
ResBlock 512
4x
ResBlock 512
GroupNorm Swish
ResBlock X>>
T-Causal Conv 3x3x3, X-Y
GroupNorm Swish Conv 1x1x1, 512->18
T-Causal Conv 3x3x3,
T-Causal Corv 1x1x1. X>>
18->512
GroupNorrm
LFQ.
T-Causal Conv 3x3x3, Yoy
Figure 7: MAGVIT-v2 tokenizer architecture. T-Causal Conv refers temporally causal convo- lution.
16

---

Published as a conference paper at ICLR 2024
A.3 VIDEO COMPRESSION EVALUATION
Which video looks closer to the original?
Original
mothod
sobot snodlesnn) acket de carees meda
Figure 8: Rating interface for subjective compression evaluation.
To rate the quality of the different methods, we use a two-alternative forced choice rating method- ology (Fechner, 1860). As this methodology produces a sequence of binary decisions, we calculate Elo scores (Elo & Sloan, 2008) based on pairwise preferences to quantify the relative visual quality between the models. The study was conducted on the 30 videos of the MCL-JCV dataset (Wang al., 2016), scaled down to a resolution 640x360 pixels. Sixteen raters are engaged, each pro- viding responses to average of roughly 800 pairwise-preference preference questions. The questions presented with an interface that parallels the one used for the Challenge on Learned Image Com- pression (ht tnttp:/ / /ccoprprrsssseeeeeeeeeeee cension..... cc/), ce/) extended to comparing videos, as shown in Fig. 8. Raters are instructed to compare the two videos and are not allowed to pause the videos.
A.4 VIDEO UNDERSTANDING EXPERIMENTS
Tokens prediction targets. BEIT (Bao al., 2021) and BEVT (Wang et al., 2022) class of models pretrain visual encoders on pixel inputs by predicting tokens as targets in a mookeling framework, and demonstrate mate-of-the--al downstream results. We use simplified BEVT pre- training setup to test the effectiveness our video tokens as targets for masked modeling. The main difference is that this reason, we also drop the multiple decoders completely and adopt an encoder-only architecture similar BETT. Detailed pre-training and fine-tuning setup presented in Tab. 6. In Tab. 4 of the main paper, we show that our video tokens are effective targets for masked modeling based video understanding.
drop the image-stream from pre-training and only the video stream and for
Tokens inputs. In Tab. 4, we show that we can re-use video understanding models trained on pixels using our video tokens as input, with very minimal performance drop. For this experiment, train factorized variant of the ViT model (Arnab et al., 2021) pixels, and evaluate it on de- tokenized pixels from our model. We use the same syper-parameters as used in Arnab et al. (2021) with Base sized model operating 32 frames of inputs 224p resolution. For the Kinetics-600 as the Kinetics-400 experiments.
experiment, we use the same
B ADDITIONAL RESULTS
For better visualization, the generated video samples can be viewed at https: http///aggvt....teeeee. cm..
edu/ /v2.
17

---

Published as a conference paper at ICLR 2024
Table Experimental configurations with tokens as targets.
Config
SSv2 Pre- Pre-Training
SSv2 Fine-tuning
inputs input size
pixels
pixels
16 x 224 x 224 x 3
16 224 x 224 3
targets
tokens
classes
encoder
ViT-B
VīT-B
decoder
linear
linear
masking
block-tube (Wang et al., 2022)
none
masking ratio
0.75
0.0
mask temporal length
16
0
batch size
1024
512
training epochs
800
50
ViT sequence length
8 x 16 x 16
8 x 16 x 16
optimization
optimizer
AdamW
Adam W
optimizer momentum
0.9
0.9
layer decay
0.75
0.75
weight decay
0.05
0.05 cosine decay
learning rate schedule
cosine decay
warmup epochs
40
5
data augmentations
random horizontal flip
true
false
label smoothing
0.1
0.1
mixup
none
0.8
cutmix
none
1.0
droppath
0.0
0.1
dropout
0.1
0.0
random color augmentation
false
false
Ouss
MAGVIT
VVC
HEVC
Ous
MAGVIT VVC
HEVe
Ours
MAGVIT
YVC
HEVE
0.95
PSNR1
32
MS-S5MM
0.90
0.10
per
bris piael (hpp))
(a) LPIPS
(b) PSNR1
(c) MS-SSIM1
Figure 9: Video compression metries, supplementary to Tab.
Where are the text-to-image results? We want emphasize that our goal is to develop video tokenizer, and many of the proposed techniques are designed specifically for videos. Text-to-image may be out of the scope of our paper. We are currently training text-to-video models that require considerable computational resources. Due to time constraints, these results are not available at the moment. We intend to add the generated videos in the next revision. However, it is important to note that comparing these text-to-image or rext-to-vides models scientifically is challenging. These models were trained different datasets, and some were even based on proprictary or non-public data, all under varying training conditions.
18

---

Published as a conference paper at ICLR 2024
Table 7: Class-conditional image generation on ImageNet 256x256. Guidance indicates the classifier-free diffusion guidance (Ho & Salimans, 2021). indicates usage of extra training data. We adopt the evaluation protocol and implementation ADM.
wlo guidance IS1
w/ guidance IS1
Type
Method
# Params Steps
FID.
FID
GAN
BigGAN-deep (Brock et al., 2018)
6.95
171.4
160M
GAN
StyleGAN-XL (Sauer al., 2022)
2.30
265.1
166M
Dift. VAE
LDM- (Romfach et al., 2022]
10.56
103.5"
"3.60
247.7
400M
250
Diff. + VAE"
DiT-XL// (Peehles Xie, 2022)
9.62
121.5
2.27
278.2
675M
250
Diff. BAE
Binary latent diffusion ( (Wang et al., 2023)
8.21
162.3
172M
64
Diffusion
ADM+Upsample (Dhariwal & Nichol, 2021)
7.49
127.5
3.94
215.8
608M
2000
Diff. VAE
MDT (Gao al., 2023)
143.0
283.0
676M
250
Diff. VAE*
MaskDiT (Zheng et al., 2023)
178.0
2.28
276.6
736M
Diffusion
CDM (Ho ct al., 2022b)
4.88
158.7
8100
Diffusson
RIN (Jabri et al., 2023)
3,42
182.0
410M
1000
Diffusion
simple difffusion (Hloogeboom al., 2023)
2.77
21188
2.44
256.3
2B
512
Diffusion
VDM++ (Kingma Gao, 2023)
2.40
225.3
2.12
267.7
2B
512
AR-LM VQ
VQGAN (Esser et al., 2021)
15.78
78.3
1.4B
256
MLM + VQ
MaskGIT (Chang ct al., 2022)
6.18
182.1
227M
MLM + VQ
Token-Critic (Lezama et al., 2022)
4.69
174.5
368M
36
MLM VQ
Contextual RQ-Transformer (Lee al., 2022)
3.41
224.6
1.4B
72
MLM + VQ
DPC (Lezama et al., 2023)
244.8
454M
MLM LFQ
MAGVIT-v2 (this paper)
3.65
200.5
1.78
319.4
307M
64
Table 8: Video generation results: class-conditional generation UCF-101 with AR-LM models. We use the same transformer configuration as MLM experiments but without vocabulary factor- ization and weight tying. As a result, the AR-LM with MAGVIT-v2 uses more parameters in the embedding table and the softmax layer.
Tokenizer
FVD.
#Params
#Steps
MAGVIT (Yu et al., 2023a)
265
306M
1024
MAGVIT-v2 (this paper)
109
840M
1280
19


Reviews:
[{"review_id": "6Q12UZEOwW", "reviewer": "Reviewer_rVyC", "summary": "The paper proposes a novel visual tokenizer based on lookup-free quantization (LFQ). With the growth of the vocabulary size LFQ consistently improve both reconstruction and generation quality, which is in stark contrast with Vector Quantization (VQ) where an increased vocabulary size reduces reconstruction error but hurts generation results. The tokenizer can be integrated with MAGVIT and achieves state-of-the-art performance on video generation. The tokenizer can also improve video compression and video recognition.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Starting from an interesting finding that enlarging the vocabulary improves reconstruction quality but hurts generation results, the paper proposes solutions (LFQ) to tame both reconstruction and generation simultaneously.\n\n2. A detailed study of architecture modifications that improves up MAGVIT supported by extensive ablations. \n\n3. The tokenizer is proved to benefit video generation, compression and recognition. It will potentially have a huge impact on the general audience of video understanding.", "weaknesses": "1. For video compression results, it would be better if there is a PSNR/LPIPS/MS-SSIM-bpp curve comparing the performance across different bpps.\n\n2. In the video recognition setup, it seems unnecessary to detokenize the visual tokens back to pixels since BEVT and BEIT can work with tokenized input. I understand one of the main reasons is that the underlying recognition model is the ViViT which takes raw pixels as input (as stated in the draft). However, you may also have a comparison with BEVT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XeMEGPcs1v", "reviewer": "Reviewer_sfJe", "summary": "The paper focuses on learning a video / image tokenizer to discretize video / images so that they can be modeled using a Language Model. They specifically introduce one innovation in this setting: A lookup free quantizer. They show that in this limit of using a large vocabulary and no lookup, the tokenizer reconstruction and LM generation quality both increase with vocabulary size. The authors also show that the learned tokenizer performs very well as a compression scheme.", "questions": "* The question “Why masked LM and not AR LM for image / video generation?” for evaluating the tokenizer was not clearly answered.\n* No explicit definition of the objective for training the tokenizer (loss function)\n* No mention of decoder in VQ-VAE and VQ-VAE loss used when we use no lookup\n* More motivation needed on why the authors choose to use a causal encoder for tokenizer when doing masked LM for image / video generation\n* It’s not clear why the authors tackle video generation if the aim was to understand the fundamentals of tokenization. It may be desirable for them to clearly motivate why they study videos and not images alone?\n* It may be interesting for the reader to understand the computational complexity of both the tokenizer (encoder) and the detokenizer (decoder) and how they compare with video or audio codecs", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper does a good job of motivating the core thesis of the paper: How to design a tokenizer for image / video? The authors also do a good job of presenting this idea to the uninitiated. It is also pretty clear that the work is significant to academia and industry given that it helps unify image generation with image understanding and natural language generation and understanding techniques. The ideas in the paper are well explained. The authors also do a thorough job of running experiments to substantiate many claims including numerous ablations. Some of the important technical insights like the lookup free quantizer (and in general lower dimensional code words) helping in generation quality are substantiated by experimental results", "weaknesses": "My main concern is the completeness of the exposition in the paper. The authors assume that the reader is familiar with the state of the art in video tokenization and details do get rather buried in the many “deltas” relative to the baseline. I do understand the space limitations but it might be helpful if the authors try to make the core system / model design more explicit. Lot of the ideas like factorization of the output space in the decoder (and associated weight tying) for example are just mentioned in passing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pVbccFjRLl", "reviewer": "Reviewer_k12R", "summary": "This paper introduces a novel visual tokenizer designed to enhance large language models in producing high-quality images and videos. Experimental results show that, when integrated with the proposed tokenizer, LLM surpasses diffusion models in standard benchmarks such as ImageNet, UCF-101, and Kinetics-600. Additionally, the paper presents promising results in video compression and representation learning.", "questions": "1.\tIn Figure 1, it's highlighted that the VQ generation FID sees a pivotal change at a vocabulary size of 2^14, while the LFG generation FID consistently improves. I'm curious to understand how the LFG generation FID would respond to even larger vocabulary sizes.\n2.\tRegarding Table 3, what could be the reason behind the proposed method's PSNR and MS-SSIM values being inferior to those of the standard video codec?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThis paper presents the first evidence of large language models surpassing diffusion models on the ImageNet benchmark.\n2.\tThe paper proposes a novel lookup-free quantization approach, providing a promising direction for expanding vocabulary size in LLM-based visual generation.\n3.\tThe motivation is clear, and the overall presentation is coherent and easy to follow.\n4.\tGood results on visual generation video compression, and video representation learning.", "weaknesses": "1.\tWhile the presented method is tailored for masked LM, many of the prevailing and powerful LLMs, such as LLaMA [A], employ an autoregressive approach. Incorporating results from AR-LM would greatly enhance the paper's relevance to the community.\n2.\tIn Table 4, despite the good action recognition performance showcased by the proposed method, it doesn't conclusively establish its efficacy as a viable self-supervised pre-training target. Notably, some pivotal baselines, like pixel colors and the image descriptor from MaskFeat [B], are missing.\n\n[A] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., ... & Lample, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.\n\n[B] Wei, C., Fan, H., Xie, S., Wu, C. Y., Yuille, A., & Feichtenhofer, C. Masked feature prediction for self-supervised visual pre-training. In CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)