LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior
Authors: Hanyu Wang
Saksham Suri
Yixuan Ren
Hao Chen
Abhinav Shrivastava
Venue: ICLR 2025 Oral (2025)
Keywords: Video Generation
Visual Tokenization
Primary area: generative models
TL;DR: A holistic video tokenizer with a learned autoregressive generative prior.

Abstract: We present LARP, a novel video tokenizer designed to overcome limitations in current video tokenization methods for autoregressive (AR) generative models. Unlike traditional patchwise tokenizers that directly encode local visual patches into discrete tokens, LARP introduces a holistic tokenization scheme that gathers information from the visual content using a set of learned holistic queries. This design allows LARP to capture more global and semantic representations, rather than being limited to local patch-level information. Furthermore, it offers flexibility by supporting an arbitrary number of discrete tokens, enabling adaptive and efficient tokenization based on the specific requirements of the task. To align the discrete token space with downstream AR generation tasks, LARP integrates a lightweight AR transformer as a training-time prior model that predicts the next token on its discrete latent space. By incorporating the prior model during training, LARP learns a latent space that is not only optimized for video reconstruction but is also structured in a way that is more conducive to autoregressive generation. Moreover, this process defines a sequential order for the discrete tokens, progressively pushing them toward an optimal configuration during training, ensuring smoother and more accurate AR generation at inference time. Comprehensive experiments demonstrate LARPs strong performance, achieving state-of-the-art FVD on the UCF101 class-conditional video generation benchmark. LARP enhances the compatibility of AR models with videos and opens up the potential to build unified high-fidelity multimodal large language models (MLLMs). Project page:  https://hywang66.github.io/larp/

Published as a conference paper at ICLR 2025
LARP: TOKENIZING VIDEOS WITH A LEARNED AUTOREGRESSIVE GENERATIVE PRIOR
Hanyu Wang, Saksham Suri, Yixuan Ren,
Hao Chen
Abhinav Shrivastava'
University of Maryland, College Park
(hywang66, sakshams, yxren @udd edu fchenh, abhi nav) @cs. umd edu Project page: https::/hyyytttt
gith http::://yyyrrgg..gg
Effectiyeness
300
250
Alt
dlass-condiion video generation
100
5o
Kinelis-EG0 800 framo prodiction
1024
Token Numbers
Comparison Across Generative Nodels
LARP Decoder
LARP Decoder
200
3FVD
230
Prior
Holsee
Talons
250
LLMs
150
12.67x
LARP Encoder
2.85x
Stage Tokenizer Training
Stage AR Generalive training
NAGVIT
WASVIT+2
LARPIOurd
(a) LARP Framewark
Figure 1: LARP highlights. (a) LARP is video tokenizer for two-stage video generative models. In the first stage, LARP tokenizer is trained with lightweight AR prior model to learn AR-friendly latent space. second stage, an AR generative model is trained on LARP's discrete tokens to synthesize high-fidelity videos. (b) The incorporation of the AR prior model significantly improves the generation FVD (gFVD) across various token number configurations. (c) LARP shows much smaller gap between its reconstruction FVD (rFVD) and generation FVD (gFVD), indicating the effectiveness of the optimized latent space it has learned.
ABSTRACT
We present LARP, a novel video tokenizer designed to overcome limitations in current video tokenization methods for autoregressive (AR) generative models. Unlike traditional patchwise tokenizers that directly encode local visual patches into discrete tokens, LARP introduces holistic tokenization scheme that gathers information from the visual content using set of learned holistic queries. This design allows LARP to capture more global and semantic representations, rather than being limited to local patch-level information. Furthermore, it offers flexi- bility by supporting an arbitrary number of discrete tokens, enabling adaptive and efficient tokenization based on the specific requirements of the task. To align the discrete token space with downstream AR generation tasks, LARP integrates a lightweight AR transformer token on its discrete latent space. By incorporating the prior model during train- ing, LARP learns latent space that not only optimized for video reconstruction but is also structured in a way that is more conducive to autoregressive generation. Moreover, this process defines sequential order for the discrete tokens, progres- sively pushing them toward an optimal configuration during training, ensuring smoother and more accurate AR generation inference time. Comprehensive experiments demonstrate LARP's strong performance, achieving state-of-the-art FVD on the UCF101 class-conditional video generation benchmark. LARP en- hances the compatibility of AR models with videos and opens up the potential to build unified high-
training-tinn prior model that predicts the next
high-fidelity fidelity multimodal large language models (MLLMs).
"Co-corresponding authors. Now rescarch scientist ByteDance.

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
The field of generative modeling has experienced significant advancements, largely driven by the success of autoregressive (AR) models in the development of large language models (LLMs) (Bai
jet a 2023 Brown 2020; Radford et al. 2019 Google et al. [023]] Touvron et al.) 2023a b). Build-
ing on AR transformers (Vaswani 2017). these models are considered pivotal for the future of Al due
to their exceptional performance (Hendrycks et al., 2020; (02T), impressive scalability (Henighan
ct al. 2020; Kaplan et al.) 2020 Rae et al. 2021), and versatile flexibility (Radford et al. 2019
Brown 2020).
Inspired by the success of LLMs, recent works have begun to employ AR transformers for visual
generation (Van Den Oord et al. 2017 Razavi et al. 2019 Esser et al. 2021 Hong et al. 2022 Ge
et al. 2022 Kondratyuk et al. [023 Wang et al. 2024). Additionally, several recent developments
have extended LLMs to handle multimodal inputs and outputs (Lu et al.) 2022
Zheng et al. 2024).
further demonstrating the promising potential of AR models in visual content generation. All of
these methods employ a visual tokenizer to convert continuous visual signals into sequences of discrete tokens, allowing them to be autoregressively modeled in the same way as natural language is modeled by LLMs. Typically, a visual tokenizer consists of a visual encoder, quantization module (Van Den Oord et al. 2017] Yu et al., 2023b), and a visual decoder. The generative modeling occurs in the quantized discrete latent space, with the decoder mapping the generated discrete token sequences back to continuous visual signals. It is evident that the visual tokenizer plays a pivotal role, as it directly influences the quality of the generated content. Building on this insight, several works have focused on improving the visual tokenizer (Lee et al. 2022 Yu et al., 2023b), making solid progress in enhancing the compression ratio and reconstruction fidelity of visual tokenization.
Most existing visual tokenizers follow a patchwise tokenization paradigm (Van Den Oord et al,, 2017 Esser et al. 2021: Wang et al. 2024 Yu et al. 2023b), where the discrete tokens are quan- tized from the encoded patches of the original visual inputs. While these approaches are intuitive for visual data with spatial or spatialtemporal structures, they restrict the tokenizers' ability to cap- ture global and holistic representations of the entire input. This limitation becomes even more AR models, which rely on sequential processing and require locally
pronounced when applied encoded tokens to be transformed into linear ID sequences. Previous research (Esser et al., 202T) has demonstrated that the method of flattening these patch tokens into a sequence is critical to the generation quality of AR models. Although most existing works adopt a raster scan order for this transformation due to its simplicity, it remains uncertain whether this is the optimal strategy. In addition, there are no clear guidelines for determining the most effective flattening order.
On the other hand, although the reconstruction fidelity of a visual tokenizer sets an upper bound on the generation fidelity of AR models, the factors that determine the gap between them remain unclear. In fact, higher reconstruction quality has been widely reported to sometimes lead to worse generation fidelity (Zhang et al. 2023 Yu et al., 2024). This discrepancy highlights the limitations of the commonly used reconstruction -wwpssrruuttooooouueeeeeeee focused design of visual tokenizers and underscores the im- portance of ensuring desirable properties in the latent space of the tokenizer. However, very few works have attempted to address this aspect in improving image tokenizers (Gu et al.) 2024 Zhang al. 2023), and for video tokenizers, it has been almost entirely overlooked.
In this paper, we present LARP, a video tokenizer with a Learned AutoRegressive generative Prior, designed address the underexplored challenges identified in previous work. By leverag- ing a ViT-style spatialtemporal patchifier (Dosovitskiy 2020) and a transformer encoder architec- ture (Vaswani 2017 LARP forms an autoencoder and employs stochastic vector quantizer (Van) Den Oord et al. 2017) to tokenize videos into holistic token sequences. Unlike traditional patch- wise tokenizers, which directly encode input patches into discrete tokens, LARP introduces a set of learned queries (Carion et al.] 2020 Li et al. 2023) that are concatenated with the input patch sequences and then encoded into holistic discrete tokens. An illustrative comparison between the patchwise tokenizer and LARP is shown in Figure 2 (a) and the left part of Figure 2 (b). By de- coupling the direct correspondence between discrete tokens and input patches, LARP allows for flexible number of discrete tokens, enabling trade-off between tokenization quality and latent representation length. This design also empowers LARP to produce more holistic and semantic representations of video content.
2

---

Published as a conference paper at ICLR 2025
To further align LARP's latent space with AR generative models, incorporate lightweight AR transformer as a prior model. autoregressively models LARP's latent space during training, providing signals to encourage learning a latent space that is well-suited for AR models. Importantly, the prior model is trained simultaneously with the main modules of LARP, but is discarded during inference, adding
memory or computational overhead to the tokenizer. Notably, by combining holistic tokenization with the co-training of the AR prior model, LARP automatically determines an order for latent discrete tokens in AR generation and optimizes the tokenizer perform optimally within that structure. This approach eliminates the need to manually define a flattening order, which remains an unsolved challenge for traditional tokenizers.
To evaluate the effectiveness of the LARP tokenizer, we train series of Llama-like (Touvron et al, 2023a by Sun et al. 2024) autoregressive (AR) generation models. Leveraging the holistic tokens and the learned AR generative prior, LARP achieves a Frechét Video Distance (FVD) (Unterthiner al. 2018) score of 57 on the UCF101 class-conditional video generation benchmark (Soomro) 2012). establishing a new state-of-the-ar among all published video generative models, including proprietary and closed-source approaches like MAGVIT-v2 (Yu al. 2023b). To summarize, our key contributions listed follows:
We present LARP, a novel video tokenizer that enables flexible, holistic tokenization, al- lowing for more semantic and global video representations.
LARP features learned AR generative prior, achieved by co-training AR prior model, which effectively aligns LARP's latent space with the downstream AR generation task.
LARP significantly improves video generation quality for AR models across varying ken sequence lengths, achieving state-of-the-art FVD performance on the UCF101 class- conditional video generation benchmark and outperforming all AR methods on the K600 frame prediction benchmark.
2 RELATED WORK
2.1 DISCRETE VISUAL TOKENIZATION
To enable AR models to generative high resolution visual contents, various discrete visual tok- enization methods have been developed. The seminal work VQ-VAE (Van Den Oord et al. 2017 Razavi et al.) 2019) introduces vector quantization to encode continuous images into discrete to- kens, allowing them be modeled by PixeICNN (Van den Oord al., (016). VQGAN (Esser et al. 2021) improves visual compression rate and perceptual reconstruction quality by incorporat- ing GAN loss (Goodfellow et al., 2014) training the autoencoder. Building on this, several works focus improving tokenizer efficiency (Cao al.) 2023) and enhancing generation quality (Gu ctt al. 2024; Zheng et al. 2022 Zhang et al. 2023). Leveraging the powerful ViT (Dosovitskiy) architecture, VIT-V VQGAN Yu et al. 2021) improves VQGAN on image generation tasks.
2020
Inspired by the success of image tokenization, researchers extend VQGAN to videos using 3D CNNs (Ge et al. 2022 Yan al. [021] Yu 2023a). C-ViViT VVillegas al. 2022) employs the temporal-cause VTT architecture to tokenize videos, while more recent work, MAGVIT-v2 YYu al. 2023b), introduces lookup-free quantization, significantly expanding the size of the quantiza- tion codebook. OmniTokenizer (Wang et al.] (204) unifies image and video tokenization using the same tokenizer model and weights for both tasks.
It is worth noting that all of the above tokenizers follow the patchwise tokenization paradigm dis- cussed in Section [ and are therefore constrained by patch-to-token correspondence. Very recently, a concurrent work Yu et al. 2024) proposes compact tokenization approach for images. However, it neither defines flattening order for the discrete tokens nor introduces any prior or regularization to improve downstream generation performance.
2.2 VISUAL GENERATION
Visual generation has been a research. The first major breakthrough comes with the rise of Generative Adversarial Networks (GANs) [Goodfellow et al. 2014: Karras et al.) 2019 2020, Skorokhodov et al., (022). known for their intuitive mechanism and fast inference capabilities. AR methods are also widely applied in
area of interest in machine learning and computer vision
3

---

Published as a conference paper at ICLR 2025
Input Video Patches
Video Patches
Holisicc Queries
Scheduled Sampling Mixing
Encoder
LARP Encoder
Quantizer
SVQ Quantizer Q
AR Prior Model
Patch Queries
Patch Tokens
Holisfic Tokens
Cosine Similarily & Softmax
Decoder
LARP Decoder
[L. مما AT
Multinamial Probabiliiies
Reconstructed Video Patchea
Reconstructed Video Patches
Eeeior
(a) Patchwise Tokenizer
LARP Tokenizer with the AR Prior Model
3 represent video patches, circles O indicate continuous embeddings, and squares O denote discrete tokens. (a) Patchwise video tokenizer used previous works. (b) Left: The LARP tokenizer tokenizes videos in holistic scheme, gathering information from the video using a set of learned queries. Right: The AR prior model, trained with LARP predicts the next holistic token, enabling latent space optimized for AR generation. The AR prior model is forwarded in two rounds per iteration. The red arrow represents the first round, and the purple arrows represent the second round. The reconstruction loss Lie is omitted for simplicity.
Figure 2: Method overview, Cubes
visual generation. Early works (Van Den Oord et al. 2016 Van den Oord et al. 2016 Chen et al. 2020) model pixel sequences autoregressively, but are limited in their ability to synthesize high- resolution content due to the extreme length of pixel sequences. Recent advancements in visual tokenization make AR generative models for visual content more practical. While all tokenizers discussed in Section 2. are suitable for AR generation, many focus BERT-style (Devlin 2018) masked visual generation (Chang et al. 2022). such as in Yu et al. (2023a by (024)) Diffusion mod- els (Ho al. 2020; Song et al. 20200; Peebles & Xie (023) have recently emerged to dominate image (Dhariwal & Nichol, 202 and video synthesis (Ho et al.) 2022). delivering impressive vi- sual generation quality. By utilizing VAEs (Kingma 2013) reduce resolution, latent diffusion
models (Rombach et al. 2022 Blattmann et al. 2023) further scale up, enabling multimodal visual
generation (Betker et al.
2023 Saharia et al. 2022 Podell et al. 2023 Brooks et 2024).
3 METHOD
3.1 PRELIMINARY
Patchwise Video Tokenization. As discussed in Section T existing video tokenizers adopt a patch- wise tokenization scheme, where latent tokens encoded from the spatialtemporal patches of the input video. Typically, a patchwise video tokenizer consists of an encoder E, a decoder D, and quantizer Q. Given video input V E x x3 is encoded, quantized, and reconstructed as:
Z = E(V),
= QZ))
V=
(1)
Tw xd refers to the spatialtemporally downsampled video feature maps with latent dimensions per location, X E x''' xw" denotes the quantized discrete tokens, and V the reconstructed video. fr,fH, are the downsampling factors for the spatialtemporal dimensions T, H, W. respectively.
where Z
E RT
Despite different implementations of the encoder E, decoder D, and quantizer Q. all patchwise tok- enizers maintain a fixed downsampling factor for each spatialtemporal dimension. The latent vector Zij.ke Rd at each position is typically the direct output of its spatialtemporally corresponding
4

---

Published as a conference paper at ICLR 2025
input video patch (e.g., same spatialtemporal location in CNNs, token position in transformers). While this design intuitive for 3D signals like video, limits the discrete tokens to low-level patch features, hindering their ability to capture higher-level, holistic information. Moreover, this formu- lation introduces the challenge of flattening patch tokens into a unidirectional sequence, which is critical for AR generation.
Autoregressive Modeling, Given a sequence of discrete tokens a = (,,, ....... Tn). we can train neural network to model the probability distribution pa( autoregressively as follows:
(x)
II pe (T. 21 I.,.,,,,,-1111))
where 0 denotes the neural network parameters. This model can be conveniently trained by opti- mizing the negative ongllikelihood (NI..) of po (xr) During inference, it iteratively predicts the next token 2y by sampling from po (x: " 1, f). based on the previously generated tokens.
While autoregressive modeling imposes direct constraints data modality, it does require the data to be both discrete and sequential, which necessitates the use of a visual tokenizer when applied images or videos.
3.2 HOLISTIC VIDEO TOKENIZATION
Patchify. LARP employs the transformer architecture ( Vaswani 2017 due to its exceptional perfor- mance and scalability. Following the ViT framework (Dosovitskiy 2020), we split the input video into spatialtemporal patches, and linearly encode each patch into continuous transformer patch em- beddings. Formally, given a video input E R x3 the video is linearly patchified as follows:
P P(V),
E =flatten(P),
where denotes the linear patchify operation, P P W. xd
is the spatialtemporal patches projected onto d dimensions, and E E Rnxd is the flattened d-dimentional patch embeddings. Here, fr, fH. fw are the downsampling factors for dimensions T, H. W, respectively, and m J * Jx x Tw nature, and therefore cannot be directly used to generate holistic discrete tokens.
#1
W is the total number of tokens. Importantly, the patch embeddings E remain local in
design a holistic video tokenizer, it is crucial to avoid directly encoding individual patches into discrete tokens. To achieve this, we adapt the philosophy of Carion ct al. (2020); [i et al. (2023) to learn a set of fixed input queries to capture the holistic information from the video, as illustrated in the left section of Figure (b). For simplicity, LARP employs transformer encoder architecture, as opposed to the transformer encoder-decoder structure used
Query-based Transformer.
in Carion et al. (2020). In-context conditioning is applied enable information mixing between different patch and query tokens.
Formally, we define n learnable holistic query embedding Qr. E Rnnd where each embedding is d-dimensional. These query embeddings are concatenated with the patch embeddings E along the token dimension. The resulting sequence, now of length mm) is then input the LARP encoder E and quantizer Q as follows:
Z = E(2 || E), x - =(Z
(4)
where || denotes the concatenation operation, Z is the latent embeddings, and (01 Fn) denotes the quantized discrete tokens. Note that only Z i.e., the latent embeddings correspond- ing the queries embeddings, are quantized and used. This ensures that each discrete token x;; has equal chance to represent any video patch, eliminating both soft and hard local patch constraints.
The LARP decoder is also implemented a transformer encoder neural network. During the decoding stage, LARP follows similar approach, utilizing m learnable patch query embeddings Qp [mad The decoding process is defined as:
2 = e (x), (x),
= reshape(D(QPp Z L:m..)
(5)
Herre and throughout this paper, "transformer encoder" refers to the specific parallel transformer encoder architecture defined in Dosovitskiy (220))
5

---

Published as a conference paper at ICLR 2025
where Q_1 denotes the de-quantization operation that maps discrete tokens z back to the continuous latent embeddings 2 Roxd These embeddings are concatenated with the patch query embed- dings Qr, and the combined sequence of length m + n is decoded into a sequence of continuous vectors. The first m vectors are reshaped to reconstruct the video V E RRT'XXXXX x3
Crucially, although the latent tokens are now both holistic and discrete, no specific flattening order is imposed due to the unordered nature of the holistic query set and the parallel processing property of the transformer encoder. As result, x not immediately suitable for AR modeling.
Stochastic Vector Quantization. While vector quantization (VQ) (Van Den Oord et al. 2017 has
been widely adopted in previous visual quantizers (Esser al. 2021 Ge et al. 2022). determinis-
tic nature limits the tokenizer's ' ability to explore inter-code correlations, resulting less semantically rich codes. To address these limitations, LARP employs a stochastic vector quantization (SVQ)
paradigm to implement the quantizer Q. Similar to VQ. SVQ maintains a codebook C' R which stores vectors, each of dimension d'. The optimization objective Lsvo includes a weighted sum of the commitment loss and the codebook loss, as defined in Van Den Oord et al. (2017). The key difference lies in the look-up operation. While VQ uses arg min operation to find the closest code by minimizing the distance between the input vector RR' and all codes in C. SVQ intro- duces stochasticity in this process. Specifically, SVQ computes the cosine similarities 8 between the input vector and all code vectors in C, interprets these similarities softmax normalization to obtain the probabilities p. One index x is then sampled from the resulting multinomial distribution P(z). Formally, the SVQ process x ((v) is defined as:
logits, and applies
V C.
p = (s),
(6)
| ||||
I~ P(x)
where denotes the indicator function.
To maintain the differentiability of SVQ, we apply the 2013). The de-quantization operation is performed via straight-through estimator (Bengio al.
straightforward index look-up, 0 --1
C.. similar to the standard VQ process.
) Ge et al. (2022); Yu et al. (023a), the reconstructive training loss of LARP, Cree. is composed of L, reconstruction loss, LPIPS perceptual
Reconstructive Training. Following Esser et al. (2021);
loss (Zhang al. 2018). GAN loss (Goodfellow et al., 2014). and SVQ loss Lsvo.
3.3 LEARNING AN AUTOREGRESSIVE GENERATIVE PRIOR
Continuous Autoregressive Transformer. To better align LARP's latent space with AR generative models, we introduce a lightweight AR transformer as prior model, which provides gradients to push the latent space toward structure optimized for AR generation. A key challenge in design- ing the prior model lies in its discrete nature. Simply applying an AR model to the discrete token sequence would prevent gradients from being back propppropeg the LARP encoder. Furthermore, unlike the stable discrete latent spaces of fully trained tokenizers, LARP's latent space is continu- ously evolving during training, which can destabilize AR modeling and reduce the quality of the signals it provides to the encoder. To address these issues, we modify a standard AR transformer into a continuous AR transformer by redefining its input and output layers, as depicted in the right section of Figure (b).
The input layer of standard AR transformer is typically an embedding look-up layer. In the prior model of LARP, this is replaced with linear projection that takes the de- de-quantized latents 2 as input, ensuring proper gradient flow during training. The output layer of a standard AR transformer predicts the logits of the next token. While this does not block gradient propagation, lacks aware- ness of the vector values in the codebook, making it unsuitable for the continuously evolving latent space during training. In contrast, the output layer of LARP's AR prior model makes predictions following the SVQ scheme described in Section 3.2 It predicts an estimate of the next token's embedding, e dd which has the same shape as codebook vectors C, Similar to SVQ, the pre- dicted embedding e is used to compute cosine similarities with all code vectors in C, as described in Equation (6) These similarities then noftmaxx-nrmalia and interpreted as probabilities,

---

Published as a conference paper at ICLR 2025
rFVD gFVID
IFVD
200
200
gFVD
150
150
FVD
100
50
50
LARPS
LARP-8 LARP Tokenizer Size
LARP-L
256
512
1024
Token Numbers
(a) Scaling LARP tokenizer sizes.
(b) Scaling Number of discrete tokens.
Figure 3: Scaling LARP tokenizer size and number of tokens.
which are used to compute the negative log-likelihood (NIL) loss with the input tokens as the ground truth. To predict the next token, sample is drawn from the resulting multinomial distribu- tion using Equation (7). This output layer design ensures that the AR prior model remains aware of the continuously evolving codebook, enabling it to make more accurate predictions and provide more precise signals to effectively train the LARP tokenizer.
Scheduled Sampling. Exposure bias (Ranzato et al. 2015) is a well-known challenge in AR mod- eling. During training. the model is fed the ground-truth data to predict the next token. However, during inference, the model mustrelyy on its own previous predictions, which may contain errors, cre- ating mismatch between training and inference conditions. While the AR prior model in LARP is only used during training, it encounters a similar issue: as the codebook evolves, the semantic mean- ing of discrete tokens shift, making the input sequence misaligned with the prior model's learned representations. To address this problem, we employ the scheduled sampling technique (Bengio et al. 2015) Mihaylova & Martins 2019) within the AR prior model of LARP. Specifically, after the first forward pass of the prior model, we randomly mix the predicted output sequence with the original input sequence the token level. This mixed sequence is then fed into the AR prior model for a second forward pass, The NLL loss is computed for both rounds of predictions and averaged, helping to reduce exposure bias and ensure more robust training.
Integration. Although the AR prior model functions as a standalone module, it is trained jointly with the LARP tokenizer in an end-to-end manner. Once the NLL loss Larior is computed, it is combined with the reconstructive loss free to optimize the parameters of both the prior model and the tokenizer. Formally, the total loss is defined as:
L = Lrce + aCprior
Since a is is typically set to a where is the loss weight, and Cree is defined in Section 3.2
small value, we apply a higher learning rate to the parameters of the prior model to ensure effective learning. Importantly, the prior model is used solely to encourage an AR-friendi discrete latent space for LARP during training. It is discarded at inference time, meaning it has no effect on the inference speed or memory footprint.
4 EXPERIMENTS
4.1
SETUP
Dataset. We conduct video reconstruction and generation experiments using the Kinetics-600 (Crroiaa et al.) 2018) and UCF-101 2012) datasets. In all experiments, we use 16-frame video clips with spatial resolution of 128x x 128 for both training and evaluation follow- ing Ge et al. (2022) Yu et al. (003a b).
Implementation Details. LARP first patchifies the input video. In all experiments, the patch sizes set to fr 4, fH = 8, and fw 8, respectively. As result, a 16x128x 128 video clip is split into 4x 16x16 = 1024 video patches, which are projected into 1024 continuous patch embeddings in the first layer of LARP. For the SVQ quantizer, we utilize factorized codebook with a size of
7

---

Published as a conference paper at ICLR 2025
Method
#Params
#Tokens
rFVD
gFVD
Tokenizer
Generator
K600
UCF
Diffusion-baser generative models with continuous video tokenizers
VidcoFusion (Luo et al. 2023
2B
173
HPDM (Skorokhodov et al. 2024
725M
66
MLM generative models with discrete video tokenizers
MAGVIT-MLM (Yu
2023a
158M
306M
1024
25
9.9
76
MAGVII--2-MLM Yu et al.
2023b
307M
1280
8.6
4.3
58
AR generative models with discrete video tokenizers
Cog Vidco (Hong ct al. 2022
9.4B
2065
109.2
626
TATS (Ge et al. 2022
32M
321M
1024
162
332
MAGVTI-AR Yu
2023a
158M
306M
1024
25
265
MAGVIT-v2-AIT Yu et al.
2023b
840M
1280
8.6
109
OmniTokenizer /Wang al.
2024
82.2M
650M
1280
42
32.9
191
LARP-L (Ours)
173M
343M
1024
24
6.2
107
LARP-L-Long (Ours)
173M
343M
1024
20
6.2
102
LARP-L-Long (Ours)
173M
632M
1024
20
5.1
57
Table 1: Comparison video generation results. Results are grouped by the type of genera- tive models. The scores for MAGVIT-AR and MAGVIT--2-AR are taken from the appendix of MAGVIT-v2 (Yu et al. 2023b). LARP-L-Long denotes the LARP-L trained for more epochs. Our best results obtained with larger AR generator.
8192 and a dimension d' 8, following the recommendations ecommendations of al. (2021) The softmax normalization in Equation (6) is applied with temperature of 0.03. The AR prior model in LARP is adapted from a small GPT-2 model (Radford et al.] 2019). consisting only 21.7M parameters. Scheduled sampling for the AR prior model employs linear warm-up for the mixing rate, starting from and reaching peak of 0.5 at 30% the total training steps. set AR prior loss weight 0.06 in main experiments, and use learning rate multiplier of 50.
We employ a Llama-likes Touvron et al. 2023a b): Sun et al. (2024) transformer as our AR generative model. One class token [cls] and one separator token [sep] used in the class-conditional generation task on UCF101 and frame prediction task on K600, respectively.
Frechét Video Distance (FVD) (Unterthiner et al., 2018) serves the main evaluation metric for both reconstruction and generation experiments.
4.2 SCALING
To explore the effect of scaling the LARP tokenizer, begin by varying its size while keeping the number of latent tokens fixed at 1024. As shown in Figure 3 (a), we compare the reconstruction FVD (rFVD) and generation FVD (gFVD) for three scaled versions of LARP LARP-L., LARP-B, and LARP-S, with parameter counts of 173.0M, 116.3M, and 39.8M, respectively. All results are reported the UCF-101 dataset. Interestingly, while rFVD consistently improves the tokenizer size increases, gFVD saturates when scaling from LARP-B to LARP-L, suggesting that gFVD can follow different trend from rFVD. Notably, shown in Figure (c), LARP has already achieved the smallest gap between rFVD and gFVD, further demonstrating the effectiveness of the optimized latent space has learned.
One of LARP's key features is its holistic video tokenization, which supports an arbitrary number of latent discrete tokens. Intuitively, using more tokens slows down the AR generation process but improves reconstruction quality. Conversely, using fewer tokens significantly speeds up the process but may lead to lower reconstruction quality due to the smaller information bottleneck. To evaluate this trade-off, from 1024 to 512 and 256. The corresponding rFVD and gFVD results on the UCF-101 dataset are reported in Figure 3 (b). It is expected that both rFVD and gFVD increase when fewer tokens used to represent video. However, the rate degradation in gFVD slows down when reducing from 512 to 256 tokens compared to rFVD, indicating improved generative representation efficiency.
use LARP-B and the default AR model, scaling down the number of latent tokens
8

---

Published as a conference paper at ICLR 2025
OnmiToheenzze izer
LARP
Figure 4: Video reconstruction comparison with Tokenzzer (Wang et al. 2024
4.3 VIDEO GENERATION COMPARISON
For video generation, we compare LARP with other state-of-the-an published video generative mod- els, including diffusion- based models, Masked Language Modeling (MLM) methods, and AR meth- ods. We use the UCF-10) 101 coas-cooniitioal generation benchmark and the K600 frame prediction benchmark, where the first 5 frames are provided to predict the next 11 frames in a 16-frame video clip. As shown Table M LARP outperforms all other video generators on the UCF-101 dataset, setting
state-of-the-art FVD of 57. Notably, within the family of AR generative models, LARP significantly surpasses all other AR methods by a large margin on both the UCF-101 and K600 datasets, including the closed-source MAGVIT-v2-AR (Yu al.) (023b). Moreover, the last two rows of Table demonstrate that using a larger AR generator can significantly improve LARP's generation quality, highlighting the scalability of LARP's representation.
4.4 LATENT SPACE ANALYSIS
We analyze LARP's latent space by examining the role of individual LARP tokens in video recon- struction. Our findings show that only certain subsets of tokens have significant impact on quality. The specific set of important tokens differs across videos, reflecting the content adaptive nature of LARP tokens as different subsets become active based on the video content. Additionally, spatial- temporal analysis reveals that LARP tokens function holistic video representations, influencing global structures rather than isolated patches, with their effects aligning with semantic features rather than appearing random. Further details and visualizations provided in Appendix B.3
4.5 VISUALIZATION
Video Reconstruction. In Figure we compare video reconstruction quality of LARP with Om- niTokenizer (Wang al. 2024). LARP consistently outperforms OmniTokenizer,, particularly in complex scenes and regions, further validating the rFVDD comparison results shown in Table [
Dass-Conditional Video Generation. We present class-conditional video generation results in Fig- ure 5 LARP constructs discrete latent space that better suited for AR generation, which enables the synthesis of high-fidelity videos, not only improving the quality of individual frames but also enhancing overall temporal consistency. Additional results provided in the appendix.
Configuration
PSNR
LPIPS|
rFVD.
gFVD
LARP-B
27.88
0.0855
31
107
No AR prior model
27.95
0.0830
23
190
No scheduled sampling in AR prior model
27.85
0.0856
27
142
Deterministic quantization
27.65
0.0884
27
149
Small AR prior model loss weight (a (1.3)
27.83
0.0866
120
No CFG
27.88
0.0855
121
Table 2: Ablation study. All configurations are modified from LARP-B model.

---

Published as a conference paper at ICLR 2025
Figure Class-conditional video generation results on the UCF-101 dataset using LARP.
Video Frame Prediction. Video frame prediction results are displayed in Figure The vertical yellow line marks the boundary between the conditioned frames and the predicted frames. We use frames as input to predict the following 11 frames, forming a 16-frame video clip. which is temporally downsampled to 8 frames for display. Additional results provided in the appendix.
Figure 6: Video frame prediction results on the K600 dataset using LARP.
4.6 ABLATION STUDY
To assess the impact of the different components proposed in Section we perform an ablation study, with results shown in Table 2 Clearly, the AR prior model contributes the most to the ex- ceptional performance of LARP. the AR prior model remains consistent across different token numbers. The scheduled sampling for the AR prior model and the use of SVQ are also critical, as both are closely tied to the AR prior model's effectiveness. The loss weight of the AR prior model and the use of CFG have relatively minor effects on the generative performance. Interestingly, the model without AR prior achieves the best reconstruction results but the worst generation results, highlighting the effectiveness of the AR prior model in enhancing LARP's discrete latent space for generative tasks.
further validated Figure [ (b), the improvement from using
5 CONCLUSION AND FUTURE WORK
In this paper, we introduce LARP, novel video tokenizer tailored specifically for autoregres- sive (AR) generative models. By introducing holistic tokenization scheme with learned queries, LARP captures more global and semantic video representations, offering greater flexibility in the number of discrete tokens. The integration of lightweight AR prior model during training opti- mizes the latent space for AR generation and defines an optimal token order, significantly improving performance in AR tasks. Extensive experiments generation, and video frame prediction demonstrate LARP's ability to achieve state-of. ooie--ffthhe f--th--an art FVD scores. The promising results of LARP not only highlight its efficacy in video generation tasks but also suggest its potential for broader applications, including the development of multimodal large language models (MLLMss) to handle video generation and understanding in unified framework.
video reconstruction, class-conditional video
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by NSF CAREER Award (#2238769) and an Amazon Research Award (Fall 2023) to AS. The authors acknowledge UMD's supercomputing resources made avail- able for conducting this research. The U.S. Govermment authorized reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessar- ily representing the official policies or endorsements, either expressed or implied, of NSF, Amazon, or the U.S. Government.
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradicnts through stochastic neurons for conditional computation. arXiv preprint arXiv: 338.3432,2 2013.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dall-e33 pdf, 2(3):8, 2023.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhor, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion mod- cls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. Technical Report, 2024. URL Com/ research/ / diddooo--- generation momeeraaio-oodelssseee as wor 1d simulators
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Shiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu, Xin Zhao, Deli Zhao, and Kaigi Huang. Efficient- vggan: Towards high-resolution image generation with efficient vision transformers. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 7368-7377, 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Joao Carreira, Eric Noland, Andras note about kinetics-600. arXiv preprint / 2018.
Chloe Hillier, and Andrew Zisserman. A short
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11321 2022.
Mark Chen, Ales Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference machine learning, pp. 1691 1703. PMLR, 2020.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192: 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 12873-12883, 2021.
songwei Ge. Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and sensisvne transformer. In European Conference on Computer Vision, pp. 118. Springer, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Gemini Team Google, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.
2023.
Yuchao Gu, Xintao Wang. Yixiao Ge, Ying Shan, and Mike Zheng Shou. Rethinking the objectives of vector-quantized tokenizers for image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7631-7640, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint XXX::2009.033000 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874 2021.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, al. Scaling laws for autoregressive generative modeling, arXiv preprint arXiv:2010.14:3 2020.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6833 66511, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Informaioon Processing Systems, 35:8633- 8646, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205. /5568, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wwXiv:2001.08361 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1401-4410 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10-081199 2020.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint 2014.
12

---

Published as a conference paper at ICLR 2025
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang. Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: large language model for zero-shot video generation. arXiv preprint 2rXXiv2312.14125, 2023.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-1153 11532, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh In- ternational Conference on Learning Representations, 2022.
Zhengxiong Luo. Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint arXiv:2303.0888 083200 2023.
Tsvetomila Mihaylova and André FT Martins. Scheduled sampling for transformers. arXiv preprint arXiv:1906.07651, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4195-4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint vrXiv:2307.00952. 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 2:21::211..11446, 2021.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint 15Xiv:/511.067 2015.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494, 2022.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3626-3636, 2022.
Ivan Skorokhodov, Willi Menapace, Aliaksandr Siarohin, and Sergey Tulyakov. Hierarchical patch diffusion models for nigh-resolution video generation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 7569-7579, 2024.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
13

---

Published as a conference paper at ICLR 2025
K Soomro. Ucf101: A dataset of 101 human actions classes from vidcos in the wild. arXiv preprint arXiv:1212.0402, 2012.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06523, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023a,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In Proceedings the IEEF/CVF conference on computer vision and pattern recognition, pp. 7921-7931, 2021.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: 1EI2:v1811.11111 2018.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pp. 1747-1756. PMLR, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Ruben Villegas, Mohammad Babacizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406.09 2024.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104. 10157, 2021.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang. Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint 1rXiv:2110.04627. 2021.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10459-10488 10469, 2023a.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023b.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang Chich Chen. An image is worth 32 tokens for reconstruction and generation. arXiv preprint 47X.v:2406.07550, 2024.
14

---

Published as a conference paper at ICLR 2025
Jiahui Zhang. Fangneng Zhan, Christian Theobalt, and Shijian Lu. Regularized vector quantization for tokenized image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18467-18476, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35:23412-23422 23425,, 2022.
Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, and Zongqing Lu. Unicode: Learning a unified codebook for multimodal large language models. arXiv preprint arXi::403.00077,, 2024.
A ADDITIONAL IMPLEMENTATION DETAILS
A.1 ADDITIONAL IMPLEMENTATION DETAILS OF THE LARP TOKENIZER.
During the training of LARP, GAN loss (Goodfellow et al. 2014) is employed to enhance re- construction quality. We use a ViT-based discriminator (Dosovitskiy 2020) with identical patchify settings to those of the LARP tokenizer. The discriminator updated once for every five updates of the LARP tokenizer and is trained with learning rate that is 30% the LARP tokenizer's learn- ing rate. To stabilize discriminator training, LeCam regularization (Tenng et al. 2021 is applied,
following the approach of Yu et al. (203a)) A GAN loss weight of 0.3 is used throughout the training.
Fixed sin-cos positional encoding (Vaswani 2017) is used in both the encoder and decoder of LARP. In the encoder, fixed 3D positional encoding is applied to each video patch, while in the decoder, fixed ID positional encoding is added to each holistic token. Notably, since the patch queries and holistic queries are position-wise learnable parameters, they do not require additional positional encodings.
In the SVQ module, set the total quantization loss weight 0.1. Additionally, follow Esser iet al. (202I) by using a commitment loss weight of 0.25 and codebook loss weight of 1.0. Both the L1 reconstruction loss and the LPIPS perceptual loss are assigned weight of 1.0.
In most experiments, we train the LARP tokenizer for 75 epochs on a combined dataset of UCF-101 and K600 with batch size of 64, totaling approximately 500k training steps. Random horizontal flipping is used as data augmentation technique. Specifically, LARP-L-Long in Table for 150 epochs with a batch size of 128.
i is trained
The Adam optimizer (Kingma 2014) is used with base learning rate of 1e 4, 31 = 0.9, and B2 = 0.95, following warm-up cosine learning rate schedule.
A.2 ADDITIONAL IMPLEMENTATION DETAILS OF THE AR GENERATIVE MODEL
We use Llama-like transformers as our AR generative models. Unlike the original implementation and Sun et al. (0224), we utilize absolute learned positional encodings. token dropout probability of 0.1 is applied during training, with both residual and feedforward dropout probabilities also set to 0.1. Additionally, when training the AR generative models, the SVQ module of the LARP tokenizer is set to be deterministic, ensuring a more accurate latent representation.
Our default AR generative model consists of 632M parameters, as specified in Table [ trained on the training split of the UCF-101 dataset for 1000 epochs with batch size of 32. The model used in the last row of Table [ which also has 632M parameters, is trained for 3000 epochs on UCF-101 with batch size of 64.
The Adam W optimizer (Loshchilov] 2017) is used with B, = 0.9. 83 = 0.95, a weight decay of 0.05, and base learning rate of Ge 4, following warm-up cosine learning rate schedule.
15

---

Published as a conference paper at ICLR 2025
1.6
LARP-B(256)
1.5
LARP-B(512)
LARP-B
1.4
LARP-L
Loss
1.3
LARP-L-Long
0
20
40
60
80
100
120
140
Epoch
Figure 7: Training loss of LARP under different configurations.
When generating vidcos, apply small Classifier-Free Guidance (CFG) scale of 1.25 (Ho & Salimans 2022). We do not use top-k or top-p sampling methods.
For the frame prediction task, to create the 5-frame conditioning holistic tokens, we first take the initial 5 frames from the ground truth video, then repeat the 5th frame 11 times and pad these repetitions with the initial 5 frames (using the same padding). This forms complete 16-frame conditioning video. This conditioning video is then input to the LARP tokenizer to generate the conditioning holistic tokens. At inference time, the AR model predicts the holistic tokens for the full video, conditioned on the holistic tokens from the padded conditioning video.
B ADDITIONAL EXPERIMENTS
B.1 TRAINING STABILITY
While training LARP involves multiple loss functions, the process remains stable across different configurations. In Figure 7 we show the training loss curves LARP under various training epochs and regimes. Here, (ARP-B(256) and LARP-B(512) denote LARP -B trained with 256 and 512 holistic tokens, respectively. LARP-L-Long is trained with a batch size of 128 for 150 epochs, while the other models are trained with batch size of 64 for 75 epochs.
As shown, all loss eurves decrease smoothly and converge the end of training. Note that the initial slope change in LARP-L-Long's curve is due to the lincar warm-up learning rate schedule applied during the first 8 epochs.
B.2 HOLISTIC REPRESENTATION
The holistic tokenization scheme of LARP enables global and semantic representations, effectively capturing the spatiotemporal redundancy in videos and representing them accurately within a com- pact latent space. validate LARP's capability represent videos globally and holistically, manually introduce information redundancy into videos and measure the improvement in recon- struction quality.
Specifically, we fix the total number of video frames to 16, consistent with our main experiments, and repeat shorter videos multiple times to form 16-frame videos. This increases information re- dundancy the number of unique frames decreases with higher repetition, implying improved reconstruction quality for tokenizers capable of exploiting global and holistic information. We also compare LARP against patchwise video tokenizer, OmniTokenizer Notably, although OmniTok- enizer operates in patchwise manner, its transformer architecture allows it to leverage some global information. For repetition levels, value of indicates repetition, where all frames unique,
16

---

Published as a conference paper at ICLR 2025
OmniTokenizer
32
LARP (Ours)
31
PSNR
Repetition Level
Figure 8: Reconstruction PSNR across different repetition levels.
while value of 16 represents full repetition, where single frame is repeated to create the entire video.
Figure 8 reports the reconstruction PSNR for various repetition levels. The results demonstrate the impact of increasing information redundancy on reconstruction quality for both Omninoken and LARP. As the repetition level increases, both tokenizers show improved PSNR, reflecting their ability to leverage the added redundancy. However, LARP consistently outperforms Omni Tokenizer across all repetition levels, with a significantly larger improvement PSNR as the redundancy increases. These results validate LARP's effectiveness in capturing spatial-temporal redundancy, enabling it to represent videos more accurately within compact latent space.
B.3 ANALYSIS AND VISUALIZATION OF LARP'S LATENT SPACE
To gain deeper insights into the structure and properties of LARP's latent space, we randomly select 10 videos examples (shown in Figure 9) and analyze the impact of individual LARP tokens the reconstruction quality these videos. Using the LARP-L-Long tokenizer, we first calculate the latent LARP tokens for all selected videos and measure their reconstruction PSNR values based the unmodified LARP tokens. Next, for each video, we systematically modify a specific LARP token by assigning
random value, then reconstruct the video and compute the PSNR again. The influence of the modified token on reconstruction quality is quantified by the difference between reconstructions. This
the PSNR values obtained from the unmodified and the wangll-tooen-moddifee process is repeated for all 1024 LARP tokens across the latent space and for all 10 videos. The resulting PSNR differences are visualized in the scatter plots Figure 10
Interestingly, we observe that for each video, the majority of LARP tokens have minimal impact on reconstruction quality, while only a small subset of tokens carry the critical information necessary to represent the video accurately. However, the specific set of important tokens varies significantly across different videos, emphasizing the content-adaptive nature of LARP tokens, as different sub- sets of tokens are activated to represent different content.
Beyond the video-level analysis of LARP tokens conducted above, we also examine their impact at the spatial-temporal level. For LARP tokens of different videos, we select specific token index and set the values of tokens at this index all videos to We then compare the spatial- temporal reconstruction differences before and after this modification and visualize these differ- ences as heatmaps, as shown in Figure II The brightness of the heatmaps have been enhanced for improved visibility.
is evident that this token't 's influence spans the entire video, unlike patchwise tokens, which primar- ily affect specific spatial-temporal video patches. This finding confirms that LARP tokens as
17

---

Published as a conference paper at ICLR 2025
holistic representations for videos. Another intriguing observation is that the LARP token's impact not random; rather, it is concentrated on specific objects and structures, reflecting the semantic properties of the LARP's latent space.
B.4 ANALYSIS LARP'S AUTOREGRESSIVE PRIOR
The AR prior learned by LARP plays a crucial role in its video generation performance, shown in Figure ! (b). In this section, we present additional analysis and visualizations to illustrate the properties the learned AR prior.
N-gram Histogram. Given the discrete latent space of LARP the AR prior is inherently abstract. To facilitate effective visualization, we analyze a simplified version of the AR prior, namely the n- gram prior, which represents the AR prior restricted to an n-token context window. Specifically, uni- formly conditioned all 101 classes, we sample one million token sequences from UCF101 videos using the LARP AR model and perform the same process with baseline model that excludes the AR prior. We then count the occurrence frequencies of all n-grams across the video sequences. Fi- nally, plot histograms of the unique n-grams for both LARP and the no- no-prior bascline to compare their distributions.
Due to computational constraints, perform this analysis for 2-grams and 3-grams. Figure 12 presents the 2-gram histograms for LARP and its no-prior baseline. [3 a zoomed-in version of Figure [2] showing only 2-grams that occur fewer than 30k times. Figure [4] displays the 3-gram histograms.
histograms, the x-axis represents the frequency, or the number of occurrences of particular n-gram, while the y-axis represents the number of unique n-grams that occur with the given frequency across all sequences.
It is evident that compared the histograms with the AR prior, those without the AR prior significantly more left-skewed. This indicates that the distribution of the latent space without the AR prior closer uniform distribution, where all n-grams have similar occurrence frequencies. In contrast, the histograms with the AR prior reveal that a small number of graam occur significantly more frequently than others, reflecting the autoregressive nature of the space they reside in.
Class Dominance Score. We define a class dominance score for 2- 2-grams to compare the latent space with and without the AR prior. Specifically, using the sampled UCF101 video token sequences, we count the occurrence frequencies of all 2-grams for each UCF101 class. For every 2-grams b; that occurs more than fixed threshold d globally, we identify the UCFI01 class сj it dominates, i.c., the class to which the majority of occurrences of b, belong. The class dominance score quantifies the dominance b, over its dominating class Cj.
Mathematically, the class dominance score S(b,) is defined as
(m(bi.
S(bb)
max (E(1....100) n(b;))
(mm())))
where n(b., 0;) denotes the number of occurrences of 6; in video token sequences belonging to class Cj, and n(b:) represents the total number of occurrences of b, across video token sequences.
After applying the threshold d, we obtain the set of class dominance scores:
(S(b)) (b:)) 2 d).
(10)
where n(bb,) ensures that only 2-grams with sufficient occurrences are included. This set is computed for both the latent space with and without the AR prior, then sorted and plotted in Figure 15 The x-axis denotes different 2-grams, and the y-axis represents the class dominance score.
worth noting that the area corresponding to the sequences without the AR prior is confined to small region in the bottom-left comner. This indicates that most 2-grams in the no-prior sequences are uniformly distributed across all UCF101 classes. In contrast, the area for sequences with the AR prior is significantly larger, reflecting that many frequent 2-grams in these sequences correlated with specific UCF101 classes. This significant disparity further highlights the impact of the AR prior.
18

---

Published as a conference paper at ICLR 2025
C ADDITIONAL VISUALIZATION RESULTS
C.I VIDEO RECONSTRUCTION COMPARISON
Additional video reconstruction results are provided in Figure T6 Across a variety of scenes and
regions, LARP consistently demonstrates superior reconstruction quality compared to OmniTok- enizer Wang al. (2024).
C.2 CLASS-CONDITIONAL VIDEO GENERATION ON UCF-101 DATASET
We provide additional class-conditional video generation results in Figure 17 These results fur- ther demonstrate LARP's ability to generate high-quality videos with both strong per- framfr fidelity and temporal consistency across various action classes in the UCF-101 dataset. The generated videos show diverse scene dynamics, capturing fine-grained details and natural motion, highlighting LARP's effectiveness in handling complex generative tasks within this challenging dataset. I
Generated video files (in MP4 format) are available in the supplementary materials.
C.3 VIDEO FRAME PREDICTION ON K600 DATASET
We present additional video frame prediction results in Figure 18 further demonstrating LARP's ca- pacity to accurately predict future frames in the K600 dataset. These results showcase LARP's abil- ity to handle a wide range of dynamic scenes, capturing temporal dependencies with natural motion and smooth transitions between predicted frames. The predictions highlight LARP's effectiveness in scenarios involving complex motion and scene diversity, underscoring its strong generalization capabilities in video frame prediction tasks.
The predicted frames and the ground truth videos (in MP4 format) are available in the supplementary materials.
D LIMITATIONS
While LARP significantly enhances video generation quality, it still has certain limitations. Like other transformer-based video tokenizers [Wang et al. (2024), LARP performs best with fixed- resolution videos due the constraints of positional encoding. Additionally, artifacts may appear in LARP-generated videos when the scenes are particularly complex. Fortunately, scaling the AR generative model is expected improve video quality and reduce these artifacts, as suggested by the scaling laws of AR models Henighan et al. (2020): Sun et al. (2024).
19

---

Published as a conference paper at ICLR 2025
Video
Video
Video
Video 4
Video
Video
Video
D5
201
Video
Video
Video
Figure Ground truth videos used for LARP token space analysis.
20

---

Published as a conference paper at ICLR 2025
Video
Video
0.8
0.8
PSNR difference
PSNR difference
0.4
0.4
0.2
0.2
0.0
200
400
600
800
1000
200
400
600
800
1000
Index
Index
0.8
Video 3
Video
PSNR difference
PSNR difference
0.4
0.4
0.2
0.2
200
400
600
800
1000
200
400
600
800
1000
Index
Index
Video
0.8
PSNR differecc
PSNR difference
0.4
0.4
0.2
0.2
200
400
600
800
1000
200
400
600
800
1000
Index
Index
Video 7
Video
PSNR difference
0.6
PSWR difference
0.4
0.4
0.0
200
400
600
800
1000
200
400
600
800
1000
Index
Index
Video 9
Video
PSNR difference
0.6
PSNRR difference
0.5
0.4
0.4
0.0
0.0
200
600
800
1000
200
400
600
800
1000
Index
Index
Figure 10: PSNR differences induced by modifying individual LARP tokens.
21

---

Published as a conference paper at ICLR 2025
Figure 11: Heatmap of reconstruction errors caused by modifying a single LARP token. The tokens with the same specific index are modified across all videos. In each group, the first row shows the ground truth video frames, while the second row displays the corresponding heatmaps.
22

---

Published as a conference paper at ICLR 2025
10a
Number unique 2-grams
10a
10a
100k 250k
300k
Frequency of 2-grams, with AR prior
Frequency of 2-grams, without AR prior
Figure 12: Histogram of 2-grams.
10a
Number unique 2-grams
107
10a
101
6k
12k 15k 18k 21k 24k 27k
30k
3k
12k 15k 18k 21k
24k 27k 30k
Frequency of 2-grams, with AR prior
Frequency of 2-grams, without AR prior
Figure 13: Histogram of 2-grams (frequency 30k).
10²
Number unique
10a
10a
0
3k
24k Frequency of 3-grams, with prior
6k
9k
12k
15k
18k
21k
27k
30k
3k 6k
9k 12k 15k 18k 21k 24k 27k 30k
Frequency 3-grams, without AR prior
Figure 14: Histogram 3-grams.
23

---

Published as a conference paper at ICLR 2025
With AR Prior
Without AR Prior
Class dominance score
0.8
0.4
0.2
500
1000
1500
2000
2500
2-gram
Figure 15: Class dominance score.
24

---

Published as a conference paper at ICLR 2025
GT
OmniTokkntze
LARP
GT
OmniTokentzer
LARP
GT
Onni OmniTTokenaa Tokenteer
LARP
GT
Onin Tokeebkeeeze nzer
LARP
Figure 16:
Additional video reconstruction comparison with Tokinzzeeizer
(Wang et al.
2024).
25

---

Published as a conference paper at ICLR 2025
www.dponhvee
Figure 17: Additional class-conditional generation results on UCF-101 dataset.
26

---

Published as a conference paper at ICLR 2025
HOS ANE
FOR MAIOR
NOS YOUR JUEE
Figure 18: Additional video frame prediction results on K600 dataset.
27


Reviews:
[{"review_id": "s3rCUlZwgy", "reviewer": "Reviewer_KrLv", "summary": "The paper introduces LARP, a novel method for tokenizing videos for autoregressive (AR) generative models. LARP employs a holistic approach to capture global video information more effectively and incorporates an AR prior model to enhance video generation quality. The method demonstrates strong performance in generating high-quality videos and holds potential for developing models capable of integrating both video and language.", "questions": "How does the training process scale with larger datasets or more complex video content?\nWhat are the hardware requirements for training and deploying LARP?\nAre there specific types of videos or scenarios where LARP may underperform, and if so, what are they?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a novel method for video tokenization that moves beyond traditional patchwise encoding, making it both interesting and innovative.\n\n2. The paper provides a comprehensive set of experiments, including video reconstruction, class-conditional video generation, and video frame prediction, effectively demonstrating LARP's capabilities across different tasks.\n\n3. LARP achieves state-of-the-art FVD scores on benchmarks like UCF101, indicating that it is a competitive approach in the field of video generation.", "weaknesses": "1. The overall computational cost and complexity of training LARP, especially with the AR prior model, may be a concern.\n2. It remains unclear how stable the training process is over longer periods or under different training regimes.\n3. The paper may not sufficiently discuss scenarios in which LARP underperforms or fails, which is essential for understanding the model’s limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQhxG20iOe", "reviewer": "Reviewer_qHTB", "summary": "The submission introduces LARP, a novel video tokenizer designed to optimize video tokenization for autoregressive (AR) generative models. The approach deviates from traditional patch-wise tokenizers by adopting a holistic strategy, which captures global and semantic representations of video content. This unique structure allows for adaptive tokenization, with a focus on enhancing AR compatibility through an integrated lightweight AR prior model. The experimental results show state-of-the-art performance in video generation benchmarks, particularly in UCF-101 and K600 datasets, establishing the proposed method as a competitive alternative for high-fidelity video synthesis.", "questions": "* What are the differences between the proposed holistic tokenization approach and [1] beyond the AR prior part? Could the authors clarify from the aspects of input/output format, model architecture, training objective, etc.?\n* Would the holistic tokens hurt the appearance preservation capabilities in editing tasks such as inpainting and outpainting?\n* In the K600 frame prediction benchmark, what is the input to the LARP encoder at inference time? Could the authors clarify how to handle 5-frame condition with the patch size described in Sec. 4.1?\n\n[1] Yu et al. An Image is Worth 32 Tokens for Reconstruction and Generation. arXiv 2406.07550", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method addresses an interesting and important topic with great potential for multimodal LLMs - how to tokenize videos into a sequence of tokens more suitable for LLM learning.\n* The introduced AR prior model is a simple yet effective method to produce tokens more friendly for autoregressive generation.\n* Experiment results demonstrate the effectiveness of the proposed method in reducing the gap between generation quality and reconstruction quality, highlighting better learnability of the produced tokens.\n* State-of-the-art performance is presented on the UCF benchmark, with K600 results outperforming AR baselines and ablation studies verifying key design choices.", "weaknesses": "* While results have been presented on class-conditional generation and frame prediction tasks, the benefit or penalty from using the proposed method on other tasks such as video editing and stylization remains unclear. \n* While not being applied to videos before, the holistic token approach appears similar to [1] on images. The differences other than the AR prior part need to be clarified.\n\n[1] Yu et al. An Image is Worth 32 Tokens for Reconstruction and Generation. arXiv 2406.07550", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WxTuPDjoA1", "reviewer": "Reviewer_3WZp", "summary": "This paper presented a novel video tokenization technique for autoregressive video generation. It leveraged several query embeddings to gather information from input videos, which enabled the tokenizer to capture more global representations and support arbitrary number of tokens. It further integrated a lightweight autoregressive transformer model as a prior during training of the tokenizer, which can result in a better latent space for downstream autoregressive video generation. This paper also presented experimental results on the effectiveness of the proposed tokenizer when applied to video generation tasks.", "questions": "- As claimed and validated in the paper, an autoregressive prior is critical to improve generation performance. In this paper, it is implemented by leveraging an additional autoregressive transformer model. As presented in line 251, this paper used a transformer encoder architecture as a video tokenizer. If this transformer encoder is replaced by an encoder-decoder structure, will it serve as an autoregressive prior?\n- Could the authors further explain why an autoregressive prior during tokenizer training can lead to better generation performance? An argument is that, lower autoregressive prior loss does not necessarily lead to better generation performance, since in an extreme case, if all the tokens are the same, then the autoregressive prior loss can be extremely low, but the codebook will collapse and the downstream generation task will definitely fail.\n- This paper introduced the holistic video tokenization technique implemented by query-based transformers. Without this holistic video tokenization technique, the autoregressive prior can also be applied to the “patch tokens” as in Figure 2 (a). Despite the flexibility of supporting an arbitrary number of discrete tokens, the claim of “global and semantic representations” seems not validated in the paper. Are there experimental results supporting this claim?\n- Table 1 shows the reconstruction comparison among different tokenizers. With 1280 tokens, the rFVD of MAGVIT-v2 is 8.6. With 1024 tokens, the rFVD of LARP-L-Long is 20. Since the tokenizer presented in the paper can support an arbitrary number of discrete tokens, what will the rFID be if 1280 tokens are used?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper presented an inspiring message that properly considering the downstream video generation tasks when training video tokenizers can result in much better generation performance.\n- Extensive experiments and ablations demonstrated the effectiveness of incorporating the autoregressive prior to tokenizer training.", "weaknesses": "- There are several typos in the paper:\n    - line 270: “back to the continues” -> “back to the continuous”\n    - line 459: “hilighting” → “highlighting”\n- More ablation studies can be conducted to analyze the effectiveness of holistic video tokenization and autoregressive generation further, as detailed in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dV196uJtui", "reviewer": "Reviewer_cs2m", "summary": "This paper proposes a video tokenizer for autoregressive (AR) generative models. The authors find that existing tokenizers use only patchwise tokenization without effective ordering, and face the design gap between reconstruction and AR generation. To address both limitations, they propose a new tokenizer with two key designs: (1) holistic tokenization that gathers information with learnable queries; (2) a lightweight AR transformer that optimizes the latent space for AR generation. Based on the proposed tokenizer, the authors trained a set of AR video generative models that achieve competitive performance on the Kinetics-600 and UCF-101 datasets.", "questions": "* My main question after reading the paper is: The video data is already nicely ordered by timestamps, why not use this ordering directly for tokenization? Does the learned tokenizer reflect this natural order in its latent space? If not, I think the video tokenizer might be suboptimal.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The paper is generally well written and easy to follow.\n* The authors provide an insightful analysis of the two limitations of AR tokenization, and the proposed tokenizer with AR prior effectively addresses these limitations.\n* Extensive quantitative and qualitative results are presented to support the effectiveness of the proposed method in video generation.", "weaknesses": "* The first contribution (holistic tokenization) is similar to TiTok [1] in that they both use learnable queries to tokenize the input data into a 1d sequence. This somewhat weakens the contribution of the paper. It would be better if the authors could detail their design differences in this part and justify the advantage of these differences through experiments.\n* For the second contribution (AR prior model), there is no analysis of its accuracy or generalizability. Otherwise, it seems to me difficult for a small 21.7M AR model to predict the next tokens in a highly complex video latent space. The previous work VideoPoet [2] and Video-LaVIT [3] all require a 7B+ LLM to achieve this.\n* Since the work focuses on generation performance, it would be better to introduce more metrics other than FVD, such as FID and IS. These metrics are also commonly used on UCF-101 and will significantly increase the soundness of the paper.\n* There is a lack of intuitive analysis of the learned AR prior and latent space in the paper. Adding additional visualizations of the token ordering (e.g., by heatmap) would help readers understand how it actually works in the overall procedure. Otherwise, readers can only infer its functionality by looking at the performance improvement.\n\n---\n\n[1] Yu, et al. An Image is Worth 32 Tokens for Reconstruction and Generation. NeurIPS 2024.\n\n[2] Kondratyuk, et al. VideoPoet: A Large Language Model for Zero-Shot Video Generation. ICML 2024\n\n[3] Jin, et al. Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization. ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)