LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior

## Abstract

Abstract We present LARP, a novel video tokenizer designed to overcome limitations in current video tokenization methods for autoregressive (AR) generative models.
Unlike traditional patchwise tokenizers that directly encode local visual patches into discrete tokens, LARP introduces a holistic tokenization scheme that gathers information from the visual content using a set of learned holistic queries.
This design allows LARP to capture more global and semantic representations, rather than being limited to local patch-level information.
Furthermore, it offers flexibility by supporting an arbitrary number of discrete tokens, enabling adaptive and efficient tokenization based on the specific requirements of the task.
To align the discrete token space with downstream AR generation tasks, LARP integrates a lightweight AR transformer as a training-time prior model that predicts the next token on its discrete latent space.
By incorporating the prior model during training, LARP learns a latent space that is not only optimized for video reconstruction but is also structured in a way that is more conducive to autoregressive generation.
Moreover, this process defines a sequential order for the discrete tokens, progressively pushing them toward an optimal configuration during training, ensuring smoother and more accurate AR generation at inference time.
Comprehensive experiments demonstrate LARP’s strong performance, achieving state-of-the-art FVD on the UCF101 class-conditional video generation benchmark. LARP enhances the compatibility of AR models with videos and opens up the potential to build unified high-fidelity multimodal large language models (MLLMs).

## 1 Introduction

The field of generative modeling has experienced significant advancements, largely driven by the success of autoregressive (AR) models in the development of large language models (LLMs) (Bai et al., ; Brown, ; Radford et al., ; Google et al., ; Touvron et al., ; ).
Building on AR transformers (Vaswani, ), these models are considered pivotal for the future of AI due to their exceptional performance (Hendrycks et al., ; ), impressive scalability (Henighan et al., ; Kaplan et al., ; Rae et al., ), and versatile flexibility (Radford et al., ; Brown, ).

Inspired by the success of LLMs, recent works have begun to employ AR transformers for visual generation (Van Den Oord et al., ; Razavi et al., ; Esser et al., ; Hong et al., ; Ge et al., ; Kondratyuk et al., ; Wang et al., ). Additionally, several recent developments have extended LLMs to handle multimodal inputs
and outputs (Lu et al., ; Zheng et al., ), further demonstrating the promising potential of AR models in visual content generation.
All of these methods employ a visual tokenizer to convert continuous visual signals into sequences of discrete tokens, allowing them to be autoregressively modeled in the same way as natural language is modeled by LLMs. Typically, a visual tokenizer consists of a visual encoder, a quantization module (Van Den Oord et al., ; Yu et al., ), and a visual decoder. The generative modeling occurs in the quantized discrete latent space, with the decoder mapping the generated discrete token sequences back to continuous visual signals. It is evident that the visual tokenizer plays a pivotal role, as it directly influences the quality of the generated content. Building on this insight, several works have focused on improving the visual tokenizer (Lee et al., ; Yu et al., ), making solid progress in enhancing the compression ratio and reconstruction fidelity of visual tokenization.

Most existing visual tokenizers follow a patchwise tokenization paradigm (Van Den Oord et al., ; Esser et al., ; Wang et al., ; Yu et al., ),
where the discrete tokens are quantized from the encoded patches of the original visual inputs.
While these approaches are intuitive for visual data with spatial or spatialtemporal structures, they restrict the tokenizers’ ability to capture global and holistic representations of the entire input.
This limitation becomes even more pronounced when applied to AR models, which rely on sequential processing and require locally encoded tokens to be transformed into linear 1D sequences.
Previous research (Esser et al., ) has demonstrated that the method of flattening these patch tokens into a sequence is critical to the generation quality of AR models. Although most existing works adopt a raster scan order for this transformation due to its simplicity, it remains uncertain whether this is the optimal strategy. In addition, there are no clear guidelines for determining the most effective flattening order.

On the other hand, although the reconstruction fidelity of a visual tokenizer sets an upper bound on the generation fidelity of AR models, the factors that determine the gap between them remain unclear. In fact, higher reconstruction quality has been widely reported to sometimes lead to worse generation fidelity (Zhang et al., ; Yu et al., ). This discrepancy highlights the limitations of the commonly used reconstruction-focused design of visual tokenizers and underscores the importance of ensuring desirable properties in the latent space of the tokenizer.
However, very few works have attempted to address this aspect in improving image tokenizers (Gu et al., ; Zhang et al., ), and for video tokenizers, it has been almost entirely overlooked.

In this paper, we present LARP, a video tokenizer with a Learned AutoRegressive generative Prior, designed to address the underexplored challenges identified in previous work.
By leveraging a ViT-style spatialtemporal patchifier (Dosovitskiy, ) and a transformer encoder architecture (Vaswani, ), LARP forms an autoencoder and employs a stochastic vector quantizer (Van Den Oord et al., ) to tokenize videos into holistic token sequences.
Unlike traditional patchwise tokenizers, which directly encode input patches into discrete tokens, LARP introduces a set of learned queries (Carion et al., ; Li et al., ) that are concatenated with the input patch sequences and then encoded into holistic discrete tokens.
An illustrative comparison between the patchwise tokenizer and LARP is shown in Figure 2 (a) and the left part of Figure 2 (b).
By decoupling the direct correspondence between discrete tokens and input patches, LARP allows for a flexible number of discrete tokens, enabling a trade-off between tokenization quality and latent representation length. This design also empowers LARP to produce more holistic and semantic representations of video content.

To further align LARP’s latent space with AR generative models, we incorporate a lightweight
AR
transformer as a prior model. It autoregressively models LARP’s latent space during training, providing signals to encourage learning a latent space that is well-suited for AR models. Importantly, the prior model is trained simultaneously with the main modules of LARP , but it is discarded during inference, adding zero memory or computational overhead to the tokenizer.
Notably, by combining holistic tokenization with the co-training of the AR prior model,
LARP automatically determines an order for latent discrete tokens in AR generation and optimizes the tokenizer to perform optimally within that structure.
This approach eliminates the need to manually define a flattening order, which remains an unsolved challenge for traditional tokenizers.

To evaluate the effectiveness of the LARP tokenizer, we train a series of Llama-like (Touvron et al., ; ; Sun et al., ) autoregressive (AR) generation models. Leveraging the holistic tokens and the learned AR generative prior, LARP achieves a Frechét Video Distance (FVD) (Unterthiner et al., ) score of 57 on the UCF101 class-conditional video generation benchmark (Soomro, ), establishing a new state-of-the-art among all published video generative models, including proprietary and closed-source approaches like MAGVIT-v2 (Yu et al., ). To summarize, our key contributions are listed as follows:

- •
We present LARP, a novel video tokenizer that enables flexible, holistic tokenization, allowing for more semantic and global video representations.
- •
LARP features a learned AR generative prior, achieved by co-training an AR prior model, which effectively aligns LARP’s latent space with the downstream AR generation task.
- •
LARP significantly improves video generation quality for AR models across varying token sequence lengths, achieving state-of-the-art FVD performance on the UCF101 class-conditional video generation benchmark and outperforming all AR methods on the K600 frame prediction benchmark.

## 2 Related Work

### 2.1 Discrete Visual Tokenization

To enable AR models to generative high resolution visual contents, various discrete visual tokenization methods have been developed.
The seminal work VQ-VAE (Van Den Oord et al., ; Razavi et al., ) introduces vector quantization to encode continuous images into discrete tokens, allowing them to be modeled by PixelCNN (Van den Oord et al., ).
VQGAN (Esser et al., ) improves visual compression rate and perceptual reconstruction quality by incorporating GAN loss (Goodfellow et al., ) in training the autoencoder. Building on this, several works focus on improving tokenizer efficiency (Cao et al., ) and enhancing generation quality (Gu et al., ; Zheng et al., ; Zhang et al., ).
Leveraging the powerful ViT (Dosovitskiy, ) architecture, ViT-VQGAN (Yu et al., ) improves VQGAN on image generationt tasks.

Inspired by the success of image tokenization, researchers extend VQGAN to videos using 3D CNNs (Ge et al., ; Yan et al., ; Yu et al., ). C-ViViT (Villegas et al., ) employs the temporal-causal ViT architecture to tokenize videos, while more recent work, MAGVIT-v2 (Yu et al., ), introduces lookup-free quantization, significantly expanding the size of the quantization codebook.
OmniTokenizer (Wang et al., ) unifies image and video tokenization using the same tokenizer model and weights for both tasks.

It is worth noting that all of the above tokenizers follow the patchwise tokenization paradigm discussed in Section 1, and are therefore constrained by patch-to-token correspondence.
Very recently, a concurrent work (Yu et al., ) proposes a compact tokenization approach for images. However, it neither defines a flattening order for the discrete tokens nor introduces any prior or regularization to improve downstream generation performance.

### 2.2 Visual Generation

Visual generation has been a long-standing area of interest in machine learning and computer vision research.
The first major breakthrough comes with the rise of Generative Adversarial Networks (GANs) (Goodfellow et al., ; Karras et al., ; ; Skorokhodov et al., ), known for their intuitive mechanism and fast inference capabilities.
AR methods are also widely applied in visual generation. Early works (Van Den Oord et al., ; Van den Oord et al., ; Chen et al., ) model pixel sequences autoregressively, but are limited in their ability to synthesize high-resolution content due to the extreme length of pixel sequences.
Recent advancements in visual tokenization make AR generative models for visual content more practical.
While all tokenizers discussed in Section 2.1 are suitable for AR generation, many focus on BERT-style (Devlin, ) masked visual generation (Chang et al., ), such as in Yu et al. (; ; ).
Diffusion models (Ho et al., ; Song et al., ; Peebles & Xie, ) have recently emerged to dominate image (Dhariwal & Nichol, ) and video synthesis (Ho et al., ), delivering impressive visual generation quality.
By utilizing VAEs (Kingma, ) to reduce resolution, latent diffusion models (Rombach et al., ; Blattmann et al., ) further scale up, enabling multimodal visual generation (Betker et al., ; Saharia et al., ; Podell et al., ; Brooks et al., ).

## 3 Method

Figure: Figure 2: Method overview. Cubes represent video patches, circles indicate continuous embeddings, and squares denote discrete tokens. (a) Patchwise video tokenizer used in previous works. (b) Left: The LARP tokenizer tokenizes videos in a holistic scheme, gathering information from the video using a set of learned queries. Right: The AR prior model, trained with LARP , predicts the next holistic token, enabling a latent space optimized for AR generation. The AR prior model is forwarded in two rounds per iteration. The red arrow represents the first round, and the purple arrows represent the second round. The reconstruction loss ${\mathcal{L}}_{\text{rec}}$ is omitted for simplicity.
Refer to caption: x2.png

### 3.1 Preliminary

Patchwise Video Tokenization.
As discussed in Section 1, existing video tokenizers adopt a patchwise tokenization scheme, where
latent tokens are encoded from the spatialtemporal patches of the input video.
Typically, a patchwise video tokenizer consists of an encoder ${\mathcal{E}}$, a decoder ${\mathcal{D}}$, and a quantizer ${\mathcal{Q}}$. Given a video input ${\bm{\mathsfit{V}}}\in{\mathbb{R}}^{T{\times}H{\times}W{\times}3}$, it is encoded, quantized, and reconstructed as:

$$ $\displaystyle{\bm{\mathsfit{Z}}}={\mathcal{E}}({\bm{\mathsfit{V}}}),\qquad{\bm {\mathsfit{X}}}={\mathcal{Q}}({\bm{\mathsfit{Z}}}),\qquad\hat{{\bm{\mathsfit{V }}}}={\mathcal{D}}({\bm{\mathsfit{X}}}),$ (1) $$

where ${\bm{\mathsfit{Z}}}\in{\mathbb{R}}^{\frac{T}{f_{T}}{\times}\frac{H}{f_{H}}{
\times}\frac{W}{f_{W}}{\times}d}$ refers to the spatialtemporally downsampled video feature maps with $d$ latent dimensions per location, ${\bm{\mathsfit{X}}}\in{\mathbb{N}}^{T^{\prime}{\times}H^{\prime}{\times}W^{
\prime}}$ denotes the quantized discrete tokens, and $\hat{{\bm{\mathsfit{V}}}}$ is the reconstructed video. $f_{T},f_{H},f_{W}$ are the downsampling factors for the spatialtemporal dimensions $T,H,W$, respectively.

Despite different implementations of the encoder ${\mathcal{E}}$, decoder ${\mathcal{D}}$, and quantizer ${\mathcal{Q}}$, all patchwise tokenizers maintain a fixed downsampling factor for each spatialtemporal dimension.
The latent vector ${\bm{\mathsfit{Z}}}_{i,j,k,:}\in{\mathbb{R}}^{d}$ at each position is typically the direct output of its spatialtemporally corresponding input video patch (e.g., same spatialtemporal location in CNNs, or token position in transformers).
While this design is intuitive for 3D signals like video, it limits the discrete tokens to low-level patch features, hindering their ability to capture higher-level, holistic information.
Moreover, this formulation introduces the challenge of flattening patch tokens into a unidirectional sequence, which is critical for AR generation.

Autoregressive Modeling.
Given a sequence of discrete tokens ${\bm{x}}=(x_{1},x_{2},\dots,x_{n})$, we can train a neural network to model the probability distribution $p_{\theta}({\bm{x}})$ autoregressively as follows:

$$ $\displaystyle p_{\theta}({\bm{x}})=\prod_{i=1}^{n}p_{\theta}\left(x_{i}\mid x_ {1},\dots,x_{i-1},\theta\right),$ (2) $$

where $\theta$ denotes the neural network parameters. This model can be conveniently trained by optimizing the negative log-likelihood (NLL) of $p_{\theta}({\bm{x}})$. During inference, it iteratively predicts the next token $x_{i}$
by sampling from $p_{\theta}\left(x_{i}\mid x_{1},\dots,x_{i-1},\theta\right)$, based on the previously generated tokens.

While autoregressive modeling imposes no direct constraints on data modality, it does require the data to be both discrete and sequential, which necessitates the use of a visual tokenizer when applied to images or videos.

### 3.2 Holistic Video Tokenization

Patchify. LARP employs the transformer architecture (Vaswani, ) due to its exceptional performance and scalability. Following the ViT framework (Dosovitskiy, ), we split the input video into spatialtemporal patches, and linearly encode each patch into continuous transformer patch embeddings. Formally, given a video input ${\bm{\mathsfit{V}}}\in{\mathbb{R}}^{T{\times}H{\times}W{\times}3}$, the video is linearly patchified as follows:

$$ $\displaystyle{\bm{\mathsfit{P}}}={\mathcal{P}}({\bm{\mathsfit{V}}}),\quad{\bm{ E}}=\operatorname{flatten}({\bm{\mathsfit{P}}}),$ (3) $$

where ${\mathcal{P}}$ denotes the linear patchify operation, ${\bm{\mathsfit{P}}}\in{\mathbb{R}}^{\frac{T}{f_{T}}{\times}\frac{H}{f_{H}}{
\times}\frac{W}{f_{W}}{\times}d}$ is the spatialtemporal patches projected onto $d$ dimensions, and ${\bm{E}}\in{\mathbb{R}}^{m{\times}d}$ is the flattened $d$-dimentional patch embeddings. Here, $f_{T},f_{H},f_{W}$ are the downsampling factors for dimensions $T,H,W$, respectively, and $m=\frac{T}{f_{T}}{\times}\frac{H}{f_{H}}{\times}\frac{W}{f_{W}}$ is the total number of tokens. Importantly, the patch embeddings ${\bm{E}}$ remain local in nature, and therefore cannot be directly used to generate holistic discrete tokens.

Query-based Transformer. To design a holistic video tokenizer, it is crucial to avoid directly encoding individual patches into discrete tokens.
To achieve this, we adapt the philosophy of Carion et al. (); Li et al. () to learn a set of fixed input queries to capture the holistic information from the video, as illustrated in the left section of Figure 2 (b).
For simplicity, LARP employs a transformer encoder (^1^11Here and throughout this paper, “transformer encoder” refers to the specific parallel transformer encoder architecture defined in Dosovitskiy ()) architecture, as opposed to the transformer encoder-decoder structure used in Carion et al. (). In-context conditioning is applied to enable information mixing between different patch and query tokens.

Formally, we define $n$ learnable holistic query embedding ${\bm{Q}}_{L}\in{\mathbb{R}}^{n{\times}d}$, where each embedding is $d$-dimensional.
These query embeddings are concatenated with the patch embeddings ${\bm{E}}$ along the token dimension.
The resulting sequence, now of length $(n+m)$, is then input to the LARP encoder ${\mathcal{E}}$ and quantizer ${\mathcal{Q}}$ as follows:

$$ $\displaystyle{\bm{Z}}={\mathcal{E}}({\bm{Q}}_{L}\parallel{\bm{E}}),\quad{\bm{x }}={\mathcal{Q}}({\bm{Z}}_{1:n,:}),$ (4) $$

where $\parallel$ denotes the concatenation operation, ${\bm{Z}}$ is the latent embeddings, and ${\bm{x}}=(x_{1},\dots,x_{n})$ denotes the quantized discrete tokens.
Note that only ${\bm{Z}}_{1:n,:}$, i.e., the latent embeddings corresponding to the queries embeddings, are quantized and used.
This ensures that each discrete token $x_{i}$ has equal chance to represent any video patch, eliminating both soft and hard local patch constraints.

The LARP decoder is also implemented as a transformer encoder neural network.
During the decoding stage, LARP follows a similar approach, utilizing $m$ learnable patch query embeddings ${\bm{Q}}_{P}\in{\mathbb{R}}^{m{\times}d}$. The decoding process is defined as:

$$ $\displaystyle\hat{{\bm{Z}}}={\mathcal{Q}}^{-1}({\bm{x}}),\quad\hat{V}= \operatorname{reshape}({\mathcal{D}}({\bm{Q}}_{P}\parallel\hat{{\bm{Z}}})_{1:m ,:}),$ (5) $$

where ${\mathcal{Q}}^{-1}$ denotes the de-quantization operation that maps discrete tokens ${\bm{x}}$ back to the continues latent embeddings $\hat{{\bm{Z}}}\in{\mathbb{R}}^{n{\times}d}$.
These embeddings are concatenated with the patch query embeddings ${\bm{Q}}_{P}$, and the combined sequence of length $m+n$ is decoded into a sequence of continuous vectors.
The first $m$ vectors are reshaped to reconstruct the video $\hat{V}\in{\mathbb{R}}^{T{\times}H{\times}W{\times}3}$.

Crucially, although the latent tokens ${\bm{x}}$ are now both holistic and discrete, no specific flattening order is imposed due to the unordered nature of the holistic query set and the parallel processing property of the transformer encoder. As a result, ${\bm{x}}$ is not immediately suitable for AR modeling.

Stochastic Vector Quantization.
While vector quantization (VQ) (Van Den Oord et al., ) has been widely adopted in previous visual quantizers (Esser et al., ; Ge et al., ), its deterministic nature limits the tokenizer’s ability to explore inter-code correlations, resulting
less semantically rich codes.
To address these limitations, LARP employs a stochastic vector quantization (SVQ) paradigm to implement the quantizer ${\mathcal{Q}}$.
Similar to VQ, SVQ maintains a codebook ${\bm{C}}\in{\mathbb{R}}^{c{\times}d^{\prime}}$, which stores $c$ vectors, each of dimension $d^{\prime}$. The optimization objective ${\mathcal{L}}_{\text{SVQ}}$ includes a weighted sum of the commitment loss and the codebook loss, as defined in Van Den Oord et al. ().
The key difference lies in the look-up operation. While VQ uses an $\operatorname*{arg\,min}$ operation to find the closest code by minimizing the distance between the input vector ${\bm{v}}\in{\mathbb{R}}^{d^{\prime}}$ and all codes in ${\bm{C}}$, SVQ introduces stochasticity in this process.
Specifically, SVQ computes the cosine similarities ${\bm{s}}$ between the input vector ${\bm{v}}$ and all code vectors in ${\bm{C}}$, interprets these similarities as logits, and applies a softmax normalization to obtain the probabilities ${\bm{p}}$. One index $x$ is then sampled from the resulting multinomial distribution $P(x)$. Formally, the SVQ process $x={\mathcal{Q}}({\bm{v}})$ is defined as:

$$ $\displaystyle{\bm{s}}=\frac{{\bm{v}}\cdot{\bm{C}}_{i}}{\|{\bm{v}}\|\|{\bm{C}}_ {i}\|},\quad{\bm{p}}=\mathrm{softmax}({\bm{s}})$ , (6) $$

$$ $\displaystyle x\sim P(x)=\prod_{j=1}^{n}{\bm{p}}_{i}^{\bm{1}_{x=j}},$ (7) $$

where $\bm{1}$ denotes the indicator function. To maintain the differentiability of SVQ, we apply the straight-through estimator (Bengio et al., ). The de-quantization operation is performed via a straightforward index look-up, $\hat{{\bm{v}}}={\mathcal{Q}}^{-1}(x)={\bm{C}}_{x}$, similar to the standard VQ process.

Reconstructive Training.
Following Esser et al. (); Ge et al. (); Yu et al. (), the reconstructive training loss of LARP, ${\mathcal{L}}_{\text{rec}}$, is composed of $L_{1}$ reconstruction loss, LPIPS perceptual loss (Zhang et al., ), GAN loss (Goodfellow et al., ), and SVQ loss ${\mathcal{L}}_{\text{SVQ}}$.

### 3.3 Learning an Autoregressive Generative Prior

Continuous Autoregressive Transformer. To better align LARP’s latent space with AR generative models, we introduce a lightweight AR transformer as a prior model, which provides gradients to push the latent space toward a structure optimized for AR generation.
A key challenge in designing the prior model lies in its discrete nature. Simply applying an AR model to the discrete token sequence would prevent gradients from being back-propagated to the LARP encoder.
Furthermore, unlike the stable discrete latent spaces of fully trained tokenizers, LARP’s latent space
is continuously evolving
during training,
which can destabilize AR modeling and reduce the quality of the signals it provides to the encoder.
To address these issues, we modify a standard AR transformer into a continuous AR transformer by redefining its input and output layers, as depicted in the right section of Figure 2 (b).

The input layer of a standard AR transformer is typically an embedding look-up layer. In the prior model of LARP, this is replaced with a linear projection that takes the de-quantized latents $\hat{{\bm{Z}}}$ as input, ensuring proper gradient flow during training.
The output layer of a standard AR transformer predicts the logits of the next token. While this does not block gradient propagation, it lacks awareness of the vector values in the codebook, making it unsuitable for the
continuously evolving
latent space during training.
In contrast, the output layer of LARP’s AR prior model makes predictions following the SVQ scheme described in Section 3.2.
It predicts an estimate of the next token’s embedding, $\bar{{\bm{v}}}\in{\mathbb{R}}^{d^{\prime}}$, which has the same shape as a codebook vectors ${\bm{C}}_{i}$.
Similar to SVQ, the predicted embedding $\bar{{\bm{v}}}$ is used to compute cosine similarities with all code vectors in ${\bm{C}}$, as described in Equation 6. These similarities are then $\mathrm{softmax}$-normalized and interpreted as probabilities, which are used to compute the negative log-likelihood (NLL) loss with the input tokens as the ground truth. To predict the next token, a sample is drawn from the resulting multinomial distribution using Equation 7.
This output layer design ensures that the AR prior model remains aware of the continuously evolving codebook, enabling it to make more accurate predictions and provide more precise signals to effectively train the LARP tokenizer.

Scheduled Sampling.
Exposure bias (Ranzato et al., ) is a well-known challenge in AR modeling. During training, the model is fed the ground-truth data to predict the next token. However, during inference, the model must rely on its own previous predictions, which may contain errors, creating a mismatch between training and inference conditions.
While the AR prior model in LARP is only used during training, it encounters a similar issue:
as the codebook evolves, the semantic meaning of discrete tokens can shift, making the input sequence misaligned with the prior model’s learned representations.
To address this problem, we employ the scheduled sampling technique (Bengio et al., ; Mihaylova & Martins, ) within the AR prior model of LARP. Specifically, after the first forward pass of the prior model, we randomly mix the predicted output sequence with the original input sequence at the token level.
This mixed sequence is then fed into the AR prior model for a second forward pass.
The NLL loss is computed for both rounds of predictions and averaged, helping to reduce exposure bias and ensure more robust training.

Integration. Although the AR prior model functions as a standalone module, it is trained jointly with the LARP tokenizer in an end-to-end manner.
Once the NLL loss ${\mathcal{L}}_{\text{prior}}$ is computed, it is combined with the reconstructive loss ${\mathcal{L}}_{\text{rec}}$ to optimize the parameters of both the prior model and the tokenizer. Formally, the total loss is defined as:

$$ $\displaystyle{\mathcal{L}}={\mathcal{L}}_{\text{rec}}+\alpha{\mathcal{L}}_{ \text{prior}},$ (8) $$

where $\alpha$ is the loss weight, and ${\mathcal{L}}_{\text{rec}}$ is defined in Section 3.2. Since $\alpha$ is is typically set to a small value, we apply a higher learning rate to the parameters of the prior model to ensure effective learning. Importantly, the prior model is used solely to encourage an AR-friendly discrete latent space for LARP during training. It is discarded at inference time, meaning it has no effect on the inference speed or memory footprint.

## 4 Experiments

Figure: (a) Scaling LARP tokenizer sizes.
Refer to caption: x3.png

### 4.1 Setup

Dataset.
We conduct video reconstruction and generation experiments using the Kinetics-600 (K600)(Carreira et al., ) and UCF-101(Soomro, ) datasets. In all experiments, we use 16-frame video clips with a spatial resolution of $128{\times}128$ for both training and evaluation following Ge et al. (); Yu et al. (; ).

Implementation Details.
LARP first patchifies the input video. In all experiments, the patch sizes are set to $f_{T}=4$, $f_{H}=8$, and $f_{W}=8$, respectively. As a result, a $16{\times}128{\times}128$ video clip is split into $4{\times}16{\times}16=1024$ video patches, which are projected into 1024 continuous patch embeddings in the first layer of LARP.
For the SVQ quantizer, we utilize a factorized codebook with a size of 8192 and a dimension of $d^{\prime}=8$, following the recommendations of Yu et al. ().
The softmax normalization in Equation 6 is applied with a temperature of 0.03. The AR prior model in LARP is adapted from a small GPT-2 model (Radford et al., ), consisting of only 21.7M parameters. Scheduled sampling for the AR prior model employs a linear warm-up for the mixing rate, starting from 0 and reaching a peak of 0.5 at $30\%$ of the total training steps. We set AR prior loss weight $\alpha=0.06$ in our main experiments, and use a learning rate multiplier of $50$.

We employ a Llama-like Touvron et al. (; ); Sun et al. () transformer as our AR generative model. One class token [cls] and one separator token [sep] are used in the class-conditional generation task on UCF101 and frame prediction task on K600, respectively.

Frechét Video Distance (FVD) (Unterthiner et al., ) serves as the main evaluation metric for both reconstruction and generation experiments.

**Table 1: Comparison of video generation results. Results are grouped by the type of generative models. The scores for MAGVIT-AR and MAGVIT-v2-AR are taken from the appendix of MAGVIT-v2 (Yu et al., ). LARP-L-Long denotes the LARP-L trained for more epochs. Our best results are obtained with a larger AR generator.**
| Method | #Params | #Tokens | rFVD$\downarrow$ | gFVD$\downarrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Tokenizer | Generator |  |  | K600 | UCF |
| Diffusion-based generative models with continuous video tokenizers |  |  |  |  |  |  |
| \hdashlineVideoFusion (Luo et al., ) | - | 2B | - | - | - | 173 |
| HPDM (Skorokhodov et al., ) | - | 725M | - | - | - | 66 |
| MLM generative models with discrete video tokenizers |  |  |  |  |  |  |
| \hdashlineMAGVIT-MLM (Yu et al., ) | 158M | 306M | 1024 | 25 | 9.9 | 76 |
| MAGVIT-v2-MLM (Yu et al., ) | - | 307M | 1280 | 8.6 | 4.3 | 58 |
| AR generative models with discrete video tokenizers |  |  |  |  |  |  |
| \hdashlineCogVideo (Hong et al., ) | - | 9.4B | 2065 | - | 109.2 | 626 |
| TATS (Ge et al., ) | 32M | 321M | 1024 | 162 | - | 332 |
| MAGVIT-AR (Yu et al., ) | 158M | 306M | 1024 | 25 | - | 265 |
| MAGVIT-v2-AR (Yu et al., ) | - | 840M | 1280 | 8.6 | - | 109 |
| OmniTokenizer (Wang et al., ) | 82.2M | 650M | 1280 | 42 | 32.9 | 191 |
| LARP-L (Ours) | 173M | 343M | 1024 | 24 | 6.2 | 107 |
| LARP-L-Long (Ours) | 173M | 343M | 1024 | 20 | 6.2 | 102 |
| LARP-L-Long (Ours) | 173M | 632M | 1024 | 20 | 5.1 | 57 |

### 4.2 Scaling

To explore the effect of scaling the LARP tokenizer, we begin by varying its size while keeping the number of latent tokens fixed at 1024. As shown in Figure 3 (a), we compare the reconstruction FVD (rFVD) and generation FVD (gFVD) for three scaled versions of LARP : LARP-L, LARP-B, and LARP-S, with parameter counts of 173.0M, 116.3M, and 39.8M, respectively. All results are reported on the UCF-101 dataset.
Interestingly, while rFVD consistently improves as the tokenizer size increases, gFVD saturates when scaling from LARP-B to LARP-L, suggesting that gFVD can follow a different trend from rFVD. Notably, as shown in Figure 1 (c), LARP has already achieved the smallest gap between rFVD and gFVD, further demonstrating the effectiveness of the optimized latent space it has learned.

One of LARP’s key features is its holistic video tokenization, which supports an arbitrary number of latent discrete tokens. Intuitively, using more tokens slows down the AR generation process but improves reconstruction quality. Conversely, using fewer tokens significantly speeds up the process but may lead to lower reconstruction quality due to the smaller information bottleneck.
To evaluate this trade-off, we use LARP-B and the default AR model, scaling down the number of latent tokens from 1024 to 512 and 256. The corresponding rFVD and gFVD results on the UCF-101 dataset are reported in Figure 3 (b).
It is expected that both rFVD and gFVD increase when fewer tokens are used to represent a video. However, the rate of degradation in gFVD slows down when reducing from 512 to 256 tokens compared to rFVD, indicating improved generative representation efficiency.

### 4.3 Video Generation Comparison

For video generation, we compare LARP with other state-of-the-art published video generative models, including diffusion-based models, Masked Language Modeling (MLM) methods, and AR methods.
We use the UCF-101 class-conditional generation benchmark and the K600 frame prediction benchmark, where the first 5 frames are provided to predict the next 11 frames in a 16-frame video clip.
As shown in Table 1, LARP outperforms all other video generators on the UCF-101 dataset, setting a new state-of-the-art FVD of 57. Notably, within the family of AR generative models, LARP significantly surpasses all other AR methods by a large margin on both the UCF-101 and K600 datasets, including the closed-source MAGVIT-v2-AR (Yu et al., ).
Moreover, the last two rows of Table 1 demonstrate that using a larger AR generator can significantly improve LARP’s generation quality, hilighting the scalability of LARP’s representation.

### 4.4 Visualization

Video Reconstruction. In Figure 4, we compare video reconstruction quality of LARP with OmniTokenizer (Wang et al., ). LARP consistently outperforms OmniTokenizer, particularly in complex scenes and regions, further validating the rFVD comparison results shown in Table 1.

Figure: Figure 4: Video reconstruction comparison with OmniTokenizer (Wang et al., ).
Refer to caption: extracted/5960086/figures/vis_recon.png

Class-Conditional Video Generation.
We present class-conditional video generation results in Figure 5. LARP constructs a discrete latent space that better suited for AR generation, which enables the synthesis of high-fidelity videos, not only improving the quality of individual frames but also enhancing overall temporal consistency. Additional results are provided in the appendix.

Figure: Figure 5: Class-conditional video generation results on the UCF-101 dataset using LARP.
Refer to caption: extracted/5960086/figures/vis_ucf_gen.png

Video Frame Prediction.
Video frame prediction results are displayed in Figure 6. The vertical yellow line marks the boundary between the conditioned frames and the predicted frames. We use 5 frames as input to predict the following 11 frames, forming a 16-frame video clip, which is temporally downsampled to 8 frames for display. It is evident that LARP effectively predicts frames with diverse scenes and natural motions. Additional results are provided in the appendix.

Figure: Figure 6: Video frame prediction results on the K600 dataset using LARP.
Refer to caption: extracted/5960086/figures/visualization_k600_fp.png

**Table 2: Ablation study. All configurations are modified from LARP-B model.**
| Configuration | PSNR$\uparrow$ | LPIPS$\downarrow$ | rFVD$\downarrow$ | gFVD$\downarrow$ |
| --- | --- | --- | --- | --- |
| LARP-B | 27.88 | 0.0855 | 31 | 107 |
| No AR prior model | 27.95 | 0.0830 | 23 | 190 |
| No scheduled sampling in AR prior model | 27.85 | 0.0856 | 27 | 142 |
| Deterministic quantization | 27.65 | 0.0884 | 27 | 149 |
| Small AR prior model loss weight ($\alpha=0.03$) | 27.83 | 0.0866 | 28 | 120 |
| No CFG | 27.88 | 0.0855 | 31 | 121 |

### 4.5 Ablation Study

To assess the impact of the different components proposed in Section 3, we perform an ablation study, with results shown in Table 2. Clearly, the AR prior model contributes the most to the exceptional performance of LARP. As further validated in Figure 1 (b), the improvement from using the AR prior model remains consistent across different token numbers.
The scheduled sampling for the AR prior model and the use of SVQ are also critical, as both are closely tied to the AR prior model’s effectiveness. The loss weight of the AR prior model and the use of CFG have relatively minor effects on the generative performance.
Interestingly, the model without the AR prior achieves the best reconstruction results but the worst generation results, highlighting the effectiveness of the AR prior model in enhancing LARP’s discrete latent space for generative tasks.

## 5 Conclusion and Future Work

In this paper, we introduce LARP, a novel video tokenizer tailored specifically for autoregressive (AR) generative models. By introducing a holistic tokenization scheme with learned queries, LARP captures more global and semantic video representations, offering greater flexibility in the number of discrete tokens. The integration of a lightweight AR prior model during training optimizes the latent space for AR generation and defines an optimal token order, significantly improving performance in AR tasks. Extensive experiments on video reconstruction, class-conditional video generation, and video frame prediction demonstrate LARP’s ability to achieve state-of-the-art FVD scores. The promising results of LARP not only highlight its efficacy in video generation tasks but also suggest its potential for broader applications, including the development of multimodal large language models (MLLMs) to handle video generation and understanding in a unified framework.

#### Acknowledgments

This work was partially supported by NSF CAREER Award (#2238769) and an Amazon Research Award (Fall 2023) to AS. The authors acknowledge UMD’s supercomputing resources made available for conducting this research. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of NSF, Amazon, or the U.S. Government.

## Appendix A Additional Implementation Details

### A.1 Additional Implementation Details of the LARP Tokenizer.

During the training of LARP, a GAN loss (Goodfellow et al., ) is employed to enhance reconstruction quality. We use a ViT-based discriminator (Dosovitskiy, ) with identical patchify settings to those of the LARP tokenizer. The discriminator is updated once for every five updates of the LARP tokenizer and is trained with a learning rate that is 30% of the LARP tokenizer’s learning rate. To stabilize discriminator training, LeCam regularization (Tseng et al., ) is applied, following the approach of Yu et al. (). A GAN loss weight of 0.3 is used throughout the training.

Fixed sin-cos positional encoding (Vaswani, ) is used in both the encoder and decoder of LARP. In the encoder, fixed 3D positional encoding is applied to each video patch, while in the decoder, fixed 1D positional encoding is added to each holistic token. Notably, since the patch queries and holistic queries are position-wise learnable parameters, they do not require additional positional encodings.

In the SVQ module, we set the total quantization loss weight to 0.1. Additionally, we follow Esser et al. () by using a commitment loss weight of 0.25 and a codebook loss weight of 1.0. Both the $L_{1}$ reconstruction loss and the LPIPS perceptual loss are assigned a weight of 1.0.

In most experiments, we train the LARP tokenizer for 75 epochs on a combined dataset of UCF-101 and K600 with a batch size of 64, totaling approximately 500k training steps. Random horizontal flipping is used as a data augmentation technique.
Specifically, LARP-L-Long in Table 1 is trained for 150 epochs with a batch size of 128.

The Adam optimizer (Kingma, ) is used with a base learning rate of $1e-4$, $\beta_{1}=0.9$, and $\beta_{2}=0.95$, following a warm-up cosine learning rate schedule.

### A.2 Additional implementation details of the AR generative model

We use Llama-like transformers as our AR generative models. Unlike the original implementation and Sun et al. (), we utilize absolute learned positional encodings. A token dropout probability of 0.1 is applied during training, with both residual and feedforward dropout probabilities also set to 0.1. Additionally, when training the AR generative models, the SVQ module of the LARP tokenizer is set to be deterministic, ensuring a more accurate latent representation.

Our default AR generative model consists of 632M parameters, as specified in Table 1. It is trained on the training split of the UCF-101 dataset for 1000 epochs with a batch size of 32. The model used in the last row of Table 1, which also has 632M parameters, is trained for 3000 epochs on UCF-101 with a batch size of 64.

The AdamW optimizer (Loshchilov, ) is used with $\beta_{1}=0.9$, $\beta_{2}=0.95$, a weight decay of 0.05, and a base learning rate of $6e-4$, following a warm-up cosine learning rate schedule.

When generating videos, we apply a small Classifier-Free Guidance (CFG) scale of 1.25 (Ho & Salimans, ). We do not use top-k or top-p sampling methods.

## Appendix B Additional Visualization Results

### B.1 Video Reconstruction Comparison

Additional video reconstruction results are provided in Figure 7. Across a variety of scenes and regions, LARP consistently demonstrates superior reconstruction quality compared to OmniTokenizer Wang et al. ().

### B.2 Class-conditional video generation on UCF-101 dataset

We provide additional class-conditional video generation results in Figure 8. These results further demonstrate LARP’s ability to generate high-quality videos with both strong per-frame fidelity and temporal consistency across various action classes in the UCF-101 dataset. The generated videos show diverse scene dynamics, capturing fine-grained details and natural motion, highlighting LARP’s effectiveness in handling complex generative tasks within this challenging dataset. I

Generated video files (in MP4 format) are available in the supplementary materials.

### B.3 Video Frame Prediction on K600 Dataset

We present additional video frame prediction results in Figure 9, further demonstrating LARP’s capacity to accurately predict future frames in the K600 dataset. These results showcase LARP’s ability to handle a wide range of dynamic scenes, capturing temporal dependencies with natural motion and smooth transitions between predicted frames. The predictions highlight LARP’s effectiveness in scenarios involving complex motion and scene diversity, underscoring its strong generalization capabilities in video frame prediction tasks.

The predicted frames and the ground truth videos (in MP4 format) are available in the supplementary materials.

Figure: Figure 7: Additional video reconstruction comparison with OmniTokenizer (Wang et al., ).
Refer to caption: extracted/5960086/figures/visualization_recon_supp.png

Figure: Figure 8: Additional class-conditional generation results on UCF-101 dataset.
Refer to caption: extracted/5960086/figures/visualization_ucf_gen_supp.png

Figure: Figure 9: Additional video frame prediction results on K600 dataset.
Refer to caption: extracted/5960086/figures/visualization_k600_fp_supp.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: How does the training process scale with larger datasets or more complex video content?
What are the hardware requirements for training and deploying LARP?
Are there specific types of videos or scenarios where LARP may underperform, and if so, what are they?
soundness: 2
strengths: 1. This paper introduces a novel method for video tokenization that moves beyond traditional patchwise encoding, making it both interesting and innovative.

2. The paper provides a comprehensive set of experiments, including video reconstruction, class-conditional video generation, and video frame prediction, effectively demonstrating LARP's capabilities across different tasks.

3. LARP achieves state-of-the-art FVD scores on benchmarks like UCF101, indicating that it is a competitive approach in the field of video generation.
summary: The paper introduces LARP, a novel method for tokenizing videos for autoregressive (AR) generative models. LARP employs a holistic approach to capture global video information more effectively and incorporates an AR prior model to enhance video generation quality. The method demonstrates strong performance in generating high-quality videos and holds potential for developing models capable of integrating both video and language.
weaknesses: 1. The overall computational cost and complexity of training LARP, especially with the AR prior model, may be a concern.
2. It remains unclear how stable the training process is over longer periods or under different training regimes.
3. The paper may not sufficiently discuss scenarios in which LARP underperforms or fails, which is essential for understanding the model’s limitations.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: * What are the differences between the proposed holistic tokenization approach and [1] beyond the AR prior part? Could the authors clarify from the aspects of input/output format, model architecture, training objective, etc.?
* Would the holistic tokens hurt the appearance preservation capabilities in editing tasks such as inpainting and outpainting?
* In the K600 frame prediction benchmark, what is the input to the LARP encoder at inference time? Could the authors clarify how to handle 5-frame condition with the patch size described in Sec. 4.1?

[1] Yu et al. An Image is Worth 32 Tokens for Reconstruction and Generation. arXiv 2406.07550
soundness: 3
strengths: * The proposed method addresses an interesting and important topic with great potential for multimodal LLMs - how to tokenize videos into a sequence of tokens more suitable for LLM learning.
* The introduced AR prior model is a simple yet effective method to produce tokens more friendly for autoregressive generation.
* Experiment results demonstrate the effectiveness of the proposed method in reducing the gap between generation quality and reconstruction quality, highlighting better learnability of the produced tokens.
* State-of-the-art performance is presented on the UCF benchmark, with K600 results outperforming AR baselines and ablation studies verifying key design choices.
summary: The submission introduces LARP, a novel video tokenizer designed to optimize video tokenization for autoregressive (AR) generative models. The approach deviates from traditional patch-wise tokenizers by adopting a holistic strategy, which captures global and semantic representations of video content. This unique structure allows for adaptive tokenization, with a focus on enhancing AR compatibility through an integrated lightweight AR prior model. The experimental results show state-of-the-art performance in video generation benchmarks, particularly in UCF-101 and K600 datasets, establishing the proposed method as a competitive alternative for high-fidelity video synthesis.
weaknesses: * While results have been presented on class-conditional generation and frame prediction tasks, the benefit or penalty from using the proposed method on other tasks such as video editing and stylization remains unclear. 
* While not being applied to videos before, the holistic token approach appears similar to [1] on images. The differences other than the AR prior part need to be clarified.

[1] Yu et al. An Image is Worth 32 Tokens for Reconstruction and Generation. arXiv 2406.07550

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - As claimed and validated in the paper, an autoregressive prior is critical to improve generation performance. In this paper, it is implemented by leveraging an additional autoregressive transformer model. As presented in line 251, this paper used a transformer encoder architecture as a video tokenizer. If this transformer encoder is replaced by an encoder-decoder structure, will it serve as an autoregressive prior?
- Could the authors further explain why an autoregressive prior during tokenizer training can lead to better generation performance? An argument is that, lower autoregressive prior loss does not necessarily lead to better generation performance, since in an extreme case, if all the tokens are the same, then the autoregressive prior loss can be extremely low, but the codebook will collapse and the downstream generation task will definitely fail.
- This paper introduced the holistic video tokenization technique implemented by query-based transformers. Without this holistic video tokenization technique, the autoregressive prior can also be applied to the “patch tokens” as in Figure 2 (a). Despite the flexibility of supporting an arbitrary number of discrete tokens, the claim of “global and semantic representations” seems not validated in the paper. Are there experimental results supporting this claim?
- Table 1 shows the reconstruction comparison among different tokenizers. With 1280 tokens, the rFVD of MAGVIT-v2 is 8.6. With 1024 tokens, the rFVD of LARP-L-Long is 20. Since the tokenizer presented in the paper can support an arbitrary number of discrete tokens, what will the rFID be if 1280 tokens are used?
soundness: 3
strengths: - This paper presented an inspiring message that properly considering the downstream video generation tasks when training video tokenizers can result in much better generation performance.
- Extensive experiments and ablations demonstrated the effectiveness of incorporating the autoregressive prior to tokenizer training.
summary: This paper presented a novel video tokenization technique for autoregressive video generation. It leveraged several query embeddings to gather information from input videos, which enabled the tokenizer to capture more global representations and support arbitrary number of tokens. It further integrated a lightweight autoregressive transformer model as a prior during training of the tokenizer, which can result in a better latent space for downstream autoregressive video generation. This paper also presented experimental results on the effectiveness of the proposed tokenizer when applied to video generation tasks.
weaknesses: - There are several typos in the paper:
    - line 270: “back to the continues” -> “back to the continuous”
    - line 459: “hilighting” → “highlighting”
- More ablation studies can be conducted to analyze the effectiveness of holistic video tokenization and autoregressive generation further, as detailed in the Questions section.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * My main question after reading the paper is: The video data is already nicely ordered by timestamps, why not use this ordering directly for tokenization? Does the learned tokenizer reflect this natural order in its latent space? If not, I think the video tokenizer might be suboptimal.
soundness: 2
strengths: * The paper is generally well written and easy to follow.
* The authors provide an insightful analysis of the two limitations of AR tokenization, and the proposed tokenizer with AR prior effectively addresses these limitations.
* Extensive quantitative and qualitative results are presented to support the effectiveness of the proposed method in video generation.
summary: This paper proposes a video tokenizer for autoregressive (AR) generative models. The authors find that existing tokenizers use only patchwise tokenization without effective ordering, and face the design gap between reconstruction and AR generation. To address both limitations, they propose a new tokenizer with two key designs: (1) holistic tokenization that gathers information with learnable queries; (2) a lightweight AR transformer that optimizes the latent space for AR generation. Based on the proposed tokenizer, the authors trained a set of AR video generative models that achieve competitive performance on the Kinetics-600 and UCF-101 datasets.
weaknesses: * The first contribution (holistic tokenization) is similar to TiTok [1] in that they both use learnable queries to tokenize the input data into a 1d sequence. This somewhat weakens the contribution of the paper. It would be better if the authors could detail their design differences in this part and justify the advantage of these differences through experiments.
* For the second contribution (AR prior model), there is no analysis of its accuracy or generalizability. Otherwise, it seems to me difficult for a small 21.7M AR model to predict the next tokens in a highly complex video latent space. The previous work VideoPoet [2] and Video-LaVIT [3] all require a 7B+ LLM to achieve this.
* Since the work focuses on generation performance, it would be better to introduce more metrics other than FVD, such as FID and IS. These metrics are also commonly used on UCF-101 and will significantly increase the soundness of the paper.
* There is a lack of intuitive analysis of the learned AR prior and latent space in the paper. Adding additional visualizations of the token ordering (e.g., by heatmap) would help readers understand how it actually works in the overall procedure. Otherwise, readers can only infer its functionality by looking at the performance improvement.

---

[1] Yu, et al. An Image is Worth 32 Tokens for Reconstruction and Generation. NeurIPS 2024.

[2] Kondratyuk, et al. VideoPoet: A Large Language Model for Zero-Shot Video Generation. ICML 2024

[3] Jin, et al. Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization. ICML 2024

Meta-review: This paper presents a method for learning tokenization that goes beyond patch-based representations and integrates a lightweight AR model, demonstrating improvements in AR generative modeling.

Strengths:

* The efficacy of the proposed method is well-validated.
* Comprehensive experiments achieve state-of-the-art results.
* The authors provide an insightful analysis of two key limitations of AR tokenization.

In the end, all reviewers are in favor of accepting this submission. The topic of the paper may attract interest in the field of visual generation using large language models.

Decision: Accept (Oral)