ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation

## Abstract

Abstract Text-to-video (T2V) models have recently undergone rapid and substantial advancements. Nevertheless, due to limitations in data and computational resources, achieving efficient generation of long videos with rich motion dynamics remains a significant challenge.
To generate high-quality, dynamic, and temporally consistent long videos, this paper presents ARLON, a novel framework that boosts diffusion Transformers with autoregressive ( AR ) models for long ( LON ) video generation, by integrating the coarse spatial and long-range temporal information provided by the AR model to guide the DiT model effectively.
Specifically, ARLON incorporates several key innovations:
1) A latent Vector Quantized Variational Autoencoder (VQ-VAE) compresses the input latent space of the DiT model into compact and highly quantized visual tokens, bridging the AR and DiT models and balancing the learning complexity and information density;
2) An adaptive norm-based semantic injection module integrates the coarse discrete visual units from the AR model into the DiT model, ensuring effective guidance during video generation;
3) To enhance the tolerance capability of noise introduced from the AR inference, the DiT model is trained with coarser visual latent tokens incorporated with an uncertainty sampling module.
Experimental results demonstrate that ARLON significantly outperforms the baseline OpenSora-V1.2 on eight out of eleven metrics selected from VBench, with notable improvements in dynamic degree and aesthetic quality, while delivering competitive results on the remaining three and simultaneously accelerating the generation process. In addition, ARLON achieves state-of-the-art performance in long video generation, outperforming other open-source models in this domain.
Detailed analyses of the improvements in inference efficiency are presented, alongside a practical application that demonstrates the generation of long videos using progressive text prompts.
See demos of ARLON at http://aka.ms/arlon .

## 1 Introduction

Text-to-video (T2V) models have recently undergone rapid advancements, driven by both Transformer architectures (Vaswani, ) and diffusion models (Ho et al., ). Autoregressive (AR) models, such as decoder-only Transformers, offer notable advantages in scalability and long-range in-context learning, demonstrating strong potential for video generation from text (Yan et al., ; Ge et al., ; Hong et al., ; Yu et al., ; Kondratyuk et al., ).
Meanwhile, diffusion-based models, including U-Net and Diffusion Transformers (DiT), have set a new benchmark in high-quality video generation, establishing themselves as dominant approaches in the field (Ho et al., ; ; Singer et al., ; Chen et al., ; Zhou et al., ; Wang et al., ; ; Blattmann et al., ; Guo et al., ; Zeng et al., ; Ma et al., ; Peebles & Xie, ; Zheng et al., ; Lab & etc., ; Yang et al., ; Ju et al., ).
However, despite the rapid progress in DiT models, several key challenges remain: 1) High training cost, especially for high-resolution videos, resulting in insufficient motion dynamics as training is restricted to short video segments within each batch; and 2) The inherent complexity and time-consuming nature of generating videos entirely through denoising based solely on text conditions; and 3) Difficulty in generating long videos with consistent motion and diverse content.

Previous research typically employed autoregressive approaches for long video generation with DiT models (Weng et al., ; Gao et al., ; Henschel et al., ), generating successive video segments conditioned on the last frames of the previous segment.
However, computational constraints restrict the length of these conditioned segments, resulting in limited historical context for the generation of each new segment.
Additionally, when given the same text prompts, generated short video segments often feature identical content, increasing the risk of repetition throughout the entire overall long video.
Diffusion models, while excellent at producing high-quality videos, struggle to capture long-range dependencies and tend to generate less dynamic motion. Additionally, the requirement for multiple denoising steps makes the process both computationally expensive and slow. By contrast, AR models are more effective at maintaining semantic information over long sequences, such as motion consistency and subject identity. They excel at generating continuous actions without the repetition issues that diffusion models often encounter in long video generation, and the inference of AR models typically achieves faster inference times.

Figure: Figure 1: Generation process for long videos with autoregressive transformer and DiT
Refer to caption: x1.png

Leveraging AR for coarse predictions, we can enhance the diffusion model’s capacity to capture richer dynamics and maintain continuity in long video sequences. The AR-generated features can also serve as an initialization to accelerate the diffusion process, resulting in faster and more efficient generation of long-form videos.
Building on this concept, we propose ARLON, a novel framework that effectively combines the advantages of autoregressive Transformer and DiT models for long video generation.
As shown in Fig. 1, ARLON first generates long-term, coarse-grained discrete visual units (AR codes) autoregressively using a decoder-only Transformer. These discrete AR codes are then segmented and sequentially fed into the DiT model by the proposed semantic injection module, which autoregressively generates high-quality video segments.
As shown in Figure 2, to bridge the feature spaces between the AR Transformer and DiT model, as well as to balance the learning complexity of the AR model and the information density of the visual tokens, we employ a 3D latent VQ-VAE to compact and quantize the input latent features of the DiT model into discrete tokens.
Various architectures of the semantic injection module, such as MLP adapter, adaptive norm modules, and ControlNet, are explored to ensure the coarse-grained AR codes guide the DiT model effectively.
However, unlike scenarios where conditioned images, videos, or motion trajectories (Chen et al., ; Zhang et al., ; Peng et al., ) are available, the tokens generated by autoregressive models in text-based video generation scenarios tend to be noisy, leading to a noticeable drop in accuracy when transitioning from teacher-forcing training to autoregressive inference.
To address this, we propose two key innovations: 1) training the DiT model using coarse visual latent tokens generated by a different latent VQ-VAE with higher compression rate than those used for AR model training, and 2) integrating an uncertainty sampling module into the semantic injection module to further enhance model performance.

Our ARLON model is evaluated using the VBench (Huang et al., ) video generation benchmark. Experimental results demonstrate that ARLON outperforms the baseline OpenSora-V1.2 on eight out of eleven metrics, while also delivering competitive performance across other metrics. Additionally, ARLON achieves state-of-the-art performance by effectively generating high-quality, temporally coherent, and dynamically rich long videos, surpassing other open-source models in this area.
Our contributions can be summarized in three points:

- •
We present ARLON, a novel framework that seamlessly combines the strengths of autoregressive Transformers and Diffusion Transformers (DiT). In this approach, the AR model supplies coarse spatial and long-range temporal information, effectively guiding the DiT model to generate long, high-quality videos with rich dynamic motion.
- •
To bridge the AR and DiT models while balancing learning complexity and information density, a latent VQ-VAE is introduced to compress the DiT model’s input space into compact, highly quantized visual tokens. These tokens are then used to train an autoregressive Transformer model, generating visual tokens based on the input text prompt. To reduce the noise inevitably introduced during AR inference, we introduce two noise-resilient strategies for the DiT model training: coarser visual latent tokens and uncertainty sampling.
- •
Both quantitative and qualitative analyses of the improvement in inference efficiency are given. In addition, long video generation using progressive text prompts is implemented, where each subsequent prompt builds on the previous.

Figure: Figure 2: Overview of ARLON Training
Refer to caption: x2.png

## 2 Method

As illustrated in Figure 2, our ARLON comprises two primary components: autoregressive latent modeling and semantic-aware condition generation. Given a text prompt, the autoregressive (AR) model predicts coarse visual latent tokens, which are constructed from a 3D VAE encoder followed by a latent VQ-VAE encoder based on the target video. These predicted visual latent tokens encapsulate both the coarse spatial information and consistent semantic information. Based on these tokens, a latent VQ-VAE decoder generates continuous latent features, which serve as semantic conditions to guide the DiT model with a semantic injection module. To mitigate the noise inevitably introduced during AR inference, we introduce two noise-robust training strategies: 1) coarser visual latent tokens, and 2) uncertainty sampling module.

### 2.1 Autoregressive

Latent VQ-VAE To align the feature spaces between the AR and DiT, and obtain quantized and compact discrete visual tokens, a latent VQ-VAE nested within the 3D VAE of the DiT model is constructed. Following Yan et al. (), the latent VQ-VAE encoder $\bm{E}_{latent}$ consists of 3D convolutional neural network (CNN) blocks and residual attention blocks, followed by a decoder $\bm{D}_{latent}$ structured as the reverse of the encoder. Let the inputs of the VQ-VAE be denoted as $\bm{X}\in\mathbb{R}^{T\times H\times W\times C}$. If the spatial and temporal compression factors of the 3D CNN encoder are $r$ and $o$ respectively, the encoder produces latent embeddings $\bm{V}\in\mathbb{R}^{\frac{T}{r}\times\frac{H}{o}\times\frac{W}{o}\times h}$. Each embedding vector $\bm{v}\in\mathbb{R}^{h}$ in $\bm{V}$ is quantized to the closest entry $\bm{c}\in\mathbb{R}^{m}$ in the learned codebook $\bm{C}\in\mathbb{R}^{K\times m}$.The index of each entry $\bm{c}$ is used to represent the latent embeddings $\bm{V}$ as $\bm{Q}=\{1,2,..,K\}^{\frac{T}{r}\times\frac{H}{o}\times\frac{W}{o}}$. For decoding, given the indices of video tokens, we can retrieve the corresponding entry $\bm{c}$, which is used to obtain reconstructed video embeddings $\bm{F}$ using the latent VQ-VAE decoder.

Autoregressive Modeling
We employ a causal Transformer decoder as the language model to autoregressively generate discrete visual tokens from textual input. Specifically, the indices of visual tokens $\bm{Q}$ are subsequently decomposed into 1D spacetime patches $\bm{Q}^{\text{AR}}=[\bm{q}_{1},\bm{q}_{2},...,\bm{q}_{N}]$, with <EOS> appended in the end of whole video, and <FRAME> inserted at the end of each frame, where ${N}=(\frac{T}{r}+1)\times\frac{H}{o}\times\frac{W}{o}$ and $q_{i}\in\{1,2,...,K\}$. These indices are converted into embeddings by the video code embedding layer, added with a learnable position embedding.
The text based condition $\bm{Y}$ is the contextual embedding of the video captions, generated by the T5 encoder (Raffel et al., ).
The AR model, comprising blocks of multi-head attention and feed-forward layers, takes the concatenation of text and visual
embeddings as input to model the dependency between these information, and the model is optimized to maximize the following probability

$$ $p(\bm{Q}^{AR}|\bm{Y};\bm{\Theta}_{AR})=\prod_{n=1}^{N}p(\bm{q}_{n}|\bm{Y},\bm{ Q}^{AR}_{<n}];\bm{\Theta}_{AR}).$ (1) $$

### 2.2 Motion Semantic Injection

STDiT Our ARLON framework is built on a spatial-temporal Transformer (Zheng et al., ), which serves as the backbone model. Given an input image latent $z$, a 3D embedding layer first projects the image into non-overlapping patches, which are then flattened. These flattened features are subsequently augmented with spatial and temporal position embeddings using ROPE (Su et al., ). The augmented features are then processed through a series of spatial-temporal DiT blocks, which conclude with an unpatchify layer predicting the noise. Each spatial-temporal DiT block comprises sequential modules for spatial-attention, temporal-attention, and cross-attention:

$$ $\displaystyle\mathbf{X}=\mathbf{X}+\operatorname{SpatialAttn}(\operatorname{LN }(\mathbf{X})),$ (2) $\displaystyle\mathbf{X}=\text{ rearrange }(\mathbf{X},({b}t){s}d\rightarrow({b }s){t}d),$ $\displaystyle\mathbf{X}=\mathbf{X}+\operatorname{TempAttn}(\operatorname{LN}( \mathbf{X})),$ $$

where $({b}t){s}d\rightarrow({b}s){t}d$ means rearranging the tensor by merging the batch size $b$ with the spatial dimension $s$ and isolating the temporal dimension $t$ for subsequent temporal attention processing. The text information is incorporated with DiT features using cross-attention, providing auxiliary information to enhance spatial-temporal consistency in video generation.

AR Semantic Condition Videos can be compressed into a coarse latent space using a video VAE and latent VQ-VAE. As the AR model predicts tokens within latent VQ-VAE space, we leverage the reconstructed latent features from the latent VQ-VAE decoder as semantic conditions for training the diffusion model. These conditional features are subsequently employed to determine the coarse spatial and temporal content in the video. Given the video $x$, the corresponding conditional features $\bm{F}$ can be extracted as:

$$ $\bm{F}=\bm{D}_{latent}(\bm{E}_{latent}(\bm{E}_{video}(x))).$ (3) $$

Following the STDiT model, we initially project the coarse latent feature $\bm{F}$ with a 3D embedding layer to generate the input condition $\bm{F}_{0}$, followed with several adapter layers to inject the semantic information into the video generation process. As shown in Figure2, the adapter block contains several residential MLP block:

$$ $\bm{F}_{i+1}=\text{Adapter}_{i}(\text{LayerNorm}(\bm{F}_{i}))+\bm{F}_{i}.$ (4) $$

Semantic Injection
To incorporate coarse semantic information into video generation, we inject the AR semantic condition into the DiT model to guide the diffusion process. Rather than directly adding the condition to each block, we utilize a gated adaptive normalization mechanism for condition injection. As shown in the upper part of Figure 3, the input latent variable $\bm{X}_{i}$ is first processed with a layer normalization, and the conditional latent variable $\bm{F}_{i}$ is processed with an uncertainty sampling (will be introduced in Section 2.3) to get $\hat{\bm{F}}_{i}$, which is projected into three parameters: scale $\bm{\gamma}_{i}$, shift $\bm{\beta}_{i}$, and gated parameter $\bm{\alpha}_{i}$, followed by the application of adaptive layer normalization to inject the conditioning information into the original latent variable. Additionally, to regulate the latent strength, we introduce an extra gated layer, initialized to zero, to adaptively control the injected features by adding them to the original DiT feature $\bm{X_{i}}$:

$$ $\displaystyle\bm{\alpha}_{i},\bm{\beta}_{i},\bm{\gamma}_{i}=$ $\displaystyle\text{MLP}\left(\hat{\bm{F}}_{i}\right),$ (5) $\displaystyle\text{Fusion}(\bm{X}_{i},\bm{\alpha}_{i},\bm{\beta}_{i},\bm{ \gamma}_{i})=$ $\displaystyle\bm{\alpha}_{i}\odot\text{MLP}\left(\bm{\gamma}_{i}\odot\text{ LayerNorm}(\bm{X_{i}})+\bm{\beta}_{i}\right)+\bm{X_{i}}.$ $$

Figure: Figure 3: Semantic injection and uncertainty sampling.
Refer to caption: x3.png

### 2.3 Training Strategy

In the training phase, each training sample consists of three inputs: the original video, a textual prompt $\bm{Y}$, and the AR semantic condition $\bm{F}$. For each video, we first convert it into the latent space $\bm{X}^{0}$. Subsequently, a timestep $t$ is randomly sampled from the interval $[0,T]$, and noise is added to the video latent $\bm{X}^{0}$, resulting in $X^{t}$. Our ARLON is then optimized using the following procedure:

$$ $\mathcal{L}=\mathbb{E}_{\bm{X}^{0},t,\bm{F},\epsilon\sim\mathcal{N}(0,1)}\left [\left\|\epsilon-\epsilon_{\theta}\left(\bm{X}^{t},t,\bm{Y},\bm{F}\right) \right\|_{2}^{2}\right].$ (6) $$

To enable overlapping long video generation and image-to-video generation, we randomly unmask frames, leaving them noise-free to serve as conditioning frames. To tolerate the errors inevitably introduced during AR inference, we implement two noise-resilient training strategies: coarser visual latent tokens and uncertainty sampling.

Coarser Visual Latent Tokens During the training phase, we employ two variants of latent VQ-VAE with distinct compression ratios to enhance the diffusion process to tolerate noisy AR prediction results. Specifically, for AR training, our latent VAE utilizes a compression ratio of $(r,o,o)$, whereas for the DiT model training, we adopt a compression rate of $(r,2o,2o)$. By using noisier and coarser information in the model training, DiT model is encouraged to capture more general patterns, thereby reducing the risk of specific errors inevitably introduced by AR prediction results.

Uncertainty Sampling
To simulate the autoregressive (AR) prediction variance, we introduce an uncertainty sampling module. As depicted in the lower part of Figure 3, this mechanism generates noise through the uncertainty sampling module rather than strictly relying on the original coarse latent features $\bm{F}_{i}$, and it is applied randomly after each adapter block during model training. Instead of injecting standard Gaussian noise, the noise is drawn from the original distribution of the latent features $\bm{F}_{i}$. The mean $\bm{\mu}_{i}$ and standard deviation $\bm{\sigma}_{i}$ of the noise are first calculated from the original coarse latent feature $\bm{F}_{i}$, and a normalization layer is applied to produce the whitened feature $\bar{\bm{F}}_{i}=\frac{\bm{F}_{i}-\bm{\mu}_{i}}{\bm{\sigma}_{i}}$. Following Chen et al. (), the sampled feature $\hat{\bm{F}}_{i}$ is calculated as:

$$ $\hat{\bm{F}}_{i}=\hat{\bm{\sigma}_{i}}\odot\bar{\bm{F}}_{i}+\hat{\bm{\mu}_{i}} ,\quad\hat{\bm{\sigma}_{i}}\sim N\left(\bm{1},\bm{\sigma}_{i}\right),\quad\hat {\bm{\mu}_{i}}\sim N\left(\bm{\mu}_{i},\bm{\sigma}_{i}\right),$ (7) $$

where $\hat{\bm{\sigma}_{i}}$ and $\hat{\bm{\mu}_{i}}$ represent noisy vectors sampled from the modeled mean and variance distribution of the target feature. To ensure that the sampled feature distribution closely approximates the original, the mean value of $\hat{\bm{\sigma}_{i}}$ distribution is set to 1.

## 3 Related Work

### 3.1 Text-to-video Generation

In recent years, substantial research has been dedicated to the development of text-to-video generation (T2I) models. These efforts can be broadly categorized into two main types: language-model-based and diffusion-model-based methods.
For diffusion-model-based approaches, pioneering works such as VDM (Ho et al., ) employ a 3D U-Net diffusion model for video generation. Imagen Video (Ho et al., ) and Make-a-Video (Singer et al., ) introduce spatiotemporally factorized models to generate high-definition videos. Subsequently, VideoCraft (Chen et al., ) and Magic Video (Zhou et al., ) utilize Video VAE and larger datasets to enhance the generalization capabilities of video models. Magic Video V2 (Wang et al., ) and Lavie (Wang et al., ) propose cascaded models for high-quality and aesthetically pleasing video generation. Moreover, SVD (Blattmann et al., ), Animatediff (Guo et al., ), and PixelDance (Zeng et al., ) employ T2I models to generate images and subsequently animate them into videos. Meanwhile, Latte (Ma et al., ) explores the training efficiency of video generation using a DiT model (Peebles & Xie, ), and SORA accelerates the investigation of DiT models. Recently, more DiT-based diffusion models have emerged, including OpenSora (Zheng et al., ), OpenSoraPlan (Lab & etc., ), CogvideoX (Yang et al., ), and Mira (Ju et al., ). While diffusion-based methods can generate high-quality videos, they are typically trained on fixed-length short videos (e.g., 16 frames), which limits their ability to produce longer videos. Furthermore, these methods predominantly focus on videos with small dynamic ranges.
Language-model-based methods leverage the transformer architecture to predict the next latent code of video representations in an autoregressive manner. VideoGPT (Yan et al., ) and TATS (Ge et al., ) utilize GPT-like transformer models to generate extended video sequences. CogVideo (Hong et al., ) employs a transformer to produce key frames, followed by a second upsampling stage to achieve higher frame rates. Recently, Magvit2 (Yu et al., ) introduced a novel lookup-free quantization approach, enhancing visual quality in language-based models. VideoPoet (Kondratyuk et al., ) incorporates a mixture of multimodal inputs into large language models (LLMs) for synthesizing video tokens. Although transformer-based models can effectively capture long-range dependencies, they demand substantial resources for training long videos, and their quality still requires improvement.

### 3.2 Long Video Generation

The generation of long videos presents significant challenges due to inherent temporal complexity and resource constraints. Previous autoregressive models (Ge et al., ) utilize sliding-window attention mechanisms to facilitate the generation of longer videos. However, despite these advantages, ensuring the quality of the generated videos remains problematic. Diffusion-based models (Ma et al., ; Zheng et al., ) typically employ conditional mask inputs for overlapping generation. Although these mask generation methods can produce long videos, they often encounter issues related to temporal inconsistency. Recent advancements, such as StreamingT2V (Henschel et al., ), have introduced the injection of key frames into diffusion processes to enhance temporal consistency across different video segments. Additionally, some training-free approaches (Qiu et al., ; Lu et al., ) leverage noise rescheduling techniques to improve temporal consistency. Moreover, VideoTetris (Tian et al., ) presents a compositional framework for video generation.
Our approach aims to integrate an autoregressive model to capture long-range dependencies while employing the diffusion-based DiT model to generate high-quality videos.

## 4 Experiments

### 4.1 Experimental Setup

#### Dataset.

For the training of the latent VAE and AR transformer model, about 5.7M video clips are used, consisting of Openvid-1M (Nan et al., ), ChronoMagic-ProH (Yuan et al., ) and OpenSora-plan (including Mixkit, Pexels and Pixabay) (Lab & etc., ).
For training the DiT model, we use 0.7M video clips from OpenVidHD-0.4M and Mixkit. To evaluate the performance of text-to-video generation, we utilize prompts from the Vbench benchmark (Huang et al., ) for comparison against other state-of-the-art models.
In the ablation studies, 100 prompts are randomly selected from OpenVid-1M, excluded from the training dataset.

**Table 1: Performance comparison of Text-to-video (T2V) generation betwen our ARLON and other open-source or commercial models on VBench benchmark. ${\dagger}$ stands for our reproduced results.**
| Models | Dynamic<br>Degree | Aesthetic<br>Quality | Imaging<br>Quality | Subject<br>Consist | Background<br>Consist | Motion<br>Smooth | Temporal<br>Flicker | Temporal<br>Style | Overall<br>Consist | Scene | Object |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Kling | 46.94 | 61.21 | 65.62 | 98.33 | 97.60 | 99.40 | 99.30 | 24.17 | 26.42 | 50.86 | 87.24 |
| Gen-2 | 18.89 | 66.96 | 67.42 | 97.61 | 97.61 | 99.58 | 99.56 | 24.12 | 26.17 | 48.91 | 90.92 |
| Pika-V1.0 | 47.50 | 62.04 | 61.87 | 96.94 | 97.36 | 99.50 | 99.74 | 24.22 | 25.94 | 49.83 | 88.72 |
| VideoCrafter-2.0 | 42.50 | 63.13 | 67.22 | 96.85 | 98.22 | 97.73 | 98.41 | 25.84 | 28.23 | 55.29 | 92.55 |
| LaVie | 49.72 | 54.94 | 61.90 | 91.41 | 97.47 | 96.38 | 98.30 | 25.93 | 26.41 | 52.69 | 91.82 |
| LaVie-Interpolation | 46.11 | 54.00 | 59.78 | 92.00 | 97.33 | 97.82 | 98.77 | 26.01 | 26.42 | 52.62 | 90.68 |
| Show-1 | 44.44 | 57.35 | 58.66 | 95.53 | 98.02 | 98.24 | 99.12 | 25.28 | 27.46 | 47.03 | 93.07 |
| CogVideo | 42.44 | 38.18 | 41.03 | 92.19 | 96.20 | 96.47 | 97.64 | 7.80 | 7.70 | 28.24 | 73.40 |
| OpenSoraPlan-V1.1 | 47.72 | 56.85 | 62.28 | 95.73 | 96.73 | 98.28 | 99.03 | 23.87 | 26.52 | 27.12 | 76.30 |
| OpenSora-V1.2 | 47.22 | 56.18 | 60.94 | 94.45 | 97.90 | 98.20 | 99.47 | 24.55 | 27.07 | 42.47 | 83.37 |
| ARLON (Ours) | 52.77 | 61.01 | 60.98 | 93.41 | 97.10 | 98.92 | 99.37 | 25.33 | 27.27 | 54.43 | 89.80 |

#### Evaluation Metrics.

To assess the text-to-video generation, we employ evaluation metrics consistent with those used in VBench and Vbench-Long: Dynamic Degree, Aesthetic Quality, Imaging Quality, Subject Consistency, Background Consistency, Motion Smoothness, Temporal Flickering, Temporal Style, Overall Consistency, Scene and Object Class. The first six metrics are used in the ablation studies.

#### Implementation Details.

The time-space compression ratio of the latent VAE is 4$\times$8$\times$8 and 4$\times$16$\times$16 for the training of the AR model and DiT model, and the dimension and vocabulary size of the codebook are 256 and 2048 respectively.
The AR model has the transformer structure with 12 layers, 16 attention heads, an embedding dimension of 1024, a feed-forward layer dimension of 4096, and a dropout of 0.1. DiT is initialized with OpenSora-V1.2, fixed during model training. The uncertainty sampling is employed randomly with a probability of 0.1. We use the Adam optimizer with a learning rate of $2\times 10^{-5}$ for fine-tuning. The model is trained at a resolution of $512\times 512$ and with a frame range from 51 to 136.

### 4.2 Results and Discussions

Figure: Figure 4: Comparison of qualitative results for text-to-video generation.
Refer to caption: x4.png

#### Text-to-video Generation

Our ARLON model is compared with other state-of-the-art open-source and commercial closed-source text-to-video generation models on VBench, including Kling(^1^11https://klingai.kuaishou.com/), Gen-2(^2^22https://runwayml.com/ai-tools/gen-2-text-to-video), Pika-V1.0(^3^33https://pika.art/home), VideoCrafter-2.0 (Chen et al., ), LaVie (Wang et al., ), LaVie-Interpolation (Wang et al., ), Show-1 (Zhang et al., ), CogVideo (Hong et al., ), OpenSoraPlan-V1.1 (Lab & etc., ) and OpenSora (Zheng et al., ).
As shown in Table 1, we conducted a quantitative comparison between ARLON and ten other text-to-video models, where ARLON demonstrates superior performance in terms of dynamic degree while maintaining strong results across other metrics. Compared to the baseline OpenSora-V1.2, ARLON outperforms eight out of eleven metrics, with particularly significant improvements in dynamic degree, aesthetic quality, scene and object metrics, while achieving competitive results in the remaining three.
Figure 4 shows the qualitative comparisons between ARLON and OpenSora. The scenes from OpenSora are mostly static, while ours exhibits significant camera movement and aligns better with the text.

Figure: Figure 5: Qualitative comparisons between StreamingT2V, FreeNoise, OpenSora, and ARLON. Each video contains 600 frames.
Refer to caption: x5.png

**Table 2: Long video generation (600 frames) results of ARLON and other models on VBench.**
| Models | Subject<br>Consist | Background<br>Consist | Motion<br>Smooth | Dynamic<br>Degree | Aesthetic<br>Quality | Imaging<br>Quality | Overall<br>Consist |
| --- | --- | --- | --- | --- | --- | --- | --- |
| FreeNoise | 96.59 | 97.48 | 98.36 | 17.44 | 47.39 | 63.88 | 25.78 |
| StreamingT2V | 87.31 | 94.64 | 93.83 | 85.64 | 44.57 | 53.64 | 23.65 |
| OpenSora-V1.2 | 96.30 | 97.39 | 98.94 | 44.79 | 56.68 | 51.64 | 26.36 |
| ARLON (Ours) | 97.11 | 97.56 | 98.50 | 50.42 | 56.85 | 53.85 | 26.55 |

Figure: Figure 6: Effects of incorporating layout information provided by AR codes into different layers.
Refer to caption: x6.png

**Table 3: Ablation study on 1) the semantic injection module, encompassing its architecture and the number of DiT layers inserting this module; and 2) the training strategies, using coarse visual latent tokens (a latent VAE with a higher compression ratio), introducing Gaussian random noise to the latent features $\bm{F}$ and performing uncertainty sampling with $\bm{F}$.**
| Compress<br>Ratio | Fusion<br>Architect | Number<br>of Layers | Gaussian<br>Noise | Uncertainty<br>Sampling | Subject<br>Consist | Background<br>Consist | Motion<br>Smooth | Dynamic<br>Degree | Aesthetic<br>Quality | Imaging<br>Quality |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline OpenSora-1.2 | 97.79 | 97.86 | 99.36 | 19.00 | 52.46 | 61.19 |  |  |  |  |
| 4$\times$8$\times$8 | Adaptive Norm | 14 | ✗ | ✗ | 94.71 | 96.22 | 99.07 | 42.00 | 53.91 | 58.50 |
| 4$\times$16$\times$16 | ControlNet | 14 | ✗ | ✗ | 95.81 | 96.58 | 99.10 | 46.00 | 55.18 | 62.89 |
| MLP Adapter | 14 | ✗ | ✗ | 97.99 | 97.97 | 99.31 | 21.00 | 56.40 | 64.74 |  |
| Adaptive Norm | 3 | ✗ | ✗ | 97.56 | 97.71 | 99.27 | 28.00 | 55.32 | 65.03 |  |
| Adaptive Norm | 8 | ✗ | ✗ | 97.29 | 97.55 | 99.21 | 31.00 | 55.55 | 65.09 |  |
| Adaptive Norm | 14 | ✗ | ✗ | 97.40 | 97.72 | 99.24 | 32.00 | 56.78 | 65.08 |  |
| Adaptive Norm | 14 | ✓ | ✗ | 97.04 | 97.35 | 99.20 | 34.00 | 54.98 | 64.21 |  |
| Adaptive Norm | 14 | ✗ | ✓ | 97.39 | 97.55 | 99.24 | 34.00 | 56.90 | 65.33 |  |

#### Long Video Generation

We compare our ARLON model with other open-source text-to-long video generation models: StreamingT2V (Henschel et al., ), FreeNoise (Qiu et al., ), and OpenSora-V1.2 (Zheng et al., ). As shown in Table 2, while FreeNoise achieves the highest image quality, its low motion metrics indicate that the generated videos are mostly static or contain minimal movement. In contrast, StreamingT2V exhibits high levels of dynamism, but this comes at the cost of consistency and overall video quality. As the length of the generated video increases, the temporal coherence between different segments diminishes.
Compared to the baseline model, OpenSora-V1.2, ARLON demonstrates significant improvements across almost all metrics.
While the increase in dynamism leads to a slight reduction in motion smoothness, we consider this trade-off acceptable for the more dynamic and engaging video content ARLON produces.
We also present several long video examples in Figure 5, while FreeNoise generates almost static videos, with minimal movement, such as the bear remaining stationary throughout. In contrast, our ARLON strikes a better balance, generating videos that not only exhibit dynamic motion but also maintain a high level of temporal consistency and natural flow.

Figure: Figure 7: Aesthetic quality and imaging quality as a function of denoising steps.
Refer to caption: x7.png

Figure: Figure 8: Generated videos of ARLON and OpenSora-V1.2 with 3 and 5 denoising steps.
Refer to caption: x9.png

### 4.3 Ablation Study

The highly compact and quantized tokens generated by autoregressive (AR) models can introduce noise and errors when transitioning from teacher-forcing training to autoregressive inference. Effectively handling those noise and errors, several key factors are explored in the DiT model training: 1) the design of the semantic injection module, including its architecture and the number of DiT layers where it is applied; and 2) the training strategies employed for the DiT model to enhance its robustness.
The results in Table 3 reveal several key points:
$\bullet$
Employing much coarser granularity during the DiT model training phase, specifically, utilizing the latent VAE with a compression ratio of $4\times 16\times 16$, compared to a ratio of $4\times 8\times 8$ during inference, can generate more inaccurate and noisier visual latent representations, which could simulate the errors introduced in the AR inference, thereby improving its robustness, and maintaining the consistency and qualities of the generated videos.
$\bullet$
While the MLP adapter-based semantic injection achieves higher consistency and ControlNet demonstrates more dynamic motion, both methods fall short in certain metrics. Although ControlNet excels in dynamic motion, outperforming other methods by a significant margin, it struggles with subject consistency, particularly as observed subjectively (as shown in Appendix, Figure 10). The adaptive norm method, however, strikes a more balanced performance across all criteria.
$\bullet$
As depicted in Figure 6, the first sub-figure presents a clip of the AR code-reconstructed video, while the second shows the baseline without AR codes. The third to sixth sub-figures correspond to videos with AR codes injected into the last 14, first 3, 8 and 14 layers of the DiT model (with 28 layers in total) respectively. Injecting AR codes into the last 14 layers provides insufficient layout information, resulting in a video similar to the baseline. In contrast, injecting codes into the first 3, 8, and 14 layers ensures that the layout information in the generated video aligns with AR codes, with greater control achieved as more layers are involved. Similar trends can be found in Table 3, injecting AR codes into the first 14 layers produces the best dynamic degree and aesthetic quality.
$\bullet$
To further improve the robustness of the DiT model, two approaches are applied: adding random noise to the latent feature $\bm{F}$ and employing uncertainty sampling as discussed in Section 2.3. Both methods improve the dynamic motion in generated videos, with uncertainty sampling further enhancing aesthetic and image quality.

Figure: Figure 9: Qualitative comparisons of the videos generated using progressive prompts.
Refer to caption: x10.png

### 4.4 Analysis

We conduct a detailed analysis of ARLON’s inference efficiency compared to OpenSora-V1.2.
The AR codes, which capture key spatial information, serve as an effective initialization, significantly accelerating the DiT model’s denoising process.
Figure 7 illustrates the curves representing aesthetic quality and imaging quality, as a function of the number of denoising steps.
Notably, ARLON achieves the similar performance in just 5 to 10 steps, compared to the 30 steps required by the baseline.
Taking the inference of a 68-frame video on a single A100 as an example, the OpenSora-V1.2 requires about 47 seconds for 30 steps of inference, while our DiT model equipped with the semantic injection module takes approximately 11 to 19 seconds for 5 to 10 steps, with additional approximately 6 seconds for the AR codes inferring. This results in a relative efficiency improvement of 47-64% over OpenSora-V1.2. Figure 8 provides a visual comparison between ARLON and the baseline OpenSora-V1.2, using 3 and 5 denoising steps. The results clearly show that ARLON generates videos with significantly higher quality.

### 4.5 Applications

One practical application of ARLON is long video generation using progressive text prompts. The procedure is as follows: assuming two text prompts, $\bm{X}_{1}$ and $\bm{X}_{2}$, are given, the corresponding AR codes $\bm{Q}_{1}$ are first generated based on $\bm{X}_{1}$. Then, the last frame of $\bm{Q}_{1}$ is used as a condition, along with $\bm{X}_{2}$, to generate the corresponding AR code $\bm{Q}_{2}$. Finally, the DiT model utilizes ($\bm{X}_{1}$, $\bm{Q}_{1}$) and ($\bm{X}_{2}$, $\bm{Q}_{2}$) to generate the video. A qualitative result is presented in Figure
9. Long videos generated by other models either remain unchanged after a prompt transition or change drastically. In contrast, our model transitions seamlessly, maintaining consistency throughout the entire video.

## 5 Conclusion

In this paper, we propose ARLON, a novel framework that boosts diffusion Transformers with autoregressive (AR) models for long (LON) video generation.
The AR model provides coarse spatial and long-range temporal information, guiding the DiT model to generate long high-quality videos with rich dynamics.
Utilizing a latent VQ-VAE, the input latent of the DiT model is compacted and highly quantized into discrete tokens for the training and inference of the AR model.
To integrate coarse spatial and temporal features into the DiT model, an adaptive norm based semantic injection module is proposed.
To improve the robustness of ARLON, the DiT model is trained with coarser visual latent tokens incorporated with an uncertainty sampling module.
Massive experiments demonstrate that our ARLON model delivers state-of-the-art performance on text-based long video generation. Detailed analysis of the inference efficiency and qualitative results of progressive prompt based long video generation are also provided.

## Appendix A Appendix

### A.1 Preliminaries

In this section, we introduce the preliminaries of a stable diffusion model, which operates the diffusion process in a latent space for computationally efficient. In this model, the image $x$ is mapped into a compressed latent space $z=\mathbb{E}(x)$ via a pre-trained auto-encoder, such as VQGAN or VQVAE. In the forward process, random noise is gradually added to the latent space, formulated as:

$$ $q\left(\bm{z}_{t}\mid\bm{z}_{t-1}\right)=\mathcal{N}\left(\bm{z}_{t};\sqrt{ \alpha_{t}}\bm{z}_{t-1},\left(1-\alpha_{t}\right)\mathbf{I}\right),$ (8) $$

where $t\in\{1,\ldots,T\}$, $T$ is the number of time steps during the diffusion process. and $q\left(\bm{z}_{t}\mid\bm{z}_{t-1}\right)$ is the noised $z_{t}$ at $t$ step given $z_{t-1}$, and $\left(1-\alpha_{t}\right)$ denotes the noise strength. Alternatively, we can formulated $z_{t}$ from $z_{0}$ as follows:

$$ $z_{t}=\sqrt{\bar{\alpha}_{t}}z_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon_{t}\text{ , }\epsilon_{t}\sim\mathcal{N}(0,\mathbf{I}),$ (9) $$

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$. The denoising process involves learning a reverse diffusion process to iteratively remove the noise added during the forward process. This is achieved by training a neural network to predict the original latent representation from the noisy version by minimizing:

$$ $l_{\epsilon}=\left\|\epsilon-\epsilon_{\theta}\left(z_{t},t,c\right)\right\|_{ 2}^{2},$ (10) $$

where $c$ denotes the conditional textual description, for the inference, random noise is sampled from Gaussian distribution, and DDIM is utilized for denoising a latent representation, followed by a VAE decoder to reconstruct the image.

### A.2 Extra Examples

Figure: Figure 10: Ablation study on different semantic injection approaches. Although ControlNet has higher dynamism, it also produces more distortions, such as the severe deformation of the dog in the left video and the facial distortions in the right video.
Refer to caption: x11.png

Figure: Figure 11: More generated videos of ARLON and OpenSora-V1.2 with 3 and 5 denoising steps, as a complement to Figure 8.
Refer to caption: x12.png

Figure: Figure 12: Examples of text to videos generation of ARLON and OpenSora-V1.2.
Refer to caption: x13.png

Figure: Figure 13: More text to videos generation of ARLON and OpenSora-V1.2.
Refer to caption: x14.png

Figure: Figure 14: Examples of text to long video generation (600 frames) using FreeNoise, StreamingT2V, OpenSora-V1.2, and ARLON.
Refer to caption: x15.png

Figure: Figure 15: Examples of Video Condition Generation: We extract the coarse latent from reference videos and replace it with our semantic latent, which is then injected into the diffusion process. The generated video maintains the same pose and position as the reference video while ensuring consistency with the provided textural prompts.
Refer to caption: x16.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the weakness part.
soundness: 2
strengths: 1. The integration of Autoregressive models with Diffusion Transformers and innovations like VQ-VAE for token compression, adaptive semantic injection, and uncertainty sampling show originality in addressing long video generation challenges.
2. The generated video spans 600 frames, making it relatively long.
summary: The manuscript introduces ARLON, a text-to-video framework that efficiently generates high-quality, dynamic, and temporally consistent long videos. By combining Autoregressive models with Diffusion Transformers, ARLON employs innovations like VQ-VAE for token compression, an adaptive semantic injection module, and an uncertainty sampling module to enhance efficiency and noise tolerance. It reduces denoising steps and outperforms OpenSora-V1.2 in both quality and speed, achieving state-of-the-art performance.
weaknesses: 1. In Table 2, why does StreamingT2V have a higher Dynamic Degree score (85.64) compared to ARLON (50.42)?
2. The paper lacks a detailed comparison of the number of parameters in ARLON versus baseline models.
3. There is no analysis of ARLON's memory footprint during training and inference, which would clarify its computational efficiency relative to models like OpenSora-V1.2.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - How many seconds can the models generate for the longest videos and how is the performance? For How many seconds do the longest videos that your method generates can last? In my understanding, this is the key advantage of the hierarchical generation framework.
- The main limitation of this work seems to be the huge computational cost of training, but the related information (type and number of GPU, training time) is not provided. It would be nice to know this information.
soundness: 3
strengths: - The motivation for using the AR model to provide semantic guidance is clear and nice.
- It is a very good extension of existing architectures.
- Good presentation.
- Good qualitative and quantitive results.
summary: This paper proposes a long video synthesis pipeline, ARLON. The main idea of this paper is to combine DiT with autoregressive transformers that provide long-range temporal information. To bridge the DiT and the AR transformer, the pipeline novelly adopts 1) a Latent VQ-VAE to enable the AR model to learn on and the DiT to learn on different latent spaces, reducing learning complexity and allowing the AR model to manage coarse temporal dependencies; 2) an adaptive norm-based semantic injection module to guide the DiT using AR generated tokens. Novel training strategies of coarser visual latent tokens for DiT and uncertainty sampling are also proposed to make the training process more stable and generalizable. 

Results are compared with current t2v models over VBench and Vbench-long and achieve notable improvements especially on long video generation.
weaknesses: I overall like this paper, but there are several points for improvement.

- No ablation on the impact of model structure and training data size.
- No discussion on failure cases and limitations.
- There might be some missing references such as nuwa-XL and Phenaki. GAN-based long video generation might also be related.

*[1] NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation*

*[2] Phenaki: Variable Length Video Generation from Open Domain Textual Descriptions*

I am not an expert in this field of training large video generation models. I will adjust the final score with other reviewers' comments and also based on the response from the author.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - What is the motivation of using a latent VQ-VAE nested inside a pretrained 3D Autoencoder, instead of training a single-stage pixel-to-latent tokenizer? And what is the additional computational cost or gain in comparison?

- The proposed coarse latent token with different compression ratio, while how is this ablated in Table 3 or any other ablation studies? (Does the 4×8×8 row refers to both the same scale training and the 4×16×16 rows refer to different scale training?)
soundness: 2
strengths: - The idea of make DiT and AR models working in one latent space is novel, and many technical improvements are designed to bridge their gap.

- The proposed method reaches a large reduction of denoising step of comparable generation quality.
summary: This paper proposes to leverage autoregressive models to guide the training of diffusion transformers for text-to-video generation task. The proposed framework incorporate with a latent VQ-VAE, coarser visual latent tokens and a uncertainty sampling module to connect DiT and AR models and inject the information from AR models to DiT training. Massive experiments are conducted with abundant quantitative metrics and visualizations are reported to demonstrate the performance of the proposed model.
weaknesses: - It's better to use bold fonts and underscores in Table 1. According to the listed numbers, the proposed method doesn't reach top 3 in many columns. And the reproduced notation is missing.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Figure1, the previously generated video seems to be used as a reference for the subsequent generation, but this is not illustrated in Figure 2. Does the DiT generate videos in an autoregressive way?
2. In Table 1 and Table 2, do the higher scores of metrics indicate better performance?
3. For long video generation in section 4.2 as well as the demo videos on the webpage, the authors only compare their proposed method with open-source text-to-long video generation models. Why not compare with the commercial closed-source text-to-video generation models like in Table 1?
soundness: 3
strengths: 1.This paper innovatively combines the strengths of autoregressive (AR) Transformers and Diffusion Transformers (DiT) for generating long video with rich dynamic motion.
2. To mitigate the effect of error introduced from AR inference to DiT, the authors introduce two noise-resilient strategies during DiT training.
3. The paper is written and presented clearly and easy to follow.
4. The long video results of the proposed method show improvement on dynamic degree, and long video generation results using progressive text prompts are more consistent throughout the entire video.
summary: This paper proposes a new text-to-video(T2V) framework consisting of autoregressive (AR) Transformers and Diffusion Transformers (DiT). Based on the input text prompt, the AR model predicts quantized visual tokens of a latent VQ-VAE nested within the 3D VAE of the DiT model. The coarse latent, reconstructed from the predicted tokens, serves as semantic condition to guide the DiT through adaptive normalization for video generation. To mitigate the effect of error introduced from AR inference, the authors introduce two noise-resilient strategies during DiT training, using coarser latent tokens and uncertainty sampling to make the semantic condition noisier.
weaknesses: 1. From Table 1, the proposed method lags behind compared methods in many metrics other than dynamic degree, such as Imaging Quality and Subject Consistency. In Table 2, the dynamic degree of proposed method (50.42) is significantly lower than that of the StreamingT2V(85.64). 
2. From the demo videos on the webpage, there is some room for improvement for the proposed method compared to others. For example, the result of "A teddy bear is swimming in the ocean." lacks of subject consistency, and its motion is not realistic, which may be consistent with the quantitative results in Table 1 and Table 2.
3. Although the authors introduce noise-resilient strategies for the DiT model training to mitigate the error issue from AR inference, I am concerned that these strategies cannot truly simulate the error of AR inference, which may limit the model performance.

Meta-review: This paper explores the generation of long videos guided by an autoregressive language model (LM) to produce conditions for a diffusion transformer (DiT) model. It introduces new techniques to bridge the gap between the LM and the DiT, including a new VQ-VAE that quantizes the DiT model’s input into visual tokens and tolerates noise robustness.

The reviewers generally acknowledge the paper’s strengths, such as its clear motivation, innovative techniques, and empirical results for generating long video sequences. While three out of four reviewers favor accepting the submission, one reviewer leans against it. 

Unfortunately, the opposing reviewer (N9At) did not respond to requests from either the authors or the AC to engage with the authors' rebuttal. 

The AC reviewed both the opposing review and the authors’ response and believes the raised concerns may have been addressed. Of the three main questions raised by Reviewer N9At, two were requests for clarification, to which the authors provided answers. The third question, which sought additional explanation of a performance difference to the baseline, is viewed by the AC as not a significant concern, and the authors provided a reasonable response.

Decision: Accept (Poster)