ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation

Authors: Zongyi Li, Shujie HU, Shujie LIU, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, Furu Wei
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: transformer; video generation; diffusion

Abstract:
Text-to-video (T2V) models have recently undergone rapid and substantial advancements. Nevertheless, due to limitations in data and computational resources, achieving efficient generation of long videos with rich motion dynamics remains a significant challenge. 
To generate high-quality, dynamic, and temporally consistent long videos, this paper presents ARLON,  a novel framework that boosts diffusion Transformers with autoregressive (\textbf{AR}) models for long (\textbf{LON}) video generation, by integrating the coarse spatial and long-range temporal information provided by the AR model to guide the DiT model effectively.
Specifically, ARLON incorporates several key innovations: 
1) A latent Vector Quantized Variational Autoencoder (VQ-VAE) compresses the input latent space of the DiT model into compact and highly quantized visual tokens, bridging the AR and DiT models and balancing the learning complexity and information density;
2) An adaptive norm-based semantic injection module integrates the coarse discrete visual units from the AR model into the DiT model, ensuring effective guidance during video generation; 
3) To enhance the tolerance capability of noise introduced from the AR inference, the DiT model is trained with coarser visual latent tokens incorporated with an uncertainty sampling module. 
Experimental results demonstrate that ARLON significantly outperforms the baseline OpenSora-V1.2 on eight out of eleven metrics selected from VBench, with notable improvements in dynamic degree and aesthetic quality, while delivering competitive results on the remaining three and simultaneously accelerating the generation process. In addition, ARLON achieves state-of-the-art performance in long video generation, outperforming other open-source models in this domain. 
Detailed analyses of the improvements in inference efficiency are presented, alongside a practical application that demonstrates the generation of long videos using progressive text prompts. Project page: \url{http://aka.ms/arlon}.

Decision: accept (poster)
Reviewer scores: 5, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730533381768

thread_type:
official_review

<thread_object>
creator: reviewer_45zf

<content>
summary:
This paper proposes a new text-to-video(T2V) framework consisting of autoregressive (AR) Transformers and Diffusion Transformers (DiT). Based on the input text prompt, the AR model predicts quantized visual tokens of a latent VQ-VAE nested within the 3D VAE of the DiT model. The coarse latent, reconstructed from the predicted tokens, serves as semantic condition to guide the DiT through adaptive normalization for video generation. To mitigate the effect of error introduced from AR inference, the authors introduce two noise-resilient strategies during DiT training, using coarser latent tokens and uncertainty sampling to make the semantic condition noisier.

strengths:
1.This paper innovatively combines the strengths of autoregressive (AR) Transformers and Diffusion Transformers (DiT) for generating long video with rich dynamic motion.
2. To mitigate the effect of error introduced from AR inference to DiT, the authors introduce two noise-resilient strategies during DiT training.
3. The paper is written and presented clearly and easy to follow.
4. The long video results of the proposed method show improvement on dynamic degree, and long video generation results using progressive text prompts are more consistent throughout the entire video.

weaknesses:
1. From Table 1, the proposed method lags behind compared methods in many metrics other than dynamic degree, such as Imaging Quality and Subject Consistency. In Table 2, the dynamic degree of proposed method (50.42) is significantly lower than that of the StreamingT2V(85.64). 
2. From the demo videos on the webpage, there is some room for improvement for the proposed method compared to others. For example, the result of "A teddy bear is swimming in the ocean." lacks of subject consistency, and its motion is not realistic, which may be consistent with the quantitative results in Table 1 and Table 2.
3. Although the authors introduce noise-resilient strategies for the DiT model training to mitigate the error issue from AR inference, I am concerned that these strategies cannot truly simulate the error of AR inference, which may limit the model performance.

questions:
1. In Figure1, the previously generated video seems to be used as a reference for the subsequent generation, but this is not illustrated in Figure 2. Does the DiT generate videos in an autoregressive way?
2. In Table 1 and Table 2, do the higher scores of metrics indicate better performance?
3. For long video generation in section 4.2 as well as the demo videos on the webpage, the authors only compare their proposed method with open-source text-to-long video generation models. Why not compare with the commercial closed-source text-to-video generation models like in Table 1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681059679

thread_type:
official_review

<thread_object>
creator: reviewer_gqnt

<content>
summary:
This paper proposes to leverage autoregressive models to guide the training of diffusion transformers for text-to-video generation task. The proposed framework incorporate with a latent VQ-VAE, coarser visual latent tokens and a uncertainty sampling module to connect DiT and AR models and inject the information from AR models to DiT training. Massive experiments are conducted with abundant quantitative metrics and visualizations are reported to demonstrate the performance of the proposed model.

strengths:
- The idea of make DiT and AR models working in one latent space is novel, and many technical improvements are designed to bridge their gap.

- The proposed method reaches a large reduction of denoising step of comparable generation quality.

weaknesses:
- It's better to use bold fonts and underscores in Table 1. According to the listed numbers, the proposed method doesn't reach top 3 in many columns. And the reproduced notation is missing.

questions:
- What is the motivation of using a latent VQ-VAE nested inside a pretrained 3D Autoencoder, instead of training a single-stage pixel-to-latent tokenizer? And what is the additional computational cost or gain in comparison?

- The proposed coarse latent token with different compression ratio, while how is this ablated in Table 3 or any other ablation studies? (Does the 4×8×8 row refers to both the same scale training and the 4×16×16 rows refer to different scale training?)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693492339

thread_type:
official_review

<thread_object>
creator: reviewer_rwnx

<content>
summary:
This paper proposes a long video synthesis pipeline, ARLON. The main idea of this paper is to combine DiT with autoregressive transformers that provide long-range temporal information. To bridge the DiT and the AR transformer, the pipeline novelly adopts 1) a Latent VQ-VAE to enable the AR model to learn on and the DiT to learn on different latent spaces, reducing learning complexity and allowing the AR model to manage coarse temporal dependencies; 2) an adaptive norm-based semantic injection module to guide the DiT using AR generated tokens. Novel training strategies of coarser visual latent tokens for DiT and uncertainty sampling are also proposed to make the training process more stable and generalizable. 

Results are compared with current t2v models over VBench and Vbench-long and achieve notable improvements especially on long video generation.

strengths:
- The motivation for using the AR model to provide semantic guidance is clear and nice.
- It is a very good extension of existing architectures.
- Good presentation.
- Good qualitative and quantitive results.

weaknesses:
I overall like this paper, but there are several points for improvement.

- No ablation on the impact of model structure and training data size.
- No discussion on failure cases and limitations.
- There might be some missing references such as nuwa-XL and Phenaki. GAN-based long video generation might also be related.

*[1] NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation*

*[2] Phenaki: Variable Length Video Generation from Open Domain Textual Descriptions*

I am not an expert in this field of training large video generation models. I will adjust the final score with other reviewers' comments and also based on the response from the author.

questions:
- How many seconds can the models generate for the longest videos and how is the performance? For How many seconds do the longest videos that your method generates can last? In my understanding, this is the key advantage of the hierarchical generation framework.
- The main limitation of this work seems to be the huge computational cost of training, but the related information (type and number of GPU, training time) is not provided. It would be nice to know this information.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730958081592

thread_type:
official_review

<thread_object>
creator: reviewer_n9at

<content>
summary:
The manuscript introduces ARLON, a text-to-video framework that efficiently generates high-quality, dynamic, and temporally consistent long videos. By combining Autoregressive models with Diffusion Transformers, ARLON employs innovations like VQ-VAE for token compression, an adaptive semantic injection module, and an uncertainty sampling module to enhance efficiency and noise tolerance. It reduces denoising steps and outperforms OpenSora-V1.2 in both quality and speed, achieving state-of-the-art performance.

strengths:
1. The integration of Autoregressive models with Diffusion Transformers and innovations like VQ-VAE for token compression, adaptive semantic injection, and uncertainty sampling show originality in addressing long video generation challenges.
2. The generated video spans 600 frames, making it relatively long.

weaknesses:
1. In Table 2, why does StreamingT2V have a higher Dynamic Degree score (85.64) compared to ARLON (50.42)?
2. The paper lacks a detailed comparison of the number of parameters in ARLON versus baseline models.
3. There is no analysis of ARLON's memory footprint during training and inference, which would clarify its computational efficiency relative to models like OpenSora-V1.2.

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732257937980

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Model Details

comment:
Caption: The batch size for training both the AR and DiT models is set to 2×68-frame 512×512 video clips, and the inference is evaluated by generating a 68-frame (68f) or 578-frame (578f) 512×512 video. Both the training and inference processes are executed on a single 40G A100 GPU.

| **Method**             | **OpenSora1.2 (baseline)**      | **ARLON (ours)**                                 |
| ---------------------- | ------------------------------- | ------------------------------------------------ |
| Param. (AR)            | -                               | 192M                                             |
| Param. (DIT)           | 1.2B                            | 92M (trainable) + 1.2B (frozen)                  |
| Param. (3D VAE)        | 384M                            | 384M                                             |
| Param. (latent VAE)    | -                               | 30M                                              |
| Traning Memory         | 36007M                          | 7701M (AR) ; 36815MB (DiT)                       |
| Inference Memory       | 24063M                          | 2269M (AR) ; 25215M (DiT)                        |
| Inference speed (68f)  | 47.3 s                          | 5.7s (AR)+18.9s (DiT)                            |
| Inference speed (578f) | 47.3 × 11 s                     | 57.2s (AR) + (18.9 × 11) s (DiT)                 |
| Inference FLOPs (68f)  | 42626G × 30 (step)              | 200G (AR) + 46461G×10 (step) (DiT)               |
| Inference FLOPs (578f) | 42626G × 30 (step) × 11 (times) | 1547G (AR) + 46461G×10 (step) × 11 (times) (DiT) |

From the results of the table, we can observe that:

1. the increase in the number of parameters (**192M + 92M + 30M**) of ARLON is minimal compared to the **1.2B** parameters of the baseline, OpenSora-V1.2. This is primarily due to our adoption of an efficiency adapter approach during training, which enables us to introduce fewer parameters while preserving performance. Additionally, the latent VQ-VAE operates within a compressed latent space, and the AR model is training and generates highly quantized tokens, both of which significantly contribute to the minimized parameter requirements.
2. it is evident that our method does not require significant additional memory and computational resources compared to the baseline model. Specifically, during the training and inference phases, there are **2.2% and 4.8%** relative increases respectively (the AR and DiT models can be trained independently).
3. Conversely, by leveraging the AR code as an efficient initialization, our model is capable of generating high-quality videos with significantly fewer steps. Consequently, our inference time and total FLOPs are superior to those of the baseline, thereby significantly accelerating the denoising process. Specifically, our model achieves a **48-49%** relative improvement in inference speed and a **64%** relative reduction in computational FLOPs for 68-frame or 578-frame video generation (**578 can be expressed as 68 × 11 - 17 × 10**. Here, 11 represents the number of times the DiT model generates 68-frame video segments, while 17 signifies the number of frames in the conditioned video. Additionally, 10 indicates the number of times the DiT model generates videos under specific conditions).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732257970012

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary

comment:
We would first like to express our gratitude to Program Chairs, Senior Area Chairs, and Area Chairs for their efforts, as well as to the dedicated reviewers for their insightful comments on our paper.

Additionally, we appreciate the reviewers for highlighting the **strengths** of our work:

1. Novel framework, that seamlessly combines the strengths of autoregressive Transformers and Diffusion Transformers (DiT).
2. Novel technical designs, including a latent VQ-VAE, adaptive semantic injection and noise-resilient strategies.
3. State-of-the-art long video generation performance, and the faster generation process with comparable performance.
4. The paper is written and presented clearly and easy to follow.

On the other hand, based on the constructive feedback, suggestions and comments from reviewers, we will make the following revisions to our paper:

1. We would like to emphasize that ARLON's primary strength lies in its ability to generate **long videos with remarkable consistency and quality**, while also delivering **impressive dynamic scores.** To more effectively showcase our contributions, we have adjusted the sequence of our presentation: we now begin with the results of the long videos, followed by those of the short videos, thereby **swapping the original order of Tables 1 and 2**, as well as **that of Figures 4 and 5.**
2. In Appendix, we have added **i)** comprehensive implemental details, encompassing the number of parameters, memory requirements for both training and inference,  inference speed, and FLOPS, for both the baseline OpenSora-V1.2 and our advanced ARLON models; **ii)** the failure cases and analysis for these cases; **iii)** Limitations and Broader Impact; **iv)** results and analysis of various size of training data for DiT model.
3. In related work, We have added NUWA-XL and Phenaki, and GAN-based long video generations, and discussed their relevance to our approach. In addition, all of them have been added into Reference.
4. For Table 2 (Table 1 in the original manuscript), we have divided it into two sections. The lower section presents the results of ARLON and OpenSora-V1.2, with our superior results highlighted with specific improvements. The upper section, meanwhile, displays the results of other text-to-video models. In addition, we have added the sentence "the higher scores of metrics indicate better performance." to the table captions.
5. For Table 3, we have changed the column name "Compress Ratio" to "Compress Ratio in DiT", and added the description "The compression ratio of the latent VQ-VAE for AR model is 4x8x8" into the table caption.
6. In Section 4.3, we have revised the content to avoid any confusion for the readers ("which could simulate the errors" -> "which could make the DiT model tolerate the errors").

Once again, we sincerely thank you for your invaluable input and the time you have dedicated to reviewing our paper. Your constructive feedback and insightful suggestions have been instrumental in enhancing the quality of our research.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper explores the generation of long videos guided by an autoregressive language model (LM) to produce conditions for a diffusion transformer (DiT) model. It introduces new techniques to bridge the gap between the LM and the DiT, including a new VQ-VAE that quantizes the DiT model’s input into visual tokens and tolerates noise robustness.

The reviewers generally acknowledge the paper’s strengths, such as its clear motivation, innovative techniques, and empirical results for generating long video sequences. While three out of four reviewers favor accepting the submission, one reviewer leans against it. 

Unfortunately, the opposing reviewer (N9At) did not respond to requests from either the authors or the AC to engage with the authors' rebuttal. 

The AC reviewed both the opposing review and the authors’ response and believes the raised concerns may have been addressed. Of the three main questions raised by Reviewer N9At, two were requests for clarification, to which the authors provided answers. The third question, which sought additional explanation of a performance difference to the baseline, is viewed by the AC as not a significant concern, and the authors provided a reasonable response.