ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation
Authors: Zongyi Li
Shujie HU
Shujie LIU
Long Zhou
Jeongsoo Choi
Lingwei Meng
Xun Guo
Jinyu Li
Hefei Ling
Furu Wei
Venue: ICLR 2025 Poster (2025)
Keywords: transformer; video generation; diffusion
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Text-to-video (T2V) models have recently undergone rapid and substantial advancements. Nevertheless, due to limitations in data and computational resources, achieving efficient generation of long videos with rich motion dynamics remains a significant challenge. 
To generate high-quality, dynamic, and temporally consistent long videos, this paper presents ARLON,  a novel framework that boosts diffusion Transformers with autoregressive (\textbf{AR}) models for long (\textbf{LON}) video generation, by integrating the coarse spatial and long-range temporal information provided by the AR model to guide the DiT model effectively.
Specifically, ARLON incorporates several key innovations: 
1) A latent Vector Quantized Variational Autoencoder (VQ-VAE) compresses the input latent space of the DiT model into compact and highly quantized visual tokens, bridging the AR and DiT models and balancing the learning complexity and information density;
2) An adaptive norm-based semantic injection module integrates the coarse discrete visual units from the AR model into the DiT model, ensuring effective guidance during video generation; 
3) To enhance the tolerance capability of noise introduced from the AR inference, the DiT model is trained with coarser visual latent tokens incorporated with an uncertainty sampling module. 
Experimental results demonstrate that ARLON significantly outperforms the baseline OpenSora-V1.2 on eight out of eleven metrics selected from VBench, with notable improvements in dynamic degree and aesthetic quality, while delivering competitive results on the remaining three and simultaneously accelerating the generation process. In addition, ARLON achieves state-of-the-art performance in long video generation, outperforming other open-source models in this domain. 
Detailed analyses of the improvements in inference efficiency are presented, alongside a practical application that demonstrates the generation of long videos using progressive text prompts. Project page: \url{http://aka.ms/arlon}.

Published as a conference paper at ICLR 2025
ARLON: BOOSTING DIFFUSION TRANSFORMERS WITH AUTOREGRESSIVE MODELS FOR LONG VIDEO GENERATION
Zongyi Li Shujie Hu²*, Shujie Liu Long Zhou", Jeongsoo Choi', Lingwei Meng",
Xun Guo Jinyu Li,, Hefei Ling', Furu Wei
Huazhong University of Science and Technology
2 The Chinese University of Hong Kong
3 Microsoft Corporation
KAIST
ABSTRACT
Text-to-video (T2V) models have recently undergone rapid and substantial ad- vancements. Nevertheless, due limitations in data and computational resources, achieving efficient generation of long videos with rich motion dynamics remains a significant challenge. To generate high-quality, dynamic. and temporally con- sistent long videos, this paper presents ARLON, a novel framework that boosts diffusion Transformers with autoregressive (AR) models for long (LON) video generation, by integrating the coarse spatial and long-range temporal informa- tion provided by the AR model to guide the DiT model effectively. Specifically, ARLON incorporates several key innovations: 1) A latent Vector Quantized Varia- tional Autoencoder (VQ-VAB) compresses the input latent space of the DiT model into compact and highly quantized visual tokens, bridging the AR and DiT mod- els and balancing the learning complexity and information density; 2) An adaptive norm-based semantic injection module integrates the coarse discrete visual units from the AR model into the DiT model, ensuring effective guidance during video generation: 3) To enhance the tolerance capability of noise introduced from the AR inference, the DiT model is trained with coarser visual latent tokens incor- porated with an uncertainty sampling module. Experimental results demonstrate that ARLON significantly outperforms the baseline OpenSora-Vil V1.2 on eight out of eleven metrics selected from VBench, with notable improvements dynamic degree and aesthetic quality, while delivering competitive results on the remain- ing three and simultaneously accelerating the generation process. In addition, ARLON achieves fate-of-the-art performance in long video generation, outper- forming other open-soure models this domain. Detailed analyses of the im- provements in inference efficiency are presented, alongside practical application that demonstrates the generation of long videos using progressive text prompts. Project page: http:// aka ws. mssarlon
INTRODUCTION
Text-to-video (T2V) models have recently undergone rapid advancements, driven by both Trans- former architectures (Vaswani 2017) and diffusion models (Ho et al., 2020). Autoregressive (AR)
models, such as decoder-only Transtormers, offer notable ad advantages in scalability and long-range
in-context learning (Wang al. 2023a). demonstrating strong potential for video generation from text (Yan et al., | Ge et al., 2022, Hong et al.] 2002]] Yu et al., 2023] Kondratyuk et al.) (023). Meanwhile, diffusion-based models, including U-Net and Diffusion Transformers (DiT), have new benchmark in high-quality video generation, establishing themselves as dominant approaches
in the field (Zheng et al., 2024, Lab & etc. 2024). However, despite the rapid progress in DiT
models, several key challenges remain: 1) High training cost, especially for high-resolution videos,
"quual contribution. Work was done during internship at Microsoft Research Asia. Corresponding author.

---

Published as a conference paper at ICLR 2025
resulting in insufficient motion dynamies as training is restricted to short video segments within each batch; and 2) The inherent complexity and time-consuming nature of generating videos en- tirely through denoising based solely text conditions; and 3) Difficulty in generating long videos with consistent motion and diverse content.
Previous research typically employed autoregressive approaches for long video generation with DiT models (Weng et al,, [024] Gao et al.] 2024; Henschel et al.] 2024), generating successive video segments conditioned on the last frames of the previous segment. However, computational con- straints restrict the length of these conditioned segments, resulting in limited historical context for the generation of each new segment. Additionally, when given the same text prompts, generated short video segments often feature identical content, increasing the risk of repetition throughout the entire overall long video. Diffusion models, while excellent at producing high-quality videos, strug- gle to capture long-range dependencies and tend to generate less dynamic motion. Additionally, the requirement for multiple denoising steps makes the process both computationally expensive and slow. By contrast, AR models are more effective at maintaining semantic information over long sequences, such as motion consistency and subject identity. They excel at generating continuous actions without the repetition issues that diffusion models often encounter in long video generation, and the inference of AR models typically achieves faster inference times.
For the First segment
For the Second sepment
condition
AR Transformer
Diffusion
Diffusion
Text
Noise
Video
Noise
Video
Figure 1: Generation process for long videos with autoregressive transformer and DiT.
Leveraging AR for coarse predictions, we enhance the diffusion model's capacity capture richer dynamics and maintain continuity in long video sequences. The AR-generated features can also serve as initialization to accelerate the diffusion process, resulting in faster and more efficient generation of long-form videos. Building on this concept. we propose ARLON, a novel framework that effectively combines the advantages of autoregressive Transformer and DiT models for long video generation. As shown in Fig. D ARLON first generates long-term, coarse-grainet discrete visual units (AR codes) autoregressively using a decoder-only Transformer. These discrete AR codes are then segmented and sequentially fed into the DiT model by the proposed semantic injection module, which autoregressively generates high-quality video segments. Specifically, the first N seconds of AR codes guide the DiT model to generate the first video segment as illustrated in the middle part of Fig. [ The second N second of AR codes, along with the last M seconds of the first video segment, serve the condition to generate the subsequent video segment.
As shown in Figure 2, to bridge the feature spaces between the AR Transformer and DiT model, as well as balance the learning complexity of the AR model and the information density of the visual tokens, we employ a 3D) latent VQ- VAE to compact and quantize the input latent features of the DiT model into discrete tokens. Various architectures of the semantic injection module, such as MLP adapter, adaptive norm modules, and ControlNet, are explored to ensure the coarse-grained AR codes guide the DiT model effectively. However, unlike scenarios where conditioned images, videos, or motion trajectories (Chen et al. [023bb Zhang et al. (2024) Peng et al. 2024) are available, the tokens generated by autoregressive models text-based video generation scenarios tend to be noisy, leading to noticeable drop in accuracy when transitioning from teacher-forcing training to autoregressive inference. To address this, we propose two key innovations: 1) training the DiT model model using coarse visual latent tokens generated by different latent VQ-VAE with higher compression rate than those used for AR model training, and 2) integrating an uncertainty sampling module into the semantic injection module to further enhance model performance.
Our ARLON model is evaluated using the VBench (Huang et al., 2024) video generation bench- mark. Experimental results demonstrate that ARLON outperforms the baseline OpenSora-Vil V1.2 on eight out of eleven metrics, while also delivering competitive performance across other metrics. Additionally, ARLON achieves state-of-the-art performance by effectively generating high-quality,
2

---

Published as a conference paper at ICLR 2025
temporally coherent, and dynamically rich long videos, surpassing other open-source models in this area. Our contributions can be summarized in three points:
We present ARLON, novel framework that seamlessly combines the strengths of autore- gressive Transformers and Diffusion Transformers (DiT), In this approach, the AR model supplies coarse spatial and long-range temporal information, effectively guiding the DiT model to generate long, high-quality videos with rich dynamic motion.
To bridge the AR and DiT models while balancing learning complexity and information density, latent VQ-VAE introduced to compress the DiT model's input space into com- pact, highly quantized visual tokens. These tokens are then used to train an autoregressive Transformer model, generating visual tokens based on the input text prompt. To reduce the noise inevitably introduced during AR inference, we introduce two noise-resilient strate- gies for the DiT model training: coarser visual latent tokens and uncertainty sampling.
Both quantitative and qualitative analyses of the improvement inference efficiency are given. In addition, long video generation using progressive text prompts is implemented. where each subsequent prompt builds on the previous.
Latent VQ-VAE Compression
Autoregressive Modeling
AR Tranefomeer
Laterd VO-VRE
VO-WAE
Coars Lateet
Semantic-aware Condition Generation
Text Enoodar (15-XXL)
STDIT
"A etreappunn can wii gompler valagg sarosesded
Video Fautures
patchely
Time dap
Figure 2: Overview of the ARLON framework, which consists of three key components: Latent VQ-VAE Compression, Autoregressive Modeling, and Semantic-aware Condition Generation.
2 METHOD
As illustrated in Figure 2 our ARLON comprises three primary components: Latent VQ-VAE Compression, Autoregressive Modeling. and Semantic-aware Condition Generation. Given a text prompt, the autoregressive (AR) model predicts coarse visual latent tokens, which constructed from a 3D VAE encoder followed by latent VQ- encoder based on the target video. These pre- dicted visual latent tokens encapsulate both the coarse spatial information and consistent semantic information. Based on these tokens, a latent VQ-VAE decoder generates continuous latent features, which serve as semantic conditions to guide the DiT model with semantic injection module. To mitigate the noise inevitably introduced during AR inference, we introduce two moise-robust training strategies: 1) coarser visual latent tokens, and 2) uncertainty sampling module.
2.1 AUTOREGRESSIVE
Latent VQ-VAE To align the feature spaces between the AR and DiT, and obtain quantized and compact discrete visual tokens, a latent VQ-VAE nested within the 3D VAE of the DiT model constructed. Following Yan et al. (202T), the latent VQ-VAE encoder Elatent consists of 3D convolutional neural network (CNN) blocks and residual attention blocks, followed by decoder Distent structured as the reverse of the encoder. Let the inputs of the VQ-VAE be denoted as e R7 x xC If the spatial and temporal compression factors of the 3D CNN encoder are and
respectively, the encoder produces latent embeddings V R
x/ Each embedding vector
3

---

Published as a conference paper at ICLR 2025
R in V is quantized to the closest entry R in the learned codebook C E R x The index of each entry is used represent the latent embeddings V as Q (1, 21, x For decoding, given the indices of video tokens, we can retrieve the corresponding entry c, which is used obtain reconstructed video embeddings F using the latent VQ-VAE decoder.
Autoregressive Modeling We employ a causal Transformer decoder as the language model to au- toregressively generate discrete visual tokens from textual input. Specifically, the indices of visual tokens Q are subsequently decomposed into ID spacetime patches QAR = [1 92..... qv], .v], with <EOS> appended in the end of whole video, and <FRAME> inserted at the end of each frame, where N ( + x H. x W and E (1, 2, K). These indices are converted into embeddings by the video code embedding layer, added with learnable position embedding. The text based condition Y is the contextual embedding of the video captions, generated by the T5 encoder (Raffel cct al. 2020). The AR model, comprising blocks of multi-head attention and feed-forward layers, takes the concatenation of text and visual embeddings as input to model the dependency between these information, and the model is optimized to maximize the following probability
N = II p(q. Y, Q : AR)
P(QARIY;
(1)
2.2 SEMANTIC-AWARE CONDITION GENERATION
STDIT Our ARLON framework is built on a spatial-tempont Transformer (Zheng et al. 2024) which serves the backbone model. Given an input image latent 3D embedding layer first projects the image into non-overlapping patches, which are then flattened. These flattened features subsequently augmented with spatial and temporal position embeddings using ROPE (Su al.) 2024). The augmented features are then processed through a series of spatial-temporal DiT blocks, which conclude with an unpatchify layer predicting the noise. Each spatial-temporal DiT block comprises sequential modules for spatial- apatial-attention, caapoorallattettoon and cross-attention:
X X + SpatialAttm((N((X)).
X rearrange (X, (bt)sd - (bs)td),
X X + TempAttn(LLN(X))
where (bt)sd tbs) means rearranging the tensor by merging the batch size b with the spatial dimension and isolating the temporal dimension for subsequent temporal attention processing. The text information is incorporated with DiT features using cross-attention, providing auxiliary information to enhance spatial-tempotes consistency in video generation.
AR Semantic Condition Videos can be compressed into a coarse latent space using video VAE and latent VQ-VAE. As the AR model predicts tokens within latent VQ- space, we leverage the reconstructed latent features from the latent VQ- VAEE decoder as semantic conditions for training the diffusion model. These conditional features are subsequently employed to determine the coarse spatial and temporal content in the video. Given the video x, the corresponding conditional features F can be extracted as:
F P=D Eratent (Evide((x))).
Following the STDIT model, we initially project the coarse latent feature F with 3D embedding layer to generate the input condition Fo. followed with several adapter layers to inject the semantic information into the video generation process. As shown in Figure the adapter block contains several residential MLP block:
F (Lapeer,LayyrNoom, F.)) + F.
Semantic Injection To incorporate coarse semantic information into video generation, we inject the AR semantic condition into the DiT model to guide the diffusion process. Rather than directly adding the condition to each block, we utilize gated adaptive normalization mechanism for condi- tion injection. As shown in the upper part of Figure 3 the input latent variable X; is first processed with a layer normalization, and the conditional latent variable F, is processed with uncertainty sampling (will be introduced in Section 2.3) get F,, which is projected into three parameters: scale 7% shift 3,, and gated parameter a,, followed by the application of adaptive layer normaliza- tion inject the conditioning information into the original latent variable. Additionally, to regulate
4

---

Published as a conference paper at ICLR 2025
the latent strength, introduce extra gated layer, initialized to zero, to adaptively control the injected features by adding them to the original DiT feature X₃:
0i B.. -MLP FF))
(5)
Fusion X;, 04, Bi 7:) =04 ② ML. (7:: @ LayerNorm(X;) + B.) + X..
2.3 TRAINING STRATEGY
In the training phase, each training sample sists of three inputs: the original video, a textual prompt Y, and the AR semantic condition F. For each video, we first convert into the latent space X Subsequently, timestep t randomly sam- pled from the interval [o, T), and noise is added to the video latent X resulting in X². Our ARLON is then optimized using the following procedure:
Nem Jelle
X;
ME
Injection
|||e - cB (x', L,
F
ExO,
F
Sample Pi
To enable overlapping long video generation and smage-to-video generation, randomly unmask frames, leaving them noise-free to serve as condi- tioning frames. To tolerate the errors inevitably in- troduced during AR inference, we implement two noise-resilient training strategies: coarser visual la- tent tokens and uncertainty sampling.
Uncertainty Sampliing
((.
Figure 3: Semantic injection and uncertainty sampling.
Coarser Visual Latent Tokens During the training phase, we employ two variants of latent VQ- VAE with distinet compression ratios to enhance the diffusion process to tolerate noisy AR predic- tion results. Specifically, for AR training, our latent VAE utilizes a compression ratio of (r, o, o), whereas for the DiT model training, we adopt compression rate of (r, 20, 20). By using noisier and coarser information in the model training, DiT model is encouraged to capture more general patterns, thereby reducing the risk of specific errors inevitably introduced by AR prediction results.
Uncertainty Sampling To simulate the autoregressive (AR) prediction variance, we introduce an uncertainty sampling module. As depicted in the lower part of Figure 3 this mechanism generates noise through the uncertainty sampling module rather than strictly relying on the original coarse latent features F,, and it is applied randomly after each adapter block during model training. Instead of injecting standard Gaussian noise, the noise is drawn from the original distribution of the latent features F, The mean μı; and standard deviation σ; of the noise coarse latent feature F,, and normalization layer is applied to produce the whitened feature F; F- Following Chen et al. 2022), the sampled feature F; is calculated as:
first calculated from the original
F =
3 F i., o, N (1,o)), A (145,
where o; and μå; represent noisy vectors sampled from the modeled mean and variance distribu- tion of the target feature. To ensure that the sampled feature distribution closely approximates the original. the mean value of o distribution is set to 1.
RELATED WORK
3.1 TEXT-TO-VIDED GENERATION
In recent years, substantial research has been dedicated to the development of text-to-video gener-
ation (T2V) models (Ho et al.) 2022b a Singer et al. 2022; Chen et al.) 2023a Zhou et al. 2022
Wang et al. 2024 2023b Blattmann al. 2023 Guo et al.) 2023 Zeng et al. 2024 Ma et al.
2024 Peebles Xie 2023 Zheng et al. 2024: Lab & etc. 2024 Yang et al. 2024: Ju et al. 2024).
These efforts can be broadly categorized into two main types: language model-
and diffusion-
model model-based methods. For diffusion-mout ceaisiionmodee-baaeeee based approaches, pioneering works such as VDM (Ho
al. 2022b) employ a 3D U-Net diffusion model for video generation. Imagen Video (Ho al.
5

---

Published as a conference paper at ICLR 2025
2022a) and Make-a-Vided (Singer et al. (022)) introduce spatiotemporally factorized models to gen- erate high-definition videos. Subsequently, VideoCraft (Chen et al. 2023a) and Magic Video (Zhou et al. 2022) utilize Video VAE and larger datasets to enhance the generalization capabilities of video models. Magic Video V2 (Wang et al. (2024) and Lavie (Wang et al., 2023b) propose cascaded mod- els for high-quality and aesthetically pleasing video generation. Moreover, SVD (Blattmann al.) 2023), Animatediff (Guo et al., 2023), and PixelDance (Zeng et al. (024) employ T2I models to generate images and subsequently animate them into videos. Meanwhile, Latte (Ma al. 2024) explores the training efficiency of video generation using a DiT model (Peebles & (023) and SORA accelerates the investigation of DiT models. Recently, more DiT-based diffusion models have emerged, including OpenSora (Zhngg et al. (024), OpenSoraPlan (Lab & etc.) 2024). CogvideoX (Yang et al., 2024). and Mira (Ju et al. 2024). While diffusion-based methods can generate high- quality videos, they are typically trained on fixed-length short videos (e.g.. 16 frames), which limits their ability to produce longer videos. Furthermore, these methods predominanty focus on videos with small dynamic ranges. Language-model-bee based methods leverage the transformer architecture to predict the next latent code of video representations in an autoregressive manner. VideoGPT (Yan) eet al., (021)) and TATS (Ge et al.] (022) utilize GPT-like transformer models to generate extended video sequences. Video (Hong et al. (022) employs a transformer to produce key frames, fol- lowed by second upsampling stage to achieve higher frame rates. Recently, Magvit2 ( YYu et al.) 2023) introduced a novel lookup-frce free quantization approach, enhancing visual quality in language- based models. VideoPoet (Kondratyuk et al. (203) incorporates a mixture of multimodal inputs into large language models (LLMs) for synthesizing video tokens. Although transformer- based models can effectively capture long-range dependencies, they demand substantial resources for training long videos, and their quality still requires improvement.
3.2
LONG VIDEO GENERATION
The generation of long vidcos presents significant challenges due to inherent temporal complexity and resource constraints. Previous autoregressive GAN-based models (Gie et al. 2022 Yu et al,
2022 Skorokhodov et al. 2022) utilize sliding-window window attention mechanisms facilitate the gen- eration of longer videos. However, despite these advantages, ensuring the quality of the generated videos remains problematic. Phenaki (Villegas et al. 2022) proposes a model for realistic video synthesis from textual prompts using a novel video representation with causal attention for variable- length videos, enabling the generation of arbitrary long videos. NUWA-XL (Yin et al.) (023) intro- duces a Diffusion over Diffusion architecture for extremely long video generation, allowing parallel generation with "coarse-to- fine" process to reduce the training-inference gap. Recent diffusion- based models (Ma et al.) 2024 ZZheng et al., (024) typically employ conditional mask inputs for overlapping generation. Although these mask generation methods can produce long videos, they often encounter issues related to temporal inconsistency. Recent advancements, such as Stream- ingT2V (Henschel et al. 2024), have introduced the injection of key frames into diffusion processes to enhance temporal consistency across different video segments. Additionally, some training-free approaches (Qiu et al. 2023 Lu al. 2024) leverage noise rescheduling techniques to improve tem- poral consistency. Moreover, Video (Tian et al. (024) presents compositional framework for video generation. Our proposed method effectively integrates an autoregressive model for long-term coherence with a diffusion-basee of existing techniques such as sliding window and diffusion-over ciriuiioonover--fffuiinnn methods. This approach ensures video integrity and detail coherence over extended periods without repetition.
DiT model for short-term continuity, overcoming the limitations
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Dataset. For the training of the latent VAE and AR transformer model, about 5.7M video clips are used, consisting of Openvid-IM (Nan et al. 2024), ChronoMagic-PProll (Yuan et al. 2024) and OpenSora-plan (including Mixkit, Pexels and Pixabay) (Lab & etc. (024). For training the DiT model. we use 0.7M video clips from OpenVidDDD VidHD-O.4M and Mixkit. To evaluate the performance of text-to-video generation, we utilize prompts from the Vbench benchmark (Huang et al. 2024) for comparison against other state-of-the-an models. In the ablation studies, 100 prompts are randomly selected from Open VM, excluded from the training dataset.
Evaluation Metrics. To assess the text-to-video generation, we employ evaluation metries con- sistent with those used in VBench and bbench-Long: Dynamic Degree, Aesthetic Quality, Imaging Quality, Subject Consistency, Background Consistency, Motion Smoothness, Temporal Flickering,

---

Published as a conference paper at ICLR 2025
Temporal Style, Overall Consistency, Scene and Object Class. The first six metries used in the ablation studies.
Implementation Details. The time-space compression ratio of the latent VAE is 4x8x8 and 4x 16x 16 for the training of the AR model and DiT model, and the dimension and vocabulary size of the codebook are 256 and 2048 respectively. The AR model has the transformer structure with 12 layers. 16 attention heads, an embedding dimension of 1024, a foed-forward layer dimension of 4096, and dropout of 0.1. DiT is initialized with OpenSora- DoenSora-V1.2. V1.2, fixed during model training. The uncertainty sampling is employed randomly with a probability of 0.1. We use the Adam optimizer with learning rate of 2 10-5 for fine-tuning. The model is trained at resolution of 512 512 and with a frame range from 51 to 136.
Table 1:
Long video generation (600 frames) results of ARLON and other models on VBench. The higher scores of metrics indicate better performance.
Subject Consist
Background Consist
Motion Smooth
Dynamic Degree
Aesthetic Quality
Imaging Quality
Overall Consist
Models
FreeNoise
96.59
97.48
98.36
17.44
47.39
63.88
25.78
StreamingT2V
87.31
94.64
93.83
85.64
44.57
53,64
23.65
OpenSora-Vi V1.2
96.30
97.39
98.94
44.79
56.68
51.64
26.36
ARLON (Ours)
97.11
97.56
98.50
50.42
56.85
53.85
26.55
Freenoise Streaming t2v
erstockk
OpenSora
"a teddy bear is swimming in the ocean."
Frame 100
Frame 300
Frame 400
Frame 600
Figure 4: Qualitative comparisons between StreamingT2y T2V, FreeNoise, OpenSora, and ARLON.
Each video contains 600 frames.
4.2 RESULTS DISCUSSIONS
Long Video Generation We compare our ARLON model with other open-source text-to-long video generation models: Streaming (Henschel et al. 2024), FreeNoise (Qiu et al. 2023). and OpenSora-VIL [Zheng et al., 2024). As shown in Table I while FreeNoise achieves the highest image quality, its low motion metrics indicate that the generated videos are mostly static or contain minimal movement. contrast, Str exhibits high levels of dynamism, but this comes at the cost of consistency and overall video quality. As the length of the generated video increases, the temporal coherence between different segments diminishes. Compared to the baseline model, OpenSora-V1.2. ARLON demonstrates significant improvements across almost all metrics. While the increase in dynamism leads to a slight reduction in motion smoothness, we consider this trade- off acceptable for the more dynamic and engaging video content ARLON produces. We also present several long video examples in Figure 4 while FreeNoise generates almost static videos, with min- imal movement, such the bear remaining stationary throughout. In contrast, our ARLON strikes a better balance, generating videos that not only exhibit dynamic motion but also maintain a high level of temporal consistency and natural flow.

---

Published as a conference paper at ICLR 2025
Table Performance comparison of Text-to-video (T2V) generation between our ARLON and other open-source commercial models on VBench benchmark. The higher scores of metrics indicate better performance. The highlighted number in the top right corner reflects the improvements we achieved in comparison to OpenSora-V1.2.
Models
Dynamic Degree
Aesthetic Quality
Imaging Quality
Subject Comsist
Background Consist
Motion Smooth
Temporal Flicker
Temporal Style
Overall Consist
Scene
Object
Kling
46.9
61.2
65.6
97.6
99.4
99.3
24.2
26.4
50.9
87.2
Gen-2
18.9
67.4
99.6
24.1
26.2
48.9
90.9
VL.a
47.5
62.0
61.9
96.9
97.4
99.5
99.7
24.2
25.9
49,8
88.7
VideoCrafter-2.0
42.5
63.1
67.2
96.8
98.2
97.7
98.4
25.8
28.2
55.3
92.6
LaVie
49.7
54.9
61.9
91.4
97.5
96.4
98.3
25.9
26.4
52.7
91.8
LaVie Interpoaaooonnnnonn
46.1
54.0
59.8
92.0
97.3
97.8
98.8
26.0
26.4
52.6
90.7
Show-1
44.4
58.7
98.2
99.1
25.2
27.5
47.0
93.1
CogVideo
42.4
38.2
41.0
96.5
97.6
7.8
7.7
28.2
73.4
OpenSoraPlan-VI.
47.7
56.9
62.3
95.7
96.7
98.3
99.0
23.9
26.5
27.1
76.3
OpenSora- VT.2
47.2
56.2
60.9
94.5
97.9
98.2
99.5
24.6
27.1
42.5
83.4
ARLON (Ours)
52.8
61.07
61.0 10
93,4
97.1
10.8
98.9
99.4
25.3
27.3
54.4
11.9
89.816.4
ascmennmmmal hore (ult mouniaaa atasnnu srrvee wisting throggh Aip.
"Aca pende reting Chiwar/ood resteurant"
OpenSora
Ours
Figure 5: Comparison of qualitative results for text-to- text-to-video generation.
Text-to-video Generation Our ARLON model compared with other state-of-the-art open- source and commercial closed closed-source text-to-video generation models on VBench, including Kling Gen-22 Pika-V1.0 VideoCrafter-2.0 (Chen al. 2024). LaVie (Wang al. 2023b), La Interpolation (Wang et al., 2023b), Show-1 1 (Zhang et al. 2023). Video (Hong et al. 2022). OpenSoraPlan-Ve VI.1 (Lab & etc. 2024) and OpenSora (Zheng et al.] (024). As shown in Table we conducted a quantitative comparison between ARLON and ten other text-to-video models, where ARLON demonstrates superior performance in terms of dynamic degree while maintaining strong results across other metrics. Compared to the baseline OpenSora- V1.2, ARLON outperforms eight out of eleven metrics, with particularly significant improvements in dynamic degree, aesthetic quality, scene and object metrics, while achieving competitive results in the remaining three. Figure S) shows the qualitative comparisons between ARLON and OpenSora. The scenes from OpenSora mostly static, while ours exhibits significant camera movement and aligns better with the text.
2
4.3 ABLATION STUDY
The highly compact and quantized tokens generated by autoregressive (AR) models can introduce noise and errors when transitioning from teacher-forcing training to autoregressive inference. Effec- tively handling those noise and errors, several key factors explored in the DiT model training: 1) the design of the semantic injection module, including its architecture and the number of DiT layers where it applied; and the training strategies employed for the DiT model to enhance its robustness. The results in Table reveal several key points:
Employing much coarser granularity during the DiT model training phase, specifically, utilizing the latent VAE with a compression ratio of 4 x 16 x 16, compared to ratio of 4 x 8 x 8 during inference, can generate more inaccurate and noisier visual latent representations, which could make the DIT model tolerate the errors, thereby improving its robustness, and maintaining the consistency and qualities of the generated videos.
While the MLP adapter-based semantic injection achieves higher consistency and ControlNet
'hutp://klligaii http://linnaii.uuussu..oooom kuaishou.com//
8

---

Published as a conference paper at ICLR 2025
demonstrates more dynamic motion, both methods fall short in certain metrics. Although Control- Net excels dynamic motion, outperforming other methods by significant margin, it struggles with subject consistency, particularly
observed subjectively (as shown in Appendix, Figure 10 The adaptive norm method, however, strikes a more balanced performance across all criteria.
As depicted in Figure 6 the first sub-figure presents clip of the AR code- redesrrecostructeed video, while the second shows the baseline without AR codes. The third to sixth sub- sub-figures correspond to videos with AR codes injected into the last 14, first 3, 8 and 14 layers of the DiT model (with 28 layers in total) respectively. Injecting AR codes into the last 14 layers provides insufficient layout information, resulting
video similar to the baseline. In contrast, injecting codes into the first 3, 8, and 14 layers ensures that the layout information in the generated video aligns with AR codes, with greater control achieved as more layers are involved. Similar trends can found in Table 3 injecting AR codes into the first 14 layers produces the best dynamic degree and aesthetic quality. the DiT model, two approaches are applicd: adding ran- dom noise to the latent feature F and employing uncertainty sampling as discussed in Section 2.3 Both methods improve the dynamic motion in generated videos, with uncertainty sampling further enhancing aesthetic and image quality.
To further improve the robustness
breathtaking view of a rocky coactine. coastliee mosoic small boulders. deep water crashes agoinst the shore, creating a frothy white foom that controsts beautifuuly with the surrounding landscape."
AR Prediction
Baseline (a/AR)
Last 14 layers
First 3 layers
First 8 layers
First 14 layers
Figure 6: Effects of incorporating layout information provided by AR codes into different layers.
Table 3: Ablation study on the semantic injection module, encompassing its architecture and the number of DiT layers inserting this module; and 2) the training strategies, using coarse visual latent tokens latent VAE with a higher compression ratio. The compression ratio of the latent VQ- "Q-VAE for AR model is 4x8 x8). introducing Gaussian random noise to the latent features F and performing uncertainty sampling with F.
Compress Ratio in DiT
Fusion Aschitect
Number of Layers
Gaussian Noise
Uncertainty Sampling
Subject Consist
Background Consist
Motion Smooth
Dynamic Degree
Aesthetic Quality
Imaging Quality
Buseline OpenSora-1.
1.2
97.79
97.86
99.36
19.00
52.46
61.19
4x8x8
Adaptive Norm
14
X
94.71
96.22
99.07
42.00
53.91
58.50
ControlNet
14
X
X
95.81
96.58
99.10
46.00
55.18
62,89
MLP Adapter
14
X
97.99
97.97
99.31
21.00
56.40
64.74
Adaptive Norm
3
97.56
97.71
99.27
28.00
55.32
63.03
4x16x16
Adaptive Norm
97.55
99.21
55.55
65.09
Adaptive Norm
14
X 97.40
97.72
99.24
32.00
56.78
65 55.08
Adaptive Norm
14
X
97.04
97.35
99.20
34.00
54.98
64.21
Adaptive Norm
14
V
97.39
97.55
99.24
34,00
56,90
65.33
Aczehetc Oualkr
50
45
80
35
30
23
0ff
Figure 7: Aesthetic quality and imaging quality as a function denoising steps.
4.4 ANALYSIS
We conduct a detailed analysis of ARLON's inference efficiency compared to OpenSora- V1.2. The AR codes, which capture key spatial information, serve effective initialization, significantly accelerating the DiT model's denoising process. Figure illlstrrates the curves representing aes- thetic quality and imaging quality, function of the number of denoising steps. Notably, ARLON

---

Published as a conference paper at ICLR 2025
achieves the similar performance in just 5 10 steps, compared to the 30 steps required by the baseline. Taking the inference of 68 68-frame video on single A100 as an example, the OpenSora- V1.2 requires about 47 seconds for 30 steps of inference, while DiT model equipped with the semantic injection module takes approximately 11 to 19 seconds for 5 to 10 steps, with additional approximately 6 seconds for the AR codes inferring. This results in relative efficiency improve- ment of 47-64% over OpenSora-V1.2 V1.2. Tigure@provice the baseline SrenSora-V1.22 using? and denoising steps. The results clearly show that ARLON generates videos with significantly higher quality.
visual comparison between ARLON and
denoising step
5 denoising step
Baseline
Ours
Figure 8: Generated videos of ARLON and OpenSora- V1.2 with 3 and 5 denoising steps.
Streaming t2v
Freenobe
OpenSora
Ours
Promptt: majestic dormant volcano rivs in the canter a tranque landocape, witn ihs rupged viopes contraxing the bash forest below, vet against clear abin and ralling isll, evcking a smme of solitd and timeless aecerity" Prompt2: VAt enupting volcano donilaaes the scene, with fiery love and oof controsting the tush forest below, os dark cloude ond lightning fMt sky. coptwring the chons and power rature's fory."
+ firamsiiona to)
Figure 9: Qualitative comparisons of the videos generated using progressive prompts.
4.5 APPLICATIONS
One practical application of ARLON long video generation using progressive text prompts. The procedure is follows: assuming two text prompts, X, and X2, are given, the corresponding AR codes Q are first generated based on X,. Then, the last frame Q: is used as a condition, along with X2, to generate the corresponding AR code Q2. Finally, the DiT model utilizes (X,, Q and (X2, Q2) to generate the video. A qualitative result is presented in Figure Long videos generated by other models either remain unchanged after prompt transition change drastically. In contrast, our model transitions seamlessly, maintaining consistency throughout the entire video.
CONCLUSION
In this paper, we propose ARLON, novel framework that boosts diffusion Transformers with au- toregressive (AR) models for long (L.ON) video generation. Utilizing latent VQ- the input latent of the DIT model is compacted and highly quantized into discrete tokens for the training and inference the AR model. To integrate coarse spatial and temporal features into the DiT model, an adaptive norm based semantic injection module is proposed. To improve the robustness of ARLON, the DiT model is trained with coarser visual latent tokens incorporated with an uncertainty sam- pling module. Massive experiments demonstrate that ARLON model delivers ff-te-o-thhe-- art performance on text-based long video generation. Detailed analysis of the inference efficiency and qualitative results of progressive prompt based long video generation are also provided.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
Zongyi Li was supported by the Natural Science Foundation of China (62372203, 62302186) and the Major Scientific and Technological Project of Shenzhen (202316021).
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310.199) 19512, 2023a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7310-7320, 2024.
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-vide:: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840, 2023b.
Yiyang Chen, Zhedong Zheng, Wei Ji, Leigang Qu, and Tat-Seng Chua. Composed image retrieval with text feedback via muuli--griine uncertainty regularization. arXiv preprint arXiv:2211.0739. 2022.
Kaifeng Gao, Jiaxin Shi, Hanwang Zhang, Chunping Wang, and Jun Xiao. Vid-gpt: Introducing gpt-style autoregressive generation in video diffusion models. arXiv preprint arXiv:2406. 2024.
songwei Ge, Thomas Hayes, Harry Yang. Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and sen-ssenieivve transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Mancesh Agrawala, Dahua Lin, and Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint arXiv:2307 04725, 2023.
Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan. Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403. 147733 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022b.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205.15865 2022.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
11

---

Published as a conference paper at ICLR 2025
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06358, 2024.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14122 2023.
PKU-Yuan Lab and Tuzhan Al etc. Open-sora-plan. April 2024. doi: 10.5281/2enodo. 10948109. URL htttts::///oi..ooo 528111..111111 00..5/2/zzzzo.....
Yu Lu, Yuanzhi Liang, Linchao Zhu, and Yi Yang. Freelong: Training-free long video generation with spectralblend temporal attention. arXiv preprint arXiv:2407. 19918, 2024.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401 arX.v22401.03048, 2024.
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- Im: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.0233 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Bohao Peng, Jian Wang. Yuechen Zhang, Wenbo Li, Bing-Chang Yang, and Jiaya Jia. Controlnext: Powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.06070 2024.
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei
Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiv preprint arXiv:2310.15169, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-vin video: Text-to-vided generation without tixt-video data. arXiv preprint arXiv:2209. 14792 2022.
Ivan Skorokhodoy, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3626-3636, 2022.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo. and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 128:13 2024.
Ye Tian, Ling Yang. Haotian Yang, Yuan Gao, Yufan Deng, Jingmin Chen, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, al. Videotetris: Towards compositional text-to-video generation. arXiv preprint arXiv:2406.00 044277 2024.
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Heman Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference Learning Representations, 2022.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech 2023a.
synthesizers. arXiv preprint arXiv:2301.0211
12

---

Published as a conference paper at ICLR 2025
Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew. Hanshu Yan, et al. Magicvideo-v2: Multi-stage high-aesthetic video generation. arXiv preprint arXiv:2401 04468. 2024.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103 2023b.
Wenming Weng. Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jianmin Bao, Yuhui Yuan, et al. Art-v: Auto-regressive text-to-video generation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7395-7405. 2024.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104 10157, 2021.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072. 2024.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang. Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint arXiv:2303. 13346 2023.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion- tokenizer key to visual generation. arXiv preprint arXiv:2310.05737 2023.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Waen Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. arXiv preprint arXiv:2202.10571, 2022.
Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. Chronomagic-bench: A benchmark for metamorphic evaluation of text-to-time-lapse video generation. arXiv preprint arXiv:2406. 18522, 2024.
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8850-8860, 2024.
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to- video generation. arXiv preprint arXiv:2309. 15888 2023.
Zhenghao Zhang, Junchao Liao, Menghao Li, Long Qin, and Weizhi Wang. Tora: Trajectory- oriented diffusion transformer for video generation. arXiv preprint arXiv:2407.21705. 2024.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. March 2024. UtLp:ttps:///iii htwp://ww..ccm/pccciiccc/////aae
Sora
Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.1101 2022.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 PRELIMINARIES
In this section, we introduce the preliminaries of stable diffusion model, which operates the diffu- sion process in a latent space for computationally efficient. In this model, the image r is mapped into a compressed latent space= : = Ex() via a pre-trained auto-encoder, such as VQGAN or VQVAE. In the forward process, random noise is gradually added to the latent space, formulated as:
(Zz = = N (z1: va (1 - I),
(8)
where t E {1...., ,). T is the number of time steps during the diffusion process. and q (zs 1==1)) the noised 2; at l step given 2-1, and (1 a.) denotes the noise strength. Alternatively, we can formulated on from En as follows:
24 = Va + -I- 6 ~ NN.,
where 0g = II= 0g The denoising process involves learning reverse diffusion process to iter- atively remove the noise added during the forward process. This is achieved by training a neural network to predict the original latent representation from the noisy version by minimizing:
= || - co
(10)
where e denotes the conditional textual description, for the inference, random noise is sampled from Gaussian distribution, and DDIM is utilized for denoising latent representation, followed by a VAE decoder to reconstruct the image.
A.2 EXTENDED MODEL COMPARISONS
Table 4: Comparative Analysis of Model Parameters, Training Memory, Inference Speed, and Com- putational Efficiency for OpenSora- V1.2 and ARLON. The batch size for training both the AR and DiT models is set 2x68- frame 512x512 video clips, and the inference is evaluated by generating a 68-frame (68f) or 578-frame (578f) 512x512 video. Both the training and inference processes are executed on single 40G A100 GPU.
Method
OpenSora- V1.2 (baseline)
ARLON (ours)
Param. (AR)
192M
Param, (DiT)
1.2B
92M (trainable) 1.2B (frozen)
Param. (3D VAE)
384M
384M
Param. (latent VQ-VAE)
30M
Training Memory
36007M
7701M (AR); 36815M (DIT)
Inference Memory
24063M
2269M (AR): 25215M (DIT)
Inference speed (68f)
47.3
5.7s (AR)+18.9, (DiT)
Inference speed (5780
47.3 11
$7.2s (AR) (18.9 (DiT)
Inference FLOPss (681)
42626G x 30 (step)
200G (AR) 46461Gx 10 (step) (DIT)
Inference FL.PPs (578F)
42626G x 30 (step) 11 (times) 1547G (AR) + 46461Gx 10 (step) x 11 (times) (DIT)
We have conducted a comparative analysis of model parameters, training memory, inference speed, and computational efficiency for OpenSora-VIs V1.2 and ARLON, as shown in Table 4 From the sults in the table, we can observe that:
The increase in the number of parameters (192M 92M 30M) of ARLON is minimal com- pared to the 1.2B parameters of the baseline, OpenSora-V V1.2. This is primarily due to our adoption of an efficiency adapter approach during training, which enables us introduce fewer parameters while preserving performance. Additionally, the latent VQ-VAE operates within a compressed latent space, and the AR model training and generates highly quantized tokens, both of which signifi- cantly contribute to the minimized parameter requirements.
It is evident that our method does not require significant additional memory and computational resources compared to the baseline model. Specifically, during the training and inference phases, there are 2.2% and 4.8% relative increases respectively (the AR and DiT models can be trained in- dependently).
Conversely, by leveraging the AR code as an efficient initialization, our model is capable of gen- erating high-quality videos with significantly fewer steps. Consequently, inference time and
14

---

Published as a conference paper at ICLR 2025
total FLOPs superior to those of the baseline, thereby significantly accelerating the denoising process. Specifically, our model achieves a 48-49% relative improvement in inference speed and a 64% relative reduction in computational FLOPs for 68-frame or 578-frame video generation (578 can be expressed as 68 11 17 x 10. Here, 11 represents the number of times the DiT model gen- erates 68-frame video segments, while 17 signifies the number of frames in the conditioned video. Additionally, 10 indicates the number of times the DiT model generates videos under specific con- ditions).
A.3 EFFECT OF TRAINING DATA SIZE FOR DIT MODEL
Table 5: Comparative Analysis of Different Training Data Sizes.
Subject Consist
Background Consist
Motion Smooth
Dynamic Degrec
Aesthetic Quality
Imaging Quality
Models
Openvid- 1M
95.02
96.35
98.16
30.00
52.34
59.15
Op 0.4M
97.78
97.83
99.25
30.00
55.42
64.11
Openvid-HQ 0.4M+Mixkit 0.3M
97.39
97.55
99.24
34.00
56.90
65.33
In 5 we further illustrate the impact varying training data sizes. Firstly, when comparing our model's performance on the Open Vid IM dataset with that OpenVid-Hi (which contains higher quality videos, totaling 0.4M),, we observed marked improvement our model's performance on OpenVid-Ho This indicates that the quality of the data plays crucial role in the task of video generation. Furthermore, when we combined the Open Vid-HQ and Mixkit (the quality of videos also high) datasets our training set (approximately 0.7M), improvements in both quality and dynamic degree are obtained. This suggests that in the context video generation, prioritizing high-quality videos while also utilizing larger dataset can effectively enhance the overall quality of generated videos.
A.4 LIMITATIONS
Although ARLON achieves state-of-the-art performance long video generation, it also exhibits some specific constraints. First, ARLON is built upon OpenSora-V1.2. which potentially caps the upper limit of video quality. Nonetheless, this limitation can be mitigated by substituting the DiT model with more advanced alternatives, such as CogVideoX train ARLON at 2K resolution, the sequence length of AR codes will become excessively long, mak- ing both training and inference impractical. Viable solutions involve employing higher compres- sion ratio in VQ-VAE, or selectively retaining essential information while disregarding irrelevant details. Additionally, for the AR model, parallel prediction emerges as an alternative approach.
MovieGen. Second, if we aim to
We also have presented some failure cases, as shown in Figure [6] For example, generating hand movements such as applying makeup or cating is challenging for creating realistic videos that con- form to the physical world, especially regarding hand details. Our future research endeavors will delve into addressing these issues.
A.5 BROADER IMPACT
Synthetic video generation is powerful technology that be misused create fake videos videos containing harmful and troublesome content, hence is important to limit and safely deploy these models. From a safety perspective, open-sourced, and we do not add any new restrictions nor relax any existing ones to OpenSora- V1.2. If you suspect that ARLON is being used in manner that is abusive illegal or infringes on your rights or the rights of other people, you can report it to us.
emphasize that the training data of ARLON are all
15

---

Published as a conference paper at ICLR 2025
A.6 EXTRA EXAMPLES
happy Congl playing pork, sunset
ControINet
Ours
concerned expression is seen dimil wearing white turtleneck. lighting subdued, highlighting the womon's face and the texture of her clathing. The overall style of the video dromatk and moody, with focus on the womon's emotional state.
ControlNet
Ours
Figure 10: Ablation study on different semantic injection approaches. Although ControlNet has higher dynamism, it also produces more distortions, such as the severe deformation of the dog in the left video and the facial distortions in the right video.
Baeelie
Ours 3tepp3
Baveline Stepps
Durs Stegpms
Figure 11: More generated videos of ARLON and OpenSora-Va V1.2 with and 5 denoising steps, as a complement Figure
16

---

Published as a conference paper at ICLR 2025
detailed desse up of wibrant underwater caral reaf, teeming with colorfel fish swimming gracefulty. The scone it with notural. suodued ligh, highlighling
the deep see ambionce. Gentle woves riple through, brigging small bubbier into view, opoinst backdhop that fodes into deep blue depths."
OpenSora
Ours
happy Corgi playnn
OpenSora
Ours
Figure 12: Examples of text videos generation of ARLON and OpenSora- V1.2.
"A high-quolity 3D render of femole character with curly blonde hair and striking blue eyes, weoring block tank top, standing before dramatic, backdrop, expression
lighting."
OpenSora
Ours
tranquil coastal beach in spring with golden sands and lush green cliffs suddenly disrupted by intence shaking, turning thythmic chootic, experience"
OpenSora
Ours
Figure 13: More text to videos generation of ARLON and OpenSora- V1.2.
17

---

Published as a conference paper at ICLR 2025
Streaming t2v
Freenoise
OpenSora
Ours
colors ranging from deep purples bright aranges. of tropical fish, including ongelfish, clownfish, and tongs, dort gracefully through the water, their scales shimmering in the filtered sunlight"
mesmerizing underwater world, vibront coral reefs
Frame
Frame 100
Frame 200
Frame 300
Frame 400
Frame 500
Frame 600
Figure 14: Examples of text to long video generation (600 frames) using FreeNoise, StreamingT2V,
OpenSora-Ve V1.2,, and ARLON.
Reference Video
Generated Video1
"A bearded mon with short hair, wearing sunglasses black leather jocket with red lining, looks upwards seriously in a close-up portroit against o green-to-blue gradient background, suggesting nighttime."
Generated Video2
"A high-quality 3D render of female charocter with curly bionde hair and striking blue eyes, wearing black tank standing before dramatic, fiery backdrop, with serious expression and detailed, realistic lighting."
Figure 15: Examples Video Condition Generation: We extract the coarse latent from reference
videos and replace it with our semantic latent, which is then injected into the diffusion process. The generated video maintains the same pose and position as the reference video while ensuring consistency with the provided textural prompts.
18

---

Published as a conference paper at ICLR 2025
young woman with flawless skin and a serene expression sits o vonity, bathed saft morning light.
Video1
A fluffy orange tabby cat with white paws and bushy toil sits on a polished wooden floor, eagerly eating from ceramic bowl decorated with fish patterns.
Video2
Figure 16: Examples failure cases.
19


Reviews:
[{"review_id": "5ZgexS6TU6", "reviewer": "Reviewer_N9At", "summary": "The manuscript introduces ARLON, a text-to-video framework that efficiently generates high-quality, dynamic, and temporally consistent long videos. By combining Autoregressive models with Diffusion Transformers, ARLON employs innovations like VQ-VAE for token compression, an adaptive semantic injection module, and an uncertainty sampling module to enhance efficiency and noise tolerance. It reduces denoising steps and outperforms OpenSora-V1.2 in both quality and speed, achieving state-of-the-art performance.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The integration of Autoregressive models with Diffusion Transformers and innovations like VQ-VAE for token compression, adaptive semantic injection, and uncertainty sampling show originality in addressing long video generation challenges.\n2. The generated video spans 600 frames, making it relatively long.", "weaknesses": "1. In Table 2, why does StreamingT2V have a higher Dynamic Degree score (85.64) compared to ARLON (50.42)?\n2. The paper lacks a detailed comparison of the number of parameters in ARLON versus baseline models.\n3. There is no analysis of ARLON's memory footprint during training and inference, which would clarify its computational efficiency relative to models like OpenSora-V1.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kYZpYwG7PU", "reviewer": "Reviewer_Rwnx", "summary": "This paper proposes a long video synthesis pipeline, ARLON. The main idea of this paper is to combine DiT with autoregressive transformers that provide long-range temporal information. To bridge the DiT and the AR transformer, the pipeline novelly adopts 1) a Latent VQ-VAE to enable the AR model to learn on and the DiT to learn on different latent spaces, reducing learning complexity and allowing the AR model to manage coarse temporal dependencies; 2) an adaptive norm-based semantic injection module to guide the DiT using AR generated tokens. Novel training strategies of coarser visual latent tokens for DiT and uncertainty sampling are also proposed to make the training process more stable and generalizable. \n\nResults are compared with current t2v models over VBench and Vbench-long and achieve notable improvements especially on long video generation.", "questions": "- How many seconds can the models generate for the longest videos and how is the performance? For How many seconds do the longest videos that your method generates can last? In my understanding, this is the key advantage of the hierarchical generation framework.\n- The main limitation of this work seems to be the huge computational cost of training, but the related information (type and number of GPU, training time) is not provided. It would be nice to know this information.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation for using the AR model to provide semantic guidance is clear and nice.\n- It is a very good extension of existing architectures.\n- Good presentation.\n- Good qualitative and quantitive results.", "weaknesses": "I overall like this paper, but there are several points for improvement.\n\n- No ablation on the impact of model structure and training data size.\n- No discussion on failure cases and limitations.\n- There might be some missing references such as nuwa-XL and Phenaki. GAN-based long video generation might also be related.\n\n*[1] NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation*\n\n*[2] Phenaki: Variable Length Video Generation from Open Domain Textual Descriptions*\n\nI am not an expert in this field of training large video generation models. I will adjust the final score with other reviewers' comments and also based on the response from the author.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QbwjmS8VJV", "reviewer": "Reviewer_gQNT", "summary": "This paper proposes to leverage autoregressive models to guide the training of diffusion transformers for text-to-video generation task. The proposed framework incorporate with a latent VQ-VAE, coarser visual latent tokens and a uncertainty sampling module to connect DiT and AR models and inject the information from AR models to DiT training. Massive experiments are conducted with abundant quantitative metrics and visualizations are reported to demonstrate the performance of the proposed model.", "questions": "- What is the motivation of using a latent VQ-VAE nested inside a pretrained 3D Autoencoder, instead of training a single-stage pixel-to-latent tokenizer? And what is the additional computational cost or gain in comparison?\n\n- The proposed coarse latent token with different compression ratio, while how is this ablated in Table 3 or any other ablation studies? (Does the 4×8×8 row refers to both the same scale training and the 4×16×16 rows refer to different scale training?)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea of make DiT and AR models working in one latent space is novel, and many technical improvements are designed to bridge their gap.\n\n- The proposed method reaches a large reduction of denoising step of comparable generation quality.", "weaknesses": "- It's better to use bold fonts and underscores in Table 1. According to the listed numbers, the proposed method doesn't reach top 3 in many columns. And the reproduced notation is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEYgX3qHIn", "reviewer": "Reviewer_45zF", "summary": "This paper proposes a new text-to-video(T2V) framework consisting of autoregressive (AR) Transformers and Diffusion Transformers (DiT). Based on the input text prompt, the AR model predicts quantized visual tokens of a latent VQ-VAE nested within the 3D VAE of the DiT model. The coarse latent, reconstructed from the predicted tokens, serves as semantic condition to guide the DiT through adaptive normalization for video generation. To mitigate the effect of error introduced from AR inference, the authors introduce two noise-resilient strategies during DiT training, using coarser latent tokens and uncertainty sampling to make the semantic condition noisier.", "questions": "1. In Figure1, the previously generated video seems to be used as a reference for the subsequent generation, but this is not illustrated in Figure 2. Does the DiT generate videos in an autoregressive way?\n2. In Table 1 and Table 2, do the higher scores of metrics indicate better performance?\n3. For long video generation in section 4.2 as well as the demo videos on the webpage, the authors only compare their proposed method with open-source text-to-long video generation models. Why not compare with the commercial closed-source text-to-video generation models like in Table 1?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper innovatively combines the strengths of autoregressive (AR) Transformers and Diffusion Transformers (DiT) for generating long video with rich dynamic motion.\n2. To mitigate the effect of error introduced from AR inference to DiT, the authors introduce two noise-resilient strategies during DiT training.\n3. The paper is written and presented clearly and easy to follow.\n4. The long video results of the proposed method show improvement on dynamic degree, and long video generation results using progressive text prompts are more consistent throughout the entire video.", "weaknesses": "1. From Table 1, the proposed method lags behind compared methods in many metrics other than dynamic degree, such as Imaging Quality and Subject Consistency. In Table 2, the dynamic degree of proposed method (50.42) is significantly lower than that of the StreamingT2V(85.64). \n2. From the demo videos on the webpage, there is some room for improvement for the proposed method compared to others. For example, the result of \"A teddy bear is swimming in the ocean.\" lacks of subject consistency, and its motion is not realistic, which may be consistent with the quantitative results in Table 1 and Table 2.\n3. Although the authors introduce noise-resilient strategies for the DiT model training to mitigate the error issue from AR inference, I am concerned that these strategies cannot truly simulate the error of AR inference, which may limit the model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)