Pyramidal Flow Matching for Efficient Video Generative Modeling
Authors: Yang Jin
Zhicheng Sun
Ningyuan Li
Kun Xu
Kun Xu
Hao Jiang
Nan Zhuang
Quzhe Huang
Yang Song
Yadong MU
Zhouchen Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Model
Flow Matching
Video Generation
Primary area: generative models

Abstract: Video generation requires modeling a vast spatiotemporal space, which demands significant computational resources and data usage. To reduce the complexity, the prevailing approaches employ a cascaded architecture to avoid direct training with full resolution latent. Despite reducing computational demands, the separate optimization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video generative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, we craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models are open-sourced at https://pyramid-flow.github.io.

Published as a conference paper at ICLR 2025
PYRAMIDAL FLOW MATCHING FOR EFFICIENT VIDEO GENERATIVE MODELING
Yang Jin', Zhicheng Sun', Ningyuan Li Kun Xu, Kun Xu², Hao Jiang', Nan Zhuang", Quzhe Huang, Yang Song, Yadong Mu? Zhouchen 1.56+
'Peking University. "Kuaishou Technology. Beijing University Posts and Telecommunication. "State Key Lab of General AI, School of Intelligence Science and Technology, Peking University, Institute for Artificial Intelligence. Peking University,
Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China
ABSTRACT
Video generation requires modeling a vast spatiotemporal space, which demands significant computational resources and data usage. To reduce the complexity, the prevailing approaches employ a cascaded architecture to avoid direct training with full resolution latent. Despite reducing computational demands, the separate op- timization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video gener- ative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. be optimized in end-to-end manner and with sin-
The entire framework gle unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos 768p resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models are open-sourced at https: ptttps://pyrrai... www.rrrammdffl.ggi.i....m flw.ggitha..io
INTRODUCTION
Video is media form that records the evolvement of the physical world. Teaching the AI system to generate various video content plays a vital role in simulating the real-world dynamics (Hu et al.)
2023 Brooks et al. 2024) and interacting with humans (Bruce et al.] 20244 Valevski et al. 2024). Nowadays, the cutting-edge diffusion models (Ho et al., 2022c Blatmann et al. [023a OpenAl
2024) and autoregressive models (Yan et al., 2021 Hong et al. 2023 Kondratyuk et al. (024) have
made remarkable breakthroughs in generating realistic and long-duration video through scaling of data and computation. However, the necessity of modeling significantly large spatiotemporal space makes the training of such video generative models computationally and data intensive.
To ease the computational burden of generating high-dimensional video data, a crucial component compress the original video pixels into a lower-dimensional latent space using VAE (Kingmal & Welling 2014 Esser et al. 2021; Rombach et al. 2022). However, the regular compression rate (typically 8 3 still results in excessive tokens, especially for high-resolution samples. In light of
this, prevalent approaches utilize a cascaded architecture (Ho et al. 20226 Pernias et al., 2024]
Teng et al., 2024) to break down the high-resolution generation process into multiple stages, where samples are first created in a highly compressed latent space and then successively upsampled us- ing additional super-resolution models. Although the cascaded pipeline avoids directly learning at high resolution and reduces the computational demands, the requirement for employing distinct models at different resolutions separately sacrifices flexibility and scalability. Besides, the separate optimization of multiple sub-models also hinders the sharing of their acquired knowledge.
This work presents an efficient video generative modeling framework that transcends the limitations of the previous cascaded approaches. Our motivation stems from the observation in Fig. La that the initial timesteps in diffusion models are quite noisy and uninformative. This suggests that operating at full resolution throughout the entire generation trajectory may not be necessary. To this end, reinterpret the original generation trajectory as a series of pyramid stages that operate on compressed
Yadong Mu and Zhouchen Lin are corresponding authors.

---

Published as a conference paper at ICLR 2025
Frame index:
==0
1=1
T=T
1=0 1=1 L=T
Noise
(a) Video diffusion model like Sora (OpenAl 2024
(b) Our proposed pyramidal flow matching
Figure 1: A motivating example for pyramidal flow matching: (a) Existing diffusion models operate at full resolution, spending a lot of computation on very noisy latents. (b) Our method harnesses the flexibility of flow matching interpolate between latents of different resolutions. This allows for simultaneous generation and decompression of visual content with better computational efficiency. Note that the black arrows are denoising trajectories, and the blue ones are their temporal conditions.
representations of different scales. Notably, the efficacy of image pyramids (Adelson et al. 1984)
has been widely validated for discriminative neural networks ([.in et al., 2017: Wang et al. 2020) and more recently for diffusion models (Ho al. [022b] Pernias et al. 2024 Teng et al. 2024 and multimodal LLMs (Yu et al., 2023 Tian et al. (2024). Here, we investigate two types of pyramids: the spatial pyramid within a frame and the temporal one between consecutive frames (as illustrated in Fig. Lb). In such pyramidal generation trajectory, only the final stage operates at full resolution, drastically reducing redundant computations in earlier timesteps. The main advantages are twofold: (1) The generation trajectories of different pyramid stages interlinked, with the subsequent stage continuing to generate from the previous ones. This eliminates the need for each stage to regenerate from pure noise in some cascade models. (2) Instead of relying on separate models for each image pyramid, we integrate them into a single unified model for end-to-end optimization, which admits drastically-exppeitee training with more elegant implementation as validated by experiments.
Based on the aforementioned pyramidal representations, we introduce novel pyramidal flow match- ing algorithm that builds upon recent prevalent flow matching framework (Lipman et al., 2023 Liu et al. 2023; Albergo & Vanden- Vanden-Eijnden 2023). Specifically, we devise a piecewise flow for each pyramid resolution, which together form a generative process from noise to data. The flow within each pyramid stage takes a similar formulation, interpolating between a pixelated (compressed) and noisier latent and a pixelate-free (decompressed) and cleaner latent. Through our design, they can be jointly optimized by the unified flow matching objective in single Diffusion Transformer (DiT) (Peebles & Xic 2023). allowing simultancous generation and decompression of visual content without multiple separate models. During inference, the output of each stage is renoised by cor- rective Gaussian noise that maintains the continuity of the probability path between stages. Further- more, we formulate the video generation in an autoregressive manner, iteratively predicting the next video latent conditioned on the generated history. Given the high redundancy in the full-resolution history, we curate temporal pyramid sequence using progressively compressed, lower-resolution history as conditions, further reducing the number of tokens and improving training efficiency.
The collaboration of the spatial and temporal pyramids results in remarkable training efficiency for video generation. Compared to the commonly used full-sequence diffusion, our method significantly reduces the number of video tokens during training (e.g., 155360 tokens versus 1 119,040 tokens for a 10-second, 241- 141-frame video), thereby reducing both computational resources required and training time. By training only on open-source datasets, our model generate high-quality 10-second videos at 768p resolution and 24 fps. The core contributions of this paper are summarized as follows:
We present pyramidal flow matching, novel video generative modeling algorithm that incorpo- rates both spatial and temporal pyramid representations. Utilizing this framework can significantly improve training efficiency while maintaining good video generation quality.
The proposed unified flow matching objective facilitates joint training of pyramid stages in single Diffusion Transformer (DIT), avoiding the separate optimization of multiple models. The support for end-to-end training further enhances its simplicity and scalability
We evaluate its effectiveness on VBench (Huang et al. 2024) and EvalCrafter (Liu et al. 2024), with highly competitive performance among video generative models trained on public datasets.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Video Generative Models have seen rapid progress with autoregressive models Yann et al.
2021
Hong al. [2023 Kondratyuk et al. 2024: Jin al. 2024) and difffusion models (Ho et al.
2022c
Blattmann et al. 2023b a). A notable breakthrough is the high-fidelity video diffusion models (Ope-
nAl n 2024 Kuaishou 2024 Luma 2024 Runway 2024) by scaling up DiT pre- pre-training (Peebles &
Xie 2023). but they induce significant training costs for long videos. An alternative line of research
integrates diffusion models with autoregressive modeling (Chen et al.] 2024a] Valevski et al.) 2024)
to natively support long video generation, but is still limited in context length and training efficiency. Our work advances both approaches terms of efficiency from compression perspective, featuring spatially compressed pyramidal flow and temporally compressed pyramidal history.
Image Pyramids (Adelson et al.
1984) have been studied extensively visual representation learn- 2005 Lin et al. 2017 Wang et al. 2020). For generative models,
ing (Lowe 2004 Dalal & Triggs
the idea is explored by cascaded diffusion models that first generate at low resolution and then per-
form super-resolution (Ho et al., [202bb] Saharia al. 2022; Zhang et al.. 2023b Gu et al. [023]
Pernias et al. 2024 Teng et al. 2024), and later extended to video (Ho et al. 2022a Singer et al.
2023). However, they require training several separate models, which prevents knowledge sharing.
Possible unified modeling solutions for pyramids include hierarchical architectures (Rombach et al.
2022 Crowson et 2024 Hatamizadeh et 2024) or via next-token prediction (Yu 2023
Tian et al.] 2024). but involve architectural changes. Instead, we propose a simple flow matching objective that allows joint training of pyramids, thus facilitating efficient video generative modeling.
METHOD
This work proposes an efficient video generative modeling scheme named pyramidal flow matching. In the following text, we first extend the flow matching algorithm (Section 3.1 to an efficient spatial pyramid representation (Section 3.2). Then, a temporal pyramid design is proposed in Section 3.3 to
further improve training efficiency. Lastly, practical implementations are discussed in Section 3.4
3.1 PRELIMINARIES ON FLOW MATCHING
Similar to diffusion models (Sohl-Dickstein et al. 2015; Song & Ermon 2019; Ho et al. 2020),
flow generative models (Papamakarios et al. 2021 Song al.] 2021 Xu et al. [022] 2 [ Lipman al., 2023 Liu et al. 2023: Alhergo & Ganden-Hijnder 2023) aim to learn a velocity field v. that maps
random noise Tp N NO,01 I)
data samples Φ q via an ordinary differential equation (ODE):
dx;
U2 (w.).
dt
Recently, Lipman et al. (2023) y: Liu et al.
(023 Albergo & Vanden- BaindenEEijnenn- (2023) proposed the
flow matching framework, which provides simple simulation-free training objective for flow gen- erative models by directly regressing the velocity v; on a conditional vector filed u. (.(()a)):
Eg.q(ae)) ) h. (ae)) (xx)) -
(2)
where u (-12) uniquely determines a conditional probability path pr ( (-121) toward data sample æ An effective choice of the conditional probability path is linear interpolation of data and noise:
t + (1 t)æo,
x; N (t (1 ()-))
and (< Zg. Notably, flow matching can be flexibly extended to interpolate between distributions other than standard Gaussians. This enables us devise a new flow matching algorithm that specializes in reducing the computational cost of video generative modeling.
3.2 PYRAMIDAL. FL.WW MATCHING
The main challenge in video generative modeling is the spatio-temporal complexity, and we address its spatial complexity first. According previous key observation in Fig. [ the initial generation
3

---

Published as a conference paper at ICLR 2025
Renoise
k-th stage
8su
n' (0,2))
Reyas
Noise
Upsample
k+1-th
8
(a) Unified modeling of pyramidal flow
(b) Handling jump points via renoising
Figure 2: Illustration of spatial pyramid. (a) The pyramidal flow is divided into multiple stages, each from pixelated and noisy starting point to pixelate-free freee and cleaner result. (b) During inference, we add a corrective noise at jump points across stages to ensure continuity of the proabability path.
steps are usually very noisy and less informative, and thus may not need to operate at full resolution latent. This motivates to study spatially compressed pyramidal flow, illustrated in Fig. 2
To alleviate redundant computation in early steps, we interpolate flow between data and compressed low-resolution noise. Let @b denote the interpolation between latents of different resolutions, and let there be K resolutions, each halving the previous one, then our flow may be expressed as:
tx (1 t1 Down(xn, 2k).
where (own(.,-) a downsampling function. Since the interpolation concerns varying-dimensional we decompose it as a piecewise flow (Yan et al. 2024) that divides [0. 1] into K time windows, where each window interpolates between successive resolutions. For the k-th time window [St. ex), let I' = (t sk)/ (ek 8k) denote the rescaled timestep, then the flow within it follows:
c t' Down( 2*) + (1 - t') Up( Down( 2
(6)
where Up(-) is an upsampling function. This way, only the last stage is performed at full resolution, while most stages are performed at lower resolutions using less computation. Under uniform stage partitioning, the idea of spatial pyramid reduces the computational cost to factor of nearly 1/K. Below, we describe the instantiation of pyramidal flow from training and inference, respectively.
3.2. UNIFIED TRAINING
In the construction of pyramidal flow, our main concern is unified modeling of different stages, as previous works (Ho et al. 2022b; Pernias et al., 2024 Teng et al. (024) all require training multiple models for separate generation and super- resolutin,, which hinders knowledge sharing.
To unify the objectives of generation and decompresion/spperrsslttui path by interpolating between different noise levels and resolutions. It starts with more noisy and pixelated latent upsampled from a lower resolution, and yields cleaner and fine ane-ggained results at a higher resolution, illustrated in Fig. 2al Formally, the conditional probability path is defined by:
we curate the probability
End:
Dey |m] ~ N(ex Down(x:, 2*), )1 -xx)²² 2I)
Start:
E2, ~ Nssx
2 (1 (1-8))21 I
(8)
where Sy Ck. and the upsampling and downsampling functions for the clean æ; are well defined, the flow trajectory,
e.g., by nearest or bilinear resampling. In addition, to enhance the straightness couple the sampling of its endpoints by enforcing the noise to be in the same direction. Namely, we first sample noise n ~ A/ I) and then jointly compute the endpoints (iog. S..)) as:
End:
Sea = Ex (ow 2*) + (1 -1--6 ex)n,
(9)
Start:
T = 5k (Doun (x1, 2*+))) + -1-38 (1-sk)n.
(10)
Thereafter, we can regress the flow model us on the conditional vector field u, ( = de, with the following flow matching objective to unify generation and decompression:
E &t,,)) (.t) (wt., ?
(11)
4

---

Published as a conference paper at ICLR 2025
Algorithm Sampling with Pyramidal Flow Matching
Require: flow model v, number of stages K, time windows ex]- Initialize starting point @n ~ (0,, I).
for k < K 1 to 0 do
Compute endpoint teg, from starting point te. based on the flow model v. Compute next starting point by upsampling dea, with renoising.
> Eq.
15]
Ensure: generated sample é1
3.2.2 INFERENCE WITH RENOISING
During inference, standard sampling algorithms can be applied within each pyramid stage. However, we must carefully handle the jump points (Campbell et al. 2023) between successive pyramid stages of different resolutions to ensure continuity of the probability path.
To ensure continuity, we first upsample the previous low lowrresouution endpoint with nearest or bilinear resampling. The result, as linear combination of the input, follows Gaussian distribution:
(p( | M(e++1 Up(Downl (z1, 2*)+)),, (
1x+1) Σ),
(12)
where E is a covariance matrix depending on the upsampling function. Comparing Eqs. (8) and (
we find it possible to match the Gaussian distributions at each jump point by a linear transformation of the upsampled result. Specifically, the following rescaling and renoising scheme would suffice:
8k
Up((ee4.1) + on', s.t. n' NO,
(13)
ck+1
where the rescaling coefficient su s*/Ex+1 allows matching the means of these distributions, and the corrective noise n' with a weight of allows matching their covariance matrices.
To derive the corrective noise and its covariance, we consider a simplest scenario with nearest neigh- bor upsampling. In this case, Σ has a blockwise structure with non-zer elements only in the 4 x 4 blocks along the diagonal (corresponding to those upsampled from the same pixel). Then, can be inferred that the corrective noise's covariance matrix E' also has a blockwise structure:
7
Σ block
1
1
1
block
7
7
7
(14)
1
7
7
7
where Block contains negative elements ? E [-1/3, 0 reduce the correlation within each block, as illustrated in Fig. 26 Since it desirable to maximally preserve the signals at each jump point, we opt add small amount of noise with y 11/3 such that it is most specialized for decorrelation. Substituting this into the above gives the update rule at jump points (see Appendix A for derivations):
+ Up($t...)+
v3(1 88))
n'
(15)
2
2
with ck41 2s1/(11 sk). The resulting inference process with renoising shown in Algorithm
!
3.3 PYRAMIDAL TEMPORAL CONDITION
Beyond the spatial complexity addressed in above sections, video presents another significant chal- lenge due to its temporal length. The prevailing full-sequence diffusion methods generate all video frames simultaneously, restricting them to fixed-length generation (consistent with training). In contrast, the autoregressive video generation paradigm supports flexible-length generation during inference. Recent advancements (Chen et al. 2024a Valevski et al., 2024) have also demonstrated its effectiveness in creating long-duration video content. However, their training still severely limited by the computational complexity arising from the full-resolution long-history condition.
We observe that there is a high redundancy in full-resolution history conditions. For example, earlier frames in video tend provide high-level semantic conditions and less related to appearance
The lower bound 1/3 ensures that the covariance matrix is semidefinite.

---

Published as a conference paper at ICLR 2025
Frame indox:
i-2 2
1-3
i-2
Extrapolation
Noise
History condition
Prediction
Interpolation
(a) Temporal pyramids rearranged in rows
(b) Position encoding in spatial and temporal pyramid
Figure 3: Illustration of temporal pyramid. (a) At each pyramid stage, the generation is conditioned on a compressed, lower-resolution history to improve training efficiency of the autoregressive model, as indicated by the rows. (b) A compatible position encoding scheme devised that extrapolates in the spatial pyramid but interpolates in the temporal pyramid to allow spatial alignment of conditions.
details. This motivates us to use compressed, lower-resolution history for autoregressive video gen- eration. As shown in Fig. 3al adopt history condition of gradually increasing resolutions:
Down(w)) (xt) 2
2,2t+))
Down (a) 2
(16)
Histury condition
Training
where the superscripts are the history latent index, and the subscript 1' indicates small noise added to history latents training mitigate errоr accumulation with autoregressive generation, as in (Chen et al. 2024a Valevski et al. 20224) After training, we use clean generated frames for inference:
2 Doun(x 2++) + Down( i-1 2,2
(17)
History condition
Prediction
The above design significantly reduces the computational and memory overhead of video generative pre-training. Let there be 7 history latents over K lower resolutions, then most frames are computed at the lowest resolution of 1/2". which reduces the number of training tokens by up to 1/4 times. As result, training efficiency is improved by up 16* times.
3.4 PRACTICAL IMPLEMENTATION
In this section, we show that the above pyramid designs can be easily implemented using standard Transformer architecture (Vaswani al. 2017) and pipelines. This is crucial for efficient and scal- able video generative pre-training based on existing acceleration frameworks.
Unlike previous methods (Ma al. 2024) that utilize factorized spatial and temporal attention to reduce computational complexity, we directly employ full sequence attention, thanks to much fewer tokens required by our pyramidal representation. Furthermore, blockwise causal attention adopted in each transformer layer, ensuring that each token cannot attend to its subsequent frames. The abla- tion results in Appendix C.2 illustrate that such casual attention design is crucial for autoregressive video generation. Another important design choice is the position encoding, as the pyramid designs introduce multiple spatial resolutions. As shown in Fig. 3b we extrapolate position encoding in the spatial pyramid for better fine-grained detail (Yang et al.) 2024), while interpolating it in the temporal pyramid input to spatially align the history conditions.
During training, different pyramidal stages are uniformly sampled in each update iteration. The autoregressive nature of our method inherently supports joint training images and videos, since the first frame in video acts as an image. We pack training samples with varying token counts together to form the length-balanced balanced training batch following Patch n' Pack (Dehghani et al. 2023). After training,
method natively possesses the capability of text-to-video and text-conditioned image-to-ve -ovideo generation. During inference sampling, the classifier-fi free guidance strategy can be employed to enhance temporal consistency and motion smoothness of the generated video.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETTINGS
Training Dataset. Our model is trained mixed corpus of open-source image and video datasets. For images, we utilize a high-aesthetic subset of LAIÓN-5B (Schuhmann et al., 2022). 1 11M from
CC-12M (Changpinyo al. 2021), 6.9M non-blurred subset of 1B (Kirillov et al. (023), 4.4M
from JourneyDB (Sun et al. 2023), and 1 14M publicly available synthetic data. For video data, we
incorporate the Web IOM (Bain al. 2021), Open (Nan al. 2024). and another IM
high-resolution non watermarkk video primarily from the Open-Sora Plan (PKU-Yuan Lab et al. 2024 After postprocessing. around 10M single-shot videos are available for training.
Evaluation Metries. We utilize the VBench (Huang et al. 2024) and EvalCrafter (Liu et al. 2024 for quantitative performance evaluation. VBench is a comprehensive benehmark that includes 16 fine-grained dimensions to systematically measure both motion quality and semantic alignment of video generative models. EvalCrafter is another large-scale evaluation benchmark including around 17 objective metrics for assessing video generation capabilities. In addition to automated evaluation metries, we also conducted a study with human participants to measure the human preference for our generated videos. The compared baselines are summarized in Appendix B
Implementation Details. We utilize the prevailing MM-DiT architecture from SD3 Medium (Esser ct al., 2024) the base model, with 2B parameters in total. It employs sinusoidal position encod- ing Vaswani et al. 2017) in the spatial dimensions. As for the temporal dimension, the ID Rotary Position Embedding (RoPE) (Su et al. 2024) is added support flexible training with different video durations. In addition, we use a 3D Variational Autoencoder (VAE) to compress videos both spatially and temporally with a downsampling ratio of 8 x 8. It shares a similar structure with MAGVIT-v2 (Yu et al. (202)) and is trained from scratch on the Web dataset (Bain et al.) 2021 The number of pyramid stages set in all the experiments. Following Valevski et al. 2024 we add some corruptive noise of strength uniformly sampled from [0, 1/3] to the history pyramid conditions, which is critical for mitigating the autoregressive generation degradation.
4.2
EFFICIENCY
The proposed pyramidal flow matching framework significantly reduces the computational and memory overhead in video generation training. Consider a video with T frame latents. where each frame contains N tokens at the original resolution. The full-sequence diffusion has TN input tokens in DiT and requires T2 computations. In contrast, our method uses only approximately TN/4K tokens and 2 N2 /16 computations even for the final pyramid stage, which significantly improves the training efficiency. Specifically, it takes only 20.7k A100 GPU hours to train a 10s video genera- tion model with 241 frames. Compared to existing models that require significant training resources, our method achieves superior video generation performance with much fewer computations. For ex- ample, the Open-Sora 1.2 (Zheng et al., 2024) requires 4.8k Ascend and 7k H100 hours to train the generation of only 97 video frames, consuming more than two times the computation of our approach, yet producing videos of worse quality. At inference, model takes just 56 seconds to create a 5-second, 384p video clip, which is comparable to full-sequence diffusion counterparts.
4.3 MAIN RESULTS
Text-to- Video Generation. We first evaluate the text-to-video generation capability of the proposed method. For each text prompt, a 5-second 121 frames video is generated for evaluation. The de- tailed quantitative results on VBench (Huang al.) 2024) and EvalCrafter (Liu al., 2024 are summarized in Tables I and respectively. Overall. our method surpasses all the compared open- sourced video generation baselines these two benchmarks. Even with only publicly accessible video data in training, it achieves comparable performance to commercial competitors trained on much larger proprictary data like Kling (Kuaishou) 2024) and Gen-3 Alpha (Runway 2024p. In par- ticular, we demonstrated exceptional performance in quality score (84.74 vs. 84.11 of Gen-3), and motion smoothness in VBench, which are crucial criteria in reflecting the visual quality of generated videos. When evaluated in EvalCrafter, our method achieves better visual and motion quality scores than most compared methods. The semantic score is relatively lower than others, mainly because we coarse-grained synthetic captions, which can be improved with more accurate video captioning.
7

---

Published as a conference paper at ICLR 2025
Table 1:
Experimental results VBench (Huang al. 2024). In terms of total score and quality score, our model even outperforms CogVideoX-5B (ang et al., 2024) with twice the model size. In the following tables, we use blue to denote the highest scores among models trained on public data.
Public Data
Total Score
Quality Score
Semantic Score
Motion
Dynamic Degree
Model
Smoothness
Gen-2
X
80.58
82.47
73.03
99.58
18.89
Pika 1.0
X
80.69
82.92
71.77
99.50
47.50
Cog VideoX-2B
X
80.91
82.18
75.83
97.73
59.86
Cog VideoX-5B
X
81.61
82.75
77.04
96.92
70.97
Kling
X
81.85
83.38
75.68
99.40
46.94
Gen-3 Alpha
X
82.32
84.11
75.17
99.23
60.14
Open-Sora Plan v1.3
77.23
80.14
65.62
99.05
30.28
Open-Sora 1.2
79.76
81.35
73.39
98.50
42.39
VideoCrafter2
80.44
82.20
73.42
97.73
42.50
T2V-Turbo
81.01
82.57
74.76
97.34
49.17
Ours
81.72
84.74
69.62
99.12
64.63
Table 2: Experimental results EvalCrafter
Liu et al.
2024
See Appendix
C.1
for raw metrics.
Public Data
Final Sum
Visual Quality
Text- Video
Motion Quality
Temporal Consistency
Model
Score
Alignment
Pika 1.0
X
250
63.05
66.97
56.43
63.81
Gen-2
X
254
69.09
63.92
55.59
65.40
ModelScope
218
53.09
54.46
52.47
57.80
Show-1
V
229
52.19
62.07
53.74
60.83
LaVie
234
57.99
68.49
52.83
54.23
VideoCrafter2
243
63.98
63.16
54.82
61.46
Ours
244
67.94
57.01
55.31
63.41
Ours vs. Open- Sora Plan v1.1
Ours vs. Open-Sora
Ours vs. Pika 1.0
Aesthetic
96 96.4%
76.7%
67.6%
Motion
92.8%%
83.1%
49, 2%
Semantic
63.5%
Ours vs. Cog VideoX-2B
Ours vs. Cog Vid
Ours vs. Kling
Aesthetic
57.0%
$2.3%
63.6%
Motion
56.6%
55.4%
32.5%
Semantic
42.1%
38.6%
63.4%
Figure 4: User preference on sampled VBench prompts. Our videos are generated 5s, 768p. 24fps.
We also present some generated 5-10 second videos in Fig. showing cinematic visual quality and validate the efficacy of pyramidal flow matching. More visualizations are provided in Appendix C.3
User study. While quantitative evaluation scores reflect the video generation capability some ex- tent, they may not align with human preferences for visual quality. Hence, an additional user study is conducted to compare our performance with baseline models, including Cog VideoX (Yang al.) 2024) and Kling (Kuaishou] 2024). We utilized 50 prompts sampled from VBench and asked 20+ participants to rank each model according to the aesthetic quality, motion smoothness, and semantic alignment of the generated videos. As seen in Fig, 4 method is preferred over open-source models such as Open-Sora and CogVideoX-22 22 especially in terms of motion smoothness. This is due to the substantial token savings achieved by pyramidal flow matching, enabling generation of 5. second (up to (0-second) 768p videos at 24 fps, while the baselines usually support video synthesis of similar length only at 8 fps. The detailed user study settings presented in Appendix
8

---

Published as a conference paper at ICLR 2025
(a) The Glenfinnan Viaduct historic railway bridge... is a stunning sight as a steam train leaves the bridge, traveling over the arch-covered viaduct. The landscape is dotted with lush greenery and rocky mountains...
(b) Beautiful, snowy Tokyo city is bustling. The camera through the bustling city street, following several people enjoying the beautiful snowy weather and shopping at nearby stalls. Gorgeous sakura petals...
(c) A side profile shot of a woman with fireworks exploding in the distance beyond her. Figure 5: Visualization of text-to-video generation results. The top videos generated 5s, 768p, 24fps, and the bottom one at 10s, 768p, 24fps. See more generated videos on our project page.
(a) A moon rises from the sky and the lights on the land are bright.
(b) Monster Illustration in flat design style of diverse family of monsters. The group includes furry brown monster, a sleek black monster with antennas, a spotted green monster, tiny polka-dotted monster, all... Figure 6: Visualization of text-conditioned togdc-to-video generation results (5s, 768p. 24fps).
Image-to-Vidde Video Generatetion. Thanks to the autoregressive property of our model and the causal attention design, the first frame of each video acts similarly to an image condition during the train- ing. Consequently, although our model is optimized solely for eext-to-video generation, it naturally accommodates text-conditioned image-to-video generation during inference. Given an image and textual prompt, it is able animate the static input image by autoregressively predicting the future frames without further fine-tuning. In Fig. 6 we illustrate qualitative examples of its image-to-vides generation performance, where each example consists of 120 newly synthesized frames spanning duration of seconds. As can be seen, our model successfully predicts reasonable subsequent mo- tion, endowing the images with rich temporal dynamic information. More generated video examples www.prrwwmi..w.oi............
are best viewed on our project page at httpps://pyaaaaddateteeeppp
4.4 ABLATION STUDY
In this section, conduct ablation studies validate the erucial component of our methods, in- cluding the spatial pyramid in denoising trajectory and the temporal pyramid in history condition. Due to limited space, the ablations for other design choices provided in Appendix 2

---

Published as a conference paper at ICLR 2025
pyramid full-res
full-res
score
pyramid
20k
30k
40%
50k
60k
Training step
Figure 7: Ablation study of spatial pyramid at 50k image training step. On the right is a quantitative comparison the FID results, where our method achieves almost three times the convergence speed.
full seq
pyramid
Figure 8: Ablation study of temporal pyramid at 100k low-resolution video training step.
Effectiveness of spatial pyramid, the generation trajectory of the proposed spatial pyramid, only the final stage operates at full resolution, which significantly reduces the number of tokens for most denoising timesteps. With the same computational resources, it can handle more samples per training batch, greatly enhancing the convergence rate. To validate its efficiency, we designed baseline that employs the standard flow matching objective for training text-to-image generation in our early experiments. This baseline is optimized using the same training data, number of tokens per batch, hyperparameter configurations, and model architecture to ensure fairness. The performance comparison illustrated in Fig. 7 It can observed that the variant using pyramidal flow demon- strates superior visual quality and prompt-following capability. We further quantitatively evaluate the FID metric of these methods on the MS-COCO benchmark (Lin et al. 2014) by randomly sam- pling 3K prompts. The FID) performance curve over training steps is presented on the right of Fig 7 Compared to standard flow matching, the convergence rate of our method is significantly improved.
Effectiveness of temporal pyramid. As mentioned in Section 4.2 the temporal pyramid design can drastically reduce the computation demands compared to traditional full-sequence diffusion. Similar to the spatial pyramid, we also established full-sequence diffusion baseline under the same exper- imental setting to investigate its training efficiency improvement. The qualitative comparison with the baseline presented in Fig- where the generated videos of pyramidal variant demonstrate much better visual quality and temporal consistency under the same training steps. In contrast, the full-sequence diffusion baseline is far from convergence. It fails to produce coherent motion, lead- ing to fragmented visual details and severe artifacts in the generated videos. This performance gap clearly highlights the training acceleration achieved by our method video generative modeling.
5 CONCLUSION
This work presents an efficient video generative modeling framework based on pyramidal visual rep- resentations. In contrast to cascaded diffusion models that use separate models for different image pyramids to improve efficiency, propose a unified pyramidal flow matching objective that simul- tancously generates and decompresses visual content across pyramid stages with a single model, effectively facilitating knowledge sharing. Furthermore, a temporal pyramid design is introduced to reduce computational redundancy in the full-resolution history of video. The proposed method is extensively evaluated VBench and EvalCrafter, demonstrating advantageous performance.
Reproducibility Statement. Our code and models are open-sourced at https: : htttp::/pyyaaa //pp///yddddd/// flow.gi thub.io Flow.github.io
The experimental settings are detailed in Section 4.1 and Appendix
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The work was supported by National Key R&D Program of China (2022ZD0160300), an internal grant of Peking University (2024JK28), grant from Kuaishou (No. DJHL-20240809-111 -10080099-111-----000000))))99) and NSF China (No. 62276004).
REFERENCES
Edward H Adelson, Charles H Anderson, James R Bergen, Peter Burt, and Joan M Ogden. Pyramid methods in image processing. RCA Engineer, 29(6):33-41 1984.
Michael Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In International Conference on Learning Representations, 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-ena retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728 1728-1738, 2021.
Black Forest Labs. FLUX. http:::/ggtubbcc////aa---ottteee /fux 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv2231.15512, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023b.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simula- tors. bttts::/openai.comm www.:aaaw....eeeeeeee//eeeeeeeee www.eeeerattoonndeeess---------e
simull lators 2024.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps. et al. Genie: Generative inter- active environments. In International Conference on Machine Learning, pp. 4603-4623 2024.
Andrew Campbell, William Harvey, Christian Weilbach, Valentin De Bortoli, Tom Rainforth, and Arnad Doucet. Trans-dimensien generative modeling via jump diffusion models. In Advances in Neural Information Processing Systems, pp. 42217 42217-42257, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. In Advances in Neurul Information Processing Systems, 2024a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7310-7320, 2024b.
Zesen Cheng. Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. VideoLLaMA 2: Advancing spatial-temporal modeling and audio understanding in video-LLMs. arXiv preprint arXiv:2406.01 440.0777667 2024.
Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z Kaplan, and Enrico Shippole. Scalable high- reghutresoltion pixcl-space image synthesis with hourglass diffusion transformers. In International Conference Machine Learning, pp. 9550-9575. 2024.
11

---

Published as a conference paper at ICLR 2025
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 886-893, 2005.
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Patch n' pack: NaViT, vision transformer for any aspect ratio and resolution. In Advances in Neural Information Processing Systems, pp. 2252-2274, 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 12873-12883, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, pp. 12606- 12633, 2024.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Miguel Angel Bautista, and Josh Susskind. F-DM: A multi-stage diffusion model via progressive signal transformation. In International Conference on Learning Representations, 2023.
Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. DiffIT
Diffusion vision transformers for image generation. In European Conference on Computer Vision, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pp. 6840-6851 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47)11-33, 2022b.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, pp. 8633- 8646, 2022c.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. CogVideo: Large-scale scale pre- training for text-to-video generation via transformers. In International Conference on Learning Representations, 2023.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. GAIA-I: A generative world model for autonomous driving. arXiv preprint arXiv:2309. 17080, 2023.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, al. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-2188 2024.
Yang Jin, Zhicheng Sun, Kun Xu, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-LaVIT: Unified video-language pre- pre- pre-training with decoupled 2024.
tokenization. In International Conference on Machine Learning, pp-
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference Learning Representations, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Ye Yen Lo, et al. Segment anything. In Proceed- ings of the IEEF/CVF International Conference Computer Vision, pp. 4015 4015-4026, 2023.
12

---

Published as a conference paper at ICLR 2025
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang. Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. VideoPoet: large language model for zero-shot video generation. In International Conference on Machine Learning, pp. 25105- 25124, 2024.
Kuaishou. Kling. https://kkl http//kiinnukkkee www.iiig....iss...ooooo 2024.
Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2V-Turbo: Breaking the quality bottleneck of video consistency model with mixed re- ward feedback. In Advances Neural Information Processing Systems, 2024.
Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. VideoDirectorGPT: Consistent multi-scene video generation via LLM-gguder planning. In Conference on Language Modeling, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pp. 740-755, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF Conference on Computer Viston and Pattern Recognition, pp. 2117-2125, 2017.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow match- ing for generative modeling. In International Conference on Learning Representations, 2023,
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023.
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. EvalCrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22139-22144 2024.
David G Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60:91-110, 2004.
Luma. Dream machine. http://umalabss.////aaaaat
2024.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03044 2024.
Kepan Nan, Rui Xie, Penghao Zhou, Tichan Fan, Zhenheng Yang, Zhijic Chen, Xiang Li, Jian Yang, and Ying Tai. Open 1M: large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02377 2024.
OpenAl. Sora. https: https://openai.com///eee//
2024.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1-64, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. Inter- national Conference on Learning Representations, 2024.
Pika. Pika 1.0. https: hptp::://ikkaarrr 2023.
PKU- Yuan Lab. Tuzhan AI, et Open-Sora Open- Sora-Plan 2024.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1):5485-555 5551, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Runway. Gen-2.
2023.
Runway. Gen-3 alpha. a alpha 2024.
httpes//ruwaayll.oo///ssssssalle
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Infor- mation Processing Systems, pp. (6479-36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, pp. 25278-25294, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-A- Aiieeo: Text-to-video generation without text- video data. In International Conference on Learning Representations, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11918-1190, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. JourneyDB: A benchmark for generative image understanding. In Advances Neural Information Processing Systems, pp. 49659 49678, 2023.
Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhuoyi Yang, and Jie Tang. Relay diffusion: Unifying diffusion process across resolutions for image synthesis. In Interna- tional Conference on Learning Representations, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive mod- eling: Scalable image generation via next-scale prediction. In Advances in Neural Information Processing Systems, 2024.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.1 14837, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, pp. 6000-6010, 2017.
14

---

Published as a conference paper at ICLR 2025
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep righ-resolttion representation learn- ing for visual recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 43 (10):3349-3364, 2020.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang. and Shiwei Zhang. Mod- elScope text-to-video technical report. arXiv preprint arXiv:2308.06:: 065771171 2023a.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang. Yinan He, Jiashuo Yu, Peiqing Yang, et al. LaVie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15103, 2023b.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5288-5296, 2016.
Yilun Xu, Ziming Liu, Max Tegmark, and Tommi Jaakkola. Poisson flow generative models. In Advances in Neural Information Processing Systems, pp. 16782-16795, 2022.
Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. PeRFlow: Piecewise rectified flow universal plug-and-plas accelerator. In Advances in Neural Informa- tion Processing Systems, 2024.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using VQ-VAE and transformers. arXiv preprint arXiv:2104. 10157, 2021.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, JiaZheng Xu, Yuanming Yang, Xiaohan Zhang, Xiaotao Gu, Guanyu Feng, et al. CogVideok VideoX: Text-to-vided diffusion models with an expert transformer. arXiv preprint arXiv:2408.0600 060772 2024.
Tianwei Yin, Qiang Zhang, Richard Zhang, William Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast causal video generators. arXiv preprint 2rX.2:2412.07772. 2024.
Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David A Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, et al. SPAE: Semantic pyramid autoencoder for multimodal generation with frozen LLMs. In Advances in Neural Information Processing Systems, pp. 52692-522222 527044, 2023.
Lijun Yu, José Lezama, Nitesh Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion- tokenizer is key to visual generation. In International Conference on Learning Representationss 2024.
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to- video generation. arXiv preprint arXiv:2309. 2023a.
Han Zhang, Ruili Feng, Zhantao Yang, Lianghua Huang, Yu Liu, Yifei Zhang, Yujun Shen, Deli Zhao, Jingren Zhou, and Fan Cheng. Dimensionality-varyiee diffusion process. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14307-14316. 2023b.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all. Sora 2024.
http:://ggthhbb.cm//pppattttttt
15

---

Published as a conference paper at ICLR 2025
A
DERIVATION
This section provides detailed derivation for Eq. (15) that handles jump points the spatial pyra- mid. For quick lookup,Tabil Table summarizes the used notations.
Symbol
Description
21
Data latent at full resolution
Noise at full resolution
K
Number of pyramid stages
ek
Timestep endpoint of k-th pyramid stage
Sk
Timestep at starting point of k-th pyramid stage
Ten
Noisy latent endpoint of k-th stage
Noisy latent at starting point k-th stage
Noisy latent at timestep
n
Noise at the resolution of current stage
Up(-)
Upsampling function, e.g. nearest-neighhor Downsampling function, e.g. bilinear
Down(...)
Table Notation in the main paper.
To ensure continuity of the probability path across different stages of the spatial pyramid. we need to make sure that the endpoints have the same probability distribution. According to Eqs. and 122 their distributions are already similar after a simple upsampling transformation:
x1 N(s Up(Doun(x)) 2))))) (1-5x)²1 2
(18)
Up ( Ø1 M(ex41 Up( Sp(DDon(x1) (21. *))))) (1 Ex+1 Σ).
(19)
Therefore, we can directly apply linear transformation with corrective Gaussian noise match their distributions:
Up( + an', s.t. n'' NOO, ck+1
8k
(20)
where the rescaling coefficient sx/ca $s/Cx+1 allows the means of these distributions to be matched, and is the noise weight. Additionally, need to match the covariance matrices of Eqs. (88) and 200
- Ek+1 1x+1) >
a 2 = -1 88) I.
(21)
To allow analysis of covariance matrices, e.g. E, we consider simplest scenario with nearest neigh- bor upsampling. In this case, Σ has a blockwise structure with non-zero elements only in the 4 x blocks along the diagonal (corresponding to those upsampled from the same pixel). Then, it can be inferred that the corrective noise's covariance matrix E' has a similar blockwise structure:
1
Σ block
7
Σ' block
7
7
(22)
1
1
7
7
where y is negative value in [-1/3, 0] for the decorrelation (its lower bound -1/3 ensures that the
covariance matrix is semidefinite). We further rewrite Eqs. (2T) and (2) by considering the equality of their diagonal and non-diagonal elements, respectively:
(1 11-ep ex41) 1 +a² -115k))
(23)
k+1
2
(1 ex+1)² +a² ==²==0. =0.
(24)
Taking into account the timestep constraints
< Sk: Ck+1 < 1, they can be solved directly:
8k
8k
6k+1
1 a /1
(25)
(1- 11-52--1
16

---

Published as a conference paper at ICLR 2025
Intuitively, it is desirable to maximally preserve the signals at each jump point, which corresponds
to minimizing the noise weight a. According to Eq. (25). this is equivalent to minimizing 7 Sub-
stituting its minimum value y -1/3 into Eq. 25 yields:
25k
a v3(1- - 9k)
ek 1+
(26)
is worth noting that Ck+1 > sk, indicating that the timestep is rolled back a bit when adding the corrective noise at each jump point. We can further obtain the renoising rule in Eq. (15):
+ 8k Up((a)+1)
v3(1
5k) n'.
(27)
2
B EXPERIMENTAL SETTINGS
Model Implementation Details. We adopt the MM-DiT architecture, based on SD3 Medium (Esser al. 2024), which comprises 24 transformer layers and total of 2B parameters. The weights the MM-DiT are initialized from the SD3 medium. Following the more recent FLUX.1 (Black Forest Labs 2024), both T5 (Raffel et al. 2020) and CLIP (Radford et al.] 2021) encoders are employed for prompts embedding. To address the redundancy in video data, we have designed 3D VAE that compresses videos both spatially and temporally into a latent space. The architecture of this VAE similar to MAGVIT-v2 YYu et al. 2024), employing 3D causal convolution to ensure that each frame depends only on the preceding frames. Leibler (KL) regularization applied to the latents. Overall, the 3D VAE achieves a compression rate of 8 X 8 x from pixels to the latent. It is trained on WebVid-10M and 6.9M SAM images from scratch. To support the tokenization of very long videos, we scatter them into multiple GPUs to distribute computation like Cog VideoX (Yang et al. 2024).
features an asymmetric encoder-decoder with Kullback-
Training Procedure Our model undergoes three-stage training procedure using 128 NVIDIA A A100 GPUs (1) Image Training. In the first stage, we utilize a pure image dataset that includes 180M images from LAION-5B (Schuhmann al., (002)). 11M from CC-12M (Changpinyo al.) 2021). 6.9M non-blurred images from SA-IB (Kirillov et al. 2023), and 4.4M from JourneyDB (Sun ct al. 2023). We keep the image's original aspect ratio and rearrange them into different buckets. It is trained for a total of 50,000 steps. requiring approximately 1536 A100 GPU hours. After this stage, the model has learned the dependencies between visual pixels, which facilitates the con- vergence of subsequent video training. (2) Low-Resolution Video Training. For this stage, we employ the WebVid-10M (Bain et al., 202T]. Open Vid-IM (Nan et al.. (0024), and another IM non-watermark video from the Open-Son Plan (PKU-Yuan Lab et al.) 2024).. We also leverage the Video-LLaMA2 (Cheng et al. 2024). a of video understanding model, to recaption each video sample. The image data from stage 1 is also utilized at proportion of 12.5% in each batch. We first train the model for 80,000 steps on 2-second video generation, followed by an additional 120,000 steps on 5-second videos. In total, it takes about 11,520 A100 GPU hours at this stage. (3) High-Resolution Video Training. The final stage employs the same strategy to continue fine- tuning the model on the aforementioned high-resolution video dataset of varying durations (5-10s). It consumes approximately 7,680 A100 GPU hours for 50,000 steps in the final stage.
Hyperparameters Setting The detailed training hyper-parameter settings for each optimization stage are reported in Table 4
Baseline Methods. For VBench (Huang et 2024). we compare with eight baseline methods,
including Open-Sora Plan V1.3 (PKU- Yuan Lab et al. 2024). Open-Sora 1.2 (Zheng et al. 2024).
VideoCrafter2 (Chen et al.) (2024b). Gen-2 (Runway) 2023), Pika 1.0 (Pika 2023), T2V-Turbo (Li et al. 2024). Cog VideoX (Yang et al. 2024). Kling (Kuaishou) 2024). and Gen-3 Alpha (Runway) 2024.. J. Among them, Open-Sora Plan, Open-Sora, Cog Video-X, Kling and Gen-3 Alpha can gener- ate long videos. For EvalCrafter (Liu et al., (224), our model is compared to six baselines, includ- ing ModelScope (Wang et al., 2023a). Show-1 (Zhang et al. 2023a), LaVie (Wang et al. 2023b), VideoCrafter2 (Chen et al. 2024b), Pika 1.0 (Pika) 2023), and Gen-2 (Runway (023)). The above models are all based on full full-sequence diffusion, while our method combines the merits of autoregres- sive generation and flow generative models to achieve better training efficiency of video generation.
User Study. To complement the quantitative evaluation in the main paper, we conduct a rigorous user study collect human preferences for these generative models. To accomplish this, we sample
17

---

Published as a conference paper at ICLR 2025
Configuration
Stage-1
Stage-2
Stage=3
Optimizer
AdamW
Adam
AdamW
Optimizer Hyperparameters
A 0.9, (92 0.999, < le-
0.9, 0.95,
le
Global batch size
1536
768
384
Learning
lc-4
lc-4
5c-5
Learning schedule
Constant with warmup
Constant with warmup
Constant with warmup
Training Steps
50k
200k
50k
Warm-up steps
lk
lk
lk
Weight decay
lc-4
lc-4
lc-4
Gradient clipping
1.0
1.0
1.0
Numerical precision
bíloat16
bfloat16
bfloat16
GPU Usage
128 NVIDIA A100
128 NVIDIA A
128 NVIDIA A100
Training Time
12h
90h
60h
Table 4: The detailed training hyperparameters of our method
prostppeept beat ond
yoleg the Some tveer Teueth bacipport oetoencde
Abeecall spbiturr rlkin bo they the Tevert grand neteoder polnt
HIGHT
Srick Matia
Plackite Noclitta
Algraedd
Figh
Sohritt
Figure 9; Interface for user study of video generative performance.
50 prompts from the VBench prompt list and randomly sample one generated video for each prompt from the baseline model. In total, six baseline models are considered, including Open-Sora Plan V1.1 (PKU- Lab et al., 2024). Open-Sora 1.2 (Zheng al., 2024), Pika 1.0 (Pika 2023). VidVoXXo22B and 5B (Yang al. 2024), and Kling (Kuaishoul 2024). We then pair these results with our generated video and ask the participant to rank their preference among three dimensions: aesthetic quality, motion smoothness, and semantic alignment, each of which represents crucial aspect of video quality. The interface for the user study is exemplified in Fig. 9 where the user accepts prompt and two generated videos (with the unnecessary information cropped, such watermark indicating which model it belongs to), and chooses between which model is better in the three dimensions. We distribute the user study 1411 valid preference choices, ensuring its effectiveness. The results of this user study are presented Fig. where our model shows a very competitive performance among the compared baselines.
than 20 participants, and collect total of
C ADDITIONAL RESULTS
C.1 QUANTITATIVE RESULTS
This section provides the full results on VBench (Huang et al. 2024) and EvalCrafter (Liu al. 2024) as a supplement to the performance comparison in the experiments section the main paper. The evaluation of our model is performed using 5-second 768p videos generated at 24 fps.
18

---

Published as a conference paper at ICLR 2025
Table 5: Detailed results on VBench (Huang al. 2024p. See Table for the summarized results. We additionally use blue to indicate the highest scores among models trained on public datasets.
Motion Consistency Consistency Flickering Smoothness
Subject
Background Temporal
Dynamic Degree
Aesthetic Quality
Imaging Quality
Object
Model
Class
Trained private datasets:
Gen-2
97.61
97.61
99.56
99.58
18.89
66.96
67.42
90.92
Pika 1.0
96.94
97.36
99.74
99.50
47.50
62.04
61.87
88.72
Cog Voggideo
96.78
96.63
98.89
97.73
59.86
60.82
61.68
83.37
Cog VidenX--X55
96.23
98.66
96.92
70.97
61.98
62.90
85.23
Kling
97.60
99.30
99.40
46.94
61.21
65.62
87.24
Gen-3 Alpha
97.10
98.61
99.23
63.34
66.82
Trained on public datasets:
Open-Sone v1.3
99.20
99.05
30,28
60.42
56.21
85.56
Open-Sora 1.2
96.75
97.61
99.53
98.50
42.39
56.85
63.34
82.22
VideoCrafter2
96.85
98.22
98.41
97.73
42.50
63.13
67.22
92.55
Tur-Turho
96.28
97.02
97.48
97.34
49.17
63.04
72.49
93.96
Ours
96.95
98.06
99.49
99.12
64.63
63.26
65.01
86.67
Multiple Objects
Human Action
Spatial Relationship
Appearance Style
Temporal Style
Overall Consistency
Model
Color
Scene
Truined on private datasels:
Gen-2
55.47
89.2
89.49
66.91
48.91
19.34
24.12
26.17
Pika 1.0
43.08
86.2
90.57
61.03
49.83
22.26
24.22
25.94
Cog Videene
62.63
79.41
69.90
51.14
24.80
24.36
26.66
Cog VideoX-5B
62.11
82.81
$3,20
24.91
25.38
27.59
Kling
68.05
93.4
89.90
73.03
50.86
19.62
24.17
26.42
Gen-3 Alpha
53.64
96.4
80.90
65.09
54.57
24.31
24.71
26.69
Truined public datasets:
Open-Sora Plan v1.3
43.58
86.8
79.30
51.61
36.73
20.03
22,47
24.47
Open-Sore Apen-Sora 1.2
51.83
91.2
90.08
68.56
42.44
23.95
24.54
26.85
VideoCrafter2
40.66
95.0
92.92
35,86
55.29
25.13
25.84
28.23
Tur-Turbo
54.65
95.2
89,90
38.67
55.58
24.42
25.51
28.16
Ours
50.71
85.6
82.87
59.53
43.20
20.91
23.09
26.23
Table 6: Raw metrics on EvalCrafter (Liu et al.
2024).
The baseline results are found on its website, See Table for the summarized results.
but there were no results for LaVie (Wang al. 2023b
CLIPP- Temp
Warping Error
Face Consistency
Action- Score
Motion
Model
VQA VQAr
IS
AC-Score
Trained on private datasets:
Pika 1.0
69.23
71.12
16.67
99.89
0,0008 0.0005
99.22
61.29
42.0
Gen-2
90.39
92.18
19.28
99.99
99.35
73.44
44,0
Truined on public datasets:
ModelScope
40.06
32.93
17.64
99.74
0.0162
98.94
72.12 12
42.0
Show-1
23.19
44.24
17.65
99.77
0.0067
99.32
81.56
50.0
VideoCrafter2
79.93
67.04
17.39
99.84
0.0085
99.44
68.17
36.0
Ours
86.09
88.31
18.49
99.90
0.0019
98.38
67.58
46.0
Flow- Score
CLIP-
BLIP- BLUE
SD- Score
Detection- Score
Color- Score
Count- Score
OCR- Score
Celebrity
Model
Score
ID Score
Trained on private datasets:
Pika 1.0
1.14
20.47
21.31
67.43
70,26
42.03
62.19
94.85
36.53
Gen-2
0.58
20.26
22.25
67.69
69.54
47.39
58.36
63.74
38,90
Trained public datasets:
ModelScope
6.99
20.36
22.54
67.93
50.01
38.72
44.18
71.32
44.56
Show-1 2.07
20.66
23.24
68.42
58.63
48.55
44.31
$8.97
37.93
VideoCrafter2
3.90
21.21
22.71
68.58
69.32
45.11
50.45
80.37
38.40
Ours
1.79
20.73
23.29
68.26
69.55
47.74
56.31
68.55
44.72
VBench (Huang al. 2024). The full experimental results VBench shown in Table As can be observed, our model achieves leading or highly competitive results among open-source and com- mercial competitors, especially for the metrics related to motion quality. For example, the dynamic
19

---

Published as a conference paper at ICLR 2025
w/o renoise
renoise
Figure 10: Ablation study of corrective renoising during the inference stage.
causal bidirection
Figure 11: Ablation study of blockwise causal attention at 100k training step.
degree metric of our model ranks 2nd among all models 64.63, validating the effectiveness of our generative model in learning temporal dynamics. For the rest of the metrics, our results are also gen- erally superior to the open-source Open-Sora Plan v1.3 (PKU- Yuan Lab al. 2024) and Open-Sora -ora 1.2 (Zeng et al., 2024), with significantly lower training computational cost as mentioned earlier. We also note that half of our results even outperformed the recent Cog VideoX-5B (Yang et al.) 2024), which is based on larger DiT model, demonstrating its modeling capacity. On the other hand, our model performs relatively inferior on metrics such as color and appearance style, which is more related to the image generation capabilities and finer-grained prompt following. This is largely due to our video captioning procedure based on video LLMs which tends to produce coarse- grained captions, thus dampening these abilities. Nevertheless, thanks to autoregressive generation framework, which decomposes video generation into first frame generation and subsequent frame generation, these image quality issues be addressed separately with ad- ditional well-captioned image data in future training stages. Similarly, due to the SD3-Medium weight initialization, which infamous for its human structure, our method achieves relatively low score in human action, which could be addressed by switching to other base models or training from scratch.
EvalCrafter (Liu et al. 2024). The raw metrics on EvalCrafter are provided in Table 6 Overall, our model delivers highly competitive performance on the majority of metrics, outperforming many previous open-source and closed-source models. In particular, the motion AC score of our method which relevant to the temporal motion quality ranks 2nd among methods, justifying the ca- pacity of our pyramid designs to learn complex spatiotemporal patterns in video. Our method also demonstrates superiority over several other metrics related to semantic alignment, including BLIP- BLUE and CLIP score. Placing top two in both metrics among the models compared, including the closed-source Gen-2 (Runway 2023), confirms the advantages of our model in text-to-video seman- tic alignment. The only metric where model performs poorly is face consistency, which is due to the temporal pyramid design adopted for compressing the history condition. We view this as an issue that can potentially be addressed by better temporal compression schemes.
C.2 ABALTION STUDY
In this section, we conduct additional ablation studies of two important design details in pro- posed pyramidal flow matching, including the corrective noise added during inference of the spatial pyramid and the blockwise causal attention used for autoregressive video generation.
Role of corrective noise. To study its efficacy in the spatial pyramid, we curate a baseline method that inferences without adding this corrective Gaussian noise. The detailed comparative results
20

---

Published as a conference paper at ICLR 2025
pyramid full-seq
PVD
4.00
300
200
Training step
(a) loxl-to-Image generation results.
(b) FVD convergence plot.
Figure 12: (a) The visualization of generated images from our pyramid-flow. Our model can syn- thesize high- high-resolution and good-quality images even using only a few million training samples. (b) The FVD comparison with full-sequence diffusion video training on MSR-VTT (Xu et al. 2016 benchmark along with optimization iterations.
of our method against this variant are shown in Fig. 10 While the baseline method has correct global structure, it fails to produce fine-grained, high-resolution image with rich details and instead produces blurred image that suffers from block-like artifacts (better observed when zooming in). This is because applying the upsampling function at the jump points between different pyramid stages of varying resolutions results in excessive correlation between spatially adjacent latent values. In comparison, our generated images have rich details and vivid colors, confirming that the adopted corrective renoising scheme effectively addresses this artifact problem in the spatial pyramid.
Effectiveness of causal attention. In Fig. 11 we study the effect of blockwise causal attention by comparing it to the bidirectional attention used in full-sequence diffusion. While an intuitive under- standing might be that bidirectional attention promotes information exchange and increases model capacity, is understudied for autoregressive video generation. In an early experiment, we trained baseline model using bidirectional attention across different latent frames, the results of which are visualized in Fig. TT] As can be seen from the sampled keyframes of the I-second videos, this model suffers from lack of temporal coherence the subject in the generated video is constantly chang- ing in shape and color. Mcanwhile, our model shows good temporal coherence with reasonable motion. We infer that this is because the history condition in bidirectional attention is influenced by the ongoing generation and thus deviates, whereas the history condition in causal attention is fixed, serving as a predetermined condition and stabilizing the autoregressive generative process.
C.3 VISUALIZATION
This section presents additional qualitative results for our text-to-video generation in comparison to the recent leading models including Gen-3 Alpha (Runway) 2024), Kling (Kuaishou) 2024 and VideoX Yang et al. 2024). The uniformly sampled frames from the generated videos are shown Figs. 14 and 15 in which our videos are generated 5s, 768p, 24fps. Overall, we observe that despite being trained only on publicly available data and using small computational budget, our model yields highly competitive visual aesthetics and motion quality among the baselines.
Specifically, the results highlight the following characteristics of our model: (1) Through generative pre-training, our model is capable of generating videos of cinematic quality and reasonable content. For example, in Fig. 14a, our generative video shows a mushroom cloud resulting from "a massive explosion" taking place in "the surface of the earth". creating sci-fi movie atmosphere. However, the current model is not fully faithful to some prompts such as the "salt desert" in Fig. [1b] which could be addressed by curating more high-quality caption data. (2) Despite that our model has only 2B parameters initialized from SD3-Medium (Esser et al. 2024), it clearly outperforms Cog VideoX- 2B of the same model size with additional training data, and is even comparable to the 5B full version in some aspects. For example, in Figs. 15a and 15b] only model and its 5B version are capable of generating reasonable sea waves according to the input prompt, while its 2B variant merely illustrates almost static sea surface. This is largely attributed to our proposed pyramidal
21

---

Published as a conference paper at ICLR 2025
1.50
1.25
1.25
1.00
0.50
0.50
0.25
0.25
0.20
0.0
62
0.4
0.6
0.8
0.0
0.5
0.8
10
(a) Our coupling with K 2, onc datapoint.
(b) Random coupling with K = 2, one datapoint.
2:00
2:40
1.75
1.25
1.00
1:00
0.35
0:00
0.00
6.6
Our coupling with K three datapoints.
(d) Random coupling with three datapoints.
Figure 13: Justification for our coupled sampling in Eqs. (9) and 10
flow matching in improving training efficiency. Overall, these results validate the effectiveness of our approach in modeling complex spatiotemporal patterns through the spatial and temporal pyramid designs. Our generated videos are best-vide best-viewed ttst / p pwwamiiddiiii lawaddffllo flw.lglw..g. thub.io Since the autoregressive video generation model natively generates a high-quality image as the first frame, pyramid-flow can also be applied to text-to-image generation. Even with only few million training images, it can show excellent visual quality, see Fig. 12a for the generated images.
C.4 ToY EXPERIMENT OF COUPLING NOISE
To validate the effectiveness of coupled sampling in Eqs. p and (10), we illustrate two variants of piecewise flow matching in toy experiment that considers mapping a few data points to uniform distribution. Two different coupling designs are considered within each time window, namely our coupling random coupling. It can be seen that our coupled sampling strategy produces much more straight flow trajectories.
The rationale for improving straightness by coupling noise is that: the straightness of the flow tra- jectory is usually compromised when there are intersections. Sampling the endpoints independently (as in vanilla flow matching) creates random directions for each trajectory and leads intersections. Instead, by coupling the sampling of these endpoints as in Eq. ( and Eq. (10), we can create more organized, possibly parallel trajectories with fewer intersections, thus improving straightness. As illustrated in Fig. 13
where coupling noise indeed leads to more straight flow trajectories.
D LIMITATIONS
Our method only supports autoregressive generation and cannot extended to keyframe interpo- lation or video interpolation. In addition, noticed that the temporal pyramid designs improve
22

---

Published as a conference paper at ICLR 2025
training efficiency sometimes lead subtle subject inconsistency, especially over the long term. While this is not a prevalent problem, we believe that developing better temporal compression meth- ods is critical to the broader upplicability of autoregressive video generative model. In addition, improving inference efficiency towards real-time is an intriguing problem (Yin et al. 2024).
There also several issues related to the training data. Since we did not include prompt rewriting procedure in the data curation, the experimental results are focused on relatively short prompts. Also, due the data filtering procedure, our model did not learn scene transitions during training. This may be overcome by introducing an additional model as the scene director (Lin et al. 2024).
23

---

Published as a conference paper at ICLR 2025
Gen-3 Alphaa
Kling
Ours
A massive explosion the surface of the earth.
Gen--3 Alpha
Kling
Ours
(b) movie trailer featuring the adventures of the year old space man wearing red wool knitted motorcyele helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors.
Gen-3 Alphu
Kling
Ours
(c) A side profile shot of woman with fireworks exploding in the distance beyond her.
Figure 14: Visualization of generated videos in comparison with the state-of-the-art closed-source models, including Gen-3 Alpha (Runway) 2024) and Kling (Kuaishou, 2024). Our model delivers cinematic visual quality comparable to these models while adhering to the textual prompt.
24

---

Published as a conference paper at ICLR 2025
Cog ValeeX SB
Cog VideoX- 2B
Ours
(a) An aerial shot of lighthouse standing tall on rocky cliff, its beacon cutting through the early dawn, waves crash against the rocks below.
VogVeleuX-S SB
Cogg Videoo 2B8
Ours
(b) Drone view of waves crashing against the rugged clifi's along Big Sur's garay point beach. The crashing blue waters create white white-tipped waves, while the golden light the setting sun illuminates the rocky shore.
VideeooXX-5 513
CodgideoX 2B
Ours
(c) series of underwater explosions, creating bubbles and splashing water.
Figure 15: Visualization of generated videos in comparison with VogVid (Yang et al. 2024
Our model outperforms Cog VideoX-2B the same model size and comparable to the 5B version.
25


Reviews:
[{"review_id": "0teSp6SxL8", "reviewer": "Reviewer_6mLb", "summary": "This paper introduces a novel pyramidal flow matching scheme for video generation, which significantly improves training efficiency while preserving generation quality. The authors also propose a unified flow-matching objective that enables joint training of the pyramid stages within a single DiT model, eliminating the need for separate optimization across multiple models seen in prior approaches. Comprehensive experimental analyses are conducted on the VBench and EvalCrafter benchmarks, with proofs and additional qualitative results included in the supplementary materials.", "questions": "Please refer to the questions (or issues) mentioned in the Weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ The proposed pyramidal flow matching scheme is novel in video generation modeling and greatly enhances training efficiency.\n+ The unified training objective is intuitive and effective.\n+ The quantitative and qualitative analyses in the paper are comprehensive.\n+ The quality of the generated videos is excellent.", "weaknesses": "* The writing in the paper could benefit from further improvement for clarity and readability.\n    - The repeated use of the term \"full-resolution\" up to the experiment section suggests that generation is being done in pixel space rather than latent space. It would be helpful to clarify this in the paper, as it may be misleading.\n\n    - The paper contains several grammatical errors and repeatedly uses unnecessary terms, such as \"sophisticated,\" which affect readability. I encourage the authors to revise the manuscript to improve clarity and flow.\n\n* Some of the derivations and assumptions in the paper are ambiguous and potentially flawed\n\n    - In Equations 7 and 8, the notation should reflect the conditional distributions $\\hat{x}{e_k}|x_1$ and $\\hat{x}{s_k}|x_1$. While both endpoints are Gaussian-distributed conditionally, the endpoint distributions $\\hat{x}{e_k} = \\int p(\\hat{x}{e_k}|x_1)p(x_1)dx_1$ are not Gaussian.\n\n    - In line 213, while the objective $\\hat{x}{e_k} - \\hat{x}{s_k}$ is correct, since we consider flow matching with $K$ windows, the vector field should instead be conditioned on the endpoints, $u_t(x_t|\\hat{x}{e_k})$. However, this is challenging because the distribution $p(x_t|\\hat{x}{e_k})$ may not be Gaussian. This objective could be derived more straightforwardly from a rectified flow perspective [1], where velocities are matched with $\\dot{X}_t$ (see section 2.3 in [1]).\n\n    - For training, are $x_1$ and $n$ at the start and end the same? Since flow matching accommodates data points from arbitrary couplings, this should not impact training validity but would benefit from additional explanation for clarity.\n\n    - Regarding the renoising procedure in Section 3.2.2, Equation 12 should ideally denote $Up(\\hat{x}{e_{k+1}}|x_1)$ rather than $Up(\\hat{x}{e_{k+1}})$, as the latter may be non-Gaussian. Consequently, the subsequent proof may be invalid if it relies on $Up(\\hat{x}{e_{k+1}})$ being Gaussian.\n\n         [1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow\n\n* Some important details and experiments are missing from the paper.\n\n    - In Sec 3.2.1, the authors claim to enforce the noise to be in the same direction to enhance the straightness of the flow trajectory (Lines 207-209). However, there is no ablation experiment to show the benefit of this design choice.\n\n    - The proof in Section 3.2.2 and Appendix A primarily relies on Equation 8. However, the authors implement the scheme in Equation 10, with no discussion on how the derivations in Section 3.2.2 and Appendix A could be generalized for Equation 10.\n    - What is the number of sampling steps in Algorithm 1?\n\n    - (Minor) Have the authors considered using temporally compressed pyramidal flow as well?\n\n\n* No experimental results are provided for the video autoencoding task\n   - In Section 4.1, the authors claim to train a video VAE for spatial and temporal compression of videos; however, there are no experimental results provided to evaluate the performance of the video VAE.\n    - Why train a new video VAE instead of utilizing existing open-source video VAEs, such as Open-Sora or Open-Sora-Plan? Wouldn't this approach have been more effective in ensuring a fair experimental comparison?\n\n* The results on VBench and EvalCrafter are quite strong. However, since most of the competing approaches were published prior to the release of these benchmarks, it would be beneficial for the authors to include the FVD metric to compare their approach with the competing baselines. Additionally, could the authors provide a comparison of the generation and training speeds of their approach relative to previous works?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sgO7BbjQgo", "reviewer": "Reviewer_fYS3", "summary": "This work presents an effective flow matching approach for text-to-video generation via both spatial and temporal pyramidal designs. With such an architecture, the training efficiency has been significantly reduced. It has a good conversation that when noise is strong, flow matching is less critical and can be performed at a low resolution. It has a unified flow matching objective instead of having different models for generation and super-resolution. The paper has validated this effectiveness by controlled experiments compared to a baseline with full-resolution with the same computational costs. The source code of this paper is expected to be open-source, which is very helpful to video generation research and industrial communities.", "questions": "In the rebuttal, I hope the authors can address my questions and concerns in the weakness section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The strength of this paper is multi-fold. \n+ It builds a flow matching model with multiple resolutions for text-to-video generation. The pyramidal flow matching allows the model to train with less computational costs and memory footprints.\n+ The whole model has a unified objective instead of optimizing separate modules for video generation and super-resolution, using a single Diffusion Transformer.\n+ The experimental results are competitive, with evaluation on two public benchmarks of VBench and EvalCrafter. The visual quality is comparable to other commercial text-to-video models.\n+ The technical explanation for inference with renoising to solve jump points is clear with supplemental materials.\n+ The model can be extended to image-to-video generation. \n+ The ablation studies are conducted to illustrate the contribution of different component in the model.", "weaknesses": "While this work has an interesting novel design for flow matching for video generation and competitive visual results, there are some unclear points and weaknesses as follows.\n- Questions about [s_k, e_k]. The authors divide [0,1] into K time windows [s_k,e_k]. Why don't the authors set e_{k+1}=s_k? Instead, the authors use e_{k+1}=2s_k/(1+s_k) and we can that e_{k+1}>s_k. This means there are overlapping time windows. Given a time step t, t may fall on more than one time step, and how do authors handle such a scenario? Also, is s_K equal to zero? Could you show what the exact values of [s_k,e_k] are?\n- The Figure 1(b) and Figure 3 are not consistent. x^i_t takes x^{i-1}^t with the same spatial resolution as part of the history in Figure 3, but Figure 1(b) shows x^i_t takes x^{i-1}_t' with lower resolution as a temporal condition. This part confuses me.\n- The model is autoregressive in the temporal domain, and does it mean that the model can generate videos with arbitrary lengths? Why the inference is limited to videos up to 10 seconds?\n- When performing the ablation study for the temporal pyramid, the baseline \"full-seq\" has only qualitative results. Can the authors provide quantitative results or something similar to the plot in Figure 7?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZ5zzAfDbf", "reviewer": "Reviewer_csnD", "summary": "The paper proposed a pyramid-based flow matching for efficient training of video diffusion models. Unlike traditional flow matching which operates on a single resolution, the proposed spatial pyramidal flow matching operates on a pyramid of resolutions. The denoising process starts from a small resolution, and the resolution will increase by 2 after several sampling steps. To address the discontinuity of the jumping point, the paper proposed a novel re-noising formula. For video generation, the paper proposed an autoregressive approach, where the generation of each frame is conditioned on the low-resolution of previous frames.", "questions": "- During training (Equation-16), noisy condition is used. During testing (Equation-17), clean generated frames are used. Will there be a training-testing distribution mismatch?\n- How is text condition added to the model?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The ideas of both spatial pyramids and temporal pyramids are novel and interesting.\n- The training efficiency is largely improved due to the novel pyramid design.", "weaknesses": "- The analysis of inference efficiency is lacking. How does the proposed method compare to previous full-attention methods for different numbers of frames?\n- Compared to full-attention methods, the proposed autoregressive method may encounter the issue of error drifting when the number of frames increases. At how many frames, the proposed method will fail?\n- It will be good to include some video results from previous methods on the project page.\n- Figure 7 and Figure 8 both show partial results before training convergence. It will be more convincing to show the training graph with more training iterations or converged training behavior. \n- The text-video alignment is worse compared to previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "epfdV24OhZ", "reviewer": "Reviewer_BWBa", "summary": "This paper introduces pyramidal flow matching, a new video generative model that combines spatial and temporal pyramid representations, which can enhance the training efficiency and maintains high video generation quality.", "questions": "1. The authors should clarify whether their proposed method can be applied to text-to-image models, as the method appears to have limited relevance to video generation.\n2. The authors should explain why they opted for flow matching over DDIM, as well as outline the advantages of flow matching.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a novel pyramidal flow matching algorithm based on the premise that the initial timesteps in diffusion models are quite noisy and uninformative. This approach builds upon the recent prevalent flow matching framework to address the shortcomings of existing methods, specifically the requirement of employing distinct models at different resolutions, which sacrifices flexibility and scalability.", "weaknesses": "1. The authors should clarify the specific implications of how the separate optimization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This is crucial as it relates to the foundational aspects of the problem design presented in this paper.\n2. The proposed method involves many parameters that require manual design. The authors should provide more explanations when designing these parameters, such as time windows and γ.\n3. Figures 2 and 3 in the paper are overly simplistic, and the authors should provide a detailed explanation of their relationship to the equations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N5SHeIEdRP", "reviewer": "Reviewer_8tU7", "summary": "The authors propose a novel video generation method to address the problems of the current methods. The SOTA (state-of-the-art) methods mostly suffer from heavy computational burdens due to cascaded architectures or separate optimization for different resolutions of video generation training. The authors address this issue with a novel pyramid flow matching framework, both in the spatial and temporal domains. In the spatial domain, the method builds a pyramid of different resolution frames, where the early stages operate on compressed, low-resolution representations (latent representations) of the frames. As the stages go higher up, the resolution increases, and the full resolution is only used in the final stage for optimization. Temporal compression is achieved by using only a lower resolution history of previous frames (to understand motion and scene), further reducing computation for long videos. This pyramidal approach reduces redundant computation by focusing resources only on necessary parts, making training more efficient. The results are comprehensive and satisfactory. Overall, a good paper to be accepted.", "questions": "What are the training data used by the other compared methods (especially the ones trained with open-source data)?\n\nHow much overlap is there between the training data used in the proposed method and the other methods?\n\nWhy did the authors keep the number of pyramid stages set to 3 in all the experiments? There should be an ablation study on the number of pyramid stages.\n\nWhat are the number of parameters and/or FLOPs (operations) used by other (open-source) methods for video generation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "The computational complexity of SOTA video generation methods is a real problem, and the pyramidal flow matching solution provided by the authors is interesting and effective.\n\nThe pyramidal flow matching framework is built on a single diffusion transformer model and can be optimized in a unified, end-to-end fashion. This saves time and enables better knowledge sharing. Better knowledge sharing can help achieve greater consistency in the generated videos, which is evident from the generated results.\n\nEvaluated on benchmarks like VBench and EvalCrafter, the model demonstrated high performance, especially compared to methods trained with open-source data.\n\nThe model also achieved competitive performance during the user study, and I have personally checked their provided anonymous website for the videos, which looked good.", "weaknesses": "The mathematical notations are not clearly defined. For example, the authors start with s_k, e_k​, x_{s_k}​​, or x_{e_k}​​, etc., without properly defining them first. This hinders the flow of the paper. Please resolve all such issues.\n\nOverall, the writing style of the paper is a bit convoluted (especially in the methods section); it should be revised for smoother understanding.\n\nThere is no ablation study on the number of pyramid stages, which is a crucial factor in their design choice.\n\nThere is no comparison between the number of parameters and FLOPs used by other open-source methods and the proposed method. This can reflect how effective the model is compared to other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)