Pyramidal Flow Matching for Efficient Video Generative Modeling

## Abstract

Abstract Video generation requires modeling a vast spatiotemporal space, which demands significant computational resources and data usage. To reduce the complexity, the prevailing approaches employ a cascaded architecture to avoid direct training with full resolution. Despite reducing computational demands, the separate optimization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video generative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, we craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours.
All code and models will be open-sourced at https://pyramid-flow.github.io .

## 1 Introduction

Video is a media form that records the evolvement of the physical world. Teaching the AI system to generate various video content plays a vital role in simulating the real-world dynamics (Hu et al., ; Brooks et al., ) and interacting with humans (Bruce et al., ; Valevski et al., ). Nowadays, the cutting-edge diffusion models (Ho et al., ; Blattmann et al., ; OpenAI, ) and autoregressive models (Yan et al., ; Hong et al., ; Kondratyuk et al., ) have made remarkable breakthroughs in generating realistic and long-duration video through scaling of data and computation. However, the necessity of modeling a significantly large spatiotemporal space makes the training of such video generative models computationally and data intensive.

To ease the computational burden of generating high-dimensional video data, a crucial component is to compress the original video pixels into a lower-dimensional latent space using a VAE (Kingma & Welling, ; Esser et al., ; Rombach et al., ). However, the regular compression rate (typically 8$\times$) still results in excessive tokens, especially for high-resolution samples. In light of this, prevalent approaches utilize a cascaded architecture (Ho et al., ; Pernias et al., ; Teng et al., ) to break down the high-resolution generation process into multiple stages, where samples are first created in a highly compressed latent space and then successively upsampled using additional super-resolution models. Although the cascaded pipeline avoids directly learning at high resolution and reduces the computational demands, the requirement for employing distinct models at different resolutions separately sacrifices flexibility and scalability. Besides, the separate optimization of multiple sub-models also hinders the sharing of their acquired knowledge.

This work presents an efficient video generative modeling framework that transcends the limitations of the previous cascaded approaches. Our motivation stems from the observation in Fig. 1(a) that the initial timesteps in diffusion models are quite noisy and uninformative. This suggests that operating at full resolution throughout the entire generation trajectory may not be necessary. To this end, we reinterpret the original generation trajectory as a series of pyramid stages that operate on compressed representations of different scales. Notably, the efficacy of image pyramids (Adelson et al., ) has been widely validated for discriminative neural networks (Lin et al., ; Wang et al., ) and more recently for diffusion models (Ho et al., ; Pernias et al., ; Teng et al., )and multimodal LLMs (Yu et al., ; Tian et al., ). Here, we investigate two types of pyramids: the spatial pyramid within a frame and the temporal one between consecutive frames (as illustrated in Fig. 1(b)). In such a pyramidal generation trajectory, only the final stage operates at full resolution, drastically reducing redundant computations in earlier timesteps. The main advantages are twofold: (1) The generation trajectories of different pyramid stages are interlinked, with the subsequent stage continuing to generate from the previous ones. This eliminates the need for each stage to regenerate from pure noise in some cascade models. (2) Instead of relying on separate models for each image pyramid, we integrate them into a single unified model for end-to-end optimization, which admits drastically-expedited training with more elegant implementation as validated by experiments.

Figure: (a) Video diffusion model like Sora (OpenAI, )
Refer to caption: x1.png

Based on the aforementioned pyramidal representations, we introduce a novel pyramidal flow matching algorithm that builds upon recent prevalent flow matching framework (Lipman et al., ; Liu et al., ; Albergo & Vanden-Eijnden, ). Specifically, we devise a piecewise flow for each pyramid resolution, which together form a generative process from noise to data. The flow within each pyramid stage takes a similar formulation, interpolating between a pixelated (compressed) and noisier latent and a pixelate-free (decompressed) and cleaner latent. Thanks to our sophisticated design, they can be jointly optimized by the unified flow matching objective in a single Diffusion Transformer (DiT) (Peebles & Xie, ), allowing simultaneous generation and decompression of visual content without multiple separate models. During inference, the output of each stage is renoised by a corrective Gaussian noise, which contributes to maintaining the continuity of the probability path between successive pyramid stages. Furthermore, we formulate the video generation in an autoregressive manner that iteratively predicts the next video latent conditioned on previously generated history. Given the high redundancy in the full-resolution history, we curate a temporal pyramid sequence using progressively compressed, lower-resolution history as conditions, thereby further reducing the token count and improving training efficiency.

The collaboration of the spatial and temporal pyramids results in remarkable training efficiency for video generation. Compared to the commonly used full-sequence diffusion, our method significantly reduces the number of video tokens during training (e.g., $\leq$15,360 tokens versus 119,040 tokens for a 10-second, 241-frame video), thereby reducing both computational resources required and training time. By training only on open-source datasets, our model generate high-quality 10-second videos at 768p resolution and 24 fps. The core contributions of this paper are summarized as follows:

- •
We present pyramidal flow matching, a novel video generative modeling algorithm that incorporates both spatial and temporal pyramid representations. Utilizing this framework can significantly improve training efficiency while maintaining good video generation quality.
- •
The proposed unified flow matching objective facilitates joint training of pyramid stages in a single Diffusion Transformer (DiT), avoiding the separate optimization of multiple models. The support for end-to-end training further enhances its simplicity and scalability.
- •
We evaluate its effectiveness on VBench (Huang et al., ) and EvalCrafter (Liu et al., ), with highly competitive performance among video generative models trained on public datasets.

## 2 Related Work

Video Generative Models
have seen rapid progress with autoregressive models (Yan et al., ; Hong et al., ; Kondratyuk et al., ; Jin et al., ) and diffusion models (Ho et al., ; Blattmann et al., ; ). A notable breakthrough is the high-fidelity video diffusion models (OpenAI, ; Kuaishou, ; Luma, ; Runway, ) by scaling up DiT pre-training (Peebles & Xie, ), but they induce significant training costs for long videos. An alternative line of research integrates diffusion models with autoregressive modeling (Chen et al., ; Valevski et al., ) to natively support long video generation, but is still limited in context length and training efficiency. Our work advances both approaches in terms of efficiency from a compression perspective, featuring a spatially compressed pyramidal flow and a temporally compressed pyramidal history.

Image Pyramids (Adelson et al., )
have been studied extensively in visual representation learning (Lowe, ; Dalal & Triggs, ; Lin et al., ; Wang et al., ). For generative models, the idea is explored by cascaded diffusion models that first generate at low resolution and then perform super-resolution (Ho et al., ; Saharia et al., ; Pernias et al., ; Teng et al., ). It is also extended to video by spatiotemporal super-resolution (Ho et al., ; Singer et al., ). However, they require training several separate models, which prevents knowledge sharing.
Possible unified modeling solutions for pyramids include hierarchical architectures (Rombach et al., ; Crowson et al., ; Hatamizadeh et al., ) or via next-token prediction (Yu et al., ; Tian et al., ), but involve architectural changes. Instead, we propose a simple flow matching objective that allows joint training of pyramids, thus facilitating efficient video generative modeling.

## 3 Method

This work proposes an efficient video generative modeling scheme named pyramidal flow matching. In the following text, we first extend the flow matching algorithm (Section 3.1) to an efficient spatial pyramid representation (Section 3.2). Then, a temporal pyramid design is proposed in Section 3.3 to further improve training efficiency. Lastly, practical implementations are discussed in Section 3.4.

### 3.1 Preliminaries on Flow Matching

Similar to diffusion models (Sohl-Dickstein et al., ; Song & Ermon, ; Ho et al., ), flow generative models (Papamakarios et al., ; Song et al., ; Xu et al., ; Lipman et al., ; Liu et al., ; Albergo & Vanden-Eijnden, ) aim to learn a velocity field ${\bm{v}}_{t}$ that maps random noise ${\bm{x}}_{0}\sim\mathcal{N}(\bm{0},{\bm{I}})$ to data samples ${\bm{x}}_{1}\sim q$ via an ordinary differential equation (ODE):

$$ $\frac{d{\bm{x}}_{t}}{dt}={\bm{v}}_{t}({\bm{x}}_{t}).$ (1) $$

Recently, Lipman et al. (); Liu et al. (); Albergo & Vanden-Eijnden () proposed the flow matching framework, which provides a simple simulation-free training objective for flow generative models by directly regressing the velocity ${\bm{v}}_{t}$ on a conditional vector filed ${\bm{u}}_{t}(\cdot|{\bm{x}}_{1})$:

$$ $\mathbb{E}_{t,q({\bm{x}}_{1}),p_{t}({\bm{x}}_{t}|{\bm{x}}_{1})}\big{\|}{\bm{v} }_{t}({\bm{x}}_{t})-{\bm{u}}_{t}({\bm{x}}_{t}|{\bm{x}}_{1})\big{\|}^{2},$ (2) $$

where ${\bm{u}}_{t}(\cdot|{\bm{x}}_{1})$ uniquely determines a conditional probability path $p_{t}(\cdot|{\bm{x}}_{1})$ toward data sample ${\bm{x}}_{1}$. An effective choice of the conditional probability path is linear interpolation of data and noise:

$$ $\displaystyle{\bm{x}}_{t}$ $\displaystyle=t{\bm{x}}_{1}+(1-t){\bm{x}}_{0},$ (3) $\displaystyle{\bm{x}}_{t}$ $\displaystyle\sim\mathcal{N}(t{\bm{x}}_{1},(1-t)^{2}{\bm{I}}),$ (4) $$

and ${\bm{u}}({\bm{x}}_{t}|{\bm{x}}_{1})={\bm{x}}_{1}-{\bm{x}}_{0}$. Notably, flow matching can be flexibly extended to interpolate between distributions other than standard Gaussians. This enables us to devise a sophisticated flow matching algorithm that specializes in reducing the computational cost of video generative modeling.

### 3.2 Pyramidal Flow Matching

The main challenge in video generative modeling is the spatio-temporal complexity, and we address its spatial complexity first. According to previous key observation in Fig. 1, the initial generation steps are usually very noisy and less informative, and thus may not need to operate at full resolution. This motivates us to study a spatially compressed pyramidal flow, illustrated in Fig. 2.

To alleviate redundant computation in early steps, we interpolate flow between data and compressed low-resolution noise. Let $\oplus$ denote the interpolation between latents of different resolutions, and let there be $K$ resolutions, each halving the previous one, then our flow may be expressed as:

$$ $\hat{\bm{x}}_{t}=t{\bm{x}}_{1}\oplus(1-t)\,\mathit{Down}({\bm{x}}_{0},2^{K}),$ (5) $$

where $\mathit{Down}(\cdot,\cdot)$ is a downsampling function. Since the interpolation concerns varying-dimensional ${\bm{x}}_{t}$, we decompose it as a piecewise flow (Yan et al., ) that divides $[0,1]$ into $K$ time windows, where each window interpolates between successive resolutions. For the $k$-th time window $[s_{k},e_{k}]$, let $t^{\prime}=(t-s_{k})/(e_{k}-s_{k})$ denote the rescaled timestep, then the flow within it follows:

$$ $\hat{\bm{x}}_{t}=t^{\prime}\,\mathit{Down}({\bm{x}}_{e_{k}},{2^{k}})+(1-t^{ \prime})\,\mathit{Up}(\mathit{Down}({\bm{x}}_{s_{k}},{2^{k+1}})),$ (6) $$

where $\mathit{Up}(\cdot)$ is an upsampling function. This way, only the last stage is performed at full resolution, while most stages are performed at lower resolutions using less computation. Under a uniform stage partitioning, the idea of spatial pyramid reduces the computational cost to a factor of nearly $1/K$. Below, we describe the instantiation of pyramidal flow from training and inference, respectively.

#### 3.2.1 Unified Training

In the construction of pyramidal flow, our main concern is unified modeling of different stages, as previous works (Ho et al., ; Pernias et al., ; Teng et al., ) all require training multiple models for separate generation and super-resolution, which hinders knowledge sharing.

Figure: (a) Unified modeling of pyramidal flow
Refer to caption: x2.png

To unify the objectives of generation and decompression/super-resolution, we curate the probability path by interpolating between different noise levels and resolutions. It starts with a more noisy and pixelated latent upsampled from a lower resolution, and yields cleaner and fine-grained results at a higher resolution, as illustrated in Fig. 2(a). Formally, the conditional probability path is defined by:

$$ $\displaystyle\text{End:}\qquad\hat{\bm{x}}_{e_{k}}$ $\displaystyle\sim\mathcal{N}(e_{k}\,\mathit{Down}({\bm{x}}_{1},{2^{k}}),(1-e_{ k})^{2}{\bm{I}}),$ (7) $\displaystyle\text{Start:}\qquad\hat{\bm{x}}_{s_{k}}$ $\displaystyle\sim\mathcal{N}(s_{k}\,\mathit{Up}(\mathit{Down}({\bm{x}}_{1},{2^ {k+1}})),(1-s_{k})^{2}{\bm{I}}),$ (8) $$

where $s_{k}<e_{k}$, and the upsampling and downsampling functions for the clean ${\bm{x}}_{1}$ are well defined, *e.g*., by nearest or bilinear resampling. In addition, to enhance the straightness of the flow trajectory, we couple the sampling of its endpoints by enforcing the noise to be in the same direction. Namely, we first sample a noise ${\bm{n}}\sim\mathcal{N}(\bm{0},{\bm{I}})$ and then jointly compute the endpoints $(\hat{\bm{x}}_{e_{k}},\hat{\bm{x}}_{s_{k}})$ as:

$$ $\displaystyle\text{End:}\qquad\hat{\bm{x}}_{e_{k}}$ $\displaystyle=e_{k}\,\mathit{Down}({\bm{x}}_{1},{2^{k}})+(1-e_{k}){\bm{n}},$ (9) $\displaystyle\text{Start:}\qquad\hat{\bm{x}}_{s_{k}}$ $\displaystyle=s_{k}\,\mathit{Up}(\mathit{Down}({\bm{x}}_{1},{2^{k+1}}))+(1-s_{ k}){\bm{n}}.$ (10) $$

Thereafter, we can regress the flow model ${\bm{v}}_{t}$ on the conditional vector field ${\bm{u}}_{t}(\hat{\bm{x}}_{t}|{\bm{x}}_{1})=\hat{\bm{x}}_{e_{k}}-\hat{\bm{x}}_
{s_{k}}$ with the following flow matching objective to unify generation and decompression:

$$ $\mathbb{E}_{k,t,(\hat{\bm{x}}_{e_{k}},\hat{\bm{x}}_{s_{k}})}\big{\|}{\bm{v}}_{ t}(\hat{\bm{x}}_{t})-(\hat{\bm{x}}_{e_{k}}-\hat{\bm{x}}_{s_{k}})\big{\|}^{2}.$ (11) $$

#### 3.2.2 Inference with Renoising

During inference, standard sampling algorithms can be applied within each pyramid stage. However, we must carefully handle the jump points (Campbell et al., ) between successive pyramid stages of different resolutions to ensure continuity of the probability path.

To ensure continuity, we first upsample the previous low-resolution endpoint with nearest or bilinear resampling. The result, as a linear combination of the input, follows a Gaussian distribution:

$$ $\mathit{Up}(\hat{\bm{x}}_{e_{k+1}})\sim\mathcal{N}(e_{k+1}\,\mathit{Up}( \mathit{Down}({\bm{x}}_{1},{2^{k+1}})),(1-e_{k+1})^{2}\,{\bm{\Sigma}}),$ (12) $$

where ${\bm{\Sigma}}$ is a covariance matrix depending on the upsampling function. Comparing Eqs. 8 and 12, we find it possible to match the Gaussian distributions at each jump point by a linear transformation of the upsampled result. Specifically, the following rescaling and renoising scheme would suffice:

$$ $\hat{\bm{x}}_{s_{k}}=\frac{s_{k}}{e_{k+1}}\,\mathit{Up}(\hat{\bm{x}}_{e_{k+1}} )+\alpha{\bm{n}}^{\prime},\quad\text{s.t. }{\bm{n}}^{\prime}\sim\mathcal{N}( \bm{0},{\bm{\Sigma}}^{\prime}),$ (13) $$

where the rescaling coefficient $s_{k}/e_{k+1}$ allows matching the means of these distributions, and the corrective noise ${\bm{n}}^{\prime}$ with a weight of $\alpha$ allows matching their covariance matrices.

Figure: Algorithm 1 Sampling with Pyramidal Flow Matching

To derive the corrective noise and its covariance, we consider a simplest scenario with nearest neighbor upsampling. In this case, ${\bm{\Sigma}}$ has a blockwise structure with non-zero elements only in the $4\times 4$ blocks along the diagonal (corresponding to those upsampled from the same pixel). Then, it can be inferred that the corrective noise’s covariance matrix ${\bm{\Sigma}}^{\prime}$ also has a blockwise structure:

$$ ${\bm{\Sigma}}_{\mathit{block}}=\begin{pmatrix}1&1&1&1\\ 1&1&1&1\\ 1&1&1&1\\ 1&1&1&1\\ \end{pmatrix}\;\Rightarrow\;{\bm{\Sigma}}^{\prime}_{\mathit{block}}=\begin{ pmatrix}1&\gamma&\gamma&\gamma\\ \gamma&1&\gamma&\gamma\\ \gamma&\gamma&1&\gamma\\ \gamma&\gamma&\gamma&1\\ \end{pmatrix},$ (14) $$

where ${\bm{\Sigma}}^{\prime}_{\mathit{block}}$ contains negative elements $\gamma\in[-1/3,0]$(^1^11The lower bound $-1/3$ ensures that the covariance matrix is semidefinite.) to reduce the correlation within each block, as illustrated in Fig. 2(b). Since it is desirable to maximally preserve the signals at each jump point, we opt to add a small amount of noise with $\gamma=-1/3$ such that it is most specialized for decorrelation. Substituting this into the above gives the update rule at jump points (see Appendix A for derivations):

$$ $\hat{\bm{x}}_{s_{k}}=\frac{1+s_{k}}{2}\,\mathit{Up}(\hat{\bm{x}}_{e_{k+1}})+ \frac{\sqrt{3}(1-s_{k})}{2}{\bm{n}}^{\prime},$ (15) $$

with $e_{k+1}=2s_{k}/(1+s_{k})$. The resulting inference process with renoising is shown in Algorithm 1.

### 3.3 Pyramidal Temporal Condition

Beyond the spatial complexity addressed in above sections, video presents another significant challenge due to its temporal length. The prevailing full-sequence diffusion methods generate all video frames simultaneously, restricting them to fixed-length generation (consistent with training). In contrast, the autoregressive video generation paradigm supports flexible-length generation during inference. Recent advancements (Chen et al., ; Valevski et al., ) have also demonstrated its effectiveness in creating long-duration video content. However, their training is still severely limited by the computational complexity arising from the full-resolution long-history condition.

Figure: (a) Temporal pyramids rearranged in rows
Refer to caption: x3.png

We observe that there is a high redundancy in full-resolution history conditions. For example, earlier frames in a video tend to provide high-level semantic conditions and are less related to appearance details. This motivates us to use compressed, lower-resolution history for autoregressive video generation. As shown in Fig. 3(a), we adopt a history condition of gradually increasing resolutions:

$$ $\underbrace{\ldots\rightarrow\mathit{Down}({\bm{x}}^{i-2}_{t^{\prime}},2^{k+1} )\rightarrow\mathit{Down}({{\bm{x}}}^{i-1}_{t^{\prime}},2^{k})}_{\text{History condition}}\rightarrow\underset{\text{Training}}{\underset{\uparrow}{\hat{\bm {x}}^{i}_{t}}},$ (16) $$

where the superscripts are the history latent index, and the subscript $t^{\prime}$ indicates small noise added to history latents in training to mitigate error accumulation with autoregressive generation, as in (Chen et al., ; Valevski et al., ). After training, we use clean generated frames for inference:

$$ $\underbrace{\ldots\rightarrow\mathit{Down}({\bm{x}}^{i-2}_{1},2^{k+1}) \rightarrow\mathit{Down}({{\bm{x}}}^{i-1}_{1},2^{k})}_{\text{History condition }}\rightarrow\underset{\text{Prediction}}{\underset{\uparrow}{\hat{\bm{x}}^{i} _{t}}}.$ (17) $$

The above design significantly reduces the computational and memory overhead of video generative pre-training. Let there be $T$ history latents over $K$ lower resolutions, then most frames are computed at the lowest resolution of $1/2^{K}$, which reduces the number of training tokens by up to $1/4^{K}$ times. As a result, training efficiency is improved by up to $16^{K}/T$ times.

### 3.4 Practical Implementation

In this section, we show that the above pyramid designs can be easily implemented using standard Transformer architecture (Vaswani et al., ) and pipelines. This is crucial for efficient and scalable video generative pre-training based on existing acceleration frameworks.

Unlike previous methods (Ma et al., ) that utilize factorized spatial and temporal attention to reduce computational complexity, we directly employ full sequence attention, thanks to much fewer tokens required by our pyramidal representation. Furthermore, blockwise causal attention is adopted in each transformer layer, ensuring that each token cannot attend to its subsequent frames. The ablation results in Section C.2 illustrate that such casual attention design is crucial for autoregressive video generation. Another important design choice is the position encoding, as the pyramid designs introduce multiple spatial resolutions. As shown in Fig. 3(b), we extrapolate position encoding in the spatial pyramid for better fine-grained detail (Yang et al., ), while interpolating it in the temporal pyramid input to spatially align the history conditions.

During training, different pyramidal stages are uniformly sampled in each update iteration. The autoregressive nature of our method inherently supports joint training of images and videos, since the first frame in a video acts as an image. We pack training samples with varying token counts together to form the length-balanced training batch following Patch n’ Pack (Dehghani et al., ). After training, our method natively possesses the capability of text-to-video and text-conditioned image-to-video generation. During inference sampling, the classifier-free guidance strategy can be employed to enhance temporal consistency and motion smoothness of the generated video.

## 4 Experiments

### 4.1 Experimental Settings

Training Dataset.
Our model is trained on a mixed corpus of open-source image and video datasets. For images, we utilize a high-aesthetic subset of LAION-5B (Schuhmann et al., ), 11M from CC-12M (Changpinyo et al., ), 6.9M non-blurred subset of SA-1B (Kirillov et al., ), 4.4M from JourneyDB (Sun et al., ), and 14M publicly available synthetic data. For video data, we incorporate the WebVid-10M (Bain et al., ), OpenVid-1M (Nan et al., ), and another 1M high-resolution non-watermark video primarily from the Open-Sora Plan (PKU-Yuan Lab et al., ). After postprocessing, around 10M single-shot videos are available for training.

Evaluation Metrics.
We utilize the VBench (Huang et al., ) and EvalCrafter (Liu et al., ) for quantitative performance evaluation. VBench is a comprehensive benchmark that includes 16 fine-grained dimensions to systematically measure both motion quality and semantic alignment of video generative models. EvalCrafter is another large-scale evaluation benchmark including around 17 objective metrics for assessing video generation capabilities. In addition to automated evaluation metrics, we also conducted a study with human participants to measure the human preference for our generated videos. The compared baselines are summarized in Appendix B.

Implementation Details.
We utilize the prevailing MM-DiT architecture from SD3 Medium (Esser et al., ) as the base model, with 2B parameters in total. It employs sinusoidal position encoding (Vaswani et al., ) in the spatial dimensions. As for the temporal dimension, the 1D Rotary Position Embedding (RoPE) (Su et al., ) is added to support flexible training with different video durations. In addition, we use a 3D Variational Autoencoder (VAE) to compress videos both spatially and temporally with a downsampling ratio of $8\times 8\times 8$. It shares a similar structure with MAGVIT-v2 (Yu et al., ) and is trained from scratch on the WebVid-10M dataset (Bain et al., ). The number of pyramid stages is set to 3 in all the experiments. Following Valevski et al. (), we add some corruptive noise of strength uniformly sampled from $[0,1/3]$ to the history pyramid conditions, which is critical for mitigating the autoregressive generation degradation.

### 4.2 Efficiency

The proposed pyramidal flow matching framework significantly reduces the computational and memory overhead in video generation training. Consider a video with $T$ frame latents, where each frame contains $N$ tokens at the original resolution. The full-sequence diffusion has $TN$ input tokens in DiT and requires $T^{2}N^{2}$ computations. In contrast, our method uses only approximately $TN/4^{K}$ tokens and $T^{2}N^{2}/16^{K}$ computations even for the final pyramid stage, which significantly improves the training efficiency. Specifically, it takes only 20.7k A100 GPU hours to train a 10s video generation model with 241 frames. Compared to existing models that require significant training resources, our method achieves superior video generation performance with much fewer computations. For example, the Open-Sora 1.2 (Zheng et al., ) requires 4.8k Ascend and 37.8k H100 hours to train the generation of only 97 video frames, consuming more than two times the computation of our approach, yet producing videos of worse quality. At inference, our model takes just 56 seconds to create a 5-second, 384p video clip, which is comparable to full-sequence diffusion counterparts.

**Table 1: Experimental results on VBench (Huang et al., ). In terms of total score and quality score, our model even outperforms CogVideoX-5B (Yang et al., ) with twice the model size. In the following tables, we use blue to denote the highest scores among models trained on public data.**
| Model | Public<br>Data | Total<br>Score | Quality<br>Score | Semantic<br>Score | Motion<br>Smoothness | Dynamic<br>Degree |
| --- | --- | --- | --- | --- | --- | --- |
| Gen-2 | $\times$ | 80.58 | 82.47 | 73.03 | 99.58 | 18.89 |
| Pika 1.0 | $\times$ | 80.69 | 82.92 | 71.77 | 99.50 | 47.50 |
| CogVideoX-2B | $\times$ | 80.91 | 82.18 | 75.83 | 97.73 | 59.86 |
| CogVideoX-5B | $\times$ | 81.61 | 82.75 | 77.04 | 96.92 | 70.97 |
| Kling | $\times$ | 81.85 | 83.38 | 75.68 | 99.40 | 46.94 |
| Gen-3 Alpha | $\times$ | 82.32 | 84.11 | 75.17 | 99.23 | 60.14 |
| Open-Sora Plan v1.1 | ✓ | 78.00 | 80.91 | 66.38 | 98.28 | 47.72 |
| Open-Sora 1.2 | ✓ | 79.76 | 81.35 | 73.39 | 98.50 | 42.39 |
| VideoCrafter2 | ✓ | 80.44 | 82.20 | 73.42 | 97.73 | 42.50 |
| T2V-Turbo | ✓ | 81.01 | 82.57 | 74.76 | 97.34 | 49.17 |
| Ours | ✓ | 81.72 | 84.74 | 69.62 | 99.12 | 64.63 |

**Table 2: Experimental results on EvalCrafter (Liu et al., ). See Section C.1 for raw metrics.**
| Model | Public<br>Data | Final Sum<br>Score | Visual<br>Quality | Text-Video<br>Alignment | Motion<br>Quality | Temporal<br>Consistency |
| --- | --- | --- | --- | --- | --- | --- |
| Pika 1.0 | $\times$ | 250 | 63.05 | 66.97 | 56.43 | 63.81 |
| Gen-2 | $\times$ | 254 | 69.09 | 63.92 | 55.59 | 65.40 |
| ModelScope | ✓ | 218 | 53.09 | 54.46 | 52.47 | 57.80 |
| Show-1 | ✓ | 229 | 52.19 | 62.07 | 53.74 | 60.83 |
| LaVie | ✓ | 234 | 57.99 | 68.49 | 52.83 | 54.23 |
| VideoCrafter2 | ✓ | 243 | 63.98 | 63.16 | 54.82 | 61.46 |
| Ours | ✓ | 244 | 67.94 | 57.01 | 55.31 | 63.41 |

### 4.3 Main Results

Text-to-Video Generation.
We first evaluate the text-to-video generation capability of the proposed method. For each text prompt, a 5-second 121 frames video is generated for evaluation. The detailed quantitative results on VBench (Huang et al., ) and EvalCrafter (Liu et al., ) are summarized in Tables 1 and 2, respectively. Overall, our method surpasses all the compared open-sourced video generation baselines in these two benchmarks. Even with only publicly accessible video data in training, it achieves comparable performance to commercial competitors trained on much larger proprietary data like Kling (Kuaishou, ) and Gen-3 Alpha (Runway, ). In particular, we demonstrated exceptional performance in quality score (84.74 vs. 84.11 of Gen-3), and motion smoothness in VBench, which are crucial criteria in reflecting the visual quality of generated videos. When evaluated in EvalCrafter, our method achieves better visual and motion quality scores than most compared methods. The semantic score is relatively lower than others, mainly because we use coarse-grained synthetic captions,

Figure: Figure 4: User preference on sampled VBench prompts. Our videos are generated at 5s, 768p, 24fps.
Refer to caption: x4.png

which can be improved with more accurate video captioning.
We also present some generated 5–10 second videos in Fig. 5, showing cinematic visual quality and validate the efficacy of pyramidal flow matching. More visualizations are provided in Section C.3.

User study.
While quantitative evaluation scores reflect the video generation capability to some extent, they may not align with human preferences for visual quality. Hence, an additional user study is conducted to compare our performance with six baseline models, including CogVideoX (Yang et al., ) and Kling (Kuaishou, ). We utilized 50 prompts sampled from VBench and asked 20+ participants to rank each model according to the aesthetic quality, motion smoothness, and semantic alignment of the generated videos. As seen in Fig. 4, our method is preferred over open-source models such as Open-Sora and CogVideoX-2B especially in terms of motion smoothness. This is due to the substantial token savings achieved by pyramidal flow matching, enabling generation of 5-second (up to 10-second) 768p videos at 24 fps, while the baselines usually support video synthesis of similar length only at 8 fps. The detailed user study settings are presented in Appendix B.

Image-to-Video Generatetion.
Thanks to the autoregressive property of our model and the causal attention design, the first frame of each video acts similarly to an image condition during the training. Consequently, although our model is optimized solely for text-to-video generation, it naturally accommodates text-conditioned image-to-video generation during inference. Given an image and a textual prompt, it is able to animate the static input image by autoregressively predicting the future frames without further fine-tuning. In Fig. 6, we illustrate qualitative examples of its image-to-video generation performance, where each example consists of 120 newly synthesized frames spanning a duration of 5 seconds. As can be seen, our model successfully predicts reasonable subsequent motion, endowing the images with rich temporal dynamic information. More generated video examples are best viewed on our project page at [https://pyramid-flow.github.io](https://pyramid-flow.github.io).

Figure: (a) The Glenfinnan Viaduct is a historic railway bridge…It is a stunning sight as a steam train leaves the bridge, traveling over the arch-covered viaduct. The landscape is dotted with lush greenery and rocky mountains…
Refer to caption: extracted/5910391/figs/t2v/rail/0.jpg

Figure: (a) A moon rises from the sky and the lights on the land are bright.
Refer to caption: extracted/5910391/figs/i2v/singapore/0.jpg

Figure: Figure 7: Ablation study of spatial pyramid at 50k image training step. On the right is a quantitative comparison of the FID results, where our method achieves almost three times the convergence speed.
Refer to caption: extracted/5910391/figs/ablation/1/baseline-2.jpg

Figure: Figure 8: Ablation study of temporal pyramid at 100k low-resolution video training step.
Refer to caption: extracted/5910391/figs/ablation/2/baseline-1/0.jpg

### 4.4 Ablation Study

In this section, we conduct ablation studies to validate the crucial component of our methods, including the spatial pyramid in denoising trajectory and the temporal pyramid in history condition. Due to limited space, the ablations for other design choices are provided in Section C.2.

Effectiveness of spatial pyramid.
In the generation trajectory of the proposed spatial pyramid, only the final stage operates at full resolution, which significantly reduces the number of tokens for most denoising timesteps. With the same computational resources, it can handle more samples per training batch, greatly enhancing the convergence rate. To validate its efficiency, we designed a baseline that employs the standard flow matching objective for training text-to-image generation in our early experiments. This baseline is optimized using the same training data, number of tokens per batch, hyperparameter configurations, and model architecture to ensure fairness. The performance comparison is illustrated in Fig. 7. It can be observed that the variant using pyramidal flow demonstrates superior visual quality and prompt-following capability. We further quantitatively evaluate the FID metric of these methods on the MS-COCO benchmark (Lin et al., ) by randomly sampling 3K prompts. The FID performance curve over training steps is presented on the right of Fig. 7. Compared to standard flow matching, the convergence rate of our method is significantly improved.

Effectiveness of temporal pyramid.
As mentioned in Section 4.2, the temporal pyramid design can drastically reduce the computation demands compared to traditional full-sequence diffusion. Similar to the spatial pyramid, we also established a full-sequence diffusion baseline under the same experimental setting to investigate its training efficiency improvement. The qualitative comparison with the baseline is presented in Fig. 8, where the generated videos of our pyramidal variant demonstrate much better visual quality and temporal consistency under the same training steps. In contrast, the full-sequence diffusion baseline is far from convergence. It fails to produce coherent motion, leading to fragmented visual details and severe artifacts in the generated videos. This performance gap clearly highlights the training acceleration achieved by our method in video generative modeling.

## 5 Conclusion

This work presents an efficient video generative modeling framework based on pyramidal visual representations. In contrast to cascaded diffusion models that use separate models for different image pyramids to improve efficiency, we propose a unified pyramidal flow matching objective that simultaneously generates and decompresses visual content across pyramid stages with a single model, effectively facilitating knowledge sharing. Furthermore, a temporal pyramid design is introduced to reduce computational redundancy in the full-resolution history of a video. The proposed method is extensively evaluated on VBench and EvalCrafter, demonstrating its advantageous performance.
All code and model weights will be open-sourced at [https://pyramid-flow.github.io](https://pyramid-flow.github.io).

Acknowledgements.
We thank Jinghan Li for assisting with the user study.

## Appendix A Derivation

This section provides detailed derivation for Eq. 15 that handles jump points in the spatial pyramid.

To ensure continuity of the probability path across different stages of the spatial pyramid, we need to make sure that the endpoints have the same probability distribution. According to Eqs. 8 and 12, their distributions are already similar after a simple upsampling transformation:

$$ $\displaystyle\hat{\bm{x}}_{s_{k}}$ $\displaystyle\sim\mathcal{N}(s_{k}\,\mathit{Up}(\mathit{Down}({\bm{x}}_{1},{2^ {k+1}})),(1-s_{k})^{2}{\bm{I}}),$ (18) $\displaystyle\mathit{Up}(\hat{\bm{x}}_{e_{k+1}})$ $\displaystyle\sim\mathcal{N}(e_{k+1}\,\mathit{Up}(\mathit{Down}({\bm{x}}_{1},{ 2^{k+1}})),(1-e_{k+1})^{2}\,{\bm{\Sigma}}).$ (19) $$

Therefore, we can directly apply a linear transformation with a corrective Gaussian noise to match their distributions:

$$ $\hat{\bm{x}}_{s_{k}}=\frac{s_{k}}{e_{k+1}}\,\mathit{Up}(\hat{\bm{x}}_{e_{k+1}} )+\alpha{\bm{n}}^{\prime},\quad\text{s.t. }{\bm{n}}^{\prime}\sim\mathcal{N}( \bm{0},{\bm{\Sigma}}^{\prime}),$ (20) $$

where the rescaling coefficient $s_{k}/e_{k+1}$ allows the means of these distributions to be matched, and $\alpha$ is the noise weight. Additionally, we need to match the covariance matrices of Eqs. 20 and 18:

$$ $\frac{s_{k}^{2}}{e_{k+1}^{2}}(1-e_{k+1})^{2}{\bm{\Sigma}}+\alpha^{2}{\bm{ \Sigma}}^{\prime}=(1-s_{k})^{2}{\bm{I}}.$ (21) $$

To allow analysis of covariance matrices, *e.g*. ${\bm{\Sigma}}$, we consider a simplest scenario with nearest neighbor upsampling. In this case, ${\bm{\Sigma}}$ has a blockwise structure with non-zero elements only in the $4\times 4$ blocks along the diagonal (corresponding to those upsampled from the same pixel). Then, it can be inferred that the corrective noise’s covariance matrix ${\bm{\Sigma}}^{\prime}$ has a similar blockwise structure:

$$ ${\bm{\Sigma}}_{\mathit{block}}=\begin{pmatrix}1&1&1&1\\ 1&1&1&1\\ 1&1&1&1\\ 1&1&1&1\\ \end{pmatrix}\;\Rightarrow\;{\bm{\Sigma}}^{\prime}_{\mathit{block}}=\begin{ pmatrix}1&\gamma&\gamma&\gamma\\ \gamma&1&\gamma&\gamma\\ \gamma&\gamma&1&\gamma\\ \gamma&\gamma&\gamma&1\\ \end{pmatrix},$ (22) $$

where $\gamma$ is a negative value in $[-1/3,0]$ for the decorrelation (its lower bound $-1/3$ ensures that the covariance matrix is semidefinite). We further rewrite Eqs. 22 and 21 by considering the equality of their diagonal and non-diagonal elements, respectively:

$$ $\displaystyle\frac{s_{k}^{2}}{e_{k+1}^{2}}(1-e_{k+1})^{2}+\alpha^{2}=(1-s_{k}) ^{2},$ (23) $\displaystyle\frac{s_{k}^{2}}{e_{k+1}^{2}}(1-e_{k+1})^{2}+\alpha^{2}\gamma=0.$ (24) $$

Taking into account the timestep constraints $0<s_{k},e_{k+1}<1$, they can be solved directly:

$$ $e_{k+1}=\frac{s_{k}\sqrt{1-\gamma}}{(1-s_{k})\sqrt{-\gamma}+s_{k}\sqrt{1- \gamma}},\quad\alpha=\frac{1-s_{k}}{\sqrt{1-\gamma}}.$ (25) $$

Intuitively, it is desirable to maximally preserve the signals at each jump point, which corresponds to minimizing the noise weight $\alpha$. According to Eq. 25, this is equivalent to minimizing $\gamma$. Substituting its minimum value $\gamma=-1/3$ into Eq. 25 yields:

$$ $e_{k+1}=\frac{2s_{k}}{1+s_{k}},\quad\alpha=\frac{\sqrt{3}(1-s_{k})}{2}.$ (26) $$

It is worth noting that $e_{k+1}>s_{k}$, indicating that the timestep is rolled back a bit when adding the corrective noise at each jump point. We can further obtain the renoising rule in Eq. 15:

$$ $\hat{\bm{x}}_{s_{k}}=\frac{1+s_{k}}{2}\,\mathit{Up}(\hat{\bm{x}}_{e_{k+1}})+ \frac{\sqrt{3}(1-s_{k})}{2}{\bm{n}}^{\prime}.$ (27) $$

## Appendix B Experimental Settings

Model Implementation Details.
We adopt the MM-DiT architecture, based on SD3 Medium (Esser et al., ), which comprises 24 transformer layers and a total of 2B parameters. The weights of the MM-DiT are initialized from the SD3 medium. Following the more recent FLUX.1 (Black Forest Labs, ), both T5 (Raffel et al., ) and CLIP (Radford et al., ) encoders are employed for prompts embedding. To address the redundancy in video data, we have designed a 3D VAE that compresses videos both spatially and temporally into a latent space. The architecture of this VAE is similar to MAGVIT-v2 (Yu et al., ), employing 3D causal convolution to ensure that each frame depends only on the preceding frames. It features an asymmetric encoder-decoder with Kullback-Leibler (KL) regularization applied to the latents. Overall, the 3D VAE achieves a compression rate of $8\times 8\times 8$ from pixels to the latent. It is trained on WebVid-10M and 6.9M SAM images from scratch. To support the tokenization of very long videos, we scatter them into multiple GPUs to distribute computation like CogVideoX (Yang et al., ).

Training Procedure
Our model undergoes a three-stage training procedure using 128 NVIDIA A100 GPUs.
(1) Image Training. In the first stage, we utilize a pure image dataset that includes 180M images from LAION-5B (Schuhmann et al., ), 11M from CC-12M (Changpinyo et al., ), 6.9M non-blurred images from SA-1B (Kirillov et al., ), and 4.4M from JourneyDB (Sun et al., ). We keep the image’s original aspect ratio and rearrange them into different buckets. It is trained for a total of 50,000 steps, requiring approximately 1536 A100 GPU hours. After this stage, the model has learned the dependencies between visual pixels, which facilitates the convergence of subsequent video training.
(2) Low-Resolution Video Training. For this stage, we employ the WebVid-10M (Bain et al., ), OpenVid-1M (Nan et al., ), and another 1M non-watermark video from the Open-Sora Plan (PKU-Yuan Lab et al., ). We also leverage the Video-LLaMA2 (Cheng et al., ), a state-of-the-art video understanding model, to recaption each video sample. The image data from stage 1 is also utilized at a proportion of 12.5% in each batch. We first train the model for 80,000 steps on 2-second video generation, followed by an additional 120,000 steps on 5-second videos. In total, it takes about 11,520 A100 GPU hours at this stage.
(3) High-Resolution Video Training. The final stage employs the same strategy to continue fine-tuning the model on the aforementioned high-resolution video dataset of varying durations (5–10s). It consumes approximately 7,680 A100 GPU hours for 50,000 steps in the final stage.

Hyperparameters Setting
The detailed training hyper-parameter settings for each optimization stage are reported in Table 3.

**Table 3: The detailed training hyperparameters of our method**
| Configuration | Stage-1 | Stage-2 | Stage-3 |
| --- | --- | --- | --- |
| Optimizer | AdamW | AdamW | AdamW |
| Optimizer Hyperparameters | $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e^{-6}$ | $\beta_{1}=0.9$, $\beta_{2}=0.95$, $\epsilon=1e^{-6}$ |  |
| Global batch size | 1536 | 768 | 384 |
| Learning rate | 1e-4 | 1e-4 | 5e-5 |
| Learning rate schedule | Constant with warmup | Constant with warmup | Constant with warmup |
| Training Steps | 50k | 200k | 50k |
| Warm-up steps | 1k | 1k | 1k |
| Weight decay | 1e-4 | 1e-4 | 1e-4 |
| Gradient clipping | 1.0 | 1.0 | 1.0 |
| Numerical precision | bfloat16 | bfloat16 | bfloat16 |
| GPU Usage | 128 NVIDIA A100 | 128 NVIDIA A100 | 128 NVIDIA A100 |
| Training Time | 12h | 90h | 60h |

Baseline Methods.
For VBench (Huang et al., ), we compare with eight baseline methods, including Open-Sora Plan V1.1 (PKU-Yuan Lab et al., ), Open-Sora 1.2 (Zheng et al., ), VideoCrafter2 (Chen et al., ), Gen-2 (Runway, ), Pika 1.0 (Pika, ), T2V-Turbo (Li et al., ), CogVideoX (Yang et al., ), Kling (Kuaishou, ), and Gen-3 Alpha (Runway, ). Among them, Open-Sora Plan, Open-Sora, CogVideo-X, Kling and Gen-3 Alpha can generate long videos. For EvalCrafter (Liu et al., ), our model is compared to six baselines, including ModelScope (Wang et al., ), Show-1 (Zhang et al., ), LaVie (Wang et al., ), VideoCrafter2 (Chen et al., ), Pika 1.0 (Pika, ), and Gen-2 (Runway, ). The above models are all based on full-sequence diffusion, while our method combines the merits of autoregressive generation and flow generative models to achieve better training efficiency of video generation.

Figure: Figure 9: Interface for user study of video generative performance.
Refer to caption: extracted/5910391/figs/user/gradio.jpg

User Study.
To complement the quantitative evaluation in the main paper, we conduct a rigorous user study to collect human preferences for these generative models. To accomplish this, we sample 50 prompts from the VBench prompt list and randomly sample one generated video for each prompt from the baseline model. In total, six baseline models are considered, including Open-Sora Plan V1.1 (PKU-Yuan Lab et al., ), Open-Sora 1.2 (Zheng et al., ), Pika 1.0 (Pika, ), CogVideoX-2B and 5B (Yang et al., ), and Kling (Kuaishou, ). We then pair these results with our generated video and ask the participant to rank their preference among three dimensions: aesthetic quality, motion smoothness, and semantic alignment, each of which represents a crucial aspect of video quality. The interface for the user study is exemplified in Fig. 9, where the user accepts a prompt and two generated videos (with the unnecessary information cropped, such as a watermark indicating which model it belongs to), and chooses between which model is better in the three dimensions. We distribute the user study to more than 20 participants, and collect a total of 1411 valid preference choices, ensuring its effectiveness. The results of this user study are presented in Fig. 4, where our model shows a very competitive performance among the compared baselines.

## Appendix C Additional Results

### C.1 Quantitative Results

**Table 4: Detailed results on VBench (Huang et al., ). See Table 1 for the summarized results. We additionally use blue to indicate the highest scores among models trained on public datasets.**
| Model | Subject<br>Consistency | Background<br>Consistency | Temporal<br>Flickering | Motion<br>Smoothness | Dynamic<br>Degree | Aesthetic<br>Quality | Imaging<br>Quality | Object<br>Class |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Trained on private datasets: |  |  |  |  |  |  |  |  |
| Gen-2 | 97.61 | 97.61 | 99.56 | 99.58 | 18.89 | 66.96 | 67.42 | 90.92 |
| Pika 1.0 | 96.94 | 97.36 | 99.74 | 99.50 | 47.50 | 62.04 | 61.87 | 88.72 |
| CogVideoX-2B | 96.78 | 96.63 | 98.89 | 97.73 | 59.86 | 60.82 | 61.68 | 83.37 |
| CogVideoX-5B | 96.23 | 96.52 | 98.66 | 96.92 | 70.97 | 61.98 | 62.90 | 85.23 |
| Kling | 98.33 | 97.60 | 99.30 | 99.40 | 46.94 | 61.21 | 65.62 | 87.24 |
| Gen-3 Alpha | 97.10 | 96.62 | 98.61 | 99.23 | 60.14 | 63.34 | 66.82 | 87.81 |
| Trained on public datasets: |  |  |  |  |  |  |  |  |
| Open-Sora Plan v1.1 | 95.73 | 96.73 | 99.03 | 98.28 | 47.72 | 56.85 | 62.28 | 76.30 |
| Open-Sora 1.2 | 96.75 | 97.61 | 99.53 | 98.50 | 42.39 | 56.85 | 63.34 | 82.22 |
| VideoCrafter2 | 96.85 | 98.22 | 98.41 | 97.73 | 42.50 | 63.13 | 67.22 | 92.55 |
| T2V-Turbo | 96.28 | 97.02 | 97.48 | 97.34 | 49.17 | 63.04 | 72.49 | 93.96 |
| Ours | 96.95 | 98.06 | 99.49 | 99.12 | 64.63 | 63.26 | 65.01 | 86.67 |
| Model | Multiple<br>Objects | Human<br>Action | Color | Spatial<br>Relationship | Scene | Appearance<br>Style | Temporal<br>Style | Overall<br>Consistency |
| Trained on private datasets: |  |  |  |  |  |  |  |  |
| Gen-2 | 55.47 | 89.2 | 89.49 | 66.91 | 48.91 | 19.34 | 24.12 | 26.17 |
| Pika 1.0 | 43.08 | 86.2 | 90.57 | 61.03 | 49.83 | 22.26 | 24.22 | 25.94 |
| CogVideoX-2B | 62.63 | 98.0 | 79.41 | 69.90 | 51.14 | 24.80 | 24.36 | 26.66 |
| CogVideoX-5B | 62.11 | 99.4 | 82.81 | 66.35 | 53.20 | 24.91 | 25.38 | 27.59 |
| Kling | 68.05 | 93.4 | 89.90 | 73.03 | 50.86 | 19.62 | 24.17 | 26.42 |
| Gen-3 Alpha | 53.64 | 96.4 | 80.90 | 65.09 | 54.57 | 24.31 | 24.71 | 26.69 |
| Trained on public datasets: |  |  |  |  |  |  |  |  |
| Open-Sora Plan v1.1 | 40.35 | 86.8 | 89.19 | 53.11 | 27.17 | 22.90 | 23.87 | 26.52 |
| Open-Sora 1.2 | 51.83 | 91.2 | 90.08 | 68.56 | 42.44 | 23.95 | 24.54 | 26.85 |
| VideoCrafter2 | 40.66 | 95.0 | 92.92 | 35.86 | 55.29 | 25.13 | 25.84 | 28.23 |
| T2V-Turbo | 54.65 | 95.2 | 89.90 | 38.67 | 55.58 | 24.42 | 25.51 | 28.16 |
| Ours | 50.71 | 85.6 | 82.87 | 59.53 | 43.20 | 20.91 | 23.09 | 26.23 |

**Table 5: Raw metrics on EvalCrafter (Liu et al., ). The baseline results are found on its website, but there were no results for LaVie (Wang et al., ). See Table 2 for the summarized results.**
| Model | VQAA | VQAT | IS | CLIP-<br>Temp | Warping<br>Error | Face<br>Consistency | Action-<br>Score | Motion<br>AC-Score |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Trained on private datasets: |  |  |  |  |  |  |  |  |  |
| Pika 1.0 | 69.23 | 71.12 | 16.67 | 99.89 | 0.0008 | 99.22 | 61.29 | 42.0 |  |
| Gen-2 | 90.39 | 92.18 | 19.28 | 99.99 | 0.0005 | 99.35 | 73.44 | 44.0 |  |
| Trained on public datasets: |  |  |  |  |  |  |  |  |  |
| ModelScope | 40.06 | 32.93 | 17.64 | 99.74 | 0.0162 | 98.94 | 72.12 | 42.0 |  |
| Show-1 | 23.19 | 44.24 | 17.65 | 99.77 | 0.0067 | 99.32 | 81.56 | 50.0 |  |
| VideoCrafter2 | 79.93 | 67.04 | 17.39 | 99.84 | 0.0085 | 99.44 | 68.17 | 36.0 |  |
| Ours | 86.09 | 88.31 | 18.49 | 99.90 | 0.0019 | 98.89 | 67.58 | 46.0 |  |
| Model | Flow-<br>Score | CLIP-<br>Score | BLIP-<br>BLUE | SD-<br>Score | Detection-<br>Score | Color-<br>Score | Count-<br>Score | OCR-<br>Score | Celebrity<br>ID Score |
| Trained on private datasets: |  |  |  |  |  |  |  |  |  |
| Pika 1.0 | 1.14 | 20.47 | 21.31 | 67.43 | 70.26 | 42.03 | 62.19 | 94.85 | 36.53 |
| Gen-2 | 0.58 | 20.26 | 22.25 | 67.69 | 69.54 | 47.39 | 58.36 | 63.74 | 38.90 |
| Trained on public datasets: |  |  |  |  |  |  |  |  |  |
| ModelScope | 6.99 | 20.36 | 22.54 | 67.93 | 50.01 | 38.72 | 44.18 | 71.32 | 44.56 |
| Show-1 | 2.07 | 20.66 | 23.24 | 68.42 | 58.63 | 48.55 | 44.31 | 58.97 | 37.93 |
| VideoCrafter2 | 3.90 | 21.21 | 22.71 | 68.58 | 69.32 | 45.11 | 50.45 | 80.37 | 38.40 |
| Ours | 1.79 | 20.73 | 23.29 | 68.26 | 69.55 | 47.74 | 56.31 | 68.55 | 44.72 |

This section provides the full results on VBench (Huang et al., ) and EvalCrafter (Liu et al., ) as a supplement to the performance comparison in the experiments section of the main paper. The evaluation of our model is performed using 5-second 768p videos generated at 24 fps.

VBench (Huang et al., ).
The full experimental results on VBench are shown in Table 4. As can be observed, our model achieves leading or highly competitive results among open-source and commercial competitors, especially for the metrics related to motion quality. For example, the dynamic degree metric of our model ranks 2nd among all models at 64.63, validating the effectiveness of our generative model in learning temporal dynamics. For the rest of the metrics, our results are also generally superior to the open-source Open-Sora Plan v1.1 (PKU-Yuan Lab et al., ) and Open-Sora 1.2 (Zheng et al., ), with significantly lower training computational cost as mentioned earlier. We also note that half of our results even outperformed the recent CogVideoX-5B (Yang et al., ), which is based on a larger DiT model, demonstrating its modeling capacity. On the other hand, our model performs relatively inferior on metrics such as color and appearance style, which is more related to the image generation capabilities and finer-grained prompt following. This is largely due to our video captioning procedure based on video LLMs which tends to produce coarse-grained captions, thus dampening these abilities. Nevertheless, thanks to our autoregressive generation framework, which decomposes video generation into first frame generation and subsequent frame generation, these image quality issues can be addressed separately with additional well-captioned image data in future training stages. Similarly, due to the SD3-Medium weight initialization, which is infamous for its human structure, our method achieves a relatively low score in human action, which could be addressed by switching to other base models or training from scratch.

EvalCrafter (Liu et al., ).
The raw metrics on EvalCrafter are provided in Table 5. Overall, our model delivers highly competitive performance on the majority of metrics, outperforming many previous open-source and closed-source models. In particular, the motion AC score of our method which is relevant to the temporal motion quality ranks 2nd among all methods, justifying the capacity of our pyramid designs to learn complex spatiotemporal patterns in video. Our method also demonstrates superiority over several other metrics related to semantic alignment, including BLIP-BLUE and CLIP score. Placing top two in both metrics among the models compared, including the closed-source Gen-2 (Runway, ), confirms the advantages of our model in text-to-video semantic alignment. The only metric where our model performs poorly is face consistency, which is due to the temporal pyramid design adopted for compressing the history condition. We view this as an issue that can potentially be addressed by more sophisticated temporal compression schemes.

### C.2 Abaltion Study

Figure: Figure 10: Ablation study of corrective renoising during the inference stage.
Refer to caption: extracted/5910391/figs/ablation/3/baseline-1.jpg

Figure: Figure 11: Ablation study of blockwise causal attention at 100k training step.
Refer to caption: extracted/5910391/figs/ablation/4/baseline-1/0.jpg

In this section, we conduct additional ablation studies of two important design details in our proposed pyramidal flow matching, including the corrective noise added during inference of the spatial pyramid and the blockwise causal attention used for autoregressive video generation.

Role of corrective noise.
To study its efficacy in the spatial pyramid, we curate a baseline method that inferences without adding this corrective Gaussian noise. The detailed comparative results of our method against this variant are shown in Fig. 10. While the baseline method has a correct global structure, it fails to produce a fine-grained, high-resolution image with rich details and instead produces a blurred image that suffers from block-like artifacts (better observed when zooming in). This is because applying the upsampling function at the jump points between different pyramid stages of varying resolutions results in excessive correlation between spatially adjacent latent values. In comparison, our generated images have rich details and vivid colors, confirming that the adopted corrective renoising scheme effectively addresses this artifact problem in the spatial pyramid.

Effectiveness of causal attention.
In Fig. 11, we study the effect of blockwise causal attention by comparing it to the bidirectional attention used in full-sequence diffusion. While an intuitive understanding might be that bidirectional attention promotes information exchange and increases model capacity, it is understudied for autoregressive video generation. In an early experiment, we trained a baseline model using bidirectional attention across different latent frames, the results of which are visualized in Fig. 11. As can be seen from the sampled keyframes of the 1-second videos, this model suffers from a lack of temporal coherence as the subject in the generated video is constantly changing in shape and color. Meanwhile, our model shows good temporal coherence with reasonable motion. We infer that this is because the history condition in bidirectional attention is influenced by the ongoing generation and thus deviates, whereas the history condition in causal attention is fixed, serving as a predetermined condition and stabilizing the autoregressive generative process.

### C.3 Visualization

Figure: (a) A massive explosion on the surface of the earth.
Refer to caption: extracted/5910391/figs/gen3/explosion/0.jpg

Figure: (a) An aerial shot of a lighthouse standing tall on a rocky cliff, its beacon cutting through the early dawn, waves crash against the rocks below.
Refer to caption: extracted/5910391/figs/cog5b/lighthouse/0.jpg

This section presents additional qualitative results for our text-to-video generation in comparison to the recent leading models including Gen-3 Alpha (Runway, ), Kling (Kuaishou, ) and CogVideoX (Yang et al., ). The uniformly sampled frames from the generated videos are shown in Figs. 13 and 12, in which our videos are generated at 5s, 768p, 24fps. Overall, we observe that despite being trained only on publicly available data and using a small computational budget, our model yields a highly competitive visual aesthetics and motion quality among the baselines.

Specifically, the results highlight the following characteristics of our model: (1) Through generative pre-training, our model is capable of generating videos of cinematic quality and reasonable content. For example, in Fig. 12(a), our generative video shows a mushroom cloud resulting from “a massive explosion” taking place in “the surface of the earth”, creating a sci-fi movie atmosphere. However, the current model is not fully faithful to some prompts such as the “salt desert” in Fig. 12(b), which could be addressed by curating more sophisticated caption data. (2) Despite that our model has only 2B parameters initialized from SD3-Medium (Esser et al., ), it clearly outperforms CogVideoX-2B of the same model size with additional training data, and is even comparable to the 5B full version in some aspects. For example, in Figs. 13(a) and 13(b), only our model and its 5B version are capable of generating reasonable sea waves according to the input prompt, while its 2B variant merely illustrates an almost static sea surface. This is largely attributed to our proposed pyramidal flow matching in improving training efficiency. Overall, these results validate the effectiveness of our approach in modeling complex spatiotemporal patterns through the spatial and temporal pyramid designs. Our generated videos are best-viewed at [https://pyramid-flow.github.io](https://pyramid-flow.github.io).

## Appendix D Limitations

Our method only supports autoregressive generation and cannot be extended to keyframe interpolation or video interpolation. In addition, we noticed that the temporal pyramid designs to improve training efficiency can sometimes lead to subtle subject inconsistency, especially over the long term. While this is not a prevalent problem, we believe that developing more sophisticated temporal compression methods is critical to the broader applicability of our video generative model.

There are also several issues related to the training data. Since we did not include a prompt rewriting procedure in the data curation, the experimental results are focused on relatively short prompts. Also, due to the data filtering procedure, our model did not learn scene transitions during training. This may be overcome by introducing an additional model as the scene director (Lin et al., ).

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please refer to the questions (or issues) mentioned in the Weaknesses section.
soundness: 2
strengths: + The proposed pyramidal flow matching scheme is novel in video generation modeling and greatly enhances training efficiency.
+ The unified training objective is intuitive and effective.
+ The quantitative and qualitative analyses in the paper are comprehensive.
+ The quality of the generated videos is excellent.
summary: This paper introduces a novel pyramidal flow matching scheme for video generation, which significantly improves training efficiency while preserving generation quality. The authors also propose a unified flow-matching objective that enables joint training of the pyramid stages within a single DiT model, eliminating the need for separate optimization across multiple models seen in prior approaches. Comprehensive experimental analyses are conducted on the VBench and EvalCrafter benchmarks, with proofs and additional qualitative results included in the supplementary materials.
weaknesses: * The writing in the paper could benefit from further improvement for clarity and readability.
    - The repeated use of the term "full-resolution" up to the experiment section suggests that generation is being done in pixel space rather than latent space. It would be helpful to clarify this in the paper, as it may be misleading.

    - The paper contains several grammatical errors and repeatedly uses unnecessary terms, such as "sophisticated," which affect readability. I encourage the authors to revise the manuscript to improve clarity and flow.

* Some of the derivations and assumptions in the paper are ambiguous and potentially flawed

    - In Equations 7 and 8, the notation should reflect the conditional distributions $\hat{x}{e_k}|x_1$ and $\hat{x}{s_k}|x_1$. While both endpoints are Gaussian-distributed conditionally, the endpoint distributions $\hat{x}{e_k} = \int p(\hat{x}{e_k}|x_1)p(x_1)dx_1$ are not Gaussian.

    - In line 213, while the objective $\hat{x}{e_k} - \hat{x}{s_k}$ is correct, since we consider flow matching with $K$ windows, the vector field should instead be conditioned on the endpoints, $u_t(x_t|\hat{x}{e_k})$. However, this is challenging because the distribution $p(x_t|\hat{x}{e_k})$ may not be Gaussian. This objective could be derived more straightforwardly from a rectified flow perspective [1], where velocities are matched with $\dot{X}_t$ (see section 2.3 in [1]).

    - For training, are $x_1$ and $n$ at the start and end the same? Since flow matching accommodates data points from arbitrary couplings, this should not impact training validity but would benefit from additional explanation for clarity.

    - Regarding the renoising procedure in Section 3.2.2, Equation 12 should ideally denote $Up(\hat{x}{e_{k+1}}|x_1)$ rather than $Up(\hat{x}{e_{k+1}})$, as the latter may be non-Gaussian. Consequently, the subsequent proof may be invalid if it relies on $Up(\hat{x}{e_{k+1}})$ being Gaussian.

         [1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

* Some important details and experiments are missing from the paper.

    - In Sec 3.2.1, the authors claim to enforce the noise to be in the same direction to enhance the straightness of the flow trajectory (Lines 207-209). However, there is no ablation experiment to show the benefit of this design choice.

    - The proof in Section 3.2.2 and Appendix A primarily relies on Equation 8. However, the authors implement the scheme in Equation 10, with no discussion on how the derivations in Section 3.2.2 and Appendix A could be generalized for Equation 10.
    - What is the number of sampling steps in Algorithm 1?

    - (Minor) Have the authors considered using temporally compressed pyramidal flow as well?


* No experimental results are provided for the video autoencoding task
   - In Section 4.1, the authors claim to train a video VAE for spatial and temporal compression of videos; however, there are no experimental results provided to evaluate the performance of the video VAE.
    - Why train a new video VAE instead of utilizing existing open-source video VAEs, such as Open-Sora or Open-Sora-Plan? Wouldn't this approach have been more effective in ensuring a fair experimental comparison?

* The results on VBench and EvalCrafter are quite strong. However, since most of the competing approaches were published prior to the release of these benchmarks, it would be beneficial for the authors to include the FVD metric to compare their approach with the competing baselines. Additionally, could the authors provide a comparison of the generation and training speeds of their approach relative to previous works?

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: In the rebuttal, I hope the authors can address my questions and concerns in the weakness section.
soundness: 4
strengths: The strength of this paper is multi-fold. 
+ It builds a flow matching model with multiple resolutions for text-to-video generation. The pyramidal flow matching allows the model to train with less computational costs and memory footprints.
+ The whole model has a unified objective instead of optimizing separate modules for video generation and super-resolution, using a single Diffusion Transformer.
+ The experimental results are competitive, with evaluation on two public benchmarks of VBench and EvalCrafter. The visual quality is comparable to other commercial text-to-video models.
+ The technical explanation for inference with renoising to solve jump points is clear with supplemental materials.
+ The model can be extended to image-to-video generation. 
+ The ablation studies are conducted to illustrate the contribution of different component in the model.
summary: This work presents an effective flow matching approach for text-to-video generation via both spatial and temporal pyramidal designs. With such an architecture, the training efficiency has been significantly reduced. It has a good conversation that when noise is strong, flow matching is less critical and can be performed at a low resolution. It has a unified flow matching objective instead of having different models for generation and super-resolution. The paper has validated this effectiveness by controlled experiments compared to a baseline with full-resolution with the same computational costs. The source code of this paper is expected to be open-source, which is very helpful to video generation research and industrial communities.
weaknesses: While this work has an interesting novel design for flow matching for video generation and competitive visual results, there are some unclear points and weaknesses as follows.
- Questions about [s_k, e_k]. The authors divide [0,1] into K time windows [s_k,e_k]. Why don't the authors set e_{k+1}=s_k? Instead, the authors use e_{k+1}=2s_k/(1+s_k) and we can that e_{k+1}>s_k. This means there are overlapping time windows. Given a time step t, t may fall on more than one time step, and how do authors handle such a scenario? Also, is s_K equal to zero? Could you show what the exact values of [s_k,e_k] are?
- The Figure 1(b) and Figure 3 are not consistent. x^i_t takes x^{i-1}^t with the same spatial resolution as part of the history in Figure 3, but Figure 1(b) shows x^i_t takes x^{i-1}_t' with lower resolution as a temporal condition. This part confuses me.
- The model is autoregressive in the temporal domain, and does it mean that the model can generate videos with arbitrary lengths? Why the inference is limited to videos up to 10 seconds?
- When performing the ablation study for the temporal pyramid, the baseline "full-seq" has only qualitative results. Can the authors provide quantitative results or something similar to the plot in Figure 7?

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - During training (Equation-16), noisy condition is used. During testing (Equation-17), clean generated frames are used. Will there be a training-testing distribution mismatch?
- How is text condition added to the model?
soundness: 2
strengths: - The ideas of both spatial pyramids and temporal pyramids are novel and interesting.
- The training efficiency is largely improved due to the novel pyramid design.
summary: The paper proposed a pyramid-based flow matching for efficient training of video diffusion models. Unlike traditional flow matching which operates on a single resolution, the proposed spatial pyramidal flow matching operates on a pyramid of resolutions. The denoising process starts from a small resolution, and the resolution will increase by 2 after several sampling steps. To address the discontinuity of the jumping point, the paper proposed a novel re-noising formula. For video generation, the paper proposed an autoregressive approach, where the generation of each frame is conditioned on the low-resolution of previous frames.
weaknesses: - The analysis of inference efficiency is lacking. How does the proposed method compare to previous full-attention methods for different numbers of frames?
- Compared to full-attention methods, the proposed autoregressive method may encounter the issue of error drifting when the number of frames increases. At how many frames, the proposed method will fail?
- It will be good to include some video results from previous methods on the project page.
- Figure 7 and Figure 8 both show partial results before training convergence. It will be more convincing to show the training graph with more training iterations or converged training behavior. 
- The text-video alignment is worse compared to previous methods.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The authors should clarify whether their proposed method can be applied to text-to-image models, as the method appears to have limited relevance to video generation.
2. The authors should explain why they opted for flow matching over DDIM, as well as outline the advantages of flow matching.
soundness: 2
strengths: This paper proposes a novel pyramidal flow matching algorithm based on the premise that the initial timesteps in diffusion models are quite noisy and uninformative. This approach builds upon the recent prevalent flow matching framework to address the shortcomings of existing methods, specifically the requirement of employing distinct models at different resolutions, which sacrifices flexibility and scalability.
summary: This paper introduces pyramidal flow matching, a new video generative model that combines spatial and temporal pyramid representations, which can enhance the training efficiency and maintains high video generation quality.
weaknesses: 1. The authors should clarify the specific implications of how the separate optimization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This is crucial as it relates to the foundational aspects of the problem design presented in this paper.
2. The proposed method involves many parameters that require manual design. The authors should provide more explanations when designing these parameters, such as time windows and γ.
3. Figures 2 and 3 in the paper are overly simplistic, and the authors should provide a detailed explanation of their relationship to the equations.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 2
questions: What are the training data used by the other compared methods (especially the ones trained with open-source data)?

How much overlap is there between the training data used in the proposed method and the other methods?

Why did the authors keep the number of pyramid stages set to 3 in all the experiments? There should be an ablation study on the number of pyramid stages.

What are the number of parameters and/or FLOPs (operations) used by other (open-source) methods for video generation?
soundness: 4
strengths: The computational complexity of SOTA video generation methods is a real problem, and the pyramidal flow matching solution provided by the authors is interesting and effective.

The pyramidal flow matching framework is built on a single diffusion transformer model and can be optimized in a unified, end-to-end fashion. This saves time and enables better knowledge sharing. Better knowledge sharing can help achieve greater consistency in the generated videos, which is evident from the generated results.

Evaluated on benchmarks like VBench and EvalCrafter, the model demonstrated high performance, especially compared to methods trained with open-source data.

The model also achieved competitive performance during the user study, and I have personally checked their provided anonymous website for the videos, which looked good.
summary: The authors propose a novel video generation method to address the problems of the current methods. The SOTA (state-of-the-art) methods mostly suffer from heavy computational burdens due to cascaded architectures or separate optimization for different resolutions of video generation training. The authors address this issue with a novel pyramid flow matching framework, both in the spatial and temporal domains. In the spatial domain, the method builds a pyramid of different resolution frames, where the early stages operate on compressed, low-resolution representations (latent representations) of the frames. As the stages go higher up, the resolution increases, and the full resolution is only used in the final stage for optimization. Temporal compression is achieved by using only a lower resolution history of previous frames (to understand motion and scene), further reducing computation for long videos. This pyramidal approach reduces redundant computation by focusing resources only on necessary parts, making training more efficient. The results are comprehensive and satisfactory. Overall, a good paper to be accepted.
weaknesses: The mathematical notations are not clearly defined. For example, the authors start with s_k, e_k​, x_{s_k}​​, or x_{e_k}​​, etc., without properly defining them first. This hinders the flow of the paper. Please resolve all such issues.

Overall, the writing style of the paper is a bit convoluted (especially in the methods section); it should be revised for smoother understanding.

There is no ablation study on the number of pyramid stages, which is a crucial factor in their design choice.

There is no comparison between the number of parameters and FLOPs used by other open-source methods and the proposed method. This can reflect how effective the model is compared to other methods.

Meta-review: The paper presents an important step towards efficient training of text-to-video generation. Such models are known to be very hard to train, requiring a substantial computational budget. Here, we are presented with a method to reinterpret denoising trajectory as a series of pyramid stages, leading to an efficient solution. The reviewers give a very positive assessment of the work listing many strengths of the work. The AC agrees. Congrats!

Decision: Accept (Poster)