Pyramidal Flow Matching for Efficient Video Generative Modeling
Authors: name: Yang Jin; name: Zhicheng Sun; name: Ningyuan Li; name: Kun Xu; name: Kun Xu; name: Hao Jiang; name: Nan Zhuang; name: Quzhe Huang; name: Yang Song; name: Yadong Mu; name: Zhouchen Lin
Published: 2024-10-08 12:10:37

Abstract: Video generation requires modeling a vast spatiotemporal space, which demands
significant computational resources and data usage. To reduce the complexity,
the prevailing approaches employ a cascaded architecture to avoid direct
training with full resolution. Despite reducing computational demands, the
separate optimization of each sub-stage hinders knowledge sharing and
sacrifices flexibility. This work introduces a unified pyramidal flow matching
algorithm. It reinterprets the original denoising trajectory as a series of
pyramid stages, where only the final stage operates at the full resolution,
thereby enabling more efficient video generative modeling. Through our
sophisticated design, the flows of different pyramid stages can be interlinked
to maintain continuity. Moreover, we craft autoregressive video generation with
a temporal pyramid to compress the full-resolution history. The entire
framework can be optimized in an end-to-end manner and with a single unified
Diffusion Transformer (DiT). Extensive experiments demonstrate that our method
supports generating high-quality 5-second (up to 10-second) videos at 768p
resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models
will be open-sourced at https://pyramid-flow.github.io.