Lumina-Video: Efficient and Flexible Video Generation with Multi-scale
  Next-DiT
Authors: name: Dongyang Liu; name: Shicheng Li; name: Yutong Liu; name: Zhen Li; name: Kai Wang; name: Xinyue Li; name: Qi Qin; name: Yufei Liu; name: Yi Xin; name: Zhongyu Li; name: Bin Fu; name: Chenyang Si; name: Yuewen Cao; name: Conghui He; name: Ziwei Liu; name: Yu Qiao; name: Qibin Hou; name: Hongsheng Li; name: Peng Gao
Published: 2025-02-10 18:58:11

Abstract: Recent advancements have established Diffusion Transformers (DiTs) as a
dominant framework in generative modeling. Building on this success,
Lumina-Next achieves exceptional performance in the generation of
photorealistic images with Next-DiT. However, its potential for video
generation remains largely untapped, with significant challenges in modeling
the spatiotemporal complexity inherent to video data. To address this, we
introduce Lumina-Video, a framework that leverages the strengths of Next-DiT
while introducing tailored solutions for video synthesis. Lumina-Video
incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple
patchifications to enhance both efficiency and flexibility. By incorporating
the motion score as an explicit condition, Lumina-Video also enables direct
control of generated videos' dynamic degree. Combined with a progressive
training scheme with increasingly higher resolution and FPS, and a multi-source
training scheme with mixed natural and synthetic data, Lumina-Video achieves
remarkable aesthetic quality and motion smoothness at high training and
inference efficiency. We additionally propose Lumina-V2A, a video-to-audio
model based on Next-DiT, to create synchronized sounds for generated videos.
Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video.