Lumina-Video: Efficient and Flexible Video Generation with Multi-scale
  Next-DiT
Authors: name: Dongyang Liu, name: Shicheng Li, name: Yutong Liu, name: Zhen Li, name: Kai Wang, name: Xinyue Li, name: Qi Qin, name: Yufei Liu, name: Yi Xin, name: Zhongyu Li, name: Bin Fu, name: Chenyang Si, name: Yuewen Cao, name: Conghui He, name: Ziwei Liu, name: Yu Qiao, name: Qibin Hou, name: Hongsheng Li, name: Peng Gao

Recent advancements have established Diffusion Transformers (DiTs) as a
dominant framework in generative modeling. Building on this success,
Lumina-Next achieves exceptional performance in the generation of
photorealistic images with Next-DiT. However, its potential for video
generation remains largely untapped, with significant challenges in modeling
the spatiotemporal complexity inherent to video data. To address this, we
introduce Lumina-Video, a framework that leverages the strengths of Next-DiT
while introducing tailored solutions for video synthesis. Lumina-Video
incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple
patchifications to enhance both efficiency and flexibility. By incorporating
the motion score as an explicit condition, Lumina-Video also enables direct
control of generated videos' dynamic degree. Combined with a progressive
training scheme with increasingly higher resolution and FPS, and a multi-source
training scheme with mixed natural and synthetic data, Lumina-Video achieves
remarkable aesthetic quality and motion smoothness at high training and
inference efficiency. We additionally propose Lumina-V2A, a video-to-audio
model based on Next-DiT, to create synchronized sounds for generated videos.
Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video.