Training-free Long Video Generation with Chain of Diffusion Model
  Experts
Authors: name: Wenhao Li; name: Yichao Cao; name: Xie Su; name: Xi Lin; name: Shan You; name: Mingkai Zheng; name: Yi Chen; name: Chang Xu
Published: 2024-08-24 01:33:28

Abstract: Video generation models hold substantial potential in areas such as
filmmaking. However, current video diffusion models need high computational
costs and produce suboptimal results due to high complexity of video generation
task. In this paper, we propose ConFiner, an efficient high-quality
video generation framework that decouples video generation into easier
subtasks: structure control and spatial-temporal refinement.
It can generate high-quality videos with chain of off-the-shelf diffusion model
experts, each expert responsible for a decoupled subtask. During the
refinement, we introduce coordinated denoising, which can merge multiple
diffusion experts' capabilities into a single sampling. Furthermore, we design
ConFiner-Long framework, which can generate long coherent video with three
constraint strategies on ConFiner. Experimental results indicate that with only
10\% of the inference cost, our ConFiner surpasses representative models like
Lavie and Modelscope across all objective and subjective metrics. And
ConFiner-Long can generate high-quality and coherent videos with up to 600
frames.