SF-V: Single Forward Video Generation Model
Authors: name: Zhixing Zhang; name: Yanyu Li; name: Yushu Wu; name: Yanwu Xu; name: Anil Kag; name: Ivan Skorokhodov; name: Willi Menapace; name: Aliaksandr Siarohin; name: Junli Cao; name: Dimitris Metaxas; name: Sergey Tulyakov; name: Jian Ren
Published: 2024-06-06 17:58:27

Abstract: Diffusion-based video generation models have demonstrated remarkable success
in obtaining high-fidelity videos through the iterative denoising process.
However, these models require multiple denoising steps during sampling,
resulting in high computational costs. In this work, we propose a novel
approach to obtain single-step video generation models by leveraging
adversarial training to fine-tune pre-trained video diffusion models. We show
that, through the adversarial training, the multi-steps video diffusion model,
i.e., Stable Video Diffusion (SVD), can be trained to perform single forward
pass to synthesize high-quality videos, capturing both temporal and spatial
dependencies in the video data. Extensive experiments demonstrate that our
method achieves competitive generation quality of synthesized videos with
significantly reduced computational overhead for the denoising process (i.e.,
around 23times speedup compared with SVD and 6times speedup compared with
existing works, with even better generation quality), paving the way for
real-time video synthesis and editing. More visualization results are made
publicly available at https://snap-research.github.io/SF-V.