ViBiDSampler: Enhancing Video Interpolation Using Bidirectional
  Diffusion Sampler
Authors: name: Serin Yang; name: Taesung Kwon; name: Jong Chul Ye
Published: 2024-10-08 03:01:54

Abstract: Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V)
diffusion models has greatly enhanced video generation, especially in terms of
keyframe interpolation. However, current image-to-video diffusion models, while
powerful in generating videos from a single conditioning frame, need adaptation
for two-frame (start & end) conditioned generation, which is essential for
effective bounded interpolation. Unfortunately, existing approaches that fuse
temporally forward and backward paths in parallel often suffer from
off-manifold issues, leading to artifacts or requiring multiple iterative
re-noising steps. In this work, we introduce a novel, bidirectional sampling
strategy to address these off-manifold issues without requiring extensive
re-noising or fine-tuning. Our method employs sequential sampling along both
forward and backward paths, conditioned on the start and end frames,
respectively, ensuring more coherent and on-manifold generation of intermediate
frames. Additionally, we incorporate advanced guidance techniques, CFG++ and
DDS, to further enhance the interpolation process. By integrating these, our
method achieves state-of-the-art performance, efficiently generating
high-quality, smooth videos between keyframes. On a single 3090 GPU, our method
can interpolate 25 frames at 1024 x 576 resolution in just 195 seconds,
establishing it as a leading solution for keyframe interpolation.