Generative Inbetweening: Adapting Image-to-Video Models for Keyframe
  Interpolation
Authors: name: Xiaojuan Wang; name: Boyang Zhou; name: Brian Curless; name: Ira Kemelmacher-Shlizerman; name: Aleksander Holynski; name: Steven M. Seitz
Published: 2024-08-27 17:57:14

Abstract: We present a method for generating video sequences with coherent motion
between a pair of input key frames. We adapt a pretrained large-scale
image-to-video diffusion model (originally trained to generate videos moving
forward in time from a single input image) for key frame interpolation, i.e.,
to produce a video in between two input frames. We accomplish this adaptation
through a lightweight fine-tuning technique that produces a version of the
model that instead predicts videos moving backwards in time from a single input
image. This model (along with the original forward-moving model) is
subsequently used in a dual-directional diffusion sampling process that
combines the overlapping model estimates starting from each of the two
keyframes. Our experiments show that our method outperforms both existing
diffusion-based methods and traditional frame interpolation techniques.