Explorative Inbetweening of Time and Space
Authors: name: Haiwen Feng; name: Zheng Ding; name: Zhihao Xia; name: Simon Niklaus; name: Victoria Abrevaya; name: Michael J. Black; name: Xuaner Zhang
Published: 2024-03-21 17:57:31

Abstract: We introduce bounded generation as a generalized task to control video
generation to synthesize arbitrary camera and subject motion based only on a
given start and end frame. Our objective is to fully leverage the inherent
generalization capability of an image-to-video model without additional
training or fine-tuning of the original model. This is achieved through the
proposed new sampling strategy, which we call Time Reversal Fusion, that fuses
the temporally forward and backward denoising paths conditioned on the start
and end frame, respectively. The fused path results in a video that smoothly
connects the two frames, generating inbetweening of faithful subject motion,
novel views of static scenes, and seamless video looping when the two bounding
frames are identical. We curate a diverse evaluation dataset of image pairs and
compare against the closest existing methods. We find that Time Reversal Fusion
outperforms related work on all subtasks, exhibiting the ability to generate
complex motions and 3D-consistent views guided by bounded frames. See project
page at https://time-reversal.github.io.