LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis
Authors: name: Hanlin Wang; name: Hao Ouyang; name: Qiuyu Wang; name: Wen Wang; name: Ka Leong Cheng; name: Qifeng Chen; name: Yujun Shen; name: Limin Wang
Published: 2024-12-19 18:59:56

Abstract: The intuitive nature of drag-based interaction has led to its growing
adoption for controlling object trajectories in image-to-video synthesis.
Still, existing methods that perform dragging in the 2D space usually face
ambiguity when handling out-of-plane movements. In this work, we augment the
interaction with a new dimension, i.e., the depth dimension, such that users
are allowed to assign a relative depth for each point on the trajectory. That
way, our new interaction paradigm not only inherits the convenience from 2D
dragging, but facilitates trajectory control in the 3D space, broadening the
scope of creativity. We propose a pioneering method for 3D trajectory control
in image-to-video synthesis by abstracting object masks into a few cluster
points. These points, accompanied by the depth information and the instance
information, are finally fed into a video diffusion model as the control
signal. Extensive experiments validate the effectiveness of our approach,
dubbed LeviTor, in precisely manipulating the object movements when producing
photo-realistic videos from static images. Project page:
https://ppetrichor.github.io/levitor.github.io/