Motion Prompting: Controlling Video Generation with Motion Trajectories
Authors: name: Daniel Geng; name: Charles Herrmann; name: Junhwa Hur; name: Forrester Cole; name: Serena Zhang; name: Tobias Pfaff; name: Tatiana Lopez-Guevara; name: Carl Doersch; name: Yusuf Aytar; name: Michael Rubinstein; name: Chen Sun; name: Oliver Wang; name: Andrew Owens; name: Deqing Sun
Published: 2024-12-03 18:59:56

Abstract: Motion control is crucial for generating expressive and compelling video
content; however, most existing video generation models rely mainly on text
prompts for control, which struggle to capture the nuances of dynamic actions
and temporal compositions. To this end, we train a video generation model
conditioned on spatio-temporally sparse or dense motion trajectories. In
contrast to prior motion conditioning work, this flexible representation can
encode any number of trajectories, object-specific or global scene motion, and
temporally sparse motion; due to its flexibility we refer to this conditioning
as motion prompts. While users may directly specify sparse trajectories, we
also show how to translate high-level user requests into detailed, semi-dense
motion prompts, a process we term motion prompt expansion. We demonstrate the
versatility of our approach through various applications, including camera and
object motion control, "interacting" with an image, motion transfer, and image
editing. Our results showcase emergent behaviors, such as realistic physics,
suggesting the potential of motion prompts for probing video models and
interacting with future generative world models. Finally, we evaluate
quantitatively, conduct a human study, and demonstrate strong performance.
Video results are available on our webpage: https://motion-prompting.github.io/