Generative Image Dynamics
Authors: name: Zhengqi Li; name: Richard Tucker; name: Noah Snavely; name: Aleksander Holynski
Published: 2023-09-14 17:54:01

Abstract: We present an approach to modeling an image-space prior on scene dynamics.
Our prior is learned from a collection of motion trajectories extracted from
real video sequences containing natural, oscillating motion such as trees,
flowers, candles, and clothes blowing in the wind. Given a single image, our
trained model uses a frequency-coordinated diffusion sampling process to
predict a per-pixel long-term motion representation in the Fourier domain,
which we call a neural stochastic motion texture. This representation can be
converted into dense motion trajectories that span an entire video. Along with
an image-based rendering module, these trajectories can be used for a number of
downstream applications, such as turning still images into seamlessly looping
dynamic videos, or allowing users to realistically interact with objects in
real pictures.