Generative Image Dynamics  Zhengqi Li Richard Tucker Noah Snavely Aleksander Holynski  Google Research  ...  Input Picture  X coefficients  Spectral Volume (Image-Space Modal Basis)  0.2Hz  0.4Hz  3.0Hz  Looping video  Interactive dynamics  ...  Y coefficients  Figure 1. We model a generative image-space prior on scene motion: from a single RGB image, our method generates a spectral volume [23], a motion representation that models dense, long-term pixel trajectories in the Fourier domain. Our learned motion priors can be used to turn a single picture into a seamlessly looping video, or into an interactive simulation of dynamics that responds to user inputs like dragging and releasing points. On the right, we visualize output videos as space-time X-t slices (along the input scanline shown on the left).  Abstract  We present an approach to modeling an image-space prior on scene motion. Our prior is learned from a collection of motion trajectories extracted from real video sequences depicting natural, oscillatory dynamics of objects such as trees, flowers, candles, and clothes swaying in the wind. We model dense, long-term motion in the Fourier domain as spectral volumes, which we find are well-suited to prediction with diffusion models. Given a single image, our trained model uses a frequency-coordinated diffusion sampling process to predict a spectral volume, which can be converted into a motion texture that spans an entire video. Along with an imagebased rendering module, the predicted motion representation can be used for a number of downstream applications, such as turning still images into seamlessly looping videos, or allowing users to interact with objects in real images, producing realistic simulated dynamics (by interpreting the spectral volumes as image-space modal bases). See our project page for more results: generative-dynamics.github.io.  1. Introduction  The natural world is always in motion, with even seemingly static scenes containing subtle oscillations as a result of wind, water currents, respiration, or other natural rhythms. Emulating this motion is crucial in visual content synthesis—human sensitivity to motion can cause imagery without motion (or with slightly unrealistic motion) to seem uncanny or unreal.  While it is easy for humans to interpret or imagine motion in scenes, training a model to learn or produce realistic scene motion is far from trivial. The motion we observe in the world is the result of a scene’s underlying physical dynamics, i.e., forces applied to objects that respond according to their unique physical properties—their mass, elasticity, etc—quantities that are hard to measure and capture at scale. Fortunately, measuring them is not necessary for certain applications: e.g., one can simulate plausible dynamics in a scene by simply analyzing some observed 2D motion [23].  This same observed motion can also serve as a supervisory signal in learning dynamics across scenes—because although observed motion is multi-modal and grounded in complex physical effects, it is nevertheless often predictable:  This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. candles will flicker in certain ways, trees will sway, and their leaves will rustle. As humans, this predictability is ingrained in our systems of perception: by viewing a still image, we can imagine plausible motions— or, since there might have been many possible such motions, a distribution of natural motions conditioned on that image. Given the facility with which humans are able to model these distributions, a natural research problem is to model them computationally.  Recent advances in generative models, in particular conditional diffusion models [44, 84, 86], have enabled us to model rich distributions, including distributions of real images conditioned on text [72–74]. This capability has enabled several new applications, such as text-conditioned generation of diverse and realistic image content. Following the success of these image models, recent work has extended these models to other domains, such as videos [7, 43] and 3D geometry [76, 99, 100, 102].  In this paper, we model a generative prior for image-space scene motion, i.e., the motion of all pixels in a single image. This model is trained on motion trajectories automatically extracted from a large collection of real video sequences. In particular, from each training video we compute motion in the form of a spectral volume [22, 23], a frequency-domain representation of dense, long-range pixel trajectories. Spectral volumes are well-suited to scenes that exhibit oscillatory dynamics, e.g., trees and flowers moving in the wind. We find that this representation is also highly effective as an output of a diffusion model for modeling scene motion. We train a generative model that, conditioned on a single image, can sample spectral volumes from its learned distribution. A predicted spectral volume can then be directly transformed into a motion texture—a set of long-range, per-pixel motion trajectories—that can be used to animate the image. The spectral volume can also be interpreted as an image-space modal basis for use in simulating interactive dynamics [22].  We predict spectral volumes from input images using a diffusion model that generates coefficients one frequency at a time, but coordinates these predictions across frequency bands through a shared attention module. The predicted motions can be used to synthesize future frames (via an image-based rendering model)—turning still images into realistic animations, as illustrated in Fig. 1.  Compared with priors over raw RGB pixels, priors over motion capture more fundamental, lower-dimensional structure that efficiently explains long-range variations in pixel values. Hence, generating intermediate motion leads to more coherent long-term generation and more fine-grained control over animations. We demonstrate the use of our trained model in several downstream applications, such as creating seamless looping videos, editing the generated motions, and enabling interactive dynamic images via image-space modal bases, i.e., simulating the response of object dynamics to user-applied forces [22].