你好，以下是一篇论文的generative image dynamics的内容，能否帮我想想和我现在的课题有什么相似之处，可能如何结合。我现在做的课题内容是单视角草图+文本生成多视角草图和多视角图片，再用多视角图片生成3dgs，最后用3dgs渲染的图片和草图衡量相似度的方法。具体来说生成多视角草图使用的是wonder3d，一个用stable diffusion微调的扩散模型，方法原来输入单视角图片输出多视角图片和多视角法线。
We present an approach to modeling an image-space
prior on scene motion. Our prior is learned from a collection
of motion trajectories extracted from real video sequences depicting natural, oscillatory dynamics of objects such as trees,
flowers, candles, and clothes swaying in the wind. We model
dense, long-term motion in the Fourier domain as spectral
volumes, which we find are well-suited to prediction with
diffusion models. Given a single image, our trained model
uses a frequency-coordinated diffusion sampling process to
predict a spectral volume, which can be converted into a motion texture that spans an entire video. Along with an imagebased rendering module, the predicted motion representation
can be used for a number of downstream applications, such
as turning still images into seamlessly looping videos, or
allowing users to interact with objects in real images, producing realistic simulated dynamics (by interpreting the spectral
volumes as image-space modal bases). See our project page
for more results: generative-dynamics.github.io.
1. Introduction
The natural world is always in motion, with even seemingly
static scenes containing subtle oscillations as a result of wind,
water currents, respiration, or other natural rhythms. Emulating this motion is crucial in visual content synthesis—human
sensitivity to motion can cause imagery without motion (or
with slightly unrealistic motion) to seem uncanny or unreal.
While it is easy for humans to interpret or imagine motion in scenes, training a model to learn or produce realistic
scene motion is far from trivial. The motion we observe in
the world is the result of a scene’s underlying physical dynamics, i.e., forces applied to objects that respond according
to their unique physical properties—their mass, elasticity,
etc—quantities that are hard to measure and capture at scale.
Fortunately, measuring them is not necessary for certain applications: e.g., one can simulate plausible dynamics in a
scene by simply analyzing some observed 2D motion [23].
This same observed motion can also serve as a supervisory signal in learning dynamics across scenes—because
although observed motion is multi-modal and grounded in
complex physical effects, it is nevertheless often predictable:
This CVPR paper is the Open Access version, provided by the Computer Vision Foundation.
Except for this watermark, it is identical to the accepted version;
the final published version of the proceedings is available on IEEE Xplore. 24142
candles will flicker in certain ways, trees will sway, and their
leaves will rustle. As humans, this predictability is ingrained
in our systems of perception: by viewing a still image, we
can imagine plausible motions— or, since there might have
been many possible such motions, a distribution of natural
motions conditioned on that image. Given the facility with
which humans are able to model these distributions, a natural
research problem is to model them computationally.
Recent advances in generative models, in particular conditional diffusion models [44, 84, 86], have enabled us to
model rich distributions, including distributions of real images conditioned on text [72–74]. This capability has enabled
several new applications, such as text-conditioned generation of diverse and realistic image content. Following the
success of these image models, recent work has extended
these models to other domains, such as videos [7, 43] and
3D geometry [76, 99, 100, 102].
In this paper, we model a generative prior for image-space
scene motion, i.e., the motion of all pixels in a single image.
This model is trained on motion trajectories automatically
extracted from a large collection of real video sequences. In
particular, from each training video we compute motion in
the form of a spectral volume [22, 23], a frequency-domain
representation of dense, long-range pixel trajectories. Spectral volumes are well-suited to scenes that exhibit oscillatory
dynamics, e.g., trees and flowers moving in the wind. We
find that this representation is also highly effective as an
output of a diffusion model for modeling scene motion. We
train a generative model that, conditioned on a single image,
can sample spectral volumes from its learned distribution. A
predicted spectral volume can then be directly transformed
into a motion texture—a set of long-range, per-pixel motion
trajectories—that can be used to animate the image. The
spectral volume can also be interpreted as an image-space
modal basis for use in simulating interactive dynamics [22].
We predict spectral volumes from input images using a
diffusion model that generates coefficients one frequency at
a time, but coordinates these predictions across frequency
bands through a shared attention module. The predicted
motions can be used to synthesize future frames (via an
image-based rendering model)—turning still images into
realistic animations, as illustrated in Fig. 1.
Compared with priors over raw RGB pixels, priors over
motion capture more fundamental, lower-dimensional structure that efficiently explains long-range variations in pixel
values. Hence, generating intermediate motion leads to more
coherent long-term generation and more fine-grained control over animations. We demonstrate the use of our trained
model in several downstream applications, such as creating
seamless looping videos, editing the generated motions, and
enabling interactive dynamic images via image-space modal
bases, i.e., simulating the response of object dynamics to
user-applied forces [22].