Supplementary Material
Generative Image Dynamics
Zhengqi LiRichard Tucker

Noah Snavely

Aleksander Holynski

Google Research

1. Spectral volumes as image-space modalbases
In this section, we provide details on the connection between
our predicted spectral volumes and image-space modal bases,
which can be used to simulate interactive dynamics throughmodal analysis. We refer readers to the original work in computer graphics and structure engineering for more thoroughtheory and analysis [3, 5, 16, 18]. In particular, we treat thepixels of the input image as a set of points (with cardinality|P |) that are linked to each other via mass-spring-damper
system. Starting from an object (modeled as a harmonic oscillator) at rest, that object’s motion response to an externalforce f (t) follows the equation of motion:
M ü(t) + C u̇(t) + Ku(t) = f (t)(1)

where u is the motion displacement of a given scene represented as a vector of size |P |, and M , C, and K are themass, damping and stiffness matrices respectively, dictating
the intrinsic dynamics of the object [16].
One can transform this equation of motion into modal
space, in which M , C, and K reduce to diagonal matrices
and we can decouple the equation of motions into a set of|P | independent single-degree-of-freedom systems:fi (t)(2)
mi
where mi , ci , and ki are the diagonalized elements of themass damping and stiffness matrices in modal space; and qi
and fi correspond to the motion displacement u and force
f in modal space. As in prior work, we do not directly estimate these matrices, but instead only use the intermediate
modal space to derive equations of response. We can further simplify Equation 2 under the common assumptions ofRayleigh damping: ci = αmi + βki , where we empiricallyset α = 0.4 and β = 0.08 in our case. This gives rise toq̈i (t) + ci q̇i (t) + ki qi (t) =fi (t)q̈i (t) + γi q̇i (t)+ωi2 qi (t) =miki, γi = α + βωi2
miwhere ω is the damped natural frequency.
ωi2 =(3)(4)

Moreover, Davis et al. [4] shows that the temporal Fouriertransform of per-pixel motion trajectory is approximately
proportional to the image-space projection of mode shapes
at resonant frequencies. Therefore, we can treat the spectral
volume as a basis and interpret the modal displacement qi
as the displacement of the object, in order to model and simulate image-space object dynamics. In particular, the final
displacement of a pixel p in image-space, Ft , can be written
as a sum of modal displacements weighted by the corresponding coefficients in the spectral volume over selected
frequency bands fj :X
Ft (p) =Sfj (p)qfj (t).
(5)fj

In our setting, we use all predicted frequency bands fromour latent diffusion modal as a basis in order to to avoid
manual mode selection performed in prior work. To simulate complex modal displacement qfj ,t , we perform explicit
Euler integration over Equation 4 to update the modal displacement, velocity, and acceleration over time:fi (t)− γi q̇i (t) − ωi2 qi (t)miq̇i (t + δt) = q̇i (t) + δt q̈i (t + δt)q̈i (t + δt) =qi (t + δt) = qi (t) + δt q̈i (t + δt)

(6)
(7)
(8)

where we set mi = 1 throughout our experiment.
Further, we project force vectors into modal space in order
to initial the complex modal displacement. In particular, as
in original work of Davis et al., we compute the magnitudeof the initial state of the modal displacement as||qfj (0)|| = ||f (0)· Sfj ||2||f (0)||2(9)and we compute the phase φ of the initial state of the modal
displacement for forces from a “drag and release” interaction
as:
f (0)
φdrag (qfj (0)) = −φ(· Sfj ).
(10)||f (0)||2(b) Learned W(a) Average-splat(c) W from motion

Figure 2. From left to right, we show a rendered future frame with(a) average splatting in RGB pixel space, (b) softmax splatting withlearnable weights [9], and (c) motion-aware feature splatting.
Scaling w/ resolution
Adaptive normalization

0.2(b) Motion derived W

Figure 1. We visualize input RGB images and corresponding
weights derived from the magnitude of motion texture.

0.1
0.0 0.0

0.2

0.4

0.4

We use motion-derived weights instead of learnable ones
because we observe that in the single-view case, learnableweights are not effective for addressing disocclusion ambiguities, as shown in the second column of Figure 2.
Moreover, we choose to perform per-frame independent
rendering instead of creating layered representations [17,19], since we found that the latter configuration can leadto significantly more visible artifacts and distortions near
object boundaries. In addition, since most of the motions weproduce are small, the method is only required to inpaintrelatively small unseen regions during rendering. Therefore,
we find that per-frame refinement is sufficient in our case.

3. Frequency-adaptive normalization
We show additional visualization of spectral volume coefficients at different frequency in Fig. 3, where we observe

0.8

1.0

1.2

0.4

0.6

0.8

1.0

1.2

Scaling w/ resolution
Adaptive normalization

0.8

Frequency

Frequency

0.2

Amplitude of Fourier coefficent at 0.4 Hz

Scaling w/ resolution
Adaptive normalization

0.6

0.2

0.4

0.1

0.2

0.0 0.0

0.2

0.4

0.6

0.8

1.0

0.0 0.00 0.25 0.50 0.75 1.00 1.25 1.50

1.2

Amplitude of Fourier coefficent at 0.8 Hz

1.0Scaling w/ resolution
Adaptive normalization

0.8Frequency

In the main manuscript we describe how we use the predictedmotion magnitude to determine the contributing weight ofeach source pixel mapped to its destination location, using
motion magnitude as a proxy for depth following the workof Davis, et al. [4].P
In particular, we compute a per-pixel
weight, W (p) = T1 t ||Ft (p)||2 as the average magnitudeof the predicted motion texture in order to determine thecontributions of colliding source pixels at destination time(shown in Fig. 1):P
I0 (p) · W (p)0P.
(11)
It (p + Ft p) =W (p)0.6

Amplitude of Fourier coefficent at 0.2 Hz

0.3

2. Image-based rendering

Scaling w/ resolution
Adaptive normalization

0.6

Amplitude of Fourier coefficent at 1.6 Hz

1.0

Scaling w/ resolution
Adaptive normalization

0.8

Frequency(a) Input image

0.25
0.20
0.15
0.10
0.05
0.00 0.0

Frequency

Frequency

0.3

0.6

0.4

0.4

0.2

0.2

0.0 0.00

0.25

0.50

0.75

1.00

1.25

Amplitude of Fourier coefficent at 2.4 Hz

0.0 0.0

0.5

1.0

1.5

Amplitude of Fourier coefficent at 3.0 Hz

Figure 3. Histogram of the amplitudes of Fourier terms acrossfrequencies after (1) scaling amplitude by image width and height(blue), or (2) frequency adaptive normalization (red).

that our frequency-adaptive normalization redistributes coefficients more evenly across different frequencies.

4. Additional implementation details
4.1. Network architecture
We use a VAE of continuous latent dimension 4 for eachfrequency slice of the spectral volume, and use base channel
128 with channel multiplier 1, 2, and 4 for the VAE. We
perform whitening to normalize encoded VAE latent features.
For the 2D diffusion model, we use base channels of 128,
channel multipliers 1, 4, and 8, and attention resolutions
32, 16, and 8 for each block. We downsample the input
RGB image using a standard ResNet encoder to produce
16-channel features and concatenate them with a 4-channel
noisy latent (or Gaussian noise map during inference) before

Input imageReference

Endo et al. [6]DMVFN [10]LFDM [15]Ours

Figure 4. Visual comparisons of generated future frames and corresponding motion fields. We show generated future frame (odd rows)
and estimated motion fields between the input and corresponding generated images. By inspecting differences with a reference image fromthe ground truth video, we observe that our approach produces more realistic textures and motions compared with baselines.

feeding them to the denoising network.
During training, we use 1,000 diffusion steps, and asquare linear noise schedule to perform latent denoising.
We adopt Adam [11] to train the LDM model for 750K steps
with batch size 96 and initial learning rate 5−5 . To avoid
overfitting, we apply random data augmentation by performing color jittering, random horizontal flips, random imagescaling, random rotation within five degrees, and random
crops. During inference, each step of DDIM sampling takes
0.5 seconds for motion prediction.
We adopt a ResNet-34 [8] as a feature extractor in ourimage-based rendering module. Specifically, we encode I0
through a feature extractor network to produce a multi-scalefeature map M = {Mj |j = 0, ..., J}. For each individual feature map Mj at scale j, we resize and scale thepredicted 2D motion field Ft according to the resolutionof Mj . With the motion field Ft and weights W , we apply softmax splatting to warp the feature map at each scale0to produce a warped feature Mj,t= Wsoftmax (Mj , Ft , W ),where Wsoftmax is the softmax splatting operation. The image

synthesis network is based on a co-modulation StyleGAN
architecture [12, 21], where we inject both style features
0mapped from Gaussian noise and warped features Mj,tinto
decoder blocks at athe corresponding scale to produce a final
rendered image Iˆt . We adopt Adam [11] to train the rendering model with batch size 32 and initial learning rate 10−4
for 300K iterations.

4.2. Data
As mentioned in the main manuscript, we collect and process a set of 3,015 videos depicting scenes with oscillation
dynamics from online footage websites as well as from ourown captures. In terms of videos from online sources, wecollect this kind of footage by using query texts like “Staticshot, trees/flowers/candles/clothes/lanterns, wind/breeze”,
and we also use more specific names of everyday trees andflowers to mine more data, (examples include maples, oak,beech, elm, pine, spruce, redwood, rose, daisy, carnation,
tulips, chrysanthemum, dahlia, sunflowers, orchid, lily, iris,cherry blossom, bushes, ivy, and dandelions). Furthermore,we remove the clips with strong camera motions by checking
if 95% pixels have average magnitude of motion trajectorylarger than one pixel; we also remove the clips without scene
motion by checking if average magnitude of estimated motion trajectories is less than one pixel.
To generate ground truth spectral volumes, we find thechoice of optical flow method to be crucial. In particular,
we observe that deep-learning based flow estimators tend
to produce over-smoothed flow fields, which results in unrealistic animation through image based rendering. Instead,
we apply a coarse-to-fine image pyramid-based flow algorithm [2, 13] between selected starting image and every
future frame within a video sequence to derive motion trajectory. We treat every 10th frame from each video as a starting
image, compute optical flows from the starting image andthe following 149 frames, and derive spectral volumes byapplying temporal FFT to the estimated per-pixel motion trajectory and selecting the first K frequency slices in Fourier
domains.

4.3. Large video diffusion model baselines
AnimateDiff [7] Since the original implementation of AnimateDiff only supports artificial images generated with Stable Diffusion as a starting frame, we perform DDIM inversion [14] to generate a video from a given real input RGB
image. We use the model from Realistic Vision V2.0 as abackbone since it is most related to the natural images andmotion we focus on. We manually add additional text inputs
according to the context of input picture. For example, weuse the prompt “flowers, wind, 8k uhd, dslr, soft lighting,high quality, film grain, Fujifilm XT3” for an input image
depicting flowers.
ModelScope [20]. The ModelScope model is based on
the work of VideoComposer [20], which supports multimodaility inputs such as text, input, sketch, motion vectors
or depth maps. Therefore, we feed our input image to themodel to generate corresponding video clip.
GEN-2 [1]. GEN-2 is a recent commercial video generation solution that supports text- and image-to-video tasks.
For each input, we not only feed the image but also manually provide text describing the image, as we found that
additional text descriptions systematically improve video
generation. For instance, for a picture of trees, we provide
the text prompt “A static shot of trees swaying in the wind,masterpiece.”
User study. We perform a user study and compare ourgenerated animations with the three baselines mentioned
above. On a randomly selected 30 videos from the test set,
we ask users “which video is more realistic?”. A total of
35 users finished the study. We found a 71.1% preference
rate for our method over AnimateDiff, a 83.7% preference
rate over ModelScope, and a 82.6% preference rate overGEN-2. We also provide visual video comparisons in thesupplementary website.

5. Additional qualitative comparisons
We provide additional comparisons of the quality of individual frames and motions synthesized by our approach andby other baselines [6, 10, 15] by visualizing the predicted
video frame Iˆt and its corresponding motion displacement
field at time t = 128. Figure 4 shows that the frames generated by our approach exhibit fewer artifacts and distortions
compared to other methods, and our corresponding 2D motion fields most resemble the reference displacement fields
estimated from the corresponding real videos. In contrast,
the background content generated by other methods tends
to drift, as shown in the flow visualizations in the evennumbered rows. Moreover, the video frames generated byother methods exhibit significant color distortion or ghosting
artifacts, suggesting that the baselines are less stable whengenerating videos with long time duration.

References[1] Gen-2. https://research.runwayml.com/gen2.
[2] Thomas Brox, Andrés Bruhn, Nils Papenberg, and JoachimWeickert. High accuracy optical flow estimation based on atheory for warping. In Proc. European Conf. on Computer
Vision (ECCV), pages 25–36. Springer, 2004.
[3] Abe Davis, Katherine L Bouman, Justin G Chen, Michael
Rubinstein, Fredo Durand, and William T Freeman. Visualvibrometry: Estimating material properties from small motion
in video. In Proceedings of the ieee conference on computer
vision and pattern recognition, pages 5335–5343, 2015.
[4] Abe Davis, Justin G Chen, and Frédo Durand. Image-spacemodal bases for plausible manipulation of objects in video.
ACM Transactions on Graphics (TOG), 34(6):1–7, 2015.
[5] Myers Abraham Davis. Visual vibration analysis. PhD thesis,
Massachusetts Institute of Technology, 2016.
[6] Yuki Endo, Yoshihiro Kanamori, and Shigeru Kuriyama. Animating landscape: Self-supervised learning of decoupled motion and appearance for single-image video synthesis. ACM
Transactions on Graphics (Proceedings of ACM SIGGRAPH
Asia 2019), 38(6):175:1–175:19, 2019.
[7] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao,
Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.
arXiv preprint arXiv:2307.04725, 2023.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patternrecognition, pages 770–778, 2016.
[9] Aleksander Holynski, Brian L Curless, Steven M Seitz, andRichard Szeliski. Animating pictures with Eulerian motion fields. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 5810–5819,2021.
[10] Xiaotao Hu, Zhewei Huang, Ailin Huang, Jun Xu, andShuchang Zhou. A dynamic multi-scale voxel flow networkfor video prediction. ArXiv, abs/2303.09875, 2023.
[11] Diederik P Kingma and Jimmy Ba. Adam: A method forstochastic optimization. arXiv preprint arXiv:1412.6980,2014.
[12] Zhengqi Li, Qianqian Wang, Noah Snavely, and AngjooKanazawa. Infinitenature-zero: Learning perpetual view generation of natural scenes from single images. In European
Conference on Computer Vision, pages 515–534. Springer,2022.
[13] Ce Liu. Beyond pixels: exploring new representations andapplications for motion analysis. PhD thesis, Massachusetts
Institute of Technology, 2009.
[14] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, andDaniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 6038–6047, 2023.
[15] Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, andMartin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 18444–18455, 2023.
[16] Ahmed A Shabana. Theory of vibration, volume 2. Springer,1991.
[17] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin
Huang. 3d photography using context-aware layered depth
inpainting. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 8028–8038,2020.
[18] Jos Stam. Stochastic dynamics: Simulating the effects ofturbulence on flexible structures. Computer Graphics Forum,
16(3), 1997.
[19] Qianqian Wang, Zhengqi Li, David Salesin, Noah Snavely,
Brian Curless, and Janne Kontkanen. 3d moments from nearduplicate photos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
3906–3915, 2022.
[20] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen,Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis
with motion controllability. arXiv preprint arXiv:2306.02018,2023.
[21] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao
Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In International Conference on Learning Representations (ICLR),
2021.