Animating Pictures with Eulerian Motion Fields
Aleksander Holynski1

Brian Curless1
1

Steven M. Seitz1

Richard Szeliski2

2
University of Washington
Facebook
eulerian.cs.washington.edu

Figure 1: Given a single input image (a), our method estimates an image-aligned motion field (b), and uses it to create a looping video (c).

Abstract

was taken. Using these priors, we can often envision theimage as if it were animated, with smoke billowing out ofa chimney, or waves rippling across a lake. In this paper,
we propose a system that learns these same motion priors
from videos of real scenes, enabling the synthesis of plausible motions for a novel static image and allowing us torender an animated video of the scene.
General scene motion is highly complex, involving perspective effects, occlusions, and transience. For the purposes of this paper, we restrict our attention to fluid motions,such as smoke, water, and clouds, which are well approximated by Eulerian motion, in particular, particle motion
through a static velocity field.
Our proposed method takes as input a single static image and produces a looping video texture. We begin by using an image-to-image translation network [29] to synthesize an Eulerian motion field. This network is trained using
pairs of images and motion fields, which are extracted froma large collection of online stock footage videos of naturalscenes. Through Euler integration, this motion field defines
each source pixel’s trajectory through the output video sequence. Given the source pixel positions in a future frame,
we render the corresponding frame using a deep warping
technique: we use an encoder network to transform the input image into a deep feature map, warp those features using
a novel temporally symmetric splatting technique, and use adecoder network to recover the corresponding warped color
image. Lastly, in order to ensure our output video loops
seamlessly, we apply a novel video looping technique that
operates in deep feature space.

In this paper, we demonstrate a fully automatic method
for converting a still image into a realistic animated looping video. We target scenes with continuous fluid motion,such as flowing water and billowing smoke. Our method relies on the observation that this type of natural motion canbe convincingly reproduced from a static Eulerian motion
description, i.e. a single, temporally constant flow field thatdefines the immediate motion of a particle at a given 2D
location. We use an image-to-image translation network to
encode motion priors of natural scenes collected from online videos, so that for a new photo, we can synthesize acorresponding motion field. The image is then animatedusing the generated motion through a deep warping technique: pixels are encoded as deep features, those features
are warped via Eulerian motion, and the resulting warped
feature maps are decoded as images. In order to produce
continuous, seamlessly looping video textures, we propose
a novel video looping technique that flows features both forward and backward in time and then blends the results. We
demonstrate the effectiveness and robustness of our method
by applying it to a large collection of examples including
beaches, waterfalls, and flowing rivers.

1. Introduction
For humans, a picture often contains much more than acollection of pixels. Drawing from our previous observations of the world, we can recognize objects, structure, andeven imagine how the scene was moving when the picture1Our contributions include (1) a novel motion representation for single-frame textural animation that uses Euler
integration to simulate motion, (2) a novel symmetric deep
splatting technique for synthesizing realistic warped frames,and (3) a novel technique for seamless video looping of textural motion.

used to inherit nearest-neighbor textural motion, assuming
a segmentation of the dynamic region is provided [18].
Transformations in latent space. Recent advances indeep learning have enabled realistic, high-resolution image
synthesis using generative adverserial networks (GANs).
Many of these systems operate by representing images orscenes as a latent feature vector, which is decoded into a
synthesized image. By perturbing the latent vector, or performing a randomized walk in the latent feature space, theresulting decoded images remain plausible, while also varying temporally [23, 8, 10]. These animations can visualize
the space of possible appearances, but do not necessarily
animate plausible motion.
Instead of a random walk, one can also directly control
movement by applying spatial warps to latent features [15].
Still, deciding how to warp the image is non-trivial — toproduce a realistic video, the applied transformations mustcorrespond with feasible motion in the scene.
Using learned motion or appearance priors. Deeplearning also enables motion synthesis from single-frame
inputs [7, 27]. Similarly, video prediction methods [35, 32,
11, 31, 19] can predict future video frames from a single
image, even modelling the inherent multi-modality of predicting the future. These techniques typically predict a setof future frames at once, and thus are limited to either low
spatial resolution or few predicted frames.
Most similar to our work, Endo et al. [6] demonstratehigh-quality motion and appearance synthesis for animating timelapses from static landscape imagery. In our evaluations, we provide comparisons to this technique, showingthat our method more reliably estimates motion for scenes
with fluids and animates videos with fewer visible artifacts.

2. Previous Work2.1. Video Textures
There is a large body of work aimed at producing looping
videos, known variously as video textures, cinemagraphs,
or live photos. These techniques typically take as input alonger video sequence and, through some analysis of the
motion, produce a single seamlessly looping video, or an
infinite (yet not obviously looping) video [21]. The term
cinemagraph often refers to selective animation of looping clips, where only certain parts of the frame, as chosen
by the user, are animated (or de-animated) [2]. Newer approaches [26, 34, 13, 12, 17] perform this task fully automatically, determining which regions are easily looped, andwhich regions contain motions that are large in magnitude
or otherwise unsuitable for looping. These approaches havealso been extended to operate on specific domains, such asvideos of faces [2], urban environments [33], panoramas[1], and continuous particle effects [3, 14]. All these methods, however, require a video as input.

2.2. Single-image animation
There are also a number of methods aimed at animating
still images. Recently, these techniques have gained popularity through commercial applications such as Plotagraph1
and Pixaloop2 , which allow users to manually “paint” motion onto an image. In the following, we focus on approaches to perform some of this annotation automatically.

3. Overview
Given a single static image I0 , we generate a looping video of length N + 1, consisting of frames It witht ∈ [0, N ]. Our pipeline begins by using an image-to-imagetranslation network to estimate a corresponding motion field
M (Section 4), which is used to define the position of eachpixel in all future frames. We use this information to animate the image through a deep warping technique (Section 5). Finally, in order to produce seamlessly looping
videos, we introduce a technique to ensure that our videos
always start and end with the same frame (Section 5.2). Our
approach is summarized in Figure 2.

Physical simulation. Instead of manually annotating thedirection and magnitude of motion, the motion of certain
objects, such as boats rocking on the water, can be physically simulated, as long as each object’s identity is known
and its extent is precisely defined [5], or automatically identified through class-specific heuristics [9]. Since each object
category is modeled independently, these methods do noteasily extend to more general scene animation.
Using videos as guidance. Alternatively, motion or appearance information can be transferred from a userprovided reference video, containing either similar scene
composition [20], aligned information from a different domain, such as semantic labels [28], or unaligned samples
from the same domain [24, 4]. Instead of a single userprovided video, a database of homogeneous videos can be

4. Motion estimation
We begin by describing the motion model and the motion estimation network. Given an image as input, wewish to synthesize plausible motion for the observed scene.
Prior work accomplishes this task through recurrent prediction of incremental flow fields [6], theoretically enabling

1 https://app.plotaverse.com
2 https://www.pixaloopapp.com

2

Figure 2: Overview: Given an input image I0 , our motion estimation network predicts a motion field M . Through Euler integration, M
is used to generate future and past displacement fields F0→t and F0→t−N , which define the source pixel locations in all other frames t.
To animate the input image using our estimated motion, we first use a feature encoder network to encode the image as a feature map D0 .
This feature map is warped by the displacement fields (using a novel symmetric splatting technique) to produce the corresponding warped
feature map Dt . The warped features are provided to the decoder network to create the output video frame It .

F0→t does not incur additional calls to the network — it
only uses information from the already-estimated M .

generation of an infinite number of future frames at highresolution. In practice, however, recurrent estimation often
results in long-term distortion, since predicted motions are
dependent on previously generated frames. In contrast, ourmotion field is only predicted once, given the input image,and thus does not degrade over time. Even though we use asingle static motion field to represent the motion of an entire video, we can still model complex motion paths. This isbecause our motion field M is a static Eulerian flow field,
i.e., a 2D map of motion vectors where each pixel’s value
defines its immediate velocity, which does not change overtime. We use M to simulate the motion of a point (particle)
from one frame to the next via Euler integration:x̂t+1 = x̂t + M (x̂t ),Note that unlike Endo et al. [6], who predict backward
flow fields for warping (i.e., using bilinear backward sampling), we predict the forward motion field, i.e., alignedwith the input image. In our evaluations, we show that predicting forward motion results in more reliable motion prediction and sharper motion estimates at object boundaries.
As a result, this enables more realistic animation of nearby
scenes with partial occlusions, since regions that are moving are more precisely delineated from those that are not.

(1)

5. Animation

where x̂t is the point’s (x, y) coordinate in frame t. In otherwords, treating each pixel as a particle, this motion field isthe flow between each frame and its adjacent future frame:
M (x̂t ) = Ft→t+1 (x̂t )Once we have estimated the displacement fields F0→t
from the input image to all future frames, we use this information to animate the image. Typically, forward warping,i.e., warping an image with a pixel-aligned displacement
field, is accomplished through a process known as splatting. This process involves sampling each pixel in the input image, computing its destination coordinate as its initial
position plus displacement, and finally assigning the source
pixel’s value to the destination coordinate. Warping an image with splatting unfortunately suffers from two significant
artifacts: (1) the output is seldom dense — it usually contains holes, which are regions to which no source pixel isdisplaced, and (2) multiple source pixels may map to thesame destination pixel, resulting in loss of detail or aliasing.
Additionally, the predicted motion fields may be imperfect,
and naively warping the input image can result in boundary artifacts. In the following section, we introduce a deep
image warping approach to resolve these issues.

(2)

To synthesize this motion field, we train an image-toimage translation network [29] on color-motion pairs, suchthat when provided with a new color image I0 , it estimatesa plausible motion field M . Given an image, M is only estimated once through an inference call to the network. Once
estimated, it can be used to define the source pixel positions
in all future frames t by recursively applying:
F0→t (x̂0 ) = F0→t−1 (x̂0 ) + M (x̂0 + F0→t−1 (x̂0 )) (3)This results in displacement fields F0→t , which define thetrajectory of each source pixel in I0 across future frames
It . These displacement fields are then used for warping the
input image, as further described in Section 5. Computing
3and are not replaced by others. This effect is especiallyprominent at motion “sources”, such as the top of a waterfall, where all predicted motion is outgoing. Although
our decoder network is intended to fill these holes, it is still
desirable to limit the complexity of the spatio-temporal inpainting task, as asking the network to animate an entire
waterfall from a small set of distant features is unlikely to
produce a compelling and temporally stable video.
Our solution to this problem leverages the fact that our
motion is textural and fluid, and thus much of the missing textural information in unknown regions can be feasibly
borrowed from other parts of the frame that lie along thesame motion path. With this intuition in mind, we describe
a symmetric splatting technique which uses reversed motion to provide valid textural information for regions which
would otherwise be unknown.
So far, the process we have described to generate an animated video involves warping the encoded feature map D0
by F0→t to produce future feature maps Vf = {D0 ...DN },which are decoded to produce the output video frames.
However, since our motion map M defines the motion between adjacent frames, we could just as easily animate theimage by generating a video of the past, i.e., instead of
warping D0 into the future, use −M to compute F0→−t ,resulting in warped feature maps Vp = {D−N ...D0 }. Decoding this feature video produces an equally plausible animation of the frame, with the main difference being thatthe large unknown regions in Vp occur at the start of thesequence, as opposed to at the end of the sequence in Vf .
In fact, because the direction of motion has been reversed, the motion sources have been replaced with motion“sinks” and vice versa (Figure 4). This means that the locations of the unknown regions in Vp are also largely complementary to those found in Vf . For instance, if our input
image contains a waterfall, Vf will begin with the input feature map D0 , and pixels will gradually flow down the waterfall, eventually accumulating at the bottom, and leaving alarge unoccupied region at the top. Conversely, Vp will begin with pixels accumulated at the top of the waterfall, and alarge hole at the bottom, and will end with D0 . We leverage
this complementarity by compositing pairs of feature maps
(one in the past, one in the future) to produce a feature map
which is typically fully dense.
We perform this composition through joint splatting: we
splat each pixel of D0 twice to the same destination frame,
once using F0→t and once using F0→t−N . Note that F0→t
does not necessarily equal −F0→−t , rather F0→−t is theresult of applying −M recursively through Eq. 3. As before, we use the softmax splatting approach with a networkpredicted per-pixel weighting metric to resolve conflicts.
This process results in a composite feature map that seldom
contains significant holes, enabling generation of longer
videos with larger magnitude motion.

Figure 3: Deep warping: Above: Naı̈ve splatting of RGB pixels
results in increasingly large unknown regions over time, shown inmagenta. Below: For the same frames, our deep warping approachsynthesizes realistic texture in these unknown regions.

5.1. Deep image warping
Given an image I0 and a displacement field F0→t , weadopt a deep warping technique to realistically warp the input frame and fill unknown regions. Our method consists ofthree steps: (1) use an encoder network to encode the input
image I0 as a deep feature map D0 , (2) use the estimateddisplacement field F0→t to splat those features to a future
frame, producing Dt , and (3) use a decoder network to convert the warped features to an output color image It . For
our encoder and decoder networks, we use variants of thearchitectures proposed in SynSin [30]. More implementation details are provided in Section 6.
As mentioned in the previous section, unlike backward
warping, splatting may result in multiple source pixels mapping to the same destination coordinate. In these cases, it isnecessary to decide which value will occupy the pixel inthe destination image. For this, we adopt softmax splatting[16], which assigns a per-pixel weighting metric Z to thesource image, and uses a softmax to determine the contributions of colliding source pixels in the destination frame:P
D0 (x̂) · exp(Z(x̂))0P(4)
Dt (x̂ ) = x̂∈X
x̂∈X exp(Z(x̂))
where X is the set of pixels which map to destination pixelx̂0 . Our method infers Z automatically as an additional
channel of the encoded feature map. The learned metricallows the network to assign importance to certain features
over others, and the softmax exponentiation avoids uniform
blending, resulting in sharper synthesized frames.
Symmetric Splatting. As feature pixels are warped
through repeated integration of our motion field M , wetypically observe increasingly large unknown regions (Figure 3), occurring when pixels vacate their original locations
4

guaranteed to be identical, and thus, our video is guaranteed
to loop seamlessly. As such, we modulate the contributionof each feature map by introducing a temporal scaling coefficient to Eq. (4):P
αt (x̂) · D0 (x̂) · exp(Z(x̂))
0P
Dt (x̂ ) = x̂∈X
(5)
x̂∈X αt (x̂) · exp(Z(x̂))
where X is the set of pixels which map to destination pixelx̂0 , either by warping forward or backward in time. For agiven frame t, we set:(tx̂ ∈ Vp(6)
αt (x̂) = N t
1 − N x̂ ∈ Vf

Figure 4: Seamless looping: An illustrated example of how seamless loops are created. Two feature videos are created by warping
D0 . The first, Vf , contains the result of integrating the motion field
M , resulting in a video starting with the input image and animating into the future. The second, Vp , instead uses −M , resulting
in a video starting in the past and ending with the input frame.
These two videos typically contain complementary unknown regions (shown in magenta). Before decoding, we combine the two
feature maps via joint splatting. We modulate the contribution ofeach using splatting weights αt , such that in the blended composite, the first and last frames are guaranteed to equal D0 , thus
ensuring a seamless loop. Note that RGB images are shown forvisualization, but these are both deep feature videos.

Although the scaling coefficient αt is linearly interpolated,
the resulting composited feature video is not a linear interpolation of Vp and Vf , since coinciding splatted featuresfrom each are typically not from the same input locations,and thus have different values of Z. Since the value of Z is
unconstrained and exponentiated, the overall magnitude of
our weighting function (αt (x̂) · exp(Z(x̂))) can vary significantly, and thus our composited feature map seldom contains equally blended features. The added coefficient αtserves as a forcing function to ensure that the composited
feature maps Dt are equal to D0 at t = 0 and t = N , but
composited features will transition from Vf to Vp at different rates per-pixel, depending on the relative magnitudes ofthe splatted Z values.
Training on regular videos. Training our deep warping
component (i.e., our encoder and decoder networks) to produce looping videos introduces an additional challenge: ourtraining dataset consists of natural non-looping videos. In
other words, the looping video we are tasking our networks
with generating does not exist, even for our training examples, and thus, it’s non-trivial to formulate a reconstruction
loss for supervision. Therefore, as illustrated in Figure 5,
we modify the task for training: instead of warping one
frame in two directions, we use two different frames, one
from the start of the video clip I0GT , and one from the end
GTIN, encoded separately as feature maps. We additionallypredict a motion field M from I0GT , which is integrated to
produce displacement fields F0→t and F0→t−N . The two
feature maps, D0 and DN , are respectively warped by F0→t
and F0→t−N to an intermediate frame t, using our joint
splatting technique with the weights defined in Eq. 5. Finally, the composited feature map Dt is decoded to an image It , and a loss is computed against the real intermediate
frame ItGT . At testing time, we perform the same process,except that instead of two input images, we use only one image, warped in both directions. This process is effectively
training the network to perform video interpolation, and at
inference time, using the network to interpolate between aframe and itself, while strictly enforcing the desired motion
by warping the feature maps.

5.2. Looping
In this section, we focus on ensuring that our output
videos loop seamlessly. To this end, we first describe a modification to the splatting weights that guarantees that the firstand last output video frames will be identical. Then, we describe an approach that enables end-to-end training without
requiring a dataset of looping videos.
Prior work [6] produces looping videos through crossfading: an animated, but non-looping, video is generated
first, and a crossfade is applied across the two ends of thevideo to smooth out any jarring frame transitions. This approach can be quite effective in certain cases, but often produces artifacts in the form of double edges and ghosting.
Instead of directly crossfading the animated video, our approach performs the transition in deep feature space, andprovides the smoothly transitioning feature maps to the decoder. This allows us to enforce smooth transitions, whilestill producing images that contain realistic texture, avoiding many of the artifacts of direct crossfading.
Looping weights. Our looping technique relies on the observation that our two warped sequences Vp and Vf eachhave the input feature map D0 on opposite ends of the sequence, as illustrated in Figure 4. With this in mind, if weare able to smoothly control the contribution of each, suchthat the first frame contains only the values in Vf and thelast frame contains only the values in Vp , our feature maps
(and our decoded images) on opposite ends of the video are5over a 2-second window. This effectively filters most motion which is cyclic, since pixels with cyclic motion willusually have been observed moving in opposing directions.
We use only training videos from stationary cameras.
The motion estimation network is trained using 5 imagemotion pairs from each of our 1096 training videos (a totalof 5480 pairs) for 35 epochs, using the default parametersfrom Pix2PixHD [29]. Prior to training, we resize all ourimages to a standard size of 1280 × 720.
The warping component is trained on 5 short video clips
from each of our 1096 training videos. A training triplet(start frame, middle frame, end frame) is selected from each
video clip at random during training, further increasing theeffective dataset size. We also apply random augmentation to our training examples, including horizontal flips andcropping. We train the network on batches of 8 imagesof size 256 × 256 for 200 epochs, using a discriminatorlearning rate of 3.5 × 10−3 and generator learning rate of
3.5 × 10−5 . We use the same losses and loss balancing coefficients shown in SynSin [30].
As in Niklaus et al. [16], and for the purpose of training
stability, we first train our two components separately. We
start by training our motion estimation network supervised
only by the ground-truth motion. Then, we train our warping component, which consists of the encoder and decoder
networks, using the ground-truth motion fields as input. Finally, we fine-tune the two end-to-end, by warping with thepredicted motion from the motion estimation network. This
final step allows each network to best adapt to the properties and error characteristics of the other. We fine-tune for20 epochs with discriminator and generator learning ratesof 1 × 10−3 and 1 × 10−5 .

Figure 5: Training: As described in Section 5.1, each frame in our
generated looping video is composed of textures from two warped
frames. To supervise this process during training, i.e., to have areal frame to compare against, we perform our symmetric splatting
using the features from two different frames, I0 and IN (instead of
I0 twice, as in inference). We enforce the motion field M to matchthe motion estimated from the ground truth video M GT , and theoutput frame It to match the real video frame ItGT . For both, we
use a combination of photometric and discriminative losses.

6. Implementation Details
In this section, we provide more details about the implementation of our method. First, we provide a summaryof the network architectures used for the motion estimation
and warping networks. Then, we provide details about our
training and inference pipelines. In order to facilitate future
work, full training and inference code will be made publicly
available at eulerian.cs.washington.edu.
Network architecture. For the feature encoder and decoder networks, we use the architectures proposed in
SynSin [30], which have shown compelling results for
single-image novel-view synthesis. Since our aim is not togenerate new viewpoints, but rather to animate the scene,
we replace the reprojection component with the softmax
splatting technique proposed in Niklaus et al. [16]. Additionally, we replace the noise-injected batch normalization layer from SynSin with the modulated convolution approach proposed in Karras et al. [10] (to which we also
provide a latent noise vector). This modification greatly
helps reduce visual artifacts and enables stable discriminator training with smaller batch sizes (a necessity for limited
GPU memory). For our motion estimation network, we use
the architecture proposed in Pix2PixHD [29].

Inference. Our looping output videos have length N =200 with a framerate of 30 frames per second. Each sequence is processed in 40 seconds on a Titan Xp GPU.

7. Results & Evaluation
We first present a quantitative analysis of our method,and show comparisons with the state-of-the-art in stillimage animation [6] (Section 7.1), as well as ablated variations of our method. Then, we show qualitative results ofour method on a diverse collection of input images (Section 7.2). We refer readers to our supplementary video for a
full collection of visual results.

Training. We focus on natural scenes with fluid textures
such as waterfalls, turbulent streams, and flowing waves.
For our training data, we collected and processed a set of1196 unique videos of textural motion from an online stockfootage website3 . We use 1096 for training, 50 for validation, and 50 for testing. To generate ground-truth motion
fields, we use a pre-trained optical flow estimator [25] tocompute the average optical flow between adjacent frames

7.1. Quantitative evaluation
In this section, we present our experiments evaluating thedifferent components of our method, i.e., (1) a novel motion
representation, (2) a novel symmetric splatting technique,and (3) a novel looping technique.
Motion representation. We evaluate the effectiveness of
our proposed motion representation (integrated Eulerian

3 www.storyblocks.com

6

Figure 6: Examples of the input images (top), alongside their corresponding synthesized motion fields (bottom). Full resolution images,
along with their corresponding animated videos, can be found in the supplementary video.

flow) by comparing our predicted motion to ground truth
pixel positions in future frames of the video. We establish
ground truth motion by densely tracking all pixels througha sequence of 60 frames, using an off-the-shelf opticalflow estimator [25]. We report the average Euclidean error between the ground truth positions and those estimatedthrough our synthesized motion field, i.e., the endpoint error. We compare our proposed method to the following variants: (1) the per-frame recurrent estimation from Endo etal. [6], (2) directly predicting F0→N and linearly interpolating intermediate motion F0→t as Nt F0→N , and (3) training
our motion network to predict the backward flow field, i.e.,
M = F1→0 (and thus all splatting is replaced by backward
warping). The results of this experiment can be found in
Figure 7. We see that our method is able to most faithfully
reproduce the ground-truth motion for our scenes. Empirically, we observe that the methods employing backward
warping produce a majority of errors at motion boundaries,
such as occlusions. We hypothesize that these differences
are because the network is more easily able to predict an
output that is spatially aligned with the input image.
Since images may often have many plausible motion directions, we ensure that the comparisons performed in thisexperiment are on video examples that contain unambiguous motion, eg. waterfalls and rivers. In order to identify
these samples, we gave each of 5 users 50 images fromdifferent scenes, and asked them to manually annotate thelikely motion direction. We retained the scenes for which
all the annotations were within 30 degrees of the medianmotion direction in our ground truth flow values. Additionally, since we prefer the motion comparison to be agnostic to animation speed, i.e., animating the scene realistically but in slow-motion is acceptable, we solve for a persequence time-scaling constant that best aligns the motion
magnitudes of the predicted and ground-truth displacement
fields. This constant is computed for all methods and is usedin all our comparisons.
The motion estimation network from Endo et al. [6] uses

Figure 7: Quantitative evaluation, motion prediction: We evaluate the quality of the predicted motion by comparing the pixel
positions in 60 future frames to those in the ground-truth video.
We compare our proposed motion representation to three alternative methods, described in Section 7.1. The shaded region shows
the range of predictions produced by Endo et al. [6]. We find thatour proposed motion representation is able to most reliably reproduce the true motion information for scenes with fluid motion. Allcomparisons are performed on images of size 1280 × 720.

a latent code as input to the network, and different latentcodes produce different predicted motions. To consider
all possible outcomes of their method, we randomly sample 100 latent codes from the training codebook and reportstatistics on the resulting synthesized motions.
Video synthesis. Second, we evaluate the choice of warping technique. Given the same flow values for a set of testing (unseen) video clips, we evaluate five future frame synthesis techniques: (1) naı̈ve color splatting, where the feature encoder and decoder are not used, and instead the color
values are warped, (2) backward warping, where the forward displacement field is inverted [22], and then backward
warping is applied, such that no holes occur during warping, (3) our method without the network inferred weights,
7

Naı̈ve color splatting
Backward Warping
Ours - Z(x̂) = 1
Ours - No Symmetric Splatting
Ours - Full

↑ PSNR
7.90
10.29
13.88
12.19
14.63

↑ SSIM
0.313
0.409
0.541
0.493
0.619

parison, we instead perform a user study, in which 100
MTurk users are asked to rank the four variants by visualquality. Table 2 shows the results of the user study, which
demonstrate that our proposed approach compares favorably against the alternatives. In the supplemental material,
we also show a visual comparison of these approaches. For
our comparison to Endo et al. [6], we use an implementation provided by the authors, trained on our dataset for therecommended 5000 epochs. Note that we only use the motion estimation component of their method, as our scenes
are not timelapses, and therefore do not have as significant
changes in overall appearance.

↓ LPIPS
0.595
0.483
0.344
0.418
0.313

Table 1: Quantitative evaluation, video synthesis: We evaluate
the quality of future frame predictions by comparing 60 synthesized frames with corresponding frames in the ground truth video.
We compare our method to four alternatives, described in Section 7.1. All variant use our proposed motion estimation network.

Endo et al. [6]Ours - No αtOurs - Crossfade
Ours - Full

S≥1
348
173
470
500

S≥2
101
183
418
472

S=3
9
0
43
448

7.2. Qualitative evaluation
For evaluation purposes, we demonstrate our system ona large collection of still images. A subset of these images,
along with their synthesized motions, can be seen in Figure
6. The dataset contains a variety of natural scenes, including waterfalls, oceans, beaches, rivers, smoke, and clouds.
In the supplementary video, we provide a larger set of input
images and final rendered animations, as well as intermediate outputs such as synthesized motion fields.

Table 2: User study: We perform a user study to compare four
techniques for producing looping videos. We collected 5 uniqueannotations for each of 100 samples. We direct users to judge
the visual quality and realism of each looping video and rank thevideos with unique scores S = [0, 3], where 3 is best. We report
the cumulative number of annotations above a certain ranking. On
average, users rank our method higher than the alternatives.

In the results, we can see that the network learns important motion cues, such as perspective (i.e. motion is larger
for objects closer to the camera), water turbulence, and detailed flow direction from surface ripples. By comparison,
we find that the generated videos using the method in Endoet al. [6] more often produces videos with unrealistic motion or incorrect motion boundaries. Additionally, since ourmethod performs warping in the deep feature domain, instead of explicitly warping RGB pixels, our results do notcontain many of the same characteristic artifacts of warping, such as shearing or rubber-sheeting. Finally, we observe that our results loop more seamlessly, without obvious
crossfading or ghosting.

i.e., Z(x̂) = 1 for all pixels, (4) our method without symmetric splatting, and (5) our full method. We again use
temporally-scaled sequences with unambiguous motion tocompare each method’s synthesized frames with the groundtruth future video frames. We perform this comparison using PSNR, SSIM, and LPIPS [36]. Table 1 shows a quantitative comparison of these techniques, demonstrating thatour proposed approach outperforms the alternatives at synthesizing future frames when the same motion is provided.
Additionally, in the supplemental material, we show a qualitative comparison of these techniques. Compared to ourapproach, we observe that standard color splatting resultsin significant sparsity, i.e. many holes with unknown color.
Backward warping instead fills these holes with interpolated (stretched) texture, which in most cases is equally jarring. Feature warping without inferred Z(x̂) values resultsin blurred details, since features are more often evenly combined. Removing symmetric splatting results in large unknown regions, which are filled in by the decoder networkwith blurry and often unrealistic texture.

8. Conclusion
In this paper, we have presented a method that can synthesize realistic motion from single photographs to produce
animated looping videos. We introduced a novel motion
representation for single-image textural animation that uses
Euler integration. This motion is used to animate the input image through a novel symmetric splatting technique, inwhich we combine texture from the future and past. Finally,
we introduced a novel video looping technique for singleframe textural animation, allowing for seamless loops of ouranimated videos.

Looping. Finally, we evaluate the choice of our looping
technique. We compare four approaches: (1) our synthesis
technique followed by the crossfading technique describedin Endo et al. [6], (2) the end-to-end pipeline described inEndo et al. [6], (3) our approach without the scaling coefficient αt introduced in Eq. 5 and (4) our proposed approach.
Since we do not have a ground truth looping video for com-

We demonstrated our method on a wide collection of images with fluid motion, and showed that our method is able
to produce plausible motion and realistic animations.
8

References[15] Elizaveta Logacheva, Roman Suvorov, Oleg Khomenko,
Anton Mashikhin, and Victor Lempitsky. Deeplandscape:Adversarial modeling of landscape video. arXiv preprint
arXiv:2008.09655, 2020. 2[16] Simon Niklaus and Feng Liu. Softmax splatting for video
frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
5437–5446, 2020. 4, 6
[17] Tae-Hyun Oh, Kyungdon Joo, Neel Joshi, Baoyuan Wang, In
So Kweon, and Sing Bing Kang. Personalized cinemagraphs
using semantic understanding and collaborative learning. In
Proceedings of the IEEE International Conference on Computer Vision, pages 5160–5169, 2017. 2[18] Makoto Okabe, Ken Anjyo, Takeo Igarashi, and Hans-Peter
Seidel. Animating pictures of fluid using video examples.
In Computer Graphics Forum, volume 28, pages 677–686.
Wiley Online Library, 2009. 2[19] Junting Pan, Chengyu Wang, Xu Jia, Jing Shao, Lu Sheng,
Junjie Yan, and Xiaogang Wang. Video generation from single semantic label map. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
3733–3742, 2019. 2[20] Ekta Prashnani, Maneli Noorkami, Daniel Vaquero, andPradeep Sen. A phase-based approach for animating images
using video examples. In Computer Graphics Forum, volume 36, pages 303–311. Wiley Online Library, 2017. 2[21] Arno Schödl, Richard Szeliski, David H Salesin, and Irfan Essa. Video textures. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 489–498. ACM Press/Addison-Wesley Publishing Co., 2000. 2[22] Jonathan Shade, Steven Gortler, L. He, and Richard Szeliski.
Layered depth images. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pages 231–242. ACM Press/Addison-Wesley Publishing Co., 1998. 7[23] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning a generative model from a single natural image. In Proceedings of the IEEE International Conference
on Computer Vision, pages 4570–4580, 2019. 2[24] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov,
Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Advances in Neural Information Processing Systems, pages 7137–7147, 2019. 2[25] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.
Pwc-net: Cnns for optical flow using pyramid, warping,and cost volume. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages 8934–
8943, 2018. 6, 7
[26] James Tompkin, Fabrizio Pece, Kartic Subr, and Jan Kautz.
Towards moment imagery: Automatic cinemagraphs. In
2011 Conference for Visual Media Production, pages 87–93.
IEEE, 2011. 2[27] Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense
optical flow prediction from a static image. In Proceedingsof the IEEE International Conference on Computer Vision,
pages 2443–2451, 2015. 2[1] Aseem Agarwala, Ke Colin Zheng, Chris Pal, Maneesh
Agrawala, Michael Cohen, Brian Curless, David Salesin, andRichard Szeliski. Panoramic video textures. ACM Transactions on Graphics (TOG), 24(3):821–827, 2005. 2[2] Jiamin Bai, Aseem Agarwala, Maneesh Agrawala, and Ravi
Ramamoorthi. Selectively de-animating video. ACM Trans.
Graph., 31(4):66–1, 2012. 2[3] Kiran S Bhat, Steven M Seitz, Jessica K Hodgins, andPradeep K Khosla. Flow-based video synthesis and editing.
In ACM Transactions on Graphics (TOG), volume 23, pages
360–363. ACM, 2004. 2[4] Chia-Chi Cheng, Hung-Yu Chen, and Wei-Chen Chiu. Timeflies: Animating a still image with time-lapse video as reference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5641–5650,
2020. 2[5] Yung-Yu Chuang, Dan B Goldman, Ke Colin Zheng, BrianCurless, David H Salesin, and Richard Szeliski. Animating
pictures with stochastic motion textures. ACM Transactions
on Graphics (TOG), 24(3):853–860, 2005. 2[6] Yuki Endo, Yoshihiro Kanamori, and Shigeru Kuriyama. Animating landscape: Self-supervised learning of decoupledmotion and appearance for single-image video synthesis.
ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2019), 38(6):175:1–175:19, 2019. 2, 3, 5, 6, 7,
8[7] Ruohan Gao, Bo Xiong, and Kristen Grauman. Im2flow:
Motion hallucination from static images for action recognition. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 5937–5947, 2018. 2[8] Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan
Wermter. Improved techniques for training single-image
gans. arXiv preprint arXiv:2003.11512, 2020. 2[9] Wei-Cih Jhou and Wen-Huang Cheng. Animating still landscape photographs through cloud motion creation. IEEE
Transactions on Multimedia, 18(1):4–13, 2015. 2[10] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten,
Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 8110–8119, 2020. 2, 6
[11] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin
Lu, and Ming-Hsuan Yang. Flow-grounded spatial-temporal
video prediction from still images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 600–
615, 2018. 2[12] Jing Liao, Mark Finch, and Hugues Hoppe. Fast computationof seamless video loops. ACM Transactions on Graphics(TOG), 34(6):197, 2015. 2[13] Zicheng Liao, Neel Joshi, and Hugues Hoppe. Automatedvideo looping with progressive dynamism. ACM Transactions on Graphics (TOG), 32(4):77, 2013. 2[14] Chih-Yang Lin, Yun-Wen Huang, and Timothy K Shih. Creating waterfall animation on a single image. Multimedia
Tools and Applications, 78(6):6637–6653, 2019. 2

9

[28] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu,Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video
synthesis. arXiv preprint arXiv:1910.12713, 2019. 2[29] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao,Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In
Proceedings of the IEEE conference on computer vision andpattern recognition, pages 8798–8807, 2018. 1, 3, 6[30] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin
Johnson. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 7467–
7477, 2020. 4, 6
[31] Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo.
Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 2364–2373, 2018. 2[32] Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis
via cross convolutional networks. In Advances in neural information processing systems, pages 91–99, 2016. 2[33] Hang Yan, Yebin Liu, and Yasutaka Furukawa. Turning anurban scene video into a cinemagraph. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 394–402, 2017. 2[34] Mei-Chen Yeh and Po-Yi Li. An approach to automatic creation of cinemagraphs. In Proceedings of the 20th ACM
international conference on Multimedia, pages 1153–1156.
ACM, 2012. 2[35] Jiangning Zhang, Chao Xu, Liang Liu, Mengmeng Wang,
Xia Wu, Yong Liu, and Yunliang Jiang. Dtvnet: Dynamic
time-lapse video generation via single still image. arXiv
preprint arXiv:2008.04776, 2020. 2[36] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness ofdeep features as a perceptual metric. In Proceedings of theIEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 8

10