Volume 30 (2011), Number 7

Pacific Graphics 2011
Bing-Yu Chen, Jan Kautz, Tong-Yee Lee, and Ming C. Lin(Guest Editors)

Creating Fluid Animationfrom a Single Image using Video Database
Makoto Okabe†1,2

Ken Anjyo‡3

1 The University of Electro-Communications

2 JST PRESTO

Rikio Onai§1
3 OLM Digital, Inc. / JST CREST

Abstract
We present a method for synthesizing fluid animation from a single image, using a fluid video database. The user
inputs a target painting or photograph of a fluid scene along with its alpha matte that extracts the fluid region ofinterest in the scene. Our approach allows the user to generate a fluid animation from the input image and to entera few additional commands about fluid orientation or speed. Employing the database of fluid examples, the corealgorithm in our method then automatically assigns fluid videos for each part of the target image. Our method
can therefore deal with various paintings and photographs of a river, waterfall, fire, and smoke. The resulting
animations demonstrate that our method is more powerful and efficient than our prior work.
Categories and Subject Descriptors (according to ACM CCS): I.4.8 [Image Processing and Computer Vision]: Scene
Analysis—Motion

1. Introduction
Pictures were first animated in lift-the-flap books, and theanimation of pictures is now recognized as a classic visual effect in the animation industry. It is also an active area of research within the field of computer graphics [HAA97, IMH05, HDK07]. In the animation of pictures,the designer specifies a single target image along with several characteristics regarding motion and uses a computer tosynthesize animated sequences derived from the input. Of
course, the level of difficulty involved in animating a picture
varies markedly according to the complexity of the scene andthe objects to be animated. It is difficult to animate a pictureof a fluid. Early research [CGZ∗ 05, OAIS09] was successful
in designing fluid animation. Here, we focus on this challenging issue using an efficient data-driven approach to handle a wide variety of fluid motions that could not be animatedpreviously.
A picture of a fluid can be animated in three ways. First,
a physics-based fluid simulation can be applied to the fluid
part of a target image. However, it is difficult to set the many
† m.o@acm.org‡ anjyo@olm.co.jp§ onai@onailab.com
c 2011 The Author(s)Computer Graphics Forum c 2011 The Eurographics Association and Blackwell Publishing Ltd. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ,
UK and 350 Main Street, Malden, MA 02148, USA.

physical parameters required to reproduce the target appearance. The second method can synthesize relatively calm
fluid motions, such as water surfaces [CGZ∗ 05]. However,
our study focuses on synthesizing more dynamic motions,such as water splashes, smoke, or fire, in addition to calm
fluids. A third method, which we described in our previous
paper, requires users to specify a video and then transfers
its fluid features to the target image [OAIS09]. This technique relies on a single video example that limits variation in
the available fluid features. Another problem is that the userspecified motion field is temporally stationary, which limits
the dynamics. The user must also expend considerable effort
to find an appropriate video example and specify the motion
field.
To address these problems, we develop a data-driven
method to create a fluid animation from a picture (Fig. 1).
The user inputs a target image and an alpha matte that extracts the fluid region of interest, while providing a few optional suggestions about fluid motion (i.e., flow direction andspeed). We employ a video database that includes hundredsof video examples of fluids, which therefore helps the user
to synthesize better quality animations with less effort thanwith previous methods. The quality of the synthesized animation is improved due to the large variety of available video
examples. We test the flexibility of our method by synthesiz-M. Okabe et al. / Creating Fluid Animation from a Single Image using Video Database

Figure 1: We employ a database of video examples of fluids (a). The user specifies a target image (b) with a few optionalsuggestions about fluid motion, e.g., sketches of flow direction, shown as orange arrows. The user also provides an alpha matteof the region of interest (c). The system synthesizes an animation (d).
ing a wide variety of fluid animations, i.e., river, waterfall,smoke, and fire.
The present study makes in three major technical contributions to solve the problem addressed in this paper.
First, we develop an image-search-based technique to efficiently extract local fluid features from a video database. We
then cut each video into smaller pieces, which are vectorquantized to construct a bag-of-features codebook for efficiently finding a video piece with an appearance that is similar to part of the picture. Second, we develop an algorithm
to assign the video pieces to the picture so that the integrated
appearance and motion become smooth. This task is formulated as a multi-label assignment problem based on theMarkov random field (MRF). The third contribution is theextension of the original synthesis algorithm. In our previous study, we decomposed a video into three components,
i.e., the average image, the motion field, and the residuals [OAIS09]. However, we found that two components (theaverage image and the differences between the average image and the original frames) produce a wider variety of dynamics than previously possible by using the video databaseeffectively. Additionally, our approach markedly reduces theuser burden, as illustrated below.
2. Previous Work
Image and Video Database Image and video searching isan active research field. Recently, this kind of search has
also been applied to image and video syntheses for computer graphics applications. Millions of images are usefulfor scene completion [HE07]. Sketch2Photo allows a user todraw rough sketches and search for adequate images from a
database to synthesize a desired scene [CCT∗ 09]. Skyfinder
searches for a user-desired sky image, modifies its appearance, and creates a composite [TYS09]. With regard to video
synthesis, SIFT Flow can be used to estimate the motion
field in a single image and also to transfer part of a video
example onto a single image [LYT∗ 08]. Webcam Clip Art
provides a video database of outdoor scenes and is usefulfor scene relighting [LEN09]. However, no existing methodaddresses the problem of animating a picture of fluid using
a video database.

Animating Pictures Many methods have been proposed for
creating an animation from a single image [HAA97,IMH05,
HDK07]. These methods are useful for touring into a picture and creating character animations, but fluid animation
is beyond their scope. Chuang et al. proposed a method tosynthesize an animation with stochastic motion from a single image [CGZ∗ 05], but this technique only supports oscillation of the water surface, such as ripples, and not flowing
fluid animation. Lin et al. synthesized animation from multiple, instead of single, high-resolution stills [LWW∗ 07].
Video Texture Synthesis Video texture synthesis has
also been well-explored [WL00, BJEYLW01, SSSE00,
DCWS03]. However, it is difficult for users to modify theappearance or motion of a synthesized animation. Wang and
Zhu analyzed fluid animation, represented it with textons,
and synthesized an animation [WZ03]. Bhat et al. proposeda sketching interface that enables users to edit a fluid animation of a video example [BSHK04]. Users can also change
the appearance of the animation, but the problem of animating a picture of fluid has not yet been solved. Kwatra et al.
developed a method that can design the animation of a texture flowing over a user-specified motion field [KEBK05],
where only a stationary motion field is demonstrated without any high-frequency fluid features. Ma et al. extended
example-based texture synthesis to allow an exemplar to affect the details of motion fields [MWGZ09]. These methods
enable users to modify an existing fluid animation and synthesize an animation, but they do not allow users to specify asingle image as an appearance constraint. Okabe et al. developed a technique for animating a picture of fluid [OAIS09];
this technique requires users to design a motion field manually and to search for an appropriate video example. We extended this work, incorporating a video database to reduce
user effort and to improve the quality of the synthesized animations.
3. System Overview
Given a target image of fluid that a user wants to create a
fluid animation, our process involves selecting appropriate
video examples from a database, assigning them onto partsof the target image, and integrating them all seamlessly intoc 2011 The Author(s)c 2011 The Eurographics Association and Blackwell Publishing Ltd.

M. Okabe et al. / Creating Fluid Animation from a Single Image using Video Database

Figure 2: System overview.

a final animation. We assume that the user inputs a single
target image and an alpha matte that extracts the fluid regionof interest. The user can also provide a desired motion fieldby sketching the flow direction and painting a speed map,
which is used as a constraint in the assignment process.
Our system consists of three components: 1) constructionof a video database of fluids (Fig. 2-a), where each video
example is cut into small pieces, 2) a best-match search for
an appropriate video example piece and assignment of thisto a part of the target image, and 3) synthesis of the final
animation through seamless integration of all the assigned
pieces and adjustment of the overall appearance. The offline process of database construction begins with gathering
original video examples of fluids (Fig. 2-b). To increase thenumber of video examples, we cut each video example into
smaller pieces (Fig. 2-c). For each video example, we compute the average image by averaging the frames (Fig. 2-e) toobtain representative appearance information. We also compute differences between the average image and the frames(Fig. 2-f) that have no significant color properties but capture high-frequency fluid features. From all of the averaged
images in the database, we construct a bag-of-features codebook and describe each average image using a histogram ofvisual words (Fig. 2-d).
Given a target image (Fig. 2-g), we cut it into pieces using
the same process used in the database construction (Fig. 2i). Next, we compute the histogram of visual words for eachpiece (Fig. 2-h), perform a best-match search between histograms of visual words (see Figs. 2-d and h), and assign
video example pieces that have appearances similar to target
image pieces. When a user-specified motion field is given, it
is used as a constraint for solving the assignment problem.
Based on the assignment results, differences are copied onto
the corresponding target image pieces (Fig. 2-j). Finally, allassigned differences are integrated seamlessly and the fic 2011 The Author(s)c 2011 The Eurographics Association and Blackwell Publishing Ltd.

nal appearance is synthesized by adjusting the appearance(Fig. 2-k).
Our data-driven transfer of fluid features to a target image
was inspired by Image Analogies [HJO∗ 01]; the transferreddifferences relate to the target image piece in the same way
as the differences originally related to the average image ofthe video example. Our method differs from Image Analogies in that it is not pixel-based but rather is a patch-based
texture synthesis, similar to Image Quilting [EF01]. When
integrating patches, Image Quilting computes a minimum
error boundary cut to remove seams between patches; wecompute alpha blending using a computationally inexpensive image pyramid. A video example piece is decomposed
into its average image and differences; the former captures
the overall low-frequency appearance and the latter has highfrequency fluid features. The high-frequency features can be
integrated seamlessly through simple alpha blending.
4. Database of Video Examples
We need to prepare as many video examples as possible, because a large amount of data is always important to achieve
high-quality data-driven image synthesis [HE07, TYS09].
However, despite the large volume of video data available
on the Internet, we found it difficult to gather the tens of
thousands of video examples suitable for our purpose. Ourcriteria for video examples are:
• The camera be fixed and focused on the fluid itself, i.e.,
the fluid must be the primary character of the video.
• The video examples of fluids need to be of sufficiently
high quality for animation synthesis.
• To simplify our method, we use only video examples withno other significant moving object other than fluids.
We gathered several hundred video examples of water scenes
from commercially available video footage collections, but
discarded any that did not satisfy our requirements. Finally,M. Okabe et al. / Creating Fluid Animation from a Single Image using Video Database

we have a total of 151 video examples of water scenes(Fig. 3). All video examples in the database have a resolution
between 640 × 480 pixels and 400 × 300 pixels.

Figure 3: Thumbnails of video examples in the database.
All video examples satisfy three requirements: 1) the fluid is
the main character in each scene, 2) each video example hasa resolution quality between 640 × 480 pixels and 400 × 300
pixels, and 3) no other moving objects appear in the scene.
To increase the amount of data, our strategy is to observe
each video example locally. For example, a video example ofa waterfall may have a different global shape from the shapeof the waterfall in the target image. However, its local parts,
such as droplets or splashes, may be used to animate a partof the target image.
To increase the number of video examples in the database,we flip each video example horizontally (thereby doubling
the number of examples); we also scale and rotate each example (Fig. 4-a). We then cut examples into smaller video
pieces (Figs. 2-c and Fig. 4-b). We make three versions ofeach video example by scaling them to 50, 75, and 100% of
their original size. We also make three versions of each video
example by rotating the original by −22.5, 0.0, and +22.5
degrees. Each version is then cut into smaller pieces with aresolution of 48 × 48 pixels, allowing neighboring pieces tooverlap each other (Fig. 4-b). Such an idea of patch libraryis also used for synthesizing facial image [MPK09].

Figure 4: Database construction.
Video example pieces that include not only fluids but also
other objects such as rocks, trees, or the sky degrade the efficiency of the database. Therefore, we remove such stationary
examples by calculating the significance of motion. For eachpiece, we compute optical flow between neighboring frames(Fig. 5-b) to obtain an average through all of the frames(Fig. 5-c). In a video example, only a pixel position with asignificant averaged motion is used; other pixels are maskedout (Fig. 5-d). In addition, any video example in which no
pixel position has significant motion is completely removed
from the database. We use OFLib to compute a dense opticalflow [ZPB07]. This algorithm has three important open parameters: θ, λ, and the number of iterations, which we set to
1.0, 0.8, and 25, respectively. Because a fluid video has highfrequency dynamics, we set parameters that reduce sensitivity to such dynamics but still yield relatively smooth results.
We have collected 246, 477 video examples of water scenes.

Figure 5: Processing a video example piece.
To ensure efficient best-match searching and efficient assignment of video examples, we construct a bag-of-features
codebook, by which each video example piece is describedusing a histogram of visual words (Fig. 2-d). The bagof-features technique works well for searches involving
small image patches such as those in our database, especially when it is combined with spatial pyramid representation [SSSFF09]: while bag-of-features has only information of presence of textures, the spatial pyramid representation adds information of location of textures that is importanteven for our small patches. We make a representative image
by averaging the frames of each video example piece whenconstructing the codebook (Fig. 2-e). Averaging involves the
motion blur that often appears in a picture of fluid; this motion is often intentionally portrayed by a painter or a photographer to visualize the trajectories of fluid motion. We extract SIFT features of each representative image by applying
a SIFT descriptor, and compute 128 dimensional feature vectors at each point on a regular grid of 9 × 9 over the 48 × 48
pixel resolution image. The use of SIFT descriptors on a regular grid is often better for image searching than using key
points detected by Gaussian differences [LP05].
We then compute vector quantization on all extracted
SIFT features to obtain the visual words used to construct thecodebook. For the vector quantization, we apply a repeatedcluster bisectioning approach, because this method is reportedly faster and yields a better quality of clustering than Kmeans [ZKF05]. Similarly to K-means, we set the number ofclusters, k, based on experimental results for settings of 100,
200, 300, and 500; we have chosen 200 as the best parameter.
Given the codebook, we generate a bag of features for each
representative image of a video example piece. We assign
c 2011 The Author(s)c 2011 The Eurographics Association and Blackwell Publishing Ltd.

M. Okabe et al. / Creating Fluid Animation from a Single Image using Video Databaseeach SIFT feature to its nearest visual word and construct a
histogram with spatial pyramid representation [LSP06].
5. Assignment of Video Example Pieces

N/2

Given a target image, we first decompose it into pieces(Fig. 2-i) in the same manner as in the database construction (Fig. 4-b). Next, we assign an appropriate video example piece to each target image piece. Our solution of theassignment problem involves three criteria: 1) appearance
similarity between a video example piece and a target image piece; 2) smoothness of motion fields between neighboring assigned pieces; and 3) appearance similarity between
neighboring assigned pieces. The first criterion means thatthe more similar a video example piece is to a target image
piece, the more likely it is that both pieces have similar fluid
motions. The second criterion means that if a part of the target image is assigned a fluid motion, its neighboring pieceis likely to have a similar fluid motion, i.e., their velocities
and temporal video textural patterns would be similar. The
third criterion prevents neighboring pieces from generating
an inconsistent appearance in the video texture.
To address the first criterion, we compute appearance similarity by comparing bag-of-features histograms (Figs. 2-dand h). As in the database construction, we extract SIFT
features on the regular grid in each target image piece anddescribe the image feature as a histogram of visual words.
Appearance similarity is computed using a histogram intersection,k

I(He , Ht ) = ∑ min(He [i], Ht [i]),

(1)

i=1

where He and Ht are the histograms of visual words of avideo example piece and a target image piece, respectively.
For the second criterion, we assume that the similarity
measure of the average motion field is a simple definitionof smoothness between neighboring motion fields (Fig. 5-c).
However, this definition is insufficient, as shown in Fig. 6,
when each video example piece has a similar average motion field; that is, when they all flow from top to bottom at asimilar speed. However, if we select motion fields and assign
them to neighboring target image pieces, the result looks unnatural because their temporal video textural patterns differ.
Fig. 6-a shows a small waterfall captured from a short distance in which the motion is temporally unstable due to visible droplets and splashes. Fig. 6-b , however, shows a large
waterfall captured from a long distance, with a temporallysmooth motion. These differences in temporal patterns arewell described by Fourier analysis; Fig. 6-a has uniformly
strong coefficients at all frequencies whereas Fig. 6-b has
strong coefficients only in the DC component and weak coefficients at other frequencies. We can compute the Fouriertransform at each pixel position x as M̂(x) = F(M(x)). F is
the one-dimensional discrete Fourier transform and conversion to the power spectrum; M(x) is a sequence of motionc 2011 The Author(s)c 2011 The Eurographics Association and Blackwell Publishing Ltd.

field at x, i.e., [M1 (x), M2 (x), ..., , MN (x)] and N is the number of frames. We can thus define the smoothness of neighboring motion fields, M i and M j , assmooth(M i , M j ) = d(M̂ i 1 , M̂ j 1 ) + σ ∑ d(M̂ i f , M̂ j f ), (2)f =2d(F, G) = ∑ (F(x) − G(x))2 ,

(3)

x∈Ω

where Ω is a set of pixel positions in the overlapping area
between neighboring target image pieces, F and G, and σ is aparameter balancing the DC component, which corresponds
to the average motion field, and the AC component, which
describes the higher-frequency dynamics.

Figure 6: Fourier analysis on motion fields.
For the third criterion, we can describe the textureness ofan assigned video example piece and measure its similarity
to neighboring pieces. Fig. 7 illustrates why this criterion
is required. The two video examples of smoke have different appearances: one has strong contrast and the other has asmooth appearance. However, because they have similar motions, flowing from left to right, the first and second criteria
allow them to be neighboring pieces and will result in a visual artifact with inconsistent neighboring textures: one has ahigh contrast appearance and its neighbor has a low contrast
appearance. This is because the first criterion is based on abag of features of a SIFT descriptor, which is invariant toillumination and contrast change and therefore too weak topreserve consistency. To solve this problem, we can describe
the textureness of each frame of a video example piece using a four-level scale-space Laplacian pyramid, and sum the
texture through all frames; this process distinguishes Fig. 7a and b, because the former has stronger coefficients than thelatter through all of the sub-bands. The textureness of each
video example piece, T , has a dimension of 48 × 48 × 4.
The similarity of neighboring textureness, texture(T i , T j ) is
computed as the Euclidean distance in the overlapped area,
similar to Eq. 3.
We solve the assignment problem by first selecting multiple video candidates for each target image piece based on
their similar appearance (Eq. 1). We set the number of candidates to 100, i.e., each target image piece has 100 labels,
then determine which label should be assigned to each target image piece. We next formulate the assignment problem
with all of the criteria using MRF, which can be expressedM. Okabe et al. / Creating Fluid Animation from a Single Image using Video Database

Figure 7: Analyzing textureness using Laplacian pyramid.

Figure 8: User-specified motion field. Here, the user wants
fire to move from bottom to top along the green arrows (aand b) with a slower speed at the bottom and a higher speed
at the top (c).

as:tween neighboring pieces. Our strategy for seamless integration is to decompose a video example piece into its average image and differences. We average all the frames ofeach video example piece as A = N1 ∑N
i=1 Fi , where Fi are
the frames of the video example piece and N is the numberof frames. We compute differences as Di = Fi − A. These
differences capture high-frequency fluid features, which canbe integrated using alpha blending without introducing artifacts, and not the low-frequency appearance (Fig. 9). To
merge smoothly overlapped areas between neighboring target image pieces, we apply an image pyramid [BA83] andperform alpha blending for each sub-band. This generates
better results than standard alpha blending along boundaries
because it can introduce more continuity at low frequencies
and also preserves high-frequency fluid features.

arg min E = ∑ Vp (li ) + λ ∑ Wp,q (li , l j ),
i

p

(4)

(p,q)

where labels, li and l j , are assigned to the neighboring target image pieces, p and q. Vp is the data term and Wp,q isthe smoothness term; these are defined using a balancing parameter ρ, as follows:
Vp (li ) = I(H p , Hli ),
li

(5)
lj

li

lj

Wp,q (li , l j ) = smooth(M , M ) + ρ · texture(T , T ). (6)To ensure energy minimization, we apply the graph-cut
method with a-expansion [BVZ01], which is provided as a
package by [SZS∗ 08]. After the graph cut is run, each target
image piece has an assigned video example piece that bestsatisfies all criteria.
User-Specified Motion Field A user can manually specify a desired motion field and influence the assignment process, which is useful for controlling dynamics or modifying a failed assignment. To make the editing process as easy
as possible, we can decompose motion field into orientationand speed maps [OAIS09]. The user draws using strokes(Fig. 8-a) and generates an orientation map by interpolating the sparsely drawn strokes using a radial basis function(Fig. 8-b). The speed map is a gray-scale image that is easilyedited using paint software (Fig. 8-c). The user can specify
the orientation and the speed map, or only one of these. It
is possible to edit a speed map from scratch, but it is easier to edit a speed map obtained from the already assigned
video example pieces. The edited motion field affects the selection of the 100 candidates: before the selection, we cannarrow down candidates in advance by discarding video example pieces with motions that differ greatly from the userspecified motion field.
6. Video Synthesis
Integration of Assigned Video Example Pieces After assignment, we expect neighboring target video example image pieces will have video textures. However, because they
are likely to come from different parts of different video examples, naïve integration of these pieces causes visible artifacts, especially discontinuities along the boundaries be-

Figure 9: Integration of neighboring video example pieces.
The left side illustrates how each target image piece has an
assigned video example piece, and the right side illustrates
how corresponding differences are copied and integrated using alpha blending with an image pyramid.
Recovery of Color Appearance The integrated differences
have video textures but no color appearance. To recover
the color of the target image, we can synthesize an image
that corresponds to the average image (Fig. 2-e) and add it
onto the integrated differences. We can synthesize this approximated average image by applying image-based motion
blur [BE01] and Gaussian blur to the target image. For motion blur, we compute a motion field over the target image,
based on the motion fields of assigned video pieces. An open
parameter to simulate shutter speed controls the effect of
motion blur. The additional Gaussian blur removes the sharpedges that are introduced by image-based motion blur; kernel size is determined in proportion to the magnitude of the
motion field.
Appearance Matching Finally, we can match the appearance of the synthesized animation more precisely to thec 2011 The Author(s)c 2011 The Eurographics Association and Blackwell Publishing Ltd.

M. Okabe et al. / Creating Fluid Animation from a Single Image using Video Databasetarget image. We apply histogram-matching between eachframe of the synthesized animation and the target image inthe same manner as [OAIS09] using Heeger and Bergen’s
texture synthesis algorithm [HB95]. To recover local appearance, we divide the image space into regular grids and apply
the process to each region independently.

Figures. 1, 11-d, and 11-g. Designing a speed map fromscratch is difficult, but when synthesizing the first version ofan animation, our system outputs the resulting speed map.
This is a gray-scale image that can be easily loaded into any
paint software. By editing the map, the user can make theanimation run faster or slower.

Dynamic Boundaries of Fluid The synthesis method described above is useful for rendering fluid features witha static overall shape, e.g., a waterfall seen from a long
distance. However, when we apply this method to fire or
smoke, the synthesized animation looks unnatural. We develop a simple ad-hoc method to achieve dynamic boundaries of fluid by advecting the alpha matte according to motion fields [BNTS07]. We can define an image-warping function Wi (I) that deforms an image I according to the motion
field Mi of the synthesized animation. Given an alpha matte
α, we repeatedly synthesize the time-varying alpha matte Bi :Bi+1 = τWi (Bi ) + (1 − τ)α, where B1 is equal to α and τ controls the dynamics of Bi . We also apply an area-preserving
adjustment each time Bi is computed to ensure that the areaof Bi is always the same. Synthesis of dynamic boundaries
requires that the user perform an additional task: manual design of the background image. The background is hidden
by fluid in the original image, but it appears when the fluid
moves. In our experiment, we used Adobe Photoshop’s clone
stamp tool and found that it took less than 10 min to design
each background.

Figure 11: Thumbnails of synthesized animations: photographs(a,b,c,e,h) and paintings (d,f,g).

Figure 10: Each frame of a fire animation has dynamic
boundaries and a corresponding alpha matte.
7. Results and Discussion
We employed an independent database for water, fire, andsmoke scenes. This involved gathering 151, 96, and 89
video examples for the water, fire, and smoke databases, respectively. From these, we obtained 246,477, 226,986 and195,318 video example pieces. We synthesized fluid animations for target images, including photographs and paintings (Figs. 1 and 11). Figure 1 shows the applicationof our technique to a tour into the picture in the supplementary video [HAA97]. We designed an alpha matte foreach target image using a scribble-based image segmentation tool [LSTS04]; this process takes less than 5 min. We
specified the orientation map for all target images; designing the orientation map requires a sparse set of user-drawn
strokes, which is a simple task and takes less than 1 min.
We specified the speed map only for the examples shown inc 2011 The Author(s)c 2011 The Eurographics Association and Blackwell Publishing Ltd.

Assignment of Video Example Pieces The supplementary
video demonstrates that our assignment algorithm works
well, using Figures 11-e and 11-f as target images. Renderings from the assigned video example pieces shown in the
supplementary video already resemble the target image. In
Figure 11-e, the top region has strong flames, with smaller
flames appearing around the tree trunks. For each region,
the assignment process copies and pastes appropriate animations from various video examples. Figure 11-f has regions with waterfalls, whitewater, and calm water surfaces;
each of these areas is based on appropriate video examples.
Waterfalls or regions with flames tend to have successful
assignment even without any user input because they havestrong image edges that correlate to the flow direction. For
example, waterfalls and flames always move from top to bottom and from bottom to top, respectively. High-frequency
regions tend to have slower, more detailed motion. However,the flow direction of the water surface in Figures 11-a and11-h is ambiguous. It is also difficult to determine the flow
direction of smoke, because image edges do not correlate
to an underlying flow direction and speed. The correlation
between appearance and motion is most noticeably absent inpaintings. In these cases, orientation and speed maps become
especially important.
Timing In our current implementation, which is not optimized, we spend an average of approximately 1 hour creating the first version of the synthesized animation. We use
a workstation with an Intel(R) Xeon(R) 3.33 GHz cpu forvideo synthesis and a remote file server for the database.
We spend 30 min on the assignment process and 30 min on
video synthesis. The assignment process is slowed not by
the energy minimization process but by computation of thesmoothness term (Eq. 6). Each node of the MRF has 100
candidates, i.e., there are 100 × 100 edges between neighboring nodes. Computation of each edge requires loading
two sets of motion data from the database, and disk accessM. Okabe et al. / Creating Fluid Animation from a Single Image using Video Databasebecomes a bottleneck. A more efficient disk-access process
will be developed in future work. We plan to assemble currently fragmented files into a single large file to eliminate therandom-seek latency [KSH04].
Open parameters Assignment of video example pieces involves three open parameters: σ to balance the DC andhigher frequency components in Eq. 2; λ to balance the dataand smoothness terms in Eq. 4; and ρ to balance the smoothness of motion fields and similarity of neighboring textureness in Eq. 6. In our experiment, λ does not significantly
affect results, probably because we rely on 100 video example candidates that have already satisfied the data terms
to some extent. σ has a significant effect on the results; anoverly large value of σ ignores the DC component, whichis equivalent to the average motion field, and thus, the resulting animation tends to have chaotic flow directions. ρ isalso important, but it has a weaker effect than the smoothness of motion fields. During the experimental process, we
set λ = 0.01, σ = 100 or 1000, and ρ = σ × 100. To reducethe computational time, multiple computers are useful to findthe best parameter set. We routinely prepare a variety of parameter sets and run them in parallel.

tages are obtained by the new algorithms with the fluid video
database, not with a single video specified by the user.
The supplementary video shows a side-by-side comparison of Figure 11-a using the previous and the present
method. Previously, we could not reproduce fluid featuresof the video example, e.g., dynamic water splashes: the results appeared as if irrelevant synthesized noise flowed along
a static motion field. Note that the motion of the resulting
animation is faster than the motion of the video example;
the residuals over a slow motion field were transferred to afast motion field, which failed to reproduce the fluid featuresof the video example. To synthesize better fluid features, theuser could search for a better video example, but it was timeconsuming to search for a video with motion that was actually similar to that designed by the user. On the other hand,the present method successfully reproduced fluid features of
the assigned video example pieces.
The present method further reduces the user’s burden,
as seen in the case of Figure 1. Here, using the previous
method, the image had to be divided into waterfall and riverparts and various videos had to be assigned independently;
in contrast, the present method processes the whole image at
once.

Duration of Synthesized Animation The duration of thesynthesized animation depends on the length of the video
examples. However, several methods are available to synthesize an infinitely flowing fluid animation from a video example [SSSE00,DCWS03,BSHK04]. By applying one of thesetechniques to our database and defining the rule of infinite
repetition for each video example, our synthesized animation can also permit infinite repetition.
7.1. Comparison with Previous Methods
It is possible to synthesize fluid animation using a textureoptimization method [KEBK05]. However, this method is
not suitable to synthesize time-varying high-frequency fluidfeatures, since it is designed to preserve the original appearance of a given texture image for all frames.

Figure 12: The result of the user study. Each blue bar represents the average score of visual quality. Each error bar
represents the standard deviation.

The difference between our method and our prior workin [OAIS09] lies in how a fluid video is decomposed: we currently decompose a video into the average image and the differences between the average image and the original frames.
On one hand, our prior work decomposed the differences
further into the stationary motion field and the residuals. We
reported that such a decomposition might cause a failure oftransferring the residuals, in applying them to the fluid in the
input image. More specifically, if the original video example was not appropriately chosen by the user, our prior approach would provide animations with visible artifacts, suchas more noises and greater viscosity than the original video
example (see Section 5.2 and 6 in [OAIS09]). Alternatively
our new method simplifies the decomposition process, while
having a wider variety of the fluid animations. These advan-

We performed a user study to evaluate the visual qualityof the resulting animations. Two professional animators andfourteen computer science students participated in the study.
We presented the resulting animations of Figures 1 and 11.
Each subject ranked the visual quality of each animation
with regard to visible artifacts on a scale ranging from 1(poorest quality) to 5 (best quality). The ranking is relative,i.e., 1 and 5 must each be assigned to at least one of the resulting animations. The results are shown in Figure 12. The
highest average score with the smallest standard deviation
was assigned to Figure 11-f, i.e., almost all subjects agreed
that the water scene had a natural appearance. The other water scenes received no scores of 1 but several scores of 5,
confirming acceptable visual quality. On the other hand, fireand smoke scenes were given lower scores. We discuss the

7.2. User Study and Quality Discussion

c 2011 The Author(s)c 2011 The Eurographics Association and Blackwell Publishing Ltd.

M. Okabe et al. / Creating Fluid Animation from a Single Image using Video Databasereasons and the limitations of our method, considering Figure 11-d as a typical case in which our method failed.
Failure of Video Example Assignment Figure 11-d had
the lowest average score. The subjects commented that it
looked noisy: the smoke differed in appearance between
areas and the smoke movements were chaotic. This wasdue to failure of the assignment process for two major reasons. First, the selection of 100 candidates of video example pieces was not successful, as the target image was nonphotorealistic and had many flat areas. Second, the assigned
video example pieces had relatively chaotic movements. We
specified the orientation and speed maps, which constrained
the average motion of each patch and created the globally
upward motions of the smoke. However, these specifications
did not constrain the local motions, which were almost random. Thus, we require more sophisticated control of fluid
motion than a single manually specified motion field.
Visible Artifacts Another limitation of the present method
is that the resulting animation inherits artifacts of the video
examples in the database. In our experiment, each video example was originally compressed by the QuickTime movie
codec. Our system extracts each frame and saves it in JPEG
format for efficient image processing. Therefore, block noise
and interlacing artifacts are included in the database. These
artifacts are not only inherited but are also sometimes exaggerated when appearance transfer process increases the image contrast. For example, the subjects commented that thebottom part of Figure 11-d had a noisy appearance. Someexaggerated JPEG block noise can actually be seen in thisimage (Fig. 13). We are currently considering simple solutions to prepare large storage and apply lossless compression
to each video example.

Figure 13: (a) The assigned video example pieces had relatively low contrast. (b) Appearance transfer enhanced thecontrast to recover the appearance of the target image, but
it also exaggerated block noises that were hidden in (a).
Preservation of Original Appearance Some subjects commented that the resulting animation looked different fromthe fluid in the targedt image. It is often difficult to preserve
the appearance of the target image, especially when it has
strong edges at significantly high frequencies. For example,
Figure 11-c shows the characteristic appearance of smoke
trajectories. However, its resulting animation does not retain
this appearance, i.e., the trajectories disappeared. The same
discussion is possible for the fire animation of Figure 11-e
c 2011 The Author(s)c 2011 The Eurographics Association and Blackwell Publishing Ltd.

in that the original shape of the flame was not reproduced
exactly. Our method is limited in that it preserves the overall
shapes of fire and smoke, but our database is not sufficiently
large to preserve details, such as the smoke trajectories.
Lighting Effect The lighting effect or the movement of an
object other than fluid is outside our scope of this study. It
is often noted that animations in which only fluid parts aremobile and other parts remain completely stationary do nothave a natural appearance. Especially, the movement of theflame in Figure 11-e must influence the lighting effect on the
ground. However, our goal is not to animate the whole sceneof a target image but to animate fluid parts.

8. Conclusions and Future Work
We have developed a data-driven method for synthesizing
a fluid animation from a single image. User input includes
a target image and an alpha matte that extracts the fluid
region of interest. A few additional suggestions regarding
fluid speed or orientation enable the user to control and refine the resulting animation. The technique described here
allows the user to produce a higher quality fluid animation
with less effort than with other methods reported previously.
The present approach suggests a number of areas for future
work. In particular, we are currently pursuing three avenues
as described below:
More Efficient Use of Video Database One of the limitations of our method is the difficulty of preserving detailed appearance of a target image. This is because ourdatabase does not include a video example that fits a target appearance. In addition to increasing the size of thedatabase, we plan to investigate more flexible use of thevideo database. For example, the patch size is currently fixed
to 48 × 48. However, if we assigned a video example piecetaking its image warping into account as described previously [BSHK04], more detailed target appearance would be
reproduced.
A More Intelligent System Our current methodology involves cutting a video example into pieces and assigningthem locally. However, most failed assignments were causedby the loss of global information. If the system could understand scene structures or fluid states, our approach would
become more automated. That is, the system would be able
to estimate fluid velocity by understanding the scene structure, which would allow it to distinguish, for example, chimney smoke from the smoke caused by an explosion, eventhough they look similar. Previous studies have attempted
to develop more intelligent systems by incorporating SIFT
flow [LYT∗ 08]. This type of system can estimate the motionof a campfire in a single image by fitting of a video sample. We plan to develop a similar technique and incorporate
it into our system.

M. Okabe et al. / Creating Fluid Animation from a Single Image using Video Database

More Dynamic Fluid Animation The present method
works well in treating the high-frequency part of the fluid,
such as water splashes. However, large fluid features, suchas the overall shape of a breaking wave, cannot be synthesized. Such global features of the fluid should be createdin the future. We plan to investigate image object-retrieval
techniques as described previously [KCCH09] to efficiently
extract large semantic regions and assign it to parts of thetarget image.
References[BA83] B URT P. J., A DELSON E. H.: A multiresolution splinewith application to image mosaics. ACM Trans. Graph. 2, 4
(1983), 217–236. 6[BE01] B ROSTOW G. J., E SSA I.: Image-based motion blurfor stop motion animation. In Proc. SIGGRAPH 2001 (2001),
pp. 561–566. 6[BJEYLW01] BAR -J OSEPH Z., E L -YANIV R., L ISCHINSKI D.,
W ERMAN M.: Texture mixing and texture movie synthesis using
statistical learning. IEEE Trans. Vis. Comput. Graph. 7, 2 (2001),
120–135. 2[BNTS07] B OUSSEAU A., N EYRET F., T HOLLOT J., S ALESIN
D.: Video watercolorization using bidirectional texture advection. In Proc. SIGGRAPH 2007 (2007), p. 104. 7[BSHK04] B HAT K. S., S EITZ S. M., H ODGINS J. K., K HOSLA
P. K.: Flow-based video synthesis and editing. ACM Trans.
Graph. 23, 3 (2004), 360–363. 2, 8, 9

[KCCH09] K UO Y.-H., C HEN K.-T., C HIANG C.-H., H SU
W. H.: Query expansion for hash-based image object retrieval.
In ACM Multimedia (2009), pp. 65–74. 10[KEBK05] K WATRA V., E SSA I., B OBICK A., K WATRA N.:
Texture optimization for example-based synthesis. In Proc. SIGGRAPH 2005 (2005), pp. 795–802. 2, 8
[KSH04] K E Y., S UKTHANKAR R., H USTON L.: Efficient nearduplicate detection and sub-image retrieval. In ACM Multimedia(2004), pp. 869–876. 8[LEN09] L ALONDE J.-F., E FROS A. A., NARASIMHAN S. G.:
Webcam clip art: appearance and illuminant transfer from timelapse sequences. ACM Trans. Graph. 28, 5 (2009), 1–10. 2[LP05] L I F.-F., P ERONA P.: A bayesian hierarchical model forlearning natural scene categories. In Proc. of CVPR ’05 (2005),
pp. 524–531. 4[LSP06] L AZEBNIK S., S CHMID C., P ONCE J.: Beyond bags offeatures: Spatial pyramid matching for recognizing natural scene
categories. In Proc. of CVPR ’06 (2006), pp. 2169–2178. 5[LSTS04] L I Y., S UN J., TANG C.-K., S HUM H.-Y.: Lazy snapping. ACM Trans. Graph. 23, 3 (2004), 303–308. 7[LWW∗ 07] L IN Z., WANG L., WANG Y., K ANG S. B., FANG
T.: High resolution animated scenes from stills. IEEE Trans. Vis.
Comput. Graph. 13, 3 (2007), 562–568. 2[LYT∗ 08] L IU C., Y UEN J., T ORRALBA A., S IVIC J., F REE MAN W. T.: Sift flow: Dense correspondence across different
scenes. In ECCV 2008 (2008), pp. 28–42. 2, 9
[MPK09] M OHAMMED U., P RINCE S. J. D., K AUTZ J.: Visiolization: generating novel facial images. ACM Trans. Graph. 28(2009). 4[BVZ01] B OYKOV Y., V EKSLER O., Z ABIH R.: Fast approximate energy minimization via graph cuts. IEEE Trans. PatternAnal. Mach. Intell. 23, 11 (2001), 1222–1239. 6[MWGZ09] M A C., W EI L.-Y., G UO B., Z HOU K.: Motion fieldtexture synthesis. ACM Trans. Graph. 28, 5 (2009), 1–8. 2[CCT∗ 09] C HEN T., C HENG M.-M., TAN P., S HAMIR A., H U
S.-M.: Sketch2photo: internet image montage. In Proc. SIGGRAPH Asia 2009 (2009), pp. 1–10. 2[OAIS09] O KABE M., A NJYO K., I GARASHI T., S EIDEL H.-P.:
Animating pictures of fluid using video examples. Computer
Graphics Forum (Proc. EUROGRAPHICS) 28, 2 (2009), 677–
686. 1, 2, 6, 7, 8

[CGZ∗ 05] C HUANG Y.-Y., G OLDMAN D. B., Z HENG K. C.,C URLESS B., S ALESIN D. H., S ZELISKI R.: Animating pictures with stochastic motion textures. In Proc. SIGGRAPH 2005
(2005), pp. 853–860. 1, 2
[DCWS03] D ORETTO G., C HIUSO A., W U Y. N., S OATTO S.:
Dynamic textures. Int. J. Comput. Vision 51, 2 (2003), 91–109.
2, 8[EF01] E FROS A. A., F REEMAN W. T.: Image quilting for texture synthesis and transfer. pp. 341–346. 3[HAA97] H ORRY Y., A NJYO K., A RAI K.: Tour into the picture:using a spidery mesh interface to make animation from a single
image. In Proc. SIGGRAPH ’97 (1997), pp. 225–232. 1, 2, 7

[SSSE00] S CHÖDL A., S ZELISKI R., S ALESIN D. H., E SSA I.:
Video textures. In Proc. SIGGRAPH 2000 (2000), pp. 489–498.
2, 8[SSSFF09] S UN M., S U H., S AVARESE S., F EI -F EI L.: A multiview probabilistic model for 3d object classes. pp. 1247 –1254.
4[SZS∗ 08] S ZELISKI R., Z ABIH R., S CHARSTEIN D., V EKSLER
O., KOLMOGOROV V., AGARWALA A., TAPPEN M., ROTHER
C.: A comparative study of energy minimization methods formarkov random fields with smoothness-based priors. IEEE
Trans. Pattern Anal. Mach. Intell. 30, 6 (2008), 1068–1080. 6[HB95] H EEGER D. J., B ERGEN J. R.: Pyramid-based texture
analysis/synthesis. In SIGGRAPH1995 (1995), pp. 229–238. 7[TYS09] TAO L., Y UAN L., S UN J.: Skyfinder: attribute-based
sky image search. In Proc. SIGGRAPH 2009 (2009), pp. 1–5. 2,
3[HDK07] H ORNUNG A., D EKKERS E., KOBBELT L.: Characteranimation from 2d pictures and 3d motion data. ACM Trans.
Graph. 26, 1 (2007), 1. 1, 2

[WL00] W EI L.-Y., L EVOY M.: Fast texture synthesis using treestructured vector quantization. In Proc. SIGGRAPH 2000 (2000),
pp. 479–488. 2[HE07] H AYS J., E FROS A. A.: Scene completion using millionsof photographs. In Proc. SIGGRAPH 2007 (2007), p. 4. 2, 3

[WZ03] WANG Y., Z HU S.-C.: Modeling textured motion: Particle, wave and sketch. In ICCV2003 (2003), pp. 213–220. 2[HJO∗ 01] H ERTZMANN A., JACOBS C. E., O LIVER N., C UR LESS B., S ALESIN D. H.: Image analogies. In Proc. SIGGRAPH
2001 (2001), pp. 327–340. 3[ZKF05] Z HAO Y., K ARYPIS G., FAYYAD U.: Hierarchical clustering algorithms for document datasets. Data Min. Knowl. Discov. 10, 2 (2005), 141–168. 4[IMH05] I GARASHI T., M OSCOVICH T., H UGHES J. F.: Asrigid-as-possible shape manipulation. ACM Trans. Graph. 24,
3 (2005), 1134–1141. 1, 2

[ZPB07] Z ACH C., P OCK T., B ISCHOF H.: A duality based approach for realtime tv-l1 optical flow. In Pattern Recognition(Proc. DAGM) (Heidelberg, Germany, 2007), pp. 214–223. 4

c 2011 The Author(s)c 2011 The Eurographics Association and Blackwell Publishing Ltd.