Extracting Fluid from a Video for Efficient Post-Production∗,‡

Makoto Okabe∗
Ken Anjyo†
Rikio Onai‡†∗
The University of Electro-Communications
OLM Digital, Inc./JST CREST
JST PRESTO

Figure 1: Top row: the input video sequence of an explosion. Bottom row: the composite of the extracted explosion with another backgroundimage. Our method automatically extracted the explosion, which has considerable transparency, especially in the smoky regions of the 4th
and 5th frames.

Abstract
We propose a method to extract fluids from a video that is captured
outside a special studio. Since such a video usually has a complex
background and the fluids overlap with much transparency, it is adifficult, time-consuming task for a designer to extract them. Our
goal is to develop an efficient method to solve the problem: we estimate the background of an input video, and then compute the foreground and alpha matte at each frame. Our method estimates the
background by observing only pixels that have little motion at eachframe. Given the estimated background, we estimate an initial alpha matte based on the color difference at every pixel between eachframe and the estimated background. Since the initial alpha matteusually includes many artifacts, we employ the gradient-domain
image processing approach to refine it: our method attenuates unrequired gradients adequately, and then integrate them to recover
the refined alpha matte. The foreground, which explains about thecolor and texture pattern of the fluid itself, is also estimated in asimilar manner. We demonstrate that our method enables to extract
the fluids from a video, which were difficult to achieve using theprevious methods.
CR Categories: I.4.8 [Image Processing and Computer Vision]:
Scene Analysis—Motion—;
Keywords: video processing, alpha matte, post-production, fluid

1

Introduction

In computer graphics production, a video database of fluids is an
important resource for a designer to create scenes with fluids re∗ m.o@acm.org† anjyo@olm.co.jp‡ onai@cs.uec.ac.jp

lated with smoke, fire, or an explosion. Although physics simulations of fluids have become powerful tools for such purposes, it isoften difficult for a non-expert designer to understand and set all thephysical parameters appropriately. Hence, constructing a composite with fluid videos is still a popular and reasonable way to createa high-quality fluid scene.
Most of the fluid videos in the current database are produced in a
special studio, e.g., the explosion of a bomb, captured in front of asingle-colored screen [Smith and Blinn 1996]. Such studios enable
us to capture fluids with clean alpha mattes, which is important for
the efficient creation of a composite. Nevertheless, we are also interested in fluid videos captured outside the studio, since they usually include larger-scale, more dynamic fluids, captured in a real environment. Also, such video materials have recently become more
readily available on the web (the top row of Fig. 1 shows an example). However, since such fluid videos are usually captured in
front of a complex background, rather than a single-colored screen,
it is time-consuming to manually separate the fluid from the background. Our goal is to develop a system to support this process,allowing the designer to efficiently use the extracted fluids for postproduction, as shown in the bottom row of Fig. 1.
Extraction of transparent fluids from a video is still a challenging
problem. As far as we know, none of the previous methods (including those of the survey [Wang and Cohen 2007]) can solve
the problem perfectly. One approach solves the problem efficiently
by introducing a strong assumption to the fluid. For example, theBayesian Matting method assumes that the fluid has only a single
color [Chuang et al. 2002], which works well for faint smoke. However, when it is applied to a colorful explosion, the extraction result
has only a flat color (Fig. 2-d). Ghanem et al. assume a texture
rather than a single color [Ghanem and Ahuja 2008]. However,since the texture cannot change temporally, it is also inadequate
for representing an explosion. There is a method for dehazing avideo [Zhang et al. 2011], but it is likewise limited to fog of a single color. A related work involves video matting combined withtransparency caused by a motion blur [Lin et al. 2011], but thistechnique has not been applied to the transparency of fluids.
Another approach is to apply the image matting method to eachframe of the video [Sun et al. 2004; Levin et al. 2008; Gastal andOliveira 2010]. In this approach, we create a trimap or scribbles
for each frame, which partitions the image into three regions: ‘defi-

Figure 2: We compare our method with previous methods. The trimap (c) is computed based on the color difference between the original (a)and the background (b) at each pixel. We use the trimap to compute the foreground color for Bayesian Matting (d). For Spectral Matting (e)and Shared Matting (f), we use the trimap as direct input to the algorithm.

Figure 3: An overview of our method.

nitely foreground,’ ‘definitely background,’ and ‘unknown.’ This
type of procedure is effective for estimating the alpha matte on
small transparent regions around an opaque object. However, since
a fluid has transparency over a large area, it is difficult to apply sucha method directly to fluid videos. Fig. 2-e shows the result of Spectral Matting [Levin et al. 2008]. This procedure extracts not only
the fluids, but also the background image over a large area. Its computational cost is an additional disadvantage; it took 214 seconds toprocess a single frame. Fig. 2-f shows the result of Shared Matting [Gastal and Oliveira 2010]. This method processes each video
frame in real time, successfully removes the background, and extracts the fire regions. However, there are many artifacts, appearing
as blobs in the faint smoke regions.
To address these issues, we develop a novel method for extracting fluids from videos with reasonable computational expense. In
the example of Fig. 1, our method automatically extracted the explosion together with the faint smoke (Fig. 2-g). Fig. 3 shows an
overview of our approach. Given a fluid video captured with a fixed,
monocular camera, we first estimate its background, and then compute the foreground and alpha matte for each frame. We estimate
the background by averaging throughout the entire sequence only
pixels that have little motion (Fig. 3-b). Most of the smoke disappears, lingering at some pixels where it is particularly dense; i.e., it
always has significant motion.
More specifically, given the estimated background, our key idea isthat the user creates an initial, rough alpha matte by manipulating
several parameters, and then our system refines it. Because it isdifficult for a computer to automatically estimate the alpha matte,
the user provides a rough alpha matte. On the other hand, because it
is difficult for the user to remove the artifacts from the rough alphamatte, our system refines it.
We compute an initial alpha matte by calculating color differences(Fig. 3-c). It roughly represents the area of the smoke, but the texture patterns of blocks of background can be seen in it. To remove
these texture patterns, we employ gradient-domain image processing to refine the alpha matte (Fig. 3-d) and estimate the foreground(Fig. 3-e), which can be used to create a composite (Fig. 3-f). We
compute horizontal and vertical image gradients for each frame, attenuate the gradients independently of the smoke, and then integratethem to obtain the refined alpha matte. The technique of attenuation is our major technical contribution. The computational time isseveral seconds per frame for 640 × 360 resolution.

2

Our method

We model the ith frame of a video sequence as a standard alphablending:
Ii (~x) = Fi (~x)αi (~
x) + B(~x)(1 − αi (~
x)),

(1)

where ~
x denotes a pixel position, and Ii , Fi , B and αi denote theoriginal frame, its foreground, background, and alpha matte. Note
that, because we assume that the input video is captured with amonocular camera, B is single through the entire sequence. In allof the following discussion, the range of a pixel value is from 0 to 1
(not 255). Given an input video, our algorithm begins by estimating
the background.

2.1

Background Estimation

We estimate the background by observing only pixels that have insignificant motion. More sophisticated approaches to backgroundestimation also exist (e.g., [Apostoloff and Fitzgibbon 2004]), but
our approach is also useful for videos like the ones used in this paper and our supplementary video.
We compute the optical flow using the OFLib package [Zach et al.
2007], which is implemented via the graphics processing unit
(GPU). It is computationally efficient, requiring about 0.1 second tocompute the optical flow between temporally neighboring frames at
640×480 resolution. Then we compute the color of the backgroundB(x) as the following weighted average:PIi (~x)Wi (~
x)
iP
,
(2)
B(~
x) =
W
(~
x
)
i
i
Wi (~
x) = e−β|Vi (~x)| ,(3)

where Vi is the optical flow computed between the ith and i + 1th
frames, Vi (~x) is the velocity at the pixel position ~x, Wi is theweight, and β is set equal to 40 throughout our experiments. Fig. 4
shows the background estimation process for a smoke video. We
show the original frames Ii in the top row, the magnitude of thecorresponding optical flow Vi in the middle row, and the corresponding weights Wi in the bottom row. Intuitively speaking, we2.3

Refinement of the Alpha Matte

The initial alpha matte constructed by the user gives the overall
region of the fluid. However, artifacts from the texture patterns of
the background remain, and we want to remove them.
Our idea for refining the initial alpha matte is based on gradientdomain image processing. Fig. 6 shows an overview of the procedure. We begin by computing the horizontal and vertical image gradients (Figs. 6-b and c) of the initial alpha matte (Fig. 6-a).
Then we process these so that only gradients related to the smoke
are preserved, and the other gradients are attenuated (Figs. 6-d ande). Finally, we integrate the processed image gradients to create
the refined alpha matte (Fig. 6-f). To obtain the image gradients,

Figure 4: Background estimation.

assume that at a pixel with low magnitude |Vi (x)|, Wi (x) assures
us a high probability of obtaining the correct background color. The
rightmost part of Fig. 4 shows the estimated background B and thePaverage of Wi ; i.e., W avg = M
i Wi /M , where M is the numberof frames. We can see some smoke remaining in B, and W avg is
dark around this area; i.e., it has low values.

2.2

Estimation of the Initial Alpha Matte

Given the estimated background B, we estimate the initial alphaby calculating the color difference between each frame
matte Ainiti
Ii and B at every pixel. We assume that a three-dimensional (3D)
vector of red, green, and blue channels is assigned to each pixel ofIi and B, and a single scalar value for the luminance channel isassigned to each pixel of Ainit. We compute the initial alpha matteiof Ii as follows:
Ainit
(~
x) =
i

1

Figure 6: An overview of the refinement of the alpha matte.
we simply compute the horizontal and vertical differences between
neighboring pixels. We formulate the integration as a constrained
least-squares optimization:
X X
X
argmin E =(ap − aq − gpq )2 +λp (ap − ainit)2 ,p
a1 ,...,aN,

pq∈ν(p)

p

(4)

(5)

which is a sigmoid function of color difference. g and t control
the steepness and the offset of the function. Fig. 5-a shows the
sigmoid function obtained by setting g and t equal to 101.4 and
0.18, respectively. Before calculating the color difference, we apply
a Gaussian blur to remove the noise from Ii and B. The size of theGaussian kernel is set at 5.0 throughout our experiments. Figs. 5b, c, and d show the resulting initial alpha matte. Note that there
are many artifacts; e.g., the texture patterns of the blocks of the
background wall can be seen in the figure.

where {a1 , ..., aN } are all of the pixel values of the refined alphamatte, p denotes a pixel position, ν(p) denotes the set of pixel positions of the horizontal and vertical neighbors of p, gpq is the processed gradient between p and q, corresponding to Figs. 6-d ande, λp is the weight of the constraint at p, and ainitis the pixel
pvalue of the initial alpha matte Ainitat p. To minimize the eni
ergy function, we use the Poisson solver with the effective preconditioner [Szeliski 2006], which is implemented via the GPU.
We set the maximum number of iterations of the conjugate gradientmethod at 80. To obtain the refined alpha matte, at every pixel wemust adequately specify 1) the gradient gpq , and 2) the weight ofthe constraint λp . We explain how to specify each of these.

1 + e−g(|Ii (~x)−B(~x)|−t)Figure 5: The sigmoid function and the initial alpha matte.
We design Eq. 4 as a heuristic with two requirements: user controllability and nonlinear smoothness. Most of the existing imagematting methods use a trimap or scribbles, designed via a user
heuristic, as input [Sun et al. 2004; Levin et al. 2008; Gastal andOliveira 2010]. Eq. 4 reflects this; it allows the user to design arough hint for an initial alpha matte, similar to the way in which a
trimap or scribbles are designed by user intuition, independently of
any equation.

To compute gpq as shown in Figs. 6-d and e, we preserve the gradients found only in Ainit, and attenuate the gradients shared by theiframe Ainitand the estimated background B. Our idea is similarito the “contrast attenuation” of Background Cut [Sun et al. 2006].
Background Cut attenuates a gradient in Ainit
when the magnitude
iof the gradient at the same pixel position in B is large. It successfully extracts only gradients found uniquely in Ainit. However,iattenuation based solely on the magnitude of the gradient is not
sufficient to satisfy our goal. We process gradients more carefully;
we attenuate the gradient at a pixel position around which a similar
texture pattern is shared by Ainit
and B.
i
To describe the features of a texture pattern, we consider a patch of
5 × 5 resolution around a pixel position (Fig. 7-a). In this patch, we
compute the oriented gradient at every pixel, and create a histogramof these oriented gradients. We include eight orientations, but thedimension of the histogram is only four, since an orientation andits reverse share a single bin, as shown in Fig. 7-a. Each bin contains the total of the magnitudes of all gradients whose orientations

correspond to the orientation the bin represents. The histogram isfinally normalized so that the total of all bins is equal to one.
At every pixel position, we compute the Euclidean distance between the histograms of Ainit
and B. Fig. 7-d shows the result. Theiintensity at each pixel position describes the difference between the
texture patterns of Ainit(Fig. 7-b) and B (Fig. 7-c). A gradient
i
init
gpq
in Ainit
is attenuated as follows:
i
2

2

init
gpq = gpq
(1 − e−dp /K ),(6)manner as Eq. 6, and integrate them to estimate the foreground. In
Eq. 5, we let ai be the intensity of the foreground, and ainitbe thepintensity of Ii . For the attenuation of the image gradient, we use
init
Eq. 6, letting gpq
be the gradient in Ii , and dp be the Euclidean
distance between the texture patterns Ii and B. As for constraints,
we specify a pixel as a constraint only when the pixel value of theinitial alpha matte is extremely large. When ainitis greater thanp0.8, we set λp equal to 0.01, but this time we ignore pixels whose
values are extremely small. Fig. 9 shows the resulting foreground.

where dp is the Euclidean distance (Fig. 7-d) at p, and K is set
equal to 0.01 throughout our experiments. The attenuated image
gradients gpq are shown in Figs. 6-d and e. The texture patternsof the blocks of the background wall are removed from the image
gradients, and also from the resulting refined alpha matte.

Figure 7: (a) We compute oriented gradients in a 5 × 5 patch andcreate a four-dimensional histogram of these gradients to describe
the features of a texture pattern. (b) The initial alpha matte Ainit.
i(c) The background B. (d) Feature similarity as the Euclidean distance between Ainit
and B. Note that the yellow boundary is overi
laid to show the corresponding region. The area of darker values
contains strong similarities, where the gradients will be attenuated.

Figure 9: (a) The original video frame, (b) the refined alpha matte,(c) the estimated foreground, and (d) the composite with a green
background.

3
Fig. 8 compares our attenuation method to that of BackgroundCut [Sun et al. 2006]. Background Cut attenuates a gradient simply when the corresponding background gradient is large. Because
the brick wall is highly textured (i.e., has many large gradients), it
attenuates the gradients in the smoke region. As a result, the extracted smoke becomes smooth and flat (Fig. 8-a). To solve thisproblem, we introduce texture pattern similarity. The gradients areattenuated only when the texture patterns match, which preserves
the original smoke texture more faithfully (Fig. 8-b).

Figure 8: Comparison of the attenuation methods.
We specify a pixel as a constraint when the pixel value of the initialalpha matte is extremely large or small. When ainitis greater thanp
0.8 or less than 0.02, we set λp equal to 0.01. Otherwise, we set λpequal to zero.

2.4

Foreground Estimation

We estimate the foreground by almost the same method used for therefinement of the alpha matte. We process the red, green, and blue
channels independently, solving the constrained least-squares optimization separately for each color channel. We compute the horizontal and vertical image gradients of Ii , attenuate them in the same

Results and Discussion

We apply our method to various fluid videos containing smokeand explosions. The videos are selected from the DynTex
database [Péteri et al. ] and the internet. Each video shows a fluid
flowing in front of a static background. The fluids have both sparseand dense parts, and the backgrounds include complex texture patterns. Information on each result is given in Table 1. All results, andcomparisons with the extractions performed by the difference mattefunction of Adobe After Effects, are also given for our supplementary video. All computations are carried out using a notebook
PC with an Intel(R) Core(TM) i7 1.87 GHz CPU and an NVIDIA
Quadro FX 3800M GPU.
Employing our system, the user begins the process of fluid extraction by loading a video into the system. The system spends several
minutes estimating the background, most of it on the computationof the optical flow. Then the user creates the initial alpha matte bymanipulating the slider bars of t and g. This process is fully interactive; i.e., the user can check the alpha matte in real time. Whena satisfactory initial alpha matte is obtained, the user pushes the
button and the system proceeds to apply gradient-domain imageprocessing to every frame. Finally, when the system finishes processing the last frame, a video sequence of the extracted fluid isobtained. We found that t = 0.18 and g = 101.4 are good default
settings for the parameters.
Figs. 9 and 10 show some successful results. In Fig. 9, because thesmoke is relatively sparse, the background is successfully estimated(Fig. 3-b). A small amount of smoke remains, but it does not createa major problem for the smoke extraction. In Fig. 10, we compare
the results obtained from the difference matte of Adobe After Effects and our method. The red arrows indicate the area where ourmethod successfully removes the texture pattern derived from thebackground: the edges and tile patterns of the background.

Figure 10: The extracted fluids from Explosion2 and Explosion3.
Fluid Video
Smoke1
Explosion1
Explosion2
Explosion3

Frames
883
209
947
609

Resolution
352×288
480×360
640×360
640×360

Time
2.92
6.31
5.0
6.0

Table 1: For each fluid video, we list the number of the frames, theresolution, and the average time (seconds) per frame required tocompute the extraction.

3.1

Limitations and Future Work

Because our method attenuates image gradients, the results tend
to be blurry, because of both the blurred foreground and the alphamatte. Fig. 1 illustrates the phenomenon; the contrast and texture
patterns of the exploding fire are reduced in the result. This suggests that we must find a more delicate technique for selecting thegradients to be attenuated. This will be the subject of our futurework.
Our method relies on the quality of the initial alpha matte, which isbased on color differences. As a result, when a pixel in a fluid has
a color very similar to that of the background, it is often estimatedas a background pixel. Fig. 11 shows an example of smoke flowing
slowly across the sky. The background (Fig. 11-b) is not successfully estimated because of the extremely dense smoke. In addition,
the similarity in color between the smoke and the background ishigh, creating an undesirable hole in the initial alpha matte (Fig. 11c). Our method attempts to remove the hole by attenuating the gradients around the hole. The boundary of the hole becomes blurry,
but remains in the refined alpha matte (Fig. 11-d). To solve thisproblem, we plan to estimate the initial alpha matte by relying not
only on color differences, but also on other information, such as
fluid motion.

References
A POSTOLOFF , N. E., AND F ITZGIBBON , A. W. 2004. Bayesianvideo matting using learnt image priors. In Proc. of CVPR ’04.
C HUANG , Y.-Y., AGARWALA , A., C URLESS , B., S ALESIN ,D. H., AND S ZELISKI , R. 2002. Video matting of complex
scenes. 243–248.
G ASTAL , E. S. L., AND O LIVEIRA , M. M. 2010. Shared sampling
for real-time alpha matting. Comput. Graph. Forum 29, 2, 575–
584.
G HANEM , B., AND A HUJA , N. 2008. Extracting a fluid dynamic
texture and the background from video. In Proc. of CVPR ’08, 1–8.
L EVIN , A., R AV-ACHA , A., AND L ISCHINSKI , D. 2008. Spectral
matting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 10, 1699–
1712.
L IN , H. T., TAI , Y.-W., AND B ROWN , M. S. 2011. Motion regularization for matting motion blurred objects. IEEE Trans. Pattern Anal. Mach. Intell. 33, 11, 2329–2336.
P ÉTERI , R., FAZEKAS , S., AND H UISKES , M. J. DynTex : a Comprehensive Database of Dynamic Textures. Pattern Recognition
Letters. http://projects.cwi.nl/dyntex/.
S MITH , A. R., AND B LINN , J. F. 1996. Blue screen matting. In
Proc. SIGGRAPH ’96, 259–268.
S UN , J., J IA , J., TANG , C.-K., AND S HUM , H.-Y. 2004. Poisson
matting. ACM Trans. Gr. 23, 3, 315–321.
S UN , J., Z HANG , W., TANG , X., AND S HUM , H. Y. 2006. Background cut. In Europ. Conf. on Computer Vision, 628–641.
S ZELISKI , R. 2006. Locally adapted hierarchical basis preconditioning. In SIGGRAPH2006, 1135–1143.
WANG , J., AND C OHEN , M. F. 2007. Image and video matting: asurvey. Found. Trends. Comput. Graph. Vis. 3, 2, 97–175.
Z ACH , C., P OCK , T., AND B ISCHOF, H. 2007. A duality based
approach for realtime tv-l1 optical flow. In Pattern Recognition(Proc. DAGM), 214–223.

Figure 11: Failure caused by small color differences.

Z HANG , J., L I , L., Z HANG , Y., YANG , G., C AO , X., AND S UN ,J. 2011. Video dehazing with spatial and temporal coherence.
Vis. Comput. 27, 749–757.