Eurographics Symposium on Rendering 2012
Fredo Durand and Diego Gutierrez(Guest Editors)

Volume 31 (2012), Number 4

Matting and Compositing for Fresnel Reflection
on Wavy Surfaces
Yuki Endo1

Yoshihiro Kanamori1
1 University of Tsukuba

Yukio Fukui1

Jun Mitani1,22 JST ERATO

Abstract
This paper introduces a framework that can extract an alpha matte from a single image with Fresnel reflection,and that can composite other objects with the image such that plausible reflections are included. Our methodhandles reflections in a plane with small undulations, for example, a water surface with waves or a glossy tabletop.
During the matting stage, our method first estimates the transmission color, which is assumed to be uniform, andthen calculates a reflection image and alpha matte based on user markups. However, accurate extraction of thematte becomes challenging when a plane has small undulations because these create perturbations in the matte.
We therefore propose a filter that can refine the matte effectively. In the compositing stage, the reflection of acomposited object is synthesized by ray tracing in real time. We demonstrate the effectiveness of our method
through comparisons with ground-truth data and results using natural images as inputs.
Categories and Subject Descriptors (according to ACM CCS): I.3.8 [Computer Graphics]: Applications—

1. Introduction
Scenes reflected on the surface of the sea or a lake are popular photographic subjects, as are seen by a quick image
search on the Internet. Water surfaces can behave like a flat
mirror, or can yield interesting warping in reflection images
if the surface is wavy. Such effects are caused by Fresnel reflection, which consists of a linear combination of reflected
and transmitted light, and can be observed in our life, forexample, on a tabletop or glossy floor.
Editing photographs with Fresnel reflection is quite interesting, especially when an image includes reflections on
wavy surfaces. Indeed, pleasing images and animations withreflections on wavy surfaces can be created from a single photograph, although this involves considerable manual labor [CGZ∗ 05, PHCS06]. Matting plays the key rolein semi-automating this creation process; the reflected image is first separated into a reflection component, transmission component, and alpha matte, and then new objects
can be composited by using this information. Matting is
ill-posed, however, and is therefore a challenging problem.
Applying previous matting methods for optical phenomena(e.g., shadow [CGC∗ 03, WTBS07] and haze [Tan08, Fat08,
HST09, KN09]) to images with Fresnel reflection is difficult due to the differences between the targets and the incompatibility of the formulations. Additionally, the matting
submitted to Eurographics Symposium on Rendering (2012)

method for refraction in glass [YTBK11] is inappropriate
for the complicated reflections observed on water surfaces.
In this paper, we propose a matting method that deals with
Fresnel reflection in a single image based on user markups.
Targeting reflections on surfaces of deep water, glossy tabletops, floors, and the like, we introduce the following three assumptions: (1) the transmission color is approximately uniform, (2) a pair composed of a source object and its reflection can be found in the input image, and (3) the reflection
surface is predominantly planar but possibly includes waves.
We first estimate the transmission color based on the first andsecond assumptions by using color transfer [RAGS01]. We
then roughly estimate the reflection component and alphamatte, as well as the camera parameters, by assuming thatthe alpha matte is smooth. However, the alpha matte willcontain high-frequency regions in the case of a wavy reflection surface. We thus propose a filter that refines the alphamatte, and use ground-truth data to validate the filter. We
also provide a compositing system that uses the calculated
information to composite new objects, including generated
plausible reflections, with the input image in real time.
Figure 1 presents an overview of our system. Our matting algorithm takes a single image with a reflection surface
as an input, as well as a user-specified region of the reflection surface and a pair of scribbles for color transfer. The al-2Endo et al. / Matting and Compositing for Fresnel Reflection on Wavy Surfaces
Scribble SΠ

Pairwise scribbles
for color transfer
Mask forreflection region Ω
Input image I

Scribble SΩ

InputsEstimate transmission component T(Section 3.2)

New object to be composited

User markups

Estimate smooth α matte(Section 3.3)

Matting (Section 3)

Estimate reflection component R(Section 3.3)

Composite result with plausible reflection

Compositing (Section 4)

Refine reflection component R(Section 3.4)

Calculate wavy α matte(Section 3.4)

Figure 1: Overview of our system. To solve matting problems, a user specifies the reflection surface and pairwise scribbles.
After several matting steps, a composite result with plausible reflection can be obtained.

gorithm then computes the reflection component, transmission component, and alpha matte, and modifies the reflection
component and alpha matte such that wave effects can be incorporated (Section 3). From this matting information, we
subsequently render reflection images of newly composited
objects by using ray tracing (Section 4).
Note that, to the authors’ knowledge, this is the first attempt to extract a complicated reflection and alpha matte forFresnel reflection from a single image. We demonstrate theeffectiveness of our method through various examples, anddiscuss its limitations in terms of the assumptions used (Sections 5 and 6).
2. Related Work
Current methods for natural image matting are outlined in asurvey [WC08] and an evaluation website [RRW∗ 09]. Thesemethods extract foreground objects and an alpha matte froma single image by the user specifying definitely foregroundand background regions using a trimap or scribbles. Unfortunately, such definite separation is not available for reflection images, and thus we require an alternative method.
Several matting methods for optical phenomena have recently been proposed. Wu et al. [WTBS07] extracted shadows from natural images by assuming that colors in a shadowed region are products of the shadow color and groundcolors based on retinex theory. Similarly, haze removal techniques suppose that an input image is a linear blending of thehaze color and background colors. Tan’s method [Tan08] enhances image contrast, and does not decompose the image.
Fattal’s [Fat08] dehazing method estimates a uniform hazecolor by assuming that the transmission and surface shading
are uncorrelated locally. In our case, this statistical approach
will fail in reflection regions where the transmission color isdominant or when sky without shading is primarily reflected.
Methods that use the dark channel prior [HST09, KN09] arebased on the hypothesis that at least one color channel tends
to be almost zero in natural images. This hypothesis is invalid in regions where a blight sky is reflected as in our case.
Most importantly, applying these methods to the situation
examined here is nontrivial due to the differences between
the targets.
A basic technique for reproducing reflection/refraction is
to use ray tracing with three-dimensional geometries. However, several methods do not use geometric information. Environment matting [ZWCS99,CZH∗ 00,WFZ02,PD03] measures the light transport of a reflective or refractive object in
the real world by means of specialized devices or multiple
photographs. Khan et al. [KRFB06] proposed a method thatcan convert an object’s material in a photograph such that theobject appears to be composed of metal or glass. The input
object is assumed to be opaque, and depths in the image are
estimated from luminance intensities. In contrast, the target
object here is not opaque and its luminance intensity does
not provide depth information.
Yeung et al. proposed a matting method for separating
glossy reflections and refractions on a glass surface in animage [YTBK11]. In their method, the reflection component
is limited to white highlights that are extracted as definitely
foreground in a trimap, and an alpha matte and warping
function of refraction are calculated based on user-specified
strokes. We consider their method to be the most relevant
to our matting method. However, the reflection component
in our case consists of a various colors and definitely foreground regions are not available. Additionally, manual specification of small-scale warping caused by a wavy reflection
surface is impractical.
submitted to Eurographics Symposium on Rendering (2012)

Endo et al. / Matting and Compositing for Fresnel Reflection on Wavy Surfaces

Separating reflection and refraction on a glass surface has
been studied in computer vision. While some of these methods take a single image as an input [LZW04, LW07] andothers multiple images [BBZZ05, GSZ08], here we give an
overview of single image methods. The key concept in thesemethods is that luminance edges belong to either reflection
or refraction components. Nevertheless, an automated approach [LZW04] is unable to achieve adequate results inmany cases, and manually specifying edges [LW07] is labor
intensive. We do not deal with complicated refractions in our
matting method, but target a uniform transmission color as
observed in deep sea water or a nontextured tabletop.
3. Reflection Matting
We start by defining our reflection matting problem and ourassumptions for solving the problem. We then describe theproposed matting algorithm for separating each componentof an input image.
3.1. Reflection Model and Assumptions
We target reflections caused by the Fresnel effect, which occurs at an interface between objects with different refractive
indices. The light intensity observed at the interface between
the objects is a linear blending of the reflected and transmitted light. For an image, we haveI(x) = α(x)R(x) + (1 − α(x))T(x),(1)

where I, R, and T are the three-channel (RGB) input image,reflection component, and transmission component, respectively. α ∈ [0, 1] is the Fresnel coefficient and x = (x, y)T is
a pixel coordinate.
The Fresnel coefficient is dependent on the incident angleof the viewing ray. At an air/water interface, for example,
α monotonically increases when the incident angle is sufficiently large. Thus, the Fresnel coefficient is small at thenear side of the camera and large at the far side, as shown in
the alpha mattes in Fig. 1 (each pixel intensity encodes theFresnel coefficient). Correspondingly, the transmitted color
is dominant at the near side and the reflected color is dominant at the far side.
Solving Eq. (1) for a single input image I is an ill-posed
problem because each pixel has only three known values
but seven unknowns. Although the current situation may
seem similar to natural image matting [WC08], which decomposes an image into foreground and background components and a matte, or to recent refraction matting techniques [YTBK11], the targets and formulations are different, as described in Section 2. We instead focus on typicalcases observed on the surfaces of deep waterbodies, glossy
tabletops, or floors, which can be seen in many photographson the Internet. Specifically, we make the following threeassumptions.
submitted to Eurographics Symposium on Rendering (2012)3

1. The transmission component T can be approximated as a
uniform color.
2. At least one real object and its corresponding reflection
can be found in an input image.
3. The reflection surface is predominantly planar but may
include waves.
Throughout this paper, we use the term real object to describe an object that is the reflection source above the surface, in contrast to its corresponding reflection. Our matting
algorithm takes a single photograph with a reflection surface
as an input, as well as a user-specified mask for the reflection
region and a pair of scribbles for color transfer. We denote
the real-object and reflection regions in an input image by Πand Ω, respectively. Our algorithm then calculates T, R, andα for Ω. Moreover, Π is located above the upper edge of Ω’sboundary ∂Ω.
On the basis of these assumptions, the matting algorithm
proceeds in several steps (see Fig. 1). We estimate T first(Section 3.2) because this is assumed to be uniform (i.e.,
constant) over the entire region and is thus easier to determine than the other components. We next calculate an alphamatte according to the Fresnel equation by using incident
angles of the viewing rays based on estimated camera parameters (Section 3.3). We then compute R using the data
generated in the previous steps (Section 3.3). Note that thereflection surface can be wavy, however, and so we refine R
and the alpha matte using a proposed filter (Section 3.4). In
the following subsections, we describe the details of thesesteps.
3.2. Estimating Transmission Component T
If the transmission component is uniform, then T in Eq. (1)is a constant vector:
I(x) = α(x)R(x) + (1 − α(x))T.

(2)

We thus consider the luminance gradients of Eq. (2):∇I(x) = (R − T )∇α(x) + α(x)∇R(x),(3)

where I, R, and T are the luminance of I, R, and T, respec∂ ∂ T
tively, and ∇ = ( ∂x, ∂y ) is the gradient operator. If the reflection surface is sufficiently flat, then the alpha matte issmooth because the differences among the incident angles
are small. Thus, ∇α is negligible. We therefore approximate
Eq. (3) as∇I(x) ≈ α(x)∇R(x).

(4)

However, this approximation might be invalid for wavy surfaces since incident angles can vary greatly in this case. To
avoid wavy surface effects, we downsample the input image
using a Gaussian filter. For example, we scale an input imageof 600 × 400 pixels down to 1/4 of its original size.
If we know α and R, we can calculate T from Eq. (2) byusing only those pixels in a partial region of Ω because T is

4Endo et al. / Matting and Compositing for Fresnel Reflection on Wavy Surfaces

constant. Conversely, α can be computed by using I and R
through Eq. (4). We therefore seek R in Ω locally. On the basis of our second assumption, R in Ω can ideally be obtainedby determining the corresponding region in Π and using thereal-object colors. However, accurate alignment of the pairis difficult because the colors and shapes in Ω are changed
compared with those in Π as a result of the Fresnel effect andwaves, respectively. An exact match of the pair thus cannot
be found, but we can convert the color distribution in Ω to be
similar to that in Π by using color transfer [RAGS01]. We
then use the color-transferred I ∈ Ω as R.
Pairwise regions for color transfer are specified throughuser-provided scribbles SΠ and SΩ . As shown in the “Usermarkups” procedure of the “Inputs” step in Fig. 1, the user
specifies a region in Π with a single green scribble SΠ , andthe corresponding region in Ω with a single red scribble SΩ .
The colors in SΠ are then transferred to those in SΩ in orderto estimate R in Ω. Note that we do not require R for every
pixel in Ω during this step, we simply collect samples toestimate T.
This color transfer works well. α can be considered constant in SΩ because the region is vertically short, and so I isequal to scaled R with a constant offset (Eq. (2)). Explicitly,
R in SΩ can be obtained by scaling the variance of I in SΠ
and offsetting the mean. The mean and variance of I in SΩthus conform to those in SΠ through color transfer.
Finally, we calculate T using the information obtained
above. The R pixel samples in SΩ are used to determine α
from Eq. (4) by projecting ∇I(x) onto ∇R(x):α̃(x) =

∇I(x) · ∇R(x)k∇R(x)k2.

(5)Here, α̃ is an intermediate value to estimate T. We omit samples if k∇R(x)k2 < 0.01 during this process to reduce errors
caused by division. We therefore calculate T for each sample through Eq. (2) using α̃, and average the values to reduce
errors.
3.3. Estimating α Matte and Reflection Component R
We obtained α values in only a partial region SΩ in the previous subsection. Next, we estimate the entire α matte. The
goal in this step is to generate a smooth alpha matte that captures low-frequency variations, and then calculate the corresponding R. We again use the downsampled I as input
here to avoid wave effects. To reliably estimate a smoothalpha matte, we exploit the analytic formulation of the Fresnel coefficient. Specifically, we use the Schlick’s approximation [Sch94] for the Fresnel coefficient:α(θ) = α0 + (1 − α0 )(1 − cos(θ))5 ,(6)

and for a planar reflection surface, the incident angle θ of aviewing ray is given by¶µφ2ytan( ) ,θ(y) = γ(y) + θo , γ(y) = arctan(7)h2

CameraCamera verticalview angle ϕImage plane
Vertical pixel
coordinates yγ

Image height h Reference
incident angle θoIncident
angle θReflection surface
Figure 2: Relation between reflection surface and camera
parameters.

where h is the image height, y is the vertical pixel coordinate, φ is the vertical view angle of the camera, and θo is thereference incident angle (Fig. 2) for the camera’s gazing direction. α0 is a constant determined by the refraction index
n of the material: α0 = n−1
n+1 . In our experiments, we simply
use a fixed value for n according to the material, for instance,n = 1.33 for water. Approximate specification of n is applicable here because a small variation of n has little effect on
the value output by Eq. (6).
To determine θ, we must estimate camera parameters θoand φ. We therefore solve the following minimization problem by using the Levenberg-Marquardt method:
¯
³
´¯2
¯
¯
argmin∑ ¯α(p)− α0 +(1−α0 )(1−cos(θ(y p )))5 ¯ (,8)θo ,φ p∈P

where P = {p = (x p , y p ) | p ∈ Ω} is a set of training sample
positions. Although Eq. (8) is nonlinear, it converges to an
appropriate minimum when the initial values are θo = 70◦
and φ = 30◦ . Note that φ can be also specified from the exchangeable image file format (EXIF) data of the input photograph, and n can be estimated during optimization of Eq. (8).
We collect samples {α(p), y p } throughout Ω, and not in apartial region as in the previous subsection, to avoid bias. To
compute α(p) in each training sample, we search for an appropriate reflection-color sample in Π. Let R(q) (q ∈ Π) be
the color of a reflection-color sample. We seek the optimalR(q) based on the following metric ε:ε(R(q), T) = kI(p) − (α̂(p) R(q) + (1 − α̂(p))T)k, (9)(I(p) − T) · (R(q) − T),α̂(p) =
(10)
kR(q) − Tk2
where the value of α̂(p) that minimizes ε is set as α(p).
Eqs. (9) and (10) are derived from Eq. (2). ε measures thedifference between I(p) and the linear blending of R(q) and
T. Intuitively, ε represents the distance between I(p) and aline segment connecting R(q) and T in RGB color space. A
smaller value of ε hence signifies that the color of the sample is better matched. This metric is motivated by robust matting [WC07], where background and foreground samples are
submitted to Eurographics Symposium on Rendering (2012)

Endo et al. / Matting and Compositing for Fresnel Reflection on Wavy Surfaces

5

G
R(x)

search window Wεreal-object
region Π
d

I(x)

R(y)

boundary ∂Ωdreflection
region Ωα(x)β(x, y)

R
I(y)

α(y)

sample position p

B

Figure 3: To calculate α(p) in each training sample, we
search for the pixel with minimal ε within Wε (blue square;enlarged for illustrative purposes), and use it as sample
R(q). The center of Wε is at a vertical d from ∂Ω.

collected based on a similar metric of natural image matting.
Note that we do not use the map α̂ as the alpha matte because
it may contain substantial noise and errors.
To efficiently search for appropriate samples R(q) in Π,
we limit the search space such that the corresponding real
object is likely to be found. Let d be the vertical distance
between the pixel under examination and the boundary ∂Ω,
as shown in Fig. 3. We choose a sample with minimal ε ina search window Wε (we use a 7 × 7 window in our experiments) centered at the position vertically symmetric to p:
q = argmin ε(R(q̃), T).

(11)

q̃∈Wε

If d is sufficiently large that the search window protrudes
outside of the image, we simply ignore the sample at p. To
precisely estimate the camera parameters, we collect 20% of
the pixels in Ω that minimize ε.
In summary, we calculate α and R as follows. We first estimate camera parameters through Eq. (8) and training samples based on Eqs. (9) and (10). We then obtain a smoothalpha matte for Ω by using the estimated camera parameters and Eq. (6), and calculate R by Eq. (2). We modify α
and R in the next subsection to account for wavy reflection
surfaces.
3.4. Modified R and αHere, we modify R and α to account for wavy surfaces,
since the alpha matte must contain high-frequency regions
because of the varying incident angles of the viewing rays.
However, the alpha matte obtained in the previous subsection is smooth, and the current R contains high-frequency
residuals.
To solve these issues, we smooth the residuals in R, andthen compute α by Eq. (2) using the smoothed R. Note
submitted to Eurographics Symposium on Rendering (2012)

T

Figure 4: Geometric interpretation of the proposed filter.
The degree of smoothing by the filter is dependent on β,which indicates the angle between I(x) − T and I(y) − T
in RGB color space, and can detect color variation of thetrue R. Our filter with β can smooth waves caused only by
variation of α.

that we must preserve the inherent edges of the reflection
and smooth only the residuals during this process. However, typical edge-preserving filters, including the bilateralfilter [TM98], are not optimal because they preserve edges
of not only the true R (i.e., its exact form) but also highfrequency residuals, which should be smoothed. Therefore,
we design a new filter as a variant of the bilateral filter, andapply it to R as follows:∑y∈Wx G(x, y)w(x, y)R(x),(12)∑y∈Wx G(x, y)w(x, y)!
Ã
(β(x, y))2,w(x, y) = exp −
(13)
σ
µ
¶
(I(x) − T) · (I(y) − T)β(x, y) = arccos. (14)
kI(x) − Tk kI(y) − TkR∗ (x) =Here, R∗ is the filtered R, Wx is a set of pixels in the K ×
K filter kernel (with kernel size K) centered at x, G is thespatial Gaussian weight, w is a proposed weight function,and σ is the variance of w. Our weight function w can detect
the variability between the inherent colors of neighboring
pixels in the reflection. If the difference between these colors
is small, then our filter increases the smoothing of R. We
set σ = 0.01 for all experiments, whereas the kernel size isadjusted for each input image according to the image-space
wavelength of the wavy surface in Ω.
We explain w(x, y) of our filter geometrically using Fig. 4.
Intuitively, β(x, y) in w(x, y) is the angle between vectors
I(x)−T and I(y)−T. Note that we should smooth R according to its true form, but we do not know the exact values of its
elements. Instead, we know I and T, and I − T has the sameorientation as R − T because I(x) − T = α(x)(R(x) − T) by
Eq. (2). Therefore, β(x, y) detects the difference between thetrue R of x and y through I − T, and β(x, y) is independent

Endo et al. / Matting and Compositing for Fresnel Reflection on Wavy Surfaces(b) Bilateral filter (a) Gaussian filter

6(d) Ground truth(c) Our filter2α (Fresnel) matte

T (inset) and R

Algorithm 1 Reflection matting.
Input: I, reflection region Ω and pairwise scribbles{SΠ , SΩ }Output: T, R and αIds ← D OWN S AMPLING(I);RΩ ← C OLORT RANSFER (Ids , SΠ , SΩ ); // Sec. 3.2
T ← C ALC T RANSMISSION C OLOR(Ids , RΩ ); // Sec. 3.2{α(p), y p } ← C OLLECT S AMPLES(Ids , T, Ω); // Sec. 3.3(θo ,φ) ← C ALC C AMERA PARAM({α(p),y p }); // Sec. 3.3
α ← C ALC A LPHA M ATTE(θo , φ, Ω); // Sec. 3.3
for each x ∈ Ω do
I(x)−(1−α(x))T
R(x) ←;
α(x)end for
if reflection region Ω is wavy thenR ← A PPLY S MOOTHING F ILTER(R, Ω); // Sec. 3.4
for each x ∈ Ω do(I−T)·(R(x)−T)
α(x) ← kR(x)−Tk2 ;end forend ifComposite results

Figure 5: Comparison among the results of our filter, theresult of conventional filters, and ground-truth data. Our filter yields the most accurate alpha matte and most plausible
reflection image in the composite result.

of α. Consequently, our filter can preserve the edges of thetrue R and smooth the residuals in the current R caused bychanges in α.
Figure 5 compares the results of our filter with those of
conventional Gaussian and bilateral filters. We use the input
image in Fig. 1, which is synthesized by ray tracing withan environment map and Stokes waves. We also compare
our results with ground-truth data of this synthesized input
image. We set the kernel sizes of all the filters to 27 × 27
because of the relatively wide waves in the input.
In the Gaussian filter results (Fig. 5(a)), the filtered R is
too smooth overall and α includes undesired edges of thereflected objects. These errors cause an inappropriate composite result where the reflections of the trees are blurred.
The bilateral filter successfully eliminates the edges of reflected objects in α (Fig. 5(b)). However, it fails to smoothwaves in R, and thus fails to capture the waves in α. As aresult, the reflected image of the gull in the composite result
is unnaturally flat. Our filter effectively improves the abovementioned errors (Fig. 5(c)). Specifically, waves in the true
α are extracted in our results, while the reflection edges arepreserved similarly to the bilateral filter. Compared with theground-truth data (Fig. 5(d)), our results extract each component more accurately in all cases.
Finally, Algorithm 1 summarizes our matting algorithm.

4. Reflection Composition
After decomposing an input image, we can generate the reflections of newly composited objects by using the calculated components and camera parameters. In the compositing step, a new reflection component Rnew for a composited
object is synthesized to create the final composite result Inew :Inew (x) = α(x)Rnew (x) + (1 − α(x))T.

(15)

We employ ray tracing to compute Rnew . Composited objects are represented as billboards that are facing the camera and in contact with the reflection surface. The incident
angles θ of the viewing rays can be estimated according to
Eq. (6) as follows:µ³ α(x) − α ´ 1 ¶0 5
θ(x) = arccos 1 −,(16)
1 − α0where x is the pixel position passed through by a viewing
ray. Note that naïvely applying ray tracing causes aliasing
in reflection images, especially around object silhouettes; areflected image in a photograph must be slightly blurred because of subpixel-scale reflection due to small waves on asurface. Therefore, we apply prefiltering using a Gaussian
filter when retrieving the colors of real objects.
In addition, we implemented two operations to increase
the flexibility of compositing. The first operation is a contact constraint. If an object faces an oblique direction, a gap
can be created between the object’s bottom and the reflection surface (Fig. 6(b)). To correct this, the user can provide
a scribble to roughly specify where the contact should be(Fig. 6(c)), and our system then extracts the object’s bottom
from within the scribble. The second operation is a heightadjustment. The user can adjust the height of objects fromthe reflection surface using the mouse wheel. This operation
submitted to Eurographics Symposium on Rendering (2012)

Endo et al. / Matting and Compositing for Fresnel Reflection on Wavy Surfaces

7

Table 1: Computational times (s) of matting results for different image sizes (pixel) and filter kernel sizes (pixel).

(a) Specifying contact constraint(b) Without contact constraint

Image
Fig.1
Fig.8(b)Fig.8(d)Fig.8(f)Fig.10(b)Fig.10(d)Fig.10(f)Image size
400 × 400
640 × 428
640 × 450
600 × 400
620 × 403
615 × 413
420 × 634

Kernel size
27 × 275×5
27 × 27
13 × 135×5(w/o filter)13 × 13

Time
7.92
0.896
7.68
1.40
0.801
0.532
1.37

(c) With contact constraint

Figure 6: Contact constraint. (a) A user can specify an object’s bottom using a green scribble, without this specification (b) the object does not make contact with the reflection
surface appropriately, whereas with this specification (c) the
contact is appropriate.

(a) Without height adjustment

PoissonOurs

Input of Poisson

Input of ours(b) With height adjustment

Figure 7: Height adjustment. A user can adjust the height offlying objects (for example, from (a) to (b)) using the mousewheel.

enables us to calculate the reflection of flying objects, suchas birds in the air (Fig. 7(b)).

Figure 9: Comparison between the results obtained by Poisson Image Editing [PGB03] and by our method. While Poisson Image Editing requires a real reflection image and can
produce unnatural results due to waves of different wavelength or different camera angles, our method can synthesize
plausible reflections even without a real reflection image.

5. Experimental Results and Discussion
We implemented our prototype system as a single-threaded
C++ program, and conducted experiments on a personal
computer with an Intel i7 processor running at 2.8 GHz andwith 8 GB RAM. The statistics of our matting results are
summarized in Table 1. Compositing is performed in realtime, as demonstrated in the video accompanying the online
version of this article.
Figure 8 shows several results of our method when natural images with water surfaces are used as inputs. Reflections
are appropriately generated according to the Fresnel effect.
That is, reflection colors on the surface are dominant at thefar side of the viewing point, whereas dark water colors are
dominant at the near side. Warping by waves is also successfully handled, especially in Figs. 8(d) and (f).
submitted to Eurographics Symposium on Rendering (2012)

The left inset in Fig. 8(f) shows a comparison between our
results and a real reflection in the photograph. We extracted
the object producing the real reflection in the input image
and synthesized a copy next to the original for a side-by-side
comparison. Our result is visually plausible, but the shapeof the reflection is subtly different from real one because of
our billboard approximation. Such a composite might also be
possible by using Poisson Image Editing [PGB03] if a realreflection image is available. However, that method can produce unnatural reflections for images with waves of different
wavelengths or different camera angles (Fig. 9). We emphasize here that Poisson Image Editing cannot generate a reflection without a real reflection image, whereas our method
can synthesize a plausible reflection even if a real reflection
image does not exist.

8Endo et al. / Matting and Compositing for Fresnel Reflection on Wavy Surfaces

(a)

(c)

(e)

(b)

(d)

(f)

Figure 8: Our synthetic results for photographs with water surfaces. Each inset shows a magnified section of the image.

(a)

(c)

(b)

(d)

(e)

(f)

Figure 10: Our method can handle reflections on (a) and (b) wet ground and off-specular reflections on glossy surfaces including (c) and (d) tabletops and (e) and (f) floors. Note that original objects (insets in (a), (c), and (e)) do not have reflection
images.

Our method is also applicable for reflections on other surfaces, such as wet ground (Figs. 10(a) and (b)), tabletops(Figs. 10(c) and (d)), and glossy floors (Figs. 10(e) and (f)).
Although the cup has a fixed size in Fig. 10(c), the relative image sizes of the cups in Fig. 10(d) are automatically adjusted according to the scene depths determined by
the estimated camera parameters (Section 3.3). In the othertwo examples, we also consider warping by wavy reflection surfaces. Note that our method can produce plausible
reflection images even though the original objects (insets in
Figs. 10(a), (c), and (e)) do not have reflection images.
Limitation of our method. The billboard approximation
used in ray tracing can result in reflections with an unnat-

ural appearance. The tops of the tea cups are rendered asreflection images in Fig. 11; but, in reality, these would not
be seen. This artifact should be alleviated by incorporating
the geometries of the real objects.
6. Conclusion and Future Work
We have presented a method of matting and compositing forreflections in images. Our system enables a user to easily edita reflection through only a few simple interactions. Under
several assumptions, our matting algorithm solves the reflection matting problem that has been shown to be intractablewith existing image matting techniques. Despite the simplisubmitted to Eurographics Symposium on Rendering (2012)

Endo et al. / Matting and Compositing for Fresnel Reflection on Wavy Surfaces

9[HST09] H E K. M., S UN J., TANG X.: Single image haze removal using dark channel prior. In CVPR (2009), pp. 1956–1963.
1, 2[KN09] K RATZ L., N ISHINO K.: Factorizing scene albedo anddepth from a single foggy image. In ICCV (2009), IEEE,
pp. 1701–1708. 1, 2
[KRFB06] K HAN E. A., R EINHARD E., F LEMING R.,
B UELTHOFF H.: Image-based material editing. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 25, 3 (2006). 2

Figure 11: Limitation of our method. Tops of the tea cups
are rendered in reflections (e.g., as marked by the red oval)
because of the billboard approximation used in ray tracing.
This is physically incorrect.

fications resulting from our assumptions, we have demonstrated that the proposed method generates visually plausible
composited reflections.
Our work is the first to produce mattes for complicated reflections, and we expect it to open new avenues of research,
including the handling of highly complex (e.g., curved) reflection surfaces and real-object geometries, and the separation of transmission components for images with shallow
water surfaces (i.e., a pond, river, or pool) where intricate
bed patterns will be visible.
Acknowledgements
We would like to thank Daisuke Miyazaki and Yosuke
Bando for their valuable comments. We also thank anonymous reviewers for their insightful suggestions. This work
was supported by Grant-in-Aid for Young Scientists (B)23700101.
References[BBZZ05] B RONSTEIN A. M., B RONSTEIN M. M.,
Z IBULEVSKY M., Z EEVI Y. Y.:
Sparse ICA for blindseparation of transmitted and reflected images. International
Journal of Imaging Systems and Technology 15, 1 (2005), 84–91.
3[CGC∗ 03] C HUANG Y.-Y., G OLDMAN D. B., C URLESS B.,
S ALESIN D. H., S ZELISKI R.: Shadow matting and compositing. ACM Trans. Graph. 22, 3 (2003), 494–500. 1[CGZ∗ 05] C HUANG Y.-Y., G OLDMAN D. B., Z HENG K. C.,C URLESS B., S ALESIN D. H., S ZELISKI R.: Animating pictures with stochastic motion textures. In ACM SIGGRAPH 2005
Papers (2005), SIGGRAPH ’05, pp. 853–860. 1[CZH∗ 00] C HUANG Y.-Y., Z ONGKER D. E., H INDORFF J.,C URLESS B., S ALESIN D. H., S ZELISKI R.: Environment matting extensions: Towards higher accuracy and real-time capture.
In Siggraph 2000, Computer Graphics Proceedings (2000). 2[Fat08] FATTAL R.: Single image dehazing. In ACM SIGGRAPH
2008 papers (2008), SIGGRAPH ’08, pp. 72:1–72:9. 1, 2
[GSZ08] G AI K., S HI Z. W., Z HANG C. S.: Blindly separating
mixtures of multiple layers with spatial shifts. In CVPR (2008),
pp. 1–8. 3
submitted to Eurographics Symposium on Rendering (2012)[LW07] L EVIN A., W EISS Y.: User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans.
Pattern Analysis and Machine Intelligence 29, 9 (2007), 1647–
1654. 3
[LZW04] L EVIN A., Z OMET A., W EISS Y.: Separating reflections from a single image using local features. In CVPR (1)(2004), pp. 306–313. 3[PD03] P EERS P., D UTRE P.: Wavelet environment matting. In
Proceedings of the 14th Eurographics Workshop on Rendering( 25–27 2003), pp. 157–166. 2[PGB03] P ÉREZ P., G ANGNET M., B LAKE A.: Poisson image
editing. ACM Transactions on Graphics 22, 3 (2003), 313–318.
7[PHCS06] PARK J., H EO N., C HOI S., S HIN S. Y.: Tour into the
picture with water surface reflection and object movements: Research articles. Comput. Animat. Virtual Worlds 17, 3-4 (2006),
315–324. 1[RAGS01] R EINHARD E., A SHIKHMIN M., G OOCH B.,
S HIRLEY P.: Color transfer between images. IEEE Computer
Graphics and Applications 21, 5 (2001), 34–41. 1, 4
[RRW∗ 09] R HEMANN C., ROTHER C., WANG J., G ELAUTZ
M., KOHLI P., ROTT P.: A perceptually motivated online benchmark for image matting. In In Proc. of Conference on Computer
Vision and Pattern Recognition (CVPR) (2009). 2[Sch94] S CHLICK C.:
An inexpensive BRDF model forphysically-based rendering. Computer Graphics Forum 13, 3(1994), 233–246. 4[Tan08] TAN R. T.: Visibility in bad weather from a single image.
In CVPR (2008), pp. 1–8. 1, 2
[TM98] T OMASI C., M ANDUCHI R.: Bilateral filtering for grayand color images. In ICCV (1998), pp. 839–846. 5[WC07] WANG J., C OHEN M. F.: Optimized color sampling forrobust matting. In CVPR (2007), pp. 1–8. 4[WC08] WANG J., C OHEN M. F.: Image and video matting: asurvey. In Foundations and Trends in Computer Graphics andVision (2008), vol. 3. 2, 3
[WFZ02] W EXLER Y., F ITZGIBBON A., Z ISSERMAN A.:
Image-based environment matting.
In Proceedings of the
13th Eurographics Workshop on Rendering (RENDERINGTECHNIQUES-02) ( 26–28 2002), pp. 279–290. 2[WTBS07] W U T.-P., TANG C.-K., B ROWN M. S., S HUM H.Y.: Natural shadow matting. ACM Trans. Graph 26, 2 (2007). 1,
2[YTBK11] Y EUNG S.-K., TANG C.-K., B ROWN M. S., K ANG
S. B.: Matting and compositing of transparent and refractive
objects. ACM Transactions on Graphics 30, 1 (2011), 2:1–2:13.
1, 2, 3[ZWCS99] Z ONGKER D. E., W ERNER D. M., C URLESS B.,
S ALESIN D.: Environment matting and compositing. In SIGGRAPH (1999), pp. 205–214. 2