Extracting Depth and Matte using a Color-Filtered Aperture
Yosuke Bando∗,‡TOSHIBA Corporation
The University of Tokyo

Bing-Yu Chen†
National Taiwan University

Tomoyuki Nishita‡
The University of Tokyo

(a)
(b)
(c)
(d)
(e)
Figure 1: (a) Top: camera lens with color filters placed in the aperture. Bottom: filter arrangement. (b) Captured image. The backgroundcolor is misaligned (see Fig. 16(c) for a closeup). (c) Estimated depth (the darker, the nearer). (d) Extracted matte. (e) Composite image.

Abstract
This paper presents a method for automatically extracting a scene
depth map and the alpha matte of a foreground object by capturing
a scene through RGB color filters placed in the camera lens aperture. By dividing the aperture into three regions through which only
light in one of the RGB color bands can pass, we can acquire three
shifted views of a scene in the RGB planes of an image in a single
exposure. In other words, a captured image has depth-dependentcolor misalignment. We develop a color alignment measure to estimate disparities between the RGB planes for depth reconstruction.
We also exploit color misalignment cues in our matting algorithm
in order to disambiguate between the foreground and backgroundregions even where their colors are similar. Based on the extracted
depth and matte, the color misalignment in the captured image canbe canceled, and various image editing operations can be applied
to the reconstructed image, including novel view synthesis, postexposure refocusing, and composition over different backgrounds.
Keywords: computational photography, computational camera,color filters, color correlation, depth estimation, alpha matting

1

Introduction

Rapid progress in the field of computational photography has
brought forth new types of cameras and imaging systems capableof capturing additional scene properties that conventional photography misses. These properties, when combined with computation,
extend the ability of imaging applications in many ways: increased
dynamic range and resolution, depth-guided editing, post-exposure
refocusing, variable lighting and reflectance, to name but a few.
∗ e-mail: yosuke1.bando@toshiba.co.jp† e-mail: robin@ntu.edu.tw‡ e-mail: {ybando, nis}@is.s.u-tokyo.ac.jp
ACM Reference Format
Bando, Y., Chen, B., Nishita, T. 2008. Extracting Depth and Matte using a Color-Filtered Aperture.
ACM Trans. Graph. 27, 5, Article 134 (December 2008), 9 pages. DOI = 10.1145/1409060.1409087
http://doi.acm.org/10.1145/1409060.1409087.
Copyright Notice
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted
without fee provided that copies are not made or distributed for profit or direct commercial advantage
and that copies show this notice on the first page or initial screen of a display along with the full citation.
Copyrights for components of this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any
component of this work in other works requires prior specific permission and/or a fee. Permissions may berequested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701, fax +1(212) 869-0481, or permissions@acm.org.
© 2008 ACM 0730-0301/2008/05-ART134 $5.00 DOI 10.1145/1409060.1409087
http://doi.acm.org/10.1145/1409060.1409087

While elaborate imaging systems and optical elements continue toemerge, one of the recent trends in this field is to make a systemcompact, or even portable [Ng et al. 2005; Georgeiv et al. 2006],
and to simplify optical elements to be attached to the conventional
camera [Levin et al. 2007; Veeraraghavan et al. 2007]. The ability
to easily switch from being a computational camera to the conventional one to capture regular photographs is also claimed as
an advantage [Green et al. 2007; Liang et al. 2008]. This trend
will serve as a driving force for making computational photography
more commonplace and affordable for ordinary users.
To boost this trend, this paper proposes a method for automaticallyextracting a scene depth map and the alpha matte of a foregroundobject with a conventional camera body and a slightly modified
camera lens with RGB color filters placed in the aperture. By dividing the aperture into three regions through which only light in one of
the RGB color bands can pass, we can acquire three shifted views ofa scene in the RGB planes of a captured image in a single exposure,which enables depth reconstruction. While this idea has alreadybeen proposed previously [Amari and Adelson 1992; Chang et al.
2002], we realize this idea in a hand-held camera without the needfor additional equipment other than color filters. We also devise abetter correspondence measure between the RGB planes which arerecorded with different bands of wavelength. Moreover, we propose
a method for extracting the matte of an in-focus foreground object,which is an entirely novel application of a color-filtered aperture.
Color misalignment cues introduced by the filters serve to constrain
the space of possible mattes that would otherwise contain erroneous
mattes when foreground and background colors are similar.
The downsides of using a color-filtered aperture are that objects
having only a single pure R, G, or B color cannot be handled, andthat the visual quality of images is spoiled by color misalignment.
We will show, however, that our method can handle many realworld objects, and we also present how to reconstruct color-aligned
images using extracted depth and matte. By showing results foroutdoor scenes and/or hairy foreground objects, we demonstrate theportability of our device and the effectiveness of our method, withseveral image editing examples such as novel view synthesis, postexposure refocusing, and composition over different backgrounds.

2

Related Work

This section reviews several research areas that are closely related
to our work. Readers can refer to [Raskar et al. 2006] for an extensive survey on computational photography.
ACM Transactions on Graphics, Vol. 27, No. 5, Article 134, Publication date: December 2008.

134:2•

Y. Bando et al.

Color-filtered aperture. The idea of using color filters in the aperture to estimate depth has been proposed previously. Amari andAdelson [1992] used a squared intensity difference measure forhigh-pass filtered images to estimate disparities. As they discussed
in their paper, however, this measure was insufficient to compensate for intensity differences between the color planes. Their prototype was not portable, and only a single result for a textured planar
surface was shown. Chang et al. [2002] normalized the intensities within a local window in each color plane before taking thesum of absolute differences between them. But as their camera
was equipped with a flashbulb for projecting a speckle pattern onto
the scene in order to generate strong edges in all the color planes,
the performance of their correspondence measure in the absenceof a flash was not shown. They also had to capture another image without flash to obtain a “normal” image. We propose a better
correspondence measure between the color planes. We believe our
matting method based on a color-filtered aperture is entirely new.
Coded aperture. Several researchers placed a patterned mask in
the aperture to change the frequency characteristics of defocus blurto facilitate blur identification/removal and depth estimation [Levinet al. 2007; Veeraraghavan et al. 2007]. These methods offeredportable imaging systems with minimal modifications to the conventional camera, which inspired us to pursue this direction. Our
approach differs in that it relies on parallax cues rather than defocus cues, which introduces a view correspondence problem but escapes ambiguity between depths farther and nearer than the focused
depth. We also propose a parallax-based matting method.
Single-lens multi-view image capture. Adelson and Wang [1992]
showed that light rays entering a camera can be captured separately
depending on their incident angle by placing a microlens array on
the image sensor, and they estimated depth from multi-view images
captured through a single main lens. Ng et al. [2005] realized thisidea in a hand-held camera, and proposed a post-exposure refocusing method by noting that the captured multi-view images correspond to the light field inside the camera [Ng 2005]. Multi-view
images can also be captured by placing an attenuation mask on theimage sensor [Veeraraghavan et al. 2007], or by splitting light rays
at the aperture [Green et al. 2007; Liang et al. 2008] or outside
the main lens [Georgeiv et al. 2006]. Our method also splits lightrays at the aperture, but requires only color filters as additional optical elements to the lens without requiring multiple exposures. Although this comes with a price of a reduced number of views (onlythree) each having only a single color plane, we can still obtain
useful information for post-exposure manipulation of images.
Matting. In image editing, matting is an important technique forextracting foreground objects in an image so that they can be composited over other images. We only review some of the most relevant work to ours here. Interested readers can refer to Chuang’s
thesis [2004] for more information. The traditional approach tomatting is to use a blue or green screen as a background [Vlahos
1971; Smith and Blinn 1996]. Extracting a matte from a single natural image (i.e., an image with general unknown background colors) requires user intervention, a typical form of which is a trimap
that segments an image into “strictly foreground,” “strictly background,” and “unknown” regions. Fractional alpha values are computed in the “unknown” region based on the information from theother two regions [Chuang et al. 2001; Levin et al. 2008; Wang
and Cohen 2007]. To automate matting, previous approaches used
multiple images. Smith and Blinn [1996] captured images of a foreground object with two different known background colors. Alternatively, Wexler et al. [2002] used a sequence of images of a translating/rotating object. Xiong and Jia [2007] captured images fromtwo viewpoints, and computed their stereo correspondences taking
into account alpha values of a foreground object. Several methods
used synchronized cameras to capture multiple images of an ob-

ject [McGuire et al. 2005; McGuire et al. 2006; Joshi et al. 2006].
Our method can automatically extract alpha mattes with a single
hand-held camera in a single exposure.

3

Color-Filtered Aperture

Fig. 1(a) shows our prototype camera lens with color filters in theaperture. We arranged the RGB regions so that their displacementwith respect to the optical center of the lens aligns with the X andY axes of the image sensor, as indicated by the arrows in Fig. 1(a)bottom. By this arrangement, a scene point farther than the focused
depth is observed with a rightward shift in the R plane, an upward
shift in the G plane, and a leftward shift in the B plane. A scene
point nearer than the focused depth will be shifted in the opposite
directions. Note that these color shifts come from geometric optics,
not from chromatic aberration. Fig. 2 illustrates this phenomenon
in 2D where the aperture is split into two (R and G) regions.
For a prototype camera lens, we cut out a disc with a triple-squareshaped hole from a piece of black cardboard, glued color filters (Fujifilter SC-58, BPB-53, and BPB-45) to it, and attached it immediately in front of the aperture diaphragm of a Canon EF 50mm f/1.8
II lens. This fabrication was done in a few hours with a box cutter
and a micro screwdriver. We used an unmodified Canon EOS40DDSLR as a camera body. Fig. 3 shows the point-spread function(PSF) of the prototype lens, which is an image of a defocused point
light source. The square shape of each filter is observed mostly asis, with only slightly rounded corners at the horizontal extremities
due to occlusion by the lens housing. Fig. 3 also shows that thethree color bands are well separated. We achieved this by applying a linear transform to RGB sensor response so as to minimize
crosstalk between the aperture filters and the image sensor (see Appendix A for details).
To align the RGB regions with the image sensor axes, manual adjustment was sufficient. Once this is done, pixel disparities will
always align with the X and Y axes of captured images, requiring no further calibration and rectification at capture time or during
post-processing. Fig. 4 shows an example photograph and its separated RGB planes. Due to the higher transmittance of the R filter,
captured images shown in this paper are relatively reddish.
backgroundfocusedobjectcolor filterslensbackgroundcolor filters

lens

image sensor(a)image sensor(b)

Figure 2: 2D illustration of the interactions between light raysfrom a scene point and a color-filtered aperture. (a) For a scene
point at the focused depth, light rays in the R band and those in the
G band converge to the same point on the image sensor. (b) For ascene point off the focused depth, light rays in the two bands reachdifferent positions on the image sensor, resulting in a color shift.

Color

Red

Green

Blue

Figure 3: Point-spread function of our lens and its RGB components. The positions of the R and B regions are opposite to those in
Fig. 1(a), as the viewpoint is at the opposite side of the filters.

ACM Transactions on Graphics, Vol. 27, No. 5, Article 134, Publication date: December 2008.

Extracting Depth and Matte using a Color-Filtered Aperture

•

134:3

Now that we can evaluate the quality of a match between the RGB
planes, we can find the disparity d that minimizes L(x, y; d) at eachpixel (x, y), from a predetermined set of disparity values (-5 to 10
in our implementation). As local estimates alone are prone to error,
we use the standard energy minimization framework using graphcuts [Boykov et al. 2001] to impose spatial smoothness constraints.
B
1

0.5
1
00

0.5
0.5

Figure 4: Example photograph taken with our lens, and its separated RGB planes. The white lines are superimposed to highlightthe background color shifts. See Fig. 15(a) for a closeup view.

4(c) d = 0, L = 0.003

(b)

B

B

1

B

1

0.5

Depth Estimation

1

0.5

0.5

1

The RGB planes Ir , Ig , and Ib of a captured image I correspondto three views of a scene. If we take a virtual center view (cyclopean view) as a reference coordinate system, the R, G, and B planes
are shifted to rightward, upward, and leftward according to the arrangement of the aperture color filters. Therefore, letting d be ahypothesized disparity at (x, y), we need to measure the quality ofa match between Ir (x+d, y), Ig (x, y −d), and Ib (x−d, y).
Clearly, we cannot expect these three values to have similar intensities because they are recorded with different bands of wavelength. To cope with this issue, inspired by Levin et al.’s matting approach [2008], we exploit the tendency of colors in natural
images to form elongated clusters in the RGB space (color lines
model) [Omer and Werman 2004]. We assume that pixel colors
within a local window w(x, y) around (x, y) belong to one cluster,
and we use the magnitude of the cluster’s elongation as a correspondence measure. More specifically, we consider a set SI (x, y; d)of pixel colors with hypothesized disparity d as SI (x, y; d) ={(Ir (s+d, t), Ig (s, t−d), Ib (s−d, t)) | (s, t) ∈ w(x, y)}, andsearch for d that minimizes the following color alignment measure:L(x, y; d) = λ0 λ1 λ2 /σr2 σg2 σb2 ,

(a)

(1)

where λ0 , λ1 , and λ2 (λ0 ≥ λ1 ≥ λ2 ≥ 0) are the eigenvalues of
the covariance matrix Σ of the color distribution SI (x, y; d), and
σr2 , σg2 , and σb2 are the diagonal elements of Σ. Note that the dependence on (x, y; d) of the right-hand side of Eq. (1) is omitted
for brevity. L(x, y; d) is the product of the variances of the color
distribution along the principal axes, normalized by the product ofthe variances along the RGB axes. It gets small when the cluster iselongated (i.e., λ0 ≫ λ1 , λ2 ) in an oblique direction with respect
to the RGB axes, meaning that the RGB components are correlated.
In fact, this measure can be interpreted as an extension of normalized cross correlation (NCC) [Lewis 1995] so that it is applicable tothree images simultaneously (see Appendix B). L(x, y; d) is in therange [0, 1], with the upper bound given by Hadamard’s inequality [Gradshteyn and Ryzhik 2000], since λ0 λ1 λ2 = det(Σ).
To illustrate the effect of this measure, we use a sample image
shown in Fig. 5(a), taken with a conventional camera lens. Since
its RGB planes are aligned, the true disparity is d = 0 everywhere,
and colors within the local window indicated by the red rectangle inFig. 5(a) actually form an elongated cluster, as shown in Fig. 5(c). If
we deliberately misalign the RGB planes by d = 1, 3, and 5 pixels,
the distribution becomes more isotropic, and the color alignment
measure becomes larger, as shown in Figs. 5(d-f).

00

0.5
0.5

R

G

10

(d) d = 1, L = 0.11

G

10

R

1
00

0.5
0.5

R

G

10

(e) d = 3, L = 0.39

1
00

0.5
0.5

R

G

10

(f) d = 5, L = 0.54

Figure 5: (a) Sample photograph taken with a conventional camera lens. (b) Closeup of the local window indicated by the red rectangle in (a). (c-f) Plots of the pixel colors within the local window
in the RGB space. The values d and L shown below each plot are
the simulated disparity and the value of Eq. (1).

5Matting

Matting is a problem of solving for foreground opacity α(x, y) ateach pixel (x, y) in the following matting equation.
I(x, y) = α(x, y)F(x, y) + (1 − α(x, y))B(x, y),(2)which models an observed image I as a convex combination of aforeground color F and a background color B. By capturing animage so that a foreground object is in focus, we can assume thatα(x, y) is aligned between the RGB planes. More precisely, regions with fractional alpha values (i.e., the silhouette of a foreground object) should be within the depth-of-field of the lens.
Slight violation of this assumption however does not lead to severe
degradation of extracted mattes, as will be shown in Sec. 6.
Solving Eq. (2) based only on the observation I is an underconstrained problem, since we have only three measurements (Ir ,Ig , and Ib ) for seven unknowns (α, Fr , Fg , Fb , Br , Bg , and Bb ).
at each pixel. Therefore, to incorporate additional constraints, weuse a trimap which we automatically generate from the disparity
map, and we also leverage the difference in misalignment between
foreground and background colors to iteratively optimize the matte.

5.1

Matte Optimization Flow

Algorithm 1 shows our iterative matte optimization procedure. For
initialization, we first roughly divide the image into foreground andbackground regions by thresholding the disparity map, and we dilate their border to construct a trimap having a conservatively wide
“unknown” region (50-70 pixels in our implementation), as shownin Fig. 6(a). We then initialize the alpha values using a trimap-based
matting method, for which we used Levin et al.’s Closed-Form Matting [2008]. While this often gives already good results, errors can

ACM Transactions on Graphics, Vol. 27, No. 5, Article 134, Publication date: December 2008.

134:4•

Y. Bando et al.

remain where foreground and background colors are similar (seeFig. 9(a) as an example). We detect and correct these errors in thesubsequent iterative optimization using color misalignment cues.
To determine how the foreground and background colors are misaligned in the “unknown” region, we make a two-layer assumption
for the scene around the foreground silhouette. And we propagate
the disparity values from the “strictly foreground” region to obtainforeground disparity map dF (x, y) as shown in Fig. 6(b). Similarly
we obtain background disparity map dB (x, y) from the “strictly
background” region (Fig. 6(c)).
In the iterative optimization, letting n denote an iteration count, wefirst estimate foreground and background colors Fn and Bn basedon
P the current matte αn , by minimizing a quadratic cost function2(x,y) ||I(x, y) − αn (x, y)Fn (x, y) − (1 − αn (x, y))Bn (x, y)||derived from Eq. (2), plus smoothness constraints on foregroundand background colors, similar to [Levin et al. 2008]. These estimated colors Fn and Bn have errors in the same regions as αnhas errors. We detect these erroneous regions by measuring how
consistent the estimated colors are with the foreground and background disparity maps dF (x, y) and dB (x, y), as we will describein Sec. 5.2. We then correct the alpha values around the detected regions to obtain the matte αn+1 for the next iteration (Sec. 5.3). We
iterate this process until change in the matte is sufficiently small.
Fig. 7 illustrates each step of the iterative optimization.
Algorithm 1: Matte optimization algorithm.
Initialization
1. Construct a trimap from the disparity map.
2. Find an initial matte α0 based on the trimap.
3. Propagate the disparity values to obtain foreground andbackground disparity maps dF and dB .
Iterative optimization
1. Estimate foreground color Fn and background color Bn
based on the current αn .
2. Compute consistency measures CFn and CBn (Sec. 5.2).
3. Update αn+1 based on CFn and CBn (Sec. 5.3).
4. Repeat until convergence.

where N = |SF (x, y; d)|, and li is the distance of the i-th color
in SF (x, y; d) from the line fitted to the elongated color cluster(i.e., the first principal axis). Intuitively, we examine whether thecolors in a local window fit the color lines model. Therefore,eF (x, y; d) becomes large when d is a wrong disparity. We define the background color lines model error eB (x, y; d) similarly.
See Appendix C for more details.
As we have two possible disparities dF (x, y) and dB (x, y) at eachpixel (x, y) in the “unknown” region, we define foreground andbackground color consistency measures by incorporating two values of color lines model errors at these two disparities:˘¯
CF (x, y) = exp (eF (x, y; dF ) − eF (x, y; dB ))/κs ,(4)˘¯CB (x, y) = exp (eB (x, y; dB ) − eB (x, y; dF ))/κs ,where κs is a scale parameter. If the estimated foreground color
around (x, y) erroneously contains the (true) background color,
CF (x, y) will be large around that region because eF (x, y; dF ) will
be large and eF (x, y; dB ) will be small. The effect of the background counterpart CB (x, y) can be similarly explained.

5.3

Solving for the Matte

Following Wang and Cohen’s Robust Matting approach [2007], we
solve for α(x, y) as a soft graph-labeling problem, where each pixel(regarded as a node in a graph) has data weights WF (x, y) andWB (x, y), and each pair of neighboring pixels has an edge weightWe (x0 , y0 ; x1 , y1 ). The data weight WF (x, y) is responsible forpulling α(x, y) toward 1, whereas WB (x, y) pulls it toward 0. Theedge weights impose spatial smoothness constraints on alpha values
by the Matting Laplacian [Levin et al. 2008]. This formulation isbeneficial in that it can be solved as a sparse linear system [Grady
2006], not graph-cuts, and that it guarantees α(x, y) to be in therange [0, 1] without additional hard constraints.
While Wang and Cohen [2007] used color samples gathered fromthe “strictly foreground” and “strictly background” regions to set
the data weights, we instead iteratively update the data weights
according to the consistency measures CFn (x, y) and CBn (x, y)
computed for the current estimate of the foreground and background colors Fn and Bn , as follows.
WFn (x, y) = κα αn (x, y)+ κc (CBn (x, y) − CFn (x, y)),WBn (x, y) = κα (1 − αn (x, y)) + κc (CFn (x, y) − CBn (x, y)),
(5)

(a)

(b)

(c)

Figure 6: (a) Trimap for the toy dog image in Fig. 4, constructedfrom the disparity map shown in Fig. 10(d) top. White: strictly
foreground. Black: strictly background. Gray: unknown. (b) Propagated foreground disparity map dF (x, y). Blue indicates an undefined region. (c) Propagated background disparity map dB (x, y).

5.2

Measuring Consistency with Disparity Maps

6

Similar to the color alignment measure in Eq. (1), we consider
a set SF (x, y; d) of pixel colors within a local window w(x, y),in this case for the foreground color F(x, y), not for the input
image I(x, y), with hypothesized disparity d as SF (x, y; d) ={(Fr (s+d, t), Fg (s, t−d), Fb (s−d, t)) | (s, t) ∈ w(x, y)}, and we
define a foreground color lines model error as follows.
NeF (x, y; d) =

1 X 2
li ,
N i=1

where κα and κc are constants. We clamp WFn (x, y) andWBn (x, y) at 0 to keep them non-negative. When the foregroundconsistency measure CFn (x, y) is smaller (i.e., more consistent)
than the background counterpart CBn (x, y), the foreground data
weight WFn (x, y) is increased while the background data weight
WBn (x, y) is decreased, so that α(x, y) is pulled toward 1 from thecurrent value αn (x, y). Conversely, α(x, y) will be pulled toward
0 if CFn (x, y) > CBn (x, y).

(3)

Results

For all of the results shown below, we set the local window size to
15 × 15 pixels, κs = 0.1, κα = 0.01, and κc = 0.02. The matte
optimization converged in about 20 iterations. The computation
time for a 720 × 480 image was 10 sec. for depth estimation, and2 min. for matting on an Intel Pentium 4 3.2GHz with 2GB RAM.
We first demonstrate the performance of our RGB correspondence
measure. We compare our disparity estimation results with those of
the previous methods [Amari and Adelson 1992; Chang et al. 2002]

ACM Transactions on Graphics, Vol. 27, No. 5, Article 134, Publication date: December 2008.

Extracting Depth and Matte using a Color-Filtered Aperture

•

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

134:5

Figure 7: Synthetic toy example demonstrating how our matte optimization works. (a) Ground truth foreground color. (b) Ground truth
background color. (c) Ground truth matte. (d) Composite image from (a-c) with the background color misaligned by 5 pixels. This image is
input to our matting algorithm. (e) Trimap. In this example we manually drew it in order to leave a wide “unknown” region. (f) Initialized
matte α0 . The center image region has large errors because the foreground and background colors are similar. These errors will be correctedin the subsequent steps using color misalignment cues from the ‘x’ shaped textures. (g) Estimated foreground color F0 based on α0 in (f).
Blue indicates an undefined region. (h) Estimated background color B0 based on α0 in (f). (i) Foreground color consistency CF0 computedfor F0 in (g). The disparity of (g) around the top center region is 5, which is inconsistent with the true foreground disparity of 0. Therefore,
CF0 became large around there. (j) Background color consistency CB0 computed for B0 in (h). The disparity of (h) around the bottom centerregion is 0, which is inconsistent with the true background disparity of 5. Therefore, CB0 became large around there. (k) Updated matte. The
alpha values were pulled toward 0 where CF0 in (i) is large, and toward 1 where CB0 in (j) is large. (l) Final matte after convergence, which
is close to the ground truth matte (c).

in Figs. 10(a-c). In order to reveal raw performance, we show local
window estimates without graph-cut optimization. As Amari andAdelson’s method relies on high-pass filtering, it mostly failed todetect disparities of the defocused scene backgrounds (Fig. 10(b)).
Chang et al.’s method performed better, but it handled color edges
and gradations poorly, presumably because these may not be accounted for by a single intensity normalization factor within a local
window (Fig. 10(c)). Our method produced better results than theprevious methods (Fig. 10(a)).

misaligned version by shifting the background color by 3 pixels
before composition. We applied the previous methods to the coloraligned synthetic image, and our method to the color-misaligned
one. Though not perfect, our method produced a better matte as
shown in Figs. 11(c-e). For quantitative evaluation, we conducted
the same experiment for five more examples shown in Fig. 12, andwe computed the mean squared errors (MSE) against the groundtruth mattes, which we plotted in Fig. 13. Our method reduced
MSE values by 33-86% compared to the other two methods.

We also compare our results with a mutual information-based
method by Kim et al. [2003], which can handle broad types of intensity relationships between images. Since their method is coupledwith iterative graph-cut optimization, our results after graph-cut optimization are also shown in Fig. 10(d). Because their correspondence measure is defined for two images, we take the average of thevalues for the three pairs of RGB planes (RG, GB, and BR). Theirmethod performed well in view of the fact that it does not assume
a priori knowledge of the intensity relationships. However, some
portions of the foreground objects were not detected (Fig. 10(e)).

As our camera is portable, and only a single exposure is required,
it is easy to capture moving objects such as animals, as shown in
Fig. 14. Using the camera’s rapid shooting capability, we can also
perform video matting. The supplemental video shows an extracted
video matte of a walking person.

Next we show our matting results. Fig. 8(a) shows the extracted
matte for the toy dog image in Fig. 4. The hairy silhouette was
extracted successfully. We can use this matte to refine the boundary of the foreground and background regions in the depth map asshown in Fig. 8(b), by compositing the foreground and backgrounddisparity maps shown in Figs. 6(b-c). In Fig. 9, we applied existing
natural image matting methods, Closed-Form Matting [Levin et al.
2008] and Robust Matting [Wang and Cohen 2007], with the trimap
given by our method. These results are not for comparison because
the previous methods are designed for color-aligned images, but thematte errors seen in Fig. 9 are indicative of the importance of ourcolor consistency measure in suppressing them.
For proper comparison, we used a ground truth matte shown in
Fig. 11(a) obtained by capturing an object in front of a simple
background and by using Bayesian Matting [Chuang et al. 2001],
followed by manual touch-up where needed. We created a synthetic “natural” image by compositing the object over a new background image, as shown in Fig. 11(b). We also created its color-

(a)

(b)

Figure 8: (a) Extracted matte for the toy dog image in Fig. 4.
(b) Refined depth map. Compare this with Fig. 10(d) top.

(a)

(b)

Figure 9: Results of existing natural image matting methods.
(a) Closed-Form Matting. (b) Robust Matting.

ACM Transactions on Graphics, Vol. 27, No. 5, Article 134, Publication date: December 2008.

134:6•

Y. Bando et al.

(a)

(b)

(c)

(d)

(e)

Figure 10: Comparison of correspondence measures between the RGB planes. Larger intensities indicate larger disparities. Top row:
results for the toy dog image in Fig. 4. Bottom row: results for the woman image in Fig. 1. (a) Our method (local estimate). (b) Amari andAdelson [1992] (local). (c) Chang et al. [2002] (local). (d) Our method (after graph-cut optimization). (e) Kim et al. [2003] (based on
mutual information with iterative graph-cut optimization).

(a)

(b)

(c)

(d)

(e)

Figure 11: Comparison using a ground truth matte. (a) Ground truth matte. (b) Synthetic natural image. (c) Our method (applied to thecolor-misaligned version of (b)). (d) Closed-Form Matting (applied to (b)). (e) Robust Matting (applied to (b)).

Fig. 14 also shows a portion of the foreground object (the hip of
the sheep) is slightly out of the depth-of-field of the lens, violating
the assumption that α(x, y) is aligned between the RGB planes in
Eq. (2). However, degradation of the extracted matte around theregion was small, as shown in Fig. 14(d).

Man

Girl

FlowerGiraffe

Tree

Figure 12: Synthetic natural images and their ground truth mattes.
Ours
Closed-Form
Robust Matting

3000
MSE value

2500

7

2000
1500
1000
500
0
cat

man

girl

Finally, we show several post-exposure image editing examples
based on the extracted depth and matte. First of all, we can restore acolor-aligned image (Fig. 15(b)) by re-compositing the foregroundand background colors after canceling their misalignment based on
the foreground and background disparity maps. Specifically, if theforeground disparity at (x, y) is d, the aligned foreground color atthat point is restored as: (Fr (x+d, y), Fg (x, y −d), Fb (x−d, y)).
Moreover, as the defocus PSF is a square whose size is given by
the disparity map, we can restore an all-in-focus background color
by deconvolution (Fig. 15(c)). By blurring the foreground color andthe all-in-focus background color differently, we can syntheticallyrefocus the image (Fig. 15(d)) [Bando and Nishita 2007]. In thepresence of hairy foreground objects, alpha mattes are indispensable for the above operations to give plausible results. The supplemental video shows interactive refocusing and view synthesis
animations. We can also composite color-aligned foreground objects over other images as shown in Figs. 1(e) and 14(e), where we
adjusted the objects’ color to match the corresponding background.
Fig. 16 shows additional color misalignment cancellation results.

flower giraffe

tree

Figure 13: MSE values of the mattes produced by our method andthe previous methods for the images shown in Figs. 11 and 12.

Discussions and Conclusions

We have presented a method for automatically extracting a scene
depth map and the alpha matte of an in-focus foreground object using a color-filtered aperture. Our method only modifies a camera
lens with off-the-shelf color filters to capture multi-view images
in a single exposure. We have proposed an effective correspondence measure between the RGB planes, and a method for employing color misalignment cues to improve the matte. We believe ourconcise camera design with various post-exposure image editing
capabilities will make computational photography a more readily
available tool for many users.

ACM Transactions on Graphics, Vol. 27, No. 5, Article 134, Publication date: December 2008.

Extracting Depth and Matte using a Color-Filtered Aperture

(a)

(b)

(c)

(d)

•

134:7

(e)

Figure 14: Results for a sheep. (a) Captured image. (b) Depth map. (c) Matte. (d) Closeup from the red rectangle in (c). (e) Composite.

(a)

(b)

(c)

(d)

Figure 15: Examples of post-exposure image editing based on the extracted depth and matte. The bottom row shows closeup views of the
top row. (a) Captured image. The colors are misaligned. (b) Color misalignment canceled. (c) Defocus blur removed. (d) Refocused.

The major limitation of our approach is that it does not work for
objects having only a single pure R, G, or B color. Combining
with depth-from-defocus methods may partially solve this problem.
However, this does not mean that objects must have achromatic colors all over. For example, the disparity of the red box in Fig. 4 iscorrectly identified as shown in Fig. 10(d), thanks to the alphabets
and the pictures of chocolates printed on the box. Therefore, ourrequirement is that objects must not be purely colored entirely, andwe think there are many real-world objects satisfying this requirement. We would like to further investigate this limitation.

hair strands and holes in foreground objects, once they are missedin the course of optimization, as shown in Fig. 17(d). We would
like to address the above issues in the future.

In our imaging system, the f-number is fixed to 1.8 (full apertureof our prototype lens) because a large aperture increases disparities
and thus increases depth resolution. Since disparities also increase
when the lens is focused near, our system typically works well forforeground objects at 0.5 to 2.5 meters away from the camera witha sufficiently distant (about twice as far away) background. For farther scenes, depth resolution will gradually decrease, and the mattequality will also deteriorate as color misalignment will be small.
By introducing color filters, the amount of incident light is decreased. Increasing the aperture filter area to compensate for thisintroduces more defocus. While this degrades depth estimation
accuracy at defocused regions, it suppresses background clutters,
which is beneficial for matting. Color filters may also affect color
demosaicing of the image sensor, although we did not observe any
loss of quality in our experiments, mainly because we downsampled the captured images for tractable computation time.
While our depth estimation works fairly robustly, our matting fails
when the foreground and background colors are similar with little
texture, as shown in Fig. 17(b), since we have few color misalignment cues. Another failure mode is that, as we use a relatively large
window (15 × 15), we cannot recover small/thin features such as

(a)

(b)

(c)

Figure 16: More color misalignment cancellation results. (a) Restored images. (b) Closeups of (a). (c) Closeups of the original.

(a)

(b)

(c)

(d)

Figure 17: Failure cases. Major errors are indicated by the arrowand circles. (a) Captured image. (b) Matte from (a). (c) Closeup of
the ground truth matte for the girl image in Fig. 12. (d) Our result.

ACM Transactions on Graphics, Vol. 27, No. 5, Article 134, Publication date: December 2008.

134:8•Y. Bando et al.

Acknowledgments
We gratefully acknowledge helpful comments and suggestions
from Takeshi Naemura, Yusuke Iguchi, Takuya Saito, and theanonymous reviewers. We would also like to thank Johanna Wolf,
Zoltan Szego, Paulo Silva, and Saori Horiuchi for their help.

References
A DELSON , E. H., AND WANG , J. Y. A. 1992. Single lens stereo
with a plenoptic camera. IEEE Trans. PAMI 14, 2, 99–106.
A MARI , Y., AND A DELSON , E. H. 1992. Single-eye range estimation by using displaced apertures with color filters. In Proc.
Int. Conf. Industrial Electronics, Control, Instrumentation, andAutomation, vol. 3, 1588–1592.
BANDO , Y., AND N ISHITA , T. 2007. Towards digital refocusingfrom a single photograph. In Proc. Pacific Graphics, 363–372.
B OYKOV, Y., V EKSLER , O., AND Z ABIH , R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. PAMI 23,
11, 1222–1239.
C HANG , I.-C., H UANG , C.-L., H SUEH , W.-J., L IN , H.-C.,C HEN , C.-C., AND Y EH , Y.-H. 2002. A novel 3-D hand-heldcamera based on tri-aperture lens. In Proc. SPIE 4925, 655–662.
C HUANG , Y.-Y., C URLESS , B., S ALESIN , D. H., AND S ZELISKI ,R. 2001. A bayesian approach to digital matting. In Proc. CVPR,
vol. 2, 264–271.
C HUANG , Y.-Y. 2004. New models and methods for matting andcompositing. PhD thesis, University of Washington.
G EORGEIV, T., Z HENG , K. C., C URLESS , B., S ALESIN , D., NA YAR , S., AND I NTWALA , C. 2006. Spatio-angular resolution
tradeoff in integral photography. In Proc. EGSR, 263–272.
G RADSHTEYN , I. S., AND RYZHIK , I. M. 2000. Table of integrals,
series, and products (sixth edition). Academic Press.
G RADY, L. 2006. Random walks for image segmentation. IEEE
Trans. PAMI 28, 11, 1768–1783.
G REEN , P., S UN , W., M ATUSIK , W., AND D URAND , F. 2007.
Multi-aperture photograpy. ACM Trans. Gr. 26, 3, 68:1–68:7.

M C G UIRE , M., M ATUSIK , W., AND Y ERAZUNIS , W. 2006.
Practical, real-time studio matting using dual imagers. In Proc.
EGSR, 235–244.
N G , R., L EVOY, M., B R ÉDIF, M., D UVAL , G., H OROWITZ , M.,
AND H ANRAHAN , P., 2005. Light field photography with ahand-held plenoptic camera. Tech. Rep. CSTR 2005-02, Stanford Computer Science, Apr.
N G , R. 2005. Fourier slice photography. ACM Trans. Gr. 24, 3,
735–744.
O MER , I., AND W ERMAN , M. 2004. Color lines: image specificcolor representation. In Proc. CVPR, vol. 2, 946–953.
R ASKAR , R., T UMBLIN , J., M OHAN , A., AGRAWAL , A., AND
L I , Y. 2006. Computational photography. In Proc. Eurographics
STAR.
S MITH , A. R., AND B LINN , J. F. 1996. Blue screen matting. In
Proc. ACM SIGGRAPH 96, 259–268.
V EERARAGHAVAN , A., R ASKAR , R., AGRAWAL , A., M OHAN ,A., AND T UMBLIN , J. 2007. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture
refocusing. ACM Trans. Gr. 26, 3, 69:1–69:12.
V LAHOS , P., 1971. Electronic composite photography. U. S. Patent
3,595,987.
WANG , J., AND C OHEN , M. F. 2007. Optimized color sampling
for robust matting. In Proc. CVPR.
W EXLER , Y., F ITZGIBBON , A., AND Z ISSERMAN , A. 2002.
Bayesian estimation of layers from multiple images. In Proc.
ECCV, 487–501.
X IONG , W., AND J IA , J. 2007. Stereo matching on objects withfractional boundary. In Proc. CVPR.

Appendix A

Let cr , cg , and cb be the mean image colors of a sheet of white paper
through the R, G, and B filters, respectively. For our prototype,
cr = (1.000, 0.335, 0.025)T ,J OSHI , N., M ATUSIK , W., AND AVIDAN ., S. 2006. Natural video
matting using camera arrays. ACM Trans. Gr. 25, 3, 779–786.
K IM , J., KOLMOGOROV, V., AND Z ABIH , R. 2003. Visual correspondence using energy minimization and mutual information.
In Proc. ICCV, vol. 2, 1033–1040.
L EVIN , A., F ERGUS , R., D URAND , F., AND F REEMAN , W. T.
2007. Image and depth from a conventional camera with a coded
aperture. ACM Trans. Gr. 26, 3, 70:1–70:9.
L EVIN , A., L ISCHINSKI , D., AND W EISS , Y. 2008. A closedform solution to natural image matting. IEEE Trans. PAMI 30,
2, 228–242.

cg = (0.153, 1.000, 0.162)T ,M C G UIRE , M., M ATUSIK , M., P FISTER , H., D URAND , F., AND
H UGHES , J. 2005. Defocus video matting. ACM Trans. Gr. 24,
3, 567–576.

(A.1)

T

cb = (0.007, 0.190, 1.000) ,where the values are normalized with respect to the maximum component. Letting M = (cr , cg , cb ), we can decompose an observedcolor co into the three aperture filters’ contributions by M −1 co .

Appendix B

Color Alignment Measure and NCC

An equivalent of Eq. (1) in 2D (e.g., in the RG space) would be:L(x, y; d) = λ0 λ1 /σr2 σg2 .

L EWIS , J. P. 1995. Fast template matching. In Proc. Vision Interface, 120–123.
L IANG , C.-K., L IN , T.-H., W ONG , B.-Y., L IU , C., AND C HEN ,H. H. 2008. Programmable aperture photography: multiplexed
light field acquisition. ACM Trans. Gr. 27, 3, 55:1–55:10.

Color Crosstalk Suppression(B.1)

Let σrg be the covariance between the R and G components, then2by λ0 λ1 = det(Σ) = σr2 σg2 − σrg, we obtain:2/σr2 σg2
L(x, y; d) = 1 − σrg(B.2)

Since NCC = σrg /σr σg ∈ [0, 1], the 2D version of the color
alignment measure L has a one-to-one correspondence to NCC.

ACM Transactions on Graphics, Vol. 27, No. 5, Article 134, Publication date: December 2008.

Extracting Depth and Matte using a Color-Filtered Aperture

Appendix C

•

134:9

Computing the Color Lines Model Error

Letting ci be the i-th color in SF (x, y; d), µ be the mean color, andv0 be a unit vector of the fitted line (the first principal eigenvector),
trigonometry gives the distance li of the point ci from the line as:
li2 = |ci − µ|2 − ((ci − µ)T v0 )2 .

(C.1)

The average of the first term is, by definition, the variance:N

1 X|ci − µ|2 = σr2 + σg2 + σb2 .
N i=1

(C.2)

For the second term, we have:N

1 X((ci − µ)T v0 )2 = v0T
N i=1

N

1 X
(ci − µ)(ci − µ)T
N i=1

!

v0

= v0T Σv0 = v0T (λ0 v0 ) = λ0 ,(C.3)
by the definitions of the covariance matrix Σ and the eigenvector v0 .
Therefore, the color lines model error can be computed as follows.
eF (x, y; d) = σr2 + σg2 + σb2 − λ0 .

(C.4)

This turns out to be similar to the color alignment measure of
Eq. (1), but we found it more effective for matting to use this unnormalized, direct error measure. Since estimation errors of background disparities are typically larger than those of foreground disparities, we discount eB (x, y; d) by scaling it by around 0.7-0.9.

ACM Transactions on Graphics, Vol. 27, No. 5, Article 134, Publication date: December 2008.