The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219

Iterative Refinement of Alpha Matte using Binary Classifier
Zixiang Lu1)

Tadahiro Fujimoto1)

1) Graduate School of Engineering, Iwate University
luzixiang (at) cg.cis.iwate-u.ac.jp
fujimoto (at) cis.iwate-u.ac.jp

Abstract
Image matting is a technique used to extract a foreground object from an input color image by estimating its opacity mask,
which is called an alpha matte. Many previous methods have estimated a poor-quality alpha matte when a foregroundobject and its background have similar colors. To overcome this problem, we treat an image matting problem as a binaryclassification problem to classify unknown pixels into foreground and background pixels. In this study, we propose alearning-based matting method based on binary classification. In our method, a binary alpha matte is first obtained bya binary classifier. The binary alpha matte is then iteratively refined to a high-quality alpha matte by repeat use of theclassifier. Although we used a support vector machine classifier and the closed-form matting method, an important meritof our method is that it provides a general mechanism to refine an alpha matte by combining various binary classifiers
and matting methods. The excellent performance of our method was revealed in several experiments, especially, for input
images having similar foreground and background colors as well as objects with small holes.

– 207 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219

1 Introduction
Image matting is a fundamental and useful technique forimage and video editing. From a foreground region separated from a background region on a given input color
image, image matting extracts the opacity mask, whichis called an alpha matte, as well as foreground and background colors. Image matting is based on the following
equation.
I i = αi F i + (1 − αi )B i .
(1)
The vectors I i , F i , and B i represent the colors of an image color, foreground, and background of a pixel pi in agiven input color image. Each color is a three-dimensional(3D) vector with red (R), green (G), and blue (B) color
values. The scalar αi ∈ [0, 1] is an opacity value, alpha.
If αi = 1 or 0, the pixel pi is a definite foreground pixel
or a definite background pixel respectively. Otherwise, thepixel pi is a mixed pixel. An accurate estimation of αi ,F i , and B i of mixed pixels pi is required to separate foreground and background regions fully. Eq.(1) consists ofthree linear equations with three known variables R, G,
B in I i and seven unknown variables αi and R, G, B in
F i and B i . This means that a matting problem is inherently ill-posed and does not have a unique solution tosatisfy Eq.(1). Thus, estimating a likely solution of αi ,F i , and B i for each pixel pi according to an input image
is essential.
Many matting methods have been proposed. Smith andBlinn [1] developed a blue screen matting method, thathas been widely used in film production. In this method,
foreground objects are photographed against a backgroundwith known colors to solve Eq.(1) easily by reducing unknown variables. Various methods have been developed to
extract foreground objects on a natural image without aknown background. These methods use either one or bothof the two kinds of interfaces to offer clues to solving amatting problem. These interfaces are known as trimapbased and scribble-based. A trimap-based interface uses
a trimap given by a user, and divides the entire image region into three subregions: a definite foreground, a definite
background , and an unknown region, as shown in Figure1(b). A definite foreground or background region has only
definite foreground or background pixels, colors of which
are used as clues. An unknown region has unknown pixels
to estimate. These are not only mixed pixels but also definite foreground or background pixels. In a scribble-based
interface, which was first proposed by Wang and Cohen[2], a user gives some foreground and background scribbles
on an input image. These scribbles contain a few definite foreground and background pixels, respectively. The
colors of these pixels are used to estimate alphas and theforeground or background colors of the remaining pixels.
In addition, a matting method generally uses one of thefollowing three approaches: sampling-based, propagationbased, and hybrid approaches [3]. Bayesian matting [4] isa well-known sampling-based method that is based on the
maximum posteriori estimation in a Bayesian framework.
Poisson matting [5] and closed-form matting (CF) [6] arepropagation-based methods. Poisson matting obtains analpha matte by solving a Poisson equation of alphas andimage color gradients. Closed-form matting uses a matting Laplacian matrix based on a color line assumption.
Most recent methods use hybrid approaches that combine
sampling-based and propagation-based approaches such as
robust color sampling matting [7], shared sampling matting [8], and large kernel matting [9]. In addition, somemethods use learning-based approaches, such as learning
based digital matting [10], support vector machine (SVM)based matting [11], and support vector regression (SVR)based matting [12]. In addition, additional excellent methods recently been developed, e.g., global sampling matting [13], KNN matting [14], local and nonlocal smoothness prior (LNSP) matting [15], color clustering matting(CCM) [16], and comprehensive sampling (CS) matting[17].
In this study, we propose a matting method founded ona learning-based approach. As described in Section 2.3
later, our method is completely different from the abovementioned learning based methods [10] [11] [12]. The key
idea of our method is the iterative refinement of an alphamatte. In our method, an alpha matte obtained by a basic
matting method we employ is refined iteratively using abinary classifier that intelligently classifies foreground andbackground pixels. The alpha matte gets better gradually as the iteration proceeds. We use the support vector
machine (SVM ) [18] as a binary classifier in a manner
different from the SVM-based matting [11], although we
can use another binary classifier. The SVM classifier is awell-known excellent binary classifier that reliably classifies foreground and background pixels even if their colors
are similar. In addition, we employ the closed-form matting (CF) method [6] as a basic matting method. Somemethods, such as those in [8], [12], [13] and [17], use the
CF method in post-processing to improve an obtained alpha matte. This usage of the CF method greatly inspired
us to invent the idea of our method, although we can employ another method as our basic matting method. The
iterative refinement mechanism proposed in this study is
not limited to the combination of the SVM and the CF
method. An important merit of our method is to provide
a general mechanism to improve an alpha matte by combining various binary classifiers and matting methods.

2 Related work
2.1 Closed-form matting
Levin et al. proposed the CF method [6]. Its main assumption is the color line model: foreground or background colors in a small window lie on a single line in RGB color
space. Based on this assumption, the alpha αi of a pixel
pi in a small window is represented as a 4D linear modelof αi ≈ aT I i + b using its color I i and constants a and b.
By minimizing the cost function J(α, a, b) defined using
the 4D linear model, a quadratic function of α, which is avector of the alphas αi of all pixels pi , is obtained.

– 208 –

J(α) = min J(α, a, b) = αT Lα.
a,b

(2)

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219

The matrix L is called matting Laplacian, which has been
widely used in many matting methods. In addition, its
elements are shown as follows.

function [19], and sigmoid. In our method, we use theGaussian radial basis function kernel.
(−

X

Lij =

(δij −

k|(i,j)∈wk

(Σk +

1
(1 + (Ii − µk )|wk |I3 )−1 (Ij − µk )))|wk |K(X i , X j ) = e,(3)

where δij is the Kronecker delta, µk and Σk are the meanand covariance matrix of the colors in window wk , |wk |is the number of pixels in wk , and I3 is a 3 × 3 identity
matrix.
To estimate an alpha matte to satisfy a constraint by
a user for example, a trimap, the CF method solves the
optimization problem.
α = argminα αT Lα + λ(α − β)T D s (α − β),(4)where λ is a large number, D s is a diagonal matrix whose
diagonal elements are 1 for constrained pixels and 0 forothers, and β is a vector that contains specified alphas forthe constrained pixels and 0 for others.
After we estimate alphas, reconstructing foreground andbackground colors is typically necessary. However, thisremains an ill-posed problem. Levin et al. proposed areconstruction method to solve this problem using the 4D
linear model, given as:minXX(αi Fic + (1 − αi )Bic − Iic )2i∈Ic+|αix |((Ficx )2 + (Bicx )2 )(5)+|αiy |((Ficy )2 + (Bicy )2 ),where Ficx , Ficy , Bicx , and Bicy are the x and y derivativesof F c and B c , and αix and αiy are the matte derivatives.
For a fixed α, the cost function (5) is quadratic and its
minimum may be found by solving a sparse set for linear
equations. In our method, we use the CF and reconstruction methods iteratively to refine an alpha matte.

2.2 Support vector machine
SVM [18] provides an excellent binary classifier based on
the structural risk minimization principle. An SVM binary classifier separates samples into two classes according
to their features. Its basic concept is to locate the optimal
separating hyperplane to separate samples in the feature
space so that the hyperplane has the maximal margin between the nearest samples in both classes.
The SVM classifier is first trained by means of a setof training data {(X i , yi )} of samples {si }, i = 1, . . . , M ,where X i ∈ Rn is a feature vector and yi ∈ {1, −1} is abinary class label. To use the “kernel trick” technique, thefeature vectors X i are mapped to a higher dimensional
space by a function φ. The function is used to define akernel function K(X i , X j ) = φ(X i )T φ(X j ), which is inturn used to define a classification function to realize theoptimal hyperplane in the space. Many kernel functions
have been proposed such as linear, polynomial, radial basis

kX i −X j k2
)
2σ 2

,

(6)

where σ is a kernel parameter. After training using thetraining data, we obtain a classification function that canclassify new samples using feature vectors given as a testing data. We used the LIBSVM [20] to develop our software.

2.3 Learning-based matting
Zheng et al. solved an alpha matting problem by treating
it as a semi-supervised learning task in machine learning[10]. They proposed local and global learning based approaches. In the local learning based approach, it is assumed that the alpha of a pixel is represented not only
by a linear combination of the alphas of its neighboring
pixels but also by a linear combination of its color components. Based on the assumption in a local region (a 7× 7 local patch), by using the ridge regression technique,a linear alpha-color local model is considered to estimate
the alphas of unknown pixels by a coefficient matrix obtained from image colors of all pixels and the alphas ofknown pixels. The linear model is extended to a nonlinear model by the kernel trick. Then, the local learning
based approach is extended to the global one by selecting for each unknown pixel its nearby known foregroundand background pixels by the shortest Euclidean distances
instead of its neighboring pixels.
Hosaka et al. proposed an optimization method to estimate an alpha matte by using the support vector machine (SVM) [11]. They defined a cost function to minimize on the basis of a Markov random field (MRF). The
cost function consists of a matting term, a smoothing
term, and a discrimination term. The matting term isdefined for the fidelity to the matting equation on single
pixels, and the smoothing term is defined for the local
smoothness between neighboring pixels. The discrimination term is defined by using the SVM to discriminate between foreground and background pixels. The SVM uses
a 15-dimensional feature vector consisting of the RGB colors of each pixel and its four nearest neighbors. The costfunction is minimized by the belief propagation to estimate
the alphas of unknown pixels and by a sampling method
to estimate their foreground and background colors.
Zhang et al. solved an alpha matting problem as a supervised regression problem by using the support vector
regression (SVR) [12]. In their method, first, an unknown
region is segmented into small circular or fan-shaped pieces
by a predefined pixel length r (they set r = 30 pixels), andthe pieces are given an order to process according to theirdistances to definite foreground and background regions.
In the process of each piece, first, candidate training samples are selected from definite foreground and backgroundpixels and previously estimated pixels around the piece.
Then, by using the similarity distance between two pixels defined by their pixel coordinates and feature vectors,
m most similar and m most dissimilar pixels (they set– 209 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219

m = 3) among the candidate training samples are selected
for every pixel in the piece as training samples for SVR toestimate the alphas of the pixels in the piece.
Our method proposed in this study is completely different from the above learning-based methods, although our
method also takes a learning-based approach. The distinctive feature of our method is to refine an alpha matte
iteratively by using a binary classifier which classifies foreground and background colors in order to finally obtain ahigh-quality matte. The binary classifier is first used toobtain an initial alpha matte in which the alpha of each
unknown pixel is either 0 or 1. Then, the classifier is usedin each iteration step to determine the confidence values of
the current estimated alphas of the unknown pixels. The
confidence values are used for the CF method, which issummarized in Section 2.1, to estimate better alphas. In
this study, we use the SVM, which is summarized in Section 2.2, as a binary classifier.

3 Iterative refinement of alpha matte
3.1 Outline
Examples obtained by our method are shown in Figure 1
in which the binary classification (c) and final alpha matte(d) are obtained from the input image (a) and trimap (b).
Our iterative refinement method estimates alphas of unknown pixels in an unknown region of a trimap according
to the following three stages.
(1) Training of binary classifier
A binary classifier to classify unknown pixels into foreground and background pixels is trained by a set of training data obtained from an input color image and its
trimap. We use SVM as a binary classifier, and otherclassifiers also can be used.
(2) Initial alpha estimation
By using the trained SVM classifier, initial alphas of unknown pixels are estimated.
Binary classification: All unknown pixels are classified
into foreground and background pixels by the SVM classifier.
Confidence evaluation: Confidence values of the unknown pixels for the classification result are evaluated.
Alpha estimation: Using the confidence values, initialalphas of the unknown pixels are estimated by Eq.(12).
(3) Iterative refinement of alpha estimation
By starting with the initial alphas, alphas of the unknown pixels are refined by iterating the following threesteps until the refinement converges.
Foreground and background estimation: Foregroundand background colors of the unknown pixels are estimatedfrom the latest estimated alphas.
Confidence evaluation: Confidence values of the estimated foreground and background colors for the unknown
pixels are evaluated using the SVM classifier.
Alpha estimation: Using the confidence values, new alphas of the unknown pixels are estimated by Eq.(16).
The details are described as follows.

3.2 Training of binary classifier
We use an SVM classifier with a Gaussian kernel [18] toclassify unknown pixels into foreground and backgroundpixels. The classifier must be given appropriate feature
vectors of pixels as training data in advance and testingdata in application. We use the feature vectors described
below and train the classifier as follows.
3.2.1 Feature vector
Many traditional segmentation methods distinguish foreground from background pixels on an input color image
by using an RGB-color vector of each pixel as its feature
vector. The RGB-color vector is a 3D vector. However,in some cases, such a feature vector does not contain adequate information for a high-quality segmentation result.
In particular, distinguishing foreground and backgroundpixels that have similar colors is very difficult. Therefore,
several methods exist to enhance the information related
to a feature vector. For example, the SVM-based matting
method [11] uses not only a pixel’s RGB-color vector but
also RGB color vectors of its four neighboring pixels inorder to construct a 15-dimensional feature vector.
For our SVM classifier, we enhance the information related to each pixel by using its position, RGB color, and
RGB color gradient to construct its feature vector. For apixel pn , we define an 11-dimensional feature vector X n= {un , vn , Rn , Gn , Bn , dRxn , dGxn , dBxn , dRyn , dGyn ,dByn }, where un and vn are pixel coordinates; Rn , Gn ,and Bn are color values; and dRxn , dGxn , dBxn anddRyn , dGyn , dByn are the gradients of the color values
in x and y directions, respectively.
3.2.2 Training of binary classifier using trimap
Our matting method uses a trimap to obtain an alphamatte from an input color image. A trimap divides the entire image region Ω into three subregions given by a user:a definite foreground region Ωf , a definite background region Ωb , and an unknown region Ωu , as shown in Figure1(b). A matting problem is solved in order to estimate aforeground color F i , a background color B i , and an alpha αi for each unknown pixel pi ∈ Ωu in order to satisfy
Eq.(1). Known pixels in Ωf and Ωb are used as clues forthe solution. A pixel pj ∈ Ωf has a foreground color F jthat is the same as its image color I j and its alpha αj is
1. A pixel pk ∈ Ωb has a background color B k that is thesame as its image color I k and its alpha αk is 0.
Our SVM classifier uses an 11-dimensional feature vector X n for a pixel pn ∈ Ω to determine whether it is aforeground or background pixel. To train the classifier, a
simple idea is to use feature vectors of all known pixels
in Ωf and Ωb as training data. However, in most natural images, nearby pixels tend to have similar foregroundand background colors and alphas. Thus, we select a subset Ω̃f ⊂ Ωf as foreground training data. For every pixel
pi ∈ Ωu , we define a selection window wi whose center is pi .
It has a small square shape (e.g., its size is 20 × 20 pixels).
If a definite foreground pixel pj ∈ Ωf is in wi , it is selected
as a foreground training pixel, that is, pj ∈ Ω̃f . In the– 210 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219(a) Input image(b) Trimap(d) Final alpha matte(c) Binary classification

Figure 1: Examples obtained by our method. In (b), definite foreground regions are white, definite background regions are black,and unknown regions are gray. In (c), (d), intensities mean alphas in [0, 1].

background, α̂i0 is set to 0. In addition, α̂j0 of all pixels
pj ∈ Ω̃f are set to 1, and α̂k0 of all pixels pk ∈ Ω̃b are set
to 0.
A confidence value con0i for each classified unknown
pixel pi is then evaluated as:
con0i = e(di −1) · e(gi −1) .
(a)

(7)

(b)

Figure 2: Training data selected on a trimap. (a) shows atrimap. (b) shows the regions of foreground andbackground training data Ω̃f and Ω̃b along the unknown region Ωu .

same manner, we select a subset Ω̃b ⊂ Ωb as backgroundtraining data. An example of selected subsets is shown in
Figure 2. In our binary classification, for a pixel pn ∈ Ω̃ =
Ω̃f ∪ Ω̃b ∪ Ωu , we give a label yn = 1 if pn is a foregroundpixel and yn = −1 if pn is a background pixel. Thus, we
use foreground training data {(X j , yj )} = {(X j , 1)} of
pixels pj ∈ Ω̃f and background training data {(X k , yk )}= {(X k , −1)} of pixels pk ∈ Ω̃b .
After training is completed, the SVM classifier obtains
an SVM function D that returns a decision value D(X n )for a feature vector X n of a pixel pn ∈ Ω̃. The function istrained to return a value D(X n ) > 0 for a foreground pixel
and D(X n ) < 0 for a background pixel. When |D(X n )| islarge, its classification has high reliability. If D(X n ) > 1or D(X n ) 6 −1, pn is a foreground or background pixel,
respectively, having high reliability. If −1 < D(X n ) < 1,
its classification reliability is low. Examples of classification are shown in Figure 1(c).

3.3 Initial alpha estimation
By means of the trained SVM classifier, initial alphas αi0of unknown pixels pi ∈ Ωu can be estimated.
All unknown pixels pi are first classified into foregroundand background pixels by the signs of the decision values
D(X i ). If a pixel pi is classified to the foreground, its
temporary alpha α̂i0 is set to 1. If pi is classified to the

The value di is calculated using the decision value Di =D(X i ) as
1
Di > avgfD /avg
0 6 Di < avgf
i
f
di =
,
(8)
Di /avgb avgb < Di < 01
Di 6 avgb
avgf
avgb

=

1
|Ω̃f |

P=1|Ω̃b |P

pj ∈Ω̃f D(X j ),(9)pk ∈Ω̃b D(X k ).

(10)

The values |Ω̃f | and |Ω̃b | are the numbers of pixels in Ω̃fand Ω̃b , respectively. The value gi is calculated using K
nearest neighbor pixels in Ω̃f ∪ Ω̃b to the pixel pi on thefeature vector space.
(
Kf /K Di > 0
gi =
.
(11)
Kb /K Di < 0
Among the K nearest neighbor pixels, Kf is the numberof pixels pj ∈ Ω̃f , and Kb is the number of pixels pk ∈ Ω̃b .
0Finally, using the temporary alphas α̂nof pixels pn ∈ Ω̃0and the confidence values coni of unknown pixels pi ∈ Ωu ,0
initial alphas αnare estimated by the CF method. The CF0
method transforms binary temporary alphas α̂n∈ {0, 1}0
into real initial alphas αn ∈ [0, 1] as a solution for minimizing the following optimization function.
α0 = argminα αT Lα + λ(α − α̂0 )T Q
(α − α̂0 ) + γ(α − α̂0 )T P 0 (α − α̂0 ).

(12)

Let N be the total number of pixels pn ∈ Ω̃. The vec0
tor α0 has N initial alphas αn, and the vector α̂0 has N– 211 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219

0
temporary alphas α̂n, n = 1, . . . , N . The N × N matting
Laplacian matrix L is obtained from the input color image
given in Eq.(3). In the N × N diagonal matrix Q, diagonal elements qnn are 1 for pn ∈ Ω̃f ∪ Ω̃b , and 0 for others.
In the N × N diagonal matrix P 0 , diagonal elements p0nn
are con0n for pn ∈ Ωu , and 0 for others. The parameters λand γ are used for weighting and normalizing; usually, λis large (e.g., 100), and γ is small (e.g., 0.1). The matrices00
Q and P 0 constrain αnto remain α̂n. The initial alphas0αnof pn ∈ Ω̃f ∪ Ω̃b are strongly constrained by Q and λ.
0
The matrix P 0 constrains αnof pn ∈ Ωu strongly if con0n
0
is large. As a solution to Eq.(12) for pn ∈ Ω̃f ∪ Ω̃b , αncan0be neither 0 nor 1. However α̂n is either 0 or 1, and should
remain as such. Thus, we do not accept the solution and000
instead set αnto α̂n. We use the solution of αnonly forpn ∈ Ωu .

3.4 Iterative refinement of alpha estimation
The alpha matte of the initial alphas αi0 often possesses
poor quality. Therefore, by starting with αi0 , our matting method refines the alphas αi of unknown pixels pi
iteratively by using the SVM classifier again until the refinement converges.
Let αit be the estimated alpha of an unknown pixel pi
after t iteration steps, t = 1, 2, . . . , tmax . In the t-th
iteration step, foreground and background colors F t−1
and
i
B t−1
of unknown pixels pi are first estimated from theilatest alphas αit−1 and the image colors I i . We use thereconstruction method proposed by the CF method [6] forthis foreground and background color estimation as Eq.(5),although other methods can be used.
Next, we evaluate the confidence of the alphas αit−1 byevaluating the confidence of the estimated colors F t−1
and
i
B t−1
instead. We construct the feature vectors of theseicolors and classify them using the SVM classifier. Ideally,
these colors should be classified to foreground or background colors with high reliability. We consider that reliable foreground and background colors guarantee a reliablealpha. We calculate the confidence value conti for each unknown pixel pi asconti = confit · conbti ,(13)

where confit and conbti are the sub-confidence values for the colors F t−1
and B t−1.
These are obiitained using the decision values Dfit−1 = D(X(F t−1))
it−1
and Dbt−1
=
D(X(B
))
returnedbytheSVMclassiiifier, where X(F t−1) and X(B t−1) are the feature vectorsiiof F t−1
and B t−1which include positions, RGB colorsiiand RGB color gradients.

confit =1

1−cos(πDfit−1 )2

conbti =

0



1

1−cos(πDbit−1 )
2




0

Dfit−1 > 1
0 < Dfit−1 < 1 ,Dfit−1 6 0

(14)

Dbt−1
6 −1
i
−1 < Dbt−1<0it−1
Dbi > 0(15)

t−1
Finally, we use the latest alphas αnof pixels pn ∈ Ω̃tand confidence values coni of unknown pixels pi ∈ Ωu totestimate new alphas αnby the CF method.

αt = argminα αT Lα + λ(α − αt−1 )T Q
(α − αt−1 ) + γ(α − αt−1 )T P t (α − αt−1 ).

(16)

The matrices L and Q and the parameters λ and γ aret
the same as in Eq.(12). The vector αt has N alphas αn
,
t−1
t−1
and the vector α
has N alphas αn , n = 1, . . . , N .
In the N × N diagonal matrix P t , diagonal elements ptnn
are contn for pn ∈ Ωu , and 0 for others. The matrix P ttt−1
constrains αnto remain αnby contn for pn ∈ Ωu . Thistconstraint is stronger as conn becomes larger. In the samemanner as described in Section 3.3, we use the solution oftt
Eq.(16) of αnonly for pn ∈ Ωu , and retain αn
as 0 or 1 forpn ∈ Ω̃f ∪ Ω̃b .
As the iteration proceeds, the quality of the estimatedalpha matte improves. The iteration stops when the difference between αt−1 and αt is smaller than the predefined
threshold.

4 Experimental results
The algorithm of this study was implemented using MATLAB and executed with an Intel i7 3.4GHz CPU and 16GB
memory. In addition, an SVM library, LIBSVM, was used,
and the parameters of the svmtrain function were given thedefault settings introduced by [18].
Figure 1(c), Figure 3(c) and Figure 4(c) show examplesof the binary classification by the SVM. These examples
reveal the fundamental excellent ability of the SVM. All
unknown pixels are classified into either foreground pixels
or background pixels reasonably. These binary classification images seem to be rough approximations of accuratealpha mattes.
Figure 3 and Figure 4 show iterative refinement processes of alpha mattes obtained by our method. The input
images, “Troll” and “GT25”, are available on a well-known
matting evaluation website [21]. In each figure, the alphamatte is gradually refined as the iteration proceeds, andthe refinement converges. These results show that the iterative refinement mechanism based on the SVM in our
method works well.
In the following, we compare our method with several
well-known methods: CF matting [6], learning based digital matting (LB) [10], KNN matting [14], SVR matting[12], LNSP matting [15], color clustering matting (CCM)[16], and comprehensive sampling (CS) matting [17]. In
order to evaluate resulting alpha mattes, we used fourtypes of errors: Sum of Absolute Differences (SAD), Mean
Squared Error (MSE), Gradient error (Grad.), and Connectivity error (Con.), which are discussed in [22] and usedon the matting website [21]. Among these methods, theSVR, LNSP, CCM, and CS are current high-ranking methods on the matting website and on average provide highquality results for the four error types. The LB and KNN
are also well-known excellent methods, and their source
codes are available.

– 212 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219

it generates poor-quality alpha mattes for some input images.

(a) Input(b) Trimap(c) Iteration: 0(d) Iteration: 1(e) Iteration: 2(f) Iteration: 4(g) Iteration: 8(h) Iteration: 16(i) Iteration: 20

Figures 8, 9, 10 and 11 show the resulting alpha mattes
obtained by applying the CF, LB, and KNN methods andour method to several input images selected from the 27
images. The ground truth images in the figures are provided in the matting website [21]. Tables 1, 2 and 3 show
the SAD, MSE, and Grad. values of the alpha mattes respectively. Our method achieves the smallest error values
in all the three error types for the 8 input images used in
Figures 8, 9 and 10. This means that, from the viewpointof error values, our method generates the highest-quality
alpha mattes for these input images. On the other hand,
our method provides large error values for the 3 input images used in Figure 11. This means that our method generates poor-quality alpha mattes.

Figure 3: Iterative refinement of “Troll”. (c) is the classification result and (d)–(i) are the refinement results.

(a) Input(b) Trimap(d) Iteration: 1(c) Iteration: 0(e) Iteration: 2

Figure 4: Iterative refinement of “GT25”. (c) is the classification result and (d)–(e) are the refinement results.

The matting website [21] provides useful benchmark input images, including “Troll” and “GT25” in Figures 3
and 4. Among these images, we used the images in the“GT dataset” to compare our method with the CF, LB,
and KNN methods, whose source codes are available. Figures 5, 6 and 7 show the graphs for the comparisons of
the SAD, MSE, and Grad. values respectively obtainedby applying the four methods to the 27 input images in
the GT dataset. In each graph, the horizontal axis indicates the index numbers of the images, and the verticalaxis indicates error values. For each input image, the error values are normalized so that the error value by the
CF method is one. Judging from the graphs, the CF and
LB methods tend to provide similar error values in all thethree error types. Compared to the CF and LB methods, the KNN method and our method tend to have largechanges in error values. The KNN method tends to provide larger error values than the other methods while itprovides small error values for some input images. As awhole, our method tends to provide the smallest error values among these methods. This means that our method
has an ability to generate high-quality alpha mattes while

The “GT04” image in Figure 8 has a part with similar foreground and background colors given by green hairand green grass. The KNN method and our method successfully generate visually reasonable alpha mattes on thepart, that is, the region including the hair of the two dollsin the middle of the zoomed images, while the CF and LB
methods fail. The “GT11” image in Figure 9 also has apart with similar foreground and background colors in thezoomed region. Judging from the error values, in Figure
9 as well as Figure 8, our method achieves the best result
although it is difficult to distinguish the results of the four
methods visually. Figure 10 shows other cases in which theresults of our method are better than those of the othermethods from the viewpoint of error values. In particular, we find visually that our method works well for the“GT02” and “GT26” images having foreground objects
with many small holes while the CF and LB methods failto treat them. We consider that, in our method, the SVM
succeeded in classifying foreground and background colors correctly even for the cases in which there were manysmall holes as well as the cases in which foreground andbackground colors were similar.
On the other hand, Figure 11 shows some cases in which
our method generates poor-quality results from the viewpoint of error values although it is difficult to visually
distinguish their qualities from those by the other methods. In order to easily compare the qualities achievedby the four methods, we show two kinds of error maps
for the “GT08” image in Figure 12. The top error maps
show, for each pixel, the difference difx between the alpha αx obtained by the method x, which is CF, LB,
KNN, or Ours, and the alpha αgt of the ground truth:
difx = αx − αgt . The red pixels mean difx > 0, the blue
pixels mean difx < 0, and the black pixels mean difx = 0.
A brighter red or blue color means a larger absolutedifference. The bottom error maps show the difference
dif gx between the gradients of the alphas αx and αgt :dif gx = gradαx − gradαgt . The red, blue, and black pixels have similar meanings to the top error maps. We consider that, in these cases, it was difficult for the SVM toappropriately classify foreground objects (e.g., long andthin hair, and fine fur) and backgrounds because of thelack of training data to adequately learn their properties.

– 213 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219

3
CF
LB
KNN
Ours

2.5

2

1.5

1

0.5

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

Figure 5: Comparison of SAD errors for GT dataset images. The horizontal axis indicates the index numbers of the images. The
vertical axis indicates SADx /SADCF , x ∈ {CF, LB, KN N, Ours}.

Table 1: SAD errors for GT dataset images.
CF
LB
KNN
Ours

GT01
1131.21130.1
710.1
633.5

GT02
4308.24273.71536.81197.7GT04
12095.5
11513.512490.4
9661.8GT05
1262.51217.61397.7836.4

GT072150.72122.92584.31603.0GT08
11453.6
11411.2
17922.2
15212.8GT113630.53546.84716.52986.3GT17
1782.31787.95429.34476.3GT18
2188.12215.31899.61324.1GT23
1608.71570.72856.12593.8GT26
18056.2
17881.2
13971.4
11955.5

11
CF
LB
KNN
Ours

10
9
8
7
6
5
4
3
2
1
0
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

Figure 6: Comparison of MSE errors for GT dataset images. The horizontal axis indicates the index numbers of the images. The
vertical axis indicates M SEx /M SECF , x ∈ {CF, LB, KN N, Ours}.

Table 2: MSE errors for GT dataset images.
CF
LB
KNN
Ours

GT01
0.000540.000530.000250.00017GT02
0.00710.007
0.00084
0.00046GT04
0.00910.00850.00520.0032GT05
0.00120.00120.00098
0.00027GT07
0.00087
0.000850.00110.00031GT08
0.0051
0.005
0.011
0.0077– 214 –

GT11
0.00160.00150.00240.0011GT17
0.00040.000410.00430.0025GT18
0.001
0.00110.000830.00038GT23
0.000250.000250.00084
0.00047GT26
0.01740.01720.00980.0084

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219

8
CF
LB
KNN
Ours

7

6

5

4

3

2

1

0
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

Figure 7: Comparison of Gradient errors for GT dataset images. The horizontal axis indicates the index numbers of the images.
The vertical axis indicates Gradx /GradCF , x ∈ {CF, LB, KN N, Ours}.

Table 3: Gradient errors for GT dataset images.
CF
LB
KNN
Ours

GT01
1776.6
1754.7
521.9
476.7

GT02
21005
20733
2287.11476.8GT04
9610.88925.3
6686
4680

GT05
2554.62438.12795.5657.5

GT07
1868.81834.32010.7756.7

GT085039.94945.611880
5483.1

GT11
5046
4744
4281.63401.9

GT17
764.1
764.2
5944.92131.2

GT18
2711.12732.52304.31048.3

GT23
671.6
641.5
3139.31019.5

GT26
48368
47720
17682
16564

(a) Input and trimap(b) CF(a) Input and trimap(b) CF(c) LB(d) KNN(c) LB(d) KNN(e) Ours(f) Ground truth(e) Ours(f) Ground truth

Figure 8: Alpha mattes for “GT04” image.

Figure 9: Alpha mattes for “GT11” image.

– 215 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219(a) Input(b) Trimap(c) CF(d) LB(e) KNN(f) Ours(g) Ground Truth

Figure 10: Alpha mattes for GT dataset images (from the top, GT01, GT02, GT05, GT07, GT18, and GT26) with small errors
by our method.

(a) Input(b) Trimap(c) CF(d) LB(e) KNN(f) Ours(g) Ground Truth

Figure 11: Alpha mattes for GT dataset images (from the top, GT08, GT17, and GT23) with large errors by our method.

– 216 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219(a) CF(b) LB(c) KNN(d) Ours

Figure 12: Error maps for “GT08” image. The top maps show, for each pixel, the difference between alphas by the respective
methods and the ground truth. The bottom maps show the difference between the gradients of alphas.

Figure 13 and Table 4 show the resulting alpha mattes and error values obtained by applying not only the
CF, LB, and KNN methods and our method but also the
SVR, LNSP, CCM, and CS methods to the “Troll” image in Figure 3. The alpha mattes and error values by
the methods except for our method are available on thematting website [21]. On the website, the “Troll” image isprovided as a typical image with “Strongly Transparent”
objects. In addition, the hair part in front of the bridge
is so difficult to treat to obtain its accurate alpha mattebecause of not only the fine detailed hair but also the similar foreground and background colors. As shown in Figure
13, no method achieves a high-quality result. The poor
qualities of the resulting alpha mattes depend on methods. The results of some methods have artifacts on thehair part caused by the outlines of the bridge. The results of some methods including ours have larger alphas
uniformly on the hair part. However, judging from theerror values in Table 4, our method competes even withthe current high-ranking methods (SVR, LNSP, CCM, andCS). In the same way as Figure 11, we consider that thepoor-quality result by our method was caused mainly by
the lack of training data. Figure 14 shows an alpha matteobtained by applying our method to the “Troll” image byusing a trimap different from the trimap used in Figure 13.
While the trimap in Figure 13 is provided in the matting
website [21], the trimap in Figure 14 was created by us soas to give additional definite background regions inside theunknown region of the trimap in Figure 13. By comparing
Figure 14(b) with Figure 13(i), we find that an improved
alpha matte was obtained by the new trimap that provided
more clues, which were used as more useful training data
for the SVM.

Table 4: Errors for “Troll” image.
CF
LB
KNN
SVR
LNSP
CCM
CS
Ours

SAD
12.7
16.0
16.2
18.7
12.2
13.8
11.2
11.9

MSE
0.5
0.8
0.8
1.1
0.5
0.5
0.4
0.5

Grad.
0.3
0.3
0.3
0.3
0.2
0.2
0.2
0.2

Con.
0.8
0.8
3.4
1.2
0.8
0.7
1.0
0.7

(a) Trimap(b) CF(c) LB(d) KNN(e) SVR(f) LNSP(g) CCM(h) CS(i) Ours

Figure 13: Alpha mattes for “Troll” image.

– 217 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219[4] Yung-Yu Chuang, Brian Curless, David H. Salesin,
and Richard Szeliski. A bayesian approach to digitalmatting. Proceedings of IEEE CVPR 2001, 2:264–
271, 2001.
(a) Trimap(b) Alpha matte

Figure 14: An alpha matte for “Troll” image by our method
using a different trimap.

From the above experiments, we find that our method
generates high-quality alpha mattes particularly for input
images having similar foreground and background colors
and objects with small holes. For several input images, ourmethod performs more effectively than other methods andcompetes even with high-ranking methods. Our method
relies on the ability of a binary classifier. Thus, the qualityof a resulting alpha matte depends on the performanceof the classifier. When the learning of the classifier bytraining data is not enough, the quality of the alpha mattetends to become poor. The effective way of the learningof the classifier needs to be explored in future.

5 Conclusion
In this study, we proposed a learning-based matting
method. In our method, an image matting problem is firsttreated as a binary classification problem, and a binaryalpha matte is obtained through a binary classifier. The
binary alpha matte is then iteratively refined to a highquality alpha matte by repeat use of the classifier. Our
method was performed better than other methods in experiments for several input images while it did not work
well for some input images.
Although we used an SVM classifier and the CF method,
an important merit of our method is that it provides a
general mechanism to refine an alpha matte by combining
various binary classifiers and matting methods. This represents a promising ability to achieve better results thanthose currently obtained, and we plan to investigate othercombinations. Besides, an effective way of the learning ofa classifier to classify foreground and background colors
correctly needs to be explored. In addition, we plan to apply our method to high-resolution images efficiently. One
idea is to use an effective trimap segmentation method.

[5] Jian Sun, Jiaya Jia, Chi-Keung Tangand, and HeungYeung Shum. Poisson matting. ACM Trans. Graph.,
23:315–321, 2004.
[6] A. Levin, D. Lischinski, and Y. Weiss. A closed form
solution to natural image matting. Pattern Analysis and Machine Intelligence, IEEE Transactions,
30:228–242, 2008.
[7] Jue Wang and Michael F. Cohen. Optimized color
sampling for robust matting. Computer Vision and
Pattern Recognition, pages 1–8, 2007.
[8] Eduardo S. L. Gastal and Manuel M. Oliveira. Sharedsampling for real-time alpha matting. Computer
Graphics Forum, 29:575–584, 2010.
[9] Kaiming He, Jian Sun, and Xiaoou Tang. Fast matting using large kernel matting laplacian matrices.
Computer Vision and Pattern Recognition (CVPR),
pages 2165 – 2172, 2010.
[10] Ye Zheng and Kambhamettu C. Learning based digital matting. Computer Vision, 2009 IEEE 12th International Conference, pages 889–896, 2009.
[11] Tadaaki Hosaka, Takumi Kobayashi, and Nobuyuki
Otsu. Image matting based on local color discrimination by svm. Pattern Recognition Letters, 30:1253 –
1263, 2009.
[12] Z. Zhang, Q. Zhu, and Y. Xie. Learning based alphamatting using support vector regression. ICIP, 2012.
[13] Kaiming He, Christoph Rhemann, Carsten Rother,
Xiaoou Tang, and Jian Sun. A global sampling
method for alpha matting. Computer Vision and Pattern Recognition (CVPR), pages 2049 – 2056, 2011.
[14] Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. Knn
matting. IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pages 869–876, 2012.
[15] X. Chen, D. Zou, and P. Tan. Image matting withlocal and nonlocal smooth priors. CVPR, 2013.

References[1] Alvy Ray Smith and James F. Blinn. Blue screen
matting. Association for Computing Machinery, Inc.,
pages 259–269, 1996.
[2] J. Wang and M. Cohen. An iterative optimization
approach for unified image segmentation and matting.
Proc. 10th IEEE Int’l Conf. Computer Vision, 2005.
[3] Jue Wang and Michael F. Cohen. Image and video
matting: A survey. Found. Trends. Comput. Graph.
Vis., 3:97–175, 2007.

[16] Y. Shi, O.C. Au, J. Pang, K. Tang, W. Sun, H. Zhang,
W. Zhu, and L. Jia. Color clustering matting. ICME,
2013.
[17] E.Shahrian, D.Rajan, B.Price, and S.Cohen. Improving image matting using comprehensive sampling sets.
CVPR, 2013.
[18] Chih wei Hsu, Chih chung Chang, and Chih jen Lin. A
practical guide to support vector classification. IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2010.

– 218 –

The Journal of the Society for Art and Science, Vol.14, No.5, pp. 207–219[19] S.Keerthi and C. j. Lin. Asymptotic behaviors of support vector machines with gaussian kernel. NeuralComputation, 15:1667–1689, 2003.
[20] C. Chang and C. Lin.
Libsvm – alibraryforsupportvectormachines.
http://www.csie.ntu.edu.tw/ cjlin/libsvm/.
[21] alphamatting
evaluation
website.
http://www.alphamatting.com/index.html.
[22] Christoph Rhemann, Carsten Rother, Jue Wang,
Margrit Gelautz, Pushmeet Kohli, and Pamela Rott.
A perceptually motivated online benchmark for image matting. Conference on Computer Vision and
Pattern Recognition (CVPR), 2009.

Zixiang Lu

Zixiang Lu is a master course student in Design and Media
Technology at the Graduate School of Engineering, Iwate
University from 2014. He received his bachelor degree from
North West Agriculture and Forestry University in 2013.
His research interest includes image processing, computer
vision and data mining.
Tadahiro Fujimoto

Tadahiro Fujimoto is currently an associate professor in
the Department of Computer and Information Sciences at
Iwate University. His research interests include computer
graphics and computer vision. He received a BE in electrical engineering, and an ME and Ph.D. in computer sciencefrom Keio University in 1990, 1992, and 2000, respectively.
He worked at Mitsubishi Research Institute from 1992 to
1995. He was a research associate in the Department ofComputer and Information Sciences at Iwate University
from 1999 to 2002, and a lecturer from 2002 to 2005. He
is a member of SAS Japan, IEICE Japan, IPS of Japan,
IEEE, and ACM.

– 219 –