# Benefits of Using Decorrelated Color Information for  Face Segmentation/Tracking

Authors: Mircea C. Ionita; Peter Corcoran
Journal: Advances in Optical Technologies
 (2008)

---

## Abstract

We analyze in this paper the benefits that can be
derived from employing color image alignment techniques in the
context of face segmentation or tracking based on texture (defined
as the patch of intensities) template matching. By making full
use of the decorrelated color information, improvements on the
accuracy of the segmentation are demonstrated. This is intended
to enhance the face segmentation algorithm by increasing its
robustness to differences in images caused by various image
acquisition devices or settings or by variations in the ambient
illumination conditions.


---

## Body

## 1. Introduction

The use of color information becomes increasingly
important in nowadays image processing applications as inexpensive color image
acquisition devices become easily available. Color image processing permits a
more extensive image representation, which expectedly leads to better results.We deal in this paper with the specific case of face
segmentation which employs face modeling techniques. This can also be viewed in
the more general context of deformable template matching, using for this
purpose a statistical model of shape variations. Extensive work has been
carried out in the area of face modeling and face segmentation using
statistical models [1–5]. These techniques have initially been developed for
gray level images. Extensions have later been proposed for color images
[6, 7]. Some advantages of using
the color extension have been demonstrated, but mostly for working in a
controlled image acquisition environment. Processing color information can thus
be challenging, especially when designing more general applications that are supposed
to work within unconstrained image acquisition conditions. We demonstrate in
this paper some positive results when using the decorrelated color information
for applications which include face segmentation and face tracking, intended to
work under no predefined constrains.The outline of this paper is as follows. In Section2, we briefly describe several decorrelated color spaces in terms of their
transforms from the common RGB color space; we
also include a comparison between these color spaces in terms of how well they
are able to decorrelate image channels on a series of test images. In Section
3, a face segmentation method is described, based on a statistical shape
model and a fixed face texture template. The limitations of the application
described in Section 3 are addressed in Section 4, introducing some texture
alignment and color transfer techniques in order to adapt the texture template
to the color distribution of the current image. These operations are
facilitated by converting the texture data to one of the decorrelated color
spaces presented in Section 2. In Section 5, we show our experiments performed
on a general face image database; the database is built as a mixture of images,
gathered mostly from various standard face image and video databases; a set of
comparative results is provided in Section 5. Finally, in Section 6 we draw
the conclusions of our work.


## 2. Image Decorrelation with Respect to Color Information

Colorwise image
decorrelation is useful for applying color image processing operations
independently on each image channel.


## 2.1. Karhunen-Loève Transform

TheKarhunen-Loève
transform (KLT) is optimal in terms of energy compaction and mean-squared
error minimization for a truncated representation. By applying KLT to a color
image, it creates image basis vectors which are orthogonal, and it thus
achieves complete decorrelation of the image channels [8–12] as follows:(1)CRGB=A⋅(I−μI),where I=[IR,IG,IB]T contains the
image color signals and μI=[μR,μG,μB]T, with μx=E[x]; E is the
mathematical expectation. CRGB is the
covariance matrix of the of the image color signals as follows:(2)CRGB=[CRRCRGCRBCGRCGGCGBCBRCBGCBB]with Cxy=E[(x−E[x])(y−E[y])], x,y∈{R,G,B}. A is the
transformation matrix formed by the eigenvectors of the covariance matrix CRGB:(3)A=[k1k2k3]T=[k11k21k31k12k22k32k13k23k33]T.Yet, KLT is data
dependant, meaning that it requires the recalculation of the transformation
matrixA for each set of
data (e.g., each new image).


## 2.2.I1I2I3 Color Space

An interesting
color space isI1I2I3, proposed by Ohta et al. [13], which realizes a
statistical minimization of the interchannel correlations (decorrelation of the RGB components) for
natural images. The conversion from RGB to I1I2I3 is given by the
simple linear transformation in (4) as follows:(4)[I1I2I3]=[1/31/31/31/20−1/2−1/41/2−1/4][RGB].I1 stands as the
achromatic (intensity) component, while I2 and I3 are the
chromatic components. We remark that the simple numeric transformation from RGB to I1I2I3 enables simple
and efficient transformation of datasets between these two color spaces.I1I2I3 was designed as
an approximation for the KLT of the RGB data to be used
for region segmentation on color images. As the transformation to I1I2I3 represents a
good approximation of the KLT for a large set
of natural images, the resulting color channels are almost completely
decorrelated.In the previous work of Ohta et al., the
discriminating power of109 linear
combinations of R, G, and B was tested on
eight different color scenes. The selected linear combinations were gathered
such that they could successfully be used for segmenting important (large area)
regions of an image, based on a histogram threshold. It was found that 82 of the linear
combinations had all positive weights, corresponding mainly to an intensity
component which is best approximated by I1; another 22 showed opposite
signs for the weights of R and B, representing the difference between the R and B components
which are best approximated by I2; finally, the remaining 4 linear
combinations could be approximated by I3. Thus it was shown that the I1, I2, and I3 components in
(4) are effective for discriminating between different regions and that they
are significant in this order [13]. We can further conclude, based on the above figures,
that the percentage of color features which are well discriminated on the
first, second, and third channels is around 76.15 %, 20.18 %, and 3.67 %,
respectively.I1I2I3 is also found
in [14] to perform
better as compared to other color space implementations like YIQ, CIELAB, and UVW for segmentation
of color images based on Markov random field (MRF) processing. In
[15], the I1I2I3 color space was
used for color image segmentation based on an MRF model and
simulated annealing due to its effectiveness in terms of the quality of the
segmentation and the reduced complexity of the transformation.


## 2.3.lαβ Color Space

Assuming that
the human visual system is ideal for processing natural scenes, Ruderman et al.
[16] developed the lαβ color space,
which also minimizes the correlation between channels for natural images. The
conversion from RGB is realized by means
of an initial transform to LMS cone space, followed by a conversion of the data
to logarithmic space (used to reduce skewness):(5)[LMS]=[0.38110.57830.04020.19670.72440.07820.02410.12880.8444][RGB],(6)Llog⁡LM=log⁡MSlog⁡S.Finally, thelαβ data is
obtained from(7)[lαβ]=[1/30001/60001/2][11111−21−10][LMS].This color space has successfully been used in
[17, 18] for image color transfer
operations, which will be described in Section
4.1.


## 2.4. Comparison between the Different Color Image Representations

The correlation
between two image channels is given by(8)ρij=E[(vi−E[vi])(vj−E[vj])]E[(vi−E[vi])2]E[(vj−E[vj])2],where vi and vj represent the ith and jth image channel
signals, respectively, (with i,j=1,3¯, i≠j), for a
certain color image representation.The total interchannel correlation is calculated as
follows:(9)Rtotal=|ρ12|+|ρ13|+|ρ23|3.The correlation coefficients have been measured for
several test images (see Figure1) in the discussed color image representations
using the above formulae [19]. Results are summarized in Table 1.Table 1
Decorrelation
results over the tested database.
Image representationCorrelation coefficients (mean/std)ρ12ρ13ρ23RtotalRGB0.93/0.060.84/0.120.95/0.070.91/0.07KLT0/00/00/00/0I1I2I30.29/0.220.25/0.210.52/0.260.35/0.15lαβ0.32/0.200.27/0.200.74/0.190.44/0.13Figure 1
General, mixed color face image database.It can be observed that theRGB representation
presents a very high interchannel correlation, while the I1I2I3 and lαβ image
representations significantly reduce this correlation. As stated above, the KLT, which is adapted to each particular image, achieves
total decorrelation of the image channels.


## 3. Face Segmentation Using Deformable Template Matching

Note that the termtexture, frequently used in
this paper, refers in the context of this work to the set of pixel intensities
across an object, also subsequent to a suitable normalization.


## 3.1. Statistical Shape Models

We are
interested in designing a shape model robust to head pose variations. The shape
is defined as the set of positions of some fiducial points on the face. The
model is statistically built from a training dataset which contains image
examples, annotated with a fixed set of landmark points. The sets of 2-D
coordinates of the landmark points define the shapes inside the image frame.
These shapes are aligned using the generalizedprocrustes analysis [20], a technique for
removing the differences in translation, rotation, and scale between the
training set of shapes. This defines the shapes in the normalized frame.LetN be the number
of training examples. Each shape example is represented as a vector s of concatenated
coordinates of its points (x1,x2,…,xL,y1,y2,…,yL)T, where L is the number
of landmark points. principal components analysis (PCA) is then applied
to the set of aligned shape vectors reducing the initial dimensionality of the
data. It can be noted that PCA is very similar
to KLT. In a geometric interpretation, KLT can be viewed
as a rotation of the coordinate system, while for PCA, the rotation of
the coordinate system is preceded by a shift of the origin to the mean point
[21]. Shape
variability is thus linearly modeled as a base (mean) shape plus a linear
combination of shape eigenvectors:(10)sm=s¯+Φsbs,where sm represents a
modeled shape, s¯ is the mean of
the aligned shapes, Φs=(ϕs1|ϕs2|⋯|ϕsp) is a matrix
having p shape
eigenvectors as its columns (p<N); finally, bs defines the set
of parameters of the shape model. p is chosen so
that a certain percentage of the total variance of the data is retained.The standard deviation for each parameter of the face
model, as resulted from the training dataset, provides its dynamic range. By
altering the model parameters within their dynamic range helps insuring that
only plausible instances of the modeled object are being generated. A
description of the way in which the optimal model parameters for a new image
can automatically be estimated follows in Section3.2.


## 3.2. Face Texture Template Optimization Algorithm

In order to optimize the face model parameters, a
texture template is also required. The separation between shape and texture is
realized using a reference shape. Based on this reference shape, the so-calledtexture
examples can be extracted. The reference shape is usually chosen as the
pointwise mean of the shape examples. The texture examples are defined in the
normalized frame of the reference shape. Each image example is then distorted
such that the points that define its attached shape, used as control points,
match the reference shape, such that the topology is preserved. An image warping
method is employed for this purpose. Image warping methods are discussed in
Section 3.3.Subsequent to the warping stage, all shape differences
between the image examples have been removed. The texture across each image
object is thus mapped into a shape-normalized representation. The resulting
images are also called the image examples in the normalized frame. For each of
these images, the corresponding pixel values across their common shape are
scanned to form the texture vectorstim=(tim1,tim2,…,timP)T, where P is the number
of texture samples.Based on previous experiments, we remark that the
variability of the shape component of the face is much more important than the
variability of the texture component in terms of a successful segmentation of
the face. Due to this fact, we consider in the following a simplified
formulation of a model-based face segmentation technique, where the modeled
image is represented by a fixed texture template; extensions could be made so
that to include texture variability, yet that was beyond the purpose of the
current work. Thus during an optimization stage (fitting the model to a query
image), the parameters to be found arep=(gsbs), where gs are the shape
2-D position, 2-D rotation, and scale parameters inside the image frame, and bs are the shape
model parameters. The optimization of the parameters p is realized by
minimizing the reconstruction error between the query image and the modeled
image. The error is evaluated in the coordinate frame of the model, that is, in the normalized texture reference frame, rather than in the coordinate frame
of the image. The difference between the query image and the modeled image is
thus given by the difference between the (normalized) image texture and the
(normalized) template texture as follows:(11)r(p)=ts−tt,and ∥r(p)∥2 is the
reconstruction error, with ∥⋅∥ marking the
Euclidean norm.A first order Taylor extension ofr(p) is given
by(12)r(p+δp)≃r(p)+∂r∂pδp.δp should be
chosen so that to minimize ∥r(p+δp)∥2. It follows that:(13)∂r∂pδp=−r(p).Normally, the gradient matrix ∂r/∂p should be
recomputed at each iteration. Yet, as the error is estimated in a normalized
texture frame, it was shown that this gradient matrix may be considered as
fixed, being thus possible to precompute it from a training dataset; these
techniques, introduced in [22],
and extended so that to also incorporate a statistical texture variation model
(as opposed to a fixed texture template described above), are called active
appearance models (AAMs). Using this technique, each parameter in p is
systematically displaced from its known optimal value retaining the normalized
texture differences. The resulted matrices are then averaged over several
displacement amounts and over several training images. The update direction of
the model parameters p is then given
by(14)δp=−Rr(p),where R=((∂r/∂p)T(∂r/∂p))−1(∂r/∂p)T is the
pseudoinverse of the  determined gradient matrix, which can be precomputed as
part of the training stage. The parameters p continue to be
updated iteratively until the error can no longer be reduced and convergence is
declared.


## 3.3. A TPS-Based Model Fitting Technique

Piecewise affine warping is extensively used in
techniques like AAM due to its reduced computational costs. A triangulation (e.g., Delauney) is used to partition the convex hull of the control points. The
points inside triangles are then mapped via an affine transformation which
uniquely assigns the corners of a triangle to their new positions. Although the
assumption that the face patches are piecewise affine within the triangles is a
satisfactory solution when there is a sufficiently large number of landmark
points, it also shows an important drawback. This refers to the fact that, when
modeling large face pose variations, corners of some triangles tend to get
reversed due to occlusions of the corresponding landmark points. This obviously
affects the image warping outcome by creating erroneous face patches. The
errors are further propagated into the fitting algorithm, resulting in an
incorrect fit. That is why the piecewise warping method works well mostly for
modeling frontal or nearly frontal faces.A more advanced and accurate warping method is
obtained by employing thethin plate splines (TPSs), introduced in
[23]. A short
description of this warping method is also given in the appendix. An initial
drawback of using the thin plate splines was represented by the fact that they
were quite expensive to calculate. The solution requires the inversion of a p×p matrix (the
bending energy matrix) which has a computational complexity of O(N3), where p is the number
of points in the dataset (i.e.,  the number of
pixels in the image); furthermore, the evaluation process is O(N2). Fortunately, important progress has been made in
order to speed this process up. An approximation approach was proved in
[24] to be very
efficient in dealing with the first problem, reducing greatly the computational
burden. As far as the evaluation process is concerned, the multilevel fast
multipole method (MLFMM) framework was described in [25] for the evaluation of
two-dimensional polyharmonic splines, while in [26] this work was extended for
the specific case of TPS, showing that a reduction of the computational
complexity from O(N2) to O(N⁢log⁡⁢N) is indeed
possible. Thus the computational difficulties involving the use of TPS have been to an
important extent removed.We show in Figures2 and 3 an example of fitting
the model based on TPS warping. The
error is evaluated relative to the number of available data points after the deformation.Face modeling. (a) Original texture template; (b) model initialization; and (c) converged model.
(a)(b)(c)Figure 3
TPS-based convergence—TPS-deformed grid for the rectangular region around the fitted shape.


## 4. Improved Model Fitting by Means of Local Color Transfer

A face detection algorithm is firstly applied for the
current image. We used here the  Viola-Jones face detector [27], which is based on the
AdaBoost algorithm [28]. A statistical relation between the face detector
estimates for the face position and size (rectangle region) and the position
and size of the reference shape inside the image frame is initially learnt
(offline) from a set of training images. This relation is then used to obtain a
more accurate initialization for the reference shape, tuned with the employed
face detection algorithm. It is also important to have a reasonably close
initialization to the real values in order to insure the convergence of the
fitting algorithm described in Section 3. Color statistics are then extracted
across the convex hull of landmark points of the initialized reference shape.


## 4.1. Image Color Transfer

According to [17], color can be transferred between two images (global
color transfer) using the formula in (15), applied in the lαβ color
space:(15)g(ct)=μs+σsσt(ct−μt),where μ and σ are,
respectively, the mean and standard deviation of the Gaussian distribution in
the considered color space.For local color transfer between two images, color
statistics (e.g., mean and variance of the Gaussian-modeled color
distribution) are gathered from the target and source image, respectively, and
used to calculate thecolor influence map (CIM). CIM contains the
weights for each pixel in the target image, determined based on their proximity
to the color range in the source image.Consider the distance between a pixel and the center
of the color distribution. For three-dimensional color data this is the
Mahalanobis distance given by(16)dM(c,μ)=(c−μ)S−1(c−μ),where S is the
covariance matrix of the three-variate color texture vector.Yet, if a decorrelated color space is used, then the
covariance matrixS is close to
being diagonal and (16) reduces to the normalized Euclidean distance (17):(17)d(c,μ)=∥c−μσ∥2,where σ is the standard
deviation vector of c over the sample
set.The weights in CIM are calculated using a function of
the above distancef(d), for which the following conditions should be met as
follows:(18)lim⁡d→∞f(d)=0;f(0)=1.The function below was proposed in [18] to be used with the lαβ color
space:(19)f(d)=e−3d2.The color transfer equation in (15) was also extended
in [18]
to(20)g(ct)=ct+f(d)(μs+σsσt(ct−μt)−ct),or, if a single color c is used as
source for color transfer,(21)g(ct)=ct+f(d)(c−μt).


## 4.2. Adaptive Texture Template Matching

Using a
decorrelated color space (see Section2), the color of the texture template
(see Figure 4(a)) can be adapted to the current image, increasing the chance of
a correct fitting (correct-face segmentation) of the face model. Experimental
results to support this premise and to confirm the benefits of employing color
adaptation techniques with the template matching algorithm follow next.Face modeling using color transfer. (a) Generating the color-adapted
template from the original template and the estimated face region in the image;
(b) converged model with no color transfer; and (c) converged
model using the color-adapted template.
(a)(b)(c)


## 5. Experiments

The
experiments have been performed on a randomly chosen subset of 16 images from
the database in Figure1. The images have been semiautomatically annotated and
the set of annotations has been used as the ground truth for calculating the
boundary errors, which give an objective measure for the fitting quality of the
face model. The boundary errors are measured between the exact shape in the
image frame (obtained from the ground truth annotations) and the optimized
model shape in the image frame. The boundary error is calculated as the point-to-point (Pt-Pt) error, which is given by the Euclidian distance between the two
shape vectors of concatenated x and y coordinates of the landmark
points. The mean and standard deviation of Pt-Pt errors is used to
evaluate the boundary errors over a whole set of images. The results are
summarized in Table 2.Table 2
Convergence
results.
Color transfer implementationPt-Pt errors (mean/std)Before fittingAfter fittingNone (on grayscale)21.00 / 2.7316.10/13.76None (on color)14.78/11.48I1I2I37.91/4.46lαβ9.86/5.33An implementation based only on the intensity (gray
scale) component has also been tested. The gray scale images have been obtained
by applying the standard mix ofRGB components in (22):(22)Grayscale=0.30R+0.59G+0.11B.The initial results (no color adaptation) show a
slight gain in the fitting accuracy over the gray scale implementation when
color information is added. However, significant increase in face segmentation
accuracy can be observed when adapting the color of the texture template using
color transfer techniques. It can also be noted that the implementation based
onI1I2I3 color space
performs slightly better in terms of segmentation accuracy, although
subjectively better color adaptation results have been observed when using the lαβ color space.
This can be explained by the fact that the I1I2I3 color space
representation is more suitable to be used together with the fitting algorithm
which is implemented in the RGB color space.The robustness to changes in the illumination
conditions was also tested using the Oulu face image database [29]. An example of color adaptation
of the texture template for this database is shown in Figure 5.Example of
color adaptation of the texture template under 16 different camera calibration
settings and illumination conditions fromOulu database (all 4×4 combinations of horizon, incandescent, fluorescent, and daylight illuminants). (a)  Initial images; (b) color-adapted
templates.
(a)(b)


## 6. Discussion and Conclusions

We analyzed in
this paper the possibility of enhancing a face segmentation/tracking method
based on texture template matching by means of color image alignment. We also
presented a model parameters optimization approach which minimized the error
between the texture template and the warped image texture across the current
shape. We employed here the TPS-based warping method which is more robust for
head pose variations.The color alignment techniques make use of the
decorrelated color statistics of the current image and template image.
Improvements of the accuracy of the segmentation have been demonstrated.From our experiments, we can conclude that the
color-adaptation method for the texture template can also be useful in face
tracking applications which employ face modeling techniques similar to the one
described in Section3. In particular, it was shown significant improvements
and increased robustness for the case of tracking a face under changes in the
illumination conditions, like the change of the type of illuminant. This may be
a real change of the illuminant or it could be caused by some wrong white
balance setting of the image acquisition device.



---

*Source: 583687-2008-04-09.xml*