2012 International Workshop on Nonlinear Circuits,Communications and Signal Processing
NCSP'12, Honolulu, Hawaii, USA. March 4-6, 2012

Nonrigid Face Alignment Using Relative Motion PriorYang Yang† , Yuehu Liu† , Yuanqi Su† , Yoshifumi Nishio‡†The Institute of Artificial Intelligence and Robotics
Xi’an Jiaotong University, Xi’an 710049, China
Email: yyang@mail.xjtu.edu.cn‡The Department of Electrical and Electronic Engineering
Tokushima University, Tokushima 770-8506, Japan
Email: nishio@ee.tokushima-u.ac.jp

AbstractInput
We propose a novel method to automatically align face images using a single template. This is a challenge work since
the appearance variance caused by various expressions. The
core of our method is to utilize the relative motion prior forconstraining nonrigid deformation parameters. The proposed
Outputmethod is independent of the appearance model and available for unseen face images. Considering the computation
efficiency, our model fitting is solved in the inverse compositional framework. Experiments on face images and videos Figure 1: Illustration of the nonrigid face alignment problem.
demonstrate the effectiveness of the proposed method.
We take a single face image with neutral expression as thetemplate, to align other expression faces automatically.

1. Introduction
Face image alignment is an important topic in the computer
vision and image processing. It has various applications in thefeature detection, motion analysis and tracking. There are twomain categories of alignment algorithms. The first category
is based on the generative model, like the Active Appearance
Model (AAM)[1][2], which achieves accurate fitting results
and high efficiency using the analysis-by-synthesis strategy.
However, as shown in the previous work[3], it falls short ofsuccessfully aligning unseen faces, since the model generalization capability is limited in training examples. Anothercategory is based on the template fitting[4][5]. Without theappearance model, template-based method is a kind of imageto-image face alignment. To capture the appearance variation,
this type of methods also needs multiple templates.
In this paper, we propose a novel face alignment method
using a single template. The goal is to fit the only neutral
expression appearance to other expression faces. Figure 1
presents the illustration of the nonrigid face alignment task.
The proposed method can also be used for the automaticallandmark labeling, which reduces the labor-intensive workfor preparing training examples and removes labeling inconsistencies among different labelers. Furthermore, it can beapplied to track face expressions in video sequences.
Same as other template-based methods, this single template alignment suffers from appearance variations across various expressions. Instead of handling the face appearance
model, we address the above problem by constraining geo-metric parameters. Motivated by the relative motion model
being successfully utilized for the expression synthesis[6] andfacial animation[7] work, we add the relative motion prior(RMP) to the face alignment algorithm. Since the RMP islearned for common face motion regulations, it has two benefits for the face alignment: 1). local fitting information will
be propagated to the entire face, which helps escaping fromthe local minima; 2). the fitting is more robust with the priorof warping parameters, since correct facial motion constraints
reduce the negative fitting caused by appearance variations.
Specifically, we tackle the optimization problem of facealignment by introducing a regularization term about theRMP. Taking the computational cost into account, we solve
the model fitting under the inverse compositional (IC)framework. The most time-consuming parameters are precomputed off-line. While in the on-line process, geometricparameters are updated efficiently. Comparison results tested
on different unseen faces demonstrate the proposed method
achieves better performances than the conventional alignment
method in term of the fitting accuracy.
The remainder of this paper is organized as follows. In
Section 2, we give a brief introduction of the image alignment problem and state the outline about model fitting: thestandard IC algorithm. Section 3 presents the proposed facealignment method and the details about its model fitting. InSection 4, the proposed method is tested by the comparison
experiments. Finally, conclusions are given in the last section.

- 257 -

-3 s.d.

2. Conventional Image Alignment and the IC Algorithm
The conventional image alignment problem is presented inthe Lucas-Kanade algorithm[4]. Let T and I denote a template image and a target image, respectively. The goal is tofind warp parameters p that minimize the cost function of theform∑∥T (x) − I(W(x; p))∥2 ,(1)E=x-1.5 s.d.

mean

1.5 s.d.

3 s.d.

1st

2nd
3rd

where W(x; p) is the warping function for the pixel coordinates x in the fitting area. W(x; p) is often chosen as thepiecewise affine warp to model nonrigid transformation. WeFigure 2: The effects of varying first three component paramapply the point distribution model (PDM) to present the faceeters in the relative motion model. We add the reconstructedshape:motion vector to a mean neutral face. Each component’s paS = S̄ + Qp,(2)
rameter is shown in a row with variation between ±3 s.d.
where S̄ and Q are the mean and basis of variation respectively.
The inverse compositional (IC) algorithm[4] is proposed to RMP can be estimated from the anisotropic Gaussians distrisolve this optimization problem efficiently. The key idea is to butions in the PCA model.
The optimization problem with the RMP is formulated tochange the role of T and I when updating △p:
minimize following energy function:∑∑E(△p) =∥T (W(x; △p)) − I(W(x; p))∥2 .
(3)
Ec (p) =∥T (x) − I(W(x; p))∥2 + λc R(p)T Σ−1r R(p),xx

And then the incremental warp W(x; △p) is inverted andcomposed with the current estimate to the warping function where the regulation term R(p) is the relative motion vector
calculated by the current face shape parameter p, by combiniteratively:
ing Eqs. (2), (5) and (6),W(x; p) ← W(x; p) ◦ W(x; △p)−1 .
(4)R(p) = Q−1(7)r (S̄ + Qp − S0 − S̄r ).
This change enables that the most time-consuming parame2). The variAnd Σr is the variance matrix diag(σ12 , σ22 , ..., σNters can be estimated off-line.
2ance σi equals to the ith eigenvalue. λc is a regularization
coefficient.
3. Nonrigid Face Alignment with the RMP
3.1. Proposed Model

3.2. Model fitting

The model fitting is to solve for optimal increments to paWe tackle the expression face alignment problem using arametersp. As an extension of the IC algorithm, we adopt thesingle template by introducing a regularization term about theinversewarping
technique to minimize Ec (p). The difference
RMP. In this section, we first give the RMP learning process.
isthat,ineachiteration,
the inverse incremental parameters
The relative motion vector is defined asshould be reflected back to the regularization term.
Sr = S − S0 ,(5)∑∥T (W(x; △p)) − I(W(x; p))∥2Ec (△p) =(8)xwhere S0 denotes the face with neutral expression. The rel∂p′∂p′
ative motion vector is modeled by the principal components+λc R(p +△p)T Σ−1R(p+△p),ranalysis (PCA) and projected into a low-dimensional space.
∂△p∂△pSr is parameterized by prwhere p′ denotes the updated warping parameters, such as
Sr = S̄r + Qr pr ,(6)
W(x; p′ ) = W(W(x; △p)−1 ; p).
(9)where Qr contains unit eigenvectors of the projection matrix
The function Ec (△p) is nonlinear with respect to △p,and S̄r is the mean value. Figure 2 shows the effect of varying
the first three component’s parameters respectively. Now the which can be linearized by taking the first order Taylor ex-- 258 -′size of face region is 126 × 126 pixels. To accommodate
global lighting variation, the intensity of face region is nor(10) malized in advance. In our experiments, the point distribution
xmodel and the relative motion model are both remained 95%
T −1
principle components and the regularization coefficient λc is+λc [R(p) + B△p] Σr [R(p) + B△p] .
set to 100 .
The Hessian matrix isSince good initial parameters are crucial to the success of∑face alignment, there have been many robust methods[2] forT
T −1
A A + λc B Σr B.
H=estimating the initialization. However, our main purpose is toxdiscuss the effect of combining the RMP. We simply locateHence, the solution of the above equation can be stated as:
the initial face shape using the detected eye corners to target()
images. For comparison purpose, same initial parameters are∑
employed in different alignment methods.
△p = H−1
AT (I(W(x; p) − T ) − λc BT Σ−1R(p).
r∂R ∂ppansion. Let A = ▽T ∂W∂p and B = ∂p ∂△p .
∑Ec (△p) ≈∥T + A△p − I(W(x; p))∥2x−1According to Eq. (7), we have ∂R∂p = Qr Q. The main remains to be described for this fitting problem is how to com∂p′pute ∂△pof B.
We set S′ as the reconstructed face shape from p′ according
to Eq. (2). Combining Eq. (9), it is computed as S′ = W(S0 −Q△p; p). Applying the chain rule,∂p′∂p′ ∂S′∂W(x; p)== −Q−1Q.
r∂△p∂x∂S′ ∂△p

(a)

(c)

(d)

(b)

(e)

(f)

(11)

We choose the piecewise affine warp for W(x; p). Giventwo corresponding triangles in the base mesh and target mesh,a set of affine transform parameters (ai , i = 1, ..., 6) of thiswarp can be computed in the closed form. For each pixel inthis base mesh, which is represented by coordinates (x, y)T ,its location after the piecewise affine warp is required as:
W(x; p) = (a1 · x + a2 · y + a3 , a4 · x + a5 · y + a6 )T .

Figure 3: Comparison of face alignment results. (a) The single template with neutral expression. (b) The test image with
the initial position and shape. (c) and (d) are the alignment results from the conventional alignment algorithm and our proposed method, respectively. More details on close-up viewsof the eye and mouth areas are shown in (e-f) for each method.

Therefore, ∂W(x;p)= (a1 , a4 )T and ∂W(x;p)= (a2 , a5 )T .
∂x
∂yTo compute the above derivative of warp function, theproblem is to decide which triangle do we use for each vertex,
since each vertex may connect several triangles. Our strategy
is to average the affine parameters of every triangle that sharesthe vertex. This way achieves a smooth warp.
The parameter B must be recomputed in each iteration,so we have analyzed the computational cost of the proposed
method. Let n and K denote the dimension of S and pr ,respectively, and N denote the number of image vectors.
The computational cost of the conventional IC algorithm isO(nN +n2 K). The extra computations in the per-iteration of
proposed method are the computing B: O(n2 K) and adding
B into the Hessian: O(n2 K). However, since (K, n ≪ N ),the extra computational cost is negligible.

Figure 3 shows some examples of alignment results. The
number of iterations for the model fitting is 10. We can seethat our method performs better than the conventional alignment introduced in Sec. 2. Notice the details shown in Fig.
3(e-f), there is a large appearance difference: the teeth, butwith the RMP, the deformed mouth shape is correctly estimated.
Tracking facial features from video sequences is anotherapplication of the proposed method. The first frame is often
supposed as the alignment template. Figure 4 shows some examples of alignment results. We can see the proposed method
achieves much better performance. For the smile expression,
the mouth shape is deformed with a natural looking. Forthe surprise expression, with the constraints of the RMP, one
point motion of the eyebrow is propagated to others, so that it
avoids some local minimum situations.
4. Experiments
Different from the above experiments, here we would like
We select 300 face images from 80 subjects in the Cohn- to quantitatively evaluate the alignment performance. To fit
Kanade Facial Expression Database[8]. 80 specific land- expression faces with a large appearance variance has more
marks are labeled around facial components. The template challenges. Therefore, we specially employ a testing database- 259 -(a)(b)

Figure 4: Alignment results for video sequences by the conventional alignment algorithm (a) and proposed method (b).
including 70 images with smile and surprise expressions.
Comparison results are demonstrated by computing the MSE
between fitting results and the ground truth. Mouth is a difficulty in the fitting process. We additionally give the evaluation results of the mouth alignment. The proposed method
gets better fitting accuracy than the conventional alignment
method on both entire faces and local mouth fittings (Fig. 5).

Acknowledgements: This work was supported by the Fundamental Research Funds for the Central Universities and by theNational Natural Science Foundation of China (90920008,91120009).
References[1] T. Cootes, G. Edwards, and C. Taylor, “Active Appearance Models,” IEEE Trans. Pattern Analysis and Machine Inteligence, vol. 23, no. 6, pp. 681-685, 2001.
[2] M. Zhou, L. Liang, J. Sun, and Y. Wang, “AAM based
Face Tracking with Temporal Matching and Face Segmentation,” IEEE Conf. on Computer Vison and Pattern
Recognition, pp. 701-708, 2010.
[3] R. Gross, I. Matthews, and S. Baker, “Generic vs. Person Specific Active Appearance Models,” Image and Vision Computing, vol. 23, no. 11, pp. 1080-1093, 2005.
[4] S.Baker and I. Matthews, “Lucas-Kanada 20 Years on:
A Unifying Framework,” Int’l J. Computer Vision, vol.
56, no. 3, pp. 221-255, 2004.

Figure 5: Alignment performance comparisons between theproposed method and the conventional method. We plot theCumulative Density Function of the MSE on both entire faces
and local mouth fittings.

5. Conclusions
In this paper, a nonrigid expression face alignment method
is proposed, using a single template of the target neutral face.
We employ the relative motion prior (RMP) as the constraintof warp parameters. The model fitting can be efficiently
solved in the inverse compositional framework. Comparedto the conventional alignment method, the proposed method
outperforms on the single template alignment.

[5] J. Zhu, L. Gool, and S. Hoi, “Unsupervised Face Alignment by Robust Nonrigid Mapping,” IEEE Conf. on
Computer Vision, pp. 1265-1272, 2009.
[6] Y. Du and X. Lin, “Emotional Facial Expression ModelBuilding,” Pattern Recognition Letters, vol. 24, no. 16,
pp. 2923-2934, 2003.
[7] Y. Yang, N. Zheng, et al., “Interactive Facial Sketch
Expression Generation Using Local Constraints,” IEEE
Conf. on Intelligent Computing and Intelligent Systems,
vol. 2, pp. 864-868, 2009.
[8] T. Kanade, J.F. Cohn, and Y. Tian, “ComprehensiveDatabase for Facial Expression Analysis,” Automatic
Face and Gesture Recognition, pp. 46-53, 2000.

- 260 -